Rework sampling and colorized logs (#140)

## Overview

This is a significant PR which has 3 very notable characteristics:

1. Omnitrace colorizes most of it's logging
2. Completely reworked the sampling 
  - Samples now record the current instruction pointers instead of strings
    - This _dramatically_ decreases the overhead of taking a sample
  - The collection of metrics during a sample are split out into another component, enabling that data collection to be disabled -- which decreases the sampling overhead even further
  - When both `OMNITRACE_SAMPLING_CPUTIME` and `OMNITRACE_SAMPLING_REALTIME` are ON:
    - `OMNITRACE_SAMPLING_CPUTIME_FREQ` and `OMNITRACE_SAMPLING_REALTIME_FREQ` can be used to individually control the sampling frequency 
  - `OMNITRACE_SAMPLING_CPUTIME_DELAY` and `OMNITRACE_SAMPLING_REALTIME_DELAY` can be used to individually control the delay time before starting
  - Now, omnitrace does not start a real-time sampler on the main thread unless `OMNITRACE_SAMPLING_REALTIME` is ON
    - In the future, an `OMNITRACE_SAMPLING_TIDS` (and real-time, cpu-time variants) configuration variable(s) will allow you to select which threads will be sampled
3. Files produced by `omnitrace` exe -- `available-instr.txt`, `instrumented-instr.txt`, etc. -- now no longer has `-instr` suffix and are placed in `instrumentation/` subfolder, i.e. `available-instr.txt` -> instrumentation/available.txt`
  - This helped de-clutter the output folder

Most of the other edits were reorganization (e.g. internal namespace changes), cleanup, and splitting up functionality.

## Bug Fixes

There is a bug fix with respect to the HSA callbacks which disabled sampling on child threads when an HSA API call was made

## Details

- created thread_info struct for mapping different thread IDs
- reorganized file structure significantly
- added categories.hpp, concepts.hpp
- moved around name trait definitions
- moved all omnitrace components into `omnitrace::component` namespace
  - there was a lot of inconsistency b/t using `tim::component` in some places and `omnitrace::component`
  - added macros like OMNITRACE_DECLARE_COMPONENT in lieu of TIMEMORY_DECLARE_COMPONENT
- OMNITRACE_CRITICAL_TRACE_NUM_THREADS -> OMNITRACE_THREAD_POOL_SIZE
- roctracer and critical_trace use same thread pool
- critical_trace functions do not lock anymore bc of thread-local TaskGroup
- added `component::local_category_region` to support using `component::category_region` without explicitly passing in name
- removed `component::omnitrace` (unused)
- migrated KokkosP and OMPT to use `component::local_category_region`
  - removed `component::user_region` as a result
- migrated omnitrace_{push,pop}_{trace,region}_hidden to use component::category_region
  - removed `component::functors` as a result
- migrated some ppdefs
- `api::omnitrace` -> `project::omnitrace`
- `api::(...)` -> `category::(...)`
- improved recording the execution time of threads
  - migrated this functionality out of pthread_create_gotcha and into thread_info
- moved mpi_gotcha, fork_gotcha, exit_gotcha, rcclp into omnitrace::component namespace
- split backtrace up into backtrace, backtrace_metrics, backtrace_timestamp components
- sampling.cpp handles setup and post-processing that was formerly in backtrace
- updated logging to use colors
- `OMNITRACE_COLORIZED_LOG` config variable
- updated docs on JSON output from timemory
- instrumentation info in instrumentation subfolder
- added testing for KokkosP entries
- added testing for ompt entries
- add_critical_trace function defined in critical_trace.hpp
- disable push_thread_state and pop_thread_state when thread state is Disabled or Completed
- add comp::page_rss to main bundle
- thread_data supports std::optional instead of std::unique_ptr
- thread_data supports tim::identity<T> to avoid unique_ptr or optional
- tracing::record_thread_start_time()
- tracing::push_timemory and tracing::pop_timemory are templated on CategoryT
- removed anonymous namespace from omnitrace::utility
- sampling backtrace stores instruction pointers instead of strings
- component::category_region updates
  - handle disabled thread state
  - handle finalized state
  - fewer debug messages
  - invoke thread_init()
  - invoke thread_init_sampling()
  - handle push/pop count based on category
  - push/pop count only modified when used
- component::cpu_freq
- components/ensure_storage.hpp
- reworked the pthread_create replacement function
- updated parallel-overhead example to report # of times locked
- OMNITRACE_MAX_UNWIND_DEPTH build option
- update timemory submodule

examples/CMakeLists.txt

@@ -2,6 +2,14 @@ cmake_minimum_required(VERSION 3.16 FATAL_ERROR)
 
 project(omnitrace-examples LANGUAGES C CXX)
 
+if("${CMAKE_BUILD_TYPE}" STREQUAL "")
+    set(CMAKE_BUILD_TYPE
+        "RelWithDebInfo"
+        CACHE STRING "Build type" FORCE)
+endif()
+
+string(TOUPPER "${CMAKE_BUILD_TYPE}" BUILD_TYPE)
+
 set(CMAKE_VISIBILITY_INLINES_HIDDEN OFF)
 set(CMAKE_CXX_VISIBILITY_PRESET "default")
 set(CMAKE_CXX_STANDARD 17)
@@ -9,6 +17,10 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON)
 set(CMAKE_CXX_CLANG_TIDY)
 set(CMAKE_INSTALL_DEFAULT_COMPONENT_NAME examples)
 
+if(OMNITRACE_BUILD_DEBUG)
+    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g -gdwarf-3 -fno-omit-frame-pointer")
+endif()
+
 option(BUILD_SHARED_LIBS "Build dynamic libraries" ON)
 
 if(CMAKE_PROJECT_NAME STREQUAL "omnitrace")

examples/lulesh/CMakeLists.txt

@@ -10,7 +10,7 @@ option(LULESH_BUILD_KOKKOS "Build Kokkos from submodule" ON)
 if(LULESH_BUILD_KOKKOS)
     add_subdirectory(external)
     if(LULESH_USE_CUDA)
-        kokkos_compilation(PROJECT)
+        kokkos_compilation(PROJECT COMPILER ${Kokkos_NVCC_WRAPPER})
     elseif(LULESH_USE_HIP AND NOT "${CMAKE_CXX_COMPILER}" MATCHES "hipcc")
         if(NOT HIPCC_EXECUTABLE)
             find_package(hip QUIET HINTS ${ROCmVersion_DIR} PATHS ${ROCmVersion_DIR})

examples/mpi/mpi.cpp

@@ -158,8 +158,8 @@ run(MPI_Comm _comm, int nitr)
     MPI_Comm_rank(_comm, &_rank);
     MPI_Comm_size(_comm, &_size);
 
-    printf("[%s][%i] running %i iterations on %i ranks...\n", _name.c_str(), _rank, nitr,
-           _size);
+    printf("[%s][%i][%s] running %i iterations on %i ranks... \n", _name.c_str(), _rank,
+           __FUNCTION__, nitr, _size);
 
     MPI_Barrier(_comm);
     for(int i = 0; i < nitr; ++i)
@@ -175,6 +175,9 @@ run(MPI_Comm _comm, int nitr)
         all2all<double, 6>(_rank, _comm);
     }
     MPI_Barrier(_comm);
+
+    printf("[%s][%i][%s] running %i iterations on %i ranks... Done\n", _name.c_str(),
+           _rank, __FUNCTION__, nitr, _size);
 }
 
 void
@@ -235,8 +238,6 @@ run_main(int argc, char** argv)
 
     print_info(MPI_COMM_WORLD, true, "MPI_COMM_WORLD");
 
-    printf("[%s]\n", _name.c_str());
-
     if(size > 1)
     {
         MPI_Comm dup;
@@ -313,6 +314,8 @@ run_main(int argc, char** argv)
 
         print_info(dup, false);
     }
+
+    printf("[%s][%i of %i] %s... Done", _name.c_str(), rank, size, __FUNCTION__);
 }
 
 int
@@ -325,13 +328,12 @@ main(int argc, char** argv)
     auto _prom = std::promise<void>{};
     auto _fut  = _prom.get_future();
 
-    std::thread _thr{ [&]() {
+    std::thread{ [&]() {
+        _prom.set_value_at_thread_exit();
         run_main(argc, argv);
-        _prom.set_value();
-    } };
+    } }.join();
 
     _fut.wait();
-    _thr.join();
 
     MPI_Finalize();
     return EXIT_SUCCESS;

examples/openmp/CMakeLists.txt

@@ -14,6 +14,9 @@ add_executable(openmp-lu ${CMAKE_CURRENT_SOURCE_DIR}/LU/lu.cpp
 if(CMAKE_CXX_COMPILER_ID MATCHES "Clang")
     find_package(OpenMP REQUIRED)
     target_link_libraries(openmp-common PUBLIC OpenMP::OpenMP_CXX)
+    set(OMNITRACE_OPENMP_USING_LIBOMP_LIBRARY
+        ON
+        CACHE INTERNAL "Used by omnitrace testing" FORCE)
 else()
     find_program(CLANGXX_EXECUTABLE NAMES clang++)
     find_library(
@@ -27,9 +30,15 @@ else()
         omnitrace_custom_compilation(COMPILER ${CLANGXX_EXECUTABLE} TARGET openmp-common)
         omnitrace_custom_compilation(COMPILER ${CLANGXX_EXECUTABLE} TARGET openmp-cg)
         omnitrace_custom_compilation(COMPILER ${CLANGXX_EXECUTABLE} TARGET openmp-lu)
+        set(OMNITRACE_OPENMP_USING_LIBOMP_LIBRARY
+            ON
+            CACHE INTERNAL "Used by omnitrace testing" FORCE)
     else()
         find_package(OpenMP REQUIRED)
         target_link_libraries(openmp-common PUBLIC OpenMP::OpenMP_CXX)
+        set(OMNITRACE_OPENMP_USING_LIBOMP_LIBRARY
+            OFF
+            CACHE INTERNAL "Used by omnitrace testing" FORCE)
     endif()
 endif()
 

examples/parallel-overhead/parallel-overhead.cpp

@@ -14,11 +14,13 @@
 
 #if USE_LOCKS > 0
 #    include <mutex>
-using auto_lock_t = std::unique_lock<std::mutex>;
-long       total  = 0;
+using auto_lock_t     = std::unique_lock<std::mutex>;
+long       total      = 0;
+long       lock_count = 0;
 std::mutex mtx{};
 #else
 std::atomic<long> total{ 0 };
+long              lock_count = 0;
 #endif
 
 long
@@ -52,6 +54,7 @@ run(size_t nitr, long n)
         auto        _v = fib(_get_n());
         auto_lock_t _lk{ mtx };
         total += _v;
+        ++lock_count;
     }
 #else
     long local = 0;
@@ -110,6 +113,7 @@ main(int argc, char** argv)
 
     printf("[%s] fibonacci(%li) x %lu = %li\n", _name.c_str(), nfib, nthread,
            static_cast<long>(total));
+    printf("[%s] number of mutex locks = %li\n", _name.c_str(), lock_count);
 
     return 0;
 }

examples/python/CMakeLists.txt

@@ -2,19 +2,22 @@ cmake_minimum_required(VERSION 3.16 FATAL_ERROR)
 
 project(omnitrace-python)
 
-set(PYTHON_FILES builtin.py external.py source.py noprofile.py)
+set(PYTHON_FILES builtin.py external.py source.py noprofile.py fill.py)
 
-if(OMNITRACE_INSTALL_EXAMPLES)
-    find_package(Python3 COMPONENTS Interpreter)
-    if(Python3_FOUND)
-        set(PYTHON_EXECUTABLE "${Python3_EXECUTABLE}")
-        foreach(_FILE ${PYTHON_FILES})
-            configure_file(${PROJECT_SOURCE_DIR}/${_FILE} ${PROJECT_BINARY_DIR}/${_FILE}
-                           @ONLY)
+find_package(Python3 COMPONENTS Interpreter)
+
+if(Python3_FOUND)
+    set(PYTHON_EXECUTABLE "${Python3_EXECUTABLE}")
+
+    foreach(_FILE ${PYTHON_FILES})
+        configure_file(${PROJECT_SOURCE_DIR}/${_FILE} ${PROJECT_BINARY_DIR}/${_FILE}
+                       @ONLY)
+
+        if(OMNITRACE_INSTALL_EXAMPLES)
             install(
                 PROGRAMS ${PROJECT_BINARY_DIR}/${_FILE}
                 DESTINATION bin
                 COMPONENT omnitrace-examples)
-        endforeach()
-    endif()
+        endif()
+    endforeach()
 endif()

examples/python/fill.py

@@ -0,0 +1,39 @@
+#!@PYTHON_EXECUTABLE@
+
+import os
+import sys
+import time
+import omnitrace
+from omnitrace.user import region as omni_user_region
+from omnitrace.profiler import config as omni_config
+
+_prefix = ""
+
+
+def loop(n):
+    pass
+
+
+@omnitrace.profile()
+def run(i, n, v):
+    for l in range(n * n):
+        loop(v + l)
+    return v + (n * n)
+
+
+if __name__ == "__main__":
+    import argparse
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-n", "--num-iterations", help="Number", type=int, default=100)
+    parser.add_argument("-v", "--value", help="Starting value", type=int, default=10)
+    args = parser.parse_args()
+
+    omni_config.include_args = True
+    _prefix = os.path.basename(__file__)
+    print(f"[{_prefix}] Executing {args.num_iterations} iterations...\n")
+    ans = 0
+    for i in range(args.num_iterations):
+        beg = ans
+        ans = run(i, args.value, beg)
+        print(f"[{_prefix}] [{i}] result of run({args.value}, {beg}) = {ans}")

examples/python/pyproject.toml

@@ -1,5 +0,0 @@
-
-[tool.black]
-line-length = 80
-target-version = ['py38']
-include = '\.py'