rocm-smi and KokkosTools support (#23)

* renamed omnitrace_thread_data to thread_data * initial implementation * Numerous fixes and updates - Updated timemory submodule - Updated perfetto submodule (pulls in fixes for TRACE_EVENT) - pthread_gotcha only after omnitrace_init_tooling - omnitrace banner - config settings for rocm-smi freq and devices - critical_trace::get_entries - OMNITRACE_BASIC_PRINT - rocm_smi perfetto category - redirect roctracer warnings for ROCm 4.5.0 - property specializations for rocm-smi components - units fixes data_tracker types - roctracer entries for pthread_create and start_thread - omnitrace-avail defaults to settings, not components - settings have conforming names - settings warn about duplicates - ptl named threads - decreased max freq for sampler SIGALRM - rocm-smi names thread - rocm-smi avoids call to hipGetDeviceCount - name roctracer activity callback threads - fixed binary rewrite test output names * Update lulesh example - supports non-UVM GPU * Lulesh tweaks + formatting * KokkosP + Mode + Roctracer sampling deadlock fix - kokkosp support - omnitrace_init_library - config::print_settings() - config::get_mode() - omnitrace::Mode - omnitrace-avail improvements (removes settings) - handle get_verbose() < 0 - disable dyninst InstrStackFrames by default - handle perf_event_paranoid > 1 by disabling PAPI - SIGALRM max freq to 5.0 - Name threads - rocm-smi handles get_use_perfetto() and get_use_timemory() - HSA_ENABLE_INTERRUPT=0 when roctracer + sampling (fixes deadlock) * Tests, API renaming, roctracer - disable renaming of thread 0 - verbprintf_bare - enable dyninst merge tramp - tweaked some omnitrace exe verbose levels - reworked roctracer::setup and roctracer::shutdown - rocm_smi::data::poll checks get_state() - omnitrace_trace_finalize -> omnitrace_finalize - omnitrace_trace_init -> omnitrace_init - omnitrace_trace_set_env -> omnitrace_set_env - omnitrace_trace_set_mpi -> omnitrace_set_mpi - sampling mode does not disable timemory - disable roctracer before shutting down rocm-smi - lulesh tests w/ and w/o kokkosp - lulesh tests for perfetto only - with --dynamic-callsites --traps --allow-overlapping - lulesh tests for timemory only - with --stdlib --dynamic-callsites --traps --allow-overlapping * Update timemory submodule - fix for TIMEMORY_PROPERTY_SPECIALIZATION * get_verbose() handling + timemory submodule update - Findroctracer.cmake uses find_package(hsakmt) * Stability fixes + rework roctracer + perfetto - reworked roctracer start up - critical_trace perfetto basic values - perfetto sampling category - sampler checks signals - peak_rss in sampling - pthread_gotcha::shutdown() - rocm_smi::device_count() - HSA_TOOLS_LIB is set - HSA_ENABLE_INTERRUPT in omnitrace exe - omnitrace exe verbosity level changes - Avoid instrumenting Impl ns in Kokkos - gpu::device_count prefers rocm_smi instead of hip - ptl blocks signals - fixed pthread_gotcha roctracer_data values - removed runtime-instrument-sampling tests - timemory submodule update * cmake formatting * timemory + roctracer updates - fix timemory issue with papi_common - fix timemory issue with units - define roctracer::is_setup() * Miscellaneous tweaks - Disable sampling during runtime instrument - Fixed warnings about dynamic callsites - Fixed backtrace output when timemory disabled - Test tweaks * cmake-format * omnitrace_target_compile_definitions * timemory submodule update * config, omnitrace, State, mpi_gotcha updates - use OMNITRACE_THROW instead of direct throw - is_attached() - is_binary_rewrite() - get_is_continuous_integration() - get_debug_init() - get_debug_finalize() - max_thread_bookmarks default to 1 - State::Init - app_thread oneTimeCode - runtime instrumentation uses waitpid - fixed init_names - include main in MPI runs - fixed sampling setup when disabled - reworked mpi_gotcha - disabled critical trace in transpose test * cmake-format * handle rocm_smi::device_count() exception * CI timeouts * Re-enable runtime-instrument + sampling [ROCm/rocprofiler-systems commit: 39f17ae8b8]
2022-02-08 17:42:17 -06:00
@@ -29,6 +29,12 @@ parse:
        RUN_ARGS: '*'
        ENVIRONMENT: '*'
        LABELS: '*'
+        PROPERTIES: '*'
+    omnitrace_target_compile_definitions:
+      kwargs:
+        PUBLIC: '*'
+        PRIVATE: '*'
+        INTERFACE: '*'
  override_spec: {}
  vartags: []
  proptags: []
@@ -9,6 +9,8 @@ on:
 env:
  BUILD_TYPE: Release
  ELFUTILS_DOWNLOAD_VERSION: 0.183
+  OMNITRACE_DEBUG_FINALIZE: ON
+  OMNITRACE_VERBOSE: 1

 jobs:
  ubuntu-focal:
@@ -22,6 +24,7 @@ jobs:
    - uses: actions/checkout@v2

    - name: Install Packages
+      timeout-minutes: 5
      run:
        sudo apt-get update &&
        sudo apt-get install -y build-essential m4 autoconf libtool python3-pip libtbb-dev libboost-{atomic,system,thread,date-time,filesystem,timer}-dev ${{ matrix.compiler }} ${{ matrix.mpi }} &&
@@ -36,6 +39,7 @@ jobs:
        echo "LD_LIBRARY_PATH=/opt/omnitrace/lib:${LD_LIBRARY_PATH}" >> $GITHUB_ENV

    - name: Configure CMake
+      timeout-minutes: 5
      run:
        cmake --version &&
        if [ -z "${{ matrix.mpi }}" ]; then USE_MPI=OFF; else USE_MPI=ON; fi &&
@@ -102,6 +106,7 @@ jobs:
    - uses: actions/checkout@v2

    - name: Install Packages
+      timeout-minutes: 5
      run:
        sudo apt-get update &&
        sudo apt-get install -y build-essential m4 autoconf libtool python3-pip ${{ matrix.compiler }} ${{ matrix.mpi }} &&
@@ -116,6 +121,7 @@ jobs:
        echo "LD_LIBRARY_PATH=/opt/omnitrace/lib:${LD_LIBRARY_PATH}" >> $GITHUB_ENV

    - name: Configure CMake
+      timeout-minutes: 5
      run:
        cmake --version &&
        if [ -z "${{ matrix.mpi }}" ]; then USE_MPI=OFF; else USE_MPI=ON; fi &&
@@ -181,6 +187,7 @@ jobs:
    - uses: actions/checkout@v2

    - name: Install Packages
+      timeout-minutes: 5
      run:
        sudo apt-get update &&
        sudo apt-get install -y build-essential m4 autoconf libtool python3-pip libboost-{atomic,system,thread,date-time,filesystem,timer}-dev libtbb-dev libiberty-dev ${{ matrix.compiler }} &&
@@ -196,6 +203,7 @@ jobs:
        echo "LD_LIBRARY_PATH=/opt/omnitrace/lib:/opt/dyninst/lib:/opt/elfutils/lib:${LD_LIBRARY_PATH}" >> $GITHUB_ENV

    - name: Install ElfUtils
+      timeout-minutes: 5
      run:
        pushd external &&
        wget https://sourceware.org/elfutils/ftp/${ELFUTILS_DOWNLOAD_VERSION}/elfutils-${ELFUTILS_DOWNLOAD_VERSION}.tar.bz2 &&
@@ -208,6 +216,7 @@ jobs:
        rm -rf elfutils*

    - name: Install Dyninst
+      timeout-minutes: 20
      run:
        cmake --version &&
        git submodule update --init external/dyninst &&
@@ -222,6 +231,7 @@ jobs:
        rm -rf build

    - name: Configure CMake
+      timeout-minutes: 5
      run:
        cmake --version &&
        cmake -B ${{ github.workspace }}/build
@@ -284,6 +294,7 @@ jobs:
    - uses: actions/checkout@v2

    - name: Install Packages
+      timeout-minutes: 5
      run:
        sudo apt-get update &&
        sudo apt-get install -y build-essential m4 autoconf libtool python3-pip ${{ matrix.compiler }} &&
@@ -299,6 +310,7 @@ jobs:
        echo "LD_LIBRARY_PATH=/opt/omnitrace/lib:/opt/dyninst/lib:${LD_LIBRARY_PATH}" >> $GITHUB_ENV

    - name: Install Dyninst
+      timeout-minutes: 20
      run:
        cmake --version &&
        git submodule update --init external/dyninst &&
@@ -318,6 +330,7 @@ jobs:
        rm -rf build

    - name: Configure CMake
+      timeout-minutes: 5
      run:
        cmake --version &&
        cmake -B ${{ github.workspace }}/build
@@ -382,6 +395,7 @@ jobs:
    - uses: actions/checkout@v2

    - name: Install Packages
+      timeout-minutes: 5
      run:
        echo '1' | sudo tee /proc/sys/kernel/perf_event_paranoid &&
        sudo apt-get update &&
@@ -389,7 +403,7 @@ jobs:
        sudo wget -q -O - https://repo.radeon.com/rocm/rocm.gpg.key | sudo apt-key add - &&
        echo "deb [arch=amd64] https://repo.radeon.com/rocm/apt/${{ matrix.rocm_version }}/ ubuntu main" | sudo tee /etc/apt/sources.list.d/rocm.list &&
        sudo apt-get update &&
-        sudo apt-get install -y build-essential m4 autoconf libtool python3-pip libboost-{atomic,system,thread,date-time,filesystem,timer}-dev libtbb-dev libiberty-dev ${{ matrix.compiler }} libnuma-dev rocm-dev rocm-utils roctracer-dev rocprofiler-dev hip-base hsa-amd-aqlprofile hsa-rocr-dev hsakmt-roct-dev ${{ matrix.mpi }} libpapi-dev &&
+        sudo apt-get install -y build-essential m4 autoconf libtool python3-pip libboost-{atomic,system,thread,date-time,filesystem,timer}-dev libtbb-dev libiberty-dev ${{ matrix.compiler }} libudev-dev libnuma-dev rocm-dev rocm-utils roctracer-dev rocprofiler-dev hip-base hsa-amd-aqlprofile hsa-rocr-dev hsakmt-roct-dev ${{ matrix.mpi }} libpapi-dev &&
        sudo python3 -m pip install --upgrade pip &&
        python3 -m pip install 'cmake==3.16.3'

@@ -402,6 +416,7 @@ jobs:
        echo "LD_LIBRARY_PATH=/opt/omnitrace/lib:/opt/dyninst/lib:/opt/elfutils/lib:${LD_LIBRARY_PATH}" >> $GITHUB_ENV

    - name: Install ElfUtils
+      timeout-minutes: 5
      run:
        pushd external &&
        wget https://sourceware.org/elfutils/ftp/${ELFUTILS_DOWNLOAD_VERSION}/elfutils-${ELFUTILS_DOWNLOAD_VERSION}.tar.bz2 &&
@@ -414,6 +429,7 @@ jobs:
        rm -rf elfutils*

    - name: Install Dyninst
+      timeout-minutes: 20
      run:
        cmake --version &&
        git submodule update --init external/dyninst &&
@@ -428,6 +444,7 @@ jobs:
        rm -rf build

    - name: Configure CMake
+      timeout-minutes: 5
      run:
        cmake --version &&
        cmake -B ${{ github.workspace }}/build
@@ -69,6 +69,9 @@ omnitrace_add_option(OMNITRACE_USE_MPI "Enable MPI support" OFF)
 omnitrace_add_option(OMNITRACE_USE_HIP "Enable HIP support" ON)
 omnitrace_add_option(OMNITRACE_USE_ROCTRACER "Enable roctracer support"
                     ${OMNITRACE_USE_HIP})
+omnitrace_add_option(
+    OMNITRACE_USE_ROCM_SMI "Enable rocm-smi support for power/temp/etc. sampling"
+    ${OMNITRACE_USE_HIP})
 omnitrace_add_option(OMNITRACE_USE_MPI_HEADERS
                     "Enable wrapping MPI functions w/o enabling MPI dependency" OFF)
 omnitrace_add_option(OMNITRACE_BUILD_DYNINST "Build dyninst from submodule" OFF)
@@ -139,7 +142,12 @@ if(OMNITRACE_BUILD_HIDDEN_VISIBILITY)
    set(CMAKE_VISIBILITY_INLINES_HIDDEN ON)
 endif()

+if(NOT DEFINED CMAKE_INTERPROCEDURAL_OPTIMIZATION)
+    set(CMAKE_INTERPROCEDURAL_OPTIMIZATION OFF)
+endif()
+
 if(OMNITRACE_BUILD_LTO)
+    omnitrace_save_variables(LTO VARIABLES CMAKE_INTERPROCEDURAL_OPTIMIZATION)
    set(CMAKE_INTERPROCEDURAL_OPTIMIZATION ON)
 endif()

@@ -153,10 +161,12 @@ set(library_sources
    ${CMAKE_CURRENT_LIST_DIR}/src/library.cpp
    ${CMAKE_CURRENT_LIST_DIR}/src/library/config.cpp
    ${CMAKE_CURRENT_LIST_DIR}/src/library/critical_trace.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/src/library/kokkosp.cpp
    ${CMAKE_CURRENT_LIST_DIR}/src/library/gpu.cpp
    ${CMAKE_CURRENT_LIST_DIR}/src/library/perfetto.cpp
    ${CMAKE_CURRENT_LIST_DIR}/src/library/ptl.cpp
    ${CMAKE_CURRENT_LIST_DIR}/src/library/sampling.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/src/library/state.cpp
    ${CMAKE_CURRENT_LIST_DIR}/src/library/thread_data.cpp
    ${CMAKE_CURRENT_LIST_DIR}/src/library/timemory.cpp
    ${CMAKE_CURRENT_LIST_DIR}/src/library/components/backtrace.cpp
@@ -185,6 +195,9 @@ set(library_headers
    ${CMAKE_CURRENT_LIST_DIR}/include/library/components/fork_gotcha.hpp
    ${CMAKE_CURRENT_LIST_DIR}/include/library/components/mpi_gotcha.hpp
    ${CMAKE_CURRENT_LIST_DIR}/include/library/components/omnitrace.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/include/library/components/rocm_smi.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/include/library/components/roctracer.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/include/library/components/roctracer_callbacks.hpp
    ${CMAKE_CURRENT_LIST_DIR}/include/library/components/pthread_gotcha.hpp
    ${perfetto_DIR}/sdk/perfetto.h)

@@ -193,11 +206,13 @@ add_library(omnitrace-library SHARED ${library_sources} ${library_headers})
 if(OMNITRACE_USE_ROCTRACER)
    target_sources(
        omnitrace-library
-        PRIVATE
-            ${CMAKE_CURRENT_LIST_DIR}/src/library/components/roctracer.cpp
-            ${CMAKE_CURRENT_LIST_DIR}/src/library/components/roctracer_callbacks.cpp
-            ${CMAKE_CURRENT_LIST_DIR}/include/library/components/roctracer.hpp
-            ${CMAKE_CURRENT_LIST_DIR}/include/library/components/roctracer_callbacks.hpp)
+        PRIVATE ${CMAKE_CURRENT_LIST_DIR}/src/library/components/roctracer.cpp
+                ${CMAKE_CURRENT_LIST_DIR}/src/library/components/roctracer_callbacks.cpp)
+endif()
+
+if(OMNITRACE_USE_ROCM_SMI)
+    target_sources(omnitrace-library
+                   PRIVATE ${CMAKE_CURRENT_LIST_DIR}/src/library/components/rocm_smi.cpp)
 endif()

 target_include_directories(omnitrace-library SYSTEM PRIVATE ${perfetto_DIR}/sdk)
@@ -214,6 +229,7 @@ target_link_libraries(
           $<BUILD_INTERFACE:omnitrace::omnitrace-compile-options>
           $<BUILD_INTERFACE:omnitrace::omnitrace-hip>
           $<BUILD_INTERFACE:omnitrace::omnitrace-roctracer>
+           $<BUILD_INTERFACE:omnitrace::omnitrace-rocm-smi>
           $<BUILD_INTERFACE:omnitrace::omnitrace-mpi>
           $<BUILD_INTERFACE:omnitrace::omnitrace-ptl>
           $<BUILD_INTERFACE:timemory::timemory-headers>
@@ -285,6 +301,11 @@ set_target_properties(
        "\$ORIGIN/../${CMAKE_INSTALL_LIBDIR}:\$ORIGIN/../${CMAKE_INSTALL_LIBDIR}/timemory/libunwind:\$ORIGIN/../${CMAKE_INSTALL_LIBDIR}/dyninst-tpls/lib"
    )

+if(CMAKE_BUILD_TYPE MATCHES "^(DEBUG|Debug)")
+    string(REPLACE " " ";" _FLAGS "${CMAKE_CXX_FLAGS_RELWITHDEBINFO}")
+    target_compile_options(omnitrace-exe PRIVATE ${_FLAGS})
+endif()
+
 install(
    TARGETS omnitrace-exe
    DESTINATION ${CMAKE_INSTALL_BINDIR}
@@ -320,6 +341,10 @@ install(
 #
 # ------------------------------------------------------------------------------#

+if(OMNITRACE_BUILD_LTO)
+    omnitrace_restore_variables(LTO VARIABLES CMAKE_INTERPROCEDURAL_OPTIMIZATION)
+endif()
+
 if(OMNITRACE_BUILD_TESTING)
    add_subdirectory(examples)
 endif()
@@ -457,6 +457,9 @@ endfunction()
 # ----------------------------------------------------------------------------------------#
 function(OMNITRACE_TARGET_COMPILE_DEFINITIONS _TARG _VIS)
    foreach(_DEF ${ARGN})
+        if(NOT "${_DEF}" MATCHES "[A-Za-z_]+=.*" AND "${_DEF}" MATCHES "^OMNITRACE_")
+            set(_DEF "${_DEF}=1")
+        endif()
        target_compile_definitions(${_TARG} ${_VIS} $<$<COMPILE_LANGUAGE:CXX>:${_DEF}>)
        if(CMAKE_CUDA_COMPILER_IS_NVIDIA)
            target_compile_definitions(${_TARG} ${_VIS}
@@ -42,7 +42,8 @@ find_program(OMNITRACE_CLANG_FORMAT_EXE NAMES clang-format-11 clang-format-mp-11

 if(OMNITRACE_CLANG_FORMAT_EXE)
    file(GLOB_RECURSE sources ${PROJECT_SOURCE_DIR}/src/*.cpp)
-    file(GLOB_RECURSE headers ${PROJECT_SOURCE_DIR}/include/*.hpp)
+    file(GLOB_RECURSE headers ${PROJECT_SOURCE_DIR}/include/*.hpp
+         ${PROJECT_SOURCE_DIR}/include/*.hpp.in)
    file(GLOB_RECURSE examples ${PROJECT_SOURCE_DIR}/examples/*.cpp
         ${PROJECT_SOURCE_DIR}/examples/*.hpp)
    file(GLOB_RECURSE external ${PROJECT_SOURCE_DIR}/examples/lulesh/external/*.cpp
@@ -9,7 +9,7 @@ include(FindPackageHandleStandardArgs)
 find_path(
    MPI_HEADERS_INCLUDE_DIR
    NAMES mpi.h
-    PATH_SUFFIXES include include/mpich include/openmpi mpich openmpi
+    PATH_SUFFIXES include/openmpi openmpi include include/mpich mpich
    HINTS ${MPI_ROOT_DIR}
    PATHS ${MPI_ROOT_DIR})

@@ -0,0 +1,70 @@
+# Distributed under the OSI-approved BSD 3-Clause License.  See accompanying file
+# Copyright.txt or https://cmake.org/licensing for details.
+
+include(FindPackageHandleStandardArgs)
+
+# ----------------------------------------------------------------------------------------#
+
+# set(_ROCM_PATHS $ENV{ROCM_HOME} /opt/rocm)
+
+# ----------------------------------------------------------------------------------------#
+
+find_path(
+    rocm-smi_ROOT_DIR
+    NAMES include/rocm_smi/rocm_smi.h
+    HINTS ${_ROCM_PATHS}
+    PATHS ${_ROCM_PATHS}
+    PATH_SUFFIXES rocm_smi)
+
+mark_as_advanced(rocm-smi_ROOT_DIR)
+
+# ----------------------------------------------------------------------------------------#
+
+find_path(
+    rocm-smi_INCLUDE_DIR
+    NAMES rocm_smi/rocm_smi.h
+    HINTS ${rocm-smi_ROOT_DIR} ${_ROCM_PATHS}
+    PATHS ${rocm-smi_ROOT_DIR} ${_ROCM_PATHS}
+    PATH_SUFFIXES rocm_smi/include rocm_smi)
+
+mark_as_advanced(rocm-smi_INCLUDE_DIR)
+
+# ----------------------------------------------------------------------------------------#
+
+find_library(
+    rocm-smi_LIBRARY
+    NAMES rocm_smi64 rocm_smi
+    HINTS ${rocm-smi_ROOT_DIR} ${_ROCM_PATHS}
+    PATHS ${rocm-smi_ROOT_DIR} ${_ROCM_PATHS}
+    PATH_SUFFIXES rocm_smi/lib rocm_smi/lib64 lib lib64)
+
+if(rocm-smi_LIBRARY)
+    get_filename_component(rocm-smi_LIBRARY_DIR "${rocm-smi_LIBRARY}" PATH CACHE)
+endif()
+
+mark_as_advanced(rocm-smi_LIBRARY)
+
+# ----------------------------------------------------------------------------------------#
+
+find_package_handle_standard_args(rocm-smi DEFAULT_MSG rocm-smi_ROOT_DIR
+                                  rocm-smi_INCLUDE_DIR rocm-smi_LIBRARY)
+
+# ------------------------------------------------------------------------------#
+
+if(rocm-smi_FOUND)
+    add_library(rocm-smi::rocm-smi INTERFACE IMPORTED)
+    add_library(rocm-smi::roctx INTERFACE IMPORTED)
+    set(rocm-smi_INCLUDE_DIRS ${rocm-smi_INCLUDE_DIR})
+    set(rocm-smi_LIBRARIES ${rocm-smi_LIBRARY})
+    set(rocm-smi_LIBRARY_DIRS ${rocm-smi_LIBRARY_DIR})
+
+    target_include_directories(rocm-smi::rocm-smi INTERFACE ${rocm-smi_INCLUDE_DIR})
+    target_link_libraries(rocm-smi::rocm-smi INTERFACE ${rocm-smi_LIBRARY})
+
+endif()
+
+# ------------------------------------------------------------------------------#
+
+unset(_ROCM_PATHS)
+
+# ------------------------------------------------------------------------------#
@@ -61,18 +61,25 @@ find_library(
    PATHS ${roctracer_ROOT_DIR} ${_ROCM_PATHS}
    PATH_SUFFIXES lib lib64)

-find_library(
-    roctracer_hsakmt_LIBRARY
-    NAMES hsakmt
-    HINTS ${roctracer_ROOT_DIR} ${_ROCM_PATHS}
-    PATHS ${roctracer_ROOT_DIR} ${_ROCM_PATHS}
-    PATH_SUFFIXES lib lib64)
+find_package(hsakmt)
+if(hsakmt_FOUND)
+    set(roctracer_hsakmt_LIBRARY
+        hsakmt::hsakmt
+        CACHE STRING "Imported hsakmt target")
+else()
+    find_library(
+        roctracer_hsakmt_LIBRARY
+        NAMES hsakmt
+        HINTS ${roctracer_ROOT_DIR} ${_ROCM_PATHS}
+        PATHS ${roctracer_ROOT_DIR} ${_ROCM_PATHS}
+        PATH_SUFFIXES lib lib64)
+endif()

 if(roctracer_LIBRARY)
    get_filename_component(roctracer_LIBRARY_DIR "${roctracer_LIBRARY}" PATH CACHE)
 endif()

-mark_as_advanced(roctracer_LIBRARY roctracer_roctx_LIBRARY)
+mark_as_advanced(roctracer_LIBRARY roctracer_roctx_LIBRARY roctracer_hsakmt_LIBRARY)

 # ----------------------------------------------------------------------------------------#

@@ -16,6 +16,8 @@ omnitrace_add_interface_library(
 omnitrace_add_interface_library(omnitrace-hip "Provides flags and libraries for HIP")
 omnitrace_add_interface_library(omnitrace-roctracer
                                "Provides flags and libraries for roctracer")
+omnitrace_add_interface_library(omnitrace-rocm-smi
+                                "Provides flags and libraries for rocm-smi")
 omnitrace_add_interface_library(omnitrace-mpi "Provides MPI or MPI headers")
 omnitrace_add_interface_library(omnitrace-ptl "Enables PTL support (tasking)")

@@ -60,7 +62,7 @@ endif()
 if(OMNITRACE_USE_HIP)
    list(APPEND CMAKE_PREFIX_PATH /opt/rocm)
    find_package(hip ${omnitrace_FIND_QUIETLY} REQUIRED)
-    target_compile_definitions(omnitrace-hip INTERFACE OMNITRACE_USE_HIP)
+    omnitrace_target_compile_definitions(omnitrace-hip INTERFACE OMNITRACE_USE_HIP)
    target_link_libraries(omnitrace-hip INTERFACE hip::host)
 endif()

@@ -73,12 +75,28 @@ endif()
 if(OMNITRACE_USE_ROCTRACER)
    list(APPEND CMAKE_PREFIX_PATH /opt/rocm)
    find_package(roctracer ${omnitrace_FIND_QUIETLY} REQUIRED)
-    target_compile_definitions(omnitrace-roctracer INTERFACE OMNITRACE_USE_ROCTRACER)
+    omnitrace_target_compile_definitions(omnitrace-roctracer
+                                         INTERFACE OMNITRACE_USE_ROCTRACER)
    target_link_libraries(omnitrace-roctracer INTERFACE roctracer::roctracer
                                                        omnitrace::omnitrace-hip)
    set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH}:${roctracer_LIBRARY_DIRS}")
 endif()

+# ----------------------------------------------------------------------------------------#
+#
+# rocm-smmi
+#
+# ----------------------------------------------------------------------------------------#
+
+if(OMNITRACE_USE_ROCM_SMI)
+    list(APPEND CMAKE_PREFIX_PATH /opt/rocm)
+    find_package(rocm-smi ${omnitrace_FIND_QUIETLY} REQUIRED)
+    omnitrace_target_compile_definitions(omnitrace-rocm-smi
+                                         INTERFACE OMNITRACE_USE_ROCM_SMI)
+    target_link_libraries(omnitrace-rocm-smi INTERFACE rocm-smi::rocm-smi)
+    set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH}:${rocm-smi_LIBRARY_DIRS}")
+endif()
+
 # ----------------------------------------------------------------------------------------#
 #
 # MPI
@@ -90,8 +108,8 @@ if(OMNITRACE_USE_MPI)
    target_link_libraries(omnitrace-mpi INTERFACE MPI::MPI_C MPI::MPI_CXX)
 elseif(OMNITRACE_USE_MPI_HEADERS)
    find_package(MPI-Headers ${omnitrace_FIND_QUIETLY} REQUIRED)
-    target_compile_definitions(omnitrace-mpi INTERFACE TIMEMORY_USE_MPI_HEADERS
-                                                       OMNITRACE_USE_MPI_HEADERS)
+    omnitrace_target_compile_definitions(
+        omnitrace-mpi INTERFACE TIMEMORY_USE_MPI_HEADERS OMNITRACE_USE_MPI_HEADERS)
    target_link_libraries(omnitrace-mpi INTERFACE MPI::MPI_HEADERS)
 endif()

@@ -143,7 +161,7 @@ if(OMNITRACE_BUILD_DYNINST)
        )

    if(OMNITRACE_DYNINST_API_RT)
-        target_compile_definitions(
+        omnitrace_target_compile_definitions(
            omnitrace-dyninst
            INTERFACE
                DYNINST_API_RT="${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}:$<TARGET_FILE_DIR:Dyninst::dyninstAPI_RT>:${CMAKE_INSTALL_PREFIX}/lib/$<TARGET_FILE_NAME:Dyninst::dyninstAPI_RT>:$<TARGET_FILE:Dyninst::dyninstAPI_RT>"
@@ -163,7 +181,7 @@ else()
            PATH_SUFFIXES lib)

        if(OMNITRACE_DYNINST_API_RT)
-            target_compile_definitions(
+            omnitrace_target_compile_definitions(
                omnitrace-dyninst INTERFACE DYNINST_API_RT="${OMNITRACE_DYNINST_API_RT}")
        endif()

@@ -214,7 +232,7 @@ else()
        endif()

        if(OMNITRACE_DYNINST_API_RT)
-            target_compile_definitions(
+            omnitrace_target_compile_definitions(
                omnitrace-dyninst INTERFACE DYNINST_API_RT="${OMNITRACE_DYNINST_API_RT}")
        endif()

@@ -253,7 +271,8 @@ else()
        target_include_directories(
            omnitrace-dyninst SYSTEM INTERFACE ${TBB_INCLUDE_DIR} ${Boost_INCLUDE_DIRS}
                                               ${DYNINST_HEADER_DIR})
-        target_compile_definitions(omnitrace-dyninst INTERFACE omnitrace_USE_DYNINST)
+        omnitrace_target_compile_definitions(omnitrace-dyninst
+                                             INTERFACE OMNITRACE_USE_DYNINST)
    endif()
 endif()

@@ -4,7 +4,16 @@ project(lulesh LANGUAGES C CXX)

 list(INSERT CMAKE_MODULE_PATH 0 ${PROJECT_SOURCE_DIR}/cmake/Modules)

-add_subdirectory(external)
+option(LULESH_BUILD_KOKKOS "Build Kokkos from submodule" ON)
+if(LULESH_BUILD_KOKKOS)
+    add_subdirectory(external)
+    if(LULESH_USE_CUDA OR LULESH_USE_HIP)
+        kokkos_compilation(PROJECT)
+    endif()
+else()
+    find_package(Kokkos REQUIRED COMPONENTS separable_compilation)
+    kokkos_compilation(PROJECT)
+endif()

 set(CMAKE_CXX_EXTENSIONS OFF)

@@ -14,7 +23,9 @@ if("${CMAKE_BUILD_TYPE}" STREQUAL "")
        CACHE STRING "CMake build type" FORCE)
 endif()

-if(DEFINED OMNITRACE_USE_MPI)
+if(DEFINED OMNITRACE_USE_MPI
+   AND NOT LULESH_USE_CUDA
+   AND NOT LULESH_USE_HIP)
    option(LULESH_USE_MPI "Enable MPI" ${OMNITRACE_USE_MPI})
 else()
    option(LULESH_USE_MPI "Enable MPI" OFF)
@@ -1,22 +1,27 @@
+# include(Utilities)
+
+option(LULESH_USE_CUDA "Enable Kokkos CUDA backend for lulesh" OFF)
+option(LULESH_USE_HIP "Enable Kokkos HIP backend for lulesh" OFF)
+
 set(Kokkos_ENABLE_SERIAL
    ON
    CACHE BOOL "Enable Serial")
-set(Kokkos_ENABLE_OPENMP
-    ON
-    CACHE BOOL "Enable OpenMP")
-if(USE_CUDA)
+
+if(LULESH_USE_CUDA)
    set(Kokkos_ENABLE_CUDA
        ON
-        CACHE BOOL "Enable CUDA")
-    set(Kokkos_ENABLE_CUDA_UVM
-        ON
-        CACHE BOOL "Enable CUDA UVM")
+        CACHE BOOL "Enable CUDA" FORCE)
    set(Kokkos_ENABLE_CUDA_LAMBDA
        ON
-        CACHE BOOL "Enable CUDA UVM")
-    set(Kokkos_ENABLE_CUDA_CONSTEXPR
+        CACHE BOOL "Enable CUDA lambda support" FORCE)
+elseif(LULESH_USE_HIP)
+    set(Kokkos_ENABLE_HIP
        ON
-        CACHE BOOL "Enable CUDA UVM")
+        CACHE BOOL "Enable HIP" FORCE)
+else()
+    set(Kokkos_ENABLE_OPENMP
+        ON
+        CACHE BOOL "Enable OpenMP")
 endif()

 checkout_git_submodule(
@@ -9,8 +9,8 @@
 /* Comm Routines */

 #    define ALLOW_UNPACKED_PLANE false
-#    define ALLOW_UNPACKED_ROW false
-#    define ALLOW_UNPACKED_COL false
+#    define ALLOW_UNPACKED_ROW   false
+#    define ALLOW_UNPACKED_COL   false

 /*
   There are coherence issues for packing and unpacking message
@@ -59,8 +59,7 @@ void
 CommRecv(Domain& domain, int msgType, Index_t xferFields, Index_t dx, Index_t dy,
         Index_t dz, bool doRecv, bool planeOnly)
 {
-    if(domain.numRanks() == 1)
-        return;
+    if(domain.numRanks() == 1) return;

    /* post recieve buffers for all incoming messages */
    int          myRank;
@@ -366,8 +365,7 @@ void
 CommSend(Domain& domain, int msgType, Index_t xferFields, Domain_member* fieldData,
         Index_t dx, Index_t dy, Index_t dz, bool doSend, bool planeOnly)
 {
-    if(domain.numRanks() == 1)
-        return;
+    if(domain.numRanks() == 1) return;

    /* post recieve buffers for all incoming messages */
    int          myRank;
@@ -926,8 +924,7 @@ CommSend(Domain& domain, int msgType, Index_t xferFields, Domain_member* fieldDa
 void
 CommSBN(Domain& domain, int xferFields, Domain_member* fieldData)
 {
-    if(domain.numRanks() == 1)
-        return;
+    if(domain.numRanks() == 1) return;

    /* summation order should be from smallest value to largest */
    /* or we could try out kahan summation! */
@@ -1405,8 +1402,7 @@ CommSBN(Domain& domain, int xferFields, Domain_member* fieldData)
 void
 CommSyncPosVel(Domain& domain)
 {
-    if(domain.numRanks() == 1)
-        return;
+    if(domain.numRanks() == 1) return;

    int           myRank;
    bool          doRecv     = false;
@@ -1893,8 +1889,7 @@ CommSyncPosVel(Domain& domain)
 void
 CommMonoQ(Domain& domain)
 {
-    if(domain.numRanks() == 1)
-        return;
+    if(domain.numRanks() == 1) return;

    int           myRank;
    Index_t       xferFields = 3; /* delv_xi, delv_eta, delv_zeta */
@@ -2,9 +2,6 @@
 #if USE_MPI
 #    include <mpi.h>
 #endif
-#if _OPENMP
-#    include <omp.h>
-#endif
 #include "lulesh.h"
 #include <cstdlib>
 #include <limits.h>
@@ -13,12 +10,12 @@
 #include <string.h>

 static KOKKOS_INLINE_FUNCTION Real_t
-                              CalcElemVolume(const Real_t x0, const Real_t x1, const Real_t x2, const Real_t x3,
-                                             const Real_t x4, const Real_t x5, const Real_t x6, const Real_t x7,
-                                             const Real_t y0, const Real_t y1, const Real_t y2, const Real_t y3,
-                                             const Real_t y4, const Real_t y5, const Real_t y6, const Real_t y7,
-                                             const Real_t z0, const Real_t z1, const Real_t z2, const Real_t z3,
-                                             const Real_t z4, const Real_t z5, const Real_t z6, const Real_t z7)
+CalcElemVolume(const Real_t x0, const Real_t x1, const Real_t x2, const Real_t x3,
+               const Real_t x4, const Real_t x5, const Real_t x6, const Real_t x7,
+               const Real_t y0, const Real_t y1, const Real_t y2, const Real_t y3,
+               const Real_t y4, const Real_t y5, const Real_t y6, const Real_t y7,
+               const Real_t z0, const Real_t z1, const Real_t z2, const Real_t z3,
+               const Real_t z4, const Real_t z5, const Real_t z6, const Real_t z7)
 {
    Real_t twelveth = Real_t(1.0) / Real_t(12.0);

@@ -122,15 +119,15 @@ Domain::Domain(Int_t numRanks, Index_t colLoc, Index_t rowLoc, Index_t planeLoc,
 , m_dvovmax(Real_t(0.1))
 , m_refdens(Real_t(1.0))
 ,
-    //
-    // set pointers to (potentially) "new'd" arrays to null to
-    // simplify deallocation.
-    //
-    m_regNumList(0)
-, m_nodeElemStart(0)
-, m_nodeElemCornerList(0)
-, m_regElemSize(0)
-, m_regElemlist(0)
+//
+// set pointers to (potentially) "new'd" arrays to null to
+// simplify deallocation.
+//
+m_regNumList(0)
+//   m_nodeElemStart(0),
+//   m_nodeElemCornerList(0),
+// m_regElemSize(0),
+// m_regElemlist(0)
 #if USE_MPI
 , commDataSend(0)
 , commDataRecv(0)
@@ -171,48 +168,27 @@ Domain::Domain(Int_t numRanks, Index_t colLoc, Index_t rowLoc, Index_t planeLoc,
    SetupCommBuffers(edgeNodes);

    // Basic Field Initialization
-    for(Index_t i = 0; i < numElem(); ++i)
-    {
-        e(i)  = Real_t(0.0);
-        p(i)  = Real_t(0.0);
-        q(i)  = Real_t(0.0);
-        ss(i) = Real_t(0.0);
-    }
+    Kokkos::deep_copy(m_e, 0.0);
+    Kokkos::deep_copy(m_p, 0.0);
+    Kokkos::deep_copy(m_q, 0.0);
+    Kokkos::deep_copy(m_ss, 0.0);

    // Note - v initializes to 1.0, not 0.0!
-    for(Index_t i = 0; i < numElem(); ++i)
-    {
-        v(i) = Real_t(1.0);
-    }
+    Kokkos::deep_copy(m_v, 1.0);

-    for(Index_t i = 0; i < numNode(); ++i)
-    {
-        xd(i) = Real_t(0.0);
-        yd(i) = Real_t(0.0);
-        zd(i) = Real_t(0.0);
-    }
+    Kokkos::deep_copy(m_xd, 0.0);
+    Kokkos::deep_copy(m_yd, 0.0);
+    Kokkos::deep_copy(m_zd, 0.0);

-    for(Index_t i = 0; i < numNode(); ++i)
-    {
-        xdd(i) = Real_t(0.0);
-        ydd(i) = Real_t(0.0);
-        zdd(i) = Real_t(0.0);
-    }
+    Kokkos::deep_copy(m_xdd, 0.0);
+    Kokkos::deep_copy(m_ydd, 0.0);
+    Kokkos::deep_copy(m_zdd, 0.0);

-    for(Index_t i = 0; i < numNode(); ++i)
-    {
-        nodalMass(i) = Real_t(0.0);
-    }
+    Kokkos::deep_copy(m_nodalMass, 0.0);

    BuildMesh(nx, edgeNodes, edgeElems);

-#if _OPENMP
    SetupThreadSupportStructures();
-#else
-    // These arrays are not used if we're not threaded
-    m_nodeElemStart      = NULL;
-    m_nodeElemCornerList = NULL;
-#endif

    // Setup region index sets. For now, these are constant sized
    // throughout the run, but could be changed every cycle to
@@ -247,30 +223,46 @@ Domain::Domain(Int_t numRanks, Index_t colLoc, Index_t rowLoc, Index_t planeLoc,
    time()            = Real_t(0.);
    cycle()           = Int_t(0);

+    // With C++17 requirement we could just run this on the device
+    // without creating temporary host copies
+    auto h_nodelist  = Kokkos::create_mirror_view(m_nodelist);
+    auto h_x         = Kokkos::create_mirror_view(m_x);
+    auto h_y         = Kokkos::create_mirror_view(m_y);
+    auto h_z         = Kokkos::create_mirror_view(m_z);
+    auto h_volo      = Kokkos::create_mirror_view(m_volo);
+    auto h_elemMass  = Kokkos::create_mirror_view(m_elemMass);
+    auto h_nodalMass = Kokkos::create_mirror_view(m_nodalMass);
+    Kokkos::deep_copy(h_nodelist, m_nodelist);
+    Kokkos::deep_copy(h_x, m_x);
+    Kokkos::deep_copy(h_y, m_y);
+    Kokkos::deep_copy(h_z, m_z);
    // initialize field data
    for(Index_t i = 0; i < numElem(); ++i)
    {
-        Real_t   x_local[8], y_local[8], z_local[8];
-        Index_t* elemToNode = nodelist(i);
+        Real_t x_local[8], y_local[8], z_local[8];
        for(Index_t lnode = 0; lnode < 8; ++lnode)
        {
-            Index_t gnode  = elemToNode[lnode];
-            x_local[lnode] = x(gnode);
-            y_local[lnode] = y(gnode);
-            z_local[lnode] = z(gnode);
+            Index_t gnode  = h_nodelist(i, lnode);
+            x_local[lnode] = h_x(gnode);
+            y_local[lnode] = h_y(gnode);
+            z_local[lnode] = h_z(gnode);
        }

        // volume calculations
        Real_t volume = CalcElemVolume(x_local, y_local, z_local);
-        volo(i)       = volume;
-        elemMass(i)   = volume;
+        h_volo(i)     = volume;
+        h_elemMass(i) = volume;
        for(Index_t j = 0; j < 8; ++j)
        {
-            Index_t idx = elemToNode[j];
-            nodalMass(idx) += volume / Real_t(8.0);
+            Index_t idx = h_nodelist(i, j);
+            h_nodalMass(idx) += volume / Real_t(8.0);
        }
    }

+    Kokkos::deep_copy(m_volo, h_volo);
+    Kokkos::deep_copy(m_elemMass, h_elemMass);
+    Kokkos::deep_copy(m_nodalMass, h_nodalMass);
+
    // deposit initial energy
    // An energy of 3.948746e+7 is correct for a problem with
    // 45 zones along a side - we need to scale it
@@ -281,10 +273,11 @@ Domain::Domain(Int_t numRanks, Index_t colLoc, Index_t rowLoc, Index_t planeLoc,
    {
        // Dump into the first zone (which we know is in the corner)
        // of the domain that sits at the origin
-        e(0) = einit;
+        Kokkos::deep_copy(Kokkos::subview(m_e, 0), einit);
+        // e(0) = einit;
    }
    // set initial deltatime base on analytic CFL calculation
-    deltatime() = (Real_t(.5) * cbrt(volo(0))) / sqrt(Real_t(2.0) * einit);
+    deltatime() = (Real_t(.5) * cbrt(h_volo(0))) / sqrt(Real_t(2.0) * einit);

 }  // End constructor

@@ -315,6 +308,10 @@ Domain::BuildMesh(Int_t nx, Int_t edgeNodes, Int_t edgeElems)
 {
    Index_t meshEdgeElems = m_tp * nx;

+    auto h_x = Kokkos::create_mirror_view(m_x);
+    auto h_y = Kokkos::create_mirror_view(m_y);
+    auto h_z = Kokkos::create_mirror_view(m_z);
+
    // initialize nodal coordinates
    Index_t nidx = 0;
    Real_t  tz   = Real_t(1.125) * Real_t(m_planeLoc * nx) / Real_t(meshEdgeElems);
@@ -326,9 +323,9 @@ Domain::BuildMesh(Int_t nx, Int_t edgeNodes, Int_t edgeElems)
            Real_t tx = Real_t(1.125) * Real_t(m_colLoc * nx) / Real_t(meshEdgeElems);
            for(Index_t col = 0; col < edgeNodes; ++col)
            {
-                x(nidx) = tx;
-                y(nidx) = ty;
-                z(nidx) = tz;
+                h_x(nidx) = tx;
+                h_y(nidx) = ty;
+                h_z(nidx) = tz;
                ++nidx;
                // tx += ds ; // may accumulate roundoff...
                tx = Real_t(1.125) * Real_t(m_colLoc * nx + col + 1) /
@@ -341,6 +338,11 @@ Domain::BuildMesh(Int_t nx, Int_t edgeNodes, Int_t edgeElems)
        tz = Real_t(1.125) * Real_t(m_planeLoc * nx + plane + 1) / Real_t(meshEdgeElems);
    }

+    Kokkos::deep_copy(m_x, h_x);
+    Kokkos::deep_copy(m_y, h_y);
+    Kokkos::deep_copy(m_z, h_z);
+
+    auto h_nodelist = Kokkos::create_mirror_view(m_nodelist);
    // embed hexehedral elements in nodal point lattice
    Index_t zidx = 0;
    nidx         = 0;
@@ -350,15 +352,14 @@ Domain::BuildMesh(Int_t nx, Int_t edgeNodes, Int_t edgeElems)
        {
            for(Index_t col = 0; col < edgeElems; ++col)
            {
-                Index_t* localNode = nodelist(zidx);
-                localNode[0]       = nidx;
-                localNode[1]       = nidx + 1;
-                localNode[2]       = nidx + edgeNodes + 1;
-                localNode[3]       = nidx + edgeNodes;
-                localNode[4]       = nidx + edgeNodes * edgeNodes;
-                localNode[5]       = nidx + edgeNodes * edgeNodes + 1;
-                localNode[6]       = nidx + edgeNodes * edgeNodes + edgeNodes + 1;
-                localNode[7]       = nidx + edgeNodes * edgeNodes + edgeNodes;
+                h_nodelist(zidx, 0) = nidx;
+                h_nodelist(zidx, 1) = nidx + 1;
+                h_nodelist(zidx, 2) = nidx + edgeNodes + 1;
+                h_nodelist(zidx, 3) = nidx + edgeNodes;
+                h_nodelist(zidx, 4) = nidx + edgeNodes * edgeNodes;
+                h_nodelist(zidx, 5) = nidx + edgeNodes * edgeNodes + 1;
+                h_nodelist(zidx, 6) = nidx + edgeNodes * edgeNodes + edgeNodes + 1;
+                h_nodelist(zidx, 7) = nidx + edgeNodes * edgeNodes + edgeNodes;
                ++zidx;
                ++nidx;
            }
@@ -366,6 +367,7 @@ Domain::BuildMesh(Int_t nx, Int_t edgeNodes, Int_t edgeElems)
        }
        nidx += edgeNodes;
    }
+    Kokkos::deep_copy(m_nodelist, h_nodelist);
 }

 ////////////////////////////////////////////////////////////////////////////////
@@ -373,32 +375,31 @@ void
 Domain::SetupThreadSupportStructures()
 {
    // set up node-centered indexing of elements
-    Index_t* nodeElemCount = Allocate<Index_t>(numNode());
-
-    for(Index_t i = 0; i < numNode(); ++i)
-    {
-        nodeElemCount[i] = 0;
-    }
+    Kokkos::View<Index_t*, Kokkos::HostSpace> nodeElemCount("nodeElemCount", numNode());
+    auto h_nodelist = Kokkos::create_mirror_view(m_nodelist);
+    Kokkos::deep_copy(h_nodelist, m_nodelist);

    for(Index_t i = 0; i < numElem(); ++i)
    {
-        Index_t* nl = nodelist(i);
        for(Index_t j = 0; j < 8; ++j)
        {
-            ++(nodeElemCount[nl[j]]);
+            ++(nodeElemCount[h_nodelist(i, j)]);
        }
    }

-    m_nodeElemStart = Allocate<Index_t>(numNode() + 1);
+    m_nodeElemStart      = Kokkos::View<Index_t*>("m_nodeElemStart", numNode() + 1);
+    auto h_nodeElemStart = Kokkos::create_mirror_view(m_nodeElemStart);

-    m_nodeElemStart[0] = 0;
+    h_nodeElemStart[0] = 0;

    for(Index_t i = 1; i <= numNode(); ++i)
    {
-        m_nodeElemStart[i] = m_nodeElemStart[i - 1] + nodeElemCount[i - 1];
+        h_nodeElemStart[i] = h_nodeElemStart[i - 1] + nodeElemCount[i - 1];
    }

-    m_nodeElemCornerList = Allocate<Index_t>(m_nodeElemStart[numNode()]);
+    m_nodeElemCornerList =
+        Kokkos::View<Index_t*>("nodeElemCornerList", h_nodeElemStart[numNode()]);
+    auto h_nodeElemCornerList = Kokkos::create_mirror_view(m_nodeElemCornerList);

    for(Index_t i = 0; i < numNode(); ++i)
    {
@@ -407,21 +408,20 @@ Domain::SetupThreadSupportStructures()

    for(Index_t i = 0; i < numElem(); ++i)
    {
-        Index_t* nl = nodelist(i);
        for(Index_t j = 0; j < 8; ++j)
        {
-            Index_t m                    = nl[j];
+            Index_t m                    = h_nodelist(i, j);
            Index_t k                    = i * 8 + j;
-            Index_t offset               = m_nodeElemStart[m] + nodeElemCount[m];
-            m_nodeElemCornerList[offset] = k;
+            Index_t offset               = h_nodeElemStart[m] + nodeElemCount[m];
+            h_nodeElemCornerList[offset] = k;
            ++(nodeElemCount[m]);
        }
    }

-    Index_t clSize = m_nodeElemStart[numNode()];
+    Index_t clSize = h_nodeElemStart[numNode()];
    for(Index_t i = 0; i < clSize; ++i)
    {
-        Index_t clv = m_nodeElemCornerList[i];
+        Index_t clv = h_nodeElemCornerList[i];
        if((clv < 0) || (clv > numElem() * 8))
        {
            fprintf(
@@ -434,8 +434,8 @@ Domain::SetupThreadSupportStructures()
 #endif
        }
    }
-
-    Release<Index_t>(&nodeElemCount);
+    Kokkos::deep_copy(m_nodeElemCornerList, h_nodeElemCornerList);
+    Kokkos::deep_copy(m_nodeElemStart, h_nodeElemStart);
 }

 ////////////////////////////////////////////////////////////////////////////////
@@ -486,12 +486,9 @@ Domain::SetupCommBuffers(Int_t edgeNodes)
 #endif

    // Boundary nodesets
-    if(m_colLoc == 0)
-        m_symmX.resize(edgeNodes * edgeNodes);
-    if(m_rowLoc == 0)
-        m_symmY.resize(edgeNodes * edgeNodes);
-    if(m_planeLoc == 0)
-        m_symmZ.resize(edgeNodes * edgeNodes);
+    if(m_colLoc == 0) Kokkos::resize(m_symmX, edgeNodes * edgeNodes);
+    if(m_rowLoc == 0) Kokkos::resize(m_symmY, edgeNodes * edgeNodes);
+    if(m_planeLoc == 0) Kokkos::resize(m_symmZ, edgeNodes * edgeNodes);
 }

 ////////////////////////////////////////////////////////////////////////////////
@@ -506,9 +503,14 @@ Domain::CreateRegionIndexSets(Int_t nr, Int_t balance)
    srand(0);
    Index_t myRank = 0;
 #endif
-    this->numReg()    = nr;
-    m_regElemSize     = Allocate<Index_t>(numReg());
-    m_regElemlist     = Allocate<Index_t*>(numReg());
+    this->numReg()     = nr;
+    m_regElemSize      = Allocate<Index_t>(numReg());
+    auto row_map       = Kokkos::View<Index_t*>("regElemlist::row_map", numReg() + 1);
+    auto h_row_map     = Kokkos::create_mirror_view(row_map);
+    auto entries       = Kokkos::View<Index_t*>("regElemlist::entries", numElem());
+    m_regElemlist      = t_regElemlist(entries, row_map);
+    auto h_regElemlist = typename t_regElemlist::HostMirror(
+        Kokkos::create_mirror_view(m_regElemlist.entries), h_row_map);
    Index_t nextIndex = 0;
    // if we only have one region just fill it
    // Fill out the regNumList with material numbers, which are always
@@ -525,14 +527,14 @@ Domain::CreateRegionIndexSets(Int_t nr, Int_t balance)
    // If we have more than one region distribute the elements.
    else
    {
-        Int_t   regionNum;
-        Int_t   regionVar;
-        Int_t   lastReg = -1;
-        Int_t   binSize;
-        Index_t elements;
-        Index_t runto           = 0;
-        Int_t   costDenominator = 0;
-        Int_t*  regBinEnd       = Allocate<Int_t>(numReg());
+        Int_t                                   regionNum;
+        Int_t                                   regionVar;
+        Int_t                                   lastReg = -1;
+        Int_t                                   binSize;
+        Index_t                                 elements;
+        Index_t                                 runto           = 0;
+        Int_t                                   costDenominator = 0;
+        Kokkos::View<Int_t*, Kokkos::HostSpace> regBinEnd("regBinEnd", numReg());
        // Determine the relative weights of all the regions.  This is based off the -b
        // flag.  Balance is the value passed into b.
        for(Index_t i = 0; i < numReg(); ++i)
@@ -612,23 +614,28 @@ Domain::CreateRegionIndexSets(Int_t nr, Int_t balance)
    // Second, allocate each region index set
    for(Index_t i = 0; i < numReg(); ++i)
    {
-        m_regElemlist[i] = Allocate<Int_t>(regElemSize(i));
+        h_row_map(i + 1) = regElemSize(i);
        regElemSize(i)   = 0;
    }
    // Third, fill index sets
    for(Index_t i = 0; i < numElem(); ++i)
    {
-        Index_t r              = regNumList(i) - 1;  // region index == regnum-1
-        Index_t regndx         = regElemSize(r)++;   // Note increment
-        regElemlist(r, regndx) = i;
+        Index_t r      = regNumList(i) - 1;  // region index == regnum-1
+        Index_t regndx = regElemSize(r)++;   // Note increment
+        h_regElemlist.entries(h_row_map(r) + regndx) = i;
    }
+    Kokkos::deep_copy(m_regElemlist.entries, h_regElemlist.entries);
+    Kokkos::deep_copy(row_map, h_row_map);
 }

 /////////////////////////////////////////////////////////////
 void
 Domain::SetupSymmetryPlanes(Int_t edgeNodes)
 {
-    Index_t nidx = 0;
+    Index_t nidx    = 0;
+    auto    h_symmZ = Kokkos::create_mirror_view(m_symmZ);
+    auto    h_symmY = Kokkos::create_mirror_view(m_symmY);
+    auto    h_symmX = Kokkos::create_mirror_view(m_symmX);
    for(Index_t i = 0; i < edgeNodes; ++i)
    {
        Index_t planeInc = i * edgeNodes * edgeNodes;
@@ -637,55 +644,72 @@ Domain::SetupSymmetryPlanes(Int_t edgeNodes)
        {
            if(m_planeLoc == 0)
            {
-                m_symmZ[nidx] = rowInc + j;
+                h_symmZ[nidx] = rowInc + j;
            }
            if(m_rowLoc == 0)
            {
-                m_symmY[nidx] = planeInc + j;
+                h_symmY[nidx] = planeInc + j;
            }
            if(m_colLoc == 0)
            {
-                m_symmX[nidx] = planeInc + j * edgeNodes;
+                h_symmX[nidx] = planeInc + j * edgeNodes;
            }
            ++nidx;
        }
    }
+    Kokkos::deep_copy(m_symmZ, h_symmZ);
+    Kokkos::deep_copy(m_symmY, h_symmY);
+    Kokkos::deep_copy(m_symmX, h_symmX);
 }

 /////////////////////////////////////////////////////////////
 void
 Domain::SetupElementConnectivities(Int_t edgeElems)
 {
-    lxim(0) = 0;
+    // With C++17 we wouldn't need to do this and could run this on the GPU
+    // using class lambdas
+    auto h_lxim = Kokkos::create_mirror_view(m_lxim);
+    auto h_lxip = Kokkos::create_mirror_view(m_lxip);
+    h_lxim(0)   = 0;
    for(Index_t i = 1; i < numElem(); ++i)
    {
-        lxim(i)     = i - 1;
-        lxip(i - 1) = i;
+        h_lxim(i)     = i - 1;
+        h_lxip(i - 1) = i;
    }
-    lxip(numElem() - 1) = numElem() - 1;
+    h_lxip(numElem() - 1) = numElem() - 1;
+    Kokkos::deep_copy(m_lxim, h_lxim);
+    Kokkos::deep_copy(m_lxip, h_lxip);

+    auto h_letam = Kokkos::create_mirror_view(m_letam);
+    auto h_letap = Kokkos::create_mirror_view(m_letap);
    for(Index_t i = 0; i < edgeElems; ++i)
    {
-        letam(i)                         = i;
-        letap(numElem() - edgeElems + i) = numElem() - edgeElems + i;
+        h_letam(i)                         = i;
+        h_letap(numElem() - edgeElems + i) = numElem() - edgeElems + i;
    }
    for(Index_t i = edgeElems; i < numElem(); ++i)
    {
-        letam(i)             = i - edgeElems;
-        letap(i - edgeElems) = i;
+        h_letam(i)             = i - edgeElems;
+        h_letap(i - edgeElems) = i;
    }
+    Kokkos::deep_copy(m_letam, h_letam);
+    Kokkos::deep_copy(m_letap, h_letap);

+    auto h_lzetam = Kokkos::create_mirror_view(m_lzetam);
+    auto h_lzetap = Kokkos::create_mirror_view(m_lzetap);
    for(Index_t i = 0; i < edgeElems * edgeElems; ++i)
    {
-        lzetam(i) = i;
-        lzetap(numElem() - edgeElems * edgeElems + i) =
+        h_lzetam(i) = i;
+        h_lzetap(numElem() - edgeElems * edgeElems + i) =
            numElem() - edgeElems * edgeElems + i;
    }
    for(Index_t i = edgeElems * edgeElems; i < numElem(); ++i)
    {
-        lzetam(i)                         = i - edgeElems * edgeElems;
-        lzetap(i - edgeElems * edgeElems) = i;
+        h_lzetam(i)                         = i - edgeElems * edgeElems;
+        h_lzetap(i - edgeElems * edgeElems) = i;
    }
+    Kokkos::deep_copy(m_lzetam, h_lzetam);
+    Kokkos::deep_copy(m_lzetap, h_lzetap);
 }

 /////////////////////////////////////////////////////////////
@@ -693,11 +717,24 @@ void
 Domain::SetupBoundaryConditions(Int_t edgeElems)
 {
    Index_t ghostIdx[6];  // offsets to ghost locations
+    auto    h_elemBC = Kokkos::create_mirror_view(m_elemBC);
+    auto    h_lzetam = Kokkos::create_mirror_view(m_lzetam);
+    auto    h_lzetap = Kokkos::create_mirror_view(m_lzetap);
+    auto    h_letam  = Kokkos::create_mirror_view(m_letam);
+    auto    h_letap  = Kokkos::create_mirror_view(m_letap);
+    auto    h_lxim   = Kokkos::create_mirror_view(m_lxim);
+    auto    h_lxip   = Kokkos::create_mirror_view(m_lxip);
+    Kokkos::deep_copy(h_lzetam, m_lzetam);
+    Kokkos::deep_copy(h_lzetap, m_lzetap);
+    Kokkos::deep_copy(h_letam, m_letam);
+    Kokkos::deep_copy(h_letap, m_letap);
+    Kokkos::deep_copy(h_lxim, m_lxim);
+    Kokkos::deep_copy(h_lxip, m_lxip);

    // set up boundary condition information
    for(Index_t i = 0; i < numElem(); ++i)
    {
-        elemBC(i) = Int_t(0);
+        h_elemBC(i) = Int_t(0);
    }

    for(Index_t i = 0; i < 6; ++i)
@@ -750,67 +787,75 @@ Domain::SetupBoundaryConditions(Int_t edgeElems)
        {
            if(m_planeLoc == 0)
            {
-                elemBC(rowInc + j) |= ZETA_M_SYMM;
+                h_elemBC(rowInc + j) |= ZETA_M_SYMM;
            }
            else
            {
-                elemBC(rowInc + j) |= ZETA_M_COMM;
-                lzetam(rowInc + j) = ghostIdx[0] + rowInc + j;
+                h_elemBC(rowInc + j) |= ZETA_M_COMM;
+                h_lzetam(rowInc + j) = ghostIdx[0] + rowInc + j;
            }

            if(m_planeLoc == m_tp - 1)
            {
-                elemBC(rowInc + j + numElem() - edgeElems * edgeElems) |= ZETA_P_FREE;
+                h_elemBC(rowInc + j + numElem() - edgeElems * edgeElems) |= ZETA_P_FREE;
            }
            else
            {
-                elemBC(rowInc + j + numElem() - edgeElems * edgeElems) |= ZETA_P_COMM;
-                lzetap(rowInc + j + numElem() - edgeElems * edgeElems) =
+                h_elemBC(rowInc + j + numElem() - edgeElems * edgeElems) |= ZETA_P_COMM;
+                h_lzetap(rowInc + j + numElem() - edgeElems * edgeElems) =
                    ghostIdx[1] + rowInc + j;
            }

            if(m_rowLoc == 0)
            {
-                elemBC(planeInc + j) |= ETA_M_SYMM;
+                h_elemBC(planeInc + j) |= ETA_M_SYMM;
            }
            else
            {
-                elemBC(planeInc + j) |= ETA_M_COMM;
-                letam(planeInc + j) = ghostIdx[2] + rowInc + j;
+                h_elemBC(planeInc + j) |= ETA_M_COMM;
+                h_letam(planeInc + j) = ghostIdx[2] + rowInc + j;
            }

            if(m_rowLoc == m_tp - 1)
            {
-                elemBC(planeInc + j + edgeElems * edgeElems - edgeElems) |= ETA_P_FREE;
+                h_elemBC(planeInc + j + edgeElems * edgeElems - edgeElems) |= ETA_P_FREE;
            }
            else
            {
-                elemBC(planeInc + j + edgeElems * edgeElems - edgeElems) |= ETA_P_COMM;
-                letap(planeInc + j + edgeElems * edgeElems - edgeElems) =
+                h_elemBC(planeInc + j + edgeElems * edgeElems - edgeElems) |= ETA_P_COMM;
+                h_letap(planeInc + j + edgeElems * edgeElems - edgeElems) =
                    ghostIdx[3] + rowInc + j;
            }

            if(m_colLoc == 0)
            {
-                elemBC(planeInc + j * edgeElems) |= XI_M_SYMM;
+                h_elemBC(planeInc + j * edgeElems) |= XI_M_SYMM;
            }
            else
            {
-                elemBC(planeInc + j * edgeElems) |= XI_M_COMM;
-                lxim(planeInc + j * edgeElems) = ghostIdx[4] + rowInc + j;
+                h_elemBC(planeInc + j * edgeElems) |= XI_M_COMM;
+                h_lxim(planeInc + j * edgeElems) = ghostIdx[4] + rowInc + j;
            }

            if(m_colLoc == m_tp - 1)
            {
-                elemBC(planeInc + j * edgeElems + edgeElems - 1) |= XI_P_FREE;
+                h_elemBC(planeInc + j * edgeElems + edgeElems - 1) |= XI_P_FREE;
            }
            else
            {
-                elemBC(planeInc + j * edgeElems + edgeElems - 1) |= XI_P_COMM;
-                lxip(planeInc + j * edgeElems + edgeElems - 1) = ghostIdx[5] + rowInc + j;
+                h_elemBC(planeInc + j * edgeElems + edgeElems - 1) |= XI_P_COMM;
+                h_lxip(planeInc + j * edgeElems + edgeElems - 1) =
+                    ghostIdx[5] + rowInc + j;
            }
        }
    }
+    Kokkos::deep_copy(m_elemBC, h_elemBC);
+    Kokkos::deep_copy(m_lzetam, h_lzetam);
+    Kokkos::deep_copy(m_lzetap, h_lzetap);
+    Kokkos::deep_copy(m_letam, h_letam);
+    Kokkos::deep_copy(m_letap, h_letap);
+    Kokkos::deep_copy(m_lxim, h_lxim);
+    Kokkos::deep_copy(m_lxip, h_lxip);
 }

 ///////////////////////////////////////////////////////////////////////////
@@ -15,8 +15,7 @@ StrToInt(const char* token, int* retVal)
    char*       endptr;
    const int   decimal_base = 10;

-    if(token == NULL)
-        return 0;
+    if(token == NULL) return 0;

    c       = token;
    *retVal = (int) strtol(c, &endptr, decimal_base);
@@ -229,11 +228,13 @@ VerifyAndWriteFinalOutput(Real_t elapsed_time, Domain& locDom, Int_t nx, Int_t n
        ((elapsed_time * 1e6) / locDom.cycle()) / (nx * nx * nx * numRanks);

    Index_t ElemId = 0;
+    auto    h_e    = Kokkos::create_mirror_view(locDom.e_view());
+    Kokkos::deep_copy(h_e, locDom.e_view());
    printf("Run completed:  \n");
    printf("   Problem size        =  %i \n", nx);
    printf("   MPI tasks           =  %i \n", numRanks);
    printf("   Iteration count     =  %i \n", locDom.cycle());
-    printf("   Final Origin Energy = %12.6e \n", locDom.e(ElemId));
+    printf("   Final Origin Energy = %12.6e \n", h_e(ElemId));

    Real_t MaxAbsDiff   = Real_t(0.0);
    Real_t TotalAbsDiff = Real_t(0.0);
@@ -243,16 +244,14 @@ VerifyAndWriteFinalOutput(Real_t elapsed_time, Domain& locDom, Int_t nx, Int_t n
    {
        for(Index_t k = j + 1; k < nx; ++k)
        {
-            Real_t AbsDiff = FABS(locDom.e(j * nx + k) - locDom.e(k * nx + j));
+            Real_t AbsDiff = FABS(h_e(j * nx + k) - h_e(k * nx + j));
            TotalAbsDiff += AbsDiff;

-            if(MaxAbsDiff < AbsDiff)
-                MaxAbsDiff = AbsDiff;
+            if(MaxAbsDiff < AbsDiff) MaxAbsDiff = AbsDiff;

-            Real_t RelDiff = AbsDiff / locDom.e(k * nx + j);
+            Real_t RelDiff = AbsDiff / h_e(k * nx + j);

-            if(MaxRelDiff < RelDiff)
-                MaxRelDiff = RelDiff;
+            if(MaxRelDiff < RelDiff) MaxRelDiff = RelDiff;
        }
    }

@@ -403,8 +403,7 @@ static void
 LULESH_PMPIO_Close(void* file, void* udata)
 {
    DBfile* db = (DBfile*) file;
-    if(db)
-        DBClose(db);
+    if(db) DBClose(db);
 }
 #    endif

@@ -3,6 +3,7 @@ cmake_minimum_required(VERSION 3.15 FATAL_ERROR)
 project(omnitrace-transpose LANGUAGES CXX)

 find_program(HIPCC_EXECUTABLE NAMES hipcc)
+mark_as_advanced(HIPCC_EXECUTABLE)

 if(NOT HIPCC_EXECUTABLE)
    message(AUTHOR_WARNING "hipcc could not be found. Cannot build transpose target")
@@ -55,8 +55,6 @@ add_critical_trace(int64_t _tid, size_t _cpu_cid, size_t _gpu_cid, size_t _paren
                   int64_t _ts_beg, int64_t _ts_val, size_t _hash, uint16_t _depth,
                   uint16_t _prio = 0)
 {
-    if(!get_use_critical_trace()) return;
-
    // clang-format off
    // these are used to create unique type mutexes
    struct critical_insert {};
@@ -29,12 +29,25 @@
 // forward decl of the API
 extern "C"
 {
+    /// handles configuration logic
+    void omnitrace_init_library(void) TIMEMORY_VISIBILITY("default");
+
+    /// starts gotcha wrappers
+    void omnitrace_init(const char*, bool, const char*) TIMEMORY_VISIBILITY("default");
+
+    /// shuts down all tooling and generates output
+    void omnitrace_finalize(void) TIMEMORY_VISIBILITY("default");
+
+    /// sets an environment variable
+    void omnitrace_set_env(const char* env_name, const char* env_val)
+        TIMEMORY_VISIBILITY("default");
+
+    /// sets whether MPI should be used
+    void omnitrace_set_mpi(bool use, bool attached) TIMEMORY_VISIBILITY("default");
+
+    /// starts an instrumentation region
    void omnitrace_push_trace(const char* name) TIMEMORY_VISIBILITY("default");
+
+    /// stops an instrumentation region
    void omnitrace_pop_trace(const char* name) TIMEMORY_VISIBILITY("default");
-    void omnitrace_trace_init(const char*, bool, const char*)
-        TIMEMORY_VISIBILITY("default");
-    void omnitrace_trace_finalize(void) TIMEMORY_VISIBILITY("default");
-    void omnitrace_trace_set_env(const char* env_name, const char* env_val)
-        TIMEMORY_VISIBILITY("default");
-    void omnitrace_trace_set_mpi(bool use, bool attached) TIMEMORY_VISIBILITY("default");
 }
@@ -43,6 +43,7 @@

 TIMEMORY_DEFINE_NS_API(api, omnitrace)
 TIMEMORY_DEFINE_NS_API(api, sampling)
+TIMEMORY_DEFINE_NS_API(api, rocm_smi)

 namespace omnitrace
 {
@@ -91,6 +91,7 @@ struct backtrace
 private:
    int64_t           m_tid        = 0;
    int64_t           m_thr_cpu_ts = 0;
+    int64_t           m_mem_peak   = 0;
    size_t            m_size       = 0;
    time_point_type   m_ts         = {};
    data_t            m_data       = {};
@@ -28,6 +28,7 @@
 #include <timemory/components/macros.hpp>
 #include <timemory/components/user_bundle/types.hpp>
 #include <timemory/enum.h>
+#include <timemory/mpl/concepts.hpp>

 TIMEMORY_DECLARE_COMPONENT(roctracer)

@@ -46,9 +47,21 @@ struct backtrace_cpu_clock
 {};
 struct backtrace_fraction
 {};
+struct backtrace_gpu_busy
+{};
+struct backtrace_gpu_temp
+{};
+struct backtrace_gpu_power
+{};
+struct backtrace_gpu_memory
+{};
 using sampling_wall_clock = data_tracker<double, backtrace_wall_clock>;
 using sampling_cpu_clock  = data_tracker<double, backtrace_cpu_clock>;
 using sampling_percent    = data_tracker<double, backtrace_fraction>;
+using sampling_gpu_busy   = data_tracker<double, backtrace_gpu_busy>;
+using sampling_gpu_temp   = data_tracker<double, backtrace_gpu_temp>;
+using sampling_gpu_power  = data_tracker<double, backtrace_gpu_power>;
+using sampling_gpu_memory = data_tracker<double, backtrace_gpu_memory>;
 using roctracer           = tim::component::roctracer;
 }  // namespace component
 }  // namespace omnitrace
@@ -59,7 +72,7 @@ TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, component::roctracer, false_type)

 #if !defined(TIMEMORY_USE_LIBUNWIND)
 TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::api::sampling, false_type)
-TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::sampling::backtrace, false_type)
+TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::backtrace, false_type)
 TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_wall_clock,
                               false_type)
 TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_cpu_clock,
@@ -68,6 +81,17 @@ TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_perc
                               false_type)
 #endif

+#if !defined(TIMEMORY_USE_LIBUNWIND) || !defined(OMNITRACE_USE_ROCM_SMI)
+TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_gpu_busy,
+                               false_type)
+TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_gpu_temp,
+                               false_type)
+TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_gpu_power,
+                               false_type)
+TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_gpu_memory,
+                               false_type)
+#endif
+
 TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::omnitrace, OMNITRACE_COMPONENT,
                                 "omnitrace", "omnitrace_component")
 TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::roctracer, OMNITRACE_ROCTRACER,
@@ -78,38 +102,87 @@ TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::sampling_cpu_clock,
                                 OMNITRACE_SAMPLING_CPU_CLOCK, "sampling_cpu_clock", "")
 TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::sampling_percent,
                                 OMNITRACE_SAMPLING_PERCENT, "sampling_percent", "")
+TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::sampling_gpu_busy,
+                                 OMNITRACE_SAMPLING_GPU_BUSY, "sampling_gpu_busy",
+                                 "sampling_gpu_util")
+TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::sampling_gpu_memory,
+                                 OMNITRACE_SAMPLING_GPU_MEMORY_USAGE,
+                                 "sampling_gpu_memory_usage", "")
+TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::sampling_gpu_power,
+                                 OMNITRACE_SAMPLING_GPU_POWER, "sampling_gpu_power", "")
+TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::sampling_gpu_temp,
+                                 OMNITRACE_SAMPLING_GPU_TEMP, "sampling_gpu_temp", "")

+TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::roctracer, "roctracer",
+                                 "High-precision ROCm API and kernel tracing", "")
 TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_wall_clock,
                                 "sampling_wall_clock", "Wall-clock timing",
-                                 "derived from statistical sampling")
+                                 "Derived from statistical sampling")
 TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_cpu_clock,
                                 "sampling_cpu_clock", "CPU-clock timing",
-                                 "derived from statistical sampling")
+                                 "Derived from statistical sampling")
 TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_percent,
                                 "sampling_percent",
                                 "Fraction of wall-clock time spent in functions",
-                                 "derived from statistical sampling")
-TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::roctracer, "roctracer",
-                                 "High-precision ROCm API and kernel tracing", "")
+                                 "Derived from statistical sampling")
+TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_gpu_busy,
+                                 "sampling_gpu_busy",
+                                 "GPU Utilization (% busy) via ROCm-SMI",
+                                 "Derived from sampling")
+TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_gpu_memory,
+                                 "sampling_gpu_memory_usage",
+                                 "GPU Memory Usage via ROCm-SMI", "Derived from sampling")
+TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_gpu_power,
+                                 "sampling_gpu_power", "GPU Power Usage via ROCm-SMI",
+                                 "Derived from sampling")
+TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_gpu_temp,
+                                 "sampling_gpu_temp", "GPU Temperature via ROCm-SMI",
+                                 "Derived from sampling")

+// statistics type
 TIMEMORY_STATISTICS_TYPE(omnitrace::component::sampling_wall_clock, double)
 TIMEMORY_STATISTICS_TYPE(omnitrace::component::sampling_cpu_clock, double)
+TIMEMORY_STATISTICS_TYPE(omnitrace::component::sampling_gpu_busy, double)
+TIMEMORY_STATISTICS_TYPE(omnitrace::component::sampling_gpu_temp, double)
+TIMEMORY_STATISTICS_TYPE(omnitrace::component::sampling_gpu_power, double)
+TIMEMORY_STATISTICS_TYPE(omnitrace::component::sampling_gpu_memory, double)

 // enable timing units
 TIMEMORY_DEFINE_CONCRETE_TRAIT(is_timing_category,
                               omnitrace::component::sampling_wall_clock, true_type)
-TIMEMORY_DEFINE_CONCRETE_TRAIT(uses_timing_units,
-                               omnitrace::component::sampling_wall_clock, true_type)
 TIMEMORY_DEFINE_CONCRETE_TRAIT(is_timing_category,
                               omnitrace::component::sampling_cpu_clock, true_type)
-TIMEMORY_DEFINE_CONCRETE_TRAIT(uses_timing_units,
-                               omnitrace::component::sampling_cpu_clock, true_type)
 TIMEMORY_DEFINE_CONCRETE_TRAIT(is_timing_category, omnitrace::component::sampling_percent,
                               true_type)
+TIMEMORY_DEFINE_CONCRETE_TRAIT(uses_timing_units,
+                               omnitrace::component::sampling_wall_clock, true_type)
+TIMEMORY_DEFINE_CONCRETE_TRAIT(uses_timing_units,
+                               omnitrace::component::sampling_cpu_clock, true_type)

+// enable percent units
+TIMEMORY_DEFINE_CONCRETE_TRAIT(uses_percent_units,
+                               omnitrace::component::sampling_gpu_busy, true_type)
+
+// enable memory units
+TIMEMORY_DEFINE_CONCRETE_TRAIT(is_memory_category,
+                               omnitrace::component::sampling_gpu_memory, true_type)
+TIMEMORY_DEFINE_CONCRETE_TRAIT(uses_memory_units,
+                               omnitrace::component::sampling_gpu_memory, true_type)
+
+// reporting categories (sum)
+TIMEMORY_DEFINE_CONCRETE_TRAIT(report_sum, omnitrace::component::sampling_gpu_busy,
+                               false_type)
+TIMEMORY_DEFINE_CONCRETE_TRAIT(report_sum, omnitrace::component::sampling_gpu_temp,
+                               false_type)
+TIMEMORY_DEFINE_CONCRETE_TRAIT(report_sum, omnitrace::component::sampling_gpu_power,
+                               false_type)
+TIMEMORY_DEFINE_CONCRETE_TRAIT(report_sum, omnitrace::component::sampling_gpu_memory,
+                               false_type)
+
+// reporting categories (mean)
 TIMEMORY_DEFINE_CONCRETE_TRAIT(report_mean, omnitrace::component::sampling_percent,
                               false_type)
-TIMEMORY_DEFINE_CONCRETE_TRAIT(report_units, omnitrace::component::sampling_percent,
-                               false_type)
+
+// reporting categories (stats)
 TIMEMORY_DEFINE_CONCRETE_TRAIT(report_statistics, omnitrace::component::sampling_percent,
                               false_type)
@@ -56,6 +56,7 @@ struct pthread_gotcha : tim::component::base<pthread_gotcha, void>

    // generate the gotcha wrappers
    static void configure();
+    static void shutdown();

    // threads can set this to avoid starting sampling on child threads
    static bool& enable_sampling_on_child_threads();
@@ -0,0 +1,165 @@
+// Copyright (c) 2018 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// with the Software without restriction, including without limitation the
+// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+// sell copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// * Redistributions of source code must retain the above copyright notice,
+// this list of conditions and the following disclaimers.
+//
+// * Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimers in the
+// documentation and/or other materials provided with the distribution.
+//
+// * Neither the names of Advanced Micro Devices, Inc. nor the names of its
+// contributors may be used to endorse or promote products derived from
+// this Software without specific prior written permission.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH
+// THE SOFTWARE.
+
+#pragma once
+
+#include "library/common.hpp"
+#include "library/components/fwd.hpp"
+#include "library/defines.hpp"
+#include "library/thread_data.hpp"
+
+#include <chrono>
+#include <cstdint>
+#include <deque>
+#include <future>
+#include <limits>
+#include <memory>
+#include <ratio>
+#include <thread>
+#include <tuple>
+#include <type_traits>
+
+namespace omnitrace
+{
+namespace rocm_smi
+{
+void
+setup();
+
+void
+shutdown();
+
+uint32_t
+device_count();
+
+struct data
+{
+    using msec_t    = std::chrono::milliseconds;
+    using usec_t    = std::chrono::microseconds;
+    using nsec_t    = std::chrono::nanoseconds;
+    using promise_t = std::promise<void>;
+
+    using timestamp_t = int64_t;
+    using power_t     = uint64_t;
+    using busy_perc_t = uint32_t;
+    using mem_usage_t = uint64_t;
+    using temp_t      = int64_t;
+
+    TIMEMORY_DEFAULT_OBJECT(data)
+
+    explicit data(uint32_t _dev_id);
+
+    void sample(uint32_t _dev_id);
+    void print(std::ostream& _os) const;
+
+    template <typename Tp                                                      = nsec_t,
+              std::enable_if_t<!std::is_same_v<std::decay_t<Tp>, nsec_t>, int> = 0>
+    static void poll(std::atomic<State>* _state, Tp&& _interval, promise_t*);
+
+    static void set_state(State);
+    static void poll(std::atomic<State>* _state, nsec_t _interval, promise_t*);
+    static void post_process(uint32_t _dev_id);
+
+    uint32_t    m_dev_id    = std::numeric_limits<uint32_t>::max();
+    timestamp_t m_ts        = 0;
+    busy_perc_t m_busy_perc = 0;
+    temp_t      m_temp      = 0;
+    power_t     m_power     = 0;
+    mem_usage_t m_mem_usage = 0;
+
+    friend std::ostream& operator<<(std::ostream& _os, const data& _v)
+    {
+        _v.print(_os);
+        return _os;
+    }
+
+private:
+    friend void omnitrace::rocm_smi::setup();
+    friend void omnitrace::rocm_smi::shutdown();
+
+    static size_t                        device_count;
+    static std::set<uint32_t>            device_list;
+    static std::unique_ptr<promise_t>    polling_finished;
+    static std::vector<data>&            get_initial();
+    static std::unique_ptr<std::thread>& get_thread();
+    static bool                          setup();
+    static bool                          shutdown();
+};
+
+template <
+    typename Tp,
+    std::enable_if_t<!std::is_same_v<std::decay_t<Tp>, std::chrono::nanoseconds>, int>>
+void
+data::poll(std::atomic<State>* _state, Tp&& _interval, promise_t* _prom)
+{
+    poll(_state, std::chrono::duration_cast<nsec_t>(_interval), _prom);
+}
+
+using bundle_t          = std::deque<data>;
+using sampler_instances = thread_data<bundle_t, api::rocm_smi>;
+
+#if !defined(OMNITRACE_USE_ROCM_SMI)
+inline void
+setup()
+{}
+
+inline void
+shutdown()
+{}
+
+inline void data::post_process(uint32_t) {}
+#endif
+}  // namespace rocm_smi
+}  // namespace omnitrace
+
+#if defined(OMNITRACE_USE_ROCM_SMI)
+#    if !defined(OMNITRACE_EXTERN_COMPONENTS) ||                                         \
+        (defined(OMNITRACE_EXTERN_COMPONENTS) && OMNITRACE_EXTERN_COMPONENTS > 0)
+
+#        include <timemory/components/base.hpp>
+#        include <timemory/components/data_tracker/components.hpp>
+#        include <timemory/operations.hpp>
+
+TIMEMORY_DECLARE_EXTERN_COMPONENT(
+    TIMEMORY_ESC(data_tracker<double, omnitrace::component::backtrace_gpu_busy>), true,
+    double)
+
+TIMEMORY_DECLARE_EXTERN_COMPONENT(
+    TIMEMORY_ESC(data_tracker<double, omnitrace::component::backtrace_gpu_temp>), true,
+    double)
+
+TIMEMORY_DECLARE_EXTERN_COMPONENT(
+    TIMEMORY_ESC(data_tracker<double, omnitrace::component::backtrace_gpu_power>), true,
+    double)
+
+TIMEMORY_DECLARE_EXTERN_COMPONENT(
+    TIMEMORY_ESC(data_tracker<double, omnitrace::component::backtrace_gpu_memory>), true,
+    double)
+
+#    endif
+#endif
@@ -52,26 +52,42 @@ struct roctracer

    static void preinit();
    static void global_init() { setup(); }
-    static void global_finalize() { tear_down(); }
+    static void global_finalize() { shutdown(); }

    static bool is_setup();
    static void setup();
-    static void tear_down();
+    static void shutdown();
    static void add_setup(const std::string&, std::function<void()>&&);
-    static void add_tear_down(const std::string&, std::function<void()>&&);
+    static void add_shutdown(const std::string&, std::function<void()>&&);
    static void remove_setup(const std::string&);
-    static void remove_tear_down(const std::string&);
+    static void remove_shutdown(const std::string&);

    void start();
    void stop();
-    void set_prefix(const char* _v) { m_prefix = _v; }
-
-private:
-    const char* m_prefix = nullptr;
 };
+
+#if !defined(OMNITRACE_USE_ROCTRACER)
+inline void
+roctracer::setup()
+{}
+
+inline void
+roctracer::shutdown()
+{}
+
+inline bool
+roctracer::is_setup()
+{
+    return false;
+}
+#endif
 }  // namespace component
 }  // namespace tim

+#if !defined(OMNITRACE_USE_ROCTRACER)
+TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, component::roctracer_data, false_type)
+#endif
+
 TIMEMORY_SET_COMPONENT_API(component::roctracer_data, project::timemory, category::timing,
                           os::supports_unix)
 TIMEMORY_DEFINE_CONCRETE_TRAIT(is_timing_category, component::roctracer_data, true_type)
@@ -90,5 +90,5 @@ roctracer_functions_t&
 roctracer_setup_routines();

 roctracer_functions_t&
-roctracer_tear_down_routines();
+roctracer_shutdown_routines();
 }  // namespace omnitrace
@@ -31,6 +31,7 @@
 #include "library/defines.hpp"
 #include "library/state.hpp"
 #include "library/timemory.hpp"
+#include "timemory/macros/language.hpp"

 #include <timemory/backends/threading.hpp>

@@ -43,10 +44,9 @@ namespace omnitrace
 // bundle of components around omnitrace_init and omnitrace_finalize
 using main_bundle_t =
    tim::lightweight_tuple<comp::wall_clock, comp::peak_rss, comp::cpu_clock,
-                           comp::cpu_util, comp::roctracer>;
+                           comp::cpu_util, pthread_gotcha_t>;

-using gotcha_bundle_t =
-    tim::lightweight_tuple<fork_gotcha_t, mpi_gotcha_t, pthread_gotcha_t>;
+using gotcha_bundle_t = tim::lightweight_tuple<fork_gotcha_t, mpi_gotcha_t>;

 // bundle of components used in instrumentation
 using instrumentation_bundle_t =
@@ -69,26 +69,63 @@ using omnitrace_thread_bundle_t =
 //
 //      Initialization routines
 //
+inline namespace config
+{
 void
-configure_settings() TIMEMORY_VISIBILITY("default");
+configure_settings();

 void
-print_config_settings(
+print_banner(std::ostream& _os = std::cout);
+
+void
+print_settings(
    std::ostream&                                                                _os,
    std::function<bool(const std::string_view&, const std::set<std::string>&)>&& _filter);

+void
+print_settings();
+
 std::string&
 get_exe_name();

+template <typename Tp>
+bool
+set_setting_value(const std::string& _name, Tp&& _v)
+{
+    auto _instance = tim::settings::shared_instance();
+    auto _setting  = _instance->find(_name);
+    if(_setting == _instance->end()) return false;
+    if(!_setting->second) return false;
+    return _setting->second->set(std::forward<Tp>(_v));
+}
+
 //
 //      User-configurable settings
 //
 std::string
 get_config_file();

+Mode
+get_mode();
+
+bool&
+is_attached();
+
+bool&
+is_binary_rewrite();
+
+bool
+get_is_continuous_integration();
+
 bool
 get_debug_env();

+bool
+get_debug_init();
+
+bool
+get_debug_finalize();
+
 bool
 get_debug();

@@ -113,6 +150,9 @@ get_use_timemory();
 bool&
 get_use_roctracer();

+bool&
+get_use_rocm_smi();
+
 bool&
 get_use_sampling();

@@ -125,6 +165,9 @@ get_use_mpip();
 bool&
 get_use_critical_trace();

+bool
+get_use_kokkosp();
+
 bool
 get_timeline_sampling();

@@ -167,8 +210,9 @@ get_trace_hsa_api_types();
 std::string&
 get_backend();

+// make this visible so omnitrace-avail can call it
 std::string&
-get_perfetto_output_filename();
+get_perfetto_output_filename() TIMEMORY_VISIBILITY("default");

 int64_t
 get_critical_trace_count();
@@ -182,8 +226,15 @@ get_sampling_freq();
 double&
 get_sampling_delay();

+double&
+get_rocm_smi_freq();
+
+std::string
+get_rocm_smi_devices();
+
 int64_t
 get_critical_trace_per_row();
+}  // namespace config

 //
 //      Runtime configuration data
@@ -175,7 +175,7 @@ struct call_chain : private std::vector<entry>
    }

    template <Device DevT>
-    void generate_perfetto(std::set<entry>& _used) const;
+    void generate_perfetto(std::set<entry>& _used, bool _basic = false) const;

    template <bool BoolV = true, typename FuncT>
    bool query(FuncT&&) const;
@@ -201,6 +201,11 @@ update(int64_t _tid = threading::get_id());
 void
 compute(int64_t _tid = threading::get_id());

+std::vector<std::pair<std::string, entry>>
+get_entries(
+    int64_t                                  _ts,
+    const std::function<bool(const entry&)>& _eval = [](const entry&) { return true; });
+
 struct id
 {};

@@ -33,6 +33,8 @@

 namespace omnitrace
 {
+inline namespace config
+{
 bool
 get_debug();

@@ -44,6 +46,7 @@ get_debug_pid();

 bool
 get_critical_trace_debug();
+}  // namespace config
 }  // namespace omnitrace

 #if defined(TIMEMORY_USE_MPI)
@@ -80,8 +83,58 @@ get_critical_trace_debug();
        fflush(stderr);                                                                  \
    }

+#if defined(TIMEMORY_USE_MPI)
+#    define OMNITRACE_CONDITIONAL_THROW(COND, ...)                                       \
+        if(COND)                                                                         \
+        {                                                                                \
+            char _msg_buffer[2048];                                                      \
+            snprintf(_msg_buffer, 2048, "[omnitrace][%i][%li] ",                         \
+                     static_cast<int>(tim::dmp::rank()), tim::threading::get_id());      \
+            auto len = strlen(_msg_buffer);                                              \
+            snprintf(_msg_buffer + len, 2048 - len, __VA_ARGS__);                        \
+            throw std::runtime_error(_msg_buffer);                                       \
+        }
+#else
+#    define OMNITRACE_CONDITIONAL_THROW(COND, ...)                                       \
+        if(COND)                                                                         \
+        {                                                                                \
+            char _msg_buffer[2048];                                                      \
+            snprintf(_msg_buffer, 2048, "[omnitrace][%i][%li] ",                         \
+                     static_cast<int>(tim::process::get_id()),                           \
+                     tim::threading::get_id());                                          \
+            auto len = strlen(_msg_buffer);                                              \
+            snprintf(_msg_buffer + len, 2048 - len, __VA_ARGS__);                        \
+            throw std::runtime_error(_msg_buffer);                                       \
+        }
+#endif
+
+#define OMNITRACE_CONDITIONAL_BASIC_THROW(COND, ...)                                     \
+    if(COND)                                                                             \
+    {                                                                                    \
+        char _msg_buffer[2048];                                                          \
+        snprintf(_msg_buffer, 2048, "[omnitrace] " __VA_ARGS__);                         \
+        auto len = strlen(_msg_buffer);                                                  \
+        snprintf(_msg_buffer + len, 2048 - len, __VA_ARGS__);                            \
+        throw std::runtime_error(_msg_buffer);                                           \
+    }
+
 #define OMNITRACE_DEBUG(...)                                                             \
    OMNITRACE_CONDITIONAL_PRINT(::omnitrace::get_debug(), __VA_ARGS__)
 #define OMNITRACE_PRINT(...) OMNITRACE_CONDITIONAL_PRINT(true, __VA_ARGS__)
 #define OMNITRACE_CT_DEBUG(...)                                                          \
    OMNITRACE_CONDITIONAL_PRINT(::omnitrace::get_critical_trace_debug(), __VA_ARGS__)
+#define OMNITRACE_BASIC_PRINT(...) OMNITRACE_CONDITIONAL_BASIC_PRINT(true, __VA_ARGS__)
+
+#define OMNITRACE_THROW(...)       OMNITRACE_CONDITIONAL_THROW(true, __VA_ARGS__)
+#define OMNITRACE_BASIC_THROW(...) OMNITRACE_CONDITIONAL_BASIC_THROW(true, __VA_ARGS__)
+
+#include <string>
+
+namespace std
+{
+inline std::string
+to_string(bool _v)
+{
+    return (_v) ? "true" : "false";
+}
+}  // namespace std
@@ -36,11 +36,17 @@
 // clang-format on

 #define TIMEMORY_USER_COMPONENT_ENUM                                                     \
-    OMNITRACE_SAMPLING_WALL_CLOCK_idx, OMNITRACE_SAMPLING_CPU_CLOCK_idx,                 \
-        OMNITRACE_SAMPLING_PERCENT_idx, OMNITRACE_COMPONENT_idx, OMNITRACE_ROCTRACER_idx,
+    OMNITRACE_COMPONENT_idx, OMNITRACE_ROCTRACER_idx, OMNITRACE_SAMPLING_WALL_CLOCK_idx, \
+        OMNITRACE_SAMPLING_CPU_CLOCK_idx, OMNITRACE_SAMPLING_PERCENT_idx,                \
+        OMNITRACE_SAMPLING_GPU_POWER_idx, OMNITRACE_SAMPLING_GPU_TEMP_idx,               \
+        OMNITRACE_SAMPLING_GPU_BUSY_idx, OMNITRACE_SAMPLING_GPU_MEMORY_USAGE_idx,

-#define OMNITRACE_COMPONENT           OMNITRACE_COMPONENT_idx
-#define OMNITRACE_ROCTRACER           OMNITRACE_ROCTRACER_idx
-#define OMNITRACE_SAMPLING_WALL_CLOCK OMNITRACE_SAMPLING_WALL_CLOCK_idx
-#define OMNITRACE_SAMPLING_CPU_CLOCK  OMNITRACE_SAMPLING_CPU_CLOCK_idx
-#define OMNITRACE_SAMPLING_PERCENT    OMNITRACE_SAMPLING_PERCENT_idx
+#define OMNITRACE_COMPONENT                 OMNITRACE_COMPONENT_idx
+#define OMNITRACE_ROCTRACER                 OMNITRACE_ROCTRACER_idx
+#define OMNITRACE_SAMPLING_WALL_CLOCK       OMNITRACE_SAMPLING_WALL_CLOCK_idx
+#define OMNITRACE_SAMPLING_CPU_CLOCK        OMNITRACE_SAMPLING_CPU_CLOCK_idx
+#define OMNITRACE_SAMPLING_PERCENT          OMNITRACE_SAMPLING_PERCENT_idx
+#define OMNITRACE_SAMPLING_GPU_POWER        OMNITRACE_SAMPLING_GPU_POWER_idx
+#define OMNITRACE_SAMPLING_GPU_TEMP         OMNITRACE_SAMPLING_GPU_TEMP_idx
+#define OMNITRACE_SAMPLING_GPU_BUSY         OMNITRACE_SAMPLING_GPU_BUSY_idx
+#define OMNITRACE_SAMPLING_GPU_MEMORY_USAGE OMNITRACE_SAMPLING_GPU_MEMORY_USAGE_idx
@@ -33,6 +33,9 @@
 #    define PERFETTO_CATEGORIES                                                          \
        perfetto::Category("host").SetDescription("Host-side function tracing"),         \
            perfetto::Category("device").SetDescription("Device-side function tracing"), \
+            perfetto::Category("rocm_smi").SetDescription("Device-level metrics"),       \
+            perfetto::Category("sampling")                                               \
+                .SetDescription("Metrics derived from sampling"),                        \
            perfetto::Category("host-critical-trace")                                    \
                .SetDescription("Host-side critical traces"),                            \
            perfetto::Category("device-critical-trace")                                  \
@@ -41,6 +44,9 @@
 #    define PERFETTO_CATEGORIES                                                          \
        perfetto::Category("host").SetDescription("Host-side function tracing"),         \
            perfetto::Category("device").SetDescription("Device-side function tracing"), \
+            perfetto::Category("rocm_smi").SetDescription("Device-level metrics"),       \
+            perfetto::Category("sampling")                                               \
+                .SetDescription("Metrics derived from sampling"),                        \
            perfetto::Category("host-critical-trace")                                    \
                .SetDescription("Host-side critical traces"),                            \
            perfetto::Category("device-critical-trace")                                  \
@@ -87,4 +93,44 @@ public:

 PERFETTO_DECLARE_DATA_SOURCE_STATIC_MEMBERS(CustomDataSource);
 #endif
+
+template <typename Tp>
+struct perfetto_counter_track
+{
+    using track_map_t = std::map<uint32_t, std::vector<perfetto::CounterTrack>>;
+    using name_map_t  = std::map<uint32_t, std::vector<std::string>>;
+    using data_t      = std::pair<name_map_t, track_map_t>;
+
+    static auto init() { (void) get_data(); }
+
+    static auto exists(size_t _idx, int64_t _n = -1)
+    {
+        bool _v = get_data().second.count(_idx) != 0;
+        if(_n < 0 || !_v) return _v;
+        return static_cast<size_t>(_n) < get_data().second.at(_idx).size();
+    }
+
+    static size_t size(size_t _idx)
+    {
+        bool _v = get_data().second.count(_idx) != 0;
+        if(!_v) return 0;
+        return get_data().second.at(_idx).size();
+    }
+
+    static auto emplace(size_t _idx, const std::string& _v, const char* _units)
+    {
+        get_data().first[_idx].emplace_back(_v);
+        get_data().second[_idx].emplace_back(get_data().first[_idx].back().c_str(),
+                                             _units);
+    }
+
+    static auto& at(size_t _idx, size_t _n) { return get_data().second.at(_idx).at(_n); }
+
+private:
+    static data_t& get_data()
+    {
+        static auto* _v = new data_t{};
+        return *_v;
+    }
+};
 }  // namespace omnitrace
@@ -0,0 +1,95 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include <iostream>
+#include <ostream>
+#include <sstream>
+#include <streambuf>
+#include <string>
+
+namespace omnitrace
+{
+inline namespace config
+{
+bool
+get_debug();
+
+int
+get_verbose();
+}  // namespace config
+
+struct redirect
+{
+    redirect(std::ostream& _os, std::string _expected)
+    : m_os{ _os }
+    , m_expected{ std::move(_expected) }
+    {
+        if(!get_debug())
+        {
+            // save stream buffer
+            m_strm_buffer = m_os.rdbuf();
+            // redirect to stringstream
+            _os.rdbuf(m_buffer.rdbuf());
+        }
+    }
+
+    ~redirect()
+    {
+        if(!m_strm_buffer) return;
+        // restore stream buffer
+        m_os.rdbuf(m_strm_buffer);
+        auto _v      = m_buffer.str();
+        _v           = replace(m_buffer.str(), '\n');
+        auto _expect = replace(m_expected, '\n');
+        if(_v != _expect)
+        {
+            if(get_verbose() > 0)
+                std::cerr << "[omnitrace::redirect] Expected:\n[omnitrace::redirect]    "
+                          << _expect
+                          << "\n[omnitrace::redirect] Found:\n[omnitrace::redirect]    "
+                          << _v << "\n";
+            if(get_verbose() <= 0 || (&m_os != &std::cerr && &m_os != &std::cout))
+                m_os << m_buffer.str() << std::flush;
+        }
+    }
+
+private:
+    template <typename Tp>
+    static std::string replace(std::string _v, Tp _c, const std::string& _s = " ")
+    {
+        while(true)
+        {
+            auto _pos = _v.find(_c);
+            if(_pos == std::string::npos) break;
+            _v = _v.replace(_pos, 1, _s);
+        }
+        return _v;
+    }
+
+    std::ostream&     m_os;
+    std::string       m_expected = {};
+    std::stringstream m_buffer{};
+    std::streambuf*   m_strm_buffer = nullptr;
+};
+}  // namespace omnitrace
@@ -46,6 +46,10 @@ using component::backtrace_cpu_clock;   // NOLINT
 using component::backtrace_fraction;    // NOLINT
 using component::backtrace_wall_clock;  // NOLINT
 using component::sampling_cpu_clock;
+using component::sampling_gpu_busy;
+using component::sampling_gpu_memory;
+using component::sampling_gpu_power;
+using component::sampling_gpu_temp;
 using component::sampling_percent;
 using component::sampling_wall_clock;

@@ -64,7 +68,7 @@ void unblock_signals(std::set<int> = {});

 using bundle_t          = tim::lightweight_tuple<backtrace>;
 using sampler_t         = tim::sampling::sampler<bundle_t, tim::sampling::dynamic>;
-using sampler_instances = omnitrace_thread_data<sampler_t, api::sampling>;
+using sampler_instances = thread_data<sampler_t, api::sampling>;

 std::unique_ptr<sampler_t>&
 get_sampler(int64_t _tid = threading::get_id());
@@ -76,7 +80,7 @@ TIMEMORY_DEFINE_CONCRETE_TRAIT(prevent_reentry, omnitrace::sampling::sampler_t,
                               std::true_type)

 TIMEMORY_DEFINE_CONCRETE_TRAIT(check_signals, omnitrace::sampling::sampler_t,
-                               std::false_type)
+                               std::true_type)

 TIMEMORY_DEFINE_CONCRETE_TRAIT(buffer_size, omnitrace::sampling::sampler_t,
                               TIMEMORY_ESC(std::integral_constant<size_t, 256>))
@@ -31,7 +31,25 @@ enum class State : unsigned short
 {
    DelayedInit = 0,
    PreInit,
+    Init,
    Active,
    Finalized
 };
+
+enum class Mode : unsigned short
+{
+    Trace = 0,
+    Sampling
+};
 }  // namespace omnitrace
+
+#include <string>
+
+namespace std
+{
+std::string
+to_string(omnitrace::State _v);
+
+std::string
+to_string(omnitrace::Mode _v);
+}  // namespace std
@@ -40,7 +40,7 @@ namespace omnitrace
 static constexpr size_t max_supported_threads = OMNITRACE_MAX_THREADS;

 template <typename Tp, typename Tag = void, size_t MaxThreads = max_supported_threads>
-struct omnitrace_thread_data
+struct thread_data
 {
    using instance_array_t  = std::array<std::unique_ptr<Tp>, MaxThreads>;
    using construct_on_init = std::true_type;
@@ -58,7 +58,7 @@ struct omnitrace_thread_data
 template <typename Tp, typename Tag, size_t MaxThreads>
 template <typename... Args>
 void
-omnitrace_thread_data<Tp, Tag, MaxThreads>::construct(Args&&... _args)
+thread_data<Tp, Tag, MaxThreads>::construct(Args&&... _args)
 {
    // construct outside of lambda to prevent data-race
    static auto&             _instances = instances();
@@ -72,14 +72,14 @@ omnitrace_thread_data<Tp, Tag, MaxThreads>::construct(Args&&... _args)

 template <typename Tp, typename Tag, size_t MaxThreads>
 std::unique_ptr<Tp>&
-omnitrace_thread_data<Tp, Tag, MaxThreads>::instance()
+thread_data<Tp, Tag, MaxThreads>::instance()
 {
    return instances().at(threading::get_id());
 }

 template <typename Tp, typename Tag, size_t MaxThreads>
-typename omnitrace_thread_data<Tp, Tag, MaxThreads>::instance_array_t&
-omnitrace_thread_data<Tp, Tag, MaxThreads>::instances()
+typename thread_data<Tp, Tag, MaxThreads>::instance_array_t&
+thread_data<Tp, Tag, MaxThreads>::instances()
 {
    static auto _v = instance_array_t{};
    return _v;
@@ -88,7 +88,7 @@ omnitrace_thread_data<Tp, Tag, MaxThreads>::instances()
 template <typename Tp, typename Tag, size_t MaxThreads>
 template <typename... Args>
 std::unique_ptr<Tp>&
-omnitrace_thread_data<Tp, Tag, MaxThreads>::instance(construct_on_init, Args&&... _args)
+thread_data<Tp, Tag, MaxThreads>::instance(construct_on_init, Args&&... _args)
 {
    construct(std::forward<Args>(_args)...);
    return instances().at(threading::get_id());
@@ -96,8 +96,8 @@ omnitrace_thread_data<Tp, Tag, MaxThreads>::instance(construct_on_init, Args&&..

 template <typename Tp, typename Tag, size_t MaxThreads>
 template <typename... Args>
-typename omnitrace_thread_data<Tp, Tag, MaxThreads>::instance_array_t&
-omnitrace_thread_data<Tp, Tag, MaxThreads>::instances(construct_on_init, Args&&... _args)
+typename thread_data<Tp, Tag, MaxThreads>::instance_array_t&
+thread_data<Tp, Tag, MaxThreads>::instances(construct_on_init, Args&&... _args)
 {
    static auto _v = [&]() {
        auto _internal = instance_array_t{};
@@ -160,6 +160,10 @@ static regexvec_t      file_exclude                  = {};
    if(verbose_level >= LEVEL) fprintf(stdout, "[omnitrace][exe] " __VA_ARGS__);         \
    fflush(stdout);

+#define verbprintf_bare(LEVEL, ...)                                                      \
+    if(verbose_level >= LEVEL) fprintf(stdout, __VA_ARGS__);                             \
+    fflush(stdout);
+
 //======================================================================================//

 template <typename... T>
@@ -477,7 +481,7 @@ dump_info(const string_t& _oname, const fmodset_t& _data, int _level, bool _fail
    {
        verbprintf(_level, "Dumping '%s'... ", _oname.c_str());
        dump_info(ofs, _data);
-        verbprintf(_level, "Done\n");
+        verbprintf_bare(_level, "Done\n");
    }
    else
    {
@@ -597,7 +601,7 @@ omnitrace_get_address_space(patch_pointer_t& _bpatch, int _cmdc, char** _cmdv,
                    _name.c_str());
            throw std::runtime_error("Failed to open binary");
        }
-        verbprintf(1, "Done\n");
+        verbprintf_bare(1, "Done\n");
    }
    else if(_pid >= 0)
    {
@@ -611,7 +615,7 @@ omnitrace_get_address_space(patch_pointer_t& _bpatch, int _cmdc, char** _cmdv,
                    (int) _pid);
            throw std::runtime_error("Failed to attach to process");
        }
-        verbprintf(1, "Done\n");
+        verbprintf_bare(1, "Done\n");
    }
    else
    {
@@ -630,7 +634,7 @@ omnitrace_get_address_space(patch_pointer_t& _bpatch, int _cmdc, char** _cmdv,
                    ss.str().c_str());
            throw std::runtime_error("Failed to create process");
        }
-        verbprintf(1, "Done\n");
+        verbprintf_bare(1, "Done\n");
    }

    return mutatee;
@@ -23,14 +23,14 @@
 //  IN THE SOFTWARE.

 #include "avail.hpp"
-
+#include "library/api.hpp"
+#include "library/components/backtrace.hpp"
 #include "library/components/fork_gotcha.hpp"
 #include "library/components/mpi_gotcha.hpp"
 #include "library/components/omnitrace.hpp"
 #include "library/components/pthread_gotcha.hpp"
 #include "library/components/roctracer.hpp"
 #include "library/config.hpp"
-#include "library/sampling.hpp"

 #include <timemory/components.hpp>
 #include <timemory/components/definition.hpp>
@@ -249,8 +249,6 @@ enum
 int
 main(int argc, char** argv)
 {
-    omnitrace::configure_settings();
-
    array_t<bool, 6>     options  = { false, false, false, false, false, false };
    array_t<string_t, 6> fields   = {};
    array_t<bool, 6>     use_mark = {};
@@ -436,7 +434,7 @@ main(int argc, char** argv)
    _parser_set_if_exists(include_hw_counters, "hw-counters");

    if(!include_components && !include_settings && !include_hw_counters)
-        include_components = true;
+        include_settings = true;

    if(markdown || include_hw_counters) padding = 6;

@@ -457,6 +455,8 @@ main(int argc, char** argv)

    if(!os) os = &std::cout;

+    omnitrace_init_library();
+
    if(include_components) write_component_info(*os, options, use_mark, fields);

    dump_log();
@@ -699,39 +699,49 @@ write_settings_info(std::ostream& os, const array_t<bool, N>& opts,
    using width_type             = array_t<int64_t, size>;
    using width_bool             = array_t<bool, size>;

-    array_type _setting_output;
-    unique_set _settings_exclude = { "OMNITRACE_ENVIRONMENT",
-                                     "OMNITRACE_COMMAND_LINE",
-                                     "cereal_class_version",
-                                     "settings",
-                                     "OMNITRACE_CUDA_EVENT_BATCH_SIZE",
-                                     "OMNITRACE_CUPTI_ACTIVITY_KINDS",
-                                     "OMNITRACE_CUPTI_ACTIVITY_LEVEL",
-                                     "OMNITRACE_CUPTI_DEVICE",
-                                     "OMNITRACE_CUPTI_EVENTS",
-                                     "OMNITRACE_CUPTI_METRICS",
-                                     "OMNITRACE_CUPTI_PCSAMPLING_NUM_COLLECT",
-                                     "OMNITRACE_CUPTI_PCSAMPLING_PERIOD",
-                                     "OMNITRACE_CUPTI_PCSAMPLING_PER_LINE",
-                                     "OMNITRACE_CUPTI_PCSAMPLING_REGION_TOTALS",
-                                     "OMNITRACE_CUPTI_PCSAMPLING_SERIALIZED",
-                                     "OMNITRACE_CUPTI_PCSAMPLING_STALL_REASONS" };
-
-    cereal::SettingsTextArchive settings_archive{ _setting_output, _settings_exclude };
-    settings::serialize_settings(settings_archive);
-
    width_type _widths = { 0, 0, 0, 0, 0, 0, 0 };
    width_bool _wusing = {
        true, !force_brief, opts[0], opts[1], opts[1], opts[1], opts[2]
    };
    width_bool _mark = { false, false, false, true, true, true, false };

+    // this settings has delayed initialization. make sure it is generated
+    (void) omnitrace::config::get_perfetto_output_filename();
+
+    array_type _setting_output;
+    unique_set _settings_exclude = { "OMNITRACE_ENVIRONMENT", "OMNITRACE_COMMAND_LINE",
+                                     "cereal_class_version", "settings" };
+
+#if !defined(TIMEMORY_USE_CRAYPAT)
+    _settings_exclude.emplace("OMNITRACE_CRAYPAT");
+#endif
+
+    cereal::SettingsTextArchive settings_archive{ _setting_output, _settings_exclude };
+    settings::serialize_settings(settings_archive);
+
+    // exclude some timemory settings which are not relevant to omnitrace
+    //  exact matches, e.g. OMNITRACE_BANNER
+    std::string _settings_rexclude_exact =
+        "^OMNITRACE_(BANNER|DESTRUCTOR_REPORT|COMPONENTS|(GLOBAL|MPIP|NCCLP|OMPT|"
+        "PROFILER|TRACE)_COMPONENTS|PYTHON_EXE|PAPI_ATTACH|PLOT_OUTPUT|SEPARATOR_FREQ|"
+        "STACK_CLEARING|TARGET_PID|THROTTLE_(COUNT|VALUE)|(AUTO|FLAMEGRAPH)_OUTPUT|"
+        "(ENABLE|DISABLE)_ALL_SIGNALS|ALLOW_SIGNAL_HANDLER|CTEST_NOTES)$";
+    //  leading matches, e.g. OMNITRACE_MPI_[A-Z_]+
+    std::string _settings_rexclude_begin =
+        "^OMNITRACE_(ERT|DART|MPI|UPCXX|ROOFLINE|CUDA|NVTX|CUPTI)_[A-Z_]+$";
+
+    // lambda for deciding which settings we want to restrict displaying
+    auto&& _remove_conditions = [&_settings_exclude, &_settings_rexclude_exact,
+                                 &_settings_rexclude_begin](const auto& itr) {
+        auto&& _v = itr.find("environ")->second;
+        bool   _a = _settings_exclude.find(_v) != _settings_exclude.end();
+        bool   _b = std::regex_match(_v, std::regex(_settings_rexclude_exact));
+        bool   _c = std::regex_match(_v, std::regex(_settings_rexclude_begin));
+        return (_a || _b || _c);
+    };
+
    _setting_output.erase(std::remove_if(_setting_output.begin(), _setting_output.end(),
-                                         [&_settings_exclude](const auto& itr) {
-                                             return _settings_exclude.find(
-                                                        itr.find("environ")->second) !=
-                                                    _settings_exclude.end();
-                                         }),
+                                         _remove_conditions),
                          _setting_output.end());

    if(alphabetical)
@@ -23,6 +23,7 @@
 #include "library.hpp"
 #include "library/components/fork_gotcha.hpp"
 #include "library/components/mpi_gotcha.hpp"
+#include "library/components/rocm_smi.hpp"
 #include "library/config.hpp"
 #include "library/critical_trace.hpp"
 #include "library/debug.hpp"
@@ -31,6 +32,7 @@
 #include "library/sampling.hpp"
 #include "library/thread_data.hpp"
 #include "library/timemory.hpp"
+#include "timemory/mpl/type_traits.hpp"

 #include <mutex>
 #include <string_view>
@@ -58,8 +60,15 @@ ensure_finalization(bool _static_init = false)
    else
    {
        OMNITRACE_CONDITIONAL_PRINT(get_debug_env(), "[%s]\n", __FUNCTION__);
+        // This environment variable forces the ROCR-Runtime to use polling to wait
+        // for signals rather than interrupts. We set this variable to avoid issues with
+        // rocm/roctracer hanging when interrupted by the sampler
+        //
+        // see:
+        // https://github.com/ROCm-Developer-Tools/roctracer/issues/22#issuecomment-572814465
+        tim::set_env("HSA_ENABLE_INTERRUPT", "0", 0);
    }
-    return scope::destructor{ []() { omnitrace_trace_finalize(); } };
+    return scope::destructor{ []() { omnitrace_finalize(); } };
 }

 auto&
@@ -105,40 +114,271 @@ using Device = critical_trace::Device;
 using Phase  = critical_trace::Phase;

 bool
-omnitrace_init_tooling()
-{
-    static bool _once = false;
-    if(get_state() != State::PreInit || _once) return false;
-    _once = true;
+omnitrace_init_tooling();
+}  // namespace

+//======================================================================================//
+///
+/// \fn void omnitrace_push_trace(const char* name)
+/// \brief the "start" function for an instrumentation region
+///
+//======================================================================================//
+
+extern "C" void
+omnitrace_push_trace(const char* name)
+{
+    // return if not active
+    if(get_state() == State::Finalized) return;
+
+    if(get_state() != State::Active && !omnitrace_init_tooling())
+    {
+        static auto _debug = get_debug_env();
+        OMNITRACE_CONDITIONAL_BASIC_PRINT(
+            _debug, "[%s] %s :: not active and perfetto not initialized\n", __FUNCTION__,
+            name);
+        return;
+    }
+
+    OMNITRACE_DEBUG("[%s] %s\n", __FUNCTION__, name);
+
+    static auto _sample_rate = std::max<size_t>(get_instrumentation_interval(), 1);
+    static thread_local size_t _sample_idx = 0;
+    auto                       _enabled    = (_sample_idx++ % _sample_rate == 0);
+    get_interval_data().emplace_back(_enabled);
+    if(_enabled) get_functors().first(name);
+    if(get_use_critical_trace())
+    {
+        auto     _ts     = comp::wall_clock::record();
+        auto     _cid    = get_cpu_cid()++;
+        uint16_t _depth  = (get_cpu_cid_stack()->empty())
+                               ? get_cpu_cid_stack(0)->size()
+                               : get_cpu_cid_stack()->size() - 1;
+        auto _parent_cid = (get_cpu_cid_stack()->empty()) ? get_cpu_cid_stack(0)->back()
+                                                          : get_cpu_cid_stack()->back();
+        get_cpu_cid_parents().emplace(_cid, std::make_tuple(_parent_cid, _depth));
+        add_critical_trace<Device::CPU, Phase::BEGIN>(
+            threading::get_id(), _cid, 0, _parent_cid, _ts, 0,
+            critical_trace::add_hash_id(name), _depth);
+    }
+}
+
+//======================================================================================//
+///
+/// \fn void omnitrace_pop_trace(const char* name)
+/// \brief the "stop" function for an instrumentation region
+///
+//======================================================================================//
+
+extern "C" void
+omnitrace_pop_trace(const char* name)
+{
+    if(get_state() == State::Active)
+    {
+        OMNITRACE_DEBUG("[%s] %s\n", __FUNCTION__, name);
+        auto& _interval_data = get_interval_data();
+        if(!_interval_data.empty())
+        {
+            if(_interval_data.back()) get_functors().second(name);
+            _interval_data.pop_back();
+        }
+        if(get_use_critical_trace())
+        {
+            if(get_cpu_cid_stack() && !get_cpu_cid_stack()->empty())
+            {
+                auto _cid = get_cpu_cid_stack()->back();
+                if(get_cpu_cid_parents().find(_cid) != get_cpu_cid_parents().end())
+                {
+                    uint64_t _parent_cid          = 0;
+                    uint16_t _depth               = 0;
+                    auto     _ts                  = comp::wall_clock::record();
+                    std::tie(_parent_cid, _depth) = get_cpu_cid_parents().at(_cid);
+                    add_critical_trace<Device::CPU, Phase::END>(
+                        threading::get_id(), _cid, 0, _parent_cid, _ts, _ts,
+                        critical_trace::add_hash_id(name), _depth);
+                }
+            }
+        }
+    }
+    else
+    {
+        static auto _debug = get_debug_env();
+        OMNITRACE_CONDITIONAL_BASIC_PRINT(_debug, "[%s] %s ignored :: not active\n",
+                                          __FUNCTION__, name);
+    }
+}
+
+//======================================================================================//
+///
+/// \fn void omnitrace_set_env(const char* name, const char* env_val)
+/// \brief Sets an initial environment variable
+///
+//======================================================================================//
+
+extern "C" void
+omnitrace_set_env(const char* env_name, const char* env_val)
+{
+    // just search env to avoid initializing the settings
+    OMNITRACE_CONDITIONAL_PRINT(get_debug_env() || get_verbose_env() > 2,
+                                "[%s] Setting env: %s=%s\n", __FUNCTION__, env_name,
+                                env_val);
+
+    tim::set_env(env_name, env_val, 0);
+
+    OMNITRACE_CONDITIONAL_THROW(
+        get_state() >= State::Init &&
+            (config::get_is_continuous_integration() || get_debug_env()),
+        "%s(\"%s\", \"%s\") called after omnitrace was initialized. state = %s",
+        __FUNCTION__, env_name, env_val, std::to_string(get_state()).c_str());
+}
+
+//======================================================================================//
+///
+/// \fn void omnitrace_set_mpi(bool use, bool attached)
+/// \brief Configures whether MPI support should be activated
+///
+//======================================================================================//
+
+namespace
+{
+bool                  _set_mpi_called        = false;
+std::function<void()> _start_gotcha_callback = []() {};
+}  // namespace
+
+extern "C" void
+omnitrace_set_mpi(bool use, bool attached)
+{
+    // just search env to avoid initializing the settings
+    OMNITRACE_CONDITIONAL_PRINT(get_debug_env() || get_verbose_env() > 2,
+                                "[%s] use: %s, attached: %s\n", __FUNCTION__,
+                                (use) ? "y" : "n", (attached) ? "y" : "n");
+
+    _set_mpi_called       = true;
+    config::is_attached() = attached;
+
+    if(use && !attached &&
+       (get_state() == State::PreInit || get_state() == State::DelayedInit))
+    {
+        tim::set_env("OMNITRACE_USE_PID", "ON", 1);
+    }
+    else if(!use)
+    {
+        trait::runtime_enabled<mpi_gotcha_t>::set(false);
+    }
+
+    OMNITRACE_CONDITIONAL_THROW(
+        get_state() >= State::Init &&
+            (config::get_is_continuous_integration() || get_debug_env()),
+        "%s(use=%s, attached=%s) called after omnitrace was initialized. state = %s",
+        __FUNCTION__, std::to_string(use).c_str(), std::to_string(attached).c_str(),
+        std::to_string(get_state()).c_str());
+
+    _start_gotcha_callback();
+}
+
+//======================================================================================//
+
+extern "C" void
+omnitrace_init_library()
+{
    auto _tid = threading::get_id();
    (void) _tid;

-    auto _mode = tim::get_env<std::string>("OMNITRACE_MODE", "");
-    OMNITRACE_CONDITIONAL_BASIC_PRINT(true, "Instrumentation mode: %s\n", _mode.c_str());
+    auto _mode = get_mode();
+
+    get_state() = State::Init;

    // configure the settings
    configure_settings();

-    if(gpu::device_count() == 0)
+    auto _debug_init  = get_debug_init();
+    auto _debug_value = get_debug();
+    if(_debug_init) config::set_setting_value("OMNITRACE_DEBUG", true);
+    scope::destructor _debug_dtor{ [_debug_value, _debug_init]() {
+        if(_debug_init) config::set_setting_value("OMNITRACE_DEBUG", _debug_value);
+    } };
+
+    OMNITRACE_DEBUG("[%s]\n", __FUNCTION__);
+
+    // below will effectively do:
+    //      get_cpu_cid_stack(0)->emplace_back(-1);
+    // plus query some env variables
+    add_critical_trace<Device::CPU, Phase::NONE>(0, -1, 0, 0, 0, 0, 0, 0);
+
+    if(gpu::device_count() == 0 && get_state() != State::Active)
    {
-        OMNITRACE_DEBUG("No HIP devices were found: disabling roctracer...\n");
+        OMNITRACE_DEBUG(
+            "No HIP devices were found: disabling roctracer and rocm_smi...\n");
        get_use_roctracer() = false;
+        get_use_rocm_smi()  = false;
    }

-    if(_mode == "sampling")
+    if(_mode == Mode::Sampling)
    {
-        OMNITRACE_PRINT(
-            "Disabling perfetto, timemory, and critical trace in %s mode...\n",
-            _mode.c_str());
+        OMNITRACE_CONDITIONAL_PRINT(get_verbose() >= 0,
+                                    "Disabling critical trace in %s mode...\n",
+                                    std::to_string(_mode).c_str());
        get_use_sampling()       = true;
-        get_use_timemory()       = false;
-        get_use_perfetto()       = false;
-        get_use_roctracer()      = false;
        get_use_critical_trace() = false;
    }

+    tim::trait::runtime_enabled<comp::roctracer>::set(get_use_roctracer());
+    tim::trait::runtime_enabled<comp::roctracer_data>::set(get_use_roctracer());
+
+    if(get_instrumentation_interval() < 1) get_instrumentation_interval() = 1;
+    get_interval_data().reserve(512);
+
+    if(get_use_kokkosp())
+    {
+        auto _force = 0;
+        if(tim::get_env<std::string>("KOKKOS_PROFILE_LIBRARY") == "libtimemory.so")
+            _force = 1;
+        tim::set_env("KOKKOS_PROFILE_LIBRARY", "libomnitrace.so", _force);
+    }
+
+#if defined(OMNITRACE_USE_ROCTRACER)
+    tim::set_env("HSA_TOOLS_LIB", "libomnitrace.so", 0);
+#endif
+}
+
+//======================================================================================//
+
+namespace
+{
+bool
+omnitrace_init_tooling()
+{
+    static bool _once = false;
+    if(get_state() != State::PreInit || get_state() == State::Init || _once) return false;
+    _once = true;
+
+    OMNITRACE_CONDITIONAL_THROW(
+        get_state() == State::Init,
+        "%s called after omnitrace_init_library() was explicitly called", __FUNCTION__);
+
+    OMNITRACE_CONDITIONAL_BASIC_PRINT(get_verbose_env() >= 0,
+                                      "Instrumentation mode: %s\n",
+                                      std::to_string(config::get_mode()).c_str());
+
+    if(get_verbose_env() >= 0) print_banner();
+
+    omnitrace_init_library();
+
+    OMNITRACE_DEBUG("[%s]\n", __FUNCTION__);
+
    auto _dtor = scope::destructor{ []() {
+        // if roctracer is not enabled, we cannot rely on OnLoad(...) via HSA tools
+        // to setup rocm-smi
+        if constexpr(!trait::is_available<comp::roctracer>::value)
+        {
+            try
+            {
+                rocm_smi::setup();
+            } catch(std::exception& _e)
+            {
+                OMNITRACE_PRINT("Exception: %s\n", _e.what());
+            }
+        }
+
        if(get_use_sampling())
        {
            pthread_gotcha::enable_sampling_on_child_threads() = false;
@@ -146,6 +386,8 @@ omnitrace_init_tooling()
            pthread_gotcha::enable_sampling_on_child_threads() = true;
            sampling::unblock_signals();
        }
+        get_main_bundle()->start();
+        get_state()= State::Active;  // set to active as very last operation
    } };

    if(get_use_sampling())
@@ -154,9 +396,8 @@ omnitrace_init_tooling()
        sampling::block_signals();
    }

-    OMNITRACE_DEBUG("[%s]\n", __FUNCTION__);
-
-    if(!get_use_timemory() && !get_use_perfetto() && !get_use_sampling())
+    if(!get_use_timemory() && !get_use_perfetto() && !get_use_sampling() &&
+       !get_use_rocm_smi())
    {
        get_state() = State::Finalized;
        OMNITRACE_DEBUG("[%s] Both perfetto and timemory are disabled. Setting the state "
@@ -165,16 +406,6 @@ omnitrace_init_tooling()
        return false;
    }

-    // below will effectively do:
-    //      get_cpu_cid_stack(0)->emplace_back(-1);
-    // plus query some env variables
-    add_critical_trace<Device::CPU, Phase::NONE>(0, -1, 0, 0, 0, 0, 0, 0);
-
-    tim::trait::runtime_enabled<comp::roctracer>::set(get_use_roctracer());
-
-    if(get_instrumentation_interval() < 1) get_instrumentation_interval() = 1;
-    get_interval_data().reserve(512);
-
    if(get_use_timemory())
    {
        comp::user_global_bundle::global_init();
@@ -204,17 +435,6 @@ omnitrace_init_tooling()
        }
    }

-    auto& _main_bundle = get_main_bundle();
-    _main_bundle->start();
-
-#if defined(OMNITRACE_USE_ROCTRACER)
-    if(get_use_roctracer())
-    {
-        assert(_main_bundle->get<comp::roctracer>() != nullptr);
-        assert(_main_bundle->get<comp::roctracer>()->get_is_running());
-    }
-#endif
-
    perfetto::TracingInitArgs               args{};
    perfetto::TraceConfig                   cfg{};
    perfetto::protos::gen::TrackEventConfig track_event_cfg{};
@@ -248,17 +468,20 @@ omnitrace_init_tooling()

    auto        _exe         = get_exe_name();
    static auto _thread_init = [_exe]() {
-        omnitrace_thread_data<omnitrace_thread_bundle_t>::construct(
-            TIMEMORY_JOIN("", _exe, "/thread-", threading::get_id()),
-            quirk::config<quirk::auto_start>{});
-        if(get_use_sampling())
-        {
-            static thread_local auto _once = std::once_flag{};
-            std::call_once(_once, sampling::setup);
-        }
+        static thread_local auto _thread_setup = [_exe]() {
+            if(threading::get_id() > 0)
+                threading::set_thread_name(
+                    TIMEMORY_JOIN(" ", "Thread", threading::get_id()).c_str());
+            thread_data<omnitrace_thread_bundle_t>::construct(
+                TIMEMORY_JOIN("", _exe, "/thread-", threading::get_id()),
+                quirk::config<quirk::auto_start>{});
+            if(get_use_sampling()) sampling::setup();
+        };
+        static thread_local auto _once = std::once_flag{};
+        std::call_once(_once, _thread_setup);
        static thread_local auto _dtor = scope::destructor{ []() {
            if(get_use_sampling()) sampling::shutdown();
-            omnitrace_thread_data<omnitrace_thread_bundle_t>::instance()->stop();
+            thread_data<omnitrace_thread_bundle_t>::instance()->stop();
        } };
        (void) _dtor;
    };
@@ -318,58 +541,6 @@ omnitrace_init_tooling()
        get_functors().second = _pop_timemory;
    }

-    if(dmp::rank() == 0)
-    {
-        static std::set<tim::string_view_t> _sample_options = {
-            "OMNITRACE_SAMPLING_FREQ", "OMNITRACE_SAMPLING_DELAY",
-            "OMNITRACE_FLAT_SAMPLING", "OMNITRACE_TIMELINE_SAMPLING",
-            "OMNITRACE_FLAT_SAMPLING", "OMNITRACE_TIMELINE_SAMPLING",
-        };
-        static std::set<tim::string_view_t> _perfetto_options = {
-            "OMNITRACE_OUTPUT_FILE",
-            "OMNITRACE_BACKEND",
-            "OMNITRACE_SHMEM_SIZE_HINT_KB",
-            "OMNITRACE_BUFFER_SIZE_KB",
-        };
-        static std::set<tim::string_view_t> _timemory_options = {
-            "OMNITRACE_ROCTRACER_FLAT_PROFILE", "OMNITRACE_ROCTRACER_TIMELINE_PROFILE"
-        };
-        // generic filter for filtering relevant options
-        auto _is_omnitrace_option = [](const auto& _v, const auto& _c) {
-            if(!get_use_roctracer() && _v.find("OMNITRACE_ROCTRACER_") == 0) return false;
-            if(!get_use_critical_trace() && _v.find("OMNITRACE_CRITICAL_TRACE_") == 0)
-                return false;
-            if(!get_use_perfetto() && _perfetto_options.count(_v) > 0) return false;
-            if(!get_use_timemory() && _timemory_options.count(_v) > 0) return false;
-            if(!get_use_sampling() && _sample_options.count(_v) > 0) return false;
-            const auto npos = std::string::npos;
-            if(_v.find("WIDTH") != npos || _v.find("SEPARATOR_FREQ") != npos ||
-               _v.find("AUTO_OUTPUT") != npos || _v.find("DART_OUTPUT") != npos ||
-               _v.find("FILE_OUTPUT") != npos || _v.find("PLOT_OUTPUT") != npos ||
-               _v.find("FLAMEGRAPH_OUTPUT") != npos)
-                return false;
-            if(!_c.empty())
-            {
-                if(_c.find("omnitrace") != _c.end()) return true;
-                if(_c.find("debugging") != _c.end() && _v.find("DEBUG") != npos)
-                    return true;
-                if(_c.find("config") != _c.end()) return true;
-                if(_c.find("dart") != _c.end()) return false;
-                if(_c.find("io") != _c.end() && _v.find("_OUTPUT") != npos) return true;
-                if(_c.find("format") != _c.end()) return true;
-                return false;
-            }
-            return (_v.find("OMNITRACE_") == 0);
-        };
-
-        if(tim::settings::verbose() > 0)
-            tim::print_env(std::cerr, [_is_omnitrace_option](const std::string& _v) {
-                return _is_omnitrace_option(_v, std::set<std::string>{});
-            });
-
-        print_config_settings(std::cerr, _is_omnitrace_option);
-    }
-
    if(get_use_perfetto() && !is_system_backend())
    {
 #if defined(CUSTOM_DATA_SOURCE)
@@ -392,126 +563,78 @@ omnitrace_init_tooling()
        tracing_session->StartBlocking();
    }

-    get_state() = State::Active;
-
    // if static objects are destroyed in the inverse order of when they are
    // created this should ensure that finalization is called before perfetto
    // ends the tracing session
    static auto _ensure_finalization = ensure_finalization();

-    if(dmp::rank() == 0) puts("");
+    if(dmp::rank() == 0 && get_verbose() >= 0) fprintf(stderr, "\n");

    return true;
 }
 }  // namespace

-//--------------------------------------------------------------------------------------//
+//======================================================================================//

 extern "C" void
-omnitrace_push_trace(const char* name)
+omnitrace_init(const char* _mode, bool _is_binary_rewrite, const char* _argv0)
 {
-    // return if not active
-    if(get_state() == State::Finalized) return;
+    // always the first
+    std::atexit(&omnitrace_finalize);

-    if(get_state() != State::Active && !omnitrace_init_tooling())
+    OMNITRACE_CONDITIONAL_BASIC_PRINT(
+        get_debug_env() || get_verbose_env() > 2,
+        "[%s] mode: %s | is binary rewrite: %s | command: %s\n", __FUNCTION__, _mode,
+        (_is_binary_rewrite) ? "y" : "n", _argv0);
+
+    tim::set_env("OMNITRACE_MODE", _mode, 0);
+    config::is_binary_rewrite() = _is_binary_rewrite;
+
+    if(!_set_mpi_called)
    {
-        OMNITRACE_DEBUG("[%s] %s :: not active and perfetto not initialized\n",
-                        __FUNCTION__, name);
-        return;
+        _start_gotcha_callback = []() { get_gotcha_bundle()->start(); };
    }
    else
    {
-        OMNITRACE_DEBUG("[%s] %s\n", __FUNCTION__, name);
-    }
-
-    static auto _sample_rate = std::max<size_t>(get_instrumentation_interval(), 1);
-    static thread_local size_t _sample_idx = 0;
-    auto                       _enabled    = (_sample_idx++ % _sample_rate == 0);
-    get_interval_data().emplace_back(_enabled);
-    if(_enabled) get_functors().first(name);
-    if(get_use_critical_trace())
-    {
-        auto     _ts     = comp::wall_clock::record();
-        auto     _cid    = get_cpu_cid()++;
-        uint16_t _depth  = (get_cpu_cid_stack()->empty())
-                               ? get_cpu_cid_stack(0)->size()
-                               : get_cpu_cid_stack()->size() - 1;
-        auto _parent_cid = (get_cpu_cid_stack()->empty()) ? get_cpu_cid_stack(0)->back()
-                                                          : get_cpu_cid_stack()->back();
-        get_cpu_cid_parents().emplace(_cid, std::make_tuple(_parent_cid, _depth));
-        add_critical_trace<Device::CPU, Phase::BEGIN>(
-            threading::get_id(), _cid, 0, _parent_cid, _ts, 0,
-            critical_trace::add_hash_id(name), _depth);
+        get_gotcha_bundle()->start();
    }
 }

-extern "C" void
-omnitrace_pop_trace(const char* name)
-{
-    if(get_state() == State::Active)
-    {
-        OMNITRACE_DEBUG("[%s] %s\n", __FUNCTION__, name);
-        auto& _interval_data = get_interval_data();
-        if(!_interval_data.empty())
-        {
-            if(_interval_data.back()) get_functors().second(name);
-            _interval_data.pop_back();
-        }
-        if(get_use_critical_trace())
-        {
-            if(get_cpu_cid_stack() && !get_cpu_cid_stack()->empty())
-            {
-                auto _cid = get_cpu_cid_stack()->back();
-                if(get_cpu_cid_parents().find(_cid) != get_cpu_cid_parents().end())
-                {
-                    uint64_t _parent_cid          = 0;
-                    uint16_t _depth               = 0;
-                    auto     _ts                  = comp::wall_clock::record();
-                    std::tie(_parent_cid, _depth) = get_cpu_cid_parents().at(_cid);
-                    add_critical_trace<Device::CPU, Phase::END>(
-                        threading::get_id(), _cid, 0, _parent_cid, _ts, _ts,
-                        critical_trace::add_hash_id(name), _depth);
-                }
-            }
-        }
-    }
-    else
-    {
-        OMNITRACE_DEBUG("[%s] %s :: not active\n", __FUNCTION__, name);
-    }
-}
+//======================================================================================//

 extern "C" void
-omnitrace_trace_init(const char* _info, bool _b, const char* _extra)
-{
-    OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug_env(), "[%s] %s | %s | %s\n",
-                                      __FUNCTION__, _info, (_b) ? "y" : "n", _extra);
-    auto& _gotcha_bundle = get_gotcha_bundle();
-    (void) _gotcha_bundle;
-}
-
-extern "C" void
-omnitrace_trace_finalize(void)
+omnitrace_finalize(void)
 {
    // return if not active
    if(get_state() != State::Active) return;

+    pthread_gotcha::enable_sampling_on_child_threads() = false;
+
+    auto _debug_init  = get_debug_finalize();
+    auto _debug_value = get_debug();
+    if(_debug_init) config::set_setting_value("OMNITRACE_DEBUG", true);
+    scope::destructor _debug_dtor{ [_debug_value, _debug_init]() {
+        if(_debug_init) config::set_setting_value("OMNITRACE_DEBUG", _debug_value);
+    } };
+
    OMNITRACE_DEBUG("[%s]\n", __FUNCTION__);

-    if(dmp::rank() == 0) puts("");
+    auto& _thread_bundle = thread_data<omnitrace_thread_bundle_t>::instance();
+    if(_thread_bundle) _thread_bundle->stop();
+
+    if(dmp::rank() == 0 && get_verbose() >= 0) fprintf(stderr, "\n");
+    if(get_verbose_env() > 0) config::print_settings();

    get_state() = State::Finalized;

    if(get_use_sampling())
    {
        OMNITRACE_DEBUG("[%s] Shutting down sampling...\n", __FUNCTION__);
-        pthread_gotcha::enable_sampling_on_child_threads() = false;
        sampling::shutdown();
        sampling::block_signals();
    }

    OMNITRACE_DEBUG("[%s] Stopping gotcha bundle...\n", __FUNCTION__);
-
    // stop the gotcha bundle
    if(get_gotcha_bundle())
    {
@@ -519,11 +642,19 @@ omnitrace_trace_finalize(void)
        get_gotcha_bundle().reset();
    }

-#if defined(OMNITRACE_USE_ROCTRACER)
+    pthread_gotcha::shutdown();
+
+    if(get_use_rocm_smi())
+    {
+        OMNITRACE_DEBUG("[%s] Shutting down rocm-smi sampling...\n", __FUNCTION__);
+        rocm_smi::shutdown();
+    }
+
    OMNITRACE_DEBUG("[%s] Shutting down roctracer...\n", __FUNCTION__);
    // ensure that threads running roctracer callbacks shutdown
-    if(get_use_roctracer()) comp::roctracer::tear_down();
-#endif
+    comp::roctracer::shutdown();
+
+    if(dmp::rank() == 0) fprintf(stderr, "\n");

    OMNITRACE_DEBUG("[%s] Stopping main bundle...\n", __FUNCTION__);
    // stop the main bundle and report the high-level metrics
@@ -551,7 +682,7 @@ omnitrace_trace_finalize(void)
    // thread-specific data will be wrong if try to stop them from
    // the main thread.
    OMNITRACE_DEBUG("[%s] Destroying thread bundle data...\n", __FUNCTION__);
-    for(auto& itr : omnitrace_thread_data<omnitrace_thread_bundle_t>::instances())
+    for(auto& itr : thread_data<omnitrace_thread_bundle_t>::instances())
    {
        if(itr && itr->get<comp::wall_clock>() &&
           !itr->get<comp::wall_clock>()->get_is_running())
@@ -559,7 +690,7 @@ omnitrace_trace_finalize(void)
            std::string _msg = JOIN("", *itr);
            auto        _pos = _msg.find(">>>  ");
            if(_pos != std::string::npos) _msg = _msg.substr(_pos + 5);
-            OMNITRACE_PRINT("%s\n", _msg.c_str());
+            OMNITRACE_CONDITIONAL_PRINT(get_verbose() >= 0, "%s\n", _msg.c_str());
        }
    }

@@ -590,7 +721,7 @@ omnitrace_trace_finalize(void)
        }
    }

-    if(get_use_critical_trace())
+    if(get_use_critical_trace() || (get_use_rocm_smi() && get_use_roctracer()))
    {
        OMNITRACE_DEBUG("[%s] Generating the critical trace...\n", __FUNCTION__);
        // increase the thread-pool size
@@ -600,7 +731,7 @@ omnitrace_trace_finalize(void)
        for(size_t i = 0; i < max_supported_threads; ++i)
        {
            using critical_trace_hash_data =
-                omnitrace_thread_data<critical_trace::hash_ids, critical_trace::id>;
+                thread_data<critical_trace::hash_ids, critical_trace::id>;

            if(critical_trace_hash_data::instances().at(i))
                critical_trace::add_hash_id(*critical_trace_hash_data::instances().at(i));
@@ -608,12 +739,34 @@ omnitrace_trace_finalize(void)

        for(size_t i = 0; i < max_supported_threads; ++i)
        {
-            using critical_trace_chain_data =
-                omnitrace_thread_data<critical_trace::call_chain>;
+            using critical_trace_chain_data = thread_data<critical_trace::call_chain>;

            if(critical_trace_chain_data::instances().at(i))
                critical_trace::update(i);  // launch update task
        }
+    }
+
+    // ensure that all the MT instances are flushed
+    if(get_use_rocm_smi())
+    {
+        size_t _ndevice = gpu::device_count();
+        OMNITRACE_CONDITIONAL_PRINT(
+            get_debug() || get_verbose() > 0,
+            "[%s] Post-processing rocm-smi samples for %zu devices...\n", __FUNCTION__,
+            _ndevice);
+        for(size_t i = 0; i < _ndevice; ++i)
+        {
+            rocm_smi::data::post_process(i);
+            rocm_smi::sampler_instances::instances().at(i).reset();
+        }
+    }
+
+    if(get_use_critical_trace())
+    {
+        OMNITRACE_DEBUG("[%s] Generating the critical trace...\n", __FUNCTION__);
+        // increase the thread-pool size
+        tasking::get_critical_trace_thread_pool().initialize_threadpool(
+            get_critical_trace_num_threads());

        // make sure outstanding hash tasks completed before compute
        OMNITRACE_PRINT("[%s] waiting for all critical trace tasks to complete...\n",
@@ -630,7 +783,10 @@ omnitrace_trace_finalize(void)
    bool _perfetto_output_error = false;
    if(get_use_perfetto() && !is_system_backend())
    {
-        OMNITRACE_DEBUG("[%s] Flushing perfetto...\n", __FUNCTION__);
+        if(get_verbose() >= 0) fprintf(stderr, "\n");
+        if(get_verbose() >= 0 || get_debug())
+            fprintf(stderr, "[%s]|%i> Flushing perfetto...\n", __FUNCTION__, dmp::rank());
+
        // Make sure the last event is closed for this example.
        perfetto::TrackEvent::Flush();

@@ -650,14 +806,12 @@ omnitrace_trace_finalize(void)
            return;
        }
        // Write the trace into a file.
-        fprintf(stderr,
-                "[%s]> Outputting '%s'. Trace data: %lu B (%.2f KB / %.2f MB / %.2f "
-                "GB)... ",
-                __FUNCTION__, get_perfetto_output_filename().c_str(),
-                (unsigned long) trace_data.size(),
-                static_cast<double>(trace_data.size()) / units::KB,
-                static_cast<double>(trace_data.size()) / units::MB,
-                static_cast<double>(trace_data.size()) / units::GB);
+        if(get_verbose() >= 0)
+            fprintf(stderr, "[%s]|%i> Outputting '%s' (%.2f KB / %.2f MB / %.2f GB)... ",
+                    __FUNCTION__, dmp::rank(), get_perfetto_output_filename().c_str(),
+                    static_cast<double>(trace_data.size()) / units::KB,
+                    static_cast<double>(trace_data.size()) / units::MB,
+                    static_cast<double>(trace_data.size()) / units::GB);
        std::ofstream ofs{};
        if(!tim::filepath::open(ofs, get_perfetto_output_filename(),
                                std::ios::out | std::ios::binary))
@@ -669,10 +823,11 @@ omnitrace_trace_finalize(void)
        else
        {
            // Write the trace into a file.
-            fprintf(stderr, "Done\n");
+            if(get_verbose() >= 0) fprintf(stderr, "Done\n");
            ofs.write(&trace_data[0], trace_data.size());
        }
        ofs.close();
+        if(get_verbose() >= 0) fprintf(stderr, "\n");
    }

    // these should be destroyed before timemory is finalized, especially the
@@ -691,33 +846,14 @@ omnitrace_trace_finalize(void)
    OMNITRACE_DEBUG("[%s] Finalizing timemory... Done\n", __FUNCTION__);

    if(_perfetto_output_error)
-        throw std::runtime_error("Unable to create perfetto output file");
-}
-
-extern "C" void
-omnitrace_trace_set_env(const char* env_name, const char* env_val)
-{
-    // just search env to avoid initializing the settings
-    OMNITRACE_CONDITIONAL_PRINT(get_debug_env(), "[%s] Setting env: %s=%s\n",
-                                __FUNCTION__, env_name, env_val);
-
-    tim::set_env(env_name, env_val, 0);
-}
-
-extern "C" void
-omnitrace_trace_set_mpi(bool use, bool attached)
-{
-    // just search env to avoid initializing the settings
-    OMNITRACE_CONDITIONAL_PRINT(get_debug_env(), "[%s] use: %s, attached: %s\n",
-                                __FUNCTION__, (use) ? "y" : "n", (attached) ? "y" : "n");
-    if(use && !attached &&
-       (get_state() == State::PreInit || get_state() == State::DelayedInit))
    {
-        tim::set_env("OMNITRACE_USE_PID", "ON", 1);
-        get_state() = State::DelayedInit;
+        OMNITRACE_THROW("Error opening perfetto output file: %s",
+                        get_perfetto_output_filename().c_str());
    }
 }

+//======================================================================================//
+
 namespace
 {
 // if static objects are destroyed randomly (relatively uncommon behavior)
@@ -21,8 +21,10 @@
 // SOFTWARE.

 #include "library/components/fwd.hpp"
+#include "library/components/rocm_smi.hpp"
 #include "library/config.hpp"
 #include "library/debug.hpp"
+#include "library/perfetto.hpp"
 #include "library/ptl.hpp"
 #include "library/sampling.hpp"

@@ -44,6 +46,7 @@
 #include <timemory/sampling/allocator.hpp>
 #include <timemory/sampling/sampler.hpp>
 #include <timemory/storage.hpp>
+#include <timemory/units.hpp>
 #include <timemory/utility/backtrace.hpp>
 #include <timemory/utility/demangle.hpp>
 #include <timemory/utility/types.hpp>
@@ -99,10 +102,10 @@ namespace omnitrace
 namespace component
 {
 using hw_counters               = typename backtrace::hw_counters;
-using signal_type_instances     = omnitrace_thread_data<std::set<int>, api::sampling>;
-using backtrace_init_instances  = omnitrace_thread_data<backtrace, api::sampling>;
-using sampler_running_instances = omnitrace_thread_data<bool, api::sampling>;
-using papi_vector_instances     = omnitrace_thread_data<hw_counters, api::sampling>;
+using signal_type_instances     = thread_data<std::set<int>, api::sampling>;
+using backtrace_init_instances  = thread_data<backtrace, api::sampling>;
+using sampler_running_instances = thread_data<bool, api::sampling>;
+using papi_vector_instances     = thread_data<hw_counters, api::sampling>;

 namespace
 {
@@ -154,9 +157,31 @@ backtrace::preinit()
    sampling_cpu_clock::label()       = "sampling_cpu_clock";
    sampling_cpu_clock::description() = "CPU clock time (via sampling)";

-    sampling_percent::label()        = "sampling_percent";
-    sampling_percent::description()  = "Percentage of samples";
-    sampling_percent::display_unit() = "%";
+    sampling_percent::label()       = "sampling_percent";
+    sampling_percent::description() = "Percentage of samples";
+
+    sampling_gpu_busy::label()       = "sampling_gpu_busy_percent";
+    sampling_gpu_busy::description() = "Utilization of GPU(s)";
+    sampling_gpu_busy::set_precision(0);
+    sampling_gpu_busy::set_format_flags(sampling_gpu_busy::get_format_flags() &
+                                        std::ios_base::showpoint);
+
+    sampling_gpu_memory::label()       = "sampling_gpu_memory_usage";
+    sampling_gpu_memory::description() = "Memory usage of GPU(s)";
+
+    sampling_gpu_power::label()        = "sampling_gpu_power";
+    sampling_gpu_power::description()  = "Power usage of GPU(s)";
+    sampling_gpu_power::unit()         = units::watt;
+    sampling_gpu_power::display_unit() = "watts";
+    sampling_gpu_power::set_precision(2);
+    sampling_gpu_power::set_format_flags(sampling_gpu_power::get_format_flags());
+
+    sampling_gpu_temp::label()        = "sampling_gpu_temperature";
+    sampling_gpu_temp::description()  = "Temperature of GPU(s)";
+    sampling_gpu_temp::unit()         = 1;
+    sampling_gpu_temp::display_unit() = "degC";
+    sampling_gpu_temp::set_precision(1);
+    sampling_gpu_temp::set_format_flags(sampling_gpu_temp::get_format_flags());
 }

 std::string
@@ -236,6 +261,7 @@ backtrace::sample(int signum)
    m_tid        = threading::get_id();
    m_ts         = clock_type::now();
    m_thr_cpu_ts = tim::get_clock_thread_now<int64_t, std::nano>();
+    m_mem_peak   = tim::get_peak_rss(RUSAGE_THREAD);
    m_data       = tim::get_unw_backtrace<128, 4, false>();
    auto* itr    = m_data.begin();
    for(; itr != m_data.end(); ++itr, ++m_size)
@@ -295,11 +321,12 @@ backtrace::configure(bool _setup, int64_t _tid)
        sampling::block_signals(*_signal_types);
        if constexpr(tim::trait::is_available<hw_counters>::value)
        {
+            perfetto_counter_track<hw_counters>::init();
            OMNITRACE_DEBUG("HW COUNTER: starting...\n");
            if(get_papi_vector(_tid)) get_papi_vector(_tid)->start();
        }

-        auto _alrm_freq = 1.0 / std::min<double>(get_sampling_freq(), 10.0);
+        auto _alrm_freq = 1.0 / std::min<double>(get_sampling_freq(), 5.0);
        auto _prof_freq = 1.0 / get_sampling_freq();
        auto _delay     = std::max<double>(1.0e-3, get_sampling_delay());

@@ -312,10 +339,14 @@ backtrace::configure(bool _setup, int64_t _tid)
        _sampler->set_delay(_delay);
        _sampler->set_frequency(_prof_freq, { SIGPROF });
        _sampler->set_frequency(_alrm_freq, { SIGALRM });
+
        static_assert(tim::trait::buffer_size<sampling::sampler_t>::value > 0,
                      "Error! Zero buffer size");
-        if(_sampler->get_buffer_size() == 0)
-            throw std::runtime_error("dynamic sampler has a zero buffer size");
+
+        OMNITRACE_CONDITIONAL_THROW(
+            _sampler->get_buffer_size() <= 0,
+            "dynamic sampler requires a positive buffer size: %zu",
+            _sampler->get_buffer_size());

        OMNITRACE_DEBUG("Sampler for thread %lu will be triggered %5.1fx per second "
                        "(every %5.2e seconds)...\n",
@@ -365,6 +396,8 @@ backtrace::get_last_hwcounters()
 void
 backtrace::post_process(int64_t _tid)
 {
+    namespace quirk = tim::quirk;
+
    configure(false, _tid);

    auto& _sampler = sampling::sampler_instances::instances().at(_tid);
@@ -392,9 +425,12 @@ backtrace::post_process(int64_t _tid)
        // debugging feature
        static bool _keep_internal =
            tim::get_env<bool>("OMNITRACE_SAMPLING_KEEP_INTERNAL", get_debug());
-        if(_keep_internal) return 1;
        const auto _npos = std::string::npos;
+        if(_keep_internal) return 1;
        if(_lbl.find("omnitrace_init_tooling") != _npos) return -1;
+        if(_lbl.find("omnitrace_push_trace") != _npos) return -1;
+        if(_lbl.find("omnitrace_pop_trace") != _npos) return -1;
+        if(_lbl.find("amd_comgr_") == 0) return -1;
        if(_check_internal)
        {
            if(std::regex_search(
@@ -423,11 +459,141 @@ backtrace::post_process(int64_t _tid)
        return _lbl.replace(_pos, _dyninst.length(), "");
    };

-    auto _data = _sampler->get_allocator().get_data();
+    using common_type_t = typename hw_counters::common_type;
+    auto _hw_cnt_labels = (get_papi_vector(_tid))
+                              ? comp::papi_common::get_events<common_type_t>()
+                              : std::vector<int>{};
+
+    auto _process_perfetto_counters = [&](const std::vector<sampling::bundle_t*>& _data) {
+        if(!perfetto_counter_track<comp::peak_rss>::exists(_tid))
+        {
+            auto _thrname = TIMEMORY_JOIN("", "[Thread ", _tid, "] ");
+            auto addendum = [&](const std::string& _v) { return _thrname + _v + " (S)"; };
+            perfetto_counter_track<comp::peak_rss>::emplace(
+                _tid, addendum("Peak Memory Usage"), "MB");
+        }
+
+        if(!perfetto_counter_track<hw_counters>::exists(_tid) &&
+           tim::trait::runtime_enabled<hw_counters>::get())
+        {
+            auto _thrname = TIMEMORY_JOIN("", "[Thread ", _tid, "] ");
+            auto addendum = [&](const std::string& _v) { return _thrname + _v + " (S)"; };
+            for(auto& itr : _hw_cnt_labels)
+            {
+                perfetto_counter_track<hw_counters>::emplace(
+                    _tid, addendum(tim::papi::get_event_info(itr).short_descr), "");
+            }
+        }
+
+        for(const auto& ditr : _data)
+        {
+            const auto* _bt = ditr->get<backtrace>();
+            if(_bt->m_tid != _tid) continue;
+
+            auto _ts = static_cast<uint64_t>(_bt->m_ts.time_since_epoch().count());
+
+            TRACE_COUNTER("sampling", perfetto_counter_track<comp::peak_rss>::at(_tid, 0),
+                          _ts, _bt->m_mem_peak / units::megabyte);
+
+            if(tim::trait::runtime_enabled<hw_counters>::get())
+            {
+                for(size_t i = 0; i < perfetto_counter_track<hw_counters>::size(_tid);
+                    ++i)
+                {
+                    if(i < _bt->m_hw_counter.size())
+                    {
+                        TRACE_COUNTER("sampling",
+                                      perfetto_counter_track<hw_counters>::at(_tid, i),
+                                      _ts, _bt->m_hw_counter.at(i));
+                    }
+                }
+            }
+        }
+    };
+
+    auto _process_perfetto = [&](const std::vector<sampling::bundle_t*>& _data,
+                                 bool                                    _rename) {
+        if(_rename)
+            threading::set_thread_name(TIMEMORY_JOIN(" ", "Thread", _tid, "(S)").c_str());
+        time_point_type _last_wall_ts = _init->get_timestamp();
+
+        for(const auto& ditr : _data)
+        {
+            const auto* _bt = ditr->get<backtrace>();
+            if(_bt->m_tid != _tid) continue;
+
+            static std::set<std::string> _static_strings{};
+            std::string                  _last = {};
+            for(const auto& itr : _bt->get())
+            {
+                auto _name = tim::demangle(_patch_label(itr));
+                auto _use =
+                    _use_label(_name, !_last.empty() &&
+                                          (_last == "start_thread" || _last == "clone"));
+                if(_use == -1) break;
+                if(_use == 0) continue;
+                auto sitr = _static_strings.emplace(_name);
+                _last     = *sitr.first;
+                auto _ts  = static_cast<uint64_t>(_bt->m_ts.time_since_epoch().count());
+
+                TRACE_EVENT_BEGIN(
+                    "sampling", perfetto::StaticString{ sitr.first->c_str() },
+                    static_cast<uint64_t>(_last_wall_ts.time_since_epoch().count()));
+                TRACE_EVENT_END("sampling", _ts);
+            }
+            _last_wall_ts = _bt->m_ts;
+        }
+    };
+
+    auto _raw_data = _sampler->get_allocator().get_data();
    // single sample that is useless (backtrace to unblocking signals)
-    if(_data.size() == 1 && _data.front().size() <= 1) _data.clear();
-    OMNITRACE_DEBUG("Post-processing %zu sampling entries for thread %lu...\n",
-                    _data.size(), _tid);
+    if(_raw_data.size() == 1 && _raw_data.front().size() <= 1) _raw_data.clear();
+
+    std::vector<sampling::bundle_t*> _data{};
+    for(auto& ditr : _raw_data)
+    {
+        _data.reserve(_data.size() + ditr.size());
+        for(auto& ritr : ditr)
+        {
+            auto* _bt = ritr.get<backtrace>();
+            if(!_bt)
+            {
+                OMNITRACE_PRINT(
+                    "Warning! Nullptr to backtrace instance for thread %lu...\n", _tid);
+                continue;
+            }
+            if(_bt->empty()) continue;
+            _data.emplace_back(&ritr);
+        }
+    }
+
+    if(_data.empty()) return;
+
+    OMNITRACE_CONDITIONAL_PRINT(
+        get_verbose() >= 0 || get_debug(),
+        "Post-processing %zu sampling entries for thread %lu...\n", _data.size(), _tid);
+
+    std::sort(_data.begin(), _data.end(),
+              [](const sampling::bundle_t* _lhs, const sampling::bundle_t* _rhs) {
+                  return _lhs->get<backtrace>()->m_ts < _rhs->get<backtrace>()->m_ts;
+              });
+
+    if(get_use_perfetto())
+    {
+        _process_perfetto_counters(_data);
+
+        if(_tid == 0 && get_mode() == Mode::Sampling)
+            _process_perfetto(_data, false);
+        else
+        {
+            auto _v = pthread_gotcha::enable_sampling_on_child_threads();
+            pthread_gotcha::enable_sampling_on_child_threads() = false;
+            std::thread{ _process_perfetto, _data, true }.join();
+            pthread_gotcha::enable_sampling_on_child_threads() = _v;
+        }
+    }
+
+    if(!get_use_timemory()) return;

    std::map<int64_t, std::map<int64_t, int64_t>> _depth_sum = {};
    auto                                          _scope     = tim::scope::config{};
@@ -441,125 +607,101 @@ backtrace::post_process(int64_t _tid)
        using bundle_t = tim::lightweight_tuple<comp::trip_count, sampling_wall_clock,
                                                sampling_cpu_clock, hw_counters>;

-        for(auto& ritr : ditr)
+        auto* _bt = ditr->get<backtrace>();
+
+        double _elapsed_wc = (_bt->m_ts - _last_wall_ts).count();
+        double _elapsed_cc = (_bt->m_thr_cpu_ts - _last_cpu_ts);
+
+        std::vector<bundle_t> _tc{};
+        _tc.reserve(_bt->size());
+
+        // generate the instances of the tuple of components and start them
+        for(const auto& itr : _bt->get())
        {
-            auto* _bt = ritr.get<backtrace>();
-            if(!_bt)
-            {
-                OMNITRACE_PRINT(
-                    "Warning! Nullptr to backtrace instance for thread %lu...\n", _tid);
-                continue;
-            }
-
-            if(_bt->empty()) continue;
-
-            double _elapsed_wc = (_bt->m_ts - _last_wall_ts).count();
-            double _elapsed_cc = (_bt->m_thr_cpu_ts - _last_cpu_ts);
-
-            std::vector<bundle_t> _tc{};
-            _tc.reserve(_bt->size());
-
-            // generate the instances of the tuple of components and start them
-            for(const auto& itr : _bt->get())
-            {
-                auto _lbl = _patch_label(itr);
-                auto _use = _use_label(_lbl, !_tc.empty() &&
-                                                 (_tc.back().key() == "start_thread" ||
+            auto _lbl = _patch_label(itr);
+            auto _use =
+                _use_label(_lbl, !_tc.empty() && (_tc.back().key() == "start_thread" ||
                                                  _tc.back().key() == "clone"));
-                if(_use == -1) break;
-                if(_use == 0) continue;
-                _tc.emplace_back(tim::string_view_t{ _lbl }, _scope);
-                _tc.back().push(_bt->m_tid);
-                _tc.back().start();
-            }
-
-            // stop the instances and update the values as needed
-            for(size_t i = 0; i < _tc.size(); ++i)
-            {
-                auto&  itr    = _tc.at(_tc.size() - i - 1);
-                size_t _depth = 0;
-                _depth_sum[_bt->m_tid][_depth] += 1;
-                itr.stop();
-                if constexpr(tim::trait::is_available<sampling_wall_clock>::value)
-                {
-                    auto* _sc = itr.get<sampling_wall_clock>();
-                    if(_sc)
-                    {
-                        auto _value = _elapsed_wc / sampling_wall_clock::get_unit();
-                        _sc->set_value(_value);
-                        _sc->set_accum(_value);
-                    }
-                }
-                if constexpr(tim::trait::is_available<sampling_cpu_clock>::value)
-                {
-                    auto* _cc = itr.get<sampling_cpu_clock>();
-                    if(_cc)
-                    {
-                        _cc->set_value(_elapsed_cc / sampling_cpu_clock::get_unit());
-                        _cc->set_accum(_elapsed_cc / sampling_cpu_clock::get_unit());
-                    }
-                }
-                if constexpr(tim::trait::is_available<hw_counters>::value)
-                {
-                    auto* _hw_counter = itr.get<hw_counters>();
-                    if(_hw_counter)
-                    {
-                        _hw_counter->set_value(_bt->m_hw_counter);
-                        _hw_counter->set_accum(_bt->m_hw_counter);
-                    }
-                }
-                itr.pop();
-            }
-            _last_wall_ts = _bt->m_ts;
-            _last_cpu_ts  = _bt->m_thr_cpu_ts;
+            if(_use == -1) break;
+            if(_use == 0) continue;
+            _tc.emplace_back(tim::string_view_t{ _lbl }, _scope);
+            _tc.back().push(_bt->m_tid);
+            _tc.back().start();
        }
-    }

-    namespace quirk = tim::quirk;
+        // stop the instances and update the values as needed
+        for(size_t i = 0; i < _tc.size(); ++i)
+        {
+            auto&  itr    = _tc.at(_tc.size() - i - 1);
+            size_t _depth = 0;
+            _depth_sum[_bt->m_tid][_depth] += 1;
+            itr.stop();
+            if constexpr(tim::trait::is_available<sampling_wall_clock>::value)
+            {
+                auto* _sc = itr.get<sampling_wall_clock>();
+                if(_sc)
+                {
+                    auto _value = _elapsed_wc / sampling_wall_clock::get_unit();
+                    _sc->set_value(_value);
+                    _sc->set_accum(_value);
+                }
+            }
+            if constexpr(tim::trait::is_available<sampling_cpu_clock>::value)
+            {
+                auto* _cc = itr.get<sampling_cpu_clock>();
+                if(_cc)
+                {
+                    _cc->set_value(_elapsed_cc / sampling_cpu_clock::get_unit());
+                    _cc->set_accum(_elapsed_cc / sampling_cpu_clock::get_unit());
+                }
+            }
+            if constexpr(tim::trait::is_available<hw_counters>::value)
+            {
+                auto* _hw_counter = itr.get<hw_counters>();
+                if(_hw_counter)
+                {
+                    _hw_counter->set_value(_bt->m_hw_counter);
+                    _hw_counter->set_accum(_bt->m_hw_counter);
+                }
+            }
+            itr.pop();
+        }
+        _last_wall_ts = _bt->m_ts;
+        _last_cpu_ts  = _bt->m_thr_cpu_ts;
+    }

    for(auto&& ditr : _data)
    {
        using bundle_t =
            tim::lightweight_tuple<sampling_percent, quirk::config<quirk::tree_scope>>;

-        for(auto& ritr : ditr)
+        auto* _bt = ditr->get<backtrace>();
+
+        std::vector<bundle_t> _tc{};
+        _tc.reserve(_bt->size());
+
+        // generate the instances of the tuple of components and start them
+        for(const auto& itr : _bt->get())
        {
-            auto* _bt = ritr.get<backtrace>();
-            if(!_bt)
-            {
-                OMNITRACE_PRINT(
-                    "Warning! Nullptr to backtrace instance for thread %lu...\n", _tid);
-                continue;
-            }
+            auto _lbl = _patch_label(itr);
+            auto _use =
+                _use_label(_lbl, !_tc.empty() && _tc.back().key() == "start_thread");
+            if(_use == -1) break;
+            if(_use == 0) continue;
+            _tc.emplace_back(tim::string_view_t{ _lbl });
+            _tc.back().push(_bt->m_tid);
+            _tc.back().start();
+        }

-            if(_bt->empty()) continue;
-
-            std::vector<bundle_t> _tc{};
-            _tc.reserve(_bt->size());
-
-            // generate the instances of the tuple of components and start them
-            for(const auto& itr : _bt->get())
-            {
-                auto _lbl = _patch_label(itr);
-                auto _use =
-                    _use_label(_lbl, !_tc.empty() && _tc.back().key() == "start_thread");
-                if(_use == -1) break;
-                if(_use == 0) continue;
-                _tc.emplace_back(tim::string_view_t{ _lbl });
-                _tc.back().push(_bt->m_tid);
-                _tc.back().start();
-            }
-
-            // stop the instances and update the values as needed
-            for(size_t i = 0; i < _tc.size(); ++i)
-            {
-                auto&  itr    = _tc.at(_tc.size() - i - 1);
-                size_t _depth = 0;
-                double _value = (1.0 / _depth_sum[_bt->m_tid][_depth]) * 100.0;
-                itr.store(std::plus<double>{}, _value);
-                itr.stop();
-                itr.pop();
-            }
+        // stop the instances and update the values as needed
+        for(size_t i = 0; i < _tc.size(); ++i)
+        {
+            auto&  itr    = _tc.at(_tc.size() - i - 1);
+            size_t _depth = 0;
+            double _value = (1.0 / _depth_sum[_bt->m_tid][_depth]) * 100.0;
+            itr.store(std::plus<double>{}, _value);
+            itr.stop();
+            itr.pop();
        }
    }
 }
@@ -21,9 +21,11 @@
 // SOFTWARE.

 #include "library/components/mpi_gotcha.hpp"
+#include "library/api.hpp"
 #include "library/components/omnitrace.hpp"
 #include "library/config.hpp"
 #include "library/debug.hpp"
+#include "timemory/backends/process.hpp"

 #include <thread>
 #include <timemory/backends/mpi.hpp>
@@ -35,7 +37,7 @@ namespace
 uint64_t    mpip_index      = std::numeric_limits<uint64_t>::max();
 std::string mpi_init_string = {};

-// this ensures omnitrace_trace_finalize is called before MPI_Finalize
+// this ensures omnitrace_finalize is called before MPI_Finalize
 void
 omnitrace_mpi_set_attr()
 {
@@ -49,9 +51,7 @@ omnitrace_mpi_set_attr()
        if(mpip_index != std::numeric_limits<uint64_t>::max())
            comp::deactivate_mpip<tim::component_tuple<omnitrace::component::omnitrace>,
                                  api::omnitrace>(mpip_index);
-        if(!mpi_init_string.empty()) omnitrace_pop_trace(mpi_init_string.c_str());
-        mpi_init_string = {};
-        omnitrace_trace_finalize();
+        omnitrace_finalize();
        return MPI_SUCCESS;
    };
    using copy_func_t = int (*)(MPI_Comm, int, void*, void*, void*, int*);
@@ -73,8 +73,10 @@ mpi_gotcha::configure()
        mpi_gotcha_t::template configure<1, int, int*, char***, int, int*>(
            "MPI_Init_thread");
        mpi_gotcha_t::template configure<2, int>("MPI_Finalize");
-        mpi_gotcha_t::template configure<3, int, tim::mpi::comm_t, int*>("MPI_Comm_rank");
-        mpi_gotcha_t::template configure<4, int, tim::mpi::comm_t, int*>("MPI_Comm_size");
+#if defined(OMNITRACE_USE_MPI_HEADERS)
+        mpi_gotcha_t::template configure<3, int, comm_t, int*>("MPI_Comm_rank");
+        mpi_gotcha_t::template configure<4, int, comm_t, int*>("MPI_Comm_size");
+#endif
    };
 }

@@ -83,15 +85,15 @@ mpi_gotcha::audit(const gotcha_data_t& _data, audit::incoming, int*, char***)
 {
    OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug_env(), "[%s] %s(int*, char***)\n",
                                      __FUNCTION__, _data.tool_id.c_str());
+
    if(get_state() == ::omnitrace::State::DelayedInit)
-    {
-        get_state()     = ::omnitrace::State::PreInit;
-        mpi_init_string = _data.tool_id;
+        get_state() = ::omnitrace::State::PreInit;
+
+    omnitrace_push_trace(_data.tool_id.c_str());
 #if !defined(TIMEMORY_USE_MPI) && defined(TIMEMORY_USE_MPI_HEADERS)
-        tim::mpi::is_initialized_callback() = []() { return true; };
-        tim::mpi::is_finalized()            = false;
+    tim::mpi::is_initialized_callback() = []() { return true; };
+    tim::mpi::is_finalized()            = false;
 #endif
-    }
 }

 void
@@ -101,14 +103,13 @@ mpi_gotcha::audit(const gotcha_data_t& _data, audit::incoming, int*, char***, in
                                      "[%s] %s(int*, char***, int, int*)\n", __FUNCTION__,
                                      _data.tool_id.c_str());
    if(get_state() == ::omnitrace::State::DelayedInit)
-    {
-        get_state()     = ::omnitrace::State::PreInit;
-        mpi_init_string = _data.tool_id;
+        get_state() = ::omnitrace::State::PreInit;
+
+    omnitrace_push_trace(_data.tool_id.c_str());
 #if !defined(TIMEMORY_USE_MPI) && defined(TIMEMORY_USE_MPI_HEADERS)
-        tim::mpi::is_initialized_callback() = []() { return true; };
-        tim::mpi::is_finalized()            = false;
+    tim::mpi::is_initialized_callback() = []() { return true; };
+    tim::mpi::is_finalized()            = false;
 #endif
-    }
 }

 void
@@ -116,16 +117,16 @@ mpi_gotcha::audit(const gotcha_data_t& _data, audit::incoming)
 {
    OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug_env(), "[%s] %s()\n", __FUNCTION__,
                                      _data.tool_id.c_str());
+
    if(mpip_index != std::numeric_limits<uint64_t>::max())
        comp::deactivate_mpip<tim::component_tuple<omnitrace::component::omnitrace>,
                              api::omnitrace>(mpip_index);
-    if(!mpi_init_string.empty()) omnitrace_pop_trace(mpi_init_string.c_str());
-    mpi_init_string = {};
+
 #if !defined(TIMEMORY_USE_MPI) && defined(TIMEMORY_USE_MPI_HEADERS)
    tim::mpi::is_initialized_callback() = []() { return false; };
    tim::mpi::is_finalized()            = true;
 #else
-    omnitrace_trace_finalize();
+    omnitrace_finalize();
 #endif
 }

@@ -134,6 +135,8 @@ mpi_gotcha::audit(const gotcha_data_t& _data, audit::incoming, comm_t, int* _val
 {
    OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug_env(), "[%s] %s()\n", __FUNCTION__,
                                      _data.tool_id.c_str());
+
+    omnitrace_push_trace(_data.tool_id.c_str());
    if(_data.tool_id == "MPI_Comm_rank")
    {
        m_rank_ptr = _val;
@@ -144,8 +147,8 @@ mpi_gotcha::audit(const gotcha_data_t& _data, audit::incoming, comm_t, int* _val
    }
    else
    {
-        OMNITRACE_PRINT("[%s] %s(<comm>, %p) :: unexpected function wrapper\n",
-                        __FUNCTION__, _data.tool_id.c_str(), _val);
+        OMNITRACE_BASIC_PRINT("[%s] %s(<comm>, %p) :: unexpected function wrapper\n",
+                              __FUNCTION__, _data.tool_id.c_str(), _val);
    }
 }

@@ -154,8 +157,8 @@ mpi_gotcha::audit(const gotcha_data_t& _data, audit::outgoing, int _retval)
 {
    OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug_env(), "[%s] %s() returned %i\n",
                                      __FUNCTION__, _data.tool_id.c_str(), (int) _retval);
-    if(_retval == tim::mpi::success_v && get_state() == ::omnitrace::State::PreInit &&
-       _data.tool_id.find("MPI_Init") == 0)
+
+    if(_retval == tim::mpi::success_v && _data.tool_id.find("MPI_Init") == 0)
    {
        omnitrace_mpi_set_attr();
        // omnitrace will set this environement variable to true in binary rewrite mode
@@ -165,8 +168,9 @@ mpi_gotcha::audit(const gotcha_data_t& _data, audit::outgoing, int _retval)
        // were excluded via a regex expression)
        if(get_use_mpip())
        {
-            OMNITRACE_CONDITIONAL_BASIC_PRINT(
-                get_debug_env(), "[%s] Activating MPI wrappers...\n", __FUNCTION__);
+            OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug_env() || get_verbose_env() > 0,
+                                              "[%s] Activating MPI wrappers...\n",
+                                              __FUNCTION__);

            // use env vars OMNITRACE_MPIP_PERMIT_LIST and OMNITRACE_MPIP_REJECT_LIST
            // to control the gotcha bindings at runtime
@@ -176,7 +180,6 @@ mpi_gotcha::audit(const gotcha_data_t& _data, audit::outgoing, int _retval)
                comp::activate_mpip<tim::component_tuple<omnitrace::component::omnitrace>,
                                    api::omnitrace>();
        }
-        omnitrace_push_trace(_data.tool_id.c_str());
    }
    else if(_retval == tim::mpi::success_v && _data.tool_id.find("MPI_Comm_") == 0)
    {
@@ -185,17 +188,17 @@ mpi_gotcha::audit(const gotcha_data_t& _data, audit::outgoing, int _retval)
            if(m_rank_ptr)
            {
                m_rank = std::max<int>(*m_rank_ptr, m_rank);
-                OMNITRACE_CONDITIONAL_BASIC_PRINT(tim::settings::verbose() > 0,
-                                                  "MPI rank: %i\n", m_rank);
                tim::mpi::set_rank(m_rank);
                tim::settings::default_process_suffix() = m_rank;
                get_perfetto_output_filename().clear();
-                (void) get_perfetto_output_filename();
+                OMNITRACE_CONDITIONAL_BASIC_PRINT(
+                    get_debug() || get_verbose() > 0, "[pid=%i] MPI rank: %i (%i)\n",
+                    process::get_id(), tim::mpi::rank(), m_rank);
            }
            else
            {
-                OMNITRACE_PRINT("[%s] %s() returned %i :: nullptr to rank\n",
-                                __FUNCTION__, _data.tool_id.c_str(), (int) _retval);
+                OMNITRACE_BASIC_PRINT("[%s] %s() returned %i :: nullptr to rank\n",
+                                      __FUNCTION__, _data.tool_id.c_str(), (int) _retval);
            }
        }
        else if(_data.tool_id == "MPI_Comm_size")
@@ -203,22 +206,25 @@ mpi_gotcha::audit(const gotcha_data_t& _data, audit::outgoing, int _retval)
            if(m_size_ptr)
            {
                m_size = std::max<int>(*m_size_ptr, m_size);
-                OMNITRACE_CONDITIONAL_BASIC_PRINT(tim::settings::verbose() > 0,
-                                                  "MPI size: %i\n", m_size);
                tim::mpi::set_size(m_size);
+                OMNITRACE_CONDITIONAL_BASIC_PRINT(
+                    get_debug() || get_verbose() > 0, "[pid=%i] MPI size: %i (%i)\n",
+                    process::get_id(), tim::mpi::size(), m_size);
            }
            else
            {
-                OMNITRACE_PRINT("[%s] %s() returned %i :: nullptr to size\n",
-                                __FUNCTION__, _data.tool_id.c_str(), (int) _retval);
+                OMNITRACE_BASIC_PRINT("[%s] %s() returned %i :: nullptr to size\n",
+                                      __FUNCTION__, _data.tool_id.c_str(), (int) _retval);
            }
        }
        else
        {
-            OMNITRACE_PRINT("[%s] %s() returned %i :: unexpected function wrapper\n",
-                            __FUNCTION__, _data.tool_id.c_str(), (int) _retval);
+            OMNITRACE_BASIC_PRINT(
+                "[%s] %s() returned %i :: unexpected function wrapper\n", __FUNCTION__,
+                _data.tool_id.c_str(), (int) _retval);
        }
    }
+    omnitrace_pop_trace(_data.tool_id.c_str());
 }
 }  // namespace omnitrace

@@ -21,10 +21,13 @@
 // SOFTWARE.

 #include "library/components/pthread_gotcha.hpp"
+#include "library/components/omnitrace.hpp"
+#include "library/components/roctracer.hpp"
 #include "library/config.hpp"
 #include "library/debug.hpp"
 #include "library/sampling.hpp"

+#include <ostream>
 #include <timemory/sampling/allocator.hpp>
 #include <timemory/utility/types.hpp>

@@ -40,6 +43,47 @@ std::set<int>
 shutdown();
 }  // namespace sampling

+namespace mpl = tim::mpl;
+
+using bundle_t  = tim::lightweight_tuple<comp::wall_clock, comp::roctracer_data>;
+using wall_pw_t = mpl::piecewise_select<comp::wall_clock>;  // only wall-clock
+using main_pw_t = mpl::piecewise_ignore<comp::wall_clock>;  // exclude wall-clock
+using omni_pw_t = mpl::piecewise_select<>;
+
+namespace
+{
+std::map<int64_t, std::shared_ptr<bundle_t>> bundles = {};
+std::mutex                                   bundles_mutex{};
+
+inline void
+start_bundle(bundle_t& _bundle)
+{
+    if(comp::roctracer::is_setup())
+    {
+        _bundle.push(main_pw_t{});
+        _bundle.start();
+    }
+    else
+    {
+        _bundle.push(omni_pw_t{});
+        _bundle.start(omni_pw_t{});
+    }
+}
+
+inline void
+stop_bundle(bundle_t& _bundle, int64_t _tid)
+{
+    _bundle.stop(wall_pw_t{});  // stop wall-clock so we can get the value
+    // update roctracer_data
+    _bundle.store(std::plus<double>{},
+                  _bundle.get<comp::wall_clock>()->get() * units::sec);
+    // stop all other components including roctracer_data after update
+    _bundle.stop(main_pw_t{});
+    // exclude popping wall-clock
+    _bundle.pop(main_pw_t{}, _tid);
+}
+}  // namespace
+
 pthread_gotcha::wrapper::wrapper(routine_t _routine, void* _arg, bool _enable_sampling,
                                 promise_t* _p)
 : m_enable_sampling{ _enable_sampling }
@@ -51,10 +95,41 @@ pthread_gotcha::wrapper::wrapper(routine_t _routine, void* _arg, bool _enable_sa
 void*
 pthread_gotcha::wrapper::operator()() const
 {
-    std::set<int> _signals{};
-    auto&         _enable_sampling = pthread_gotcha::enable_sampling_on_child_threads();
-    if(m_enable_sampling && _enable_sampling)
+    std::shared_ptr<bundle_t> _bundle{};
+    std::set<int>             _signals{};
+    auto&   _enable_sampling = pthread_gotcha::enable_sampling_on_child_threads();
+    auto    _active          = (get_state() == omnitrace::State::Active);
+    int64_t _tid             = -1;
+    auto    _is_sampling     = false;
+    auto    _dtor            = scope::destructor{ [&]() {
+        if(_is_sampling)
+        {
+            sampling::block_signals(_signals);
+            sampling::shutdown();
+        }
+
+        if(_bundle)
+        {
+            std::unique_lock<std::mutex> _lk{ bundles_mutex };
+            stop_bundle(*_bundle, _tid);
+            _bundle.reset();
+            bundles.erase(_tid);
+        }
+    } };
+
+    if(m_enable_sampling && _enable_sampling && _active)
    {
+        _tid = threading::get_id();
+        threading::set_thread_name(TIMEMORY_JOIN(" ", "Thread", _tid).c_str());
+        {
+            std::unique_lock<std::mutex> _lk{ bundles_mutex };
+            if(comp::roctracer::is_setup())
+                _bundle =
+                    bundles.emplace(_tid, std::make_shared<bundle_t>("start_thread"))
+                        .first->second;
+        }
+        if(_bundle) start_bundle(*_bundle);
+        _is_sampling     = true;
        _enable_sampling = false;
        _signals         = sampling::setup();
        _enable_sampling = true;
@@ -64,15 +139,7 @@ pthread_gotcha::wrapper::operator()() const
    if(m_promise) m_promise->set_value();

    // execute the original function
-    auto* _ret = m_routine(m_arg);
-
-    if(m_enable_sampling && _enable_sampling)
-    {
-        sampling::block_signals(_signals);
-        sampling::shutdown();
-    }
-
-    return _ret;
+    return m_routine(m_arg);
 }

 void*
@@ -95,6 +162,27 @@ pthread_gotcha::configure()
    };
 }

+void
+pthread_gotcha::shutdown()
+{
+    std::unique_lock<std::mutex> _lk{ bundles_mutex };
+    unsigned long                _ndangling = 0;
+    for(auto itr : bundles)
+    {
+        if(itr.second)
+        {
+            stop_bundle(*itr.second, itr.first);
+            ++_ndangling;
+        }
+        itr.second.reset();
+    }
+
+    OMNITRACE_CONDITIONAL_PRINT(
+        (get_verbose() > 0 || get_debug()) && _ndangling > 0,
+        "pthread_gotcha::shutdown() cleaned up %lu dangling bundles\n", _ndangling);
+    bundles.clear();
+}
+
 bool&
 pthread_gotcha::enable_sampling_on_child_threads()
 {
@@ -107,13 +195,18 @@ int
 pthread_gotcha::operator()(pthread_t* thread, const pthread_attr_t* attr,
                           void* (*start_routine)(void*), void*     arg) const
 {
-    auto _enable_sampling = enable_sampling_on_child_threads();
+    bundle_t _bundle{ "pthread_create" };
+    auto     _enable_sampling = enable_sampling_on_child_threads();

-    if(!_enable_sampling)
+    if(!get_use_sampling() || !_enable_sampling)
    {
+        // if(!get_use_sampling()) start_bundle(_bundle);
        auto* _obj = new wrapper(start_routine, arg, _enable_sampling, nullptr);
        // create the thread
-        return pthread_create(thread, attr, &wrapper::wrap, static_cast<void*>(_obj));
+        auto _ret =
+            pthread_create(thread, attr, &wrapper::wrap, static_cast<void*>(_obj));
+        // if(!get_use_sampling()) stop_bundle(_bundle, threading::get_id());
+        return _ret;
    }

    // block the signals in entire process
@@ -121,18 +214,22 @@ pthread_gotcha::operator()(pthread_t* thread, const pthread_attr_t* attr,
    tim::sampling::block_signals({ SIGALRM, SIGPROF },
                                 tim::sampling::sigmask_scope::process);

+    start_bundle(_bundle);
+
    // promise set by thread when signal handler is configured
    auto  _promise = std::promise<void>{};
    auto  _fut     = _promise.get_future();
-    auto* _obj     = new wrapper(start_routine, arg, _enable_sampling, &_promise);
+    auto* _wrap    = new wrapper(start_routine, arg, _enable_sampling, &_promise);

    // create the thread
-    auto _ret = pthread_create(thread, attr, &wrapper::wrap, static_cast<void*>(_obj));
+    auto _ret = pthread_create(thread, attr, &wrapper::wrap, static_cast<void*>(_wrap));

    // wait for thread to set promise
    OMNITRACE_DEBUG("waiting for child to signal it is setup...\n");
    _fut.wait();

+    stop_bundle(_bundle, threading::get_id());
+
    // unblock the signals in the entire process
    OMNITRACE_DEBUG("unblocking signals...\n");
    tim::sampling::unblock_signals({ SIGALRM, SIGPROF },
@@ -0,0 +1,569 @@
+// Copyright (c) 2018 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// with the Software without restriction, including without limitation the
+// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+// sell copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// * Redistributions of source code must retain the above copyright notice,
+// this list of conditions and the following disclaimers.
+//
+// * Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimers in the
+// documentation and/or other materials provided with the distribution.
+//
+// * Neither the names of Advanced Micro Devices, Inc. nor the names of its
+// contributors may be used to endorse or promote products derived from
+// this Software without specific prior written permission.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH
+// THE SOFTWARE.
+
+#if defined(NDEBUG)
+#    undef NDEBUG
+#endif
+
+#include "library/components/rocm_smi.hpp"
+#include "library/common.hpp"
+#include "library/components/fwd.hpp"
+#include "library/config.hpp"
+#include "library/critical_trace.hpp"
+#include "library/debug.hpp"
+#include "library/gpu.hpp"
+#include "library/perfetto.hpp"
+
+#include <timemory/backends/threading.hpp>
+#include <timemory/components/timing/backends.hpp>
+#include <timemory/units.hpp>
+#include <timemory/utility/locking.hpp>
+
+#include <rocm_smi/rocm_smi.h>
+
+#include <cassert>
+#include <chrono>
+#include <ios>
+#include <sstream>
+#include <sys/resource.h>
+#include <thread>
+
+#define OMNITRACE_ROCM_SMI_CALL(ERROR_CODE) ::omnitrace::rocm_smi::check_error(ERROR_CODE)
+
+namespace omnitrace
+{
+namespace rocm_smi
+{
+using tim::type_mutex;
+using auto_lock_t = tim::auto_lock_t;
+
+namespace
+{
+bool&
+is_initialized()
+{
+    static bool _v = false;
+    return _v;
+}
+
+void
+check_error(rsmi_status_t ec)
+{
+    if(ec == RSMI_STATUS_SUCCESS) return;
+    const char* _msg = nullptr;
+    auto        _err = rsmi_status_string(ec, &_msg);
+    if(_err != RSMI_STATUS_SUCCESS)
+        OMNITRACE_THROW(
+            "rsmi_status_string(%i, ...) failed. No error message available\n", (int) ec);
+    OMNITRACE_THROW("%s", _msg);
+}
+
+std::atomic<State>&
+get_rocm_smi_state()
+{
+    static std::atomic<State> _v{ State::PreInit };
+    return _v;
+}
+}  // namespace
+
+//--------------------------------------------------------------------------------------//
+
+size_t                           data::device_count     = 0;
+std::set<uint32_t>               data::device_list      = {};
+std::unique_ptr<data::promise_t> data::polling_finished = {};
+
+data::data(uint32_t _dev_id) { sample(_dev_id); }
+
+void
+data::sample(uint32_t _dev_id)
+{
+    auto _ts = tim::get_clock_real_now<size_t, std::nano>();
+    assert(_ts < std::numeric_limits<int64_t>::max());
+
+    m_dev_id = _dev_id;
+    m_ts     = _ts;
+
+    rsmi_dev_busy_percent_get(_dev_id, &m_busy_perc);
+    rsmi_dev_temp_metric_get(_dev_id, RSMI_TEMP_TYPE_EDGE, RSMI_TEMP_CURRENT, &m_temp);
+    rsmi_dev_power_ave_get(_dev_id, 0, &m_power);
+    rsmi_dev_memory_usage_get(_dev_id, RSMI_MEM_TYPE_VRAM, &m_mem_usage);
+}
+
+void
+data::print(std::ostream& _os) const
+{
+    std::stringstream _ss{};
+    _ss << "device: " << m_dev_id << ", busy = " << m_busy_perc << "%, temp = " << m_temp
+        << ", power = " << m_power << ", memory usage = " << m_mem_usage;
+    _os << _ss.str();
+}
+
+namespace
+{
+struct cpu_freq
+{};
+using freq_pair_t                                     = std::pair<size_t, double>;
+std::vector<std::vector<freq_pair_t>> cpu_frequencies = {};
+
+struct cpu_mem
+{};
+using cpu_mem_usage_pair_t                      = std::pair<size_t, int64_t>;
+std::vector<cpu_mem_usage_pair_t> cpu_mem_usage = {};
+}  // namespace
+
+void
+data::poll(std::atomic<State>* _state, nsec_t _interval, promise_t* _ready)
+{
+    threading::set_thread_name("omni.rocm_smi");
+
+    // notify thread started
+    if(_ready) _ready->set_value();
+
+    std::vector<std::unique_ptr<bundle_t>*> _bundle_data{};
+    _bundle_data.resize(device_count, nullptr);
+    for(size_t i = 0; i < device_count; ++i)
+    {
+        if(device_list.count(i) > 0)
+        {
+            _bundle_data.at(i) = &sampler_instances::instances().at(i);
+            if(!*_bundle_data.at(i)) *_bundle_data.at(i) = std::make_unique<bundle_t>();
+        }
+    }
+
+    auto                _ncpu = threading::affinity::hw_concurrency();
+    std::vector<size_t> _cpu_mhz_pos{};
+    std::ifstream       _ifs{ "/proc/cpuinfo" };
+    if(_ifs)
+    {
+        for(size_t i = 0; i < _ncpu; ++i)
+        {
+            short       _n = 0;
+            std::string _st{};
+            while(_ifs && _ifs.good())
+            {
+                std::string _s{};
+                _ifs >> _s;
+                if(!_ifs.good() || !_ifs) break;
+
+                if(_s == "cpu" || _s == "MHz" || _s == ":")
+                {
+                    ++_n;
+                    _st += _s + " ";
+                }
+                else
+                {
+                    _n  = 0;
+                    _st = {};
+                }
+
+                if(_n == 3)
+                {
+                    size_t _pos = _ifs.tellg();
+                    _cpu_mhz_pos.emplace_back(_pos + 1);
+                    _ifs >> _s;
+                    if(!_ifs.good() || !_ifs) break;
+                    OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug() || get_verbose() > 1,
+                                                      "[%zu] %s %s (pos = %zu)\n", i,
+                                                      _st.c_str(), _s.c_str(), _pos + 1);
+                    break;
+                }
+            }
+        }
+    }
+
+    cpu_frequencies.resize(_ncpu);
+
+    _ifs                = std::ifstream{ "/proc/cpuinfo", std::ifstream::binary };
+    auto _read_cpu_freq = [&_cpu_mhz_pos, &_ifs](size_t _idx) {
+        double _freq = 0;
+        _ifs.seekg(_cpu_mhz_pos.at(_idx), _ifs.beg);
+        _ifs >> _freq;
+        return _freq;
+    };
+
+    auto _read_cpu_mem_usage = []() {
+        cpu_mem_usage.emplace_back(tim::get_clock_real_now<size_t, std::nano>(),
+                                   tim::get_page_rss());
+    };
+
+    auto _read_cpu_freqs = [&_read_cpu_freq, _ncpu]() {
+        auto _ts = tim::get_clock_real_now<size_t, std::nano>();
+        for(size_t i = 0; i < _ncpu; ++i)
+        {
+            auto _freq = _read_cpu_freq(i);
+            cpu_frequencies.at(i).emplace_back(_ts, _freq);
+        }
+    };
+
+    OMNITRACE_CONDITIONAL_BASIC_PRINT(
+        get_verbose() > 0 || get_debug(),
+        "Polling rocm-smi for %zu device(s) at an interval of %f seconds...\n",
+        device_list.size(),
+        std::chrono::duration_cast<std::chrono::duration<double>>(_interval).count());
+
+    get_initial().resize(device_count);
+    for(auto itr : device_list)
+        get_initial().at(itr).sample(itr);
+
+    auto _now = std::chrono::steady_clock::now();
+    while(_state && _state->load() != State::Finalized && get_state() != State::Finalized)
+    {
+        std::this_thread::sleep_until(_now);
+        if(_state->load() != State::Active) continue;
+        _read_cpu_mem_usage();
+        _read_cpu_freqs();
+        for(auto itr : device_list)
+        {
+            OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug(),
+                                              "Polling rocm-smi for device %u...\n", itr);
+            auto& _data = *_bundle_data.at(itr);
+            if(!_data) continue;
+            _data->emplace_back(data{ itr });
+            OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug(), "    %s\n",
+                                              TIMEMORY_JOIN("", _data->back()).c_str());
+        }
+        while(_now < std::chrono::steady_clock::now())
+            _now += _interval;
+    }
+    OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug(), "Polling rocm-smi completed...\n");
+
+    if(polling_finished) polling_finished->set_value();
+}
+
+std::vector<data>&
+data::get_initial()
+{
+    static std::vector<data> _v{};
+    return _v;
+}
+
+std::unique_ptr<std::thread>&
+data::get_thread()
+{
+    static std::unique_ptr<std::thread> _v;
+    return _v;
+}
+
+void
+data::set_state(State _state)
+{
+    get_rocm_smi_state().store(_state);
+}
+
+bool
+data::setup()
+{
+    perfetto_counter_track<cpu_freq>::init();
+    perfetto_counter_track<cpu_mem>::init();
+    perfetto_counter_track<data>::init();
+
+    // shutdown if already running
+    shutdown();
+
+    OMNITRACE_DEBUG("Configuring rocm-smi...\n");
+
+    auto     _freq      = get_rocm_smi_freq();
+    uint64_t _msec_freq = (1.0 / _freq) * 1.0e3;
+
+    promise_t _prom{};
+    auto      _fut   = _prom.get_future();
+    polling_finished = std::make_unique<promise_t>();
+
+    set_state(State::PreInit);
+    get_thread() = std::make_unique<std::thread>(
+        &data::poll<msec_t>, &get_rocm_smi_state(), msec_t{ _msec_freq }, &_prom);
+
+    _fut.wait();
+    return true;
+}
+
+bool
+data::shutdown()
+{
+    auto& _thread = get_thread();
+    if(_thread)
+    {
+        OMNITRACE_DEBUG("Shutting down rocm-smi...\n");
+        set_state(State::Finalized);
+        if(polling_finished)
+        {
+            auto     _fut  = polling_finished->get_future();
+            uint64_t _freq = (1.0 / get_rocm_smi_freq()) * 1.0e3;
+            _fut.wait_for(msec_t{ 5 * _freq });
+            _thread->join();
+        }
+        else
+        {
+            uint64_t _freq = (1.0 / get_rocm_smi_freq()) * 1.0e3;
+            std::this_thread::sleep_for(msec_t{ 5 * _freq });
+            pthread_cancel(_thread->native_handle());
+            _thread->detach();
+        }
+        _thread          = std::unique_ptr<std::thread>{};
+        polling_finished = std::unique_ptr<promise_t>{};
+        return true;
+    }
+    return false;
+}
+
+#define GPU_METRIC(COMPONENT, ...)                                                       \
+    if constexpr(tim::trait::is_available<COMPONENT>::value)                             \
+    {                                                                                    \
+        auto* _val = _v.get<COMPONENT>();                                                \
+        if(_val)                                                                         \
+        {                                                                                \
+            _val->set_value(itr.__VA_ARGS__);                                            \
+            _val->set_accum(itr.__VA_ARGS__);                                            \
+        }                                                                                \
+    }
+
+void
+data::post_process(uint32_t _dev_id)
+{
+    OMNITRACE_CONDITIONAL_PRINT(get_debug() || get_verbose() > 0,
+                                "Post-processing rocm-smi data for device %u\n", _dev_id);
+
+    using component::sampling_gpu_busy;
+    using component::sampling_gpu_memory;
+    using component::sampling_gpu_power;
+    using component::sampling_gpu_temp;
+    using bundle_t = tim::lightweight_tuple<sampling_gpu_busy, sampling_gpu_temp,
+                                            sampling_gpu_power, sampling_gpu_memory>;
+
+    auto _process_frequencies = [](size_t _idx) {
+        using counter_track = perfetto_counter_track<cpu_freq>;
+        if(!counter_track::exists(_idx))
+        {
+            auto _devname = TIMEMORY_JOIN("", "[CPU ", _idx, "] ");
+            auto addendum = [&](const char* _v) { return _devname + std::string{ _v }; };
+            counter_track::emplace(_idx, addendum("Frequency (S)"), "MHz");
+        }
+
+        for(auto& itr : cpu_frequencies.at(_idx))
+        {
+            uint64_t _ts   = itr.first;
+            double   _freq = itr.second;
+            TRACE_COUNTER("sampling", counter_track::at(_idx, 0), _ts, _freq);
+        }
+    };
+
+    auto _process_cpu_mem_usage = []() {
+        using counter_track = perfetto_counter_track<cpu_mem>;
+        if(!counter_track::exists(0))
+        {
+            auto _devname = TIMEMORY_JOIN("", "[CPU] ");
+            auto addendum = [&](const char* _v) { return _devname + std::string{ _v }; };
+            counter_track::emplace(0, addendum("Memory Usage (S)"), "MB");
+        }
+
+        for(auto& itr : cpu_mem_usage)
+        {
+            uint64_t _ts        = itr.first;
+            double   _mem_usage = itr.second;
+            TRACE_COUNTER("sampling", counter_track::at(0, 0), _ts,
+                          _mem_usage / units::megabyte);
+        }
+    };
+
+    static bool _once = false;
+    if(!_once)
+    {
+        _once = true;
+        _process_cpu_mem_usage();
+        for(size_t i = 0; i < cpu_frequencies.size(); ++i)
+            _process_frequencies(i);
+    }
+
+    if(device_count < _dev_id) return;
+
+    auto& _rocm_smi_v = sampler_instances::instances().at(_dev_id);
+    auto  _rocm_smi   = (_rocm_smi_v) ? *_rocm_smi_v : std::deque<rocm_smi::data>{};
+
+    auto _process_perfetto = [&]() {
+        for(auto& itr : _rocm_smi)
+        {
+            using counter_track = perfetto_counter_track<data>;
+            if(itr.m_dev_id != _dev_id) continue;
+            if(!counter_track::exists(_dev_id))
+            {
+                auto _devname = TIMEMORY_JOIN("", "[GPU ", _dev_id, "] ");
+                auto addendum = [&](const char* _v) {
+                    return _devname + std::string{ _v };
+                };
+                counter_track::emplace(_dev_id, addendum("Busy"), "%");
+                counter_track::emplace(_dev_id, addendum("Temperature"), "deg C");
+                counter_track::emplace(_dev_id, addendum("Power"), "watts");
+                counter_track::emplace(_dev_id, addendum("Memory Usage"), "megabytes");
+            }
+            uint64_t _ts    = itr.m_ts;
+            double   _busy  = itr.m_busy_perc;
+            double   _temp  = itr.m_temp / 1.0e3;
+            double   _power = itr.m_power / 1.0e6;
+            double   _usage = itr.m_mem_usage / static_cast<double>(units::megabyte);
+            TRACE_COUNTER("rocm_smi", counter_track::at(_dev_id, 0), _ts, _busy);
+            TRACE_COUNTER("rocm_smi", counter_track::at(_dev_id, 1), _ts, _temp);
+            TRACE_COUNTER("rocm_smi", counter_track::at(_dev_id, 2), _ts, _power);
+            TRACE_COUNTER("rocm_smi", counter_track::at(_dev_id, 3), _ts, _usage);
+        }
+    };
+
+    if(get_use_perfetto()) _process_perfetto();
+
+    if(!get_use_timemory()) return;
+
+    for(auto& itr : _rocm_smi)
+    {
+        using entry_t = critical_trace::entry;
+        auto _ts      = itr.m_ts;
+        auto _entries = critical_trace::get_entries(_ts, [](const entry_t& _e) {
+            return _e.device == critical_trace::Device::GPU;
+        });
+
+        std::vector<bundle_t> _tc{};
+        _tc.reserve(_entries.size());
+        for(auto& eitr : _entries)
+        {
+            auto& _v = _tc.emplace_back(eitr.first);
+            _v.push();
+            _v.start();
+            _v.stop();
+
+            GPU_METRIC(sampling_gpu_busy, m_busy_perc)
+            GPU_METRIC(sampling_gpu_temp, m_temp / 1.0e3)  // provided in milli-degree C
+            GPU_METRIC(sampling_gpu_power,
+                       m_power * units::microwatt / static_cast<double>(units::watt))
+            GPU_METRIC(sampling_gpu_memory,
+                       m_mem_usage / static_cast<double>(units::megabyte))
+
+            _v.pop();
+        }
+    }
+}
+
+//--------------------------------------------------------------------------------------//
+
+void
+setup()
+{
+    auto_lock_t _lk{ type_mutex<api::rocm_smi>() };
+
+    if(is_initialized() || !get_use_rocm_smi()) return;
+
+    pthread_gotcha::enable_sampling_on_child_threads() = false;
+
+    // assign the data value to determined by rocm-smi
+    data::device_count = device_count();
+
+    auto _devices_v = get_rocm_smi_devices();
+    for(auto& itr : _devices_v)
+        itr = tolower(itr);
+    bool _all_devices = _devices_v.find("all") != std::string::npos || _devices_v.empty();
+    bool _no_devices  = _devices_v.find("none") != std::string::npos;
+
+    std::set<uint32_t> _devices{};
+    if(_all_devices)
+    {
+        for(uint32_t i = 0; i < data::device_count; ++i)
+            _devices.emplace(i);
+    }
+    else if(!_no_devices)
+    {
+        for(auto&& itr : tim::delimit(get_rocm_smi_devices()))
+        {
+            uint32_t idx = std::stoul(itr);
+            if(idx < data::device_count) _devices.emplace(idx);
+        }
+    }
+
+    data::device_list = _devices;
+
+    for(auto itr : _devices)
+    {
+        uint16_t dev_id = 0;
+        OMNITRACE_ROCM_SMI_CALL(rsmi_dev_id_get(itr, &dev_id));
+        // dev_id holds the device ID of device i, upon a successful call
+    }
+
+    is_initialized() = true;
+
+    data::setup();
+
+    pthread_gotcha::enable_sampling_on_child_threads() = true;
+    omnitrace::rocm_smi::data::set_state(State::Active);
+}
+
+void
+shutdown()
+{
+    auto_lock_t _lk{ type_mutex<api::rocm_smi>() };
+
+    if(!is_initialized()) return;
+
+    if(data::shutdown())
+    {
+        OMNITRACE_ROCM_SMI_CALL(rsmi_shut_down());
+    }
+
+    is_initialized() = false;
+}
+
+uint32_t
+device_count()
+{
+    uint32_t _num_devices = 0;
+    try
+    {
+        static auto _rsmi_init_once = []() { OMNITRACE_ROCM_SMI_CALL(rsmi_init(0)); };
+        static std::once_flag _once{};
+        std::call_once(_once, _rsmi_init_once);
+
+        OMNITRACE_ROCM_SMI_CALL(rsmi_num_monitor_devices(&_num_devices));
+    } catch(const std::exception& _e)
+    {
+        OMNITRACE_BASIC_PRINT("Exception: %s\n", _e.what());
+    }
+    return _num_devices;
+}
+}  // namespace rocm_smi
+}  // namespace omnitrace
+
+TIMEMORY_INSTANTIATE_EXTERN_COMPONENT(
+    TIMEMORY_ESC(data_tracker<double, omnitrace::component::backtrace_gpu_busy>), true,
+    double)
+
+TIMEMORY_INSTANTIATE_EXTERN_COMPONENT(
+    TIMEMORY_ESC(data_tracker<double, omnitrace::component::backtrace_gpu_temp>), true,
+    double)
+
+TIMEMORY_INSTANTIATE_EXTERN_COMPONENT(
+    TIMEMORY_ESC(data_tracker<double, omnitrace::component::backtrace_gpu_power>), true,
+    double)
+
+TIMEMORY_INSTANTIATE_EXTERN_COMPONENT(
+    TIMEMORY_ESC(data_tracker<double, omnitrace::component::backtrace_gpu_memory>), true,
+    double)
@@ -21,11 +21,16 @@
 // SOFTWARE.

 #include "library/components/roctracer.hpp"
+#include "library/components/pthread_gotcha.hpp"
+#include "library/components/rocm_smi.hpp"
 #include "library/components/roctracer_callbacks.hpp"
 #include "library/config.hpp"
 #include "library/defines.hpp"
+#include "library/redirect.hpp"
+#include "library/sampling.hpp"
 #include "library/thread_data.hpp"

+namespace rocm_smi = omnitrace::rocm_smi;
 using namespace omnitrace;

 namespace tim
@@ -35,11 +40,22 @@ namespace component
 void
 roctracer::preinit()
 {
-    OMNITRACE_DEBUG("[%s]\n", __FUNCTION__);
    roctracer_data::label()       = "roctracer";
    roctracer_data::description() = "ROCm tracer (activity API)";
 }

+void
+roctracer::start()
+{
+    if(tracker_type::start() == 0) setup();
+}
+
+void
+roctracer::stop()
+{
+    if(tracker_type::stop() == 0) shutdown();
+}
+
 bool
 roctracer::is_setup()
 {
@@ -53,9 +69,9 @@ roctracer::add_setup(const std::string& _lbl, std::function<void()>&& _func)
 }

 void
-roctracer::add_tear_down(const std::string& _lbl, std::function<void()>&& _func)
+roctracer::add_shutdown(const std::string& _lbl, std::function<void()>&& _func)
 {
-    roctracer_tear_down_routines().emplace_back(_lbl, std::move(_func));
+    roctracer_shutdown_routines().emplace_back(_lbl, std::move(_func));
 }

 void
@@ -73,7 +89,7 @@ roctracer::remove_setup(const std::string& _lbl)
 }

 void
-roctracer::remove_tear_down(const std::string& _lbl)
+roctracer::remove_shutdown(const std::string& _lbl)
 {
    auto& _data = roctracer_setup_routines();
    for(auto itr = _data.begin(); itr != _data.end(); ++itr)
@@ -89,31 +105,44 @@ roctracer::remove_tear_down(const std::string& _lbl)
 void
 roctracer::setup()
 {
-    if(!get_use_timemory() && !get_use_perfetto()) return;
+    trait::runtime_enabled<roctracer_data>::set(get_use_timemory());
+
+    if(!get_use_roctracer()) return;

    auto_lock_t _lk{ type_mutex<roctracer>() };
    if(roctracer_is_setup()) return;
    roctracer_is_setup() = true;
+
    OMNITRACE_DEBUG("[%s]\n", __FUNCTION__);

-    tim::set_env("HSA_TOOLS_LIB", "libomnitrace.so", 0);
-
+#if OMNITRACE_HIP_VERSION_MAJOR == 4 && OMNITRACE_HIP_VERSION_MINOR < 4
    auto _kfdwrapper = dynamic_library{ "OMNITRACE_ROCTRACER_LIBKFDWRAPPER",
                                        OMNITRACE_ROCTRACER_LIBKFDWRAPPER };
+#endif

-    ROCTRACER_CALL(roctracer_set_properties(ACTIVITY_DOMAIN_HIP_API, nullptr));
+    ROCTRACER_CALL(
+        roctracer_set_properties(ACTIVITY_DOMAIN_HIP_API, (void*) hip_api_callback));

-    // if(roctracer_default_pool() == nullptr)
+    if(roctracer_default_pool() == nullptr)
    {
        // Allocating tracing pool
        roctracer_properties_t properties{};
        memset(&properties, 0, sizeof(roctracer_properties_t));
-        properties.mode                = 0x1000;
+        // properties.mode                = 0x1000;
        properties.buffer_size         = 0x1000;
        properties.buffer_callback_fun = hip_activity_callback;
        ROCTRACER_CALL(roctracer_open_pool(&properties));
    }

+#if OMNITRACE_HIP_VERSION_MAJOR == 4 && OMNITRACE_HIP_VERSION_MINOR >= 4 &&              \
+    OMNITRACE_HIP_VERSION_MINOR <= 5
+    // HIP 4.5.0 has an invalid warning
+    redirect _rd{ std::cerr, "roctracer_enable_callback(), get_op_end(), invalid domain "
+                             "ID(4)  in: roctracer_enable_callback(hip_api_callback, "
+                             "nullptr)roctracer_enable_activity_expl(), get_op_end(), "
+                             "invalid domain ID(4)  in: roctracer_enable_activity()" };
+#endif
+
    // Enable API callbacks, all domains
    ROCTRACER_CALL(roctracer_enable_callback(hip_api_callback, nullptr));
    // Enable activity tracing, all domains
@@ -125,7 +154,7 @@ roctracer::setup()
 }

 void
-roctracer::tear_down()
+roctracer::shutdown()
 {
    auto_lock_t _lk{ type_mutex<roctracer>() };
    if(!roctracer_is_setup()) return;
@@ -133,37 +162,48 @@ roctracer::tear_down()
    OMNITRACE_DEBUG("[%s]\n", __FUNCTION__);

    // flush all the activity
-    // if(roctracer_default_pool() != nullptr)
+    if(roctracer_default_pool() != nullptr)
    {
+        OMNITRACE_DEBUG("[%s] roctracer_flush_activity\n", __FUNCTION__);
        ROCTRACER_CALL(roctracer_flush_activity());
-    }
-    // flush all buffers
-    roctracer_flush_buf();

+        // flush all buffers
+        OMNITRACE_DEBUG("[%s] roctracer_flush_buf\n", __FUNCTION__);
+        roctracer_flush_buf();
+    }
+
+    OMNITRACE_DEBUG("[%s] executing hip_exec_activity_callbacks\n", __FUNCTION__);
    // make sure all async operations are executed
    for(size_t i = 0; i < max_supported_threads; ++i)
        hip_exec_activity_callbacks(i);

    // callback for hsa
-    for(auto& itr : roctracer_tear_down_routines())
+    OMNITRACE_DEBUG("[%s] executing roctracer_shutdown_routines...\n", __FUNCTION__);
+    for(auto& itr : roctracer_shutdown_routines())
        itr.second();

+#if OMNITRACE_HIP_VERSION_MAJOR == 4 && OMNITRACE_HIP_VERSION_MINOR >= 4 &&              \
+    OMNITRACE_HIP_VERSION_MINOR <= 5
+    OMNITRACE_DEBUG("[%s] redirecting roctracer warnings\n", __FUNCTION__);
+    // HIP 4.5.0 has an invalid warning
+    redirect _rd{
+        std::cerr, "roctracer_disable_callback(), get_op_end(), invalid domain ID(4)  "
+                   "in: roctracer_disable_callback()roctracer_disable_activity(), "
+                   "get_op_end(), invalid domain ID(4)  in: roctracer_disable_activity()"
+    };
+#endif
+
    // Disable tracing and closing the pool
+    OMNITRACE_DEBUG("[%s] roctracer_disable_callback\n", __FUNCTION__);
    ROCTRACER_CALL(roctracer_disable_callback());
+
+    OMNITRACE_DEBUG("[%s] roctracer_disable_activity\n", __FUNCTION__);
    ROCTRACER_CALL(roctracer_disable_activity());
+
+    OMNITRACE_DEBUG("[%s] roctracer_close_pool\n", __FUNCTION__);
    ROCTRACER_CALL(roctracer_close_pool());
-}

-void
-roctracer::start()
-{
-    if(tracker_type::start() == 0) setup();
-}
-
-void
-roctracer::stop()
-{
-    if(tracker_type::stop() == 0) tear_down();
+    OMNITRACE_DEBUG("[%s] roctracer is shutdown\n", __FUNCTION__);
 }
 }  // namespace component
 }  // namespace tim
@@ -181,92 +221,103 @@ extern "C"
    bool OnLoad(HsaApiTable* table, uint64_t runtime_version, uint64_t failed_tool_count,
                const char* const* failed_tool_names)
    {
-        OMNITRACE_DEBUG("[%s]\n", __FUNCTION__);
+        pthread_gotcha::enable_sampling_on_child_threads() = false;
+        OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug_env() || get_verbose_env() > 0,
+                                          "[%s]\n", __FUNCTION__);
        tim::consume_parameters(table, runtime_version, failed_tool_count,
                                failed_tool_names);

-        // ONLOAD_TRACE_BEG();
-        // on_exit(exit_handler, nullptr);
-
        auto _setup = [=]() {
-            get_hsa_timer() =
-                std::make_unique<hsa_timer_t>(table->core_->hsa_system_get_info_fn);
-
-            // const char* output_prefix = getenv("ROCP_OUTPUT_DIR");
-            const char* output_prefix = nullptr;
-
-            // App begin timestamp begin_ts_file.txt
-            // begin_ts_file_handle = open_output_file(output_prefix,
-            // "begin_ts_file.txt"); const timestamp_t app_start_time =
-            // timer->timestamp_fn_ns(); fprintf(begin_ts_file_handle, "%lu\n",
-            // app_start_time);
-
-            bool                     trace_hsa_api = get_trace_hsa_api();
-            std::vector<std::string> hsa_api_vec =
-                tim::delimit(get_trace_hsa_api_types());
-
-            // Enable HSA API callbacks/activity
-            if(trace_hsa_api)
+            try
            {
-                // hsa_api_file_handle = open_output_file(output_prefix,
-                // "hsa_api_trace.txt");
+                OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug() || get_verbose() > 1,
+                                                  "[%s] setting up HSA...\n",
+                                                  __FUNCTION__);

-                // initialize HSA tracing
-                roctracer_set_properties(ACTIVITY_DOMAIN_HSA_API, (void*) table);
+                get_hsa_timer() =
+                    std::make_unique<hsa_timer_t>(table->core_->hsa_system_get_info_fn);

-                OMNITRACE_DEBUG("    HSA-trace(");
-                if(!hsa_api_vec.empty())
+                // const char* output_prefix = getenv("ROCP_OUTPUT_DIR");
+                const char* output_prefix = nullptr;
+
+                bool trace_hsa_api = get_trace_hsa_api();
+
+                // Enable HSA API callbacks/activity
+                if(trace_hsa_api)
                {
-                    for(const auto& itr : hsa_api_vec)
+                    std::vector<std::string> hsa_api_vec =
+                        tim::delimit(get_trace_hsa_api_types());
+
+                    // initialize HSA tracing
+                    roctracer_set_properties(ACTIVITY_DOMAIN_HSA_API, (void*) table);
+
+                    OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug() || get_verbose() > 1,
+                                                      "    HSA-trace(");
+                    if(!hsa_api_vec.empty())
                    {
-                        uint32_t    cid = HSA_API_ID_NUMBER;
-                        const char* api = itr.c_str();
-                        ROCTRACER_CALL(roctracer_op_code(ACTIVITY_DOMAIN_HSA_API, api,
-                                                         &cid, nullptr));
-                        ROCTRACER_CALL(roctracer_enable_op_callback(
-                            ACTIVITY_DOMAIN_HSA_API, cid, hsa_api_callback, nullptr));
+                        for(const auto& itr : hsa_api_vec)
+                        {
+                            uint32_t    cid = HSA_API_ID_NUMBER;
+                            const char* api = itr.c_str();
+                            ROCTRACER_CALL(roctracer_op_code(ACTIVITY_DOMAIN_HSA_API, api,
+                                                             &cid, nullptr));
+                            ROCTRACER_CALL(roctracer_enable_op_callback(
+                                ACTIVITY_DOMAIN_HSA_API, cid, hsa_api_callback, nullptr));

-                        OMNITRACE_DEBUG(" %s", api);
+                            OMNITRACE_CONDITIONAL_BASIC_PRINT(
+                                get_debug() || get_verbose() > 1, " %s", api);
+                        }
                    }
+                    else
+                    {
+                        ROCTRACER_CALL(roctracer_enable_domain_callback(
+                            ACTIVITY_DOMAIN_HSA_API, hsa_api_callback, nullptr));
+                    }
+                    OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug() || get_verbose() > 1,
+                                                      "\n");
                }
-                else
+
+                bool trace_hsa_activity = get_trace_hsa_activity();
+                // Enable HSA GPU activity
+                if(trace_hsa_activity)
                {
-                    ROCTRACER_CALL(roctracer_enable_domain_callback(
-                        ACTIVITY_DOMAIN_HSA_API, hsa_api_callback, nullptr));
+                    // initialize HSA tracing
+                    ::roctracer::hsa_ops_properties_t ops_properties{
+                        table,
+                        reinterpret_cast<activity_async_callback_t>(
+                            hsa_activity_callback),
+                        nullptr, output_prefix
+                    };
+                    roctracer_set_properties(ACTIVITY_DOMAIN_HSA_OPS, &ops_properties);
+
+                    OMNITRACE_CONDITIONAL_BASIC_PRINT(get_debug() || get_verbose() > 1,
+                                                      "    HSA-activity-trace()\n");
+                    ROCTRACER_CALL(roctracer_enable_op_activity(ACTIVITY_DOMAIN_HSA_OPS,
+                                                                HSA_OP_ID_COPY));
                }
-                OMNITRACE_DEBUG("\n");
-            }
-
-            bool trace_hsa_activity = get_trace_hsa_activity();
-            // Enable HSA GPU activity
-            if(trace_hsa_activity)
+            } catch(std::exception& _e)
            {
-                // initialize HSA tracing
-                ::roctracer::hsa_ops_properties_t ops_properties{
-                    table,
-                    reinterpret_cast<activity_async_callback_t>(hsa_activity_callback),
-                    nullptr, output_prefix
-                };
-                roctracer_set_properties(ACTIVITY_DOMAIN_HSA_OPS, &ops_properties);
-
-                OMNITRACE_DEBUG("    HSA-activity-trace()\n");
-                ROCTRACER_CALL(roctracer_enable_op_activity(ACTIVITY_DOMAIN_HSA_OPS,
-                                                            HSA_OP_ID_COPY));
+                OMNITRACE_BASIC_PRINT("Exception was thrown in HSA setup: %s\n",
+                                      _e.what());
            }
        };

-        auto _tear_down = []() {
+        auto _shutdown = []() {
+            OMNITRACE_DEBUG("[%s] roctracer_disable_domain_callback\n", __FUNCTION__);
            ROCTRACER_CALL(roctracer_disable_domain_callback(ACTIVITY_DOMAIN_HSA_API));

+            OMNITRACE_DEBUG("[%s] roctracer_disable_op_activity\n", __FUNCTION__);
            ROCTRACER_CALL(
                roctracer_disable_op_activity(ACTIVITY_DOMAIN_HSA_OPS, HSA_OP_ID_COPY));
        };

-        if(comp::roctracer::is_setup()) _setup();
-
        comp::roctracer::add_setup("hsa", std::move(_setup));
-        comp::roctracer::add_tear_down("hsa", std::move(_tear_down));
+        comp::roctracer::add_shutdown("hsa", std::move(_shutdown));

+        rocm_smi::setup();
+        comp::roctracer::setup();
+
+        pthread_gotcha::enable_sampling_on_child_threads() = true;
        return true;
    }

@@ -274,6 +325,7 @@ extern "C"
    void OnUnload()
    {
        OMNITRACE_DEBUG("[%s]\n", __FUNCTION__);
-        // ONLOAD_TRACE("");
+        rocm_smi::shutdown();
+        comp::roctracer::shutdown();
    }
 }
@@ -24,6 +24,7 @@
 #include "library.hpp"
 #include "library/config.hpp"
 #include "library/critical_trace.hpp"
+#include "library/sampling.hpp"
 #include "library/thread_data.hpp"

 #include <timemory/backends/threading.hpp>
@@ -46,7 +47,7 @@ auto&
 get_roctracer_hip_data(int64_t _tid = threading::get_id())
 {
    using data_t        = std::unordered_map<uint64_t, roctracer_bundle_t>;
-    using thread_data_t = omnitrace_thread_data<data_t, api::roctracer>;
+    using thread_data_t = thread_data<data_t, api::roctracer>;
    static auto& _v     = thread_data_t::instances(thread_data_t::construct_on_init{});
    return _v.at(_tid);
 }
@@ -76,9 +77,8 @@ get_roctracer_cid_data()
 auto&
 get_hip_activity_callbacks(int64_t _tid = threading::get_id())
 {
-    using thread_data_t =
-        omnitrace_thread_data<std::vector<std::function<void()>>, api::roctracer>;
-    static auto& _v = thread_data_t::instances(thread_data_t::construct_on_init{});
+    using thread_data_t = thread_data<std::vector<std::function<void()>>, api::roctracer>;
+    static auto& _v     = thread_data_t::instances(thread_data_t::construct_on_init{});
    return _v.at(_tid);
 }

@@ -214,11 +214,17 @@ hsa_api_callback(uint32_t domain, uint32_t cid, const void* callback_data, void*
 void
 hsa_activity_callback(uint32_t op, activity_record_t* record, void* arg)
 {
+    if(get_state() != State::Active || !trait::runtime_enabled<comp::roctracer>::get())
+        return;
+
    static const char* copy_op_name     = "hsa_async_copy";
    static const char* dispatch_op_name = "hsa_dispatch";
    static const char* barrier_op_name  = "hsa_barrier";
    const char**       _name            = nullptr;

+    static thread_local auto _once = (threading::set_thread_name("omni.roctracer"), true);
+    (void) _once;
+
    switch(op)
    {
        case HSA_OP_ID_DISPATCH: _name = &dispatch_op_name; break;
@@ -318,7 +324,7 @@ hip_api_callback(uint32_t domain, uint32_t cid, const void* callback_data, void*
                                          data->args.hipLaunchKernel.stream);
                if(_name != nullptr)
                {
-                    if(get_use_perfetto() || get_use_timemory())
+                    if(get_use_perfetto() || get_use_timemory() || get_use_rocm_smi())
                    {
                        tim::auto_lock_t _lk{ tim::type_mutex<key_data_mutex_t>() };
                        get_roctracer_key_data().emplace(data->correlation_id, _name);
@@ -333,7 +339,7 @@ hip_api_callback(uint32_t domain, uint32_t cid, const void* callback_data, void*
                const char* _name = hipKernelNameRef(data->args.hipModuleLaunchKernel.f);
                if(_name != nullptr)
                {
-                    if(get_use_perfetto() || get_use_timemory())
+                    if(get_use_perfetto() || get_use_timemory() || get_use_rocm_smi())
                    {
                        tim::auto_lock_t _lk{ tim::type_mutex<key_data_mutex_t>() };
                        get_roctracer_key_data().emplace(data->correlation_id, _name);
@@ -345,7 +351,7 @@ hip_api_callback(uint32_t domain, uint32_t cid, const void* callback_data, void*
            }
            default:
            {
-                if(get_use_perfetto() || get_use_timemory())
+                if(get_use_perfetto() || get_use_timemory() || get_use_rocm_smi())
                {
                    tim::auto_lock_t _lk{ tim::type_mutex<key_data_mutex_t>() };
                    get_roctracer_key_data().emplace(data->correlation_id, op_name);
@@ -373,7 +379,7 @@ hip_api_callback(uint32_t domain, uint32_t cid, const void* callback_data, void*
                get_roctracer_hip_data()->erase(itr.first);
            }
        }
-        if(get_use_critical_trace())
+        if(get_use_critical_trace() || get_use_rocm_smi())
        {
            auto     _cid        = get_cpu_cid()++;
            uint16_t _depth      = (get_cpu_cid_stack()->empty())
@@ -422,7 +428,7 @@ hip_api_callback(uint32_t domain, uint32_t cid, const void* callback_data, void*
                }
            }
        }
-        if(get_use_critical_trace())
+        if(get_use_critical_trace() || get_use_rocm_smi())
        {
            uint16_t _depth      = 0;
            uint64_t _cid        = 0;
@@ -445,6 +451,14 @@ hip_api_callback(uint32_t domain, uint32_t cid, const void* callback_data, void*
 void
 hip_activity_callback(const char* begin, const char* end, void*)
 {
+    if(get_state() != State::Active || !trait::runtime_enabled<comp::roctracer>::get())
+        return;
+
+    sampling::block_signals();
+
+    static thread_local auto _once = (threading::set_thread_name("omni.roctracer"), true);
+    (void) _once;
+
    using Device = critical_trace::Device;
    using Phase  = critical_trace::Phase;

@@ -494,7 +508,7 @@ hip_activity_callback(const char* begin, const char* end, void*)
        auto        _laps           = _indexes[_corr_id]++;  // see note #1
        const char* _name           = nullptr;
        bool        _found          = false;
-        bool        _critical_trace = get_use_critical_trace();
+        bool        _critical_trace = get_use_critical_trace() || get_use_rocm_smi();

        {
            tim::auto_lock_t _lk{ tim::type_mutex<key_data_mutex_t>() };
@@ -597,7 +611,7 @@ roctracer_setup_routines()
 }

 roctracer_functions_t&
-roctracer_tear_down_routines()
+roctracer_shutdown_routines()
 {
    static auto _v = roctracer_functions_t{};
    return _v;
@@ -54,12 +54,30 @@ get_config()
    (void) _once;
 }

+std::string
+get_setting_name(std::string _v)
+{
+    static const auto _prefix = tim::string_view_t{ "omnitrace_" };
+    for(auto& itr : _v)
+        itr = tolower(itr);
+    auto _pos = _v.find(_prefix);
+    if(_pos == 0) return _v.substr(_prefix.length());
+    return _v;
+}
+
 #define OMNITRACE_CONFIG_SETTING(TYPE, ENV_NAME, DESCRIPTION, INITIAL_VALUE, ...)        \
-    _config->insert<TYPE, TYPE>(                                                         \
-        ENV_NAME, ENV_NAME, DESCRIPTION, INITIAL_VALUE,                                  \
-        std::set<std::string>{ "custom", "omnitrace", __VA_ARGS__ })
+    {                                                                                    \
+        auto _ret = _config->insert<TYPE, TYPE>(                                         \
+            ENV_NAME, get_setting_name(ENV_NAME), DESCRIPTION, INITIAL_VALUE,            \
+            std::set<std::string>{ "custom", "omnitrace", __VA_ARGS__ });                \
+        if(!_ret.second)                                                                 \
+            OMNITRACE_PRINT("Warning! Duplicate setting: %s / %s\n",                     \
+                            get_setting_name(ENV_NAME).c_str(), ENV_NAME);               \
+    }
 }  // namespace

+inline namespace config
+{
 void
 configure_settings()
 {
@@ -67,25 +85,24 @@ configure_settings()
    if(_once) return;
    _once = true;

+    OMNITRACE_CONDITIONAL_THROW(
+        get_state() < State::Init,
+        "config::configure_settings() called before omnitrace_init_library. state = %s",
+        std::to_string(get_state()).c_str());
+
+    OMNITRACE_CONDITIONAL_THROW(
+        get_state() > State::Init,
+        "config::configure_settings() called after omnitrace was initialized. state = %s",
+        std::to_string(get_state()).c_str());
+
    static auto _config = settings::shared_instance();
-    // auto* _config = settings::instance();

    // if using timemory, default to perfetto being off
    auto _default_perfetto_v =
        !tim::get_env<bool>("OMNITRACE_USE_TIMEMORY", false, false);

-    auto _default_config_file =
-        JOIN("/", tim::get_env<std::string>("HOME", "."), "omnitrace.cfg");
-
    auto _system_backend = tim::get_env("OMNITRACE_BACKEND_SYSTEM", false, false);

-    OMNITRACE_CONFIG_SETTING(std::string, "OMNITRACE_CONFIG_FILE",
-                             "Configuration file of omnitrace and timemory settings",
-                             _default_config_file, "config");
-
-    OMNITRACE_CONFIG_SETTING(bool, "OMNITRACE_DEBUG", "Enable debugging output",
-                             _config->get_debug(), "debugging");
-
    auto _omnitrace_debug = _config->get<bool>("OMNITRACE_DEBUG");
    if(_omnitrace_debug) tim::set_env("TIMEMORY_DEBUG_SETTINGS", "1", 0);

@@ -101,6 +118,22 @@ configure_settings()
                             "backend", "roctracer");
 #endif

+#if defined(OMNITRACE_USE_ROCM_SMI)
+    OMNITRACE_CONFIG_SETTING(
+        bool, "OMNITRACE_USE_ROCM_SMI",
+        "Enable sampling GPU power, temp, utilization, and memory usage", true, "backend",
+        "rocm-smi");
+
+    OMNITRACE_CONFIG_SETTING(std::string, "OMNITRACE_ROCM_SMI_DEVICES",
+                             "Devices to query when OMNITRACE_USE_ROCM_SMI=ON", "all",
+                             "backend", "rocm-smi");
+
+    OMNITRACE_CONFIG_SETTING(
+        double, "OMNITRACE_ROCM_SMI_FREQ",
+        "Number of rocm-smi samples per second when OMNITTRACE_USE_ROCM_SMI=ON", 5.0,
+        "backend", "rocm-smi");
+#endif
+
    OMNITRACE_CONFIG_SETTING(bool, "OMNITRACE_USE_SAMPLING",
                             "Enable statistical sampling of call-stack", false,
                             "backend", "sampling");
@@ -110,6 +143,9 @@ configure_settings()
        "Enable tagging filenames with process identifier (either MPI rank or pid)", true,
        "io");

+    OMNITRACE_CONFIG_SETTING(bool, "OMNITRACE_USE_KOKKOSP",
+                             "Enable support for Kokkos Tools", true, "kokkos");
+
    OMNITRACE_CONFIG_SETTING(size_t, "OMNITRACE_INSTRUMENTATION_INTERVAL",
                             "Instrumentation only takes measurements once every N "
                             "function calls (not statistical)",
@@ -225,6 +261,40 @@ configure_settings()
    _config->get_stack_clearing()        = false;
    _config->get_time_output()           = true;
    _config->get_timing_precision()      = 6;
+    _config->get_max_thread_bookmarks()  = 1;
+    _config->get_timing_units()          = "sec";
+    _config->get_memory_units()          = "MB";
+    _config->get_papi_events()           = "PAPI_TOT_CYC, PAPI_TOT_INS";
+
+#if defined(TIMEMORY_USE_PAPI)
+    int _paranoid = 2;
+    {
+        std::ifstream _fparanoid{ "/proc/sys/kernel/perf_event_paranoid" };
+        if(_fparanoid) _fparanoid >> _paranoid;
+    }
+
+    if(_paranoid > 1)
+    {
+        OMNITRACE_CONDITIONAL_BASIC_PRINT(
+            get_verbose_env() >= 0,
+            "/proc/sys/kernel/perf_event_paranoid has a value of %i. "
+            "Disabling PAPI (requires a value <= 1)...\n",
+            _paranoid);
+        OMNITRACE_CONDITIONAL_BASIC_PRINT(
+            get_verbose_env() >= 0,
+            "In order to enable PAPI support, run 'echo N | sudo tee "
+            "/proc/sys/kernel/perf_event_paranoid' where N is < 2\n");
+        tim::trait::runtime_enabled<comp::papi_common>::set(false);
+        tim::trait::runtime_enabled<comp::papi_array_t>::set(false);
+        tim::trait::runtime_enabled<comp::papi_vector>::set(false);
+        tim::trait::runtime_enabled<comp::cpu_roofline_flops>::set(false);
+        tim::trait::runtime_enabled<comp::cpu_roofline_dp_flops>::set(false);
+        tim::trait::runtime_enabled<comp::cpu_roofline_sp_flops>::set(false);
+        _config->get_papi_events() = "";
+    }
+#else
+    _config->get_papi_quiet() = true;
+#endif

    for(auto&& itr :
        tim::delimit(_config->get<std::string>("OMNITRACE_CONFIG_FILE"), ";:"))
@@ -265,15 +335,39 @@ configure_settings()
    if(tim::mpi::is_initialized()) settings::default_process_suffix() = tim::mpi::rank();
 #endif
    OMNITRACE_CONDITIONAL_BASIC_PRINT(get_verbose_env() > 0, "configuration complete\n");
+
+#if defined(OMNITRACE_USE_ROCM_SMI)
+    auto _rsmi_freq = _config->find("OMNITRACE_ROCM_SMI_FREQ");
+    if(_rsmi_freq != _config->end())
+    {
+        double& _rsmi_freq_v =
+            static_cast<tim::tsettings<double>&>(*_rsmi_freq->second).get();
+        if(_rsmi_freq_v > 1000) _rsmi_freq_v = 1000.;
+    }
+#endif
 }

 void
-print_config_settings(
+print_banner(std::ostream& _os)
+{
+    static const char* _banner = R"banner(
+
+      ______   .___  ___. .__   __.  __  .___________..______          ___       ______  _______
+     /  __  \  |   \/   | |  \ |  | |  | |           ||   _  \        /   \     /      ||   ____|
+    |  |  |  | |  \  /  | |   \|  | |  | `---|  |----`|  |_)  |      /  ^  \   |  ,----'|  |__
+    |  |  |  | |  |\/|  | |  . `  | |  |     |  |     |      /      /  /_\  \  |  |     |   __|
+    |  `--'  | |  |  |  | |  |\   | |  |     |  |     |  |\  \----./  _____  \ |  `----.|  |____
+     \______/  |__|  |__| |__| \__| |__|     |__|     | _| `._____/__/     \__\ \______||_______|
+
+    )banner";
+    _os << _banner << std::endl;
+}
+
+void
+print_settings(
    std::ostream&                                                                _ros,
    std::function<bool(const std::string_view&, const std::set<std::string>&)>&& _filter)
 {
-    if(get_verbose() < 1) return;
-
    OMNITRACE_CONDITIONAL_BASIC_PRINT(true, "configuration:\n");

    std::stringstream _os{};
@@ -370,6 +464,61 @@ print_config_settings(
    _ros << _os.str() << std::flush;
 }

+void
+print_settings()
+{
+    if(dmp::rank() > 0) return;
+
+    static std::set<tim::string_view_t> _sample_options = {
+        "OMNITRACE_SAMPLING_FREQ", "OMNITRACE_SAMPLING_DELAY",
+        "OMNITRACE_FLAT_SAMPLING", "OMNITRACE_TIMELINE_SAMPLING",
+        "OMNITRACE_FLAT_SAMPLING", "OMNITRACE_TIMELINE_SAMPLING",
+    };
+    static std::set<tim::string_view_t> _perfetto_options = {
+        "OMNITRACE_OUTPUT_FILE",
+        "OMNITRACE_BACKEND",
+        "OMNITRACE_SHMEM_SIZE_HINT_KB",
+        "OMNITRACE_BUFFER_SIZE_KB",
+    };
+    static std::set<tim::string_view_t> _timemory_options = {
+        "OMNITRACE_ROCTRACER_FLAT_PROFILE", "OMNITRACE_ROCTRACER_TIMELINE_PROFILE"
+    };
+    // generic filter for filtering relevant options
+    auto _is_omnitrace_option = [](const auto& _v, const auto& _c) {
+        if(!get_use_roctracer() && _v.find("OMNITRACE_ROCTRACER_") == 0) return false;
+        if(!get_use_critical_trace() && _v.find("OMNITRACE_CRITICAL_TRACE_") == 0)
+            return false;
+        if(!get_use_perfetto() && _perfetto_options.count(_v) > 0) return false;
+        if(!get_use_timemory() && _timemory_options.count(_v) > 0) return false;
+        if(!get_use_sampling() && _sample_options.count(_v) > 0) return false;
+        const auto npos = std::string::npos;
+        if(_v.find("WIDTH") != npos || _v.find("SEPARATOR_FREQ") != npos ||
+           _v.find("AUTO_OUTPUT") != npos || _v.find("DART_OUTPUT") != npos ||
+           _v.find("FILE_OUTPUT") != npos || _v.find("PLOT_OUTPUT") != npos ||
+           _v.find("FLAMEGRAPH_OUTPUT") != npos)
+            return false;
+        if(!_c.empty())
+        {
+            if(_c.find("omnitrace") != _c.end()) return true;
+            if(_c.find("debugging") != _c.end() && _v.find("DEBUG") != npos) return true;
+            if(_c.find("config") != _c.end()) return true;
+            if(_c.find("dart") != _c.end()) return false;
+            if(_c.find("io") != _c.end() && _v.find("_OUTPUT") != npos) return true;
+            if(_c.find("format") != _c.end()) return true;
+            return false;
+        }
+        return (_v.find("OMNITRACE_") == 0);
+    };
+
+    tim::print_env(std::cerr, [_is_omnitrace_option](const std::string& _v) {
+        return _is_omnitrace_option(_v, std::set<std::string>{});
+    });
+
+    print_settings(std::cerr, _is_omnitrace_option);
+
+    fprintf(stderr, "\n");
+}
+
 std::string&
 get_exe_name()
 {
@@ -384,12 +533,56 @@ get_config_file()
    return static_cast<tim::tsettings<std::string>&>(*_v->second).get();
 }

+Mode
+get_mode()
+{
+    static auto _v = []() {
+        auto _mode = tim::get_env_choice<std::string>("OMNITRACE_MODE", "trace",
+                                                      { "trace", "sampling" });
+        if(_mode == "sampling") return Mode::Sampling;
+        return Mode::Trace;
+    }();
+    return _v;
+}
+
+bool&
+is_attached()
+{
+    static bool _v = false;
+    return _v;
+}
+
+bool&
+is_binary_rewrite()
+{
+    static bool _v = false;
+    return _v;
+}
+
 bool
 get_debug_env()
 {
    return tim::get_env<bool>("OMNITRACE_DEBUG", false);
 }

+bool
+get_is_continuous_integration()
+{
+    return tim::get_env<bool>("OMNITRACE_CI", false);
+}
+
+bool
+get_debug_init()
+{
+    return tim::get_env<bool>("OMNITRACE_DEBUG_INIT", false);
+}
+
+bool
+get_debug_finalize()
+{
+    return tim::get_env<bool>("OMNITRACE_DEBUG_FINALIZE", false);
+}
+
 bool
 get_debug()
 {
@@ -430,6 +623,18 @@ get_use_roctracer()
 #if defined(OMNITRACE_USE_ROCTRACER)
    static auto _v = get_config()->find("OMNITRACE_USE_ROCTRACER");
    return static_cast<tim::tsettings<bool>&>(*_v->second).get();
+#else
+    static auto _v            = false;
+    return _v;
+#endif
+}
+
+bool&
+get_use_rocm_smi()
+{
+#if defined(OMNITRACE_USE_ROCM_SMI)
+    static auto _v = get_config()->find("OMNITRACE_USE_ROCM_SMI");
+    return static_cast<tim::tsettings<bool>&>(*_v->second).get();
 #else
    static auto _v = false;
    return _v;
@@ -443,10 +648,9 @@ get_use_sampling()
    static auto _v = get_config()->find("OMNITRACE_USE_SAMPLING");
    return static_cast<tim::tsettings<bool>&>(*_v->second).get();
 #else
+    OMNITRACE_THROW(
+        "Error! sampling was enabled but omnitrace was not built with libunwind support");
    static bool _v = false;
-    if(_v)
-        throw std::runtime_error("Error! sampling was enabled but omnitrace was not "
-                                 "built with libunwind support");
    return _v;
 #endif
 }
@@ -472,6 +676,13 @@ get_use_critical_trace()
    return static_cast<tim::tsettings<bool>&>(*_v->second).get();
 }

+bool
+get_use_kokkosp()
+{
+    static auto _v = get_config()->find("OMNITRACE_USE_KOKKOSP");
+    return static_cast<tim::tsettings<bool>&>(*_v->second).get();
+}
+
 bool
 get_critical_trace_debug()
 {
@@ -634,6 +845,20 @@ get_critical_trace_count()
    return static_cast<tim::tsettings<int64_t>&>(*_v->second).get();
 }

+double&
+get_rocm_smi_freq()
+{
+    static auto _v = get_config()->find("OMNITRACE_ROCM_SMI_FREQ");
+    return static_cast<tim::tsettings<double>&>(*_v->second).get();
+}
+
+std::string
+get_rocm_smi_devices()
+{
+    static auto _v = get_config()->find("OMNITRACE_ROCM_SMI_DEVICES");
+    return static_cast<tim::tsettings<std::string>&>(*_v->second).get();
+}
+
 bool
 get_debug_tid()
 {
@@ -665,6 +890,7 @@ get_debug_pid()
                     _vlist.count(dmp::rank()) > 0;
    return _v;
 }
+}  // namespace config

 State&
 get_state()
@@ -685,8 +911,7 @@ get_cpu_cid_stack(int64_t _tid)
 {
    struct omnitrace_cpu_cid_stack
    {};
-    using thread_data_t =
-        omnitrace_thread_data<std::vector<uint64_t>, omnitrace_cpu_cid_stack>;
+    using thread_data_t = thread_data<std::vector<uint64_t>, omnitrace_cpu_cid_stack>;
    static auto&             _v       = thread_data_t::instances();
    static thread_local auto _v_check = [_tid]() {
        thread_data_t::construct((_tid > 0) ? *thread_data_t::instances().at(0)
@@ -29,6 +29,7 @@

 #include <PTL/ThreadPool.hh>
 #include <timemory/backends/dmp.hpp>
+#include <timemory/backends/threading.hpp>
 #include <timemory/hash/types.hpp>
 #include <timemory/tpls/cereal/cereal/archives/json.hpp>
 #include <timemory/tpls/cereal/cereal/cereal.hpp>
@@ -38,6 +39,7 @@

 #include <cctype>
 #include <cstdint>
+#include <exception>
 #include <iomanip>
 #include <sstream>
 #include <stdexcept>
@@ -176,7 +178,8 @@ entry::operator+=(const entry& rhs)
    }
    else
    {
-        OMNITRACE_PRINT(
+        OMNITRACE_CONDITIONAL_PRINT(
+            get_verbose() > 1,
            "Warning! Incorrect phase. entry::operator+=(entry) is only valid for "
            "Phase::BEGIN += Phase::END\n");
    }
@@ -427,46 +430,80 @@ call_chain::query(FuncT&& _func) const

 template <>
 void
-call_chain::generate_perfetto<Device::NONE>(std::set<entry>&) const
+call_chain::generate_perfetto<Device::NONE>(std::set<entry>&, bool) const
 {}

 template <>
 void
-call_chain::generate_perfetto<Device::CPU>(std::set<entry>& _used) const
+call_chain::generate_perfetto<Device::CPU>(std::set<entry>& _used, bool _basic) const
 {
    static std::set<std::string> _static_strings{};
    static std::mutex            _static_mutex{};
    for(const auto& itr : *this)
    {
-        if(!_used.emplace(itr).second) continue;
-        std::string _name = tim::demangle(tim::get_hash_identifier(itr.hash));
-        _static_mutex.lock();
-        auto sitr = _static_strings.emplace(_name);
-        _static_mutex.unlock();
-        TRACE_EVENT_BEGIN("host-critical-trace",
-                          perfetto::StaticString{ sitr.first->c_str() },
-                          static_cast<uint64_t>(itr.begin_ns));
-        TRACE_EVENT_END("host-critical-trace", static_cast<uint64_t>(itr.end_ns));
+        if(_basic)
+        {
+            if(itr.device == Device::CPU)
+            {
+                TRACE_EVENT_BEGIN("device-critical-trace", "CPU",
+                                  static_cast<uint64_t>(itr.begin_ns));
+            }
+            else if(itr.device == Device::GPU)
+            {
+                TRACE_EVENT_BEGIN("device-critical-trace", "GPU",
+                                  static_cast<uint64_t>(itr.begin_ns));
+            }
+            TRACE_EVENT_END("device-critical-trace", static_cast<uint64_t>(itr.end_ns));
+        }
+        else
+        {
+            if(!_used.emplace(itr).second) continue;
+            std::string _name = tim::demangle(tim::get_hash_identifier(itr.hash));
+            _static_mutex.lock();
+            auto sitr = _static_strings.emplace(_name);
+            _static_mutex.unlock();
+            TRACE_EVENT_BEGIN("host-critical-trace",
+                              perfetto::StaticString{ sitr.first->c_str() },
+                              static_cast<uint64_t>(itr.begin_ns));
+            TRACE_EVENT_END("host-critical-trace", static_cast<uint64_t>(itr.end_ns));
+        }
    }
 }

 template <>
 void
-call_chain::generate_perfetto<Device::GPU>(std::set<entry>& _used) const
+call_chain::generate_perfetto<Device::GPU>(std::set<entry>& _used, bool _basic) const
 {
    static std::set<std::string> _static_strings{};
    static std::mutex            _static_mutex{};
    for(const auto& itr : *this)
    {
-        if(!_used.emplace(itr).second) continue;
-        std::string _name = tim::demangle(tim::get_hash_identifier(itr.hash));
-        _static_mutex.lock();
-        auto sitr = _static_strings.emplace(_name);
-        _static_mutex.unlock();
-        TRACE_EVENT_BEGIN("device-critical-trace",
-                          perfetto::StaticString{ sitr.first->c_str() },
-                          static_cast<uint64_t>(itr.begin_ns));
-        TRACE_EVENT_END("device-critical-trace", static_cast<uint64_t>(itr.end_ns));
+        if(_basic)
+        {
+            if(itr.device == Device::CPU)
+            {
+                TRACE_EVENT_BEGIN("device-critical-trace", "CPU",
+                                  static_cast<uint64_t>(itr.begin_ns));
+            }
+            else if(itr.device == Device::GPU)
+            {
+                TRACE_EVENT_BEGIN("device-critical-trace", "GPU",
+                                  static_cast<uint64_t>(itr.begin_ns));
+            }
+            TRACE_EVENT_END("device-critical-trace", static_cast<uint64_t>(itr.end_ns));
+        }
+        else
+        {
+            if(!_used.emplace(itr).second) continue;
+            std::string _name = tim::demangle(tim::get_hash_identifier(itr.hash));
+            _static_mutex.lock();
+            auto sitr = _static_strings.emplace(_name);
+            _static_mutex.unlock();
+            TRACE_EVENT_BEGIN("device-critical-trace",
+                              perfetto::StaticString{ sitr.first->c_str() },
+                              static_cast<uint64_t>(itr.begin_ns));
+            TRACE_EVENT_END("device-critical-trace", static_cast<uint64_t>(itr.end_ns));
+        }
    }
 }

@@ -485,7 +522,7 @@ get_update_frequency()
 std::unique_ptr<call_chain>&
 get(int64_t _tid)
 {
-    static auto&             _v    = omnitrace_thread_data<call_chain>::instances();
+    static auto&             _v    = thread_data<call_chain>::instances();
    static thread_local auto _once = [_tid]() {
        if(!_v.at(0)) _v.at(0) = std::make_unique<call_chain>();
        if(!_v.at(_tid)) _v.at(_tid) = std::make_unique<call_chain>();
@@ -515,10 +552,10 @@ size_t
 add_hash_id(const std::string& _label)
 {
    using critical_trace_hash_data =
-        omnitrace_thread_data<critical_trace::hash_ids, critical_trace::id>;
+        thread_data<critical_trace::hash_ids, critical_trace::id>;

    auto _hash = tim::hash::add_hash_id(_label);
-    if(get_use_critical_trace())
+    if(get_use_critical_trace() || get_use_rocm_smi())
    {
        critical_trace_hash_data::construct();
        critical_trace_hash_data::instance()->emplace(_label);
@@ -529,7 +566,7 @@ add_hash_id(const std::string& _label)
 void
 update(int64_t _tid)
 {
-    if(!get_use_critical_trace()) return;
+    if(!get_use_critical_trace() && !get_use_rocm_smi()) return;
    std::unique_lock<std::mutex> _lk{ tasking::get_critical_trace_mutex(),
                                      std::defer_lock };
    if(!_lk.owns_lock()) _lk.lock();
@@ -604,7 +641,8 @@ save_call_graph(const std::string& _fname, const std::string& _label,
        if(_msg)
        {
            if(_func.empty()) _func = __FUNCTION__;
-            OMNITRACE_PRINT("[%s] Outputting '%s'...\n", _func.c_str(), _fname.c_str());
+            OMNITRACE_CONDITIONAL_PRINT(get_verbose() >= 0, "[%s] Outputting '%s'...\n",
+                                        _func.c_str(), _fname.c_str());
        }
        ofs << oss.str() << std::endl;
    }
@@ -648,7 +686,8 @@ save_critical_trace(const std::string& _fname, const std::string& _label,
        if(_msg)
        {
            if(_func.empty()) _func = __FUNCTION__;
-            OMNITRACE_PRINT("[%s] Outputting '%s'...\n", _func.c_str(), _fname.c_str());
+            OMNITRACE_CONDITIONAL_PRINT(get_verbose() >= 0, "[%s] Outputting '%s'...\n",
+                                        _func.c_str(), _fname.c_str());
        }
        std::stringstream oss{};
        if(_cchain.size() > 1000)
@@ -686,7 +725,8 @@ save_call_chain_text(const std::string& _fname, const call_chain& _call_chain,
        if(_msg)
        {
            if(_func.empty()) _func = __FUNCTION__;
-            OMNITRACE_PRINT("[%s] Outputting '%s'...\n", _func.c_str(), _fname.c_str());
+            OMNITRACE_CONDITIONAL_PRINT(get_verbose() >= 0, "[%s] Outputting '%s'...\n",
+                                        _func.c_str(), _fname.c_str());
        }
        ofs << _call_chain << "\n";
    }
@@ -730,7 +770,8 @@ save_call_chain_json(const std::string& _fname, const std::string& _label,
        if(_msg)
        {
            if(_func.empty()) _func = __FUNCTION__;
-            OMNITRACE_PRINT("[%s] Outputting '%s'...\n", _func.c_str(), _fname.c_str());
+            OMNITRACE_CONDITIONAL_PRINT(get_verbose() >= 0, "[%s] Outputting '%s'...\n",
+                                        _func.c_str(), _fname.c_str());
        }
        std::stringstream oss{};
        if(_call_chain.size() > 100000)
@@ -1117,27 +1158,33 @@ get_top(const std::vector<call_chain>& _chain, size_t _count)

 template <Device DevT>
 void
-generate_perfetto(const std::vector<call_chain>& _data)
+generate_perfetto(const std::vector<call_chain>& _data, bool _basic = false)
 {
    OMNITRACE_CT_DEBUG("[%s]\n", __FUNCTION__);
-    auto _func = [&](size_t _beg, size_t _end) {
+
+    auto _nrows = std::min<size_t>(get_critical_trace_per_row(), _data.size());
+
+    // run in separate thread(s) so that it ends up in unique row
+    if(_nrows < 1) _nrows = _data.size();
+
+    auto _func = [&](size_t _idx, size_t _beg, size_t _end) {
+        if(_nrows != 1)
+            threading::set_thread_name(TIMEMORY_JOIN(" ", "CriticalPath", _idx).c_str());
+        else
+            threading::set_thread_name("CritialPath");
        // ensure all hash ids exist
        copy_hash_ids();
        std::set<entry> _used{};
        for(size_t i = _beg; i < _end; ++i)
        {
            if(i >= _data.size()) break;
-            _data.at(i).generate_perfetto<DevT>(_used);
+            _data.at(i).generate_perfetto<DevT>(_used, _basic);
        }
    };

-    // run in separate thread(s) so that it ends up in unique row
-    auto _nrows = get_critical_trace_per_row();
-    if(_nrows < 1) _nrows = _data.size();
-
    for(size_t i = 0; i < _data.size(); i += _nrows)
    {
-        std::thread{ _func, i, i + _nrows }.join();
+        std::thread{ _func, i, i, i + _nrows }.join();
    }
 }

@@ -1225,6 +1272,8 @@ compute_critical_trace()

        _perf.reset().start();

+        generate_perfetto<Device::GPU>({ complete_call_chain }, true);
+
        OMNITRACE_CT_DEBUG("[%s] Finding sequences...\n", __FUNCTION__);
        // find the sequences
        std::vector<call_chain> _top{};
@@ -1280,8 +1329,7 @@ compute_critical_trace()
        _tg.join();
        _tp.destroy_threadpool();
        _computed = true;
-
-    } catch(const std::exception& e)
+    } catch(std::exception& e)
    {
        OMNITRACE_PRINT("Thread exited '%s' with exception: %s\n", __FUNCTION__,
                        e.what());
@@ -1295,5 +1343,36 @@ compute_critical_trace()
    OMNITRACE_PRINT("%s\n", _ct_msg.c_str());
 }
 }  // namespace
+
+std::vector<std::pair<std::string, entry>>
+get_entries(int64_t _ts, const std::function<bool(const entry&)>& _eval)
+{
+    copy_hash_ids();
+    auto _func = [_eval, _ts](std::vector<std::pair<std::string, entry>>* _targ,
+                              size_t*                                     _avail) {
+        copy_hash_ids();
+        squash_critical_path(complete_call_chain);
+        *_avail = complete_call_chain.size();
+        std::vector<std::pair<std::string, entry>> _v{};
+        std::sort(complete_call_chain.begin(), complete_call_chain.end());
+        for(const auto& itr : complete_call_chain)
+        {
+            if(itr.phase != Phase::DELTA) continue;
+            if(itr.begin_ns <= _ts && itr.end_ns >= _ts)
+            {
+                if(_eval(itr)) _v.emplace_back(tim::get_hash_identifier(itr.hash), itr);
+            }
+        }
+        *_targ = _v;
+    };
+    size_t                                     _n = 0;
+    std::vector<std::pair<std::string, entry>> _v{};
+    tasking::get_critical_trace_task_group().exec(_func, &_v, &_n);
+    tasking::get_critical_trace_task_group().join();
+    OMNITRACE_DEBUG("critical_trace::%s :: found %zu out of %zu entries at %li...\n",
+                    __FUNCTION__, _v.size(), _n, _ts);
+    return _v;
+}
+
 }  // namespace critical_trace
 }  // namespace omnitrace
@@ -22,11 +22,12 @@

 #include "library/gpu.hpp"

-#if defined(OMNITRACE_USE_HIP)
+#if defined(OMNITRACE_USE_ROCM_SMI)
+#    include "library/components/rocm_smi.hpp"
+#elif defined(OMNITRACE_USE_HIP)
 #    if !defined(TIMEMORY_USE_HIP)
 #        define TIMEMORY_USE_HIP 1
 #    endif
-
 #    include "timemory/components/hip/backends.hpp"
 #endif

@@ -34,18 +35,18 @@ namespace omnitrace
 {
 namespace gpu
 {
-#if defined(OMNITRACE_USE_HIP)
 int
 device_count()
 {
+#if defined(OMNITRACE_USE_ROCM_SMI)
+    // store as static since calls after rsmi_shutdown will return zero
+    static auto _v = rocm_smi::device_count();
+    return _v;
+#elif defined(OMNITRACE_USE_HIP)
    return ::tim::hip::device_count();
-}
 #else
-int
-device_count()
-{
    return 0;
-}
 #endif
+}
 }  // namespace gpu
 }  // namespace omnitrace
@@ -0,0 +1,326 @@
+// MIT License
+//
+// Copyright (c) 2020, The Regents of the University of California,
+// through Lawrence Berkeley National Laboratory (subject to receipt of any
+// required approvals from the U.S. Dept. of Energy).  All rights reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+// used by Kokkos decls
+#if !defined(TIMEMORY_LIBRARY_SOURCE)
+#    define TIMEMORY_LIBRARY_SOURCE 1
+#endif
+
+#include "library/components/omnitrace.hpp"
+#include "library/config.hpp"
+
+#include <timemory/api/kokkosp.hpp>
+
+namespace kokkosp = tim::kokkosp;
+
+//--------------------------------------------------------------------------------------//
+
+namespace tim
+{
+template <>
+inline auto
+invoke_preinit<kokkosp::memory_tracker>(long)
+{
+    kokkosp::memory_tracker::label()       = "kokkos_memory";
+    kokkosp::memory_tracker::description() = "Kokkos Memory tracker";
+}
+}  // namespace tim
+
+//--------------------------------------------------------------------------------------//
+
+namespace
+{
+std::string kokkos_banner =
+    "#---------------------------------------------------------------------------#";
+
+//--------------------------------------------------------------------------------------//
+
+bool enable_kernel_logger = false;
+
+inline void
+add_kernel_logger()
+{
+    static bool _first = true;
+    if(!_first) return;
+    _first         = false;
+    using strvec_t = std::vector<std::string>;
+
+    tim::settings::instance()->insert<bool, bool&>(
+        std::string{ "OMNITRACE_KOKKOS_KERNEL_LOGGER" }, std::string{},
+        std::string{ "Enables kernel logging" }, enable_kernel_logger,
+        strvec_t({ "--omnitrace-kokkos-kernel-logger" }));
+}
+
+inline void
+setup_kernel_logger()
+{
+    if(tim::settings::debug() || tim::settings::verbose() > 3 || enable_kernel_logger)
+    {
+        kokkosp::logger_t::get_initializer() = [](kokkosp::logger_t& _obj) {
+            _obj.initialize<kokkosp::kernel_logger>();
+        };
+    }
+}
+
+}  // namespace
+
+//--------------------------------------------------------------------------------------//
+
+extern "C"
+{
+    void kokkosp_print_help(char*) {}
+
+    void kokkosp_parse_args(int, char**) {}
+
+    void kokkosp_declare_metadata(const char* key, const char* value)
+    {
+        tim::manager::add_metadata(key, value);
+    }
+
+    void kokkosp_init_library(const int loadSeq, const uint64_t interfaceVer,
+                              const uint32_t devInfoCount, void* deviceInfo)
+    {
+        add_kernel_logger();
+
+        tim::consume_parameters(devInfoCount, deviceInfo);
+
+        printf("%s\n", kokkos_banner.c_str());
+        printf("# KokkosP: omnitrace connector (sequence is %d, version: %llu)\n",
+               loadSeq, (unsigned long long) interfaceVer);
+        printf("%s\n", kokkos_banner.c_str());
+
+        setup_kernel_logger();
+
+        tim::trait::runtime_enabled<kokkosp::memory_tracker>::set(
+            omnitrace::config::get_use_timemory());
+    }
+
+    void kokkosp_finalize_library()
+    {
+        printf("%s\n", kokkos_banner.c_str());
+        printf("# KokkosP: Finalization of omnitrace connector. Complete.\n");
+        printf("%s\n", kokkos_banner.c_str());
+
+        kokkosp::cleanup();
+    }
+
+    //----------------------------------------------------------------------------------//
+
+    void kokkosp_begin_parallel_for(const char* name, uint32_t devid, uint64_t* kernid)
+    {
+        auto pname =
+            (devid > std::numeric_limits<uint16_t>::max())  // junk device number
+                ? TIMEMORY_JOIN(" ", "[kokkos]", name)
+                : TIMEMORY_JOIN(" ", TIMEMORY_JOIN("", "[kokkos][dev", devid, ']'), name);
+        *kernid = kokkosp::get_unique_id();
+        kokkosp::logger_t{}.mark(1, __FUNCTION__, name, *kernid);
+        kokkosp::create_profiler<omnitrace::component::omnitrace>(pname, *kernid);
+        kokkosp::start_profiler<omnitrace::component::omnitrace>(*kernid);
+    }
+
+    void kokkosp_end_parallel_for(uint64_t kernid)
+    {
+        kokkosp::logger_t{}.mark(-1, __FUNCTION__, kernid);
+        kokkosp::stop_profiler<omnitrace::component::omnitrace>(kernid);
+        kokkosp::destroy_profiler<omnitrace::component::omnitrace>(kernid);
+    }
+
+    //----------------------------------------------------------------------------------//
+
+    void kokkosp_begin_parallel_reduce(const char* name, uint32_t devid, uint64_t* kernid)
+    {
+        auto pname =
+            (devid > std::numeric_limits<uint16_t>::max())  // junk device number
+                ? TIMEMORY_JOIN(" ", "[kokkos]", name)
+                : TIMEMORY_JOIN(" ", TIMEMORY_JOIN("", "[kokkos][dev", devid, ']'), name);
+        *kernid = kokkosp::get_unique_id();
+        kokkosp::logger_t{}.mark(1, __FUNCTION__, name, *kernid);
+        kokkosp::create_profiler<omnitrace::component::omnitrace>(pname, *kernid);
+        kokkosp::start_profiler<omnitrace::component::omnitrace>(*kernid);
+    }
+
+    void kokkosp_end_parallel_reduce(uint64_t kernid)
+    {
+        kokkosp::logger_t{}.mark(-1, __FUNCTION__, kernid);
+        kokkosp::stop_profiler<omnitrace::component::omnitrace>(kernid);
+        kokkosp::destroy_profiler<omnitrace::component::omnitrace>(kernid);
+    }
+
+    //----------------------------------------------------------------------------------//
+
+    void kokkosp_begin_parallel_scan(const char* name, uint32_t devid, uint64_t* kernid)
+    {
+        auto pname =
+            (devid > std::numeric_limits<uint16_t>::max())  // junk device number
+                ? TIMEMORY_JOIN(" ", "[kokkos]", name)
+                : TIMEMORY_JOIN(" ", TIMEMORY_JOIN("", "[kokkos][dev", devid, ']'), name);
+        *kernid = kokkosp::get_unique_id();
+        kokkosp::logger_t{}.mark(1, __FUNCTION__, name, *kernid);
+        kokkosp::create_profiler<omnitrace::component::omnitrace>(pname, *kernid);
+        kokkosp::start_profiler<omnitrace::component::omnitrace>(*kernid);
+    }
+
+    void kokkosp_end_parallel_scan(uint64_t kernid)
+    {
+        kokkosp::logger_t{}.mark(-1, __FUNCTION__, kernid);
+        kokkosp::stop_profiler<omnitrace::component::omnitrace>(kernid);
+        kokkosp::destroy_profiler<omnitrace::component::omnitrace>(kernid);
+    }
+
+    //----------------------------------------------------------------------------------//
+
+    void kokkosp_begin_fence(const char* name, uint32_t devid, uint64_t* kernid)
+    {
+        auto pname =
+            (devid > std::numeric_limits<uint16_t>::max())  // junk device number
+                ? TIMEMORY_JOIN(" ", "[kokkos]", name)
+                : TIMEMORY_JOIN(" ", TIMEMORY_JOIN("", "[kokkos][dev", devid, ']'), name);
+        *kernid = kokkosp::get_unique_id();
+        kokkosp::logger_t{}.mark(1, __FUNCTION__, name, *kernid);
+        kokkosp::create_profiler<omnitrace::component::omnitrace>(pname, *kernid);
+        kokkosp::start_profiler<omnitrace::component::omnitrace>(*kernid);
+    }
+
+    void kokkosp_end_fence(uint64_t kernid)
+    {
+        kokkosp::logger_t{}.mark(-1, __FUNCTION__, kernid);
+        kokkosp::stop_profiler<omnitrace::component::omnitrace>(kernid);
+        kokkosp::destroy_profiler<omnitrace::component::omnitrace>(kernid);
+    }
+
+    //----------------------------------------------------------------------------------//
+
+    void kokkosp_push_profile_region(const char* name)
+    {
+        kokkosp::logger_t{}.mark(1, __FUNCTION__, name);
+        kokkosp::get_profiler_stack<omnitrace::component::omnitrace>().push_back(
+            kokkosp::profiler_t<omnitrace::component::omnitrace>(name));
+        kokkosp::get_profiler_stack<omnitrace::component::omnitrace>().back().start();
+    }
+
+    void kokkosp_pop_profile_region()
+    {
+        kokkosp::logger_t{}.mark(-1, __FUNCTION__);
+        if(kokkosp::get_profiler_stack<omnitrace::component::omnitrace>().empty()) return;
+        kokkosp::get_profiler_stack<omnitrace::component::omnitrace>().back().stop();
+        kokkosp::get_profiler_stack<omnitrace::component::omnitrace>().pop_back();
+    }
+
+    //----------------------------------------------------------------------------------//
+
+    void kokkosp_create_profile_section(const char* name, uint32_t* secid)
+    {
+        *secid     = kokkosp::get_unique_id();
+        auto pname = TIMEMORY_JOIN(" ", "[kokkos]", name);
+        kokkosp::create_profiler<omnitrace::component::omnitrace>(pname, *secid);
+    }
+
+    void kokkosp_destroy_profile_section(uint32_t secid)
+    {
+        kokkosp::destroy_profiler<omnitrace::component::omnitrace>(secid);
+    }
+
+    //----------------------------------------------------------------------------------//
+
+    void kokkosp_start_profile_section(uint32_t secid)
+    {
+        kokkosp::logger_t{}.mark(1, __FUNCTION__, secid);
+        kokkosp::start_profiler<omnitrace::component::omnitrace>(secid);
+    }
+
+    void kokkosp_stop_profile_section(uint32_t secid)
+    {
+        kokkosp::logger_t{}.mark(-1, __FUNCTION__, secid);
+        kokkosp::start_profiler<omnitrace::component::omnitrace>(secid);
+    }
+
+    //----------------------------------------------------------------------------------//
+
+    void kokkosp_allocate_data(const SpaceHandle space, const char* label,
+                               const void* const ptr, const uint64_t size)
+    {
+        kokkosp::logger_t{}.mark(0, __FUNCTION__, space.name, label,
+                                 TIMEMORY_JOIN("", '[', ptr, ']'), size);
+        kokkosp::profiler_alloc_t<>{ TIMEMORY_JOIN(" ", "[kokkos][allocate]", space.name,
+                                                   label) }
+            .store(std::plus<int64_t>{}, size);
+    }
+
+    void kokkosp_deallocate_data(const SpaceHandle space, const char* label,
+                                 const void* const ptr, const uint64_t size)
+    {
+        kokkosp::logger_t{}.mark(0, __FUNCTION__, space.name, label,
+                                 TIMEMORY_JOIN("", '[', ptr, ']'), size);
+        kokkosp::profiler_alloc_t<>{ TIMEMORY_JOIN(" ", "[kokkos][deallocate]",
+                                                   space.name, label) }
+            .store(std::plus<int64_t>{}, size);
+    }
+
+    //----------------------------------------------------------------------------------//
+
+    void kokkosp_begin_deep_copy(SpaceHandle dst_handle, const char* dst_name,
+                                 const void* dst_ptr, SpaceHandle src_handle,
+                                 const char* src_name, const void* src_ptr, uint64_t size)
+    {
+        kokkosp::logger_t{}.mark(1, __FUNCTION__, dst_handle.name, dst_name,
+                                 TIMEMORY_JOIN("", '[', dst_ptr, ']'), src_handle.name,
+                                 src_name, TIMEMORY_JOIN("", '[', src_ptr, ']'), size);
+
+        auto name = TIMEMORY_JOIN(" ", "[kokkos][deep_copy]",
+                                  TIMEMORY_JOIN('=', dst_handle.name, dst_name),
+                                  TIMEMORY_JOIN('=', src_handle.name, src_name));
+
+        auto& _data = kokkosp::get_profiler_stack<omnitrace::component::omnitrace>();
+        _data.emplace_back(name);
+        _data.back().audit(dst_handle, dst_name, dst_ptr, src_handle, src_name, src_ptr,
+                           size);
+        _data.back().start();
+        _data.back().store(std::plus<int64_t>{}, size);
+    }
+
+    void kokkosp_end_deep_copy()
+    {
+        kokkosp::logger_t{}.mark(-1, __FUNCTION__);
+        auto& _data = kokkosp::get_profiler_stack<omnitrace::component::omnitrace>();
+        if(_data.empty()) return;
+        _data.back().store(std::minus<int64_t>{}, 0);
+        _data.back().stop();
+        _data.pop_back();
+    }
+
+    //----------------------------------------------------------------------------------//
+
+    void kokkosp_profile_event(const char* name)
+    {
+        kokkosp::profiler_t<omnitrace::component::omnitrace>{}.mark(name);
+    }
+
+    //----------------------------------------------------------------------------------//
+}
+
+//--------------------------------------------------------------------------------------//
+
+TIMEMORY_INITIALIZE_STORAGE(kokkosp::memory_tracker)
+
+//--------------------------------------------------------------------------------------//
@@ -24,6 +24,7 @@
 #include "library/config.hpp"
 #include "library/debug.hpp"
 #include "library/defines.hpp"
+#include "library/sampling.hpp"

 #include <PTL/ThreadPool.hh>
 #include <timemory/utility/declaration.hpp>
@@ -39,10 +40,14 @@ auto _thread_pool_cfg = []() {
    _v.init         = true;
    _v.use_affinity = false;
    _v.use_tbb      = false;
-    _v.initializer  = []() {};
-    _v.finalizer    = []() {};
-    _v.priority     = 5;
-    _v.pool_size    = 1;
+    _v.initializer  = []() {
+        sampling::block_signals();
+        threading::set_thread_name(
+            TIMEMORY_JOIN('.', "ptl", PTL::Threading::GetThreadId()).c_str());
+    };
+    _v.finalizer = []() {};
+    _v.priority  = 5;
+    _v.pool_size = 1;
    return _v;
 }();
 }
@@ -76,10 +76,10 @@ namespace omnitrace
 namespace sampling
 {
 using hw_counters               = typename component::backtrace::hw_counters;
-using signal_type_instances     = omnitrace_thread_data<std::set<int>, api::sampling>;
-using backtrace_init_instances  = omnitrace_thread_data<backtrace, api::sampling>;
-using sampler_running_instances = omnitrace_thread_data<bool, api::sampling>;
-using papi_vector_instances     = omnitrace_thread_data<hw_counters, api::sampling>;
+using signal_type_instances     = thread_data<std::set<int>, api::sampling>;
+using backtrace_init_instances  = thread_data<backtrace, api::sampling>;
+using sampler_running_instances = thread_data<bool, api::sampling>;
+using papi_vector_instances     = thread_data<hw_counters, api::sampling>;

 namespace
 {
@@ -134,6 +134,7 @@ get_signal_types(int64_t _tid)
 std::set<int>
 setup()
 {
+    if(!get_use_sampling()) return std::set<int>{};
    return backtrace::configure(true);
 }

@@ -0,0 +1,53 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/state.hpp"
+
+#include <string>
+
+namespace std
+{
+std::string
+to_string(omnitrace::State _v)
+{
+    switch(_v)
+    {
+        case omnitrace::State::DelayedInit: return "DelayedInit";
+        case omnitrace::State::PreInit: return "PreInit";
+        case omnitrace::State::Init: return "Init";
+        case omnitrace::State::Active: return "Active";
+        case omnitrace::State::Finalized: return "Finalized";
+    }
+    return {};
+}
+
+std::string
+to_string(omnitrace::Mode _v)
+{
+    switch(_v)
+    {
+        case omnitrace::Mode::Trace: return "Trace";
+        case omnitrace::Mode::Sampling: return "Sampling";
+    }
+    return {};
+}
+}  // namespace std
@@ -23,52 +23,55 @@
 #include "omnitrace.hpp"

 #include <chrono>
+#include <cstdint>
 #include <cstdlib>
+#include <map>
 #include <string>
 #include <sys/stat.h>
 #include <sys/types.h>
 #include <thread>
+#include <utility>
+#include <vector>

 bool debug_print   = false;
 int  verbose_level = tim::get_env<int>("TIMEMORY_RUN_VERBOSE", 0);

-static bool     binary_rewrite                                    = false;
-static bool     is_attached                                       = false;
-static bool     loop_level_instr                                  = false;
-static bool     werror                                            = false;
-static bool     stl_func_instr                                    = false;
-static bool     use_mpi                                           = false;
-static bool     is_static_exe                                     = false;
-static bool     is_driver                                         = false;
-static bool     allow_overlapping                                 = false;
-static bool     instr_dynamic_callsites                           = false;
-static bool     instr_traps                                       = false;
-static bool     instr_loop_traps                                  = false;
-static size_t   batch_size                                        = 50;
-static strset_t extra_libs                                        = {};
-static size_t   min_address_range                                 = (1 << 8);  // 256
-static size_t   min_loop_address_range                            = (1 << 8);  // 256
-static std::vector<std::pair<uint64_t, string_t>> hash_ids        = {};
-static std::map<string_t, bool>                   use_stubs       = {};
-static std::map<string_t, procedure_t*>           beg_stubs       = {};
-static std::map<string_t, procedure_t*>           end_stubs       = {};
-static strvec_t                                   init_stub_names = {};
-static strvec_t                                   fini_stub_names = {};
-static strset_t                                   used_stub_names = {};
-static std::vector<call_expr_pointer_t>           env_variables   = {};
-static std::map<string_t, call_expr_pointer_t>    beg_expr        = {};
-static std::map<string_t, call_expr_pointer_t>    end_expr        = {};
-static const auto                                 npos_v          = string_t::npos;
-static string_t                                   instr_mode      = "trace";
-static string_t    instr_push_func    = "omnitrace_push_trace";
-static string_t    instr_pop_func     = "omnitrace_pop_trace";
-static string_t    instr_push_hash    = "omnitrace_push_trace_hash";
-static string_t    instr_pop_hash     = "omnitrace_pop_trace_hash";
-static string_t    print_instrumented = {};
-static string_t    print_available    = {};
-static string_t    print_overlapping  = {};
-static std::string modfunc_dump_dir   = {};
-static auto regex_opts = std::regex_constants::egrep | std::regex_constants::optimize;
+namespace
+{
+bool                                       binary_rewrite          = false;
+bool                                       is_attached             = false;
+bool                                       loop_level_instr        = false;
+bool                                       werror                  = false;
+bool                                       stl_func_instr          = false;
+bool                                       use_mpi                 = false;
+bool                                       is_static_exe           = false;
+bool                                       is_driver               = false;
+bool                                       allow_overlapping       = false;
+bool                                       instr_dynamic_callsites = false;
+bool                                       instr_traps             = false;
+bool                                       instr_loop_traps        = false;
+size_t                                     batch_size              = 50;
+strset_t                                   extra_libs              = {};
+size_t                                     min_address_range       = (1 << 8);  // 256
+size_t                                     min_loop_address_range  = (1 << 8);  // 256
+std::vector<std::pair<uint64_t, string_t>> hash_ids                = {};
+std::map<string_t, bool>                   use_stubs               = {};
+std::map<string_t, procedure_t*>           beg_stubs               = {};
+std::map<string_t, procedure_t*>           end_stubs               = {};
+strvec_t                                   init_stub_names         = {};
+strvec_t                                   fini_stub_names         = {};
+strset_t                                   used_stub_names         = {};
+std::vector<call_expr_pointer_t>           env_variables           = {};
+std::map<string_t, call_expr_pointer_t>    beg_expr                = {};
+std::map<string_t, call_expr_pointer_t>    end_expr                = {};
+const auto                                 npos_v                  = string_t::npos;
+string_t                                   instr_mode              = "trace";
+string_t                                   print_instrumented      = {};
+string_t                                   print_available         = {};
+string_t                                   print_overlapping       = {};
+std::string                                modfunc_dump_dir        = {};
+auto regex_opts = std::regex_constants::egrep | std::regex_constants::optimize;
+}  // namespace

 std::string
 get_absolute_exe_filepath(std::string exe_name);
@@ -132,10 +135,10 @@ main(int argc, char** argv)
    bpatch->setLivenessAnalysis(false);
    bpatch->setBaseTrampDeletion(false);
    bpatch->setTrampRecursive(false);
-    bpatch->setMergeTramp(false);
+    bpatch->setMergeTramp(true);

    std::set<std::string> dyninst_defs = { "TypeChecking", "SaveFPR", "DelayedParsing",
-                                           "InstrStackFrames" };
+                                           "MergeTramp" };

    int _argc = argc;
    int _cmdc = 0;
@@ -598,7 +601,8 @@ main(int argc, char** argv)
    //  Helper function for adding regex expressions
    //
    auto add_regex = [](auto& regex_array, const string_t& regex_expr) {
-        if(!regex_expr.empty()) regex_array.push_back(std::regex(regex_expr, regex_opts));
+        if(!regex_expr.empty())
+            regex_array.emplace_back(std::regex(regex_expr, regex_opts));
    };

    add_regex(func_include, tim::get_env<string_t>("OMNITRACE_REGEX_INCLUDE", ""));
@@ -638,7 +642,7 @@ main(int argc, char** argv)
    //
    //----------------------------------------------------------------------------------//

-    int dyninst_verb = 1;
+    int dyninst_verb = 2;
    if(parser.exists("dyninst-options"))
    {
        dyninst_defs = parser.get<std::set<std::string>>("dyninst-options");
@@ -704,6 +708,26 @@ main(int argc, char** argv)
    //
    //----------------------------------------------------------------------------------//

+    // for runtime instrumentation, we need to set this before the process gets created
+    if(!binary_rewrite)
+    {
+        tim::set_env("HSA_ENABLE_INTERRUPT", "0", 0);
+        if(_pid >= 0)
+        {
+            verbprintf(-10, "#-------------------------------------------------------"
+                            "-------------------------------------------#\n");
+            verbprintf(-10, "\n");
+            verbprintf(-10, "WARNING! Sampling may result in ioctl() deadlock within "
+                            "the ROCR runtime.\n");
+            verbprintf(-10,
+                       "To avoid this, set HSA_ENABLE_INTERRUPT=0 in the environment "
+                       "before starting your ROCm/HIP application\n");
+            verbprintf(-10, "\n");
+            verbprintf(-10, "#-------------------------------------------------------"
+                            "-------------------------------------------#\n");
+        }
+    }
+
    addr_space =
        omnitrace_get_address_space(bpatch, _cmdc, _cmdv, binary_rewrite, _pid, mutname);

@@ -750,7 +774,7 @@ main(int argc, char** argv)
    if(app_modules && !app_modules->empty())
    {
        modules = *app_modules;
-        for(auto* itr : *app_modules)
+        for(auto* itr : modules)
        {
            auto* procedures = itr->getProcedures();
            if(procedures)
@@ -774,7 +798,7 @@ main(int argc, char** argv)
    if(app_functions && !app_functions->empty())
    {
        functions = *app_functions;
-        for(auto* itr : *app_functions)
+        for(auto* itr : functions)
        {
            module_t* mod = itr->getModule();
            if(mod && itr->isInstrumentable())
@@ -792,9 +816,9 @@ main(int argc, char** argv)
    }

    verbprintf(0, "Module size before loading instrumentation library: %lu\n",
-               (long unsigned) app_modules->size());
+               (long unsigned) modules.size());

-    if(debug_print || verbose_level > 1)
+    if(debug_print || verbose_level > 2)
    {
        module_function::reset_width();
        for(const auto& itr : available_module_functions)
@@ -931,21 +955,18 @@ main(int argc, char** argv)

    verbprintf(0, "Finding functions in image...\n");

-    auto* entr_trace = find_function(app_image, instr_push_func.c_str());
-    auto* exit_trace = find_function(app_image, instr_pop_func.c_str());
-    auto* entr_hash  = find_function(app_image, instr_push_hash.c_str());
-    auto* exit_hash  = find_function(app_image, instr_pop_hash.c_str());
-    auto* init_func  = find_function(app_image, "omnitrace_trace_init");
-    auto* fini_func  = find_function(app_image, "omnitrace_trace_finalize");
-    auto* env_func   = find_function(app_image, "omnitrace_trace_set_env");
-    auto* mpi_func   = find_function(app_image, "omnitrace_trace_set_mpi");
+    auto* entr_trace = find_function(app_image, "omnitrace_push_trace");
+    auto* exit_trace = find_function(app_image, "omnitrace_pop_trace");
+    auto* entr_hash  = find_function(app_image, "omnitrace_push_trace_hash");
+    auto* exit_hash  = find_function(app_image, "omnitrace_pop_trace_hash");
+    auto* init_func  = find_function(app_image, "omnitrace_init");
+    auto* fini_func  = find_function(app_image, "omnitrace_finalize");
+    auto* env_func   = find_function(app_image, "omnitrace_set_env");
+    auto* mpi_func   = find_function(app_image, "omnitrace_set_mpi");
    auto* hash_func  = find_function(app_image, "omnitrace_add_hash_id");

    if(!main_func && main_fname == "main") main_func = find_function(app_image, "_main");

-    verbprintf(0, "Instrumenting with '%s' and '%s'...\n", instr_push_func.c_str(),
-               instr_pop_func.c_str());
-
    if(mpi_init_func && mpi_fini_func) use_mpi = true;

    bool use_mpip = false;
@@ -1085,10 +1106,10 @@ main(int argc, char** argv)
    using pair_t = std::pair<procedure_t*, string_t>;

    for(const auto& itr :
-        { pair_t(main_func, main_fname), pair_t(entr_trace, instr_push_func),
-          pair_t(exit_trace, instr_pop_func), pair_t(init_func, "omnitrace_trace_init"),
-          pair_t(fini_func, "omnitrace_trace_finalize"),
-          pair_t(env_func, "omnitrace_trace_set_env") })
+        { pair_t(main_func, main_fname), pair_t(entr_trace, "omnitrace_push_trace"),
+          pair_t(exit_trace, "omnitrace_pop_trace"), pair_t(init_func, "omnitrace_init"),
+          pair_t(fini_func, "omnitrace_finalize"),
+          pair_t(env_func, "omnitrace_set_env") })
    {
        if(itr.first == main_func && !is_driver) continue;
        if(!itr.first)
@@ -1182,7 +1203,7 @@ main(int argc, char** argv)
    verbprintf(2, "Getting call expressions... ");

    auto main_call_args = omnitrace_call_expr(main_sign.get());
-    auto init_call_args = omnitrace_call_expr(default_components, binary_rewrite, cmdv0);
+    auto init_call_args = omnitrace_call_expr(instr_mode, binary_rewrite, cmdv0);
    auto fini_call_args = omnitrace_call_expr();
    auto umpi_call_args = omnitrace_call_expr(use_mpi, is_attached);
    auto none_call_args = omnitrace_call_expr();
@@ -1208,11 +1229,19 @@ main(int argc, char** argv)
        }
    }

+    std::string _libname = {};
+    for(auto&& itr : sharedlibname)
+        _libname = get_absolute_lib_filepath(itr);
+    if(_libname.empty()) _libname = "libomnitrace.so";
+
    auto env_vars = parser.get<strvec_t>("env");
    env_vars.emplace_back(TIMEMORY_JOIN('=', "OMNITRACE_MODE", instr_mode));
+    env_vars.emplace_back(TIMEMORY_JOIN('=', "HSA_ENABLE_INTERRUPT", "0"));
+    env_vars.emplace_back(TIMEMORY_JOIN('=', "HSA_TOOLS_LIB", _libname));
    env_vars.emplace_back(TIMEMORY_JOIN('=', "OMNITRACE_MPI_INIT", "OFF"));
    env_vars.emplace_back(TIMEMORY_JOIN('=', "OMNITRACE_MPI_FINALIZE", "OFF"));
-    env_vars.emplace_back(TIMEMORY_JOIN('=', "OMNITRACE_COMPONENTS", default_components));
+    env_vars.emplace_back(
+        TIMEMORY_JOIN('=', "OMNITRACE_TIMEMORY_COMPONENTS", default_components));
    env_vars.emplace_back(
        TIMEMORY_JOIN('=', "OMNITRACE_USE_MPIP",
                      (binary_rewrite && use_mpi && use_mpip) ? "ON" : "OFF"));
@@ -1229,7 +1258,7 @@ main(int argc, char** argv)
        }
        tim::set_env(p.at(0), p.at(1));
        auto _expr = omnitrace_call_expr(p.at(0), p.at(1));
-        env_variables.push_back(_expr.get(env_func));
+        env_variables.emplace_back(_expr.get(env_func));
    }

    //----------------------------------------------------------------------------------//
@@ -1240,7 +1269,7 @@ main(int argc, char** argv)

    for(const auto& itr : env_variables)
    {
-        if(itr) init_names.push_back(itr.get());
+        if(itr) init_names.emplace_back(itr.get());
    }

    for(const auto& itr : beg_expr)
@@ -1248,7 +1277,7 @@ main(int argc, char** argv)
        if(itr.second)
        {
            verbprintf(1, "+ Adding %s instrumentation...\n", itr.first.c_str());
-            init_names.push_back(itr.second.get());
+            init_names.emplace_back(itr.second.get());
        }
        else
        {
@@ -1256,55 +1285,35 @@ main(int argc, char** argv)
        }
    }

-    if(use_mpi && umpi_call) init_names.push_back(umpi_call.get());
-
-    if(init_call && binary_rewrite) init_names.push_back(init_call.get());
+    if(umpi_call) init_names.emplace_back(umpi_call.get());
+    if(init_call) init_names.emplace_back(init_call.get());

    if(binary_rewrite)
    {
-        if(mpi_init_func && mpi_fini_func)
-        {
-            verbprintf(2, "Patching MPI init functions\n");
-            if(init_call)
-                insert_instr(addr_space, mpi_init_func, init_call, BPatch_exit, nullptr,
-                             nullptr);
-            if(fini_call)
-                insert_instr(addr_space, mpi_fini_func, fini_call, BPatch_entry, nullptr,
-                             nullptr);
-        }
-        else
-        {
-            verbprintf(2, "Adding main begin and end snippets...\n");
-            init_names.push_back(main_beg_call.get());
-            fini_names.push_back(main_end_call.get());
-        }
+        verbprintf(2, "Adding main begin and end snippets...\n");
+        if(main_beg_call) init_names.emplace_back(main_beg_call.get());
+        if(main_end_call) fini_names.emplace_back(main_end_call.get());
    }
    else if(app_thread)
    {
        verbprintf(2, "Patching main function\n");
-        if(init_call)
-            insert_instr(addr_space, main_func, init_call, BPatch_entry, nullptr,
-                         nullptr);
-        if(!use_mpi)
-        {
+        if(main_beg_call)
            insert_instr(addr_space, main_func, main_beg_call, BPatch_entry, nullptr,
                         nullptr);
+        if(main_end_call)
            insert_instr(addr_space, main_func, main_end_call, BPatch_exit, nullptr,
                         nullptr);
-        }
-        if(fini_call)
-            insert_instr(addr_space, main_func, fini_call, BPatch_exit, nullptr, nullptr);
    }
    else
    {
        verbprintf(0, "No binary_rewrite and no app_thread!...\n");
    }

-    if(fini_call) fini_names.push_back(fini_call.get());
+    if(fini_call) fini_names.emplace_back(fini_call.get());

    for(const auto& itr : end_expr)
    {
-        if(itr.second) fini_names.push_back(itr.second.get());
+        if(itr.second) fini_names.emplace_back(itr.second.get());
    }

    //----------------------------------------------------------------------------------//
@@ -1316,7 +1325,7 @@ main(int argc, char** argv)
    //----------------------------------------------------------------------------------//
    std::vector<std::function<void()>> instr_procedure_functions;
    auto instr_procedures = [&](const procedure_vec_t& procedures) {
-        verbprintf(2, "Instrumenting %lu procedures...\n",
+        verbprintf(3, "Instrumenting %lu procedures...\n",
                   (unsigned long) procedures.size());
        for(auto* itr : procedures)
        {
@@ -1356,7 +1365,7 @@ main(int argc, char** argv)

            if(name.get().empty())
            {
-                verbprintf(1, "Skipping function [empty name]: %s\n", fname);
+                verbprintf(2, "Skipping function [empty name]: %s\n", fname);
                continue;
            }

@@ -1366,7 +1375,7 @@ main(int argc, char** argv)
            if(is_static_exe && has_debug_info && string_t{ fname } == "_fini" &&
               string_t{ modname } == "DEFAULT_MODULE")
            {
-                verbprintf(1, "Skipping function [DEFAULT_MODULE]: %s\n", fname);
+                verbprintf(2, "Skipping function [DEFAULT_MODULE]: %s\n", fname);
                continue;
            }

@@ -1376,7 +1385,7 @@ main(int argc, char** argv)
               overlapping_module_functions.find(module_function{ mod, itr }) !=
                   overlapping_module_functions.end())
            {
-                verbprintf(1, "Skipping function [overlapping]: %s / %s\n",
+                verbprintf(2, "Skipping function [overlapping]: %s / %s\n",
                           name.m_name.c_str(), name.get().c_str());
                continue;
            }
@@ -1402,8 +1411,33 @@ main(int argc, char** argv)
                    ? std::max<size_t>(_loop_entries->size(), basic_loop.size())
                    : basic_loop.size();
            auto _has_loop_entries = (_num_loop_entries > 0);
+            auto _skip_range =
+                (_has_loop_entries) ? false : (_address_range < min_address_range);
+            auto _skip_loop_range =
+                (_has_loop_entries) ? (_address_range < min_loop_address_range) : false;

-            if(_address_range < min_address_range && !_has_loop_entries && !_force_instr)
+            if(_force_instr && _skip_range)
+            {
+                verbprintf(
+                    1,
+                    "Instrumenting function [dynamic-callsite]: %s / %s despite not "
+                    "satisfy minimum address range (address range = %lu, minimum "
+                    "= %lu) because contains dynamic callsites\n",
+                    name.m_name.c_str(), name.get().c_str(),
+                    (unsigned long) _address_range, (unsigned long) min_address_range);
+            }
+            else if(_force_instr && _skip_loop_range)
+            {
+                verbprintf(
+                    1,
+                    "Instrumenting function [dynamic-callsite]: %s / %s despite not "
+                    "satisfy minimum loop address range (address range = %lu, minimum "
+                    "= %lu) because contains dynamic callsites\n",
+                    name.m_name.c_str(), name.get().c_str(),
+                    (unsigned long) _address_range,
+                    (unsigned long) min_loop_address_range);
+            }
+            else if(_skip_range)
            {
                verbprintf(1,
                           "Skipping function [min-address-range]: %s / %s (address "
@@ -1413,8 +1447,7 @@ main(int argc, char** argv)
                           (unsigned long) min_address_range);
                continue;
            }
-            else if(_address_range < min_loop_address_range && _has_loop_entries &&
-                    !_force_instr)
+            else if(_skip_loop_range)
            {
                verbprintf(1,
                           "Skipping function [min-loop-address-range]: %s / %s (address "
@@ -1424,16 +1457,6 @@ main(int argc, char** argv)
                           (unsigned long) min_loop_address_range);
                continue;
            }
-            else if(_force_instr)
-            {
-                verbprintf(1,
-                           "Enabling function [dynamic-callsite]: %s / %s despite not "
-                           "satisfy minimum address range (address range = %lu, minimum "
-                           "= %lu) because contains dynamic callsites\n",
-                           name.m_name.c_str(), name.get().c_str(),
-                           (unsigned long) _address_range,
-                           (unsigned long) min_address_range);
-            }

            bool _entr_success =
                query_instr(itr, BPatch_entry, nullptr, nullptr, instr_traps);
@@ -1562,7 +1585,7 @@ main(int argc, char** argv)
                continue;
            }

-            verbprintf(1, "Parsing module: %s\n", modname);
+            verbprintf(3, "Parsing module: %s\n", modname);
            bpvector_t<procedure_t*>* p = m->getProcedures();
            if(!p) continue;

@@ -1590,7 +1613,7 @@ main(int argc, char** argv)
    //
    //----------------------------------------------------------------------------------//

-    if(is_attached)
+    if(app_thread)
    {
        assert(app_thread != nullptr);
        verbprintf(1, "Executing initial snippets...\n");
@@ -1827,58 +1850,52 @@ main(int argc, char** argv)
    {
        verbprintf(0, "Executing...\n");

-        // bpatch->setDebugParsing(false);
-        // bpatch->setTypeChecking(false);
-        // bpatch->setDelayedParsing(true);
-        // bpatch->setInstrStackFrames(true);
-        // bpatch->setLivenessAnalysis(false);
-        // addr_space->beginInsertionSet();
-
-        verbprintf(4, "Registering fork callbacks...\n");
-        auto _prefork  = bpatch->registerPreForkCallback(&omnitrace_fork_callback);
-        auto _postfork = bpatch->registerPostForkCallback(&omnitrace_fork_callback);
-
-        auto _wait_exec = [&]() {
-            while(!app_thread->isTerminated())
+        if(!app_thread->isTerminated())
+        {
+            app_thread->detach(true);
+            pid_t cpid = app_thread->getPid();
+            pid_t w;
+            int   status = 0;
+            do
            {
-                verbprintf(3, "Continuing execution...\n");
-                app_thread->continueExecution();
-                verbprintf(4, "Process is not terminated...\n");
-                bpatch->waitForStatusChange();
-                std::this_thread::sleep_for(std::chrono::milliseconds{ 100 });
-                verbprintf(4, "Process status change...\n");
-                if(app_thread->isStopped())
+                w = waitpid(cpid, &status, WUNTRACED);
+                if(w == -1)
                {
-                    verbprintf(4, "Process is stopped, continuing execution...\n");
-                    if(!app_thread->continueExecution())
-                    {
-                        fprintf(stderr, "continueExecution failed\n");
-                        exit(EXIT_FAILURE);
-                    }
+                    perror("waitpid");
+                    exit(EXIT_FAILURE);
                }
+
+                if(WIFEXITED(status))
+                {
+                    code = WEXITSTATUS(status);
+                }
+                else if(WIFSIGNALED(status))
+                {
+                    code = WTERMSIG(status);
+                }
+                else if(WIFSTOPPED(status))
+                {
+                    code = WSTOPSIG(status);
+                }
+                else if(WIFCONTINUED(status))
+                {
+                    code = WIFCONTINUED(status);
+                }
+            } while(!WIFEXITED(status) && !WIFSIGNALED(status));
+        }
+        else
+        {
+            if(app_thread->terminationStatus() == ExitedNormally)
+            {
+                if(app_thread->isTerminated()) verbprintf(0, "End of omnitrace\n");
            }
-        };
-
-        verbprintf(4, "Entering wait for status change mode...\n");
-        _wait_exec();
-
-        if(app_thread->terminationStatus() == ExitedNormally)
-        {
-            if(app_thread->isTerminated())
-                printf("\nEnd of omnitrace\n");
-            else
-                _wait_exec();
+            else if(app_thread->terminationStatus() == ExitedViaSignal)
+            {
+                auto sign = app_thread->getExitSignal();
+                fprintf(stderr, "\nApplication exited with signal: %i\n", int(sign));
+            }
+            code = app_thread->getExitCode();
        }
-        else if(app_thread->terminationStatus() == ExitedViaSignal)
-        {
-            auto sign = app_thread->getExitSignal();
-            fprintf(stderr, "\nApplication exited with signal: %i\n", int(sign));
-        }
-
-        // addr_space->finalizeInsertionSet(false, nullptr);
-
-        code = app_thread->getExitCode();
-        consume_parameters(_prefork, _postfork);
    }

    // cleanup
@@ -1888,6 +1905,9 @@ main(int argc, char** argv)
    for(int i = 0; i < _cmdc; ++i)
        delete[] _cmdv[i];
    delete[] _cmdv;
+
+    verbprintf(0, "End of omnitrace\n");
+    verbprintf(1, "Exit code: %i\n", code);
    return code;
 }

@@ -1943,7 +1963,7 @@ instrument_module(const string_t& file_name)
    static std::regex corelib_regex(
        "^lib(c|z|rt|dl|dw|util|zstd|elf|pthread|open[\\-]rte|open[\\-]pal|"
        "hwloc|numa|event|udev|dyninstAPI_RT|gcc_s|tcmalloc|profiler|tbbmalloc|"
-        "tbbmalloc_proxy)(-|\\.)",
+        "tbbmalloc_proxy|event_pthreads|ltdl)(-|\\.)",
        regex_opts);
    // these are all due to TAU
    static std::regex prefix_regex(
@@ -2049,11 +2069,13 @@ instrument_entity(const string_t& function_name)
        regex_opts);
    static std::regex exclude_cxx("(std::_Sp_counted_base|std::use_facet)", regex_opts);
    static std::regex leading(
-        "^(_|\\.|frame_dummy|\\(|targ|new|delete|operator new|operator "
-        "delete|std::allocat|"
-        "nvtx|gcov|main\\.cold|TAU|tau|Tau|dyn|RT|dl|sys|pthread|posix|clone|virtual "
-        "thunk|non-virtual thunk|transaction "
-        "clone|RtsLayer|DYNINST|PthreadLayer|threaded_func|targ8|PMPI)",
+        "^(_|\\.|frame_dummy|\\(|targ|new|delete|operator new|operator delete|"
+        "std::allocat|nvtx|gcov|main\\.cold|TAU|tau|Tau|dyn|RT|"
+        "sys|pthread|posix|clone|"
+        "virtual thunk|non-virtual thunk|transaction clone|"
+        "RtsLayer|DYNINST|PthreadLayer|threaded_func|PMPI|"
+        "Kokkos::Impl::|Kokkos::Experimental::Impl::|Kokkos::impl_|"
+        "Kokkos::[A-Za-z]+::impl_|Kokkos::Tools::|Kokkos::Profiling::)",
        regex_opts);
    static std::regex trailing("(\\.part\\.[0-9]+|\\.constprop\\.[0-9]+|\\.|\\.[0-9]+)$",
                               regex_opts);
@@ -2328,7 +2350,8 @@ get_absolute_lib_filepath(std::string lib_name)
                             std::regex_match(lib_name, std::regex("^[A-Za-z0-9].*"))))
    {
        auto _lib_orig = lib_name;
-        auto _paths = tim::delimit(tim::get_env<std::string>("LD_LIBRARY_PATH", ""), ":");
+        auto _paths    = tim::delimit(
+            std::string{ ".:" } + tim::get_env<std::string>("LD_LIBRARY_PATH", ""), ":");
        for(auto& pitr : _paths)
        {
            if(file_exists(TIMEMORY_JOIN('/', pitr, lib_name)))
@@ -101,6 +101,14 @@ get_whole_function_names()
                     "daemon",
                     "des_setparity",
                     "div",
+                     "dlopen",
+                     "dlsym",
+                     "dlerror",
+                     "dladdr",
+                     "dlinfo",
+                     "dlvsym",
+                     "dlmopen",
+                     "dl_iterate_phdr",
                     "dysize",
                     "endutxent",
                     "envz_add",
@@ -26,7 +26,29 @@ endif()
 set(_base_environment
    "OMNITRACE_USE_PERFETTO=ON"
    "OMNITRACE_USE_TIMEMORY=ON"
-    "OMNITRACE_USE_SAMPLING=OFF"
+    "OMNITRACE_USE_SAMPLING=ON"
+    "OMNITRACE_TIME_OUTPUT=OFF"
+    "OMP_PROC_BIND=spread"
+    "OMP_PLACES=threads"
+    "OMP_NUM_THREADS=2"
+    "LD_LIBRARY_PATH=${PROJECT_BINARY_DIR}:${OMNITRACE_DYNINST_API_RT_DIR}:$ENV{LD_LIBRARY_PATH}"
+    )
+
+set(_perfetto_environment
+    "OMNITRACE_USE_PERFETTO=ON"
+    "OMNITRACE_USE_TIMEMORY=OFF"
+    "OMNITRACE_USE_SAMPLING=ON"
+    "OMNITRACE_TIME_OUTPUT=OFF"
+    "OMP_PROC_BIND=spread"
+    "OMP_PLACES=threads"
+    "OMP_NUM_THREADS=2"
+    "LD_LIBRARY_PATH=${PROJECT_BINARY_DIR}:${OMNITRACE_DYNINST_API_RT_DIR}:$ENV{LD_LIBRARY_PATH}"
+    )
+
+set(_timemory_environment
+    "OMNITRACE_USE_PERFETTO=OFF"
+    "OMNITRACE_USE_TIMEMORY=ON"
+    "OMNITRACE_USE_SAMPLING=ON"
    "OMNITRACE_TIME_OUTPUT=OFF"
    "OMP_PROC_BIND=spread"
    "OMP_PLACES=threads"
@@ -48,12 +70,15 @@ set(_fast_environment
    "LD_LIBRARY_PATH=${PROJECT_BINARY_DIR}:${OMNITRACE_DYNINST_API_RT_DIR}:$ENV{LD_LIBRARY_PATH}"
    )

+# -------------------------------------------------------------------------------------- #
+
 function(OMNITRACE_ADD_TEST)
    cmake_parse_arguments(
        TEST
        "" # options
-        "NAME;TARGET;MPI;NUM_PROCS" # single value args
-        "REWRITE_ARGS;RUNTIME_ARGS;RUN_ARGS;ENVIRONMENT;LABELS" # multiple value args
+        "NAME;TARGET;MPI;NUM_PROCS;REWRITE_TIMEOUT;RUNTIME_TIMEOUT" # single value args
+        "REWRITE_ARGS;RUNTIME_ARGS;RUN_ARGS;ENVIRONMENT;LABELS;PROPERTIES" # multiple
+                                                                           # value args
        ${ARGN})

    if("${TEST_MPI}" STREQUAL "")
@@ -68,10 +93,20 @@ function(OMNITRACE_ADD_TEST)
        set(TEST_NUM_PROCS 0)
    endif()

+    if(NOT TEST_REWRITE_TIMEOUT)
+        set(TEST_REWRITE_TIMEOUT 120)
+    endif()
+
+    if(NOT TEST_RUNTIME_TIMEOUT)
+        set(TEST_RUNTIME_TIMEOUT 300)
+    endif()
+
    if(NOT DEFINED TEST_ENVIRONMENT OR "${TEST_ENVIRONMENT}" STREQUAL "")
        set(TEST_ENVIRONMENT "${_test_environment}")
    endif()

+    list(APPEND TEST_ENVIRONMENT "OMNITRACE_CI=ON")
+
    if(TARGET ${TEST_TARGET})
        if(DEFINED TEST_MPI
           AND ${TEST_MPI}
@@ -88,14 +123,14 @@ function(OMNITRACE_ADD_TEST)

        add_test(
            NAME ${TEST_NAME}-baseline
-            COMMAND $<TARGET_FILE:${TEST_TARGET}> ${TEST_RUN_ARGS}
+            COMMAND ${COMMAND_PREFIX} $<TARGET_FILE:${TEST_TARGET}> ${TEST_RUN_ARGS}
            WORKING_DIRECTORY $<TARGET_FILE_DIR:${TEST_TARGET}>)

        add_test(
            NAME ${TEST_NAME}-binary-rewrite
            COMMAND
                $<TARGET_FILE:omnitrace-exe> -o
-                $<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_TARGET}.inst ${TEST_REWRITE_ARGS}
+                $<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_NAME}.inst ${TEST_REWRITE_ARGS}
                -- $<TARGET_FILE:${TEST_TARGET}>
            WORKING_DIRECTORY $<TARGET_FILE_DIR:${TEST_TARGET}>)

@@ -103,20 +138,20 @@ function(OMNITRACE_ADD_TEST)
            NAME ${TEST_NAME}-binary-rewrite-sampling
            COMMAND
                $<TARGET_FILE:omnitrace-exe> -o
-                $<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_TARGET}.samp -M sampling
+                $<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_NAME}.samp -M sampling
                ${TEST_REWRITE_ARGS} -- $<TARGET_FILE:${TEST_TARGET}>
            WORKING_DIRECTORY $<TARGET_FILE_DIR:${TEST_TARGET}>)

        add_test(
            NAME ${TEST_NAME}-binary-rewrite-run
-            COMMAND ${COMMAND_PREFIX}
-                    $<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_TARGET}.inst ${TEST_RUN_ARGS}
+            COMMAND ${COMMAND_PREFIX} $<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_NAME}.inst
+                    ${TEST_RUN_ARGS}
            WORKING_DIRECTORY $<TARGET_FILE_DIR:${TEST_TARGET}>)

        add_test(
            NAME ${TEST_NAME}-binary-rewrite-run-sampling
-            COMMAND ${COMMAND_PREFIX}
-                    $<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_TARGET}.samp ${TEST_RUN_ARGS}
+            COMMAND ${COMMAND_PREFIX} $<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_NAME}.samp
+                    ${TEST_RUN_ARGS}
            WORKING_DIRECTORY $<TARGET_FILE_DIR:${TEST_TARGET}>)

        add_test(
@@ -127,10 +162,8 @@ function(OMNITRACE_ADD_TEST)

        add_test(
            NAME ${TEST_NAME}-runtime-instrument-sampling
-            COMMAND
-                $<TARGET_FILE:omnitrace-exe> -M sampling --env
-                OMNITRACE_OUTPUT_PREFIX=sampling- ${TEST_RUNTIME_ARGS} --
-                $<TARGET_FILE:${TEST_TARGET}> ${TEST_RUN_ARGS}
+            COMMAND $<TARGET_FILE:omnitrace-exe> -M sampling ${TEST_RUNTIME_ARGS} --
+                    $<TARGET_FILE:${TEST_TARGET}> ${TEST_RUN_ARGS}
            WORKING_DIRECTORY $<TARGET_FILE_DIR:${TEST_TARGET}>)

        set_tests_properties(${TEST_NAME}-binary-rewrite-run
@@ -143,36 +176,52 @@ function(OMNITRACE_ADD_TEST)
            _TEST
            baseline binary-rewrite binary-rewrite-run binary-rewrite-sampling
            binary-rewrite-run-sampling runtime-instrument runtime-instrument-sampling)
-            string(REPLACE "-run-" "-" _PREFIX "${TEST_NAME}-${_TEST}-")
+            string(REPLACE "-run-" "-" _prefix "${TEST_NAME}-${_TEST}-")
            set(_environ "${TEST_ENVIRONMENT}")
+            set(_labels "${_TEST}")
+            set(_timeout ${TEST_REWRITE_TIMEOUT})
            list(APPEND _environ "OMNITRACE_OUTPUT_PATH=omnitrace-tests-output"
-                 "OMNITRACE_OUTPUT_PREFIX=${_PREFIX}")
-            set(_LABELS "${_TEST}")
-            string(REPLACE "-run" "" _LABELS "${_TEST}")
-            string(REPLACE "-sampling" ";sampling" _LABELS "${_LABELS}")
-            set_tests_properties(
-                ${TEST_NAME}-${_TEST} PROPERTIES ENVIRONMENT "${_environ}" TIMEOUT 600
-                                                 LABELS "${_LABELS};${TEST_LABELS}")
+                 "OMNITRACE_OUTPUT_PREFIX=${_prefix}")
+            string(REPLACE "-run" "" _labels "${_TEST}")
+            string(REPLACE "-sampling" ";sampling" _labels "${_labels}")
+            if("${_TEST}" MATCHES "runtime-instrument")
+                set(_timeout ${TEST_RUNTIME_TIMEOUT})
+            endif()
+            set(_props)
+            if("${_TEST}" MATCHES "run|baseline")
+                set(_props ${TEST_PROPERTIES})
+                if(NOT "RUN_SERIAL" IN_LIST _props)
+                    list(APPEND _props RUN_SERIAL ON)
+                endif()
+            endif()
+            if(TEST ${TEST_NAME}-${_TEST})
+                set_tests_properties(
+                    ${TEST_NAME}-${_TEST}
+                    PROPERTIES ENVIRONMENT "${_environ}" TIMEOUT ${_timeout} LABELS
+                               "${_labels};${TEST_LABELS}" ${_props})
+            endif()
        endforeach()
    endif()
 endfunction()

+# -------------------------------------------------------------------------------------- #
+
 omnitrace_add_test(
    NAME transpose
    TARGET transpose
    MPI ${TRANSPOSE_USE_MPI}
    NUM_PROCS ${NUM_PROCS}
-    REWRITE_ARGS -e -v 1
+    REWRITE_ARGS -e -v 2
    RUNTIME_ARGS -e -v 1 --label file line return args
    RUN_ARGS ""
-    ENVIRONMENT "${_base_environment};OMNITRACE_CRITICAL_TRACE=ON")
+    ENVIRONMENT "${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF")

 omnitrace_add_test(
    NAME transpose-no-save-fpr
    TARGET transpose
    MPI ${TRANSPOSE_USE_MPI}
    NUM_PROCS ${NUM_PROCS}
-    REWRITE_ARGS -e -v 1 --dyninst-options DelayedParsing TypeChecking
+    REWRITE_ARGS -e -v 2 --dyninst-options DelayedParsing TypeChecking
    RUNTIME_ARGS
        -e
        -v
@@ -191,7 +240,7 @@ omnitrace_add_test(
 omnitrace_add_test(
    NAME parallel-overhead
    TARGET parallel-overhead
-    REWRITE_ARGS -e -v 1 --min-address-range-loop=64
+    REWRITE_ARGS -e -v 2 --min-address-range-loop=64
    RUNTIME_ARGS
        -e
        -v
@@ -202,13 +251,13 @@ omnitrace_add_test(
        line
        return
        args
-    RUN_ARGS 10 ${NUM_THREADS}
+    RUN_ARGS 10 ${NUM_THREADS} 1000
    ENVIRONMENT "${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF")

 omnitrace_add_test(
    NAME parallel-overhead-no-save-fpr
    TARGET parallel-overhead
-    REWRITE_ARGS -e -v 1 --min-address-range-loop=32 --dyninst-options DelayedParsing
+    REWRITE_ARGS -e -v 2 --min-address-range-loop=32 --dyninst-options DelayedParsing
                 TypeChecking
    RUNTIME_ARGS
        -e
@@ -223,7 +272,7 @@ omnitrace_add_test(
        --dyninst-options
        DelayedParsing
        TypeChecking
-    RUN_ARGS 10 ${NUM_THREADS}
+    RUN_ARGS 10 ${NUM_THREADS} 1000
    ENVIRONMENT "${_fast_environment}")

 omnitrace_add_test(
@@ -231,7 +280,7 @@ omnitrace_add_test(
    TARGET lulesh
    MPI ${LULESH_USE_MPI}
    NUM_PROCS 8
-    REWRITE_ARGS -e -v 1
+    REWRITE_ARGS -e -v 2
    RUNTIME_ARGS
        -e
        -v
@@ -242,6 +291,69 @@ omnitrace_add_test(
        return
        args
        -ME
-        [==['lib(gomp|m-)']==]
+        [==[lib(gomp|m-)]==]
    RUN_ARGS -i 10 -s 20 -p
-    ENVIRONMENT "${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF")
+    ENVIRONMENT
+        "${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=OFF")
+
+omnitrace_add_test(
+    NAME lulesh-kokkosp
+    TARGET lulesh
+    MPI ${LULESH_USE_MPI}
+    NUM_PROCS 8
+    REWRITE_ARGS -e -v 2
+    RUNTIME_ARGS
+        -e
+        -v
+        1
+        --label
+        file
+        line
+        return
+        args
+        -ME
+        [==[lib(gomp|m-)]==]
+    RUN_ARGS -i 10 -s 20 -p
+    ENVIRONMENT
+        "${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=ON")
+
+omnitrace_add_test(
+    NAME lulesh-perfetto
+    TARGET lulesh
+    MPI ${LULESH_USE_MPI}
+    NUM_PROCS 8
+    REWRITE_ARGS -e -v 2
+    RUNTIME_ARGS
+        -e
+        -v
+        1
+        -l
+        --dynamic-callsites
+        --traps
+        --allow-overlapping
+        -ME
+        [==[libgomp]==]
+    RUN_ARGS -i 10 -s 20 -p
+    ENVIRONMENT
+        "${_perfetto_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=OFF")
+
+omnitrace_add_test(
+    NAME lulesh-timemory
+    TARGET lulesh
+    MPI ${LULESH_USE_MPI}
+    NUM_PROCS 8
+    REWRITE_ARGS -e -v 2 -l --dynamic-callsites --traps --allow-overlapping
+    RUNTIME_ARGS
+        -e
+        -v
+        1
+        -l
+        --dynamic-callsites
+        -ME
+        [==[libgomp]==]
+        -d
+        wall_clock
+        peak_rss
+    RUN_ARGS -i 10 -s 20 -p
+    ENVIRONMENT
+        "${_timemory_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=OFF")