Causal profiling (#229)

* Addition of basic structure * Reworked categories * More causal integration additions * Causal implementation * Update examples * delete virtual_speedup files * Update perfetto submodule to v31.0 * Update dyninst submodule * Update timemory submodule * ElfUtils build for libdw * OMNITRACE_LIKELY and OMNITRACE_UNLIKELY * Update common lib join * Examples updates for causal profiling * config updates with causal options - OMNITRACE_CAUSAL_FIXED_LINE - OMNITRACE_CAUSAL_FIXED_SPEEDUP - OMNITRACE_CAUSAL_FILE - OMNITRACE_CAUSAL_BINARY_SCOPE - OMNITRACE_CAUSAL_SOURCE_SCOPE - version info in banner - support increments in parse_numeric_range - fix occasional deadlock in first call to get_config * PTL general task group * Always include PID in debug/verbose messages * Add blocking/unblocking gotchas to runtime init bundle * CausalState * thread_data updates - generic component_bundle_cache * Improve handling of causal in category_region * components updates - backtrace_causal component - backtrace::get_data member func - decrease ignore_depth in backtrace::sample(int) - handle "omnitrace_main" in backtrace::filter_and_patch(...) - tweak internal thread state scope for pthread_mutex_gotcha wrappers * simplify tracing get_instrumentation_bundles usage * sampling updates - include backtrace_causal component - disable backtrace_metrics if using causal and not using perfetto - disable backtrace and backtrace_timestamp when using causal - post_process_causal * causal updates - more checks in blocking_gotcha and unblocking_gotcha start/stop - miscellaneous overhaul of data - experiment update * Remove virtual speedup * libomnitrace code_object * causal-profiling test * libomnitrace library.cpp updates - handle causal profiling - fini_bundle * Disable causal profiling by default * Updated causal code and example - example: three execution variants: cpu + rng, cpu, rng - example: three instrumentation variants: none, omni, coz - fix blocking gotcha credit - rework perform_experiment_impl - get_eligible_address_ranges - compute_eligible_lines - support fixed lines/speedups/functions - update selected_entry to support function mode - fix causal::delay - experiment updates * omnitrace_progress / omnitrace_user_progress - with accompanying omnitrace_annotated_progress / omnitrace_user_annotated_progress * Update timemory submodule * CausalMode - mode indicated whether causal predictions source be at line-level or function-level * code_object, config, runtime, sampling, thread_data - code_object: address_range - code_object: basic::line_info serialize(), name(), hash() - config updates - two signals for causal sampling - thread_data init fixes * pthread updates - pthread_create_gotcha processes delays - pthread_mutex_gotcha does not wrap pthread_join in causal mode * backtrace_causal update - dynamic delay period stats * main wrapper uses basename of argv[0] * update elfio submodule * perf support (currently unused) * Fix experiment JSON serialization - static_vector.hpp (unused) * causal executable + config options updates - omnitrace-causal exe simplifies running multiple causal configs - changed the causal config option names * Support both throughput and latency points * process-causal-json.py script - will be used later for testing * stable_vector * Rework thread_data * Improve omnitrace-causal exe - better verbosity handling - correct diagnosis of status for child process - execvpe when only one iteration (debugging) * Update timemory submodule * exe --version - omnitrace, omnitrace-avail, and omnitrace-sample all support --version on command-line * OMNITRACE_INTERNAL_API + OMNITRACE_{LIKELY,UNLIKELY} * omnitrace-causal cmake format * omnitrace config update - OMNITRACE_CAUSAL_FILE_CLOBBER * custom exception - wraps STL exception and gets stacktrace during construction * exit_gotcha supports _Exit * use global construct_on_init + max threads - add some safety when exceeding max # of threads * update code_object binary filter - exclude dyninst and tbbmalloc library * containers: c_array, static_vector, stable_vector - moved utility::c_array to container::c_array - created static_vector: std::vector bound to std::array - created stable_vector: vector with stable references * grow thread_data when new thread created * causal updates - data: improve compute_eligible_lines to ignore lambdas - data: use new thread_data - delay: use new thread_data - experiment: properly support latency points - experiment: support file clobber - experiment: ensure non-zero experiment time - progress_point: use new thread_data - backtrace_causal: use new thread_data * Update causal-profiling tests * fix omnitrace-causal backslash escaping * process-causal-json script * restructure causal implementation - update verbose messages for omnitrace-causal diagnose_status - migrated causal implementation in sampling.cpp to causal/sampling.cpp - OMNITRACE_USE_CAUSAL does not require OMNITRACE_USE_SAMPLING - added Mode::Causal - causal sampling uses same signals as regular sampling - moved tracing::thread_init to implementation file - combined tracing::thread_init and tracing::thread_init_sampling - added causal/components folder - pthread_create_gotcha::wrapper_config - omnitrace_preload checks OMNITRACE_USE_CAUSAL - updates mode accordingly * update timemory submodule * update timemory submodule * causal example updates - causal for lulesh * perf code + utility - helpers - relocated causal perf code - placement new when generating unique ptr trait for potentially allocating during sampling - additions to utility header - removed previously added helpers.hpp * update timemory submodule * Default env variables for omnitrace-causal - activate OMNITRACE_USE_KOKKOSP, etc. * update stable_vector and static_vector - static vector can use atomic for size tracking for thread-safe situations * update causal example header - CAUSAL_PROGRESS_NAMED - use CAUSAL_ prefix for some macros * Tweak lulesh example - use CAUSAL_PROGRESS instead of CAUSAL_BEGIN and CAUSAL_END * omnitrace-sample support for causal mode - set OMNITRACE_USE_SAMPLING to off when OMNITRACE_MODE=causal * refactor and cleanup code_object - scope filter - fixes to address_range * overhaul causal data + causal config options - full support for function and line mode - support static vector of instruction pointers - improve line info mapping resolution - remove thread-locality from miscellanous functions where unnecessary - causal options for {binary,source,function,fileline} exclusion * causal experiment, sampling, and backtrace updates - is_selected + unwind address array - experiment warning about progress points - increased buffer size for backtrace_casual sampler - backtrace_causal only stores IP addresses instead of full unwind info * category_region updates - minor refactor - local_category_region::mark * Update causal tests * Bump version to 1.8.0 * omnitrace-causal args + CLOBBER -> RESET - renamed OMNITRACE_CAUSAL_FILE_CLOBBER to OMNITRACE_CAUSAL_FILE_RESET - updated omnitrace-causal exe to support recently added configuration options - other miscellaneous tweaks to data.cpp, experiment.cpp, and sampling.cpp * Refactor causal and code_object - code_object.hpp and code_object.cpp moved into binary folder - causal components namespaced into omnitrace::causal::component - moved sample_data out of backtrace_causal and into own file - renamed backtrace_causal to causal::component::backtrace * preload omnitrace_init + OMNITRACE_DEBUG_MARK - env OMNITRACE_DEBUG_MARK - fix omnitrace_init call when LD_PRELOAD-ing omnitrace * Fix fileline support + line-info output names + experiment log - line-info log files are prefixed with experiment name - don't print experiment duration when E2E - account for fileline scope in analysis * KokkosP: OMNITRACE_KOKKOSP_NAME_LENGTH_MAX - config option to limit the name of kokkos tool callbacks - remove [kokkos] from KokkosP names * Update causal example - minor tweaks to decrease probability of overlapping regions in binary * omnitrace-causal update - prefix N / Ntot in environment printout * Miscellaneous updates - causal::finish_experimenting() - OMNITRACE_CAUSAL_RANDOM_SEED - KokkosP causal updates - exclude some callbacks, make some callbacks unique, etc. - address_range::operator+=(address_range) - combine contiguous ranges in binary/analysis.cpp when file, func, line is same and address range is contiguous - bfd_line_info reads inline info - wait for perform_experiment_impl to complete - causal::delay updates - delay::process checks if experiment is active - uses threading::get_id() - experiment scales duration up for larger speedup experiments - line info samples includes excluded lines - sampler uses CLOCK_REALTIME - blocking_gotcha updates - is no longer fully static - adds audit routine which sets the postblock value to zero if try/timed routine fails - category::host was added to causal_throughput_categories_t - pthread_create_gotcha sets new threads local parent delay - was using internal value, now uses sequent value * Causal improvements to KokkosP * Updates to experiment time scaling - use stats instead of just max * binary/link_map.{hpp,cpp} * update process-causal-json.py * Folded fileline scope into source scope * Update documentation - Add documentation for causal profiling - Replace 'Omnitrace' with 'OmniTrace' everywhere * Update causal-helpers.cmake + omnitrace-testing.cmake - split tests/CMakeLists.txt partially into omnitrace-testing.cmake * omnitrace/causal.h - OMNITRACE_CAUSAL_PROGRESS - OMNITRACE_CAUSAL_PROGRESS_NAMED - OMNITRACE_CAUSAL_BEGIN - OMNITRACE_CAUSAL_END * selected_entry + remove default filters for lambdas and operator() - selected entry stores range and binary load address * update process-causal-json.py * format examples/lulesh/CMakeLists.txt * causal-helpers find_package(Threads) * OMNITRACE_KOKKOSP_KERNEL_LOGGER - was OMNITRACE_KOKKOS_KERNEL_LOGGER * quiet find of coz-profiler * Fix rocm_smi exception handling * Update timemory submodule (binutils) - fix binutls compile error on some systems - bump binutils to v2.40 * Fix miscellaneous tests * OMNITRACE_KOKKOSP_PREFIX * revert rocm_smi handling * ElfUtils updates - default to download version 0.188 - add -Wno-error=null-dereference due to GCC 12 compiler error * Update causal example * Remove OMNITRACE_VERBOSE from global workflow envs * Reliable causal test * disable compilation of causal perf files * Remove set_current_selection with unwind stack * update timemory submodule * fix for segfault on bionic - locking in TLS dtor was causing segfault * remove experiment::is_selected(unwind_stack_t) * update default init of selected_entry * Fix for when IP is not offset by load address * Update CMakeLists.txt * Miscellaneous updates - OMNITRACE_WARNING_OR_CI_THROW - OMNITRACE_REQUIRE - OMNITRACE_PREFER - fixed issues with no ASLR - added load address variable and ipaddr() func to basic/bfd line info - removed get_basic() from dwarf_line_info - TIMEMORY_PREFER -> OMNITRACE_PREFER - removed previously added binary_address and range variables from selected_entry * Removed superfluous CausalState * Additional causal tests (lulesh + kokkos) * filter, prefer, analysis ASLR handling - removed default filter on cold functions - fixed OMNITRACE_PREFER - fixed analysis ASLR handling * Tweak line-info output * Removed some superfluous code - causal/delay - causal/selected_entry * Exclude main.cold in function mode * Update validate-perfetto-proto.py - account for occasional http errors * Add sampling test disabling tmp files * argparser for process-causal-json - support validation - support filtering * Avoid pthread_{lock,unlock} in sampling offload - use homemade atomic_mutex/atomic_lock since contention will be low and using pthread tools might trigger our wrappers * Rename process-causal-json.py - validate-causal-json.py * rework omnitrace_add_causal_test - capable of performing validation - added validation tests * Fix kokkosp_begin_deep_copy + causal * Tweak address range in bfd_line_info::read_pc * Tweak analysis and data IP handling - look for gaps * Disable scaling experiment time by speedup * Revert change in max threads during CI * binary updates - significant overhaul of binary analysis implementation - removed "basic_line_info" and "bfd_line_info" in lieu of "symbol" class - symbol class has basic BFD info + vector of inlines + vector of dwarf info * Updated causal to use new binary analysis - Fix symbol.cpp includes * Updated formatting target - include *.cmake files * Updated causal tests - causal tests should be stable now * Update timemory and dyninst submodules - TPLs are stripped + built w/o debug info * Increase tolerance for causal validation speedups - higher speedups have more variance (increased to +/- 5 from 3) * Support causal output for MPI - i.e. tag with MPI rank * omnitrace-causal launcher argument * improve experiment sampling output * causal data updates - call compute lines once - fixed filtered cached binary info - debugging info when experiment fails to start * Tweaked causal validation tests * dwarf_entry ranges * CI updates - increase max threads to 64 * Tweak causal E2E validation tests - more threads - shorter thread runtime - more iterations * Fix shadowed variable * fix symbol read_bfd last PC calculation * fix maybe-uninitialized warning * omnitrace-causal launcher update - only inject "omnitrace-causal --" once - throw error if no matches found * Update causal profiling docs for launcher * fix address range boundaries [ROCm/rocprofiler-systems commit: 9618ddefba]
2023-01-24 18:53:23 -06:00
@@ -16,6 +16,7 @@ endif()
 # executables
 add_subdirectory(omnitrace-avail)
 add_subdirectory(omnitrace-critical-trace)
+add_subdirectory(omnitrace-causal)
 add_subdirectory(omnitrace-sample)
 add_subdirectory(omnitrace)

@@ -40,6 +40,7 @@
 #include <timemory/components/skeletons.hpp>
 #include <timemory/mpl/types.hpp>
 #include <timemory/timemory.hpp>
+#include <timemory/unwind/bfd.hpp>
 #include <timemory/utility/types.hpp>
 #include <timemory/utility/utility.hpp>

@@ -113,6 +114,7 @@ int gpu_count = 0;
 int
 main(int argc, char** argv)
 {
+    tim::unwind::set_bfd_verbose(3);
    tim::set_env("OMNITRACE_INIT_TOOLING", "OFF", 1);
    omnitrace_init_library();

@@ -172,6 +174,9 @@ main(int argc, char** argv)
    parser_t parser("omnitrace-avail");

    parser.enable_help();
+    parser.enable_version("omnitrace-avail", "v" OMNITRACE_VERSION_STRING,
+                          OMNITRACE_GIT_DESCRIBE, OMNITRACE_GIT_REVISION);
+
    parser.set_help_width(40);
    parser.add_argument({ "--debug" }, "Enable debug messages")
        .max_count(1)
@@ -0,0 +1,27 @@
+# ------------------------------------------------------------------------------#
+#
+# omnitrace-causal target
+#
+# ------------------------------------------------------------------------------#
+
+add_executable(
+    omnitrace-causal
+    ${CMAKE_CURRENT_LIST_DIR}/omnitrace-causal.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/omnitrace-causal.hpp ${CMAKE_CURRENT_LIST_DIR}/impl.cpp)
+
+target_compile_definitions(omnitrace-causal PRIVATE TIMEMORY_CMAKE=1)
+target_include_directories(omnitrace-causal PRIVATE ${CMAKE_CURRENT_LIST_DIR})
+target_link_libraries(
+    omnitrace-causal
+    PRIVATE omnitrace::omnitrace-compile-definitions omnitrace::omnitrace-headers
+            omnitrace::omnitrace-common-library)
+set_target_properties(
+    omnitrace-causal PROPERTIES BUILD_RPATH "\$ORIGIN:\$ORIGIN/../${CMAKE_INSTALL_LIBDIR}"
+                                INSTALL_RPATH "${OMNITRACE_EXE_INSTALL_RPATH}")
+
+omnitrace_strip_target(omnitrace-causal)
+
+install(
+    TARGETS omnitrace-causal
+    DESTINATION ${CMAKE_INSTALL_BINDIR}
+    OPTIONAL)
@@ -0,0 +1,985 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "omnitrace-causal.hpp"
+
+#include "common/defines.h"
+#include "common/delimit.hpp"
+#include "common/environment.hpp"
+#include "common/join.hpp"
+#include "common/setup.hpp"
+
+#include <regex>
+#include <timemory/environment.hpp>
+#include <timemory/log/color.hpp>
+#include <timemory/utility/argparse.hpp>
+#include <timemory/utility/console.hpp>
+#include <timemory/utility/filepath.hpp>
+#include <timemory/utility/join.hpp>
+
+#include <array>
+#include <chrono>
+#include <cmath>
+#include <cstdint>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <iostream>
+#include <stdexcept>
+#include <string>
+#include <string_view>
+#include <sys/wait.h>
+#include <thread>
+#include <unistd.h>
+#include <vector>
+
+namespace color    = ::tim::log::color;
+namespace filepath = ::tim::filepath;
+namespace console  = ::tim::utility::console;
+namespace argparse = ::tim::argparse;
+using namespace timemory::join;
+using tim::get_env;
+using tim::log::colorized;
+using tim::log::stream;
+
+namespace std
+{
+std::string
+to_string(bool _v)
+{
+    return (_v) ? "true" : "false";
+}
+}  // namespace std
+
+namespace
+{
+int  verbose       = 0;
+auto updated_envs  = std::set<std::string_view>{};
+auto original_envs = std::set<std::string>{};
+auto child_pids    = std::set<pid_t>{};
+auto launcher      = std::string{};
+
+inline signal_handler&
+get_signal_handler(int _sig)
+{
+    static auto _v  = std::unordered_map<int, signal_handler>{};
+    auto        itr = _v.emplace(_sig, signal_handler{});
+    return itr.first->second;
+}
+
+void
+create_signal_handler(int sig, signal_handler& sh, void (*func)(int))
+{
+    if(sig < 1) return;
+    sh.m_custom_sigaction.sa_handler = func;
+    sigemptyset(&sh.m_custom_sigaction.sa_mask);
+    sh.m_custom_sigaction.sa_flags = SA_RESTART;
+    if(sigaction(sig, &sh.m_custom_sigaction, &sh.m_original_sigaction) == -1)
+    {
+        std::cerr << "Failed to create signal handler for " << sig << std::endl;
+    }
+}
+
+void
+forward_signal(int sig)
+{
+    for(auto itr : child_pids)
+    {
+        TIMEMORY_PRINTF_WARNING(stderr, "Killing pid=%i with signal %i...\n", itr, sig);
+        kill(itr, sig);
+        diagnose_status(itr, wait_pid(itr));
+    }
+    signal(sig, SIG_DFL);
+    kill(getpid(), sig);
+}
+}  // namespace
+
+int
+get_verbose()
+{
+    verbose = get_env("OMNITRACE_CAUSAL_VERBOSE",
+                      get_env<int>("OMNITRACE_VERBOSE", verbose, false));
+    auto _debug =
+        get_env("OMNITRACE_CAUSAL_DEBUG", get_env<bool>("OMNITRACE_DEBUG", false, false));
+    if(_debug) verbose += 8;
+    return verbose;
+}
+
+void
+forward_signals(const std::set<int>& _signals)
+{
+    for(auto itr : _signals)
+        create_signal_handler(itr, get_signal_handler(itr), &forward_signal);
+}
+
+void
+add_child_pid(pid_t _v)
+{
+    child_pids.emplace(_v);
+}
+
+void
+remove_child_pid(pid_t _v)
+{
+    child_pids.erase(_v);
+}
+
+int
+wait_pid(pid_t _pid, int _opts)
+{
+    int   _status = 0;
+    pid_t _pid_v  = -1;
+    _opts |= WUNTRACED;
+    do
+    {
+        if((_opts & WNOHANG) > 0)
+            std::this_thread::sleep_for(std::chrono::milliseconds{ 100 });
+        _pid_v = waitpid(_pid, &_status, _opts);
+    } while(_pid <= 0);
+    return _status;
+}
+
+int
+diagnose_status(pid_t _pid, int _status)
+{
+    auto _verbose = get_verbose();
+    if(_verbose >= 3)
+    {
+        fflush(stderr);
+        fflush(stdout);
+        std::cout << std::flush;
+        std::cerr << std::flush;
+    }
+
+    bool _normal_exit      = (WIFEXITED(_status) > 0);
+    bool _unhandled_signal = (WIFSIGNALED(_status) > 0);
+    bool _core_dump        = (WCOREDUMP(_status) > 0);
+    bool _stopped          = (WIFSTOPPED(_status) > 0);
+    int  _exit_status      = WEXITSTATUS(_status);
+    int  _stop_signal      = (_stopped) ? WSTOPSIG(_status) : 0;
+    int  _ec               = (_unhandled_signal) ? WTERMSIG(_status) : 0;
+
+    if(_verbose >= 4)
+    {
+        TIMEMORY_PRINTF_INFO(
+            stderr,
+            "diagnosing status for process %i :: status: %i... normal exit: %s, "
+            "unhandled signal: %s, core dump: %s, stopped: %s, exit status: %i, stop "
+            "signal: %i, exit code: %i\n",
+            _pid, _status, std::to_string(_normal_exit).c_str(),
+            std::to_string(_unhandled_signal).c_str(), std::to_string(_core_dump).c_str(),
+            std::to_string(_stopped).c_str(), _exit_status, _stop_signal, _ec);
+    }
+    else if(_verbose >= 3)
+    {
+        TIMEMORY_PRINTF_INFO(stderr,
+                             "diagnosing status for process %i :: status: %i ...\n", _pid,
+                             _status);
+    }
+
+    if(!_normal_exit)
+    {
+        if(_ec == 0) _ec = EXIT_FAILURE;
+        if(_verbose >= 0)
+        {
+            TIMEMORY_PRINTF_FATAL(
+                stderr, "process %i terminated abnormally. exit code: %i\n", _pid, _ec);
+        }
+    }
+
+    if(_stopped)
+    {
+        if(_verbose >= 0)
+        {
+            TIMEMORY_PRINTF_FATAL(stderr,
+                                  "process %i stopped with signal %i. exit code: %i\n",
+                                  _pid, _stop_signal, _ec);
+        }
+    }
+
+    if(_core_dump)
+    {
+        if(_verbose >= 0)
+        {
+            TIMEMORY_PRINTF_FATAL(
+                stderr, "process %i terminated and produced a core dump. exit code: %i\n",
+                _pid, _ec);
+        }
+    }
+
+    if(_unhandled_signal)
+    {
+        if(_verbose >= 0)
+        {
+            TIMEMORY_PRINTF_FATAL(stderr,
+                                  "process %i terminated because it received a signal "
+                                  "(%i) that was not handled. exit code: %i\n",
+                                  _pid, _ec, _ec);
+        }
+    }
+
+    if(!_normal_exit && _exit_status > 0)
+    {
+        if(_verbose >= 0)
+        {
+            if(_exit_status == 127)
+            {
+                TIMEMORY_PRINTF_FATAL(
+                    stderr, "execv in process %i failed. exit code: %i\n", _pid, _ec);
+            }
+            else
+            {
+                TIMEMORY_PRINTF_FATAL(
+                    stderr,
+                    "process %i terminated with a non-zero status. exit code: %i\n", _pid,
+                    _ec);
+            }
+        }
+    }
+
+    return _ec;
+}
+
+std::string
+get_realpath(const std::string& _v)
+{
+    auto* _tmp = realpath(_v.c_str(), nullptr);
+    auto  _ret = std::string{ _tmp };
+    free(_tmp);
+    return _ret;
+}
+
+void
+print_command(const std::vector<char*>& _argv, std::string_view _prefix)
+{
+    if(verbose >= 1)
+        stream(std::cout, color::info())
+            << _prefix << "Executing '" << join(array_config{ " " }, _argv) << "'...\n";
+
+    std::cerr << color::end() << std::flush;
+}
+
+std::vector<char*>
+get_initial_environment()
+{
+    std::vector<char*> _env;
+    if(environ != nullptr)
+    {
+        int idx = 0;
+        while(environ[idx] != nullptr)
+        {
+            auto* _v = environ[idx++];
+            original_envs.emplace(_v);
+            _env.emplace_back(strdup(_v));
+        }
+    }
+
+    update_env(_env, "OMNITRACE_MODE", "causal");
+    update_env(_env, "OMNITRACE_USE_CAUSAL", true);
+    update_env(_env, "OMNITRACE_USE_SAMPLING", false);
+    update_env(_env, "OMNITRACE_USE_PERFETTO", false);
+    update_env(_env, "OMNITRACE_USE_TIMEMORY", false);
+    update_env(_env, "OMNITRACE_USE_PROCESS_SAMPLING", false);
+    update_env(_env, "OMNITRACE_CRITICAL_TRACE", false);
+
+    return _env;
+}
+
+void
+prepare_command_for_run(char* _exe, std::vector<char*>& _argv)
+{
+    if(!launcher.empty())
+    {
+        bool _injected = false;
+        auto _new_argv = std::vector<char*>{};
+        for(auto* itr : _argv)
+        {
+            if(!_injected && std::regex_search(itr, std::regex{ launcher }))
+            {
+                _new_argv.emplace_back(_exe);
+                _new_argv.emplace_back(strdup("--"));
+                _injected = true;
+            }
+            _new_argv.emplace_back(itr);
+        }
+
+        if(!_injected)
+        {
+            throw std::runtime_error(
+                join("", "omnitrace-causal was unable to match \"", launcher,
+                     "\" to any arguments on the command line: \"",
+                     join(array_config{ " ", "", "" }, _argv), "\""));
+        }
+
+        std::swap(_argv, _new_argv);
+    }
+}
+
+void
+prepare_environment_for_run(std::vector<char*>& _env)
+{
+    if(launcher.empty())
+    {
+        update_env(_env, "LD_PRELOAD",
+                   get_realpath(get_internal_libpath("libomnitrace-dl.so")), true);
+    }
+}
+
+std::string
+get_internal_libpath(const std::string& _lib)
+{
+    auto _exe = std::string_view{ realpath("/proc/self/exe", nullptr) };
+    auto _pos = _exe.find_last_of('/');
+    auto _dir = std::string{ "./" };
+    if(_pos != std::string_view::npos) _dir = _exe.substr(0, _pos);
+    return omnitrace::common::join("/", _dir, "..", "lib", _lib);
+}
+
+void
+print_updated_environment(std::vector<char*> _env, std::string_view _prefix)
+{
+    if(get_verbose() < 0) return;
+
+    std::sort(_env.begin(), _env.end(), [](auto* _lhs, auto* _rhs) {
+        if(!_lhs) return false;
+        if(!_rhs) return true;
+        return std::string_view{ _lhs } < std::string_view{ _rhs };
+    });
+
+    std::vector<std::string_view> _updates = {};
+    std::vector<std::string_view> _general = {};
+
+    for(auto* itr : _env)
+    {
+        if(itr == nullptr) continue;
+
+        auto _is_omni = (std::string_view{ itr }.find("OMNITRACE") == 0);
+        auto _updated = false;
+        for(const auto& vitr : updated_envs)
+        {
+            if(std::string_view{ itr }.find(vitr) == 0)
+            {
+                _updated = true;
+                break;
+            }
+        }
+
+        if(_updated)
+            _updates.emplace_back(itr);
+        else if(verbose >= 1 && _is_omni)
+            _general.emplace_back(itr);
+    }
+
+    if(_general.size() + _updates.size() == 0 || verbose < 0) return;
+
+    std::cerr << std::endl;
+
+    for(auto& itr : _general)
+        stream(std::cerr, color::source()) << _prefix << itr << "\n";
+    for(auto& itr : _updates)
+        stream(std::cerr, color::source()) << _prefix << itr << "\n";
+
+    std::cerr << color::end() << std::flush;
+}
+
+template <typename Tp>
+void
+update_env(std::vector<char*>& _environ, std::string_view _env_var, Tp&& _env_val,
+           bool _append, std::string_view _join_delim)
+{
+    updated_envs.emplace(_env_var);
+
+    auto _key = join("", _env_var, "=");
+    for(auto& itr : _environ)
+    {
+        if(!itr) continue;
+        if(std::string_view{ itr }.find(_key) == 0)
+        {
+            if(_append)
+            {
+                if(std::string_view{ itr }.find(join("", _env_val)) ==
+                   std::string_view::npos)
+                {
+                    auto _val = std::string{ itr }.substr(_key.length());
+                    free(itr);
+                    itr = strdup(
+                        join('=', _env_var, join(_join_delim, _env_val, _val)).c_str());
+                }
+            }
+            else
+            {
+                free(itr);
+                itr = strdup(omnitrace::common::join('=', _env_var, _env_val).c_str());
+            }
+            return;
+        }
+    }
+    _environ.emplace_back(
+        strdup(omnitrace::common::join('=', _env_var, _env_val).c_str()));
+}
+
+template <typename Tp>
+void
+add_default_env(std::vector<char*>& _environ, std::string_view _env_var, Tp&& _env_val)
+{
+    auto _key = join("", _env_var, "=");
+    for(auto& itr : _environ)
+    {
+        if(!itr) continue;
+        if(std::string_view{ itr }.find(_key) == 0) return;
+    }
+
+    updated_envs.emplace(_env_var);
+    _environ.emplace_back(
+        strdup(omnitrace::common::join('=', _env_var, _env_val).c_str()));
+}
+
+void
+remove_env(std::vector<char*>& _environ, std::string_view _env_var)
+{
+    auto _key   = join("", _env_var, "=");
+    auto _match = [&_key](auto itr) { return std::string_view{ itr }.find(_key) == 0; };
+
+    _environ.erase(std::remove_if(_environ.begin(), _environ.end(), _match),
+                   _environ.end());
+
+    for(const auto& itr : original_envs)
+    {
+        if(std::string_view{ itr }.find(_key) == 0)
+            _environ.emplace_back(strdup(itr.c_str()));
+    }
+}
+
+std::vector<char*>
+parse_args(int argc, char** argv, std::vector<char*>& _env,
+           std::vector<std::map<std::string_view, std::string>>& _causal_envs)
+{
+    using parser_t     = argparse::argument_parser;
+    using parser_err_t = typename parser_t::result_type;
+
+    auto help_check = [](parser_t& p, int _argc, char** _argv) {
+        std::set<std::string> help_args = { "-h", "--help", "-?" };
+        return (p.exists("help") || _argc == 1 ||
+                (_argc > 1 && help_args.find(_argv[1]) != help_args.end()));
+    };
+
+    auto _pec        = EXIT_SUCCESS;
+    auto help_action = [&_pec, argc, argv](parser_t& p) {
+        if(_pec != EXIT_SUCCESS)
+        {
+            std::stringstream msg;
+            msg << "Error in command:";
+            for(int i = 0; i < argc; ++i)
+                msg << " " << argv[i];
+            msg << "\n\n";
+            stream(std::cerr, color::fatal()) << msg.str();
+            std::cerr << std::flush;
+        }
+
+        p.print_help();
+        exit(_pec);
+    };
+
+    const auto* _desc = R"desc(
+    Causal profiling usually requires multiple runs to reliably resolve the speedup estimates.
+    This executable is designed to streamline that process.
+    For example (assume all commands end with '-- <exe> <args>'):
+
+        omnitrace-causal -n 5 -- <exe>                  # runs <exe> 5x with causal profiling enabled
+
+        omnitrace-causal -s 0 5,10,15,20                # runs <exe> 2x with virtual speedups:
+                                                        #   - 0
+                                                        #   - randomly selected from 5, 10, 15, and 20
+
+        omnitrace-causal -F func_A func_B func_(A|B)    # runs <exe> 3x with the function scope limited to:
+                                                        #   1. func_A
+                                                        #   2. func_B
+                                                        #   3. func_A or func_B
+    General tips:
+    - Insert progress points at hotspots in your code or use omnitrace's runtime instrumentation
+        - Note: binary rewrite will produce a incompatible new binary
+    - Collect a flat profile via sampling
+        - E.g., omnitrace-sample -F -- <exe> <args>
+        - Inspect sampling_wall_clock.txt and sampling_cpu_clock.txt for functions to target
+    - Run omnitrace-causal in "function" mode first (does not require debug info)
+    - Run omnitrace-causal in "line" mode when you are targeting one function (requires debug info)
+        - Preferably, use predictions from the "function" mode to determine which function to target
+    - Limit the virtual speedups to a smaller pool, e.g., 0,5,10,25,50, to get reliable predictions quicker
+    - Make use of the binary, source, and function scope to limit the functions/lines selected for experiments
+        - Note: source scope requires debug info
+    )desc";
+
+    auto parser = parser_t{ basename(argv[0]), _desc };
+
+    parser.on_error([](parser_t&, const parser_err_t& _err) {
+        stream(std::cerr, color::fatal()) << _err << "\n";
+        exit(EXIT_FAILURE);
+    });
+
+    auto _add_separator = [&](std::string _v, const std::string& _desc) {
+        parser.add_argument({ "" }, "");
+        parser
+            .add_argument({ join("", "[", _v, "]") },
+                          (_desc.empty()) ? _desc : join({ "", "(", ")" }, _desc))
+            .color(color::info());
+        parser.add_argument({ "" }, "");
+    };
+
+    parser.enable_help();
+    parser.enable_version("omnitrace-causal", "v" OMNITRACE_VERSION_STRING,
+                          OMNITRACE_GIT_DESCRIBE, OMNITRACE_GIT_REVISION);
+
+    auto _cols = std::get<0>(console::get_columns());
+    if(_cols > parser.get_help_width() + 8)
+        parser.set_description_width(
+            std::min<int>(_cols - parser.get_help_width() - 8, 120));
+
+    _add_separator("DEBUG OPTIONS", "");
+    parser.add_argument({ "--monochrome" }, "Disable colorized output")
+        .max_count(1)
+        .dtype("bool")
+        .action([&](parser_t& p) {
+            auto _colorized = !p.get<bool>("monochrome");
+            colorized()     = _colorized;
+            p.set_use_color(_colorized);
+            update_env(_env, "OMNITRACE_COLORIZED_LOG", (_colorized) ? "1" : "0");
+            update_env(_env, "COLORIZED_LOG", (_colorized) ? "1" : "0");
+        });
+    parser.add_argument({ "--debug" }, "Debug output")
+        .max_count(1)
+        .action([&](parser_t& p) {
+            update_env(_env, "OMNITRACE_DEBUG", p.get<bool>("debug"));
+        });
+    parser.add_argument({ "-v", "--verbose" }, "Verbose output")
+        .count(1)
+        .action([&](parser_t& p) {
+            auto _v = p.get<int>("verbose");
+            verbose = _v;
+            update_env(_env, "OMNITRACE_VERBOSE", _v);
+        });
+
+    std::string _config_file      = {};
+    std::string _config_folder    = "omnitrace-causal-config";
+    bool        _generate_configs = false;
+    bool        _add_defaults     = true;
+
+    _add_separator("GENERAL OPTIONS", "");
+    parser.add_argument({ "-c", "--config" }, "Base configuration file")
+        .min_count(0)
+        .dtype("filepath")
+        .action([&](parser_t& p) {
+            _config_file =
+                join(array_config{ ":" }, p.get<std::vector<std::string>>("config"));
+        });
+    parser
+        .add_argument(
+            { "-l", "--launcher" },
+            "When running MPI jobs, omnitrace-causal needs to be *before* the executable "
+            "which launches the MPI processes (i.e. before `mpirun`, `srun`, etc.). Pass "
+            "the name of the target executable (or a regex for matching to the name of "
+            "the target) for causal profiling, e.g., `omnitrace-causal -l foo -- mpirun "
+            "-n 4 foo`. This ensures that the omnitrace library is LD_PRELOADed on the "
+            "proper target")
+        .count(1)
+        .dtype("executable")
+        .action([&](parser_t& p) { launcher = p.get<std::string>("launcher"); });
+    parser
+        .add_argument({ "-g", "--generate-configs" },
+                      "Generate config files instead of passing environment variables "
+                      "directly. If no arguments are provided, the config files will be "
+                      "placed in ${PWD}/omnitrace-causal-config folder")
+        .min_count(0)
+        .max_count(1)
+        .dtype("folder")
+        .action([&](parser_t& p) {
+            _generate_configs = true;
+            auto _dir         = p.get<std::string>("generate-configs");
+            if(!_dir.empty()) _config_folder = std::move(_dir);
+            if(!filepath::exists(_config_folder)) filepath::makedir(_config_folder);
+        });
+    parser
+        .add_argument({ "--no-defaults" },
+                      "Do not activate default features which are recommended for causal "
+                      "profiling. For example: PID-tagging of output files and "
+                      "timestamped subdirectories are disabled by default. Kokkos tools "
+                      "support is added by default (OMNITRACE_USE_KOKKOSP=ON) because, "
+                      "for Kokkos applications, the Kokkos-Tools callbacks are used for "
+                      "progress points. Activation of OpenMP tools support is similar")
+        .min_count(0)
+        .max_count(1)
+        .dtype("bool")
+        .action([&](parser_t& p) { _add_defaults = !p.get<bool>("no-defaults"); });
+
+    _add_separator("CAUSAL PROFILING OPTIONS (General)",
+                   "These settings will be applied to all causal profiling runs");
+    parser.add_argument({ "-m", "--mode" }, "Causal profiling mode")
+        .count(1)
+        .dtype("string")
+        .choices({ "function", "line" })
+        .choice_alias("function", { "func" })
+        .action([&](parser_t& p) {
+            update_env(_env, "OMNITRACE_CAUSAL_MODE", p.get<std::string>("mode"));
+        });
+
+    parser
+        .add_argument({ "-o", "--output-name" },
+                      "Output filename of causal profiling data w/o extension")
+        .min_count(1)
+        .dtype("filename")
+        .action([&](parser_t& p) {
+            update_env(_env, "OMNITRACE_CAUSAL_FILE", p.get<std::string>("output-name"));
+        });
+
+    bool _reset = false;
+
+    parser
+        .add_argument({ "-r", "--reset" },
+                      "Overwrite any existing experiment results during the first run")
+        .max_count(1)
+        .dtype("bool")
+        .action([&](parser_t& p) { _reset = p.get<bool>("reset"); });
+
+    parser
+        .add_argument({ "-e", "--end-to-end" },
+                      "Single causal experiment for the entire application runtime")
+        .max_count(1)
+        .dtype("bool")
+        .action([&](parser_t& p) {
+            update_env(_env, "OMNITRACE_CAUSAL_END_TO_END", p.get<bool>("end-to-end"));
+        });
+
+    parser
+        .add_argument({ "-w", "--wait" },
+                      "Set the wait time (i.e. delay) before starting the first causal "
+                      "experiment (in seconds)")
+        .count(1)
+        .dtype("seconds")
+        .action([&](parser_t& p) {
+            update_env(_env, "OMNITRACE_CAUSAL_DELAY", p.get<double>("wait"));
+        });
+
+    parser
+        .add_argument(
+            { "-d", "--duration" },
+            "Set the length of time (in seconds) to perform causal experimentationafter "
+            "the first experiment is started. Once this amount of time has elapsed, no "
+            "more causal experiments will be started but any currently running "
+            "experiment will be allowed to finish.")
+        .count(1)
+        .dtype("seconds")
+        .action([&](parser_t& p) {
+            update_env(_env, "OMNITRACE_CAUSAL_DURATION", p.get<double>("duration"));
+        });
+
+    int64_t _niterations       = 1;
+    auto    _virtual_speedups  = std::vector<std::string>{};
+    auto    _function_scopes   = std::vector<std::string>{};
+    auto    _binary_scopes     = std::vector<std::string>{};
+    auto    _source_scopes     = std::vector<std::string>{};
+    auto    _function_excludes = std::vector<std::string>{};
+    auto    _binary_excludes   = std::vector<std::string>{};
+    auto    _source_excludes   = std::vector<std::string>{};
+
+    parser
+        .add_argument({ "-n", "--iterations" },
+                      "Number of times to repeat the combination of run configurations")
+        .count(1)
+        .dtype("int")
+        .action([&](parser_t& p) { _niterations = p.get<int64_t>("iterations"); });
+
+    _add_separator(
+        "CAUSAL PROFILING OPTIONS (Combinatorial)",
+        "Each individual argument to these options will multiply the number runs by the "
+        "number of arguments and the number of iterations. E.g. -n 2 -B \"MAIN\" -F "
+        "\"foo\" \"bar\" will produce 4 runs: 2 iterations x 1 binary scope x 2 function "
+        "scopes (MAIN+foo, MAIN+bar, MAIN+foo, MAIN+bar)");
+
+    parser
+        .add_argument({ "-s", "--speedups" },
+                      "Pool of virtual speedups to sample from during experimentation. "
+                      "Each space designates a group and multiple speedups can be "
+                      "grouped together by commas, e.g. -s 0 0,10,20-50 is two groups: "
+                      "group #1 is '0' and group #2 is '0 10 20 25 30 35 40 45 50'")
+        .min_count(0)
+        .max_count(-1)
+        .dtype("integers")
+        .action([&](parser_t& p) {
+            _virtual_speedups = p.get<std::vector<std::string>>("speedups");
+        });
+
+    parser
+        .add_argument({ "-B", "--binary-scope" },
+                      "Restricts causal experiments to the binaries matching the list of "
+                      "regular expressions. Each space designates a group and multiple "
+                      "scopes can be grouped together with a semi-colon")
+        .min_count(0)
+        .max_count(-1)
+        .dtype("integers")
+        .action([&](parser_t& p) {
+            _binary_scopes = p.get<std::vector<std::string>>("binary-scope");
+        });
+
+    parser
+        .add_argument({ "-S", "--source-scope" },
+                      "Restricts causal experiments to the source files or source file + "
+                      "lineno pairs (i.e. <file> or <file>:<line>) matching the list of "
+                      "regular expressions. Each space designates a group and multiple "
+                      "scopes can be grouped together with a semi-colon")
+        .min_count(0)
+        .max_count(-1)
+        .dtype("integers")
+        .action([&](parser_t& p) {
+            _source_scopes = p.get<std::vector<std::string>>("source-scope");
+        });
+
+    parser
+        .add_argument(
+            { "-F", "--function-scope" },
+            "Restricts causal experiments to the functions matching the list of "
+            "regular expressions. Each space designates a group and multiple "
+            "scopes can be grouped together with a semi-colon")
+        .min_count(0)
+        .max_count(-1)
+        .dtype("regex-list")
+        .action([&](parser_t& p) {
+            _function_scopes = p.get<std::vector<std::string>>("function-scope");
+        });
+
+    parser
+        .add_argument(
+            { "-BE", "--binary-exclude" },
+            "Excludes causal experiments from being performed on the binaries matching "
+            "the list of regular expressions. Each space designates a group and multiple "
+            "excludes can be grouped together with a semi-colon")
+        .min_count(0)
+        .max_count(-1)
+        .dtype("integers")
+        .action([&](parser_t& p) {
+            _binary_excludes = p.get<std::vector<std::string>>("binary-exclude");
+        });
+
+    parser
+        .add_argument(
+            { "-SE", "--source-exclude" },
+            "Excludes causal experiments from being performed on the code from the "
+            "source files or source file + lineno pair (i.e. <file> or <file>:<line>) "
+            "matching the list of regular expressions. Each space designates a group and "
+            "multiple excludes can be grouped together with a semi-colon")
+        .min_count(0)
+        .max_count(-1)
+        .dtype("integers")
+        .action([&](parser_t& p) {
+            _source_excludes = p.get<std::vector<std::string>>("source-exclude");
+        });
+
+    parser
+        .add_argument(
+            { "-FE", "--function-exclude" },
+            "Excludes causal experiments from being performed on the functions matching "
+            "the list of regular expressions. Each space designates a group and multiple "
+            "excludes can be grouped together with a semi-colon")
+        .min_count(0)
+        .max_count(-1)
+        .dtype("regex-list")
+        .action([&](parser_t& p) {
+            _function_excludes = p.get<std::vector<std::string>>("function-exclude");
+        });
+
+#if OMNITRACE_HIP_VERSION > 0 && OMNITRACE_HIP_VERSION < 50300
+    update_env(_env, "HSA_ENABLE_INTERRUPT", 0);
+#endif
+
+    auto _inpv = std::vector<char*>{};
+    auto _outv = std::vector<char*>{};
+    bool _hash = false;
+    for(int i = 0; i < argc; ++i)
+    {
+        if(_hash)
+        {
+            _outv.emplace_back(argv[i]);
+        }
+        else if(std::string_view{ argv[i] } == "--")
+        {
+            _hash = true;
+        }
+        else
+        {
+            _inpv.emplace_back(argv[i]);
+        }
+    }
+
+    auto _cerr = parser.parse_args(_inpv.size(), _inpv.data());
+    if(help_check(parser, argc, argv))
+        help_action(parser);
+    else if(_cerr)
+        throw std::runtime_error(_cerr.what());
+
+    if(_niterations < 1) _niterations = 1;
+    auto _get_size = [](const auto& _v) { return std::max<size_t>(_v.size(), 1); };
+
+    auto _causal_envs_tmp = std::vector<std::map<std::string_view, std::string>>{};
+    auto _fill = [&_causal_envs_tmp](std::string_view _env_var, const auto& _data,
+                                     bool _quote) {
+        if(_data.empty()) return;
+        if(_causal_envs_tmp.empty()) _causal_envs_tmp.emplace_back();
+        auto _tmp = _causal_envs_tmp;
+        _causal_envs_tmp.clear();
+        _causal_envs_tmp.reserve(_data.size() * _tmp.size());
+        for(auto ditr : _data)
+        {
+            if(_quote)
+            {
+                ditr.insert(0, "\"");
+                ditr += "\"";
+            }
+
+            // duplicate the env, add the env variable, emplace back
+            for(auto itr : _tmp)
+            {
+                itr[_env_var] = ditr;
+                _causal_envs_tmp.emplace_back(itr);
+            }
+        }
+    };
+
+    if(_add_defaults)
+    {
+        add_default_env(_env, "OMNITRACE_TIME_OUTPUT", false);
+        add_default_env(_env, "OMNITRACE_USE_PID", false);
+        add_default_env(_env, "OMNITRACE_USE_KOKKOSP", true);
+
+#if defined(OMNITRACE_USE_OMPT) && OMNITRACE_USE_OMPT > 0
+        add_default_env(_env, "OMNITRACE_USE_OMPT", true);
+#endif
+
+#if(defined(OMNITRACE_USE_MPI) && OMNITRACE_USE_MPI > 0) ||                              \
+    (defined(OMNITRACE_USE_MPI_HEADERS) && OMNITRACE_USE_MPI_HEADERS > 0)
+        add_default_env(_env, "OMNITRACE_USE_MPIP", true);
+#endif
+
+#if defined(OMNITRACE_USE_ROCTRACER) && OMNITRACE_USE_ROCTRACER > 0
+        add_default_env(_env, "OMNITRACE_ROCTRACER_HIP_API", true);
+        add_default_env(_env, "OMNITRACE_ROCTRACER_HSA_API", true);
+#endif
+
+#if defined(OMNITRACE_USE_RCCL) && OMNITRACE_USE_RCCL > 0
+        add_default_env(_env, "OMNITRACE_USE_RCCLP", true);
+#endif
+    }
+
+    _fill("OMNITRACE_CAUSAL_BINARY_EXCLUDE", _binary_excludes, _generate_configs);
+    _fill("OMNITRACE_CAUSAL_SOURCE_EXCLUDE", _source_excludes, _generate_configs);
+    _fill("OMNITRACE_CAUSAL_FUNCTION_EXCLUDE", _function_excludes, _generate_configs);
+
+    _fill("OMNITRACE_CAUSAL_BINARY_SCOPE", _binary_scopes, _generate_configs);
+    _fill("OMNITRACE_CAUSAL_SOURCE_SCOPE", _source_scopes, _generate_configs);
+    _fill("OMNITRACE_CAUSAL_FUNCTION_SCOPE", _function_scopes, _generate_configs);
+
+    _fill("OMNITRACE_CAUSAL_FIXED_SPEEDUP", _virtual_speedups, false);
+
+    // make sure at least one env exists
+    if(_causal_envs_tmp.empty()) _causal_envs_tmp.emplace_back();
+
+    // duplicate for the number of iterations
+    _causal_envs.clear();
+    _causal_envs.reserve(_niterations * _causal_envs_tmp.size());
+    for(int64_t i = 0; i < _niterations; ++i)
+    {
+        for(const auto& itr : _causal_envs_tmp)
+            _causal_envs.emplace_back(itr);
+    }
+
+    if(_generate_configs)
+    {
+        auto _is_omni_cfg = [](std::string_view itr) {
+            return (itr.find("OMNITRACE") == 0 && itr.find("OMNITRACE_MODE") != 0 &&
+                    itr.find("OMNITRACE_CONFIG_FILE") != 0 &&
+                    itr.find('=') < itr.length());
+        };
+
+        auto _omni_env = std::map<std::string, std::string>{};
+        for(auto* itr : _env)
+        {
+            if(_is_omni_cfg(itr))
+            {
+                auto _env_var = std::string{ itr };
+                auto _pos     = _env_var.find('=');
+                auto _env_val = _env_var.substr(_pos + 1);
+                _env_var      = _env_var.substr(0, _pos);
+                _omni_env.emplace(_env_var, _env_val);
+            }
+        }
+
+        _env.erase(std::remove_if(_env.begin(), _env.end(), _is_omni_cfg), _env.end());
+
+        _causal_envs_tmp = std::move(_causal_envs);
+        _causal_envs.clear();
+        auto _write_config =
+            [_omni_env](std::ostream&                                  _os,
+                        const std::map<std::string_view, std::string>& _data) {
+                size_t _width = 0;
+                for(const auto& itr : _omni_env)
+                    _width = std::max(_width, itr.first.length());
+
+                for(const auto& itr : _data)
+                    _width = std::max(_width, itr.first.length());
+
+                _os << "# omnitrace common settings\n";
+                for(const auto& itr : _omni_env)
+                    _os << std::setw(_width + 1) << std::left << itr.first << " = "
+                        << itr.second << "\n";
+
+                _os << "\n# omnitrace causal settings\n";
+                for(const auto& itr : _data)
+                    _os << std::setw(_width + 1) << std::left << itr.first << " = "
+                        << itr.second << "\n";
+            };
+
+        int nwidth = (std::log10(_causal_envs_tmp.size()) + 1);
+        for(size_t i = 0; i < _causal_envs_tmp.size(); ++i)
+        {
+            std::stringstream fname{};
+            fname.fill('0');
+            fname << _config_folder << "/causal-" << std::setw(nwidth) << i << ".cfg";
+            std::ofstream _ofs{ fname.str() };
+            _write_config(_ofs, _causal_envs_tmp.at(i));
+            auto _cfg_name = (_config_file.empty())
+                                 ? fname.str()
+                                 : join(array_config{ ":" }, _config_file, fname.str());
+            auto _cfg =
+                std::map<std::string_view, std::string>{ { "OMNITRACE_CONFIG_FILE",
+                                                           _cfg_name } };
+            _causal_envs.emplace_back(_cfg);
+        }
+    }
+
+    if(_reset)
+        _causal_envs.front().emplace(std::string_view{ "OMNITRACE_CAUSAL_FILE_RESET" },
+                                     std::string{ "true" });
+
+    return _outv;
+}
+
+// explicit instantiation for usage in omnitrace-causal.cpp
+template void
+update_env(std::vector<char*>&, std::string_view, const std::string& _env_val,
+           bool _append, std::string_view);
@@ -0,0 +1,136 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "omnitrace-causal.hpp"
+
+#include <timemory/log/macros.hpp>
+
+#include <algorithm>
+#include <cmath>
+#include <iomanip>
+#include <iostream>
+#include <map>
+#include <sstream>
+#include <string_view>
+#include <unistd.h>
+
+int
+main(int argc, char** argv)
+{
+    auto _base_env   = get_initial_environment();
+    auto _causal_env = std::vector<std::map<std::string_view, std::string>>{};
+
+    bool _has_double_hyphen = false;
+    for(int i = 1; i < argc; ++i)
+    {
+        auto _arg = std::string_view{ argv[i] };
+        if(_arg == "--" || _arg == "-?" || _arg == "-h" || _arg == "--help" ||
+           _arg == "--version")
+            _has_double_hyphen = true;
+    }
+
+    std::vector<char*> _argv = {};
+    if(_has_double_hyphen)
+    {
+        _argv = parse_args(argc, argv, _base_env, _causal_env);
+    }
+    else
+    {
+        _argv.reserve(argc);
+        for(int i = 1; i < argc; ++i)
+            _argv.emplace_back(argv[i]);
+        _causal_env.resize(1);
+    }
+
+    prepare_command_for_run(argv[0], _argv);
+    prepare_environment_for_run(_base_env);
+
+    if(get_verbose() >= 3)
+    {
+        TIMEMORY_PRINTF_INFO(stderr, "causal environments to be executed:\n");
+        size_t _n = 0;
+        for(auto& citr : _causal_env)
+        {
+            auto _env = _base_env;
+            for(const auto& eitr : citr)
+                update_env(_env, eitr.first, eitr.second);
+            auto _prefix = std::to_string(_n++) + ":  ";
+            print_updated_environment(_env, _prefix);
+        }
+    }
+
+    if(!_argv.empty())
+    {
+        if(_causal_env.size() == 1)
+        {
+            auto _env = _base_env;
+            for(const auto& eitr : _causal_env.front())
+                update_env(_env, eitr.first, eitr.second);
+            print_updated_environment(_env, "0: ");
+            print_command(_argv, "0: ");
+            _argv.emplace_back(nullptr);
+            _env.emplace_back(nullptr);
+            return execvpe(_argv.front(), _argv.data(), _env.data());
+        }
+
+        forward_signals({ SIGINT, SIGTERM, SIGQUIT });
+        size_t _ncount = 0;
+        size_t _width  = std::log10(_causal_env.size()) + 1;
+        for(auto& citr : _causal_env)
+        {
+            auto _n        = _ncount++;
+            auto _main_pid = getpid();
+            auto _pid      = fork();
+
+            if(get_verbose() >= 3)
+            {
+                TIMEMORY_PRINTF_INFO(stderr, "process %i returned %i from fork...\n",
+                                     getpid(), _pid);
+            }
+
+            if(_pid == 0)
+            {
+                auto _prefix = std::stringstream{};
+                _prefix << std::setw(_width) << std::right << _n << "/"
+                        << std::setw(_width) << std::left << _causal_env.size() << ": ["
+                        << _main_pid << " -> " << getpid() << "] ";
+
+                auto _env = _base_env;
+                for(const auto& eitr : citr)
+                    update_env(_env, eitr.first, eitr.second);
+                print_updated_environment(_env, _prefix.str());
+                print_command(_argv, _prefix.str());
+                _argv.emplace_back(nullptr);
+                _env.emplace_back(nullptr);
+                return execvpe(_argv.front(), _argv.data(), _env.data());
+            }
+            else
+            {
+                add_child_pid(_pid);
+                auto _status = wait_pid(_pid);
+                auto _ret    = diagnose_status(_pid, _status);
+                remove_child_pid(_pid);
+                if(_ret != 0) return _ret;
+            }
+        }
+    }
+}
@@ -0,0 +1,93 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#define TIMEMORY_PROJECT_NAME "omnitrace-causal"
+
+#include <csignal>
+#include <map>
+#include <sched.h>
+#include <set>
+#include <string>
+#include <string_view>
+#include <vector>
+
+int
+get_verbose();
+
+std::string
+get_realpath(const std::string&);
+
+void
+print_command(const std::vector<char*>& _argv, std::string_view);
+
+void print_updated_environment(std::vector<char*>, std::string_view);
+
+std::vector<char*>
+get_initial_environment();
+
+void
+prepare_command_for_run(char*, std::vector<char*>&);
+
+void
+prepare_environment_for_run(std::vector<char*>&);
+
+std::string
+get_internal_libpath(const std::string& _lib);
+
+template <typename Tp>
+void
+update_env(std::vector<char*>&, std::string_view, Tp&&, bool _append = false,
+           std::string_view _join_delim = ":");
+
+template <typename Tp>
+void
+add_default_env(std::vector<char*>&, std::string_view, Tp&&);
+
+void
+remove_env(std::vector<char*>&, std::string_view);
+
+std::vector<char*>
+parse_args(int argc, char** argv, std::vector<char*>&,
+           std::vector<std::map<std::string_view, std::string>>&);
+
+using sigaction_t = struct sigaction;
+
+struct signal_handler
+{
+    sigaction_t m_custom_sigaction   = {};
+    sigaction_t m_original_sigaction = {};
+};
+
+void
+forward_signals(const std::set<int>&);
+
+void add_child_pid(pid_t);
+
+void remove_child_pid(pid_t);
+
+int
+wait_pid(pid_t, int = 0);
+
+int
+diagnose_status(pid_t, int);
@@ -104,7 +104,9 @@ get_initial_environment()
    auto* _omni_libpath =
        realpath(get_internal_libpath("libomnitrace.so").c_str(), nullptr);

-    update_env(_env, "OMNITRACE_USE_SAMPLING", true);
+    auto _mode = get_env<std::string>("OMNITRACE_MODE", "sampling", false);
+
+    update_env(_env, "OMNITRACE_USE_SAMPLING", (_mode != "causal"));
    update_env(_env, "OMNITRACE_CRITICAL_TRACE", false);
    update_env(_env, "OMNITRACE_USE_PROCESS_SAMPLING", false);

@@ -147,6 +149,8 @@ get_internal_libpath(const std::string& _lib)
 void
 print_updated_environment(std::vector<char*> _env)
 {
+    if(get_env<int>("OMNITRACE_VERBOSE", 0) < 0) return;
+
    std::sort(_env.begin(), _env.end(), [](auto* _lhs, auto* _rhs) {
        if(!_lhs) return false;
        if(!_rhs) return true;
@@ -335,6 +339,8 @@ parse_args(int argc, char** argv, std::vector<char*>& _env)
    };

    parser.enable_help();
+    parser.enable_version("omnitrace-sample", "v" OMNITRACE_VERSION_STRING,
+                          OMNITRACE_GIT_DESCRIBE, OMNITRACE_GIT_REVISION);

    auto _cols = std::get<0>(tim::utility::console::get_columns());
    if(_cols > parser.get_help_width() + 8)
@@ -36,7 +36,8 @@ main(int argc, char** argv)
    for(int i = 1; i < argc; ++i)
    {
        auto _arg = std::string_view{ argv[i] };
-        if(_arg == "--" || _arg == "-?" || _arg == "-h" || _arg == "--help")
+        if(_arg == "--" || _arg == "-?" || _arg == "-h" || _arg == "--help" ||
+           _arg == "--version")
            _has_double_hyphen = true;
    }

@@ -21,6 +21,7 @@
 // SOFTWARE.

 #include "omnitrace.hpp"
+#include "common/defines.h"
 #include "fwd.hpp"
 #include "log.hpp"

@@ -349,10 +350,6 @@ main(int argc, char** argv)
                  << std::endl;
    }

-    verbprintf(0, "\n");
-    verbprintf(0, "command :: '%s'...\n", cmd_string(_cmdc, _cmdv).c_str());
-    verbprintf(0, "\n");
-
    if(_cmdc > 0) cmdv0 = _cmdv[0];

    // now can loop through the options.  If the first character is '-', then we know
@@ -364,6 +361,9 @@ main(int argc, char** argv)
    string_t extra_help = "-- <CMD> <ARGS>";

    parser.enable_help();
+    parser.enable_version("omnitrace", "v" OMNITRACE_VERSION_STRING,
+                          OMNITRACE_GIT_DESCRIBE, OMNITRACE_GIT_REVISION);
+
    parser.add_argument({ "" }, "");
    parser.add_argument({ "[DEBUG OPTIONS]" }, "");
    parser.add_argument({ "" }, "");
@@ -875,6 +875,10 @@ main(int argc, char** argv)
        return 0;
    }

+    verbprintf(0, "\n");
+    verbprintf(0, "command :: '%s'...\n", cmd_string(_cmdc, _cmdv).c_str());
+    verbprintf(0, "\n");
+
    if(err)
    {
        std::cerr << err << std::endl;
@@ -296,6 +296,7 @@ omnitrace_add_bin_test(
         _P
         ~PERFETTO
         ~PROCESS_SAMPLING
+         ~KOKKOSP
         --csv
         --brief
         --advanced
@@ -8,11 +8,11 @@

 ## Overview

-> ***[Omnitrace](https://github.com/AMDResearch/omnitrace) is an AMD open source research project and is not supported as part of the ROCm software stack.***
+> ***[OmniTrace](https://github.com/AMDResearch/omnitrace) is an AMD open source research project and is not supported as part of the ROCm software stack.***

-[Browse Omnitrace source code on Github](https://github.com/AMDResearch/omnitrace)
+[Browse OmniTrace source code on Github](https://github.com/AMDResearch/omnitrace)

-[Omnitrace](https://github.com/AMDResearch/omnitrace) is designed for both high-level profiling and
+[OmniTrace](https://github.com/AMDResearch/omnitrace) is designed for both high-level profiling and
 comprehensive tracing of applications running on the CPU or the CPU+GPU via dynamic binary instrumentation,
 call-stack sampling, and various other means for determining currently executing function and line information.

@@ -24,17 +24,17 @@ The JSON output files are compatible with the python package [hatchet](https://g
 the performance data into pandas dataframes and facilitate multi-run comparisons, filtering, visualization in Jupyter notebooks,
 and much more.

-[Omnitrace](https://github.com/AMDResearch/omnitrace) has two distinct configuration steps when instrumenting:
+[OmniTrace](https://github.com/AMDResearch/omnitrace) has two distinct configuration steps when instrumenting:

 1. Configuring which functions and modules are instrumented in the target binaries (i.e. executable and/or libraries)
-   - [Instrumenting with Omnitrace](instrumenting.md)
+   - [Instrumenting with OmniTrace](instrumenting.md)
 2. Configuring what the instrumentation does happens when the instrumented binaries are executed
-   - [Customizing Omnitrace Runtime](runtime.md)
+   - [Customizing OmniTrace Runtime](runtime.md)

-## Omnitrace Use Cases
+## OmniTrace Use Cases

 When analyzing the performance of an application, ***it is always best to NOT assume you know where the performance bottlenecks are***
-***and why they are happening.*** Omnitrace is a ***tool for the entire execution of application***. It is the sort of tool which is
+***and why they are happening.*** OmniTrace is a ***tool for the entire execution of application***. It is the sort of tool which is
 ideal for *characterizing* where optimization would have the greatest impact on the end-to-end execution of the application and/or
 viewing what else is happening on the system during a performance bottleneck.

@@ -44,10 +44,10 @@ to 1 microsecond (1000x speed-up) but the original application *never spent time
 you will see zero statistically significant speed-up in end-to-end runtime of your application. In other words, it does not matter
 how fast or slow the code on GPU is if the application is not bottlenecked waiting on the GPU.

-Use Omnitrace to obtain a high-level view of the entire application. Use it to determine where the performance bottlenecks are and
+Use OmniTrace to obtain a high-level view of the entire application. Use it to determine where the performance bottlenecks are and
 obtain clues to why these bottlenecks are happening. If you want ***extensive*** insight into the execution of individual kernels
 on the GPU, AMD Research is working on another tool for this but you should start with the tool which characterizes the
-broad picture: Omnitrace.
+broad picture: OmniTrace.

-With regard to the CPU, Omnitrace does not target any specific vendor, it works just as well with non-AMD CPUs as with AMD CPUs.
-With regard to the GPU, Omnitrace is currently restricted to the HIP and HSA APIs and kernels executing on AMD GPUs.
+With regard to the CPU, OmniTrace does not target any specific vendor, it works just as well with non-AMD CPUs as with AMD CPUs.
+With regard to the GPU, OmniTrace is currently restricted to the HIP and HSA APIs and kernels executing on AMD GPUs.
@@ -0,0 +1,480 @@
+# Causal Profiling
+
+```eval_rst
+.. toctree::
+   :glob:
+   :maxdepth: 3
+```
+
+## What is "Causal Profiling"?
+
+> ***If you speed up a given block of code by X%, the application will execute Y% faster***
+
+Causal profiling directs parallel application developers to where they should focus their optimization
+efforts by quantifying the potential impact of optimizations. Causal profiling is rooted in the concept
+that *software execution speed is relative*: speeding up a block of code by X% is mathematically equivalent
+to that block of code running at its current speed if all the other code running slower by X%.
+Thus, causal profiling works by performing experiments on blocks of code during program execution which
+insert pauses to slow down all other concurrently running code. During post-processing, these experiments
+are translated into calculations for the potential impact of speeding up this block of code.
+
+Consider the following C++ code executing `foo` and `bar` concurrently in two different threads
+where `foo` is 30% faster than `bar` (ideally):
+
+```cpp
+constexpr size_t FOO_N =  7 * 1000000000UL;
+constexpr size_t BAR_N = 10 * 1000000000UL;
+
+void foo()
+{
+    for(volatile size_t i = 0; i < FOO_N; ++i) {}
+}
+
+void bar()
+{
+    for(volatile size_t i = 0; i < BAR_N; ++i) {}
+}
+
+int main()
+{
+    auto _threads = { std::thread{ foo },
+                      std::thread{ bar } };
+
+    for(auto& itr : _threads)
+        itr.join();
+}
+```
+
+No matter how many optimizations are applied to `foo`, the application will always require the same amount of time
+because the end-to-end performance is limited by `bar`. However, a 5% speedup in `bar` will result in the
+end-to-end performance improving by 5% and this trend will continue linearly (10% speedup in `bar` yields 10% speedup in
+end-to-end performance, and so on) up to 30% speedup, at which point, `bar` executes as fast as `foo`;
+any speedup to `bar` beyond 30% will still only yield an end-to-end performance speedup of 30% since the application
+will be limited by performance of `foo`, as demonstrated below in the causal profiling visualization:
+
+![foobar-causal-plot](images/causal-foobar.png)
+
+The full details of the causal profiling methodology can be found in the paper [Coz: Finding Code that Counts with Causal Profiling](http://arxiv.org/pdf/1608.03676v1.pdf).
+The author's implementation is publicly available on [GitHub](https://github.com/plasma-umass/coz).
+
+## Getting Started
+
+### Progress Points
+
+Causal profiling requires "progress points" to track progress through the code in between samples. Progress points must be triggered deterministically via instrumentation.
+This can happen in three different ways:
+
+1. OmniTrace can leverage the callbacks from Kokkos-Tools, OpenMP-Tools, roctracer, etc. and the wrappers around functions for MPI, NUMA, RCCL, etc. to act as progress-points
+2. User can leverage the [runtime instrumentation capabilities](instrumenting.md#runtime-instrumentation) to insert progress-points (NOTE: binary rewrite to insert progress-points is not supported)
+3. User can leverage the [User API](user_api.md), e.g. `OMNITRACE_CAUSAL_PROGRESS`
+
+Please note with regard to #2, binary rewrite to insert progress-points is not supported: when a rewritten binary is executed, Dyninst translates the instruction pointer address in order
+to execute the instrumentation and, as a result, call-stack samples never return instruction pointer addresses in the ranges defined as valid by OmniTrace. Hopefully, a work-around will
+be found in the future.
+
+### Key Concepts
+
+| Concept          | Setting                           | Options                          | Description                                                                                                        |
+|------------------|-----------------------------------|----------------------------------|--------------------------------------------------------------------------------------------------------------------|
+| Mode             | `OMNITRACE_CAUSAL_MODE`           | `function`, `line`               | Select entire function or individual line of code for causal experiments                                           |
+| End-to-End       | `OMNITRACE_CAUSAL_END_TO_END`     | boolean                          | Perform a single experiment during the entire run (does not require progress-points)                               |
+| Fixed speedup(s) | `OMNITRACE_CAUSAL_FIXED_SPEEDUP`  | one or more values from [0, 100] | Virtual speedup or pool of virtual speedups to randomly select                                                     |
+| Binary scope     | `OMNITRACE_CAUSAL_BINARY_SCOPE`   | regular expression(s)            | Dynamic binaries containing code for experiments                                                                   |
+| Source scope     | `OMNITRACE_CAUSAL_SOURCE_SCOPE`   | regular expression(s)            | `<file>` and/or `<file>:<line>` containing code to include in experiments                                          |
+| Function scope   | `OMNITRACE_CAUSAL_FUNCTION_SCOPE` | regular expression(s)            | Restricts experiments to matching functions (function mode) or lines of code within matching functions (line mode) |
+
+#### Notes
+
+1. Binary scope defaults to `%MAIN%` (executable). Scope can be expanded to include linked libraries
+2. `<file>` and `<file>:<line>` support requires debug info (i.e. code was compiled with `-g` or, preferably, `-g3`)
+3. Function mode does not require debug info but does not support stripped binaries
+
+### Speedup Prediction Variability and `omnitrace-causal` Executable
+
+Causal profiling typically require executing the application several times in order to adequately sample all the domains of executing code, experiment speedups, etc. and resolve statistical fluctuations.
+The `omnitrace-causal` executable is designed to simplify running this procedure:
+
+```console
+$ omnitrace-causal --help
+[omnitrace-causal] Usage: ./bin/omnitrace-causal [ --help (count: 0, dtype: bool)
+                                                   --version (count: 0, dtype: bool)
+                                                   --monochrome (max: 1, dtype: bool)
+                                                   --debug (max: 1, dtype: bool)
+                                                   --verbose (count: 1)
+                                                   --config (min: 0, dtype: filepath)
+                                                   --launcher (count: 1, dtype: executable)
+                                                   --generate-configs (min: 0, dtype: folder)
+                                                   --no-defaults (min: 0, dtype: bool)
+                                                   --mode (count: 1, dtype: string)
+                                                   --output-name (min: 1, dtype: filename)
+                                                   --reset (max: 1, dtype: bool)
+                                                   --end-to-end (max: 1, dtype: bool)
+                                                   --wait (count: 1, dtype: seconds)
+                                                   --duration (count: 1, dtype: seconds)
+                                                   --iterations (count: 1, dtype: int)
+                                                   --speedups (min: 0, dtype: integers)
+                                                   --binary-scope (min: 0, dtype: integers)
+                                                   --source-scope (min: 0, dtype: integers)
+                                                   --function-scope (min: 0, dtype: regex-list)
+                                                   --binary-exclude (min: 0, dtype: integers)
+                                                   --source-exclude (min: 0, dtype: integers)
+                                                   --function-exclude (min: 0, dtype: regex-list)
+                                                 ]
+
+    Causal profiling usually requires multiple runs to reliably resolve the speedup estimates.
+    This executable is designed to streamline that process.
+    For example (assume all commands end with '-- <exe> <args>'):
+
+        omnitrace-causal -n 5 -- <exe>                  # runs <exe> 5x with causal profiling enabled
+
+        omnitrace-causal -s 0 5,10,15,20                # runs <exe> 2x with virtual speedups:
+                                                        #   - 0
+                                                        #   - randomly selected from 5, 10, 15, and 20
+
+        omnitrace-causal -F func_A func_B func_(A|B)    # runs <exe> 3x with the function scope limited to:
+                                                        #   1. func_A
+                                                        #   2. func_B
+                                                        #   3. func_A or func_B
+    General tips:
+    - Insert progress points at hotspots in your code or use omnitrace's runtime instrumentation
+        - Note: binary rewrite will produce a incompatible new binary
+    - Run omnitrace-causal in "function" mode first (does not require debug info)
+    - Run omnitrace-causal in "line" mode when you are targeting one function (requires debug info)
+        - Preferably, use predictions from the "function" mode to determine which function to target
+    - Limit the virtual speedups to a smaller pool, e.g., 0,5,10,25,50, to get reliable predictions quicker
+    - Make use of the binary, source, and function scope to limit the functions/lines selected for experiments
+        - Note: source scope requires debug info
+
+
+Options:
+    -h, -?, --help                 Shows this page
+    --version                      Prints the version and exit
+
+    [DEBUG OPTIONS]
+
+    --monochrome                   Disable colorized output
+    --debug                        Debug output
+    -v, --verbose                  Verbose output
+
+    [GENERAL OPTIONS]
+
+    -c, --config                   Base configuration file
+    -l, --launcher                 When running MPI jobs, omnitrace-causal needs to be *before* the executable which launches the MPI processes (i.e.
+                                   before `mpirun`, `srun`, etc.). Pass the name of the target executable (or a regex for matching to the name of the
+                                   target) for causal profiling, e.g., `omnitrace-causal -l foo -- mpirun -n 4 foo`. This ensures that the omnitrace
+                                   library is LD_PRELOADed on the proper target
+    -g, --generate-configs         Generate config files instead of passing environment variables directly. If no arguments are provided, the config files
+                                   will be placed in ${PWD}/omnitrace-causal-config folder
+    --no-defaults                  Do not activate default features which are recommended for causal profiling. For example: PID-tagging of output files
+                                   and timestamped subdirectories are disabled by default. Kokkos tools support is added by default
+                                   (OMNITRACE_USE_KOKKOSP=ON) because, for Kokkos applications, the Kokkos-Tools callbacks are used for progress points.
+                                   Activation of OpenMP tools support is similar
+
+    [CAUSAL PROFILING OPTIONS (General)]
+                                   (These settings will be applied to all causal profiling runs)
+
+    -m, --mode [ function (func) | line ]
+                                   Causal profiling mode
+    -o, --output-name              Output filename of causal profiling data w/o extension
+    -r, --reset                    Overwrite any existing experiment results during the first run
+    -e, --end-to-end               Single causal experiment for the entire application runtime
+    -w, --wait                     Set the wait time (i.e. delay) before starting the first causal experiment (in seconds)
+    -d, --duration                 Set the length of time (in seconds) to perform causal experimentationafter the first experiment is started. Once this
+                                   amount of time has elapsed, no more causal experiments will be started but any currently running experiment will be
+                                   allowed to finish.
+    -n, --iterations               Number of times to repeat the combination of run configurations
+
+    [CAUSAL PROFILING OPTIONS (Combinatorial)]
+                                   (Each individual argument to these options will multiply the number runs by the number of arguments and the number of
+                                   iterations. E.g. -n 2 -B "MAIN" -F "foo" "bar" will produce 4 runs: 2 iterations x 1 binary scope x 2 function scopes
+                                   (MAIN+foo, MAIN+bar, MAIN+foo, MAIN+bar))
+
+    -s, --speedups                 Pool of virtual speedups to sample from during experimentation. Each space designates a group and multiple speedups can
+                                   be grouped together by commas, e.g. -s 0 0,10,20-50 is two groups: group #1 is '0' and group #2 is '0 10 20 25 30 35 40
+                                   45 50'
+    -B, --binary-scope             Restricts causal experiments to the binaries matching the list of regular expressions. Each space designates a group
+                                   and multiple scopes can be grouped together with a semi-colon
+    -S, --source-scope             Restricts causal experiments to the source files or source file + lineno pairs (i.e. <file> or <file>:<line>) matching
+                                   the list of regular expressions. Each space designates a group and multiple scopes can be grouped together with a
+                                   semi-colon
+    -F, --function-scope           Restricts causal experiments to the functions matching the list of regular expressions. Each space designates a group
+                                   and multiple scopes can be grouped together with a semi-colon
+    -BE, --binary-exclude          Excludes causal experiments from being performed on the binaries matching the list of regular expressions. Each space
+                                   designates a group and multiple excludes can be grouped together with a semi-colon
+    -SE, --source-exclude          Excludes causal experiments from being performed on the code from the source files or source file + lineno pair (i.e.
+                                   <file> or <file>:<line>) matching the list of regular expressions. Each space designates a group and multiple excludes
+                                   can be grouped together with a semi-colon
+    -FE, --function-exclude        Excludes causal experiments from being performed on the functions matching the list of regular expressions. Each space
+                                   designates a group and multiple excludes can be grouped together with a semi-colon
+```
+
+#### Examples
+
+```bash
+#!/bin/bash -e
+
+module load omnitrace
+
+N=20
+I=3
+
+# when providing speedups to omnitrace-causal, speedup
+# groups are separated by a space so "0,10" results in
+# one speedup group where omnitrace samples from
+# the speedup set of {0, 10}. Passing "0 10" (without
+# quotes to omnitrace-causal multiplies the
+# number of runs by 2, where the first half of the
+# runs instruct omnitrace to only use 0 as the
+# speedup and the second half of the runs instruct
+# omnitrace to only use 10 as the speedup.
+SPEEDUPS="0,0,0,10,20,30,40,50,50,75,75,75,90,90,90"
+# thus, -s ${SPEEDUPS} only multiplies the number
+# of runs by 1 whereas -S ${SPEEDUPS_E2E} multiplies
+# the number of runs by 15:
+#   - 3 runs with speedup of 0
+#   - 1 run for each of the speedups 10, 20, 30, and 40
+#   - 2 runs with speedup of 50
+#   - 3 runs with speedup of 75
+#   - 3 runs with speedup of 90
+SPEEDUPS_E2E=$(echo "${SPEEDUPS}" | sed 's/,/ /g')
+
+
+# 20 iterations in function mode with 1 speedup group
+# and source scope set to .cpp files
+#
+# outputs to files:
+#   - causal/experiments.func.coz
+#   - causal/experiments.func.json
+#
+# total executions: 20
+#
+omnitrace-causal        \
+    -n ${N}             \
+    -s ${SPEEDUPS}      \
+    -m function         \
+    -o experiments.func \
+    -S ".*\\.cpp"       \
+    --                  \
+    ./causal-omni-cpu "${@}"
+
+
+# 20 iterations in line mode with 1 speedup group
+# and source scope restricted to lines 155 and 165
+# in the causal.cpp file.
+#
+# outputs to files:
+#   - causal/experiments.line.coz
+#   - causal/experiments.line.json
+#
+# total executions: 20
+#
+omnitrace-causal                \
+    -n ${N}                     \
+    -s ${SPEEDUPS}              \
+    -m line                     \
+    -o experiments.line         \
+    -S "causal\\.cpp:(155|165)" \
+    --                          \
+    ./causal-omni-cpu "${@}"
+
+
+# 3 iterations in function mode of 15 singular speedups
+# in end-to-end mode with 2 different function scopes
+# where one is restricted to "cpu_slow_func" and
+# another is restricted to "cpu_fast_func".
+#
+# outputs to files:
+#   - causal/experiments.func.e2e.coz
+#   - causal/experiments.func.e2e.json
+#
+# total executions: 90
+#
+omnitrace-causal            \
+    -n ${I}                 \
+    -s ${SPEEDUPS_E2E}      \
+    -m func                 \
+    -e                      \
+    -o experiments.func.e2e \
+    -F "cpu_slow_func"      \
+       "cpu_fast_func"      \
+    --                      \
+    ./causal-omni-cpu "${@}"
+
+# 3 iterations in line mode of 15 singular speedups
+# in end-to-end mode with 2 different source scopes
+# where one is restricted to line 155 in causal.cpp
+# and another is restricted to line 165 in causal.cpp.
+#
+# outputs to files:
+#   - causal/experiments.line.e2e.coz
+#   - causal/experiments.line.e2e.json
+#
+# total executions: 90
+#
+omnitrace-causal            \
+    -n ${I}                 \
+    -s ${SPEEDUPS_E2E}      \
+    -m line                 \
+    -e                      \
+    -o experiments.line.e2e \
+    -S "causal\\.cpp:155"   \
+       "causal\\.cpp:165"   \
+    --                      \
+    ./causal-omni-cpu "${@}"
+
+
+export OMP_NUM_THREADS=8
+export OMP_PROC_BIND=spread
+export OMP_PLACES=threads
+
+# set number of iterations to 5
+N=5
+
+# 5 iterations in function mode of 1 speedup
+# group with the source scope restricted
+# to files containing "lulesh" in their filename
+# and exclude functions which start with "Kokkos::"
+# or "std::enable_if".
+#
+# outputs to files:
+#   - causal/experiments.func.coz
+#   - causal/experiments.func.json
+#
+# total executions: 5
+#
+# First of 5 executions overwrites any
+# existing causal/experiments.func.(coz|json)
+# file due to "--reset" argument
+#
+omnitrace-causal                            \
+    --reset                                 \
+    -n ${N}                                 \
+    -s ${SPEEDUPS}                          \
+    -m func                                 \
+    -o experiments.func                     \
+    -S "lulesh.*"                           \
+    -FE "^(Kokkos::|std::enable_if)"        \
+    --                                      \
+    ./lulesh-omni -i 50 -s 200 -r 20 -b 5 -c 5 -p
+
+
+# 5 iterations in line mode of 1 speedup
+# group with the source scope restricted
+# to files containing "lulesh" in their filename
+# and exclude functions which start with "exec_range"
+# or "execute" and which contain either
+# "construct_shared_allocation" or "._omp_fn." in
+# the function name.
+#
+# outputs to files:
+#   - causal/experiments.line.coz
+#   - causal/experiments.line.json
+#
+# total executions: 5
+#
+# First of 5 executions overwrites any
+# existing causal/experiments.line.(coz|json)
+# file due to "--reset" argument
+#
+omnitrace-causal                            \
+    --reset                                 \
+    -n ${N}                                 \
+    -s ${SPEEDUPS}                          \
+    -m line                                 \
+    -o experiments.line                     \
+    -S "lulesh.*"                           \
+    -FE "^(exec_range|execute);construct_shared_allocation;\\._omp_fn\\." \
+    --                                      \
+    ./lulesh-omni -i 50 -s 200 -r 20 -b 5 -c 5 -p
+
+
+# 5 iterations in line mode of 1 speedup
+# group with the source scope restricted
+# to files whose basename is "lulesh.cc"
+# for 3 different functions:
+#   - ApplyMaterialPropertiesForElems
+#   - CalcHourglassControlForElems
+#   - CalcVolumeForceForElems
+#
+# outputs to files:
+#   - causal/experiments.line.targeted.coz
+#   - causal/experiments.line.targeted.json
+#
+# total executions: 15
+#
+# First of 5 executions overwrites any
+# existing causal/experiments.line.(coz|json)
+# file due to "--reset" argument
+#
+omnitrace-causal                            \
+    --reset                                 \
+    -n ${N}                                 \
+    -s ${SPEEDUPS}                          \
+    -m line                                 \
+    -o experiments.line.targeted            \
+    -F "ApplyMaterialPropertiesForElems"    \
+       "CalcHourglassControlForElems"       \
+       "CalcVolumeForceForElems"            \
+    -S "lulesh\\.cc"                        \
+    --                                      \
+    ./lulesh-omni -i 50 -s 200 -r 20 -b 5 -c 5 -p
+```
+
+#### Using `omnitrace-causal` with other launchers (e.g. `mpirun`)
+
+The `omnitrace-causal` executable is intended to assist with application replay and is designed to always be at the start of the command-line (i.e. the primary process).
+`omnitrace-causal` typically adds a `LD_PRELOAD` of the OmniTrace libraries into the environment before launching the command in order to inject the functionality
+required to start the causal profiling tooling. However, this is problematic when the target application for causal profiling requires another command-line
+tool in order to run, e.g. `foo` is the target application but executing `foo` requires `mpirun -n 2 foo`. If one were to simply do `omnitrace-causal -- mpirun -n 2 foo`,
+then the causal profiling would be applied to `mpirun` instead of `foo`. `omnitrace-causal` remedies this by providing a command-line option `-l` / `--launcher`
+to indicate the target application is using a launcher script/executable. The argument to the command-line option is the name of (or regex for) the target application
+on the command-line. When `--launcher` is used, `omnitrace-causal` will generate all the replay configurations and execute them but delay adding the `LD_PRELOAD`, instead it
+will inject a call to itself into the command-line right before the target application. This recursive call to itself will inherit the configuration from
+parent `omnitrace-causal` executable, insert an `LD_PRELOAD` into the environment, and then invoke an `execv` to replace itself with the new process launched by the target
+application.
+
+In other words, the following command:
+
+```console
+omnitrace-causal -l foo -n 3 -- mpirun -n 2 foo`
+```
+
+Effectively results in:
+
+```console
+mpirun -n 2 omnitrace-causal -- foo
+mpirun -n 2 omnitrace-causal -- foo
+mpirun -n 2 omnitrace-causal -- foo
+```
+
+### Visualizing the Causal Output
+
+OmniTrace generates a `causal/experiments.json` and `causal/experiments.coz` in `${OMNITRACE_OUTPUT_PATH}/${OMNITRACE_OUTPUT_PREFIX}`. A standalone GUI for viewing the causal profiling
+results in under development but until this is available, visit [plasma-umass.org/coz/](https://plasma-umass.org/coz/) and open the `*.coz` file.
+
+## OmniTrace vs. Coz
+
+This section is intended for readers who are familiar with the [Coz profiler](https://github.com/plasma-umass/coz).
+OmniTrace provides several additional features and utilities for causal profiling:
+
+|                      |             [Coz](https://github.com/plasma-umass/coz)              |   [OmniTrace](https://github.com/AMDResearch/omnitrace)    | Notes                         |
+|----------------------|:-------------------------------------------------------------------:|:----------------------------------------------------------:|-------------------------------|
+| Debug info           |        requires debug info in DWARF v3 format (`-gdwarf-3`)         |        optional, supports any DWARF format version         | See Note #1 below             |
+| Experiment selection |                           `<file>:<line>`                           |              `<function>` or `<file>:<line>`               | See Note #2 below             |
+| Experiment speedups  | Randomly samples b/t 0..100 in increments of 5 or one fixed speedup |             Supports specifying smaller subset             | Set Note #3 below             |
+| Scope options        |                  Supports binary and source scopes                  |        Supports binary, source, and function scopes        | See Note #4, #5, and #6 below |
+| Scope inclusion      |          Uses `%` as wildcard for binary and source scopes          | Full regex support for binary, source, and function scopes |                               |
+| Scope exclusion      |                            Not supported                            |   Supports regexes for excluding binary/source/function    | See Note #7 below             |
+| Call-stack sampling  |                             Linux perf                              |                         libunwind                          | See Note #8 below             |
+
+### Notes
+
+1. OmniTrace supports a "function" mode which does not require debug info
+2. OmniTrace supports selecting entire range of instruction pointers for a function instead of instruction pointer for one line. In large codes, "function" mode
+   can resolve in fewer iterations and once a target function is identified, one can switch to line mode and limit the function scope to the target function
+3. OmniTrace supports randomly sampling from subsets, e.g. { 0, 0, 5, 10 } where 0% is randomly selected 50% of time and 5% and 10% are randomly selected 25% of the time
+4. OmniTrace and COZ have same definition for binary scope: the binaries loaded at runtime (e.g. executable and linked libraries)
+5. OmniTrace "source scope" supports both `<file>` and `<file>:<line>` formats in contrast to COZ "source scope" which requires `<file>:<line>` format
+6. OmniTrace supports a "function" scope which narrows the functions/lines which are eligible for causal experiments to those within the matching functions
+7. OmniTrace supports a second filter on scopes for removing binary/source/function caught by inclusive match, e.g. `BINARY_SCOPE=.*` + `BINARY_EXCLUDE=libmpi.*`
+   initially includes all binaries but exclude regex removes MPI libraries
@@ -137,6 +137,7 @@ breathe_projects_source = {
            "omnitrace/types.h",
            "omnitrace/categories.h",
            "omnitrace/user.h",
+            "omnitrace/causal.h",
        ],
    )
 }
@@ -8,7 +8,7 @@

 ## Overview

-[Omnitrace](https://github.com/AMDResearch/omnitrace) is designed to be highly extensible. Internally, it leverages the
+[OmniTrace](https://github.com/AMDResearch/omnitrace) is designed to be highly extensible. Internally, it leverages the
 [timemory performance analysis toolkit](https://github.com/NERSC/timemory) to
 manage extensions, resources, data, etc.

@@ -23,6 +23,8 @@ manage extensions, resources, data, etc.
  - Periodic software interrupts per-thread
 - Process-level sampling
  - Background thread records process-, system- and device-level metrics while the application executes
+- Causal profiling
+  - Quantifies the potential impact of optimizations in parallel codes
 - Critical trace generation

 ### Data Analysis
@@ -32,6 +34,7 @@ manage extensions, resources, data, etc.
  - Ideal for running at scale
 - Comprehensive traces
  - Every individual event/measurement
+- Application speedup predictions resulting from potential optimizations in functions and lines of code (causal profiling)
 - Critical trace analysis (alpha)

 ### Parallelism API Support
@@ -101,7 +101,7 @@ e.g., omnitrace's meaning of the term "module" when instrumenting Python.

 ## Data Collection Mode(s)

-Omnitrace supports several modes of recording trace and profiling data for your application:
+OmniTrace supports several modes of recording trace and profiling data for your application:

 | Mode                        | Descriptions                                                                                                                                                  |
 |-----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -1,4 +1,4 @@
-# Welcome to the [Omnitrace](https://github.com/AMDResearch/omnitrace) Documentation!
+# Welcome to the [OmniTrace](https://github.com/AMDResearch/omnitrace) Documentation!

 ```eval_rst
 .. toctree::
@@ -14,6 +14,7 @@
   runtime
   sampling
   instrumenting
+   causal_profiling
   critical_trace
   output
   user_api
@@ -8,7 +8,7 @@

 ## Operating System

-Omnitrace is only supported on Linux.
+OmniTrace is only supported on Linux.

 - Ubuntu 18.04
 - Ubuntu 20.04
@@ -76,16 +76,16 @@ mkdir /opt/omnitrace
 ./omnitrace-1.0.0-ubuntu-18.04-ROCm-405000-OMPT-PAPI.sh --prefix=/opt/omnitrace --exclude-subdir
 ```

-## Installing Omnitrace from source
+## Installing OmniTrace from source

 ### Build Requirements

-Omnitrace needs a GCC compiler with full support for C++17 and CMake v3.16 or higher.
+OmniTrace needs a GCC compiler with full support for C++17 and CMake v3.16 or higher.
 The Clang compiler may be used in lieu of the GCC compiler if Dyninst is already installed.

 - GCC compiler v7+
  - Older GCC compilers may be supported but are not tested
-  - Clang compilers are generally supported for [Omnitrace](https://github.com/AMDResearch/omnitrace) but not Dyninst
+  - Clang compilers are generally supported for [OmniTrace](https://github.com/AMDResearch/omnitrace) but not Dyninst
 - [CMake](https://cmake.org/) v3.16+

 > ***If the system installed cmake is too old, installing a new version of cmake can be done through several methods.***
@@ -114,8 +114,8 @@ and Dyninst requires TBB), and the CMake option to build the package alongside o

 | Third-Party Library | Minimum Version | Required By | CMake Option                              |
 |---------------------|-----------------|-------------|-------------------------------------------|
-| Dyninst             | 10.0            | Omnitrace   | `OMNITRACE_BUILD_DYNINST` (default: OFF)  |
-| Libunwind           |                 | Omnitrace   | `OMNITRACE_BUILD_LIBUNWIND` (default: ON) |
+| Dyninst             | 10.0            | OmniTrace   | `OMNITRACE_BUILD_DYNINST` (default: OFF)  |
+| Libunwind           |                 | OmniTrace   | `OMNITRACE_BUILD_LIBUNWIND` (default: ON) |
 | TBB                 | 2018.6          | Dyninst     | `DYNINST_BUILD_TBB` (default: OFF)        |
 | ElfUtils            | 0.178           | Dyninst     | `DYNINST_BUILD_ELFUTILS` (default: OFF)   |
 | LibIberty           |                 | Dyninst     | `DYNINST_BUILD_LIBIBERTY` (default: OFF)  |
@@ -144,7 +144,7 @@ and Dyninst requires TBB), and the CMake option to build the package alongside o

 ### Installing DynInst

-#### Building Dyninst alongside Omnitrace
+#### Building Dyninst alongside OmniTrace

 The easiest way to install Dyninst is to configure omnitrace with `OMNITRACE_BUILD_DYNINST=ON`. Depending on the version of Ubuntu, the apt package manager may have current enough
 versions of Dyninst's Boost, TBB, and LibIberty dependencies (i.e. `apt-get install libtbb-dev libiberty-dev libboost-dev`); however, it is possible to request Dyninst to install
@@ -173,7 +173,7 @@ spack load -r dyninst

 ### Installing omnitrace

-Omnitrace has cmake configuration options for supporting MPI (`OMNITRACE_USE_MPI` or `OMNITRACE_USE_MPI_HEADERS`), HIP kernel tracing (`OMNITRACE_USE_ROCTRACER`),
+OmniTrace has cmake configuration options for supporting MPI (`OMNITRACE_USE_MPI` or `OMNITRACE_USE_MPI_HEADERS`), HIP kernel tracing (`OMNITRACE_USE_ROCTRACER`),
 sampling ROCm devices (`OMNITRACE_USE_ROCM_SMI`), OpenMP-Tools (`OMNITRACE_USE_OMPT`), hardware counters via PAPI (`OMNITRACE_USE_PAPI`), among others.
 Various additional features can be enabled via the [`TIMEMORY_USE_*` CMake options](https://timemory.readthedocs.io/en/develop/installation.html#cmake-options).
 Any `OMNITRACE_USE_<VAL>` option which has a corresponding `TIMEMORY_USE_<VAL>` option means that the support within timemory for this feature has been integrated
@@ -204,9 +204,9 @@ cmake --build omnitrace-build --target install
 source /opt/omnitrace/share/omnitrace/setup-env.sh
 ```

-#### MPI Support within Omnitrace
+#### MPI Support within OmniTrace

-[Omnitrace](https://github.com/AMDResearch/omnitrace) can have full (`OMNITRACE_USE_MPI=ON`) or partial (`OMNITRACE_USE_MPI_HEADERS=ON`) MPI support.
+[OmniTrace](https://github.com/AMDResearch/omnitrace) can have full (`OMNITRACE_USE_MPI=ON`) or partial (`OMNITRACE_USE_MPI_HEADERS=ON`) MPI support.
 The only difference between these two modes is whether or not the results collected via timemory and/or perfetto can be aggregated into a single
 output file during finalization. When full MPI support is enabled, combining the timemory results always occurs whereas combining the perfetto
 results is configurable via the `OMNITRACE_PERFETTO_COMBINE_TRACES` setting.
@@ -311,7 +311,7 @@ omnitrace --simulate -o foo.inst -- foo

 ### Excluding and Including Modules and Functions

-[Omnitrace](https://github.com/AMDResearch/omnitrace) has a set of 6 command-line options which each accept one or more regular expressions for customizing the scope of which module and/or functions are
+[OmniTrace](https://github.com/AMDResearch/omnitrace) has a set of 6 command-line options which each accept one or more regular expressions for customizing the scope of which module and/or functions are
 instrumented. Multiple regexes per option are treated as an OR operation, e.g. `--module-include libfoo libbar` is effectively that same as `--module-include 'libfoo|libbar'`.

 If you would like to force the inclusion of certain modules and/or function without changing any of the heuristics, use the `--module-include` and/or `--function-include` options.
@@ -616,7 +616,7 @@ background system-level thread sampling by default.
 Tracing capabilities which do not rely on instrumentation, such as the HIP API and kernel tracing
 (which is collected via roctracer), will still be available.

-[Omnitrace](https://github.com/AMDResearch/omnitrace)'s sampling capabilities are always available, even in trace mode, but is deactivated by default.
+[OmniTrace](https://github.com/AMDResearch/omnitrace)'s sampling capabilities are always available, even in trace mode, but is deactivated by default.
 In order to activate sampling in trace mode, simply set `OMNITRACE_USE_SAMPLING=ON` in the environment
 or in an omnitrace configuration file.

@@ -116,7 +116,8 @@ WARN_LOGFILE           = doc/warnings.log
 INPUT                  = @SOURCE_DIR@/README.md \
                         @SOURCE_DIR@/source/lib/omnitrace-user/omnitrace/types.h \
                         @SOURCE_DIR@/source/lib/omnitrace-user/omnitrace/categories.h \
-                         @SOURCE_DIR@/source/lib/omnitrace-user/omnitrace/user.h
+                         @SOURCE_DIR@/source/lib/omnitrace-user/omnitrace/user.h \
+                         @SOURCE_DIR@/source/lib/omnitrace-user/omnitrace/causal.h
 INPUT_ENCODING         = UTF-8
 FILE_PATTERNS          = *.h \
                         *.hh \
@@ -1,4 +1,4 @@
-# Omnitrace Output
+# OmniTrace Output

 ```eval_rst
 .. toctree::
@@ -55,7 +55,7 @@ $ omnitrace -- ./foo

 ## Metadata

-[Omnitrace](https://github.com/AMDResearch/omnitrace) will output a metadata.json file. This metadata file will contain
+[OmniTrace](https://github.com/AMDResearch/omnitrace) will output a metadata.json file. This metadata file will contain
 information about the settings, environment variables, output files, and info about the system and the run:

 - Hardware cache sizes
@@ -237,7 +237,7 @@ information about the settings, environment variables, output files, and info ab

 ### Core Configuration Settings

-> ***See also: [Customizing Omnitrace Runtime](runtime.md)***
+> ***See also: [Customizing OmniTrace Runtime](runtime.md)***

 | Setting                   | Value              | Description                                                                                       |
 |---------------------------|--------------------|---------------------------------------------------------------------------------------------------|
@@ -6,7 +6,7 @@
   :maxdepth: 3
 ```

-[Omnitrace](https://github.com/AMDResearch/omnitrace) supports profiling Python code at the source-level and/or the script-level.
+[OmniTrace](https://github.com/AMDResearch/omnitrace) supports profiling Python code at the source-level and/or the script-level.
 Python support is enabled via the `OMNITRACE_USE_PYTHON` and the `OMNITRACE_PYTHON_VERSIONS="<MAJOR>.<MINOR>` CMake options.
 Alternatively, to build multiple python versions, use `OMNITRACE_PYTHON_VERSIONS="<MAJOR>.<MINOR>;[<MAJOR>.<MINOR>]"`,
 and `OMNITRACE_PYTHON_ROOT_DIRS="/path/to/version;[/path/to/version]"` instead of `OMNITRACE_PYTHON_VERSION`.
@@ -30,9 +30,9 @@ export PYTHONPATH=/opt/omnitrace/lib/python3.8/site-packages:${PYTHONPATH}
 If using either the `share/omnitrace/setup-env.sh` script or the modulefile in `share/modulefiles/omnitrace`, prefixing the `PYTHONPATH`
 environment variable is automatically handled.

-## Running Omnitrace on a Python Script
+## Running OmniTrace on a Python Script

-Omnitrace provides an `omnitrace-python` helper bash script which effectively handles ensuring `PYTHONPATH` is properly set and the correct python interpreter is used.
+OmniTrace provides an `omnitrace-python` helper bash script which effectively handles ensuring `PYTHONPATH` is properly set and the correct python interpreter is used.
 Thus the following are effectively equivalent:

 ```bash
@@ -57,7 +57,7 @@ optional arguments:
                        Logging verbosity
  -b, --builtin         Put 'profile' in the builtins. Use '@profile' to decorate a single function, or 'with profile:' to profile a single section of code.
  -c FILE, --config FILE
-                        Omnitrace configuration file
+                        OmniTrace configuration file
  -s FILE, --setup FILE
                        Code to execute before the code to profile
  -F [BOOL], --full-filepath [BOOL]
@@ -168,7 +168,7 @@ And executed with `omnitrace-python -b -- ./example.py`, omnitrace would produce
 |-----------------------------------------------------------|
 ```

-## Omnitrace Python Source Instrumentation
+## OmniTrace Python Source Instrumentation

 Starting from the unmodified `example.py` script above, we start by importing the `omnitrace` module:

@@ -232,7 +232,7 @@ The results for both of the source-level instrumentation modes are identical to
 > ***When `omnitrace-python` is used without built-ins, the profiling results will likely be cluttered by***
 > ***numerous functions called during the importing of more complex modules, e.g. `import numpy`.***

-### Omnitrace Python Source Instrumentation Configuration
+### OmniTrace Python Source Instrumentation Configuration

 Within the Python source code, the profiler can be configured by directly modifying the `omnitrace.profiler.config` data fields.

@@ -1,4 +1,4 @@
-# Configuring Omnitrace Runtime
+# Configuring OmniTrace Runtime

 ```eval_rst
 .. toctree::
@@ -49,7 +49,7 @@ match to nearly all common expressions for boolean logic: ON, OFF, YES, NO, TRUE

 ### Exploring Components

-[Omnitrace](https://github.com/AMDResearch/omnitrace) uses [timemory](https://github.com/NERSC/timemory) extensively to provide various capabilities and manage
+[OmniTrace](https://github.com/AMDResearch/omnitrace) uses [timemory](https://github.com/NERSC/timemory) extensively to provide various capabilities and manage
 data and resources. By default, when `OMNITRACE_USE_TIMEMORY=ON`, omnitrace will only collect wall-clock
 timing values; however, by modifying the `OMNITRACE_TIMEMORY_COMPONENTS` setting, omnitrace can be configured to
 collect hardware counters, CPU-clock timers, memory usage, context-switches, page-faults, network statistics,
@@ -72,7 +72,7 @@ omnitrace-avail --components --available --string --brief

 ### Exploring Hardware Counters

-[Omnitrace](https://github.com/AMDResearch/omnitrace) supports collecting hardware counters via PAPI and ROCm.
+[OmniTrace](https://github.com/AMDResearch/omnitrace) supports collecting hardware counters via PAPI and ROCm.
 Generally, PAPI is used to collect CPU-based hardware counters and ROCm is used to collect GPU-based hardware
 counters; although it is possible to install PAPI with ROCm support and collect GPU-based hardware counters
 via PAPI but this is not recommended because CPU hardware counters via PAPI cannot be collected simultaneously.
@@ -143,7 +143,7 @@ OMNITRACE_PAPI_EVENTS = PAPI_TOT_INS        perf::CACHE-REFERENCES  perf::CACHE-

 #### OMNITRACE_ROCM_EVENTS

-Omnitrace reads the ROCm events from the `${ROCM_PATH}/lib/rocprofiler/metrics.xml` file. Use the `ROCP_METRICS` environment
+OmniTrace reads the ROCm events from the `${ROCM_PATH}/lib/rocprofiler/metrics.xml` file. Use the `ROCP_METRICS` environment
 variable to point omnitrace to a different XML metrics file, e.g., `export ROCP_METRICS=${PWD}/custom_metrics.xml`.
 `omnitrace-avail -H -c GPU` will show event names with a suffix of `:device=N` where `N` is the device number.
 For example, if you have two devices, you will see:
@@ -201,7 +201,7 @@ OMNITRACE_CRITICAL_TRACE_SERIALIZE_NAMES           = false
 OMNITRACE_DEBUG                                    = false
 OMNITRACE_DL_VERBOSE                               = 0
 OMNITRACE_INSTRUMENTATION_INTERVAL                 = 1
-OMNITRACE_KOKKOS_KERNEL_LOGGER                     = false
+OMNITRACE_KOKKOSP_KERNEL_LOGGER                    = false
 OMNITRACE_PAPI_EVENTS                              = PAPI_TOT_CYC
 OMNITRACE_PERFETTO_BACKEND                         = inprocess
 OMNITRACE_PERFETTO_BUFFER_SIZE_KB                  = 1024000
@@ -302,7 +302,7 @@ $ omnitrace-avail -S -bd
 | OMNITRACE_INPUT_PREFIX                  | Explicitly specify the prefix for in... |
 | OMNITRACE_INSTRUMENTATION_INTERVAL      | Instrumentation only takes measureme... |
 | OMNITRACE_JSON_OUTPUT                   | Write json output files                 |
-| OMNITRACE_KOKKOS_KERNEL_LOGGER          | Enables kernel logging                  |
+| OMNITRACE_KOKKOSP_KERNEL_LOGGER         | Enables kernel logging                  |
 | OMNITRACE_MAX_DEPTH                     | Set the maximum depth of label hiera... |
 | OMNITRACE_MAX_THREAD_BOOKMARKS          | Maximum number of times a worker thr... |
 | OMNITRACE_MAX_WIDTH                     | Set the maximum width for component ... |
@@ -1161,7 +1161,7 @@ $ omnitrace-avail -H -bd

 ## Creating a Configuration File

-[Omnitrace](https://github.com/AMDResearch/omnitrace) supports 3 configuration file formats: JSON, XML, and plain text.
+[OmniTrace](https://github.com/AMDResearch/omnitrace) supports 3 configuration file formats: JSON, XML, and plain text.
 Use `omnitrace-avail -G <filename> -F txt json xml` to generate default configuration files of each format and, optionally,
 include the `--all` flag for descriptions, etc.
 Configuration files are specified via the `OMNITRACE_CONFIG_FILE` environment variable
@@ -192,7 +192,7 @@ are interpreted as belonging to omnitrace and all arguments following the double
 application and it's arguments. The double-hyphen is only necessary when passing command line arguments to the target
 which also use hyphens. E.g. `omnitrace-sample ls` works but, in order to run `ls -la`, use `omnitrace-sample -- ls -la`.

-[Configuring Omnitrace Runtime](runtime.md) establish the precedence of environment variable values over values specified in the configuration files. This enables
+[Configuring OmniTrace Runtime](runtime.md) establish the precedence of environment variable values over values specified in the configuration files. This enables
 the user to configure the omnitrace runtime to their preferred default behavior in a file such as `~/.omnitrace.cfg` and then easily override
 those settings via something like `OMNITRACE_ENABLED=OFF omnitrace-sample -- foo`.
 Similarly, the command line arguments passed to `omnitrace-sample` take precedence over environment variables.
@@ -4,6 +4,7 @@
 .. doxygenfile:: omnitrace/types.h
 .. doxygenfile:: omnitrace/categories.h
 .. doxygenfile:: omnitrace/user.h
+.. doxygenfile:: omnitrace/causal.h
 ```

 By default, when omnitrace detects any `omnitrace_user_start_*` or `omnitrace_user_stop_*` function, instrumentation
@@ -13,6 +14,26 @@ recorded, regardless of whether whether `omnitrace_user_start_*` or `omnitrace_u

 ## Example

+### Compilation
+
+#### CMake
+
+```cmake
+find_package(omnitrace REQUIRED COMPONENTS user)
+
+add_executable(foo foo.cpp)
+
+target_link_libraries(foo PRIVATE omnitrace::omnitrace-user-library)
+```
+
+#### General
+
+Assuming omnitrace installed in `/opt/omnitrace`:
+
+```bash
+g++ -I/opt/omnitrace foo.cpp -o foo -lomnitrace-user
+```
+
 ### User API Implementation

 ```cpp
@@ -71,6 +71,7 @@
 #define OMNITRACE_VISIBILITY(MODE)  OMNITRACE_ATTRIBUTE(visibility(MODE))
 #define OMNITRACE_PUBLIC_API        OMNITRACE_VISIBILITY("default")
 #define OMNITRACE_HIDDEN_API        OMNITRACE_VISIBILITY("hidden")
+#define OMNITRACE_INTERNAL_API      OMNITRACE_VISIBILITY("internal")
 #define OMNITRACE_INLINE            OMNITRACE_ATTRIBUTE(always_inline) inline
 #define OMNITRACE_NOINLINE          OMNITRACE_ATTRIBUTE(noinline)
 #define OMNITRACE_HOT               OMNITRACE_ATTRIBUTE(hot)
@@ -80,6 +81,8 @@
 #define OMNITRACE_WEAK              OMNITRACE_ATTRIBUTE(weak)
 #define OMNITRACE_PACKED            OMNITRACE_ATTRIBUTE(__packed__)
 #define OMNITRACE_PACKED_ALIGN(VAL) OMNITRACE_PACKED OMNITRACE_ATTRIBUTE(__aligned__(VAL))
+#define OMNITRACE_LIKELY(...)       __builtin_expect((__VA_ARGS__), 1)
+#define OMNITRACE_UNLIKELY(...)     __builtin_expect((__VA_ARGS__), 0)

 #if defined(OMNITRACE_CI) && OMNITRACE_CI > 0
 #    if defined(NDEBUG)
@@ -22,9 +22,13 @@

 #pragma once

+#include <array>
+#include <initializer_list>
 #include <ios>
 #include <sstream>
 #include <string>
+#include <string_view>
+#include <tuple>
 #include <type_traits>

 #if !defined(OMNITRACE_FOLD_EXPRESSION)
@@ -127,6 +131,42 @@ join(QuoteStrings&&, DelimT&& _delim, Args&&... _args)
        return (_ret.length() > _len) ? _ret.substr(_len) : std::string{};
    }
 }
+
+template <typename... Args>
+auto
+join(std::array<std::string_view, 3>&& _delim, Args&&... _args)
+{
+    return join("", std::get<0>(_delim),
+                join(std::get<1>(_delim), std::forward<Args>(_args)...),
+                std::get<2>(_delim));
+}
+
+template <typename... Args>
+auto
+join(QuoteStrings&&, std::array<std::string_view, 3>&& _delim, Args&&... _args)
+{
+    return join(QuoteStrings{}, "", std::get<0>(_delim),
+                join(std::get<1>(_delim), std::forward<Args>(_args)...),
+                std::get<2>(_delim));
+}
+
+template <typename DelimB, typename DelimT, typename DelimE, typename... Args>
+auto
+join(std::tuple<DelimB, DelimT, DelimE>&& _delim, Args&&... _args)
+{
+    return join("", std::get<0>(_delim),
+                join(std::get<1>(_delim), std::forward<Args>(_args)...),
+                std::get<2>(_delim));
+}
+
+template <typename DelimB, typename DelimT, typename DelimE, typename... Args>
+auto
+join(QuoteStrings&&, std::tuple<DelimB, DelimT, DelimE>&& _delim, Args&&... _args)
+{
+    return join(QuoteStrings{}, "", std::get<0>(_delim),
+                join(std::get<1>(_delim), std::forward<Args>(_args)...),
+                std::get<2>(_delim));
+}
 }  // namespace
 }  // namespace common
 }  // namespace omnitrace
@@ -266,6 +266,9 @@ struct OMNITRACE_HIDDEN_API indirect
                        "omnitrace_register_source");
        OMNITRACE_DLSYM(omnitrace_register_coverage_f, m_omnihandle,
                        "omnitrace_register_coverage");
+        OMNITRACE_DLSYM(omnitrace_progress_f, m_omnihandle, "omnitrace_progress");
+        OMNITRACE_DLSYM(omnitrace_annotated_progress_f, m_omnihandle,
+                        "omnitrace_annotated_progress");

        OMNITRACE_DLSYM(kokkosp_print_help_f, m_omnihandle, "kokkosp_print_help");
        OMNITRACE_DLSYM(kokkosp_parse_args_f, m_omnihandle, "kokkosp_parse_args");
@@ -343,8 +346,10 @@ struct OMNITRACE_HIDDEN_API indirect
            _cb.stop_thread_trace          = &omnitrace_user_stop_thread_trace_dl;
            _cb.push_region                = &omnitrace_user_push_region_dl;
            _cb.pop_region                 = &omnitrace_user_pop_region_dl;
+            _cb.progress                   = &omnitrace_user_progress_dl;
            _cb.push_annotated_region      = &omnitrace_user_push_annotated_region_dl;
            _cb.pop_annotated_region       = &omnitrace_user_pop_annotated_region_dl;
+            _cb.annotated_progress         = &omnitrace_user_annotated_progress_dl;
            (*omnitrace_user_configure_f)(OMNITRACE_USER_REPLACE_CONFIG, _cb, nullptr);
        }
    }
@@ -370,6 +375,9 @@ public:
                                            omnitrace_annotation_t*, size_t) = nullptr;
    int (*omnitrace_pop_category_region_f)(omnitrace_category_t, const char*,
                                           omnitrace_annotation_t*, size_t)  = nullptr;
+    void (*omnitrace_progress_f)(const char*)                                = nullptr;
+    void (*omnitrace_annotated_progress_f)(const char*, omnitrace_annotation_t*,
+                                           size_t)                           = nullptr;

    // libomnitrace-user functions
    int (*omnitrace_user_configure_f)(int, user_cb_t, user_cb_t*) = nullptr;
@@ -772,6 +780,12 @@ extern "C"
        return OMNITRACE_DL_INVOKE(get_indirect().omnitrace_pop_region_f, name);
    }

+    int omnitrace_user_progress_dl(const char* name)
+    {
+        OMNITRACE_DL_INVOKE(get_indirect().omnitrace_progress_f, name);
+        return 0;
+    }
+
    int omnitrace_user_push_annotated_region_dl(const char*             name,
                                                omnitrace_annotation_t* _annotations,
                                                size_t                  _annotation_count)
@@ -792,6 +806,28 @@ extern "C"
                                   _annotation_count);
    }

+    int omnitrace_user_annotated_progress_dl(const char*             name,
+                                             omnitrace_annotation_t* _annotations,
+                                             size_t                  _annotation_count)
+    {
+        OMNITRACE_DL_INVOKE(get_indirect().omnitrace_annotated_progress_f, name,
+                            _annotations, _annotation_count);
+        return 0;
+    }
+
+    void omnitrace_progress(const char* _name)
+    {
+        return OMNITRACE_DL_INVOKE(get_indirect().omnitrace_progress_f, _name);
+    }
+
+    void omnitrace_annotated_progress(const char*             _name,
+                                      omnitrace_annotation_t* _annotations,
+                                      size_t                  _annotation_count)
+    {
+        return OMNITRACE_DL_INVOKE(get_indirect().omnitrace_annotated_progress_f, _name,
+                                   _annotations, _annotation_count);
+    }
+
    //----------------------------------------------------------------------------------//
    //
    //      KokkosP
@@ -1033,11 +1069,13 @@ omnitrace_preload()
    {
        // reset_omnitrace_preload();
        omnitrace_preinit_library();
+        auto _causal = get_env("OMNITRACE_USE_CAUSAL", false);
+        auto _mode   = get_env("OMNITRACE_MODE", (_causal) ? "causal" : "sampling");
        OMNITRACE_DL_LOG(1, "[%s] invoking %s(%s)\n", __FUNCTION__, "omnitrace_init",
-                         ::omnitrace::join(::omnitrace::QuoteStrings{}, ", ", "sampling",
-                                           false, "main")
+                         ::omnitrace::join(::omnitrace::QuoteStrings{}, ", ", _mode,
+                                           false, "omnitrace")
                             .c_str());
-        omnitrace_init("sampling", false, "omnitrace");
+        omnitrace_init(_mode.c_str(), false, nullptr);
        omnitrace_init_tooling();
    }

@@ -92,6 +92,9 @@ extern "C"
                                   const char* source) OMNITRACE_PUBLIC_API;
    void omnitrace_register_coverage(const char* file, const char* func,
                                     size_t address) OMNITRACE_PUBLIC_API;
+    void omnitrace_progress(const char*) OMNITRACE_PUBLIC_API;
+    void omnitrace_annotated_progress(const char*, omnitrace_annotation_t*,
+                                      size_t) OMNITRACE_PUBLIC_API;

 #if defined(OMNITRACE_DL_SOURCE) && (OMNITRACE_DL_SOURCE > 0)
    void omnitrace_preinit_library(void) OMNITRACE_HIDDEN_API;
@@ -111,6 +114,9 @@ extern "C"
    int omnitrace_user_pop_annotated_region_dl(const char*, omnitrace_annotation_t*,
                                               size_t) OMNITRACE_HIDDEN_API;

+    int omnitrace_user_progress_dl(const char* name) OMNITRACE_HIDDEN_API;
+    int omnitrace_user_annotated_progress_dl(const char*, omnitrace_annotation_t*,
+                                             size_t) OMNITRACE_HIDDEN_API;
    // KokkosP
    struct OMNITRACE_HIDDEN_API SpaceHandle
    {
@@ -97,11 +97,11 @@ omnitrace_main(int argc, char** argv, char** envp)
    const char* mode = getenv("OMNITRACE_MODE");
    omnitrace_init(mode ? mode : "sampling", false, argv[0]);
    omnitrace_init_tooling();
-    omnitrace_push_trace("main(int argc, char** argv)");
+    omnitrace_push_trace(basename(argv[0]));

    int ret = main_real(argc, argv, envp);

-    omnitrace_pop_trace("main(int argc, char** argv)");
+    omnitrace_pop_trace(basename(argv[0]));
    omnitrace_finalize();

    return ret;
@@ -17,6 +17,7 @@ target_sources(
    omnitrace-user-library
    PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/user.cpp
            ${CMAKE_CURRENT_SOURCE_DIR}/omnitrace/user.h
+            ${CMAKE_CURRENT_SOURCE_DIR}/omnitrace/causal.h
            ${CMAKE_CURRENT_SOURCE_DIR}/omnitrace/types.h)
 target_include_directories(
    omnitrace-user-library PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>
@@ -38,6 +39,7 @@ omnitrace_strip_target(omnitrace-user-library)
 install(
    FILES ${CMAKE_CURRENT_SOURCE_DIR}/omnitrace/user.h
          ${CMAKE_CURRENT_SOURCE_DIR}/omnitrace/types.h
+          ${CMAKE_CURRENT_SOURCE_DIR}/omnitrace/causal.h
          ${CMAKE_CURRENT_SOURCE_DIR}/omnitrace/categories.h
    DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/omnitrace)

@@ -0,0 +1,86 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+/** @file causal.h */
+
+#ifndef OMNITRACE_CAUSAL_H_
+#define OMNITRACE_CAUSAL_H_
+
+/**
+ * @defgroup OMNITRACE_CASUAL_GROUP OmniTrace Causal Profiling Defines
+ *
+ * @{
+ */
+
+#if !defined(OMNITRACE_CAUSAL_ENABLED)
+/** Preprocessor switch to enable/disable instrumentation for causal profiling */
+#    define OMNITRACE_CAUSAL_ENABLED 1
+#endif
+
+#if OMNITRACE_CAUSAL_ENABLED > 0
+#    include <omnitrace/user.h>
+
+#    if !defined(OMNITRACE_CAUSAL_LABEL)
+/** @cond OMNITRACE_HIDDEN_DEFINES */
+#        define OMNITRACE_CAUSAL_STR2(x) #        x
+#        define OMNITRACE_CAUSAL_STR(x)  OMNITRACE_CAUSAL_STR2(x)
+/** @endcond */
+/** Default label for a causal progress point */
+#        define OMNITRACE_CAUSAL_LABEL __FILE__ ":" OMNITRACE_CAUSAL_STR(__LINE__)
+#    endif
+#    if !defined(OMNITRACE_CAUSAL_PROGRESS)
+/** Adds a throughput progress point with label `<file>:<line>` */
+#        define OMNITRACE_CAUSAL_PROGRESS omnitrace_user_progress(OMNITRACE_CAUSAL_LABEL);
+#    endif
+#    if !defined(OMNITRACE_CAUSAL_PROGRESS_NAMED)
+/** Adds a throughput progress point with user defined label. Each instance should use a
+ * unique label. */
+#        define OMNITRACE_CAUSAL_PROGRESS_NAMED(LABEL) omnitrace_user_progress(LABEL);
+#    endif
+#    if !defined(OMNITRACE_CAUSAL_BEGIN)
+/** Starts a latency progress point (region of interest) with user defined label. Each
+ * instance should use a unique label. */
+#        define OMNITRACE_CAUSAL_BEGIN(LABEL) omnitrace_user_push_region(LABEL);
+#    endif
+#    if !defined(OMNITRACE_CAUSAL_END)
+/** End the latency progress point (region of interest) for the matching user defined
+ * label. */
+#        define OMNITRACE_CAUSAL_END(LABEL) omnitrace_user_pop_region(LABEL);
+#    endif
+#else
+#    if !defined(OMNITRACE_CAUSAL_PROGRESS)
+#        define OMNITRACE_CAUSAL_PROGRESS
+#    endif
+#    if !defined(OMNITRACE_CAUSAL_PROGRESS_NAMED)
+#        define OMNITRACE_CAUSAL_PROGRESS_NAMED(LABEL)
+#    endif
+#    if !defined(OMNITRACE_CAUSAL_BEGIN)
+#        define OMNITRACE_CAUSAL_BEGIN(LABEL)
+#    endif
+#    if !defined(OMNITRACE_CAUSAL_END)
+#        define OMNITRACE_CAUSAL_END(LABEL)
+#    endif
+#endif
+
+/** @} */
+
+#endif  // OMNITRACE_CAUSAL_H_
@@ -49,8 +49,10 @@ extern "C"
        omnitrace_trace_func_t            stop_thread_trace;
        omnitrace_region_func_t           push_region;
        omnitrace_region_func_t           pop_region;
+        omnitrace_region_func_t           progress;
        omnitrace_annotated_region_func_t push_annotated_region;
        omnitrace_annotated_region_func_t pop_annotated_region;
+        omnitrace_annotated_region_func_t annotated_progress;

        /// @var start_trace
        /// @brief callback for enabling tracing globally
@@ -64,10 +66,14 @@ extern "C"
        /// @brief callback for starting a trace region
        /// @var pop_region
        /// @brief callback for ending a trace region
+        /// @var progress
+        /// @brief callback for marking an causal profiling event
        /// @var push_annotated_region
        /// @brief callback for starting a trace region + annotations
        /// @var pop_annotated_region
        /// @brief callback for ending a trace region + annotations
+        /// @var annotated_progress
+        /// @brief callback for marking an causal profiling event + annotations
    } omnitrace_user_callbacks_t;

    /// @enum OMNITRACE_USER_CONFIGURE_MODE
@@ -104,7 +110,7 @@ extern "C"
 #ifndef OMNITRACE_USER_CALLBACKS_INIT
 #    define OMNITRACE_USER_CALLBACKS_INIT                                                \
        {                                                                                \
-            NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL                               \
+            NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL                   \
        }
 #endif

@@ -104,6 +104,13 @@ extern "C"
    extern int omnitrace_user_pop_annotated_region(const char*, omnitrace_annotation_t*,
                                                   size_t) OMNITRACE_PUBLIC_API;

+    /// mark causal progress
+    extern int omnitrace_user_progress(const char*) OMNITRACE_PUBLIC_API;
+
+    /// mark causal progress with annotations
+    extern int omnitrace_user_annotated_progress(const char*, omnitrace_annotation_t*,
+                                                 size_t) OMNITRACE_PUBLIC_API;
+
    /// @fn int omnitrace_user_configure(omnitrace_user_configure_mode_t mode,
    ///                                  omnitrace_user_callbacks_t inp,
    ///                                  omnitrace_user_callbacks_t* out)
@@ -80,6 +80,11 @@ extern "C"
        return invoke(_callbacks.pop_region, id);
    }

+    int omnitrace_user_progress(const char* id)
+    {
+        return invoke(_callbacks.progress, id);
+    }
+
    int omnitrace_user_push_annotated_region(const char* id, annotation_t* _annotations,
                                             size_t _annotation_count)
    {
@@ -94,6 +99,12 @@ extern "C"
                      _annotation_count);
    }

+    int omnitrace_user_annotated_progress(const char* id, annotation_t* _annotations,
+                                          size_t _annotation_count)
+    {
+        return invoke(_callbacks.annotated_progress, id, _annotations, _annotation_count);
+    }
+
    int omnitrace_user_configure(omnitrace_user_configure_mode_t mode,
                                 omnitrace_user_callbacks_t      inp,
                                 omnitrace_user_callbacks_t*     out)
@@ -121,8 +132,10 @@ extern "C"
                _update(_v.stop_thread_trace, inp.stop_thread_trace);
                _update(_v.push_region, inp.push_region);
                _update(_v.pop_region, inp.pop_region);
+                _update(_v.progress, inp.progress);
                _update(_v.push_annotated_region, inp.push_annotated_region);
                _update(_v.pop_annotated_region, inp.pop_annotated_region);
+                _update(_v.annotated_progress, inp.annotated_progress);

                _callbacks = _v;
                break;
@@ -141,8 +154,10 @@ extern "C"
                _update(_v.stop_thread_trace, inp.stop_thread_trace);
                _update(_v.push_region, inp.push_region);
                _update(_v.pop_region, inp.pop_region);
+                _update(_v.progress, inp.progress);
                _update(_v.push_annotated_region, inp.push_annotated_region);
                _update(_v.pop_annotated_region, inp.pop_annotated_region);
+                _update(_v.annotated_progress, inp.annotated_progress);

                _callbacks = _v;
                break;
@@ -32,6 +32,7 @@ target_link_libraries(
        $<BUILD_INTERFACE:omnitrace::omnitrace-compile-definitions>
        $<BUILD_INTERFACE:omnitrace::omnitrace-perfetto>
        $<BUILD_INTERFACE:omnitrace::omnitrace-timemory>
+        $<BUILD_INTERFACE:omnitrace::omnitrace-elfutils>
        $<BUILD_INTERFACE:omnitrace::omnitrace-bfd>
        $<BUILD_INTERFACE:omnitrace::omnitrace-mpi>
        $<BUILD_INTERFACE:omnitrace::omnitrace-ptl>
@@ -57,7 +58,8 @@ add_library(omnitrace::omnitrace-object-library ALIAS omnitrace-object-library)
 target_sources(
    omnitrace-object-library
    PRIVATE ${CMAKE_CURRENT_LIST_DIR}/library.cpp ${CMAKE_CURRENT_LIST_DIR}/regions.cpp
-            ${CMAKE_CURRENT_LIST_DIR}/api.cpp ${CMAKE_CURRENT_LIST_DIR}/api.hpp)
+            ${CMAKE_CURRENT_LIST_DIR}/progress.cpp ${CMAKE_CURRENT_LIST_DIR}/api.cpp
+            ${CMAKE_CURRENT_LIST_DIR}/api.hpp)

 add_subdirectory(library)

@@ -100,6 +100,19 @@ omnitrace_pop_category_region(omnitrace_category_t _category, const char* _name,
    return 0;
 }

+extern "C" void
+omnitrace_progress(const char* _name)
+{
+    omnitrace_progress_hidden(_name);
+}
+
+extern "C" void
+omnitrace_annotated_progress(const char* _name, omnitrace_annotation_t* _annotations,
+                             size_t _annotation_count)
+{
+    omnitrace_annotated_progress_hidden(_name, _annotations, _annotation_count);
+}
+
 extern "C" void
 omnitrace_init_library(void)
 {
@@ -86,6 +86,13 @@ extern "C"
    void omnitrace_register_coverage(const char* file, const char* func,
                                     size_t address) OMNITRACE_PUBLIC_API;

+    /// mark causal progress
+    void omnitrace_progress(const char*) OMNITRACE_PUBLIC_API;
+
+    /// mark causal progress with annotations
+    void omnitrace_annotated_progress(const char*, omnitrace_annotation_t*,
+                                      size_t) OMNITRACE_PUBLIC_API;
+
    // these are the real implementations for internal calling convention
    void omnitrace_init_library_hidden(void) OMNITRACE_HIDDEN_API;
    bool omnitrace_init_tooling_hidden(void) OMNITRACE_HIDDEN_API;
@@ -108,4 +115,7 @@ extern "C"
                                          const char*) OMNITRACE_HIDDEN_API;
    void omnitrace_register_coverage_hidden(const char*, const char*,
                                            size_t) OMNITRACE_HIDDEN_API;
+    void omnitrace_progress_hidden(const char*) OMNITRACE_HIDDEN_API;
+    void omnitrace_annotated_progress_hidden(const char*, omnitrace_annotation_t*,
+                                             size_t) OMNITRACE_HIDDEN_API;
 }
@@ -26,12 +26,16 @@

 #include "api.hpp"
 #include "common/setup.hpp"
+#include "library/causal/data.hpp"
+#include "library/causal/experiment.hpp"
+#include "library/causal/sampling.hpp"
 #include "library/components/exit_gotcha.hpp"
 #include "library/components/fork_gotcha.hpp"
 #include "library/components/fwd.hpp"
 #include "library/components/mpi_gotcha.hpp"
 #include "library/components/pthread_gotcha.hpp"
 #include "library/components/rocprofiler.hpp"
+#include "library/concepts.hpp"
 #include "library/config.hpp"
 #include "library/coverage.hpp"
 #include "library/critical_trace.hpp"
@@ -52,6 +56,7 @@
 #include "library/utility.hpp"
 #include "omnitrace/categories.h"  // in omnitrace-user

+#include <timemory/process/threading.hpp>
 #include <timemory/signals/signal_handlers.hpp>
 #include <timemory/signals/types.hpp>
 #include <timemory/hash/types.hpp>
@@ -67,6 +72,8 @@
 #include <mutex>
 #include <string_view>
 #include <utility>
+#include <cstdlib>
+#include <stdexcept>

 using namespace omnitrace;

@@ -92,9 +99,39 @@ namespace
 auto _timemory_manager  = tim::manager::instance();
 auto _timemory_settings = tim::settings::shared_instance();

+bool
+ensure_initialization(bool _offset, int64_t _glob_n, int64_t _offset_n)
+{
+    auto _exit_info = component::exit_gotcha::get_exit_info();
+    if(_exit_info.is_known && _exit_info.exit_code != EXIT_SUCCESS) return _offset;
+
+    auto _tid         = utility::get_thread_index();
+    auto _max_threads = grow_data(_tid + 1);
+
+    if(_tid > 0 && _tid < _max_threads)
+    {
+        const auto& _info = thread_info::get();
+        OMNITRACE_BASIC_VERBOSE_F(3,
+                                  "thread info: %s, offset: %s, global counter: %li, "
+                                  "offset counter: %li, max threads: %li\n",
+                                  std::to_string(static_cast<bool>(_info)).c_str(),
+                                  std::to_string(_offset).c_str(), _glob_n, _offset_n,
+                                  _max_threads);
+    }
+
+    return _offset;
+}
+
 auto
 ensure_finalization(bool _static_init = false)
 {
+    if(_static_init)
+    {
+        auto _idx = threading::add_callback(&ensure_initialization);
+        if(_idx < 0)
+            throw exception<std::runtime_error>("failure adding threading callback");
+    }
+
    const auto& _info = thread_info::init();
    const auto& _tid  = _info->index_data;
    if(_tid)
@@ -149,6 +186,51 @@ is_system_backend()

 using Device = critical_trace::Device;
 using Phase  = critical_trace::Phase;
+
+template <typename... Tp>
+struct fini_bundle
+{
+    using data_type = std::tuple<Tp...>;
+
+    TIMEMORY_DEFAULT_OBJECT(fini_bundle)
+
+    fini_bundle(std::string_view _label)
+    : m_label{ _label }
+    {}
+
+    template <typename... Args>
+    void start(Args&&... _args)
+    {
+        TIMEMORY_FOLD_EXPRESSION(tim::operation::start<Tp>{}(
+            std::get<Tp>(m_data), std::forward<Args>(_args)...));
+    }
+
+    template <typename... Args>
+    void stop(Args&&... _args)
+    {
+        TIMEMORY_FOLD_EXPRESSION(tim::operation::stop<Tp>{}(
+            std::get<Tp>(m_data), std::forward<Args>(_args)...));
+    }
+
+    std::string as_string(bool _print_prefix = true) const
+    {
+        std::stringstream _ss;
+        if(_print_prefix && m_label.length() > 0) _ss << m_label << " : ";
+        _ss << timemory::join::join(", ", std::get<Tp>(m_data)...);
+        return _ss.str();
+    }
+
+    std::string_view m_label = {};
+    data_type        m_data  = {};
+};
+
+template <typename... Tp>
+struct fini_bundle<tim::lightweight_tuple<Tp...>>
+{
+    using base_type = fini_bundle<Tp...>;
+};
+
+using fini_bundle_t = typename fini_bundle<main_bundle_t>::base_type;
 }  // namespace

 //======================================================================================//
@@ -365,13 +447,21 @@ omnitrace_init_tooling_hidden()
            OMNITRACE_SCOPED_SAMPLING_ON_CHILD_THREADS(false);
            process_sampler::setup();
        }
-        if(get_use_sampling())
+        if(get_use_causal())
        {
-            OMNITRACE_SCOPED_SAMPLING_ON_CHILD_THREADS(false);
-            sampling::setup();
+            {
+                OMNITRACE_SCOPED_SAMPLING_ON_CHILD_THREADS(false);
+                causal::sampling::setup();
+            }
+            push_enable_sampling_on_child_threads(get_use_causal());
+            sampling::unblock_signals();
        }
-        if(get_use_sampling())
+        else if(get_use_sampling())
        {
+            {
+                OMNITRACE_SCOPED_SAMPLING_ON_CHILD_THREADS(false);
+                sampling::setup();
+            }
            push_enable_sampling_on_child_threads(get_use_sampling());
            sampling::unblock_signals();
        }
@@ -400,6 +490,8 @@ omnitrace_init_tooling_hidden()

    tasking::setup();

+    if(get_use_causal()) causal::start_experimenting();
+
    if(get_use_timemory())
    {
        comp::user_global_bundle::global_init();
@@ -457,13 +549,14 @@ omnitrace_init_tooling_hidden()
 //======================================================================================//

 extern "C" void
-omnitrace_init_hidden(const char* _mode, bool _is_binary_rewrite, const char* _argv0)
+omnitrace_init_hidden(const char* _mode, bool _is_binary_rewrite, const char* _argv0_c)
 {
    static int  _total_count = 0;
    static auto _args = std::make_pair(std::string_view{ _mode }, _is_binary_rewrite);

    auto _count   = _total_count++;
    auto _mode_sv = std::string_view{ _mode };
+    auto _argv0   = (_argv0_c) ? std::string{ _argv0_c } : config::get_exe_name();
    // this function may be called multiple times if multiple libraries are instrumented
    // we want to guard against multiple calls which with different arguments
    if(_count > 0 &&
@@ -495,21 +588,23 @@ omnitrace_init_hidden(const char* _mode, bool _is_binary_rewrite, const char* _a
                "called after omnitrace was initialized. state = %s. Mode-based settings "
                "(via -M <MODE> passed to omnitrace exe) may not be properly "
                "configured.\n",
-                _mode, std::to_string(_is_binary_rewrite).c_str(), _argv0,
+                _mode, std::to_string(_is_binary_rewrite).c_str(), _argv0.c_str(),
                std::to_string(get_state()).c_str());
        }
    }

-    tracing::get_finalization_functions().emplace_back([_argv0]() {
+    tracing::get_finalization_functions().emplace_back([_argv0_c]() {
        OMNITRACE_CI_THROW(get_state() != State::Active,
                           "Finalizer function for popping main invoked in non-active "
                           "state :: state = %s\n",
                           std::to_string(get_state()).c_str());
        if(get_state() == State::Active)
        {
+            auto _name = (_argv0_c) ? std::string{ _argv0_c } : config::get_exe_name();
            // if main hasn't been popped yet, pop it
-            OMNITRACE_BASIC_VERBOSE(2, "Running omnitrace_pop_trace(%s)...\n", _argv0);
-            omnitrace_pop_trace_hidden(_argv0);
+            OMNITRACE_BASIC_VERBOSE(2, "Running omnitrace_pop_trace(%s)...\n",
+                                    _name.c_str());
+            omnitrace_pop_trace_hidden(_name.c_str());
        }
    });

@@ -521,7 +616,7 @@ omnitrace_init_hidden(const char* _mode, bool _is_binary_rewrite, const char* _a
    OMNITRACE_CONDITIONAL_BASIC_PRINT_F(
        get_debug_env() || get_verbose_env() > 2,
        "mode: %s | is binary rewrite: %s | command: %s\n", _mode,
-        (_is_binary_rewrite) ? "y" : "n", _argv0);
+        (_is_binary_rewrite) ? "y" : "n", _argv0.c_str());

    tim::set_env("OMNITRACE_MODE", _mode, 0);
    config::is_binary_rewrite() = _is_binary_rewrite;
@@ -561,6 +656,8 @@ omnitrace_finalize_hidden(void)
 {
    // disable thread id recycling during finalization
    threading::recycle_ids() = false;
+    // disable initialization callback
+    threading::remove_callback(&ensure_initialization);

    set_thread_state(ThreadState::Completed);

@@ -629,13 +726,6 @@ omnitrace_finalize_hidden(void)
    tim::signals::enable_signal_detection({ tim::signals::sys_signal::Interrupt },
                                          [](int) {});

-    std::string              _bundle_name = OMNITRACE_FUNCTION;
-    comp::user_global_bundle _bundle{ _bundle_name.c_str() };
-    _bundle.clear();
-    _bundle.insert<comp::wall_clock, comp::cpu_clock, comp::peak_rss, comp::page_rss,
-                   comp::cpu_util>();
-    _bundle.start();
-
    OMNITRACE_DEBUG_F("Copying over all timemory hash information to main thread...\n");
    // copy these over so that all hashes are known
    auto& _hzero = tracing::get_timemory_hash_ids(0);
@@ -663,6 +753,9 @@ omnitrace_finalize_hidden(void)
        get_main_bundle()->stop();
    }

+    fini_bundle_t _finalization{};
+    _finalization.start();
+
    if(get_use_rcclp())
    {
        OMNITRACE_VERBOSE_F(1, "Shutting down RCCLP...\n");
@@ -741,6 +834,12 @@ omnitrace_finalize_hidden(void)
        rocprofiler::rocm_cleanup();
    }

+    if(get_use_causal())
+    {
+        OMNITRACE_VERBOSE_F(1, "Shutting down causal sampling...\n");
+        causal::sampling::shutdown();
+    }
+
    if(get_use_sampling())
    {
        OMNITRACE_VERBOSE_F(1, "Shutting down sampling...\n");
@@ -764,6 +863,7 @@ omnitrace_finalize_hidden(void)
    // if they are still running (e.g. thread-pool still alive), the
    // thread-specific data will be wrong if try to stop them from
    // the main thread.
+    auto _thr_verbose = (config::get_use_causal()) ? 1 : 0;
    for(auto& itr : thread_data<thread_bundle_t>::instances())
    {
        if(itr && itr->get<comp::wall_clock>() &&
@@ -772,7 +872,7 @@ omnitrace_finalize_hidden(void)
            std::string _msg = JOIN("", *itr);
            auto        _pos = _msg.find(">>>  ");
            if(_pos != std::string::npos) _msg = _msg.substr(_pos + 5);
-            OMNITRACE_VERBOSE_F(0, "%s\n", _msg.c_str());
+            OMNITRACE_VERBOSE_F(_thr_verbose, "%s\n", _msg.c_str());
        }
    }

@@ -785,6 +885,12 @@ omnitrace_finalize_hidden(void)
        sampling::post_process();
    }

+    if(get_use_causal())
+    {
+        OMNITRACE_VERBOSE_F(1, "Finishing the causal experiments...\n");
+        causal::finish_experimenting();
+    }
+
    if(get_use_critical_trace() || (get_use_rocm_smi() && get_use_roctracer()))
    {
        OMNITRACE_VERBOSE_F(1, "Generating the critical trace...\n");
@@ -929,18 +1035,6 @@ omnitrace_finalize_hidden(void)
        }
    }

-    _bundle.stop();
-    auto _get_metric = [](auto* _v, std::string_view _tail) -> std::string {
-        return (_v) ? JOIN("", *_v, _tail) : std::string{};
-    };
-
-    OMNITRACE_VERBOSE_F(0, "Finalization metrics: %s%s%s%s%s\n",
-                        _get_metric(_bundle.get<comp::wall_clock>(), ", ").c_str(),
-                        _get_metric(_bundle.get<comp::peak_rss>(), ", ").c_str(),
-                        _get_metric(_bundle.get<comp::page_rss>(), ", ").c_str(),
-                        _get_metric(_bundle.get<comp::cpu_clock>(), ", ").c_str(),
-                        _get_metric(_bundle.get<comp::cpu_util>(), "").c_str());
-
    if(_timemory_manager && _timemory_manager != nullptr)
    {
        _timemory_manager->add_metadata([](auto& ar) {
@@ -967,6 +1061,8 @@ omnitrace_finalize_hidden(void)
                                          "omnitrace", _cfg);
    }

+    _finalization.stop();
+
    if(_perfetto_output_error)
    {
        OMNITRACE_THROW("Error opening perfetto output file: %s",
@@ -984,7 +1080,7 @@ omnitrace_finalize_hidden(void)

    config::finalize();

-    OMNITRACE_VERBOSE_F(0, "Finalized\n");
+    OMNITRACE_VERBOSE_F(0, "Finalized: %s\n", _finalization.as_string().c_str());
 }

 //======================================================================================//
@@ -9,8 +9,10 @@ set(library_sources
    ${CMAKE_CURRENT_LIST_DIR}/critical_trace.cpp
    ${CMAKE_CURRENT_LIST_DIR}/debug.cpp
    ${CMAKE_CURRENT_LIST_DIR}/dynamic_library.cpp
-    ${CMAKE_CURRENT_LIST_DIR}/kokkosp.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/exception.cpp
    ${CMAKE_CURRENT_LIST_DIR}/gpu.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/kokkosp.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/locking.cpp
    ${CMAKE_CURRENT_LIST_DIR}/mproc.cpp
    ${CMAKE_CURRENT_LIST_DIR}/ompt.cpp
    ${CMAKE_CURRENT_LIST_DIR}/perfetto.cpp
@@ -19,28 +21,30 @@ set(library_sources
    ${CMAKE_CURRENT_LIST_DIR}/runtime.cpp
    ${CMAKE_CURRENT_LIST_DIR}/sampling.cpp
    ${CMAKE_CURRENT_LIST_DIR}/state.cpp
-    ${CMAKE_CURRENT_LIST_DIR}/thread_data.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/thread_deleter.cpp
    ${CMAKE_CURRENT_LIST_DIR}/thread_info.cpp
    ${CMAKE_CURRENT_LIST_DIR}/timemory.cpp
    ${CMAKE_CURRENT_LIST_DIR}/tracing.cpp)

 set(library_headers
    ${CMAKE_CURRENT_LIST_DIR}/categories.hpp
-    ${CMAKE_CURRENT_LIST_DIR}/config.hpp
    ${CMAKE_CURRENT_LIST_DIR}/common.hpp
    ${CMAKE_CURRENT_LIST_DIR}/concepts.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/config.hpp
    ${CMAKE_CURRENT_LIST_DIR}/coverage.hpp
    ${CMAKE_CURRENT_LIST_DIR}/cpu_freq.hpp
    ${CMAKE_CURRENT_LIST_DIR}/critical_trace.hpp
    ${CMAKE_CURRENT_LIST_DIR}/debug.hpp
    ${CMAKE_CURRENT_LIST_DIR}/dynamic_library.hpp
    ${CMAKE_CURRENT_LIST_DIR}/gpu.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/locking.hpp
    ${CMAKE_CURRENT_LIST_DIR}/mproc.hpp
    ${CMAKE_CURRENT_LIST_DIR}/ompt.hpp
    ${CMAKE_CURRENT_LIST_DIR}/perfetto.hpp
    ${CMAKE_CURRENT_LIST_DIR}/process_sampler.hpp
    ${CMAKE_CURRENT_LIST_DIR}/ptl.hpp
    ${CMAKE_CURRENT_LIST_DIR}/rcclp.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/redirect.hpp
    ${CMAKE_CURRENT_LIST_DIR}/rocm.hpp
    ${CMAKE_CURRENT_LIST_DIR}/rocm_smi.hpp
    ${CMAKE_CURRENT_LIST_DIR}/rocprofiler.hpp
@@ -49,6 +53,7 @@ set(library_headers
    ${CMAKE_CURRENT_LIST_DIR}/sampling.hpp
    ${CMAKE_CURRENT_LIST_DIR}/state.hpp
    ${CMAKE_CURRENT_LIST_DIR}/thread_data.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/thread_deleter.hpp
    ${CMAKE_CURRENT_LIST_DIR}/thread_info.hpp
    ${CMAKE_CURRENT_LIST_DIR}/timemory.hpp
    ${CMAKE_CURRENT_LIST_DIR}/tracing.hpp
@@ -81,7 +86,22 @@ if(OMNITRACE_USE_ROCM_SMI)
                   PRIVATE ${CMAKE_CURRENT_LIST_DIR}/rocm_smi.cpp)
 endif()

+add_subdirectory(binary)
+add_subdirectory(causal)
 add_subdirectory(components)
+add_subdirectory(containers)
 add_subdirectory(coverage)
 add_subdirectory(rocm)
 add_subdirectory(tracing)
+
+set(ndebug_sources
+    ${CMAKE_CURRENT_LIST_DIR}/components/mpi_gotcha.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/components/backtrace_metrics.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/rcclp.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/kokkosp.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/rocm_smi.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/ompt.cpp)
+
+set_source_files_properties(
+    ${ndebug_sources} DIRECTORY ${PROJECT_SOURCE_DIR}/source/lib/omnitrace
+    PROPERTIES COMPILE_DEFINITIONS NDEBUG COMPILE_OPTIONS "-g0;-O3")
@@ -0,0 +1,22 @@
+#
+set(binary_sources
+    ${CMAKE_CURRENT_LIST_DIR}/address_multirange.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/address_range.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/analysis.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/dwarf_entry.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/link_map.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/scope_filter.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/symbol.cpp)
+
+set(binary_headers
+    ${CMAKE_CURRENT_LIST_DIR}/address_multirange.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/address_range.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/analysis.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/dwarf_entry.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/fwd.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/binary_info.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/link_map.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/scope_filter.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/symbol.hpp)
+
+target_sources(omnitrace-object-library PRIVATE ${binary_sources} ${binary_headers})
@@ -0,0 +1,75 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/binary/address_multirange.hpp"
+
+#include <algorithm>
+#include <cstdint>
+#include <string>
+#include <utility>
+
+namespace omnitrace
+{
+namespace binary
+{
+address_multirange&
+address_multirange::operator+=(std::pair<coarse, uintptr_t>&& _v)
+{
+    coarse_range = address_range{ std::min(coarse_range.low, _v.second),
+                                  std::max(coarse_range.high, _v.second) };
+    return *this;
+}
+
+address_multirange&
+address_multirange::operator+=(std::pair<coarse, address_range>&& _v)
+{
+    coarse_range = address_range{ std::min(coarse_range.low, _v.second.low),
+                                  std::max(coarse_range.high, _v.second.high) };
+
+    return *this;
+}
+
+address_multirange&
+address_multirange::operator+=(uintptr_t _v)
+{
+    *this += std::make_pair(coarse{}, _v);
+
+    for(auto&& itr : m_fine_ranges)
+        if(itr.contains(_v)) return *this;
+
+    m_fine_ranges.emplace(address_range{ _v });
+    return *this;
+}
+
+address_multirange&
+address_multirange::operator+=(address_range _v)
+{
+    *this += std::make_pair(coarse{}, _v);
+
+    for(auto&& itr : m_fine_ranges)
+        if(itr.contains(_v)) return *this;
+
+    m_fine_ranges.emplace(_v);
+    return *this;
+}
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,76 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/binary/address_range.hpp"
+#include "library/binary/fwd.hpp"
+
+#include <timemory/utility/macros.hpp>
+
+#include <cstdint>
+#include <utility>
+
+namespace omnitrace
+{
+namespace binary
+{
+struct address_multirange
+{
+    struct coarse
+    {};
+
+    TIMEMORY_DEFAULT_OBJECT(address_multirange)
+
+    address_multirange& operator+=(std::pair<coarse, uintptr_t>&&);
+    address_multirange& operator+=(std::pair<coarse, address_range>&& _v);
+    address_multirange& operator+=(uintptr_t _v);
+    address_multirange& operator+=(address_range _v);
+
+    template <typename Tp>
+    bool contains(Tp&& _v) const;
+
+    address_range coarse_range = {};
+
+    auto size() const { return m_fine_ranges.size(); }
+    auto empty() const { return m_fine_ranges.empty(); }
+    auto range_size() const { return coarse_range.size(); }
+
+private:
+    std::set<address_range> m_fine_ranges = {};
+};
+
+template <typename Tp>
+OMNITRACE_INLINE bool
+address_multirange::contains(Tp&& _v) const
+{
+    using type = concepts::unqualified_type_t<Tp>;
+    static_assert(std::is_integral<type>::value ||
+                      std::is_same<type, address_range>::value,
+                  "Error! operator+= supports only integrals or address_ranges");
+
+    if(!coarse_range.contains(_v)) return false;
+    return std::any_of(m_fine_ranges.begin(), m_fine_ranges.end(),
+                       [_v](auto&& itr) { return itr.contains(_v); });
+}
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,190 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/binary/address_range.hpp"
+#include "library/debug.hpp"
+
+namespace omnitrace
+{
+namespace binary
+{
+address_range::address_range(uintptr_t _v)
+: low{ _v }
+, high{ _v }
+{}
+
+address_range::address_range(uintptr_t _low, uintptr_t _high)
+: low{ _low }
+, high{ _high }
+{
+    TIMEMORY_REQUIRE(high >= low)
+        << "Error! address_range high must be >= low. low=" << as_hex(low)
+        << ", high=" << as_hex(high) << "\n";
+}
+
+bool
+address_range::is_range() const
+{
+    return (low < high);
+}
+
+std::string
+address_range::as_string(int _depth) const
+{
+    std::stringstream _ss{};
+    _ss << std::hex;
+    _ss << std::setw(2 * _depth) << "";
+    _ss.fill('0');
+    _ss << "0x" << std::setw(16) << low << "-"
+        << "0x" << std::setw(16) << high;
+    return _ss.str();
+}
+
+uintptr_t
+address_range::size() const
+{
+    return (low == high) ? 1 : (high > low) ? (high - low + 1) : (low - high + 1);
+}
+
+bool
+address_range::is_valid() const
+{
+    return (low <= high && (low + 1) > 1);
+}
+
+bool
+address_range::contains(uintptr_t _v) const
+{
+    return (is_range()) ? (low <= _v && high > _v) : (_v == low);
+}
+
+bool
+address_range::contains(address_range _v) const
+{
+    return (*this == _v) || (contains(_v.low) && contains(_v.high));
+}
+
+bool
+address_range::overlaps(address_range _v) const
+{
+    if(contains(_v)) return false;
+    int64_t _lhs_diff = (high - low);
+    int64_t _rhs_diff = (_v.high - _v.low);
+    int64_t _diff     = (std::max(high, _v.high) - std::min(low, _v.low));
+    return (_diff < (_lhs_diff + _rhs_diff));
+}
+
+bool
+address_range::contiguous_with(address_range _v) const
+{
+    return (_v.low == high || low == _v.high);
+}
+
+bool
+address_range::operator==(address_range _v) const
+{
+    // if arg is range and this is not range, call this function with arg
+    // if(_v.is_range() && !is_range()) return false;
+    // check if arg is in range
+    // if(is_range() && !_v.is_range()) return false;
+    // both are ranges or both are just address
+    return std::tie(low, high) == std::tie(_v.low, _v.high);
+}
+
+bool
+address_range::operator<(address_range _v) const
+{
+    if(is_range() && !_v.is_range())
+    {
+        return (low == _v.low) ? true : (low < _v.low);
+    }
+    else if(!is_range() && _v.is_range())
+    {
+        return (low == _v.low) ? false : (low < _v.low);
+    }
+    else if(!is_range() && !_v.is_range())
+    {
+        return (low < _v.low);
+    }
+    return std::tie(low, high) < std::tie(_v.low, _v.high);
+    // if(_v.low == _v.high && _v.low >= low && _v.low < high) return false;
+    // return (low == _v.low) ? (high > _v.high) : (low < _v.low);
+}
+
+bool
+address_range::operator>(address_range _v) const
+{
+    return !(*this < _v) && !(*this == _v);
+}
+
+address_range&
+address_range::operator+=(uintptr_t _v)
+{
+    if(is_valid())
+    {
+        low += _v;
+        high += _v;
+    }
+    else
+    {
+        low  = _v;
+        high = _v;
+    }
+    return *this;
+}
+
+address_range&
+address_range::operator-=(uintptr_t _v)
+{
+    if(is_valid())
+    {
+        low -= _v;
+        high -= _v;
+    }
+    else
+    {
+        low  = _v;
+        high = _v;
+    }
+    return *this;
+}
+
+address_range&
+address_range::operator+=(address_range _v)
+{
+    if(!contiguous_with(_v))
+        throw exception<std::runtime_error>(
+            "attempting to add two address ranges that are not contiguous");
+
+    low  = std::min(low, _v.low);
+    high = std::max(high, _v.high);
+    return *this;
+}
+
+hash_value_t
+address_range::hash() const
+{
+    return (is_range()) ? tim::get_combined_hash_id(hash_value_t{ low }, high)
+                        : hash_value_t{ low };
+}
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,104 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/binary/fwd.hpp"
+#include "library/common.hpp"
+#include "library/timemory.hpp"
+
+#include <timemory/hash/types.hpp>
+#include <timemory/utility/macros.hpp>
+
+#include <cstdint>
+#include <limits>
+
+namespace omnitrace
+{
+namespace binary
+{
+struct address_range
+{
+    // set to low to max and high to min to support std::min(...)
+    // and std::max(...) assignment
+    uintptr_t low  = std::numeric_limits<uintptr_t>::max();
+    uintptr_t high = std::numeric_limits<uintptr_t>::min();
+
+    TIMEMORY_DEFAULT_OBJECT(address_range)
+
+    explicit address_range(uintptr_t _v);
+    address_range(uintptr_t _low, uintptr_t _high);
+
+    bool contains(uintptr_t) const;
+    bool contains(address_range) const;
+    bool overlaps(address_range) const;
+    bool contiguous_with(address_range) const;
+
+    bool operator==(address_range _v) const;
+    bool operator!=(address_range _v) const { return !(*this == _v); }
+    bool operator<(address_range _v) const;
+    bool operator>(address_range _v) const;
+
+    address_range& operator+=(uintptr_t);
+    address_range& operator-=(uintptr_t);
+    address_range& operator+=(address_range);
+
+    bool         is_range() const;
+    hash_value_t hash() const;
+    std::string  as_string(int _depth = 0) const;
+    bool         is_valid() const;
+    uintptr_t    size() const;
+    explicit     operator bool() const { return is_valid(); }
+
+    template <typename ArchiveT>
+    void serialize(ArchiveT& ar, const unsigned)
+    {
+        ar(cereal::make_nvp("low", low));
+        ar(cereal::make_nvp("high", high));
+    }
+};
+}  // namespace binary
+
+inline binary::address_range
+operator+(binary::address_range _lhs, uintptr_t _v)
+{
+    return (_lhs += _v);
+}
+
+inline binary::address_range
+operator+(uintptr_t _v, binary::address_range _lhs)
+{
+    return (_lhs += _v);
+}
+}  // namespace omnitrace
+
+namespace std
+{
+template <>
+struct hash<::omnitrace::binary::address_range>
+{
+    using address_range_t = ::omnitrace::binary::address_range;
+
+    auto operator()(const address_range_t& _v) const { return _v.hash(); }
+    auto operator()(address_range_t&& _v) const { return _v.hash(); }
+};
+}  // namespace std
@@ -0,0 +1,194 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/config.hpp"
+
+#if !defined(TIMEMORY_USE_BFD)
+#    error "BFD support not enabled"
+#endif
+
+#define PACKAGE "omnitrace"
+
+#include <bfd.h>
+
+#include "library/binary/address_range.hpp"
+#include "library/binary/analysis.hpp"
+#include "library/binary/binary_info.hpp"
+#include "library/binary/dwarf_entry.hpp"
+#include "library/binary/fwd.hpp"
+#include "library/binary/scope_filter.hpp"
+#include "library/binary/symbol.hpp"
+#include "library/common.hpp"
+#include "library/config.hpp"
+#include "library/debug.hpp"
+#include "library/state.hpp"
+#include "library/utility.hpp"
+
+#include <timemory/log/macros.hpp>
+#include <timemory/unwind/bfd.hpp>
+#include <timemory/unwind/dlinfo.hpp>
+#include <timemory/utility/filepath.hpp>
+#include <timemory/utility/join.hpp>
+#include <timemory/utility/procfs/maps.hpp>
+
+#include <cstdint>
+#include <cstdlib>
+#include <dlfcn.h>
+#include <regex>
+#include <set>
+#include <stdexcept>
+
+namespace omnitrace
+{
+namespace binary
+{
+namespace
+{
+binary_info
+parse_line_info(const std::string& _name)
+{
+    auto _info = binary_info{};
+
+    auto& _bfd = _info.bfd;
+    _bfd       = std::make_shared<bfd_file>(_name);
+
+    OMNITRACE_VERBOSE(0, "[binary] Reading line info for '%s'...\n", _name.c_str());
+
+    if(_bfd && _bfd->is_good())
+    {
+        auto& _section_map = _info.sections;
+        auto  _section_set = std::set<asection*>{};
+        auto  _processed   = std::set<uintptr_t>{};
+        for(auto&& itr : _bfd->get_symbols())
+        {
+            if(itr.symsize == 0) continue;
+            auto& _sym = _info.symbols.emplace_back(symbol{ itr });
+            // if(itr.symsize == 0) continue;
+            auto* _section = static_cast<asection*>(itr.section);
+            _section_set.emplace(_section);
+            _processed.emplace(itr.address);
+            _info.ranges.emplace_back(
+                address_range{ itr.address, itr.address + itr.symsize });
+            _sym.read_bfd(*_bfd);
+        }
+
+        for(auto* itr : _section_set)
+        {
+            auto*         _section     = const_cast<asection*>(itr);
+            bfd_vma       _section_vma = bfd_section_vma(_section);
+            bfd_size_type _section_len = bfd_section_size(_section);
+            auto          _section_range =
+                address_range{ _section_vma, _section_vma + _section_len };
+            _section_map[_section_range] = _section;
+        }
+
+        TIMEMORY_REQUIRE(_section_set.size() == _section_map.size())
+            << "section set size (" << _section_set.size() << ") != section map size ("
+            << _section_map.size() << ")\n";
+
+        _info.debug_info = dwarf_entry::process_dwarf(_bfd->fd, _info.ranges);
+
+        for(auto& itr : _info.symbols)
+            itr.read_dwarf(_info.debug_info);
+
+        _info.sort();
+    }
+
+    OMNITRACE_VERBOSE(1, "[binary] Reading line info for '%s'... %zu entries\n",
+                      _bfd->name.c_str(), _info.symbols.size());
+
+    return _info;
+}
+}  // namespace
+
+std::vector<binary_info>
+get_binary_info(const std::vector<std::string>&  _files,
+                const std::vector<scope_filter>& _filters)
+{
+    auto _satisfies_filter = [&_filters](auto _scope, const std::string& _value) {
+        for(const auto& itr : _filters)  // NOLINT
+        {
+            // if the filter is for the specified scope and itr does not satisfy the
+            // include/exclude mode, return false
+            if((itr.scope & _scope) == _scope && !itr(_value)) return false;
+        }
+        return true;
+    };
+
+    auto _satisfies_binary_filter = [&_satisfies_filter](const std::string& _value) {
+        return _satisfies_filter(scope_filter::BINARY_FILTER, _value);
+    };
+
+    // filter function used by procfs::get_contiguous_maps
+    // ensures that we do not process omnitrace/gotcha/libunwind libraries
+    // and do not process the libraries outside of the binary scope
+    auto _filter = [&_satisfies_binary_filter](const procfs::maps& _v) {
+        if(_v.pathname.empty()) return false;
+        auto _path = filepath::realpath(_v.pathname, nullptr, false);
+        return (filepath::exists(_path) && _satisfies_binary_filter(_path));
+    };
+
+    auto _data = std::vector<binary_info>{};
+    _data.reserve(_files.size());
+    {
+        auto _exists = std::set<std::string>{};
+        for(const auto& itr : _files)
+        {
+            auto _filename = filepath::realpath(itr, nullptr, false);
+            if(filepath::exists(_filename) && _satisfies_binary_filter(_filename) &&
+               _exists.find(_filename) == _exists.end())
+            {
+                _data.emplace_back(parse_line_info(_filename));
+                _exists.emplace(_filename);
+            }
+        }
+    }
+
+    // get the memory maps
+    auto _maps = procfs::get_contiguous_maps(process::get_id(), _filter, true);
+
+    for(auto& itr : _data)
+    {
+        for(const auto& mitr : _maps)
+            if(itr.bfd->name == mitr.pathname) itr.mappings.emplace_back(mitr);
+    }
+
+    for(auto& itr : _data)
+    {
+        for(const auto& mitr : itr.mappings)
+        {
+            auto mrange = address_range{ mitr.load_address, mitr.last_address };
+            for(auto& sitr : itr.symbols)
+            {
+                auto _addr = sitr.address + mitr.load_address;
+                if(mrange.contains(_addr)) sitr.load_address = mitr.load_address;
+            }
+        }
+    }
+
+    for(auto& itr : _data)
+        itr.sort();
+
+    return _data;
+}
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,59 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "common/defines.h"
+#include "library/binary/fwd.hpp"
+#include "library/common.hpp"
+#include "library/defines.hpp"
+#include "library/exception.hpp"
+
+#include <timemory/hash/types.hpp>
+#include <timemory/mpl/concepts.hpp>
+#include <timemory/tpls/cereal/cereal/cereal.hpp>
+#include <timemory/unwind/bfd.hpp>
+#include <timemory/unwind/types.hpp>
+#include <timemory/utility/procfs/maps.hpp>
+
+#include <cstdint>
+#include <deque>
+#include <map>
+#include <memory>
+#include <regex>
+#include <string>
+#include <tuple>
+#include <variant>
+
+namespace omnitrace
+{
+namespace binary
+{
+namespace procfs = ::tim::procfs;  // NOLINT
+
+using bfd_file     = ::tim::unwind::bfd_file;
+using hash_value_t = ::tim::hash_value_t;
+
+std::vector<binary_info>
+get_binary_info(const std::vector<std::string>&, const std::vector<scope_filter>&);
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,76 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/binary/address_range.hpp"
+#include "library/binary/dwarf_entry.hpp"
+#include "library/binary/fwd.hpp"
+#include "library/binary/symbol.hpp"
+#include "library/utility.hpp"
+
+#include <timemory/utility/procfs/maps.hpp>
+
+#include <deque>
+#include <memory>
+#include <vector>
+
+namespace omnitrace
+{
+namespace binary
+{
+struct binary_info
+{
+    std::shared_ptr<bfd_file>                bfd        = {};
+    std::vector<procfs::maps>                mappings   = {};
+    std::deque<symbol>                       symbols    = {};
+    std::deque<dwarf_entry>                  debug_info = {};
+    std::vector<address_range>               ranges     = {};
+    std::unordered_map<address_range, void*> sections   = {};
+
+    void sort();
+
+    template <typename RetT = void>
+    RetT* find_section(uintptr_t);
+};
+
+inline void
+binary_info::sort()
+{
+    utility::filter_sort_unique(mappings);
+    utility::filter_sort_unique(symbols);
+    utility::filter_sort_unique(ranges);
+    utility::filter_sort_unique(debug_info);
+}
+
+template <typename RetT>
+inline RetT*
+binary_info::find_section(uintptr_t _addr)
+{
+    for(const auto& sitr : sections)
+    {
+        if(sitr.first.contains(_addr)) return static_cast<RetT*>(sitr.second);
+    }
+    return nullptr;
+}
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,201 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/binary/dwarf_entry.hpp"
+#include "library/binary/fwd.hpp"
+#include "library/timemory.hpp"
+#include "library/utility.hpp"
+
+#include <dwarf.h>
+#include <elfutils/libdw.h>
+
+namespace omnitrace
+{
+namespace binary
+{
+namespace
+{
+using utility::combine;
+
+auto
+get_dwarf_address_ranges(Dwarf_Die* _die)
+{
+    auto _ranges = std::vector<address_range>{};
+
+    if(dwarf_tag(_die) != DW_TAG_compile_unit) return _ranges;
+
+    Dwarf_Addr _low_pc;
+    Dwarf_Addr _high_pc;
+    dwarf_lowpc(_die, &_low_pc);
+    dwarf_highpc(_die, &_high_pc);
+
+    _ranges.emplace_back(address_range{ _low_pc, _high_pc });
+
+    Dwarf_Addr _base_addr;
+    ptrdiff_t  _offset = 0;
+    do
+    {
+        _ranges.emplace_back(address_range{ 0, 0 });
+    } while((_offset = dwarf_ranges(_die, _offset, &_base_addr, &_ranges.back().low,
+                                    &_ranges.back().high)) > 0);
+    // will always have one extra
+    _ranges.pop_back();
+
+    return _ranges;
+}
+
+auto
+get_dwarf_entry(Dwarf_Die* _die)
+{
+    auto _line_info = std::deque<dwarf_entry>{};
+
+    if(dwarf_tag(_die) != DW_TAG_compile_unit) return _line_info;
+
+    Dwarf_Lines* _lines     = nullptr;
+    size_t       _num_lines = 0;
+    if(dwarf_getsrclines(_die, &_lines, &_num_lines) == 0)
+    {
+        _line_info.resize(_num_lines);
+        for(size_t j = 0; j < _num_lines; ++j)
+        {
+            auto& itr   = _line_info.at(j);
+            auto* _line = dwarf_onesrcline(_lines, j);
+            if(_line)
+            {
+                int       _lineno  = 0;
+                uintptr_t _address = 0;
+                dwarf_lineno(_line, &_lineno);
+                dwarf_linecol(_line, &itr.col);
+                dwarf_linebeginstatement(_line, &itr.begin_statement);
+                dwarf_lineendsequence(_line, &itr.end_sequence);
+                dwarf_lineblock(_line, &itr.line_block);
+                dwarf_lineepiloguebegin(_line, &itr.epilogue_begin);
+                dwarf_lineprologueend(_line, &itr.prologue_end);
+                dwarf_lineisa(_line, &itr.isa);
+                dwarf_linediscriminator(_line, &itr.discriminator);
+                dwarf_lineaddr(_line, &_address);
+                itr.address = address_range{ _address };
+                if(_lineno > 0) itr.line = _lineno;
+                const auto* _file = dwarf_linesrc(_line, nullptr, nullptr);
+                if(!_file) _file = dwarf_diename(_die);
+                itr.file = filepath::realpath(_file, nullptr, false);
+            }
+        }
+    }
+
+    return _line_info;
+}
+}  // namespace
+
+bool
+dwarf_entry::operator<(const dwarf_entry& _rhs) const
+{
+    return std::tie(address, line, col, discriminator) <
+           std::tie(_rhs.address, _rhs.line, _rhs.col, _rhs.discriminator);
+}
+
+bool
+dwarf_entry::operator==(const dwarf_entry& _rhs) const
+{
+    return std::tie(address, line, col, discriminator, vliw_op_index, isa, file) ==
+           std::tie(_rhs.address, _rhs.line, _rhs.col, _rhs.discriminator,
+                    _rhs.vliw_op_index, _rhs.isa, _rhs.file);
+}
+
+bool
+dwarf_entry::operator!=(const dwarf_entry& _rhs) const
+{
+    return !(*this == _rhs);
+}
+
+bool
+dwarf_entry::is_valid() const
+{
+    return (*this != dwarf_entry{} && !file.empty());
+}
+
+std::deque<dwarf_entry>
+dwarf_entry::process_dwarf(int _fd, std::vector<address_range>& _ranges)
+{
+    auto* _dwarf_v   = dwarf_begin(_fd, DWARF_C_READ);
+    auto  _line_info = std::deque<dwarf_entry>{};
+
+    size_t    cu_header_size = 0;
+    Dwarf_Off cu_off         = 0;
+    Dwarf_Off next_cu_off    = 0;
+    for(; dwarf_nextcu(_dwarf_v, cu_off, &next_cu_off, &cu_header_size, nullptr, nullptr,
+                       nullptr) == 0;
+        cu_off = next_cu_off)
+    {
+        Dwarf_Off cu_die_off = cu_off + cu_header_size;
+        Dwarf_Die cu_die;
+        if(dwarf_offdie(_dwarf_v, cu_die_off, &cu_die) != nullptr)
+        {
+            Dwarf_Die* _die = &cu_die;
+            if(dwarf_tag(_die) == DW_TAG_compile_unit)
+            {
+                combine(_line_info, get_dwarf_entry(_die));
+                combine(_ranges, get_dwarf_address_ranges(_die));
+            }
+        }
+    }
+
+    dwarf_end(_dwarf_v);
+    utility::filter_sort_unique(_line_info);
+    utility::filter_sort_unique(_ranges);
+
+    return _line_info;
+}
+
+template <typename ArchiveT>
+void
+dwarf_entry::serialize(ArchiveT& ar, const unsigned int)
+{
+#define OMNITRACE_SERIALIZE_MEMBER(MEMBER) ar(::tim::cereal::make_nvp(#MEMBER, MEMBER));
+
+    OMNITRACE_SERIALIZE_MEMBER(file)
+    OMNITRACE_SERIALIZE_MEMBER(line)
+    OMNITRACE_SERIALIZE_MEMBER(col)
+    OMNITRACE_SERIALIZE_MEMBER(address)
+    OMNITRACE_SERIALIZE_MEMBER(discriminator)
+    // OMNITRACE_SERIALIZE_MEMBER(begin_statement)
+    // OMNITRACE_SERIALIZE_MEMBER(end_sequence)
+    // OMNITRACE_SERIALIZE_MEMBER(line_block)
+    // OMNITRACE_SERIALIZE_MEMBER(prologue_end)
+    // OMNITRACE_SERIALIZE_MEMBER(epilogue_begin)
+    // OMNITRACE_SERIALIZE_MEMBER(vliw_op_index)
+    // OMNITRACE_SERIALIZE_MEMBER(isa)
+}
+
+template void
+dwarf_entry::serialize<cereal::JSONInputArchive>(cereal::JSONInputArchive&,
+                                                 const unsigned int);
+
+template void
+dwarf_entry::serialize<cereal::MinimalJSONOutputArchive>(
+    cereal::MinimalJSONOutputArchive&, const unsigned int);
+
+template void
+dwarf_entry::serialize<cereal::PrettyJSONOutputArchive>(cereal::PrettyJSONOutputArchive&,
+                                                        const unsigned int);
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,62 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/binary/address_range.hpp"
+#include "library/binary/fwd.hpp"
+
+namespace omnitrace
+{
+namespace binary
+{
+struct dwarf_entry
+{
+    TIMEMORY_DEFAULT_OBJECT(dwarf_entry)
+
+    bool          begin_statement = false;
+    bool          end_sequence    = false;
+    bool          line_block      = false;
+    bool          prologue_end    = false;
+    bool          epilogue_begin  = false;
+    unsigned int  line            = 0;
+    int           col             = 0;
+    unsigned int  vliw_op_index   = 0;
+    unsigned int  isa             = 0;
+    unsigned int  discriminator   = 0;
+    address_range address         = { 0, 0 };
+    std::string   file            = {};
+
+    bool is_valid() const;
+
+    bool     operator<(const dwarf_entry&) const;
+    bool     operator==(const dwarf_entry&) const;
+    bool     operator!=(const dwarf_entry&) const;
+    explicit operator bool() const { return is_valid(); }
+
+    static std::deque<dwarf_entry> process_dwarf(int _fd, std::vector<address_range>&);
+
+    template <typename ArchiveT>
+    void serialize(ArchiveT&, const unsigned int);
+};
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,62 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "common/defines.h"
+#include "library/common.hpp"
+#include "library/defines.hpp"
+#include "library/exception.hpp"
+
+#include <timemory/hash/types.hpp>
+#include <timemory/mpl/concepts.hpp>
+#include <timemory/tpls/cereal/cereal/cereal.hpp>
+#include <timemory/unwind/bfd.hpp>
+#include <timemory/unwind/types.hpp>
+#include <timemory/utility/procfs/maps.hpp>
+
+#include <cstdint>
+#include <deque>
+#include <map>
+#include <memory>
+#include <regex>
+#include <string>
+#include <tuple>
+#include <variant>
+
+namespace omnitrace
+{
+namespace binary
+{
+namespace procfs = ::tim::procfs;  // NOLINT
+
+using bfd_file     = ::tim::unwind::bfd_file;
+using hash_value_t = ::tim::hash_value_t;
+
+struct address_range;
+struct address_multirange;
+struct scope_filter;
+struct symbol;
+struct dwarf_entry;
+struct binary_info;
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,133 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/binary/link_map.hpp"
+#include "library/common.hpp"
+#include "library/config.hpp"
+#include "library/debug.hpp"
+#include "library/timemory.hpp"
+
+#include <timemory/utility/filepath.hpp>
+
+#include <cstdint>
+#include <dlfcn.h>
+#include <link.h>
+#include <set>
+#include <string>
+#include <string_view>
+
+namespace omnitrace
+{
+namespace binary
+{
+std::set<link_file>
+get_link_map(const char* _lib, const std::string& _exclude_linked_by,
+             const std::string& _exclude_re)
+{
+    auto _get_chain = [](const char* _name) {
+        void* _handle = dlopen(_name, RTLD_LAZY | RTLD_NOLOAD);
+        auto  _chain  = std::set<std::string>{};
+        if(_handle)
+        {
+            struct link_map* _link_map = nullptr;
+            dlinfo(_handle, RTLD_DI_LINKMAP, &_link_map);
+            struct link_map* _next = _link_map;
+            while(_next)
+            {
+                if(_name == nullptr && _next == _link_map &&
+                   std::string_view{ _next->l_name }.empty())
+                {
+                    // only insert exe name if dlopened the exe and
+                    // empty name is first entry
+                    _chain.emplace(config::get_exe_realpath());
+                }
+                else if(!std::string_view{ _next->l_name }.empty())
+                {
+                    _chain.emplace(_next->l_name);
+                }
+                _next = _next->l_next;
+            }
+        }
+        return _chain;
+    };
+
+    auto _full_chain = _get_chain(_lib);
+    auto _excl_chain = (_exclude_linked_by.empty())
+                           ? std::set<std::string>{}
+                           : _get_chain(_exclude_linked_by.c_str());
+    auto _fini_chain = std::set<link_file>{};
+
+    for(const auto& itr : _full_chain)
+    {
+        if(_excl_chain.find(itr) == _excl_chain.end())
+        {
+            if(_exclude_re.empty() || !std::regex_search(itr, std::regex{ _exclude_re }))
+                _fini_chain.emplace(itr);
+            else
+                _excl_chain.emplace(itr);
+        }
+    }
+
+    auto _name = (!_lib) ? config::get_exe_realpath() : std::string{ _lib };
+    for(const auto& itr : _fini_chain)
+    {
+        OMNITRACE_VERBOSE(2, "[linkmap][%s]: %s\n", filepath::basename(_name),
+                          itr.real().c_str());
+    }
+
+    for(const auto& itr : _excl_chain)
+    {
+        OMNITRACE_VERBOSE(3, "[linkmap][%s]: %s\n", _exclude_linked_by.c_str(),
+                          link_file{ itr }.real().c_str());
+    }
+
+    return _fini_chain;
+}
+
+bool
+link_file::operator<(const link_file& _rhs) const
+{
+    if(name == _rhs.name) return false;
+
+    auto _lhs_base = base();
+    auto _lhs_real = real();
+    auto _rhs_base = _rhs.base();
+    auto _rhs_real = _rhs.real();
+
+    if(_lhs_base == _rhs_base || _lhs_real == _rhs_real) return false;
+
+    return (_lhs_real < _rhs_real);
+}
+
+std::string_view
+link_file::base() const
+{
+    return std::string_view{ filepath::basename(name) };
+}
+
+std::string
+link_file::real() const
+{
+    return filepath::realpath(name, nullptr, false);
+}
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,54 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include <cstdint>
+#include <set>
+#include <string>
+#include <string_view>
+
+namespace omnitrace
+{
+namespace binary
+{
+struct link_file
+{
+    link_file(std::string_view&& _v)
+    : name{ _v }
+    {}
+
+    std::string_view base() const;
+    std::string      real() const;
+    bool             operator<(const link_file&) const;
+
+    std::string name = {};
+};
+
+// default parameters: get the linked binaries for the exe but exclude the linked binaries
+// from libomnitrace
+std::set<link_file>
+get_link_map(const char*        _lib               = nullptr,
+             const std::string& _exclude_linked_by = "libomnitrace.so",
+             const std::string& _exclude_re        = "libomnitrace-([a-zA-Z]+)\\.so");
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,46 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/binary/scope_filter.hpp"
+#include "library/exception.hpp"
+
+#include <regex>
+
+namespace omnitrace
+{
+namespace binary
+{
+bool
+scope_filter::operator()(std::string_view _value) const
+{
+    if(mode == FILTER_INCLUDE)
+        return (expression.empty())
+                   ? true
+                   : std::regex_search(_value.data(), std::regex{ expression });
+    else if(mode == FILTER_EXCLUDE)
+        return (expression.empty())
+                   ? false
+                   : !std::regex_search(_value.data(), std::regex{ expression });
+    throw exception<std::runtime_error>{ "invalid scope filter mode" };
+}
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,75 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/defines.hpp"
+
+#include <cstdint>
+#include <string>
+
+namespace omnitrace
+{
+namespace binary
+{
+struct scope_filter
+{
+    enum filter_mode : uint8_t
+    {
+        FILTER_INCLUDE = 0,
+        FILTER_EXCLUDE
+    };
+
+    enum filter_scope : uint8_t
+    {
+        UNIVERSAL_FILTER = (1 << 0),
+        BINARY_FILTER    = (1 << 1),
+        SOURCE_FILTER    = (1 << 2),
+        FUNCTION_FILTER  = (1 << 3)
+    };
+
+    filter_mode  mode       = FILTER_INCLUDE;
+    filter_scope scope      = UNIVERSAL_FILTER;
+    std::string  expression = {};
+
+    bool operator()(std::string_view _value) const;
+
+    template <typename ContainerT>
+    static bool satisfies_filter(const ContainerT&, filter_scope,
+                                 std::string_view) OMNITRACE_PURE;
+};
+
+template <typename ContainerT>
+inline bool
+scope_filter::satisfies_filter(const ContainerT& _filters, filter_scope _scope,
+                               std::string_view _value)
+{
+    for(const auto& itr : _filters)  // NOLINT
+    {
+        // if the filter is for the specified scope and itr does not satisfy the
+        // include/exclude mode, return false
+        if((itr.scope & _scope) > 0 && !itr(_value)) return false;
+    }
+    return true;
+}
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,356 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/config.hpp"
+#include "library/debug.hpp"
+
+#if !defined(TIMEMORY_USE_BFD)
+#    error "BFD support not enabled"
+#endif
+
+#define PACKAGE     "omnitrace"
+#define L_LNNO_SIZE 4
+
+#include <bfd.h>
+#include <coff/external.h>
+#include <coff/internal.h>
+#include <cstddef>
+#include <cstdio>
+#include <dwarf.h>
+#include <elf-bfd.h>
+#include <elfutils/libdw.h>
+#include <libcoff.h>
+
+#include "library/binary/dwarf_entry.hpp"
+#include "library/binary/fwd.hpp"
+#include "library/binary/scope_filter.hpp"
+#include "library/binary/symbol.hpp"
+#include "library/timemory.hpp"
+#include "library/utility.hpp"
+
+#include <timemory/mpl/concepts.hpp>
+
+namespace omnitrace
+{
+namespace binary
+{
+namespace
+{
+std::vector<inlined_symbol>
+read_inliner_info(bfd* _inp)
+{
+    auto _data = std::vector<inlined_symbol>{};
+    while(true)
+    {
+        const char*  _file = nullptr;
+        const char*  _func = nullptr;
+        unsigned int _line = 0;
+        if(bfd_find_inliner_info(_inp, &_file, &_func, &_line) != 0)
+        {
+            if(_file && _func && _line > 0)
+                _data.emplace_back(inlined_symbol{
+                    _line, filepath::realpath(_file, nullptr, false), _func });
+        }
+        else
+        {
+            break;
+        }
+    }
+
+    return _data;
+}
+}  // namespace
+
+symbol::symbol(const base_type& _v)
+: base_type{ _v }
+, address{ _v.address, _v.address + _v.symsize }
+{}
+
+bool
+symbol::operator==(const symbol& _rhs) const
+{
+    return std::tie(address, base_type::name) ==
+           std::tie(_rhs.address, _rhs.base_type::name);
+}
+
+bool
+symbol::operator<(const symbol& _rhs) const
+{
+    return std::tie(address, base_type::binding, base_type::visibility, base_type::name) <
+           std::tie(_rhs.address, _rhs.base_type::binding, base_type::visibility,
+                    base_type::name);
+}
+
+bool
+symbol::operator()(const std::vector<scope_filter>& _filters) const
+{
+    using sf = scope_filter;
+
+    // apply filters to the main symbol
+    return (sf::satisfies_filter(_filters, sf::FUNCTION_FILTER, demangle(func)) &&
+            (sf::satisfies_filter(_filters, sf::SOURCE_FILTER, file) ||
+             sf::satisfies_filter(_filters, sf::SOURCE_FILTER, join(':', file, line))));
+}
+
+symbol&
+symbol::operator+=(const symbol& _rhs)
+{
+    if(address.contiguous_with(_rhs.address) &&
+       std::tie(line, load_address, func, file) ==
+           std::tie(_rhs.line, _rhs.load_address, _rhs.func, _rhs.file))
+    {
+        address += _rhs.address;
+        utility::combine(inlines, _rhs.inlines);
+        utility::combine(dwarf_info, _rhs.dwarf_info);
+    }
+    else
+    {
+        throw exception<std::runtime_error>("incompatible symbol+=");
+    }
+
+    return *this;
+}
+
+symbol::operator bool() const
+{
+    return address.is_valid() && (file.length() + func.length() + line) > 0;
+}
+
+size_t
+symbol::read_dwarf(const std::deque<dwarf_entry>& _info)
+{
+    for(const auto& itr : _info)
+    {
+        if(address.contains(itr.address)) dwarf_info.emplace_back(itr);
+    }
+
+    // make sure the dwarf info is sorted by address (low to high)
+    std::sort(dwarf_info.begin(), dwarf_info.end(),
+              [](const dwarf_entry& _lhs, const dwarf_entry& _rhs) {
+                  return _lhs.address < _rhs.address;
+              });
+
+    // helper for getting the end address
+    auto _get_next_address = [&](auto nitr, uintptr_t _low) {
+        while(++nitr != dwarf_info.end())
+        {
+            if(nitr->address.low > _low)
+            {
+                return nitr->address.low;
+            }
+        }
+        // return the end address of the symbol
+        return address.high;
+    };
+    // convert the single addresses into ranges
+    for(auto itr = dwarf_info.begin(); itr != dwarf_info.end(); ++itr)
+    {
+        // if address is already a range, do not update it
+        if(!itr->address.is_range())
+            itr->address = address_range{ itr->address.low,
+                                          _get_next_address(itr, itr->address.low) };
+    }
+
+    return dwarf_info.size();
+}
+
+bool
+symbol::read_bfd(bfd_file& _bfd)
+{
+    auto*         _section = static_cast<asection*>(section);
+    bfd_vma       _vma     = bfd_section_vma(_section);
+    bfd_size_type _size    = bfd_section_size(_section);
+
+    auto& _pc     = address.low;
+    auto& _pc_end = address.high;
+
+    if(_pc < _vma || _pc >= _vma + _size) return false;
+    // add one to vma + size because address range is exclusive of last address
+    if(_pc_end > _vma + _size) _pc_end = (_vma + _size);
+
+    auto* _inp  = static_cast<bfd*>(_bfd.data);
+    auto* _syms = reinterpret_cast<asymbol**>(_bfd.syms);
+
+    {
+        const char*  _file          = nullptr;
+        const char*  _func          = nullptr;
+        unsigned int _line          = 0;
+        unsigned int _discriminator = 0;
+
+        // if(bfd_find_nearest_line(_inp, _section, _syms, _pc - _vma, &_file,
+        //                         &_func, &_line) != 0)
+        if(bfd_find_nearest_line_discriminator(_inp, _section, _syms, _pc - _vma, &_file,
+                                               &_func, &_line, &_discriminator) != 0)
+        {
+            if(_file) file = _file;
+            if(_func) func = _func;
+            if(_file && strnlen(_file, 1) > 0)
+                file = _file;
+            else if(!_file || strnlen(_file, 1) == 0)
+                file = bfd_get_filename(_inp);
+            if(!func.empty())
+            {
+                file    = filepath::realpath(file, nullptr, false);
+                line    = _line;
+                inlines = read_inliner_info(_inp);
+                return true;
+            }
+        }
+    }
+
+    return false;
+}
+
+symbol
+symbol::clone() const
+{
+    auto _sym         = symbol{ static_cast<base_type>(*this) };
+    _sym.line         = line;
+    _sym.load_address = load_address;
+    _sym.address      = address;
+    _sym.func         = func;
+    _sym.file         = file;
+
+    return _sym;
+}
+
+template <typename Tp>
+Tp
+symbol::get_inline_symbols(const std::vector<scope_filter>& _filters) const
+{
+    using sf         = scope_filter;
+    using value_type = typename Tp::value_type;
+
+    auto _data = Tp{};
+
+    for(const auto& itr : inlines)
+    {
+        if(sf::satisfies_filter(_filters, sf::FUNCTION_FILTER, demangle(itr.func)) &&
+           (sf::satisfies_filter(_filters, sf::SOURCE_FILTER, itr.file) ||
+            sf::satisfies_filter(_filters, sf::SOURCE_FILTER,
+                                 join(':', itr.file, itr.line))))
+        {
+            if constexpr(concepts::is_unqualified_same<value_type, symbol>::value)
+            {
+                auto _sym = clone();
+                _sym.func = itr.func;
+                _sym.line = itr.line;
+                _sym.file = itr.file;
+                _data.emplace_back(_sym);
+            }
+            else if constexpr(concepts::is_unqualified_same<value_type,
+                                                            inlined_symbol>::value)
+            {
+                _data.emplace_back(itr);
+            }
+        }
+    }
+
+    return _data;
+}
+
+template <typename Tp>
+Tp
+symbol::get_debug_line_info(const std::vector<scope_filter>& _filters) const
+{
+    using sf         = scope_filter;
+    using value_type = typename Tp::value_type;
+
+    auto _data = Tp{};
+
+    if(sf::satisfies_filter(_filters, sf::FUNCTION_FILTER, demangle(func)))
+    {
+        for(const auto& itr : dwarf_info)
+        {
+            if(sf::satisfies_filter(_filters, sf::SOURCE_FILTER, itr.file) ||
+               sf::satisfies_filter(_filters, sf::SOURCE_FILTER,
+                                    join(':', itr.file, itr.line)))
+            {
+                if constexpr(concepts::is_unqualified_same<value_type, symbol>::value)
+                {
+                    auto _sym    = clone();
+                    _sym.address = itr.address;
+                    _sym.file    = itr.file;
+                    _sym.line    = itr.line;
+                    _data.emplace_back(_sym);
+                }
+                else if constexpr(concepts::is_unqualified_same<value_type,
+                                                                dwarf_entry>::value)
+                {
+                    _data.emplace_back(itr);
+                }
+            }
+        }
+    }
+
+    return _data;
+}
+
+template <typename ArchiveT>
+void
+inlined_symbol::serialize(ArchiveT& ar, const unsigned int)
+{
+    using ::tim::cereal::make_nvp;
+    ar(make_nvp("func", func), make_nvp("file", file), make_nvp("line", line));
+}
+
+template <typename ArchiveT>
+void
+symbol::serialize(ArchiveT& ar, const unsigned int)
+{
+    using ::tim::cereal::make_nvp;
+    ar(make_nvp("address", address), make_nvp("load_address", load_address),
+       make_nvp("line", line), make_nvp("func", func), make_nvp("file", file),
+       make_nvp("inlines", inlines), make_nvp("dwarf_info", dwarf_info));
+    if constexpr(concepts::is_output_archive<ArchiveT>::value)
+        ar(cereal::make_nvp("dfunc", demangle(func)));
+}
+
+template void
+symbol::serialize<cereal::JSONInputArchive>(cereal::JSONInputArchive&,
+                                            const unsigned int);
+
+template void
+symbol::serialize<cereal::MinimalJSONOutputArchive>(cereal::MinimalJSONOutputArchive&,
+                                                    const unsigned int);
+
+template void
+symbol::serialize<cereal::PrettyJSONOutputArchive>(cereal::PrettyJSONOutputArchive&,
+                                                   const unsigned int);
+
+template std::deque<symbol>
+symbol::get_inline_symbols<std::deque<symbol>>(
+    const std::vector<scope_filter>& _filters) const;
+
+template std::vector<inlined_symbol>
+symbol::get_inline_symbols<std::vector<inlined_symbol>>(
+    const std::vector<scope_filter>& _filters) const;
+
+template std::deque<symbol>
+symbol::get_debug_line_info<std::deque<symbol>>(
+    const std::vector<scope_filter>& _filters) const;
+
+template std::vector<dwarf_entry>
+symbol::get_debug_line_info<std::vector<dwarf_entry>>(
+    const std::vector<scope_filter>& _filters) const;
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,96 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/binary/address_range.hpp"
+#include "library/binary/fwd.hpp"
+
+#include <timemory/unwind/bfd.hpp>
+
+#include <cstdint>
+#include <deque>
+#include <string_view>
+#include <vector>
+
+namespace omnitrace
+{
+namespace binary
+{
+struct inlined_symbol
+{
+    unsigned int line = 0;
+    std::string  file = {};
+    std::string  func = {};
+
+    template <typename ArchiveT>
+    void serialize(ArchiveT&, const unsigned int);
+};
+
+struct symbol : private tim::unwind::bfd_file::symbol
+{
+    using base_type = tim::unwind::bfd_file::symbol;
+
+    symbol() = default;
+    symbol(const base_type& _v);
+
+    ~symbol()                 = default;
+    symbol(const symbol&)     = default;
+    symbol(symbol&&) noexcept = default;
+
+    symbol& operator=(const symbol&) = default;
+    symbol& operator=(symbol&&) noexcept = default;
+
+    bool     operator==(const symbol&) const;
+    bool     operator<(const symbol&) const;
+    bool     operator()(const std::vector<scope_filter>&) const;
+    symbol&  operator+=(const symbol&);
+    explicit operator bool() const;
+
+    bool          read_bfd(bfd_file&);
+    size_t        read_dwarf(const std::deque<dwarf_entry>&);
+    address_range ipaddr() const { return address + load_address; }
+    symbol        clone() const;
+
+    template <typename Tp = std::deque<symbol>>
+    Tp get_inline_symbols(const std::vector<scope_filter>&) const;
+
+    template <typename Tp = std::deque<symbol>>
+    Tp get_debug_line_info(const std::vector<scope_filter>&) const;
+
+    template <typename ArchiveT>
+    void serialize(ArchiveT&, const unsigned int);
+
+    using base_type::binding;
+    using base_type::section;
+    using base_type::visibility;
+
+    unsigned int                line         = 0;
+    uintptr_t                   load_address = 0;
+    address_range               address      = {};
+    std::string                 func         = {};
+    std::string                 file         = {};
+    std::vector<inlined_symbol> inlines      = {};
+    std::vector<dwarf_entry>    dwarf_info   = {};
+};
+}  // namespace binary
+}  // namespace omnitrace
@@ -0,0 +1,22 @@
+#
+set(causal_sources
+    ${CMAKE_CURRENT_LIST_DIR}/data.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/delay.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/experiment.cpp
+    # ${CMAKE_CURRENT_LIST_DIR}/perf.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/sample_data.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/sampling.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/selected_entry.cpp)
+
+set(causal_headers
+    ${CMAKE_CURRENT_LIST_DIR}/data.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/delay.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/experiment.hpp
+    # ${CMAKE_CURRENT_LIST_DIR}/perf.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/sample_data.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/sampling.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/selected_entry.hpp)
+
+target_sources(omnitrace-object-library PRIVATE ${causal_sources} ${causal_headers})
+
+add_subdirectory(components)
@@ -0,0 +1,16 @@
+#
+set(component_sources
+    ${CMAKE_CURRENT_LIST_DIR}/backtrace.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/blocking_gotcha.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/causal_gotcha.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/progress_point.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/unblocking_gotcha.cpp)
+
+set(component_headers
+    ${CMAKE_CURRENT_LIST_DIR}/backtrace.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/blocking_gotcha.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/causal_gotcha.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/progress_point.hpp
+    ${CMAKE_CURRENT_LIST_DIR}/unblocking_gotcha.hpp)
+
+target_sources(omnitrace-object-library PRIVATE ${component_sources} ${component_headers})
@@ -0,0 +1,227 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/causal/components/backtrace.hpp"
+#include "library/causal/data.hpp"
+#include "library/causal/delay.hpp"
+#include "library/causal/experiment.hpp"
+#include "library/concepts.hpp"
+#include "library/config.hpp"
+#include "library/debug.hpp"
+#include "library/runtime.hpp"
+#include "library/state.hpp"
+#include "library/thread_data.hpp"
+#include "library/thread_info.hpp"
+#include "library/tracing.hpp"
+#include "library/utility.hpp"
+
+#include <timemory/components/timing/backends.hpp>
+#include <timemory/components/timing/wall_clock.hpp>
+#include <timemory/mpl/concepts.hpp>
+#include <timemory/mpl/types.hpp>
+#include <timemory/process/threading.hpp>
+#include <timemory/units.hpp>
+#include <timemory/utility/backtrace.hpp>
+
+#include <atomic>
+#include <ctime>
+#include <type_traits>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace component
+{
+namespace
+{
+using ::tim::backtrace::get_unw_signal_frame_stack_raw;
+
+auto&
+get_delay_statistics()
+{
+    using thread_data_t =
+        thread_data<identity<tim::statistics<int64_t>>, category::sampling>;
+
+    static_assert(
+        use_placement_new_when_generating_unique_ptr<thread_data_t>::value,
+        "delay statistics thread data should use placement new to allocate unique_ptr");
+
+    static auto& _v = thread_data_t::instance(construct_on_init{});
+    return _v;
+}
+
+auto&
+get_in_use()
+{
+    using thread_data_t = thread_data<identity<bool>, category::sampling>;
+
+    static_assert(
+        use_placement_new_when_generating_unique_ptr<thread_data_t>::value,
+        "sampling is_use thread data should use placement new to allocate unique_ptr");
+
+    static auto& _v = thread_data_t::instance(construct_on_init{});
+    return _v;
+}
+
+struct scoped_in_use
+{
+    scoped_in_use(int64_t _tid = utility::get_thread_index())
+    : value{ get_in_use()->at(_tid) }
+    {
+        value = true;
+    }
+    ~scoped_in_use() { value = false; }
+
+    bool& value;
+};
+
+auto
+is_in_use(int64_t _tid = threading::get_id())
+{
+    return get_in_use()->at(_tid);
+}
+
+}  // namespace
+
+void
+backtrace::start()
+{
+    // do not delete these lines. The thread data needs to be allocated
+    // before it is called in sampler or else a deadlock will occur when
+    // the sample interrupts a malloc call
+    (void) get_delay_statistics();
+    (void) get_in_use();
+}
+
+void
+backtrace::stop()
+{}
+
+void
+backtrace::sample(int _sig)
+{
+    constexpr size_t  depth        = ::omnitrace::causal::unwind_depth;
+    constexpr int64_t ignore_depth = ::omnitrace::causal::unwind_offset;
+
+    // update the last sample for backtrace signal(s) even when in use
+    static thread_local int64_t _last_sample = 0;
+
+    if(is_in_use())
+    {
+        if(_sig == get_realtime_signal()) _last_sample = tracing::now();
+        return;
+    }
+    scoped_in_use _in_use{};
+
+    m_index = causal::experiment::get_index();
+    m_stack = get_unw_signal_frame_stack_raw<depth, ignore_depth>();
+
+    // the batch handler timer delivers a signal according to the thread CPU
+    // clock, ensuring that setting the current selection and processing the
+    // delays only happens when the thread is active
+    if(_sig == get_cputime_signal())
+    {
+        if(!causal::experiment::is_active())
+            causal::set_current_selection(m_stack);
+        else
+            causal::delay::process();
+    }
+    else if(_sig == get_realtime_signal())
+    {
+        auto  _this_sample = tracing::now();
+        auto& _period_stat = get_delay_statistics()->at(threading::get_id());
+        if(_last_sample > 0) _period_stat += (_this_sample - _last_sample);
+        _last_sample = _this_sample;
+
+        if(causal::experiment::is_active() && causal::experiment::is_selected(m_stack))
+        {
+            m_selected = true;
+            causal::experiment::add_selected();
+            // compute the delay time based on the rate of taking samples,
+            // unless we have taken less than 10, in which case, we just
+            // use the pre-computed value.
+            auto _delay =
+                (_period_stat.get_count() < 10)
+                    ? causal::experiment::get_delay()
+                    : (_period_stat.get_mean() * causal::experiment::get_delay_scaling());
+            causal::delay::get_local() += _delay;
+        }
+    }
+    else
+    {
+        OMNITRACE_THROW("unhandled signal %i\n", _sig);
+    }
+}
+
+template <typename Tp>
+Tp
+backtrace::get_period(uint64_t _units)
+{
+    using cast_type = std::conditional_t<std::is_floating_point<Tp>::value, Tp, double>;
+
+    double _realtime_freq =
+        (get_use_sampling_realtime()) ? get_sampling_real_freq() : 0.0;
+    double _cputime_freq = (get_use_sampling_cputime()) ? get_sampling_cpu_freq() : 0.0;
+
+    auto    _freq        = std::max<double>(_realtime_freq, _cputime_freq);
+    double  _period      = 1.0 / _freq;
+    int64_t _period_nsec = static_cast<int64_t>(_period * units::sec) % units::sec;
+    return static_cast<Tp>(_period_nsec) / static_cast<cast_type>(_units);
+}
+
+tim::statistics<int64_t>
+backtrace::get_period_stats()
+{
+    scoped_in_use _in_use{};
+    auto          _data = tim::statistics<int64_t>{};
+    if(!get_delay_statistics()) return _data;
+    for(size_t i = 0; i < get_delay_statistics()->size(); ++i)
+    {
+        scoped_in_use _thr_in_use{ static_cast<int64_t>(i) };
+        const auto&   itr = get_delay_statistics()->at(i);
+        if(itr.get_count() > 1) _data += itr;
+    }
+    return _data;
+}
+
+void
+backtrace::reset_period_stats()
+{
+    scoped_in_use _in_use{};
+    for(size_t i = 0; i < get_delay_statistics()->size(); ++i)
+    {
+        scoped_in_use _thr_in_use{ static_cast<int64_t>(i) };
+        get_delay_statistics()->at(i).reset();
+    }
+}
+}  // namespace component
+}  // namespace causal
+}  // namespace omnitrace
+
+#define INSTANTIATE_BT_CAUSAL_PERIOD(TYPE)                                               \
+    template TYPE omnitrace::causal::component::backtrace::get_period<TYPE>(uint64_t);
+
+INSTANTIATE_BT_CAUSAL_PERIOD(float)
+INSTANTIATE_BT_CAUSAL_PERIOD(double)
+INSTANTIATE_BT_CAUSAL_PERIOD(int64_t)
+INSTANTIATE_BT_CAUSAL_PERIOD(uint64_t)
@@ -0,0 +1,91 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/causal/data.hpp"
+#include "library/causal/sample_data.hpp"
+#include "library/common.hpp"
+#include "library/components/fwd.hpp"
+#include "library/defines.hpp"
+#include "library/timemory.hpp"
+
+#include <timemory/components/base.hpp>
+#include <timemory/macros/language.hpp>
+#include <timemory/mpl/concepts.hpp>
+#include <timemory/tpls/cereal/cereal/cereal.hpp>
+#include <timemory/units.hpp>
+#include <timemory/utility/unwind.hpp>
+
+#include <chrono>
+#include <cstdint>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace component
+{
+struct backtrace
+: tim::component::empty_base
+, tim::concepts::component
+{
+    using value_type        = void;
+    using sample_data_set_t = std::set<sample_data>;
+
+    static std::string label() { return "causal::backtrace"; }
+    static std::string description()
+    {
+        return "Causal profiling data collected in backtrace";
+    }
+
+    backtrace()                     = default;
+    ~backtrace()                    = default;
+    backtrace(const backtrace&)     = default;
+    backtrace(backtrace&&) noexcept = default;
+
+    backtrace& operator=(const backtrace&) = default;
+    backtrace& operator=(backtrace&&) noexcept = default;
+
+    static void start();
+    static void stop();
+
+    void sample(int = -1);
+
+    auto get_selected() const { return m_selected; }
+    auto get_index() const { return m_index; }
+    auto get_stack() const { return m_stack; }
+
+    template <typename Tp = uint64_t>
+    static Tp get_period(uint64_t _units = units::nsec);
+
+    static tim::statistics<int64_t> get_period_stats();
+    static void                     reset_period_stats();
+
+private:
+    bool                  m_selected = false;
+    uint32_t              m_index    = 0;
+    causal::unwind_addr_t m_stack    = {};
+};
+}  // namespace component
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,154 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/causal/components/blocking_gotcha.hpp"
+#include "library/causal/delay.hpp"
+#include "library/causal/experiment.hpp"
+#include "library/config.hpp"
+#include "library/debug.hpp"
+#include "library/runtime.hpp"
+#include "library/state.hpp"
+
+#include <timemory/components/macros.hpp>
+#include <timemory/hash/types.hpp>
+
+#include <csignal>
+#include <cstdint>
+#include <pthread.h>
+#include <stdexcept>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace component
+{
+std::string
+blocking_gotcha::label()
+{
+    return "causal_blocking_gotcha";
+}
+
+std::string
+blocking_gotcha::description()
+{
+    return "Handles executing all necessary pauses before the thread performs some "
+           "blocking function";
+}
+
+void
+blocking_gotcha::preinit()
+{
+    configure();
+}
+
+void
+blocking_gotcha::configure()
+{
+    blocking_gotcha_t::get_initializer() = []() {
+        if(!config::get_use_causal()) return;
+
+        blocking_gotcha_t::configure(
+            comp::gotcha_config<0, int, pthread_mutex_t*>{ "pthread_mutex_lock" });
+
+        blocking_gotcha_t::configure(
+            comp::gotcha_config<1, int, pthread_mutex_t*>{ "pthread_mutex_trylock" });
+
+        blocking_gotcha_t::configure(
+            comp::gotcha_config<2, int, pthread_rwlock_t*>{ "pthread_rwlock_wrlock" });
+
+        blocking_gotcha_t::configure(
+            comp::gotcha_config<3, int, pthread_rwlock_t*>{ "pthread_rwlock_trywrlock" });
+
+        blocking_gotcha_t::configure(
+            comp::gotcha_config<4, int, pthread_spinlock_t*>{ "pthread_spin_lock" });
+
+        blocking_gotcha_t::configure(
+            comp::gotcha_config<5, int, pthread_spinlock_t*>{ "pthread_spin_trylock" });
+
+        blocking_gotcha_t::configure(
+            comp::gotcha_config<6, int, pthread_t, void**>{ "pthread_join" });
+
+        blocking_gotcha_t::configure(
+            comp::gotcha_config<7, int, pthread_cond_t*, pthread_mutex_t*>{
+                "pthread_cond_wait" });
+
+        blocking_gotcha_t::configure(
+            comp::gotcha_config<8, int, pthread_cond_t*, pthread_mutex_t*,
+                                const struct timespec*>{ "pthread_cond_timedwait" });
+
+        blocking_gotcha_t::configure(
+            comp::gotcha_config<9, int, const sigset_t*, int*>{ "sigwait" });
+
+        blocking_gotcha_t::configure(
+            comp::gotcha_config<10, int, const sigset_t*, int*, siginfo_t*>{
+                "sigwaitinfo" });
+
+        blocking_gotcha_t::configure(
+            comp::gotcha_config<11, int, const sigset_t*, int*, siginfo_t*,
+                                const struct timespec*>{ "sigtimedwait" });
+
+        blocking_gotcha_t::configure(
+            comp::gotcha_config<12, int, const sigset_t*>{ "sigsuspend" });
+    };
+}
+
+void
+blocking_gotcha::shutdown()
+{
+    blocking_gotcha_t::disable();
+}
+
+void
+blocking_gotcha::start()
+{
+    if(causal::experiment::is_active() &&
+       get_thread_state() == ::omnitrace::ThreadState::Enabled && delay_value == 0)
+        delay_value = causal::delay::get_global().load();
+}
+
+void
+blocking_gotcha::audit(const comp::gotcha_data& _data, audit::outgoing, int _ret)
+{
+    // if one of the try/timed functions did not succeed, reset the delay value to zero
+    if(_ret != 0 && _ret != ETIMEDOUT &&
+       std::set<size_t>{ 1, 3, 5, 8, 11 }.count(_data.index) > 0)
+    {
+        delay_value = 0;
+    }
+}
+
+void
+blocking_gotcha::stop()
+{
+    if(delay_value > 0 && causal::experiment::is_active() &&
+       get_thread_state() == ::omnitrace::ThreadState::Enabled)
+    {
+        causal::delay::postblock(delay_value);
+        delay_value = 0;
+    }
+}
+}  // namespace component
+}  // namespace causal
+}  // namespace omnitrace
+
+TIMEMORY_INVOKE_PREINIT(omnitrace::causal::component::blocking_gotcha)
@@ -0,0 +1,76 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/common.hpp"
+#include "library/defines.hpp"
+#include "library/timemory.hpp"
+
+#include <timemory/components/gotcha/backends.hpp>
+#include <timemory/mpl/macros.hpp>
+#include <timemory/utility/types.hpp>
+
+#include <array>
+#include <cstddef>
+#include <string>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace component
+{
+using timespec_t = struct timespec;
+// this is used to wrap pthread_mutex()
+struct blocking_gotcha : comp::base<blocking_gotcha, void>
+{
+    static constexpr size_t gotcha_capacity = 13;
+
+    TIMEMORY_DEFAULT_OBJECT(blocking_gotcha)
+
+    // string id for component
+    static std::string label();
+    static std::string description();
+    static void        preinit();
+
+    // generate the gotcha wrappers
+    static void configure();
+    static void shutdown();
+
+    void start();
+    void audit(const comp::gotcha_data&, audit::outgoing, int);
+    void stop();
+
+private:
+    int64_t delay_value = 0;
+};
+
+using blocking_gotcha_t =
+    comp::gotcha<blocking_gotcha::gotcha_capacity,
+                 tim::lightweight_tuple<blocking_gotcha>, category::causal>;
+}  // namespace component
+}  // namespace causal
+}  // namespace omnitrace
+
+OMNITRACE_DEFINE_CONCRETE_TRAIT(prevent_reentry, causal::component::blocking_gotcha_t,
+                                false_type)
@@ -0,0 +1,95 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/causal/components/causal_gotcha.hpp"
+#include "library/causal/components/blocking_gotcha.hpp"
+#include "library/causal/components/unblocking_gotcha.hpp"
+#include "library/config.hpp"
+
+#include <timemory/backends/threading.hpp>
+#include <timemory/utility/macros.hpp>
+#include <timemory/utility/types.hpp>
+
+#include <array>
+#include <vector>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace component
+{
+namespace
+{
+using bundle_t = tim::lightweight_tuple<blocking_gotcha_t, unblocking_gotcha_t>;
+
+auto&
+get_bundle()
+{
+    static auto _v = std::unique_ptr<bundle_t>{};
+    if(!_v) _v = std::make_unique<bundle_t>("causal_gotcha");
+    return _v;
+}
+
+bool is_configured = false;
+}  // namespace
+
+//--------------------------------------------------------------------------------------//
+
+void
+causal_gotcha::configure()
+{
+    if(!is_configured)
+    {
+        blocking_gotcha::configure();
+        unblocking_gotcha::configure();
+        is_configured = true;
+    }
+}
+
+void
+causal_gotcha::shutdown()
+{
+    if(is_configured)
+    {
+        blocking_gotcha::shutdown();
+        unblocking_gotcha::shutdown();
+        is_configured = false;
+    }
+}
+
+void
+causal_gotcha::start()
+{
+    configure();
+    get_bundle()->start();
+}
+
+void
+causal_gotcha::stop()
+{
+    get_bundle()->stop();
+    shutdown();
+}
+}  // namespace component
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,54 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/common.hpp"
+#include "library/defines.hpp"
+#include "library/timemory.hpp"
+
+#include <cstdint>
+#include <future>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace component
+{
+struct causal_gotcha : tim::component::base<causal_gotcha, void>
+{
+    TIMEMORY_DEFAULT_OBJECT(causal_gotcha)
+
+    // string id for component
+    static std::string label() { return "causal_gotcha"; }
+
+    // generate the gotcha wrappers
+    static void configure();
+    static void shutdown();
+
+    static void start();
+    static void stop();
+};
+}  // namespace component
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,241 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/causal/components/progress_point.hpp"
+#include "library/causal/experiment.hpp"
+#include "library/common.hpp"
+#include "library/concepts.hpp"
+#include "library/debug.hpp"
+#include "library/thread_data.hpp"
+#include "library/timemory.hpp"
+
+#include <timemory/hash/types.hpp>
+#include <timemory/mpl/type_traits.hpp>
+#include <timemory/units.hpp>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace component
+{
+namespace
+{
+using progress_allocator_t = tim::data::ring_buffer_allocator<progress_point>;
+using progress_map_t       = std::unordered_map<tim::hash_value_t, progress_point*>;
+
+auto&
+get_progress_map()
+{
+    using thread_data_t = thread_data<identity<progress_map_t>>;
+    static auto& _v     = thread_data_t::instance(construct_on_init{});
+    return _v;
+}
+
+progress_map_t&
+get_progress_map(int64_t _tid)
+{
+    return get_progress_map()->at(_tid);
+}
+
+auto&
+get_progress_allocator(int64_t _tid)
+{
+    static auto& _v = thread_data<progress_allocator_t>::instances(construct_on_init{});
+    return _v.at(_tid);
+}
+}  // namespace
+
+std::unordered_map<tim::hash_value_t, progress_point>
+progress_point::get_progress_points()
+{
+    auto _data = std::unordered_map<tim::hash_value_t, progress_point>{};
+    if(!get_progress_map()) return _data;
+    for(const auto& titr : *get_progress_map())
+    {
+        for(const auto& itr : titr)
+        {
+            if(itr.second)
+            {
+                auto& ditr = _data[itr.first];
+                ditr += *itr.second;
+                ditr.set_hash(itr.second->get_hash());
+                itr.second->set_value(0);
+            }
+        }
+    }
+    return _data;
+}
+
+std::string
+progress_point::label()
+{
+    return "progress_point";
+}
+
+std::string
+progress_point::description()
+{
+    return "Tracks progress point latency and throughput for causal profiling";
+}
+
+void
+progress_point::start()
+{
+    ++m_arrival;
+}
+
+void
+progress_point::stop()
+{
+    ++m_departure;
+}
+
+void
+progress_point::mark()
+{
+    ++m_delta;
+}
+
+void
+progress_point::set_value(int64_t _v)
+{
+    m_delta     = _v;
+    m_arrival   = _v;
+    m_departure = _v;
+}
+
+progress_point&
+progress_point::operator+=(const progress_point& _v)
+{
+    if(this != &_v)
+    {
+        m_delta += _v.m_delta;
+        m_arrival += _v.m_arrival;
+        m_departure += _v.m_departure;
+    }
+    return *this;
+}
+
+progress_point&
+progress_point::operator-=(const progress_point& _v)
+{
+    if(this != &_v)
+    {
+        m_delta -= _v.m_delta;
+        m_arrival -= _v.m_arrival;
+        m_departure -= _v.m_departure;
+    }
+    return *this;
+}
+
+bool
+progress_point::is_throughput_point() const
+{
+    return (m_delta != 0);
+}
+
+bool
+progress_point::is_latency_point() const
+{
+    return (m_arrival != 0 || m_departure != 0);
+}
+
+int64_t
+progress_point::get_delta() const
+{
+    return m_delta;
+}
+
+int64_t
+progress_point::get_arrival() const
+{
+    if(!is_latency_point()) return m_arrival;
+    // when it is a latency point, we want the difference to be greater than zero
+    return (m_arrival >= m_departure) ? (m_arrival + 1) : m_arrival;
+}
+
+int64_t
+progress_point::get_departure() const
+{
+    // if(!is_latency_point()) return m_departure;
+    // return (m_departure <= m_arrival) ? m_departure : (m_departure + 1);
+    return m_departure;
+}
+
+int64_t
+progress_point::get_latency_delta() const
+{
+    return (get_arrival() - get_departure());
+}
+
+int64_t
+progress_point::get_laps() const
+{
+    return std::max(get_delta(), get_latency_delta());
+}
+
+void
+progress_point::print(std::ostream& os) const
+{
+    os << tim::get_hash_identifier(m_hash) << " :: ";
+    tim::operation::base_printer<progress_point>(os, *this);
+}
+}  // namespace component
+}  // namespace causal
+}  // namespace omnitrace
+
+namespace tim
+{
+namespace operation
+{
+namespace causal = omnitrace::causal;
+
+void
+push_node<causal::component::progress_point>::operator()(type&        _obj, scope::config,
+                                                         hash_value_t _hash,
+                                                         int64_t      _tid) const
+{
+    auto itr = causal::component::get_progress_map(_tid).emplace(_hash, nullptr);
+    if(itr.second && !itr.first->second)
+    {
+        auto& _alloc = causal::component::get_progress_allocator(_tid);
+        auto* _val   = _alloc->allocate(1);
+        _alloc->construct(_val);
+        _val->set_hash(_hash);
+        itr.first->second = _val;
+    }
+    _obj.set_hash(_hash);
+    _obj.set_iterator(itr.first->second);
+}
+
+void
+pop_node<causal::component::progress_point>::operator()(type& _obj, int64_t) const
+{
+    auto* itr = _obj.get_iterator();
+    if(itr && !(_obj.get_is_invalid() || _obj.get_is_running()))
+    {
+        *itr += _obj;
+    }
+}
+}  // namespace operation
+}  // namespace tim
@@ -0,0 +1,157 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/common.hpp"
+#include "library/components/fwd.hpp"
+#include "library/defines.hpp"
+
+#include <timemory/components/base.hpp>
+#include <timemory/hash/types.hpp>
+#include <timemory/mpl/concepts.hpp>
+#include <timemory/mpl/type_traits.hpp>
+#include <timemory/tpls/cereal/cereal.hpp>
+#include <timemory/utility/types.hpp>
+
+#include <cstdint>
+#include <unordered_map>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace component
+{
+struct progress_point : comp::base<progress_point, void>
+{
+    using base_type     = comp::base<progress_point, void>;
+    using value_type    = int64_t;
+    using hash_type     = tim::hash_value_t;
+    using iterator_type = progress_point*;
+
+    static std::string label();
+    static std::string description();
+
+    TIMEMORY_DEFAULT_OBJECT(progress_point)
+
+    void            start();
+    void            stop();
+    void            mark();
+    void            set_value(int64_t);
+    progress_point& operator+=(const progress_point&);
+    progress_point& operator-=(const progress_point&);
+
+    bool is_throughput_point() const;
+    bool is_latency_point() const;
+    void print(std::ostream& os) const;
+
+    void    set_hash(hash_type _v) { m_hash = _v; }
+    void    set_iterator(iterator_type _v) { m_iterator = _v; }
+    auto    get_iterator() const { return m_iterator; }
+    auto    get_hash() const { return m_hash; }
+    int64_t get_delta() const;
+    int64_t get_arrival() const;
+    int64_t get_departure() const;
+    int64_t get_latency_delta() const;
+    int64_t get_laps() const;
+
+    template <typename ArchiveT>
+    void load(ArchiveT& ar, const unsigned)
+    {
+        namespace cereal = ::tim::cereal;
+        auto _name       = std::string{};
+
+        ar(cereal::make_nvp("name", _name));
+        ar(cereal::make_nvp("delta", m_delta));
+        ar(cereal::make_nvp("arrival", m_arrival));
+        ar(cereal::make_nvp("departure", m_departure));
+        m_hash = tim::hash::add_hash_id(_name);
+    }
+
+    template <typename ArchiveT>
+    void save(ArchiveT& ar, const unsigned) const
+    {
+        namespace cereal = ::tim::cereal;
+        ar(cereal::make_nvp("hash", m_hash));
+        ar(cereal::make_nvp("name", std::string{ tim::get_hash_identifier(m_hash) }));
+        ar(cereal::make_nvp("delta", m_delta));
+        ar(cereal::make_nvp("arrival", m_arrival));
+        ar(cereal::make_nvp("departure", m_departure));
+    }
+
+    static std::unordered_map<tim::hash_value_t, progress_point> get_progress_points();
+
+private:
+    hash_type       m_hash      = 0;
+    int64_t         m_delta     = 0;
+    int64_t         m_arrival   = 0;
+    int64_t         m_departure = 0;
+    progress_point* m_iterator  = nullptr;
+};
+}  // namespace component
+}  // namespace causal
+}  // namespace omnitrace
+
+OMNITRACE_DEFINE_CONCRETE_TRAIT(uses_storage, causal::component::progress_point,
+                                false_type)
+OMNITRACE_DEFINE_CONCRETE_TRAIT(flat_storage, causal::component::progress_point,
+                                true_type)
+OMNITRACE_DEFINE_CONCRETE_TRAIT(uses_timing_units, causal::component::progress_point,
+                                true_type)
+OMNITRACE_DEFINE_CONCRETE_TRAIT(is_timing_category, causal::component::progress_point,
+                                true_type)
+
+namespace tim
+{
+namespace operation
+{
+template <>
+struct push_node<omnitrace::causal::component::progress_point>
+{
+    using type = omnitrace::causal::component::progress_point;
+
+    TIMEMORY_DEFAULT_OBJECT(push_node)
+
+    push_node(type& _obj, scope::config _scope, hash_value_t _hash,
+              int64_t _tid = threading::get_id())
+    {
+        (*this)(_obj, _scope, _hash, _tid);
+    }
+
+    void operator()(type& _obj, scope::config, hash_value_t _hash,
+                    int64_t _tid = threading::get_id()) const;
+};
+
+template <>
+struct pop_node<omnitrace::causal::component::progress_point>
+{
+    using type = omnitrace::causal::component::progress_point;
+
+    TIMEMORY_DEFAULT_OBJECT(pop_node)
+
+    pop_node(type& _obj, int64_t _tid = threading::get_id()) { (*this)(_obj, _tid); }
+
+    void operator()(type& _obj, int64_t _tid = threading::get_id()) const;
+};
+}  // namespace operation
+}  // namespace tim
@@ -0,0 +1,135 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/causal/components/unblocking_gotcha.hpp"
+#include "library/causal/delay.hpp"
+#include "library/causal/experiment.hpp"
+#include "library/config.hpp"
+#include "library/debug.hpp"
+#include "library/runtime.hpp"
+
+#include <timemory/components/macros.hpp>
+#include <timemory/hash/types.hpp>
+
+#include <csignal>
+#include <cstdint>
+#include <pthread.h>
+#include <stdexcept>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace component
+{
+std::string
+unblocking_gotcha::label()
+{
+    return "causal_unblocking_gotcha";
+}
+
+std::string
+unblocking_gotcha::description()
+{
+    return "Handles executing all necessary pauses before the thread performs some "
+           "blocking function";
+}
+
+void
+unblocking_gotcha::preinit()
+{
+    configure();
+}
+
+void
+unblocking_gotcha::configure()
+{
+    unblocking_gotcha_t::get_initializer() = []() {
+        if(!config::get_use_causal()) return;
+
+        unblocking_gotcha_t::configure(
+            comp::gotcha_config<0, int, pthread_mutex_t*>{ "pthread_mutex_unlock" });
+
+        unblocking_gotcha_t::configure(
+            comp::gotcha_config<1, int, pthread_rwlock_t*>{ "pthread_rwlock_unlock" });
+
+        unblocking_gotcha_t::configure(
+            comp::gotcha_config<2, int, pthread_spinlock_t*>{ "pthread_spin_unlock" });
+
+        unblocking_gotcha_t::configure(
+            comp::gotcha_config<3, int, pthread_barrier_t*>{ "pthread_barrier_wait" });
+
+        unblocking_gotcha_t::configure(
+            comp::gotcha_config<4, int, pthread_cond_t*>{ "pthread_cond_signal" });
+
+        unblocking_gotcha_t::configure(
+            comp::gotcha_config<5, int, pthread_cond_t*>{ "pthread_cond_broadcast" });
+
+        unblocking_gotcha_t::configure(
+            comp::gotcha_config<6, int, pthread_t, int>{ "pthread_kill" });
+
+        unblocking_gotcha_t::configure(
+            comp::gotcha_config<7, void, void*>{ "pthread_exit" });
+    };
+}
+
+void
+unblocking_gotcha::shutdown()
+{
+    unblocking_gotcha_t::disable();
+}
+
+void
+unblocking_gotcha::start()
+{
+    if(causal::experiment::is_active() &&
+       get_thread_state() == ::omnitrace::ThreadState::Enabled)
+        causal::delay::process();
+}
+
+void
+unblocking_gotcha::stop()
+{
+    if(causal::experiment::is_active() &&
+       get_thread_state() == ::omnitrace::ThreadState::Enabled)
+        causal::delay::credit();
+}
+
+void
+unblocking_gotcha::set_data(const comp::gotcha_data& _data)
+{
+    OMNITRACE_SCOPED_THREAD_STATE(ThreadState::Internal);
+    auto   _hash  = tim::add_hash_id(_data.tool_id);
+    auto&& _ident = tim::get_hash_identifier(_hash);
+    if(_ident != _data.tool_id)
+        throw ::omnitrace::exception<std::runtime_error>(
+            JOIN("", "Error! resolving hash for \"", _data.tool_id, "\" (", _hash,
+                 ") returns ", _ident.c_str()));
+#if defined(OMNITRACE_CI)
+    OMNITRACE_VERBOSE_F(3, "data set for '%s'...\n", _data.tool_id.c_str());
+#endif
+}
+}  // namespace component
+}  // namespace causal
+}  // namespace omnitrace
+
+TIMEMORY_INVOKE_PREINIT(omnitrace::causal::component::unblocking_gotcha)
@@ -0,0 +1,75 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/common.hpp"
+#include "library/defines.hpp"
+#include "library/timemory.hpp"
+
+#include <timemory/components/gotcha/backends.hpp>
+#include <timemory/mpl/macros.hpp>
+
+#include <array>
+#include <cstddef>
+#include <string>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace component
+{
+using timespec_t = struct timespec;
+// this is used to wrap pthread_mutex()
+struct unblocking_gotcha : comp::base<unblocking_gotcha, void>
+{
+    static constexpr size_t gotcha_capacity = 8;
+
+    TIMEMORY_DEFAULT_OBJECT(unblocking_gotcha)
+
+    // string id for component
+    static std::string label();
+    static std::string description();
+    static void        preinit();
+
+    // generate the gotcha wrappers
+    static void configure();
+    static void shutdown();
+
+    static void start();
+    static void stop();
+
+    static void set_data(const comp::gotcha_data&);
+};
+
+using unblocking_gotcha_t =
+    comp::gotcha<unblocking_gotcha::gotcha_capacity,
+                 tim::lightweight_tuple<unblocking_gotcha>, category::causal>;
+}  // namespace component
+}  // namespace causal
+}  // namespace omnitrace
+
+OMNITRACE_DEFINE_CONCRETE_TRAIT(prevent_reentry, causal::component::unblocking_gotcha_t,
+                                false_type)
+OMNITRACE_DEFINE_CONCRETE_TRAIT(static_data, causal::component::unblocking_gotcha_t,
+                                true_type)
@@ -0,0 +1,949 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/causal/data.hpp"
+#include "library/binary/address_multirange.hpp"
+#include "library/binary/analysis.hpp"
+#include "library/binary/binary_info.hpp"
+#include "library/binary/fwd.hpp"
+#include "library/binary/link_map.hpp"
+#include "library/binary/scope_filter.hpp"
+#include "library/causal/delay.hpp"
+#include "library/causal/experiment.hpp"
+#include "library/causal/sampling.hpp"
+#include "library/causal/selected_entry.hpp"
+#include "library/config.hpp"
+#include "library/debug.hpp"
+#include "library/ptl.hpp"
+#include "library/runtime.hpp"
+#include "library/state.hpp"
+#include "library/thread_data.hpp"
+#include "library/thread_info.hpp"
+#include "library/utility.hpp"
+
+#include <timemory/data/atomic_ring_buffer.hpp>
+#include <timemory/hash/types.hpp>
+#include <timemory/log/logger.hpp>
+#include <timemory/mpl/concepts.hpp>
+#include <timemory/units.hpp>
+#include <timemory/unwind/dlinfo.hpp>
+#include <timemory/unwind/processed_entry.hpp>
+#include <timemory/utility/procfs/maps.hpp>
+
+#include <algorithm>
+#include <atomic>
+#include <chrono>
+#include <cmath>
+#include <cstdint>
+#include <cstdlib>
+#include <initializer_list>
+#include <random>
+#include <regex>
+#include <sstream>
+#include <thread>
+#include <utility>
+#include <vector>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace
+{
+using random_engine_t    = std::mt19937_64;
+using progress_bundles_t = component_bundle_cache<component::progress_point>;
+
+auto speedup_seeds     = std::vector<size_t>{};
+auto speedup_divisions = get_env<uint16_t>("OMNITRACE_CAUSAL_SPEEDUP_DIVISIONS", 5);
+auto speedup_dist      = []() {
+    size_t                _n = std::max<size_t>(1, 100 / speedup_divisions);
+    std::vector<uint16_t> _v(_n, uint16_t{ 0 });
+    std::generate(_v.begin(), _v.end(),
+                  [_value = 0]() mutable { return (_value += speedup_divisions); });
+    // approximately 25% of bins should be zero speedup
+    size_t _nzero = std::ceil(_v.size() / 4.0);
+    _v.resize(_v.size() + _nzero, 0);
+    std::sort(_v.begin(), _v.end());
+    OMNITRACE_CI_THROW(_v.back() > 100, "Error! last value is too large: %i\n",
+                       (int) _v.back());
+    return _v;
+}();
+
+auto perform_experiment_impl_completed = std::unique_ptr<std::promise<void>>{};
+auto num_progress_points               = std::atomic<size_t>{ 0 };
+
+template <typename ContextT>
+auto&
+get_engine()
+{
+    static auto _seed = []() -> hash_value_t {
+        auto _seed_v =
+            config::get_setting_value<uint64_t>("OMNITRACE_CAUSAL_RANDOM_SEED").second;
+        if(_seed_v == 0) _seed_v = std::random_device{}();
+        return _seed_v;
+    }();
+
+    static thread_local auto _v =
+        random_engine_t{ tim::get_combined_hash_id(_seed, utility::get_thread_index()) };
+    return _v;
+}
+
+binary::address_multirange&
+get_eligible_address_ranges()
+{
+    static auto _v = binary::address_multirange{};
+    return _v;
+}
+
+using sf = binary::scope_filter;
+
+auto
+get_filters(std::set<binary::scope_filter::filter_scope> _scopes = {
+                sf::BINARY_FILTER, sf::SOURCE_FILTER, sf::FUNCTION_FILTER })
+{
+    auto _filters = std::vector<binary::scope_filter>{};
+
+    // exclude internal libraries used by omnitrace
+    if(_scopes.count(sf::BINARY_FILTER) > 0)
+        _filters.emplace_back(
+            sf{ sf::FILTER_EXCLUDE, sf::BINARY_FILTER,
+                "lib(omnitrace[-\\.]|dyninst|tbbmalloc|gotcha\\.|unwind\\.so\\.99)" });
+
+    // in function mode, it generally doesn't help to experiment on main function since
+    // telling the user to "make the main function" faster is literally useless since it
+    // contains everything that could be made faster
+    if(config::get_causal_mode() == CausalMode::Function &&
+       _scopes.count(sf::FUNCTION_FILTER) > 0)
+        _filters.emplace_back(sf{ sf::FILTER_EXCLUDE, sf::FUNCTION_FILTER,
+                                  "( main\\(|^main$|^main\\.cold$)" });
+
+    bool _use_default_excludes =
+        config::get_setting_value<bool>("OMNITRACE_CAUSAL_FUNCTION_EXCLUDE_DEFAULTS")
+            .second;
+
+    if(_use_default_excludes && _scopes.count(sf::FUNCTION_FILTER) > 0)
+    {
+        // symbols starting with leading underscore are generally system functions
+        _filters.emplace_back(sf{ sf::FILTER_EXCLUDE, sf::FUNCTION_FILTER, "^_" });
+
+        if(config::get_causal_mode() == CausalMode::Function)
+        {
+            // exclude STL implementation functions
+            _filters.emplace_back(sf{ sf::FILTER_EXCLUDE, sf::FUNCTION_FILTER, "::_M" });
+        }
+    }
+
+    // in function mode, it generally doesn't help to claim
+    // "make main function" faster since it contains everything
+    // that could be made faster
+    if(config::get_causal_mode() == CausalMode::Function &&
+       _scopes.count(sf::FUNCTION_FILTER) > 0)
+    {
+        _filters.emplace_back(sf{ sf::FILTER_EXCLUDE, sf::FUNCTION_FILTER,
+                                  "(^main$|^main.cold$|int main\\()" });
+    }
+
+    using utility::get_regex_or;
+
+    auto _source_end_converter = [](const std::string& _v) { return _v + "$"; };
+
+    // include handling
+    {
+        auto _binary_include = get_regex_or(config::get_causal_binary_scope(), "");
+        auto _source_include =
+            get_regex_or(config::get_causal_source_scope(), _source_end_converter, "");
+        auto _function_include = get_regex_or(config::get_causal_function_scope(), "");
+
+        auto _current_include =
+            std::make_tuple(_binary_include, _source_include, _function_include);
+        static auto _former_include = decltype(_current_include){};
+
+        if(_former_include != _current_include)
+        {
+            if(!_binary_include.empty())
+                OMNITRACE_VERBOSE(0, "[causal] binary scope     : %s\n",
+                                  _binary_include.c_str());
+            if(!_source_include.empty())
+                OMNITRACE_VERBOSE(0, "[causal] source scope     : %s\n",
+                                  _source_include.c_str());
+            if(!_function_include.empty())
+                OMNITRACE_VERBOSE(0, "[causal] function scope   : %s\n",
+                                  _function_include.c_str());
+            _former_include = _current_include;
+        }
+
+        if(!_binary_include.empty() && _scopes.count(sf::BINARY_FILTER) > 0)
+            _filters.emplace_back(
+                sf{ sf::FILTER_INCLUDE, sf::BINARY_FILTER, _binary_include });
+
+        if(!_source_include.empty() && _scopes.count(sf::SOURCE_FILTER) > 0)
+            _filters.emplace_back(
+                sf{ sf::FILTER_INCLUDE, sf::SOURCE_FILTER, _source_include });
+
+        if(!_function_include.empty() && _scopes.count(sf::FUNCTION_FILTER) > 0)
+            _filters.emplace_back(
+                sf{ sf::FILTER_INCLUDE, sf::FUNCTION_FILTER, _function_include });
+    }
+
+    // exclude handling
+    {
+        auto _binary_exclude = get_regex_or(config::get_causal_binary_exclude(), "");
+        auto _source_exclude =
+            get_regex_or(config::get_causal_source_exclude(), _source_end_converter, "");
+        auto _function_exclude = get_regex_or(config::get_causal_function_exclude(), "");
+
+        auto _current_exclude =
+            std::make_tuple(_binary_exclude, _source_exclude, _function_exclude);
+        static auto _former_exclude = decltype(_current_exclude){};
+
+        if(_former_exclude != _current_exclude)
+        {
+            if(!_binary_exclude.empty())
+                OMNITRACE_VERBOSE(0, "[causal] binary exclude   : %s\n",
+                                  _binary_exclude.c_str());
+            if(!_source_exclude.empty())
+                OMNITRACE_VERBOSE(0, "[causal] source exclude   : %s\n",
+                                  _source_exclude.c_str());
+            if(!_function_exclude.empty())
+                OMNITRACE_VERBOSE(0, "[causal] function exclude : %s\n",
+                                  _function_exclude.c_str());
+            _former_exclude = _current_exclude;
+        }
+
+        if(!_binary_exclude.empty() && _scopes.count(sf::BINARY_FILTER) > 0)
+            _filters.emplace_back(
+                sf{ sf::FILTER_EXCLUDE, sf::BINARY_FILTER, _binary_exclude });
+
+        if(!_source_exclude.empty() && _scopes.count(sf::SOURCE_FILTER) > 0)
+            _filters.emplace_back(
+                sf{ sf::FILTER_EXCLUDE, sf::SOURCE_FILTER, _source_exclude });
+
+        if(!_function_exclude.empty() && _scopes.count(sf::FUNCTION_FILTER) > 0)
+            _filters.emplace_back(
+                sf{ sf::FILTER_EXCLUDE, sf::FUNCTION_FILTER, _function_exclude });
+    }
+
+    return _filters;
+}
+
+using binary_info_t = std::vector<binary::binary_info>;
+std::pair<binary_info_t, binary_info_t>&
+get_cached_binary_info()
+{
+    static auto _v = []() {
+        // get the linked binaries for the exe (excluding ones from libomnitrace)
+        auto _link_map = binary::get_link_map();
+        auto _files    = std::vector<std::string>{};
+        _files.reserve(_link_map.size());
+        for(const auto& itr : _link_map)
+            _files.emplace_back(itr.real());
+
+        auto _discarded = std::vector<binary::binary_info>{};
+        auto _requested = binary::get_binary_info(_files, get_filters());
+        return std::make_pair(_requested, _discarded);
+    }();
+    return _v;
+}
+
+bool
+satisfies_filter(const binary::scope_filter::filter_scope& _scope,
+                 const std::string&                        _value)
+{
+    static auto _filters = get_filters();
+    return binary::scope_filter::satisfies_filter(_filters, _scope, _value);
+}
+
+auto
+compute_eligible_lines_impl()
+{
+    const auto& _binary_info = get_cached_binary_info().first;
+    auto&       _filter_info = get_cached_binary_info().second;
+    auto        _filters     = get_filters();
+
+    auto& _eligible_ar = get_eligible_address_ranges();
+    for(const auto& litr : _binary_info)
+    {
+        for(const auto& ditr : litr.mappings)
+        {
+            _eligible_ar +=
+                std::make_pair(binary::address_multirange::coarse{},
+                               address_range_t{ ditr.load_address, ditr.last_address });
+        }
+
+        for(const auto& ditr : litr.symbols)
+        {
+            _eligible_ar += ditr.address + ditr.load_address;
+        }
+
+        auto& _filtered    = _filter_info.emplace_back();
+        _filtered.bfd      = litr.bfd;
+        _filtered.mappings = litr.mappings;
+        _filtered.ranges   = litr.ranges;
+        _filtered.sections = litr.sections;
+
+        for(const auto& ditr : litr.symbols)
+        {
+            auto _sym = ditr.clone();
+
+            _sym.inlines =
+                ditr.get_inline_symbols<std::vector<binary::inlined_symbol>>(_filters);
+
+            _sym.dwarf_info =
+                ditr.get_debug_line_info<std::vector<binary::dwarf_entry>>(_filters);
+
+            if(ditr(_filters) || (_sym.inlines.size() + _sym.dwarf_info.size()) > 0)
+            {
+                _filtered.symbols.emplace_back(_sym);
+            }
+        }
+
+        for(const auto& ditr : litr.debug_info)
+        {
+            if(sf::satisfies_filter(_filters, sf::SOURCE_FILTER, ditr.file) ||
+               sf::satisfies_filter(_filters, sf::SOURCE_FILTER,
+                                    join(':', ditr.file, ditr.line)))
+            {
+                _filtered.debug_info.emplace_back(ditr);
+            }
+        }
+
+        _filtered.sort();
+    }
+
+    OMNITRACE_VERBOSE(
+        0, "[causal] eligible address ranges: %zu, coarse address range: %zu [%s]\n",
+        _eligible_ar.size(), _eligible_ar.range_size(),
+        _eligible_ar.coarse_range.as_string().c_str());
+
+    if(_eligible_ar.empty())
+    {
+        auto _cfg         = settings::compose_filename_config{};
+        _cfg.subdirectory = "causal/binary-info";
+        _cfg.use_suffix   = config::get_use_pid();
+        save_line_info(_cfg, config::get_verbose());
+    }
+
+    OMNITRACE_CONDITIONAL_THROW(
+        _eligible_ar.empty(),
+        "Error! binary analysis (after filters) resulted in zero eligible instruction "
+        "pointer addresses for causal experimentation");
+}
+
+void
+save_maps_info_impl(std::ostream& _ofs)
+{
+    auto _maps_file = join("/", "/proc", process::get_id(), "maps");
+    auto _ifs       = std::ifstream{ _maps_file };
+    auto _maps      = std::stringstream{};
+    if(_ifs)
+    {
+        _maps << _maps_file << "\n";
+        while(_ifs)
+        {
+            std::string _line{};
+            getline(_ifs, _line);
+            if(!_line.empty()) _maps << "    " << _line << "\n";
+        }
+    }
+    _ofs << _maps.str();
+}
+
+void
+save_line_info_impl(std::ostream&                           _ofs,
+                    const std::vector<binary::binary_info>& _binary_data)
+{
+    auto _write_impl = [&_ofs](const binary::binary_info& _data) {
+        for(const auto& itr : _data.mappings)
+        {
+            _ofs << itr.pathname << " [" << as_hex(itr.load_address) << " - "
+                 << as_hex(itr.last_address) << "]\n";
+        }
+
+        auto _emitted_dwarf_addresses = std::set<uintptr_t>{};
+        for(const auto& itr : _data.symbols)
+        {
+            auto _addr     = itr.address;
+            auto _addr_off = itr.address + itr.load_address;
+            _ofs << "    " << as_hex(_addr_off) << " [" << as_hex(_addr)
+                 << "] :: " << itr.file;
+            if(itr.line > 0) _ofs << ":" << itr.line;
+            if(!itr.func.empty()) _ofs << " [" << tim::demangle(itr.func) << "]";
+            _ofs << "\n";
+
+            for(const auto& ditr : itr.inlines)
+            {
+                _ofs << "        " << ditr.file << ":" << ditr.line;
+                if(!ditr.func.empty()) _ofs << " [" << tim::demangle(ditr.func) << "]";
+                _ofs << "\n";
+            }
+
+            for(const auto& ditr : itr.dwarf_info)
+            {
+                _ofs << "        " << as_hex(ditr.address) << " :: " << ditr.file << ":"
+                     << ditr.line;
+                _ofs << "\n";
+                _emitted_dwarf_addresses.emplace(ditr.address.low);
+            }
+        }
+
+        for(const auto& itr : _data.debug_info)
+        {
+            if(_emitted_dwarf_addresses.count(itr.address.low) > 0) continue;
+            _ofs << "    " << as_hex(itr.address) << " :: " << itr.file << ":"
+                 << itr.line;
+            _ofs << "\n";
+        }
+
+        _ofs << "\n" << std::flush;
+    };
+
+    for(const auto& itr : _binary_data)
+        _write_impl(itr);
+}
+
+void
+compute_eligible_lines()
+{
+    static auto _once = std::once_flag{};
+    std::call_once(_once, compute_eligible_lines_impl);
+}
+
+void
+perform_experiment_impl(std::shared_ptr<std::promise<void>> _started)  // NOLINT
+{
+    using clock_type      = std::chrono::high_resolution_clock;
+    using duration_nsec_t = std::chrono::duration<double, std::nano>;
+    using duration_sec_t  = std::chrono::duration<double, std::ratio<1>>;
+
+    const auto& _thr_info = thread_info::init(true);
+    OMNITRACE_SCOPED_THREAD_STATE(ThreadState::Internal);
+    OMNITRACE_CONDITIONAL_THROW(!_thr_info->is_offset,
+                                "Error! causal profiling thread should be offset");
+
+    if(!perform_experiment_impl_completed)
+        perform_experiment_impl_completed = std::make_unique<std::promise<void>>();
+
+    perform_experiment_impl_completed->set_value_at_thread_exit();
+
+    compute_eligible_lines();
+
+    // notify that thread has started
+    if(_started) _started->set_value();
+
+    // pause at least one second to determine sampling rate
+    // std::this_thread::sleep_for(std::chrono::seconds{ 1 });
+
+    if(!config::get_causal_end_to_end())
+    {
+        // wait for at least one progress point to start
+        while(num_progress_points.load(std::memory_order_relaxed) == 0)
+        {
+            std::this_thread::sleep_for(std::chrono::milliseconds{ 1 });
+        }
+    }
+
+    double _delay_sec =
+        config::get_setting_value<double>("OMNITRACE_CAUSAL_DELAY").second;
+    double _duration_sec =
+        config::get_setting_value<double>("OMNITRACE_CAUSAL_DURATION").second;
+    auto _duration_nsec = duration_nsec_t{ _duration_sec * units::sec };
+
+    if(_delay_sec > 0.0)
+    {
+        OMNITRACE_VERBOSE(1, "[causal] delaying experimentation for %.2f seconds...\n",
+                          _delay_sec);
+        uint64_t _delay_nsec = _delay_sec * units::sec;
+        std::this_thread::sleep_for(std::chrono::nanoseconds{ _delay_nsec });
+    }
+
+    auto _impl_count        = 0;
+    auto _start_time        = clock_type::now();
+    auto _exceeded_duration = [&]() {
+        if(_duration_sec > 1.0e-3)
+        {
+            auto _elapsed = clock_type::now() - _start_time;
+            if(_elapsed >= _duration_nsec)
+            {
+                OMNITRACE_VERBOSE(
+                    1,
+                    "[causal] stopping experimentation after %.2f seconds "
+                    "(elapsed: %.2f seconds)...\n",
+                    _duration_sec,
+                    std::chrono::duration_cast<duration_sec_t>(_elapsed).count());
+                causal::sampling::post_process();
+                return true;
+            }
+        }
+        return false;
+    };
+
+    while(get_state() < State::Finalized)
+    {
+        auto _impl_no = _impl_count++;
+        auto _experim = experiment{};
+
+        // loop until started or finalized
+        while(!_experim.start())
+        {
+            if(get_state() == State::Finalized)
+            {
+                auto _memory = std::stringstream{};
+                auto _binary = std::stringstream{};
+                auto _scoped = std::stringstream{};
+                auto _sample = std::stringstream{};
+                save_maps_info_impl(_memory);
+                save_line_info_impl(_binary, get_cached_binary_info().first);
+                save_line_info_impl(_scoped, get_cached_binary_info().second);
+
+                auto _samples = std::map<uintptr_t, size_t>{};
+                for(const auto& itr : get_samples())
+                {
+                    for(const auto& iitr : itr.second)
+                    {
+                        _samples[iitr.address] += iitr.count;
+                    }
+                }
+
+                for(const auto& itr : _samples)
+                {
+                    if(itr.second > 0)
+                    {
+                        auto _linfo = get_line_info(itr.first, true);
+                        // if(_linfo.size() > 1) _linfo.pop_front();
+                        for(const auto& iitr : _linfo)
+                        {
+                            _sample << "    " << std::setw(8) << itr.second
+                                    << " :: " << as_hex(itr.first) << " [" << iitr.file
+                                    << ":" << iitr.line << "][" << demangle(iitr.func)
+                                    << "]\n";
+                        }
+
+                        if(_linfo.empty())
+                        {
+                            _sample << "    " << std::setw(8) << itr.second
+                                    << " :: " << as_hex(itr.first) << "\n";
+                        }
+                    }
+                }
+
+                std::cerr << std::flush;
+                auto _cerr = tim::log::warning_stream(std::cerr);
+                _cerr << "\nmaps:\n\n" << _memory.str() << "\n";
+                _cerr << "\nbinary:\n\n" << _binary.str() << "\n";
+                _cerr << "\nscoped:\n\n" << _scoped.str() << "\n";
+                _cerr << "\nsample:\n\n" << _sample.str() << "\n";
+                std::cerr << std::flush;
+
+                OMNITRACE_CONDITIONAL_THROW(_impl_no == 0, "experiment never started");
+                return;
+            }
+        }
+
+        // wait for the experiment to complete
+        if(config::get_causal_end_to_end())
+        {
+            mark_progress_point(config::get_exe_name(), true);
+            while(get_state() < State::Finalized)
+            {
+                std::this_thread::yield();
+                std::this_thread::sleep_for(std::chrono::milliseconds{ 100 });
+                if(_exceeded_duration()) return;
+            }
+        }
+        else
+        {
+            _experim.wait();
+        }
+
+        while(!_experim.stop())
+        {
+            if(get_state() == State::Finalized) return;
+        }
+
+        if(_exceeded_duration()) return;
+    }
+}
+
+// thread-safe read/write ring-buffer via atomics
+using pc_ring_buffer_t = tim::data_storage::atomic_ring_buffer<uintptr_t>;
+// latest_eligible_pcs is an array of unwind_depth size -> samples will
+// use lowest indexes for most recent functions address in the call-stack
+auto latest_eligible_pc = []() {
+    auto _arr = std::array<std::unique_ptr<pc_ring_buffer_t>, unwind_depth>{};
+    for(auto& itr : _arr)
+        itr = std::make_unique<pc_ring_buffer_t>(units::get_page_size() /
+                                                 (sizeof(uintptr_t) + 1));
+    return _arr;
+}();
+}  // namespace
+
+//--------------------------------------------------------------------------------------//
+
+bool
+is_eligible_address(uintptr_t _v)
+{
+    return get_eligible_address_ranges().coarse_range.contains(_v);
+}
+
+void
+save_line_info(const settings::compose_filename_config& _cfg, int _verbose)
+{
+    auto _write = [_verbose](const std::string& ofname, const auto& _data) {
+        auto _ofs = std::ofstream{};
+        if(tim::filepath::open(_ofs, ofname))
+        {
+            if(_verbose >= 0)
+                operation::file_output_message<binary::symbol>{}(
+                    ofname, std::string{ "causal_symbol_info" });
+            save_line_info_impl(_ofs, _data);
+            save_maps_info_impl(_ofs);
+        }
+        else
+        {
+            throw ::omnitrace::exception<std::runtime_error>("Error opening " + ofname);
+        }
+    };
+
+    _write(tim::settings::compose_output_filename(
+               join('-', config::get_causal_output_filename(), "binary"), "txt", _cfg),
+           get_cached_binary_info().first);
+    _write(tim::settings::compose_output_filename(
+               join('-', config::get_causal_output_filename(), "scoped"), "txt", _cfg),
+           get_cached_binary_info().second);
+}
+
+void
+set_current_selection(unwind_addr_t _stack)
+{
+    if(experiment::is_active()) return;
+
+    size_t _n = 0;
+    for(auto itr : _stack)
+    {
+        auto& _pcs = latest_eligible_pc.at(_n);
+        if(_pcs && is_eligible_address(itr))
+        {
+            _pcs->write(&itr);
+            // increment after valid found -> first valid pc for call-stack
+            ++_n;
+        }
+    }
+}
+
+selected_entry
+sample_selection(size_t _nitr, size_t _wait_ns)
+{
+    OMNITRACE_SCOPED_THREAD_STATE(ThreadState::Internal);
+
+    size_t _n = 0;
+
+    auto _select_address = [&](auto& _address_vec) {
+        // this isn't necessary bc of check before calling this lambda but
+        // kept because of size() - 1 in distribution range
+        if(OMNITRACE_UNLIKELY(_address_vec.empty()))
+        {
+            OMNITRACE_WARNING(0, "no addresses for sample selection...\n");
+            return selected_entry{};
+        }
+
+        while(!_address_vec.empty())
+        {
+            // randomly select an address
+            auto _dist =
+                std::uniform_int_distribution<size_t>{ 0, _address_vec.size() - 1 };
+            auto _idx = _dist(get_engine<selected_entry>());
+
+            uintptr_t _addr        = _address_vec.at(_idx);
+            uintptr_t _sym_addr    = 0;
+            uintptr_t _lookup_addr = _addr;
+            auto      _dl_info     = unwind::dlinfo::construct(_addr);
+
+            _address_vec.erase(_address_vec.begin() + _idx);
+
+            if(get_causal_mode() == CausalMode::Function)
+                _sym_addr = (_dl_info.symbol) ? _dl_info.symbol.address() : _addr;
+
+            // lookup the PC line info at either the address or the symbol address
+            auto linfo = get_line_info(_lookup_addr, false);
+
+            // unlikely this will be empty but just in case
+            if(linfo.empty()) continue;
+
+            // debugging for continuous integration
+            if(OMNITRACE_UNLIKELY(config::get_is_continuous_integration() ||
+                                  config::get_debug()))
+            {
+                auto _location =
+                    (_dl_info.location)
+                        ? filepath::realpath(std::string{ _dl_info.location.name },
+                                             nullptr, false)
+                        : std::string{};
+                for(const auto& itr : linfo)
+                {
+                    if(OMNITRACE_UNLIKELY(config::get_debug()))
+                    {
+                        OMNITRACE_WARNING(
+                            0, "[%s][%s][%s][%s] %s [%s:%i][%s][%zu]\n",
+                            as_hex(_lookup_addr).c_str(), as_hex(_addr).c_str(),
+                            as_hex(_sym_addr).c_str(),
+                            (_location.empty()) ? "" : _location.data(),
+                            demangle(itr.func).c_str(), itr.file.c_str(), itr.line,
+                            itr.address.as_string().c_str(), itr.address.size());
+                    }
+                }
+            }
+
+            auto& _linfo_v = (config::get_causal_mode() == CausalMode::Function)
+                                 ? linfo.front()
+                                 : linfo.back();
+            return selected_entry{ _addr, _sym_addr, _linfo_v };
+            // return selected_entry{ address_range_t{ _addr },
+            //                       address_range_t{ _sym_addr },
+            //                       { _linfo_v.second } };
+        }
+        return selected_entry{};
+    };
+
+    while(_n++ < _nitr)
+    {
+        auto _addresses = std::deque<uintptr_t>{};
+        for(auto& aitr : latest_eligible_pc)
+        {
+            if(OMNITRACE_UNLIKELY(!aitr))
+            {
+                OMNITRACE_WARNING(0, "invalid ring buffer...\n");
+                continue;
+            }
+
+            auto _naddrs = aitr->count();
+            if(_naddrs == 0) continue;
+
+            for(size_t i = 0; i < _naddrs; ++i)
+            {
+                uintptr_t _addr = 0;
+                if(!aitr->is_empty() && aitr->read(&_addr) != nullptr)
+                {
+                    if(_addr > 0) _addresses.emplace_back(_addr);
+                }
+            }
+
+            if(!_addresses.empty())
+            {
+                auto _selection = _select_address(_addresses);
+                if(_selection) return _selection;
+            }
+        }
+
+        std::this_thread::yield();
+        std::this_thread::sleep_for(std::chrono::nanoseconds{ _wait_ns });
+    }
+
+    return selected_entry{};
+}
+
+std::deque<binary::symbol>
+get_line_info(uintptr_t _addr, bool _include_discarded)
+{
+    static auto _glob_filters  = get_filters({ sf::BINARY_FILTER });
+    static auto _scope_filters = get_filters();
+    auto        _data          = std::deque<binary::symbol>{};
+    auto        _get_line_info = [&](const auto& _info, const auto& _filters) {
+        // search for exact matches first
+        for(const binary::binary_info& litr : _info)
+        {
+            auto _local_data = std::deque<binary::symbol>{};
+
+            for(const auto& ditr : litr.symbols)
+            {
+                auto _ipaddr = ditr.ipaddr();
+                if(!_ipaddr.contains(_addr)) continue;
+
+                if(config::get_causal_mode() == CausalMode::Function)
+                {
+                    // check if the primary symbol satisfy the constraints
+                    if(ditr(_filters)) _local_data.emplace_back(ditr);
+
+                    // the primary symbol may not satisfy the constraints but the inlined
+                    // functions may
+                    utility::combine(_local_data, ditr.get_inline_symbols(_filters));
+                }
+                else if(config::get_causal_mode() == CausalMode::Line)
+                {
+                    auto _debug_data = std::deque<binary::symbol>{};
+                    for(const auto& itr : ditr.get_debug_line_info(_filters))
+                    {
+                        if(itr.ipaddr().contains(_addr)) _debug_data.emplace_back(itr);
+                    }
+                    utility::combine(_local_data, _debug_data);
+                }
+                else
+                {
+                    throw exception<std::runtime_error>(
+                        join(" ", "Causal mode not supported:",
+                             std::to_string(config::get_causal_mode())));
+                }
+            }
+
+            if(!_local_data.empty())
+            {
+                // combine and only allow first match
+                utility::combine(_data, _local_data);
+                break;
+            }
+        }
+    };
+
+    if(_include_discarded)
+        _get_line_info(get_cached_binary_info().first, _glob_filters);
+    else
+        _get_line_info(get_cached_binary_info().second, _scope_filters);
+
+    return _data;
+}
+
+void
+push_progress_point(std::string_view _name)
+{
+    if(config::get_causal_end_to_end()) return;
+
+    ++num_progress_points;
+
+    auto  _hash   = tim::add_hash_id(_name);
+    auto& _data   = progress_bundles_t::instance(utility::get_thread_index());
+    auto* _bundle = _data.construct(_hash);
+    _bundle->push();
+    _bundle->start();
+}
+
+void
+pop_progress_point(std::string_view _name)
+{
+    if(config::get_causal_end_to_end()) return;
+
+    auto& _data = progress_bundles_t::instance(utility::get_thread_index());
+    if(_data.empty()) return;
+    if(_name.empty())
+    {
+        auto* itr = _data.back();
+        itr->stop();
+        itr->pop();
+        _data.pop_back();
+        return;
+    }
+    else
+    {
+        auto _hash = tim::add_hash_id(_name);
+        for(auto itr = _data.rbegin(); itr != _data.rend(); ++itr)
+        {
+            if((*itr)->get_hash() == _hash)
+            {
+                (*itr)->stop();
+                (*itr)->pop();
+                _data.destroy(itr);
+                return;
+            }
+        }
+    }
+}
+
+void
+mark_progress_point(std::string_view _name, bool _force)
+{
+    if(config::get_causal_end_to_end() && !_force) return;
+
+    ++num_progress_points;
+
+    auto  _hash   = tim::add_hash_id(_name);
+    auto& _data   = progress_bundles_t::instance(utility::get_thread_index());
+    auto* _bundle = _data.construct(_hash);
+    _bundle->push();
+    _bundle->mark();
+    _bundle->pop();
+    _data.pop_back();
+}
+
+uint16_t
+sample_virtual_speedup()
+{
+    if(speedup_dist.empty())
+        return 0;
+    else if(speedup_dist.size() == 1)
+        return speedup_dist.front();
+    else
+    {
+        struct virtual_speedup
+        {};
+        auto _dist =
+            std::uniform_int_distribution<size_t>{ size_t{ 0 }, speedup_dist.size() - 1 };
+        return speedup_dist.at(_dist(get_engine<virtual_speedup>()));
+    }
+}
+
+void
+start_experimenting()
+{
+    OMNITRACE_SCOPED_THREAD_STATE(ThreadState::Internal);
+
+    auto _user_speedup_dist = config::get_causal_fixed_speedup();
+    if(!_user_speedup_dist.empty())
+    {
+        speedup_dist.clear();
+        for(auto itr : _user_speedup_dist)
+        {
+            OMNITRACE_CONDITIONAL_ABORT_F(itr > 100,
+                                          "Virtual speedups must be in range [0, 100]. "
+                                          "Invalid virtual speedup: %lu\n",
+                                          itr);
+            speedup_dist.emplace_back(static_cast<uint16_t>(itr));
+        }
+    }
+
+    compute_eligible_lines();
+
+    auto _cfg         = settings::compose_filename_config{};
+    _cfg.subdirectory = "causal/binary-info";
+    _cfg.use_suffix   = config::get_use_pid();
+    save_line_info(_cfg, config::get_verbose());
+
+    if(get_state() < State::Finalized)
+    {
+        OMNITRACE_SCOPED_SAMPLING_ON_CHILD_THREADS(false);
+        auto _promise = std::make_shared<std::promise<void>>();
+        std::thread{ perform_experiment_impl, _promise }.detach();
+        _promise->get_future().wait_for(std::chrono::seconds{ 2 });
+    }
+}
+
+void
+finish_experimenting()
+{
+    if(perform_experiment_impl_completed)
+    {
+        perform_experiment_impl_completed->get_future().wait_for(
+            std::chrono::seconds{ 5 });
+        perform_experiment_impl_completed.reset();
+    }
+    sampling::post_process();
+    experiment::save_experiments();
+}
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,76 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/binary/analysis.hpp"
+#include "library/binary/fwd.hpp"
+#include "library/causal/fwd.hpp"
+#include "library/containers/c_array.hpp"
+#include "library/containers/static_vector.hpp"
+#include "library/defines.hpp"
+#include "library/thread_data.hpp"
+#include "library/utility.hpp"
+
+#include <timemory/hash/types.hpp>
+#include <timemory/tpls/cereal/cereal/cereal.hpp>
+#include <timemory/utility/procfs/maps.hpp>
+#include <timemory/utility/unwind.hpp>
+
+#include <deque>
+#include <dlfcn.h>
+#include <map>
+
+namespace omnitrace
+{
+namespace causal
+{
+void
+save_line_info(const settings::compose_filename_config&, int _verbose);
+
+std::deque<binary::symbol>
+get_line_info(uintptr_t _addr, bool include_discarded = true);
+
+bool is_eligible_address(uintptr_t);
+
+void set_current_selection(unwind_addr_t);
+
+selected_entry
+sample_selection(size_t _nitr = 1000, size_t _wait_ns = 10000);
+
+void push_progress_point(std::string_view);
+
+void pop_progress_point(std::string_view);
+
+void
+mark_progress_point(std::string_view, bool force = false);
+
+uint16_t
+sample_virtual_speedup();
+
+void
+start_experimenting();
+
+void
+finish_experimenting();
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,193 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/causal/delay.hpp"
+#include "library/causal/experiment.hpp"
+#include "library/runtime.hpp"
+#include "library/state.hpp"
+#include "library/thread_data.hpp"
+#include "library/thread_info.hpp"
+#include "library/tracing.hpp"
+#include "library/utility.hpp"
+
+#include <timemory/components/macros.hpp>
+#include <timemory/mpl/concepts.hpp>
+#include <timemory/mpl/types.hpp>
+#include <timemory/process/threading.hpp>
+
+#include <atomic>
+#include <chrono>
+#include <random>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace
+{
+auto&
+get_delay_data()
+{
+    using thread_data_t = thread_data<identity<int64_t>, delay>;
+    static auto& _v     = thread_data_t::construct(
+        construct_on_init{}, []() { return delay::get_global().load(); });
+    return _v;
+}
+
+int64_t
+compute_sleep_for_overhead()
+{
+    using random_engine_t = std::mt19937_64;
+    auto   _engine        = random_engine_t{ std::random_device{}() };
+    auto   _dist          = std::uniform_int_distribution<int64_t>{ 0, 5 };
+    size_t _ntot          = 250;
+    size_t _nwarm         = 50;
+    auto   _stats         = tim::statistics<double>{};
+    for(size_t i = 0; i < _ntot; ++i)
+    {
+        auto    _val = _dist(_engine);
+        int64_t _beg = tracing::now();
+        std::this_thread::sleep_for(std::chrono::nanoseconds{ _val });
+        int64_t _end = tracing::now();
+        if(i < _nwarm) continue;
+        auto _diff = (_end - _beg);
+        OMNITRACE_CONDITIONAL_THROW(
+            _diff < _val, "Error! sleep_for(%zu) [nanoseconds] >= %zu", _val, _diff);
+        _stats += (_diff - _val);
+    }
+
+    OMNITRACE_BASIC_VERBOSE(2,
+                            "[causal] overhead of std::this_thread::sleep_for(...) "
+                            "invocation = %6.3f usec +/- %e\n",
+                            _stats.get_mean() / units::usec,
+                            _stats.get_stddev() / units::usec);
+
+    tim::manager::instance()->add_metadata([_stats](auto& ar) {
+        ar(tim::cereal::make_nvp("causal thread sleep overhead [nsec]", _stats));
+    });
+
+    (void) get_delay_data();
+
+    return _stats.get_mean();
+}
+
+int64_t sleep_for_overhead = compute_sleep_for_overhead();
+}  // namespace
+
+void
+delay::process()
+{
+    if(!trait::runtime_enabled<delay>::get()) return;
+    if(get_state() >= ::omnitrace::State::Finalized) return;
+
+    if(causal::experiment::is_active())
+    {
+        if(get_global() < get_local())
+        {
+            auto _diff = (get_local() - get_global());
+            if(_diff > sleep_for_overhead) get_global() += _diff;
+        }
+        else if(get_global() > get_local())
+        {
+            auto _beg = tracing::now();
+            std::this_thread::sleep_for(
+                std::chrono::nanoseconds{ get_global() - get_local() });
+            get_local() += (tracing::now() - _beg);
+        }
+    }
+    else
+    {
+        get_local() = get_global();
+    }
+}
+
+void
+delay::credit()
+{
+    if(!trait::runtime_enabled<delay>::get()) return;
+    if(get_state() >= ::omnitrace::State::Finalized) return;
+
+    auto _diff = get_global() - get_local();
+    if(_diff > 0)
+    {
+        get_local() += _diff;
+    }
+}
+
+void
+delay::preblock()
+{
+    if(!trait::runtime_enabled<delay>::get()) return;
+    if(get_state() >= ::omnitrace::State::Finalized) return;
+
+    auto _diff = get_global() - get_local();
+    if(_diff > 0)
+    {
+        get_local() += _diff;
+    }
+}
+
+void
+delay::postblock(int64_t _preblock_global_delay_value)
+{
+    if(!trait::runtime_enabled<delay>::get()) return;
+    if(get_state() >= ::omnitrace::State::Finalized) return;
+
+    get_local() += (get_global() - _preblock_global_delay_value);
+}
+
+int64_t
+delay::sync()
+{
+    auto _v = get_global().load(std::memory_order_seq_cst);
+    if(get_delay_data()) get_delay_data()->fill(_v);
+    return _v;
+}
+
+std::atomic<int64_t>&
+delay::get_global()
+{
+    static auto _v = std::atomic<int64_t>{ 0 };
+    return _v;
+}
+
+int64_t&
+delay::get_local(int64_t _tid)
+{
+    auto&                    _data     = get_delay_data();
+    static thread_local auto _thr_init = []() {
+        using thread_data_t = thread_data<identity<int64_t>, delay>;
+        thread_data_t::construct(construct_on_thread{ threading::get_id() },
+                                 get_global().load());
+        return true;
+    }();
+    return _data->at(_tid);
+    (void) _thr_init;
+}
+
+uint64_t
+delay::compute_total_delay(uint64_t _baseline)
+{
+    return get_global().load() - _baseline;
+}
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,62 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/common.hpp"
+#include "library/components/fwd.hpp"
+#include "library/defines.hpp"
+#include "library/thread_data.hpp"
+
+#include <timemory/components/base.hpp>
+#include <timemory/macros/language.hpp>
+#include <timemory/mpl/concepts.hpp>
+
+#include <atomic>
+#include <cstdint>
+
+namespace omnitrace
+{
+namespace causal
+{
+struct delay
+: tim::component::empty_base
+, tim::concepts::component
+{
+    using value_type = void;
+
+    TIMEMORY_DEFAULT_OBJECT(delay)
+
+    static void    process();
+    static void    credit();
+    static void    preblock();
+    static void    postblock(int64_t);
+    static int64_t sync();
+
+    static std::atomic<int64_t>& get_global();
+    static int64_t&              get_local(int64_t _tid = threading::get_id());
+
+    static int64_t  get(int64_t _tid = threading::get_id());
+    static uint64_t compute_total_delay(uint64_t);
+};
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,671 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/causal/experiment.hpp"
+#include "common/defines.h"
+#include "library/causal/components/backtrace.hpp"
+#include "library/causal/components/progress_point.hpp"
+#include "library/causal/data.hpp"
+#include "library/causal/delay.hpp"
+#include "library/config.hpp"
+#include "library/debug.hpp"
+#include "library/state.hpp"
+#include "library/thread_data.hpp"
+#include "library/thread_info.hpp"
+#include "library/tracing.hpp"
+
+#include <string>
+#include <timemory/components/timing/backends.hpp>
+#include <timemory/hash/types.hpp>
+#include <timemory/mpl/policy.hpp>
+#include <timemory/tpls/cereal/archives.hpp>
+#include <timemory/tpls/cereal/cereal.hpp>
+#include <timemory/tpls/cereal/cereal/archives/json.hpp>
+#include <timemory/tpls/cereal/types.hpp>
+#include <timemory/units.hpp>
+
+#include <chrono>
+#include <ratio>
+#include <regex>
+#include <thread>
+#include <vector>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace
+{
+using backtrace_causal = omnitrace::causal::component::backtrace;
+namespace cereal       = ::tim::cereal;
+
+auto    current_experiment_value  = experiment{};
+auto    current_selected_count    = std::atomic<uint64_t>{ 0 };
+auto    current_experiment        = std::atomic<experiment*>{ nullptr };
+auto    experiment_history        = std::vector<experiment>{};
+int64_t global_scaling            = 1;
+int64_t global_scaling_increments = 0;
+bool    use_exp_speedup_scaling =
+    get_env<bool>("OMNITRACE_CAUSAL_SCALE_EXPERIMENT_TIME_BY_SPEEDUP", false);
+}  // namespace
+
+bool
+experiment::sample::operator==(const sample& _v) const
+{
+    return std::tie(address, info.line, info.file, info.func, location) ==
+           std::tie(_v.address, _v.info.line, _v.info.file, _v.info.func, _v.location);
+}
+
+bool
+experiment::sample::operator<(const sample& _v) const
+{
+    if(info.line > 0 && _v.info.line > 0)
+    {
+        return std::tie(info.line, info.file) == std::tie(_v.info.line, _v.info.file);
+    }
+    else if((info.line + _v.info.line) > 0)
+    {
+        return std::tie(info.file, location, info.line) <
+               std::tie(_v.info.file, _v.location, _v.info.line);
+    }
+    return (location < _v.location);
+}
+
+const auto&
+experiment::sample::operator+=(const sample& _v) const
+{
+    if(*this == _v && this != &_v) count += _v.count;
+    return *this;
+}
+
+template <typename ArchiveT>
+void
+experiment::sample::serialize(ArchiveT& ar, const unsigned)
+{
+    namespace cereal = ::tim::cereal;
+    ar(cereal::make_nvp("location", location), cereal::make_nvp("count", count),
+       cereal::make_nvp("info", info));
+}
+
+template <typename ArchiveT>
+void
+experiment::record::serialize(ArchiveT& ar, const unsigned)
+{
+    namespace cereal = ::tim::cereal;
+    ar(cereal::make_nvp("startup_time", startup),
+       cereal::make_nvp("experiments", experiments),
+       cereal::make_nvp("runtime", runtime));
+    auto _samples = std::vector<sample>{};
+    if constexpr(concepts::is_input_archive<ArchiveT>::value)
+    {
+        ar(cereal::make_nvp("samples", _samples));
+        for(auto& itr : _samples)
+            samples.emplace(std::move(itr));
+    }
+    else
+    {
+        ar(cereal::make_nvp("samples", samples));
+    }
+}
+
+template <typename ArchiveT>
+void
+experiment::serialize(ArchiveT& ar, const unsigned)
+{
+    namespace cereal = ::tim::cereal;
+
+    ar(cereal::make_nvp("index", index),
+       cereal::make_nvp("virtual_speedup", virtual_speedup),
+       cereal::make_nvp("sampling_period", sampling_period),
+       cereal::make_nvp("start_time", start_time), cereal::make_nvp("end_time", end_time),
+       cereal::make_nvp("experiment_time", experiment_time),
+       cereal::make_nvp("batch_size", batch_size), cereal::make_nvp("duration", duration),
+       cereal::make_nvp("scaling_factor", scaling_factor),
+       cereal::make_nvp("selected", selected),
+       cereal::make_nvp("sample_delay", sample_delay),
+       cereal::make_nvp("delay_scaling", delay_scaling),
+       cereal::make_nvp("total_delay", total_delay),
+       cereal::make_nvp("global_delay", global_delay),
+       cereal::make_nvp("selection", selection));
+
+    if constexpr(concepts::is_input_archive<ArchiveT>::value)
+    {
+        auto _ppts = std::vector<component::progress_point>{};
+        init_progress.clear();
+        fini_progress.clear();
+        ar(cereal::make_nvp("progress_points", _ppts));
+        for(auto itr : _ppts)
+            fini_progress.emplace(itr.get_hash(), itr);
+    }
+    else
+    {
+        auto _ppts = std::vector<component::progress_point>{};
+        {
+            auto ppts = fini_progress;
+            for(auto& pitr : ppts)
+                pitr.second.set_hash(pitr.first);
+            for(auto pitr : init_progress)
+                ppts[pitr.first] -= pitr.second;
+            _ppts.reserve(ppts.size());
+            for(auto& pitr : ppts)
+                _ppts.emplace_back(pitr.second);
+        }
+        ar(cereal::make_nvp("progress_points", _ppts));
+    }
+
+    ar(cereal::make_nvp("period_stats", period_stats));
+}
+
+std::string
+experiment::label()
+{
+    return "causal_experiment";
+}
+
+std::string
+experiment::description()
+{
+    return "Records an experiment for causal profiling";
+}
+
+const std::atomic<experiment*>&
+experiment::get_current_experiment()
+{
+    return current_experiment;
+}
+
+bool
+experiment::start()
+{
+    if(running && tracing::now() < start_time + experiment_time) return false;
+
+    selection = sample_selection();
+    if(!selection) return false;
+
+    // sampling period in nanoseconds
+    sampling_period = backtrace_causal::get_period(units::nsec);
+    // adjust for the real sampling period
+    period_stats = causal::component::backtrace::get_period_stats();
+    if(period_stats.get_count() > 10) sampling_period = period_stats.get_mean();
+
+    // experiment time is scaled up for longer speedups
+    index           = experiment_history.size() + 1;
+    virtual_speedup = sample_virtual_speedup();
+    delay_scaling   = virtual_speedup / 100.0;
+    if(use_exp_speedup_scaling) scaling_factor *= (1.0 + delay_scaling);
+
+    experiment_time = global_scaling * scaling_factor * sampling_period * batch_size;
+    sample_delay    = sampling_period * delay_scaling;
+    total_delay     = delay::sync();
+    init_progress   = component::progress_point::get_progress_points();
+    start_time      = tracing::now();
+
+    OMNITRACE_VERBOSE(0, "Starting causal experiment #%-3u: %s\n", index,
+                      as_string().c_str());
+
+    current_experiment_value = *this;
+    current_selected_count.store(0);
+    current_experiment.store(this);
+    return true;
+}
+
+bool
+experiment::wait() const
+{
+    auto _now  = tracing::now();
+    auto _wait = experiment_time - (_now - start_time);
+    auto _end  = _now + _wait;
+    auto _incr = std::min<uint64_t>(_wait / 100, 1000000);
+    while(tracing::now() < _end && get_state() < State::Finalized)
+    {
+        std::this_thread::yield();
+        std::this_thread::sleep_for(std::chrono::nanoseconds{ _incr });
+    }
+    return (tracing::now() >= _end);
+}
+
+bool
+experiment::stop()
+{
+    auto _now = tracing::now();
+    if(_now < start_time + experiment_time) return false;
+
+    current_experiment.store(nullptr);
+    selected        = current_selected_count.load();
+    running         = false;
+    end_time        = _now;
+    experiment_time = (end_time - start_time);
+    global_delay    = delay::compute_total_delay(0);
+    total_delay     = (global_delay - total_delay);
+    duration      = (experiment_time > total_delay) ? (experiment_time - total_delay) : 0;
+    fini_progress = component::progress_point::get_progress_points();
+    period_stats  = causal::component::backtrace::get_period_stats();
+
+    // sync data
+    delay::sync();
+
+    // for larger speedups, we increased the experiment time, so we want to artificially
+    // increase num by the same factor. E.g. 10 throughput points at speedup 50 should
+    // really look like 15
+    double _scale_num  = 1.0 + ((use_exp_speedup_scaling) ? delay_scaling : 0.0);
+    auto   _prog_stats = tim::statistics<int64_t>{};
+    for(auto fitr : fini_progress)
+    {
+        auto    _pt = fitr.second - init_progress[fitr.first];
+        int64_t _num =
+            std::max<int64_t>({ _pt.get_laps(), _pt.get_arrival(), _pt.get_departure() });
+        if(_num > 0) _prog_stats += (_num * _scale_num);
+    }
+
+    auto _mean = (_prog_stats.get_count() > 0) ? _prog_stats.get_mean() : 0;
+    auto _high = (_prog_stats.get_count() > 0) ? _prog_stats.get_max() : 0;
+    if(_high < 5)
+    {
+        global_scaling *= 2;
+        ++global_scaling_increments;  // keep track of how many successive increments have
+                                      // been performed
+    }
+    else if(_mean > 10 && global_scaling > 1)
+    {
+        global_scaling /= 2;
+        global_scaling_increments = 0;
+    }
+
+    if(OMNITRACE_UNLIKELY(global_scaling_increments >= 5))
+    {
+        OMNITRACE_WARNING(
+            0,
+            "Warning! causal experimentation hasn't seen at least 5 progress points "
+            "in the last %li experiments. Progress points are necessary for measuring "
+            "the effect of the virtual speed-up. Please visit "
+            "https://amdresearch.github.io/omnitrace/ for documentation on progress "
+            "points and how to add them\n",
+            global_scaling_increments);
+    }
+
+    if(_high > 0) experiment_history.emplace_back(*this);
+
+    std::this_thread::sleep_for(std::chrono::nanoseconds{ sampling_period * batch_size });
+    return true;
+}
+
+std::string
+experiment::as_string() const
+{
+    std::stringstream _ss{};
+    auto _dur = static_cast<double>(experiment_time) / static_cast<double>(units::sec);
+    _ss << std::boolalpha << "speed-up: " << std::setw(3) << virtual_speedup
+        << "%, period: " << std::setw(4) << std::fixed << std::setprecision(2)
+        << (sampling_period / static_cast<double>(units::msec)) << " msec";
+    if(!config::get_causal_end_to_end())
+        _ss << ", duration: " << std::setw(5) << std::fixed << std::setprecision(3)
+            << _dur << " sec";
+    _ss << " :: experiment: " << as_hex(selection.address) << " ";
+    //_ss << " [" << selection.info.ipaddr().as_string() << "]";
+    if(selection.symbol_address > 0 && selection.address != selection.symbol_address)
+        _ss << "(symbol@" << as_hex(selection.symbol_address) << ") ";
+    if(!selection.symbol.file.empty() && selection.symbol.line > 0)
+        _ss << "[" << filepath::basename(selection.symbol.file) << ":"
+            << selection.symbol.line << "]";
+
+    auto _patch = [](std::string _v) {
+        auto _pos       = std::string::npos;
+        using strpair_t = std::pair<std::string_view, std::string>;
+        for(const auto& itr :
+            { strpair_t{
+                  "::basic_string<char, std::char_traits<char>, std::allocator<char> > ",
+                  "::string" },
+              strpair_t{ "::__cxx11::", "::" } })
+        {
+            while((_pos = _v.find(itr.first)) != std::string::npos)
+                _v = _v.replace(_pos, itr.first.length(), itr.second);
+        }
+        return _v;
+    };
+    auto _func = _patch(demangle(selection.symbol.func));
+    _ss << "['" << _func << "']";
+
+    return _ss.str();
+}
+
+// in nanoseconds
+uint64_t
+experiment::get_delay()
+{
+    if(!current_experiment.load()) return 0;
+    return current_experiment_value.sample_delay;
+}
+
+double
+experiment::get_delay_scaling()
+{
+    if(!current_experiment.load()) return 0;
+    return current_experiment_value.delay_scaling;
+}
+
+uint32_t
+experiment::get_index()
+{
+    if(!is_active()) return 0;
+    return current_experiment_value.index;
+}
+
+bool
+experiment::is_active()
+{
+    return (current_experiment.load(std::memory_order_relaxed) != nullptr);
+}
+
+bool
+experiment::is_selected(unwind_addr_t _stack)
+{
+    if(is_active())
+    {
+        for(auto itr : _stack)
+            if(current_experiment_value.selection.contains(itr)) return true;
+    }
+    return false;
+}
+
+void
+experiment::add_selected()
+{
+    if(current_experiment.load() == nullptr) return;
+    ++current_selected_count;
+}
+
+std::vector<experiment>
+experiment::get_experiments()
+{
+    return experiment_history;
+}
+
+void
+experiment::save_experiments()
+{
+    auto _cfg         = settings::compose_filename_config{};
+    _cfg.subdirectory = "causal";
+    _cfg.use_suffix   = config::get_use_pid();
+    save_experiments(config::get_causal_output_filename(), _cfg);
+}
+
+void  // NOLINTNEXTLINE
+experiment::save_experiments(std::string _fname_base, const filename_config_t& _cfg)
+{
+    const auto& _info0 = thread_info::get(0, InternalTID);
+
+    // if(experiment_history.size() > 1)
+    //    experiment_history.erase(experiment_history.begin());
+
+    auto current_record    = record{};
+    current_record.startup = _info0->lifetime.first;
+
+    // update experiments
+    {
+        for(auto& itr : experiment_history)
+        {
+            if(itr.duration == 0 || itr.experiment_time == 0) continue;
+            current_record.experiments.emplace_back(std::move(itr));
+        }
+        experiment_history.clear();
+    }
+
+    // update runtime value
+    {
+        uint64_t _beg_runtime = std::numeric_limits<uint64_t>::max();
+        uint64_t _end_runtime = std::numeric_limits<uint64_t>::min();
+        for(auto& itr : current_record.experiments)
+        {
+            if(itr.duration == 0) continue;
+            if(itr.experiment_time == 0) continue;
+            _beg_runtime = std::min<uint64_t>(_beg_runtime, itr.start_time);
+            _end_runtime = std::max<uint64_t>(_end_runtime, itr.end_time);
+        }
+        current_record.runtime = (_end_runtime - _beg_runtime);
+    }
+
+    // update sample data
+    {
+        auto _add_sample = [&current_record](sample&& _v) {
+            auto fitr = current_record.samples.find(_v);
+            if(fitr != current_record.samples.end())
+                *fitr += _v;
+            else
+                current_record.samples.emplace(std::move(_v));
+        };
+
+        auto _total_samples = std::map<uintptr_t, size_t>{};
+        for(const auto& itr : get_samples())
+        {
+            for(const auto& sitr : itr.second)
+            {
+                _total_samples[sitr.address] += sitr.count;
+            }
+        }
+
+        auto _binfo_cfg         = settings::compose_filename_config{};
+        _binfo_cfg.subdirectory = "causal/binary-info";
+        _binfo_cfg.use_suffix   = config::get_use_pid();
+        save_line_info(_binfo_cfg, config::get_verbose());
+
+        for(const auto& itr : _total_samples)
+        {
+            auto _addr  = itr.first;
+            auto _count = itr.second;
+            if(_count > 0)
+            {
+                auto _linfo = get_line_info(_addr, true);
+                for(const auto& iitr : _linfo)
+                {
+                    auto _name = (iitr.line > 0) ? join(":", iitr.file, iitr.line)
+                                                 : demangle(iitr.func);
+
+                    _name = join(" :: ", as_hex(_addr), _name);
+                    _add_sample(sample{ _count, _addr, _name, iitr });
+                }
+
+                if(_linfo.empty() && config::get_debug())
+                {
+                    _add_sample(
+                        sample{ _count, _addr, as_hex(_addr), sample::line_info{} });
+                }
+            }
+        }
+    }
+
+    bool _causal_output_reset =
+        config::get_setting_value<bool>("OMNITRACE_CAUSAL_FILE_RESET").second;
+
+    // if(current_record.experiments.empty()) return;
+
+    {
+        auto _saved_experiments = (_causal_output_reset)
+                                      ? std::vector<experiment::record>{}
+                                      : load_experiments(_fname_base, _cfg, false);
+        _saved_experiments.emplace_back(current_record);
+        std::stringstream oss{};
+        {
+            auto ar =
+                tim::policy::output_archive<cereal::PrettyJSONOutputArchive>::get(oss);
+
+            ar->setNextName("omnitrace");
+            ar->startNode();
+            ar->setNextName("causal");
+            ar->startNode();
+            (*ar)(cereal::make_nvp("records", _saved_experiments));
+            ar->finishNode();
+            ar->finishNode();
+        }
+
+        auto _fname = tim::settings::compose_output_filename(_fname_base, "json", _cfg);
+        auto ofs    = std::ofstream{};
+        if(tim::filepath::open(ofs, _fname))
+        {
+            if(get_verbose() >= 0)
+                operation::file_output_message<experiment>{}(
+                    _fname, std::string{ "causal_experiments" });
+            ofs << oss.str() << "\n";
+        }
+        else
+        {
+            OMNITRACE_THROW("Error opening causal experiments output file: %s",
+                            _fname.c_str());
+        }
+    }
+
+    auto _fname = tim::settings::compose_output_filename(_fname_base, "coz", _cfg);
+
+    // read in existing data
+    auto _existing = std::stringstream{};
+    if(!_causal_output_reset)
+    {
+        std::ifstream ifs{ _fname };
+        if(ifs)
+        {
+            while(ifs && ifs.good())
+            {
+                std::string _line;
+                std::getline(ifs, _line);
+                _existing << _line << "\n";
+            }
+        }
+    }
+
+    std::ofstream ofs{};
+    ofs.setf(std::ios::fixed);
+    if(tim::filepath::open(ofs, _fname))
+    {
+        if(get_verbose() >= 0)
+            operation::file_output_message<experiment>{}(
+                _fname, std::string{ "causal_experiments" });
+
+        ofs << _existing.str();
+        ofs << "startup\ttime=" << current_record.startup << "\n";
+
+        for(auto& itr : current_record.experiments)
+        {
+            auto& _selection = itr.selection;
+            auto& _line_info = _selection.symbol;
+
+            std::string _name = (_selection.symbol_address > 0)
+                                    ? _line_info.func
+                                    : join(":", _line_info.file, _line_info.line);
+
+            OMNITRACE_CONDITIONAL_THROW(
+                _name.empty(),
+                "Error! causal experiment selection has no name: address=%s, file=%s, "
+                "line=%u, func=%s",
+                as_hex(_line_info.address).c_str(), _line_info.file.c_str(),
+                _line_info.line, _line_info.func.c_str());
+
+            ofs << "experiment\tselected=" << demangle(_name)
+                << "\tspeedup=" << std::setprecision(2)
+                << static_cast<double>(itr.virtual_speedup / 100.0)
+                << "\tduration=" << itr.duration << "\tselected-samples=" << itr.selected
+                << "\n";
+
+            auto ppts = itr.fini_progress;
+            for(auto pitr : itr.init_progress)
+                ppts[pitr.first] -= pitr.second;
+
+            for(auto pitr : ppts)
+            {
+                // if(pitr.second.get_laps() == 0) continue;
+                if(get_causal_end_to_end() && pitr.second.get_laps() > 1) continue;
+                if(pitr.second.is_throughput_point() && pitr.second.get_delta() != 0)
+                {
+                    ofs << "throughput-point\tname="
+                        << tim::demangle(tim::get_hash_identifier(pitr.first))
+                        << "\tdelta=" << pitr.second.get_delta() << "\n";
+                    if(get_causal_end_to_end()) break;
+                }
+                if(pitr.second.is_latency_point())
+                {
+                    if(get_causal_end_to_end()) continue;
+                    auto _delta = std::max<int64_t>(pitr.second.get_latency_delta(), 1);
+                    ofs << "latency-point\tname="
+                        << tim::demangle(tim::get_hash_identifier(pitr.first))
+                        << "\tarrivals=" << pitr.second.get_arrival()
+                        << "\tdepartures=" << pitr.second.get_departure()
+                        << "\tdifference=" << _delta << "\n";
+                }
+            }
+        }
+
+        ofs << "runtime\ttime=" << current_record.runtime << "\n";
+
+        for(const auto& itr : current_record.samples)
+        {
+            ofs << "samples\tlocation=" << itr.location << "\tcount=" << itr.count;
+            if(config::get_debug()) ofs << "\taddress=" << as_hex(itr.address);
+            ofs << "\n";
+        }
+    }
+    else
+    {
+        OMNITRACE_THROW("Error opening causal experiments output file: %s",
+                        _fname.c_str());
+    }
+}
+
+std::vector<experiment::record>
+experiment::load_experiments(bool _throw_on_error)
+{
+    auto _cfg         = settings::compose_filename_config{};
+    _cfg.subdirectory = "causal";
+    _cfg.use_suffix   = config::get_use_pid();
+    return load_experiments(config::get_causal_output_filename(), _cfg, _throw_on_error);
+}
+
+std::vector<experiment::record>
+experiment::load_experiments(std::string _fname, const filename_config_t& _cfg,
+                             bool _throw_on_error)
+{
+    _fname = tim::settings::compose_input_filename(_fname, "json", _cfg);
+
+    auto ifs   = std::ifstream{};
+    auto _data = std::vector<experiment::record>{};
+    if(tim::filepath::open(ifs, _fname))
+    {
+        auto ar = tim::policy::input_archive<cereal::JSONInputArchive>::get(ifs);
+
+        ar->setNextName("omnitrace");
+        ar->startNode();
+        ar->setNextName("causal");
+        ar->startNode();
+        (*ar)(cereal::make_nvp("records", _data));
+        ar->finishNode();
+        ar->finishNode();
+    }
+    else
+    {
+        if(_throw_on_error)
+        {
+            OMNITRACE_THROW("Error opening causal experiments input file: %s",
+                            _fname.c_str());
+        }
+    }
+
+    return _data;
+}
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,139 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/binary/dwarf_entry.hpp"
+#include "library/binary/symbol.hpp"
+#include "library/causal/components/backtrace.hpp"
+#include "library/causal/components/progress_point.hpp"
+#include "library/causal/data.hpp"
+#include "library/causal/sample_data.hpp"
+#include "library/causal/selected_entry.hpp"
+#include "library/containers/c_array.hpp"
+#include "library/defines.hpp"
+#include "library/utility.hpp"
+
+#include <timemory/hash/types.hpp>
+#include <timemory/mpl/concepts.hpp>
+#include <timemory/tpls/cereal/cereal.hpp>
+#include <timemory/tpls/cereal/cereal/cereal.hpp>
+#include <timemory/utility/unwind.hpp>
+
+#include <atomic>
+#include <cstdint>
+#include <unordered_map>
+
+namespace omnitrace
+{
+namespace causal
+{
+using hash_value_t = ::tim::hash_value_t;
+
+struct experiment
+{
+    using progress_points_t =
+        std::unordered_map<tim::hash_value_t, component::progress_point>;
+    using experiments_t     = std::vector<experiment>;
+    using filename_config_t = settings::compose_filename_config;
+    using sample_dataset_t  = std::set<sample_data>;
+    using period_stats_t    = tim::statistics<int64_t>;
+
+    struct sample
+    {
+        using line_info = binary::symbol;
+
+        mutable uint64_t count    = 0;
+        uintptr_t        address  = 0;
+        std::string      location = {};
+        line_info        info     = {};
+
+        bool        operator==(const sample&) const;
+        bool        operator<(const sample&) const;
+        const auto& operator+=(const sample&) const;
+
+        template <typename ArchiveT>
+        void serialize(ArchiveT& ar, const unsigned);
+    };
+
+    struct record
+    {
+        int64_t                 startup     = 0;
+        uint64_t                runtime     = 0;
+        std::vector<experiment> experiments = {};
+        std::set<sample>        samples     = {};
+
+        template <typename ArchiveT>
+        void serialize(ArchiveT& ar, const unsigned);
+    };
+
+    static std::string                     label();
+    static std::string                     description();
+    static const std::atomic<experiment*>& get_current_experiment();
+
+    TIMEMORY_DEFAULT_OBJECT(experiment)
+
+    bool        start();
+    bool        wait() const;  // returns false if interrupted
+    bool        stop();
+    std::string as_string() const;
+
+    template <typename ArchiveT>
+    void serialize(ArchiveT& ar, const unsigned version);
+
+    // in nanoseconds
+    static uint64_t      get_delay();
+    static double        get_delay_scaling();
+    static uint32_t      get_index();
+    static bool          is_active();
+    static bool          is_selected(unwind_addr_t);
+    static void          add_selected();
+    static experiments_t get_experiments();
+
+    static void                save_experiments();
+    static void                save_experiments(std::string, const filename_config_t&);
+    static std::vector<record> load_experiments(bool _throw_on_err = true);
+    static std::vector<record> load_experiments(std::string, const filename_config_t&,
+                                                bool = true);
+
+    bool              running         = false;
+    uint16_t          virtual_speedup = 0;   /// 0-100 in multiples of 5
+    uint32_t          index           = 0;   /// experiment number
+    uint64_t          sampling_period = 0;   /// period b/t samples [nsec]
+    uint64_t          start_time      = 0;   /// start of experiment [nsec]
+    uint64_t          end_time        = 0;   /// end of experiment [nsec]
+    uint64_t          experiment_time = 0;   /// how long the experiment ran [nsec]
+    uint64_t          duration        = 0;   /// runtime - delays [nsec]
+    uint64_t          batch_size      = 10;  /// batch factor for experiment/cooloff
+    uint64_t          scaling_factor  = 50;  /// scaling factor for experiment time
+    uint64_t          sample_delay    = 0;   /// how long to delay [nsec]
+    uint64_t          total_delay     = 0;   /// total delays [nsec]
+    uint64_t          selected        = 0;   /// num times selected line sampled
+    uint64_t          global_delay    = 0;
+    double            delay_scaling   = 0.0;  /// virtual_speedup / 100.
+    selected_entry    selection       = {};   /// which line was selected
+    progress_points_t init_progress   = {};   /// progress points at start
+    progress_points_t fini_progress   = {};   /// progress points at end
+    period_stats_t    period_stats    = {};   /// stats for sampling period
+};
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,52 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/binary/fwd.hpp"
+#include "library/containers/static_vector.hpp"
+#include "library/defines.hpp"
+
+#include <timemory/hash/types.hpp>
+#include <timemory/unwind/stack.hpp>
+#include <timemory/utility/procfs/maps.hpp>
+
+#include <cstddef>
+#include <cstdint>
+#include <map>
+#include <utility>
+
+namespace omnitrace
+{
+namespace unwind = ::tim::unwind;
+
+namespace causal
+{
+static constexpr size_t unwind_depth  = 8;
+static constexpr size_t unwind_offset = 0;
+using unwind_stack_t                  = unwind::stack<unwind_depth>;
+using unwind_addr_t                   = container::static_vector<uintptr_t, unwind_depth>;
+using hash_value_t                    = tim::hash_value_t;
+
+struct selected_entry;
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,555 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/causal/perf.hpp"
+#include "library/timemory.hpp"
+#include "library/utility.hpp"
+
+#include <timemory/log/logger.hpp>
+#include <timemory/log/macros.hpp>
+#include <timemory/units.hpp>
+
+#include <asm/unistd.h>
+#include <fcntl.h>
+#include <linux/perf_event.h>
+#include <poll.h>
+#include <signal.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace perf
+{
+namespace
+{
+struct SizeParams
+{
+    const size_t num_pages = 2;
+    const size_t page      = units::get_page_size();
+    const size_t data      = num_pages * page;
+    const size_t mmap      = data + page;
+};
+const SizeParams sizes = {};
+}  // namespace
+
+long
+perf_event_open(struct perf_event_attr* hw_event, pid_t _pid, int _cpu, int group_fd,
+                unsigned long flags)
+{
+    return syscall(__NR_perf_event_open, hw_event, _pid, _cpu, group_fd, flags);
+}
+
+/// Move constructor
+perf_event::perf_event(perf_event&& rhs) noexcept
+{
+    // Release resources if the current perf_event is initialized and not equal to this
+    // one
+    if(m_fd != -1 && m_fd != rhs.m_fd)
+    {
+        ::close(m_fd);
+        TIMEMORY_INFO << "Closed perf event fd " << m_fd;
+    }
+
+    if(m_mapping != nullptr && m_mapping != rhs.m_mapping) munmap(m_mapping, sizes.mmap);
+
+    // take rhs perf event's file descriptor and replace it with -1
+    m_fd     = rhs.m_fd;
+    rhs.m_fd = -1;
+
+    // take rhs perf_event's mapping and replace it with nullptr
+    m_mapping     = rhs.m_mapping;
+    rhs.m_mapping = nullptr;
+
+    // Copy over the sample type and read format
+    m_sample_type = rhs.m_sample_type;
+    m_read_format = rhs.m_read_format;
+}
+
+/// Close the perf_event file descriptor and unmap the ring buffer
+perf_event::~perf_event() { close(); }
+
+/// Move assignment
+perf_event&
+perf_event::operator=(perf_event&& rhs) noexcept
+{
+    if(&rhs == this) return *this;
+
+    // Release resources if the current perf_event is initialized and not equal to this
+    // one
+    if(m_fd != -1 && m_fd != rhs.m_fd) ::close(m_fd);
+    if(m_mapping != nullptr && m_mapping != rhs.m_mapping) munmap(m_mapping, sizes.mmap);
+
+    // take rhs perf event's file descriptor and replace it with -1
+    m_fd     = rhs.m_fd;
+    rhs.m_fd = -1;
+
+    // take rhs perf_event's mapping and replace it with nullptr
+    m_mapping     = rhs.m_mapping;
+    rhs.m_mapping = nullptr;
+
+    // Copy over the sample type and read format
+    m_sample_type = rhs.m_sample_type;
+    m_read_format = rhs.m_read_format;
+
+    return *this;
+}
+
+// Open a perf_event file and map it (if sampling is enabled)
+bool
+perf_event::open(struct perf_event_attr& _pe, pid_t _pid, int _cpu)
+{
+    m_sample_type = _pe.sample_type;
+    m_read_format = _pe.read_format;
+
+    // Set some mandatory fields
+    _pe.size     = sizeof(struct perf_event_attr);
+    _pe.disabled = 1;
+
+    // Open the file
+    m_fd = perf_event_open(&_pe, _pid, _cpu, -1, 0);
+    if(m_fd == -1)
+    {
+        std::string path = "/proc/sys/kernel/perf_event_paranoid";
+
+        FILE* file = fopen(path.c_str(), "r");
+        OMNITRACE_PREFER(file != nullptr)
+            << "Failed to open " << path << ": " << strerror(errno);
+
+        if(file == nullptr) return false;
+
+        char value_str[3];
+        int  res = fread(value_str, sizeof(value_str), 1, file);
+        TIMEMORY_REQUIRE(res != -1)
+            << "Failed to read from " << path << ": " << strerror(errno);
+
+        if(res == -1) return false;
+
+        value_str[2] = '\0';
+        int value    = atoi(value_str);
+
+        TIMEMORY_WARNING << "Failed to open perf event. "
+                         << "Consider tweaking " << path << " to 2 or less "
+                         << "(current value is " << value << "), "
+                         << "or run omnitrace as a privileged user (with CAP_SYS_ADMIN).";
+        return false;
+    }
+
+    // If sampling, map the perf event file
+    if(_pe.sample_type != 0 && _pe.sample_period != 0)
+    {
+        void* ring_buffer =
+            mmap(nullptr, sizes.mmap, PROT_READ | PROT_WRITE, MAP_SHARED, m_fd, 0);
+
+        OMNITRACE_PREFER(ring_buffer != MAP_FAILED)
+            << "Mapping perf_event ring buffer failed. "
+            << "Make sure the current user has permission "
+               "to invoke the perf tool, and that "
+            << "the program being profiled does not use "
+               "an excessive number of threads (>1000).\n";
+
+        if(ring_buffer == MAP_FAILED) return false;
+
+        m_mapping = reinterpret_cast<struct perf_event_mmap_page*>(ring_buffer);
+    }
+
+    return true;
+}
+
+bool
+perf_event::open(double _freq, uint32_t _batch_size, pid_t _pid, int _cpu)
+{
+    uint64_t               _period = (1.0 / _freq) * units::sec;
+    struct perf_event_attr _pe;
+
+    memset(&_pe, 0, sizeof(_pe));
+    _pe.type           = PERF_TYPE_SOFTWARE;
+    _pe.config         = PERF_COUNT_SW_TASK_CLOCK;
+    _pe.sample_type    = PERF_SAMPLE_IP | PERF_SAMPLE_CALLCHAIN;
+    _pe.sample_period  = _period;
+    _pe.wakeup_events  = _batch_size;
+    _pe.sample_period  = _period;
+    _pe.wakeup_events  = _batch_size;  // This is ignored on linux 3.13 (why?)
+    _pe.exclude_idle   = 1;
+    _pe.exclude_kernel = 1;
+    _pe.precise_ip     = 0;
+    _pe.disabled       = 1;
+
+    return open(_pe, _pid, _cpu);
+}
+
+/// Read event count
+uint64_t
+perf_event::get_count() const
+{
+    uint64_t count;
+    TIMEMORY_REQUIRE(read(m_fd, &count, sizeof(uint64_t)) == sizeof(uint64_t))
+        << "Failed to read event count from perf_event file";
+    return count;
+}
+
+/// Start counting events
+void
+perf_event::start() const
+{
+    if(m_fd != -1)
+    {
+        TIMEMORY_REQUIRE(ioctl(m_fd, PERF_EVENT_IOC_ENABLE, 0) != -1)
+            << "Failed to start perf event: " << strerror(errno);
+    }
+}
+
+/// Stop counting events
+void
+perf_event::stop() const
+{
+    if(m_fd != -1)
+    {
+        TIMEMORY_REQUIRE(ioctl(m_fd, PERF_EVENT_IOC_DISABLE, 0) != -1)
+            << "Failed to stop perf event: " << strerror(errno) << " (" << m_fd << ")";
+    }
+}
+
+void
+perf_event::close()
+{
+    if(m_fd != -1)
+    {
+        ::close(m_fd);
+        m_fd = -1;
+    }
+
+    if(m_mapping != nullptr)
+    {
+        munmap(m_mapping, sizes.mmap);
+        m_mapping = nullptr;
+    }
+}
+
+void
+perf_event::set_ready_signal(int sig) const
+{
+    // Set the perf_event file to async
+    TIMEMORY_REQUIRE(fcntl(m_fd, F_SETFL, fcntl(m_fd, F_GETFL, 0) | O_ASYNC) != -1)
+        << "failed to set perf_event file to async mode";
+
+    // Set the notification signal for the perf file
+    TIMEMORY_REQUIRE(fcntl(m_fd, F_SETSIG, sig) != -1)
+        << "failed to set perf_event file signal";
+
+    // Set the current thread as the owner of the file (to target signal delivery)
+    TIMEMORY_REQUIRE(fcntl(m_fd, F_SETOWN, gettid()) != -1)
+        << "failed to set the owner of the perf_event file";
+}
+
+void
+perf_event::iterator::next()
+{
+    struct perf_event_header _hdr;
+
+    // Copy out the record header
+    perf_event::copy_from_ring_buffer(m_mapping, m_index, &_hdr,
+                                      sizeof(struct perf_event_header));
+
+    // Advance to the next record
+    m_index += _hdr.size;
+}
+
+perf_event::iterator::iterator(perf_event& _source, struct perf_event_mmap_page* _mapping)
+: m_source{ _source }
+, m_mapping{ _mapping }
+{
+    if(_mapping != nullptr)
+    {
+        m_index = _mapping->data_tail;
+        m_head  = _mapping->data_head;
+    }
+    else
+    {
+        m_index = 0;
+        m_head  = 0;
+    }
+}
+
+perf_event::iterator::~iterator()
+{
+    if(m_mapping != nullptr)
+    {
+        m_mapping->data_tail = m_index;
+    }
+}
+
+perf_event::iterator&
+perf_event::iterator::operator++()
+{
+    next();
+    return *this;
+}
+
+bool
+perf_event::iterator::operator!=(const iterator& other) const
+{
+    return has_data() != other.has_data();
+}
+
+perf_event::record
+perf_event::iterator::get()
+{
+    // Copy out the record header
+    perf_event::copy_from_ring_buffer(m_mapping, m_index, _buf,
+                                      sizeof(struct perf_event_header));
+
+    // Get a pointer to the header
+    struct perf_event_header* header = reinterpret_cast<struct perf_event_header*>(_buf);
+
+    // Copy out the entire record
+    perf_event::copy_from_ring_buffer(m_mapping, m_index, _buf, header->size);
+
+    return perf_event::record(m_source, header);
+}
+
+bool
+perf_event::iterator::has_data() const
+{
+    // If there is no ring buffer, there is no data
+    if(m_mapping == nullptr)
+    {
+        return false;
+    }
+
+    // If there isn't enough data in the ring buffer to hold a header, there is no data
+    if(m_index + sizeof(struct perf_event_header) >= m_head)
+    {
+        return false;
+    }
+
+    struct perf_event_header _hdr;
+    perf_event::copy_from_ring_buffer(m_mapping, m_index, &_hdr,
+                                      sizeof(struct perf_event_header));
+
+    // If the first record is larger than the available data, nothing can be read
+    if(m_index + _hdr.size > m_head)
+    {
+        return false;
+    }
+
+    return true;
+}
+
+void
+perf_event::copy_from_ring_buffer(struct perf_event_mmap_page* _mapping, ptrdiff_t _index,
+                                  void* _dest, size_t _nbytes)
+{
+    uintptr_t _base    = reinterpret_cast<uintptr_t>(_mapping) + sizes.page;
+    size_t    _beg_idx = _index % sizes.data;
+    size_t    _end_idx = _beg_idx + _nbytes;
+
+    if(_end_idx <= sizes.data)
+    {
+        memcpy(_dest, reinterpret_cast<void*>(_base + _beg_idx), _nbytes);
+    }
+    else
+    {
+        size_t _chunk_size2 = _end_idx - sizes.data;
+        size_t _chunk_size1 = _nbytes - _chunk_size2;
+
+        void* _dest2 =
+            reinterpret_cast<void*>(reinterpret_cast<uintptr_t>(_dest) + _chunk_size1);
+
+        memcpy(_dest, reinterpret_cast<void*>(_base + _beg_idx), _chunk_size1);
+        memcpy(_dest2, reinterpret_cast<void*>(_base), _chunk_size2);
+    }
+}
+
+uint64_t
+perf_event::record::get_ip() const
+{
+    TIMEMORY_ASSERT(is_sample() && m_source.is_sampling(sample::ip))
+        << "Record does not have an ip field";
+    return *locate_field<sample::ip, uint64_t*>();
+}
+
+uint64_t
+perf_event::record::get_pid() const
+{
+    TIMEMORY_ASSERT(is_sample() && m_source.is_sampling(sample::pid_tid))
+        << "Record does not have a `pid` field";
+    return locate_field<sample::pid_tid, uint32_t*>()[0];
+}
+
+uint64_t
+perf_event::record::get_tid() const
+{
+    TIMEMORY_ASSERT(is_sample() && m_source.is_sampling(sample::pid_tid))
+        << "Record does not have a `tid` field";
+    return locate_field<sample::pid_tid, uint32_t*>()[1];
+}
+
+uint64_t
+perf_event::record::get_time() const
+{
+    TIMEMORY_ASSERT(is_sample() && m_source.is_sampling(sample::time))
+        << "Record does not have a 'time' field";
+    return *locate_field<sample::time, uint64_t*>();
+}
+
+uint32_t
+perf_event::record::get_cpu() const
+{
+    TIMEMORY_ASSERT(is_sample() && m_source.is_sampling(sample::cpu))
+        << "Record does not have a 'cpu' field";
+    return *locate_field<sample::cpu, uint32_t*>();
+}
+
+container::c_array<uint64_t>
+perf_event::record::get_callchain() const
+{
+    TIMEMORY_ASSERT(is_sample() && m_source.is_sampling(sample::callchain))
+        << "Record does not have a callchain field";
+
+    uint64_t* _base = locate_field<sample::callchain, uint64_t*>();
+    uint64_t  _size = *_base;
+    // Advance the callchain array pointer past the size
+    _base++;
+    return container::wrap_c_array(_base, _size);
+}
+
+template <sample SampleT, typename Tp>
+Tp
+perf_event::record::locate_field() const
+{
+    uintptr_t p =
+        reinterpret_cast<uintptr_t>(m_header) + sizeof(struct perf_event_header);
+
+    // Walk through the fields in the sample structure. Once the requested field is
+    // reached, return. Skip past any unrequested fields that are included in the sample
+    // type
+
+    // ip
+    if constexpr(SampleT == sample::ip) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::ip)) p += sizeof(uint64_t);
+
+    // pid, tid
+    if constexpr(SampleT == sample::pid_tid) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::pid_tid)) p += sizeof(uint32_t) + sizeof(uint32_t);
+
+    // time
+    if constexpr(SampleT == sample::time) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::time)) p += sizeof(uint64_t);
+
+    // addr
+    if constexpr(SampleT == sample::addr) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::addr)) p += sizeof(uint64_t);
+
+    // id
+    if constexpr(SampleT == sample::id) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::id)) p += sizeof(uint64_t);
+
+    // stream_id
+    if constexpr(SampleT == sample::stream_id) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::stream_id)) p += sizeof(uint64_t);
+
+    // cpu
+    if constexpr(SampleT == sample::cpu) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::cpu)) p += sizeof(uint32_t) + sizeof(uint32_t);
+
+    // period
+    if constexpr(SampleT == sample::period) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::period)) p += sizeof(uint64_t);
+
+    // value
+    if constexpr(SampleT == sample::read) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::read))
+    {
+        uint64_t read_format = m_source.get_read_format();
+        if(read_format & PERF_FORMAT_GROUP)
+        {
+            // Get the number of values in the read format structure
+            uint64_t nr = *reinterpret_cast<uint64_t*>(p);
+            // The default size of each entry is a u64
+            size_t sz = sizeof(uint64_t);
+            // If requested, the id will be included with each value
+            if(read_format & PERF_FORMAT_ID) sz += sizeof(uint64_t);
+            // Skip over the entry count, and each entry
+            p += sizeof(uint64_t) + nr * sz;
+        }
+        else
+        {
+            // Skip over the value
+            p += sizeof(uint64_t);
+            // Skip over the id, if included
+            if(read_format & PERF_FORMAT_ID) p += sizeof(uint64_t);
+        }
+
+        // Skip over the time_enabled field
+        if(read_format & PERF_FORMAT_TOTAL_TIME_ENABLED) p += sizeof(uint64_t);
+        // Skip over the time_running field
+        if(read_format & PERF_FORMAT_TOTAL_TIME_RUNNING) p += sizeof(uint64_t);
+    }
+
+    // callchain
+    if constexpr(SampleT == sample::callchain) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::callchain))
+    {
+        uint64_t nr = *reinterpret_cast<uint64_t*>(p);
+        p += sizeof(uint64_t) + nr * sizeof(uint64_t);
+    }
+
+    // raw
+    if constexpr(SampleT == sample::raw) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::raw))
+    {
+        uint32_t raw_size = *reinterpret_cast<uint32_t*>(p);
+        p += sizeof(uint32_t) + raw_size;
+    }
+
+    // branch_stack
+    if constexpr(SampleT == sample::branch_stack) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::branch_stack))
+        TIMEMORY_FATAL << "Branch stack sampling is not supported";
+
+    // regs
+    if constexpr(SampleT == sample::regs) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::regs))
+        TIMEMORY_FATAL << "Register sampling is not supported";
+
+    // stack
+    if constexpr(SampleT == sample::stack) return reinterpret_cast<Tp>(p);
+    if(m_source.is_sampling(sample::stack))
+        TIMEMORY_FATAL << "Stack sampling is not supported";
+
+    // end
+    if constexpr(SampleT == sample::last) return reinterpret_cast<Tp>(p);
+
+    TIMEMORY_FATAL << "Unsupported sample field requested!";
+}
+}  // namespace perf
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,256 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/containers/c_array.hpp"
+#include "library/defines.hpp"
+
+#include <cstddef>
+#include <cstdint>
+#include <functional>
+#include <linux/perf_event.h>
+#include <sys/types.h>
+
+#if __GLIBC__ == 2 && __GLIBC_MINOR__ < 30
+#    include <sys/syscall.h>
+#    define gettid() syscall(SYS_gettid)
+#endif
+
+// Workaround for missing hw_breakpoint.h include file:
+//   This include file just defines constants used to configure watchpoint registers.
+//   This will be constant across x86 systems.
+enum
+{
+    HW_BREAKPOINT_X = 4
+};
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace perf
+{
+/// An enum class with all the available sampling data
+enum class sample : uint64_t
+{
+    ip        = PERF_SAMPLE_IP,
+    pid_tid   = PERF_SAMPLE_TID,
+    time      = PERF_SAMPLE_TIME,
+    addr      = PERF_SAMPLE_ADDR,
+    id        = PERF_SAMPLE_ID,
+    stream_id = PERF_SAMPLE_STREAM_ID,
+    cpu       = PERF_SAMPLE_CPU,
+    period    = PERF_SAMPLE_PERIOD,
+
+#if defined(PREF_SAMPLE_READ)
+    read = PERF_SAMPLE_READ,
+#else
+    read         = 0,
+#endif
+
+    callchain = PERF_SAMPLE_CALLCHAIN,
+    raw       = PERF_SAMPLE_RAW,
+
+#if defined(PERF_SAMPLE_BRANCH_STACK)
+    branch_stack = PERF_SAMPLE_BRANCH_STACK,
+#else
+    branch_stack = 0,
+#endif
+
+#if defined(PERF_SAMPLE_REGS_USER)
+    regs = PERF_SAMPLE_REGS_USER,
+#else
+    regs         = 0,
+#endif
+
+#if defined(PERF_SAMPLE_STACK_USER)
+    stack = PERF_SAMPLE_STACK_USER,
+#else
+    stack        = 0,
+#endif
+
+    last = PERF_SAMPLE_MAX
+};
+
+struct perf_event
+{
+    enum class record_type;
+    struct record;
+    struct sample_record;
+
+    /// Default constructor
+    perf_event() = default;
+    /// Move constructor
+    perf_event(perf_event&& other) noexcept;
+
+    /// Close the perf event file and unmap the ring buffer
+    ~perf_event();
+
+    /// Move assignment is supported
+    perf_event& operator=(perf_event&& other) noexcept;
+
+    perf_event(const perf_event&) = delete;
+    perf_event& operator=(const perf_event&) = delete;
+
+    /// Open a perf_event file using the given options structure
+    bool open(struct perf_event_attr& pe, pid_t pid = 0, int cpu = -1);
+    bool open(double, uint32_t, pid_t pid = 0, int cpu = -1);
+
+    /// Read event count
+    uint64_t get_count() const;
+
+    /// Start counting events and collecting samples
+    void start() const;
+
+    /// Stop counting events
+    void stop() const;
+
+    /// Close the perf_event file and unmap the ring buffer
+    void close();
+
+    /// Configure the perf_event file to deliver a signal when samples are ready to be
+    /// processed
+    void set_ready_signal(int sig) const;
+
+    /// Check if this perf_event was configured to collect a type of sample data
+    inline bool is_sampling(sample s) const
+    {
+        return (m_sample_type & static_cast<uint64_t>(s)) != 0u;
+    }
+
+    /// Get the configuration for this perf_event's read format
+    inline uint64_t get_read_format() const { return m_read_format; }
+
+    /// An enum to distinguish types of records in the mmapped ring buffer
+    enum class record_type
+    {
+        mmap       = PERF_RECORD_MMAP,
+        lost       = PERF_RECORD_LOST,
+        comm       = PERF_RECORD_COMM,
+        exit       = PERF_RECORD_EXIT,
+        throttle   = PERF_RECORD_THROTTLE,
+        unthrottle = PERF_RECORD_UNTHROTTLE,
+        fork       = PERF_RECORD_FORK,
+        read       = PERF_RECORD_READ,
+        sample     = PERF_RECORD_SAMPLE,
+
+#if defined(PERF_RECORD_MMAP2)
+        mmap2 = PERF_RECORD_MMAP2
+#else
+        mmap2 = 0
+#endif
+    };
+
+    class iterator;
+
+    /// A generic record type
+    struct record
+    {
+        friend class perf_event::iterator;
+
+        record_type get_type() const { return static_cast<record_type>(m_header->type); }
+
+        inline bool is_mmap() const { return get_type() == record_type::mmap; }
+        inline bool is_lost() const { return get_type() == record_type::lost; }
+        inline bool is_comm() const { return get_type() == record_type::comm; }
+        inline bool is_exit() const { return get_type() == record_type::exit; }
+        inline bool is_throttle() const { return get_type() == record_type::throttle; }
+        inline bool is_unthrottle() const
+        {
+            return get_type() == record_type::unthrottle;
+        }
+        inline bool is_fork() const { return get_type() == record_type::fork; }
+        inline bool is_read() const { return get_type() == record_type::read; }
+        inline bool is_sample() const { return get_type() == record_type::sample; }
+        inline bool is_mmap2() const { return get_type() == record_type::mmap2; }
+
+        uint64_t                     get_ip() const;
+        uint64_t                     get_pid() const;
+        uint64_t                     get_tid() const;
+        uint64_t                     get_time() const;
+        uint32_t                     get_cpu() const;
+        container::c_array<uint64_t> get_callchain() const;
+
+    private:
+        record(const perf_event& source, struct perf_event_header* header)
+        : m_source(source)
+        , m_header(header)
+        {}
+
+        template <sample SampleT, typename Tp = void*>
+        Tp locate_field() const;
+
+        const perf_event&         m_source;
+        struct perf_event_header* m_header;
+    };
+
+    class iterator
+    {
+    public:
+        iterator(perf_event& source, struct perf_event_mmap_page* mapping);
+        ~iterator();
+
+        void   next();
+        record get();
+        bool   has_data() const;
+
+        iterator& operator++();
+        record    operator*() { return get(); }
+        bool      operator!=(const iterator& other) const;
+
+    private:
+        perf_event&                  m_source;
+        size_t                       m_index   = 0;
+        size_t                       m_head    = 0;
+        struct perf_event_mmap_page* m_mapping = nullptr;
+
+        // Buffer to hold the current record. Just a hack until records play nice with the
+        // ring buffer
+        uint8_t _buf[4096];
+    };
+
+    /// Get an iterator to the beginning of the memory mapped ring buffer
+    iterator begin() { return iterator(*this, m_mapping); }
+
+    // Get an iterator to the end of the memory mapped ring buffer
+    iterator end() { return iterator(*this, nullptr); }
+
+private:
+    // Copy data out of the mmap ring buffer
+    static void copy_from_ring_buffer(struct perf_event_mmap_page* mapping,
+                                      ptrdiff_t index, void* dest, size_t bytes);
+
+    /// File descriptor for the perf event
+    long m_fd = -1;
+
+    /// Memory mapped perf event region
+    struct perf_event_mmap_page* m_mapping = nullptr;
+
+    /// The sample type from this perf_event's configuration
+    uint64_t m_sample_type = 0;
+    /// The read format from this perf event's configuration
+    uint64_t m_read_format = 0;
+};
+}  // namespace perf
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,71 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/causal/sample_data.hpp"
+
+#include <chrono>
+#include <cstdint>
+#include <map>
+#include <set>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace
+{
+auto samples = std::map<uint32_t, std::set<sample_data>>{};
+}
+
+std::set<sample_data>
+get_samples(uint32_t _index)
+{
+    return samples[_index];
+}
+
+std::map<uint32_t, std::set<sample_data>>
+get_samples()
+{
+    return samples;
+}
+
+void
+add_sample(uint32_t _index, uintptr_t _v)
+{
+    auto& _samples = samples[_index];
+    auto  _value   = sample_data{ _v };
+    _value.count   = 1;
+    auto itr       = _samples.find(_value);
+    if(itr == _samples.end())
+        _samples.emplace(_value);
+    else
+        itr->count += 1;
+}
+
+void
+add_samples(uint32_t _index, const std::vector<uintptr_t>& _v)
+{
+    for(const auto& itr : _v)
+        add_sample(_index, itr);
+}
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,64 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/defines.hpp"
+#include "library/timemory.hpp"
+
+#include <chrono>
+#include <cstdint>
+
+namespace omnitrace
+{
+namespace causal
+{
+struct sample_data
+{
+    uintptr_t        address = 0x0;
+    mutable uint64_t count   = 0;
+
+    bool operator==(sample_data _v) const { return (address == _v.address); }
+    bool operator!=(sample_data _v) const { return !(*this == _v); }
+    bool operator<(sample_data _v) const { return (address < _v.address); }
+    bool operator>(sample_data _v) const { return (address > _v.address); }
+    bool operator<=(sample_data _v) const { return (address <= _v.address); }
+    bool operator>=(sample_data _v) const { return (address >= _v.address); }
+
+    template <typename ArchiveT>
+    void serialize(ArchiveT& ar, const unsigned)
+    {
+        ar(cereal::make_nvp("address", address), cereal::make_nvp("count", count));
+    }
+};
+
+std::map<uint32_t, std::set<sample_data>>
+get_samples();
+
+void
+add_samples(uint32_t, const std::vector<uintptr_t>&);
+
+std::set<sample_data> get_samples(uint32_t);
+
+void add_sample(uint32_t, uintptr_t);
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,375 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/causal/sampling.hpp"
+#include "library/causal/components/backtrace.hpp"
+#include "library/causal/data.hpp"
+#include "library/common.hpp"
+#include "library/concepts.hpp"
+#include "library/config.hpp"
+#include "library/debug.hpp"
+#include "library/ptl.hpp"
+#include "library/runtime.hpp"
+#include "library/sampling.hpp"
+#include "library/state.hpp"
+#include "library/thread_data.hpp"
+#include "library/thread_info.hpp"
+#include "library/utility.hpp"
+
+#include <timemory/macros.hpp>
+#include <timemory/sampling/allocator.hpp>
+#include <timemory/sampling/sampler.hpp>
+#include <timemory/units.hpp>
+#include <timemory/utility/backtrace.hpp>
+#include <timemory/variadic.hpp>
+
+#include <csignal>
+#include <cstring>
+#include <ctime>
+#include <mutex>
+#include <sstream>
+#include <string>
+#include <type_traits>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace sampling
+{
+using ::tim::sampling::dynamic;
+using ::tim::sampling::timer;
+
+using causal_bundle_t  = tim::lightweight_tuple<causal::component::backtrace>;
+using causal_sampler_t = tim::sampling::sampler<causal_bundle_t, dynamic>;
+}  // namespace sampling
+}  // namespace causal
+}  // namespace omnitrace
+
+OMNITRACE_DEFINE_CONCRETE_TRAIT(prevent_reentry, causal::sampling::causal_sampler_t,
+                                std::true_type)
+
+OMNITRACE_DEFINE_CONCRETE_TRAIT(provide_backtrace, causal::sampling::causal_sampler_t,
+                                std::false_type)
+
+OMNITRACE_DEFINE_CONCRETE_TRAIT(buffer_size, causal::sampling::causal_sampler_t,
+                                TIMEMORY_ESC(std::integral_constant<size_t, 4096>))
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace sampling
+{
+namespace
+{
+using causal_sampler_allocator_t = typename causal_sampler_t::allocator_t;
+using causal_sampler_bundle_t    = typename causal_sampler_t::bundle_type;
+using causal_sampler_buffer_t = tim::data_storage::ring_buffer<causal_sampler_bundle_t>;
+
+struct causal_sampling
+{};
+
+std::set<int>
+configure(bool _setup, int64_t _tid = threading::get_id());
+
+std::shared_ptr<causal_sampler_allocator_t>&
+get_causal_sampler_allocator(bool _construct)
+{
+    static auto _v = std::shared_ptr<causal_sampler_allocator_t>{};
+    if(!_v && _construct) _v = std::make_shared<causal_sampler_allocator_t>();
+    return _v;
+}
+
+auto&
+get_causal_sampler_signals()
+{
+    using thread_data_t = thread_data<identity<std::set<int>>, causal_sampling>;
+    static auto& _v     = thread_data_t::instance(construct_on_init{});
+    return _v;
+}
+
+auto&
+get_causal_sampler_running()
+{
+    using thread_data_t = thread_data<identity<bool>, causal_sampling>;
+    static auto& _v     = thread_data_t::instance(construct_on_init{});
+    return _v;
+}
+
+auto&
+get_causal_samplers()
+{
+    using thread_data_t =
+        thread_data<identity<std::unique_ptr<causal_sampler_t>>, causal_sampling>;
+    static auto& _v = thread_data_t::instance(construct_on_init{});
+    return _v;
+}
+
+std::set<int>&
+get_causal_sampler_signals(int64_t _tid)
+{
+    auto& _data = get_causal_sampler_signals();
+    if(static_cast<size_t>(_tid) >= _data->size())
+        _data->resize(_tid + 1, std::set<int>{});
+    return _data->at(_tid);
+}
+
+bool&
+get_causal_sampler_running(int64_t _tid)
+{
+    auto& _data = get_causal_sampler_running();
+    if(static_cast<size_t>(_tid) >= _data->size()) _data->resize(_tid + 1, false);
+    return _data->at(_tid);
+}
+
+auto&
+get_causal_sampler(int64_t _tid)
+{
+    auto& _data = get_causal_samplers();
+    if(static_cast<size_t>(_tid) >= _data->size()) _data->resize(_tid + 1);
+    return _data->at(_tid);
+}
+
+void
+causal_offload_buffer(int64_t, causal_sampler_buffer_t&& _buf)
+{
+    auto _data      = std::move(_buf);
+    auto _processed = std::map<uint32_t, std::vector<uintptr_t>>{};
+    while(!_data.is_empty())
+    {
+        auto _bundle = causal_sampler_bundle_t{};
+        _data.read(&_bundle);
+        auto* _bt_causal = _bundle.get<causal::component::backtrace>();
+        if(_bt_causal)
+        {
+            for(auto&& itr : _bt_causal->get_stack())
+            {
+                if(itr > 0) _processed[_bt_causal->get_index()].emplace_back(itr);
+            }
+        }
+    }
+    _data.destroy();
+
+    if(!_processed.empty())
+    {
+        tasking::general::get_task_group().exec([_processed]() {
+            static std::mutex _mutex;
+            auto              _lk = std::scoped_lock<std::mutex>{ _mutex };
+            for(const auto& itr : _processed)
+                add_samples(itr.first, itr.second);
+        });
+    }
+}
+
+std::set<int>
+configure(bool _setup, int64_t _tid)
+{
+    const auto& _info         = thread_info::get(_tid, SequentTID);
+    auto&       _causal       = get_causal_sampler(_tid);
+    auto&       _running      = get_causal_sampler_running(_tid);
+    auto&       _signal_types = get_causal_sampler_signals(_tid);
+
+    OMNITRACE_CONDITIONAL_THROW(get_use_sampling(),
+                                "Internal error! configuring causal profiling not "
+                                "permitted when sampling is enabled");
+
+    OMNITRACE_SCOPED_SAMPLING_ON_CHILD_THREADS(false);
+
+    if(_setup && _signal_types.empty()) _signal_types = get_sampling_signals(_tid);
+
+    if(_setup && !_causal && !_running && !_signal_types.empty())
+    {
+        auto _verbose = std::min<int>(get_verbose() - 2, 2);
+        if(get_debug_sampling()) _verbose = 2;
+
+        // if this thread has an offset ID, that means it was created internally
+        // and is probably here bc it called a function which was instrumented.
+        // thus we should not start a sampler for it
+        if(_tid > 0 && _info && _info->is_offset) return std::set<int>{};
+        // if the thread state is disabled or completed, return
+        if(_info && _info->index_data->sequent_value == _tid &&
+           get_thread_state() == ThreadState::Disabled)
+            return std::set<int>{};
+
+        (void) get_debug_sampling();  // make sure query in sampler does not allocate
+        assert(_tid == threading::get_id());
+
+        auto _causal_alloc = get_causal_sampler_allocator(true);
+        _causal = std::make_unique<causal_sampler_t>(_causal_alloc, "omnitrace", _tid,
+                                                     _verbose);
+
+        TIMEMORY_REQUIRE(_causal) << "nullptr to causal profiling instance";
+
+        _causal->set_flags(SA_RESTART);
+        _causal->set_verbose(_verbose);
+        _causal->set_offload(&causal_offload_buffer);
+
+        _causal->configure(timer{ get_realtime_signal(), CLOCK_REALTIME, SIGEV_THREAD_ID,
+                                  1000.0, 1.0e-6, _tid, threading::get_sys_tid() });
+
+        _causal->configure(timer{ get_cputime_signal(), CLOCK_THREAD_CPUTIME_ID,
+                                  SIGEV_THREAD_ID, 1000.0, 1.0e-6, _tid,
+                                  threading::get_sys_tid() });
+
+        _running = true;
+        if(_tid == 0) causal::component::backtrace::start();
+        _causal->start();
+    }
+    else if(!_setup && _causal && _running)
+    {
+        OMNITRACE_DEBUG("Destroying causal sampler for thread %lu...\n", _tid);
+        _running = false;
+
+        if(_tid == threading::get_id() && !_signal_types.empty())
+            block_signals(_signal_types);
+
+        if(_tid == 0)
+        {
+            block_samples();
+
+            // this propagates to all threads
+            _causal->ignore(_signal_types);
+
+            for(int64_t i = 1; i < OMNITRACE_MAX_THREADS; ++i)
+            {
+                if(get_causal_sampler(i)) get_causal_sampler(i)->reset();
+            }
+        }
+
+        _causal->stop();
+
+        OMNITRACE_DEBUG("Causal sampler destroyed for thread %lu\n", _tid);
+    }
+
+    return _signal_types;
+}
+
+void
+post_process_causal(int64_t _tid, const std::vector<causal_bundle_t>& _data);
+}  // namespace
+
+std::set<int>
+get_signal_types(int64_t _tid)
+{
+    return (get_causal_sampler_signals()) ? get_causal_sampler_signals(_tid)
+                                          : std::set<int>{};
+}
+
+std::set<int>
+setup()
+{
+    if(!get_use_causal()) return std::set<int>{};
+    return configure(true);
+}
+
+std::set<int>
+shutdown()
+{
+    auto _v = configure(false);
+    return _v;
+}
+
+void
+block_samples()
+{
+    trait::runtime_enabled<causal_sampler_t>::set(false);
+}
+
+void
+unblock_samples()
+{
+    trait::runtime_enabled<causal_sampler_t>::set(true);
+}
+
+void
+block_signals(std::set<int> _signals)
+{
+    if(_signals.empty()) _signals = get_signal_types(threading::get_id());
+    if(_signals.empty()) return;
+
+    ::omnitrace::sampling::block_signals(_signals);
+}
+
+void
+unblock_signals(std::set<int> _signals)
+{
+    if(_signals.empty()) _signals = get_signal_types(threading::get_id());
+    if(_signals.empty()) return;
+
+    ::omnitrace::sampling::unblock_signals(_signals);
+}
+
+void
+post_process()
+{
+    OMNITRACE_SCOPED_THREAD_STATE(ThreadState::Internal);
+
+    OMNITRACE_VERBOSE(2 || get_debug_sampling(),
+                      "Stopping causal sampling components...\n");
+
+    for(size_t i = 0; i < max_supported_threads; ++i)
+    {
+        auto& _causal = get_causal_sampler(i);
+        if(_causal) _causal->stop();
+    }
+
+    configure(false, 0);
+
+    for(size_t i = 0; i < max_supported_threads; ++i)
+    {
+        auto& _causal = get_causal_sampler(i);
+        auto  _causal_data =
+            (_causal) ? _causal->get_data() : std::vector<sampling::causal_bundle_t>{};
+
+        if(!_causal_data.empty()) post_process_causal(i, _causal_data);
+    }
+
+    for(size_t i = 0; i < max_supported_threads; ++i)
+    {
+        get_causal_sampler(i).reset();
+    }
+
+    if(get_causal_sampler_allocator(false))
+    {
+        get_causal_sampler_allocator(false).reset();
+    }
+}
+
+namespace
+{
+void
+post_process_causal(int64_t, const std::vector<causal_bundle_t>& _data)
+{
+    for(const auto& itr : _data)
+    {
+        const auto* _bt_causal = itr.get<causal::component::backtrace>();
+        for(auto&& ditr : _bt_causal->get_stack())
+        {
+            if(ditr > 0) add_sample(_bt_causal->get_index(), ditr);
+        }
+    }
+}
+}  // namespace
+}  // namespace sampling
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,62 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/concepts.hpp"
+#include "library/defines.hpp"
+
+#include <cstdint>
+#include <memory>
+#include <set>
+#include <type_traits>
+
+namespace omnitrace
+{
+namespace causal
+{
+namespace sampling
+{
+std::set<int>
+get_signal_types(int64_t _tid);
+
+void
+block_samples();
+
+void
+unblock_samples();
+
+void block_signals(std::set<int> = {});
+
+void unblock_signals(std::set<int> = {});
+
+std::set<int>
+setup();
+
+std::set<int>
+shutdown();
+
+void
+post_process();
+}  // namespace sampling
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,52 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include "library/causal/selected_entry.hpp"
+#include "library/common.hpp"
+#include "library/timemory.hpp"
+
+namespace omnitrace
+{
+namespace causal
+{
+template <typename ArchiveT>
+void
+selected_entry::serialize(ArchiveT& ar, const unsigned int)
+{
+    using ::tim::cereal::make_nvp;
+    ar(make_nvp("address", address), make_nvp("symbol_address", symbol_address),
+       make_nvp("info", symbol));
+}
+
+template void
+selected_entry::serialize<cereal::JSONInputArchive>(cereal::JSONInputArchive&,
+                                                    const unsigned int);
+
+template void
+selected_entry::serialize<cereal::MinimalJSONOutputArchive>(
+    cereal::MinimalJSONOutputArchive&, const unsigned int);
+
+template void
+selected_entry::serialize<cereal::PrettyJSONOutputArchive>(
+    cereal::PrettyJSONOutputArchive&, const unsigned int);
+}  // namespace causal
+}  // namespace omnitrace
@@ -0,0 +1,70 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include "library/binary/dwarf_entry.hpp"
+#include "library/binary/fwd.hpp"
+#include "library/binary/symbol.hpp"
+#include "library/causal/fwd.hpp"
+#include "library/debug.hpp"
+#include "library/defines.hpp"
+
+#include <timemory/hash/types.hpp>
+#include <timemory/unwind/dlinfo.hpp>
+#include <timemory/unwind/stack.hpp>
+#include <timemory/utility/macros.hpp>
+#include <timemory/utility/procfs/maps.hpp>
+
+#include <cstddef>
+#include <cstdint>
+#include <dlfcn.h>
+#include <map>
+#include <utility>
+
+namespace omnitrace
+{
+namespace causal
+{
+struct selected_entry
+{
+    TIMEMORY_DEFAULT_OBJECT(selected_entry)
+
+    uintptr_t      address        = 0x0;
+    uintptr_t      symbol_address = 0x0;
+    binary::symbol symbol         = {};
+
+    template <typename ArchiveT>
+    void serialize(ArchiveT&, const unsigned int);
+
+    bool     contains(uintptr_t) const;
+    explicit operator bool() const { return (address > 0 && symbol.address); }
+};
+
+inline bool
+selected_entry::contains(uintptr_t _v) const
+{
+    return (_v == address || (symbol_address > 0 && _v == symbol_address) ||
+            symbol.ipaddr().contains(_v));
+}
+}  // namespace causal
+}  // namespace omnitrace
@@ -113,12 +113,19 @@ namespace threading = ::tim::threading;  // NOLINT
 namespace scope     = ::tim::scope;      // NOLINT
 namespace policy    = ::tim::policy;     // NOLINT
 namespace trait     = ::tim::trait;      // NOLINT
+namespace cereal    = ::tim::cereal;     // NOLINT

 using ::tim::auto_lock_t;   // NOLINT
 using ::tim::demangle;      // NOLINT
 using ::tim::get_env;       // NOLINT
+using ::tim::set_env;       // NOLINT
 using ::tim::try_demangle;  // NOLINT
 using ::tim::type_mutex;    // NOLINT
+
+struct construct_on_thread
+{
+    int64_t index = threading::get_id();
+};
 }  // namespace omnitrace

 // same sort of functionality as python's " ".join([...])
@@ -118,6 +118,7 @@ backtrace::filter_and_patch(const std::vector<entry_type>& _data)
        bool       _keep_internal = get_sampling_keep_internal();
        const auto _npos          = std::string::npos;
        if(_keep_internal) return 1;
+        if(_lbl.find("omnitrace_main") != _npos) return 0;
        if(_lbl.find("omnitrace::common::") != _npos) return 0;
        if(_lbl.find("omnitrace::") != _npos) return 0;
        if(_lbl.find("tim::") != _npos) return 0;
@@ -184,7 +185,7 @@ backtrace::sample(int)
 {
    using namespace tim::backtrace;
    constexpr bool   with_signal_frame = false;
-    constexpr size_t ignore_depth      = 4;
+    constexpr size_t ignore_depth      = 3;
    // ignore depth based on:
    // 1. this frame
    // 2. tim::sampling::sampler<...>::sample(...) [always inline]
@@ -79,6 +79,7 @@ struct backtrace
    bool                    empty() const;
    size_t                  size() const;
    std::vector<entry_type> get() const;
+    data_t                  get_data() const { return m_data; }

 private:
    data_t m_data = {};
@@ -96,8 +96,7 @@ struct perfetto_rusage
 unique_ptr_t<std::vector<std::string>>&
 get_papi_labels(int64_t _tid)
 {
-    static auto& _v =
-        papi_label_instances::instances(papi_label_instances::construct_on_init{});
+    static auto& _v = papi_label_instances::instances(construct_on_init{});
    return _v.at(_tid);
 }

@@ -119,8 +118,7 @@ get_backtrace_metrics_init(int64_t _tid)
 unique_ptr_t<bool>&
 get_sampler_running(int64_t _tid)
 {
-    static auto& _v = sampler_running_instances::instances(
-        sampler_running_instances::construct_on_init{}, false);
+    static auto& _v = sampler_running_instances::instances(construct_on_init{}, false);
    return _v.at(_tid);
 }
 }  // namespace
@@ -22,10 +22,12 @@

 #pragma once

+#include "library/causal/data.hpp"
 #include "library/config.hpp"
 #include "library/critical_trace.hpp"
 #include "library/defines.hpp"
 #include "library/runtime.hpp"
+#include "library/state.hpp"
 #include "library/timemory.hpp"
 #include "library/tracing.hpp"
 #include "library/tracing/annotation.hpp"
@@ -41,6 +43,9 @@ namespace tim
 {
 namespace quirk
 {
+struct causal : concepts::quirk_type
+{};
+
 struct perfetto : concepts::quirk_type
 {};

@@ -53,6 +58,34 @@ namespace omnitrace
 {
 namespace component
 {
+using tim::is_one_of;
+using tim::type_list;
+
+// these categories increment push/pop counts, which are used for sanity checks since
+// they should ALWAYS be popped if they were pushed
+using tracing_count_categories_t =
+    type_list<category::host, category::mpi, category::pthread, category::rocm_hip,
+              category::rocm_hsa, category::rocm_rccl>;
+
+// these categories are added to the critical trace
+using critical_trace_categories_t = type_list<category::host>;
+
+// convert these categories to throughput points
+using causal_throughput_categories_t =
+    type_list<category::host, category::kokkos, category::ompt, category::rocm_hip,
+              category::rocm_hsa, category::rocm_rccl, category::rocm_roctx>;
+
+// define this outside of category region functions so that the
+// static thread_local is global instead of per-template instantiation
+inline ThreadState
+get_thread_status()
+{
+    static thread_local auto _thread_init_once = std::once_flag{};
+    std::call_once(_thread_init_once, tracing::thread_init);
+
+    return get_thread_state();
+}
+
 // timemory component which calls omnitrace functions
 // (used in gotcha wrappers)
 template <typename CategoryT>
@@ -70,6 +103,9 @@ struct category_region : comp::base<category_region<CategoryT>, void>
    template <typename... OptsT, typename... Args>
    static void stop(std::string_view name, Args&&...);

+    template <typename... OptsT, typename... Args>
+    static void mark(std::string_view name, Args&&...);
+
    template <typename... OptsT, typename... Args>
    static void audit(const gotcha_data_t&, audit::incoming, Args&&...);

@@ -96,7 +132,7 @@ category_region<CategoryT>::start(std::string_view name, Args&&... args)

    // unconditionally return if thread is disabled or finalized
    if(get_thread_state() == ThreadState::Disabled) return;
-    if(get_state() == State::Finalized) return;
+    if(get_state() >= State::Finalized) return;

    OMNITRACE_SCOPED_THREAD_STATE(ThreadState::Internal);

@@ -105,31 +141,36 @@ category_region<CategoryT>::start(std::string_view name, Args&&... args)
    // tooling one time and as it exits set it to active and return true.
    if(get_state() != State::Active && !omnitrace_init_tooling_hidden()) return;

-    tracing::thread_init();
-
-    // thread initialization may have disabled the thread
-    if(get_thread_state() == ThreadState::Disabled) return;
-
-    tracing::thread_init_sampling();
+    if(get_thread_status() == ThreadState::Disabled) return;

    constexpr bool _ct_use_timemory =
-        (sizeof...(OptsT) == 0 ||
-         tim::is_one_of<quirk::timemory, tim::type_list<OptsT...>>::value);
+        (sizeof...(OptsT) == 0 || is_one_of<quirk::timemory, type_list<OptsT...>>::value);

    constexpr bool _ct_use_perfetto =
-        (sizeof...(OptsT) == 0 ||
-         tim::is_one_of<quirk::perfetto, tim::type_list<OptsT...>>::value);
+        (sizeof...(OptsT) == 0 || is_one_of<quirk::perfetto, type_list<OptsT...>>::value);

-    OMNITRACE_CONDITIONAL_PRINT(tracing::debug_push,
-                                "[%s][PID=%i][state=%s] omnitrace_push_region(%s)\n",
-                                category_name, process::get_id(),
-                                std::to_string(get_state()).c_str(), name.data());
+    constexpr bool _ct_use_causal =
+        (sizeof...(OptsT) == 0 || is_one_of<quirk::causal, type_list<OptsT...>>::value);

-    if constexpr(tim::is_one_of<CategoryT, tim::type_list<category::host>>::value)
+    OMNITRACE_CONDITIONAL_PRINT(
+        tracing::debug_push,
+        "[%s][PID=%i][state=%s][thread_state=%s] omnitrace_push_region(%s)\n",
+        category_name, process::get_id(), std::to_string(get_state()).c_str(),
+        std::to_string(get_thread_state()).c_str(), name.data());
+
+    if constexpr(is_one_of<CategoryT, tracing_count_categories_t>::value)
    {
        ++tracing::push_count();
    }

+    if constexpr(_ct_use_causal)
+    {
+        if constexpr(!is_one_of<CategoryT, causal_throughput_categories_t>::value)
+        {
+            if(get_use_causal()) causal::push_progress_point(name);
+        }
+    }
+
    if constexpr(_ct_use_perfetto)
    {
        if(get_use_perfetto())
@@ -146,7 +187,7 @@ category_region<CategoryT>::start(std::string_view name, Args&&... args)
        }
    }

-    if constexpr(tim::is_one_of<CategoryT, tim::type_list<category::host>>::value)
+    if constexpr(is_one_of<CategoryT, critical_trace_categories_t>::value)
    {
        using Device = critical_trace::Device;
        using Phase  = critical_trace::Phase;
@@ -178,22 +219,24 @@ category_region<CategoryT>::stop(std::string_view name, Args&&... args)
    OMNITRACE_SCOPED_THREAD_STATE(ThreadState::Internal);

    constexpr bool _ct_use_timemory =
-        (sizeof...(OptsT) == 0 ||
-         tim::is_one_of<quirk::timemory, tim::type_list<OptsT...>>::value);
+        (sizeof...(OptsT) == 0 || is_one_of<quirk::timemory, type_list<OptsT...>>::value);

    constexpr bool _ct_use_perfetto =
-        (sizeof...(OptsT) == 0 ||
-         tim::is_one_of<quirk::perfetto, tim::type_list<OptsT...>>::value);
+        (sizeof...(OptsT) == 0 || is_one_of<quirk::perfetto, type_list<OptsT...>>::value);

-    OMNITRACE_CONDITIONAL_PRINT(tracing::debug_pop,
-                                "[%s][PID=%i][state=%s] omnitrace_pop_region(%s)\n",
-                                category_name, process::get_id(),
-                                std::to_string(get_state()).c_str(), name.data());
+    constexpr bool _ct_use_causal =
+        (sizeof...(OptsT) == 0 || is_one_of<quirk::causal, type_list<OptsT...>>::value);
+
+    OMNITRACE_CONDITIONAL_PRINT(
+        tracing::debug_pop,
+        "[%s][PID=%i][state=%s][thread_state=%s] omnitrace_pop_region(%s)\n",
+        category_name, process::get_id(), std::to_string(get_state()).c_str(),
+        std::to_string(get_thread_state()).c_str(), name.data());

    // only execute when active
    if(get_state() == State::Active)
    {
-        if constexpr(tim::is_one_of<CategoryT, tim::type_list<category::host>>::value)
+        if constexpr(is_one_of<CategoryT, tracing_count_categories_t>::value)
        {
            ++tracing::pop_count();
        }
@@ -216,7 +259,19 @@ category_region<CategoryT>::stop(std::string_view name, Args&&... args)
            }
        }

-        if constexpr(tim::is_one_of<CategoryT, tim::type_list<category::host>>::value)
+        if constexpr(_ct_use_causal)
+        {
+            if constexpr(is_one_of<CategoryT, causal_throughput_categories_t>::value)
+            {
+                if(get_use_causal()) causal::mark_progress_point(name);
+            }
+            else
+            {
+                if(get_use_causal()) causal::pop_progress_point(name);
+            }
+        }
+
+        if constexpr(is_one_of<CategoryT, critical_trace_categories_t>::value)
        {
            using Device = critical_trace::Device;
            using Phase  = critical_trace::Phase;
@@ -249,6 +304,41 @@ category_region<CategoryT>::stop(std::string_view name, Args&&... args)
    }
 }

+template <typename CategoryT>
+template <typename... OptsT, typename... Args>
+void
+category_region<CategoryT>::mark(std::string_view name, Args&&...)
+{
+    constexpr bool _ct_use_causal =
+        (sizeof...(OptsT) == 0 || is_one_of<quirk::causal, type_list<OptsT...>>::value);
+
+    if constexpr(!_ct_use_causal) return;
+
+    // skip if category is disabled
+    if(!trait::runtime_enabled<CategoryT>::get()) return;
+
+    // the expectation here is that if the state is not active then the call
+    // to omnitrace_init_tooling_hidden will activate all the appropriate
+    // tooling one time and as it exits set it to active and return true.
+    if(get_state() != State::Active && !omnitrace_init_tooling_hidden()) return;
+
+    // unconditionally return if thread is disabled or finalized
+    if(get_thread_state() >= ThreadState::Completed) return;
+
+    OMNITRACE_SCOPED_THREAD_STATE(ThreadState::Internal);
+
+    if(get_use_causal())
+    {
+        OMNITRACE_CONDITIONAL_PRINT(
+            tracing::debug_mark,
+            "[%s][PID=%i][state=%s][thread_state=%s] omnitrace_progress(%s)\n",
+            category_name, process::get_id(), std::to_string(get_state()).c_str(),
+            std::to_string(get_thread_state()).c_str(), name.data());
+
+        causal::mark_progress_point(name);
+    }
+}
+
 template <typename CategoryT>
 template <typename... OptsT, typename... Args>
 void
@@ -347,6 +437,13 @@ struct local_category_region : comp::base<local_category_region<CategoryT>, void
        return impl_type::template stop<OptsT...>(m_prefix, std::forward<Args>(args)...);
    }

+    template <typename... OptsT, typename... Args>
+    auto mark(Args&&... args)
+    {
+        if(m_prefix.empty()) return;
+        return impl_type::template mark<OptsT...>(m_prefix, std::forward<Args>(args)...);
+    }
+
    template <typename... OptsT, typename... Args>
    auto audit(Args&&... args)
        -> decltype(impl_type::template audit<OptsT...>(std::declval<std::string_view>(),
@@ -120,9 +120,9 @@ comm_data::audit(const gotcha_data& _data, audit::incoming, const void*, int cou
    auto      _name = std::string_view{ _data.tool_id };
    tracker_t _a{ _name };
    add(_a, count * _size);
-    tracker_t _b{ JOIN('_', _name, "dst", dst) };
+    tracker_t _b{ JOIN('/', _name, JOIN('=', "dst", dst)) };
    add(_b, count * _size);
-    add(JOIN('_', _name, "dst", dst, "tag", tag), count * _size);
+    add(JOIN('/', _name, JOIN('=', "dst", dst), JOIN('=', "tag", tag)), count * _size);
 }

 // MPI_Recv
@@ -139,9 +139,9 @@ comm_data::audit(const gotcha_data& _data, audit::incoming, void*, int count,
    auto      _name = std::string_view{ _data.tool_id };
    tracker_t _a{ _name };
    add(_a, count * _size);
-    tracker_t _b{ JOIN('_', _name, "dst", dst) };
+    tracker_t _b{ JOIN('/', _name, JOIN('=', "dst", dst)) };
    add(_b, count * _size);
-    add(JOIN('_', _name, "dst", dst, "tag", tag), count * _size);
+    add(JOIN('/', _name, JOIN('=', "dst", dst), JOIN('=', "tag", tag)), count * _size);
 }

 // MPI_Isend
@@ -158,9 +158,9 @@ comm_data::audit(const gotcha_data& _data, audit::incoming, const void*, int cou
    auto      _name = std::string_view{ _data.tool_id };
    tracker_t _a{ _name };
    add(_a, count * _size);
-    tracker_t _b{ JOIN('_', _name, "dst", dst) };
+    tracker_t _b{ JOIN('/', _name, JOIN('=', "dst", dst)) };
    add(_b, count * _size);
-    add(JOIN('_', _name, "dst", dst, "tag", tag), count * _size);
+    add(JOIN('/', _name, JOIN('=', "dst", dst), JOIN('=', "tag", tag)), count * _size);
 }

 // MPI_Irecv
@@ -177,9 +177,9 @@ comm_data::audit(const gotcha_data& _data, audit::incoming, void*, int count,
    auto      _name = std::string_view{ _data.tool_id };
    tracker_t _a{ _name };
    add(_a, count * _size);
-    tracker_t _b{ JOIN('_', _name, "dst", dst) };
+    tracker_t _b{ JOIN('/', _name, JOIN('=', "dst", dst)) };
    add(_b, count * _size);
-    add(JOIN('_', _name, "dst", dst, "tag", tag), count * _size);
+    add(JOIN('/', _name, JOIN('=', "dst", dst), JOIN('=', "tag", tag)), count * _size);
 }

 // MPI_Bcast
@@ -196,7 +196,7 @@ comm_data::audit(const gotcha_data& _data, audit::incoming, void*, int count,
    auto      _name = std::string_view{ _data.tool_id };
    tracker_t _t{ _name };
    add(_t, count * _size);
-    add(JOIN('_', _name, "root", root), count * _size);
+    add(JOIN('/', _name, JOIN('=', "root", root)), count * _size);
 }

 // MPI_Allreduce
@@ -232,20 +232,22 @@ comm_data::audit(const gotcha_data& _data, audit::incoming, const void*, int sen
    tracker_t _t{ _name };
    add(_t, sendcount * _send_size + recvcount * _recv_size);
    {
-        tracker_t _b{ JOIN('_', _name, "send") };
+        tracker_t _b{ JOIN('/', _name, "send") };
        add(_b, sendcount * _send_size);
-        tracker_t _c{ JOIN('_', _name, "send", dst) };
+        tracker_t _c{ JOIN('/', _name, JOIN('=', "send", dst)) };
        add(_b, sendcount * _send_size);
-        add(JOIN('_', _name, "send", "tag", sendtag), sendcount * _send_size);
-        add(JOIN('_', _name, "send", dst, "tag", sendtag), sendcount * _send_size);
+        add(JOIN('/', _name, "send", JOIN('=', "tag", sendtag)), sendcount * _send_size);
+        add(JOIN('/', _name, JOIN('=', "send", dst), JOIN('=', "tag", sendtag)),
+            sendcount * _send_size);
    }
    {
-        tracker_t _b{ JOIN('_', _name, "recv") };
+        tracker_t _b{ JOIN('/', _name, "recv") };
        add(_b, recvcount * _recv_size);
-        tracker_t _c{ JOIN('_', _name, "recv", src) };
+        tracker_t _c{ JOIN('/', _name, JOIN('=', "recv", src)) };
        add(_b, recvcount * _recv_size);
-        add(JOIN('_', _name, "recv", "tag", recvtag), recvcount * _recv_size);
-        add(JOIN('_', _name, "recv", src, "tag", recvtag), recvcount * _recv_size);
+        add(JOIN('/', _name, "recv", JOIN('=', "tag", recvtag)), recvcount * _recv_size);
+        add(JOIN('/', _name, JOIN('=', "recv", src), JOIN('=', "tag", recvtag)),
+            recvcount * _recv_size);
    }
 }

@@ -267,10 +269,10 @@ comm_data::audit(const gotcha_data& _data, audit::incoming, const void*, int sen
    auto      _name = std::string_view{ _data.tool_id };
    tracker_t _t{ _name };
    add(_t, sendcount * _send_size + recvcount * _recv_size);
-    tracker_t _r(JOIN('_', _name, "root", root));
+    tracker_t _r(JOIN('/', _name, JOIN('=', "root", root)));
    add(_r, sendcount * _send_size + recvcount * _recv_size);
-    add(JOIN('_', _name, "root", root, "send"), sendcount * _send_size);
-    add(JOIN('_', _name, "root", root, "recv"), recvcount * _recv_size);
+    add(JOIN('/', _name, JOIN('=', "root", root), "send"), sendcount * _send_size);
+    add(JOIN('/', _name, JOIN('=', "root", root), "recv"), recvcount * _recv_size);
 }

 // MPI_Alltoall
@@ -290,8 +292,8 @@ comm_data::audit(const gotcha_data& _data, audit::incoming, const void*, int sen
    auto      _name = std::string_view{ _data.tool_id };
    tracker_t _t{ _name };
    add(_t, sendcount * _send_size + recvcount * _recv_size);
-    add(JOIN('_', _name, "send"), sendcount * _send_size);
-    add(JOIN('_', _name, "recv"), recvcount * _recv_size);
+    add(JOIN('/', _name, "send"), sendcount * _send_size);
+    add(JOIN('/', _name, "recv"), recvcount * _recv_size);
 }
 #endif

@@ -311,7 +313,7 @@ comm_data::audit(const gotcha_data& _data, audit::incoming, const void*, const v
    auto      _name = std::string_view{ _data.tool_id };
    tracker_t _t{ _name };
    add(_t, count * _size);
-    add(JOIN('_', _name, "root", root), count * _size);
+    add(JOIN('/', _name, JOIN('=', "root", root)), count * _size);
 }

 // ncclSend
@@ -350,7 +352,7 @@ comm_data::audit(const gotcha_data& _data, audit::incoming, const void*, size_t

    tracker_t _t{ _name };
    add(_t, count * _size);
-    add(JOIN('_', _name, _label, peer), count * _size);
+    add(JOIN('/', _name, JOIN('=', _label, peer)), count * _size);
 }

 // ncclBroadcast
@@ -367,7 +369,7 @@ comm_data::audit(const gotcha_data& _data, audit::incoming, const void*, const v
    auto      _name = std::string_view{ _data.tool_id };
    tracker_t _t{ _name };
    add(_t, count * _size);
-    add(JOIN('_', _data.tool_id, "root", root), count * _size);
+    add(JOIN('/', _data.tool_id, JOIN('=', "root", root)), count * _size);
 }

 // ncclAllReduce
@@ -29,6 +29,7 @@
 #include "library/timemory.hpp"

 #include <timemory/backends/threading.hpp>
+#include <timemory/process/threading.hpp>
 #include <timemory/utility/types.hpp>

 #include <cstddef>
@@ -45,6 +46,7 @@ exit_gotcha::configure()
        exit_gotcha_t::configure<0, void>("abort");
        exit_gotcha_t::configure<1, void, int>("exit");
        exit_gotcha_t::configure<2, void, int>("quick_exit");
+        exit_gotcha_t::configure<3, void, int>("_Exit");
    };
 }

@@ -56,6 +58,8 @@ template <typename FuncT, typename... Args>
 void
 invoke_exit_gotcha(const exit_gotcha::gotcha_data& _data, FuncT _func, Args... _args)
 {
+    threading::clear_callbacks();
+
    if(config::settings_are_configured())
    {
        OMNITRACE_VERBOSE(0, "%s called %s(%s)...\n", get_exe_name().c_str(),
@@ -38,7 +38,7 @@ namespace component
 {
 struct exit_gotcha : tim::component::base<exit_gotcha, void>
 {
-    static constexpr size_t gotcha_capacity = 3;
+    static constexpr size_t gotcha_capacity = 4;

    using gotcha_data  = tim::component::gotcha_data;
    using exit_func_t  = void (*)(int);
@@ -56,7 +56,7 @@ struct exit_gotcha : tim::component::base<exit_gotcha, void>
    static inline void start() {}
    static inline void stop() {}

-    // exit
+    // exit / _Exit / quick_exit
    void operator()(const gotcha_data&, exit_func_t, int) const;
    // abort
    void operator()(const gotcha_data&, abort_func_t) const;
@@ -21,6 +21,7 @@
 // SOFTWARE.

 #include "library/components/pthread_create_gotcha.hpp"
+#include "library/causal/delay.hpp"
 #include "library/components/category_region.hpp"
 #include "library/components/roctracer.hpp"
 #include "library/config.hpp"
@@ -133,14 +134,10 @@ stop_bundle(bundle_t& _bundle, int64_t _tid, Args&&... _args)
 //--------------------------------------------------------------------------------------//

 pthread_create_gotcha::wrapper::wrapper(routine_t _routine, void* _arg,
-                                        bool _enable_sampling, bool _offset,
-                                        int64_t _parent, promise_t _p)
-: m_enable_sampling{ _enable_sampling }
-, m_offset{ _offset }
-, m_parent_tid{ _parent }
-, m_routine{ _routine }
+                                        wrapper_config _config)
+: m_routine{ _routine }
 , m_arg{ _arg }
-, m_promise{ std::move(_p) }
+, m_config{ std::move(_config) }
 {}

 void*
@@ -150,26 +147,34 @@ pthread_create_gotcha::wrapper::operator()() const

    if(is_shutdown && *is_shutdown)
    {
-        if(m_promise) m_promise->set_value();
+        if(m_config.promise) m_config.promise->set_value();
        // execute the original function
        return m_routine(m_arg);
    }

    push_thread_state(ThreadState::Internal);

-    int64_t _tid         = -1;
-    void*   _ret         = nullptr;
-    auto    _is_sampling = false;
-    auto    _bundle      = std::shared_ptr<bundle_t>{};
-    auto    _signals     = std::set<int>{};
-    auto    _coverage    = (get_mode() == Mode::Coverage);
-    // const auto& _parent_info = thread_info::get(m_parent_tid, InternalTID);
-    auto _dtor = [&]() {
+    int64_t     _tid         = -1;
+    void*       _ret         = nullptr;
+    auto        _is_sampling = false;
+    auto        _bundle      = std::shared_ptr<bundle_t>{};
+    auto        _signals     = std::set<int>{};
+    auto        _coverage    = (get_mode() == Mode::Coverage);
+    const auto& _parent_info = thread_info::get(m_config.parent_tid, InternalTID);
+    auto        _dtor        = [&]() {
        set_thread_state(ThreadState::Internal);
        if(_is_sampling)
        {
-            sampling::block_signals(_signals);
-            sampling::shutdown();
+            if(m_config.enable_causal)
+            {
+                causal::sampling::block_signals(_signals);
+                causal::sampling::shutdown();
+            }
+            else if(m_config.enable_sampling)
+            {
+                sampling::block_signals(_signals);
+                sampling::shutdown();
+            }
        }

        if(_tid >= 0)
@@ -190,8 +195,8 @@ pthread_create_gotcha::wrapper::operator()() const
    auto _active = (get_state() == ::omnitrace::State::Active && bundles != nullptr &&
                    bundles_mutex != nullptr);

-    const auto& _info = thread_info::init(m_offset);
-    if(_active && !_coverage && !m_offset)
+    const auto& _info = thread_info::init(m_config.offset);
+    if(_active && !_coverage && !m_config.offset)
    {
        _tid = _info->index_data->sequent_value;
        threading::set_thread_name(TIMEMORY_JOIN(" ", "Thread", _tid).c_str());
@@ -210,8 +215,19 @@ pthread_create_gotcha::wrapper::operator()() const
                          .first->second;
        }
        if(_bundle) start_bundle(*_bundle);
-        get_cpu_cid_stack(_tid, m_parent_tid);
-        if(m_enable_sampling)
+        get_cpu_cid_stack(_tid, m_config.parent_tid);
+        if(m_config.enable_causal)
+        {
+            // children inherit the parent delay data
+            if(_parent_info && _parent_info->index_data)
+                causal::delay::get_local(_tid) =
+                    causal::delay::get_local(_parent_info->index_data->sequent_value);
+            _is_sampling = true;
+            OMNITRACE_SCOPED_SAMPLING_ON_CHILD_THREADS(false);
+            _signals = causal::sampling::setup();
+            causal::sampling::unblock_signals();
+        }
+        else if(m_config.enable_sampling)
        {
            _is_sampling = true;
            OMNITRACE_SCOPED_SAMPLING_ON_CHILD_THREADS(false);
@@ -221,7 +237,7 @@ pthread_create_gotcha::wrapper::operator()() const
    }

    // notify the wrapper that all internal work is completed
-    if(m_promise) m_promise->set_value();
+    if(m_config.promise) m_config.promise->set_value();

    // Internal -> Enabled
    pop_thread_state();
@@ -347,11 +363,14 @@ pthread_create_gotcha::operator()(pthread_t* thread, const pthread_attr_t* attr,
    OMNITRACE_SCOPED_THREAD_STATE(ThreadState::Internal);

    auto _coverage     = (_mode == Mode::Coverage);
-    auto _use_sampling = get_use_sampling();
+    auto _use_sampling = config::get_use_sampling();
+    auto _use_causal   = config::get_use_causal();
    auto _offset       = (!_enabled || !_active || _info->is_offset);
    auto _use_bundle   = (_active && !_coverage && !_offset);
    auto _enable_sampling =
        (_use_sampling && _sample_child && _active && !_coverage && !_offset);
+    auto _enable_causal =
+        (_use_causal && _sample_child && _active && !_coverage && !_offset);

    static bool debug_threading_get_id =
        get_env<bool>(TIMEMORY_SETTINGS_PREFIX "DEBUG_THREADING_GET_ID", false);
@@ -360,12 +379,13 @@ pthread_create_gotcha::operator()(pthread_t* thread, const pthread_attr_t* attr,
    OMNITRACE_VERBOSE(
        _verbose,
        "Creating new thread :: global_state=%s, thread_state=%s, mode=%s, active=%s, "
-        "coverage=%s, use_sampling=%s, sample_children=%s, tid=%li, use_bundle=%s, "
-        "enable_sampling=%s, thread_info=(%s)...\n",
+        "coverage=%s, use_causal=%s, use_sampling=%s, sample_children=%s, tid=%li, "
+        "use_bundle=%s, enable_causal=%s, enable_sampling=%s, thread_info=(%s)...\n",
        std::to_string(_glob_state).c_str(), std::to_string(_thr_state).c_str(),
        std::to_string(_mode).c_str(), std::to_string(_active).c_str(),
-        std::to_string(_coverage).c_str(), std::to_string(_use_sampling).c_str(),
-        std::to_string(_sample_child).c_str(), _tid, std::to_string(_use_bundle).c_str(),
+        std::to_string(_coverage).c_str(), std::to_string(_use_causal).c_str(),
+        std::to_string(_use_sampling).c_str(), std::to_string(_sample_child).c_str(),
+        _tid, std::to_string(_use_bundle).c_str(), std::to_string(_enable_causal).c_str(),
        std::to_string(_enable_sampling).c_str(), JOIN("", *_info).c_str());

    if(debug_threading_get_id)
@@ -391,9 +411,11 @@ pthread_create_gotcha::operator()(pthread_t* thread, const pthread_attr_t* attr,
    }

    set_thread_state(ThreadState::Disabled);
-    auto  _blocked = get_sampling_signals();
-    auto  _promise = (_active) ? std::make_shared<std::promise<void>>() : promise_t{};
-    auto* _wrap    = new wrapper(func, arg, _enable_sampling, _offset, _tid, _promise);
+    auto _blocked = get_sampling_signals();
+    auto _promise = (_active) ? std::make_shared<std::promise<void>>() : promise_t{};
+    auto _config =
+        wrapper_config{ _enable_causal, _enable_sampling, _offset, _tid, _promise };
+    auto* _wrap = new wrapper{ func, arg, _config };
    set_thread_state(ThreadState::Internal);

    // block the signals in entire process
@@ -409,6 +431,9 @@ pthread_create_gotcha::operator()(pthread_t* thread, const pthread_attr_t* attr,
        start_bundle(*_bundle, audit::incoming{}, thread, attr, func, arg);
    }

+    // threads must process their delays before creating a new thread
+    causal::delay::process();
+
    // create the thread
    auto _ret = (*m_wrappee)(thread, attr, &wrapper::wrap, static_cast<void*>(_wrap));

@@ -42,20 +42,26 @@ struct pthread_create_gotcha : tim::component::base<pthread_create_gotcha, void>
    using wrappee_t = int (*)(pthread_t*, const pthread_attr_t*, routine_t, void*);
    using promise_t = std::shared_ptr<std::promise<void>>;

+    struct wrapper_config
+    {
+        bool      enable_causal   = false;
+        bool      enable_sampling = false;
+        bool      offset          = false;
+        int64_t   parent_tid      = 0;
+        promise_t promise         = {};
+    };
+
    struct wrapper
    {
-        wrapper(routine_t _routine, void* _arg, bool, bool, int64_t, promise_t);
+        wrapper(routine_t _routine, void* _arg, wrapper_config _cfg);
        void* operator()() const;

        static void* wrap(void* _arg);

    private:
-        bool      m_enable_sampling = false;
-        bool      m_offset          = false;
-        int64_t   m_parent_tid      = 0;
-        routine_t m_routine         = nullptr;
-        void*     m_arg             = nullptr;
-        promise_t m_promise         = {};
+        routine_t      m_routine = nullptr;
+        void*          m_arg     = nullptr;
+        wrapper_config m_config  = {};
    };

    TIMEMORY_DEFAULT_OBJECT(pthread_create_gotcha)
@@ -23,7 +23,6 @@
 #include "library/components/pthread_gotcha.hpp"
 #include "library/components/pthread_create_gotcha.hpp"
 #include "library/components/pthread_mutex_gotcha.hpp"
-#include "library/components/roctracer.hpp"
 #include "library/config.hpp"
 #include "library/debug.hpp"
 #include "library/runtime.hpp"
@@ -156,7 +156,7 @@ pthread_mutex_gotcha::configure()
                    "pthread_spin_unlock" });
        }

-        if(config::get_trace_thread_join())
+        if(config::get_trace_thread_join() && !get_use_causal())
        {
            pthread_mutex_gotcha_t::configure(
                comp::gotcha_config<12, int, pthread_t, void**>{ "pthread_join" });
@@ -208,10 +208,9 @@ pthread_mutex_gotcha::operator()(uintptr_t&& _id, int (*_callee)(Args...),
    uint32_t _depth      = 0;
    int64_t  _ts         = 0;

-    OMNITRACE_SCOPED_THREAD_STATE(ThreadState::Internal);
-
    if(_id < std::numeric_limits<uintptr_t>::max() && get_use_critical_trace())
    {
+        OMNITRACE_SCOPED_THREAD_STATE(ThreadState::Internal);
        std::tie(_cid, _parent_cid, _depth) = create_cpu_cid_entry();
        _ts                                 = comp::wall_clock::record();
    }
@@ -222,11 +221,11 @@ pthread_mutex_gotcha::operator()(uintptr_t&& _id, int (*_callee)(Args...),

    if(_id < std::numeric_limits<uintptr_t>::max() && get_use_critical_trace())
    {
+        OMNITRACE_SCOPED_THREAD_STATE(ThreadState::Internal);
        add_critical_trace<Device::CPU, Phase::DELTA>(
            threading::get_id(), _cid, 0, _parent_cid, _ts, comp::wall_clock::record(), 0,
            _id, get_hashes().at(m_data->index), _depth);
    }
-
    tim::consume_parameters(_id, _cid, _parent_cid, _depth, _ts);
    return _ret;
 }
@@ -61,7 +61,7 @@ unique_ptr_t<rocm_data_t>&
 rocm_data(int64_t _tid)
 {
    using thread_data_t = thread_data<rocm_data_t, rocm_event>;
-    static auto& _v     = thread_data_t::instances(thread_data_t::construct_on_init{});
+    static auto& _v     = thread_data_t::instances(construct_on_init{});
    return _v.at(_tid);
 }

@@ -25,20 +25,33 @@
 #include "library/defines.hpp"

 #include <timemory/mpl/concepts.hpp>
+#include <timemory/utility/types.hpp>

 #include <memory>
 #include <optional>
+#include <type_traits>

 namespace omnitrace
 {
 namespace concepts = ::tim::concepts;  // NOLINT

+static constexpr size_t max_supported_threads = OMNITRACE_MAX_THREADS;
+
 template <typename Tp>
 struct thread_deleter;

 // unique ptr type for omnitrace
 template <typename Tp>
 using unique_ptr_t = std::unique_ptr<Tp, thread_deleter<Tp>>;
+
+using construct_on_init = std::true_type;
+
+using tim::identity;    // NOLINT
+using tim::identity_t;  // NOLINT
+
+template <typename Tp>
+struct use_placement_new_when_generating_unique_ptr : std::false_type
+{};
 }  // namespace omnitrace

 namespace tim
@@ -43,6 +43,7 @@
 #include <timemory/settings/types.hpp>
 #include <timemory/utility/argparse.hpp>
 #include <timemory/utility/declaration.hpp>
+#include <timemory/utility/filepath.hpp>
 #include <timemory/utility/join.hpp>
 #include <timemory/utility/signals.hpp>

@@ -314,6 +315,10 @@ configure_settings(bool _init)
                             !_config->get<bool>("OMNITRACE_USE_PERFETTO"), "backend",
                             "timemory");

+    OMNITRACE_CONFIG_SETTING(bool, "OMNITRACE_USE_CAUSAL",
+                             "Enable causal profiling analysis", false, "backend",
+                             "causal", "analysis");
+
    OMNITRACE_CONFIG_SETTING(bool, "OMNITRACE_USE_ROCTRACER",
                             "Enable ROCm API and kernel tracing", true, "backend",
                             "roctracer", "rocm");
@@ -366,9 +371,19 @@ configure_settings(bool _init)
        false, "rocm", "rccl", "backend");

    OMNITRACE_CONFIG_CL_SETTING(
-        bool, "OMNITRACE_KOKKOS_KERNEL_LOGGER", "Enables kernel logging", false,
+        bool, "OMNITRACE_KOKKOSP_KERNEL_LOGGER", "Enables kernel logging", false,
        "--omnitrace-kokkos-kernel-logger", "kokkos", "debugging", "advanced");

+    OMNITRACE_CONFIG_SETTING(int64_t, "OMNITRACE_KOKKOSP_NAME_LENGTH_MAX",
+                             "Set this to a value > 0 to help avoid unnamed Kokkos Tools "
+                             "callbacks. Generally, unnamed callbacks are the demangled "
+                             "name of the function, which is very long",
+                             0, "kokkos", "debugging", "advanced");
+
+    OMNITRACE_CONFIG_SETTING(std::string, "OMNITRACE_KOKKOSP_PREFIX",
+                             "Set to [kokkos] to maintain old naming convention", "",
+                             "kokkos", "debugging", "advanced");
+
    OMNITRACE_CONFIG_SETTING(bool, "OMNITRACE_USE_OMPT",
                             "Enable support for OpenMP-Tools", false, "openmp", "ompt",
                             "backend");
@@ -682,6 +697,101 @@ configure_settings(bool _init)
        std::string, "OMNITRACE_TMPDIR", "Base directory for temporary files",
        get_env<std::string>("TMPDIR", "/tmp"), "io", "data", "advanced");

+    OMNITRACE_CONFIG_SETTING(
+        std::string, "OMNITRACE_CAUSAL_MODE",
+        "Perform causal experiments at the function-scope or line-scope. Ideally, use "
+        "function first to locate function with highest impact and then switch to line "
+        "mode + OMNITRACE_CAUSAL_FUNCTION_SCOPE set to the function being targeted.",
+        std::string{ "function" }, "causal", "analysis", "advanced");
+
+    OMNITRACE_CONFIG_SETTING(
+        double, "OMNITRACE_CAUSAL_DELAY",
+        "Length of time to wait (in seconds) before starting the first causal experiment",
+        0.0, "causal", "analysis");
+
+    OMNITRACE_CONFIG_SETTING(
+        double, "OMNITRACE_CAUSAL_DURATION",
+        "Length of time to perform causal experimentation (in seconds) after the first "
+        "experiment has started. After this amount of time has elapsed, no more causal "
+        "experiments will be performed and the application will continue without any "
+        "overhead from causal profiling. Any value <= 0 means until the application "
+        "completes",
+        0.0, "causal", "analysis");
+
+    OMNITRACE_CONFIG_SETTING(
+        bool, "OMNITRACE_CAUSAL_END_TO_END",
+        "Perform causal experiment over the length of the entire application", false,
+        "causal", "analysis", "advanced");
+
+    OMNITRACE_CONFIG_SETTING(std::string, "OMNITRACE_CAUSAL_FILE",
+                             "Name of causal output filename (w/o extension)",
+                             std::string{ "experiments" }, "causal", "analysis",
+                             "advanced", "io");
+
+    OMNITRACE_CONFIG_SETTING(
+        bool, "OMNITRACE_CAUSAL_FILE_RESET",
+        "Overwrite any existing causal output file instead of appending to it", false,
+        "causal", "analysis", "advanced", "io");
+
+    OMNITRACE_CONFIG_SETTING(
+        uint64_t, "OMNITRACE_CAUSAL_RANDOM_SEED",
+        "Seed for random number generator which selects speedups and experiments -- "
+        "please note that the lines selected for experimentation are not reproducible "
+        "but the speedup selection is. If set to zero, std::random_device{}() will be "
+        "used.",
+        0, "causal", "analysis");
+
+    OMNITRACE_CONFIG_SETTING(std::string, "OMNITRACE_CAUSAL_FIXED_SPEEDUP",
+                             "List of virtual speedups between 0 and 100 (inclusive) to "
+                             "sample from for causal profiling",
+                             std::string{}, "causal", "analysis", "advanced");
+
+    OMNITRACE_CONFIG_SETTING(
+        std::string, "OMNITRACE_CAUSAL_BINARY_SCOPE",
+        "Limits causal experiments to the binaries matching the provided list of regular "
+        "expressions (separated by tab, semi-colon, and/or quotes (single or double))",
+        std::string{ "%MAIN%" }, "causal", "analysis");
+
+    OMNITRACE_CONFIG_SETTING(
+        std::string, "OMNITRACE_CAUSAL_SOURCE_SCOPE",
+        "Limits causal experiments to the source files or source file + lineno pair "
+        "(i.e. <file> or <file>:<line>) matching the provided list of regular "
+        "expressions (separated by tab, semi-colon, and/or quotes (single or double))",
+        std::string{}, "causal", "analysis");
+
+    OMNITRACE_CONFIG_SETTING(
+        std::string, "OMNITRACE_CAUSAL_FUNCTION_SCOPE",
+        "List of <function> regex entries for causal profiling (separated by tab, "
+        "semi-colon, and/or quotes (single or double))",
+        std::string{}, "causal", "analysis");
+
+    OMNITRACE_CONFIG_SETTING(
+        std::string, "OMNITRACE_CAUSAL_BINARY_EXCLUDE",
+        "Excludes binaries matching the list of provided regexes from causal experiments "
+        "(separated by tab, semi-colon, and/or quotes (single or double))",
+        std::string{}, "causal", "analysis");
+
+    OMNITRACE_CONFIG_SETTING(
+        std::string, "OMNITRACE_CAUSAL_SOURCE_EXCLUDE",
+        "Excludes source files or source file + lineno pair (i.e. <file> or "
+        "<file>:<line>) matching the list of provided regexes from causal experiments "
+        "(separated by tab, semi-colon, and/or quotes (single or double))",
+        std::string{}, "causal", "analysis");
+
+    OMNITRACE_CONFIG_SETTING(
+        std::string, "OMNITRACE_CAUSAL_FUNCTION_EXCLUDE",
+        "Excludes functions matching the list of provided regexes from causal "
+        "experiments (separated by tab, semi-colon, and/or quotes (single or double))",
+        std::string{}, "causal", "analysis");
+
+    OMNITRACE_CONFIG_SETTING(
+        bool, "OMNITRACE_CAUSAL_FUNCTION_EXCLUDE_DEFAULTS",
+        "This controls adding a series of function exclude regexes to avoid "
+        "experimenting on STL implementation functions, etc. which are, "
+        "generally, not helpful. Details: excludes demangled function names "
+        "starting with '_' or containing '::_M'.",
+        true, "causal", "analysis", "advanced");
+
    // set the defaults
    _config->get_flamegraph_output()     = false;
    _config->get_ctest_notes()           = false;
@@ -819,8 +929,9 @@ configure_settings(bool _init)
    auto _cmd     = tim::read_command_line(process::get_id());
    auto _cmd_env = tim::get_env<std::string>("OMNITRACE_COMMAND_LINE", "");
    if(!_cmd_env.empty()) _cmd = tim::delimit(_cmd_env, " ");
-    auto _exe = (_cmd.empty()) ? "exe" : _cmd.front();
-    auto _pos = _exe.find_last_of('/');
+    auto _exe          = (_cmd.empty()) ? "exe" : _cmd.front();
+    get_exe_realpath() = filepath::realpath(_exe, nullptr, false);
+    auto _pos          = _exe.find_last_of('/');
    if(_pos < _exe.length() - 1) _exe = _exe.substr(_pos + 1);
    get_exe_name() = _exe;
    _config->set_tag(_exe);
@@ -948,12 +1059,15 @@ configure_mode_settings()
        }
    };

+    auto _use_causal = get_setting_value<bool>("OMNITRACE_USE_CAUSAL");
+    if(_use_causal.first && _use_causal.second) set_env("OMNITRACE_MODE", "causal", 1);
+
    if(get_mode() == Mode::Coverage)
    {
        set_default_setting_value("OMNITRACE_USE_CODE_COVERAGE", true);
        _set("OMNITRACE_USE_PERFETTO", false);
        _set("OMNITRACE_USE_TIMEMORY", false);
-        //_set("OMNITRACE_USE_CAUSAL", false);
+        _set("OMNITRACE_USE_CAUSAL", false);
        _set("OMNITRACE_USE_ROCM_SMI", false);
        _set("OMNITRACE_USE_ROCTRACER", false);
        _set("OMNITRACE_USE_ROCPROFILER", false);
@@ -964,6 +1078,15 @@ configure_mode_settings()
        _set("OMNITRACE_USE_PROCESS_SAMPLING", false);
        _set("OMNITRACE_CRITICAL_TRACE", false);
    }
+    else if(get_mode() == Mode::Causal)
+    {
+        _set("OMNITRACE_USE_CAUSAL", true);
+        _set("OMNITRACE_USE_PERFETTO", false);
+        _set("OMNITRACE_USE_TIMEMORY", false);
+        _set("OMNITRACE_CRITICAL_TRACE", false);
+        _set("OMNITRACE_USE_SAMPLING", false);
+        _set("OMNITRACE_USE_PROCESS_SAMPLING", false);
+    }
    else if(get_mode() == Mode::Sampling)
    {
        set_default_setting_value("OMNITRACE_USE_SAMPLING", true);
@@ -973,8 +1096,10 @@ configure_mode_settings()

    if(gpu::device_count() == 0)
    {
+#if OMNITRACE_HIP_VERSION > 0
        OMNITRACE_BASIC_VERBOSE(1, "No HIP devices were found: disabling roctracer, "
                                   "rocprofiler, and rocm_smi...\n");
+#endif
        _set("OMNITRACE_USE_ROCPROFILER", false);
        _set("OMNITRACE_USE_ROCTRACER", false);
        _set("OMNITRACE_USE_ROCM_SMI", false);
@@ -1010,7 +1135,7 @@ configure_mode_settings()
    {
        _set("OMNITRACE_USE_PERFETTO", false);
        _set("OMNITRACE_USE_TIMEMORY", false);
-        //_set("OMNITRACE_USE_CAUSAL", false);
+        _set("OMNITRACE_USE_CAUSAL", false);
        _set("OMNITRACE_USE_ROCM_SMI", false);
        _set("OMNITRACE_USE_ROCTRACER", false);
        _set("OMNITRACE_USE_ROCPROFILER", false);
@@ -1117,6 +1242,7 @@ configure_disabled_settings()

    _handle_use_option("OMNITRACE_USE_SAMPLING", "sampling");
    _handle_use_option("OMNITRACE_USE_PROCESS_SAMPLING", "process_sampling");
+    _handle_use_option("OMNITRACE_USE_CAUSAL", "causal");
    _handle_use_option("OMNITRACE_USE_KOKKOSP", "kokkos");
    _handle_use_option("OMNITRACE_USE_PERFETTO", "perfetto");
    _handle_use_option("OMNITRACE_USE_TIMEMORY", "timemory");
@@ -1436,6 +1562,18 @@ get_exe_name()
    return _v;
 }

+std::string&
+get_exe_realpath()
+{
+    static std::string _v = []() {
+        auto _cmd_line = tim::read_command_line(process::get_id());
+        if(!_cmd_line.empty())
+            return filepath::realpath(_cmd_line.front(), nullptr, false);
+        return std::string{};
+    }();
+    return _v;
+}
+
 std::string
 get_config_file()
 {
@@ -1449,15 +1587,18 @@ get_mode()
    if(!settings_are_configured())
    {
        auto _mode = tim::get_env_choice<std::string>(
-            "OMNITRACE_MODE", "trace", { "trace", "sampling", "coverage" });
+            "OMNITRACE_MODE", "trace", { "trace", "sampling", "causal", "coverage" });
        if(_mode == "sampling")
            return Mode::Sampling;
+        else if(_mode == "causal")
+            return Mode::Causal;
        else if(_mode == "coverage")
            return Mode::Coverage;
        return Mode::Trace;
    }
    static auto _m =
        std::unordered_map<std::string_view, Mode>{ { "trace", Mode::Trace },
+                                                    { "causal", Mode::Causal },
                                                    { "sampling", Mode::Sampling },
                                                    { "coverage", Mode::Coverage } };
    static auto _v = get_config()->find("OMNITRACE_MODE");
@@ -1564,6 +1705,13 @@ get_use_timemory()
    return static_cast<tim::tsettings<bool>&>(*_v->second).get();
 }

+bool&
+get_use_causal()
+{
+    static auto _v = get_config()->find("OMNITRACE_USE_CAUSAL");
+    return static_cast<tim::tsettings<bool>&>(*_v->second).get();
+}
+
 bool
 get_use_roctracer()
 {
@@ -1671,7 +1819,7 @@ get_use_kokkosp()
 bool
 get_use_kokkosp_kernel_logger()
 {
-    static auto _v = get_config()->find("OMNITRACE_KOKKOS_KERNEL_LOGGER");
+    static auto _v = get_config()->find("OMNITRACE_KOKKOSP_KERNEL_LOGGER");
    return static_cast<tim::tsettings<bool>&>(*_v->second).get();
 }

@@ -2214,5 +2362,152 @@ get_tmp_file(std::string _basename, std::string _ext)
    _existing_files.emplace(_fname, std::move(_v));
    return _existing_files.at(_fname);
 }
+
+CausalMode
+get_causal_mode()
+{
+    if(!settings_are_configured())
+    {
+        auto _mode = tim::get_env_choice<std::string>("OMNITRACE_CAUSAL_MODE", "function",
+                                                      { "line", "function" });
+        if(_mode == "line") return CausalMode::Line;
+        return CausalMode::Function;
+    }
+    static auto _causal_mode = []() {
+        auto _m = std::unordered_map<std::string_view, CausalMode>{
+            { "line", CausalMode::Line },
+            { "func", CausalMode::Function },
+            { "function", CausalMode::Function }
+        };
+        auto _v = get_config()->find("OMNITRACE_CAUSAL_MODE");
+        try
+        {
+            return _m.at(static_cast<tim::tsettings<std::string>&>(*_v->second).get());
+        } catch(std::runtime_error& _e)
+        {
+            auto _mode = static_cast<tim::tsettings<std::string>&>(*_v->second).get();
+            std::stringstream _ss{};
+            for(const auto& itr : _v->second->get_choices())
+                _ss << ", " << itr;
+            auto _msg = (_ss.str().length() > 2) ? _ss.str().substr(2) : std::string{};
+            OMNITRACE_THROW("[%s] invalid causal mode %s. Choices: %s\n", __FUNCTION__,
+                            _mode.c_str(), _msg.c_str());
+        }
+        return CausalMode::Function;
+    }();
+    return _causal_mode;
+}
+
+bool
+get_causal_end_to_end()
+{
+    static auto _v = get_config()->find("OMNITRACE_CAUSAL_END_TO_END");
+    return static_cast<tim::tsettings<bool>&>(*_v->second).get();
+}
+
+std::vector<int64_t>
+get_causal_fixed_speedup()
+{
+    static auto _v = get_config()->find("OMNITRACE_CAUSAL_FIXED_SPEEDUP");
+    return parse_numeric_range<int64_t, std::vector<int64_t>>(
+        static_cast<tim::tsettings<std::string>&>(*_v->second).get(),
+        "causal fixed speedup", 5);
+}
+
+std::string
+get_causal_output_filename()
+{
+    static auto _v     = get_config()->find("OMNITRACE_CAUSAL_FILE");
+    auto        _fname = static_cast<tim::tsettings<std::string>&>(*_v->second).get();
+    for(auto&& itr : std::initializer_list<std::string>{ ".txt", ".json", ".xml" })
+    {
+        auto _pos = _fname.find(itr);
+        // if extension is found at end of string, remove
+        if(_pos != std::string::npos && (_pos + itr.length()) == _fname.length())
+            _fname = _fname.substr(0, _fname.length() - itr.length());
+    }
+    return _fname;
+}
+
+namespace
+{
+std::vector<std::string>
+format_causal_scopes(std::vector<std::string> _value, const std::string& _tag)
+{
+    const auto _config   = get_config();
+    const auto _main_re  = std::regex{ "(^|[^a-zA-Z])(MAIN|%MAIN%)($|[^a-zA-Z])" };
+    const auto _space_re = std::regex{ "^([ ]*)(.*)([ ]*)$" };
+    for(auto& itr : _value)
+    {
+        // replace any output/input keys, e.g. %argv0%
+        itr = settings::format(itr, _tag);
+        // replace MAIN or %MAIN% with (<exe_basename>|<exe_realpath>)
+        if(std::regex_search(itr, _main_re))
+        {
+            itr = std::regex_replace(
+                itr, _main_re,
+                join("", "$1", "(", get_exe_name(), "|", get_exe_realpath(), ")", "$3"));
+        }
+        // trim leading and trailing spaces since we didn't delimit spaces
+        if(std::regex_search(itr, _space_re))
+            itr = std::regex_replace(itr, _space_re, "$2");
+    }
+    return _value;
+}
+}  // namespace
+
+std::vector<std::string>
+get_causal_binary_scope()
+{
+    auto&&      _config = get_config();
+    static auto _v      = _config->find("OMNITRACE_CAUSAL_BINARY_SCOPE");
+    return format_causal_scopes(
+        tim::delimit(static_cast<tim::tsettings<std::string>&>(*_v->second).get(),
+                     "\t\"';"),
+        _config->get_tag());
+}
+
+std::vector<std::string>
+get_causal_source_scope()
+{
+    static auto _v = get_config()->find("OMNITRACE_CAUSAL_SOURCE_SCOPE");
+    return tim::delimit(static_cast<tim::tsettings<std::string>&>(*_v->second).get(),
+                        "\t\"';");
+}
+
+std::vector<std::string>
+get_causal_function_scope()
+{
+    static auto _v = get_config()->find("OMNITRACE_CAUSAL_FUNCTION_SCOPE");
+    return tim::delimit(static_cast<tim::tsettings<std::string>&>(*_v->second).get(),
+                        "\t\"';");
+}
+
+std::vector<std::string>
+get_causal_binary_exclude()
+{
+    auto&&      _config = get_config();
+    static auto _v      = _config->find("OMNITRACE_CAUSAL_BINARY_EXCLUDE");
+    return format_causal_scopes(
+        tim::delimit(static_cast<tim::tsettings<std::string>&>(*_v->second).get(),
+                     "\t\"';"),
+        _config->get_tag());
+}
+
+std::vector<std::string>
+get_causal_source_exclude()
+{
+    static auto _v = get_config()->find("OMNITRACE_CAUSAL_SOURCE_EXCLUDE");
+    return tim::delimit(static_cast<tim::tsettings<std::string>&>(*_v->second).get(),
+                        "\t\"';");
+}
+
+std::vector<std::string>
+get_causal_function_exclude()
+{
+    static auto _v = get_config()->find("OMNITRACE_CAUSAL_FUNCTION_EXCLUDE");
+    return tim::delimit(static_cast<tim::tsettings<std::string>&>(*_v->second).get(),
+                        "\t\"';");
+}
 }  // namespace config
 }  // namespace omnitrace
@@ -79,6 +79,9 @@ print_settings(bool include_env = true);
 std::string&
 get_exe_name();

+std::string&
+get_exe_realpath();
+
 template <typename Tp>
 bool
 set_setting_value(const std::string& _name, Tp&& _v)
@@ -166,6 +169,9 @@ get_use_perfetto() OMNITRACE_HOT;
 bool&
 get_use_timemory() OMNITRACE_HOT;

+bool&
+get_use_causal() OMNITRACE_HOT;
+
 bool
 get_use_roctracer() OMNITRACE_HOT;

@@ -365,7 +371,7 @@ struct tmp_file
    void close();
    void remove();

-    operator bool() const { return stream.is_open() && stream.good(); }
+    explicit operator bool() const { return stream.is_open() && stream.good(); }

    std::string  filename = {};
    std::fstream stream   = {};
@@ -373,5 +379,35 @@ struct tmp_file

 std::shared_ptr<tmp_file>
 get_tmp_file(std::string _basename, std::string _ext = "dat");
+
+CausalMode
+get_causal_mode();
+
+bool
+get_causal_end_to_end();
+
+std::vector<int64_t>
+get_causal_fixed_speedup();
+
+std::string
+get_causal_output_filename();
+
+std::vector<std::string>
+get_causal_binary_scope();
+
+std::vector<std::string>
+get_causal_source_scope();
+
+std::vector<std::string>
+get_causal_function_scope();
+
+std::vector<std::string>
+get_causal_binary_exclude();
+
+std::vector<std::string>
+get_causal_source_exclude();
+
+std::vector<std::string>
+get_causal_function_exclude();
 }  // namespace config
 }  // namespace omnitrace
--- a/Εμφάνιση Περισσότερων
+++ b/Εμφάνιση Περισσότερων