Causal profiling (#229)

* Addition of basic structure * Reworked categories * More causal integration additions * Causal implementation * Update examples * delete virtual_speedup files * Update perfetto submodule to v31.0 * Update dyninst submodule * Update timemory submodule * ElfUtils build for libdw * OMNITRACE_LIKELY and OMNITRACE_UNLIKELY * Update common lib join * Examples updates for causal profiling * config updates with causal options - OMNITRACE_CAUSAL_FIXED_LINE - OMNITRACE_CAUSAL_FIXED_SPEEDUP - OMNITRACE_CAUSAL_FILE - OMNITRACE_CAUSAL_BINARY_SCOPE - OMNITRACE_CAUSAL_SOURCE_SCOPE - version info in banner - support increments in parse_numeric_range - fix occasional deadlock in first call to get_config * PTL general task group * Always include PID in debug/verbose messages * Add blocking/unblocking gotchas to runtime init bundle * CausalState * thread_data updates - generic component_bundle_cache * Improve handling of causal in category_region * components updates - backtrace_causal component - backtrace::get_data member func - decrease ignore_depth in backtrace::sample(int) - handle "omnitrace_main" in backtrace::filter_and_patch(...) - tweak internal thread state scope for pthread_mutex_gotcha wrappers * simplify tracing get_instrumentation_bundles usage * sampling updates - include backtrace_causal component - disable backtrace_metrics if using causal and not using perfetto - disable backtrace and backtrace_timestamp when using causal - post_process_causal * causal updates - more checks in blocking_gotcha and unblocking_gotcha start/stop - miscellaneous overhaul of data - experiment update * Remove virtual speedup * libomnitrace code_object * causal-profiling test * libomnitrace library.cpp updates - handle causal profiling - fini_bundle * Disable causal profiling by default * Updated causal code and example - example: three execution variants: cpu + rng, cpu, rng - example: three instrumentation variants: none, omni, coz - fix blocking gotcha credit - rework perform_experiment_impl - get_eligible_address_ranges - compute_eligible_lines - support fixed lines/speedups/functions - update selected_entry to support function mode - fix causal::delay - experiment updates * omnitrace_progress / omnitrace_user_progress - with accompanying omnitrace_annotated_progress / omnitrace_user_annotated_progress * Update timemory submodule * CausalMode - mode indicated whether causal predictions source be at line-level or function-level * code_object, config, runtime, sampling, thread_data - code_object: address_range - code_object: basic::line_info serialize(), name(), hash() - config updates - two signals for causal sampling - thread_data init fixes * pthread updates - pthread_create_gotcha processes delays - pthread_mutex_gotcha does not wrap pthread_join in causal mode * backtrace_causal update - dynamic delay period stats * main wrapper uses basename of argv[0] * update elfio submodule * perf support (currently unused) * Fix experiment JSON serialization - static_vector.hpp (unused) * causal executable + config options updates - omnitrace-causal exe simplifies running multiple causal configs - changed the causal config option names * Support both throughput and latency points * process-causal-json.py script - will be used later for testing * stable_vector * Rework thread_data * Improve omnitrace-causal exe - better verbosity handling - correct diagnosis of status for child process - execvpe when only one iteration (debugging) * Update timemory submodule * exe --version - omnitrace, omnitrace-avail, and omnitrace-sample all support --version on command-line * OMNITRACE_INTERNAL_API + OMNITRACE_{LIKELY,UNLIKELY} * omnitrace-causal cmake format * omnitrace config update - OMNITRACE_CAUSAL_FILE_CLOBBER * custom exception - wraps STL exception and gets stacktrace during construction * exit_gotcha supports _Exit * use global construct_on_init + max threads - add some safety when exceeding max # of threads * update code_object binary filter - exclude dyninst and tbbmalloc library * containers: c_array, static_vector, stable_vector - moved utility::c_array to container::c_array - created static_vector: std::vector bound to std::array - created stable_vector: vector with stable references * grow thread_data when new thread created * causal updates - data: improve compute_eligible_lines to ignore lambdas - data: use new thread_data - delay: use new thread_data - experiment: properly support latency points - experiment: support file clobber - experiment: ensure non-zero experiment time - progress_point: use new thread_data - backtrace_causal: use new thread_data * Update causal-profiling tests * fix omnitrace-causal backslash escaping * process-causal-json script * restructure causal implementation - update verbose messages for omnitrace-causal diagnose_status - migrated causal implementation in sampling.cpp to causal/sampling.cpp - OMNITRACE_USE_CAUSAL does not require OMNITRACE_USE_SAMPLING - added Mode::Causal - causal sampling uses same signals as regular sampling - moved tracing::thread_init to implementation file - combined tracing::thread_init and tracing::thread_init_sampling - added causal/components folder - pthread_create_gotcha::wrapper_config - omnitrace_preload checks OMNITRACE_USE_CAUSAL - updates mode accordingly * update timemory submodule * update timemory submodule * causal example updates - causal for lulesh * perf code + utility - helpers - relocated causal perf code - placement new when generating unique ptr trait for potentially allocating during sampling - additions to utility header - removed previously added helpers.hpp * update timemory submodule * Default env variables for omnitrace-causal - activate OMNITRACE_USE_KOKKOSP, etc. * update stable_vector and static_vector - static vector can use atomic for size tracking for thread-safe situations * update causal example header - CAUSAL_PROGRESS_NAMED - use CAUSAL_ prefix for some macros * Tweak lulesh example - use CAUSAL_PROGRESS instead of CAUSAL_BEGIN and CAUSAL_END * omnitrace-sample support for causal mode - set OMNITRACE_USE_SAMPLING to off when OMNITRACE_MODE=causal * refactor and cleanup code_object - scope filter - fixes to address_range * overhaul causal data + causal config options - full support for function and line mode - support static vector of instruction pointers - improve line info mapping resolution - remove thread-locality from miscellanous functions where unnecessary - causal options for {binary,source,function,fileline} exclusion * causal experiment, sampling, and backtrace updates - is_selected + unwind address array - experiment warning about progress points - increased buffer size for backtrace_casual sampler - backtrace_causal only stores IP addresses instead of full unwind info * category_region updates - minor refactor - local_category_region::mark * Update causal tests * Bump version to 1.8.0 * omnitrace-causal args + CLOBBER -> RESET - renamed OMNITRACE_CAUSAL_FILE_CLOBBER to OMNITRACE_CAUSAL_FILE_RESET - updated omnitrace-causal exe to support recently added configuration options - other miscellaneous tweaks to data.cpp, experiment.cpp, and sampling.cpp * Refactor causal and code_object - code_object.hpp and code_object.cpp moved into binary folder - causal components namespaced into omnitrace::causal::component - moved sample_data out of backtrace_causal and into own file - renamed backtrace_causal to causal::component::backtrace * preload omnitrace_init + OMNITRACE_DEBUG_MARK - env OMNITRACE_DEBUG_MARK - fix omnitrace_init call when LD_PRELOAD-ing omnitrace * Fix fileline support + line-info output names + experiment log - line-info log files are prefixed with experiment name - don't print experiment duration when E2E - account for fileline scope in analysis * KokkosP: OMNITRACE_KOKKOSP_NAME_LENGTH_MAX - config option to limit the name of kokkos tool callbacks - remove [kokkos] from KokkosP names * Update causal example - minor tweaks to decrease probability of overlapping regions in binary * omnitrace-causal update - prefix N / Ntot in environment printout * Miscellaneous updates - causal::finish_experimenting() - OMNITRACE_CAUSAL_RANDOM_SEED - KokkosP causal updates - exclude some callbacks, make some callbacks unique, etc. - address_range::operator+=(address_range) - combine contiguous ranges in binary/analysis.cpp when file, func, line is same and address range is contiguous - bfd_line_info reads inline info - wait for perform_experiment_impl to complete - causal::delay updates - delay::process checks if experiment is active - uses threading::get_id() - experiment scales duration up for larger speedup experiments - line info samples includes excluded lines - sampler uses CLOCK_REALTIME - blocking_gotcha updates - is no longer fully static - adds audit routine which sets the postblock value to zero if try/timed routine fails - category::host was added to causal_throughput_categories_t - pthread_create_gotcha sets new threads local parent delay - was using internal value, now uses sequent value * Causal improvements to KokkosP * Updates to experiment time scaling - use stats instead of just max * binary/link_map.{hpp,cpp} * update process-causal-json.py * Folded fileline scope into source scope * Update documentation - Add documentation for causal profiling - Replace 'Omnitrace' with 'OmniTrace' everywhere * Update causal-helpers.cmake + omnitrace-testing.cmake - split tests/CMakeLists.txt partially into omnitrace-testing.cmake * omnitrace/causal.h - OMNITRACE_CAUSAL_PROGRESS - OMNITRACE_CAUSAL_PROGRESS_NAMED - OMNITRACE_CAUSAL_BEGIN - OMNITRACE_CAUSAL_END * selected_entry + remove default filters for lambdas and operator() - selected entry stores range and binary load address * update process-causal-json.py * format examples/lulesh/CMakeLists.txt * causal-helpers find_package(Threads) * OMNITRACE_KOKKOSP_KERNEL_LOGGER - was OMNITRACE_KOKKOS_KERNEL_LOGGER * quiet find of coz-profiler * Fix rocm_smi exception handling * Update timemory submodule (binutils) - fix binutls compile error on some systems - bump binutils to v2.40 * Fix miscellaneous tests * OMNITRACE_KOKKOSP_PREFIX * revert rocm_smi handling * ElfUtils updates - default to download version 0.188 - add -Wno-error=null-dereference due to GCC 12 compiler error * Update causal example * Remove OMNITRACE_VERBOSE from global workflow envs * Reliable causal test * disable compilation of causal perf files * Remove set_current_selection with unwind stack * update timemory submodule * fix for segfault on bionic - locking in TLS dtor was causing segfault * remove experiment::is_selected(unwind_stack_t) * update default init of selected_entry * Fix for when IP is not offset by load address * Update CMakeLists.txt * Miscellaneous updates - OMNITRACE_WARNING_OR_CI_THROW - OMNITRACE_REQUIRE - OMNITRACE_PREFER - fixed issues with no ASLR - added load address variable and ipaddr() func to basic/bfd line info - removed get_basic() from dwarf_line_info - TIMEMORY_PREFER -> OMNITRACE_PREFER - removed previously added binary_address and range variables from selected_entry * Removed superfluous CausalState * Additional causal tests (lulesh + kokkos) * filter, prefer, analysis ASLR handling - removed default filter on cold functions - fixed OMNITRACE_PREFER - fixed analysis ASLR handling * Tweak line-info output * Removed some superfluous code - causal/delay - causal/selected_entry * Exclude main.cold in function mode * Update validate-perfetto-proto.py - account for occasional http errors * Add sampling test disabling tmp files * argparser for process-causal-json - support validation - support filtering * Avoid pthread_{lock,unlock} in sampling offload - use homemade atomic_mutex/atomic_lock since contention will be low and using pthread tools might trigger our wrappers * Rename process-causal-json.py - validate-causal-json.py * rework omnitrace_add_causal_test - capable of performing validation - added validation tests * Fix kokkosp_begin_deep_copy + causal * Tweak address range in bfd_line_info::read_pc * Tweak analysis and data IP handling - look for gaps * Disable scaling experiment time by speedup * Revert change in max threads during CI * binary updates - significant overhaul of binary analysis implementation - removed "basic_line_info" and "bfd_line_info" in lieu of "symbol" class - symbol class has basic BFD info + vector of inlines + vector of dwarf info * Updated causal to use new binary analysis - Fix symbol.cpp includes * Updated formatting target - include *.cmake files * Updated causal tests - causal tests should be stable now * Update timemory and dyninst submodules - TPLs are stripped + built w/o debug info * Increase tolerance for causal validation speedups - higher speedups have more variance (increased to +/- 5 from 3) * Support causal output for MPI - i.e. tag with MPI rank * omnitrace-causal launcher argument * improve experiment sampling output * causal data updates - call compute lines once - fixed filtered cached binary info - debugging info when experiment fails to start * Tweaked causal validation tests * dwarf_entry ranges * CI updates - increase max threads to 64 * Tweak causal E2E validation tests - more threads - shorter thread runtime - more iterations * Fix shadowed variable * fix symbol read_bfd last PC calculation * fix maybe-uninitialized warning * omnitrace-causal launcher update - only inject "omnitrace-causal --" once - throw error if no matches found * Update causal profiling docs for launcher * fix address range boundaries [ROCm/rocprofiler-systems commit: 9618ddefba]
2023-01-24 18:53:23 -06:00
@@ -18,7 +18,7 @@ set(CMAKE_CXX_CLANG_TIDY)
 set(CMAKE_INSTALL_DEFAULT_COMPONENT_NAME examples)

 if(OMNITRACE_BUILD_DEBUG)
-    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g  -fno-omit-frame-pointer")
+    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g3 -fno-omit-frame-pointer")
 endif()

 option(BUILD_SHARED_LIBS "Build dynamic libraries" ON)
@@ -36,6 +36,12 @@ if(OMNITRACE_INSTALL_EXAMPLES)
    include(GNUInstallDirs)
 endif()

+set(OMNITRACE_EXAMPLE_ROOT_DIR
+    ${CMAKE_CURRENT_LIST_DIR}
+    CACHE INTERNAL "")
+# defines function for creating causal profiling exes
+include(${CMAKE_CURRENT_LIST_DIR}/causal-helpers.cmake)
+
 add_subdirectory(transpose)
 add_subdirectory(parallel-overhead)
 add_subdirectory(code-coverage)
@@ -46,3 +52,4 @@ add_subdirectory(python)
 add_subdirectory(lulesh)
 add_subdirectory(rccl)
 add_subdirectory(rewrite-caller)
+add_subdirectory(causal)
@@ -0,0 +1,104 @@
+#
+#   function for
+#
+include_guard(DIRECTORY)
+
+if(NOT TARGET omnitrace::omnitrace-user-library)
+    find_package(omnitrace REQUIRED COMPONENTS user)
+endif()
+
+if(NOT coz-profiler_FOUND)
+    find_package(coz-profiler QUIET)
+endif()
+
+if(NOT TARGET omni-causal-examples)
+    add_custom_target(omni-causal-examples)
+endif()
+
+function(omnitrace_causal_example_executable _NAME)
+    cmake_parse_arguments(
+        CAUSAL "" "" "SOURCES;DEFINITIONS;INCLUDE_DIRECTORIES;LINK_LIBRARIES" ${ARGN})
+
+    function(omnitrace_causal_example_interface _TARGET)
+        if(NOT TARGET ${_TARGET})
+            find_package(Threads REQUIRED)
+            add_library(${_TARGET} INTERFACE)
+            target_link_libraries(${_TARGET} INTERFACE Threads::Threads ${CMAKE_DL_LIBS})
+        endif()
+    endfunction()
+
+    omnitrace_causal_example_interface(omni-causal-example-lib-debug)
+    omnitrace_causal_example_interface(omni-causal-example-lib-no-debug)
+
+    target_compile_options(omni-causal-example-lib-debug
+                           INTERFACE -g3 -fno-omit-frame-pointer)
+    target_compile_options(omni-causal-example-lib-no-debug INTERFACE -g0)
+
+    add_executable(${_NAME} ${CAUSAL_SOURCES})
+    target_compile_definitions(${_NAME} PRIVATE USE_COZ=0 USE_OMNI=0
+                                                ${CAUSAL_DEFINITIONS})
+    target_include_directories(${_NAME} PRIVATE ${OMNITRACE_EXAMPLE_ROOT_DIR}/causal
+                                                ${CAUSAL_INCLUDE_DIRECTORIES})
+    target_link_libraries(
+        ${_NAME} PRIVATE ${CAUSAL_LINK_LIBRARIES} omnitrace::omnitrace-user-library
+                         omni-causal-example-lib-debug)
+
+    add_executable(${_NAME}-omni ${CAUSAL_SOURCES})
+    target_compile_definitions(${_NAME}-omni PRIVATE USE_COZ=0 USE_OMNI=1
+                                                     ${CAUSAL_DEFINITIONS})
+    target_include_directories(${_NAME}-omni PRIVATE ${OMNITRACE_EXAMPLE_ROOT_DIR}/causal
+                                                     ${CAUSAL_INCLUDE_DIRECTORIES})
+    target_link_libraries(
+        ${_NAME}-omni PRIVATE ${CAUSAL_LINK_LIBRARIES} omnitrace::omnitrace-user-library
+                              omni-causal-example-lib-debug)
+
+    add_executable(${_NAME}-ndebug ${CAUSAL_SOURCES})
+    target_compile_definitions(${_NAME}-ndebug PRIVATE USE_COZ=0 USE_OMNI=0
+                                                       ${CAUSAL_DEFINITIONS})
+    target_include_directories(
+        ${_NAME}-ndebug PRIVATE ${OMNITRACE_EXAMPLE_ROOT_DIR}/causal
+                                ${CAUSAL_INCLUDE_DIRECTORIES})
+    target_link_libraries(
+        ${_NAME}-ndebug
+        PRIVATE ${CAUSAL_LINK_LIBRARIES} omnitrace::omnitrace-user-library
+                omni-causal-example-lib-no-debug)
+
+    add_executable(${_NAME}-omni-ndebug ${CAUSAL_SOURCES})
+    target_compile_definitions(${_NAME}-omni-ndebug PRIVATE USE_COZ=0 USE_OMNI=1
+                                                            ${CAUSAL_DEFINITIONS})
+    target_include_directories(
+        ${_NAME}-omni-ndebug PRIVATE ${OMNITRACE_EXAMPLE_ROOT_DIR}/causal
+                                     ${CAUSAL_INCLUDE_DIRECTORIES})
+    target_link_libraries(
+        ${_NAME}-omni-ndebug
+        PRIVATE ${CAUSAL_LINK_LIBRARIES} omnitrace::omnitrace-user-library
+                omni-causal-example-lib-no-debug)
+
+    add_dependencies(omni-causal-examples ${_NAME} ${_NAME}-omni ${_NAME}-ndebug
+                     ${_NAME}-omni-ndebug)
+
+    if(coz-profiler_FOUND)
+        omnitrace_causal_example_interface(omni-causal-example-lib-coz)
+        target_compile_options(omni-causal-example-lib-coz
+                               INTERFACE -g3 -gdwarf-3 -fno-omit-frame-pointer)
+
+        add_executable(${_NAME}-coz ${CAUSAL_SOURCES})
+        target_compile_definitions(${_NAME}-coz PRIVATE USE_COZ=1 USE_OMNI=0
+                                                        ${CAUSAL_DEFINITIONS})
+        target_include_directories(
+            ${_NAME}-coz PRIVATE ${OMNITRACE_EXAMPLE_ROOT_DIR}/causal
+                                 ${CAUSAL_INCLUDE_DIRECTORIES})
+        target_link_libraries(${_NAME}-coz PRIVATE ${CAUSAL_LINK_LIBRARIES}
+                                                   omni-causal-example-lib-coz coz::coz)
+
+        add_dependencies(omni-causal-examples ${_NAME}-coz)
+    endif()
+
+    if(OMNITRACE_INSTALL_EXAMPLES)
+        install(
+            TARGETS ${_NAME} ${_NAME}-omni ${_NAME}-coz
+            DESTINATION bin
+            COMPONENT omnitrace-examples
+            OPTIONAL)
+    endif()
+endfunction()
@@ -0,0 +1,42 @@
+cmake_minimum_required(VERSION 3.16 FATAL_ERROR)
+
+project(omnitrace-causal-example LANGUAGES CXX)
+
+if(OMNITRACE_DISABLE_EXAMPLES)
+    get_filename_component(_DIR ${CMAKE_CURRENT_LIST_DIR} NAME)
+
+    if(${PROJECT_NAME} IN_LIST OMNITRACE_DISABLE_EXAMPLES OR ${_DIR} IN_LIST
+                                                             OMNITRACE_DISABLE_EXAMPLES)
+        return()
+    endif()
+endif()
+
+set(CMAKE_BUILD_TYPE "Release")
+find_package(Threads REQUIRED)
+if(NOT TARGET omnitrace::omnitrace-user-library)
+    find_package(omnitrace REQUIRED COMPONENTS user)
+endif()
+
+add_library(causal-interface-library INTERFACE)
+target_compile_options(causal-interface-library INTERFACE -g3 -gdwarf-3
+                                                          -fno-omit-frame-pointer)
+target_link_libraries(causal-interface-library INTERFACE Threads::Threads
+                                                         ${CMAKE_DL_LIBS})
+
+omnitrace_causal_example_executable(
+    "causal-both"
+    SOURCES causal.cpp impl.cpp
+    LINK_LIBRARIES causal-interface-library
+    DEFINITIONS USE_RNG=1 USE_CPU=1)
+
+omnitrace_causal_example_executable(
+    "causal-rng"
+    SOURCES causal.cpp impl.cpp
+    LINK_LIBRARIES causal-interface-library
+    DEFINITIONS USE_RNG=1 USE_CPU=0)
+
+omnitrace_causal_example_executable(
+    "causal-cpu"
+    SOURCES causal.cpp impl.cpp
+    LINK_LIBRARIES causal-interface-library
+    DEFINITIONS USE_RNG=0 USE_CPU=1)
@@ -0,0 +1,167 @@
+#include "causal.hpp"
+
+#include <chrono>
+#include <cmath>
+#include <cstdio>
+#include <cstdlib>
+#include <ctime>
+#include <iostream>
+#include <mutex>
+#include <random>
+#include <string>
+#include <thread>
+#include <unistd.h>
+#include <vector>
+
+using mutex_t     = std::timed_mutex;
+using auto_lock_t = std::unique_lock<mutex_t>;
+using clock_type  = std::chrono::high_resolution_clock;
+using nanosec     = std::chrono::nanoseconds;
+
+namespace
+{
+std::chrono::duration<double, std::milli> t_ms;
+std::chrono::duration<double, std::milli> slow_ms;
+std::chrono::duration<double, std::milli> fast_ms;
+
+template <typename... Args>
+inline void
+consume_variables(Args&&...)
+{}
+}  // namespace
+
+template <bool>
+bool
+rng_func_impl(int64_t n, uint64_t rseed);
+
+template <bool>
+bool
+cpu_func_impl(int64_t n, int nloop);
+
+void
+rng_slow_func(int64_t n, uint64_t rseed) __attribute__((noinline));
+
+void
+rng_fast_func(int64_t n, uint64_t rseed) __attribute__((noinline));
+
+void
+cpu_slow_func(int64_t n, int nloop) __attribute__((noinline));
+
+void
+cpu_fast_func(int64_t n, int nloop) __attribute__((noinline));
+
+#if USE_CPU > 0
+#    define CPU_SLOW_FUNC(...) cpu_slow_func(__VA_ARGS__)
+#    define CPU_FAST_FUNC(...) cpu_fast_func(__VA_ARGS__)
+#else
+#    define CPU_SLOW_FUNC(...) consume_variables(__VA_ARGS__)
+#    define CPU_FAST_FUNC(...) consume_variables(__VA_ARGS__)
+#endif
+
+#if USE_RNG > 0
+#    define RNG_SLOW_FUNC(...) rng_slow_func(__VA_ARGS__)
+#    define RNG_FAST_FUNC(...) rng_fast_func(__VA_ARGS__)
+#else
+#    define RNG_SLOW_FUNC(...) consume_variables(__VA_ARGS__)
+#    define RNG_FAST_FUNC(...) consume_variables(__VA_ARGS__)
+#endif
+
+int
+main(int argc, char** argv)
+{
+    uint64_t rseed    = std::random_device{}();
+    int      nitr     = 200;
+    double   frac     = 70;
+    int64_t  slow_val = 100000000L;
+
+    if(argc > 1) frac = std::stod(argv[1]);
+    if(argc > 2) nitr = std::stoi(argv[2]);
+    if(argc > 3) rseed = std::stoul(argv[3]);
+    if(argc > 4) slow_val = std::stol(argv[4]);
+
+    int64_t fast_val = (frac / 100.0) * slow_val;
+    double  rfrac    = (fast_val / static_cast<double>(slow_val));
+    if(argc > 5) fast_val = std::stol(argv[5]);
+
+    printf("\nIterations: %i, fraction: %6.2f, random seed: %lu :: slow = %zu, "
+           "fast = %zu, expected ratio = %6.2f\n",
+           nitr, frac, rseed, slow_val, fast_val, rfrac * 100.0);
+
+    auto _t = clock_type::now();
+    for(int i = 0; i < nitr; ++i)
+    {
+        if(i == 0 || i + 1 == nitr || i % (nitr / 5) == 0)
+            printf("executing iteration: %i\n", i);
+        //
+        auto&& _slow_func = [](auto _nsec, auto _seed, auto _nloop) {
+            auto _t = clock_type::now();
+            CPU_SLOW_FUNC(_nsec, _nloop);
+            RNG_SLOW_FUNC(_nsec / 5, _seed);
+            slow_ms += (clock_type::now() - _t);
+        };
+        //
+        auto&& _fast_func = [](auto _nsec, auto _seed, auto _nloop) {
+            auto _t = clock_type::now();
+            CPU_FAST_FUNC(_nsec, _nloop);
+            RNG_FAST_FUNC(_nsec / 5, _seed);
+            fast_ms += (clock_type::now() - _t);
+        };
+        //
+        CAUSAL_BEGIN("main_iteration");
+        //
+        auto _threads = std::vector<std::thread>{};
+        _threads.emplace_back(std::move(_slow_func), slow_val, rseed, 10000);
+        _threads.emplace_back(std::move(_fast_func), fast_val, rseed, 10000);
+        for(auto& itr : _threads)
+            itr.join();
+        CAUSAL_END("main_iteration");
+        CAUSAL_PROGRESS;
+    }
+    t_ms += clock_type::now() - _t;
+    auto rms = (fast_ms.count() / slow_ms.count());
+    printf("slow_func() took %10.3f ms\n", slow_ms.count());
+    printf("fast_func() took %10.3f ms\n", fast_ms.count());
+    printf("total is %18.3f ms\n", t_ms.count());
+    printf("ratio is %18.3f %s\n", 100.0 * rms, "%");
+    printf("rdiff is %18.3f %s\n", 100.0 * (rms - rfrac), "%");
+}
+//
+//
+//
+void
+rng_slow_func(int64_t n, uint64_t rseed)
+{
+    // clang-format off
+    while(rng_func_impl<false>(n, rseed) != false) {}
+    // clang-format on
+}
+//
+//
+//
+void
+rng_fast_func(int64_t n, uint64_t rseed)
+{
+    // clang-format off
+    while(rng_func_impl<true>(n, rseed) != true) {}
+    // clang-format on
+}
+//
+//
+//
+void
+cpu_slow_func(int64_t n, int nloop)
+{
+    // clang-format off
+    while(cpu_func_impl<false>(n, nloop) != false) {}
+    // clang-format on
+}
+//
+//
+//
+void
+cpu_fast_func(int64_t n, int nloop)
+{
+    // clang-format off
+    while(cpu_func_impl<true>(n, nloop) != true) {}
+    // clang-format on
+}
@@ -0,0 +1,46 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#define CAUSAL_STR2(x) #x
+#define CAUSAL_STR(x)  CAUSAL_STR2(x)
+#define CAUSAL_LABEL   __FILE__ ":" CAUSAL_STR(__LINE__)
+
+#if defined(USE_OMNI) && USE_OMNI > 0
+#    include <omnitrace/causal.h>
+#    define CAUSAL_PROGRESS              OMNITRACE_CAUSAL_PROGRESS
+#    define CAUSAL_PROGRESS_NAMED(LABEL) OMNITRACE_CAUSAL_PROGRESS_NAMED(LABEL)
+#    define CAUSAL_BEGIN(LABEL)          OMNITRACE_CAUSAL_BEGIN(LABEL)
+#    define CAUSAL_END(LABEL)            OMNITRACE_CAUSAL_END(LABEL)
+#elif defined(USE_COZ) && USE_COZ > 0
+#    include <coz.h>
+#    define CAUSAL_PROGRESS              COZ_PROGRESS_NAMED(CAUSAL_LABEL)
+#    define CAUSAL_PROGRESS_NAMED(LABEL) COZ_PROGRESS_NAMED(LABEL)
+#    define CAUSAL_BEGIN(LABEL)          COZ_BEGIN(LABEL)
+#    define CAUSAL_END(LABEL)            COZ_END(LABEL)
+#else
+#    define CAUSAL_PROGRESS
+#    define CAUSAL_PROGRESS_NAMED(LABEL)
+#    define CAUSAL_BEGIN(LABEL)
+#    define CAUSAL_END(LABEL)
+#endif
@@ -0,0 +1,135 @@
+// MIT License
+//
+// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#include <chrono>
+#include <cmath>
+#include <cstdio>
+#include <cstdlib>
+#include <ctime>
+#include <iostream>
+#include <mutex>
+#include <optional>
+#include <random>
+#include <stdexcept>
+#include <string>
+#include <sys/times.h>
+#include <thread>
+#include <unistd.h>
+
+using mutex_t     = std::timed_mutex;
+using auto_lock_t = std::unique_lock<mutex_t>;
+using clock_type  = std::chrono::high_resolution_clock;
+using nanosec     = std::chrono::nanoseconds;
+
+namespace
+{
+inline __attribute__((always_inline)) int64_t
+clock_tick() noexcept;
+
+template <typename Precision, typename Ret = int64_t>
+inline __attribute__((always_inline)) Ret
+clock_tick() noexcept;
+
+template <typename Tp = int64_t, typename Precision = std::nano>
+inline __attribute__((always_inline)) Tp
+get_clock_now(clockid_t clock_id) noexcept;
+
+template <typename Tp = int64_t, typename Precision = std::nano>
+inline __attribute__((always_inline)) Tp
+get_clock_cpu_now() noexcept;
+}  // namespace
+
+//
+//  This implementation works well for Omnitrace
+//  while COZ makes poor predictions
+//
+template <bool V>
+bool
+rng_func_impl(int64_t n, uint64_t rseed)
+{
+    int64_t _n    = 0;
+    auto    _rng  = std::mt19937_64{ rseed };
+    auto    _dist = std::uniform_int_distribution<int64_t>{ 1, 1 };
+    // clang-format off
+    while(_n < n) _n += _dist(_rng);
+    // clang-format on
+    return V;
+}
+
+template bool rng_func_impl<true>(int64_t, uint64_t);
+template bool rng_func_impl<false>(int64_t, uint64_t);
+
+//
+//  This implementation works well for COZ
+//  while Omnitrace makes poor predictions
+//
+template <bool V>
+bool
+cpu_func_impl(int64_t n, int nloop)
+{
+    auto _t       = clock_type::now();
+    auto _cpu_now = get_clock_cpu_now();
+    auto _cpu_end = _cpu_now + n;
+    // clang-format off
+    while(get_clock_cpu_now() < _cpu_end) { for(volatile int i = 0; i < nloop; ++i) {} }
+    // clang-format on
+    return V;
+}
+
+template bool
+cpu_func_impl<true>(int64_t, int);
+template bool
+cpu_func_impl<false>(int64_t, int);
+
+namespace
+{
+int64_t
+clock_tick() noexcept
+{
+    static int64_t _val = ::sysconf(_SC_CLK_TCK);
+    return _val;
+}
+
+template <typename Precision, typename Ret>
+Ret
+clock_tick() noexcept
+{
+    return static_cast<Ret>(Precision::den) / static_cast<Ret>(clock_tick());
+}
+
+template <typename Tp, typename Precision>
+Tp
+get_clock_now(clockid_t clock_id) noexcept
+{
+    constexpr Tp    factor = Precision::den / static_cast<Tp>(std::nano::den);
+    struct timespec ts;
+    clock_gettime(clock_id, &ts);
+    return (ts.tv_sec * std::nano::den + ts.tv_nsec) * factor;
+}
+
+template <typename Tp, typename Precision>
+Tp
+get_clock_cpu_now() noexcept
+{
+    return get_clock_now<Tp, Precision>(CLOCK_THREAD_CPUTIME_ID);
+}
+}  // namespace
@@ -12,6 +12,8 @@ if(OMNITRACE_DISABLE_EXAMPLES)
 endif()

 set(CMAKE_BUILD_TYPE "RelWithDebInfo")
+string(REGEX REPLACE " -g(|[0-2]) " " -g3 " CMAKE_CXX_FLAGS_RELWITHDEBINFO
+                     "${CMAKE_CXX_FLAGS_RELWITHDEBINFO}")

 list(INSERT CMAKE_MODULE_PATH 0 ${PROJECT_SOURCE_DIR}/cmake/Modules)

@@ -63,9 +65,11 @@ endif()
 file(GLOB headers ${PROJECT_SOURCE_DIR}/*.h ${PROJECT_SOURCE_DIR}/*.hxx)
 file(GLOB sources ${PROJECT_SOURCE_DIR}/*.cc)

-add_executable(lulesh ${sources} ${headers})
-target_include_directories(lulesh PRIVATE ${PROJECT_SOURCE_DIR}/includes)
-target_link_libraries(lulesh PRIVATE Kokkos::kokkos lulesh-mpi)
+omnitrace_causal_example_executable(
+    "lulesh"
+    SOURCES ${sources} ${headers}
+    LINK_LIBRARIES Kokkos::kokkos lulesh-mpi
+    INCLUDE_DIRECTORIES ${PROJECT_SOURCE_DIR}/includes)

 if(OMNITRACE_INSTALL_EXAMPLES)
    if(LULESH_BUILD_KOKKOS)
@@ -76,8 +80,4 @@ if(OMNITRACE_INSTALL_EXAMPLES)
        set_target_properties(lulesh PROPERTIES INSTALL_RPATH
                                                "\$ORIGIN/../${CMAKE_INSTALL_LIBDIR}")
    endif()
-    install(
-        TARGETS lulesh
-        DESTINATION bin
-        COMPONENT omnitrace-examples)
 endif()
@@ -13,6 +13,8 @@

 #include "lulesh.h"

+#include "causal.hpp"
+
 static Kokkos::View<Real_t*> buffer;
 static size_t                buffer_size;
 static size_t                buffer_offset;
@@ -46,6 +48,7 @@ TimeIncrement(Domain& domain)

    if((domain.dtfixed() <= Real_t(0.0)) && (domain.cycle() != Int_t(0)))
    {
+        // CAUSAL_BEGIN("TimeIncrement_Iteration")
        Real_t ratio;
        Real_t olddt = domain.deltatime();

@@ -86,6 +89,8 @@ TimeIncrement(Domain& domain)
            newdt = domain.dtmax();
        }
        domain.deltatime() = newdt;
+        CAUSAL_PROGRESS_NAMED("TimeIncrement_Iteration");
+        // CAUSAL_END("TimeIncrement_Iteration")
    }

    if((targetdt > domain.deltatime()) &&
@@ -2241,12 +2246,15 @@ main(int argc, char* argv[])
        while((locDom.time() < locDom.stoptime()) && (locDom.cycle() < opts.its))
        {
            Kokkos::Tools::startSection(_time_incrp);
+            //CAUSAL_BEGIN("Iteration")
            TimeIncrement(locDom);
            Kokkos::Tools::stopSection(_time_incrp);

            Kokkos::Tools::startSection(_leap_frogp);
            LagrangeLeapFrog(locDom);
            Kokkos::Tools::stopSection(_leap_frogp);
+            CAUSAL_PROGRESS_NAMED("Iteration")
+            //CAUSAL_END("Iteration")

            if((opts.showProg != 0) && (opts.quiet == 0) && (myRank == 0))
            {
@@ -2254,6 +2262,7 @@ main(int argc, char* argv[])
                       double(locDom.time()), double(locDom.deltatime()));
            }
            Kokkos::Tools::markEvent("completed_timestep");
+            CAUSAL_PROGRESS
        }

        Kokkos::Tools::destroyProfileSection(_time_incrp);
@@ -13,11 +13,17 @@ endif()

 set(CMAKE_BUILD_TYPE "Release")
 find_package(Threads REQUIRED)
+
+add_library(parallel-overhead-compile-options INTERFACE)
+target_compile_options(parallel-overhead-compile-options INTERFACE -g)
+
 add_executable(parallel-overhead parallel-overhead.cpp)
-target_link_libraries(parallel-overhead PRIVATE Threads::Threads)
+target_link_libraries(parallel-overhead PRIVATE Threads::Threads
+                                                parallel-overhead-compile-options)

 add_executable(parallel-overhead-locks parallel-overhead.cpp)
-target_link_libraries(parallel-overhead-locks PRIVATE Threads::Threads)
+target_link_libraries(parallel-overhead-locks PRIVATE Threads::Threads
+                                                      parallel-overhead-compile-options)
 target_compile_definitions(parallel-overhead-locks PRIVATE USE_LOCKS=1)

 if(OMNITRACE_INSTALL_EXAMPLES)