Causal profiling (#229)

* Addition of basic structure

* Reworked categories

* More causal integration additions

* Causal implementation

* Update examples

* delete virtual_speedup files

* Update perfetto submodule to v31.0

* Update dyninst submodule

* Update timemory submodule

* ElfUtils build for libdw

* OMNITRACE_LIKELY and OMNITRACE_UNLIKELY

* Update common lib join

* Examples updates for causal profiling

* config updates with causal options

- OMNITRACE_CAUSAL_FIXED_LINE
- OMNITRACE_CAUSAL_FIXED_SPEEDUP
- OMNITRACE_CAUSAL_FILE
- OMNITRACE_CAUSAL_BINARY_SCOPE
- OMNITRACE_CAUSAL_SOURCE_SCOPE
- version info in banner
- support increments in parse_numeric_range
- fix occasional deadlock in first call to get_config

* PTL general task group

* Always include PID in debug/verbose messages

* Add blocking/unblocking gotchas to runtime init bundle

* CausalState

* thread_data updates

- generic component_bundle_cache

* Improve handling of causal in category_region

* components updates

- backtrace_causal component
- backtrace::get_data member func
- decrease ignore_depth in backtrace::sample(int)
- handle "omnitrace_main" in backtrace::filter_and_patch(...)
- tweak internal thread state scope for pthread_mutex_gotcha wrappers

* simplify tracing get_instrumentation_bundles usage

* sampling updates

- include backtrace_causal component
- disable backtrace_metrics if using causal and not using perfetto
- disable backtrace and backtrace_timestamp when using causal
- post_process_causal

* causal updates

- more checks in blocking_gotcha and unblocking_gotcha start/stop
- miscellaneous overhaul of data
- experiment update

* Remove virtual speedup

* libomnitrace code_object

* causal-profiling test

* libomnitrace library.cpp updates

- handle causal profiling
- fini_bundle

* Disable causal profiling by default

* Updated causal code and example

- example: three execution variants: cpu + rng, cpu, rng
- example: three instrumentation variants: none, omni, coz
- fix blocking gotcha credit
- rework perform_experiment_impl
- get_eligible_address_ranges
- compute_eligible_lines
- support fixed lines/speedups/functions
- update selected_entry to support function mode
- fix causal::delay
- experiment updates

* omnitrace_progress / omnitrace_user_progress

- with accompanying omnitrace_annotated_progress / omnitrace_user_annotated_progress

* Update timemory submodule

* CausalMode

- mode indicated whether causal predictions source be at line-level or function-level

* code_object, config, runtime, sampling, thread_data

- code_object: address_range
- code_object: basic::line_info serialize(), name(), hash()
- config updates
- two signals for causal sampling
- thread_data init fixes

* pthread updates

- pthread_create_gotcha processes delays
- pthread_mutex_gotcha does not wrap pthread_join in causal mode

* backtrace_causal update

- dynamic delay period stats

* main wrapper uses basename of argv[0]

* update elfio submodule

* perf support (currently unused)

* Fix experiment JSON serialization

- static_vector.hpp (unused)

* causal executable + config options updates

- omnitrace-causal exe simplifies running multiple causal configs
- changed the causal config option names

* Support both throughput and latency points

* process-causal-json.py script

- will be used later for testing

* stable_vector

* Rework thread_data

* Improve omnitrace-causal exe

- better verbosity handling
- correct diagnosis of status for child process
- execvpe when only one iteration (debugging)

* Update timemory submodule

* exe --version

- omnitrace, omnitrace-avail, and omnitrace-sample all support --version on command-line

* OMNITRACE_INTERNAL_API + OMNITRACE_{LIKELY,UNLIKELY}

* omnitrace-causal cmake format

* omnitrace config update

- OMNITRACE_CAUSAL_FILE_CLOBBER

* custom exception

- wraps STL exception and gets stacktrace during construction

* exit_gotcha supports _Exit

* use global construct_on_init + max threads

- add some safety when exceeding max # of threads

* update code_object binary filter

- exclude dyninst and tbbmalloc library

* containers: c_array, static_vector, stable_vector

- moved utility::c_array to container::c_array
- created static_vector: std::vector bound to std::array
- created stable_vector: vector with stable references

* grow thread_data when new thread created

* causal updates

- data: improve compute_eligible_lines to ignore lambdas
- data: use new thread_data
- delay: use new thread_data
- experiment: properly support latency points
- experiment: support file clobber
- experiment: ensure non-zero experiment time
- progress_point: use new thread_data
- backtrace_causal: use new thread_data

* Update causal-profiling tests

* fix omnitrace-causal backslash escaping

* process-causal-json script

* restructure causal implementation

- update verbose messages for omnitrace-causal diagnose_status
- migrated causal implementation in sampling.cpp to causal/sampling.cpp
- OMNITRACE_USE_CAUSAL does not require OMNITRACE_USE_SAMPLING
- added Mode::Causal
- causal sampling uses same signals as regular sampling
- moved tracing::thread_init to implementation file
- combined tracing::thread_init and tracing::thread_init_sampling
- added causal/components folder
- pthread_create_gotcha::wrapper_config
- omnitrace_preload checks OMNITRACE_USE_CAUSAL
  - updates mode accordingly

* update timemory submodule

* update timemory submodule

* causal example updates

- causal for lulesh

* perf code + utility - helpers

- relocated causal perf code
- placement new when generating unique ptr trait for potentially allocating during sampling
- additions to utility header
- removed previously added helpers.hpp

* update timemory submodule

* Default env variables for omnitrace-causal

- activate OMNITRACE_USE_KOKKOSP, etc.

* update stable_vector and static_vector

- static vector can use atomic for size tracking for thread-safe situations

* update causal example header

- CAUSAL_PROGRESS_NAMED
- use CAUSAL_ prefix for some macros

* Tweak lulesh example

- use CAUSAL_PROGRESS instead of CAUSAL_BEGIN and CAUSAL_END

* omnitrace-sample support for causal mode

- set OMNITRACE_USE_SAMPLING to off when OMNITRACE_MODE=causal

* refactor and cleanup code_object

- scope filter
- fixes to address_range

* overhaul causal data + causal config options

- full support for function and line mode
- support static vector of instruction pointers
- improve line info mapping resolution
- remove thread-locality from miscellanous functions where unnecessary
- causal options for {binary,source,function,fileline} exclusion

* causal experiment, sampling, and backtrace updates

- is_selected + unwind address array
- experiment warning about progress points
- increased buffer size for backtrace_casual sampler
- backtrace_causal only stores IP addresses instead of full unwind info

* category_region updates

- minor refactor
- local_category_region::mark

* Update causal tests

* Bump version to 1.8.0

* omnitrace-causal args + CLOBBER -> RESET

- renamed OMNITRACE_CAUSAL_FILE_CLOBBER to OMNITRACE_CAUSAL_FILE_RESET
- updated omnitrace-causal exe to support recently added configuration options
- other miscellaneous tweaks to data.cpp, experiment.cpp, and sampling.cpp

* Refactor causal and code_object

- code_object.hpp and code_object.cpp moved into binary folder
- causal components namespaced into omnitrace::causal::component
- moved sample_data out of backtrace_causal and into own file
- renamed backtrace_causal to causal::component::backtrace

* preload omnitrace_init + OMNITRACE_DEBUG_MARK

- env OMNITRACE_DEBUG_MARK
- fix omnitrace_init call when LD_PRELOAD-ing omnitrace

* Fix fileline support + line-info output names + experiment log

- line-info log files are prefixed with experiment name
- don't print experiment duration when E2E
- account for fileline scope in analysis

* KokkosP: OMNITRACE_KOKKOSP_NAME_LENGTH_MAX

- config option to limit the name of kokkos tool callbacks
- remove [kokkos] from KokkosP names

* Update causal example

- minor tweaks to decrease probability of overlapping regions in binary

* omnitrace-causal update

- prefix N / Ntot in environment printout

* Miscellaneous updates

- causal::finish_experimenting()
- OMNITRACE_CAUSAL_RANDOM_SEED
- KokkosP causal updates
  - exclude some callbacks, make some callbacks unique, etc.
- address_range::operator+=(address_range)
- combine contiguous ranges in binary/analysis.cpp when file, func, line is same and address range is contiguous
- bfd_line_info reads inline info
- wait for perform_experiment_impl to complete
- causal::delay updates
  - delay::process checks if experiment is active
  - uses threading::get_id()
- experiment scales duration up for larger speedup experiments
- line info samples includes excluded lines
- sampler uses CLOCK_REALTIME
- blocking_gotcha updates
  - is no longer fully static
  - adds audit routine which sets the postblock value to zero if try/timed routine fails
- category::host was added to causal_throughput_categories_t
- pthread_create_gotcha sets new threads local parent delay
  - was using internal value, now uses sequent value

* Causal improvements to KokkosP

* Updates to experiment time scaling

- use stats instead of just max

* binary/link_map.{hpp,cpp}

* update process-causal-json.py

* Folded fileline scope into source scope

* Update documentation

- Add documentation for causal profiling
- Replace 'Omnitrace' with 'OmniTrace' everywhere

* Update causal-helpers.cmake + omnitrace-testing.cmake

- split tests/CMakeLists.txt partially into omnitrace-testing.cmake

* omnitrace/causal.h

- OMNITRACE_CAUSAL_PROGRESS
- OMNITRACE_CAUSAL_PROGRESS_NAMED
- OMNITRACE_CAUSAL_BEGIN
- OMNITRACE_CAUSAL_END

* selected_entry + remove default filters for lambdas and operator()

- selected entry stores range and binary load address

* update process-causal-json.py

* format examples/lulesh/CMakeLists.txt

* causal-helpers find_package(Threads)

* OMNITRACE_KOKKOSP_KERNEL_LOGGER

- was OMNITRACE_KOKKOS_KERNEL_LOGGER

* quiet find of coz-profiler

* Fix rocm_smi exception handling

* Update timemory submodule (binutils)

- fix binutls compile error on some systems
- bump binutils to v2.40

* Fix miscellaneous tests

* OMNITRACE_KOKKOSP_PREFIX

* revert rocm_smi handling

* ElfUtils updates

- default to download version 0.188
- add -Wno-error=null-dereference due to GCC 12 compiler error

* Update causal example

* Remove OMNITRACE_VERBOSE from global workflow envs

* Reliable causal test

* disable compilation of causal perf files

* Remove set_current_selection with unwind stack

* update timemory submodule

* fix for segfault on bionic

- locking in TLS dtor was causing segfault

* remove experiment::is_selected(unwind_stack_t)

* update default init of selected_entry

* Fix for when IP is not offset by load address

* Update CMakeLists.txt

* Miscellaneous updates

- OMNITRACE_WARNING_OR_CI_THROW
- OMNITRACE_REQUIRE
- OMNITRACE_PREFER
- fixed issues with no ASLR
-  added load address variable and ipaddr() func to basic/bfd line info
- removed get_basic() from dwarf_line_info
- TIMEMORY_PREFER -> OMNITRACE_PREFER
- removed previously added binary_address and range variables from selected_entry

* Removed superfluous CausalState

* Additional causal tests (lulesh + kokkos)

* filter, prefer, analysis ASLR handling

- removed default filter on cold functions
- fixed OMNITRACE_PREFER
- fixed analysis ASLR handling

* Tweak line-info output

* Removed some superfluous code

- causal/delay
- causal/selected_entry

* Exclude main.cold in function mode

* Update validate-perfetto-proto.py

- account for occasional http errors

* Add sampling test disabling tmp files

* argparser for process-causal-json

- support validation
- support filtering

* Avoid pthread_{lock,unlock} in sampling offload

- use homemade atomic_mutex/atomic_lock since contention will be low and using pthread tools might trigger our wrappers

* Rename process-causal-json.py

- validate-causal-json.py

* rework omnitrace_add_causal_test

- capable of performing validation
- added validation tests

* Fix kokkosp_begin_deep_copy + causal

* Tweak address range in bfd_line_info::read_pc

* Tweak analysis and data IP handling

- look for gaps

* Disable scaling experiment time by speedup

* Revert change in max threads during CI

* binary updates

- significant overhaul of binary analysis implementation
- removed "basic_line_info" and "bfd_line_info" in lieu of "symbol" class
  - symbol class has basic BFD info + vector of inlines + vector of dwarf info

* Updated causal to use new binary analysis

- Fix symbol.cpp includes

* Updated formatting target

- include *.cmake files

* Updated causal tests

- causal tests should be stable now

* Update timemory and dyninst submodules

- TPLs are stripped + built w/o debug info

* Increase tolerance for causal validation speedups

- higher speedups have more variance (increased to +/- 5 from 3)

* Support causal output for MPI

- i.e. tag with MPI rank

* omnitrace-causal launcher argument

* improve experiment sampling output

* causal data updates

- call compute lines once
- fixed filtered cached binary info
- debugging info when experiment fails to start

* Tweaked causal validation tests

* dwarf_entry ranges

* CI updates

- increase max threads to 64

* Tweak causal E2E validation tests

- more threads
- shorter thread runtime
- more iterations

* Fix shadowed variable

* fix symbol read_bfd last PC calculation

* fix maybe-uninitialized warning

* omnitrace-causal launcher update

- only inject "omnitrace-causal --" once
- throw error if no matches found

* Update causal profiling docs for launcher

* fix address range boundaries

source/lib/omnitrace/library.cpp

@@ -26,12 +26,16 @@
 
 #include "api.hpp"
 #include "common/setup.hpp"
+#include "library/causal/data.hpp"
+#include "library/causal/experiment.hpp"
+#include "library/causal/sampling.hpp"
 #include "library/components/exit_gotcha.hpp"
 #include "library/components/fork_gotcha.hpp"
 #include "library/components/fwd.hpp"
 #include "library/components/mpi_gotcha.hpp"
 #include "library/components/pthread_gotcha.hpp"
 #include "library/components/rocprofiler.hpp"
+#include "library/concepts.hpp"
 #include "library/config.hpp"
 #include "library/coverage.hpp"
 #include "library/critical_trace.hpp"
@@ -52,6 +56,7 @@
 #include "library/utility.hpp"
 #include "omnitrace/categories.h"  // in omnitrace-user
 
+#include <timemory/process/threading.hpp>
 #include <timemory/signals/signal_handlers.hpp>
 #include <timemory/signals/types.hpp>
 #include <timemory/hash/types.hpp>
@@ -67,6 +72,8 @@
 #include <mutex>
 #include <string_view>
 #include <utility>
+#include <cstdlib>
+#include <stdexcept>
 
 using namespace omnitrace;
 
@@ -92,9 +99,39 @@ namespace
 auto _timemory_manager  = tim::manager::instance();
 auto _timemory_settings = tim::settings::shared_instance();
 
+bool
+ensure_initialization(bool _offset, int64_t _glob_n, int64_t _offset_n)
+{
+    auto _exit_info = component::exit_gotcha::get_exit_info();
+    if(_exit_info.is_known && _exit_info.exit_code != EXIT_SUCCESS) return _offset;
+
+    auto _tid         = utility::get_thread_index();
+    auto _max_threads = grow_data(_tid + 1);
+
+    if(_tid > 0 && _tid < _max_threads)
+    {
+        const auto& _info = thread_info::get();
+        OMNITRACE_BASIC_VERBOSE_F(3,
+                                  "thread info: %s, offset: %s, global counter: %li, "
+                                  "offset counter: %li, max threads: %li\n",
+                                  std::to_string(static_cast<bool>(_info)).c_str(),
+                                  std::to_string(_offset).c_str(), _glob_n, _offset_n,
+                                  _max_threads);
+    }
+
+    return _offset;
+}
+
 auto
 ensure_finalization(bool _static_init = false)
 {
+    if(_static_init)
+    {
+        auto _idx = threading::add_callback(&ensure_initialization);
+        if(_idx < 0)
+            throw exception<std::runtime_error>("failure adding threading callback");
+    }
+
     const auto& _info = thread_info::init();
     const auto& _tid  = _info->index_data;
     if(_tid)
@@ -149,6 +186,51 @@ is_system_backend()
 
 using Device = critical_trace::Device;
 using Phase  = critical_trace::Phase;
+
+template <typename... Tp>
+struct fini_bundle
+{
+    using data_type = std::tuple<Tp...>;
+
+    TIMEMORY_DEFAULT_OBJECT(fini_bundle)
+
+    fini_bundle(std::string_view _label)
+    : m_label{ _label }
+    {}
+
+    template <typename... Args>
+    void start(Args&&... _args)
+    {
+        TIMEMORY_FOLD_EXPRESSION(tim::operation::start<Tp>{}(
+            std::get<Tp>(m_data), std::forward<Args>(_args)...));
+    }
+
+    template <typename... Args>
+    void stop(Args&&... _args)
+    {
+        TIMEMORY_FOLD_EXPRESSION(tim::operation::stop<Tp>{}(
+            std::get<Tp>(m_data), std::forward<Args>(_args)...));
+    }
+
+    std::string as_string(bool _print_prefix = true) const
+    {
+        std::stringstream _ss;
+        if(_print_prefix && m_label.length() > 0) _ss << m_label << " : ";
+        _ss << timemory::join::join(", ", std::get<Tp>(m_data)...);
+        return _ss.str();
+    }
+
+    std::string_view m_label = {};
+    data_type        m_data  = {};
+};
+
+template <typename... Tp>
+struct fini_bundle<tim::lightweight_tuple<Tp...>>
+{
+    using base_type = fini_bundle<Tp...>;
+};
+
+using fini_bundle_t = typename fini_bundle<main_bundle_t>::base_type;
 }  // namespace
 
 //======================================================================================//
@@ -365,13 +447,21 @@ omnitrace_init_tooling_hidden()
             OMNITRACE_SCOPED_SAMPLING_ON_CHILD_THREADS(false);
             process_sampler::setup();
         }
-        if(get_use_sampling())
+        if(get_use_causal())
         {
-            OMNITRACE_SCOPED_SAMPLING_ON_CHILD_THREADS(false);
-            sampling::setup();
+            {
+                OMNITRACE_SCOPED_SAMPLING_ON_CHILD_THREADS(false);
+                causal::sampling::setup();
+            }
+            push_enable_sampling_on_child_threads(get_use_causal());
+            sampling::unblock_signals();
         }
-        if(get_use_sampling())
+        else if(get_use_sampling())
         {
+            {
+                OMNITRACE_SCOPED_SAMPLING_ON_CHILD_THREADS(false);
+                sampling::setup();
+            }
             push_enable_sampling_on_child_threads(get_use_sampling());
             sampling::unblock_signals();
         }
@@ -400,6 +490,8 @@ omnitrace_init_tooling_hidden()
 
     tasking::setup();
 
+    if(get_use_causal()) causal::start_experimenting();
+
     if(get_use_timemory())
     {
         comp::user_global_bundle::global_init();
@@ -457,13 +549,14 @@ omnitrace_init_tooling_hidden()
 //======================================================================================//
 
 extern "C" void
-omnitrace_init_hidden(const char* _mode, bool _is_binary_rewrite, const char* _argv0)
+omnitrace_init_hidden(const char* _mode, bool _is_binary_rewrite, const char* _argv0_c)
 {
     static int  _total_count = 0;
     static auto _args = std::make_pair(std::string_view{ _mode }, _is_binary_rewrite);
 
     auto _count   = _total_count++;
     auto _mode_sv = std::string_view{ _mode };
+    auto _argv0   = (_argv0_c) ? std::string{ _argv0_c } : config::get_exe_name();
     // this function may be called multiple times if multiple libraries are instrumented
     // we want to guard against multiple calls which with different arguments
     if(_count > 0 &&
@@ -495,21 +588,23 @@ omnitrace_init_hidden(const char* _mode, bool _is_binary_rewrite, const char* _a
                 "called after omnitrace was initialized. state = %s. Mode-based settings "
                 "(via -M <MODE> passed to omnitrace exe) may not be properly "
                 "configured.\n",
-                _mode, std::to_string(_is_binary_rewrite).c_str(), _argv0,
+                _mode, std::to_string(_is_binary_rewrite).c_str(), _argv0.c_str(),
                 std::to_string(get_state()).c_str());
         }
     }
 
-    tracing::get_finalization_functions().emplace_back([_argv0]() {
+    tracing::get_finalization_functions().emplace_back([_argv0_c]() {
         OMNITRACE_CI_THROW(get_state() != State::Active,
                            "Finalizer function for popping main invoked in non-active "
                            "state :: state = %s\n",
                            std::to_string(get_state()).c_str());
         if(get_state() == State::Active)
         {
+            auto _name = (_argv0_c) ? std::string{ _argv0_c } : config::get_exe_name();
             // if main hasn't been popped yet, pop it
-            OMNITRACE_BASIC_VERBOSE(2, "Running omnitrace_pop_trace(%s)...\n", _argv0);
-            omnitrace_pop_trace_hidden(_argv0);
+            OMNITRACE_BASIC_VERBOSE(2, "Running omnitrace_pop_trace(%s)...\n",
+                                    _name.c_str());
+            omnitrace_pop_trace_hidden(_name.c_str());
         }
     });
 
@@ -521,7 +616,7 @@ omnitrace_init_hidden(const char* _mode, bool _is_binary_rewrite, const char* _a
     OMNITRACE_CONDITIONAL_BASIC_PRINT_F(
         get_debug_env() || get_verbose_env() > 2,
         "mode: %s | is binary rewrite: %s | command: %s\n", _mode,
-        (_is_binary_rewrite) ? "y" : "n", _argv0);
+        (_is_binary_rewrite) ? "y" : "n", _argv0.c_str());
 
     tim::set_env("OMNITRACE_MODE", _mode, 0);
     config::is_binary_rewrite() = _is_binary_rewrite;
@@ -561,6 +656,8 @@ omnitrace_finalize_hidden(void)
 {
     // disable thread id recycling during finalization
     threading::recycle_ids() = false;
+    // disable initialization callback
+    threading::remove_callback(&ensure_initialization);
 
     set_thread_state(ThreadState::Completed);
 
@@ -629,13 +726,6 @@ omnitrace_finalize_hidden(void)
     tim::signals::enable_signal_detection({ tim::signals::sys_signal::Interrupt },
                                           [](int) {});
 
-    std::string              _bundle_name = OMNITRACE_FUNCTION;
-    comp::user_global_bundle _bundle{ _bundle_name.c_str() };
-    _bundle.clear();
-    _bundle.insert<comp::wall_clock, comp::cpu_clock, comp::peak_rss, comp::page_rss,
-                   comp::cpu_util>();
-    _bundle.start();
-
     OMNITRACE_DEBUG_F("Copying over all timemory hash information to main thread...\n");
     // copy these over so that all hashes are known
     auto& _hzero = tracing::get_timemory_hash_ids(0);
@@ -663,6 +753,9 @@ omnitrace_finalize_hidden(void)
         get_main_bundle()->stop();
     }
 
+    fini_bundle_t _finalization{};
+    _finalization.start();
+
     if(get_use_rcclp())
     {
         OMNITRACE_VERBOSE_F(1, "Shutting down RCCLP...\n");
@@ -741,6 +834,12 @@ omnitrace_finalize_hidden(void)
         rocprofiler::rocm_cleanup();
     }
 
+    if(get_use_causal())
+    {
+        OMNITRACE_VERBOSE_F(1, "Shutting down causal sampling...\n");
+        causal::sampling::shutdown();
+    }
+
     if(get_use_sampling())
     {
         OMNITRACE_VERBOSE_F(1, "Shutting down sampling...\n");
@@ -764,6 +863,7 @@ omnitrace_finalize_hidden(void)
     // if they are still running (e.g. thread-pool still alive), the
     // thread-specific data will be wrong if try to stop them from
     // the main thread.
+    auto _thr_verbose = (config::get_use_causal()) ? 1 : 0;
     for(auto& itr : thread_data<thread_bundle_t>::instances())
     {
         if(itr && itr->get<comp::wall_clock>() &&
@@ -772,7 +872,7 @@ omnitrace_finalize_hidden(void)
             std::string _msg = JOIN("", *itr);
             auto        _pos = _msg.find(">>>  ");
             if(_pos != std::string::npos) _msg = _msg.substr(_pos + 5);
-            OMNITRACE_VERBOSE_F(0, "%s\n", _msg.c_str());
+            OMNITRACE_VERBOSE_F(_thr_verbose, "%s\n", _msg.c_str());
         }
     }
 
@@ -785,6 +885,12 @@ omnitrace_finalize_hidden(void)
         sampling::post_process();
     }
 
+    if(get_use_causal())
+    {
+        OMNITRACE_VERBOSE_F(1, "Finishing the causal experiments...\n");
+        causal::finish_experimenting();
+    }
+
     if(get_use_critical_trace() || (get_use_rocm_smi() && get_use_roctracer()))
     {
         OMNITRACE_VERBOSE_F(1, "Generating the critical trace...\n");
@@ -929,18 +1035,6 @@ omnitrace_finalize_hidden(void)
         }
     }
 
-    _bundle.stop();
-    auto _get_metric = [](auto* _v, std::string_view _tail) -> std::string {
-        return (_v) ? JOIN("", *_v, _tail) : std::string{};
-    };
-
-    OMNITRACE_VERBOSE_F(0, "Finalization metrics: %s%s%s%s%s\n",
-                        _get_metric(_bundle.get<comp::wall_clock>(), ", ").c_str(),
-                        _get_metric(_bundle.get<comp::peak_rss>(), ", ").c_str(),
-                        _get_metric(_bundle.get<comp::page_rss>(), ", ").c_str(),
-                        _get_metric(_bundle.get<comp::cpu_clock>(), ", ").c_str(),
-                        _get_metric(_bundle.get<comp::cpu_util>(), "").c_str());
-
     if(_timemory_manager && _timemory_manager != nullptr)
     {
         _timemory_manager->add_metadata([](auto& ar) {
@@ -967,6 +1061,8 @@ omnitrace_finalize_hidden(void)
                                           "omnitrace", _cfg);
     }
 
+    _finalization.stop();
+
     if(_perfetto_output_error)
     {
         OMNITRACE_THROW("Error opening perfetto output file: %s",
@@ -984,7 +1080,7 @@ omnitrace_finalize_hidden(void)
 
     config::finalize();
 
-    OMNITRACE_VERBOSE_F(0, "Finalized\n");
+    OMNITRACE_VERBOSE_F(0, "Finalized: %s\n", _finalization.as_string().c_str());
 }
 
 //======================================================================================//