* roctracer: use multiple tracks for HIP streams
Use different perfetto tracks for each stream, and set the name of
these tracks to the stream pointer values. Setting the name like this
matches the args in the API traces.
This fixes overlapping work on multiple streams appearing as a call
stack.
* Fix -pedantic
* Run clang-format
* Add option to disable per stream tracks in perfetto
* Updated scheme for roctracer activity + general roctracer fixes
- Per-device tracks
- Handle HSA OPS in ROCm 5.3
- the changes in ROCm 5.3 were causing HSA ops to get discarded
- Default for OMNITRACE_ROCTRACER_DISCARD_INVALID is now zero
- i.e. default behavior is to flip beg_ns and end_ns when beg_ns > end_ns
* Flush perfetto at end of hip_activity_callback
- fixes unterminated regions
* GitHub Actions and run-ci script updates
- improve reliability
* Set OMNITRACE_TMPDIR in testing
- files in /tmp get occasionally deleted during CI
Co-authored-by: Gergely Meszaros <gergely@streamhpc.com>
* Testing and CI support for Ubuntu 22.04
* Fixes for ROCm
- Jammy does not have ROCm installers
* Name, timeout, and python updates
- renamed ubuntu-jammy-external.yml to ubuntu-jammy.yml
- increased all 5 minute timeouts to 10 minutes
- include python 3.10 in testing
* Update dyninst to remove interposed definition of _r_debug
* Rebuild Dyninst + test install script
* Revert container change
* git safe directory
* pushd -> cd
* fix MPI include
* Fix testing step
* OMPI_ALLOW_RUN_AS_ROOT
* Test script changes
* Fix mismatched malloc / delete[]
* Jammy workflow tweaks
* CPack tweak for boost deb deps
* pthread_mutex_gotcha config returns when not enabled
* fix echoing config in CI
* USE_CLANG_OMP
- option to disable using LLVM OpenMP when building OpenMP test executables
- Jammy workflow sets USE_CLANG_OMP=OFF
* Dyninst submodule boost download
- updated containers workflow to include jammy
- updated workflow to use ci
* Updates to workflows + replace test-install.sh
- test-install.sh in this branch was replaced with one in main branch
* Expand jammy test-install.sh args
* Fix openmp-cg-sampling-duration test
* update timemory submodule
- use-after-free violation in popen::pclose
* revert some tweaks to sampling-duration test
* Fix env of test-install.sh
* cmake format
* jammy bash
* CPack install for jammy
* formatting workflow action version bump
* Update timemory submodule
- libunwind submodule via timemory sets SOVERSION to 99 to avoid ABI conflicts with v8
* Fix help menu for omnitrace-sample
* Support other boolean forms in test-install.sh
* Update docker files and build-docker.sh
- consolidated cases in build-docker.sh
- support rocm version of 0.0 (no rocm install)
- support rocm v5.3
- updated centos handling
* update opensuse actions/checkout version
* Tweaks to ubuntu-focal testing
- actions/checkout@v3
- use test-install script
* update cpack
- ubuntu 22.04
- rocm 5.3
- rename os matrix field to os-version
- remove CI_ROCM_VERSION (no longer necessary)
- remove default-rocm-version matrix field (no longer necessary)
- CentOS packaging
* fix argparsing and omnitrace-sample tests in install-tests.sh
* focal rocm test install workflow fix
* Fix omnitrace-sample build
* Dockerfile.centos + build-docker.sh updates
* Update actions/upload-artifact version
* Dockerfile.ubuntu: install rocm-device-libs
* Refactor cpack
* fix cpack if quotes
* Dockerfile.ubuntu rocm < 5 installs rocm-dev
* build-release.sh defaults to boost version 1.79.0
- More to come in later commit, below is just tidying some stuff up
- clang-tidy
- mpi_gotcha quiet about not finding funcs
- update to new papi config
- sampling block_samples / unblock_samples
- disable calling component's sample functions within sampler
- release doesn't strip library
- remove HSA and ROCP env variables from modulefile / setup-env
- preliminary support for LD_PRELOAD usage
- default sampling rate is 300 interrupts / second
- fixes various deadlock issues at startup
## Overview
This is a significant PR which has 3 very notable characteristics:
1. Omnitrace colorizes most of it's logging
2. Completely reworked the sampling
- Samples now record the current instruction pointers instead of strings
- This _dramatically_ decreases the overhead of taking a sample
- The collection of metrics during a sample are split out into another component, enabling that data collection to be disabled -- which decreases the sampling overhead even further
- When both `OMNITRACE_SAMPLING_CPUTIME` and `OMNITRACE_SAMPLING_REALTIME` are ON:
- `OMNITRACE_SAMPLING_CPUTIME_FREQ` and `OMNITRACE_SAMPLING_REALTIME_FREQ` can be used to individually control the sampling frequency
- `OMNITRACE_SAMPLING_CPUTIME_DELAY` and `OMNITRACE_SAMPLING_REALTIME_DELAY` can be used to individually control the delay time before starting
- Now, omnitrace does not start a real-time sampler on the main thread unless `OMNITRACE_SAMPLING_REALTIME` is ON
- In the future, an `OMNITRACE_SAMPLING_TIDS` (and real-time, cpu-time variants) configuration variable(s) will allow you to select which threads will be sampled
3. Files produced by `omnitrace` exe -- `available-instr.txt`, `instrumented-instr.txt`, etc. -- now no longer has `-instr` suffix and are placed in `instrumentation/` subfolder, i.e. `available-instr.txt` -> instrumentation/available.txt`
- This helped de-clutter the output folder
Most of the other edits were reorganization (e.g. internal namespace changes), cleanup, and splitting up functionality.
## Bug Fixes
There is a bug fix with respect to the HSA callbacks which disabled sampling on child threads when an HSA API call was made
## Details
- created thread_info struct for mapping different thread IDs
- reorganized file structure significantly
- added categories.hpp, concepts.hpp
- moved around name trait definitions
- moved all omnitrace components into `omnitrace::component` namespace
- there was a lot of inconsistency b/t using `tim::component` in some places and `omnitrace::component`
- added macros like OMNITRACE_DECLARE_COMPONENT in lieu of TIMEMORY_DECLARE_COMPONENT
- OMNITRACE_CRITICAL_TRACE_NUM_THREADS -> OMNITRACE_THREAD_POOL_SIZE
- roctracer and critical_trace use same thread pool
- critical_trace functions do not lock anymore bc of thread-local TaskGroup
- added `component::local_category_region` to support using `component::category_region` without explicitly passing in name
- removed `component::omnitrace` (unused)
- migrated KokkosP and OMPT to use `component::local_category_region`
- removed `component::user_region` as a result
- migrated omnitrace_{push,pop}_{trace,region}_hidden to use component::category_region
- removed `component::functors` as a result
- migrated some ppdefs
- `api::omnitrace` -> `project::omnitrace`
- `api::(...)` -> `category::(...)`
- improved recording the execution time of threads
- migrated this functionality out of pthread_create_gotcha and into thread_info
- moved mpi_gotcha, fork_gotcha, exit_gotcha, rcclp into omnitrace::component namespace
- split backtrace up into backtrace, backtrace_metrics, backtrace_timestamp components
- sampling.cpp handles setup and post-processing that was formerly in backtrace
- updated logging to use colors
- `OMNITRACE_COLORIZED_LOG` config variable
- updated docs on JSON output from timemory
- instrumentation info in instrumentation subfolder
- added testing for KokkosP entries
- added testing for ompt entries
- add_critical_trace function defined in critical_trace.hpp
- disable push_thread_state and pop_thread_state when thread state is Disabled or Completed
- add comp::page_rss to main bundle
- thread_data supports std::optional instead of std::unique_ptr
- thread_data supports tim::identity<T> to avoid unique_ptr or optional
- tracing::record_thread_start_time()
- tracing::push_timemory and tracing::pop_timemory are templated on CategoryT
- removed anonymous namespace from omnitrace::utility
- sampling backtrace stores instruction pointers instead of strings
- component::category_region updates
- handle disabled thread state
- handle finalized state
- fewer debug messages
- invoke thread_init()
- invoke thread_init_sampling()
- handle push/pop count based on category
- push/pop count only modified when used
- component::cpu_freq
- components/ensure_storage.hpp
- reworked the pthread_create replacement function
- updated parallel-overhead example to report # of times locked
- OMNITRACE_MAX_UNWIND_DEPTH build option
- update timemory submodule
tracing NS + category region component
- made library.cpp impl more broadly available
- support for perfetto args
- MPI wrappers encode args and return type
- new categories / perfetto categories
- omnitrace_library category -> libomnitrace
- omnitrace_dl_library -> libomnitrace-dl
Jonathan R. Madsen