Graf commitů

35 Commity

Autor SHA1 Zpráva Datum
Jonathan R. Madsen 0ee32b3755 Improved sampling performance (#74)
* Improved sampling performance

* Sampling tweaks

- backtrace::get() returns vector of string_view
- further performance improvements
- tweaked _use_label
- wrapped samples in perfetto "samples [omnitrace]" block
- samples in TID=0 in sampling mode are in separate thread row

* Fix empty HW counter desc + category for sampling

- fallback to to metric name if papi event info description not found
- add perfetto sampling category

* Limit the SIGALRM frequency
2022-07-12 18:08:17 -05:00
Jonathan R. Madsen e099c84640 pthread_rwlock deadlock fix (#82)
- found when using ROCm-enabled OpenMPI with rocHPL
  - when wrapping pthread_rwlock_rdlock, pthread_rwlock_wrlock, and pthread_rwlock_unlock, omnitrace has been found to deadlock for some unknown reason
- New configuration variable: OMNITRACE_TRACE_THREAD_RW_LOCKS which defaults to false
2022-07-11 20:59:57 -05:00
Jonathan R. Madsen f6b6be3ddc Fix empty OMNITRACE_CONFIG_FILE and suppressing config and parsing (#81)
* Fix empty OMNITRACE_CONFIG_FILE and suppressing config and parsing
2022-07-11 18:46:12 -05:00
Jonathan R. Madsen f82845388a Handle OMNITRACE_ENABLED + minor updates (#78)
- handle OMNITRACE_ENABLED=OFF by disabling everything
- use set_data to get wrappee in pthread_create_gotcha
- clear roctracer_data storage if roctracer not initialized
2022-06-29 19:39:55 -05:00
Jonathan R. Madsen 2e1fd5a3c4 HIP API args in perfetto + new perfetto categories (#76)
* HIP API perfetto args + updated perfetto categories

- Support for HIP API args field in perfetto
- PERFETTO_CATEGORIES -> OMNITRACE_PERFETTO_CATEGORIES
- Changed perfetto categories for several trace events and trace counters
- migrated several TRACE_EVENT_* to use omnitrace::tracing::{push,pop}_perfetto_ts(...)

* Tweaked category_region to encode the type of args as well as value

- Affects MPI args field in perfetto

* Improved testing in ubuntu-focal.yml

- "Test Install" step sources setup-env.sh
- "Test Install" step tests python support
- "Test Install" step tests reading ~/.omnitrace.cfg
- Avoid installing boost and tbb libs when building from submodule

* validate-perfetto-proto.py accepts -m / --categories

* Remove reference from category_region typeids

* Tweak opensuse action name

* Tweak the "Test Install" Step of ubuntu-focal
2022-06-29 16:26:02 -05:00
Jonathan R. Madsen 1877ebf47b omnitrace-avail generate config (#69)
* Config updates

- See PR #69 for details

- change type of OMNITRACE_DL_VERBOSE
- add "deprecated" category to OMNITRACE_ROCM_SMI_DEVICES
- reduce size of perfetto shared memory size hint
- deprecate OMNITRACE_OUTPUT_FILE in favor of OMNITRACE_PERFETTO_FILE
- set papi event choices
- read config file after reading command line
- fix update of OMNITRACE_DL_VERBOSE
- mark several settings as hidden
- timemory update support hidden attribute for settings
- rework get_perfetto_output_filename()
- Hide settings from not available backends

* Rework omnitrace-avail to support dumping configurations

* Overwrite query, tests, output flag

- Support using -O flag when dumping config
- Support checking before overwriting existing config
- Support --force to overwrite existing config
- Fix get_component_info not including omnitrace components
- Testing for dumping config

* Update documentation on omnitrace-avail

* Fix issue with timemory format + "/__w/"

* Update output prefix keys docs

* Rename --dump-config to --generate-config

* Hide MPI related options

- OMNITRACE_PERFETTO_COMBINE_TRACES and OMNITRACE_COLLAPSE_PROCESSES are hidden w/o MPI support
2022-06-28 01:36:04 -05:00
Jonathan R. Madsen efe1edd253 Fix PID resolution + OMNITRACE_VERSION + fix various configs (#71)
* Fix to pid via mpi_gotcha

* OMNITRACE_VERSION defines

* call perfetto on hsa_activity_callback thread

* Test label tweak

* Config fixes

- Change type of OMNITRACE_DL_VERBOSE
- Update OMNITRACE_DL_VERBOSE properly
- Add OMNITRACE_ROCM_SMI_DEVICES to deprecated group
- Set default_process_suffix

* metadata for OMNITRACE_VERSION and OMNITRACE_HIP_VERSION
2022-06-27 23:01:24 -05:00
Jonathan R. Madsen 5105e2c94f tracing NS + category region component + MPI args (#52)
tracing NS + category region component

- made library.cpp impl more broadly available
- support for perfetto args
- MPI wrappers encode args and return type
- new categories / perfetto categories
- omnitrace_library category -> libomnitrace
- omnitrace_dl_library -> libomnitrace-dl
2022-06-24 16:08:06 -05:00
Jonathan R. Madsen 27e4e82376 Deprecate omnitrace use thread sampling (#68)
* Deprecate OMNITRACE_USE_THREAD_SAMPLING

* Reworked config based on OMNITRACE_MODE

- config::set_default_setting_value(...)
- config::get_mode() is now dynamically deduced
- moved tweaking defaults from library.cpp to config::configure_mode_settings(...)
- timemory submodule update fixing vsetting issue

* runtime.md update

* revert accidental lambda name change

* Reintroduce (deprecated) OMNITRACE_ROCM_SMI_DEVICES

- add handle_deprecated_setting(...) for this deprecated setting
2022-06-24 15:03:15 -05:00
Jonathan R. Madsen 335f309b8f libomnitrace uses common headers (#62)
* libomnitrace uses common headers

- JOIN(...) uses ::omnitrace::common::join instead of ::tim::mpl::apply<>::join

* Update lib/common CMake
2022-06-22 15:19:34 -05:00
Jonathan R. Madsen 354bbf2a32 Rename OMNITRACE_ROCM_SMI_DEVICES to OMNITRACE_SAMPLING_GPUS (#58)
- support ranges in OMNITRACE_SAMPLING_GPUS
2022-06-22 15:01:13 -05:00
Jonathan R. Madsen dfda902092 Improved the determination of MPI rank (#61)
* Improved the determination of MPI rank

* C-style cast of MPI_Comm
2022-06-21 00:27:52 -05:00
Jonathan R. Madsen 8eff363ed3 Remove reliance on MPI_Comm_rank (#56)
* Remove reliance on MPI_Comm_rank

- read /proc/<PID>/tasks/<PID>/children of parent process to deduce the rank
- Old format relied on user calling MPI_Comm_rank(MPI_COMM_WORLD, ...)
- if MPI_Comm_rank called with subcommunicators only, multiple ranks would write to same file

* Tweak mpi example
2022-06-20 00:50:49 -05:00
Jonathan R. Madsen f27f062e88 Fixes OMNITRACE_SUPPRESS_CONFIG handling (#53)
Fixes OMNITRACE_SUPPRESS_CONFIG

- now, if set to ON, the config file will not be read
2022-06-20 00:49:57 -05:00
Jonathan R. Madsen 0ef9a1531a Inclusive range for OMNITRACE_SAMPLING_CPUS (#54)
- previously, a value of 0-3 would be CPUs 0, 1, and 2
- now, a value of 0-3 would be CPUs 0, 1, 2, and 3
2022-06-20 00:46:55 -05:00
Jonathan R. Madsen 32a8d40d25 Define new function attributes (#55)
- decorates several functions with OMNITRACE_HOT
2022-06-20 00:44:31 -05:00
Jonathan R. Madsen a142b2029d Fix the main library stop routine for timemory (#39)
* Fix the main library stop routine for timemory

- the main pop_timemory function was popping too many calls
- this primarily affected recursive calls

* Lengthen the timeout for the Configure CMake step

* Fix python tests

- new validate-timemory-json.py script

* Documentation update

- Call-counts in timemory output examples in documentation were affected by the changes

* Fix the per-thread metrics during finalization

- pthread_create_mutex starts/stops the per-thread data
- removed unintentional continue statement

* Docs tweaks

* Fix lap counter on per-thread metrics
2022-06-13 15:57:44 -05:00
Jonathan R. Madsen ab395f86c4 Support omnitrace-dl as kokkos profile library (#37)
- add OnLoad and OnUnload to omnitrace-dl
- disable global fence for kokkos profiling tools
- tweak omnitrace_strip_target to use wildcards
- added dl-gen.py script for generating dlopen bindings
- added support for kokkosp_request_tool_settings
- added support for kokkosp_dual_view_sync
- added support for kokkosp_dual_view_modify
2022-06-10 18:54:22 -05:00
Jonathan R. Madsen d4b8e25765 export libomnitrace-dl.so to OMP_TOOL_LIBRARIES (#27)
* export libomnitrace-dl.so to OMP_TOOL_LIBRARIES

* Disable ompt_context_t

- ompt.cpp enables/disables context
- Update timemory submodule :: handles disabled context
2022-06-10 13:16:11 -05:00
Jonathan R. Madsen 90ab7a89fc Fix sampling counter time scales (#33)
* Fix sampling counter time scales

- All perfetto trace events have "begin_ns" and "end_ns" debug fields
- data for thread start and end timestamp in pthread_create_gotcha
- discard samples outside of thread start and end timestamps
- rename "CPU User CPU Time" perfetto counter to "CPU User Time"
- rename "CPU Kernel CPU Time" perfetto counter to "CPU Kernel Time"
- ensure CPU system samples in perfetto are set to zero at end
- backtrace uses comp::wall_clock record() for timestamps (consistency)
- "Peak Memory Usage [Thread X] (S)" renamed to "Thread Peak Memory Usage [X] (S)"
- "Context Switches [Thread X] (S)" renamed to "Thread Context Switches Usage [X] (S)"
- "Page Faults [Thread X] (S)" renamed to "Thread Page Faults Usage [X] (S)"
- "<PAPI_DESC> [Thread X] (S)" renamed to "Thread <PAPI_DESC> [X] (S)"
- samples

* Fix includes
2022-06-10 08:35:39 -05:00
Jonathan R. Madsen 1db3934f85 Rework sampling trace counter names + new trace counters (#30)
* Rework sampling trace counter names + new trace counters

- reformulate trace counter names for easier comparison (thread sampling)
- new process-level trace counters for context switches (thread sampling)
- new process-level trace counters for page faults (thread sampling)
- new process-level trace counters for CPU time (thread sampling)
- new thread-level trace counters for context switches (sampling)
- new thread-level trace counters for page faults (sampling)

* tweak header include in backtrace.cpp
2022-06-09 13:43:07 -05:00
Jonathan R. Madsen 3ca81fd8c0 Support strict settings option in timemory + expanded config syntax (#31)
Support strict settings option in timemory

- timemory settings updates
  - strict config option
  - improved variable support
    - $env: lvalues
    - support for ${VARIABLE} syntax
    - support for variable expansion in substring
  - chained config files
2022-06-08 16:58:06 -05:00
Jonathan R. Madsen 3031f173fd Fix perfetto_counter_track string lifetime (#28)
* Fix perfetto_counter_track string lifetime

- ensure the C-string pointer backing the perfetto::CounterTrack label is not still valid even after resizing container

* STL includes
2022-06-06 21:12:18 -05:00
Jonathan R. Madsen 424a3593e7 MPI headers + mutex gotcha + roctracer + kokkosp (#11)
* MPI headers, mutex gotcha + roctracer + kokkosp

- relocate internal MPI headers
- pthread_barrier in parallel-overhead
- doc fixes to DYNINST options
- minor tweaks to dynamic_library
- dlopen libamdhip64.so
- scoped thread state in kokkos
- extended pthread_mutex_gotcha

* Fix for unused-but-set-variables
2022-05-30 18:25:12 -05:00
Jonathan R. Madsen d5995846df Updates installation docs, cmake updates, internal OpenMPI header (#10)
* Updates installation docs + minor cmake tweaks

- OMNITRACE_BUILD_LIBUNWIND option
- Locally set OMNITRACE_USE_HIP=OFF if roctracer and rocm-smi are off
- Force TIMEMORY_BUILD_GOTCHA to avoid bug in gotcha not patched upstream

* MPI-Headers

- include copy of mpi.h from OpenMPI
- reworked FindMPI-Headers to support the internal OpenMPI headers
2022-05-26 10:03:31 -05:00
Jonathan R. Madsen 6af5b2a7e2 clang-tidy (#9)
- Fixed some clang-tidy warnings
- Fixed issue with omnitrace_launch_compiler + clang-tidy
2022-05-25 14:18:55 -05:00
Jonathan R. Madsen 353e8eeb69 Critical trace updates (#6)
* critical trace updates

- better handling of OMNITRACE_USE_PERFETTO in omnitrace-critical-trace exe
- changed some data types in `critical_trace::entry`
- added device ids to critical trace entries
- added process ids to critical trace entries
- added packing to critical trace entries

* Update timemory submodule
2022-05-24 19:25:54 -05:00
Jonathan R. Madsen 5b2c27cccd Minor updates for transpose, timemory submodule, roctracer, and omnitrace exe (#4)
* transpose usage message

* timemory submodule update

* roctracer updates

- Changes to verbosity of roctracer::shutdown
- protect_flush_activity prevents deadlock when error in callback

* Removed linking to timemory-cxx in omnitrace

- omnitrace exe does not link to `timemory-cxx` target
2022-05-24 18:35:33 -05:00
Jonathan R. Madsen 506c26cf82 Fixes for roctracer_callbacks PP regions (#59)
- define OMNITRACE_HIP_VERSION
- fix for ROCm < 4.3
- fix for PP blocks based on HIP version
2022-05-19 16:07:27 -05:00
Jonathan R. Madsen c2b206ba28 Timemory procfs utilities (#60)
- Serialize memory maps
 - Utilize tim::utility::procfs::cpuinfo::freq in cpu_freqs.cpp
2022-05-19 16:07:11 -05:00
Jonathan R. Madsen 8146426e8b Install perfetto tools option (#58)
* Install perfetto tools option

- E.g. traced, perfetto, etc.

* Fix copying of perfetto directory

* Require curl for installing perfetto tools

* Fix to locating tools/ninja
2022-05-11 15:05:09 -05:00
Jonathan R. Madsen 346f8cd0bc Option rename + minor fixes (#57)
- Set choices of OMNITRACE_BACKEND option
- rename OMNITRACE_SHMEM_SIZE_HINT_KB option
- rename OMNITRACE_BUFFER_SIZE_KB option
- rename OMNITRACE_COMBINE_PERFETTO_TRACES
- rename OMNITRACE_BACKEND option
- default to OMNITRACE_COLLAPSE_PROCESSES for combining perfetto traces
- OMNITRACE_PERFETTO_FILL_POLICY option
- fix unused variables due to constexpr in add_critical_trace
- rename perfetto config from "track_event" to "omnitrace"
- fix build-release.sh + python
- handle config file updating OMNITRACE_DL_VERBOSE in omnitrace-dl
- rename roctrace.cfg to omnitrace.cfg
- accept "on" and "off" for get_sampling_cpus()
2022-05-10 17:30:45 -05:00
Jonathan R. Madsen b208047741 Support for tracing mutex locking (#52)
* Parallel overhead example with locks

* Support tracing mutex locking + more

- support wrapping pthread_mutex_lock
- support wrapping pthread_mutex_unlock
- support wrapping pthread_mutex_trylock
- get_perfetto_combined_traces setting
- OMNITRACE_TRACE_THREAD_LOCKS option
- ThreadState
- critical trace includes queue id
- enabled/disabled settings in timemory
- fix OMNITRACE_TIMEMORY_COMPONENTS
- fix reading config
- fix setting categories
- applied ThreadState::Internal in various places
- utility::get_filled_array
- utility::get_reserved_vector
- utility::get_thread_index
- fork_gotcha messages about forks
- split out some pthread_gotcha functionality into pthread_create_gotcha
- handle queue id in roctracer callbacks

* Update timemory and PTL submodules

* Misc CMake updates

- Includes fix to omnitrace-static-lib{gcc,stdcxx}

* Misc cleanup to pthread_mutex_gotcha and backtrace

* Fix to duplicate field in module_function json

* Improvement to debug messages

* omnitrace-dl and common improvements

- tweak to delimit
- common::ignore message
- common::join quoting of strings
- omnitrace_set_env ignores if inited and active
- omnitrace_set_mpi ignores if inited and active

* nsync for transpose example

* Fix to thread_deleter<void> functor invoke

* Fix thread state and HIP stream enums
2022-05-08 04:40:10 -05:00
Jonathan R. Madsen 134b33320d Code coverage updates (#50)
* code coverage updates

- python support
- refactored source

* remove code_coverage::operator+ and operator+=

* impl/coverage.hpp
2022-05-08 01:40:56 -05:00
Jonathan R. Madsen 1f66e23fdd Reorganize source/lib/omnitrace (#51)
- Got rid of `source/lib/omnitrace/include` and `source/lib/omnitrace/src` and merged into `source/lib/omnitrace`
- Updated perfetto submodule to v25.0
- Updated papi submodule
2022-05-02 13:08:51 -05:00