Graf commitů

224 Commity

Autor SHA1 Zpráva Datum
cfallows-amd 0c32dfd6bc Upgrade ROCm-SMI to AMD SMI (#86)
* Integrating amd-smi into rocprofiler-systems due to rocm-smi deprecation.
* No functionality changes to users other than naming conventions.
* New tracks available in perfetto- gpu busy percentage metrics now splits gfx busy into separate gfx, umc, and mm engine measurements.

---------

Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
Co-authored-by: David Galiffi <David.Galiffi@amd.com>
2025-01-30 21:32:07 -05:00
Peter Park 0a15d355e0 Update copyright year to 2025 (#83) 2025-01-29 16:53:16 -05:00
Maarten Arnst 043a8010a9 Update to KOKKOS_TOOLS_LIBS env var (#69) 2025-01-29 16:53:15 -05:00
Pranjal Swarup 0263e951ff Merge proto files from multiprocess run into one file. (#63)
- Added script to merge multiprocess output automatically to one file when there are multiprocess proto files written into output folder
- Execute the merge multiprocess script from the rank 0 process
- Added the scripts folder path to env path, via setup-env.sh
- Installed merge_multiprocess_output.sh to /share/rocprofiler-systems/bin dir

Co-authored-by: David Galiffi <David.Galiffi@amd.com>
2024-12-18 17:34:02 -05:00
Sajina PK 3fa37c991e Enable VCN tracing in Perfetto output (#65)
Enable VCN activity tracing on different instances from the GPU metrics fetched using rsmi_dev_gpu_metrics_info_get in the ROCm-SMI. library.

The tracing can be controlled with ROCPROFSYS_ROCM_SMI_METRICS by setting the value as vcn_activity, Currently this configuration takes the following values: busy, temp, power, mem_usage, vcn_activity.
By default, all the 5 values will be enabled.

Signed-off-by: Sajina P Kandy <Sajina.PuthalathKandy@amd.com>
Co-authored-by: Sajina Kandy <sputhala-amd@amd.com>
2024-12-18 15:56:48 -05:00
David Galiffi 88aa2d3cbe Update to use rocprofiler-sdk (#55)
- Renames the CMake option "ROCPROFSYS_USE_HIP" to "ROCPROFSYS_USE_ROCM"
- Remove the "ROCPROFSYS_USE_ROCM_SMI option. Controlled with the "ROCPROFSYS_USE_ROCM" option, instead.
   - Runtime configuration can still toggle ROCPROFSYS_USE_ROCM_SMI to disable the sampling.
- Rename ROCPROFSYS_HIP_VERSION macro to ROCPROFSYS_ROCM_VERSION and remove blocks for `ROCPROFSYS_ROCM_VERSION < 60000`
- Remove ROCPROFSYS_USE_ROCTRACER and ROCPROFSYS_USE_ROCPROFILER
- Update test cases
- Update docker files and workflows to install cmake 3.21, which is required for the rocprofiler-sdk findPackage script.
- Removed rocm-6.2 from workflows due to a rocprofiler-sdk API change.
2024-12-13 18:48:39 -05:00
David Galiffi 398ea62629 Adding installer for Ubuntu 24.04 (#14)
* Add installers for ubuntu 24.04

* Formatting change to the ubuntu-focal and ubuntu-jammy workflows

* Initial Ubuntu 24.04 workflow - just build test
2024-12-11 19:36:04 -05:00
Pran Swarup 51446f715f Fix GPU resource data of GPU power and temperature is not present on … (#23)
* Fix GPU resource data of GPU power and temperature is not present on MI300A traces
2024-11-13 15:02:43 -05:00
David Galiffi 7dce5926a7 OMPT Target Offload Support (#17)
- Porting from https://github.com/ROCm/omnitrace/pull/411
- Improve OMPT support
- Add OpenMP target example to testing
- Update Timemory submodule to use ROCm/Timemory rather than NERSC/Timemory
- Update `actions/upload-artifacts` to v4
- Standardize the `cmake_minimum_required` to 3.18.4 across workflows, project, and examples
- Updated Ubuntu 20.04 workflows
2024-11-07 16:49:32 -05:00
David Galiffi 21d9ab79fd Fix for proto files not being viewable in Perfetto UI (#16)
- Fix for proto files not being viewable in Perfetto UI
  - Ported from https://github.com/ROCm/omnitrace/pull/411

- Update Workflows

- Use V47 trace_processor_shell for certain OS releases.
  - RedHat 8, SUSE 15.5, and Ubuntu 20.04 are no longer compatible with the latest trace_processor_shell.
  - Incompatible version of GLIBC.

- Remove notes about Perfetto workaround in documentation.

---------

Signed-off-by: David Galiffi <David.Galiffi@amd.com>
2024-11-05 10:14:25 -05:00
David Galiffi 032d39f15c Update branding to ROCm Systems Profiler in documentation (#2)
* Update branding in docs

* Rename image used in documentation

* Update names of code samples.

In the code snippets, the "-" is not valid. ex., rocprof-sys_ --> rocprofsys_

* Update ASCII art

* update Doxyfile strip_from_path

* Add a "Formerly known as" message.

* Fixed typo in product name

ROCm Systems Profiler, not ROCm Profiler System

* Add "Omnitrace" back to the metadata keywords

* Update "install via package manager" section

* Update paths to user API files

* Rename configuration and environment settings

* Update Doxyfiles

Update publisher name & ID to "AMD".
Update bundle ID to "rocprofiler-systems"

* Update docs/what-is-rocprof-sys.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/conceptual/data-collection-modes.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/tutorials/video-tutorials.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/conceptual/rocprof-sys-feature-set.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/configuring-runtime-options.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/configuring-validating-environment.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/general-tips-using-rocprof-sys.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/reference/rocprof-sys-glossary.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/reference/development-guide.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/instrumenting-rewriting-binary-application.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/install/quick-start.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Note that videos were recorded using the "Omnitrace" name.

* Rebase and update some file paths

* Update paths to doc images

* Update Omnitrace references in code snippets

* Rename examples still using the "omni" prefix.

* Update docs/how-to/performing-causal-profiling.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/profiling-python-scripts.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/sampling-call-stack.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/understanding-rocprof-sys-output.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/install/install.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

---------

Co-authored-by: Peter Park <peter.park@amd.com>
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
2024-10-17 15:19:19 -04:00
David Galiffi d07bf508a9 Rename Omnitrace to ROCm Systems Profiler (#4)
The Omnitrace program is being renamed. 

Full name: "ROCm Systems Profiler"
Package name: "rocprofiler-systems"
Binary / Library names: "rocprof-sys-*"

---------
Co-authored-by: Xuan Chen <xuchen@amd.com>
Signed-off-by: David Galiffi <David.Galiffi@amd.com>
2024-10-15 11:20:40 -04:00
David Galiffi 6c28e49d11 Prepare for branch renaming. (#367)
* Add `amd-staging` and `amd-mainline` to workflow
* Update links in documentation to use amd-mainline

---------

Signed-off-by: David Galiffi <David.Galiffi@amd.com>
2024-09-26 12:21:31 -04:00
ajanicijamd 96d7b8f0ab Update Perfetto and fix tests (#378)
Fix for "SWDEV-479652" - Perfetto-based tests are failing.

Updated version of perfetto submodule to v46.0.
Modified Omnitrace code that uses Perfetto, so it can compile.
Modified the testing code, so it can run the version of trace_processor_shell provided (v46.0).

---------

Signed-off-by: Aleksandar Janicijevic <Aleksandar.Janicijevic@amd.com>
Co-authored-by: David Galiffi <David.Galiffi@amd.com>
2024-09-13 13:43:26 -04:00
jamesxu2 395c330dad Fix SWDEV-473314 by avoiding empty unique_ptr dereference (#371)
- Revert part of https://github.com/ROCm/omnitrace/pull/78/commits/b134a68110b2c96ce11293d93ad56f38e211fd06
	modifying source/lib/omnitrace/library/components/roctracer.cpp
2024-08-27 10:20:48 -04:00
Jonathan R. Madsen 18833a0a5e Executables append omnitrace library directory to LD_LIBRARY_PATH (#356)
- omnitrace-run, omnitrace-sample, and omnitrace-causal now automatically append the LD_LIBRARY_PATH with the directory containing the omnitrace libraries
  - this helps ensure that binary rewritten exes can resolve omnitrace-rt library location
2024-07-02 18:55:37 -05:00
Jonathan R. Madsen 0cf017251e Build omnitrace-rt library (#355)
* Build omnitrace-rt library

- Explicitly build dyninstAPI_RT as omnitrace-rt so that the SONAME in the ELF is omnitrace-rt instead of dyninstAPI_RT
- Create symbolic link lib/omnitrace/libdyninstAPI_RT.so which points to lib/libomnitrace-rt.so
- Simplify build tree location of libomnitrace-rt.so since it is ../lib from the bin directory even in the build tree
- Update dyninst submodule with minor tweaks to dyninstAPI_RT/CMakeLists.txt

* Update source/lib/omnitrace-rt/cmake/platform.cmake

* Use ftpmirror.gnu.org instead of ftp.gnu.org

- in timemory and dyninst submodules
- minor .clang-tidy tweak
2024-06-27 16:51:43 -05:00
Jonathan R. Madsen 219b2e988e Workflow, submodules, and thread info Updates (#352)
* Update CI workflows

- use node20 workflow packages

* Update tests/source/CMakeLists.txt

- Use OMNITRACE_TRACE and OMNTRACE_PROFILE instead of perfetto/timemory

* Update timemory submodule

- argparse: requires -> required
- parse callbacks

* Update thread_info.cpp

- fix causal::delay::get_local usage

* Update timemory submodule

* Update kokkos submodule

- release 3.7.02

* Revert opensuse.yml and ubuntu-bionic.yml to use node16 workflows

* Update docs.yml
2024-06-20 17:47:31 -05:00
Jonathan R. Madsen f8d52a6dbf Add release links to installation.md (#351) 2024-06-19 15:10:42 -05:00
David Galiffi 59e7b4b501 Check for an empty vector before popping (#350)
Protect from possible seg. fault

Signed-off-by: David Galiffi <David.Galiffi@amd.com>
2024-06-19 14:59:23 -04:00
David Galiffi f9ae806b78 Removing static version asserts. (#347)
It is causing failures on our internal builds

Signed-off-by: David Galiffi <David.Galiffi@amd.com>
2024-06-15 12:33:08 +05:30
ajanicijamd 74ed76529a Changed macros verbprintf and verbprintf_bare so they write to stdout… (#346)
Flush stdout when listing keys + bump verbose level for GPU count
2024-06-14 16:50:25 -04:00
ajanicijamd f0bd9126a5 Patch for omnitrace-sample crash with HIP API. (#344)
Fix HIP-API CTest failure

Check if stack is empty before popping data off of it.

Signed-off-by: Aleksandar Janicijevic <Aleksandar.Janicijevic@amd.com>
2024-06-07 15:23:18 -04:00
Jonathan R. Madsen 9499e2f521 Remove Critical Trace Support (#327)
* Delete core critical-trace files

* Update docs and README

* Update workflows

* Update testing

* Update cmake

* Remove critical trace usage in source code

* Update source/docs/critical_trace.md

- fix spelling

* Formatting

* Update bin/omnitrace-avail/avail.cpp

- statically allocate shared pointers for timemory manager and hash id/aliases to prevent use-after-free errors
2024-04-23 09:35:44 -05:00
David Galiffi b81db80926 Updated links to point to the ROCm organization. (#337)
Signed-off-by: David Galiffi <David.Galiffi@amd.com>
2024-04-23 01:44:02 -05:00
David Galiffi b1e5c356aa Fix link to ROCm install instructions (#333)
Now pointing to https://rocm.docs.amd.com/projects/install-on-linux/en/latest/.
2024-03-27 18:13:06 -05:00
Jonathan R. Madsen 15127c0d43 OMNITRACE_ROCM_SMI_METRICS (#331)
* OMNITRACE_ROCM_SMI_METRICS

- configuration variable OMNITRACE_ROCM_SMI_METRICS for specifying which rocm-smi metrics to collect
- auto-disable metric collection when rsmi_dev_X_get returns RSMI_STATUS_NOT_SUPPORTED

* Bump version to 1.11.1

* Python formatting

* Update python/libpyomnitrace.cpp

- fix usage of substr (ignored return value)

* Update python/gui/source/gui.py

- Fix E721
  - do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
2024-02-08 07:06:23 -06:00
Jonathan R. Madsen 77d52814e9 Fix omnitrace-avail component list (#328)
* Fix omnitrace-avail component list

- remove omnitrace components from `omnitrace-avail -C` since these are no-ops in OMNITRACE_TIMEMORY_COMPONENTS

* Fix omnitrace-avail-filter-wall-clock-available test
2024-01-10 20:00:46 -06:00
Jonathan R. Madsen edd6f57cf3 Fix thread-limit bug in roctracer (#326)
Update roctracer.cpp

- fix call to hip_exec_activity_callbacks when more runtime threads than compile time max
2024-01-10 19:10:45 -06:00
Jonathan R. Madsen adefde707c Fix cpack on SLES (#324)
Update lib/core/gpu.cpp

- use const std::array instead of constexpr std::array due to internal compiler errors on systems with older GCC compilers
2024-01-10 09:38:09 -06:00
Ben Richard 5de4163d66 Deprecate OMNITRACE_USE_PERFETTO, OMNITRACE_USE_TIMEMORY (#306)
* Rename OMNITRACE_USE_PERFETTO to OMNITRACE_TRACE

* Rename OMNITRACE_USE_TIMEMORY to OMNITRACE_PROFILE

* Revert change to Perfetto.cmake

* Fix formatting

clang-format-11 was complaining about formatting
2024-01-10 07:20:54 -06:00
Tal Ben-Nun 82cbe3f306 Add option to skip barrier marker events in traces (#320)
* Add option to skip barrier marker events in traces

* Formatting

* Apply review suggestions

Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com>

* clang-format

* Formatting

---------

Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com>
Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>
2024-01-10 07:17:32 -06:00
Jonathan R. Madsen 608287ddad HIP API backtraces (#323)
* Update lib/core/config.cpp

- Add OMNITRACE_ROCTRACER_HIP_API_BACKTRACE option

* Update lib/omnitrace/library/roctracer.cpp

- support perfetto debug annotation of backtrace in HIP API call

* Fix backtrace resolution and ordering in UI (#1)

* Fix backtrace resolution for non-omnitrace libraries

* Nicer Perfetto UI on long backtraces

* Make Perfetto annotation consistent

* clang-format

---------

Co-authored-by: Tal Ben-Nun <tbennun@users.noreply.github.com>
2024-01-10 06:15:32 -06:00
Jonathan R. Madsen 7bc50f5a0a Roctracer flush activity fix + perfetto.cfg (#317)
* Fix roctracer_flush_activity

- invoke roctracer_flush_activity() before disabling domains

* create comp::roctracer::flush()

- real issue was the global state when roctracer_flush_activity() was called

* formatting

* Update lib/omnitrace/library/components/roctracer.hpp

- provide definition of comp::roctracer::flush when OMNITRACE_USE_ROCTRACER is not defined

* omnitrace.cfg -> perfetto.cfg

- rename provided perfetto config file (omnitrace.cfg) to perfetto.cfg to avoid confusion

* Update lib/core

- gpu.hpp: defines for OMNITRACE_USE_{HIP,ROCTRACER,ROCPROFILER,ROCM_SMI}
- gpu.cpp
  - include core/hip_runtime.hpp
  - fix serialization of hipDeviceProp_t
- add hip_runtime.hpp
  -  ensure proper inclusion of hip_runtime.h
- add rccl.hpp
  - ensure proper inclusion of rccl.h

* Update lib/omnitrace/library

- rcclp.cpp
  - update includes for rccl
- roctracer.hpp
  - update includes for hip_runtime
- components/comm_data.hpp
  - update includes for rccl
- components/rcclp.hpp
  - update includes for rccl

* Update bin/omnitrace-avail/avail.cpp

- update includes for hip_runtime

* Update examples/rccl/CMakeLists.txt

- fix find_package for rccl when CI enabled

* Update CMakeLists.txt

- set cmake policy CMP0135 to NEW for cmake >= 3.24
  - Enable DOWNLOAD_EXTRACT_TIMESTAMP with ExternalProject_Add + URL download method

* Update timemory submodule

* Update pybind11 submodule

* Update pybind11 submodule

* Update lib/core/rccl.hpp

- include rccl.h only if OMNITRACE_USE_RCCL > 0

* Update lib/core/{gpu,hip_runtime}.hpp

* Update lib/core/gpu.cpp

- reintroduce some ppdefs

* Update lib/core/gpu.cpp

- fix ifdef on OMNITRACE_HIP_VERSION

* Update lib/core/gpu.cpp

- fix static assert for OMNITRACE_HIP_VERSION_MINOR when HIP version 4.x or older (unreliable minor versions)

* Update lib/core/gpu.cpp

- fix ifdef on OMNITRACE_HIP_VERSION

* Update lib/core/config.cpp

- disable OMNITRACE_PERFETTO_COMBINE_TRACES by default

* Update lib/core/perfetto.cpp

- if unable to open perfetto temp file, return the ReadTraceBlocking()

* Update lib/core/config.*

- flush tmpfile before closing
2024-01-10 05:02:22 -06:00
Jonathan R. Madsen 518c83e0f9 Dynamic expansion of thread data (#294)
* Tests for exceeding OMNITRACE_MAX_THREADS

- tests which exceeds OMNITRACE_MAX_THREADS value for thread creation

* CMake Formatting.cmake update

- include source files in /tests/source directory

* Add unknown-hash= to OMNITRACE_ABORT_FAIL_REGEX

- fail if a timemory hash is not resolved to a name

* Tests for exceeding OMNITRACE_MAX_THREADS

- update

* omnitrace-sample update

- remove env disabling of critical-trace and process-sampling

* core library update

- make_unique in concepts.hpp
- add OMNITRACE_USE_ROCM_SMI to "process_sampling" category
- remove forced disabling of critical-trace in sampling mode
- parentheses for OMNITRACE_PREFER
- use tim::get_hash_id instead of tim::get_combined_hash_id

* core library update (containers)

- added aligned_static_vector.hpp
  - similar to static_vector.hpp but attempts to align to cache line size
- alignment template parameter for stable_vector
- added missing aliases in static_vector
  - consistent with aligned_static_vector aliases

* thread_info update

- track the peak number of threads created
- thread_info::get_peak_num_threads() returns the peak number of threads

* thread_data update

- generic thread_data inherits from base_thread_data
- thread_data reworked to support dynamic expansion
- base_thread_data updated to invoke private_instance() function
- thread_data<optional<T>> uses stable_vector aligned to cache line width
- thread_data<identity<T>> uses stable_vector aligned to cache line width
- thread_data for optional and identity provide private private_instance function + friend to base_thread_data
- component_bundle_cache<T> is now thread_data<component_bundle_cache_impl<T>>

* causal update

- thread_data<T>::instances -> thread_data<T>::instance(construct_on_thread{ ... })
- loop over max_supported_threads (constexpr) -> loop over thread_info::get_peak_num_threads()
- tim::get_combined_hash_id -> tim::get_hash_id
- update progress_bundle usage to new thread_data API

* backtrace/backtrace_metrics component update

- backtrace_metrics update
  - update to new thead_data API
  - add thread CPU time row in perfetto
  - fix potential bug when rusage categories are disabled
  - fix bug in operator-= not subtracting cpu time of rhs
- backtrace update
  - skip all child call-stack below 'tim::openmp::' if sampling_keep_internal = false

* pthread_gotcha component update

- pthread_gotcha::shutdown() invokes pthread_create_gotcha::shutdown()

* pthread_create_gotcha component update

- minor tweak to {start,stop}_bundle functions: pass in thread id
- update to new thread_data API
- track native handles of internal threads
- implement system with pthread_kill to stop dangling bundles

* rocprofiler/roctracer component update

- update to new thread_data API
- loop over max_supported_threads (constexpr) -> loop over thread_info::get_peak_num_threads()

* critical trace (library) update

- update to new thread_data API
- tim::get_combined_hash_id -> tim::get_hash_id

* coverage update

- update to new thread_data API

* tasking update

- update to new thread_data API
- loop over max_supported_threads (constexpr) -> loop over thread_info::get_peak_num_threads()

* roctracer update

- update to new thread_data API
- loop over max_supported_threads (constexpr) -> loop over thread_info::get_peak_num_threads()

* rocm_smi update

- update to new thread_data API

* runtime.cpp update

- update to new thread_data API

* sampling.cpp update

- update to new thread_data API
- loop over max_supported_threads (constexpr) -> loop over thread_info::get_peak_num_threads()

* ompt.cpp update

- invoke pthread_gotcha::shutdown before invoking OMPT finalize function
  - this prevents signals from being delivered to OpenMP threads

* tracing.hpp and tracing.cpp update

- replace get_timemory_hash_{ids,aliases} functions with copy_timemory_hash_ids function
- update to new thread_data API
- loop over max_supported_threads (constexpr) -> loop over thread_info::get_peak_num_threads()
- tim::get_combined_hash_id -> tim::get_hash_id
- improvements to + error checking in thread_init function

* library.cpp update

- move copying timemory hash id/aliases to tracing.cpp
- update to new thread_data API
- loop over max_supported_threads (constexpr) -> loop over thread_info::get_peak_num_threads()

* Update BuildSettings.cmake

- add -Wno-interference-size to suppress warning about use of std::hardware_destructive_interference

* Update fork example

- improve scheme for waiting on child processes via waitpid instead of wait
- support running main routine multiple times
- push/pop regions in child process

* Update lib/common/defines.h.in

- allow use to specify misc values via -D <name>=<value>
  - OMNITRACE_CACHELINE_SIZE
  - OMNITRACE_CACHELINE_SIZE_MIN
  - OMNITRACE_ROCM_MAX_COUNTERS
- remove unused defines
  - OMNITRACE_ROCM_LOOK_AHEAD
  - OMNITRACE_MAX_ROCM_QUEUES

* Update rocprofiler.hpp

- OMNITRACE_MAX_ROCM_COUNTERS -> OMNITRACE_ROCM_MAX_COUNTERS

* Update aligned_static_vector

- set cacheline_align_v from max of OMNITRACE_CACHELINE_SIZE and OMNITRACE_CACHELINE_SIZE_MIN

* Update tracing.cpp

- acquire locks for updating main hash ids/aliases
- only propagate ids/aliases when finalizing

* Update pthread_create_gotcha.cpp

- make sure hash for "start_thread" exists on main thread

* Update causal end to end tests

- if OMNITRACE_BUILD_NUMBER is 1, set OMNITRACE_VERBOSE=0
2023-10-16 18:04:47 -05:00
Jonathan R. Madsen 227980f32b Fix roctracer data race (#309)
- roctracer_type_mutex was per-thread, causing lack of sync b/t callback and activity
2023-09-27 18:06:16 -05:00
Fei Zheng f88614dbcd A few fixes for the document (#302)
a few doc fixes
2023-09-27 14:43:06 -05:00
Jonathan R. Madsen 1216fd99a7 Reduce release packaging (#300)
* Bump version to 1.10.3

* Drop releases for ROCm < 5.3

- ROCm is no longer providing release for Ubuntu 18.04 starting with 5.3 so omnitrace is dropping support for Ubuntu 18.04 + ROCm
- Dropping ROCm 5.2 releases for Ubuntu 20.04
- Dropping ROCm 5.2 releases for OpenSUSE 15.4

* Update redhat workflow

- Test RedHat 9.1 + ROCm 5.5
- Test RedHat 9.1 + ROCm 5.6

* Update ubuntu-focal workflow

- drop ROCm 5.2 testing
- add ROCm 5.6 testing

* Update Findroctracer.cmake

- provide /opt/amdgpu to HINTS/PATHS for drm and drm_amdgpu libraries

* Update Findrocprofiler.cmake

- prefer librocprofiler64.so.1

* Update librocprofiler64.so to librocprofiler64.so.1

- search for the SOVERSION 1 library of librocprofiler64.so if ROCm > 5.5.0

* Update Findrocprofiler.cmake

- link to libpciaccess for ROCm 5.5.0

* Update redhat CI workflow

- install libpciaccess for rocm CI

* Update cpack workflow

- Remove all RHEL 9.0 packaging
- Remove all packaging for ROCm 5.3 on OSes supporting where releases are provided for 5.4, 5.5, and 5.6

* Update ubuntu focal workflow

- remove rocm 5.3 jobs
2023-09-12 20:19:26 -05:00
Jonathan R. Madsen 5276c957fb Sampling post-processing Perfetto fix (#298)
sampling post-processing perfetto fix

- avoid creating overflow sampling perfetto tracks when there is no data
- fix the parent region begin/end timestamps for the sampling tracks
2023-07-06 02:40:49 -05:00
Jonathan R. Madsen 6c9b66d938 PyTorch Python fork fix part 2 (#292)
PyTorch Python fork fix part 2

- store script file in environment for robustness against restart after fork
2023-06-22 17:14:40 -05:00
Jonathan R. Madsen a85f141afe PyTorch Python fork fix (#291)
* PyTorch Python fork fix

- fixes issue where forking process in PyTorch causes omnitrace/__main__.py to fail due to missing script argument

* Update source/python/omnitrace/__main__.py

Remove debugging "print" LOC
2023-06-21 22:30:47 -05:00
Jonathan R. Madsen 1aca8c177b Perfetto annotation from timemory components (#289)
* Annotate perfetto with timemory component data

- support perfetto annotations via timemory component data, e.g. use PAPI component for exact HW counter annotations

* Tests for perfetto annotation via timemory data

* Update omnitrace-instrument

- remove --default-components argument as this overrides any components set in configuration file
- required by perfetto annotation via timemory data tests

* filter unavailable timemory components

- filter out unavailable timemory components before attempting to invoke the annotate operation on the bundle

* update annotate tests

- account for no PAPI support

* update lulesh-timemory test

- replace '-d wall_clock peak_rss' with '--env OMNITRACE_TIMEMORY_COMPONENTS="wall_clock peak_rss"'

* annotate tests update

- fix misnamed test

* annotate tests update

- restrict binary rewrite to run function to force instrumentation despite heuristics

* annotate tests update

- print {available,overlapping,excluded,instrumented} functions during binary rewrite

* annotate tests update

- add allow-overlapping flag

* Support PAPI with CAP_SYS_ADMIN

- do not disable PAPI if perf_event_paranoid > 2 but has CAP_SYS_ADMIN capability
2023-06-19 19:18:04 -05:00
Jonathan R. Madsen 223536896b Fix rocprofiler usage in ROCm >= 5.5.x (#288)
Fix rocprofiler usage in ROCm >= 5.5.x

- starting with ROCm 5.5.0, rocprofiler throws exception if OnLoad + dlopen librocprofiler
- CI skipped for this PR since CI does not support GPU usage (tested locally)
2023-06-15 23:28:45 -05:00
Jonathan R. Madsen de9f0e4c10 Fix thread index values (#287)
* Update PTL

- PTL submodule waits for threads to start before proceeding

* Initialize perfetto after init_bundle

- perfetto thread creation after pthread_create wrapped

* backtrace component update

- exclude gotcha call-tree

* callchain component update

- callchain::get sorts based on timestamp
- callchain::sample supports duplicate IPs (recursion)

* Bump version to 1.10.1
2023-06-15 22:37:33 -05:00
Jonathan R. Madsen 262f1e9299 kokkosp update: disable deep copy tracing by default (#286)
kokkosp update

- disable tracing deep copies by default because inconsistent begin/end corrupts call-stack hierarchy
- tracing deep copy operations can be enabled via `OMNITRACE_KOKKOSP_DEEP_COPY`
2023-06-15 17:55:38 -05:00
Jonathan R. Madsen 3e2fa69a14 CI timeout + line-info in releases (#279)
* Update perfetto args.gn.in

- remove enable_perfetto_tools_trace_to_text (unused)

* core timeout implementation

- requires OMNITRACE_CI=ON
- requires OMNITRACE_CI_TIMEOUT=<sec>
- adds pthread_self and std::this_thread::get_id to thread info
- pthread_create_gotcha stores native handles (pthread_self)

* Testing updates

- improve detection of segfault/failures with PASS_REGEX exists
- add OMNITRACE_CI_TIMEOUT env variable to all tests

* Line-info in releases

- e.g. -g1 + more options to minimize size of debug info

* Fix typo in config exit action message

* OMNITRACE_UNLIKELY around debug/verbose messages

* format fixes

* Overflow tests + capability check

* transpose example update

- link to threads library

* roctracer/rocprofiler update

- in ROCm 5.5.0, cannot include rocprofiler.h and roctracer.h in same file due to conflicting enum defs
- Moved HSA tracing setup/shutdown to component::roctracer

* roctracer update

- fix definition of roctracer::setup when disabled

* Update fork example

- detach threads on main PID
- flush io outputs when printing info

* Update overflow tests

- pass regular expressions
- overflow on PERF_COUNT_SW_CPU_CLOCK event

* fork gotcha update

- use getpid() instead of getppid()

* update fork example

- wait on threads calling fork

* timeout update

- wait on timeout thread to launch before proceeding
2023-06-14 11:55:22 -05:00
Jonathan R. Madsen 9de3a6b0b4 Linux Perf Support + Causal Profiling Updates (#276)
* causal backtrace updates

- fix initial causal sampling period value

* causal delay updates

- tweak handling of sleep_for_overhead

* Fix experiment global scaling for prog pts

- results in drastically improved predictions

* pthread_mutex_gotcha updates

- disable all wrappers during causal profiling

* validate-causal-json.py updates

- support decimal stddev
- fix setting stddev from command-line

* causal perform_experiment_impl update

- handle start failing because finalizing

* deprecate causal::component::sample_rate

- appears to not help at all

* Rework sample info

* Increase causal unwind_depth

- use OMNITRACE_MAX_UNWIND_DEPTH

* validate-causal-json updates

- min experiments
  - exclude reporting predictions with less than X experiments at a given speedup
- percent samples
  - only print samples within X% of the peak (default: 95%)

* Update timemory submodule

- extensions to sampling for signals delivered via non-timer method
  - e.g. via HW counter overflow

* dwarf_entry::operator< updates

- sort via file

* causal profiling docs updates

- info about backends
- info about installing/enabling perf

* config updates: causal backend

- CausalBackend enum
- OMNITRACE_CAUSAL_BACKEND: perf, timer, auto
- omnitrace-causal option: --backend

* debug update

- use spin_mutex instead of std::mutex

* address_range::contains update

- range from 0-100 contains range from 10-100 but was returning false because high was == 100 not < 100

* symbol::operator< update

- handle load address differences

* sampling updates (non-causal)

- update get_timer to get_trigger + dynamic_cast

* container::static_vector updates

- support construction from container::c_array
- update_size private member func for handling atomic m_size

* Move perf files

- moved library/causal/perf.{hpp,cpp} to library/perf.{hpp,cpp}

* causal example update

- created impl.hpp (forward decls)
- renamed {cpu,rng}_func_impl to {cpu,rng}_impl_func
- only create two threads which run N iterations instead of two threads each iteration

* Update timemory submodule

- updates to unwind::processed_entry
- updates to procfs::maps

* Updated causal documentation

- fixed line numbers changed by modifications to causal example

* omnitrace-causal exe updates

- set OMNITRACE_THREAD_POOL_SIZE to zero by default

* core/containers updates

- static_vector: provide data() member function
- c_array pop_front() and pop_back() member functions

* core: config and argparse updates + perf

- core/perf.{hpp,cpp}
  - forward decl of enums
  - config-related capabilities
- argparse: --sample-overflow
- renamed some config functions
  - e.g. get_sampling_cpu_freq -> get_sampling_cputime_freq
- added config settings related to overflow sampling via perf
- added timer_sampling and overflow_sampling categories

* Update timemory submodule

- sampling allocator flushing

* binary updates

- lookup_ipaddr_entry
- use bfd_find_nearest_line instead of bfd_find_nearest_line_discriminator
  - discriminators are not used
- explicit instantiations of inlined_symbol::serialize

* Bump VERSION to 1.10.0

* sampling and perf updates

- support overflow sampling via Linux Perf
- update perf namespace
- update perf::perf_event
  - update record ctor: pointer instead of const ref
  - update open member func: return optional string
  - add m_batch_size member variable
- sampling updates
  - support overflow sampling
  - flush allocators
  - increase buffer size from 1024 to 2048
  - restructure post-processing in light of perf overflow supports
  - improve offload memory usage only load buffers for thread
  - load_offload_buffer(tid) uses thread-specific filepos
- component updates
  - backtrace_metrics::operator-=
  - backtrace_metrics::operator-
  - backtrace::sample does not record for overflow signal
  - callchain: perf overflow sample

* core updates

- component::sampling_percent does not report self + uses_percent_units

* causal updates

- tweak get_line_info
- overloads for set_current_selection (uint64_t, c_array, std::array)
- delay
  - use sampling::pause/sampling::resume
- experiment
  - experiment::sample derives from unwind::processed_entry
  - experiment::samples is vector instead of set
  - fixed samples
  - overloads for is_selected (uint64_t, c_array, std::array)
  - scaling factor defaults to 100 instead of 50
  - serialize updates follow change to experiment::sample
  - modify algorithm for increasing/decreasing experiment length
- sample_data
  - use map<uintptr, uint64_t> instead of set<sample_data>
  - get_samples returns vector<sample_data> instead of set<sample_data>
- sampling
  - support overflow via Linux Perf
  - update causal_offload_buffer
  - flush sampling allocator
- backtrace
  - overflow component

* libomnitrace-dl updates

- handle dl::InstrumentMode::PythonProfile

* testing updates (causal)

- causal line 155 -> causal line 100
- causal line 165 -> causal line 110

* formatting

* exit_gotcha updates

- exit_info for abort()
- message about non-zero exit code

* testing updates

- fail regex for causal tests
- validate-causal-json: >= min_experiments instead of > min_experiments
- handle OMNITRACE_DEBUG_SETTINGS in omnitrace_write_test_config

* causal sampling updates

- add new lines where appropriate

* causal data updates

- reorder diagnostic info when experiment fails to start

* binary updates

- symbol address range from address to address + symsize + 1
  - add 1 based on debug info

* causal data updates

- sample_selection wait_ns defaults to 1,000 instead of 10,000
- sample_selection wait scaled by iteration number
- save_line_info_impl verbosity
- print latest_eligible_pc when experiment does not start

* causal sampling + component updates

- perf backend disables component::backtrace
- ensure get_sampling_(realtime|cputime|overflow)_signal do not malloc

* causal: remove period stats

* validate-causal-json update

- fix --help

* causal data updates

- improve eligible pc history reporting when experiment fails to start

* causal data updates

- fix compute_eligible_lines_impl
  - eligible address ranges returning too many ranges
  - occasionally, overwrite all *true* eligible address ranges

* causal data updates

- reduce scoped ranges to symbol ranges
- is_eligible_address() returns true contains (not just coarse)
- revert some sample_selection behavior

* binary address_multirange updates

- make coarse_range private
- fix operator+=(pair<coarse, uintptr_t>)

* causal example update

- fix nsync to default to once per iteration

* binary analysis updates

- tweak header file includes

* causal updates

- remove factoring in sleep_for_overhead
- invoke delay::process() even if experiment is not active

* causal data updates

- update latest_eligible_pc structure

* update omnitrace-install.py.in

- fix support for fedora
  - /etc/os-release does not have ID_LIKE
  - fallback to RHEL 8.7 if version not specified

* update omnitrace-install.py.in

- fix support for debian
  - /etc/os-release does not have ID_LIKE
  - version mapping

* Update documentation

- update docs on installation

* causal data and experiment updates

- data: reset_sample_selection

* causal set_current_selection debugging

- debug messages for failed e2e runs

* causal data and backtrace component updates

- data: set_current_selection returns the number of eligible addresses added
- backtrace: if cputime signal has selected zero IPs > 5x, then realtime signal starts contributing call-stacks

* core library updates

- move config::parse_numeric_range to utility namespace
- add core/utility.cpp
- support range:increment, e.g. 5-25:10 expands to '5 15 25' instead of '5 10 15 20 25'

* omnitrace-causal update

- end-to-end expands all speedups
- support range:increment in speedups

* causal backtrace updates

- remove select_ival (realtime signal always contributes when select_count == 0)

* containers: static_vector update

- explicit c_array constructor
- explicit std::array constructor

* causal data updates

- remove set_current_selection(uint64_t)
- remove set_current_selection(std::array)
- sample_selection increase default wait time
- report eligible PC candidates
- move reset_sample_selection to perform_experiment_impl
- decrease latest_eligible_pc array size
- set_current_selection does not guard for experiment::active

* core debug updates

- OMNITRACE_PRINT_COLOR macros

* causal data updates

- tweak to experiment never started message

* causal gotcha updates

- remove unused code

* critical trace updates

- remove unused code

* omnitrace-causal

- OMNITRACE_LAUNCHER

* causal data updates

- don't fail on end-to-end + omnitrace-causal

* causal backtrace updates

- reintroduce select_ival behavior

* causal data updates

- tweak verbose messages about number of PC candidates

* core mproc updates

- utilities for waiting on child PID and diagnosing status
  - omnitrace::mproc::wait_pid
  - omnitrace::mproc::diagnose_status

* omnitrace-run updates

- support --fork argument for executing via fork in current process + execvpe on child instead of execvpe in current process

* omnitrace-causal updates

- wait_pid and diagnose_status just call equivalent functions in omnitrace::mproc

* ubuntu-focal workflow update

- attempt to launch ubuntu-focal-codecov job with CAP_SYS_ADMIN and use perf backend

* tests reorg and updates

- remove binary-rewrite-sampling and runtime-instrument-sampling tests
- rename *-preload tests (which use omnitrace-sample exe) to *-sampling
- split tests/CMakeLists.txt into several tests/omnitrace-<category>-tests.cmake files
- tweak to causal-both-omni-func test
  - add args: -n 2 -b timer

* update validate-causal-json.py

- better reasoning info for adjusting tolerance
- always apply tolerance adjustments in CI mode

* causal e2e tests update

- add label "causal-e2e" label
- tweak params
  - old: 80 12 432525 500000000
  - new: 80 50 432525 100000000
- disable processor affinity for slow-func/line-100 tests
  - artificially inflates some speedups with perf

* unblocking_gotcha updates

- overload operator() according to gotcha function index

* blocking_gotcha updates

- overload operator() according to gotcha function index
- fix bug where potentially post block functors (e.g. pthread_mutex_trylock) throw error if lock is not acquired.

* parse_numeric_range update

- support unordered_set

* config update

- OMNITRACE_DEBUG_{TIDS,PIDS} use parse_numeric_range
2023-04-13 02:14:35 -05:00
Jonathan R. Madsen cc14b52584 Casual Profiling GUI (#265)
* Long workload name layout fix (#269)

* changed layout to fit experiment names

- added span to show full name
- shortened name to fit dropwdown
- changed layout for added consistency

* layout Fixes

- refresh button is to the right
- header is more consistent across different width screens

* header layout update

- center div makes turns into multiple lines if not all items fit

* slight improvement for header/graph spacing

* Fixed refresh button shape and function

- moved find_causal_files to parser so that main and gui can access
- resized refresh width to allow for same shape across different screens

* all graphs now have the same width

- graphs now have same width

- chart headers start well below the header with filters

---------

Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com>

* Causal GUI: Linting, synced y-range, remove unused imports/variables, and bug fixes (#274)

* GUI: python linting workflow

- runs flake8 on code in source/python/gui

* GUI: flake8 settings in source/python/gui/setup.cfg

- ignore E501 errors (line too long)
- ignore W503 errors (line break before binary operator)

* GUI: setup.py updates

- remove unused imports

* GUI: __main__.py updates

- stddev is float value
- remove unused imports
- effectively propagate the --stddev argument

* GUI: gui.py updates

- remove unused imports
- fix light mode
- sync initial y range of all causal plots
- fix error bars for causal data
- set x-ticks to 5
- set y-ticks to 10
- only display top 99% of samples
- separate global declarations and assignments to global values
- remove unused variable assignments
- fix mislabeled function_regex and exp_regex
- change if X == False to if not X

* GUI: header.py updates

- remove unused imports
- fix mislabeled function_regex and exp_regex

* GUI: parser.py updates

- add set_num_stddev function for manipulating global num_stddev value
- remove unused variables
- fix latency point object (duplicated __init__ function)
- fix handling latency in JSON
- fix formatting of validation format error message
- replace if X == False with if not X
- fix unused dataframe creation in add_latency
- fix flake8 do not assign lambda for name_wo_ext (use def)

* GUI: gui.py updates

- replace misnamed "func_list" with "experiment_list"
- replace misnamed "exp_list" with "progpt_list"

* GUI: fix python workflow

- quote python versions to avoid truncating 3.10 to 3.1

---------

Co-authored-by: JoseSantosAMD <87447437+JoseSantosAMD@users.noreply.github.com>
2023-04-11 23:36:24 -05:00
Jonathan R. Madsen b39a683eab omnitrace-avail updates (#272)
* omnitrace-avail updates

- enables text wrapping for descriptions
- reworks the HW counters display layout
  - added new column "Device" which has either "CPU" or "GPU"
- support sorting HW counters alphabetically
- fixed some minor csv issues
- reorganize the order of the argparse arguments

* Fix tests
2023-03-30 15:06:29 -05:00
Jonathan R. Madsen ab8894082b rocprofler_iterate_info workaround v2 (#271)
Revert agent_info->dev_index in hsa_rsrc_factory.cpp

- disable using the driver_node_id and, instead, start at zero
  because it breaks the lookup for rocprofiler_pool_fetch
  lookup in rocprofiler.cpp. On my system (one AMD GPU and
  one NVIDIA GPU), it has a value of 1, not 0 and the pool
  size is 1 -- resulting in segfault
2023-03-30 11:15:19 -05:00