## Motivation
Enable UCX communication tracing and communication metadata
## Technical Details
Implement UCX API wrappers to trace transport-layer communication. This adds communication data tracking and exposes “UCX Comm Send/Recv” timelines, enabling detailed analysis of MPI, OpenSHMEM, and other UCX-based runtime communication patterns.
- Implements function interception for UCX functions across multiple categories using gotcha component.
- Extended comm_data component to track UCX send/recv operations - Added ucx_send and ucx_recv labels for Perfetto counter tracks. Integrated UCX data tracking with existing MPI/RCCL tracking infrastructure.
- Added ROCPROFSYS_USE_UCX configuration option (enabled by default).
- Created FindUCX.cmake module for UCX header detection. Falls back to internal UCX headers if system headers not found.
- Updated all Dockerfiles to include UCX dependencies.
## Motivation
<!-- Explain the purpose of this PR and the goals it aims to achieve. -->
- __Reduced Code Duplication__: Version parsing logic moved from individual Dockerfiles to the central build script
- __Improved Edge Case Handling__: Better handling of ROCm versions with and without patch numbers (e.g., `6.2` vs `6.2.0`)
- __Easier Maintenance__: Future version-related changes only need to be made in one place
- __Cleaner Dockerfiles__: Simplified Dockerfiles focus on package installation rather than complex shell logic
- __Updated Platform Support__: Refreshed container matrix to reflect current platform/ROCm version combinations
- __Fix OpenSUSE Docker Generation__: OpenSUSE container generation fails due to a change to the `binutils-gold` package
- __Error Handling__: Fix bug where errors in docker image build were being masked, allowing workflow to pass anyway.
## Technical Details
<!-- Explain the changes along with any relevant GitHub links. -->
- Updated `Dockerfile.opensuse` and `Dockerfile.opensuse.ci` docker files to remove `binutils-gold`
- Not needed since we build `binutils` with systems anyways
- Updated `rocprofiler-systems-containers.yml` to remove `pushd/popd` commands and just run the shell scripts
- There was a silent failure observed here, which I verified in this PR before adding the fix for openSUSE
- Refactor ROCm version parsing. Move this logic to the `build-docker.sh` script to reduce duplication.
- Fix bug that caused ROCm 7.0 to fail installation. The trailing `.0` was being trimmed.
- Fixed inconsistencies in `containers.yml` that lead to invalid ROCm-OS_VERSION combinations.
- Formatting fixes
- Removed trailing whitespace
- Fix docker build warnings. Use an `=` rather than ` ` when assigning an environment variable.
* Add GHCR retry logic
* Add retries to Install ROCm Packages step in rocprofiler-systems-redhat.yml
* Update containers-ci.yml file to use latest RHEL9/10 releases
* Use build-docker-ci script in rocprofiler-systems-containers
* Remove working-directory from step in rocprofiler-systems-redhat.yml
* Remove shell bash from Install ROCm Packages step
* Revert RHEL version change in rocprofiler-systems-redhat.yml
- Add `nlohmann-json-dev` (or equivalent) to CI Docker images for RHEL, SUSE, and Ubuntu.
- Add `gmock-dev` and `gtest-dev` (or equivalent) to CI Docker images for RHEL, SUSE, and Ubuntu.
- Add `--set solver classic` to conda config to resolve an issue setting up the conda environment
- Fix Perfetto package installation on ubuntu noble image.
- Add a check and log error if pip installation fail
---------
Co-authored-by: jbonnell-amd <jason.bonnell@amd.com>
- Rename the GHCR packages for rocprofiler Docker images to reduce the number of packages that will be released on the repository
- Changed package name to only include the OS instead of OS+Version - version moved to the tag instead.
- Updated Dockerfile.*.ci files to specify target ROCm version from tarball in name.
* Initial skeleton code for rocprofiler-systems-continuous-integration.yml
* Add python3-devel to opensuse and rhel ci images
* Update rocprofiler-systems-containers.yml to include TheRock tarballs
* Update pip install command for Dockerfile.ubuntu.ci
* Fix pip install again for Dockerfile.ubuntu.ci
* Remove skeleton workflow for CI
* Add new ci-gfx containers for TheRock installs
* Add set -e and pipefail to ci Dockerfiles to detect errors
* Upgrade pip in Dockerfile.ubuntu.ci
* revert pipefail set -e change
* Replace build-docker-ci.sh script with Docker step for ci-base
* Add support for gfx950, add containers-ci-gfx.yml
* Add working-directory to matrix setup steps
* Try changing containers-ci-gfx.yml
* make more changes to containers-ci-gfx.yml
* Remove build-docker-ci.sh script from gfx step, fix typo in Dockerfile
* Remove gfx110X and gfx120X for now
* Update ci-gfx docker workflow to use ghcr.io
* Temporary change to test one image
* Enable push to test out ghcr package
* Add labels to debug oauth issue
* add pacakages permissions to step
* add rocprofiler-systems-ghcr.yml workflow
* Remove cache from Docker push action step
* Add prefix to tag
* Add back gfx94X and gfx950 support, add back no push on PR
* Remove gfx container creation from rocprofiler-systems-containers.yml
* Add a gfx950 image for now
* Revert change
* Create CI dockers for debian 12
* Create Debian workflow
Signed-off-by: David Galiffi <David.Galiffi@amd.com>
* Fixing typo
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update default value for script's "VERSIONS" variable
* Fix Docker build warnings
LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format
* Refactored the check for `pip install --break-system-packages`
---------
Signed-off-by: David Galiffi <David.Galiffi@amd.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Use matrix data from containers.yml in rocprofiler-systems-containers.yml
* add on pull_request to rocprofiler-systems-containers.yml
* update yq command in rocprofiler-systems-containers.yml
* ensure no indenting in json output
* update output definition in prepare_release_matrix
* added container-ci.yml for ci containers
* move ordering of steps in rocprofiler-systems-containers.yml
* Update projects/rocprofiler-systems/docker/containers-ci.yml
Co-authored-by: David Galiffi <David.Galiffi@amd.com>
* only push containers when running on schedule or push
* Formatting fix
Cleaning trailing whitespace.
---------
Co-authored-by: David Galiffi <David.Galiffi@amd.com>
- matrix -m argument for build-docker.sh that lists compatible OS + ROCm combinations.
- ${DISTRO} is now case-insensitive.
- Added note to README.md to mention this flag.
- Removed --build-arg AMDGPU_RPM=${ROCM_RPM}, which is no longer used
[ROCm/rocprofiler-systems commit: 67bc147780]
- Bringing in recent changes from rocm-6.4 branch (https://github.com/ROCm/rocprofiler-systems/pull/171)
- Add libdrm-devel to rhel and suse files
- Update ROCm installation method in Ubuntu file
- Add additional output to `test-release.sh` to catch failures due to a Python version not included
- Add Python 3.13 to Dockers
[ROCm/rocprofiler-systems commit: 83a9eb3d7c]
* Update runners to `ubuntu-latest`.
The `ubuntu-20.04` runner is deprecated and will be removed.
* Add 'vim' and 'perfetto' to CI docker images
For convenience when using the images locally.
[ROCm/rocprofiler-systems commit: a25554359b]
* Integrating amd-smi into rocprofiler-systems due to rocm-smi deprecation.
* No functionality changes to users other than naming conventions.
* New tracks available in perfetto- gpu busy percentage metrics now splits gfx busy into separate gfx, umc, and mm engine measurements.
---------
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
Co-authored-by: David Galiffi <David.Galiffi@amd.com>
[ROCm/rocprofiler-systems commit: 0c32dfd6bc]
- Fixed type in script name to fix CTest. Should be: `run-rocprof-sys-pid.sh`
- Add `/usr/lib64/openmpi/bin` to Docker Image PATH to help ctests to discover `mpiexec`
- `pip install perfetto` in the CI Docker file, for convenience.
[ROCm/rocprofiler-systems commit: 4c4d0dea85]
- Renames the CMake option "ROCPROFSYS_USE_HIP" to "ROCPROFSYS_USE_ROCM"
- Remove the "ROCPROFSYS_USE_ROCM_SMI option. Controlled with the "ROCPROFSYS_USE_ROCM" option, instead.
- Runtime configuration can still toggle ROCPROFSYS_USE_ROCM_SMI to disable the sampling.
- Rename ROCPROFSYS_HIP_VERSION macro to ROCPROFSYS_ROCM_VERSION and remove blocks for `ROCPROFSYS_ROCM_VERSION < 60000`
- Remove ROCPROFSYS_USE_ROCTRACER and ROCPROFSYS_USE_ROCPROFILER
- Update test cases
- Update docker files and workflows to install cmake 3.21, which is required for the rocprofiler-sdk findPackage script.
- Removed rocm-6.2 from workflows due to a rocprofiler-sdk API change.
[ROCm/rocprofiler-systems commit: 88aa2d3cbe]
* Add installers for ubuntu 24.04
* Formatting change to the ubuntu-focal and ubuntu-jammy workflows
* Initial Ubuntu 24.04 workflow - just build test
[ROCm/rocprofiler-systems commit: 398ea62629]
* Update cmake version installed in dockerfiles
* Standardize the cmake_minimum_required to 3.18.4 across dockerfiles
* Fix link to perl repo in opensuse docker.
---------
Signed-off-by: David Galiffi <David.Galiffi@amd.com>
[ROCm/rocprofiler-systems commit: cef228bfbd]
Updated OS test matrix to match ROCm 6.2.
Update build and CI docker files
Remove the "docs" workflow, because "read-the-docs" is now being used for ROCm documentation
[ROCm/rocprofiler-systems commit: b15c9e94fc]
* Add ROCm 6.2 to CPACK workflow.
* Update APT-Get packages
- hipcc is no longer included in hip-runtime-amd meta packages. This change standardizes the installs with the other OSes.
* Bump version to 1.11.4
---------
Signed-off-by: David Galiffi <David.Galiffi@amd.com>
[ROCm/rocprofiler-systems commit: 6b0627f5b7]
* Add texinfo to Ubuntu and RHEL docker files
* Add the `bison` package to Ubuntu dockerfile
* Update the CI docker files too.
---------
Signed-off-by: David Galiffi <David.Galiffi@amd.com>
[ROCm/rocprofiler-systems commit: d0d97490b4]
* Add ROCm 6.1 to packages
* Bump version to 1.11.3
* Add 6.1 support to the docker build support.
Simplified this by adding 6.* to case statements, now that repo links have been standardized.
[ROCm/rocprofiler-systems commit: 99c9618c39]
* Update build-docker.sh
- support rocm 6.0
* Update cpack workflow
- support rocm 6.0
* Update CI testing workflow paths-ignore
- changes to {docs,cpack,containers,formatting}.yml and docker do not require testing
* Update docker for OpenSUSE
- always use --non-interactive with zypper
- tweak to PERL_REPO when OS version >= 15.4
[ROCm/rocprofiler-systems commit: cfaace38a8]
* Support ROCm 5.5 in docker
* Update containers workflow
- add Ubuntu and OpenSUSE container builds for ROCm 5.4 and 5.5
- add RHEL builds
* Update cpack workflow
- build on PR against main when cpack.yml or docker files updated
- removed packaging for ROCm < 5.2 for many OSes
- added packaging for ROCm 5.5
* Update OpenSUSE workflow
- add python 3.11 to OMNITRACE_PYTHON_ENVS
- upload-artifacts name includes strategy.job-index (prevent overwrite)
- only upload artifacts on failure
- continue on error if upload artifacts fails
* Update RedHat workflow
- provide run-name
- add python 3.11 to OMNITRACE_PYTHON_ENVS
- upload-artifacts name includes strategy.job-index (prevent overwrite)
- only upload artifacts on failure
- continue on error if upload artifacts fails
* Update Ubuntu (Bionic) workflow
- add python 3.11 to OMNITRACE_PYTHON_ENVS
- upload-artifacts name includes strategy.job-index (prevent overwrite)
- only upload artifacts on failure
- continue on error if upload artifacts fails
* Update Ubuntu (Focal) workflow
- add python 3.11 to OMNITRACE_PYTHON_ENVS
- upload-artifacts name includes strategy.job-index (prevent overwrite)
- only upload artifacts on failure
- continue on error if upload artifacts fails
- remove testing of ROCm 4.3, 5.0, 5.1
- add testing of ROCm 5.5
* Update Ubuntu (Jammy) workflow
- add python 3.11 to OMNITRACE_PYTHON_ENVS
- upload-artifacts name includes strategy.job-index (prevent overwrite)
- only upload artifacts on failure
- continue on error if upload artifacts fails
- add testing of ROCm latest
* Dockerfile.{rhel,opensuse} update
- remove use of amdgpu-install in favor of installing rocm-dev package
- In ROCm 5.5, amdgpu-install changed meaning of --usecase=rocm (added rocmdev use case)
* redhat workflow update
- remove use of amdgpu-install in favor of installing rocm-dev package
- In ROCm 5.5, amdgpu-install changed meaning of --usecase=rocm (added rocmdev use case)
* build-docker.sh update
- add '--progress plain' to docker build commands
* Ubuntu (jammy) workflow update
- fix rocm installation
* Update Dockerfile.rhel
- add LIBRARY_PATH for /opt/amdgpu/lib64 for redhat
* Update Dockerfile.rhel
- install libpciaccess for rocm
[ROCm/rocprofiler-systems commit: 693f753a9e]
Remove docker hiplibsdk from amdgpu-install
- amdgpu-install use case hiplibsdk is not necessary and bloats the install
- same as above for package rocm-hip-sdk
[ROCm/rocprofiler-systems commit: 2ce0cb4a19]
* Fixes for Python 3.11
* Add python 3.11 to scripts
- also tweak to to{upper,lower} bash functions
* Fix PAPI RPM packaging in RedHat
- fix error from #!/usr/bin/python in papi_hl_output_writer.py
- requires either python2 or python3 instead of python
* cpack updates
- only generate STGZ for RedHat
- support `--generators` arg in build-release.sh
- support 7z, zip, and other zip generators
- fix build-release.sh with `--mpi`
- support setting CONDA_ROOT
* Support rhel/fedora/centos in omnitrace-install.py
* RedHat status badge
* Fix support for Python 3.11 + tweak ubuntu ci
- Remove installing clang and mpich in Ubuntu CI container
- Fallback on conda-forge for Python 3.11
- Enable entrypoint-rhel.sh for RHEL CI
- Pull latest container by default
* Update ElfUtils and PAPI builds
- quieter build output
- disable-nls for ElfUtils
- use -s flag for make
* Development Guide Docs
[ROCm/rocprofiler-systems commit: 83f9ed8696]
- additional miscellaneous tweaks to workflows and docker scripts, e.g. install perfetto python bindings
- improves the stability of MPI finalization
- reduces some debug messages within timemory when `OMNITRACE_DEBUG=ON`
- fixes issue found in RHEL where libunwind is using mutex and omnitrace was not treating this as an internal mutex call
- this may have been affecting the causal profiling slightly (tests seem a bit more stable now)
- fix data race in timemory
* Add RedHat CI and release packaging
- additional miscellaneous tweaks to workflows and docker scripts, e.g. install perfetto python bindings
* Fix URL for ROCm packages in redhat workflow
* Fix dnf --enable-repo for ROCm perl packages
* Dockerfile.rhel and redhat.yml updates
- Fix dnf repo for ROCm PERL packages
- Disable python in CI (interpreter segfaults)
- Exclude parallel-overhead-locks tests due to inclusion of internal locks
- This needs to be remedied in the future
* Exclude _dl_relocate_static_pie from instrumentation
* Testing updates
- OMNITRACE_SAMPLING_KEEP_INTERNAL=OFF for parallel-overhead-locks
* Fix redhat workflow
* redhat.yml update
- remove if condition on config/build/test step
* Update timemory submodule
- tweaks to verbosity messages
* Set thread state before unw_step
- on Redhat, unw_step calls mutex
* Update timemory submodule
- verbosity changes
- gotcha uses spin_lock/spin_mutex
* Remove using gsplit-dwarf unless OMNITRACE_BUILD_NUMBER > 2
* Re-enable parallel-overhead-locks tests in redhat workflow
* Always disable timemory manager metadata auto output
* testing updates
- tweak parallel-overhead-locks-timemory to higher instruction count min
- OMNITRACE_SAMPLING_KEEP_INTERNAL=OFF for parallel-overhead-locks-perfetto
* Update timemory submodule
- quiet realpath queries
* omnitrace exe updates
- detect text files
- improved bin/lib locating
* cmake format
* test-install.sh and redhat workflow updates
- handle testing when ls is script
- re-enable python testing on redhat workflow
- invoke test-install.sh in redhat workflow
* Misc guards for finalization
* omnitrace-exe, testing updates
- test-install.sh: LS_EXEC -> LS_NAME
- handle /usr/bin/ls being script in source/bin/tests
- improve locating the binary
* Fix mpi_gotcha compile error
* omnitrace-exe updates
- improve file locating
* formatting
* Misc fixes
- remove -static-libstdc++ for RHEL packaging (rocky-linux doesn't distribute static lib)
* omnitrace-exe paths
* Replace realpath with absolute
- using absolute path to symlink fixes issues with locating libdyninstAPI_RT at runtime
* omnitrace exe updates
- judicious use of realpath
* Update timemory submodule
- fix update main hash ids/aliases data race in merge
* bin tests update
- change working directory of omnitrace-exe-simulate-lib-basename
* omnitrace exe updates
- Update resolved exe/lib messaging
* bin tests update
- change working directory of omnitrace-exe-simulate-lib-basename
[ROCm/rocprofiler-systems commit: 1688a027d8]
* library metadata/functions JSON
- remove always tagging metadata.json and functions.json with PID
* roctracer options for HIP API vs. HIP activity
* opensuse docker update for ROCm
- remove adding perl repo (does not exist)
[ROCm/rocprofiler-systems commit: bb591c82cb]
Sajina PK
Mario Limonciello
Jason Bonnell
Kian Cossettini
David Galiffi
Milan Radosavljevic
Pranjal Swarup
ajanicijamd
cfallows-amd
Jonathan R. Madsen