* Fix for multi process workload profiling
Native counter collection tool updates:
* Do not dump empty counter data for a process
* Use PID instead of UUID for dumped csv files to facilitate correlation
* Handle merging multiple pairs of rocpd (from sdk tool) and csv (from
native tool) files
* Handle merging multiple pairs of csv (from sdk tool) and csv (from
native tool) files
Rocpd output format updates:
* Merge multiple rocpd databases into a single csv
* Reset dispatch id and kernel id for unique dispatches and unique
kernels respectively
* Retain multiple rocpd databases per run for multi process workloads
* Add test case for multiprocess profiling using rocflop workload
* Add rocflop
* Fix native counter csv to rocprofv3 csv conversion
* Use kernel_id instead of dispatch_id to correlate native counter csv
and kernel trace csv
* python formatting using ruff 0.14 instead of 0.13
* Improve Iteration multiplexing
* Improve iteration multiplexing documentation by adding usage note and
listing caveats
* Bugfixes for iteration mulitplexing
* Use merge iteration multiplexing in analysis webui and db mode
* Do not remove Dispatch_ID column in merge iteration multiplexing
since it is needed for analysis of top dispatches based on
duration
* Bugfixes for analysis logic
* Graceful handling of missing counters in case of iteration
multiplexing
* Improved warnings when metrics could not be calculated due to
missing counter data
* Fix the check to prevent showing table when a column is full of
N/A
* Improve detection of empty values when metric evaludation fails
due to missing counter data
* Bugfixes for profile logic
* Fix kernel filtering during roofline benchmark phase
* Update changelog for bugfixes
* Remove unnecessary columns when merging dispatches for iteration multiplexing
* bugfix
* Better analysis warnings
* fix to_std() in parser
* Use median in merge iteration multiplex
* Address review comments
* Fix cmake formatting
* fix None handling of parser util functions
* Enable stochastic counter accuracy test
* fix cmake formatting
* Use native tool for counter collection
* Add native counter collection tool which uses rocprofiler-sdk C++
library public API to get counter collection data
* This is enabled by default, unless --no-native-tool option is
provided or ROCPROF=rocprofv3 env. var. is provided
* This tool is only supported for ROCm version >=7.x.x
* This tool is not supported for attach/detach scenario
* Build native tool shared object during build time
* If using rocprof-compute without building then runtime compilation of
t push native tool shared object is performed
* rocprofiler-sdk tools is still used for services other than counter
collection and data collected by native tool is merged into the
rocpd/csv output of rocprofiler-sdk tool
* Make `rocpd` choice the default choice for `--format-rocprof-output`
option
* If `rocpd` public API from rocprofiler-sdk library is not present,
then fallback to `csv` choice
* In this case only `pmc_perf.csv` is written in workload folder
instead of multiple `csv` files for each profiling run
* Remove `json` choice from `--format-rocprof-output` option since it
functions identical to `csv` option
* Rename option `--rocprofiler-sdk-library-path` to
`--rocprofiler-sdk-tool-path` since we LD_PRELOAD the
rocprofiler-sdk tool shared object and not the rocprofiler-sdk library
shared object
* Fix the meaning of `--dispatch` option in `profile` mode to mention
dispatch iteration filtering instead of dispatch id filtering
* --dispatch option in analyze mode does dispatch id filtering
* Move standalone binary creation logic from cmake file to docker file
* fix native counter collection tool during attach/detach
* improve logging
* fix attach detach with native tool
* fix attach detach with native tool
* do not support attach/detach in native tool
* Update changelog
* add standalone binary creation functionality in cmake
* address review comments
* address review comments
* fix formatting
* address review comments
* Adding paths for cmake to search. Also updated min. cmake requirement to 3.21 as this was when hip was supported.
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* Update hip compiler ID check, sometimes comes up as Clang, sometimes ROCMClang- depends on setup.
Updated formatting.
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* RHEL8.10 unable to compile due to defaulting to old c++ version, need to force c++17
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* Updating changelog per docs team recommendations
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* Apply suggestions from code review to changelog
Co-authored-by: Pratik Basyal <pratik.basyal@amd.com>
* Do not required HIP complier to build native counter collection tool
* fix cmake
* gersemi formatting on latest cmake change
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* ex ci updated dependencies to include rocprofiler-sdk, but cmake was still not capturing the path- there was a commit that added to the cmake_prefix_path entry that specified rocprof-sdk's cmake location ut was too specific for the search paths in find_package's config mode.
removing the cmake_prefix_path var and adding hints to find_package call instead, and specifying config mode so it knows how to construct the search paths
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* gersemi run for formatting
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* Still need prefix path, should not have been removed in last commit but does need to be shortened to just the rocm path to allow for find_package config mode to do the job
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* include cstdint for uint32_t
* Run formatting on helper.cpp
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* Remove rocm 7.2 release stuff from version and changelog and handle it in separate pr
* fix version
* fix changelog
* fix changelog
* run ruff formatter
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* fix rocprofiler-sdk attach so path
---------
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
Co-authored-by: Carrie Fallows <Carrie.Fallows@amd.com>
Co-authored-by: Pratik Basyal <pratik.basyal@amd.com>
Force tencentos to use rhel-based bin since tencent is branched off of centos, which is branch of fedora.
Verified rocprof-compute run correctly selects bin to use, and the roofline benchmark values look similar between runs on rhel vs tencentos4 docker images on same system.
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* fix crash created by path and arg for pc_sampling and add copyright for mat_mul
* resolve fomat issue of line too long
* bugfixes
* copy gfx9 config template to analysis config in src
---------
Co-authored-by: Wang <ywang103@ctr2-alola-login-01.amd.com>
Co-authored-by: Vignesh Edithal <Vignesh.Edithal@amd.com>
* initial commit
* add csv support extraction for non kernel selection mode
* add --kernel-trace for rocprofiler-sdk mode
* make non kernel selective mode runnable
* make kernel selection work with -k
* remove upper case of arg hint
* update documentation
* display same kernel name at only one place and merge instruction id with same obj id as well as offset
* remove kernel name's display for single kernel selection
* change log added
---------
Co-authored-by: Fei Zheng <44449748+feizheng10@users.noreply.github.com>
* Upgrade min python version from 3.8 to 3.9
* Set min version for textual-fspicker for TUI support
* Update workflows to use python 3.9 instead of 3.8
* fix formatting
* fix bug
---------
Co-authored-by: Vignesh Edithal <Vignesh.Edithal@amd.com>
* Set default rocprof interface as rocprofiler-sdk
* Remove rocrprofv1 and rocprofv2 interfaces
* Remove deprecation notice for rocprof v1/v2/v3 interfaces
* Make rocprofiler-sdk the default interface and make rocprofv3 interface opt-in using ROCPROF=rocprofv3
* Add deprecation notice for rocprofv3
* Simplify the roofline binary pickup process by determining which base distribution the system OS is based off of, and select the correct binary.
* Add more OS distribution support to roofline by modifying the detection parameters and adding an AZL binary
* Update changelog to include roofline support additions
---------
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
Analysis data dump
* Add `--output-format` and `--output-name` option to analyze mode
* Remove `--output` and `-save-dfs` option to analyze mode
* Add documentation on `rocpd` output format and analysis database file
* Create sqlite3 database using object relation mapping (ORM) provided
by sqlalchemy library
* Fix metrics config to remove metrics marked as `null`, fix `Unit` header, add
missing `title`
* Add test cases to ensure analysis data dump work
* Use own counter definition
* Do not depend on rocprofiler-sdk counter definition
* Add missing counter definitions for MI100, MI200, MI300, MI350 series
* Counters added based on register specification
* This prevents some missing metrics
* Enable SQC_DCACHE_INFLIGHT_LEVEL counter and associated metrics
* Enable TCP_TCP_LATENCY counter and associated counter for all GPUs
except MI300
* Update TCC_EA_* counters for MI100 to TCC_EA0_*
* Update MI100 metrics which depend on TCC_EA0_* counters
* Enable accumulation counters for MI100
* Improve rocprof list avail usage to get a better idea of supported
counters
* Update CHANGELOG
* Move accumulation counters to counter definition
---------
Co-authored-by: Vignesh Edithal <Vignesh.Edithal@amd.com>
RHEL8 now being supported until EOS, rebuilding rhel8 bins for rocm7 (previously built for rocm7 rhel9).
Remove roofline-rhel9-rocm7, replace with new roofline-rhel8-rocm7.
Update check for roof bin.
Update any doc mentions of rhel min support version back to rhel8.
---------
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* Add `rocpd` choice for `--format-rocprof-output` option
* Add rocpd_data.py which defines SQL queries to extract data from rocpd database
* Use sqlite3 package to read the database
* Add `--retain-rocpd-output` option in profile mode to retain raw
rocpd database
* Add warning notice to say `--format-rocprof-output rocpd` will be
default in future release
For rocpd output:
* Use only `pmc_perf.csv` instead of reading individual coll_level results csv files
* Post process csv files using pandas in analysis mode instead of profile mode
* Use ACCUM counters instead of SQ_ACCUM_PREV_HIRES
* Add test cases for rocpd output format
* Fix code formatting issues
* Update CHANGELOG
[ROCm/rocprofiler-compute commit: 03d27c0ba0]
* Analysis report block based filtering is the default now
* Update documentation
* Update CHANGELOG
* Fix tests
* Replace hardware block based filtering tests with report block
based filtering tests
[ROCm/rocprofiler-compute commit: 98bb0f4237]
* Fix roofline rocm version bug
* Fix utils bug
* Remove unnecessary tests
* Do not check textual-fspicker package in cmake build
* Use rocprofv3 to test MI 100 and fix tests
[ROCm/rocprofiler-compute commit: 000fd4f5b2]
* Update current bins to have rocm6 suffix. Add new rocm7 bins, built on rocm7.0 latest due to hip updates.
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* Re-add rocm version check for roof bins.
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* Rebuild roofline binaries on top of latest rocm7 (#16379) after recent mainline promotions.
Adjusting version and distro combinations of bins following rocm6 vs rocm 7 supported OS.
*rhel8 not supported on rocm7, also not built anymore
*sles15 not supported on rocm7 but is still being built
*ubuntu stays as 22.04 and above for rocm7
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* Minor fixes after testing.
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* Fixed bad copy after finding it in testing ctest.
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* Remove runpath from new bin
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* Rework target_binary map return in detect_roofline- we should not be returning maps of different sizes or with different keys for the same method. Expected output should be consistent in case we run into bad position, or for testing purposes. Manually tested all possible roofline bin expected cases to comfirm functionality and expected user output.
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
* Update changelog with new roofline distro minimums
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
---------
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
[ROCm/rocprofiler-compute commit: 3df0d1dded]
Support for python3.8 and above means union defined in one of the methods was throwing errors for anything less than python3.10. Swapping out | operand for Optional[] resolves errors on systems using <3.10. No functional changes.
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
[ROCm/rocprofiler-compute commit: a59b1ea6e6]
* Add custom rocprofiler-sdk counter definitions file for MI 100
* Update CHANGELOG to mention that accumulation counters will not be
collected when profiling on MI 100 using rocprofiler-sdk/rocprofv3
* Migrate accum_counters.yaml to code
[ROCm/rocprofiler-compute commit: a95a45d69a]
* added progress printout for profiler
* added comments and fixed readability
* removed redundant newlines
* moved format_time helper function to utils
* removed tqdm and redundant time calc
[ROCm/rocprofiler-compute commit: 036866fc12]
* Revert of https://github.com/ROCm/rocprofiler-compute/pull/738
* Change default rocprof backend interface to rocprofv3
* Add MI 350 support in documentation
* Added known issue that MI 100 profiling will not work unless rocprofv1
is explicitly opted in
* Remove MI 50 soc gfx python class since MI 50 is not supported
[ROCm/rocprofiler-compute commit: d27ee69b52]
* Remove rocscope related code
* Add deprecation warning for database update mode which is used for grafana and mongodb functionality
[ROCm/rocprofiler-compute commit: cdd41dee40]
* Setting ROCPROF=rocprofiler-sdk environment variable will use rocprofiler-sdk C++ library instead of rocprofv3 python script
* Add runtime option --rocprofiler-sdk-library-path to use custom version of rocprofiler sdk library
* Add --rocprofiler-sdk-library-path conftest option for tests
* Setup appropriate environment variables to inject rocprofiler sdk code to user command
* Add env. vars. for counter collection and filtering
* Add env. vars. for pc sampling
* Use python bindings to list counters supported by rocprofiler sdk
[ROCm/rocprofiler-compute commit: 5cb86e31fc]
* Avoid crash when profiling data not generated
-Handle case where program has no kernel launches
-Improve error messages
-Avoid roofline when profiling data is missing
Signed-off-by: benrichard-amd <ben.richard@amd.com>
* Update other soc_gfx files to catch missing pmc_perf.csv
* Fix formatting
* Fix incorrectly ordered imports
---------
Signed-off-by: benrichard-amd <ben.richard@amd.com>
[ROCm/rocprofiler-compute commit: 35493f440c]
Rework of roofline binaries generated from rocm-amdgpu-bench
- removed arch identifier in bin name
- removed rocm5 bins altogether
Updated required distros for roofline
- updated distro checks and bin naming
- moved up ubuntu20.04->22.04 and sles15.3->15.6 per rocm support
Enabled ctests for mi350 for test_roof_*
- removed mi350 series check to skip these specific tests
---------
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
[ROCm/rocprofiler-compute commit: 41e73650d5]
* Add test for 9fx942 number of xcds.
* Improve the structure of mi gpu specs, add num_xcds_spec_class test.
* Add to ctest.
---------
Signed-off-by: xuchen-amd <xuchen@amd.com>
[ROCm/rocprofiler-compute commit: 85bfa73e2c]
vedithal-amd
Ben Richard
Kunal Malviya
abchoudh-amd
cfallows-amd
ywang103-amd
xuchen-amd
Fei Zheng
systems-assistant[bot]
jamessiddeley-amd
Vladimir Indic
Daniel Su