* Skipping AllReduce FP8 test on 9 to 16 ranks (gfx90a) as it's using Tree algorithm not RING
---------
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>
[ROCm/rccl commit: 5f691aaf65]
* Skipping AllReduce FP8 test on 9 to 16 ranks (gfx90a) as it's using Tree algorithm not RING
---------
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>
* Add opt-in to disable perfetto annotations
Add an env option `ROCPROFILER_DISABLE_PERFETTO_ANNOTATIONS`
to disable perfetto function-arg annotations.
If there are a large number of records, the tests that use this tool timeout
on some machines
* Update iteration kind
* Remove test_retired_correlation_ids for page-migration
[ROCm/rocprofiler-sdk commit: bbacf70ec7]
* Add opt-in to disable perfetto annotations
Add an env option `ROCPROFILER_DISABLE_PERFETTO_ANNOTATIONS`
to disable perfetto function-arg annotations.
If there are a large number of records, the tests that use this tool timeout
on some machines
* Update iteration kind
* Remove test_retired_correlation_ids for page-migration
Added debug log for when no flops are recorded (total_flops is 0), so AI points will not be plotted.
Removed commented out print statement that is not functional- contains nonexistent method call.
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
[ROCm/rocprofiler-compute commit: 1c237c1382]
Added debug log for when no flops are recorded (total_flops is 0), so AI points will not be plotted.
Removed commented out print statement that is not functional- contains nonexistent method call.
Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com>
- Remove hdp and ipc pointers from BlockHandle, align RO stats with RO contexts
- Add run commands for `rocshmem_g` and `rocshmem_p` API tests in driver.sh
- Allocate rocshmem API return buffers based on number of device contexts.
- Associate status flag address with blocking calls and remove threadId dependency
- Associated the status flag address with each blocking call request to notify the GPU thread.
- Removed dependency on threadId for determining the appropriate status flag index.
- Move status flag buffer allocation to backend.
- Initialize allocated memeory to zero
[ROCm/rocshmem commit: df4ad2c04d]
- Remove hdp and ipc pointers from BlockHandle, align RO stats with RO contexts
- Add run commands for `rocshmem_g` and `rocshmem_p` API tests in driver.sh
- Allocate rocshmem API return buffers based on number of device contexts.
- Associate status flag address with blocking calls and remove threadId dependency
- Associated the status flag address with each blocking call request to notify the GPU thread.
- Removed dependency on threadId for determining the appropriate status flag index.
- Move status flag buffer allocation to backend.
- Initialize allocated memeory to zero
There are many reasons why there may already be a CMAKE_MODULE_PATH defined.
The idiomatic way to extend it in a project is via list(APPEND).
https://github.com/ROCm/rocminfo/pull/97 submitted by @stellaraccident
Co-authored-by: Stella Laurenzo <Stella.Laurenzo@amd.com>
[ROCm/rocminfo commit: 8503ec36ca]
There are many reasons why there may already be a CMAKE_MODULE_PATH defined.
The idiomatic way to extend it in a project is via list(APPEND).
https://github.com/ROCm/rocminfo/pull/97 submitted by @stellaraccident
Co-authored-by: Stella Laurenzo <Stella.Laurenzo@amd.com>
* rocprofiler_stream_id_t: opaque handle for a stream
- e.g. HIP stream
- the same HIP stream may map to different HSA queues at different points in the application
- added to:
- rocprofiler_buffer_tracing_hip_api_record_t
- rocprofiler_buffer_tracing_memory_copy_record_t
- rocprofiler_callback_tracing_hip_api_data_t
- rocprofiler_callback_tracing_memory_copy_data_t
---------
Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>
Co-authored-by: Mark Meserve <mark.meserve@amd.com>
Co-authored-by: Elwazir, Ammar <Ammar.Elwazir@amd.com>
Co-authored-by: Ammar ELWazir <aelwazir@amd.com>
Co-authored-by: Jakaraddi, Manjunath <Manjunath.Jakaraddi@amd.com>
Co-authored-by: Bhardwaj, Gopesh <Gopesh.Bhardwaj@amd.com>
Co-authored-by: Nagaraj, Sriraksha <Sriraksha.Nagaraj@amd.com>
Co-authored-by: U, Srihari <Srihari.U@amd.com>
Co-authored-by: Madsen, Jonathan <Jonathan.Madsen@amd.com>
Co-authored-by: Welton, Benjamin <Benjamin.Welton@amd.com>
Co-authored-by: Benjamin Welton <ben@amd.com>
Co-authored-by: Indic, Vladimir <Vladimir.Indic@amd.com>
Co-authored-by: Benjamin Welton <bewelton@amd.com>
[ROCm/rocprofiler-sdk commit: ccd1e54293]
* rocprofiler_stream_id_t: opaque handle for a stream
- e.g. HIP stream
- the same HIP stream may map to different HSA queues at different points in the application
- added to:
- rocprofiler_buffer_tracing_hip_api_record_t
- rocprofiler_buffer_tracing_memory_copy_record_t
- rocprofiler_callback_tracing_hip_api_data_t
- rocprofiler_callback_tracing_memory_copy_data_t
---------
Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>
Co-authored-by: Mark Meserve <mark.meserve@amd.com>
Co-authored-by: Elwazir, Ammar <Ammar.Elwazir@amd.com>
Co-authored-by: Ammar ELWazir <aelwazir@amd.com>
Co-authored-by: Jakaraddi, Manjunath <Manjunath.Jakaraddi@amd.com>
Co-authored-by: Bhardwaj, Gopesh <Gopesh.Bhardwaj@amd.com>
Co-authored-by: Nagaraj, Sriraksha <Sriraksha.Nagaraj@amd.com>
Co-authored-by: U, Srihari <Srihari.U@amd.com>
Co-authored-by: Madsen, Jonathan <Jonathan.Madsen@amd.com>
Co-authored-by: Welton, Benjamin <Benjamin.Welton@amd.com>
Co-authored-by: Benjamin Welton <ben@amd.com>
Co-authored-by: Indic, Vladimir <Vladimir.Indic@amd.com>
Co-authored-by: Benjamin Welton <bewelton@amd.com>
Adds iteration based multiplexing to counter collection. Counter groups can now be specified. These counter groups are collected on a device individually until a specified interval period is reached. When the interval is reached, the next counter group is set to be collected on subsequent kernel executions.
Supplies two new argument types that can be included in YAML/JSON inputs:
pmc_groups: an array of arrays containing the counter groups to run (i.e. [ ["SQ_WAVES", "GRBM_COUNT"], ["GRBM_GUI_ACTIVE"])
pmc_group_interval: the number of kernel invocations on a GPU of a group before rotating to the next group
Note: originally there was a random_seed_generator proposed in the linked ticket, that was not implemented since there are very few instances where you would want the selection of the groups to be randomly generated (and if you do, you can randomly generate the pattern and place it as a large list of groups in pmc_group).
All existing counter functionality should be preserved (selection of counters on specific devices only, profiling of only specific kernels, etc).
---------
Co-authored-by: Benjamin Welton <bewelton@amd.com>
[ROCm/rocprofiler-sdk commit: aa88dd44c7]
Adds iteration based multiplexing to counter collection. Counter groups can now be specified. These counter groups are collected on a device individually until a specified interval period is reached. When the interval is reached, the next counter group is set to be collected on subsequent kernel executions.
Supplies two new argument types that can be included in YAML/JSON inputs:
pmc_groups: an array of arrays containing the counter groups to run (i.e. [ ["SQ_WAVES", "GRBM_COUNT"], ["GRBM_GUI_ACTIVE"])
pmc_group_interval: the number of kernel invocations on a GPU of a group before rotating to the next group
Note: originally there was a random_seed_generator proposed in the linked ticket, that was not implemented since there are very few instances where you would want the selection of the groups to be randomly generated (and if you do, you can randomly generate the pattern and place it as a large list of groups in pmc_group).
All existing counter functionality should be preserved (selection of counters on specific devices only, profiling of only specific kernels, etc).
---------
Co-authored-by: Benjamin Welton <bewelton@amd.com>
* [SWDEV-518071] Return HSA not loaded status (device counter collection)
This is a state that a caller would want to know about to understand if
they got no counters because of a failure or if they were trying to
collect counters too early (as is the case in the sample, which can
attempt to collect counters before HSA is inited).
* Minor edit
* format
* [SWDEV-518081] Simplify Metric Loading (#243)
* [SWDEV-518071] Return HSA not loaded status (device counter collection)
This is a state that a caller would want to know about to understand if
they got no counters because of a failure or if they were trying to
collect counters too early (as is the case in the sample, which can
attempt to collect counters before HSA is inited).
* [SWDEV-518324] Add AST update support
Allows the ability for ASTs to be updated (instead of an unchangable
static value). Adds a shared pointer return type to protect against
static destructors/modifications from invalidating potentially in use
AST definitions. No functionality/use changes in this PR.
* [SWDEV-518593] Add updatable dimension cache + fix string issues (#252)
* [SWDEV-518593] Add updatable dimension cache + fix string issues
Updates dimension cache to use the same design pattern as AST/Metrics.
Fixes the string scoping issue seen in ASTs, which appears here as well.
* Add rocprofiler_create_counter
Creates derived counters based on input from the API. This PR does three
things:
1. Adds the API + test case
2. Validates that an AST can be constructed from the counter supplied.
3. Updates metrics, ast, and dimension caches to include the new metric.
Metric should be available for use immediately after the call completes.
Due to the regeneration of ASTs, this call should not be performed in
performance sensitive code.
* Suggestion fixes
---------
Co-authored-by: Benjamin Welton <bewelton@amd.com>
* Minor tweak
---------
Co-authored-by: Benjamin Welton <bewelton@amd.com>
Co-authored-by: Venkateshwar Reddy Kandula <vkandula@amd.com>
---------
Co-authored-by: Benjamin Welton <bewelton@amd.com>
Co-authored-by: Venkateshwar Reddy Kandula <vkandula@amd.com>
* Fixes for comments
---------
Co-authored-by: Benjamin Welton <bewelton@amd.com>
Co-authored-by: Kandula, Venkateshwar reddy <Venkateshwarreddy.Kandula@amd.com>
Co-authored-by: Venkateshwar Reddy Kandula <vkandula@amd.com>
---------
Co-authored-by: Benjamin Welton <bewelton@amd.com>
Co-authored-by: Kandula, Venkateshwar reddy <Venkateshwarreddy.Kandula@amd.com>
Co-authored-by: Venkateshwar Reddy Kandula <vkandula@amd.com>
[ROCm/rocprofiler-sdk commit: 007285272b]
* [SWDEV-518071] Return HSA not loaded status (device counter collection)
This is a state that a caller would want to know about to understand if
they got no counters because of a failure or if they were trying to
collect counters too early (as is the case in the sample, which can
attempt to collect counters before HSA is inited).
* Minor edit
* format
* [SWDEV-518081] Simplify Metric Loading (#243)
* [SWDEV-518071] Return HSA not loaded status (device counter collection)
This is a state that a caller would want to know about to understand if
they got no counters because of a failure or if they were trying to
collect counters too early (as is the case in the sample, which can
attempt to collect counters before HSA is inited).
* [SWDEV-518324] Add AST update support
Allows the ability for ASTs to be updated (instead of an unchangable
static value). Adds a shared pointer return type to protect against
static destructors/modifications from invalidating potentially in use
AST definitions. No functionality/use changes in this PR.
* [SWDEV-518593] Add updatable dimension cache + fix string issues (#252)
* [SWDEV-518593] Add updatable dimension cache + fix string issues
Updates dimension cache to use the same design pattern as AST/Metrics.
Fixes the string scoping issue seen in ASTs, which appears here as well.
* Add rocprofiler_create_counter
Creates derived counters based on input from the API. This PR does three
things:
1. Adds the API + test case
2. Validates that an AST can be constructed from the counter supplied.
3. Updates metrics, ast, and dimension caches to include the new metric.
Metric should be available for use immediately after the call completes.
Due to the regeneration of ASTs, this call should not be performed in
performance sensitive code.
* Suggestion fixes
---------
Co-authored-by: Benjamin Welton <bewelton@amd.com>
* Minor tweak
---------
Co-authored-by: Benjamin Welton <bewelton@amd.com>
Co-authored-by: Venkateshwar Reddy Kandula <vkandula@amd.com>
---------
Co-authored-by: Benjamin Welton <bewelton@amd.com>
Co-authored-by: Venkateshwar Reddy Kandula <vkandula@amd.com>
* Fixes for comments
---------
Co-authored-by: Benjamin Welton <bewelton@amd.com>
Co-authored-by: Kandula, Venkateshwar reddy <Venkateshwarreddy.Kandula@amd.com>
Co-authored-by: Venkateshwar Reddy Kandula <vkandula@amd.com>
---------
Co-authored-by: Benjamin Welton <bewelton@amd.com>
Co-authored-by: Kandula, Venkateshwar reddy <Venkateshwarreddy.Kandula@amd.com>
Co-authored-by: Venkateshwar Reddy Kandula <vkandula@amd.com>
Yiltan
mberenjk
Mustafa Abduljabbar
Zhang, Ava
Wenkai Du
Baraldi, Giovanni
Nilesh M Negi
Mallya, Ameya Keshava
Kuricheti, Mythreya
cfallows-amd
Avinash Kethineedi
Khatri, Shweta
Stella Laurenzo
Park, Peter
Trowbridge, Ian
Welton, Benjamin
Nagaraj, Sriraksha