Remove the hipInitActivityCallback and use the new hipRegister/
RemoveActivityCallback which allows distinct memory pools to be used
for HIP_OPS activities.
Enable the multi_pool_activities test.
Change-Id: I6f6feaedecc9c36285bea975caf24dbf8f5f624b
The code is easier to read if calling HIPActivityCallbackTracker
enable/disable_check directly. Both enable/disable_check return the
new mask, and the check whether a callback is already installed is
clearer.
Change-Id: Ic90d34489b5b4d9929dc08b4d9e93cc974b136b1
The HIP runtime is now allocating the hip_api_data and record on its
stack so we don't need the thread local record_data_pair stack anymore.
Refactor the API callback function to handle both the case where
synchronous user callbacks are requested and the case where asynchronous
records are requested (enable_callback & enable_activity respectively).
If the callback argument (memory pool) is not null, then activity
records are requested.
Remove CorrelationIdRegister and CorrelationIdLookup. These were used
by the HIP runtime to associate a HIP record id to a ROCtracer
correlation id. Instead, the HIP runtime is now using the correlation
ID returned in the hip_api_data_t.
Added a test to check enabling/disabling concurrent callbacks and
activities.
Change-Id: I5850cfead9861eb3602a3e8fcb7b22580d5fc979
These functions have little value as it is very unlikely an application
would want to enable all the domains.
Change-Id: I4743e8ddf6743e60c95c7ba5240950d2ef734301
Disable automatic dependency detection when generating rocrtst RPMs.
This was adding unnecessary dependency on libhwloc, which is now
provided with the rocrtst package.
This matches behavior for DEB packages where there is no dependency
list for rocrtst.
Change-Id: If4a93f5b4c039b2f45e9445f60f65eefe84e32eb
Invalid peer links are labeled as N/A during topology report creation.
This invalid link type could be triggered by having a configuration
with CPU XGMI iolinks and disable XGMI peer to peer access. This can
be done by setting the driver parameter 'use_xgmi_p2p = 0'.
Signed-off-by: Alex Sierra <Alex.Sierra@amd.com>
Change-Id: Ifb09a8f3266a3f07686615dfb45781d6cfe55e83
Devices with CPU XGMI iolink do not support PCIe peer access. Therefore,
they should not be reported as accessible links in the topology.
Signed-off-by: Alex Sierra <Alex.Sierra@amd.com>
Change-Id: I3ee51796945dc0966200dee03886510e8f1846b7
Queue ctx_save_restore memory is allocated with size
ctx_save_restore_size + debug_memory_size, use the same size
in free_queue to free ctx_save_restore memory.
Change-Id: I4902ff15fb82ddea64b8342b89776a1bf5c38d13
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Avoiding segfault when an invalid SharedMemoryHandle is passed in
when calling fmm_register_shared_memory.
Change-Id: I0e0bbed01487fc10afcbb170eb9330e70b209d14
Signed-off-by: David Yat Sin <David.YatSin@amd.com>
The purpose of this patch is to modify the column header of the default
'./rocm-smi' command from 'Temp' to 'Temp (DieEdge)' for clarity.
Signed-off-by: Ori Messinger <Ori.Messinger@amd.com>
Change-Id: I127a9214be97a1185c3db010f1c9176d1f412ec9
- Sampling duration config values
- OMNITRACE_SAMPLING_DURATION
- OMNITRACE_PROCESS_SAMPLING_DURATION
- Disables sampling after this time (in seconds) has elapsed
- Sampling thread-id config values
- OMNITRACE_SAMPLING_TIDS
- OMNITRACE_SAMPLING_CPUTIME_TIDS
- OMNITRACE_SAMPLING_REALTIME_TIDS
- Allows user to select certain threads for sampling
- Miscellaneous
- Tweaked the finalization verbosity messages
- moved sampling-on-child-threads into runtime.hpp and runtime.cpp
- fixed submodule dyninst header install
## Overview
This is a significant PR which has 3 very notable characteristics:
1. Omnitrace colorizes most of it's logging
2. Completely reworked the sampling
- Samples now record the current instruction pointers instead of strings
- This _dramatically_ decreases the overhead of taking a sample
- The collection of metrics during a sample are split out into another component, enabling that data collection to be disabled -- which decreases the sampling overhead even further
- When both `OMNITRACE_SAMPLING_CPUTIME` and `OMNITRACE_SAMPLING_REALTIME` are ON:
- `OMNITRACE_SAMPLING_CPUTIME_FREQ` and `OMNITRACE_SAMPLING_REALTIME_FREQ` can be used to individually control the sampling frequency
- `OMNITRACE_SAMPLING_CPUTIME_DELAY` and `OMNITRACE_SAMPLING_REALTIME_DELAY` can be used to individually control the delay time before starting
- Now, omnitrace does not start a real-time sampler on the main thread unless `OMNITRACE_SAMPLING_REALTIME` is ON
- In the future, an `OMNITRACE_SAMPLING_TIDS` (and real-time, cpu-time variants) configuration variable(s) will allow you to select which threads will be sampled
3. Files produced by `omnitrace` exe -- `available-instr.txt`, `instrumented-instr.txt`, etc. -- now no longer has `-instr` suffix and are placed in `instrumentation/` subfolder, i.e. `available-instr.txt` -> instrumentation/available.txt`
- This helped de-clutter the output folder
Most of the other edits were reorganization (e.g. internal namespace changes), cleanup, and splitting up functionality.
## Bug Fixes
There is a bug fix with respect to the HSA callbacks which disabled sampling on child threads when an HSA API call was made
## Details
- created thread_info struct for mapping different thread IDs
- reorganized file structure significantly
- added categories.hpp, concepts.hpp
- moved around name trait definitions
- moved all omnitrace components into `omnitrace::component` namespace
- there was a lot of inconsistency b/t using `tim::component` in some places and `omnitrace::component`
- added macros like OMNITRACE_DECLARE_COMPONENT in lieu of TIMEMORY_DECLARE_COMPONENT
- OMNITRACE_CRITICAL_TRACE_NUM_THREADS -> OMNITRACE_THREAD_POOL_SIZE
- roctracer and critical_trace use same thread pool
- critical_trace functions do not lock anymore bc of thread-local TaskGroup
- added `component::local_category_region` to support using `component::category_region` without explicitly passing in name
- removed `component::omnitrace` (unused)
- migrated KokkosP and OMPT to use `component::local_category_region`
- removed `component::user_region` as a result
- migrated omnitrace_{push,pop}_{trace,region}_hidden to use component::category_region
- removed `component::functors` as a result
- migrated some ppdefs
- `api::omnitrace` -> `project::omnitrace`
- `api::(...)` -> `category::(...)`
- improved recording the execution time of threads
- migrated this functionality out of pthread_create_gotcha and into thread_info
- moved mpi_gotcha, fork_gotcha, exit_gotcha, rcclp into omnitrace::component namespace
- split backtrace up into backtrace, backtrace_metrics, backtrace_timestamp components
- sampling.cpp handles setup and post-processing that was formerly in backtrace
- updated logging to use colors
- `OMNITRACE_COLORIZED_LOG` config variable
- updated docs on JSON output from timemory
- instrumentation info in instrumentation subfolder
- added testing for KokkosP entries
- added testing for ompt entries
- add_critical_trace function defined in critical_trace.hpp
- disable push_thread_state and pop_thread_state when thread state is Disabled or Completed
- add comp::page_rss to main bundle
- thread_data supports std::optional instead of std::unique_ptr
- thread_data supports tim::identity<T> to avoid unique_ptr or optional
- tracing::record_thread_start_time()
- tracing::push_timemory and tracing::pop_timemory are templated on CategoryT
- removed anonymous namespace from omnitrace::utility
- sampling backtrace stores instruction pointers instead of strings
- component::category_region updates
- handle disabled thread state
- handle finalized state
- fewer debug messages
- invoke thread_init()
- invoke thread_init_sampling()
- handle push/pop count based on category
- push/pop count only modified when used
- component::cpu_freq
- components/ensure_storage.hpp
- reworked the pthread_create replacement function
- updated parallel-overhead example to report # of times locked
- OMNITRACE_MAX_UNWIND_DEPTH build option
- update timemory submodule
Now that HsaNodeProperties is passed in to
topology_get_node_props_from_drm, check that pointer instead of the
pointer for MarketingName (which throws a compiler warning)
Signed-off-by: kent.russell@amd.com <kent.russell@amd.com>
Change-Id: If76b24e1bab5a62e514ab440b6316c7b7cd264c1
This test checks that asynchronous activities can be enabled in distinct
memory pools. It enables activity reporting for HIP kernel dispatches in
one memory pool, and memory copy reporting in another memory pool.
The output of this test to stdout should be a series of kernel dispatch
records (10) followed by a series of memory copy records (10). The
records should not be interleaved.
Change-Id: Idb5cca7e650b2312a1955909932364f914737856
* Generic comm_data component
- moved rccl_comm_data to comm_data
- comm_data includes communication data for MPI
* fix timemory include with quotes
* Only support MPI comm data with full MPI support
* Increase timeouts + kill perfetto
* Update timemory submodule
* Fix missing command killall
* set +e in Kill Perfetto workflow step
* Updated MPI example to include MPI_Send and MPI_Recv calls
* Update timemory submodule with storage merge fix
* Perfetto comm data
- tracing::now<T>() function
* Fix timemory header include
Add agent info query HSA_AMD_AGENT_INFO_ASIC_FAMILY_ID.
Then we can remove the codes to parse family id.
Signed-off-by: Lang Yu <Lang.Yu@amd.com>
Change-Id: I3ac4746d3015e89b32322ebc0f8a3084f98677a4
Query family id info from drm render node, then
ROCr can query this info directly from Thunk
instead of parsing the info by itself.
Signed-off-by: Lang Yu <Lang.Yu@amd.com>
Change-Id: I030bd27ab2379fbf87f3d787302c3b8613456278
Required due to LLVM retirement of llvm::apply_tuple, instead using
std::apply which was introduced in C++17.
Signed-off-by: Graham Sider <Graham.Sider@amd.com>
Change-Id: I6646ebcca7d71d3e1bcf340ccfa3db2c15a3110a
The plugin's file scope global variables destructors could be called
before roctracer_plugin_finalize is called, making the global variables
undefined by the time roctracer_plugin_finalize is called.
To avoid this issue, remove all non-pod global variables from the file
plugin.
Change-Id: I4b620d67d460d9c99adfd81cbf46b0e64540c503
--showdeviceid
Fix for false-positive "FRU is corrupted" messages,
since str(sn).isalphanum() triggers on empty struct.
--showproductname
fix script termination on non-alphanum product name
Change-Id: I78d4998e156f9b0d9f45338bed2a0d30b789e220
Failure with new CWSR tests reported for GFX10, for now add to blacklist.
Signed-off-by: Graham Sider <Graham.Sider@amd.com>
Change-Id: I5b2bd9ec61c64ad66e1c34ba2c192bece808f56f
This reverts commit c904cc5856.
The change from using RUNPATH to RPATH was not approved formally.
Reverting this patch until this gets approved.
Change-Id: Ibc1a8f9d5dfa6694adacccfd9e3b0d053660e848
This patch restructures the CWSR basic test and allows for
creating parameterized CWSR tests. This patch introduces four
parameterizations. These tests behave as follows:
This test dispatches the IterateIsa shader, which continuously
increments a vgpr for (num_witems / WAVE_SIZE) waves. While this shader
is running, dequeue/requeue requests are sent in a loop to trigger
CWSRs.
This test defines a CWSR threshold. Once the number of CWSRs triggered
reaches the threshold, a known-value is filled into the inputBuf to
signal the shader to exit.
4 parameterized tests are defined:
KFDCWSRTest.BasicTest/0
KFDCWSRTest.BasicTest/1
KFDCWSRTest.BasicTest/2
KFDCWSRTest.BasicTest/3
0: 1 work-item, CWSR threshold of 10
1: 256 work-items, CWSR threshold of 50
2: 512 work-items, CWSR threshold of 100
3: 1024 work-items, CWSR threshold of 1000
Tuple Format: (num_witems, cwsr_thresh)
num_witems: Defines the number of work-items.
cwsr_thresh: Defines the number of CWSRs to trigger.
Signed-off-by: Graham Sider <Graham.Sider@amd.com>
Change-Id: I639eb7bd75b14ee70e190b4bd19dcf34096fc7bf
The debugger can now request snapshot copies with entry size and
set/clear watchpoints by device.
v3: drop min version check to v10.0
v2: check runtime allowance from v10.3 to 13.x
Signed-off-by: Jonathan Kim <jonathan.kim@amd.com>
Change-Id: I9befefb596201a11591de218db29a9317b41e69b
The allocation logic of the SPI does not take into account compute
user thread management settings for masking CUs with the exception of
skipping fully disabled SEs. This means that occupancy limited
dispatches such as cooperative launch may over allocate onto hardware
resources that are not immediately available, resulting in a potential
barrier logic hang as occupying work groups are waiting on enqueued
work groups to reach the barrier.
Further work will have to be done to get the per-SA CU enablement count
from the KFD in order to correctly clip the cooperative CU limit based
on the CU mask, which will require breaking the current ABI.
For now, report that cooperative launch is not supported while a CU
mask has been applied to prevent potential shader hangs.
Change-Id: I8be4bb47d65ceb62d805f36ef6ef3996d756021f
This function has been deprecated since ROCm-2.9, use ROCTX's
roctxMark(const char* message) as a replacement for roctracer_mark.
Change-Id: Ie4aeae1db238453fc4451746cc9a338032ba817f
New environment variable HSA_OVERRIDE_CPU_AFFINITY_DEBUG to
enable/disable overriding CPU affinity.
Default value is enabled(1).
This is a temporary variable and may be removed in the future.
Change-Id: Id6a7c611730471ddc276ca333fde1e57046bf32a
- Multithreaded Applications and plugin destruction
- Fixing Async-copy trace in file plugin
- Adding the assert checkups for every trace buffer flush function
Change-Id: I96e096fd7ee2604931200a0b446edb5ce49959dd
Don't set the color variables if tput is not available, not working, or
if ncolors < 8.
Move the color variables outside of eval to avoid calling tput over and
over again.
Change-Id: Id51a742b77ad0f7c99c1c7c5d05bed0f423b75de
* adding python-builtin-validation
* adding python-builtin-test check
* adding validation for external.py including and excluding inefficient
* validation for python-external(including & excluding inefficient) & python-builtin
* fixing label mixmatch for builtin and added only perfetto validation for external, no timemory validation for external
* python-validation-builtin
Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com>
Fix for regression in commit:
8a0fe6a832
When running rocrtstNeg.Queue_Validation_InvalidWorkGroupSize, each
time rocrtst::LoadKernelFromObjFile is called, a new CodeObject is
created and not deleted until end of the whole test. Each CodeObject
keeps an open file descriptor of the kernel file and this can exceed
maximum allowed open files on some systems. Deleting the CodeObjects
after each iteration in the test.
Change-Id: I388e56f95f7b671ecc29d5ecb4eb8ac2d0ddc412
`__pycache__` might be created when a python script is ran. Which
prevents `rpm -e` and `dpkg --remove` from completely removing the
application. This patch removes `__pycache__` early in the uninstall
process.
A similar issue is resolved in:
rocm_smi_lib Change-Id: I695bd085d4a43b678b563b4c35f6d2e8ddfa7d7c
Signed-off-by: Galantsev, Dmitrii <dmitrii.galantsev@amd.com>
Change-Id: I9fe0cd61570c2bd83cf9a45c95837ee6ad11e84b
I6520b51aac34060b5b90f94a016cec1827a4973f happens after uninstall, which
leaves a dangling directory under /opt/rocm/libexec/rocm_smi.
Removing __pycache__ before uninstall fixes the issue.
Signed-off-by: Galantsev, Dmitrii <dmitrii.galantsev@amd.com>
Change-Id: I695bd085d4a43b678b563b4c35f6d2e8ddfa7d7c
- Added File plugin as the default plugin
- Moved the flush functions to the plugins
- Improved the flush to file implementation
Change-Id: I80dd448eb8147a8ea4aa63b39bd1d0a4baf7252b
Laurent Morichetti
David Yat Sin
Alex Sierra
Philip Yang
Hao Zhou
Jonathan R. Madsen
Ori Messinger
jie1zhan
kent.russell@amd.com
Lang Yu
Graham Sider
Elena Sakhnovitch
Jonathan Kim
Ammar ELWazir
Mike Li
Tamima Rashid
Galantsev, Dmitrii