* Add kernel profiling time info to counter collection records
- lib/rocprofiler-sdk/kernel_dispatch
- added profiling_time.{hpp,cpp}
- restructured tracing.cpp
- updated queue.cpp AsyncSignalHandler
- gets kernel dispatch profiling time and passes to dispatch_complete and signal callbacks
- structured some header includes to reduce cyclic include probability
- originally, including kernel_dispatch/tracing.hpp in hsa/queue.hpp created a lot of cyclic includes
* Fix kernel_dispatch.cpp includes
* Fix kernel_dispatch.cpp
- include <cstring>
- replace use of ROCPROFILER_HSA_AMD_EXT_API_ID_NONE with ROCPROFILER_KERNEL_DISPATCH_LAST
[ROCm/rocprofiler-sdk commit: b15e498945]
* Add kernel profiling time info to counter collection records
- lib/rocprofiler-sdk/kernel_dispatch
- added profiling_time.{hpp,cpp}
- restructured tracing.cpp
- updated queue.cpp AsyncSignalHandler
- gets kernel dispatch profiling time and passes to dispatch_complete and signal callbacks
- structured some header includes to reduce cyclic include probability
- originally, including kernel_dispatch/tracing.hpp in hsa/queue.hpp created a lot of cyclic includes
* Fix kernel_dispatch.cpp includes
* Fix kernel_dispatch.cpp
- include <cstring>
- replace use of ROCPROFILER_HSA_AMD_EXT_API_ID_NONE with ROCPROFILER_KERNEL_DISPATCH_LAST
This fixes an issue for missing HW events when out of HW events.
We cannot determine whether a HW event has occurred unless we call the
underlying drivers with hsaKmtWaitOnMultipleEvents_Ext. Previous logic
in Signal::WaitAny would switch to ACTIVE_WAIT state if we run out of
hardware events (signal->EopEvent() == NULL) and this would cause the
hsaKmtWaitOnMultipleEvents_Ext call to be skipped. But also, when we
have some signals without hardware events, calling
hsaKmtWaitOnMultipleEvents_Ext with a timeout of 0 so that we can poll
for remaining signals adds overhead with an IOCTL call and may cause
extra delay. Separating AsyncEventLoop into two separate threads so
that:
1. We can have a new Signal::WaitAnyExceptions to wait for HW events
This function can be simpler as it does not have to perform all the
timer calculations because it is expected to be always waiting on
hsaKmtWaitOnMultipleEvents_Ext through the lifetime of a process.
2. Signal::WaitAny does not need to have extra code to check for HW
exceptions as it only needs to handle HSA_EVENTTYPE_SIGNAL events. It
can also skip the calls to hsaKmtWaitOnMultipleEvents_Ext if needed.
Change-Id: I52ba99fd6e483e0cb477b7931a0dcc03520aa523
Signed-off-by: David Yat Sin <David.YatSin@amd.com>
Delete queues used internally in agent destructor to make sure any
memory allocated by the queue objects are freed before the agent memory
regions are destroyed.
Change-Id: I4768c9cf66f77ac00a5a355f373f7f22dc266e47
If user application tries to free memory that is currently being used by
the underlying HW device, the hsaKmtFreeMemory function call will fail.
This would be caused by an incorrect call by the user application. A
system memory error is raised and the user application is expected to
abort when this happens.
Note: This leaves the allocation_map_ table in an inconsistent state as
this address entry is removed from it while the pointer is not actually
free'd. But re-organising the FreeMemory() function would require the
memory_lock_ to be held for much longer and may affect performance.
Since this is a very unlikely and invalid use case, we prefer to leave
the FreeMemory() function as is.
Change-Id: I24279eb98620c32d34f4c5ad1b7a0a30cb65835d
Signed-off-by: David Yat Sin <David.YatSin@amd.com>
Skip coredump generation when receiving HSA_STATUS_ERROR_MEMORY_FAULT.
We also receive a system error of type HSA_EVENTTYPE_MEMORY and generate
the coredump there. Trying to generate coredump from 2 places sometimes
causes unnecessary error message because both places try to create a
coredump file with the same name.
Change-Id: If3f03bab2c24ad71dfeff39ab411bb9ac08b337e
Signed-off-by: David Yat Sin <David.YatSin@amd.com>
This patch adds output (to stderr) to indicate step in the core dump
creation failed to improve debuggability.
Change-Id: I349692e278c2d744136d7fba7f7c2e5a7ada0c06
Signed-off-by: David Yat Sin <David.YatSin@amd.com>
It is possible for the runtime to receive an interrupt while trying to
access VRAM data using /proc/self/mem. In such case, pread(2) would
return -1 and set errno to -EINTR. This is not an error case, the
pread(2) call just need to be restarted, however current implementation
would tread it as an error.
This patch changes the the implementation to correctly retry on EINTR.
While at it, this patch also handles cases where pread(2) reads less
data than originally requested.
Change-Id: I6a72fc5eda4afd90319f0d24b35c9eac6d1ff41c
Signed-off-by: David Yat Sin <David.YatSin@amd.com>
Force mem_flags to be explicit passed in then calling Queue constructor
to avoid ambiguity with calls to Queue constructor trying to only pass
the agent_node_id.
Change-Id: Ib6fedcb9e52d6c9f35f9051dfa989343456ca368
Signed-off-by: David Yat Sin <David.YatSin@amd.com>
kmod dependency is not at all required for rocminfo. Removing the same from the package dependency list
Change-Id: I58f9c4305585c5dd770ea3c6a6298c30c89c31b5
* Tidying ATT dispatch API. ATT Agent to be initialized with rest of profiler. Removing read_index-based wait.
* Formatting
* Adding some input validation
* Add perf test for agent
* Removing async
[ROCm/rocprofiler-sdk commit: fa1b9e67ab]
* Tidying ATT dispatch API. ATT Agent to be initialized with rest of profiler. Removing read_index-based wait.
* Formatting
* Adding some input validation
* Add perf test for agent
* Removing async
In rocrtst helper_funcs.h, a function argument that gets
written to was previously incorrectly marked as const.
Change-Id: If8cc6555ebfa974b9665d9d5b93de01bb45fde2c
When source or destination pitch is set to zero in hip_Memcpy2D struct
it should default to WidthInBytes + [src/dst]XInBytes
Change-Id: Id57b53cab40ba72ced231258da9356554c4868c3
- Fixes -0.0 and +0.0 comparison. For atomicMax if the value on
address is -0.0 and on val is +0.0, gfx90a's unsafe atomics will swap
them. This behavior should be consistent with cas loop as well.
- _system variants of atomicMax and atomicMin are resulting in
incorrect output. Updated these to use the similar implementation as
atomicMax and atomicMin.
Change-Id: I20df36ee29ae0434a6b564f2ba71193fe41cfa59
This fixes issues like:
1.
/run/lock/rdcd.lock: Bad file descriptor
Failed to determine owner of lock file.: Numerical result out of range
2.
rdc.service: Failed to determine group credentials: No such process
rdc.service: Failed at step GROUP spawning rdcd: No such process
Change-Id: I0ef5eb6ab72d036a3ea8dcb81f7a9108d279f7d6
Signed-off-by: Galantsev, Dmitrii <dmitrii.galantsev@amd.com>
This fixes issues like:
1.
/run/lock/rdcd.lock: Bad file descriptor
Failed to determine owner of lock file.: Numerical result out of range
2.
rdc.service: Failed to determine group credentials: No such process
rdc.service: Failed at step GROUP spawning rdcd: No such process
Change-Id: I0ef5eb6ab72d036a3ea8dcb81f7a9108d279f7d6
Signed-off-by: Galantsev, Dmitrii <dmitrii.galantsev@amd.com>
[ROCm/rdc commit: c015f0fcaa]
Previously we relied on "render" to be enough and only used "video" as a
fallback. On some systems like SLES this might not be sufficient.
One issue happened when starting rocprofiler as part of RDC
initialization:
what(): hsa error code: 4104 HSA_STATUS_ERROR_OUT_OF_RESOURCES
The issue only happened when RDC was started with systemd.
Turns out "rdc" user (under which systemctl starts RDC) only had render
but not video group. Adding video group solved the issue.
Change-Id: Idf6a9521ae72a0b28a428869aa7ab1edde3ae259
Signed-off-by: Galantsev, Dmitrii <dmitrii.galantsev@amd.com>
Previously we relied on "render" to be enough and only used "video" as a
fallback. On some systems like SLES this might not be sufficient.
One issue happened when starting rocprofiler as part of RDC
initialization:
what(): hsa error code: 4104 HSA_STATUS_ERROR_OUT_OF_RESOURCES
The issue only happened when RDC was started with systemd.
Turns out "rdc" user (under which systemctl starts RDC) only had render
but not video group. Adding video group solved the issue.
Change-Id: Idf6a9521ae72a0b28a428869aa7ab1edde3ae259
Signed-off-by: Galantsev, Dmitrii <dmitrii.galantsev@amd.com>
[ROCm/rdc commit: 4ebc34095c]
Jonathan R. Madsen
David Yat Sin
Ioannis Assiouras
Shweta.Khatri
Lancelot Six
kjayapra-amd
Swati Rawat
Chris Freehill
Ranjith Ramakrishnan
James Xu
Julia Jiang
Joseph Macaranas
Giovanni Lenzi Baraldi
Rahul Manocha
Gopesh Bhardwaj
Vladana Stojiljkovic
Satyanvesh Dittakavi
Rakesh Roy
Galantsev, Dmitrii
SrirakshaNag