Add a state indicator to retain ExternalSignals when needed.
Co-operative group launch uses external signals to indicate a dependency
to the next command.
Change-Id: I6d0daa006e2377c3bbf4aeca0fd5b63c7ac8fbbb
[ROCm/clr commit: 1fbd75b825]
Crash was due to the fact that external signal structure was stale even
after destroyign the command. That is because we skipped wait due to a
missing check.
Detect external signals and dispatch a barrier in ReleaseGpuMemoryFence.
Also clear external_signals_ at ProfilingBegin.
Change-Id: I991387edcfe928b511bf5e780988ee131321ed5a
[ROCm/clr commit: 3239222516]
The optimization is controlled with ROCR_SKIP_KERNEL_ARG_COPY.
This is initial check-in for experiments. Extra changes are
necessary for full support:
- handle graph capture with the original sysmem alloc
- avoid memobject references, otherwise there is a race condition with
reusage of the arg buffer
- Remove arg setup from hip
Change-Id: Ib0af710f93e79834711fa4049a7c66093711e68b
[ROCm/clr commit: 7e12cf6318]
The kernel arg pool will be divided into 8 chunks to avoid long stalls,
when the pool will be reused.
Change-Id: I228e6ca1c09e428c1775f1e5b685220a9a5d71af
[ROCm/clr commit: f78b3a8919]
Reuse FillMemory function, that should fix the cache syncs from the host
Change-Id: Ieebec5fc3ed3a322b88d5187c8dca4805ec6f84b
[ROCm/clr commit: 24442be35a]
There is a possible race condition when signal reuse can have
access to a destroyed Timestamp object, because the callback
was running asynchronously. Use reference counter and lock
to allow asynchronous timestamp update
Change-Id: I6224f7c62cb0a03a7466fcc512e5e5afb06736fa
[ROCm/clr commit: ec89348291]
Add ref counting to ProfilingSignal class to track the last release.
If a signal was used in the marker, then don't reuse it,
but create a new one for internal usage.
Don't rely on HSA callback for the command status update if there
are no pending dispatches.
Change-Id: I19f14ed9d80acfe79993b343b2187635f8428a20
[ROCm/clr commit: ff15c0893e]
Runtime can't assign internal HSA signals for HIP events, because
HIP application can destroy the HIP stream or signal reuse may
occur internally. Switch to global HSA signals for HIP events.
Change-Id: Ieaea2d6b039e492b2e7c5112782a8f4e601e50a1
[ROCm/clr commit: ce8dad2ecc]
Add lock protection for signal processing
If signal is reused, then disable reference to it from HIP
Increase the pool signal size to 32
Change-Id: I7d529b35910f83ce577c9eca6d3386759611ccc0
[ROCm/clr commit: a1629cad26]
- Create an env var ROC_ACTIVE_WAIT_TIMEOUT to set active wait timeout
- Record profiling informaion if marker_ts_ property is valid.
Change-Id: If0d8aec8d9b0715027cf0f7c3dc8a4c722a6bae6
[ROCm/clr commit: b416ad7b9d]
With HIP API callback runtime has to stall the queue until the
callback is done. Rocclr will introduce SW blocking HSA signal,
which will be released after the callback is done.
Change-Id: I6411f3efab31b468e3b87ebb5c8d155e116b613d
[ROCm/clr commit: d93df7037c]
- Make sure barrier with dependent signals issues before queue
index reservation
- Don't issue extra barrier if it's already a barrier command
with dependent signals
Change-Id: I179a8b7adac79eed698f4a4d9eca2606d8e913aa
[ROCm/clr commit: 148a5cac39]
This addresses the rocprof hang seen with direct dispatch. The
workaround queues the handler back if any of the signal value in the batch
is not decremented. To rememmber the last position in the list, we save
the parsed command in the current timestamp struct.
Change-Id: I02959e463cfe3cee83c54808ffd6e6f48f43b4e8
[ROCm/clr commit: e5e635f9bf]
MT doesn't use GPU waits, but CPU for sync between engines.
Change the threshold values for CPU waits for direct dispatch.
That will bring behavior closer to MT.
Change-Id: Ia41c3cb812614962aff2746b6cf858f1bf77dda2
[ROCm/clr commit: ca2ea70a6c]
ROC_BARRIER_SYNC will not work with direct dispatch.
Remove and cleanup.
Change-Id: I81368b2e65039477bd0343bb92708dab48867db6
[ROCm/clr commit: aa38af8c96]
The copy image workaround could be recursively used by ROCclr blit kernel.
Avoid such situation by using stack variable.
Change-Id: Iadaa8cad9216220194760dd461a9533bb236aea0
[ROCm/clr commit: 2bd1836049]
- Don't notify if the batch is empty, because that means
the current command was processed already.
- Disable pinning optimization to avoid a race condition on stall.
- TS marker submition requires extra AQL barrier
to track the status.
Change-Id: I17eff4ad12ac66cfe1bb44048bebb1891805279d
[ROCm/clr commit: 24299e25bd]
GPU waits have noticeable overheads on compute with extra
AQL barrier packet and on SDMA with power saving features. This
change introduces a wait on CPU for 30 us in case the app has tiny
operations.
Change-Id: I761ba3af595f3f48544980058a9077dda15aa5f9
[ROCm/clr commit: ac387f9b03]
- Avoid GPU wait on the marker submission and update the command
batch after HSA signal callback upon HSA barrier completion.
Change-Id: I5c1c97212aefc2ae4b99aa9e2a81627ee9a38c1c
[ROCm/clr commit: 6966d8098e]
- The logic will trace compute, sdma read/write operations and
apply signals when necessary
- ROC_CPU_WAIT_FOR_SIGNAL, ROC_SYSTEM_SCOPE_SIGNAL
and ROC_SKIP_COPY_SYNC were added to control the tracking
Change-Id: I9e8e6174c63bf7784f7ab00964e2918c8667d364
[ROCm/clr commit: dbc7abaecf]
- ROCR fails the call for some reason, then the signal will
become invalid and can hang on a wait. The logic will reset the
active signal in such cases
Change-Id: Ia131420200f1bbd7c9a162b8f1b06db8cecf41c6
[ROCm/clr commit: ce2e5eba6b]
- There is a performance regression with a HW wait for HSA signal
on ROCr async operation. For now move the logic back to CPU wait.
- Fix profiling issue with multiple HSA signal per single timestamp
object. Some copies require multiple ROCR calls and if profiling is
required, then the execution time is derived from all used signals.
Change-Id: Id003e4abb8c2de378eedc152a7e389500fc6f4ce
[ROCm/clr commit: 5a8946190a]
Implement the global class for signals tracking per device queue.
Switch to the new tracking mechanism.
Change-Id: I3c4dda04b34e6d18d6a95510d84102909633b415
[ROCm/clr commit: 8698aeef0d]
Use barrier packets for every profile marker that gets submitted
and use the completion signal to get GPU ts. This gives most accurate
dispatch time. Club cache flushes with profile marker if there is a
pending dispatch that needs cache flush. This optimization saves on
extra barrier and helps wall time
Change-Id: Ib62d6d7aabf4743827b561be6c9c5afa813203da
[ROCm/clr commit: 59c6cb0268]
OCL can't distinguish different copy types, but ROC profiler
expects SDMA transfer visibility. Add extra code to detect
a transfer with the host memory and substitute OCL command
Change-Id: I5290acd0e10bc082e00c1d4ae1474a075de7f165
[ROCm/clr commit: bd340d8cbf]
Enable this optimization when the barrier is disabled, since
reuse requires a signal wait.
Use the size of pending AQL signals as the size of signal pool.
Change-Id: I2754a0f8b67e19d2601c58945e10fdf0e8be1624
[ROCm/clr commit: a5661192b6]
The change reuses HSA signals for dispatches as a wait signal.
Skipping the barrier requires to disable L2 cache for sysmem
allocations and extra tracking for HDP access with the large bar.
ROC_BARRIER_SYNC=0 activates the new logic. Barrier sync is
still used by default.
ROC_ACTIVE_WAIT=1 enables unconditional active wait in ROCr.
The change also consolidated ROCr wait logic under single function.
Change-Id: I6bd1be30aa88258da1b1f9de319ef5a45852afd8
[ROCm/clr commit: d9397590de]
The current implementation creates default reference in the stack and assigns it to class member cuMasks_, so whenever the content of the stack changes, cuMask_ would change.
Change-Id: Iefab63c335d504b83c4ae90bd34ae76c6afb8f3c
[ROCm/clr commit: 8ef5da00c7]
- Expose ROCclr interfaces for HIP usage
- ROCr interfaces aren't available in staging, thus control the
build with AMD_HMM_SUPPORT define
Change-Id: Iadc2bcc230e78d3b0dc22b235189c8cc80843446
[ROCm/clr commit: c5afd5d412]
SWDEV-234947
SWDEV-236298
Instead of forcing a barrier packet, just inject system scope on the next packet.
Change-Id: If9bcee23e08dfe5db731235e2fcb30582cbd4c1c
[ROCm/clr commit: 6a5af4056e]
Saleel Kudchadker
German Andryeyev
agunashe
pghafari
Alex Xie
Ravi C Akkenapally
Payam
Jason Tang
Aryan Salmanpour
Christophe Paquot