- For D2H cases avoid passing dependent signals to SDMA, the signals
take a while to resolve on SDMA engine
Change-Id: I569635228af977847f201c82ca897002f8f2f4a8
[ROCm/clr commit: 78d0ff2dbc]
- This change tries to save extra synchronization packets we may insert
as we didnt track the completion signals for every command. We track
the current enqueued command until it exits the enqueue stage. We also
record the exit scope to know if we flushed the caches
- Handle correct release scopes and store completion signal as HW events
- Use a new finishCommand implementation to only wait for the command
passed as the argument
Change-Id: Ie4350c5dd24f5d48dfa6ccbabd892f0544caadcc
[ROCm/clr commit: e03e4f3b5d]
- Use getBuffer/releaseBuffer in BlitManager
- Cleanup XferBuffer as we use ManagedBuffer for both reads/writes
Change-Id: I2661b85dd012763b17a38a743fec1b1d79125f67
[ROCm/clr commit: 37d606d193]
- If any kernel uses device heap, the launch needs to be preceeded by an
init kernel, Save on the extra barrier packet launch/flush between the
init heap kernel and user kernel
Change-Id: I8ebc6246188200e5f673dc464bc76a53bcb8b7c6
[ROCm/clr commit: ca530c660b]
- Fix regression for D2H pinned copies which adds systemscope release.
- Skip cpu wait for D2H unpinned copies as we can pass the signal of the
barrier to rocr copy.
- Fix an old bug in sdmaEngineRetainCount_ logic
- Improve logging
Change-Id: If074bddb05564b15949b0d5f9bf12acd3692174e
[ROCm/clr commit: 4c95ee5e1e]
- When using shader copy, make sure to use release scope for the AQL
packet. This is a potential bug but is hidden as hipMemcpyAsync always
needs synchronization(which inserts a barrier with release scope). For
hipMemcpy we use a barrier packet to make sure its blocking. Eitherways
a barrier gets always used and hides in some ways a potential bug.
Change-Id: I57fb7f769c3179e76d712471c0905104c801d7ba
[ROCm/clr commit: c9dd95bf6c]
- When we use blit(compute) copies, two subsequent copies may read for
the same source buffer, the buffer may get modified by the host in
between and if the src buffer was allocated with non-coherent flag, the
device may simply use stale value from previous cacheline fetch. This is
a corner case.
Change-Id: I2ce261c6f6fa4e5bb608f116548e5cc711ae6f3c
[ROCm/clr commit: b63005d550]
This reverts commit 0830d95f6d.
Reason for revert: There needs to be memcpy size change
Change-Id: If4f51769731e54743ac705b19b4f81b2d5925d5a
[ROCm/clr commit: 446ed661a0]
We are passing this arg as an address, and memcpy complains about
overreading (8 bytes instead of 4).
Change-Id: Ica9207f6c5f6056a4bfc968280c76e779ded13ae
[ROCm/clr commit: a6f2a2c2af]
- Refactor blit code and clean ASAN instrumentation
- Use unified function for rocr copy
- Enable shader copy path for unpinned writeBuffer/readBuffer paths
- Set GPU_FORCE_BLIT_COPY_SIZE=16 which means we will use BLIT copy for
pinned copies or unpinned H2D/D2H copies < 16KB
Change-Id: I42045cca79234b340dbf53dafb93044199736ae4
[ROCm/clr commit: 7863eb92dc]
Although unpinned copies require synchronizations
in HIP, runtime can avoid syncs for H2D copies with
a staging buffer
Change-Id: If2203c6bc0cbd89742823688dc8e89e9acd873b2
[ROCm/clr commit: 29cc678d8d]
If the graph has kernels that does device side allocation, during packet capture, heap is
allocated because heap pointer has to be added to the AQL packet, and initialized during
graph launch.
Handle race with wait when 2 kernels with device heap are enqueued on multiple streams.
Change-Id: I45933b77fcaf7bc8fdf1bc906462e32b5d8d3688
[ROCm/clr commit: 57156c524d]
This reverts commit 96e25898c8.
Reason for revert: SWDEV-455075, SWDEV-461507 - This change forces to
use ROCr's copy path. Reintroducing hostBlit copy path for
host-to-host copies.
Change-Id: Ic3c45b49e481c9dcdaa7611f61071778790b7e6c
[ROCm/clr commit: a9f05e22db]
If we are using the mask returned by getLastUsedSdmaEngine() then we
need to apply the SDMA Read/Write mask to it before using with HSA
copy_on_engine API.
Change-Id: I6e5dc6c187eeb3c61ee159e9d2a0fa7b4737c06e
[ROCm/clr commit: 3f0bcf7834]
The copies can get blocked if the last SDMA engine is used by another
copy and this can lead to perf drop in some of the tests like Gromacs.
Resetting the last engine by checking the engine status and fetching the
new mask after few copies can avoid this.
Change-Id: I8fe8ea678db508d291c6242f3741fa9215e99921
[ROCm/clr commit: 1b25484f0f]
__amd_streamOpsWrite blitkernel in device-libs has only 3 args.
so getting rid of the 4th unused arg (sizeBytes)
Change-Id: I81cc1107f8b424bf58558c93a2495a1b878aef91
[ROCm/clr commit: e643406caa]
Use only 16 workgroups for compute P2P copies.
That should be enough to utilize XGMI bandwidth.
Change-Id: I60dfe019279bb95f93c8874244c1738aad1896d8
[ROCm/clr commit: 31101c6219]
The new copy kernel can limit the number of launched workgoups.
It can copy in chunks of 16 bytes or 4 bytes.
Workgoup size is increased to 512 or 1024
Change-Id: Ic3fefa2d5bda6afebd1acc4d41ad310b138af6df
[ROCm/clr commit: ed4e1fec98]
- Add the new fillBuffer kernel, which allows to launch a limited
number of workgroups for memory fill operation
- Switch fill memory to 16 bytes write by default
- Allow to limit the workgroups with DEBUG_CLR_LIMIT_BLIT_WG
Change-Id: Ibad1822f2d42b2fc71bcfc1917c31409c0623e8e
[ROCm/clr commit: f1dc81f427]
Add hipMemcpyDeviceToDeviceNoCU to force a non blit copy path. This
helps in cases where an app may determine that CU may be busy and copies
with SDMA may be quicker.
Change-Id: I59b415dd8f6022c244e8d75f265464d5c635df1e
[ROCm/clr commit: f316a30e5d]
- Track last SDMA engine per queue, this results in better scheduling
- Reset last SDMA engine upon batch completion. That ensures we dont get
blocked if the same engine is used by another concurrent copy
Change-Id: Id53111980da7ee41d5c932fb44e4aab5b1e065a3
[ROCm/clr commit: bf8baeecb3]
- Rename HIP_USE_SDMA_QUERY to DEBUG_CLR_USE_SDMA_QUERY as this is
supposed to be a temporary env var for debug purposes only.
Change-Id: If6ebd52ab87624375a3df24ceccdcc05c60a65af
[ROCm/clr commit: 770b2a4711]
Blit manager requires an image view to reduce the amount
of copy kernels. Creation/destruction of a view in ROCr is
an expensive operation. Thus, runtime can cache views for fast access.
Change-Id: Ia67d775b481cc8326d91215ca22d4a73c1dddb59
[ROCm/clr commit: d29755452b]
- Remove large bar memcpy path. Since we end up waiting for a barrier,
its defeating the true intent of the copy, Also memcpy over PCIE\XGMI is
introducing variability in perf for HPC apps like GROMACS
Change-Id: I3b5c9d9ce93333959c39023bf4f703e2ccb6e3af
[ROCm/clr commit: 0a3d4bd4d4]
Saleel Kudchadker
Rahul Manocha
kjayapra-amd
Jimbo Xie
Sourabh Betigeri
Jatin Jaikishan Chaudhary
Anusha GodavarthySurya
German Andryeyev
Ioannis Assiouras
Satyanvesh Dittakavi
Ajay
Jaydeep Patel
victzhan