This patch fixes this potential issue that filling AQL header before
filling the AQL body. The hsa spec specifies "Packet processors may
process AQL packets after the packet format field is updated, but
before the doorbell is signaled."
However, the hipGraph AQL package with valid header will be filled
before fill the body, which may have the potential issue that CP
receive invalid AQL body.
Change-Id: I84af798c19ee2b8805ba19732b0eabdea2958a96
[ROCm/clr commit: 3959b5be1e]
- Added the optimized multi stream path in graph execution. It uses a fixed number of async streams in the execution
- Optimize the launch latency, where commands
creation and execution is done at the same time
- Optimize the scheduling to use less barriers and waiting signals if
the same queue can be detected
- The new path is controlled by DEBUG_HIP_FORCE_GRAPH_QUEUES
environment variable, where 0 will use the original path and any other
value will force the number of asynchronous queues for execution
- DEBUG_HIP_FORCE_ASYNC_QUEUE can force single queue async
execution in graphs(applicable for Navi families only)
Change-Id: I7eb40bc15c45f508d6911868a6f6d4c3598d380e
[ROCm/clr commit: 9db52f9a46]
=> Added support to capture multiple AQL Packets.
=> Added Interface to callback to hip runtime from rocclr to allocate
kernel args from the graph kernel arg pool.
=> Enabled Support to capture memset node.
Change-Id: I7e1c2ba06927459e024653058af142bd82192c43
[ROCm/clr commit: bd3a35bde1]
If only external signals were provided, then just process it
without adding internal signals
Change-Id: Iaefd65d0f8b0a64b9f6a864a9bd73de20a29dfa4
[ROCm/clr commit: 18187cd8fe]
Capture AQL packets during GraphInstantiation and enqueue AQL packets during graph launch.
Added support to capture single graph memset node.
Capture support for memset node is currently disabled.
Memset capture will be enabled when capture for multiple packets are supported..
Change-Id: I14dfbc41731025cc3a548a730558915def3fa384
[ROCm/clr commit: 346da4bb40]
This change modifies the readback mechanism to use a pointer to volatile
instead of a volatile pointer. This ensures that the compiler does not
optimize away the read operation.
Change-Id: I79ff925d615aa8cc4f950e8ff4b7e608fcb179a4
[ROCm/clr commit: ea50d2c0c2]
If the graph has kernels that does device side allocation, during packet capture, heap is
allocated because heap pointer has to be added to the AQL packet, and initialized during
graph launch.
Handle race with wait when 2 kernels with device heap are enqueued on multiple streams.
Change-Id: I45933b77fcaf7bc8fdf1bc906462e32b5d8d3688
[ROCm/clr commit: 57156c524d]
Adding a safety check prevents an invalid memory access
if timestamps and kernelNames vectors are of different size.
The patch also moves the addKernelNames for the accumulate command
into dispatchAqlPacket function.
Change-Id: Iea0927e1253800403a1ae3f3d72de1e7d96476c3
[ROCm/clr commit: d44f44a5b1]
This reverts commit 96e25898c8.
Reason for revert: SWDEV-455075, SWDEV-461507 - This change forces to
use ROCr's copy path. Reintroducing hostBlit copy path for
host-to-host copies.
Change-Id: Ic3c45b49e481c9dcdaa7611f61071778790b7e6c
[ROCm/clr commit: a9f05e22db]
- Print kernelname for graph launches, its hard to correlate packets
otherwise
- Print correlation_id if any
Change-Id: Ib8db7a00e4e7c98f570e71029e61d86f5dccc2ed
[ROCm/clr commit: 72d23a02c5]
- Remove Last graph node optimization and instead submit a barrier NOP
packet always. This simplifies the code.
Change-Id: Ied443173ba47a08b6df148ac7e3ead712acda11c
[ROCm/clr commit: badf2b0880]
The Readback and Avoid HDP Flush memory ordering workaround is
used as a fallback solution only when HDP flush register is invalid
Change-Id: Ic284eba1f95ed22b0270d3abeb904fb902015b1a
[ROCm/clr commit: 6cb7b6ec6b]
- Add LOG_TS mask for printing signal times
- Read raw ticks from signals
Change-Id: Ibdd0bf06c790729f6c65083a4784c97a3c3219e0
[ROCm/clr commit: 948ca5a931]
The "optimized" version of memcpy is outdated and
was used in win32 only.
Change-Id: I7f2e0e9051e37cec95438266824b5b0025c324c6
[ROCm/clr commit: 7448113cfc]
Dispatching multiple packets with ring the doorbell once is not supported by the lower layers
Change-Id: I7665a2dcdd4ef9e47dadfe410180fed64c5a4ee0
[ROCm/clr commit: d7f352dbed]
- Create a vector to allow multiple TS to be stored in Command.
- This would mean we dont wait for entire batch in Accumulate command
to finish when we exhaust signals.
- Reduce the number of signals created at init to 64. This min value
may still need to be tuned but the KFD allows max of 4094 interrupt
signals per device.
- Store kernel names whenever they are available and not just when
profiling. If we dynamically enable profiling like for Torch, a crash
can happen if hipGraphInstantiate wasnt included in Torch profile scope
beacuse we previously entered kernel names only when profiler is
attached.
Change-Id: I34e7881a25bbc763f82fdeb3408a8ea58e1ec006
[ROCm/clr commit: c157bfb202]
- With https://gerrit-git.amd.com/c/lightning/ec/llvm-project/+/1002628 applied, at -O0 Kernel::dynamicParallelism() returns true but virtual queue isn't created
- This causes segfault inside VirtualGPU::submitKernelInternal() when getVQVirtualAddress() is called
Change-Id: Ia7af042adad2329e870c142caaac3e8fa886f8b8
[ROCm/clr commit: d1fff7cea2]
Under ROCr physical allocations don't have initial VA and require extra
flag in ROCclr. Add an option to have a mempool of physical allocations.
Change-Id: I4d062fe0dd8113d4eaf6e8b51749ed56d8701d1e
[ROCm/clr commit: f296159f62]
- Sometimes we want to mask out kernel names, use right level for kernel
logging
Change-Id: Ideae9647c57b86ae390ff2f4131f6d8c6df5c086
[ROCm/clr commit: f1adecd186]
- Print SWq for AQL packets, this helps correlating a stream to the HWq
mapped
Change-Id: I610430c0872a1abc6636027c00163ec46983cd65
[ROCm/clr commit: 984c86f407]
- Enable Device kernel args for MI300* for now.
- Fix a perf issue which impacts graph instantiate when dev kernel args
are enabled.
Change-Id: I962e58fd9d8dd1a8db95e601cb03a8e9c7bac97f
[ROCm/clr commit: 68f40f78dd]
Node can be enabled/disabled only for kernel, memcpy and memset nodes.
If the node is disabled it becomes empty node.
To maintain ordering just enqueue marker with respective node dependencies.
Change-Id: I710f3e88ab4e76c81f6f86a40a7dc61fd4c7e440
[ROCm/clr commit: e0e63eb04d]
- Implement workaround to ensure HDP writes are done by writing and
reading the HDP MMIO register.
- Implement the same workaround for graphs, we no longer need sentinel
write/readback
Change-Id: I0d3027b46a1f61131ec62e3c8c669ff5184fa6b2
[ROCm/clr commit: f138e0d113]
Display queue base pointer in the log. This can be co-related with AQL
packets
Change-Id: I544f9b6db6ae01c85e57e4b3f0b3fffefcd7c2ed
[ROCm/clr commit: 0567c3b720]
- The correlation_id had random junk values which we were inserting in
the dispatch AQL packet even when no profiler was attached but if we had
a valid timestamp.
- Also make sure we dont even write the reserved2 field in the AQL
packet if no profiler attached.
Change-Id: Icdb7493198c1bb5e2d786a97e027288660854cd7
[ROCm/clr commit: 9a6ddae7b2]
This issue is due to the fact that we set fence_dirty_ flag for every
kernel launch. Whenever any next HIP API is caused, the stream logic
assumes fence is dirty and queues a marker, when its not needed.
Change-Id: I10a49ee951daa92221aa6be75e2334849579a45d
[ROCm/clr commit: d6b4892600]
- For new AccumulateCommand we enqueue nop barrier packet. So during stream sync we need system flush.
Change-Id: I0b97b626bcdae582ef95e7c95030d78df1fa5a54
[ROCm/clr commit: 748c6b3520]
Julia Jiang
kjayapra-amd
Shane Xiao
German Andryeyev
Anusha GodavarthySurya
Ioannis Assiouras
Sourabh Betigeri
Jacob Lambert
Saleel Kudchadker
Satyanvesh Dittakavi
Rakesh Roy
cadolphe
Alex Xie
ksankisa
Jatin Chaudhary