If left non-zero the event loop will keep reinvoking the callback,
preventing AqlQueue::ExceptionHandler from running.
Change-Id: If85fbaf62f04ffd327ecf9d649aa23afad4442ce
Test for HQD preemption during stalled context restore. Added for
regression testing against new microcode.
Change-Id: I13eb7d1c598062390e12cf8a5237e53b6489f232
Signed-off-by: Jay Cornwall <jay.cornwall@amd.com>
The queue control stack size cannot exceed 0x7000 on ASICs
gfx1010 through gfx1031. The lower limit is not achievable
with AQL so this should have no practical effect.
Fixes control stack size overflow on large ASICs.
Change-Id: Ib78cf6e4c5f096044bf8de24debe211689891caa
Signed-off-by: Jay Cornwall <jay.cornwall@amd.com>
Debugger path is taken for (trap_id >= 3) and single step exceptions.
Other traps/exceptions behave as before.
Change-Id: I276c0eb69953709968353a57717ee017d22348a2
- Use new buffer resource descriptor layout
- Handle wave32 scratch allocation error from CP
- Make wavefront size a property of scratch allocation requests
- Repurpose wave64-specific amd_queue_t.scratch_workitem_byte_size field
- Clear index_stride field in V# on gfx10, calculated per-dispatch by CP
Change-Id: If2acdf6430772abd4d6a8c792fc8c11260764dda
Originally reserved 100 SGPRs per wave. Pre-gfx10 needs 102 SGPRs
and gfx10 needs 128 SGPRs. Reserve 128 SGPRs per wave for all ASICs
to simplify calculation.
Also double VGPR register size for gfx908 family
Change-Id: I98b741cbfa051f49ed37ff25d99f851f124be7b6
Signed-off-by: Jay Cornwall <Jay.Cornwall@amd.com>
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
If M0[23] is set then the driver will interpret the interrupt as a
debug event, rather than a signal event.
Clear M0 before sending the interrupt. All paths here are terminal so
it's not necessary to save/restore M0.
Change-Id: Ibd85b8cc6f8556941f2308a2c3fa3c68702cd606
Report traps and fatal exceptions through a wavefront's
amd_queue_t.queue_inactive_signal. Previously, only traps were
reported and requireed the compiler to pass in the signal pointer
in s[0:1].
The signal is obtained through a mapping from doorbell index to
amd_queue_t*. The doorbell is fetched within a wavefront through
the gfx9+ S_SENDMSG(MSG_GET_DOORBELL) instruction.
Change-Id: I319b45f2e15dfcfe4db8f4065da1136e9539a42b
Assembler toolchains are moving from SP3 to LLVM. Replace trap handler
source code with LLVM equivalent.
Fix a trap issue with SQ_WAVE_IB_STS restore. Mostly harmless as all
traps are currently considered fatal to the wavefront.
Change-Id: Iacecd9dd31a1d96a083c8b8327f442f33c861f9f
KFD no longer reports MemoryAccessFault.Failure with retry fault
implementation. ROCr ignores the memory event when Failure = 0.
Use the Flags field instead, which will be non-zero when the
event is triggered.
Change-Id: Ie90799a303b0b2f1b476b20ffafdde79ae137182
llvm.debugtrap and other trap IDs are reserved and should not place
the queue into an error state.
Change-Id: I98193a35ac7da94c4a42ee75d87754ee552ebea0
- Ignore exceptions passed to the second-level handler
- Restore SQ_WAVE_IB_STS and SQ_WAVE_STATUS before exiting trap
Change-Id: I872c111c030d94eae644ae073df3c2e508f42f45
On discrete GPUs place the EOP queue in VRAM. The reader/writer of this
queue is the CP and the size is small. Dispatch latency improves
through lower read latency in AQL completion phase.
Change-Id: Id8351dcddbd21fd7c7d699803c96434c9132db71
Signed-off-by: Jay Cornwall <Jay.Cornwall@amd.com>
Hardware block testing is done with the workgroup state offset
initialized to the control stack size on all ASICs. MEC microcode
assumes this space is available when the workgroup state offset is
reset after a context restore event.
Fixes context save area overrun when the full save area is used.
Change-Id: I8eeb62f97140c6fe409fe78b4497d833584feea8
Signed-off-by: Jay Cornwall <Jay.Cornwall@amd.com>
When a fatal memory fault occurs the scheduler context-saves all queues
in the process and notifies the runtime through the memory event. The
saved state contains all GPR/LDS data at the moment of the fault.
Retrieve this state and present it to the user if HSA_DEBUG_FAULT is set
to "analyze" and the wavefront caused the fault. If amdgcn-capable objdump
is in the PATH invoke this to disassemble code around the PC.
Queue lifetime is now managed by the runtime to allow querying the
context save state for all active queues.
Change-Id: I6fee662fad1c4f9aa125bf5c53d7d0ea1ab32f95
The trap protocol changed between gfx8 and gfx9. The return address
is in trap temporaries [0,1] on gfx9 rather than [4,5] on gfx8.
Unfortunately SP3 changes the meaning of the ttmp register aliases
in gfx9, further confusing the issue.
Clean up later when LLVM assembly build is introduced to the runtime.
Change-Id: I84ea9bf3736f060dd95d0361f9d5a0f9a3576178
Should point directly to amd_queue_t.write_dispatch_id. Only noticeable
with HWS enabled which is not yet stable.
Change-Id: I169906d45225379a3ca2729ff04d298fdbb9a9fb
- Build system fixes
- No user-mode high-precision timer by default, use clock_gettime
- Use C11 aligned_alloc pending C++17 std::aligned_alloc
Change-Id: I268365bdfd11d1e817a89584b9e086ee5b86e1dc
Gfx9 requires monotonic write pointer and doorbell.
Cound fields are 1-based compared with 0-based pre-Gfx9.
- Restructure implementation to use monotonic ring indices
- Remove redundant submission size checks (handled by AcquireWriteAddress)
- Unify copy/fill per-command limit (documentation is unclear)
Change-Id: I57c1675221d2e63aa319fee700d9951671e1bd65
Ensure that the write index and ring buffer contents are visible
to the HW before sending the doorbell. The latter is a write-combined
MMIO store and must be ordered with prior cacehable non-MMIO stores.
Also be more explicit about memory semantics for doorbell stores.
Change-Id: Ie4d96a7ee2a507237a8dbe7705fdf234d62ce9ba
gfx802 requires a workaround for a VM TLB bug in which lookups use
the ACTIVE bit of the 8th PTE within any aligned group of 8 PTEs.
Until this is fixed in amdgpu the GPUVM doorbell logic will fail.
Change-Id: I5ec7b1fcd8b7677011a141d27cfc486c45d9a415
Signed-off-by: Jay Cornwall <Jay.Cornwall@amd.com>
The PM4 IB must have executable permission.
A second part of this fix concerns robustness when this is not the case.
This remains under investigation.
This fix will shortly be cleaned up in a refactoring pass to consolidate
calls to hsaKmtAllocMemory.
Change-Id: I326fe01949a77669e0b07c3cadc9fd44b8065055
Due to a misinterpretation of the HSA specification the microcode has,
until now, been responsible for ensuring a coherent view of the
amd_kernel_code_t object when acquire_fence_scope is set to agent or system.
To correct this the runtime must instead assume this responsibility.
Introduce GpuAgentInt::InvalidateCodeCaches to perform this operation
on-demand. Invoke this after code object allocation. Extend the Queue
implementations to support PM4 command submission, through which the
PM4 command ACQUIRE_MEM can be submitted to perform cache invalidation.
Submit through a runtime-managed queue shared with the blit implementation.
This change depends on microcode support and this is checked against the
running version. Older microcode builds will perform cache invalidation
themselves, so it is acceptable for this change to do nothing in that case.
Change-Id: I268dd2b83af3decdd9ad07430a81df8a2ecb6bd2
This option was disabled by default to address issues writing to stderr
in Windows applications. The lack of an error message for memory access
faults is confusing to users, however.
Enable the error message by default on Linux only.
Change-Id: I1f44ba42362f8874abdc7c8e63ddd54a855b5394
The runtime needs a queue on which to submit cache management commands.
Device-to-device blit copy already creates a queue unconditionally.
We can share this queue for both purposes.
This change restructures the BlitKernel interface to accept, rather than
create, a queue. GpuAgent creates queues as needed for both cache
management and blit compute.
Fix queue full detection in AcquireWriteIndex (<= vs <).
Change-Id: I61d0c6b9d04f2dba74872f0676ad791435778ba4
This is the first part of transitioning to the LLVM-based assembler.
SP3 is deprecated and all references to the library are removed.
Pending LLVM support, relevant shaders have been precompiled.
Change-Id: I7d44cef5ded1836c4a74b77881af5bea8803d2c1
On multi-node systems only the first CPU node was recognized in the
signal consumer list, causing fallback to non-interrupt signals.
Change-Id: I9bd0706bafbe046be9d7f210d05fa4cf1fcd16fa
Reduces the number of blit queues from 3 to 2, when SDMA is unavailable,
improving the availability of queue slots for applications.
Change-Id: I8860d2b6c6d6527494b9fc35d164099e1313886a
max_single_fill_size_ overflowed the packet field size. Reduce by one dword.
[git-p4: depot-paths = "//depot/stg/hsa/drivers/hsa/runtime/": change = 1259263]
Jay Cornwall