Includes some workarounds and HMM.
Conflicts:
opensrc/hsa-runtime/core/runtime/amd_topology.cpp
opensrc/hsa-runtime/core/util/flag.h
Change-Id: I22976f07964a43dbb228a6231777dbd599112b8d
[ROCm/ROCR-Runtime commit: 7333c77e22]
Sramecc is misreported in kfd 4.0 and prior. To prevent possible
corruption due to d16 instructions, deny use of gfx906 with older
kfds and correct misreport for gfx908. Denial of gfx906 may be
overridden by setting HSA_IGNORE_SRAMECC_MISREPORT=1.
Change-Id: I7d5c3a716fad01c348f8b88cd508cedbf914c989
[ROCm/ROCR-Runtime commit: 45fbe5b192]
Park the wave, if it is stopped, to avoid halting it at an s_endpgm
instruction if the architecture does not support it.
Free ttmp6 by converting the dispatch_ptr into a queue packet index
(25-bit) and storing it in ttmp7[24:0].
Save the exception PC in ttmp11[22:7] ttmp6[31:0].
Change-Id: Iaa3c5baf5b488c0b534044d338f12bffa63ddce2
[ROCm/ROCR-Runtime commit: ea6ee0aa81]
Legacy p2p copy path incorrectly transfered in whole pages rather than
the requested size.
Change-Id: I9aa7337754f9e32f587a0cc5305f8ffeb6196f10
[ROCm/ROCR-Runtime commit: 34ac62274a]
Replace the stop reasons ttmp11.trap_raised and ttmp11.excp_raised
with ttmp11.wave_stopped which indicates that the trap handler has
halted the wave as the result of an event (trap, single-step or
exception).
If the wave is stopped because of a trap, also record the trap_id in
ttmp11.saved_trap_id[7:0].
Save status.halt in ttmp11.saved_status_halt, so that it can be
restored when resuming a wave (changing a wave's state from stopped to
running or single-stepping).
Change-Id: I7322f59b60e8cc1b92bf5f067dba606a3109ef49
[ROCm/ROCR-Runtime commit: 9ca79d072a]
To support single stepping the instruction preceding an s_endpgm,
unwind the PC by 8 bytes and set ttmp11[9] to notify the debugger
that the wave is halted with a modified PC.
Bump the debug r_version for this new trap handler ABI.
Change-Id: I55e4e0d65576f92da14a336266c31c513baab547
[ROCm/ROCR-Runtime commit: 8aec53969f]
Each SE must be assigned equal numbers of slots and slots
must be assigned in units of whole groups.
Change-Id: I8f3677237fa6f2e2d25e3e78210c5a7a0ad792f3
[ROCm/ROCR-Runtime commit: 7bc6aac5d2]
- Use consistent naming in Isa class.
- Remove unused Isa methods.
- Simplify Isa methods.
Change-Id: I7c4045d08fbfe0d94b3181db8ebc5e5ed8c8cc82
[ROCm/ROCR-Runtime commit: 6bbf6b1c9c]
Store target ID string in isa registry and use for returning agent and
isa name.
Change-Id: I72a20d8ff963c73d86392158aff3853e4c9bfdbd
[ROCm/ROCR-Runtime commit: 853ccc762e]
Avoids calling to KFD to map/unmap scratch allocations for
every large scratch using dispatch.
Change-Id: I9fab5705251ec82b03e4f2f2ca6da7cdccabefb9
[ROCm/ROCR-Runtime commit: 27e044ae4d]
Improves HIP event performance in directed benchmarks where
clock sync latency is significant.
Change-Id: I78b724a14a8f5b6a9a2b9f4d85afe9d8b81808a6
[ROCm/ROCR-Runtime commit: 32d0fcafa9]
For small copies cache flush latency is larger than data transfer
latency in local VRAM. Select SDMA for small copies.
Environment key HSA_FORCE_SDMA_SIZE is added for easy adjustment
of the small copy size. This may be removed after tuning is done.
Change-Id: I733fa0ae01c616617c5de50e71226b51fd589ef2
[ROCm/ROCR-Runtime commit: 2a0c6774fb]
Make explicit reference to hsa_api_trace.cpp from
initialization of hsa_table_interface.cpp. Breaks
the ability to use hsa_table_interface.cpp in plugins.
Change-Id: I22a42d3a132512b0d9ec7a1ca629b169e7f8eba7
[ROCm/ROCR-Runtime commit: f4fe7ddf47]
The excess declarations mark implemenation functions as default
visibility. Normally this is not an issue since our linker script
will specify which visible symbols will be permitted into the dynamic
symbol table. However, for static linking methods which apply linker
directives during incremental linking symbol visibility must be correct
in the (non-dynamic) symbol table.
Change-Id: I13dc8dd1019368e8943920d36335a91f0c555a92
[ROCm/ROCR-Runtime commit: f6e6eae86d]
The size of the m0 payload for MSG_INTERRUPT has changed in gfx10. It is
now 23bit wide instead of 24bit wide in gfx9.
Since we are generating different binaries for gfx9 and gfx10, we can
conditionally set DEBUG_INTERRUPT_CONTEXT_ID_BIT to 23 for gfx9 and
22 for gfx10.
Change-Id: Ifc15a9fa4399d35328ab58b742f791f1660bcd9a
[ROCm/ROCR-Runtime commit: 23df617150]
- Make code object reader use mmap when loading from a file on Linux.
- Support computing code object URI for memory either fro the loaded
host executables, or from all mmapped files. Define the environment
variable HSA_LOADER_ENABLE_MMAP_URI to non 0 to search the mmap
files, otherwise only the loaded executables will be seatched.
- For mmap search, determine file size and ommit offset and size URI
fragment when the code object is the whole file even when specifying
a file size explicitly or specifying memory that has been mmaped.
- Always return a non-empty code object URI.
- When a code object reader is created, complete all fields to ensure
it can be used in a multi-threaded manner using only const
operations.
- Add missing exception handlers in the AMD vendor extentions.
- More rigorous checking for errors.
Change-Id: I07797b1dc60c5c64245142d77becf9f7c9643395
[ROCm/ROCR-Runtime commit: 91cb98dab6]
For static linking all non-API functions must have internal linkage
or be in project namespace to avoid colliding with application symbols.
Change-Id: I3398e40a387cca36f3ddef2b978306c765161d72
[ROCm/ROCR-Runtime commit: 0be21c54a6]
Save and restore exec_lo/exec_hi around the mGetDoorbellId macro in
the signal_error case just like we do in the signal_debugger case.
Also reset the wave_id (ttmp4/ttmp5) to 0 since it isn't preserved.
0 will be detected as a new wave by the debugger api library.
Change-Id: I5123caa9431154ec1584bae85e42648c97c64c37
[ROCm/ROCR-Runtime commit: db6a781f0c]
Set ttmp11[8] and send a signal for the debugger when the handler
is entered because of an address watch exception.
Change-Id: Icc83a79027bb7ca1e50e19e2f00464cb9ca862f3
[ROCm/ROCR-Runtime commit: da6d892058]
Adds the following:
- New factory method to create a code object reader from
file with offset and size.
- A pair of queries on a loaded code object to get the URI name/length.
- A bump to the AMD vendor loader extension API and its associated table.
Change-Id: I17c83e9c2447d29a43c438459395365f786a3611
[ROCm/ROCR-Runtime commit: 9eb735ec24]
Memory from the suballocator may be exported via IPC. If this
happens then the allocating process should not reuse that memory
since it would still be connected to the remote process. IPC exported
memory must be released back to the driver.
Change-Id: I2ab0c814f63191f753fc3640cc4140ee144bf07f
[ROCm/ROCR-Runtime commit: 29b660c91e]
The 1st level trap handler jumps to the 2nd level trap handler on
context save requests or regular traps if (mode.debug_en
&& !status.halt) is true.
If we return from the 2nd level trap handler without status.halt=1, then
we need to make sure mode.debug_en is cleared, or we will re-enter the
2nd level trap handler again and again when trapsts.savectx is set.
Change-Id: I4db6369de8c91a32842f488a4df5c9d94fa65aa9
[ROCm/ROCR-Runtime commit: 584ef1e1ca]
In the case where SQ_WAVE_TRAPSTS_XNACK_ERROR_MASK is set, we also need
to set the TTMP11_EXCP_RAISED_BIT in ttmp11. If we don't, the debugger
may think that the wave is halted at launch (halted without events).
Change-Id: I8c19605bbfc145275728de4ad1979d3ba8bb478a
[ROCm/ROCR-Runtime commit: 838c6bd6ad]
The ROCR trap handler should check for all end program instructions
and not halt on them. Mask off the imm16 before comparing the
instruction to the s_endpgm opcode.
Change-Id: I669ffc7f5b699d7daf0c8ec5761ed7bb193f07a7
[ROCm/ROCR-Runtime commit: df03a377f5]
New trap handler ABI: Record in ttmp11[8:7] the event that caused the
trap handler to be entered. We currently record 2 events, trap_raised
if an s_trap instruction was executed, or excp_raised if an exception
(MEM_VIOL or ILLEGAL_INST) was raised.
Change-Id: Ie278c8277437b3b67c2737dcd1a12fe6511df428
[ROCm/ROCR-Runtime commit: 00da82f951]
scratch_used_large_ was uninitialized leading to the observed hang.
DynamicScratchHandler would wait for a large scratch release despite no
large scratch having yet been allocated. Fixes .
The patch also removes a potential race between AddScratchNotifier and
ReleaseQueueScratch. The race condition does not exist today since both
scratch alloc and release run on the same thread. The changes will
prevent this potential race from manifesting if the async event handler
is ever updated to use multiple threads.
Also enhances scratch occupancy reduction reporting. Reporting now
prints the initial request size as well as the allocated size and the
effect on occupancy this has. Occupancy is computed in terms of the
requesting dispatch grid size so may be >100%.
Change-Id: I0fc5ee01467ff4c29bdd25d545177c97862c3bd9
[ROCm/ROCR-Runtime commit: 6c556002d8]
Queues should transition to ref counting for all queues eventually.
That cleanup will be part of shared queue pooling support.
Change-Id: I217ff5d573156678b9559da6fb81baa8cd31c617
[ROCm/ROCR-Runtime commit: 0a43a107b1]
Debugger path is taken for (trap_id >= 3) and single step exceptions.
Other traps/exceptions behave as before.
Change-Id: I276c0eb69953709968353a57717ee017d22348a2
[ROCm/ROCR-Runtime commit: 78e754935c]
Sean Keely
Laurent Morichetti
Tony Tye
Konstantin Zhuravlyov
Ramesh Errabolu
Qingchuan Shi
Jay Cornwall