AqlPacket::string should check the packet type is in range of the array
used to print its name.
Change-Id: I33dabbd941d086929526d842c9dbc0bd7305acd5
[ROCm/ROCR-Runtime commit: 7955fb01ec]
An AQL packet header field is stored using an atomic release, and needs
to be read using atomic acquire if it may be written by another thread.
Change-Id: I1d75587fd93f9c6216deebffc9a627b404a7e749
[ROCm/ROCR-Runtime commit: 395ad3b77b]
Define AMD_AQL_FORMAT_INTERCEPT_MARKER AMD vendor AQL packet. Add
support to intercept queue to invoke a callback for these packets.
Change-Id: Ia58d5fe2171f563632b4edd6343e02585f49d149
[ROCm/ROCR-Runtime commit: 23b4ce501d]
It is possible that packet rewriting an initial packet for the intercept
queue produces more packets that the size of the wrapped queue. The code
would never submit the such a set of packets as it attempted to submit
all or none. This can result in an infinite loop.
This is corrected to submit what will fit if the rewrite is larger than
the wrapped queue.
Change-Id: I8f03228c2e15151287e25de46eaee998f829c62a
[ROCm/ROCR-Runtime commit: 9f4d651d14]
Describe the assumption being made when checking if there is a retry
barrier packet on the queue. Also enforce the consequential requirement
of the minimum queue size.
Change-Id: I0efaffc5a79b9e2fdab3655b8b74270118a5c2ff
[ROCm/ROCR-Runtime commit: ca99795c58]
Using new ExtendedCoherent KFD HSA memory flag to achieve system
scope coherence on atomic instructions. Non-compliant systems may
have the need to perform explicit HDP flushes to achieve system
scope coherence using this flag.
Change-Id: Ic6b47c0e97285086fa1f52bbfa4597b81cadafeb
[ROCm/ROCR-Runtime commit: 4eb6ed7799]
Silence warnings on more stringent compile checks for lack of override
declaration.
Signed-off-by: Jonathan Kim <jonathan.kim@amd.com>
Change-Id: Iaa54dfc3dd74f5ee55763cafbbcf2db73493bb21
[ROCm/ROCR-Runtime commit: 6b4365ae4c]
On busy systems, the memory allocation can take long duration and
increase calls to hsa_signal_create/hsa_amd_signal_create. This
mitigates this issue.
Change-Id: Ib7640273262ebc3dbf1f07049ce5da10b1d6b158
[ROCm/ROCR-Runtime commit: 9a127193a8]
Generates shader bytecode stream in amd_blit_shaders_v2.h at build time
Change-Id: I5228ec5442a78d074fd85ca9cd7f7a156dd84da3
[ROCm/ROCR-Runtime commit: 4e675ce730]
Some GFX9 devices will drop commands if ring buffer submission is less
than 64 DWORDs. Pad submission with a NOP head an trailing null
DWORDs in this case.
Change-Id: I850af490fb699f7efe8aef96d97c600a8e76516b
[ROCm/ROCR-Runtime commit: cdd0728d9b]
When oversubscribing SDMA gangs, a circular deadlock can occur since
gang enqueue is staggered with respect to SDMA engine leader based
on source to destination.
As a result, an enqueued leader may be waiting on a gang item that is
waiting on another enqueued leader or gang item and so on.
To prevent this, first lock the submission to ensure dma status query
and submissions are atomic. Once this is in place, be more stringent
with ganging in that all SDMA engines must be available in order to gang.
Finally, re-enable SDMA ganging by default.
Change-Id: I4511e3487db9d26475b5aece4897f10168cc5322
[ROCm/ROCR-Runtime commit: 8f21793a3e]
Use all available SDMA engines capped by xGMI bandwith for
all D2D copies within a hive.
By default, set the latency boundary copy size as 4KB and below.
Any copy size in within this boundary will not gang.
Avoid oversubscribing engines by not ganging on engines with
pending non-ganged work.
An enviroment variable HSA_ENABLE_SDMA_GANG has been provided
to override default ganging behaviour.
Change-Id: Iccde76aa1af1d47ea2a151789432c9db4f0ffa8d
[ROCm/ROCR-Runtime commit: 7df0167821]
Add agent info query to return nearest CPU agent. This can be used to
determine which CPU agent is in the same NUMA region as the GPU agent.
Change-Id: I5400b4347ffbf4d2a836df31c4de443a38b0ecd1
[ROCm/ROCR-Runtime commit: 469defa78a]
Silence out of order initializer compile warnings during memory region
initialization.
Change-Id: Idbbdd93d3ea8cda289d25a473b3882b920b2e8d8
[ROCm/ROCR-Runtime commit: 42274cfc59]
Aqlpacket:IsValid() function: Replaced bitwise AND operator (&) with the logical
AND operator (&&) when evaluating AQL packet type
Change-Id: I59980bc206cc7eff424023fff0bb92b618aa8c70
[ROCm/ROCR-Runtime commit: a2d0adf9be]
Support function to retain allocation handle for memory mappings.
The get allocation properties function will return the current
allocation properties for existing memory mappings.
This is part of patch series for Virtual Memory API.
Change-Id: I0a53a11b6efc2b5bf9d463512a489a2abd812551
[ROCm/ROCR-Runtime commit: 687eb043d4]
Support exporting and importing dmabuf file descriptors for memory
mappings. The exported dmabuf file descriptors are shareable posix
file descriptors that can be used for cross-vendor, cross-device
and cross-process memory sharing.
This is part of patch series for Virtual Memory API.
Change-Id: I3673fc009f7e73bc26be8349e19f66e20d0607c5
[ROCm/ROCR-Runtime commit: b03c96c264]
Mapping memory handles to virtual memory addresses do not make them
accessible. The set access function is needed to make the memory
mappings accessible to specific agents. The get access function
returns current access properties for individual agents.
This is part of patch series for Virtual Memory API.
Change-Id: I152ba0557fd2a802eb9d840568b68cdd1911b72c
[ROCm/ROCR-Runtime commit: 13fbd8a232]
Add support for mapping and unmapping memory handles to virtual
address ranges.
This is part of patch series for Virtual Memory API.
Change-Id: If512d49ff4211e68f2064249add607a3200e458a
[ROCm/ROCR-Runtime commit: 179dcf1c77]
Add support for creating and releasing memory handles. Memory
handles are memory allocations on device memory without a virtual
address.
This is part of patch series for Virtual Memory API.
Change-Id: I5dfb162eb1661621cce171b2870a3c93b24d840e
[ROCm/ROCR-Runtime commit: e4a84c4a9c]
Add support for reserving virtual address ranges. Virtual address
ranges are addresses without any memory backing. These address ranges
need to be mapped to memory handles later.
This is part of patch series for Virtual Memory API.
Change-Id: I5d066e7421d6896f933f524312afc230a13d594e
[ROCm/ROCR-Runtime commit: 1085311f1a]
Change initialize libdrm device and file descriptor initialization
to use new APIs from Thunk. Libdrm recommends that we re-use the same
file descriptor thoughout the life of a process instead of re-creating
new one each time.
This is part of patch series for Virtual Memory API.
Change-Id: I1c0b8d1bd660cd25478b5f94c84071b90d93fc6c
[ROCm/ROCR-Runtime commit: a55f11025b]
Checks whether version of libdrm library installed on current
system supports the amdgpu_device_get_fd API. This API is
required to support the virtual memory API functions. The
amdgpu_device_get_fd function was introduced in libdrm-2.4.109.
Using a runtime check test instead of static dependency to be
able to support previous APIs on older versions of libdrm.
Add query for virtual memory API support.
This is part of patch series for Virtual Memory API.
Change-Id: Iec831eb24b5d1689c392e50ae86f4d52d4870ac4
[ROCm/ROCR-Runtime commit: e65edb35fc]
Unless SDMA blits have actually been used for copies, prevent the DMA
copy status from querying the blit's pending byte status to avoid
creating an unnecessary HW queue.
Change-Id: Ied1fbed73c08f0408f0e3583f9b56f2768c71708
[ROCm/ROCR-Runtime commit: 92467fd282]
Querying pending bytes on a blit kernel is unnecessary when runtime
runs out of SDMA resource since we are returning an SDMA availabilty
mask.
Change-Id: I347efba0c85b70ea3ba8749d76a499afc23909e8
[ROCm/ROCR-Runtime commit: 8c60f04a99]
Added HSA_AMD_MEMORY_POOL_GLOBAL_FLAG_EXT_SCOPE_FINE_GRAINED flag to enable extended scope memory region
where the device-scope atomics act as system-scope atomics
Change-Id: I79fc3207cb630dfc68bed2f8aabd75f35fe80b12
[ROCm/ROCR-Runtime commit: 77bf357647]
KFD kernel version 1.13 starts to support event age
tracking which help elimating unncessary busy wait.
Change-Id: Ib447ed6e0350f3110a4d6b9b80a0388000dd0e72
Signed-off-by: James Zhu <James.Zhu@amd.com>
[ROCm/ROCR-Runtime commit: 5871b28503]
Earlier, hsa-runtime was unable to find symbols from a stripped ELF-image becasue
no support to find symbols from ".dynsym" section.
Looking for symbols in .dynsym is enabled by LOADER_USE_DYNSYM=1
environment variable
Change-Id: I4f0e8dd0eb053a6066d4d49b670c52e51149531a
[ROCm/ROCR-Runtime commit: 4142a77375]
Adding support for gfx941 and gfx942 ISAs.
gfx940 ISA will use sc0:1 sc1:1 on load/store operations
gfx942 ISA will use default load/store operations
Change-Id: If1efbef86f59e2cf2d48fe359cd4166405a0a579
[ROCm/ROCR-Runtime commit: 41f6d0426d]
When compiling in ASAN mode, remap the first page of device allocations
to system memory. ASAN's memory allocator uses a small amount of extra
memory to store data for housekeeping purpose. But because this memory
is from the GPU memory pool, it might have uncommon memory type for host
to access. Mapping this section of memory to the host makes this memory
accessible to ASAN.
Change-Id: I36f659d616a4d15558372592439a8723c5c84a69
Signed-off-by: Bing Ma <Bing.Ma@amd.com>
[ROCm/ROCR-Runtime commit: 50e754d08b]
Some workloads running on multi-GPU create 1 process per GPU. So each
process creates a GPU agent on every GPU, but will only create queues on
one GPU. This would cause un-necessary scratch reservation.
Change-Id: I50a216f0bcc0b5f707f3943147390b0ecec1ac22
[ROCm/ROCR-Runtime commit: 38e832a682]
Scratch cache reserved memory is only available for scratch memory use
so do not report this memory as available to the user via the
HSA_AMD_AGENT_INFO_MEMORY_AVAIL api.
Change-Id: I52f96e62536458bcaa52b9f4be5de856d5680dc4
[ROCm/ROCR-Runtime commit: 3477fbc661]
Some applications will keep trying to allocate device memory until the
allocation fails. This causes all device memory to be used up and we are
then unable to allocate scratch memory for dispatches. Reserve enough
memory for 1 small scratch allocation.
Change-Id: I968400d41540ba1aca8f28581f229693eec02225
[ROCm/ROCR-Runtime commit: 8ebf5f9c48]
Adds hsa_amd_portable_export_dmabuf and hsa_amd_portable_close_dmabuf
which allow obtaining dmabuf handles to rocr allocations. These handles
may be shared with other APIs to support cross vendor & cross device
memory sharing.
Adds query to return whether dmabuf export is supported
Signed-off-by: Jonathan Kim <Jonathan.Kim@amd.com>
Signed-off-by: David Yat Sin <David.YatSin@amd.com>
Change-Id: I7f98501087d9563d07fc2cb428cc886b1e518b1e
[ROCm/ROCR-Runtime commit: 42243c1e8f]
Use mwaitx instructions when busy waiting for signals to reduce CPU
energy usage.
This can be disabled by setting HSA_ENABLE_MWAITX=0
Change-Id: Ic207895a491b2bf6dacba47ef0921df3faad5b5a
[ROCm/ROCR-Runtime commit: cc48dfdbff]
ROCr internally uses the same allocation_map_ list to track memory
allocations that are both for internal allocations and allocations by
users of ROCr library. In some edge cases, the library user would call
hsa_amd_pointer_info on an invalid pointer, but ROCR would return the
pointer as valid because this pointer belongs to a memory range that
was allocated internally within ROCr. Adding a flag to differentiate
between internal and external allocations.
Change-Id: I98c52bd85f3985d1ba1b0e3101d2254b003412cf
[ROCm/ROCR-Runtime commit: 59685f4492]
Track and report the size, in bytes, of pending unexecuted blit
commands. To be used in copy ganging.
Change-Id: Ia7453ff88571e927df771c6c819b73c17e67708e
[ROCm/ROCR-Runtime commit: 27596aef0c]
Currently, Wavefront::GetInfo(HSA_WAVEFRONT_INFO_SIZE.. always returns
64. Instead, return the proper wavefront size based on the ISA.
Temporarily, we only return 1 wavefront size for each ISA. As we do not
have mechanism from upper layers to determine correct wavefront when
there are multiple wavefronts supported. We are temporarily
returning 32 for all gfx1xxx cards even though they support 64 as the
kernels for gfx1xxx are compiled for wavefront-32 by default.
Change-Id: Ic6c2917b7e6d3704daf742d243f5ec7f49430de9
[ROCm/ROCR-Runtime commit: f7e3782b42]
MES devices need GART mappings and therefore need non-paged memory. But
using non-paged memory introduces performance regression where it can
take over 80 ms to see the signal changes if the memory is in the wrong
NUMA node. Currently, we cannot control NUMA affinity when allocating
non-paged memory. Using non-paged memory allocation only on devices that
have MES scheduler
Change-Id: Ib27fb01d75247aa4f2bb2aa4503c6af5a98afda0
[ROCm/ROCR-Runtime commit: c1e836b6ab]
Using previous method of std::thread for SVM profiler task was causing
segfaults on thread launch on RHEL 8 if libhsa-runtime library is loaded
using dlopen.
Change-Id: Ic010cd6ae9bc6e6ed0605de02b93f6aae8ed3e97
[ROCm/ROCR-Runtime commit: 0e4c7336ff]
Amount of memory requested by user may be aligned-up internally to
the memory pool granularity. The extra padded memory should not be
considered when validating pointers from the user. Also return the
user requested size when user queries pointer information.
Change-Id: I28b25448ea03c836b44fafdb34b7330cf6887424
[ROCm/ROCR-Runtime commit: 39632a713e]
Non-paged allocation for queue memory necessary for binding wptr to
GART. Required to support usermode queue oversubscription with MES for
GFX11.
Adds AllocateNonPaged entry to MemoryRegion::AllocateEnum for clarity;
aliases AllocateIPC.
Signed-off-by: Graham Sider <Graham.Sider@amd.com>
Change-Id: I1a97a1820da26cf2433d9c237b2e6d2b0b8628b4
[ROCm/ROCR-Runtime commit: 061aa04147]
Tony Tye
Shweta Khatri
David Yat Sin
Jonathan Kim
James Zhu
raghavmedicherla
Sean Keely
Alexander Turek
Graham Sider