To allow non-POD global variables to last until the last thread
has exited, use "new" to allocate the memory instead of static
allocation.
Change-Id: Ica571b61ff8068a52e472c49cb1c44917e60c8c8
[ROCm/ROCR-Runtime commit: 0878deda17]
Also restore dead signals cleanup for old path when HSA_WAIT_ANY_DEBUG
is used.
Change-Id: I51a7404991443c9f6cbf57b4b9e9faa694b9538c
[ROCm/ROCR-Runtime commit: 700f1d9abd]
An ASAN run of the release build revealed some elements of
the supported_isas static map were still using stack data. This
change makes it use heap data so it will persist.
Change-Id: Ie51887e88b9e2dec27acfc97ea45a6219fea971c
[ROCm/ROCR-Runtime commit: c7521a5f2a]
SDMA queue resources are limited when all SDMA copies are bottle necked
into 2 engines. Callers will not be able to make the best decisions
to allocate queue resources fairly so have ROCr fallback to old round
robin behaviour dictated by KFD.
Change-Id: I93d52297976d74e20129c5eb1dcfbfa5aa5067a7
[ROCm/ROCR-Runtime commit: 7f8676e177]
These are mostly AIE related, but there are a couple of others.
Change-Id: I549e004772160ca282d4c94dc9d94dd2ccae8b1c
[ROCm/ROCR-Runtime commit: 08699069d6]
- Add the new path to avoid WaitAny() calls in AsyncEventsLoopp() with
HSA_WAIT_ANY_DEBUG key. The new path is selected by default.
The optimizaiton combines all logic of WaitAny() in a single processing loop
and avoids extra memory allocations or ref counting. Also it won't spin
on the CPU if all events are busy.
Change-Id: I197ce60d0d023fbb672f700d6e87702686f1f55a
[ROCm/ROCR-Runtime commit: 0fc7369ba5]
On GPUs where EOP is handled in asic, the read_dispatch_id is not always
updated after each packet. Look for the first dispatch packet that needs
scratch memory before allocating scratch.
Change-Id: Ibf4b4b485f99bf2fabfe48e9609ca99111fdafbe
[ROCm/ROCR-Runtime commit: d90fbee9c4]
The supported_isas static unordered_map was adding stack
allocated Isa objects. Instead, make the objects statically
allocated, as supported_isas itself is.
Change-Id: I23405e218290d48deea6f984f76c57e7b43e314e
[ROCm/ROCR-Runtime commit: fd99b74287]
When ROCr is built as a static library, global variables
were often not initialized to valid values at their first
use. This change addresses that problem.
Change-Id: I550fa41feb3bc04b9cc686bcfb4acf2a7b651a88
[ROCm/ROCR-Runtime commit: 9b13bcd0ac]
Devices older than GFX90a hit a segfault on queue unmap when an
SDMA queue has been assigned a fixed engine. Bypass fixing the
engine for these devices for now.
Change-Id: I7d2f882d2377f004a7bb65f3b397396db07ce6d3
[ROCm/ROCR-Runtime commit: 1d6ff45673]
Fix memory leak for memory regions objects when GPU is masked using
ROCR_VISIBLE_DEVICES.
Change-Id: I610842a18adbc3cdc854b12650844e271bc00592
[ROCm/ROCR-Runtime commit: dbae8da515]
To correctly map to all GPUs after an import, use the new extended
registration call that can import a virtual address without having to
specify a target node.
Change-Id: Ifca8f6f6ee24fa99b2af357dcc3ea1de3ab234f7
[ROCm/ROCR-Runtime commit: 0ae064fe2d]
When hsa_amd_vmem_set_access is called, do not remove permissions for
unspecified agents. Also updating documentation in header to clarify
this.
Change-Id: I3bb4cf08ba399f85cc67b17fd13a4a40d862415f
[ROCm/ROCR-Runtime commit: 73f6bfa747]
Socket server accept calls do not guarantee synchronous actions
post-accept. This can result in a race condition.
To resolve this, first limit the socket server's listen backlog to a
single connection. This will force competing clients to busy-retry
until timeout.
Second, make the DMABUF IPC file descriptor send-receive and import
calls into an atomic routine per connection.
By doing these fixes, not only to we resolve potential races but
we guarantee that any exporter process will create at most one
file descriptor that will only last for the duration of the import
transaction. This alleviates any concern on running into system
limits for the number of open file descriptors per process.
Change-Id: I6d8b14795a680d89a2707e082fa027d525792e05
[ROCm/ROCR-Runtime commit: 909b82d463]
Discarding blocks for reallocation on IPC export for better memory
performance trigger memory violations with DMA BUF exports so bypass
this for now as application performance drops haven't been observed
with the bypass.
The raw fragment should be passed to the DMA Buf export call as well
since offsets will be implicitly applied in the Thunk/KFD for
export/import calls.
Also, use the agent information directly from the pointer
information so that the export call doesn't have to scan memory to find
this. Pass the node ID in the handle so that the import call doesn't
have to make two thunk imports to fetch the node ID for GPU memory
imports.
Finally, allow the user to use DMA Buf IPC via
HSA_ENABLE_IPC_MODE_LEGACY=0 for developer testing as legacy mode will
be applied by default.
Change-Id: Ie8fe267f8768fa5df37126078406f7065f69ff4e
[ROCm/ROCR-Runtime commit: 32bb0764b7]
When an entry is deleted from the array, it's set to nullptr
but not removed. Most other functions that
iterate over the array check if the entry is nullptr
but this loop in IterateExecutables did not.
Change-Id: I763b361eea59f6df201bb86ead0234e95f2cf79c
[ROCm/ROCR-Runtime commit: f3664fd124]
Return false if trying to free a NULL pointer (or invalid size)
internally in ROCr. This is to detect errors within ROCr when trying
to free NULL pointers. If a user of ROCr tries to free a NULL
pointer, this condition should be caught at the beginning of the
Runtime::FreeMemory(...) function and return HSA_STATUS_SUCCESS. This
matches the behavior of the free(...) or delete functions that
silently ignores calls when the passed a NULL pointer.
Change-Id: I84bc26928b35023e19cd9f214b42c6ee9508029c
[ROCm/ROCR-Runtime commit: 0af7a54ebe]
Refactor VMemorySetAccess so that it can be re-used in the following
patch.
Change-Id: I341241da7a59724bb3611172f0d26b0689d7bb46
[ROCm/ROCR-Runtime commit: 8f1b05660a]
Adds support for AllocateMemoryOnly inside XDNA driver.
Move the IsLocalMemory() check inside the KFD driver
since the XDNA driver can, and needs to, create handles
on system memory buffer objects.
Changed handle variable name from thunk_handle to user_mode_driver_handle,
which is more representative if we support non-GPU drivers.
Change-Id: I95db9d575afd1ab0ff2de74cea5175d9a12a721b
[ROCm/ROCR-Runtime commit: 4bf102dc6b]
Adds support for the packet interface for interacting with
the Embedded Runtime (ERT) on AIE agents. The ERT is what
interprets command packets send to the AIE agent work
queues.
Change-Id: Id28fb98056b2c046354c446bdc9568d74385bea1
[ROCm/ROCR-Runtime commit: 6abb993f65]
Adds support for initialzing the XDNA driver so that
a hardware context can be created for an AIE queue.
Right now this initializes the device heap in the driver,
gets the relevant tile parameters for the AIE agent,
and creates a hardware context that backs the AIE queue.
Change-Id: Ib90e1bc67a8637f6db3ff2bebe34677843796417
[ROCm/ROCR-Runtime commit: 931733d51a]
Increase queue-size for co-op queues to 16K to improve performance on
some workloads
Change-Id: I4d3bf0ecbd30ebb648b68d9c5fdabadc670a386c
[ROCm/ROCR-Runtime commit: 924e11ba7f]
GFX 9.4.x has better performance for CPU-GPU copies when using
engines in reverse order from other devices.
Change-Id: I1eaebf0e837bb7f44712f40d5115df618f6a73d7
[ROCm/ROCR-Runtime commit: 509e8d863a]
If the KFD doesn't support targeting SDMA engines, ensure that ROCr
selects the correct downstream queue type by using an invalid engine.
Change-Id: Ia6848126f67f3d35ab37248633e8e0e6e2d77fff
[ROCm/ROCR-Runtime commit: 24b25003b0]
- Use HSA_ALLOCATE_QUEUE_DEV_MEM=1 to create AQL queue in device
memory.
- Before writing AQL packet header to the queue use an SFENCE to ensure
that there is no reodering of the writes over PCIE
Change-Id: I5eacdc35108c4a1e245c75ae349b7495451aa60d
[ROCm/ROCR-Runtime commit: 3baaa6e9c0]
Remove KFD-specific Allocate/Free calls from the AMD::MemoryRegion.
The KFD-driver-specific Allocate/Free calls are now implemented in
the KfdDriver. Future changes will migrate the remaining KFD-specific
calls out of AMD::MemoryRegion.
This allows the MemoryRegion to be used across AMD drivers like the
XDNA driver.
Change-Id: Ib6a2a9e5e1a15e61644d2592beb3a8e6578c3010
[ROCm/ROCR-Runtime commit: 68669f4e1a]
Adds the initial KFD driver interface and use it to open the
KFD from amd_topology.cpp.
This change is to show the direction of the Driver interface for
initially supporting the KFD and to get feedback on the approach.
For now we wrap relevant ROCt calls behind this generic driver
interface so that we can generalize core ROCr components like
MemoryRegion, Runtime, etc.
Now that ROCt is incorporated into ROCr, we can more fully integrate
ROCt into the Driver interface. Ideally, we get to a point where
the generic Driver interface can support KFD, XDNA, and potential
future drivers.
Change-Id: I4573fd6af1f8398233ee9d3814d9f3139dd0279c
[ROCm/ROCR-Runtime commit: c42ff44a6a]
This change adds the initial classes for the AIE agent and AIE AQL
queue.
An AIE agent list is added to the core runtime object.
Change-Id: I84b02f52171b80726dfb2c8431582a3ea2986eb3
[ROCm/ROCR-Runtime commit: 8ea62f1cea]
Rewriting logic to fix issue where pthread_create would return errors
other than EINVAL, and these errors would be ignored.
Change-Id: I573958724dcf886c20e8c14e6a9182303b3ffa06
[ROCm/ROCR-Runtime commit: c8dd4d2b3b]
Konstantin Zhuravlyov
Chris Freehill
Jaydeep Patel
Yiannis Papadopoulos
Jonathan Kim
German Andryeyev
David Yat Sin
Young Hui
James Xu
Sam Wu
Tony Gutierrez
Saleel Kudchadker
Shweta Khatri