Implement PC Sampling session create and destroy APIs.
Change-Id: I93370d3d01b74ee15e71b8b0e20feb8f0066a3dc
Signed-off-by: David Yat Sin <David.YatSin@amd.com>
Signed-off-by: Vladimir Indic <Vladimir.Indic@amd.com>
Change-Id: Ib0c64356a1a4616b12d5dbeebe16273fe2a84abe
[ROCm/ROCR-Runtime commit: 632f9e60f7]
Add new PC Sampling API to list the supported PC Sampling methods and
options on a specific agent. If there is already a PC Sampling session
active on this agent, the list of methods returned will be reduced to
methods that can be run simultaneously with the current active session.
Change-Id: I42ac2b8f30d5c368faf8ed4cf37ca4134db22985
[ROCm/ROCR-Runtime commit: 295acf6b27]
Create allocator helper function to provide fine-grained memory on
a specific agent.
Change-Id: I32ba9aceb9c9dc708b140a0c45158e6e7a018844
[ROCm/ROCR-Runtime commit: 71f1a6726c]
ExecutePM4() function can optionally accept extra arguments for
acquire fence scope, release fence scope andcompletion signal. When
a completion signal is provided, ExecutePM4() does not wait for the
commands to complete.
Change-Id: Ib2a433b7bce1cb6260be8b76fe902335bd5dfada
[ROCm/ROCR-Runtime commit: 721e56ef5c]
Hard limit for scratch is 4GB per XCC and checks in case user specifies
values exceeding this value
Change-Id: Ib3cade762ff66c7e7d6a2d311e482cacbcf2b0de
[ROCm/ROCR-Runtime commit: d7adc94e3f]
- Per-executable contexts should be used from now on
- Global contexts are left as is for now for backwards
compatibility and will be phased out in follow up
patches.
Change-Id: I6291abf865c7ed24ee71f5065e539afc23f5ce64
[ROCm/ROCR-Runtime commit: b983c19729]
This reverts commit 5c520f4544c654e5f18e05cabd1c63d64473cfab.
Reason for revert: This patch is introducing a synchronization related bug in Unit_hipGetSetDevice_MultiThreaded testcase.
Change-Id: I367e4d4f1d75b21658ac1127c58982894a97cedb
[ROCm/ROCR-Runtime commit: 244ad319ac]
Temporary change to set the AllocateGTTAccess flag and node_id
on MES devices.
Change-Id: I22385d11b17b76cfb44278fa0d8a09bc8721cea6
[ROCm/ROCR-Runtime commit: efe455c2fa]
Instead of caching shared memory fds for export on the exporter side,
only export the FD in the async handler when requested.
The importer should request export fd closure once import is done.
Change-Id: I469e0cd1749beeb9c506c8a6461745fb039d9c3b
[ROCm/ROCR-Runtime commit: e911335cee]
Add new tools table and functions to notify in case of an event
Change-Id: I47f0c2f3c8e02d7bcb74d649903eb4f86721c154
[ROCm/ROCR-Runtime commit: a67af3807f]
Optimizations include:
- Greedy gang by placing gang leaders on first D2D sdma blit context
to avoid dead locking with other gang leaders and items. Note that
this is fine since we can't avoid an oversubscription problem when
there is only 1 xGMI link anyways, so treat all xGMI links as a single
pipe for ganging.
- Non-leader gang items don't have to poll on dependency signals so this
opens up more non-blocking SDMA channels.
- unlock gang lock when gangs are not needed.
- Change gang factor lookup from vector pair to map and register all
gpus in gang factor lookup regardless of link type so that we can take
advantage of the O(logN) direct key/value lookup time.
Fixes include:
- HSA_PAGE_SIZE_4KB was an incorrect macro to use for gang size limit.
As a result, small copies ended up ganging and hitting latency limit.
Use hardcoded 4096 bytes instead.
- Cap auxillary gang factor to the number of non-XGMI SDMA engines.
Change-Id: Ic23fde131502906a807134a04599aa6d012e8cbb
[ROCm/ROCR-Runtime commit: 62f3f250ce]
When using multi-GPU for each BO, a new dmabuf_fd needs to be imported
into libdrm.
Change-Id: Iaa2415c8f655a1ce8e92b0878517a11ff014a1d5
[ROCm/ROCR-Runtime commit: 8b85f9e668]
As the KFD IPC IOCTLs will not be upstreamed, change runtime
implementation to use DMA bufs.
DMA buf fds will be passed over abstract unix domain sockets.
The exporter spins a thread that creates a socket server.
The importer connects to the server to fetch the fd.
libDRM will be required to do a manual import and GPU map for
memory that is not already imported and mapped.
For now, use the legacy IPC implementation by default as a
follow on patch will disable the HSA_ENABLE_IPC_MODE_LEGACY
environment variable.
Change-Id: Ifd8469e9adfc81f8a1ea78d6010fb10b515ba1b4
[ROCm/ROCR-Runtime commit: 5dfebdbca9]
Implement HybridMutex to improve latencies compared to KernelMutex when
there is contention between several threads calling hsa_signal_create
and hsa_amd_signal_async_handler.
Change-Id: If53377033e749b0050727964c9303f09b02527cc
[ROCm/ROCR-Runtime commit: 8d3fee5095]
Fixes issue where t1_ counters may not be updated when doing dispatch
profiling, causing a divide by 0.
Change-Id: I91060ac3f9fd2183d277e6e7cd810398a453a87f
[ROCm/ROCR-Runtime commit: 3d1563ee68]
Add query to return flags for GPU agent memory properties and AQL
extensions.
Implement flag to determine that GPU agent is an APU
Change-Id: Ic04c51290b2b9763e14989c117f35a2e22297453
[ROCm/ROCR-Runtime commit: c86837d8d6]
Member added to KFDVersion to report if KFD supports core dump
mechanism. This is done through hsaKmtRuntimeEnable API call while
the topology is being built. It also dictates if core dump will be
generated by either KFD or hsa-runtime.
Signed-off-by: Alex Sierra <alex.sierra@amd.com>
Change-Id: I2e9d4166563402f78613d728446feb692c52d9d1
[ROCm/ROCR-Runtime commit: 54604654bd]
This API consists in one function to be called from a fault event at the
hsa-runtime to generate a core dump.
Signed-off-by: Alex Sierra <alex.sierra@amd.com>
Change-Id: Ib1b90d5beb13f93c4e8ebd21fd61705ebb12ca5d
[ROCm/ROCR-Runtime commit: 514b222368]
The alternate scratch memory is used for dispatches that have a low
number of waves but relatively large wave size.
This allows us to keep the tmpring_size.bits.WAVES field of the main
scratch to full occupancy.
Change-Id: I32d240fac4b7d38200d1eebc1b0fdc8a823920d3
[ROCm/ROCR-Runtime commit: a7a3358067]
For devices where the CP FW supports asynchronous scratch reclaim, ROCr
is able to claw-back scratch memory that was assigned to an AQL queue.
With that ability, ROCr does not have to rely on using USO
(use-scratch-once) when assigning large amounts of memory to a queue.
If we reach a situation where we are running low on device memory, ROCr
will attempt to claw-back the scratch memory.
Change-Id: Iddf8ec84e37ab8b9fdc58bafbe2b61fe2acb6eb7
[ROCm/ROCR-Runtime commit: dca8f3a21d]
Separate the event handler and scratch handler portions of the code into
separate functions.
Change-Id: Ifdb7461e816b0f2d3c1c0a74d6f020b4d6fc736c
[ROCm/ROCR-Runtime commit: 64070a9acc]
SDMA4.4 and SDMA5.2+ has increased it's available copy size to 2^30 bytes
represented by exponent as bits set in the COUNT field of the
linear copy.
Also note that the full 2^22 byte limit is available from SDMA4 onwards
as it has corrected the 0x3fffe0 HW limitation from SDMA3.
As copy limit has increase, this can change system performance
so provide env var HSA_ENABLE_SDMA_COPY_SIZE_OVERRIDE=0 to fall
back to the original 0x3fffe0 limit for debugging purposes.
Change-Id: I0fb6e5378f68e5b8a00ff559271691a943ee06ee
[ROCm/ROCR-Runtime commit: 81c64228e0]
Add handler to handle HW exception events reported by underlying
drivers. These events are generally caused by GPU resets and need the
application to abort.
As an improvement, in the future, we can provide additional information
about the exception (e.g mode-reset level)
Change-Id: If3fb5f19f9fce181a9d3b5e34a5506725856e7b0
[ROCm/ROCR-Runtime commit: 750212e50e]
AqlPacket::string should check the packet type is in range of the array
used to print its name.
Change-Id: I33dabbd941d086929526d842c9dbc0bd7305acd5
[ROCm/ROCR-Runtime commit: 7955fb01ec]
An AQL packet header field is stored using an atomic release, and needs
to be read using atomic acquire if it may be written by another thread.
Change-Id: I1d75587fd93f9c6216deebffc9a627b404a7e749
[ROCm/ROCR-Runtime commit: 395ad3b77b]
Define AMD_AQL_FORMAT_INTERCEPT_MARKER AMD vendor AQL packet. Add
support to intercept queue to invoke a callback for these packets.
Change-Id: Ia58d5fe2171f563632b4edd6343e02585f49d149
[ROCm/ROCR-Runtime commit: 23b4ce501d]
It is possible that packet rewriting an initial packet for the intercept
queue produces more packets that the size of the wrapped queue. The code
would never submit the such a set of packets as it attempted to submit
all or none. This can result in an infinite loop.
This is corrected to submit what will fit if the rewrite is larger than
the wrapped queue.
Change-Id: I8f03228c2e15151287e25de46eaee998f829c62a
[ROCm/ROCR-Runtime commit: 9f4d651d14]
Describe the assumption being made when checking if there is a retry
barrier packet on the queue. Also enforce the consequential requirement
of the minimum queue size.
Change-Id: I0efaffc5a79b9e2fdab3655b8b74270118a5c2ff
[ROCm/ROCR-Runtime commit: ca99795c58]
Using new ExtendedCoherent KFD HSA memory flag to achieve system
scope coherence on atomic instructions. Non-compliant systems may
have the need to perform explicit HDP flushes to achieve system
scope coherence using this flag.
Change-Id: Ic6b47c0e97285086fa1f52bbfa4597b81cadafeb
[ROCm/ROCR-Runtime commit: 4eb6ed7799]
Silence warnings on more stringent compile checks for lack of override
declaration.
Signed-off-by: Jonathan Kim <jonathan.kim@amd.com>
Change-Id: Iaa54dfc3dd74f5ee55763cafbbcf2db73493bb21
[ROCm/ROCR-Runtime commit: 6b4365ae4c]
On busy systems, the memory allocation can take long duration and
increase calls to hsa_signal_create/hsa_amd_signal_create. This
mitigates this issue.
Change-Id: Ib7640273262ebc3dbf1f07049ce5da10b1d6b158
[ROCm/ROCR-Runtime commit: 9a127193a8]
Generates shader bytecode stream in amd_blit_shaders_v2.h at build time
Change-Id: I5228ec5442a78d074fd85ca9cd7f7a156dd84da3
[ROCm/ROCR-Runtime commit: 4e675ce730]
Some GFX9 devices will drop commands if ring buffer submission is less
than 64 DWORDs. Pad submission with a NOP head an trailing null
DWORDs in this case.
Change-Id: I850af490fb699f7efe8aef96d97c600a8e76516b
[ROCm/ROCR-Runtime commit: cdd0728d9b]
When oversubscribing SDMA gangs, a circular deadlock can occur since
gang enqueue is staggered with respect to SDMA engine leader based
on source to destination.
As a result, an enqueued leader may be waiting on a gang item that is
waiting on another enqueued leader or gang item and so on.
To prevent this, first lock the submission to ensure dma status query
and submissions are atomic. Once this is in place, be more stringent
with ganging in that all SDMA engines must be available in order to gang.
Finally, re-enable SDMA ganging by default.
Change-Id: I4511e3487db9d26475b5aece4897f10168cc5322
[ROCm/ROCR-Runtime commit: 8f21793a3e]
Use all available SDMA engines capped by xGMI bandwith for
all D2D copies within a hive.
By default, set the latency boundary copy size as 4KB and below.
Any copy size in within this boundary will not gang.
Avoid oversubscribing engines by not ganging on engines with
pending non-ganged work.
An enviroment variable HSA_ENABLE_SDMA_GANG has been provided
to override default ganging behaviour.
Change-Id: Iccde76aa1af1d47ea2a151789432c9db4f0ffa8d
[ROCm/ROCR-Runtime commit: 7df0167821]
Add agent info query to return nearest CPU agent. This can be used to
determine which CPU agent is in the same NUMA region as the GPU agent.
Change-Id: I5400b4347ffbf4d2a836df31c4de443a38b0ecd1
[ROCm/ROCR-Runtime commit: 469defa78a]
Silence out of order initializer compile warnings during memory region
initialization.
Change-Id: Idbbdd93d3ea8cda289d25a473b3882b920b2e8d8
[ROCm/ROCR-Runtime commit: 42274cfc59]
Aqlpacket:IsValid() function: Replaced bitwise AND operator (&) with the logical
AND operator (&&) when evaluating AQL packet type
Change-Id: I59980bc206cc7eff424023fff0bb92b618aa8c70
[ROCm/ROCR-Runtime commit: a2d0adf9be]
Support function to retain allocation handle for memory mappings.
The get allocation properties function will return the current
allocation properties for existing memory mappings.
This is part of patch series for Virtual Memory API.
Change-Id: I0a53a11b6efc2b5bf9d463512a489a2abd812551
[ROCm/ROCR-Runtime commit: 687eb043d4]
Support exporting and importing dmabuf file descriptors for memory
mappings. The exported dmabuf file descriptors are shareable posix
file descriptors that can be used for cross-vendor, cross-device
and cross-process memory sharing.
This is part of patch series for Virtual Memory API.
Change-Id: I3673fc009f7e73bc26be8349e19f66e20d0607c5
[ROCm/ROCR-Runtime commit: b03c96c264]
Mapping memory handles to virtual memory addresses do not make them
accessible. The set access function is needed to make the memory
mappings accessible to specific agents. The get access function
returns current access properties for individual agents.
This is part of patch series for Virtual Memory API.
Change-Id: I152ba0557fd2a802eb9d840568b68cdd1911b72c
[ROCm/ROCR-Runtime commit: 13fbd8a232]
Add support for mapping and unmapping memory handles to virtual
address ranges.
This is part of patch series for Virtual Memory API.
Change-Id: If512d49ff4211e68f2064249add607a3200e458a
[ROCm/ROCR-Runtime commit: 179dcf1c77]
Add support for creating and releasing memory handles. Memory
handles are memory allocations on device memory without a virtual
address.
This is part of patch series for Virtual Memory API.
Change-Id: I5dfb162eb1661621cce171b2870a3c93b24d840e
[ROCm/ROCR-Runtime commit: e4a84c4a9c]
David Yat Sin
Konstantin Zhuravlyov
Shweta Khatri
pvanhout
Jonathan Kim
Mythreya
Alex Sierra
Tony Tye