The value of STATE_PRIV is captured by the 1st level trap handler, and
passed on to the second level trap handler. The value is to be restored
before exit. However it is possible for the value of
STATE_PRIV.BARRIER_COMPLETE to change while the wave is in the trap
handler (all the other waves in the workgroup has signaled the
work-gropu barrier), and in this case restoring STATE_PRIV in full would
result in STATE_PRIV.BARRIER_COMPLETE to be cleared.
Restore every bits of STATE_PRIV except for BARRIER_COMPLETE before
return to prevent this race.
Change-Id: I76c875bced7d23c58670b28f257d22c933f99fc5
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 9e625307d2]
GFX94x runs into performance regression when doing large packet
enqueues.
Drop back to legacy packet sizes for now.
Change-Id: I595838ebada66c6c5143bfdb2f56c83ee71654a9
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: b8aae52404]
Removing extra bits set in forbiddenBlock that seemed to be set for
debugging and are causing unexpected image formats to be used.
Change-Id: I29c9e319907027a2b0b6bf7c1c0c8558eb6a36f4
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: e721eb509b]
Packet for GFX12 is incompatible with pre-GFX12 as some fields changed
location. Implement code path and packet specific to GFX12.
This fixes some issues with SDMA blits and 3D images.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: I56c204aaa12160e563ec960bd3b226cfa94e142d
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 6d147dd3b1]
Add new files image_manager_gfx12.{h,cpp}.
Implement BUF/IMG/SAMP desc changes for GFX12.
Implement compute surface info code using AddrLib3 API (new starting
from GFX12).
Implement algorithm for choosing "best" swizzle mode (starting
from AddrLib3/GFX12, AddrLib provides only list of suitable swizzle mode,
up to client, ROCr, to choose the best). Algorithm implemented follows
behaviour in GFX11 and behaviour for GFX12 on other platforms.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: Ib344c86228a98bbac5acdab421ee2ef9b1e84eef
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: f8a015f53e]
Added GFX12 implementation for InitScratchSRD and for compute_tmpring.
Implementation for compute_tmpring could be combined with GFX11 with some
refactoring as a possible future improvement.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: I8013cbe4438786bf41bbfd03f6a5d3b9ef51e7bf
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: def4a6c326]
Updated struct definitions, field size changes and new fields in
registers.h.
Added resource_gfx12.h and updated fields in BUF/IMG/SAMP descriptor
structs based on documentation.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: I08f05ba30f54c40e7b823a6a105829a1e8590b3d
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 8165da63cc]
Do not allow extended-scope fine-grain memory on gfx120x devices.
Change-Id: I1e6e6c1860de00160cca9d8137b129c7e32c0526
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 7dd90f8361]
There is an issue in the gfx12 trap handler where the EXCP_FLAG_PRIV
is only fetched under certain conditions (trap_id != 0) while it should
have been fetched unconditionally. As a consequence, the interrupt
payload might contain invalid data, leading to incorrect exceptions
being reported by the runtime. Debugger is mostly un-affected as it
will inspect the wave's state to figure out what exception(s) have been
reported for each wave.
Also, it is not necessary to check for the host trap bit if trap_id is
!= 0 in gfx12, there is on trap ID anymore for host trap.
This patch implements those fixes.
Co-Authored-By: Laurent Morichetti <laurent.morichetti@amd.com>
Change-Id: Ib72cd8cc5d935ca643e241da7fccd3f96201b09d
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 7a3bf30769]
The constant declarations in trap_handler_gfx12.s have been sorted
alphabetically, which causes inconsistencies. Fix the order of
declarations where it makes sense.
Change-Id: I5b05d87a5afbe1ff3362746801a1c9373537b49e
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: ff9b11fd89]
Given the differences between previous architectures and gfx12, this
patch implements the gfx12 2nd level trap handler in a separate source
file, and adjusts the build system.
Change-Id: I65192ffbbcd66a4f78d2d0c3fb1739a92cac95d4
Signed-off-by: Lancelot SIX <lancelot.six@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 855015377c]
For GFX12, the workgroup id is passed in ttmp9 (trap temp register) instead of the scalar register.
Normal shader code (i.e. not priv, not trap handler) can only read the ttmp registers.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: I42404d8c8c0ee9c746e23879fd30b2d16cfa1787
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 40cc6559f1]
Add timeout to AQLQueue destructor signal wait to prevent indefinite hang
Change-Id: I6c6c98a7bdd27d39569af1d667aa9aa7e9596535
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 4e9647704d]
This reverts commit ef95ccf81e59b8608861e8f2f256d981eee19df7.
Reason for revert: Causing performance regressions on some systems
Change-Id: I82951350cafbd57c495852d6f90023a3373f04f6
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 1cee8656df]
Add target gfx1200 to several files.
Add cases for GFX12 in a few switch statements.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: Ib90032f5b9d5a3306060f13a43d970108a1399df
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 2f14acd9c1]
Generate static package by combining binary and dev components.
Binary and dev component dependencies are added to the static package dependencies
No dependency to rocprofiler-register
Package name will have suffix static-dev/devel
Change-Id: I2f9680f13dbffc9eb7ced9fa9b28e360c47ebcca
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 696d8fae9e]
Add a new driver interface as a core ROCr component.
The driver component provides an interface for ROCr to interact with
agent kernel-model drivers in a generic way. This interface will be used
to interact with the XDNA NPU driver. Eventually, the ROCt library's
functionality should be implemented behind this interface.
For now the interface provides basic queue and memory allocation
for supporting HSA queues and signals and matches the thunk API
closely.
Change-Id: I37ac9f2dcbadc86ce45999f76b0e9ce753fd0c06
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 69ba32fa95]
Create a new top-level CMakeLists.txt file to control building thunk
and ROCr. kfdtest and rocrtest are built separately.
Most of the cmake code that existed for thunk, ROCr, rocrtst and kfdtest
still reside in their respective CMakeLists.txt files, except the
CPack packaging directives which have been moved to the top-level
CMakeLists.txt.
Change-Id: I1a537359029504af8b1abb324bc6f0d75d98471e
[ROCm/ROCR-Runtime commit: 662f6817d7]
Minor instructions changes for GFX12.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: Iab2c430bb5d7d8fa2b166d07fd33ea15aca3a5cd
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 8917561625]
Since PC Sampling is still under experiment, we can't
bump KFD_IOCTL_MINOR_VERSION to enable pc sampling.
KFD_IOCTL_MINOR_VERSION 16 already includes all pc sampling
code, so use version 16 to enable pc sampling implicitly for
customer to try-out this new feature.
Need update the version accordingly when pc sampling upstream.
Change-Id: I65840128f94e8f347c0617971c0aa4b7e478691a
Signed-off-by: James Zhu <James.Zhu@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 338721c24a]
Minor instructions changes for GFX12.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: I57cca6393d4b4aae869a2bc9862d75eef1f29ed7
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 9665499f83]
Minor instructions changes for GFX12.
Change-Id: I78a37fa37950b378cdd2a1618c71c97c6ba66aac
Signed-off-by: David Belanger <david.belanger@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 39f4fbee10]
KFD ioctl version is 1.16 on upstream for contiguous memory support.
Remove pc_sampling version, should be added after pc_sample upstream.
Change-Id: I6e6c3340bc8e371d68dd7741b02578be2fdef801
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 6e6f445f75]
Update amdp2ptest.h to sync with the same file from rdma test driver
folder.
Add ContiguousVRAMAllocation to verify rdma get pages will get
contiguous VRAM pages, skipped RDMA getpages if amdp2ptest.ko is not
loaded.
Change rdma buffer mmap with MAP_SHARED flag, because MAP_PRIVATE goes
to COW path, which requires mmap the entire vma and cannot support
multiple sg nents.
Change-Id: I5fbb1902251f1454616d4404a4b048a88996d4f7
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: e076a4ee82]
mmap system call parameter vma->vm_start, vm_end is mmap virtual address
start, end, vma->vm_pgoff is rdma buffer GPU address, which is used to
find the sg_table dma_address.
Handle multiple sg table nents case because sg->length is limited to max
2GB.
Change-Id: I677dd6662ee58f0b5c93f8eef32b7009e1e890d8
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 9d9fbceefb]
The application may use parent process KFD handle or invalid KFD handle,
add CHECK_KFD_OPEN in all APIs to catch this application bug earlier
without calling to KFD.
Change-Id: I0391e91eeca8e6752fc9c23f0742445b823ea9b0
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: c98a8dc179]
New API to support optional alignment parameter for memory allocations.
The alignment should be larger than or equal to page size and a power
of 2.
Change-Id: Ic3fec43b3c4281f74dd33a57ab4143dcf76e1186
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: a31e84eaef]
Since the amdgpu driver commit 1f4ac94b59aebebf.
https://lore.kernel.org/all/a121a72c-b441-4f42-94a3-4597b7f19e7d@amd.com/T/
gtt and vram are available for compute.
So, the vramSize obtained by function GetSysMemSize is actually about 50% system memory.
But small APUs don't have large system memory, and kernel memory limit is smaller for them.
Therefore, it will fail to register SVM Range for SysBuffer and SysBuffer2.
Example:
System Memory size: 3373M Kernel memory limit:1791M
VRAM Memory Size: 256M GTT Memory Size: 1686M
Signed-off-by: Jesse Zhang <Jesse.Zhang@amd.com>
Change-Id: Ib3826933100ab7b432cb476caaf2d91cc9cdb948
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 1abd02af32]
hsaKmtRegisterMemory* can only register OS allocated userptr.
v2: Apply changes to all hsaKmtRegisterMemory* stuff.(Philip)
v3: Unlock aperture->fmm_mutex to aviod deadlock.
Suggested-by: Felix Kuehling <Felix.Kuehling@amd.com>
Change-Id: I1045af7edb4da8206cb878f64c0176ba4fc59f60
Signed-off-by: Lang Yu <Lang.Yu@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 4844a70d94]
Update CMakeLists.txt to use Thunk pkgconfig.
Add rdma contiguous memory allocation test, to verify if KFD rdma get
pages to pin buffer on contiguous VRAM pages.
Change-Id: I7cc617fc083ce1998c214c327c130f033ce41d6f
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 3f00c88910]
Update Makefile for newer kernel version, and support build with dkms
amdgpu driver. Use symbol_request to get KFD peerdirect interface.
Sync up with KFD peerdirect interface changes, remove the free callback
which is not used any more.
Change-Id: I01d8906d9ffa427a058a26e88e36f6b80e9e22c2
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 88dabfcc04]
Small APUs use same memory allocation approach with APP APU now,
skip these tests as well.
Change-Id: I13c953cc53da071f6f36af0d4a0153a48ea066fe
Signed-off-by: Lang Yu <Lang.Yu@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 5eb2a2660c]
To differentiate discrete and integrated GPU more flexibly in runtime,
this will aid in querying HSA_AMD_MEMORY_PROPERTY_AGENT_IS_APU
and hipDeviceAttributeIntegrated.
Change-Id: Ic8a6c9aea3b4bd19c4d5f6729af7e64c328fc61d
Signed-off-by: Lang Yu <Lang.Yu@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: ae3ede062f]
Add test cases excluded from GFX11 to GFX12 list if they are also not
stable on GFX12.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: Ifeab24f8ea94085250ea86128a3e401479bdb53d
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 24578e10c1]
Minor changes to instructions for GFX12.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: Iac5be900e3755099d83010fb1a2066b4dbb52dda
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: bde8e7a212]
Updated ShaderStore shader (used by CWSR test) for GFX12.
Workgroup ID now pass in a different register.
Minor changes for new scope syntax.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: I6fdabc8b62cba201d7777a736d3d43cfae28ca4c
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: e086c383fe]
New watchpoint exception status bits have to been assign to the first 4 least
significant bits so change test verification mask to check against the
first watch point ID accordingly.
Signed-off-by: Jonathan Kim <jonathan.kim@amd.com>
Change-Id: If83950207ea9f66cd230c23e7386a97b3893c2eb
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 3b842c39f1]
Fix traphandler for KFD debugger testing.
Signed-off-by: Jonathan Kim <jonathan.kim@amd.com>
Change-Id: Ib8f5aac3d1b99e4463ac56b5f6d5dee2c367c447
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: a2e9226784]
Lancelot SIX
Jonathan Kim
Shweta.Khatri
David Yat Sin
David Belanger
Sreekant Somasekharan
Ranjith Ramakrishnan
Tony Gutierrez
Lang Yu
Chris Freehill
James Zhu
Philip Yang
Jesse Zhang
Yiyang Wu