New API to support alignment parameter when reserving virtual addresses.
If the alignment is 0, then the default size is used. Otherwise the
alignment needs to be a power of 2 and greater than or equal to page
size.
Existing hsa_amd_vmem_address_reserve marked for future deprecation.
Change-Id: I17cee75420183dea5842fc1ecc2514cdcd760bac
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 08c44fbda6]
The elf libraries are installed in /usr/lib64 in RHEL.
Removed invalid paths
Change-Id: I8c2b5525c1e3b62a2bd4e31a442d9931005c2f30
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 14ed20e0cc]
This is to fix the situation where libhsa-runtime exists in
/usr/lib. The preinstall script will check for this and ask
the user if they want to delete the old version, or else
abandon install.
Change-Id: I0976b6ec95b9752c95031f1a73fc49a150b02b23
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 4474dcff2c]
The current trap handler defined:
.set SQ_WAVE_EXCP_FLAG_USER_MATH_EXCP_SHIFT , 0
.set SQ_WAVE_EXCP_FLAG_USER_MATH_EXCP_SIZE , 6
.set SQ_WAVE_TRAP_CTRL_MATH_EXCP_SHIFT , 0
.set SQ_WAVE_TRAP_CTRL_MATH_EXCP_SIZE , 6
However, the ALU exception in EXCP_FLAG_USER go from bit 0 (alu_invalid)
to bit 6 (alu_int_div0), making it a total of 7 bits, not 6. Similarly,
the corresponding bits in TRAP_CTRL go from bit 0 to 6 as well.
Fix the incorrect size to be sure to properly detect the int_div0
exception.
Change-Id: I60c2d94a447b71ca0ce26a87b7f55b055b9aef8e
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: cb8705627f]
This patch is to remove duplicated definition of GFX1150.
Change-Id: I4a8b8bce5c2721748c4d64e1da13b59feae2139a
Signed-off-by: Yifan Zhang <yifan1.zhang@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 1d1a32d725]
This avoids conflicts in case application is loading another copy of
addrlib.
Change-Id: Ifb4a10270c867366d5eed0a8c015257b415189a5
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: f1a13b6d87]
This reverts commit b156e906d9c192bd487d10a8900e3eb6090ef547.
Reason for revert: Memory violation test causing a timeout in subsequent test.
Change-Id: If3a217575af545a47d6d67bebba4a2c640a43b81
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 2e1f363d2f]
The value of STATE_PRIV is captured by the 1st level trap handler, and
passed on to the second level trap handler. The value is to be restored
before exit. However it is possible for the value of
STATE_PRIV.BARRIER_COMPLETE to change while the wave is in the trap
handler (all the other waves in the workgroup has signaled the
work-gropu barrier), and in this case restoring STATE_PRIV in full would
result in STATE_PRIV.BARRIER_COMPLETE to be cleared.
Restore every bits of STATE_PRIV except for BARRIER_COMPLETE before
return to prevent this race.
Change-Id: I76c875bced7d23c58670b28f257d22c933f99fc5
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 9e625307d2]
GFX94x runs into performance regression when doing large packet
enqueues.
Drop back to legacy packet sizes for now.
Change-Id: I595838ebada66c6c5143bfdb2f56c83ee71654a9
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: b8aae52404]
Removing extra bits set in forbiddenBlock that seemed to be set for
debugging and are causing unexpected image formats to be used.
Change-Id: I29c9e319907027a2b0b6bf7c1c0c8558eb6a36f4
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: e721eb509b]
Packet for GFX12 is incompatible with pre-GFX12 as some fields changed
location. Implement code path and packet specific to GFX12.
This fixes some issues with SDMA blits and 3D images.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: I56c204aaa12160e563ec960bd3b226cfa94e142d
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 6d147dd3b1]
Add new files image_manager_gfx12.{h,cpp}.
Implement BUF/IMG/SAMP desc changes for GFX12.
Implement compute surface info code using AddrLib3 API (new starting
from GFX12).
Implement algorithm for choosing "best" swizzle mode (starting
from AddrLib3/GFX12, AddrLib provides only list of suitable swizzle mode,
up to client, ROCr, to choose the best). Algorithm implemented follows
behaviour in GFX11 and behaviour for GFX12 on other platforms.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: Ib344c86228a98bbac5acdab421ee2ef9b1e84eef
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: f8a015f53e]
Added GFX12 implementation for InitScratchSRD and for compute_tmpring.
Implementation for compute_tmpring could be combined with GFX11 with some
refactoring as a possible future improvement.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: I8013cbe4438786bf41bbfd03f6a5d3b9ef51e7bf
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: def4a6c326]
Updated struct definitions, field size changes and new fields in
registers.h.
Added resource_gfx12.h and updated fields in BUF/IMG/SAMP descriptor
structs based on documentation.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: I08f05ba30f54c40e7b823a6a105829a1e8590b3d
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 8165da63cc]
Do not allow extended-scope fine-grain memory on gfx120x devices.
Change-Id: I1e6e6c1860de00160cca9d8137b129c7e32c0526
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 7dd90f8361]
There is an issue in the gfx12 trap handler where the EXCP_FLAG_PRIV
is only fetched under certain conditions (trap_id != 0) while it should
have been fetched unconditionally. As a consequence, the interrupt
payload might contain invalid data, leading to incorrect exceptions
being reported by the runtime. Debugger is mostly un-affected as it
will inspect the wave's state to figure out what exception(s) have been
reported for each wave.
Also, it is not necessary to check for the host trap bit if trap_id is
!= 0 in gfx12, there is on trap ID anymore for host trap.
This patch implements those fixes.
Co-Authored-By: Laurent Morichetti <laurent.morichetti@amd.com>
Change-Id: Ib72cd8cc5d935ca643e241da7fccd3f96201b09d
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 7a3bf30769]
The constant declarations in trap_handler_gfx12.s have been sorted
alphabetically, which causes inconsistencies. Fix the order of
declarations where it makes sense.
Change-Id: I5b05d87a5afbe1ff3362746801a1c9373537b49e
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: ff9b11fd89]
Given the differences between previous architectures and gfx12, this
patch implements the gfx12 2nd level trap handler in a separate source
file, and adjusts the build system.
Change-Id: I65192ffbbcd66a4f78d2d0c3fb1739a92cac95d4
Signed-off-by: Lancelot SIX <lancelot.six@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 855015377c]
For GFX12, the workgroup id is passed in ttmp9 (trap temp register) instead of the scalar register.
Normal shader code (i.e. not priv, not trap handler) can only read the ttmp registers.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: I42404d8c8c0ee9c746e23879fd30b2d16cfa1787
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 40cc6559f1]
Add timeout to AQLQueue destructor signal wait to prevent indefinite hang
Change-Id: I6c6c98a7bdd27d39569af1d667aa9aa7e9596535
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 4e9647704d]
This reverts commit ef95ccf81e59b8608861e8f2f256d981eee19df7.
Reason for revert: Causing performance regressions on some systems
Change-Id: I82951350cafbd57c495852d6f90023a3373f04f6
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 1cee8656df]
Add target gfx1200 to several files.
Add cases for GFX12 in a few switch statements.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: Ib90032f5b9d5a3306060f13a43d970108a1399df
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 2f14acd9c1]
Generate static package by combining binary and dev components.
Binary and dev component dependencies are added to the static package dependencies
No dependency to rocprofiler-register
Package name will have suffix static-dev/devel
Change-Id: I2f9680f13dbffc9eb7ced9fa9b28e360c47ebcca
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 696d8fae9e]
Add a new driver interface as a core ROCr component.
The driver component provides an interface for ROCr to interact with
agent kernel-model drivers in a generic way. This interface will be used
to interact with the XDNA NPU driver. Eventually, the ROCt library's
functionality should be implemented behind this interface.
For now the interface provides basic queue and memory allocation
for supporting HSA queues and signals and matches the thunk API
closely.
Change-Id: I37ac9f2dcbadc86ce45999f76b0e9ce753fd0c06
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 69ba32fa95]
Create a new top-level CMakeLists.txt file to control building thunk
and ROCr. kfdtest and rocrtest are built separately.
Most of the cmake code that existed for thunk, ROCr, rocrtst and kfdtest
still reside in their respective CMakeLists.txt files, except the
CPack packaging directives which have been moved to the top-level
CMakeLists.txt.
Change-Id: I1a537359029504af8b1abb324bc6f0d75d98471e
[ROCm/ROCR-Runtime commit: 662f6817d7]
Minor instructions changes for GFX12.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: Iab2c430bb5d7d8fa2b166d07fd33ea15aca3a5cd
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 8917561625]
Since PC Sampling is still under experiment, we can't
bump KFD_IOCTL_MINOR_VERSION to enable pc sampling.
KFD_IOCTL_MINOR_VERSION 16 already includes all pc sampling
code, so use version 16 to enable pc sampling implicitly for
customer to try-out this new feature.
Need update the version accordingly when pc sampling upstream.
Change-Id: I65840128f94e8f347c0617971c0aa4b7e478691a
Signed-off-by: James Zhu <James.Zhu@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 338721c24a]
Minor instructions changes for GFX12.
Signed-off-by: David Belanger <david.belanger@amd.com>
Change-Id: I57cca6393d4b4aae869a2bc9862d75eef1f29ed7
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 9665499f83]
Minor instructions changes for GFX12.
Change-Id: I78a37fa37950b378cdd2a1618c71c97c6ba66aac
Signed-off-by: David Belanger <david.belanger@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 39f4fbee10]
KFD ioctl version is 1.16 on upstream for contiguous memory support.
Remove pc_sampling version, should be added after pc_sample upstream.
Change-Id: I6e6c3340bc8e371d68dd7741b02578be2fdef801
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 6e6f445f75]
Update amdp2ptest.h to sync with the same file from rdma test driver
folder.
Add ContiguousVRAMAllocation to verify rdma get pages will get
contiguous VRAM pages, skipped RDMA getpages if amdp2ptest.ko is not
loaded.
Change rdma buffer mmap with MAP_SHARED flag, because MAP_PRIVATE goes
to COW path, which requires mmap the entire vma and cannot support
multiple sg nents.
Change-Id: I5fbb1902251f1454616d4404a4b048a88996d4f7
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: e076a4ee82]
mmap system call parameter vma->vm_start, vm_end is mmap virtual address
start, end, vma->vm_pgoff is rdma buffer GPU address, which is used to
find the sg_table dma_address.
Handle multiple sg table nents case because sg->length is limited to max
2GB.
Change-Id: I677dd6662ee58f0b5c93f8eef32b7009e1e890d8
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 9d9fbceefb]
The application may use parent process KFD handle or invalid KFD handle,
add CHECK_KFD_OPEN in all APIs to catch this application bug earlier
without calling to KFD.
Change-Id: I0391e91eeca8e6752fc9c23f0742445b823ea9b0
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: c98a8dc179]
New API to support optional alignment parameter for memory allocations.
The alignment should be larger than or equal to page size and a power
of 2.
Change-Id: Ic3fec43b3c4281f74dd33a57ab4143dcf76e1186
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: a31e84eaef]
Since the amdgpu driver commit 1f4ac94b59aebebf.
https://lore.kernel.org/all/a121a72c-b441-4f42-94a3-4597b7f19e7d@amd.com/T/
gtt and vram are available for compute.
So, the vramSize obtained by function GetSysMemSize is actually about 50% system memory.
But small APUs don't have large system memory, and kernel memory limit is smaller for them.
Therefore, it will fail to register SVM Range for SysBuffer and SysBuffer2.
Example:
System Memory size: 3373M Kernel memory limit:1791M
VRAM Memory Size: 256M GTT Memory Size: 1686M
Signed-off-by: Jesse Zhang <Jesse.Zhang@amd.com>
Change-Id: Ib3826933100ab7b432cb476caaf2d91cc9cdb948
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 1abd02af32]
hsaKmtRegisterMemory* can only register OS allocated userptr.
v2: Apply changes to all hsaKmtRegisterMemory* stuff.(Philip)
v3: Unlock aperture->fmm_mutex to aviod deadlock.
Suggested-by: Felix Kuehling <Felix.Kuehling@amd.com>
Change-Id: I1045af7edb4da8206cb878f64c0176ba4fc59f60
Signed-off-by: Lang Yu <Lang.Yu@amd.com>
Signed-off-by: Chris Freehill <cfreehil@amd.com>
[ROCm/ROCR-Runtime commit: 4844a70d94]
Chris Freehill
David Yat Sin
AravindanC
Yifan Zhang
Ranjith Ramakrishnan
Vladimir Indic
Lancelot SIX
Shweta Khatri
Jonathan Kim
David Belanger
Sreekant Somasekharan
Tony Gutierrez
Lang Yu
James Zhu
Philip Yang
Jesse Zhang
Yiyang Wu