export HSA_CHECK_USERPTR=1 to check user pointers on registration. If
the pointer doesn't point to a valid mapping, there will be a segfault.
Change-Id: I459c0902cbc90338517fbf79678871ebfbe5183b
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 34ddde0c50]
Guard pages help catch out-of-bounds memory accesses by applications
by generating VM faults (GPU) and segfaults (CPU).
Remove address space reservation from scratch aperture. That address
space is managed by the Thunk client. Guard pages would cause Thunk's
address space management to get out of sync with the client's.
Change-Id: I2e5aee2923a90186358cc7b0e131baf547996df6
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 11862b9f61]
Unless HSA_USERPTR_FOR_PAGED_MEM is explicitly set, don't use userptr
for all paged memory. This will also allow us to work around some 4.9
issues, and then we can explicitly set HSA_USERPTR_FOR_PAGED_MEM for
all usage once those issues are resolved.
Change-Id: I25ce22b73ae6e93f1567f2318d9d2b47d4a44e69
[ROCm/ROCR-Runtime commit: c991951288]
Use KFD_IOC_ALLOC_MEM_FLAGS_COHERENT when allocating fine-grained
memory and doorbell BOs so that they will be mapped with MTYPE_UC
on GFX9 hardware.
Change-Id: I51adf45b13105f479e6bcdaf54955b467920ee9a
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: Shaoyun Liu <Shaoyun.Liu@amd.com>
[ROCm/ROCR-Runtime commit: 8cb89b6926]
This avoids unnecessary evictions and failed restores due to the
munmap of userptr BOs that are just about to be freed.
Change-Id: Icf2f0b73991455556a201c54c05ea7e20af80f47
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 74ebfca9f0]
If fork() is called, clear all duplicated data that is invalid in the
child process.
Change-Id: I4e27198060db593c630c6337b7071dfbd0d80b83
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
[ROCm/ROCR-Runtime commit: f1f62d863c]
Implement three new APIs for IPC buffer sharing:
-hsaKmtShareMemory()
-hsaKmtRegisterSharedHandle()
-hsaKmtRegisterSharedHandleToNodes()
Add new ioclts necessary for the above APIs.
Change-Id: Ia2b4d0dc91ec64bff959395d11c0536467404792
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
[ROCm/ROCR-Runtime commit: 559e31d6ff]
A memory region is allowed to be registered multiple times when the memory
is specified by a user pointer. If it's registered with the same user
pointer but with different sizes, it's treated as different instances and
multiple VM objects are created with different GPU address.
Change-Id: I49627111bb5db36d18f1133b252fb62a611f06a4
[ROCm/ROCR-Runtime commit: 2a50ebba98]
Allocate doorbells for dGPUs in the SVM aperture and map them for
GPU access. This is necessary to allow GPU-initiated submissions to
user mode queues.
Depends on new doorbell BO allocation flag in KFD.
Change-Id: I0737bef4a4764bb4a66c43846707ead2108f6601
[ROCm/ROCR-Runtime commit: 2e0a6eb371]
Add vm_find_object_by_userptr_range so QueryPointerInfo can find the
object as well when the pointer is not the starting address but it's
inside the memory range. Also rename vm_find_object_xxx functions to
_by_address and _by_address_range to be consistent.
Change-Id: I5c2b3a05b41493e32b7fd9154665bf078b043606
[ROCm/ROCR-Runtime commit: 4911c91389]
Add CPUVM aperture to keep track of memory allocation that is not known
to GPU driver. Together with GPUVM, this patch adds the pointer attributes
support to APU.
Change-Id: If13f9cf01ff8b9f709b99b66661e7505246adf4c
[ROCm/ROCR-Runtime commit: 19f2676ea7]
Add two pointer attributes APIs:
hsaKmtQueryPointerInfo - allow the user to query the memory information
using a pointer. This pointer can point to any address inside the
range known to HSA.
hsaKmtSetMemoryUserData - allow the user to attach data to a pointer to
add memory tracking information. This pointer must match the start
address of a memory allocation or registration.
TODO: This patch implements support on dGPU. Needs to add APU.
Change-Id: I4711809274248434901f0794f50ebfa13a7371a8
[ROCm/ROCR-Runtime commit: 51e4d27c37]
For APUs, use /proc/cpuinfo to get Marketing name.
Change-Id: I4a17516d26a092683f36631032be00ad44f7e7fe
Signed-off-by: Lan Xiao <Lan.Xiao@amd.com>
[ROCm/ROCR-Runtime commit: df593aa076]
HSA thunk API is currently reporting engineering name to MarketingName
and returning NULL when querying for AMDName.
-Change current name reporting from MarketingName to AMDName.
-Use libpci to get MarketingName
Change-Id: I819a6de7b067a2e724a6695e7d800274b83a71f8
Signed-off-by: Lan Xiao <Lan.Xiao@amd.com>
[ROCm/ROCR-Runtime commit: 9cbbf30be7]
Intermediate size was stored in a 32-bit variable. This resulted in
4GB allocations to fail in KFD due to 0 size. Larger allocations
would allocate the wrong amount of memory.
Change-Id: If19dedf64952f1d2edd813793241e12c0362d220
[ROCm/ROCR-Runtime commit: 82b3fad320]
This option may help debug synchronization or coherency issues
involving the GPU caches. It works only on dGPUs, by changing the
cache policy of the GPUVM default aperture to "cohrent", which is
implemented as non-cached on current dGPU hardware.
Change-Id: I544ac9cc5c0cf1fa5c4e30f67aa42b3b5e44ae67
[ROCm/ROCR-Runtime commit: 06d391c6c9]
The Runtime requested this information so they can tell easily
whether a pointer is part of HSA shared address space or not.
Change-Id: If2041ed34031636677d692bc2dc6625634027ed4
[ROCm/ROCR-Runtime commit: 0ed29f5191]
Lstopo doesn't have system memory mappings at low addresses. Make
sure we leave enough GPUVM address space for kernel allocations
(currently only CWSR) before the start of the user-managed SVM
aperture.
Change-Id: Ic197f7bd5a3cfb150a0da2bfdbc848664e7869be
[ROCm/ROCR-Runtime commit: cac0c08496]
Non-canonical GPUVM aperture doesn't exist on dGPUs. Remove comments
and code that say otherwise.
Fix alignment of GPUVM aperture for gfx801. Requires the same workaround
as gfx802. It's not used for anything on gfx801 yet, but will be soon.
Change-Id: I88607fe7b340081cc0715b85f28fdbf5f1bb0ad7
[ROCm/ROCR-Runtime commit: b837c3e7b0]
Use the object size when freeing address space, instead of the
parameter passed in by the caller. The parameter may be incorrect
due to app or runtime bugs, or when the buffers is an AQL ring
buffer with double mapping workaround.
Change-Id: I00bb31d4520ef969a49d6d5ea723e8a33418acc3
[ROCm/ROCR-Runtime commit: 006f3ee41b]
The alignment performed in vm_find_object_by_address isn't sufficient
because it doesn't take into account the offset from the start of the
page.
This fixes a bug where certain unaligned userpointers and sizes fail
to register correctly.
Change-Id: I17872e264467a619f5e1bedb7e1ed3d994a856bf
[ROCm/ROCR-Runtime commit: 8a0161d6bb]
Use the aligned size of the buffer objects for CPU unmapping in
__fmm_release instead of relying on the unaligned size passed in by
the caller.
Change-Id: If986ec24e9a05d32981549fddbf143221fc40bac
[ROCm/ROCR-Runtime commit: 7a383f9d88]
Allocate SVM address space for the registered memory and use new
userptr support in KFD to create a system memory BO associated with
the given user pointer. Map this BO at the SVM address for CPU
access.
MapMemoryToGPU can be used with the registered user pointer and
will return the SVM address as alternate GPUVA.
Change-Id: I4886e193c51fb6870a567878870c36bf8b5c3748
[ROCm/ROCR-Runtime commit: 85f9efb1a0]
Allocations from GPU nodes will return VRAM, not system memory.
Only non-paged allocation from GPU nodes is supported. System
memory can only be allocated from CPU nodes (usually node 0).
The HostAccess flag is no longer used to distinguish the memory
type. It only indicates, whether the memory is mapped for CPU
access.
Maintain compatibility with broken KfdTests by returning system
memory for paged-memory requested from GPU nodes.
Change-Id: I514defede735f55e6de436f41944125b6f2c4ccf
[ROCm/ROCR-Runtime commit: 887b32fe86]
All tonga page size alignment is done in the memory management
functions in fmm.c. All other code only specifies the minimum
alignment it needs and lets fmm.c handle the HW-specific
alignment.
Clean up aligned-exec memory allocation in queue.c to remove
hard-coded TONGA_PAGE_SIZE alignments and remove code duplication.
Make sure alignments are consistent between allocate and free.
Change-Id: Ia8923448173d1cef315af24cebff12adef385cb0
[ROCm/ROCR-Runtime commit: cc9fc386bd]
Stop using NUM_OF_SUPPORTED_GPUS. For now the definitions itself cannot
be removed as ioctl code is in upstream Kernel.
Change-Id: If846625a8ad5062d5483e762850c793d3c00b9d0
[ROCm/ROCR-Runtime commit: ce83dc623f]
Fix hsaKmtRegisterMemory to be a no-op for now and move the multi-GPU
implementation to hsaKmtRegisterMemoryToNodes. Make GPU memory mappings
of host memory visible to all GPUs by default. Device memory is still
visible to the allocating GPU only by default (but can be overridden
with hsaKmtRegisterMemoryToNodes for experimenting with P2P).
Change-Id: I73408afbe3b10c8dad2ab3a780f58413249692e6
[ROCm/ROCR-Runtime commit: 063ad3ad9e]
This is required when we have a debug session
Change-Id: If9d6d2d23a9016b6ca9562e02a91fc16e0354ee4
Signed-off-by: Yair Shachar <Yair.Shachar@amd.com>
[ROCm/ROCR-Runtime commit: 681f4dcecc]
When the Thunk is initialized multiple times in the lifetime of a single process
, some global resources are leaked. This can happen when dlopen and dlclose are
used to load the library at runtime, rather than linking the runtime against
the Thunk. This patch adds the destructor to release global resources when
dlclose is called.
Change-Id: Ia00da0d41f095d0b2706f98c0e75effedd596f49
[ROCm/ROCR-Runtime commit: 582b70f9c3]
This will also fix out of bound access in functions
fmm_get_aperture_base_and_limit and fmm_release
Change-Id: Icf064c46647e69a069126171dbacdf3d5b27f972
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
[ROCm/ROCR-Runtime commit: a4cf02d797]
dgpu_aperture and dgpu_alt_aperture will be shared by all dGPUs.
Change-Id: I814495e43b51acabdc6266cfa8d83db5a062e20d
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
[ROCm/ROCR-Runtime commit: 2903a610e1]
Break from the for-loop once dgpu VM range is found, otherwise the
length is reduced by half
Change-Id: Ie602054c16ea69ea1cbb75e804ead551bc3615c0
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
[ROCm/ROCR-Runtime commit: 5a55383baf]
Felix Kuehling
Harish Kasiviswanathan
Kent Russell
Amber Lin
Sean Keely
Lan Xiao
Ben Goz
shaoyunl
Yair Shachar