The NULL pointer check was the only way for that function to fail. And it
was done after the pointer was accessed. Simplify this by just returning
the result as a return value instead of using a pointer as output
parameter. This way the function can never fail and the caller doesn't
need to do any error handling.
Declare the function in libhsakmt.h instead of duplicating the
declaration in fmm.c.
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Change-Id: I91b90d66166fd3b5cdc47c73a9bbc369c45b51fe
[ROCm/ROCR-Runtime commit: 2d53430ce3]
Setting this variable to '0' will force to disable memory
registration/allocation through SVM API mechanism.
Not setting this or setting to '1', SVM API will be used only if all
GPUs support it.
Signed-off-by: Alex Sierra <Alex.Sierra@amd.com>
Change-Id: Icdf7656de09aa9988b567ec6c024953398e9bb48
[ROCm/ROCR-Runtime commit: 8a746bdaed]
Register and map userptrs through Shared Virtual Memory(SVM) API at
the Kernel level when available. Using this approach, performance
will be improve as register/unregister memory will not trigger any
system call to KFD driver.
Signed-off-by: Alex Sierra <alex.sierra@amd.com>
Change-Id: I20723cbeb340bf48b95e1115f0102c031397bc14
[ROCm/ROCR-Runtime commit: 178a619b80]
fork process copy-on-write MMU nitifier on CWSR range will evict user
queues, and then update GPU mapping and resume queues, use MADV_DONTFORK
to avoid COW MMU notifier callback on CWSR SVM range.
Use mmap to alloc SVM range for CWSR because posix_memalign don't alloc
new range in child process, this fails to register svm range as range is
invalid address in forked child process.
Change-Id: Ibaea56a691dd6f577ed2e1f2d43f4a3500b8316f
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 093cf898fb]
mmap alloc larger address range with align padding page plus guard
pages, then unmap the padding and guard pages at beginning and end
of the range, return aligned address range.
Change-Id: Iaf3c711a079c744289efbafee9b5e63aaf724765
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: b7710a1dda]
Avoiding segfault when an invalid SharedMemoryHandle is passed in
when calling fmm_register_shared_memory.
Change-Id: I0e0bbed01487fc10afcbb170eb9330e70b209d14
Signed-off-by: David Yat Sin <David.YatSin@amd.com>
[ROCm/ROCR-Runtime commit: 1c385fb257]
hsaKmtMapMemoryToGPU should not try to map VRAM on peer GPUs that don't
have an IO-Link to the memory. The new P2P mapping code in KFD will
fail otherwise.
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Change-Id: I6d59b55651b98756865a0f69eafef3e386372cf3
[ROCm/ROCR-Runtime commit: 9ac2c75171]
This allows init_process_apertures to use the whole consistent topoology
instead of taking its own partial snapshot.
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Change-Id: Ia13e7aa7fcd090ea8d6cacd4babb29a27c20207f
[ROCm/ROCR-Runtime commit: 87aca673e8]
Add KFDGpuID to HsaNodeProperties to return gpu_id to upper layer,
gpu_id is hash ID generated by KFD to distinguish GPUs on the system.
ROCr and ROCProfiler will use gpu_id to analyze SMI event message.
Change-Id: I6eabe6849230e04120674f5bc55e6ea254a532d6
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 88778e13dc]
This env variable sets the max VA alignment order size as
"PAGE_SIZE * 2^alignment order" during mapping. By default the order
size is set to 9(2MB).
Signed-off-by: Alex Sierra <alex.sierra@amd.com>
Change-Id: I01ae4e0963f4d21c7c367464e60f865bc58d7fac
[ROCm/ROCR-Runtime commit: 0e908f05bb]
Tested on Talos II with Vega 64
POWER systems allocate NUMA nodes on multiples of 8 to allow CPU
onlining / offlining
Set the correct NUMA mask bits when requesting node-bound memory
allocations
This is a cleanup/squash/rebase of:
https://github.com/RadeonOpenCompute/ROCT-Thunk-Interface/pull/47
Change-Id: Id4af6dff7e66e9d464d6b17a1e99087eb3ac8e51
Signed-off-by: Jeremy Newton <Jeremy.Newton@amd.com>
[ROCm/ROCR-Runtime commit: 5fd3c868b2]
Mapped memory areas become invalid after fork, and the child process is
required to remap the memory areas after a fork. So we mark these device
memory mappings with MADV_DONTFORK so that they are removed from the
child process after fork.
This was causing some issues when doing CRIU checkpoint/restore because
CRIU and amdgpu_plugin were not able to handle these mappings.
Change-Id: I50eb334aecea6dab7522d94da0273adcf4fb1ce0
Signed-off-by: David Yat Sin <david.yatsin@amd.com>
[ROCm/ROCR-Runtime commit: 4986f4a5c2]
For userptr, after taking aperture lock, decrease registration_count and
ensure object registration_count equal to 0 to release KFD and thunk
object.
Move decrementing of registration count from fmm_deregister_memory into
__fmm_release to avoid a race condition when dropping the aperture lock
in fmm_deregister_memory.
Change-Id: I5381fa6b8a77a1516af2554e5174e91969c338c4
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: f82927ad65]
After calling ioctl to create userptr obj, take aperture lock, check if
there is same userptr obj created after finding object, to catch the
race that multiple threads register same userptr to multiple GPUs.
If same userptr obj exist, then increase userptr registeration_count,
and free the newly create obj.
Change-Id: I63ae3a4f54da8aedd11c124d8d53ebe727b8203a
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 8f7443494a]
This reverts commit f94be2078c.
Change-Id: I954326d9a291280de7c9b7ef49a8cce0cd9dd9ad
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: a809a5bf32]
This reverts commit 559bb50c6a.
Change-Id: Ibd32ac36bcb9ee418957d25b240ef9dd9172fbd3
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 91b0fce535]
This reverts commit 14d821391e.
Change-Id: Idf636655ac01f569d8e92e9c0e100e5db7d1cb61
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 445fcd803b]
This reverts commit 8f735fb9d5.
Reason for revert: aqlprofile has merged
Change-Id: Iee0fbe8ac1e475123273b328949f68f10490d3ca
[ROCm/ROCR-Runtime commit: 268bb1ae84]
g_props is intialized in topology_take_snapshot which needs to call
hsaKmtAcquireSystemProperties. In hsaKmtOpenKFD, it doesn't call hsaKmtAcquireSystemProperties.
So it needs to change parameter of topology_is_svm_needed from node_id
to EngineId to avoid Segmentation fault on gfx902.
Signed-off-by: changzhu <Changfeng.Zhu@amd.com>
Change-Id: Iba8d20a9142510c70927454d26bdcaf579ad5574
[ROCm/ROCR-Runtime commit: 978028d3f6]
Use the database provided by libdrm-amdgpu,as this is maintained by AMD
Signed-off-by: Mike Li <Tianxinmike.Li@amd.com>
Change-Id: I319a46d833cb23173a77fdff1deae69ea58588b8
[ROCm/ROCR-Runtime commit: cb1c84d99b]
hsaKmtQueryPointerInfo return vm_obj flags for all below registered
memory types other than hsaKmtAllocMemory, and set the CoarseGrain flag
correctly for:
Graphics: always coarse grain.
Shared: hsaKmtShareMemory pass mflags with export handle to KFD to store
in KFD objs, hsaKmtRegisterSharedHandle get mflags from KFD with import
handle.
Userptr: it is already coarse-grain by default, or based on mflags
provided in hsaKmtRegisterMemoryWithFlags call.
Change-Id: Idc23e8b0cf599b02580737639da2f9ef4ccd0c0d
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: fa7b4a6268]
Query pointer info returns KFD_IOC_ALLOC_MEM_FLAGS_* flags, it should
return HsaMemFlags, fix it by renaming vm_obj->flags to mflags and
always saving HsaMemFlags.
Use consistent function parameter and variable name to avoid confusion:
mflags for HsaMemFlags and ioc_flags for KFD_IOC_ALLOC_MEM_FLAGS_*
flags.
AMDKFD_IOC_GET_DMABUF_INFO return ioc_flags, translate it to mflags
using new helper fmm_translate_ioc_to_hsa_flags.
Change-Id: If9e117c507139c0166abb1ab0df8c233ef7e48a1
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 2c796e62be]
__fmm_release is sometimes called with the aperture lock, and sometimes
without. Consistently call it with the aperture lock held and remove the
lock/unlock calls from this function.
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Change-Id: I80dddc64cc0703e5eed8e9f1eb65b75a2c7ae2eb
[ROCm/ROCR-Runtime commit: 5fac7dcc3b]
Unlock mutex if MMIO mapping fails. This happens on all GFXv8 GPUs.
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Change-Id: I1dee1cbddefd9185c24ea79377f49f8ae2c5ff57
[ROCm/ROCR-Runtime commit: 19536080a8]
Aperture locking is too fine-grained, it has race between find userptr
and allocate userptr object.
Change _fmm_allocate_device and fmm_allocate_memory_object to not take
the aperture lock, the callers take it, this implements an atomic find
userptr or allocate a new one.
Change-Id: I6773404e22c1f4382a211c5a9817df23c5534a2a
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: c4d5ee28f0]
it is to optimize memory allocation latency, which
changes alignment from 2MB to 1GB.
Signed-off-by: Alex Sierra <alex.sierra@amd.com>
Signed-off-by: Eric Huang <jinhuieric.huang@amd.com>
Change-Id: I7818e9f13b17e2c0992e75b17f978dc03a018a57
[ROCm/ROCR-Runtime commit: 973b35bc06]
Set the KFD_IOC_ALLOC_MEM_FLAGS_COHERENT flag and
KFD_IOC_ALLOC_MEM_FLAGS_UNCACHED flag to allocate
uncached coherent memory when HSA_DISABLE_CACHE
environment variable is set. At KFD driver,
Single KFD_IOC_ALLOC_MEM_FLAGS_UNCACHED flag is
not sufficient to allocate uncached memory. We
have to use both two flags to allocate uncached
memory.
Change-Id: Ie490f37b2e696314e60048f5b1b57442431696e9
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
[ROCm/ROCR-Runtime commit: ae0e74095e]
Before gfx90a, coherent memory is uncached. So it was reasonable
when environment variable HSA_DISABLE_CACHE is set, memory is mapped
as coherent. On gfx90a, coherent memory can be cached, so mapping
memory as coherent can't guarantee memory is uncached. When
HSA_DISABLE_CACHE is set, we have to map memory as uncached.
Change-Id: Ia5ed4cf0ad6aef5644dc8c9e6632b52d606f06f4
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
[ROCm/ROCR-Runtime commit: f132fb2cd0]
This change might be redundant if ROCr takes care of it
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Change-Id: I7b67143a8ad21baa61b7eda7b8e5fe0ac1e33830
[ROCm/ROCR-Runtime commit: 10674916e4]
It is to provide an option to map specific memory as
uncached on A+A HW platform.
Signed-off-by: Eric Huang <JinhuiEric.Huang@amd.com>
Change-Id: Ib665cb306a0e78aba3ea5ee2f0e46cb62ae139f8
[ROCm/ROCR-Runtime commit: 2464bfc714]
1. Create P2P links
2. Determine FRAMEBUFFER_PUBLIC/PRIVATE only based
host-accessibility, not peer-accesssibility
Signed-off-by: Gang Ba <gaba@amd.com>
Change-Id: I15fccdc60386b453e2a47849a16df15157324b21
[ROCm/ROCR-Runtime commit: bedecc5957]
This reverts commit 35b07e1e28.
Reason for revert: This commit caused a regression rocrtst memory
subtest: Maximum Single Allocation in Memory Pools failed.
Change-Id: I15330625603f893200a08cd8b5b097f9bf95361f
[ROCm/ROCR-Runtime commit: e515fd818b]
Don't update the vm_object if GPU mapping failed. Print an error message
to help diagnose underlying problems.
Change-Id: I801ab6fe6c155bd25e6c0358007c106a4a019480
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: c7e6f5a274]
Use MAP_POPULATE when allocating anonymous system memory for later
GPU mapping as a userptr. This can speed up large allocations by
more than factor 2. I suspect populating pages in this way is more
efficient than the CPU page fault code path triggered by
get_user_pages in the kernel.
Change-Id: I188bbc1462ccb650d48cbfb1080dbb8eb7ada8b5
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 8f26c0c40c]
KFD already implemented the fallback path for APU. Thunk will use flag
which exposed by kfd to configure is_dgpu instead of hardcode before.
Signed-off-by: Huang Rui <ray.huang@amd.com>
Change-Id: I445f6cf668f9484dd06cd9ae1bb3cfe7428ec7eb
[ROCm/ROCR-Runtime commit: 12813691a2]
This is needed to avoid additional references to mapped BOs in child
processes that can prevent freeing memory in the parent process and lead
to out-of-memory conditions.
Change-Id: I25c90510a14dde515cc23ea5dc1f68e8d7e37a66
Signed-off-by: Philip Cox <Philip.Cox@amd.com>
[ROCm/ROCR-Runtime commit: f7a3427c99]
The option to use kfd_fd for cpu mapping is for very old broken KFD
version, it is not used in upstreaming process. This causes issue when
multiple process uses shared system memory because the GTT address is
over 40 bits.
Change to always use render node fd to create CPU mapping.
Change-Id: Id7e7b2a2e2f13c6e62c5de170589abfff4d456b0
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 9e9771a7d9]
This is needed to allow gdb to access the memory.
Change-Id: I96c084b714e952d7b7000f0dd41e1c530fdd092f
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: e5062c4383]
libpci was only used to find a marketing string for a device.
This patch looks for a pci.ids on disk and parses it to extract the
same string, using 'Device xxxx' as the fallback on file i/o error
or missing data from the text file. Tested by checking every vendor/
device pair against the values returned from libpci.
Change-Id: I21af3157472c1824d57fcee31393c6ee8ce07330
Signed-off-by: Jon Chesterfield <Jonathan.Chesterfield@amd.com>
[ROCm/ROCR-Runtime commit: 0a1718b753]
IPC memory was previously returned as HSA_POINTER_ALLOCATED and
had garbage in the node_id field. Due to ROCR_VISIBLE_DEVICES
we need to be able to distinguish between imported memory and
regular memory because imported memory may not be owned by an
agent that is visible in the process. Differentiating these flags
allows the users to expect null agent for the owning agent.
Fixes
Change-Id: Ide3489cec1ee2072dc9697fa5cb71ddb17999d14
[ROCm/ROCR-Runtime commit: 6957202df8]
From Thunk spec, flag NoSubstitute = 0, if specific memory type is not
available on node, allocation may fall back to other memory that can
replace it on that node. mbind return failure if no memory available on
the specific node, we should ignore the mbind failure for this case.
Change-Id: I651a1bedf1852330604e56965cc17862403ebf87
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 5858aa17a9]
Check for errno == EBADF in kmtIoctl to detect misuse of the kfd_fd
in a forked child process.
Detect being in a forked child process pro-actively by implementing
a pthread_atfork callback.
Make sure all mutexes get reinitialized in the child process to avoid
deadlocks.
Check for being in a forked child process in CHECK_KFD_OPENED so that
all hsaKmt functions will return the appropriate status
HSAKMT_STATUS_KERNEL_IO_CHANNEL_NOT_OPENED.
Update InvalidKFDHandleTest to expect that error code.
Change-Id: I0238e5fba344dcaa454e97a35db2e2dcc8d1f607
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 87e10cd0b4]
On NUMA system, allocate queue ctx save restore area on the closest NUMA
node to the GPU which the queue is going to run. This will improve
performance on NUMA system generally by reducing schedule latency and
fix the multi-node rccl-tests unstable performance issue.
If the closest NUMA node has no memory available, set flags NoNUMABind=1
to bypass mbind, to use default NUMA memory policy to allocate system
memory.
Change-Id: Ic62bfa5bb2efbf4f6ae79ff403e9610ddf18d45c
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 59c857476f]
FREE_MEMORY_OF_GPU ioctl could fail, e.g., if memory is still mapped
to GPU. Handle this failure by return error in fmm_release/HsaKmtFreeMemory
Change-Id: I5461db39964f733cf97376d50e44906a9b4c0f13
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
[ROCm/ROCR-Runtime commit: fa0cb9ebeb]
Without this change, the failure was hard to notice when it happened.
Change-Id: I99c3e8cea0d0cbd3bcfe79069410e6e870e225bf
Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
[ROCm/ROCR-Runtime commit: 16fa78b134]
numa_max_node() return the highest node number available on the current
system, number of NUMA nodes should be numa_max_node() + 1.
Change-Id: I20a6c17af071e73e853cb5ea6d0304c8aca52681
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 71cf3cf5d3]
Felix Kuehling
Alex Sierra
Philip Yang
David Yat Sin
Timothy Pearson
Kent Russell
changzhu
Mike Li
Graham Sider
Oak Zeng
Harish Kasiviswanathan
Eric Huang
Gang Ba
Huang Rui
Philip Cox
Jon Chesterfield
Sean Keely
Yong Zhao