This env variable sets the max VA alignment order size as
"PAGE_SIZE * 2^alignment order" during mapping. By default the order
size is set to 9(2MB).
Signed-off-by: Alex Sierra <alex.sierra@amd.com>
Change-Id: I01ae4e0963f4d21c7c367464e60f865bc58d7fac
Tested on Talos II with Vega 64
POWER systems allocate NUMA nodes on multiples of 8 to allow CPU
onlining / offlining
Set the correct NUMA mask bits when requesting node-bound memory
allocations
This is a cleanup/squash/rebase of:
https://github.com/RadeonOpenCompute/ROCT-Thunk-Interface/pull/47
Change-Id: Id4af6dff7e66e9d464d6b17a1e99087eb3ac8e51
Signed-off-by: Jeremy Newton <Jeremy.Newton@amd.com>
Mapped memory areas become invalid after fork, and the child process is
required to remap the memory areas after a fork. So we mark these device
memory mappings with MADV_DONTFORK so that they are removed from the
child process after fork.
This was causing some issues when doing CRIU checkpoint/restore because
CRIU and amdgpu_plugin were not able to handle these mappings.
Change-Id: I50eb334aecea6dab7522d94da0273adcf4fb1ce0
Signed-off-by: David Yat Sin <david.yatsin@amd.com>
For userptr, after taking aperture lock, decrease registration_count and
ensure object registration_count equal to 0 to release KFD and thunk
object.
Move decrementing of registration count from fmm_deregister_memory into
__fmm_release to avoid a race condition when dropping the aperture lock
in fmm_deregister_memory.
Change-Id: I5381fa6b8a77a1516af2554e5174e91969c338c4
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
After calling ioctl to create userptr obj, take aperture lock, check if
there is same userptr obj created after finding object, to catch the
race that multiple threads register same userptr to multiple GPUs.
If same userptr obj exist, then increase userptr registeration_count,
and free the newly create obj.
Change-Id: I63ae3a4f54da8aedd11c124d8d53ebe727b8203a
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
g_props is intialized in topology_take_snapshot which needs to call
hsaKmtAcquireSystemProperties. In hsaKmtOpenKFD, it doesn't call hsaKmtAcquireSystemProperties.
So it needs to change parameter of topology_is_svm_needed from node_id
to EngineId to avoid Segmentation fault on gfx902.
Signed-off-by: changzhu <Changfeng.Zhu@amd.com>
Change-Id: Iba8d20a9142510c70927454d26bdcaf579ad5574
Use the database provided by libdrm-amdgpu,as this is maintained by AMD
Signed-off-by: Mike Li <Tianxinmike.Li@amd.com>
Change-Id: I319a46d833cb23173a77fdff1deae69ea58588b8
hsaKmtQueryPointerInfo return vm_obj flags for all below registered
memory types other than hsaKmtAllocMemory, and set the CoarseGrain flag
correctly for:
Graphics: always coarse grain.
Shared: hsaKmtShareMemory pass mflags with export handle to KFD to store
in KFD objs, hsaKmtRegisterSharedHandle get mflags from KFD with import
handle.
Userptr: it is already coarse-grain by default, or based on mflags
provided in hsaKmtRegisterMemoryWithFlags call.
Change-Id: Idc23e8b0cf599b02580737639da2f9ef4ccd0c0d
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Query pointer info returns KFD_IOC_ALLOC_MEM_FLAGS_* flags, it should
return HsaMemFlags, fix it by renaming vm_obj->flags to mflags and
always saving HsaMemFlags.
Use consistent function parameter and variable name to avoid confusion:
mflags for HsaMemFlags and ioc_flags for KFD_IOC_ALLOC_MEM_FLAGS_*
flags.
AMDKFD_IOC_GET_DMABUF_INFO return ioc_flags, translate it to mflags
using new helper fmm_translate_ioc_to_hsa_flags.
Change-Id: If9e117c507139c0166abb1ab0df8c233ef7e48a1
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
__fmm_release is sometimes called with the aperture lock, and sometimes
without. Consistently call it with the aperture lock held and remove the
lock/unlock calls from this function.
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Change-Id: I80dddc64cc0703e5eed8e9f1eb65b75a2c7ae2eb
Unlock mutex if MMIO mapping fails. This happens on all GFXv8 GPUs.
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Change-Id: I1dee1cbddefd9185c24ea79377f49f8ae2c5ff57
Aperture locking is too fine-grained, it has race between find userptr
and allocate userptr object.
Change _fmm_allocate_device and fmm_allocate_memory_object to not take
the aperture lock, the callers take it, this implements an atomic find
userptr or allocate a new one.
Change-Id: I6773404e22c1f4382a211c5a9817df23c5534a2a
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
it is to optimize memory allocation latency, which
changes alignment from 2MB to 1GB.
Signed-off-by: Alex Sierra <alex.sierra@amd.com>
Signed-off-by: Eric Huang <jinhuieric.huang@amd.com>
Change-Id: I7818e9f13b17e2c0992e75b17f978dc03a018a57
Set the KFD_IOC_ALLOC_MEM_FLAGS_COHERENT flag and
KFD_IOC_ALLOC_MEM_FLAGS_UNCACHED flag to allocate
uncached coherent memory when HSA_DISABLE_CACHE
environment variable is set. At KFD driver,
Single KFD_IOC_ALLOC_MEM_FLAGS_UNCACHED flag is
not sufficient to allocate uncached memory. We
have to use both two flags to allocate uncached
memory.
Change-Id: Ie490f37b2e696314e60048f5b1b57442431696e9
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
Before gfx90a, coherent memory is uncached. So it was reasonable
when environment variable HSA_DISABLE_CACHE is set, memory is mapped
as coherent. On gfx90a, coherent memory can be cached, so mapping
memory as coherent can't guarantee memory is uncached. When
HSA_DISABLE_CACHE is set, we have to map memory as uncached.
Change-Id: Ia5ed4cf0ad6aef5644dc8c9e6632b52d606f06f4
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
This change might be redundant if ROCr takes care of it
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Change-Id: I7b67143a8ad21baa61b7eda7b8e5fe0ac1e33830
It is to provide an option to map specific memory as
uncached on A+A HW platform.
Signed-off-by: Eric Huang <JinhuiEric.Huang@amd.com>
Change-Id: Ib665cb306a0e78aba3ea5ee2f0e46cb62ae139f8
1. Create P2P links
2. Determine FRAMEBUFFER_PUBLIC/PRIVATE only based
host-accessibility, not peer-accesssibility
Signed-off-by: Gang Ba <gaba@amd.com>
Change-Id: I15fccdc60386b453e2a47849a16df15157324b21
This reverts commit 8f26c0c40c.
Reason for revert: This commit caused a regression rocrtst memory
subtest: Maximum Single Allocation in Memory Pools failed.
Change-Id: I15330625603f893200a08cd8b5b097f9bf95361f
Don't update the vm_object if GPU mapping failed. Print an error message
to help diagnose underlying problems.
Change-Id: I801ab6fe6c155bd25e6c0358007c106a4a019480
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Use MAP_POPULATE when allocating anonymous system memory for later
GPU mapping as a userptr. This can speed up large allocations by
more than factor 2. I suspect populating pages in this way is more
efficient than the CPU page fault code path triggered by
get_user_pages in the kernel.
Change-Id: I188bbc1462ccb650d48cbfb1080dbb8eb7ada8b5
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
KFD already implemented the fallback path for APU. Thunk will use flag
which exposed by kfd to configure is_dgpu instead of hardcode before.
Signed-off-by: Huang Rui <ray.huang@amd.com>
Change-Id: I445f6cf668f9484dd06cd9ae1bb3cfe7428ec7eb
This is needed to avoid additional references to mapped BOs in child
processes that can prevent freeing memory in the parent process and lead
to out-of-memory conditions.
Change-Id: I25c90510a14dde515cc23ea5dc1f68e8d7e37a66
Signed-off-by: Philip Cox <Philip.Cox@amd.com>
The option to use kfd_fd for cpu mapping is for very old broken KFD
version, it is not used in upstreaming process. This causes issue when
multiple process uses shared system memory because the GTT address is
over 40 bits.
Change to always use render node fd to create CPU mapping.
Change-Id: Id7e7b2a2e2f13c6e62c5de170589abfff4d456b0
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
This is needed to allow gdb to access the memory.
Change-Id: I96c084b714e952d7b7000f0dd41e1c530fdd092f
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
libpci was only used to find a marketing string for a device.
This patch looks for a pci.ids on disk and parses it to extract the
same string, using 'Device xxxx' as the fallback on file i/o error
or missing data from the text file. Tested by checking every vendor/
device pair against the values returned from libpci.
Change-Id: I21af3157472c1824d57fcee31393c6ee8ce07330
Signed-off-by: Jon Chesterfield <Jonathan.Chesterfield@amd.com>
IPC memory was previously returned as HSA_POINTER_ALLOCATED and
had garbage in the node_id field. Due to ROCR_VISIBLE_DEVICES
we need to be able to distinguish between imported memory and
regular memory because imported memory may not be owned by an
agent that is visible in the process. Differentiating these flags
allows the users to expect null agent for the owning agent.
Fixes
Change-Id: Ide3489cec1ee2072dc9697fa5cb71ddb17999d14
From Thunk spec, flag NoSubstitute = 0, if specific memory type is not
available on node, allocation may fall back to other memory that can
replace it on that node. mbind return failure if no memory available on
the specific node, we should ignore the mbind failure for this case.
Change-Id: I651a1bedf1852330604e56965cc17862403ebf87
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Check for errno == EBADF in kmtIoctl to detect misuse of the kfd_fd
in a forked child process.
Detect being in a forked child process pro-actively by implementing
a pthread_atfork callback.
Make sure all mutexes get reinitialized in the child process to avoid
deadlocks.
Check for being in a forked child process in CHECK_KFD_OPENED so that
all hsaKmt functions will return the appropriate status
HSAKMT_STATUS_KERNEL_IO_CHANNEL_NOT_OPENED.
Update InvalidKFDHandleTest to expect that error code.
Change-Id: I0238e5fba344dcaa454e97a35db2e2dcc8d1f607
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
On NUMA system, allocate queue ctx save restore area on the closest NUMA
node to the GPU which the queue is going to run. This will improve
performance on NUMA system generally by reducing schedule latency and
fix the multi-node rccl-tests unstable performance issue.
If the closest NUMA node has no memory available, set flags NoNUMABind=1
to bypass mbind, to use default NUMA memory policy to allocate system
memory.
Change-Id: Ic62bfa5bb2efbf4f6ae79ff403e9610ddf18d45c
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
FREE_MEMORY_OF_GPU ioctl could fail, e.g., if memory is still mapped
to GPU. Handle this failure by return error in fmm_release/HsaKmtFreeMemory
Change-Id: I5461db39964f733cf97376d50e44906a9b4c0f13
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
Without this change, the failure was hard to notice when it happened.
Change-Id: I99c3e8cea0d0cbd3bcfe79069410e6e870e225bf
Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
numa_max_node() return the highest node number available on the current
system, number of NUMA nodes should be numa_max_node() + 1.
Change-Id: I20a6c17af071e73e853cb5ea6d0304c8aca52681
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
on NUMA system, node 0 may have no memory, application pass node id
0 to hsaKmtAllocMemory will fail because mbind to specify the allocation
from node 0 return EINVAL.
Add new flag NoNUMABind for application to pass it to hsaKmtAllocMemory
to skip mbind.
hsaKmtCreateEvent and hsaKmtCreateQueue specify the new flag NoNUMABind
to allocate system memory for event page and CWSR area, don't bind the
system memory to a specific NUMA node.
Change-Id: I854e5a57502c7807c4c5ff2e441d499ae515c309
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Docker seccomp by default blocks mbind system call, so mbind return
failed on docker. thunk should not fail this otherwise application
cannot allocate system memory on docker.
Use pr_warn_once and pr_err_once to avoid duplicate same error messages
Change-Id: I61a7c0e4abaa3dcfe7abf2ea48db90f669f9638a
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
The memory size alignment workaround for a TLB bug on gfx802 was
breaking userptrs because it would attempt to get_user_pages beyond
the end of a VMA. Refine this workaround based on our understanding
of the HW bug. It only affects L2 cacheline allocation, which is
decided by the last page in the cache line (8 entries = 32KB of
address space). Thus aligning memory allocation so that the last
page falls on the end of a 8 entry TLB cache line allows caching
to work correctly.
Imported images require specific alignments. If their size is not
naturally aligned with 8 cache lines, it may have bad TLB cache
performance.
This patch will only have the desired effect if redundant size
padding in KFD is also removed.
Change-Id: I984cbe7fa61fec04d70fa387aaf9aab370eabeb9
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
By using user-allocated pages instead of kernel-allocated pages
from TTM, we're not subject to TTM's self imposed limits on kernel
memory usage. This also paves the way for for more wide-spread use
of HMM on upstream kernels.
Change-Id: Iac82964c98a441e29b7f1986d1be1bb5ccb1e569
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
child process clone vm objects from svm->apertures if parent process
doesn't free memory before fork. fmm_clear_all_mem suppose to clear the
apertures in forked child process but this only works if gpu_vm is not
NULL. parent process call hsaKmtCloseKFD reset gpu_vm to NULL and then
fork, then child process will not clear svm->apertures.
As a result, the child process will allocate vm object with same address
and add to aperture, there are duplicate vm objects with same address
in aperture. Then mapping to GPU will find the wrong vm object and
create incorrect GPU mapping cause rocrtst IPC test VM fault. The issue
happened with HSA_USERPTR_FOR_PAGED_MEM=1.
The fix is to clear vm objects in all apertures in clear_after_fork.
Change-Id: I92e42a967075a634a3f475b915c8242d82077ecb
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
This reverts commit 632ad3a749.
This change causes KFDTest failed on gfx803. The first hsaKmtCreateEvent
call allocate system memory for events_page because global variable
events_page is NULL. And this events page vm address should not be freed
until the process exit.
The change to destrory objects in hsaKmtCloseKFD removes events page. As
a result, KFDTest call hsaKmtOpenKFD again and then allocate memory will
get same events_page vm address on gfx803, and map this vm failed because
the vm conflict with events_page mapping.
KFDTest passed on VG10, gfx906 because allocate memory get different vm
address. hsaKmtCreateEvent still works fine as the driver keeps the
events page mapping of the process.
We should only destroy objects in fork cloned child process regardless
if gpu_vm is NULL or not.
Change-Id: I174ef65321cbd6074c855c2021318fe961c8c72c
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Otherwise the parent call hsaKmtCloseKFD and then fork child process,
child process will duplicate the vm_objects from the parent.
Change-Id: Ia6ffc51cbae983b6a7cdc58ccf3b11ebe4087d97
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Ignore requests to bind to invalid NUMA nodes. This affects only
legacy applications (such as KFDTest) that allocate system memory
as paged memory with a GPU node ID.
Change-Id: I81e514af6d0c1ab2ed5229adeeca1fa0ab2a0685
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
mbind walks through pages to setup vma memory policy. So we need do mmap
to create vma mappings first, then call mbind. mbind will do nothing if
vma does not exist.
And add numa available check before executing mbind, and return NULL to
hsaKmtAllocMemory if mbind failed.
Change-Id: I28ab661885d807ca51ef90e87230669dc80f10ec
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Alex Sierra
Timothy Pearson
David Yat Sin
Philip Yang
Kent Russell
changzhu
Mike Li
Graham Sider
Felix Kuehling
Oak Zeng
Harish Kasiviswanathan
Eric Huang
Gang Ba
Huang Rui
Philip Cox
Jon Chesterfield
Sean Keely
Yong Zhao