Define dgpu_mem_init before it's used and keep the code close to the
rest of the aperture initialization code.
Change-Id: I14ad11a364524a15affee9186b1298ba7d56d2c9
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Invisible device memory is mmapped as PROT_NONE.
Normal CPU access to the memory is still not allowed but
struct vm_area_struct will be created for the memory address
so ptrace can access the memory via the vma.
Change-Id: I07c69208716c920ccce33e6b494b610b61a0a7c1
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
This reverts commit f7689d4fef,
Plus a bug fix to patch "Cleanup fmm.c":
Call id_in_array with correct parameter. The third parameter
of id_in_array is size in byte of the array, not the number
of array items. Call it correctly.
Change-Id: I72d8e2fcc0df32af76c72967386e92c1be18c159
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
to fmm_allocate_memory_object. This function name was confusingly
similar to fmm_allocate_device and __fmm_allocate_device. The new name
reflects its function better: allocate the VM object and the kernel
mode buffer object.
Change-Id: I6604d228004b4d41e871d4de784786823608b5d6
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
This reverts commit b4c89c1ea7.
This change caused a regression ()
Revert temporarily
Change-Id: Ic3829264151e37d1f8c6927c6f464006234ba17f
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
This reverts commit 019f7cbd20.
This change caused a regression ()
Revert temporarily
Change-Id: I5af59d319afeb7f0b03e5a09e8397e3853b8b37b
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
1. Renamed _fmm_map_to_gpu to _fmm_map_to_apu_local
to reflect the real semantics of this function
2. Renamed _fmm_map_to_gpu_gtt to _fmm_map_to_gpu
because this function is used to map both gtt
and local memory
3. Call _fmm_map_to_gpu in _fmm_map_to_apu_local
to get rid of duplicated codes
Change-Id: Id8e3ebfffe0a3c27ebdcac8a8f4dc3738d67d10a
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
1. Initialize pointers to NULL in vm_create_and_init_object
2. Added helper function to add/remove device ids to/from mapped arrary
3. Only map nodes that were not mapped currently
4. Remove unnecessary condition check on object frees
Change-Id: I7aed6d40c7464be0d168d5796229af55451e0f34
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
_fmm_unmap_from_gpu is called in fmm_map_to_gpu_nodes
to unmap buffer from nodes that is already mapped to
but not in the new map nodes list. Previously, the unmap
was called unconditionally even though the size of the
array to unmap is 0. This fixes the issue by calling
the unmap func only when the unmap array size is not 0.
Also releases the fmm_mutex on error returns
Change-Id: Iadd8383caf7ebb92f02618798c5efd138a352aaa
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
Buffer mapping to devices and buffer registration to
devices can be changed b/t two pointer info queries.
Thus update buffer mapping info and registration info
only when mapping and registration changed. This is
done by free mapped_node_id_array on mapping to new
device and free registered_node_id_array on registration
and re-allocate them on next ptr info query.
Also uses fmm_mutex to avoid race conditions in case
of calling hsaKmtQueryPointerInfo concurrently with
calling of buffer mapping or registration
Change-Id: Ibc2e20be1fc0147066f873dfa44b21f5015104b7
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
Also separated a function for removing CPU mapping
and reserving address, as a refactoring of codes
Change-Id: I1feb85b0b2ec942487f899ec3192c7c47dd7c7d5
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
Previously kfd device is used to map memory for CPU access.
However this is not compatible with how TTM handles CPU mapping
on eviction - memory won't be unmapped and remapped on restore.
This fixes the issue by mmapping memory using DRM render device.
This patch requires a coordinated kernel driver change to work.
To make it compatible with old kernel driver, some temporary codes
are included. Once the coordinated kernel driver is checked in,
the temporary codes can be removed.
Change-Id: Ie7b304c4a82b7e8d5ab703acb81d66430af4f0bc
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
Because of HW design change, GPUVM aperture is no longer needed on GFX9
APUs. However, on APUs some functionalities still depend on GPUVM
aperture, so we choose to use SVM aperture instead to assume
the functionality of previous GPUVM aperture.
Change-Id: Ife7f0d598dd7989f2bcf7cdf3466d5a68703ca60
Signed-off-by: Yong Zhao <yong.zhao@amd.com>
Use mbind to specify the NUMA node for system memory allocation. This
only works with HSA_USERPTR_FOR_PAGED_MEM=1.
Change-Id: I88e7815d5a5aefcc4c22358c1a4a1635d7677ef3
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Marking heap memory as executable using mprotect() is not allowed
by SELinux. mprotect() calls that try to do this will fail on systems
with SELinux enabled. This is also a security risk, so it should be
fixed even on systems that allow this.
Any memory we want to mark as executable must be allocated using mmap().
See https://www.akkadia.org/drepper/selinux-mem.html
The two places where we try to mark heap memory as executable both use
posix_memalign() to allocate the heap memory. In both cases, the
alignment value passed into this function is always equal to PAGE_SIZE,
which means that they are safe to replace with mmap(), which guarantees
alignment to PAGE_SIZE. In this case PAGE_SIZE has been set to
sysconf(_SC_PAGESIZE);
v2:
- Use MAP_PRIVATE instead of MAP_SHARED. This matches the behavior
of memory allocated by posix_memalign()
- Ignore alignment hints instead of returning error when we can't
accommodate them.
- Drop alignment parameter of allocate_exec_aligned_memory() since
the only alignment supported is sysconf(_SC_PAGESIZE).
- Remove extra parameter from fmm_release().
- Add error path to fmm_allocate_host_cpu() for when mmap fails.
v3:
- Avoid use after free.
Change-Id: I7d51279790d9700bc3fa761c44bfde1c1936019b
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Applications may try to allocate lots host memory and reaches the mmap
limit (/proc/sys/vm/max_map_count). When Applications fails to allocate
memory and calls hsaKmtFreeMemory to release the memory, Thunk fails to
reduce the maps count so the following hsaKmtAllocMemory calls continue
to fail, which doesn't make sense to the application. This patch checks
the mmap to NORESERVE return value. If it fails and the error number is
ENOMEM, reduce the map count by munmap and map it again immediately.
Change-Id: I127cb479dfd86b199172eef269d59426f23859ea
Signed-off-by: Amber Lin <Amber.Lin@amd.com>
Support all fragment sizes up to 2MB by aligning buffers according
to their size.
Change-Id: I82b7ef8be6f1507d941e5c97edb6618adf8c66de
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Due to max_mmap_count issue, set default of guard page as disabled.
Change-Id: Ic9dfe69b621733e9fac86831b008a122994a67e7
Signed-off-by: Amber Lin <Amber.Lin@amd.com>
Use the mapped_device_id_array size when allocating temp_node_id_array
for unmapping queues in fmm_map_to_gpu_nodes. registered_device_id_array
size may be 0. Also, this temporary array is small enough to allocate it
on the stack. Malloc and free are overkill here.
Fix potential memory leak when registering the same device ID array
multiple times.
Change-Id: I83f09fd0925d9de7cf11bf72ba0ebb77273f587d
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Add pr_debug to all memory APIs and pr_err to some failure cases.
Change-Id: I8b519a1228cc19e6c04118fd87432e7f48f3cbf9
Signed-off-by: Amber Lin <Amber.Lin@amd.com>
Simplify fmm_map_to_gpu_nodes code. Also fix a memory leak in this change.
Change-Id: I3487338b78c915de44588d0206bac4c53e728c60
Signed-off-by: Amber Lin <Amber.Lin@amd.com>
Fall back to older apertures API and old events page size if the new APIs
fail. This allows running on current upstream kernels (with only minor
fixes) on gfx801 and enables testing of further changes during upstreaming.
Change-Id: I9d86d4f576e52fcbb5bc158d80f1bf41261e4e87
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Non paged system memory is allocated with node id 0. However, since a
gpu node is required for allocating system memory via KFD, the first
dgpu is used. In hsaKmtShareMemory() if system memory use the same
(first) dgpu.
Change-Id: I85789a89a4e4f7888e3826826401ea89ce4d1718
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
This should allow us to take advantage of BigK fragments and huge pages
and improve TLB efficiency for VRAM allocations. Huge pages only work
with 4-level page tables (gfx900 and up). BigK fragments work on older
GPUs.
Change-Id: I02e1fbf74de554e16fdaf44e44d03b47df45c3b0
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Imported graphics buffers are most likely images. Align them for
tiled image access. 64KB seems to do the trick.
This fixes VM faults with OpenCL graphics interop.
Change-Id: I7f60e205d93fff9407e0d00d3dbb02cc4990b863
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Libraries normally don't print messages. We use pr_err, pr_warn,
pr_info, and pr_debug to print messages to stderr when prints are
enabled for debugging.
Change-Id: I9caf719343aa618c88e7b500f9737a46702e424a
Signed-off-by: Amber Lin <Amber.Lin@amd.com>
The KFD implementation has been removed and will not be upstreamed.
This API has been superseded by hsaKmtRegisterGraphicsHandleToNodes.
Change-Id: I5f2d8da3260974618cdb6ea3fdcd77d37b82c9cb
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Amber Lin <Amber.Lin@amd.com>
Use checkpatch.pl to fix the majority of errors. Some that remain and
will be excluded:
Use of typedefs/externs/volatile/sscanf
Lines over 80 characters
Remaining errors are due to misunderstanding the * symbol with typedefs
Also use this opportunity to spell manageable properly
Change-Id: I0b335e9cb3e1eea38bee27eaa1f582b2c9b09b38
export HSA_CHECK_USERPTR=1 to check user pointers on registration. If
the pointer doesn't point to a valid mapping, there will be a segfault.
Change-Id: I459c0902cbc90338517fbf79678871ebfbe5183b
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Guard pages help catch out-of-bounds memory accesses by applications
by generating VM faults (GPU) and segfaults (CPU).
Remove address space reservation from scratch aperture. That address
space is managed by the Thunk client. Guard pages would cause Thunk's
address space management to get out of sync with the client's.
Change-Id: I2e5aee2923a90186358cc7b0e131baf547996df6
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Unless HSA_USERPTR_FOR_PAGED_MEM is explicitly set, don't use userptr
for all paged memory. This will also allow us to work around some 4.9
issues, and then we can explicitly set HSA_USERPTR_FOR_PAGED_MEM for
all usage once those issues are resolved.
Change-Id: I25ce22b73ae6e93f1567f2318d9d2b47d4a44e69
Use KFD_IOC_ALLOC_MEM_FLAGS_COHERENT when allocating fine-grained
memory and doorbell BOs so that they will be mapped with MTYPE_UC
on GFX9 hardware.
Change-Id: I51adf45b13105f479e6bcdaf54955b467920ee9a
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: Shaoyun Liu <Shaoyun.Liu@amd.com>
This avoids unnecessary evictions and failed restores due to the
munmap of userptr BOs that are just about to be freed.
Change-Id: Icf2f0b73991455556a201c54c05ea7e20af80f47
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
If fork() is called, clear all duplicated data that is invalid in the
child process.
Change-Id: I4e27198060db593c630c6337b7071dfbd0d80b83
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Felix Kuehling
Oak Zeng
Yong Zhao
Laurent Morichetti
Tom Stellard
Amber Lin
Harish Kasiviswanathan
Kent Russell
Sean Keely