Move debug trap support capabilities to their own
structure to fix thunk spec vs header mismatch.
Change-Id: I6694601bfa36097502c8ab932e082d7a4645d5b2
Signed-off-by: Philip Cox <Philip.Cox@amd.com>
[ROCm/ROCR-Runtime commit: 105edd4bb4]
GPU Resource management can disable some of the GPU nodes.
The Kernel driver could be not aware of this.
Get from Kernel driver information of all the nodes and then filter it.
Change-Id: I4eeb126a5efce2192c35f5d2b72be1811e9ded32
Signed-off-by: Mike Li <Tianxinmike.Li@amd.com>
[ROCm/ROCR-Runtime commit: 3144a84b9a]
The existing call sysconf (_SC_NPROCESSORS_ONLN) provides the number of
processors available to the scheduler. When a KFD process is run under a
container environment, only a subset (cpuset) of processors are
available to the current process.
For getting CPU cache information use sched_getaffinity() to get the
number of processors available to the current process.
Change-Id: Ieac02f1f61c17e24ac34ba502968c69d3bc631cb
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
[ROCm/ROCR-Runtime commit: fb79a0efe2]
Handle the case that svm.dgpu_aperture does not exist in vm_find_object.
Change-Id: Ic0983d4f321f1b6248514f2fa25162976e90bd75
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: be574169c1]
Some nodes are unavailable based on the task's cgroup hierarchy. Handle
this situation by ignoring those nodes
Change-Id: I72f9e822d2ec8cf15732df95e427d5549a75b55d
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
[ROCm/ROCR-Runtime commit: 7876bb70a9]
With GPU resource management, some nodes are unavailable based on the
cgroup hierarchy of the task. Kernel via sysfs specifies all the
iolinks. Skip the links which are not accessible.
Also iolinks specified by the kernel refer to sysfs Node IDs. Map it to
relevant user Node IDs
v2: NodeFrom mapped from sysfs Node to User Node
Change-Id: I95312ee6ca51b89fe9e6ca2a9185c2ea1e94afc4
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
[ROCm/ROCR-Runtime commit: 866ef20054]
This reverts commit 84bb9072c0.
This fixes ambiguity when looking up GPU addresses with
hsaKmtQueryPointerInfo.
hsa_amd_agents_allow_access uses hsaKmtQueryPointerInfo, and
depends on finding the correct object from a GPU address. Finding
the wrong userptr object based on its CPU address leads to
incorrect GPU mappings and results in VM faults.
Change-Id: I7c5f571ee6e1f9d32687aa3eab6d96944ad032be
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: a9bd6e6f8b]
Added a helper vm_find_object that can be used everywhere we need to
lookup objects by their address and optionally size. This unifies
all subtly different, partially incomplete, or broken ways of doing
this in various functions:
* map
* unmap
* register
* deregister
* free
* get_mem_info
* set_mem_user_data
At the same time fix some subtle problems for userptr lookup that
got a bit more complex when the userptr address can match the GPU
address.
Change-Id: I98572d1734fc7688a1d68f6a784e02c8dee90af5
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 855f1a32a9]
PCIe P2P (indirect) IOLinks should only be created if the remote GPU
is large-BAR
Change-Id: I55cbb5e37c5d41267583e07aca6bdcc708403029
Signed-off-by: shaoyunl <Shaoyun.Liu@amd.com>
[ROCm/ROCR-Runtime commit: 30a4ab39f3]
Currently, all HSA nodes are exposed to user. So the existing
implementation assumes a one to one mapping between user
NodeId and sysfs nodeId.
GPU Resource Management will provide control over the exposed
HSA nodes. This means not all HSA nodes will be exposed to the user.
Decouple it.
The mapping from user NodeId to sysfs NodeId will be local
to topology.c and topology helper functions. For others NodeId
should be sequential from 0 to Number of Nodes exposed to user.
v1: initial implementation
v2: map node id within the topology_* functions
v3: remove two static globals
v4: add bounds check got node id
Change-Id: Id12147ece41d682430f398944bbb339ca906eb1b
Signed-off-by: Mike Li <Tianxinmike.Li@amd.com>
[ROCm/ROCR-Runtime commit: 3437a356c7]
Add initial support for the kfd debugger trap support
for GFX9 chips.
- Adding support for Enable/Disable trap support
- Setting debug trap support data
- Setting wave launch trap override
- Setting wave launch mode
Change-Id: If39f2395c4b6cf56249cf76f1c44cfcbdcef891c
Signed-off-by: Philip Cox <Philip.Cox@amd.com>
[ROCm/ROCR-Runtime commit: db92d5af23]
AMDKFD_IOC_WAIT_EVENTS with multiple events and wait_for_all = 0
returns success after any of the events have signaled. So we can't
blindly assume that a memory fault event that was in the list has
actually signaled. Check the gpu_id as an indicator whether there
really was a memory fault before processing it further.
Change-Id: I6cc311bfc184c631beaf684027176a6ca42e05c1
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 9271e69ddf]
If the CPU addr of a userptr is accessible by the GPU, try to use it
instead of allocating a different GPU address. If something else is
already registered with an overlapping address range, we still need to
allocate a GPU address, because KFD does not support overlapping GPUVM
mappings.
Change-Id: I452963ee45a454f735755a0b43122b9aee5d55be
Signed-off-by: Felix Kuehling <felix.kuehling@gmail.com>
[ROCm/ROCR-Runtime commit: ab181c46c0]
If the GPU virtual address space is >= 47 bits, don't reserve virtual
address space at startup and use mmap to allocate virtual addresses.
Change-Id: Ic935b03c8e78271829fc8e6cfd0e543184aff818
Signed-off-by: Felix Kuehling <felix.kuehling@gmail.com>
[ROCm/ROCR-Runtime commit: 80f2cc644c]
So far we have assumed that userptrs are always memory outside
reserved SVM apertures that are mapped into the SVM aperture for
GPU access.
With an unreserved SVM aperture that covers the entire virtual
address range, this distinction will no longer be true. Userptrs
will generally be inside the unreserved SVM aperture. Take that
into consideration when registering, mapping and unmapping virtual
addresses.
We now need a retry logic when looking up buffers from addresses.
If it is not found by its GPU address, try it as a userptr.
We also need to consider the new possibility that a userptr is
registered at the same address for CPU and GPU access. So a buffer
found by its GPU address may also turn out to be a userptr. In
that case use a stricter lookup using the userptr and size (if
the size is known), to identify the correct one of multiple
overlapping objects.
Change-Id: Ia43633aaa40f9fd2a74918ae969a631d2ff68419
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 40c46cc6cb]
Make dgpu_aperture and dgpu_alt_aperture pointers that can point to
the same actual aperture. This will be useful on GFXv9 and later,
where the MType is not defined by the aperture and we want to have
a single aperture covering the entire virtual address space.
aperture->is_coherent can no longer be a reliable indicator of
coherency. Replace it with different conditions based on mem flags
and svm.disable_cache (from HSA_DISABLE_CACHE environment).
Change-Id: Iefc415b87b8abd96e3916586485a0a55d9b27c19
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: d57026f447]
This prepares the code for an alternative aperture management method
that needs to unmap memory differently.
Change-Id: I5494aa5420f85edb8f7857f00c17e1d2e6479a51
Signed-off-by: Felix Kuehling <felix.kuehling@gmail.com>
[ROCm/ROCR-Runtime commit: 2d2181b478]
Only scratch_physical, for scratch-backing memory is managed by the Thunk.
Change-Id: I4716981aa908d9569584dc35f40ffd270a2f9014
Signed-off-by: Felix Kuehling <felix.kuehling@gmail.com>
[ROCm/ROCR-Runtime commit: 9d96af0150]
This parameter was used for non-canonical GPUVM allocations on GFX7/8 APUs
only, to prevent getting NULL pointers from valid allocation after
subtracting the aperture base. The same can be achieved less intrusively
by reserving address space at the start of the aperture during
initialization.
Change-Id: I0aae773f069c2b228824ba464b0612a4d8b489ce
Signed-off-by: Felix Kuehling <felix.kuehling@gmail.com>
[ROCm/ROCR-Runtime commit: 842359a826]
Hit queue create failure when do kfdtest with --gtest_repet=-1
fix: d89e9165("Remove the use of IS_DGPU()")
Change-Id: I04fa73f90cef13a5517dbaceb89c41dc0f821a79
Signed-off-by: xinhui pan <xinhui.pan@amd.com>
[ROCm/ROCR-Runtime commit: eb5539fb10]
Because gfx700 has local memory but other APUs don't, we should reflect
that in the code. Meanwhile, fix a bug that on gfx902 svm aperture is not
added when calling hsaKmtGetNodeMemoryProperties().
Change-Id: Id840f2db0b14fda9ee713b219a9474c15f8a9771
Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
[ROCm/ROCR-Runtime commit: 110e754f64]
On some asics, like tonga, the memory alignment size is as big as 0x8000.
fmm_allocate* alloc vm area with size passed in which is not aligned mostly.
But __fmm_release free vm area with vm_object_t->size which is aligned.
That might cause aperture_release_area fail to free the vm area as the
size might be bigger than zone itself or it just free another vm area
nearby unexpected.
This patch somehow will alloc more space than it needed on tonga.
gfx900+ is not affected.
Change-Id: I5a88c92b08c4e6f6bc05881798f769b55d6debe9
Signed-off-by: xinhui pan <xinhui.pan@amd.com>
[ROCm/ROCR-Runtime commit: 8fbf4a26ec]
Previously we used the first dgpu mem, but after careful examination, we
found it only needs to be a GPU, so we modify the code to reflect that as
well.
Change-Id: I069d9b8e247aed55c1f885b79f743ea8e03ddf93
Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
[ROCm/ROCR-Runtime commit: fe04dd6890]
The information can be obtained directly from node id. Also improve the
whole logic for future compatibility.
Change-Id: I130733be4e7930d5953d5e81409905e60c2ec35e
Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
[ROCm/ROCR-Runtime commit: 4bb90d048c]
Unset ret_addr when unmapping the address space reservation. Otherwise
it may try to unmap it again later.
Remember the actual map_size and use it instead of len outside the
reservation loops.
Change-Id: I1a6b3fecfb59e22a713e5ed49c3ed37914cb6fb5
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: c21927f425]
node is used repeatedly and excessively, which caused unnecessary confusion.
Change-Id: I4ae4171887df5e5b85209a5af8a636e6d72e5e82
Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
[ROCm/ROCR-Runtime commit: 08b6685dd5]
simple test of mapping many system memory to gpu.
before
[ RUN ] KFDMemoryTest.MMap
[ ] Using ISA for GFXIP 9.0
[ ] successfully register/map 32GB system memory to gpu
[ OK ] KFDMemoryTest.MMap (36932 ms)
after
[ RUN ] KFDMemoryTest.MMap
[ ] Using ISA for GFXIP 9.0
[ ] successfully register/map 32GB system memory to gpu
[ OK ] KFDMemoryTest.MMap (11441 ms)
So there is 11s VS 36s improvement.
Looks like we can do something similar with vm_area too.
Change-Id: I0349aacdeddec3534016d28176f0fabf632c61fc
Signed-off-by: xinhui pan <xinhui.pan@amd.com>
[ROCm/ROCR-Runtime commit: ab9017715f]
Compare with gpu_mem_count instead of deprecated NUM_OF_SUPPORTED_GPUS
to prevent overflows in case no dGPUs are present.
Change-Id: I71fcb7503ba4c20bffadbdb04cefc4e4027a7df7
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: d3228f363e]
The new logic is easier to follow.
Change-Id: I69759a45c5dedaefeff831a2367253d3a4486bd3
Signed-off-by: Yong Zhao <yong.zhao@amd.com>
[ROCm/ROCR-Runtime commit: 4eaaf9694d]
There were two doorbells, one embedded in another, which are very confusing.
Change the member variable name to mapping to differentiate them. Also,
rename doorbells_mutex to just mutext for brevity.
Change-Id: Iaa14a1a3ee09449a9089fc1fb39c916fdf32fb44
Signed-off-by: Yong Zhao <yong.zhao@amd.com>
[ROCm/ROCR-Runtime commit: 5972fac417]
If opening drm render device fails (usually when the user is not a member
of video group), fmm_init_process_apertures() still returns success,
resulting in weird segfault in a later stage.
Change-Id: Ifbde4481629988944ad7f384d59753c88e287fa9
Signed-off-by: Yong Zhao <yong.zhao@amd.com>
[ROCm/ROCR-Runtime commit: 77ec699460]
Avoid warnings of the type
error: 'strncpy' specified bound 64 equals destination size
With the destination being 0-initialized, subtracting 1 from the
destination buffer size will ensure that the destination will be a
0-terminated string, even when it's truncated.
Change-Id: I7c3a90482065ce4d020db215e3e41348de51a083
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: fb551a44af]
Add back missing pthread_mutex_lock.
Handle all error cases in fmm_release.
Change-Id: I8efa561ddadfd769cede5bf86300215ba3fb3dd1
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 4e766615d7]
__fmm_release actually fails to find the object if address is not
pagesize aligned. And the caller did not notice this as __fmm_release
has no err code return.
So to fix this, move the object lookup in caller, and use vm-object
instead. Also fmm_release will pass up the error code.
Change-Id: Ib8ea1ea5ae844844fd20e8e01f0fdb841d218f2c
Signed-off-by: xinhui pan <xinhui.pan@amd.com>
[ROCm/ROCR-Runtime commit: 8ee5647814]
When KFD is already opened, Opening it again should return
HSAKMT_STATUS_KERNEL_ALREADY_OPENED to align with the specification.
Change-Id: Ib10a2d2c48781600bea7d072557d03ccb1a2bc19
Signed-off-by: Yong Zhao <yong.zhao@amd.com>
[ROCm/ROCR-Runtime commit: 7a8566dc03]
Upstream KFD doesn't support mapping doorbells to GPUVM yet. Fall
back to the old method.
Change-Id: I452a6fc59b88329b833844e3914c480c2f13c82d
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 0462744965]
Make it more similar with upstream.
Change-Id: I982ccfd4045d96e3c30bc84d38d0e03db8de9b08
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 7495e74257]
The main point is to move update_ctx_save_restore_size() out of if()
condition.
Change-Id: I58a1a4f3edca2d1c510fdd0e31e59b5c41e92a14
Signed-off-by: Yong Zhao <yong.zhao@amd.com>
[ROCm/ROCR-Runtime commit: 43f119bcbc]
Philip Cox
Gang Ba
Mike Li
Harish Kasiviswanathan
Felix Kuehling
shaoyunl
Felix Kuehling
xinhui pan
Yong Zhao
Slava Grigorev
Oak Zeng