Don't set kfd_open_count=1 unless hsaKmtOpenKFD actually succeeds.
This prevents returning HSAKMT_STATUS_KERNEL_ALREADY_OPENED in
subsequent calls when KFD is actually closed.
Signed-off-by: Sean Keely <Sean.Keely@amd.com>
Change-Id: Ia870b5faa8626826a6c8795aa10784d376cf2e80
[ROCm/ROCR-Runtime commit: 884fed4f04]
libpci was only used to find a marketing string for a device.
This patch looks for a pci.ids on disk and parses it to extract the
same string, using 'Device xxxx' as the fallback on file i/o error
or missing data from the text file. Tested by checking every vendor/
device pair against the values returned from libpci.
Change-Id: I21af3157472c1824d57fcee31393c6ee8ce07330
Signed-off-by: Jon Chesterfield <Jonathan.Chesterfield@amd.com>
[ROCm/ROCR-Runtime commit: 0a1718b753]
Read device unique id from sysfs and expose it in HsaNodeProperties.
For devices not supported the value will be 0
Signed-off-by: Divya Shikre <DivyaUday.Shikre@amd.com>
Change-Id: I97b8689dfa090971c6876de6feaa97652e28c03d
[ROCm/ROCR-Runtime commit: ebe7de1f99]
If child process explicitly calls hsaKmtReleaseSystemProperties(), it
fails. Allow child process to release and acquire system properties.
Change-Id: I649a4600212711b2ad4474f605f3ca39a4003d03
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
[ROCm/ROCR-Runtime commit: 31530da7c6]
IPC memory was previously returned as HSA_POINTER_ALLOCATED and
had garbage in the node_id field. Due to ROCR_VISIBLE_DEVICES
we need to be able to distinguish between imported memory and
regular memory because imported memory may not be owned by an
agent that is visible in the process. Differentiating these flags
allows the users to expect null agent for the owning agent.
Fixes
Change-Id: Ide3489cec1ee2072dc9697fa5cb71ddb17999d14
[ROCm/ROCR-Runtime commit: 6957202df8]
From Thunk spec, flag NoSubstitute = 0, if specific memory type is not
available on node, allocation may fall back to other memory that can
replace it on that node. mbind return failure if no memory available on
the specific node, we should ignore the mbind failure for this case.
Change-Id: I651a1bedf1852330604e56965cc17862403ebf87
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 5858aa17a9]
Check for errno == EBADF in kmtIoctl to detect misuse of the kfd_fd
in a forked child process.
Detect being in a forked child process pro-actively by implementing
a pthread_atfork callback.
Make sure all mutexes get reinitialized in the child process to avoid
deadlocks.
Check for being in a forked child process in CHECK_KFD_OPENED so that
all hsaKmt functions will return the appropriate status
HSAKMT_STATUS_KERNEL_IO_CHANNEL_NOT_OPENED.
Update InvalidKFDHandleTest to expect that error code.
Change-Id: I0238e5fba344dcaa454e97a35db2e2dcc8d1f607
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 87e10cd0b4]
NumCpQueues and NumSdmaQueuesPerEngine should be got by kfd driver not hardcode.
So add two data fields in HsaNodeProperties then thunk is able to get it from sysfs
that exposed by kfd.
v2: change NumCpQueues/NumSdmaQueuesPerEngine to one byte.
v3: merge two commits as one to avoid ABI update two times.
Change-Id: Ie386e4685f13493e22db6e207a399db6a4c5b9dc
Signed-off-by: Huang Rui <ray.huang@amd.com>
[ROCm/ROCR-Runtime commit: 06464b917d]
adds api and test to get newly create queue snapshot per ptraced process.
Change-Id: Ife97123a5b930e837ccaa386801145ef23c2cc2c
Signed-off-by: Jonathan Kim <Jonathan.Kim@amd.com>
[ROCm/ROCR-Runtime commit: 8b01a1c4c5]
This patch adds the support for gfx90c apu. So far we treat it as "dgpu" and
gfx900. Will update hsa gfxip table while the isa/llvm is implemented on gfx90c.
Change-Id: I6ef164bf3e751fe6dd6287cac212a500dce84b1a
Signed-off-by: Huang Rui <ray.huang@amd.com>
[ROCm/ROCR-Runtime commit: fdba74c2fb]
On NUMA system, allocate queue ctx save restore area on the closest NUMA
node to the GPU which the queue is going to run. This will improve
performance on NUMA system generally by reducing schedule latency and
fix the multi-node rccl-tests unstable performance issue.
If the closest NUMA node has no memory available, set flags NoNUMABind=1
to bypass mbind, to use default NUMA memory policy to allocate system
memory.
Change-Id: Ic62bfa5bb2efbf4f6ae79ff403e9610ddf18d45c
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 59c857476f]
This patch adds the non-privileged PMC blocks for GFX10/gfx1010.
Change-Id: I4b98cb2159d71113c12920ca7fd10e45096b4e2c
Signed-off-by: Ori Messinger <Ori.Messinger@amd.com>
[ROCm/ROCR-Runtime commit: e7f45fae8a]
FREE_MEMORY_OF_GPU ioctl could fail, e.g., if memory is still mapped
to GPU. Handle this failure by return error in fmm_release/HsaKmtFreeMemory
Change-Id: I5461db39964f733cf97376d50e44906a9b4c0f13
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
[ROCm/ROCR-Runtime commit: fa0cb9ebeb]
Now the gfx1012 device IDs are okay to reveal.
Change-Id: I9da2a036b74ec7b6b8b1fb7587597a5847f02205
Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
[ROCm/ROCR-Runtime commit: 1c7755d2da]
Without this change, the failure was hard to notice when it happened.
Change-Id: I99c3e8cea0d0cbd3bcfe79069410e6e870e225bf
Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
[ROCm/ROCR-Runtime commit: 16fa78b134]
When CONFIG_NUMA is not enabled in the kernel config, only one CPU node
presents on the system and /sys/devices/system/node/nodeX directories
don't exist. Read CPU cache information from /sys/devices/system/cpu in
this situation.
Change-Id: I017ff17dd72678a0551edcc77446664501aa42ca
Signed-off-by: Amber Lin <Amber.Lin@amd.com>
[ROCm/ROCR-Runtime commit: 23541e0289]
The debug trap accesses the data0/data1 registers, so we do not
want the userspace to write values to it. We remove the calls to
set the data0/data1 register values.
Change-Id: Iaba842a4c445f339f16a39fe1994526ff78a2f3c
Signed-off-by: Philip Cox <Philip.Cox@amd.com>
[ROCm/ROCR-Runtime commit: 6933540c81]
To support adding new features to the kfd debugger, and not break
functionality, we need to be able to check the kfd debugger support
version info from the kernel.
Change-Id: Icd88e4edab8430c35eaed588e62d892c1b5c62ec
Signed-off-by: Philip Cox <Philip.Cox@amd.com>
[ROCm/ROCR-Runtime commit: dbbd189b33]
When fail to get CPU dirs from //sys/devices/system/node/nodeX directory,
the error message should print node_dir, not path.
Change-Id: If76a51918c8dd55fa6605a62f3d29f9efc6fadb3
Signed-off-by: Amber Lin <Amber.Lin@amd.com>
[ROCm/ROCR-Runtime commit: 5a09880620]
numa_max_node() return the highest node number available on the current
system, number of NUMA nodes should be numa_max_node() + 1.
Change-Id: I20a6c17af071e73e853cb5ea6d0304c8aca52681
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 71cf3cf5d3]
on NUMA system, node 0 may have no memory, application pass node id
0 to hsaKmtAllocMemory will fail because mbind to specify the allocation
from node 0 return EINVAL.
Add new flag NoNUMABind for application to pass it to hsaKmtAllocMemory
to skip mbind.
hsaKmtCreateEvent and hsaKmtCreateQueue specify the new flag NoNUMABind
to allocate system memory for event page and CWSR area, don't bind the
system memory to a specific NUMA node.
Change-Id: I854e5a57502c7807c4c5ff2e441d499ae515c309
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 42392f093f]
Docker seccomp by default blocks mbind system call, so mbind return
failed on docker. thunk should not fail this otherwise application
cannot allocate system memory on docker.
Use pr_warn_once and pr_err_once to avoid duplicate same error messages
Change-Id: I61a7c0e4abaa3dcfe7abf2ea48db90f669f9638a
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 4da09813a3]
The gfx version item is yet to be added.
Change-Id: Ia6c487447e5a5df80c0c12fe150939175068024b
Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
[ROCm/ROCR-Runtime commit: 3ecd83e52d]
Force all the GPUs to a certain type, use the below command:
HSA_FORCE_ASIC_TYPE="10.1.0 1 gfx1010 14"
meaning major.minor.step dgpu asic_name asic_id
This will faciliate the cooperation across the teams for bringing up
ASICs which reuse existing device IDs.
Change-Id: I40fe4c9b46d3ccb3e38ea52250e80e82fb50fb0f
Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
[ROCm/ROCR-Runtime commit: d6539ddc24]
The memory size alignment workaround for a TLB bug on gfx802 was
breaking userptrs because it would attempt to get_user_pages beyond
the end of a VMA. Refine this workaround based on our understanding
of the HW bug. It only affects L2 cacheline allocation, which is
decided by the last page in the cache line (8 entries = 32KB of
address space). Thus aligning memory allocation so that the last
page falls on the end of a 8 entry TLB cache line allows caching
to work correctly.
Imported images require specific alignments. If their size is not
naturally aligned with 8 cache lines, it may have bad TLB cache
performance.
This patch will only have the desired effect if redundant size
padding in KFD is also removed.
Change-Id: I984cbe7fa61fec04d70fa387aaf9aab370eabeb9
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: e320913e9e]
snprintf throws a warning from -Wformat-truncation where the string
could be truncated. We address this by referencing the maximum size that
can be returned from a file according to MAXNAMLEN . This should safely
guard us from truncating the path value.
Change-Id: If1d208990d8775e9494835b0deb890d2616fd15b
Signed-off-by: Kent Russell <kent.russell@amd.com>
[ROCm/ROCR-Runtime commit: ccac07cb14]
Replace cpuid call with sysfs data to get CPU cache information. With this
change, x86 check is also removed since sysfs applies to other platforms.
CPU cache information can be retrieved from
/sys/devices/system/node/nodeX/cpuY/cache where Y is processor number
represented in /proc/cpuinfo at "processor" entry.
Change-Id: Ic47df6d5dafaf1aae5b46b1fdee42691c697e49e
Signed-off-by: Amber Lin <Amber.Lin@amd.com>
[ROCm/ROCR-Runtime commit: 4fa930af5a]
PCI domain has moved to 32-bits to accommodate virtualization,
so a 32-bit integer is exposed for domain to reflect this change.
Change-Id: I0d767acadcdc8e4277db203b5865dd67dd001cef
Signed-off-by: Ori Messinger <ori.messinger@amd.com>
[ROCm/ROCR-Runtime commit: f2173254e4]
enable thunk query api to report if queue is newly created
test new queue bit test on clear events.
also fixup cleanup to disable debug trap.
Change-Id: I3ebe2d85da66f28b8c82f0e68461ee7d32ec0b0d
Signed-off-by: Jonathan Kim <Jonathan.Kim@amd.com>
Reviewed-by: Philip Cox <Philip.Cox@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 1ff5cb33b2]
Originally reserved 100 SGPRs per wave. Pre-gfx10 needs 102 SGPRs
and gfx10 needs 128 SGPRs. Reserve 128 SGPRs per wave for all ASICs
to simplify calculation.
Also double VGPR register size for gfx908 family
Change-Id: I98b741cbfa051f49ed37ff25d99f851f124be7b6
Signed-off-by: Jay Cornwall <Jay.Cornwall@amd.com>
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
[ROCm/ROCR-Runtime commit: 814e0f0bdc]
By using user-allocated pages instead of kernel-allocated pages
from TTM, we're not subject to TTM's self imposed limits on kernel
memory usage. This also paves the way for for more wide-spread use
of HMM on upstream kernels.
Change-Id: Iac82964c98a441e29b7f1986d1be1bb5ccb1e569
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 626957a263]
child process clone vm objects from svm->apertures if parent process
doesn't free memory before fork. fmm_clear_all_mem suppose to clear the
apertures in forked child process but this only works if gpu_vm is not
NULL. parent process call hsaKmtCloseKFD reset gpu_vm to NULL and then
fork, then child process will not clear svm->apertures.
As a result, the child process will allocate vm object with same address
and add to aperture, there are duplicate vm objects with same address
in aperture. Then mapping to GPU will find the wrong vm object and
create incorrect GPU mapping cause rocrtst IPC test VM fault. The issue
happened with HSA_USERPTR_FOR_PAGED_MEM=1.
The fix is to clear vm objects in all apertures in clear_after_fork.
Change-Id: I92e42a967075a634a3f475b915c8242d82077ecb
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: a11cb2a633]
This reverts commit bb4291dcac.
This change causes KFDTest failed on gfx803. The first hsaKmtCreateEvent
call allocate system memory for events_page because global variable
events_page is NULL. And this events page vm address should not be freed
until the process exit.
The change to destrory objects in hsaKmtCloseKFD removes events page. As
a result, KFDTest call hsaKmtOpenKFD again and then allocate memory will
get same events_page vm address on gfx803, and map this vm failed because
the vm conflict with events_page mapping.
KFDTest passed on VG10, gfx906 because allocate memory get different vm
address. hsaKmtCreateEvent still works fine as the driver keeps the
events page mapping of the process.
We should only destroy objects in fork cloned child process regardless
if gpu_vm is NULL or not.
Change-Id: I174ef65321cbd6074c855c2021318fe961c8c72c
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 7fc6a9f7c2]
/proc/cpuinfo are opened, read, and closed multiple times. Once for vendor
name and multiple times for model name -- each node opens once. For example
in a 2 CPUs + 4 GPUs system, it'll be opened 7 times. This patch reads it
one time and stores it in a cpuinfo buffer. This cpuinfo buffer is freed
when the snapshot is done.
Also replace returns with gotos inside the snapshot to avoid possible
memory leak.
Change-Id: Iaf26a6c7e7323a8651d137c3706179449b9e3c80
Signed-off-by: Amber Lin <Amber.Lin@amd.com>
[ROCm/ROCR-Runtime commit: 1fddfd316a]
Add data out for enable trap to return poll fd to user space.
Add query debug events interface.
Change-Id: Ia4afde1cf167e6aa61d502380a8b329ee89d5f44
Signed-off-by: Jonathan Kim <Jonathan.Kim@amd.com>
[ROCm/ROCR-Runtime commit: 836dfd0752]
Otherwise the parent call hsaKmtCloseKFD and then fork child process,
child process will duplicate the vm_objects from the parent.
Change-Id: Ia6ffc51cbae983b6a7cdc58ccf3b11ebe4087d97
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 632ad3a749]
The NodeId parameter is redundant and can be retrieved
from QueueId parameter.
Change-Id: I12853849b868b304bd27633fa7653ba644d69026
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
[ROCm/ROCR-Runtime commit: f40f166e20]
Ignore requests to bind to invalid NUMA nodes. This affects only
legacy applications (such as KFDTest) that allocate system memory
as paged memory with a GPU node ID.
Change-Id: I81e514af6d0c1ab2ed5229adeeca1fa0ab2a0685
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 4d7b0990e4]
mbind walks through pages to setup vma memory policy. So we need do mmap
to create vma mappings first, then call mbind. mbind will do nothing if
vma does not exist.
And add numa available check before executing mbind, and return NULL to
hsaKmtAllocMemory if mbind failed.
Change-Id: I28ab661885d807ca51ef90e87230669dc80f10ec
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
[ROCm/ROCR-Runtime commit: 67f366243d]
Srinivasan Subramanian
Sean Keely
Jon Chesterfield
Divya Shikre
Harish Kasiviswanathan
Philip Yang
Felix Kuehling
Huang Rui
Yong Zhao
Jonathan Kim
Ori Messinger
Oak Zeng
Amber Lin
Philip Cox
shaoyunl
Jack Zhang
Kent Russell
Jay Cornwall