The information can be obtained directly from node id. Also improve the
whole logic for future compatibility.
Change-Id: I130733be4e7930d5953d5e81409905e60c2ec35e
Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>
[ROCm/ROCR-Runtime commit: 4bb90d048c]
There were two doorbells, one embedded in another, which are very confusing.
Change the member variable name to mapping to differentiate them. Also,
rename doorbells_mutex to just mutext for brevity.
Change-Id: Iaa14a1a3ee09449a9089fc1fb39c916fdf32fb44
Signed-off-by: Yong Zhao <yong.zhao@amd.com>
[ROCm/ROCR-Runtime commit: 5972fac417]
Upstream KFD doesn't support mapping doorbells to GPUVM yet. Fall
back to the old method.
Change-Id: I452a6fc59b88329b833844e3914c480c2f13c82d
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 0462744965]
The main point is to move update_ctx_save_restore_size() out of if()
condition.
Change-Id: I58a1a4f3edca2d1c510fdd0e31e59b5c41e92a14
Signed-off-by: Yong Zhao <yong.zhao@amd.com>
[ROCm/ROCR-Runtime commit: 43f119bcbc]
On discrete GPUs place the EOP queue in VRAM. The reader/writer of this
queue is the CP and the size is small. Dispatch latency improves
through lower read latency in AQL completion phase.
Change-Id: Id8351dcddbd21fd7c7d699803c96434c9132db71
Signed-off-by: Jay Cornwall <Jay.Cornwall@amd.com>
[ROCm/ROCR-Runtime commit: e2c353dc0d]
Marking heap memory as executable using mprotect() is not allowed
by SELinux. mprotect() calls that try to do this will fail on systems
with SELinux enabled. This is also a security risk, so it should be
fixed even on systems that allow this.
Any memory we want to mark as executable must be allocated using mmap().
See https://www.akkadia.org/drepper/selinux-mem.html
The two places where we try to mark heap memory as executable both use
posix_memalign() to allocate the heap memory. In both cases, the
alignment value passed into this function is always equal to PAGE_SIZE,
which means that they are safe to replace with mmap(), which guarantees
alignment to PAGE_SIZE. In this case PAGE_SIZE has been set to
sysconf(_SC_PAGESIZE);
v2:
- Use MAP_PRIVATE instead of MAP_SHARED. This matches the behavior
of memory allocated by posix_memalign()
- Ignore alignment hints instead of returning error when we can't
accommodate them.
- Drop alignment parameter of allocate_exec_aligned_memory() since
the only alignment supported is sysconf(_SC_PAGESIZE).
- Remove extra parameter from fmm_release().
- Add error path to fmm_allocate_host_cpu() for when mmap fails.
v3:
- Avoid use after free.
Change-Id: I7d51279790d9700bc3fa761c44bfde1c1936019b
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: e2ed9cf79a]
Hardware block testing is done with the workgroup state offset
initialized to the control stack size on all ASICs. MEC microcode
assumes this space is available when the workgroup state offset is
reset after a context restore event.
Fixes context save area overrun when the full save area is used.
Change-Id: I8eeb62f97140c6fe409fe78b4497d833584feea8
Signed-off-by: Jay Cornwall <Jay.Cornwall@amd.com>
[ROCm/ROCR-Runtime commit: 4fbffcdd9c]
ctl_stack_copy is allocated from malloc. It should be freed by free.
Change-Id: Ib924da20200d91f52f106fe173464d47862759a8
Signed-off-by: Amber Lin <Amber.Lin@amd.com>
[ROCm/ROCR-Runtime commit: 6e113e2634]
For items in HsaQueueInfo, control stack information comes from KFD, CU
mask information is maintained in Thunk, and others (queue detail error
and queue type extended) are ignored (value = 0) at this point.
Change-Id: Ib21370b0f52b2bb4ebe6a9b4b6ec6139cccb25ca
Signed-off-by: Amber Lin <Amber.Lin@amd.com>
[ROCm/ROCR-Runtime commit: 683fc96325]
Use checkpatch.pl to fix the majority of errors. Some that remain and
will be excluded:
Use of typedefs/externs/volatile/sscanf
Lines over 80 characters
Remaining errors are due to misunderstanding the * symbol with typedefs
Also use this opportunity to spell manageable properly
Change-Id: I0b335e9cb3e1eea38bee27eaa1f582b2c9b09b38
[ROCm/ROCR-Runtime commit: b78e0e152a]
Integrate the supported device ID list distributed in topology, queue, and
pmc into one place: topology.
Change-Id: If035cf8e4a6fc6caff6c94ec627647cfb11c3d79
[ROCm/ROCR-Runtime commit: 4827b09119]
The control stack memory for CWSR is allocate in kernel together with MQD
allocation.
Change-Id: Ib1c0ab9402df3431e9555649394320380d6c6dd8
Signed-off-by: shaoyun.liu <shaoyun.liu@amd.com>
[ROCm/ROCR-Runtime commit: 116e5c5e8b]
On SOC15 chips, the ABI for the create_queue ioctl is changed to
allow doorbell allocation independent of the queue ID. This is
necessary to accommodate doorbell routing to specific engines in
the BIF.
Change-Id: Ie98d0a758758149dd5fc09ae088afccc29904124
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: 7de66d149b]
On gfx900 we need 64-bit for all doorbells and SDMA WPTRs.
Change-Id: I9b922e16442e967599ae3c928308451d5cc470b3
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
[ROCm/ROCR-Runtime commit: d7063dd102]
This is in preparation for gfx900, which uses 64-bit doorbells. We
maintain the same number of doorbells per process by making the
doorbell page size bigger.
KFD will need to implement the same rule.
Change-Id: I3c4110869b191b83943b5a390a48edfc94d941d8
[ROCm/ROCR-Runtime commit: 48207af92a]
events.c and queues.c were accidently changed to 755 by change
fc70f0c30976f4021f7d763bfc10d76a76029553. Change them back.
Change-Id: If51c0b91139afc23e9051cf94c83d61fc20297e6
Signed-off-by: Oak Zeng <Oak.Zeng@amd.com>
[ROCm/ROCR-Runtime commit: b3c3f7bae1]
Add gfx803 10/11 device IDs that were recently added to KFD.
Change-Id: Id40b117ae47bacedefa6e333fdfdf58dea92cd2d
Signed-off-by: Amber Lin <Amber.Lin@amd.com>
[ROCm/ROCR-Runtime commit: 72b842a6dc]
If fork() is called, clear all duplicated data that is invalid in the
child process.
Change-Id: I4e27198060db593c630c6337b7071dfbd0d80b83
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
[ROCm/ROCR-Runtime commit: f1f62d863c]
CWSR buffers can be large on dGPUs (~21MB on gfx803). Allocating them
in VRAM limits the number of queues that can be created unnecessarily.
Also make freeing of per-queue buffers symmetric with allocation. All
buffers are now allocated with allocate_exec_aligned_memory on dGPUs
and APUs, so use free_exec_aligned_memory to free them.
Change-Id: I45e8cb1801857d0268750202cdd422426611e457
[ROCm/ROCR-Runtime commit: 4181b408fc]
gfx802 requires a workaround for a VM TLB bug in which lookups use
the ACTIVE bit of the 8th PTE within any aligned group of 8 PTEs.
Until this is fixed in amdgpu the GPUVM doorbell logic will fail.
Change-Id: I5ec7b1fcd8b7677011a141d27cfc486c45d9a415
Signed-off-by: Jay Cornwall <Jay.Cornwall@amd.com>
[ROCm/ROCR-Runtime commit: 5493ae420b]
Allocate doorbells for dGPUs in the SVM aperture and map them for
GPU access. This is necessary to allow GPU-initiated submissions to
user mode queues.
Depends on new doorbell BO allocation flag in KFD.
Change-Id: I0737bef4a4764bb4a66c43846707ead2108f6601
[ROCm/ROCR-Runtime commit: 2e0a6eb371]
The thunk spec requires that CUMaskCount be divisible by 32. Check this
and return INVALID_PARAMETER if it is not.
Change-Id: I4e0c8502d996d3da31224b817a5d4ff2c6054e13
[ROCm/ROCR-Runtime commit: 70b1b5b17e]
Allocations from GPU nodes will return VRAM, not system memory.
Only non-paged allocation from GPU nodes is supported. System
memory can only be allocated from CPU nodes (usually node 0).
The HostAccess flag is no longer used to distinguish the memory
type. It only indicates, whether the memory is mapped for CPU
access.
Maintain compatibility with broken KfdTests by returning system
memory for paged-memory requested from GPU nodes.
Change-Id: I514defede735f55e6de436f41944125b6f2c4ccf
[ROCm/ROCR-Runtime commit: 887b32fe86]
This is thunk part of the CWSR support.
1. SDMA queue don't support CWSR , no necessary to allocate the context save/restore memory
2. Allocate the context save/restore memory in local frame buffer for dGPU
Change-Id: Ie83506f0cced2a5a537c49d68125796d831c2764
[ROCm/ROCR-Runtime commit: 4e6c25e55b]
All tonga page size alignment is done in the memory management
functions in fmm.c. All other code only specifies the minimum
alignment it needs and lets fmm.c handle the HW-specific
alignment.
Clean up aligned-exec memory allocation in queue.c to remove
hard-coded TONGA_PAGE_SIZE alignments and remove code duplication.
Make sure alignments are consistent between allocate and free.
Change-Id: Ia8923448173d1cef315af24cebff12adef385cb0
[ROCm/ROCR-Runtime commit: cc9fc386bd]
Stop using NUM_OF_SUPPORTED_GPUS. For now the definitions itself cannot
be removed as ioctl code is in upstream Kernel.
Change-Id: If846625a8ad5062d5483e762850c793d3c00b9d0
[ROCm/ROCR-Runtime commit: ce83dc623f]
Fix hsaKmtRegisterMemory to be a no-op for now and move the multi-GPU
implementation to hsaKmtRegisterMemoryToNodes. Make GPU memory mappings
of host memory visible to all GPUs by default. Device memory is still
visible to the allocating GPU only by default (but can be overridden
with hsaKmtRegisterMemoryToNodes for experimenting with P2P).
Change-Id: I73408afbe3b10c8dad2ab3a780f58413249692e6
[ROCm/ROCR-Runtime commit: 063ad3ad9e]
Fix TONGA_PAGE_SIZE value and move it to libhsakmt.h for usiing it
consistently in all places that require the same alignment for the
same reason. Create a generic alignment helper macro to replace some
incorrect hand-coded size alignments.
Move virtual address and size alignments down into aperture management
functions. Alignment is a per-aperture property that is set during
fmm_init_process_apertures. Doing the alignment there ensures that
all allocations in the same aperture are aligned the same way. Finding
objects by size and address can take the alignment into account.
Also align the size of physical allocations to back aligned virtual
address allocations. CPU mappings do not need to be aligned.
Map anonymous pages over released memory mappings to allow the
backing pages to be released, while keeping the address space
reserved.
Add alignment parameter to free_exec_aligned_memory_gpu to match the
interface of allocate_exec_aligned_memory_cpu. It doesn't make sense
to allow an alignment parameter in one but assume a specific
alignment in the other.
Change-Id: I74226ca6938f4948f643e5aee1d474720cd89e78
[ROCm/ROCR-Runtime commit: 6a5ca4bc5a]
Create new device_info and add device ID. Add helper macros to
identify chip families (VI, discrete). For now gfx803 behaves like
gfx802. But if necessary we can have gfx802 or gfx803-specific
code paths or workarounds in the future.
Change-Id: I61b4ffef7dd7796bb34cb01fbff0089bd49507bb
[ROCm/ROCR-Runtime commit: 0fc0a5b526]
1. Add IOCTL defines to set trap handler
2. Add control stack size information on create queue argument.
3. Increase the total save&restore area size for carrizo to include the control stack size.
Signed-off-by: Shaoyun Liu <Shaoyun.liu@amd.com>
Change-Id: Iccf15e073b7db2519e96e7f7b46a89d57ab9a4df
[ROCm/ROCR-Runtime commit: 2d63ee7b8f]
Yong Zhao
Felix Kuehling
Shaoyun Liu
Jay Cornwall
Laurent Morichetti
Tom Stellard
Amber Lin
Kent Russell
ozeng
Harish Kasiviswanathan
Ben Goz
Philip Cox