AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SWDEV-541514 - Docs update 2025-09-15 (#993)

Browse Source 
Co-authored-by: Julia Jiang <56359287+jujiang-del@users.noreply.github.com>
This commit is contained in:
Istvan Kiss
2025-09-24 18:57:00 +02:00
committed by GitHub
parent 0575606e49
commit 83fb0c8c47
39 changed files with 2973 additions and 807 deletions
Download Patch File Download Diff File Expand all files Collapse all files
projects/hip/docs/how-to/hip_runtime_api/asynchronous.rst
+1 -1
View File
@@ -88,7 +88,7 @@ developer may have to reduce the block size of the kernels. The kernel runtimes
can be misleading for concurrent kernel runs, that is why during optimization
it is a good practice to check the trace files, to see if one kernel is blocking
another kernel, while they are running in parallel. For more information about
the application tracing, check::doc:`rocprofiler:/how-to/using-rocprof`.
application tracing, see :doc:`rocprofiler:how-to/using-rocprof`.
When running kernels in parallel, the execution time can increase due to
contention for shared resources. This is because multiple kernels may attempt
projects/hip/docs/how-to/hip_runtime_api/call_stack.rst
+1 -1
View File
@@ -21,7 +21,7 @@ and AMD GPUs use different approaches. NVIDIA GPUs have the independent thread
scheduling feature where each thread has its own call stack and effective
program counter. On AMD GPUs threads are grouped; each warp has its own call
stack and program counter. Warps are described and explained in the
:ref:`inherent_thread_hierarchy`
:ref:`inherent_thread_model`
If a thread or warp exceeds its stack size, a stack overflow occurs, causing
kernel failure. This can be detected using debuggers.
projects/hip/docs/how-to/hip_runtime_api/error_handling.rst
+4 -5
View File
@@ -21,11 +21,10 @@ without changing it. To get a human readable version of the errors,
.. note::
:cpp:func:`hipGetLastError` returns the returned error code of the last HIP
runtime API call even if it's ``hipSuccess``, while ``cudaGetLastError``
returns the error returned by any of the preceding CUDA APIs in the same
host thread. :cpp:func:`hipGetLastError` behavior will be matched with
``cudaGetLastError`` in ROCm release 7.0.
:cpp:func:`hipGetLastError` returns the last actual HIP API error caught in the current thread
during the application execution. Prior to ROCm 7.0, ``hipGetLastError`` might also return
``hipSuccess`` or ``hipErrorNotReady`` from the last HIP runtime API call, which are not errors.
Best practices of HIP error handling:
projects/hip/docs/how-to/hip_runtime_api/hipgraph.rst
-6
View File
@@ -8,12 +8,6 @@
HIP graphs
********************************************************************************
.. note::
The HIP graph API is currently in Beta. Some features can change and might
have outstanding issues. Not all features supported by CUDA graphs are yet
supported. For a list of all currently supported functions see the
:ref:`HIP graph API documentation<graph_management_reference>`.
HIP graphs are an alternative way of executing tasks on a GPU that can provide
performance benefits over launching kernels using the standard
method via streams. A HIP graph is made up of nodes and edges. The nodes of a
projects/hip/docs/how-to/hip_runtime_api/memory_management/coherence_control.rst
+2
View File
@@ -64,6 +64,8 @@ To check the availability of fine- and coarse-grained memory pools, use
Segment: GLOBAL; FLAGS: COARSE GRAINED
...
.. _hip-memory-coherence-table:
The APIs, flags and respective memory coherence control are listed in the
following table:
projects/hip/docs/how-to/hip_runtime_api/memory_management/host_memory.rst
+2
View File
@@ -108,6 +108,8 @@ C++ application.
:cpp:func:`hipMalloc` and :cpp:func:`hipFree` are blocking calls. However, HIP
also provides non-blocking versions :cpp:func:`hipMallocAsync` and
:cpp:func:`hipFreeAsync`, which require a stream as an additional argument.
For asynchronous memory allocations made with ``hipMallocAsync`` and ``hipMallocFromPoolAsync``
``hipFree`` does not implicitly wait for synchronization, to match the behavior of ``cudaFree``.
.. _pinned_host_memory:
projects/hip/docs/how-to/hip_runtime_api/memory_management/unified_memory.rst
+116 -65
View File
@@ -9,6 +9,14 @@
Unified memory management
*******************************************************************************
This document covers unified memory management in HIP, which encompasses several
approaches that provide a single address space accessible from both CPU and GPU.
**Unified memory** refers to the overall architectural concept of this shared
address space, while **managed memory** is one specific implementation that
provides automatic page migration between devices. Other unified memory allocators
like :cpp:func:`hipMalloc()` and :cpp:func:`hipHostMalloc()` provide different
access patterns within the same unified address space concept.
In conventional architectures CPUs and attached devices have their own memory
space and dedicated physical memory backing it up, e.g. normal RAM for CPUs and
VRAM on GPUs. This way each device can have physical memory optimized for its
@@ -39,57 +47,79 @@ model is shown in the following figure.
Unified memory enables the access to memory located on other devices via
several methods, depending on whether hardware support is available or has to be
managed by the driver.
managed by the driver. CPUs can access memory allocated via :cpp:func:`hipMalloc()`,
providing bidirectional memory accessibility within the unified address space.
Hardware supported on-demand page migration
--------------------------------------------------------------------------------
Managed memory
================================================================================
When a kernel on the device tries to access a memory address that is not in its
memory, a page-fault is triggered. The GPU then in turn requests the page from
the host or an other device, on which the memory is located. The page is then
unmapped from the source, sent to the device and mapped to the device's memory.
The requested memory is then available to the processes running on the device.
Managed Memory is an extension of the unified memory architecture in which HIP
monitors memory access and intelligently migrates data between device and
system memories, thereby improving performance and resource efficiency.
In case the device's memory is at capacity, a page is unmapped from the device's
memory first and sent and mapped to host memory. This enables more memory to be
allocated and used for a GPU, than the GPU itself has physically available.
When a kernel on the device tries to access a managed memory address that is
not in its local device memory, a page-fault is triggered. The GPU then in
turn requests the page from the host or other device on which the memory is
located. The page is unmapped from the source, sent to the device and
mapped to the device's memory. The requested memory is then available locally
to the processes running on the device, which improves performance as local
memory access outperforms remote memory access.
This level of unified memory support can be very beneficial for sparse accesses
to an array, that is not often used on the device.
Driver managed page migration
--------------------------------------------------------------------------------
If the hardware does not support on-demand page migration, then all the pages
accessed by a kernel have to be resident on the device, so they have to be
migrated before the kernel is running. Since the driver can not know beforehand,
what parts of an array are going to be accessed, all pages of all accessed
arrays have to be migrated. This can lead to significant delays on the first run
of a kernel, on top of possibly copying more memory than is actually accessed by
the kernel.
Managed memory also expands the memory capacity available to a GPU kernel. When
migrating memory into the device on page-fault, if the device's memory is
already at capacity, a page is unmapped from the device's memory first and sent
and mapped to host memory. This enables more memory to be allocated and used
for a GPU than the GPU itself has physically available. This level of support
can be very beneficial, for example, for sparse accesses to an array that is
not often used on the device.
.. _unified memory system requirements:
System requirements
================================================================================
System requirements for managed memory
--------------------------------------------------------------------------------
Unified memory is supported on Linux by all modern AMD GPUs from the Vega
series onward, as shown in the following table. Unified memory management can
be achieved by explicitly allocating managed memory using
:cpp:func:`hipMallocManaged` or marking variables with the ``__managed__``
attribute. For the latest GPUs, with a Linux kernel that supports
`Heterogeneous Memory Management (HMM)
Some AMD GPUs do not support page-faults, and thus do not support on-demand
page-fault driven migration. On these architectures, if the programmer prefers
all GPU memory accesses to be local, all pages have to migrated before the
kernel is dispatched, as the driver cannot know beforehand which parts of a
dataset are going to be accessed. This can lead to significant delays on the
first run of a kernel, and, in the example of a sparsely accessed array, can
also lead to copying more memory than is actually accessed by the kernel.
Note that on systems which do not support page-faults, managed memory APIs are
still accessible to the programmer, but managed memory operates in a degraded
fashion due to the lack of demand-driven migration. Furthermore, on these
systems it is still possible to use unified memory allocators that do not
provide managed memory features; see
:ref:`memory allocation approaches in unified memory` for more details.
Managed memory is supported on Linux by all modern AMD GPUs from the Vega
series onward, as shown in the following table. Managed memory can be
explicitly allocated using :cpp:func:`hipMallocManaged()` or marking variables
with the ``__managed__`` attribute. For the latest GPUs, with a Linux kernel
that supports `Heterogeneous Memory Management (HMM)
<https://www.kernel.org/doc/html/latest/mm/hmm.html>`_, the normal system
allocator can be used.
allocators (e.g., ``new``, ``malloc()``) can be used.
.. list-table:: Supported Unified Memory Allocators by GPU architecture
.. note::
To ensure the proper functioning of managed memory on supported GPUs, it
is **essential** to set the environment variable ``HSA_XNACK=1`` and use a
GPU kernel mode driver that supports `HMM
<https://www.kernel.org/doc/html/latest/mm/hmm.html>`_. Without this
configuration, access-driven memory migration will be disabled, and the
behavior will be similar to that of systems without HMM support.
.. list-table:: Managed Memory Support by GPU Architecture
:widths: 40, 25, 25
:header-rows: 1
:align: center
* - Architecture
- :cpp:func:`hipMallocManaged()`, ``__managed__``
- ``new``, ``malloc()``
- ``new``, ``malloc()``, ``allocate()``
* - CDNA4
- ✅
- ✅ :sup:`1`
* - CDNA3
- ✅
- ✅ :sup:`1`
@@ -98,7 +128,7 @@ allocator can be used.
- ✅ :sup:`1`
* - CDNA1
- ✅
- ✅ :sup:`1`
- ❌
* - RDNA1
- ✅
- ❌
@@ -113,11 +143,16 @@ allocator can be used.
:sup:`1` Works only with ``HSA_XNACK=1`` and kernels with HMM support. First GPU
access causes recoverable page-fault.
.. _unified memory allocators:
.. _memory allocation approaches in unified memory:
Unified memory allocators
Memory allocation approaches in unified memory
================================================================================
While managed memory provides automatic migration, unified memory encompasses
several allocation methods, each with different access patterns and migration
behaviors. The following section covers all available unified memory allocation
approaches, including but not limited to managed memory APIs.
Support for the different unified memory allocators depends on the GPU
architecture and on the system. For more information, see :ref:`unified memory
system requirements` and :ref:`checking unified memory support`.
@@ -133,17 +168,17 @@ system requirements` and :ref:`checking unified memory support`.
- **System allocated unified memory**
Starting with CDNA2, the ``new`` and ``malloc()`` system allocators allow
Starting with CDNA2, the ``new``, ``malloc()``, and ``allocate()`` (Fortran) system allocators allow
you to reserve unified memory. The system allocator is more versatile and
offers an easy transition for code written for CPUs to HIP code as the
same system allocation API is used.
same system allocation API is used. Memory allocated by these allocators can
be registered to be accessible on device using :cpp:func:`hipHostRegister()`.
To ensure the proper functioning of system allocated unified memory on supported
GPUs, it is essential to set the environment variable ``HSA_XNACK=1`` and use
a GPU kernel mode driver that supports HMM
<https://www.kernel.org/doc/html/latest/mm/hmm.html>`_. Without this
configuration, the behavior will be similar to that of systems without HMM
support.
- **HIP allocated non-managed memory**
:cpp:func:`hipMalloc()` and :cpp:func:`hipHostMalloc()` are dynamic memory
allocators available on all GPUs with unified memory support. Memory
allocated by these allocators is not migrated between device and host memory.
The table below illustrates the expected behavior of managed and unified memory
functions on ROCm and CUDA, both with and without HMM support.
@@ -161,10 +196,10 @@ functions on ROCm and CUDA, both with and without HMM support.
- Access outside the origin without HMM or ``HSA_XNACK=0``
- Allocation origin with HMM and ``HSA_XNACK=1``
- Access outside the origin with HMM and ``HSA_XNACK=1``
* - ``new``, ``malloc()``
* - ``new``, ``malloc()``, ``allocate()``
- host
- not accessible on device
- host
- first touch
- page-fault migration
* - :cpp:func:`hipMalloc()`
- device
@@ -174,13 +209,13 @@ functions on ROCm and CUDA, both with and without HMM support.
* - :cpp:func:`hipMallocManaged()`, ``__managed__``
- pinned host
- zero copy [zc]_
- host
- first touch
- page-fault migration
* - :cpp:func:`hipHostRegister()`
- undefined behavior
- undefined behavior
- host
- page-fault migration
- pinned host
- zero copy [zc]_
- pinned host
- zero copy [zc]_
* - :cpp:func:`hipHostMalloc()`
- pinned host
- zero copy [zc]_
@@ -236,9 +271,9 @@ functions on ROCm and CUDA, both with and without HMM support.
Checking unified memory support
--------------------------------------------------------------------------------
The following device attributes can offer information about which :ref:`unified
memory allocators` are supported. The attribute value is 1 if the functionality
is supported, and 0 if it is not supported.
The following device attributes can offer information about which :ref:`memory
allocation approaches in unified memory` are supported. The attribute value is
1 if the functionality is supported, and 0 if it is not supported.
.. list-table:: Device attributes for unified memory management
:widths: 40, 60
@@ -364,10 +399,11 @@ explicit memory management example is presented in the last tab.
.. tab-item:: new
.. code-block:: cpp
:emphasize-lines: 20-23
:emphasize-lines: 21-24
#include <hip/hip_runtime.h>
#include <iostream>
#include <new>
#define HIP_CHECK(expression) \
{ \
@@ -386,10 +422,10 @@ explicit memory management example is presented in the last tab.
// This example requires HMM support and the environment variable HSA_XNACK needs to be set to 1
int main() {
// Allocate memory for a, b, and c.
int *a = new int[1];
int *b = new int[1];
int *c = new int[1];
// Allocate memory with proper alignment for performance
int *a = new(std::align_val_t(128)) int[1];
int *b = new(std::align_val_t(128)) int[1];
int *c = new(std::align_val_t(128)) int[1];
// Setup input values.
*a = 1;
@@ -404,10 +440,10 @@ explicit memory management example is presented in the last tab.
// Prints the result.
std::cout << *a << " + " << *b << " = " << *c << std::endl;
// Cleanup allocated memory.
delete[] a;
delete[] b;
delete[] c;
// Cleanup allocated memory with matching aligned delete.
::operator delete[](a, std::align_val_t(128));
::operator delete[](b, std::align_val_t(128));
::operator delete[](c, std::align_val_t(128));
return 0;
}
@@ -501,9 +537,24 @@ Performance optimizations for unified memory
There are several ways, in which the developer can guide the runtime to reduce
copies between devices, in order to improve performance.
With ``numactl --membind`` bindings, developers can control where physical
allocation occurs by restricting memory allocation to specific NUMA nodes.
This approach can reduce or eliminate the need for explicit data prefetching
since memory is allocated in the desired location from the start.
Data prefetching
--------------------------------------------------------------------------------
.. warning::
Data prefetching is not always an optimization and can slow down execution,
as the API takes time to execute. If the memory is already in the right
place, prefetching will waste time. Users should profile their code to
verify whether prefetching is beneficial for their specific use case.
When prefetching is beneficial, developers can consider setting different default
locations for different devices and using prefetch between them, which can help
eliminate IPC communication overhead when memory moves between devices.
Data prefetching is a technique used to improve the performance of your
application by moving data to the desired device before it's actually
needed. ``hipCpuDeviceId`` is a special constant to specify the CPU as target.