AI/rocm-systems
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update docs 2025 04 14 (#54)

Browse Source 
* Update docs 2025 03 31

- Docs: remove virtual_rocr.rst
- Fix documentation  warnings
- Reformat HIP RTC
- Docs: Refactor HIP porting guide
- Docs: Expand HIP porting guide and CUDA driver porting guide
- Minor fix
- Docs: Update environment variables file
- Bump rocm-docs-core[api_reference] from 1.15.0 to 1.17.0 in /docs/sphinx
- Docs: Update FP8 page to show both FP8 and FP16 types
- Bump sphinxcontrib-doxylink from 1.12.4 to 1.13.0 in /docs/sphinx
- Bumps [rocm-docs-core[api_reference]](https://github.com/ROCm/rocm-docs-core) from 1.17.0 to 1.17.1.
- Remove external link
- Update programming model
- Bump rocm-docs-core[api_reference] from 1.17.1 to 1.18.1 in /docs/sphinx
- Docs: Add page for Complex Math API
- Docs: Add page about HIP error codes
- Update docs: the compilation cache is enabled by default
- Fix fns32 function mask type in doc

* Bump rocm-docs-core[api_reference] from 1.18.1 to 1.18.2 in /docs/sphinx

Bumps [rocm-docs-core[api_reference]](https://github.com/ROCm/rocm-docs-core) from 1.18.1 to 1.18.2.
- [Release notes](https://github.com/ROCm/rocm-docs-core/releases)
- [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/ROCm/rocm-docs-core/compare/v1.18.1...v1.18.2)

---
updated-dependencies:
- dependency-name: rocm-docs-core[api_reference]
  dependency-version: 1.18.2
  dependency-type: direct:production
  update-type: version-update:semver-patch

* Fix readme link

* Docs: Fix verbose paths generated by doxygen

* Handle git ssh in docs conf.py
This commit is contained in:
Kiss, Istvan authored and GitHub committed 2025-06-02 20:40:41 +05:30
1 parent b301ef2282
commit d0cf32a63a
46 files changed
+6457 -3181
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/understand/compilers.rst
+75
View File
@@ -21,6 +21,81 @@ On NVIDIA CUDA platform, ``hipcc`` takes care of invoking compiler ``nvcc``.
``amdclang++`` is based on the ``clang++`` compiler. For more
details, see the :doc:`llvm project<llvm-project:index>`.
HIPCC
================================================================================
Common Compiler Options
--------------------------------------------------------------------------------
The following table shows the most common compiler options supported by
``hipcc``.
.. list-table::
:header-rows: 1
*
- Option
- Description
*
- ``--fgpu-rdc``
- Generate relocatable device code, which allows kernels or device functions
to call device functions in different translation units.
*
- ``-ggdb``
- Equivalent to `-g` plus tuning for GDB. This is recommended when using
ROCm's GDB to debug GPU code.
*
- ``--gpu-max-threads-per-block=<num>``
- Generate code to support up to the specified number of threads per block.
*
- ``-offload-arch=<target>``
- Generate code for the given GPU target.
For a full list of supported compilation targets see the `processor names in AMDGPU's llvm documentation <https://llvm.org/docs/AMDGPUUsage.html#processors>`_.
This option can appear multiple times to generate a fat binary for multiple
targets.
The actual support of the platform's runtime may differ.
*
- ``-save-temps``
- Save the compiler generated intermediate files.
*
- ``-v``
- Show the compilation steps.
Linking
--------------------------------------------------------------------------------
``hipcc`` adds the necessary libraries for HIP as well as for the accelerator
compiler (``nvcc`` or ``amdclang++``). We recommend linking with ``hipcc`` since
it automatically links the binary to the necessary HIP runtime libraries.
Linking Code With Other Compilers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
``nvcc`` by default uses ``g++`` to generate the host code.
``amdclang++`` generates both device and host code. The code uses the same API
as ``gcc``, which allows code generated by different ``gcc``-compatible
compilers to be linked together. For example, code compiled using ``amdclang++``
can link with code compiled using compilers such as ``gcc``, ``icc`` and
``clang``. Take care to ensure all compilers use the same standard C++ header
and library formats.
libc++ and libstdc++
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
``hipcc`` links to ``libstdc++`` by default. This provides better compatibility
between ``g++`` and HIP.
In order to link to ``libc++``, pass ``--stdlib=libc++`` to ``hipcc``.
Generally, libc++ provides a broader set of C++ features while ``libstdc++`` is
the standard for more compilers, notably including ``g++``.
When cross-linking C++ code, any C++ functions that use types from the C++
standard library, such as ``std::string``, ``std::vector`` and other containers,
must use the same standard-library implementation. This includes cross-linking
between ``amdclang++`` and other compilers.
HIP compilation workflow
================================================================================
docs/understand/hardware_implementation.rst
+7 -2
View File
@@ -45,12 +45,13 @@ The amount of warps that can reside concurrently on a CU, known
as occupancy, is determined by the warp's resource usage of registers and
shared memory.
.. _gcn_cu:
.. figure:: ../data/understand/hardware_implementation/compute_unit.svg
:alt: Diagram depicting the general structure of a compute unit of an AMD
GPU.
An AMD Graphics Core Next (GCN) CU. The CDNA and RDNA CUs are based on
variations of the GCN CU.
AMD Graphics Core Next (GCN) CU
On AMD GCN GPUs the basic structure of a CU is:
@@ -102,6 +103,8 @@ The scalar unit performs instructions that are uniform within a warp. It
thereby improves efficiency and reduces the pressure on the vector ALUs and the
vector register file.
.. _cdna3_cu:
CDNA architecture
=================
@@ -121,6 +124,8 @@ multiply-accumulate operations for
Block Diagram of a CDNA3 Compute Unit.
.. _rdna3_cu:
RDNA architecture
=================
docs/understand/programming_model.rst
+358 -188
View File
@@ -7,67 +7,96 @@
.. _programming_model:
*******************************************************************************
HIP programming model
Introduction to the HIP programming model
*******************************************************************************
The HIP programming model makes it easy to map data-parallel C/C++ algorithms to
massively parallel, wide single instruction, multiple data (SIMD) architectures,
such as GPUs.
The HIP programming model enables mapping data-parallel C/C++ algorithms to massively
parallel SIMD (Single Instruction, Multiple Data) architectures like GPUs. HIP
supports many imperative languages, such as Python via PyHIP, but this document
focuses on the original C/C++ API of HIP.
While the model may be expressed in most imperative languages, (for example
Python via PyHIP) this document will focus on the original C/C++ API of HIP.
A basic understanding of the underlying device architecture helps you
While GPUs may be capable of running applications written for CPUs if properly ported
and compiled, it would not be an efficient use of GPU resources. GPUs fundamentally differ
from CPUs and should be used accordingly to achieve optimum
performance. A basic understanding of the underlying device architecture helps you
make efficient use of HIP and general purpose graphics processing unit (GPGPU)
programming in general.
programming in general. The following topics introduce you to the key concepts of
GPU-based programming and the HIP programming model.
RDNA & CDNA architecture summary
Hardware differences: CPU vs GPU
================================
GPUs in general are made up of basic building blocks called compute units (CUs),
that execute the threads of a kernel. These CUs provide the necessary resources
for the threads: the Arithmetic Logical Units (ALUs), register files, caches and
shared memory for efficient communication between the threads.
CPUs and GPUs have been designed for different purposes. CPUs quickly execute a single thread, decreasing the time for a single operation while increasing the number of sequential instructions that can be executed. This includes fetching data and reducing pipeline stalls where the ALU has to wait for previous instructions to finish.
This design allows for efficient execution of kernels while also being able to
scale from small GPUs embedded in APUs with few CUs up to GPUs designed for data
centers with hundreds of CUs. Figure :ref:`rdna3_cu` and :ref:`cdna3_cu` show
examples of such compute units.
.. figure:: ../data/understand/programming_model/cpu-gpu-comparison.svg
:alt: Diagram depicting the differences between CPU and GPU hardware.
The CPU block shows four large processing cores, lists Large Cache per
Core, and High Clock Speed of 3 to 5 gigahertz. The GPU block shows 42
smaller processing cores, lists Shared Memory across Cores, and Lower
Clock Speeds of 1 to 2 gigahertz.
For architecture details, check :ref:`hardware_implementation`.
Differences in CPUs and GPUs
.. _rdna3_cu:
With CPUs, the goal is to quickly process operations. CPUs provide low-latency processing for
serial instructions. On the other hand, GPUs have been designed to execute many similar commands, or threads,
in parallel, achieving higher throughput. Latency is the time between starting an
operation and receiving its result, such as 2 ns, while throughput is the rate of
completed operations, for example, operations per second.
.. figure:: ../data/understand/programming_model/rdna3_cu.png
:alt: Block diagram showing the structure of an RDNA3 Compute Unit. It
consists of four SIMD units, each including a vector and scalar register
file, with the corresponding scalar and vector ALUs. All four SIMDs
share a scalar and instruction cache, as well as the shared memory. Two
of the SIMD units each share an L0 cache.
For the GPU, the objective is to process as many operations in parallel, rather
than to finish a single instruction quickly. GPUs in general are made up of basic
building blocks called compute units (CUs), that execute the threads of a kernel.
As described in :ref:`hardware_implementation`, these CUs provide the necessary
resources for the threads: the Arithmetic Logical Units (ALUs), register files,
caches and shared memory for efficient communication between the threads.
Block Diagram of an RDNA3 Compute Unit.
The following describes a few hardware differences between CPUs and GPUs:
.. _cdna3_cu:
* CPU:
.. figure:: ../data/understand/programming_model/cdna3_cu.png
:alt: Block diagram showing the structure of a CDNA3 compute unit. It includes
Shader Cores, the Matrix Core Unit, a Local Data Share used for sharing
memory between threads in a block, an L1 Cache and a Scheduler. The
Shader Cores represent the vector ALUs and the Matrix Core Unit the
matrix ALUs. The Local Data Share is used as the shared memory.
- Optimized for sequential processing with a few powerful cores (4-64 typically)
- High clock speeds (3-5 GHz)
- One register file per thread. On modern CPUs you have at most 2 register files per core, called hyperthreading.
- One ALU executing the thread.
Block Diagram of a CDNA3 Compute Unit.
- Designed to quickly execute instructions of the same thread.
- Complex branch prediction.
Heterogeneous Programming
- Large L1/L2 cache per core, shared by fewer threads (maximum of 2 when hyperthreading is available).
- A disadvantage is switching execution from one thread to another (or context switching) takes a considerable amount of time: the ALU pipeline needs to be emptied, the register file has to be written to memory to free the register for another thread.
* GPU:
- Designed for parallel processing with many simpler cores (hundreds/thousands)
- Lower clock speeds (1-2 GHz)
- Streamlined control logic
- Small caches, more registers
- Register files are shared among threads. The number of threads that can be run in parallel depends on the registers needed per thread.
- Multiple ALUs execute a collection of threads having the same operations, also known as a wavefront or warp. This is called single-instruction, multiple threads (SIMT) operation as described in :ref:`programming_model_simt`.
- The collection of ALUs is called SIMD. SIMDs are an extension to the hardware architecture that allows a `single instruction` to concurrently operate on `multiple data` inputs.
- For branching threads where conditional instructions lead to thread divergence, ALUs still process the full wavefront, but the result for divergent threads is masked out. This leads to wasted ALU cycles and should be a consideration in your programming. Keep instructions consistent and leave conditionals out of threads.
- The advantage for GPUs is that context switching is easy. All threads that run on a core/compute unit have their registers on the compute unit, so they don't need to be stored to global memory, and each cycle one instruction from any wavefront that resides on the compute unit can be issued.
When programming for a heterogeneous system, which incorporates CPUs and GPUs, you must
write your program to take advantage of the strengths of the available hardware.
Use the CPU for tasks that require complex logic with conditional branching, to reduce the
time to reach a decision. Use the GPU for parallel operations of the same instruction
across large datasets, with little branching, where the volume of operations is the key.
.. _heterogeneous_programming:
Heterogeneous programming
=========================
The HIP programming model assumes two execution contexts. One is referred to as
*host* while compute kernels execute on a *device*. These contexts have
different capabilities, therefor slightly different rules apply. The *host*
execution is defined by the C++ abstract machine, while *device* execution
follows the :ref:`SIMT model<programming_model_simt>` of HIP. These execution contexts in
code are signified by the ``__host__`` and ``__device__`` decorators. There are
a few key differences between the two:
The HIP programming model has two execution contexts. The main application starts on the CPU, or
the *host* processor, and compute kernels are launched on the *device* such as `Instinct
accelerators <https://www.amd.com/en/products/accelerators/instinct.html>`_ or AMD GPUs.
The host execution is defined by the C++ abstract machine, while device execution
follows the :ref:`SIMT model<programming_model_simt>` of HIP. These two execution contexts
are signified by the ``__host__`` and ``__global__`` (or ``__device__``) decorators
in HIP program code. There are a few key differences between the two contexts:
* The C++ abstract machine assumes a unified memory address space, meaning that
one can always access any given address in memory (assuming the absence of
@@ -75,96 +104,160 @@ a few key differences between the two:
from one means nothing in another. Moreover, not all address spaces are
accessible from all contexts.
Looking at :ref:`rdna3_cu` and :ref:`cdna3_cu`, you can see that
every CU has an instance of storage backing the namespace ``__shared__``.
Even if the host were to have access to these regions of
memory, the performance benefits of the segmented memory subsystem are
Looking at the :ref:`gcn_cu` figure, you can see that every CU has an instance of storage
backing the namespace ``__shared__``. Even if the host were to have access to these
regions of memory, the performance benefits of the segmented memory subsystem are
supported by the inability of asynchronous access from the host.
* Not all C++ language features map cleanly to typical device architectures,
some are very expensive (meaning slow) to implement on GPU devices, therefor
they are forbidden in device contexts to avoid users tapping into features
that unexpectedly decimate their program's performance. Offload devices targeted
by HIP aren't general purpose devices, at least not in the sense that a CPU is.
HIP focuses on data parallel computations and as such caters to throughput
optimized architectures, such as GPUs or accelerators derived from GPU
architectures.
* Not all C++ language features map cleanly to typical GPU device architectures.
Some C++ features have poor latency when implemented on GPU devices, therefore
they are forbidden in device contexts to avoid using features that unexpectedly
decimate the program's performance. Offload devices targeted by HIP aren't general
purpose devices, at least not in the sense that a CPU is. HIP focuses on data
parallel computations and as such caters to throughput optimized architectures,
such as GPUs or accelerators derived from GPU architectures.
* Asynchrony is at the forefront of the HIP API. Computations launched on the device
* Asynchronicity is at the forefront of the HIP API. Computations launched on the device
execute asynchronously with respect to the host, and it is the user's responsibility to
synchronize their data dispatch/fetch with computations on the device.
.. note::
HIP does perform implicit synchronization on occasions, more advanced than other
APIs such as OpenCL or SYCL, in which the responsibility of synchronization mostly
depends on the user.
HIP performs implicit synchronization on occasions, unlike some
APIs where the responsibility for synchronization is left to the user.
Host programming
----------------
In heterogeneous programming, the CPU is available for processing operations but the host application has the additional task of managing data and computation exchanges between the CPU (host) and GPU (device). The host acts as the application manager, coordinating the overall workflow and directing operations to the appropriate context, handles data preparation and data transfers, and manages GPU tasks and synchronization. Here is a typical sequence of operations:
1. Initialize the HIP runtime and select the GPU: As described in :ref:`initialization`, refers to identifying and selecting a target GPU, setting up a context to let the CPU interact with the GPU.
2. Data preparation: As discussed in :ref:`memory_management`, this includes allocating the required memory on the host and device, preparing input data and transferring it from the host to the device. The data is both transferred to the device, and passed as an input parameter when launching the kernel.
3. Configure and launch the kernel on the GPU: As described in :ref:`device_program`, this defines kernel configurations and arguments, launches kernel to run on the GPU device using the triple chevron syntax or appropriate API call (for example ``hipLaunchKernelGGL``). On the GPU, multiple kernels can run on streams, with a queue of operations. Within the same stream, operations run in the order they were issued, but on multiple streams operations are independent and can execute concurrently. In the HIP runtime, kernels run on the default stream when one is not specified, but specifying a stream for the kernel lets you increase concurrency in task scheduling and resource utilization, and launch and manage multiple kernels from the host program.
4. Synchronization: As described in :ref:`asynchronous_how-to`, kernel execution occurs in the context of device streams, specifically the default (`0`) stream. You can use streams and events to manage task dependencies, overlap computation with data transfers, and manage asynchronous processes to ensure proper sequencing of operations. Wait for events or streams to finish execution and transfer results from the GPU back to the host.
5. Error handling: As described in :ref:`error_handling`, you should catch and handle potential errors from API calls, kernel launches, or memory operations. For example, use ``hipGetErrorString`` to retrieve error messages.
6. Cleanup and resource management: Validate results, clean up GPU contexts and resources, and free allocated memory on the host and devices.
This structure allows for efficient use of GPU resources and facilitates the acceleration of compute-intensive tasks while keeping the host CPU available for other tasks.
.. figure:: ../data/understand/programming_model/host-device-flow.svg
:alt: Diagram depicting a host CPU and device GPU rectangles of varying color.
There are arrows pointing between the rectangles showing from the Host
to the Device the initialization, data transfer, and Kernel execution
steps, and from the Device back to the Host the returning results.
Interaction of Host and Device in a GPU application
.. _device_program:
Device programming
------------------
The device or kernel program acts as workers on the GPU application, distributing operations to be handled quickly and efficiently. Launching a kernel in the host application starts the kernel program running on the GPU, defining the parallel operations to repeat the same instructions across many datasets. Understanding how the kernel works and the processes involved is essential to writing efficient GPU applications. Threads, blocks, and grids provide a hierarchical approach to parallel operations. Understanding the thread hierarchy is critical to distributing work across the available CUs, managing parallel operations, and optimizing memory access. The general flow of the kernel program looks like this:
1. Thread Grouping: As described in :ref:`inherent_thread_model`, threads are organized into a hierarchy consisting of threads, which are individual instances of parallel operations, blocks that group the threads, and grids that group blocks into the kernel. Each thread runs an instance of the kernel in parallel with other threads in the block.
2. Indexing: The kernel computes the unique index for each thread to access the relevant data to be processed by the thread.
3. Data Fetch: Threads fetch input data from memory previously transferred from the host to the device. As described in :ref:`memory_hierarchy`, the hierarchy of threads is influenced by the memory subsystem of GPUs. The memory hierarchy includes local memory per-thread with very fast access, shared memory for the block of threads which also supports quick access, and larger amounts of global memory visible to the whole kernel,but accesses are expensive due to high latency. Understanding the memory model is a key concept for kernel programming.
4. Computation: Threads perform the required computations on the input data, and generate any needed output. Each thread of the kernel runs the same instruction simultaneously on the different datasets. This sometimes require multiple iterations when the number of operations exceeds the resources of the CU.
5. Synchronization: When needed, threads synchronize within their block to ensure correct results when working with shared memory.
Kernels are parallel programs that execute the same instruction set across multiple threads, organized in wavefronts, as described below and as demonstrated in the `Hello World tutorial <https://github.com/ROCm/rocm-examples/tree/develop/HIP-Basic/hello_world>`_ or :doc:`../tutorial/saxpy`. However, heterogeneous GPU applications can also become quite complex, managing hundreds, thousands, or hundreds of thousands of operations with repeated data transfers between host and device to support massive parallelization, using multiple streams to manage concurrent asynchronous operations, using rich libraries of functions optimized for GPU hardware as described in the `ROCm documentation <https://rocm.docs.amd.com/en/latest/>`_.
.. _programming_model_simt:
Single instruction multiple threads (SIMT)
==========================================
The SIMT programming model behind the HIP device-side execution is a middle-ground
between SMT (Simultaneous Multi-Threading) programming known from multicore CPUs,
and SIMD (Single Instruction, Multiple Data) programming mostly known from exploiting
relevant instruction sets on CPUs (for example SSE/AVX/Neon).
The HIP kernel code, written as a series of scalar instructions for multiple
threads with different thread indices, gets mapped to the SIMD units of the GPUs.
Every single instruction, which is executed for every participating thread of a
kernel, gets mapped to the SIMD.
A HIP device compiler maps SIMT code written in HIP C++ to an inherently SIMD
architecture (like GPUs). This is done by scalarizing the entire kernel and issuing the scalar
instructions of multiple kernel instances (called threads) to each of the SIMD engine lanes, rather
than exploiting data parallelism within a single instance of a kernel and spreading
identical instructions over the available SIMD engines.
Consider the following kernel:
.. code-block:: cpp
__global__ void k(float4* a, const float4* b)
{
int tid = threadIdx.x;
int bid = blockIdx.x;
int dim = blockDim.x;
a[tid] += (tid + bid - dim) * b[tid];
}
The incoming four-vector of floating-point values ``b`` is multiplied by a
scalar and then added element-wise to the four-vector floating-point values of
``a``. On modern SIMD-capable architectures, the four-vector ops are expected to
compile to a single SIMD instruction. However, GPU execution of this kernel will
typically break down the vector elements into 4 separate threads for parallel execution,
as seen in the following figure:
This is done by grouping threads into warps, which contain as many threads as there
are physical lanes in a SIMD, and issuing that instruction to the SIMD for every
warp of a kernel. Ideally, the SIMD is always fully utilized. However, if the number of threads
can't be evenly divided by the warpSize, then the unused lanes are masked out
from the corresponding SIMD execution.
.. _simt:
.. figure:: ../data/understand/programming_model/simt.svg
:alt: Image representing the instruction flow of a SIMT program. Two identical
arrows pointing downward with blocks representing the instructions
inside and ellipsis between the arrows. The instructions represented in
the arrows are, from top to bottom: ADD, DIV, FMA, FMA, FMA and FMA.
.. figure:: ../data/understand/programming_model/simt-execution.svg
:alt: Diagram depicting the SIMT execution model. There is a red rectangle
which contains the expression a[i] = b[i] + c[i], and below that four
arrows that point to Thread 0,1,2, and 3. Each thread contains different
values for b, c, and a, showing the parallel operations of this equation.
Instruction flow of the sample SIMT program.
Instruction flow of a sample SIMT program
A kernel follows the same C++ rules as the functions on the host, but it has a special ``__global__`` label to mark it for execution on the device, as shown in the following example:
.. code-block:: cpp
__global__ void AddKernel(float* a, const float* b)
{
int global_idx = threadIdx.x + blockIdx.x * blockDim.x;
a[global_idx] += b[global_idx];
}
One of the first things you might notice is the usage of the special ``threadIdx``,
``blockIdx`` and ``blockDim`` variables. Unlike normal C++ host functions, a kernel
is not launched once, but as often as specified by the user. Each of these instances
is a separate thread, with its own values for ``threadIdx``, ``blockIdx`` and ``blockDim``.
The kernel program is launched from the host application using a language extension
called the triple chevron syntax, which looks like the following:
.. code-block:: cpp
AddKernel<<<number_of_blocks, threads_per_block>>>(a, b);
Inside the angle brackets, provide the following:
* The number of blocks to launch, which defines the grid size (relating to blockDim).
* The number of threads in a block, which defines the block size (relating to blockIdx).
* The amount of shared memory to allocate by the host, not specified above.
* The device stream to enqueue the operation on, not specified above so the default stream is used.
.. note::
The kernel can also be launched through other methods, such as the ``hipLaunchKernel()`` function.
Here, the total number of threads launched for the ``AddKernel`` program is defined by
``number_of_blocks * threads_per_block``. You define these values when launching the
kernel program to address the problem to be solved with the available resources within
the system. In other words, the thread configuration is customized to the needs of the
operations and the available hardware.
For comparison, the ``AddKernel`` program could be written in plain C++ as a ``FOR`` loop:
.. code-block:: cpp
for(int i = 0; i < (number_of_blocks * threads_per_block); ++i){
a[i] += b[i];
}
In HIP, lanes of the SIMD architecture are fed by mapping threads of a SIMT
execution, one thread down each lane of an SIMD engine. Execution parallelism
usually isn't exploited from the width of the built-in vector types, but across multiple threads via the thread ID constants ``threadIdx.x``, ``blockIdx.x``, etc.
usually isn't exploited from the width of the built-in vector types, but across
multiple threads via the thread ID constants ``threadIdx.x``, ``blockIdx.x``, etc.
.. _inherent_thread_model:
Inherent thread model
=====================
Hierarchical thread model
---------------------
The SIMT nature of HIP is captured by the ability to execute user-provided
device programs, expressed as single-source C/C++ functions or sources compiled
online/offline to binaries, in bulk.
As previously discussed, all threads of a kernel are uniquely identified by a set
of integral values called thread IDs. The hierarchy consists of three levels: thread,
blocks, and grids.
All threads of a kernel are uniquely identified by a set of integral values, called thread IDs.
The set of integers identifying a thread relate to the hierarchy in which the threads execute.
* Threads are single instances of kernel operations, running concurrently across warps
* Blocks group threads together and enable cooperation and shared memory
* Grids define the number of thread blocks for a single kernel launch
* Blocks and grids can be defined in 3 dimensions (``x``, ``y``, ``z``)
* By default, the Y and Z dimensions are set to 1
The thread hierarchy inherent to how AMD GPUs operate is depicted in the
following figure.
.. _inherent_thread_hierarchy:
The combined values represent the thread index, and relate to the sequence that the
threads execute. The thread hierarchy is integral to how AMD GPUs operate, and is
depicted in the following figure.
.. figure:: ../data/understand/programming_model/thread_hierarchy.svg
:alt: Diagram depicting nested rectangles of varying color. The outermost one
@@ -175,10 +268,13 @@ following figure.
Hierarchy of thread groups.
.. _wavefront:
Warp (or Wavefront)
The innermost grouping of threads is called a warp, or a wavefront in ISA terms. A warp
is the most tightly coupled groups of threads, both physically and logically. Threads
inside a warp are also called lanes, and the integral value identifying them is the lane ID.
The innermost grouping of threads is called a warp. A warp is the most tightly
coupled groups of threads, both physically and logically. Threads inside a warp
are executed in lockstep, with each thread executing the same instruction. Threads
in a warp are also called lanes, and the value identifying them is the lane ID.
.. tip::
@@ -187,41 +283,52 @@ Warp (or Wavefront)
calculated values to be.
The size of a warp is architecture dependent and always fixed. For AMD GPUs
the wavefront is typically 64 threads, though sometimes 32 threads. Warps are
the warp is typically 64 threads, though sometimes 32 threads. Warps are
signified by the set of communication primitives at their disposal, as
discussed in :ref:`warp-cross-lane`.
.. _inherent_thread_hierarchy_block:
Block
The middle grouping is called a block or thread block. The defining feature
of a block is that all threads in a block will share an instance of memory
which they may use to share data or synchronize with one another.
The next level of the thread hierarchy is called a thread block, or block. The
defining feature of a block is that all threads in the block have shared memory
that they can use to share data or synchronize with one another, as described in
:ref:`memory_hierarchy`.
The size of a block is user-configurable but is limited by the queryable
capabilities of the executing hardware. The unique ID of the thread within a
block is 3-dimensional as provided by the API. When linearizing thread IDs
within a block, assume the "fast index" being dimension ``x``, followed by
the ``y`` and ``z`` dimensions.
The size of a block, or the block dimension, is the user-configurable number of
threads per block, but is limited by the queryable capabilities of the executing
hardware. The unique ID of the thread within a block can be 1, 2, or 3-dimensional
as provided by the HIP API. You can configure the thread block to best represent
the data associated with the kernel instruction set.
.. note::
When linearizing thread IDs within a block, assume the *fast index* is the ``x``
dimension, followed by the ``y`` and ``z`` dimensions.
.. _inherent_thread_hierarchy_grid:
Grid
The outermost grouping is called a grid. A grid manifests as a single
dispatch of kernels for execution. The unique ID of each block within a grid
is 3-dimensional, as provided by the API and is queryable by every thread
within the block.
The top-most level of the thread hierarchy is a grid. A grid is the number of blocks
needed for a single launch of the kernel. The unique ID of each block within
a grid can be 1, 2, or 3-dimensional, as provided by the API and is queryable
by every thread within the block.
The three-dimensional thread hierarchy available to a kernel program lends itself to solutions
that align closely to the computational problem. The following are some examples:
* 1-dimensional: array processing, linear data structures, or sequential data transformation
* 2-dimensional: Image processing, matrix operations, 2 dimensional simulations
* 3-dimensional: Volume rendering, 3D scientific simulations, spatial algorithms
Cooperative groups thread model
-------------------------------
The Cooperative groups API introduces new APIs to launch, group, subdivide,
The Cooperative groups API introduces new functions to launch, group, subdivide,
synchronize and identify threads, as well as some predefined group-collective
algorithms, but most importantly a matching threading model to think in terms of.
It relaxes some restrictions of the :ref:`inherent_thread_model` imposed by the
strict 1:1 mapping of architectural details to the programming model. Cooperative
groups let you define your own set of thread groups which may fit your user-cases
better than the defaults defined by the hardware.
algorithms. Cooperative groups let you define your own set of thread groups which
may fit your use-cases better than those defined by the hardware. It relaxes some
restrictions of the :ref:`inherent_thread_model` imposed by the strict 1:1 mapping
of architectural details to the programming model.
.. note::
The implicit groups defined by kernel launch parameters are still available
@@ -229,14 +336,17 @@ better than the defaults defined by the hardware.
For further information, see :doc:`Cooperative groups </how-to/hip_runtime_api/cooperative_groups>`.
.. _memory_hierarchy:
Memory model
============
The hierarchy of threads introduced by the :ref:`inherent_thread_model` is induced
by the memory subsystem of GPUs. The following figure summarizes the memory
namespaces and how they relate to the various levels of the threading model.
.. _memory_hierarchy:
The GPU memory architecture is designed to support parallel execution across the
thread hierarchy. Understanding the following memory spaces and their relationships
to thread groupings is crucial for efficient GPU programming. The choice of memory
type and access patterns significantly impacts kernel performance. The following figure
summarizes the memory namespaces and how they relate to the various levels of the
threading model.
.. figure:: ../data/understand/programming_model/memory_hierarchy.svg
:alt: Diagram depicting nested rectangles of varying color. The outermost one
@@ -250,22 +360,33 @@ namespaces and how they relate to the various levels of the threading model.
Local or per-thread memory
Read-write storage only visible to the threads defining the given variables,
also called per-thread memory. The size of a block for a given kernel, and thereby
the number of concurrent warps, are limited by local memory usage.
This relates to an important aspect: occupancy. This is the default memory
namespace.
also called per-thread memory. This is the default memory namespace.
The size of the blocks for a given kernel, and thereby the number of concurrent
warps, are limited by local memory usage. This relates to the *occupancy* of the
CU as described in :doc:`Compute Units <./hardware_implementation>`,
an important concept in resource usage and performance optimization.
Use local memory when the data is specific to a thread, to store variables generated
by the thread, or to provide register pressure relief for the thread.
Shared memory
Read-write storage visible to all the threads in a given block.
Read-write storage visible to all the threads in a given block. Use shared memory
when the data is reused within a thread block, when cross-thread communication
is needed, or to minimize global memory transactions by using device memory
whenever possible.
Global
Read-write storage visible to all threads in a given grid. There are
specialized versions of global memory with different usage semantics which
are typically backed by the same hardware storing global.
are typically backed by the same hardware storing global.
Use global memory when you have large datasets, are transferring memory between
the host and the device, and when you are sharing data between thread blocks.
Constant
Read-only storage visible to all threads in a given grid. It is a limited
segment of global with queryable size.
segment of global with queryable size. Use constant memory for read-only data
that is shared across multiple threads, and that has a small data size.
Texture
Read-only storage visible to all threads in a given grid and accessible
@@ -274,54 +395,86 @@ Global
Surface
A read-write version of texture memory.
Memory optimizations and best practices
---------------------------------------
.. figure:: ../data/understand/programming_model/memory-access.svg
:alt: Diagram depicting an example memory access pattern for coalesced memory.
The diagram has uncoalesced access on the left side, with consecutive
threads accessing memory in a random pattern. With coalesced access on the
right showing consecutive threads accessing consecutive memory addresses.
Coalesced memory accesses
The following are a few memory access patterns and best practices to improve performance. You can find additional information in :ref:`memory_management` and :doc:`../how-to/performance_guidelines`.
* **Global memory**: Coalescing reduces the number of memory transactions.
Coalesced memory access in HIP refers to the optimization of memory transactions to maximize throughput when accessing global memory. When a kernel accesses global memory, the memory transactions typically occur in chunks of 32, 64, or 128 bytes, which must be naturally aligned. Coalescing memory accesses means aligning and organizing these accesses so that multiple threads in a warp can combine their memory requests into the fewest possible transactions. If threads access memory in a coalesced manner, meaning consecutive threads read or write consecutive memory locations, the memory controller can merge these accesses into a single transaction. This is crucial because global memory bandwidth is relatively low compared to on-chip bandwidths, and non-optimal memory accesses can significantly impact performance. If all the threads in a warp can access consecutive memory locations, memory access is fully coalesced.
To achieve coalesced memory access in HIP, you should:
1. *Align Data*: Use data types that are naturally aligned and ensure that structures and arrays are aligned properly.
2. *Optimize Access Patterns*: Arrange memory accesses so that consecutive threads in a warp access consecutive memory locations. For example, if threads access a 2D array, the array and thread block widths should be multiples of the warp size.
3. *Avoid strided access*: For example array[i * stride] can lead to memory bank conflicts and inefficient access.
4. *Pad Data*: If necessary, pad data structures to ensure alignment and coalescing.
* **Shared memory**: Avoiding bank conflicts reduces the serialization of memory transactions.
Shared memory is a small, fast memory region inside the CU. Unlike global memory, shared memory accesses do not require coalescing, but they can suffer from bank conflicts, which are another form of inefficient memory access. Shared memory is divided into multiple memory banks (usually 32 banks on modern GPUs). If multiple threads within a warp try to access different addresses that map to the same memory bank, accesses get serialized, leading to poor performance. To optimize shared memory usage, ensure that consecutive threads access different memory banks. Use padding if necessary to avoid conflicts.
* **Texture memory**: Spatial locality improves caching performance.
Texture memory is read-only memory optimized for spatial locality and caching rather than coalescing. Texture memory is cached, unlike standard global memory, and it provides optimized access patterns for 2D and spatially local data. Accessing neighboring values results in cache hits, improving performance. Therefore, instead of worrying about coalescing, optimal memory access patterns involve ensuring that threads access spatially adjacent texture elements, and the memory layout aligns well with the 2D caching mechanism.
* **Unified memory**: Structured access reduces the overhead of page migrations.
Unified memory allows the CPU and GPU to share memory seamlessly, but performance depends on access patterns. Unified memory enables automatic page migration between CPU and GPU memory. However, if different threads access different pages, it can lead to expensive page migrations and slow throughput performance. Accessing unified memory in a structured, warp-friendly manner reduces unnecessary page transfers. Ensure threads access memory in a structured, consecutive manner, minimizing page faults. Prefetch data to the GPU before computation by using ``hipMemPrefetchAsync()``. In addition, using small batch transfers as described below, can reduce unexpected page migrations when using unified memory.
* **Small batch transfers**: Enable pipelining and improve PCIe bandwidth use.
Memory transfers between the host and the device can become a major bottleneck if not optimized. One method is to use small batch memory transfers where data is transferred in smaller chunks instead of dealing with large datasets to avoid long blocking operations. Small batch transfers offer better PCIe bandwidth utilization over large data transfers. Small batch transfers offer performance improvement by offering reduced latency with small batches that run asynchronously using ``hipMemcpyAsync()`` as described in :ref:`asynchronous_how-to`, pipelining data transfers and kernel execution using separate streams. Finally, using pinned memory with small batch transfers enables faster DMA transfers without CPU involvement, greatly improving memory transfer performance.
Execution model
===============
HIP programs consist of two distinct scopes:
As previously discussed in :ref:`heterogeneous_programming`, HIP programs consist of two distinct scopes:
* The host-side API running on the host processor. There are two APIs available:
* The host-side API running on the host processor.
* The device-side kernels running on GPUs.
* The HIP runtime API which enables use of the single-source programming
model.
* The HIP driver API which sits at a lower level and most importantly differs
by removing some facilities provided by the runtime API, most
importantly around kernel launching and argument setting. It is geared
towards implementing abstractions atop, such as the runtime API itself.
Offers two additional pieces of functionality not provided by the Runtime
API: ``hipModule`` and ``hipCtx`` APIs. For further details, check
:doc:`HIP driver API </how-to/hip_porting_driver_api>`.
* The device-side kernels running on GPUs. Both the host and the device-side
APIs have synchronous and asynchronous functions in them.
.. note::
The HIP does not present two *separate* APIs link NVIDIA CUDA. HIP only extends
the HIP runtime API with new APIs for ``hipModule`` and ``hipCtx``.
Both the host and the device-side APIs have synchronous and asynchronous functions.
Host-side execution
-------------------
The part of the host-side API which deals with device management and their
queries are synchronous. All asynchronous APIs, such as kernel execution, data
movement and potentially data allocation/freeing all happen in the context of
device streams.
The host-side API dealing with device management and their queries are synchronous.
All asynchronous APIs, such as kernel execution, data movement and potentially data
allocation/freeing all happen in the context of device streams, as described in `Managing streams <../how-to/hip_runtime_api/asynchronous.html#managing-streams>`_.
Streams are FIFO buffers of commands to execute relating to a given device.
Commands which enqueue tasks on a stream all return promptly and the command is
Operations that enqueue tasks on a stream all return promptly, and the command is
executed asynchronously. All side effects of a command on a stream are visible
to all subsequent commands on the same stream. Multiple streams may point to
the same device and those streams may be fed from multiple concurrent host-side
threads. Execution on multiple streams may be concurrent but isn't required to
be.
Asynchronous APIs involving a stream all return a stream event which may be
Asynchronous APIs involving a stream all return a stream event, which can be
used to synchronize the execution of multiple streams. A user may enqueue a
barrier onto a stream referencing an event. The barrier will block until
the command related to the event does not complete, at which point all
side effects of the command shall be visible to commands following the barrier,
even if those side effects manifest on different devices.
barrier onto a stream referencing an event. The barrier will block activity on the
stream until the operation related to the event completes. After the event completes, all
side effects of the operation will be visible to subsequent commands even if those
side effects manifest on different devices.
.. figure:: ../data/understand/programming_model/stream-workflow.svg
:alt: Diagram depicting the stream and event workflow, with an example of
multiple streams working together. The diagram shows operations as red
rectangles, and events as white dots. There are three streams labelled
Stream 1, 2, and 3. The streams each have multiple operations and events
that require synchronization between the streams.
Multiple stream workflow
Streams also support executing user-defined functions as callbacks on the host.
The stream will not launch subsequent commands until the callback completes.
@@ -329,17 +482,8 @@ The stream will not launch subsequent commands until the callback completes.
Device-side execution
---------------------
The SIMT programming model behind the HIP device-side execution is a
middle-ground between SMT (Simultaneous Multi-Threading) programming known from
multicore CPUs, and SIMD (Single Instruction, Multiple Data) programming
mostly known from exploiting relevant instruction sets on CPUs (for example
SSE/AVX/Neon).
Kernel launch
-------------
Kernels may be launched in multiple ways all with different syntaxes and
intended use-cases.
Kernels may be launched in multiple ways, all with different syntaxes and
intended use cases.
* Using the triple-chevron ``<<<...>>>`` operator on a ``__global__`` annotated
function.
@@ -348,18 +492,44 @@ intended use-cases.
.. tip::
This name by default is a macro expanding to triple-chevron. In cases where
This name, by default, is a macro expanding to the triple-chevron syntax. In cases where
language syntax extensions are undesirable, or where launching templated
and/or overloaded kernel functions define the
``HIP_TEMPLATE_KERNEL_LAUNCH`` preprocessor macro before including the HIP
headers to turn it into a templated function.
* Using the launch APIs supporting the triple-chevron syntax directly.
Asynchronous execution
----------------------
.. caution::
Asynchronous operations between the host and the kernel provide a variety of opportunities,
or challenges, for managing synchronization, as described in :ref:`asynchronous_how-to`.
For instance, a basic model would be to launch an asynchronous operation on a kernel
in a stream, create an event to track the operation, continue operations in the host
program, and when the event shows that the asynchronous operation is complete, synchronize the kernel to return the results.
These APIs are intended to be used/generated by tools such as the HIP
compiler itself and not intended towards end-user code. Should you be
writing a tool having to launch device code using HIP, consider using these
over the alternatives.
However, one of the opportunities of asynchronous operation is the pipelining of operations
between launching kernels and transferring memory. In this case, you would be working
with multiple streams running concurrently, or at least overlapping in some regard,
and managing any dependencies between the streams in the host application.
The producer-consumer paradigm can be used to convert a sequential program
into parallel operations to improve performance. This process can employ multiple
streams to kick off asynchronous kernels, provide data to the kernels, perform operations,
and return the results for further processing in the host application.
These asynchronous activities call for stream management strategies. In the case
of the single stream, the only management would be the stream synchronization
when the work was complete. However, with multiple streams you have
overlapping execution of operations and synchronization becomes more complex, as shown
in the variations of the example in `Programmatic dependent launch and synchronization <../how-to/hip_runtime_api/asynchronous.html#programmatic-dependent-launch-and-synchronization>`_.
You need to manage each stream's activities, evaluate the availability of results, evaluate the critical path of the tasks, allocate resources on the hardware, and manage the execution order.
Multi-GPU and load balancing
----------------------------
For applications requiring additional computational power beyond a single device,
HIP supports utilizing multiple GPUs within a system. Large-scale applications
that need more compute power can use multiple GPUs in the system. This enables
the runtime to distribute workloads across multiple GPUs to balance the load and prevent some GPUs
from being over-utilized while others are idle.
For more information, see :ref:`multi-device`.