AI/rocm-systems
2
0
Fork 0
Code Issues Pull-Requests Actions Pakete Projekte Releases Wiki Aktivität

Add 'projects/rocshmem/' from commit '0496586829058af5cfd7f23acda2a6d0040da584'

Quellcode durchsuchen 
git-subtree-dir: projects/rocshmem
git-subtree-mainline: 5fd976da70
git-subtree-split: 0496586829
Dieser Commit ist enthalten in:
Ameya Keshava Mallya
2026-01-21 20:25:37 +00:00
Ursprung 5fd976da70 0496586829
Commit 12ab8df3bc
367 geänderte Dateien mit 81890 neuen und 0 gelöschten Zeilen
Patch-Datei herunterladen Vergleichs-Datei herunterladen Alle Dateien ausklappen Alle Dateien einklappen
projects/rocshmem/docs/.gitignore
+5
Datei anzeigen
@@ -0,0 +1,5 @@
_build/
_doxygen/
doxygen/html/
doxygen/xml/
sphinx/_toc.yml
projects/rocshmem/docs/README.md
+21
Datei anzeigen
@@ -0,0 +1,21 @@
# Building the rocSHMEM documentation
## macOS
To build html documentation locally:
```
brew install doxygen sphinx-doc
pip3.10 install -r ./sphinx/requirements.txt
python3.10 -m sphinx -T -E -b html -d _build/doctrees -D language=en . _build/html
open _build/html/index.html
```
To build pdf documentation we require a LaTeX installation on your machine.
Once LaTeX is installed, you may run the following:
```
pip3.10 install -r ./sphinx/requirements.txt
sphinx-build -M latexpdf . _build
open _build/latex/rocshmem.pdf
```
projects/rocshmem/docs/api/amo.rst
+418
Datei anzeigen
@@ -0,0 +1,418 @@
.. meta::
:description: rocSHMEM intra-kernel networking runtime for AMD dGPUs on the ROCm platform.
:keywords: rocSHMEM, API, ROCm, documentation, HIP, Networking, Communication
.. _rocshmem-api-amo:
---------------------------
Atomic memory operations
---------------------------
You can call these functions from divergent control paths at the per-thread level.
ROSHMEM_ATOMIC_FETCH
--------------------
.. cpp:function:: __device__ TYPE rocshmem_TYPENAME_atomic_fetch(TYPE *source, int pe)
.. cpp:function:: __device__ TYPE rocshmem_ctx_TYPENAME_atomic_fetch(rocshmem_ctx_t ctx, TYPE *source, int pe)
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param pe: PE of the remote process.
:returns: The value of ``dest``.
**Description:**
This function atomically returns the value of ``dest`` to the calling PE.
Valid ``TYPENAME`` and ``TYPE`` values are listed in EXTENDED_AMO_TYPES_.
SHMEM_ATOMIC_SET
----------------
.. cpp:function:: __device__ void rocshmem_TYPENAME_atomic_set(TYPE *dest, TYPE value, int pe);
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_atomic_set(rocshmem_ctx_t ctx, TYPE *dest, TYPE value, int pe);
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param value: The value to be atomically set.
:param pe: PE of the remote process.
:returns: None.
**Description:**
This function atomically sets the value ``value`` to ``dest`` on ``pe``.
Valid ``TYPENAME`` and ``TYPE`` values are listed in EXTENDED_AMO_TYPES_.
SHMEM_ATOMIC_COMPARE_SWAP
-------------------------
.. cpp:function:: __device__ TYPE rocshmem_TYPENAME_atomic_compare_swap(TYPE *dest, TYPE cond, TYPE value, TYPE pe);
.. cpp:function:: __device__ TYPE rocshmem_ctx_TYPENAME_atomic_compare_swap(rocshmem_ctx_t ctx, TYPE *dest, TYPE cond, TYPE value, TYPE pe);
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param cond: The value to be compare with.
:param value: The value to be atomically swapped.
:param pe: PE of the remote process.
:return: The old value of ``dest``.
**Description:**
This function atomically compares the value in ``dest`` with ``cond``. If they are equal, it stores ``value`` in ``dest``.
The operation returns the older value of ``dest`` to the calling PE.
The operation is blocking.
Valid ``TYPENAME`` and ``TYPE`` values are listed in STANDARD_AMO_TYPES_.
SHMEM_ATOMIC_SWAP
-----------------
.. cpp:function:: __device__ TYPE rocshmem_TYPENAME_atomic_swap(TYPE *dest, TYPE value, TYPE pe);
.. cpp:function:: __device__ TYPE rocshmem_ctx_TYPENAME_atomic_swap(rocshmem_ctx_t ctx, TYPE *dest, TYPE value, TYPE pe);
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param value: The value to be atomically swapped.
:param pe: PE of the remote process.
:return: The old value of ``dest``.
**Description:**
This function atomically swaps the value ``val`` with ``dest`` on ``pe``.
The operation is blocking.
Valid ``TYPENAME`` and ``TYPE`` values are listed in EXTENDED_AMO_TYPES_.
SHMEM_ATOMIC_FETCH_INC
----------------------
.. cpp:function:: __device__ TYPE rocshmem_TYPENAME_atomic_fetch_inc(TYPE *dest, TYPE pe);
.. cpp:function:: __device__ TYPE rocshmem_ctx_TYPENAME_atomic_fetch_inc(rocshmem_ctx_t ctx, TYPE *dest, TYPE pe);
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param pe: PE of the remote process.
:return: The old value of ``dest``.
**Description:**
This function atomically adds ``1`` to ``dest`` on ``pe``.
The operation is blocking.
Valid ``TYPENAME`` and ``TYPE`` values are listed in STANDARD_AMO_TYPES_.
SHMEM_ATOMIC_INC
----------------
.. cpp:function:: __device__ void rocshmem_TYPENAME_atomic_inc(TYPE *dest, TYPE pe);
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_atomic_inc(rocshmem_ctx_t ctx, TYPE *dest, TYPE pe);
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param pe: PE of the remote process.
:return: None.
**Description:**
This function atomically adds ``1`` to ``dest`` on ``pe``.
The operation is blocking.
Valid ``TYPENAME`` and ``TYPE`` values are listed in STANDARD_AMO_TYPES_.
SHMEM_ATOMIC_FETCH_ADD
----------------------
.. cpp:function:: __device__ TYPE rocshmem_TYPENAME_atomic_fetch_add(TYPE *dest, TYPE value, TYPE pe);
.. cpp:function:: __device__ TYPE rocshmem_ctx_TYPENAME_atomic_fetch_add(rocshmem_ctx_t ctx, TYPE *dest, TYPE value, TYPE pe);
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param value: The value to be atomically added.
:param pe: PE of the remote process.
:return: The old value of ``dest``.
**Description:**
This function atomically adds ``value`` to ``dest`` on ``pe``.
The operation is blocking.
Valid ``TYPENAME`` and ``TYPE`` values are listed in STANDARD_AMO_TYPES_.
SHMEM_ATOMIC_ADD
----------------
.. cpp:function:: __device__ void rocshmem_TYPENAME_atomic_add(TYPE *dest, TYPE value, TYPE pe);
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_atomic_add(rocshmem_ctx_t ctx, TYPE *dest, TYPE value, TYPE pe);
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param value: The value to be atomically added.
:param pe: PE of the remote process.
:return: None.
**Description:**
This function atomically adds ``value`` to ``dest`` on ``pe``.
The operation is blocking.
Valid ``TYPENAME`` and ``TYPE`` values can be seen in STANDARD_AMO_TYPES_.
SHMEM_ATOMIC_FETCH_AND
----------------------
.. cpp:function:: __device__ TYPE rocshmem_TYPENAME_atomic_fetch_and(TYPE *dest, TYPE value, TYPE pe);
.. cpp:function:: __device__ TYPE rocshmem_ctx_TYPENAME_atomic_fetch_and(rocshmem_ctx_t ctx, TYPE *dest, TYPE value, TYPE pe);
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param value: The value to be atomically ``AND``.
:param pe: PE of the remote process.
:return: The old value of ``dest``.
**Description:**
This function atomically bitwise-and ``value`` to the value at ``dest`` on ``pe``.
The operation is blocking.
Valid ``TYPENAME`` and ``TYPE`` values are listed in BITWISE_AMO_TYPES_.
SHMEM_ATOMIC_AND
----------------
.. cpp:function:: __device__ TYPE rocshmem_TYPENAME_atomic_and(TYPE *dest, TYPE value, TYPE pe);
.. cpp:function:: __device__ TYPE rocshmem_ctx_TYPENAME_atomic_and(rocshmem_ctx_t ctx, TYPE *dest, TYPE value, TYPE pe);
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param value: The value to be atomically ``AND``.
:param pe: PE of the remote process.
:return: None
**Description:**
This function atomically bitwise-and ``value`` to the value at ``dest`` on ``pe``.
The operation is blocking.
Valid ``TYPENAME`` and ``TYPE`` values are listed in BITWISE_AMO_TYPES_.
SHMEM_ATOMIC_FETCH_OR
----------------------
.. cpp:function:: __device__ TYPE rocshmem_TYPENAME_atomic_fetch_or(TYPE *dest, TYPE value, TYPE pe)
.. cpp:function:: __device__ TYPE rocshmem_ctx_TYPENAME_atomic_fetch_or(rocshmem_ctx_t ctx, TYPE *dest, TYPE value, TYPE pe)
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param value: The value to be atomically ``OR``.
:param pe: PE of the remote process.
:return: The old value of ``dest``.
**Description:**
This function atomically bitwise-or ``value`` to the value at ``dest`` on ``pe``.
The operation is blocking.
Valid ``TYPENAME`` and ``TYPE`` values are listed in BITWISE_AMO_TYPES_.
SHMEM_ATOMIC_OR
---------------
.. cpp:function:: __device__ TYPE rocshmem_TYPENAME_atomic_or(TYPE *dest, TYPE value, TYPE pe)
.. cpp:function:: __device__ TYPE rocshmem_ctx_TYPENAME_atomic_or(rocshmem_ctx_t ctx, TYPE *dest, TYPE value, TYPE pe)
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param value: The value to be atomically ``OR``.
:param pe: PE of the remote process.
:return: None.
**Description:**
This function atomically bitwise-or ``value`` to the value at ``dest`` on ``pe``.
The operation is blocking.
Valid ``TYPENAME`` and ``TYPE`` values are listed in BITWISE_AMO_TYPES_.
SHMEM_ATOMIC_FETCH_XOR
----------------------
.. cpp:function:: __device__ TYPE rocshmem_TYPENAME_atomic_fetch_xor(TYPE *dest, TYPE value, TYPE pe);
.. cpp:function:: __device__ TYPE rocshmem_ctx_TYPENAME_atomic_fetch_xor(rocshmem_ctx_t ctx, TYPE *dest, TYPE value, TYPE pe);
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param value: The value to be atomically ``XOR``.
:param pe: PE of the remote process.
:return: The old value of ``dest``.
**Description:**
This function atomically bitwise-xor ``value`` to the value at ``dest`` on ``pe``.
The operation is blocking.
Valid ``TYPENAME`` and ``TYPE`` values are listed in BITWISE_AMO_TYPES_.
SHMEM_ATOMIC_XOR
----------------
.. cpp:function:: __device__ TYPE rocshmem_TYPENAME_atomic_xor(TYPE *dest, TYPE value, TYPE pe)
.. cpp:function:: __device__ TYPE rocshmem_ctx_TYPENAME_atomic_xor(rocshmem_ctx_t ctx, TYPE *dest, TYPE value, TYPE pe)
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param value: The value to be atomically ``XOR``.
:param pe: PE of the remote process.
:return: None.
**Description:**
This function atomically bitwise-xor ``value`` to the value at ``dest`` on ``pe``.
The operation is blocking.
Valid ``TYPENAME`` and ``TYPE`` values are listed in BITWISE_AMO_TYPES_.
Supported AMO data types
------------------------
.. _STANDARD_AMO_TYPES:
.. list-table:: Standard AMO Data Types
:widths: 10 20 20
:header-rows: 1
* - TYPE
- TYPENAME
- Supported
* - int
- int
- Yes
* - long
- long
- Yes
* - long long
- longlong
- Yes
* - unsigned int
- uint
- Yes
* - unsigned long
- ulong
- Yes
* - unsigned long long
- ulonglong
- Yes
* - int32_t
- int32
- Yes
* - int64_t
- int64
- Yes
* - uint32_t
- uint32
- Yes
* - uint64_t
- uint64
- Yes
* - size_t
- size
- Yes
* - ptrdiff_t
- ptrdiff
- Yes
.. _EXTENDED_AMO_TYPES:
.. list-table:: Extended AMO Data Types
:widths: 10 20 20
:header-rows: 1
* - TYPE
- TYPENAME
- Supported
* - float
- float
- Yes
* - double
- double
- Yes
* - int
- int
- Yes
* - long
- long
- Yes
* - long long
- longlong
- Yes
* - unsigned int
- uint
- Yes
* - unsigned long
- ulong
- Yes
* - unsigned long long
- ulonglong
- Yes
* - int32_t
- int32
- Yes
* - int64_t
- int64
- Yes
* - uint32_t
- uint32
- Yes
* - uint64_t
- uint64
- Yes
* - size_t
- size
- Yes
* - ptrdiff_t
- ptrdiff
- Yes
.. _BITWISE_AMO_TYPES:
.. list-table:: Bitwise AMO Data Types
:widths: 10 20 20
:header-rows: 1
* - TYPE
- TYPENAME
- Supported
* - unsigned int
- uint
- Yes
* - unsigned long
- ulong
- Yes
* - unsigned long long
- ulonglong
- Yes
* - int32_t
- int32
- Yes
* - int64_t
- int64
- Yes
* - uint32_t
- uint32
- Yes
* - uint64_t
- uint64
- Yes
projects/rocshmem/docs/api/coll.rst
+323
Datei anzeigen
@@ -0,0 +1,323 @@
.. meta::
:description: rocSHMEM intra-kernel networking runtime for AMD dGPUs on the ROCm platform.
:keywords: rocSHMEM, API, ROCm, documentation, HIP, Networking, Communication
.. _rocshmem-api-coll:
---------------------------
Collective routines
---------------------------
ROCSHMEM_BARRIER_ALL
--------------------
.. cpp:function:: __device__ void rocshmem_barrier_all()
.. cpp:function:: __device__ void rocshmem_barrier_all_wave()
.. cpp:function:: __device__ void rocshmem_barrier_all_wg()
:returns: None.
**Description:**
This routine performs a collective barrier across all PEs in the system.
The caller is blocked until the barrier is resolved and all updates local and remote are completed.
These APIs should be called from only one thread/wavefront/workgroup within the grid to avoid undefined behavior.
ROCSHMEM_BARRIER_ALL_ON_STREAM
-------------------------------
.. cpp:function:: __host__ void rocshmem_barrier_all_on_stream(hipStream_t stream)
:param stream: HIP stream on which to enqueue the operation.
:returns: None.
**Description:**
This routine enqueues a collective barrier operation on a HIP stream. The barrier is performed
across all PEs in the system. The operation is enqueued on the specified stream and will execute
asynchronously. The caller must synchronize the stream (e.g., using ``hipStreamSynchronize``)
to ensure completion.
ROCSHMEM_BARRIER
----------------
.. cpp:function:: __device__ void rocshmem_ctx_barrier(rocshmem_ctx_t ctx, rocshmem_team_t team)
.. cpp:function:: __device__ void rocshmem_ctx_barrier_wave(rocshmem_ctx_t ctx, rocshmem_team_t team)
.. cpp:function:: __device__ void rocshmem_ctx_barrier_wg(rocshmem_ctx_t ctx, rocshmem_team_t team)
:param ctx: Context with which to perform this operation.
:returns: None.
**Description:**
This routine performs a collective barrier between all PEs in the system.
The caller is blocked until the barrier is resolved.
ROCSHMEM_TEAM_SYNC
------------------
.. cpp:function:: __device__ void rocshmem_ctx_sync(rocshmem_ctx_t ctx, rocshmem_team_t team)
.. cpp:function:: __device__ void rocshmem_ctx_sync_wave(rocshmem_ctx_t ctx, rocshmem_team_t team)
.. cpp:function:: __device__ void rocshmem_ctx_sync_wg(rocshmem_ctx_t ctx, rocshmem_team_t team)
:param ctx: Context with which to perform this operation.
:param team: Team with which to perform this operation.
:returns: None.
**Description:**
This routine registers the arrival of a PE at a barrier.
The caller is blocked until the synchronization is resolved.
Unlike the ``shmem_barrier_all`` routine, ``shmem_team_sync`` only ensures the
completion and visibility of previously issued memory stores, but does not
ensure the completion of remote memory updates issued via OpenSHMEM routines.
ROCSHMEM_SYNC_ALL
-----------------
.. cpp:function:: __device__ void rocshmem_sync_all()
.. cpp:function:: __device__ void rocshmem_sync_all_wave()
.. cpp:function:: __device__ void rocshmem_sync_all_wg()
:returns: None.
**Description:**
These routines behaves the same way as ``rocshmem_team_sync_*`` when called on the world team.
These APIs should be called from only one thread/wavefront/workgroup within the grid to avoid undefined behavior.
ROSHMEM_ALLTOALL
----------------
.. cpp:function:: __device__ void rocshmem_TYPENAME_alltoall_wg(rocshmem_team_t team, TYPE *dest, const TYPE *source, int nelems)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_alltoall_wg(rocshmem_ctx_t ctx, rocshmem_team_t team, TYPE *dest, const TYPE *source, int nelems)
:param team: The team participating in the collective.
:param dest: Destination address. Must be an address on the
symmetric heap.
:param source: Source address. Must be an address on the symmetric
heap.
:param nelems: Number of data blocks transferred per pair of PEs.
:returns: None.
**Description:**
This routine exchanges a fixed amount of contiguous data blocks between all pairs
of PEs participating in the collective routine.
This function must be called as a work-group collective.
Valid ``TYPENAME`` and ``TYPE`` values are listed in :ref:`RMA_TYPES`.
ROCSHMEM_ALLTOALLMEM_ON_STREAM
-------------------------------
.. cpp:function:: __host__ void rocshmem_alltoallmem_on_stream(rocshmem_team_t team, void *dest, const void *source, size_t size, hipStream_t stream)
:param team: The team participating in the collective.
:param dest: Destination address. Must be an address on the symmetric heap.
:param source: Source address. Must be an address on the symmetric heap.
:param size: Number of bytes to transfer per pair of PEs.
:param stream: HIP stream on which to enqueue the operation.
:returns: None.
**Description:**
This routine enqueues an alltoall collective operation on a HIP stream. The function
exchanges a fixed amount of contiguous data blocks between all pairs of PEs participating
in the collective routine. The operation is enqueued on the specified stream and will
execute asynchronously. The caller must synchronize the stream (e.g., using
``hipStreamSynchronize``) to ensure completion.
This function creates a separate context for each workgroup to avoid contention on the
default context, allowing parallel execution across multiple streams.
ROCSHMEM_BROADCAST
------------------
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_broadcast_wg(rocshmem_ctx_t ctx, rocshmem_team_t team, TYPE *dest, const TYPE *source, int nelems, int pe_root)
:param ctx: Context with which to perform this collective.
:param team: The team participating in the collective.
:param dest: Destination address. Must be an address on the
symmetric heap.
:param source: Source address. Must be an address on the symmetric
heap.
:param nelems: Number of data blocks transferred per pair of PEs.
:returns: None.
**Description:**
This routine performs a broadcast across PEs in the team.
The caller is blocked until the broadcast completes.
Valid ``TYPENAME`` and ``TYPE`` values are listed in :ref:`RMA_TYPES`.
ROCSHMEM_BROADCASTMEM_ON_STREAM
--------------------------------
.. cpp:function:: __host__ void rocshmem_broadcastmem_on_stream(rocshmem_team_t team, void *dest, const void *source, size_t nelems, int pe_root, hipStream_t stream)
:param team: The team participating in the collective.
:param dest: Destination address. Must be an address on the symmetric heap.
:param source: Source address. Must be an address on the symmetric heap.
:param nelems: Number of bytes to broadcast.
:param pe_root: Root PE (relative to team) from which to broadcast.
:param stream: HIP stream on which to enqueue the operation.
:returns: None.
**Description:**
This routine enqueues a broadcast collective operation on a HIP stream. The function broadcasts
data from the root PE to all other PEs participating in the collective routine. The operation
is enqueued on the specified stream and will execute asynchronously. The caller must synchronize
the stream (e.g., using ``hipStreamSynchronize``) to ensure completion.
This function creates a separate context for each workgroup to avoid contention on the
default context, allowing parallel execution across multiple streams.
ROCSHMEM_FCOLLECT
-----------------
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_fcollect_wg(rocshmem_ctx_t ctx, rocshmem_team_t team, TYPE *dest, const TYPE *source, int nelems)
:param ctx: Context with which to perform this collective.
:param team: The team participating in the collective.
:param dest: Destination address. Must be an address on the
symmetric heap.
:param source: Source address. Must be an address on the symmetric
heap.
:param nelems: Number of data blocks transferred per pair of PEs.
:returns: None.
**Description:**
This routine concatenates blocks of data from multiple PEs to an array in every
PE participating in the collective routine.
ROCSHMEM_REDUCTION
------------------
.. cpp:function:: __device__ int rocshmem_ctx_TYPENAME_OPNAME_reduce_wg(rocshmem_ctx_t ctx, rocshmem_team_t team, TYPE *dest, const TYPE *source, int nreduce)
:param ctx: Context with which to perform this collective.
:param team: The team participating in the collective.
:param dest: Destination address. Must be an address on the
symmetric heap.
:param source: Source address. Must be an address on the symmetric
heap.
:param nreduce: Number of data blocks transferred per pair of PEs.
:returns: Zero on successful local completion. Nonzero otherwise.
**Description:**
This routine performs an allreduce operation across PEs in the team.
Valid ``TYPENAME``, ``TYPE``, and ``OPNAME`` values are listed in :ref:`REDUCE_TYPES`.
Supported reduction types and operations
----------------------------------------
.. _REDUCE_TYPES:
.. list-table:: Reduction Types, Names and Operations
:widths: 20 20 20 20
:header-rows: 1
* - TYPE
- TYPENAME
- OPNAME
- Supported
* - char
- char
- max, min, sum, prod
- No
* - signed char
- schar
- max, min, sum, prod
- No
* - short
- short
- max, min, sum, prod
- Yes
* - int
- int
- max, min, sum, prod
- Yes
* - long
- long
- max, min, sum, prod
- Yes
* - long long
- longlong
- max, min, sum, prod
- Yes
* - ptrdiff_t
- ptrdiff
- max, min, sum, prod
- No
* - unsigned char
- uchar
- and, or, xor, max, min, sum, prod
- No
* - unsigned short
- ushort
- and, or, xor, max, min, sum, prod
- No
* - unsigned int
- uint
- and, or, xor, max, min, sum, prod
- No
* - unsigned long
- ulong
- and, or, xor, max, min, sum, prod
- No
* - unsigned long long
- ulonglong
- and, or, xor, max, min, sum, prod
- No
* - int8_t
- int8
- and, or, xor, max, min, sum, prod
- No
* - int16_t
- int16
- and, or, xor, max, min, sum, prod
- No
* - int32_t
- int32
- and, or, xor, max, min, sum, prod
- No
* - int64_t
- int64
- and, or, xor, max, min, sum, prod
- No
* - uint8_t
- uint8
- and, or, xor, max, min, sum, prod
- No
* - uint16_t
- uint16
- and, or, xor, max, min, sum, prod
- No
* - uint32_t
- uint32
- and, or, xor, max, min, sum, prod
- No
* - uint64_t
- uint64
- and, or, xor, max, min, sum, prod
- No
* - size_t
- size
- and, or, xor, max, min, sum, prod
- No
* - float
- float
- max, min, sum, prod
- Yes
* - double
- double
- max, min, sum, prod
- Yes
* - long double
- longdouble
- max, min, sum, prod
- No
* - double _Complex
- complexd
- sum, prod
- No
* - float _Complex
- complexf
- sum, prod
- No
projects/rocshmem/docs/api/ctx.rst
+59
Datei anzeigen
@@ -0,0 +1,59 @@
.. meta::
:description: rocSHMEM intra-kernel networking runtime for AMD dGPUs on the ROCm platform.
:keywords: rocSHMEM, API, ROCm, documentation, HIP, Networking, Communication
.. _rocshmem-api-ctx:
-----------------------------------
Context management routines
-----------------------------------
ROCSHMEM_CTX_CREATE
-------------------
.. cpp:function:: __device__ int rocshmem_wg_ctx_create(int64_t options, rocshmem_ctx_t *ctx)
.. cpp:function:: __device__ int rocshmem_wg_team_create_ctx(rocshmem_team_t team, long options, rocshmem_ctx_t *ctx)
:param team: Team handle to derive the context from.
:param options: Options for context creation. Ignored in current design; use the value ``0``.
:param ctx: A handle to the newly created context.
:returns: All threads returns ``0`` if the context was created successfully.
If any thread returns non-zero value, the operation fails, ctx is set to ``ROCSHMEM_CTX_INVALID`` and a
higher number of ``ROCSHMEM_MAX_NUM_CONTEXTS`` is required.
**Description:**
This routine creates an rocSHMEM context. By design, the context is private to the calling work-group.
It must be called collectively by all threads in the work-group. If the context was created successfully, a value
of zero is returned and the context handle pointed to by ctx specifies a valid context; otherwise, a nonzero value
is returned and ctx is set to ``ROCSHMEM_CTX_INVALID``. An unsuccessful context creation call is not treated as an
error and the rocSHMEM library remains in a correct state. The creation call can be reattempted after additional
resources become available.
ROCSHMEM_CTX_DESTROY
--------------------
.. cpp:function:: __device__ void rocshmem_wg_ctx_destroy(rocshmem_ctx_t *ctx)
:param ctx: Context handle.
:returns: None.
**Description:**
This routine destroys an rocSHMEM context. It must be called collectively by all threads in the work-group.
If ctx has the value ``ROCSHMEM_CTX_INVALID``, no operation is performed.
ROCSHMEM_GET_DEVICE_CTX
-----------------------
.. cpp:function:: __host__ void * rocshmem_get_device_ctx()
:param: None.
:returns: Returns ``ROCSHMEM_CTX_DEFAULT`` device pointer that users.
can query from one instance of rocSHMEM host library and
use later for dynamic module initialization in
kernel bitcode device library in the same application.
**Description:**
This routine queries rocSHMEM default device context from host API.
projects/rocshmem/docs/api/env_variables.rst
+96
Datei anzeigen
@@ -0,0 +1,96 @@
.. meta::
:description: rocSHMEM environment variables reference
:keywords: rocSHMEM, ROCm, API, environment variables, environment, reference
.. _rocshmem-api-env-variables:
********************************************************************
rocSHMEM environment variables
********************************************************************
This section describes the important environment variables used to
control the behavior of rocSHMEM.
.. list-table::
:header-rows: 1
:widths: 35,14,51
* - **Environment variable**
- **Default value**
- **Value**
* - | ``ROCSHMEM_HEAP_SIZE``
| Defines the size of the rocSHMEM symmetric heap in bytes (per PE).
- ``1073741824`` (1 GB)
- | Size in bytes (per PE).
| Note: the heap is on GPU memory.
* - | ``ROCSHMEM_MAX_NUM_CONTEXTS``
| Defines the number of contexts an application can use.
- ``32``
- Maximum number of contexts.
* - | ``ROCSHMEM_MAX_NUM_TEAMS``
| Defines the number of teams an application can use.
- ``40``
- Maximum number of teams.
* - | ``ROCSHMEM_BACKEND``
| When rocSHMEM is compiled for all backends, this enviroment variable
| selects which backend to execute. The default value is an empty string and rocSHMEM auto-selects the most appropriate backend.
- `` ``
- | ``ipc``: IPC Backend
| ``ro``: Reverse Offload Backend
| ``gda``: GPU Direct Async Backend
* - | ``ROCSHMEM_UNIQUEID_WITH_MPI``
| Defines whether rocSHMEM is expected to use MPI when using the uniqueId based initialization.
- ``0``
- | ``0``: Do not use MPI.
| ``1``: Use MPI.
* - | ``ROCSHMEM_DISABLE_MIXED_IPC``
| Defines whether to force using the network conduit even when IPC is available.
- ``0``
- | ``0``: Use IPC when available.
| ``1``: Force network conduit.
* - | ``ROCSHMEM_USE_IB_HCA``
| Defines which NIC that this PE should be bound to. The default value is an empty string and rocSHMEM auto-detects the most appropriate NIC.
- `` ``
- | Example value: ``bnxt_re0``
* - | ``ROCSHMEM_BOOTSTRAP_SOCKET_IFNAME``
| Chooses the interface to bootstrap rocSHMEM with.
| Only valid when not using MPI.
| The default value is an empty string and rocSHMEM auto-detects the most appropriate interface.
- `` ``
- | Example value: ``eno8303``
* - | ``ROCSHMEM_GDA_PROVIDER``
| When rocSHMEM is compiled with support for multiple NIC vendors,
| the enviroment variable selects the desired provider.
| The default value is an empty string and rocSHMEM auto-detects the most appropriate NIC.
- `` ``
- | ``bnxt``: Broadcom Thor 2
| ``pensando``: AMD Pensando Pollara
| ``ionic``: AMD Pensando Pollara (alias)
| ``mlx5``: Mellanox ConnectX-7
* - | ``ROCSHMEM_GDA_ALTERNATE_QP_PORTS``
| Enables or disables alternating QP mappings across rocSHMEM contexts.
- ``1``
- | ``0``: Disabled.
| ``1``: Enabled. This helps saturate bandwidth on multiport bonded interfaces.
* - | ``ROCSHMEM_GDA_TRAFFIC_CLASS``
| When using an NIC with an Ethernet link layer, this sets the traffic class for the QPs.
- ``0``
- The traffic class number.
* - | ``ROCSHMEM_GDA_PCIE_RELAXED_ORDERING``
| Enables PCIe Relaxed Ordering when registering the symmetric heap with the RDMA NICs.
- ``0``
- | ``0``: Disabled.
| ``1``: Enabled.
projects/rocshmem/docs/api/init.rst
+161
Datei anzeigen
@@ -0,0 +1,161 @@
.. meta::
:description: rocSHMEM intra-kernel networking runtime for AMD dGPUs on the ROCm platform.
:keywords: rocSHMEM, API, ROCm, documentation, HIP, Networking, Communication
.. _rocshmem-api-init:
---------------------------------------
Library setup, exit, and query routines
---------------------------------------
ROCSHMEM_INIT
-------------
.. cpp:function:: __host__ void rocshmem_init(void)
:Parameters: None.
:returns: None.
**Description:**
This routine initializes the rocSHMEM library and underlying transport layer.
Before ``rocshmem_init`` is called,
you must select the device that this PE is associated to by calling
`hipSetDevice
<https://rocm.docs.amd.com/projects/HIP/en/docs-6.0.0/doxygen/html/group___device.html#ga43c1e7f15925eeb762195ccb5e063eae>`_.
.. WARNING::
Routine `rocshmem_wg_init` has been deprecated.
.. cpp:function:: [[deprecated]] __device__ void rocshmem_wg_init(void)
:Parameters: None.
:returns: None.
**Description:**
This routine has been deprecated, please do not use.
This routine initializes device-side rocSHMEM resources.
It must be called before any threads in this work-group invoke other rocSHMEM functions.
It must be called collectively by all threads in the work-group.
ROCSHMEM_FINALIZE
-----------------
.. cpp:function:: __host__ void rocshmem_finalize(void)
:Parameters: None.
:returns: None.
**Description:**
This routine finalizes the rocSHMEM library.
.. WARNING::
Routine `rocshmem_wg_finalize` has been deprecated.
.. cpp:function:: [[deprecated]] __device__ void rocshmem_wg_finalize(void)
:Parameters: None.
:returns: None.
**Description:**
This routine has been deprecated, please do not use.
This routine finalizes device-side rocSHMEM resources.
It must be called before work-group completion if the work-group also called ``rocshmem_wg_init``.
It must be called collectively by all threads in the work-group.
ROCSHMEM_INIT_ATTR
------------------
.. cpp:function:: __host__ int rocshmem_init_attr(unsigned int flags, rocshmem_init_attr_t *attr)
:param flags: The initialization method to be used.
:param attr: Attribute structure specifying input characteristics.
:returns int: Returns ``0`` on success; otherwise, returns a nonzero value.
**Description:**
This routine initializes the rocSHMEM runtime and underlying transport layer using
the provided mode and attributes.
The parameter ``flags`` can be either
``ROCSHMEM_INIT_WITH_UNIQUEID`` or ``ROCSHMEM_INIT_WITH_MPI_COMM``.
ROCSHMEM_GET_UNIQUEID
---------------------
.. cpp:function:: __host__ int rocshmem_get_uniqueid(rocshmem_uniqueid_t *uid)
:param uid: Pointer to a unique ID handle.
:returns: Returns ``0`` on success; otherwise, returns a nonzero value.
**Description:**
This routine returns a unique ID.
ROCSHMEM_SET_ATTR_UNIQUEID_ARGS
-------------------------------
.. cpp:function:: __host__ int rocshmem_set_attr_uniqueid_args(int rank, int nranks, rocshmem_uniqueid_t *uid, rocshmem_init_attr_t *attr)
:param rank: Rank of the calling process.
:param nranks: Number of PEs.
:param uid: Unique ID used to identify the group processes.
:param attr: Attribute structure to be passed to ``rocshmem_init_attr_t``.
:returns: Returns ``0`` on success; otherwise, returns a nonzero value.
**Description:**
This routine initializes the ``rocshmem_init_attr_t`` struct.
ROCSHMEM_N_PES
--------------
.. cpp:function:: __host__ int rocshmem_n_pes(void)
:Parameters: None.
:returns: Total number of PEs.
**Description:**
This routine queries the total number of PEs.
It can be called before ``rocshmem_init``.
.. cpp:function:: __device__ int rocshmem_n_pes(void)
.. cpp:function:: __device__ int rocshmem_ctx_n_pes(rocshmem_ctx_t ctx)
:param ctx: GPU side context handle.
:returns: Total number of PEs.
**Description:**
This routine queries the total number of PEs for a given context.
It can be called per thread with no performance penalty.
ROCSHMEM_MY_PE
--------------
.. cpp:function:: __host__ int rocshmem_my_pe(void)
:Parameters: None.
:returns: PE ID of the caller.
**Description:**
This routine queries the PE ID of the caller.
It can be called before ``rocshmem_init``.
.. cpp:function:: __device__ int rocshmem_my_pe(void)
.. cpp:function:: __device__ int rocshmem_ctx_my_pe(rocshmem_ctx_t ctx)
:param ctx: GPU side context handle.
:returns: PE ID of the caller.
**Description:**
This routine queries the PE ID of the caller.
It can be called per thread with no performance penalty.
ROCSHMEM_PTR
--------------
.. cpp:function:: __host__ void* rocshmem_ptr(const void *dest, int pe);
.. cpp:function:: __device__ void* rocshmem_ptr(const void *dest, int pe);
:param dest: Local symmetric heap allocation pointer for current PE.
:param pe: Remote PE.
:returns: Returns remote symmetric heap device pointer from host-side API.
``NULL`` is returned if a valid device pointer cannot be provided.
This pointer can be used to issue load/store from custom kernels
instead of using rocshmem device side get/put APIs for RMA operations.
**Description:**
This routine queries rocSHMEM remote symmetric heap pointer.
projects/rocshmem/docs/api/memory_management.rst
+35
Datei anzeigen
@@ -0,0 +1,35 @@
.. meta::
:description: rocSHMEM intra-kernel networking runtime for AMD dGPUs on the ROCm platform.
:keywords: rocSHMEM, API, ROCm, documentation, HIP, Networking, Communication
.. _rocshmem-api-memory-management:
---------------------------
Memory management routines
---------------------------
ROCSHMEM_MALLOC
---------------
.. cpp:function:: __host__ void *rocshmem_malloc(size_t size)
:param size: Memory allocation size in bytes.
:returns: A pointer to the allocated memory on the symmetric heap.
If a valid allocation cannot be made, it returns ``NULL``.
**Description:**
This routine allocates memory of ``size`` bytes from the symmetric heap.
This is a collective operation and must be called by all PEs.
ROCSHMEM_FREE
-------------
.. cpp:function:: __host__ void rocshmem_free(void *ptr)
:param ptr: A pointer to previously allocated memory on the symmetric heap.
:returns: None.
**Description:**
This routine frees a memory allocation from the symmetric heap.
It is a collective operation and must be called by all PEs.
projects/rocshmem/docs/api/memory_ordering.rst
+51
Datei anzeigen
@@ -0,0 +1,51 @@
.. meta::
:description: rocSHMEM intra-kernel networking runtime for AMD dGPUs on the ROCm platform.
:keywords: rocSHMEM, API, ROCm, documentation, HIP, Networking, Communication
.. _rocshmem-api-memory-ordering:
---------------------------
Memory ordering routines
---------------------------
ROCSHMEM_FENCE
--------------
.. cpp:function:: __device__ void rocshmem_fence()
.. cpp:function:: __device__ void rocshmem_fence(int pe)
.. cpp:function:: __device__ void rocshmem_ctx_fence(rocshmem_ctx_t ctx)
.. cpp:function:: __device__ void rocshmem_ctx_fence(rocshmem_ctx_t ctx, int pe)
:param ctx: Context with which to perform this operation.
:param pe: Destination ``pe``.
:returns: None.
**Description:**
This routine ensures order between messages in this context to follow OpenSHMEM semantics.
ROCSHMEM_QUIET
--------------
.. cpp:function:: __device__ void rocshmem_ctx_quiet(rocshmem_ctx_t ctx)
.. cpp:function:: __device__ void rocshmem_quiet()
:param ctx: Context with which to perform this operation.
:returns: None.
**Description:**
This routine completes all previous operations posted to this context.
ROCSHMEM_PE_QUIET
-----------------
.. cpp:function:: __device__ void rocshmem_ctx_pe_quiet(shmem_ctx_t ctx, const int *target_pes, size_t npes)
.. cpp:function:: __device__ void rocshmem_pe_quiet(const int *target_pes, size_t npes)
:param ctx: Context with which to perform this operation.
:param target_pes: Address of target PE array where the operations need to be completed
:param npes: The number of PEs in the target PE array
:returns: None.
**Description:**
This routine completes all previous operations posted to this context
for the PEs in the `target_pes` array.
projects/rocshmem/docs/api/pt2pt_sync.rst
+142
Datei anzeigen
@@ -0,0 +1,142 @@
.. meta::
:description: rocSHMEM intra-kernel networking runtime for AMD dGPUs on the ROCm platform.
:keywords: rocSHMEM, API, ROCm, documentation, HIP, Networking, Communication
.. _rocshmem-api-pt2pt-sync:
-----------------------------------------
Point-to-point synchronization routines
-----------------------------------------
ROCSHMEM_WAIT_UNTIL
-------------------
.. cpp:function:: __device__ void rocshmem_TYPENAME_wait_until(TYPE *ivars, int cmp, TYPE val)
:param ivars: Pointer to memory on the symmetric heap to wait for.
:param cmp: Operation for the comparison.
:param val: Value to compare the memory at ``ivars`` to.
:returns: None.
**Description:**
This routine blocks the caller until the condition ``(*ivars cmp val)`` is true.
Valid ``cmp`` values are listed in :ref:`CMP_VALUES`.
Valid ``TYPENAME`` and ``TYPE`` values are listed in :ref:`STANDARD_AMO_TYPES`.
ROCSHMEM_WAIT_UNTIL_ALL
-----------------------
.. cpp:function:: __device__ void rocshmem_TYPENAME_wait_until_all(TYPE *ivars, size_t nelems, const int* status, int cmp, TYPE val)
:param ivars: Pointer to memory on the symmetric heap to wait for.
:param nelems: Number of elements in the ``ivars`` array.
:param status: Array of length ``nelems`` to exclude elements from the wait.
:param cmp: Operation for the comparison.
:param val: Value to compare.
:returns: None.
**Description:**
This routine blocks the caller until the condition ``(ivars[i] cmp val)`` is true for all ``ivars``.
Valid ``cmp`` values are listed in :ref:`CMP_VALUES`.
Valid ``TYPENAME`` and ``TYPE`` values are listed in :ref:`STANDARD_AMO_TYPES`.
ROCSHMEM_WAIT_UNTIL_ANY
-----------------------
.. cpp:function:: __device__ size_t rocshmem_TYPENAME_wait_until_any(TYPE *ivars, size_t nelems, const int* status, int cmp, TYPE val)
:param ivars: Pointer to memory on the symmetric heap to wait for.
:param nelems: Number of elements in the ``ivars`` array.
:param status: Array of length ``nelems`` to exclude elements from the wait.
:param cmp: Operation for the comparison.
:param val: Value to compare.
:returns: The index of an element in the ``ivars`` array that satisfies the wait condition. If the wait set is empty, this routine returns ``SIZE_MAX``.
**Description:**
This routine blocks the caller until any of the condition ``(ivars[i] cmp val)`` is true.
Valid ``cmp`` values are listed in :ref:`CMP_VALUES`.
Valid ``TYPENAME`` and ``TYPE`` values are listed in :ref:`STANDARD_AMO_TYPES`.
ROCSHMEM_WAIT_UNTIL_SOME
------------------------
.. cpp:function:: __device__ size_t rocshmem_TYPENAME_wait_until_some(TYPE *ivars, size_t nelems, size_t* indices, const int* status, int cmp, TYPE val)
:param ivars: Pointer to memory on the symmetric heap to wait for.
:param nelems: Number of elements in the ``ivars`` array.
:param indices: List of indices with a length of at least ``nelems``.
:param status: Array of length ``nelems`` to exclude elements from the wait.
:param cmp: Operation for the comparison.
:param val: Value to compare.
:returns: The number of indices returned in the indices array. If the wait set is empty, this routine returns ``0``.
**Description:**
This routine blocks the caller until any of the conditions ``(ivars[i] cmp val)`` is true.
Valid ``cmp`` values are listed in :ref:`CMP_VALUES`.
Valid ``TYPENAME`` and ``TYPE`` values are listed in :ref:`STANDARD_AMO_TYPES`.
ROCSHMEM_TEST
-------------
.. cpp:function:: __device__ int rocshmem_TYPENAME_test(TYPE *ivars, int cmp, TYPE val)
:param ivars: Pointer to memory on the symmetric heap to wait for.
:param cmp: Operation for the comparison.
:param val: Value to compare the memory at ``ivars`` to.
:returns: ``1`` if the evaluation is true. ``0`` otherwise.
**Description:**
This routine tests if the condition ``(*ivars cmp val)`` is true.
ROCSHMEM_SIGNAL_WAIT_UNTIL_ON_STREAM
-------------------------------------
.. cpp:function:: __host__ void rocshmem_signal_wait_until_on_stream(uint64_t *sig_addr, int cmp, uint64_t cmp_value, hipStream_t stream)
:param sig_addr: Address of the signal variable on the symmetric heap.
:param cmp: Comparison operator (e.g., ROCSHMEM_CMP_EQ, ROCSHMEM_CMP_GE, etc.).
:param cmp_value: Value to compare against.
:param stream: HIP stream on which to enqueue the operation.
:returns: None.
**Description:**
This routine enqueues a wait operation on a HIP stream. The function blocks the calling thread
until the signal variable at ``sig_addr`` satisfies the comparison condition ``(*sig_addr cmp cmp_value)``.
The wait operation is executed asynchronously on the specified stream. The caller must synchronize
the stream (e.g., using ``hipStreamSynchronize``) to ensure the wait condition has been satisfied.
Valid ``cmp`` values are listed in :ref:`CMP_VALUES`.
.. _CMP_VALUES:
Supported comparisons
---------------------
The following table lists the point-to-point comparison constants:
.. list-table:: Point-to-Point Comparison Constants
:widths: 20 20
:header-rows: 1
* - Constant
- Description
* - ROCSHMEM_CMP_EQ
- Equal
* - ROCSHMEM_CMP_NE
- Not equal
* - ROCSHMEM_CMP_GT
- Greater than
* - ROCSHMEM_CMP_GE
- Greater than or equal to
* - ROCSHMEM_CMP_LT
- Less than
* - ROCSHMEM_CMP_LE
- Less than or equal to
projects/rocshmem/docs/api/rma.rst
+278
Datei anzeigen
@@ -0,0 +1,278 @@
.. meta::
:description: rocSHMEM intra-kernel networking runtime for AMD dGPUs on the ROCm platform.
:keywords: rocSHMEM, API, ROCm, documentation, HIP, Networking, Communication
.. _rocshmem-api-rma:
-----------------------------------------
Remote memory access routines
-----------------------------------------
- Routines with the ``_wave`` and ``_wg`` suffixes require all threads in a wavefront and workgroup, respectively,
to call the routine with the same parameters.
- Routines with the ``_nbi`` substring will return as soon as the request is posted.
- Routines without the ``_nbi`` substring will block until the operation completes locally.
- Valid ``TYPENAME`` and ``TYPE`` values can be found in RMA_TYPES_.
ROCSHMEM_PUT
------------
.. cpp:function:: __device__ void rocshmem_TYPENAME_put(TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_put_wave(TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_put_wg(TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_put_nbi(TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_put_nbi_wave(TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_put_nbi_wg(TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_put(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_put_wave(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_put_wg(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_put_nbi(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_put_nbi_wave(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_put_nbi_wg(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, int pe)
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param source: Source address. Must be an address on the symmetric heap.
:param nelems: The number of elements to transfer.
:param pe: PE of the remote process.
:returns: None.
**Description:**
This routine writes contiguous data of ``nelems`` elements from source on the calling PE to ``dest`` at ``pe``.
ROCSHMEM_PUTMEM
---------------
.. cpp:function:: __device__ void rocshmem_putmem(void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_putmem_wave(void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_putmem_wg(void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_putmem_nbi(void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_putmem_nbi_wave(void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_putmem_nbi_wg(void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_putmem(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_putmem_wave(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_putmem_wg(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_putmem_nbi(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_putmem_nbi_wave(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_putmem_nbi_wg(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, int pe)
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param source: Source address. Must be an address on the symmetric heap.
:param nelems: Size of the transfer in bytes.
:param pe: PE of the remote process.
:returns: None.
**Description:**
This routine writes contiguous data of ``nelems`` bytes from source on the calling PE to ``dest`` at ``pe``.
ROCSHMEM_PUTMEM_ON_STREAM
--------------------------
.. cpp:function:: __host__ void rocshmem_putmem_on_stream(void *dest, const void *source, size_t nelems, int pe, hipStream_t stream)
:param dest: Destination address. Must be an address on the symmetric heap.
:param source: Source address. Must be an address on the symmetric heap.
:param nelems: Size of the transfer in bytes.
:param pe: PE of the remote process.
:param stream: HIP stream on which to enqueue the operation.
:returns: None.
**Description:**
This routine enqueues a putmem RMA operation on a HIP stream. The function writes contiguous
data of ``nelems`` bytes from source on the calling PE to ``dest`` at ``pe``. The operation
is enqueued on the specified stream and will execute asynchronously. The caller must
synchronize the stream (e.g., using ``hipStreamSynchronize``) to ensure completion.
ROCSHMEM_P
----------
.. cpp:function:: __device__ void rocshmem_TYPENAME_p(TYPE *dest, TYPE value, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_p(rocshmem_ctx_t ctx, TYPE *dest, TYPE value, int pe)
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param value: Value to write to ``dest`` at ``pe``.
:param pe: PE of the remote process.
:returns: None.
**Description:**
This routine writes a single value to to ``dest`` at ``pe``.
ROCSHMEM_GET
------------
.. cpp:function:: __device__ void rocshmem_TYPENAME_get(TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_get_wave(TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_get_wg(TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_get_nbi(TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_get_nbi_wave(TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_get_nbi_wg(TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_get(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_get_wave(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_get_wg(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_get_nbi(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_get_nbi_wave(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_get_nbi_wg(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, int pe)
:param ctx: Context with which to perform this operation.
:param dest: Destination address; Must be an address on the symmetric heap.
:param source: Source address. Must be an address on the symmetric heap.
:param nelems: The number of elements to transfer.
:param pe: PE of the remote process.
:returns: None.
**Description:**
This routine reads contiguous data of ``nelems`` elements from source on ``pe`` to ``dest`` on the calling PE.
ROCSHMEM_GETMEM
---------------
.. cpp:function:: __device__ void rocshmem_getmem(void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_getmem_wave(void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_getmem_wg(void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_getmem_nbi(void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_getmem_nbi_wave(void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_getmem_nbi_wg(void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_getmem(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_getmem_wave(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_getmem_wg(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_getmem_nbi(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_getmem_nbi_wave(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_getmem_nbi_wg(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, int pe)
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param source: Source address. Must be an address on the symmetric heap.
:param nelems: Size of the transfer in bytes.
:param pe: PE of the remote process.
:returns: None.
**Description:**
This routine reads contiguous data of ``nelems`` bytes from source on ``pe`` to ``dest`` on the calling PE.
ROCSHMEM_GETMEM_ON_STREAM
--------------------------
.. cpp:function:: __host__ void rocshmem_getmem_on_stream(void *dest, const void *source, size_t nelems, int pe, hipStream_t stream)
:param dest: Destination address. Must be an address on the symmetric heap.
:param source: Source address. Must be an address on the symmetric heap.
:param nelems: Size of the transfer in bytes.
:param pe: PE of the remote process.
:param stream: HIP stream on which to enqueue the operation.
:returns: None.
**Description:**
This routine enqueues a getmem RMA operation on a HIP stream. The function reads contiguous
data of ``nelems`` bytes from source on ``pe`` to ``dest`` on the calling PE. The operation
is enqueued on the specified stream and will execute asynchronously. The caller must
synchronize the stream (e.g., using ``hipStreamSynchronize``) to ensure completion.
ROCSHMEM_G
----------
.. cpp:function:: __device__ float rocshmem_ctx_float_g(rocshmem_ctx_t ctx, const float *source, int pe)
.. cpp:function:: __device__ float rocshmem_float_g(const float *source, int pe)
:param ctx: Context with which to perform this operation.
:param source: Source address. Must be an address on the symmetric heap.
:param pe: PE of the remote process.
:returns: The value read from source at ``pe``.
**Description:**
This routine reads and returns single value from source at ``pe``.
Supported RMA data types
------------------------
The following table lists the supported RMA data types:
.. _RMA_TYPES:
.. list-table:: RMA Data Types
:widths: 10 20 20
:header-rows: 1
* - TYPE
- TYPENAME
- Supported
* - float
- float
- Yes
* - double
- double
- Yes
* - long double
- longdouble
- No
* - char
- char
- Yes
* - signed char
- schar
- Yes
* - short
- short
- Yes
* - int
- int
- Yes
* - long
- long
- Yes
* - long long
- longlong
- Yes
* - unsigned char
- uchar
- Yes
* - unsigned short
- ushort
- Yes
* - unsigned int
- uint
- Yes
* - unsigned long
- ulong
- Yes
* - unsigned long long
- ulonglong
- Yes
* - int8_t
- int8
- No
* - int16_t
- int16
- No
* - int32_t
- int32
- No
* - int64_t
- int64
- Yes
* - uint8_t
- uint8
- No
* - uint16_t
- uint16
- No
* - uint32_t
- uint32
- No
* - uint64_t
- uint64
- No
* - size_t
- size
- No
* - ptrdiff_t
- ptrdiff
- No
projects/rocshmem/docs/api/sigops.rst
+125
Datei anzeigen
@@ -0,0 +1,125 @@
.. meta::
:description: rocSHMEM intra-kernel networking runtime for AMD dGPUs on the ROCm platform.
:keywords: rocSHMEM, API, ROCm, documentation, HIP, Networking, Communication
.. _rocshmem-api-sigops:
---------------------
Signaling operations
---------------------
ROCSHMEM_PUTMEM_SIGNAL
----------------------
.. cpp:function:: __device__ void rocshmem_putmem_signal(void *dest, const void *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_putmem_signal_wave(void *dest, const void *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_putmem_signal_wg(void *dest, const void *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_putmem_signal_nbi(void *dest, const void *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_putmem_signal_nbi_wave(void *dest, const void *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_putmem_signal_nbi_wg(void *dest, const void *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_putmem_signal(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_putmem_signal_wave(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_putmem_signal_wg(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_putmem_signal_nbi(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_putmem_signal_nbi_wave(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_putmem_signal_nbi_wg(rocshmem_ctx_t ctx, void *dest, const void *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param source: Source address. Must be an address on the symmetric heap.
:param nelems: The number of bytes to transfer.
:param sig_addr: Signal address. Must be an address on the symmetric heap.
:param signal: Signal value.
:param sig_op: Atomic operation to apply the signal value.
:param pe: PE of the remote process.
:returns: None.
**Description:**
This function writes contiguous data of ``nelems`` bytes from source on the calling PE to ``dest`` at ``pe``,
then applies ``sig_op`` at ``sig_addr`` with the signal value.
Valid ``sig_op values`` are listed in SIGNAL_OPERATORS_.
ROCSHMEM_PUT_SIGNAL
-------------------
.. cpp:function:: __device__ void rocshmem_TYPENAME_put_signal(TYPE *dest, const TYPE *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_put_signal_wave(TYPE *dest, const TYPE *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_put_signal_wg(TYPE *dest, const TYPE *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_put_signal_nbi(TYPE *dest, const TYPE *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_put_signal_nbi_wave(TYPE *dest, const TYPE *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_TYPENAME_put_signal_nbi_wg(TYPE *dest, const TYPE *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_put_signal(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_put_signal_wave(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_put_signal_wg(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_put_signal_nbi(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_put_signal_nbi_wave(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
.. cpp:function:: __device__ void rocshmem_ctx_TYPENAME_put_signal_nbi_wg(rocshmem_ctx_t ctx, TYPE *dest, const TYPE *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe)
:param ctx: Context with which to perform this operation.
:param dest: Destination address. Must be an address on the symmetric heap.
:param source: Source address. Must be an address on the symmetric heap.
:param nelems: The number of elements of size ``TYPE`` to transfer.
:param sig_addr: Signal address. Must be an address on the symmetric heap.
:param signal: Signal value.
:param sig_op: Atomic operation to apply the signal value.
:param pe: PE of the remote process.
:returns: None.
**Description:**
This function writes contiguous data of ``nelems`` elements of ``TYPE`` from source on the calling PE to ``dest`` at ``pe``,
then applies ``sig_op`` at ``sig_addr`` with the signal value.
Valid ``sig_op values`` are listed in SIGNAL_OPERATORS_.
Valid ``TYPENAME`` and ``TYPE`` values are listed in :ref:`RMA_TYPES`.
ROCSHMEM_PUTMEM_SIGNAL_ON_STREAM
---------------------------------
.. cpp:function:: __host__ void rocshmem_putmem_signal_on_stream(void *dest, const void *source, size_t nelems, uint64_t *sig_addr, uint64_t signal, int sig_op, int pe, hipStream_t stream)
:param dest: Destination address on the remote PE. Must be an address on the symmetric heap.
:param source: Source address on the local PE. Must be an address on the symmetric heap.
:param nelems: Size of the transfer in bytes.
:param sig_addr: Address of signal variable on the remote PE. Must be an address on the symmetric heap.
:param signal: Signal value to be written.
:param sig_op: Signal operation (ROCSHMEM_SIGNAL_SET or ROCSHMEM_SIGNAL_ADD).
:param pe: PE number of the remote PE.
:param stream: HIP stream on which to enqueue the operation.
:returns: None.
**Description:**
This routine enqueues a put-with-signal operation on a HIP stream. The function writes contiguous
data of ``nelems`` bytes from source on the calling PE to ``dest`` at ``pe``, then applies ``sig_op``
at ``sig_addr`` with the signal value. The operation is enqueued on the specified stream and will
execute asynchronously. The caller must synchronize the stream (e.g., using ``hipStreamSynchronize``)
to ensure completion.
Valid ``sig_op`` values are listed in SIGNAL_OPERATORS_.
ROCSHMEM_SIGNAL_FETCH
---------------------
.. cpp:function:: __device__ uint64_t rocshmem_signal_fetch(const uint64_t *sig_addr)
.. cpp:function:: __device__ uint64_t rocshmem_signal_fetch_wg(const uint64_t *sig_addr)
.. cpp:function:: __device__ uint64_t rocshmem_signal_fetch_wave(const uint64_t *sig_addr)
:param sig_addr: Signal address. Must be an address on the symmetric heap.
:returns: Value at ``sig_addr``.
**Description:**
This function atomically fetches the value stored at ``sig_addr``.
Signal operators
----------------
.. _SIGNAL_OPERATORS:
.. list-table:: Signal Operators
:widths: 20 40
:header-rows: 1
* - Value
- Description
* - ROCSHMEM_SIGNAL_SET
- The signaling operation routines will atomically set the signal value at ``sig_addr``.
* - ROCSHMEM_SIGNAL_ADD
- The signaling operation routines will atomically add the signal value at ``sig_addr``.
projects/rocshmem/docs/api/teams.rst
+90
Datei anzeigen
@@ -0,0 +1,90 @@
.. meta::
:description: rocSHMEM intra-kernel networking runtime for AMD dGPUs on the ROCm platform.
:keywords: rocSHMEM, API, ROCm, documentation, HIP, Networking, Communication
.. _rocshmem-api-teams:
-------------------------
Team management routines
-------------------------
ROCSHMEM_TEAM_MY_PE
-------------------
.. cpp:function:: __host__ int rocshmem_team_my_pe(rocshmem_team_t team)
:param team: The team to query.
:returns: PE ID of the caller in the provided team.
**Description:**
This routine queries the PE ID of the caller in a team.
ROCSHMEM_TEAM_N_PES
-------------------
.. cpp:function:: __host__ int rocshmem_team_n_pes(rocshmem_team_t team)
:param team: The team to query.
:returns: Number of PEs in the provided team.
**Description:**
This routine queries the number of PEs in a team.
ROCSHMEM_TEAM_TRANSLATE_PE
--------------------------
.. cpp:function:: __host__ int rocshmem_team_translate_pe(rocshmem_team_t src_team, int src_pe, rocshmem_team_t dest_team)
:param src_team: Handle of the team from which to translate.
:param src_pe: PE-of-interest's index in ``src_team``.
:param dest_team: Handle of the team to which to translate.
:returns: PE of ``src_pe`` in ``dest_team``.
If any input is invalid or if ``src_pe`` is
not in both source and destination teams, a value of ``-1`` is returned.
**Description:**
This routine translates the PE in ``src_team`` to that in ``dest_team``.
ROCSHMEM_TEAM_SPLIT_STRIDED
---------------------------
.. cpp:function:: __host__ int rocshmem_team_split_strided(rocshmem_team_t parent_team, int start, int stride, int size, const rocshmem_team_config_t *config, long config_mask, rocshmem_team_t *new_team)
:param parent_team: The team to split from.
:param start: The lowest PE number of the subset of the PEs
from the parent team that will form the new
team.
:param stride: The stride between team PE members in the
parent team that comprise the subset of PEs
that will form the new team.
:param size: The number of PEs in the new team.
:param config: Pointer to the config parameters for the new team.
:param config_mask: Bitwise mask representing parameters to use from config.
:param new_team: Pointer to the newly created team.
If an error occurs during team creation, or if the PE in
the parent team is not in the new team, the value will be
``ROCSHMEM_TEAM_INVALID``.
:returns: Zero upon successful team creation; non-zero if erroneous.
**Description:**
This routine creates a new a team of PEs. It must be called by all PEs in the parent team.
ROCSHMEM_TEAM_DESTROY
---------------------
.. cpp:function:: __host__ void rocshmem_team_destroy(rocshmem_team_t team)
:param team: The team to destroy. The behavior is undefined if
the input team is ``ROCSHMEM_TEAM_WORLD`` or any other
invalid team. If the input is ``ROCSHMEM_TEAM_INVALID``,
this function will not perform any operation.
:returns: None
**Description:**
This routine destroys a team. It must be called by all PEs in the team.
You must destroy all private contexts created in the
team before destroying this team. Otherwise, the behavior
is undefined. This call will destroy only the shareable contexts
created from the referenced team.
projects/rocshmem/docs/compile_and_run.rst
+60
Datei anzeigen
@@ -0,0 +1,60 @@
.. meta::
:description: Information on how to compile and run rocSHMEM applications.
:keywords: rocSHMEM, ROCm, library, API, compile, link, hipcc
.. _running-applications:
--------------------------------------------------
Compiling and running rocSHMEM applications
--------------------------------------------------
This topic explains how to compile and run rocSHMEM applications.
Compiling and linking with rocSHMEM
-----------------------------------
rocSHMEM is a library that can be statically linked to your application during compilation with ``hipcc``. For more information, see :doc:`HIPCC <hipcc:index>`.
When compiling your application with ``hipcc``, you must include the rocSHMEM header files and the rocSHMEM library.
Because rocSHMEM depends on MPI (Message Passing Interface), you must manually add the arguments for MPI linkage instead of using ``mpicc``.
When using ``hipcc`` directly without a build system, it's recommended to perform the compilation and linking steps separately.
Example compile and link commands are provided at the top of the example files in the ``examples`` directory:
.. code-block:: bash
# Compile
hipcc -c -fgpu-rdc -x hip rocshmem_allreduce_test.cc \
-I/opt/rocm/include \
-I$ROCSHMEM_INSTALL_DIR/include \
-I$OPENMPI_UCX_INSTALL_DIR/include/
# Link
hipcc -fgpu-rdc --hip-link rocshmem_allreduce_test.o -o rocshmem_allreduce_test \
$ROCSHMEM_INSTALL_DIR/lib/librocshmem.a \
$OPENMPI_UCX_INSTALL_DIR/lib/libmpi.so \
-L/opt/rocm/lib -lamdhip64 -lhsa-runtime64
If your project uses CMake, see
`Using CMake with AMD ROCm <https://rocmdocs.amd.com/en/latest/conceptual/cmake-packages.html>`_.
Running a rocSHMEM application
------------------------------
Applications using rocSHMEM typically deploy multiple processes, usually one per GPU.
The MPI launcher, for example, ``mpiexec`` with Open MPI, is used to start the required number
of processes. For example, to launch two ``getmem`` example processes (available when compiled from source):
.. code-block:: bash
mpiexec --map-by numa --mca pml ucx --mca osc ucx -np 2 ./build/examples/rocshmem_getmem_test
See the `Open MPI documentation <https://docs.open-mpi.org/en/main/>`_ for more information about ``mpiexec`` command line parameters.
.. note::
Some systems may have multiple MPI installations, some of which do not
have GPU support enabled. You must use the ``mpiexec`` from the expected
MPI library, especially when using the MPI built by yourself
as part of :ref:`install-dependencies`.
projects/rocshmem/docs/conf.py
+36
Datei anzeigen
@@ -0,0 +1,36 @@
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
import re
from rocm_docs import ROCmDocs
with open('../include/rocshmem/rocshmem.hpp', encoding='utf-8') as f:
match = re.search(r'constexpr char VERSION\[\] = "([0-9.]+)[^0-9.]+', f.read())
if not match:
raise ValueError("VERSION not found!")
version_number = match[1]
left_nav_title = f"rocSHMEM {version_number} documentation"
# for PDF output on Read the Docs
project = "rocSHMEM"
author = "Advanced Micro Devices, Inc."
copyright = "Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved."
version = version_number
release = version_number
external_toc_path = "./sphinx/_toc.yml"
docs_core = ROCmDocs(left_nav_title)
docs_core.run_doxygen(doxygen_root="doxygen", doxygen_path="doxygen/xml")
docs_core.setup()
external_projects_current_project = "rocshmem"
cpp_id_attributes = ["__host__", "__global__", "__device__"]
exclude_patterns = ["README.md"]
for sphinx_var in ROCmDocs.SPHINX_VARS:
globals()[sphinx_var] = getattr(docs_core, sphinx_var)
projects/rocshmem/docs/doxygen/Doxyfile
+2452
Datei anzeigen
Datei-Diff unterdrückt, da er zu groß ist Diff laden
projects/rocshmem/docs/index.rst
+42
Datei anzeigen
@@ -0,0 +1,42 @@
.. meta::
:description: rocSHMEM is a runtime that provides GPU-centric networking through an OpenSHMEM-like interface.
:keywords: rocSHMEM, ROCm, OpenSHMEM, library, API, IPC, RO
****************************
rocSHMEM documentation
****************************
The ROCm OpenSHMEM (rocSHMEM) is an intra-kernel networking library that provides GPU-centric networking through an OpenSHMEM-like interface. It simplifies application code complexity and enables finer communication and computation overlap than traditional host-driven networking. rocSHMEM uses a single symmetric heap allocated to GPU memories. For more information, see :doc:`introduction`.
The rocSHMEM public repository is located at `<https://github.com/ROCm/rocSHMEM>`_.
.. grid:: 2
:gutter: 3
.. grid-item-card:: Install
* :doc:`Install rocSHMEM <./install>`
.. grid-item-card:: How to
* :doc:`Compile and run applications <./compile_and_run>`
.. grid-item-card:: API reference
* :doc:`Library setup, exit, and query routines <./api/init>`
* :doc:`Memory management routines <./api/memory_management>`
* :doc:`Team management routines <./api/teams>`
* :doc:`Context management routines <./api/ctx>`
* :doc:`Environment variables <./api/env_variables>`
* :doc:`Remote memory access routines <./api/rma>`
* :doc:`Atomic memory operations <./api/amo>`
* :doc:`Signaling operations <./api/sigops>`
* :doc:`Collective routines <./api/coll>`
* :doc:`Point-to-point synchronization routines <./api/pt2pt_sync>`
* :doc:`Memory ordering routines <./api/memory_ordering>`
To contribute to the documentation, refer to
`Contributing to ROCm <https://rocm.docs.amd.com/en/latest/contribute/contributing.html>`_.
You can find licensing information on the
`Licensing <https://rocm.docs.amd.com/en/latest/about/license.html>`_ page.
projects/rocshmem/docs/install.rst
+222
Datei anzeigen
@@ -0,0 +1,222 @@
.. meta::
:description: Instruction on how to install rocSHMEM.
:keywords: rocSHMEM, ROCm, install, build, dependencies, MPI, UCX, Open MPI
.. _install-rocshmem:
---------------------------
Installing rocSHMEM
---------------------------
This topic describes how to install rocSHMEM.
Requirements
------------
* ROCm 6.4.0 or later, including the :doc:`HIP runtime <hip:index>`. For more information, see `ROCm installation for Linux <https://rocm.docs.amd.com/projects/install-on-linux/en/latest/>`_.
* The following AMD GPUs have been fully tested for compatibility with rocSHMEM:
* MI250X
* MI300X
* MI350X (Requires ROCm 7.0 or later)
.. note::
Other AMD GPUs might function with unknown limitations. For the complete list of supported hardware, see `ROCm System Requirements <https://rocm.docs.amd.com/projects/install-on-linux-internal/en/latest/reference/system-requirements.html>`_.
* The RO backend requires ROCm-aware Open MPI and UCX. When using the IPC or GDA backends, MPI is optional.
For more information about installing ROCm-aware Open MPI and UCX, see :ref:`install-dependencies`.
* Inter-node communication requires AMD Pollara IONIC, Broadcom Thor 2, or CX7 Infiniband NICs.
Available network backends
--------------------------
rocSHMEM supports the following network backends:
* The **IPC (Inter-Process Communication)** backend enables fast communication between GPUs on the same host using ROCm inter-process mechanisms. It does not support inter-node communication.
* The **RO (Reverse Offload)** backend enables communication between GPUs on different nodes through a NIC, using a host-based proxy to forward communication orders to and from the GPU. RO is built on an MPI-RMA compatibility layer.
* The **GDA (GPU Direct Async)** backend enables communication between GPUs on different nodes through a NIC. In this backend, the GPU directly interacts with the NIC with no host (CPU) involvement in the critical path of communication.
You can activate IPC, RO, and GDA backends in the same rocSHMEM build.
.. note::
When RO + IPC is active, all atomic operations use the RO backend, even for intra-node communication.
When GDA + IPC is active, all atomic operations use the GDA backend, even for intra-node communication.
Installing from a package manager
---------------------------------
On Ubuntu, you can install rocSHMEM by running:
.. code-block:: bash
apt install rocshmem-dev
.. note::
This installation method requires ROCm 6.4 or later. You must manually build dependencies such as Open MPI and UCX, because the distribution packaged versions don't include full accelerator support. For more information, see :ref:`install-dependencies`.
.. _install-dependencies:
Building dependencies
---------------------
GDA NIC dependencies
^^^^^^^^^^^^^^^^^^^^
- GDA on Mellanox NICs should work on any recent version of rdma-core.
- GDA on Broadcom Thor requires driver version 233.2.108.0 and firmware version 233.2.104.0 or later.
Building rocSHMEM with MPI (Optional)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocSHMEM requires ROCm-Aware Open MPI and UCX for the RO backend.
MPI is optional with the IPC and GDA backends.
Other MPI implementations, such as MPICH, have not been fully tested.
To build and configure ROCm-Aware UCX 1.17.0 or later, run:
.. code-block:: bash
git clone https://github.com/ROCm/ucx.git -b v1.17.x
cd ucx
./autogen.sh
./configure --prefix=<prefix_dir> --with-rocm=<rocm_path> --enable-mt
make -j 8
make -j 8 install
To build Open MPI 5.0.7 or later with UCX support, run:
.. code-block:: bash
git clone --recursive https://github.com/open-mpi/ompi.git -b v5.0.x
cd ompi
./autogen.pl
./configure --prefix=<prefix_dir> --with-rocm=<rocm_path> --with-ucx=<ucx_path>
make -j 8
make -j 8 install
Alternatively, you can use a script to install dependencies:
.. code-block:: bash
export BUILD_DIR=/path/to/not_rocshmem_src_or_build/dependencies
/path/to/rocshmem_src/scripts/install_dependencies.sh
.. note::
Configuration options vary by platform. Review the script to ensure it is compatible with your system.
For more information about OpenMPI-UCX support, see
`GPU-enabled Message Passing Interface <https://rocm.docs.amd.com/en/latest/how-to/gpu-enabled-mpi.html>`_.
Installing from source
--------------------------------
You can choose from three communication backends at build time for rocSHMEM: IPC, RO, and GDA.
Backend can be combined during build time.
MPI is not required to build rocSHMEM. To disable MPI, pass
the following flag to the build configuration scripts ``-DUSE_EXTERNAL_MPI=OFF``.
However, this will disable the functional and unit
tests, as they required MPI to run.
All backends build
^^^^^^^^^^^^^^^^^^
To build and install rocSHMEM with all three backends, run:
.. code-block:: bash
git clone git@github.com:ROCm/rocSHMEM.git
cd rocSHMEM
mkdir build
cd build
../scripts/build_configs/all_backends
The build script passes configuration options to CMake to set up a canonical build.
.. note::
This builds rocSHMEM with all backends. You can select IPC, RO, GDA, or any combination at runtime. However, this portability can reduce performance, so the other build scripts are recommended if you need maximum performance.
GDA backend build
^^^^^^^^^^^^^^^^^
To build and install rocSHMEM with the GDA backends, run:
.. code-block:: bash
git clone git@github.com:ROCm/rocSHMEM.git
cd rocSHMEM
mkdir build
cd build
# Choose one of the following scripts for your NIC vendor:
../scripts/build_configs/gda_bnxt # Broadcom
../scripts/build_configs/gda_ionic # AMD Pollara
../scripts/build_configs/gda_mlx5 # Mellanox
The build script passes configuration options to CMake to set up a canonical build.
RO and IPC backend build
^^^^^^^^^^^^^^^^^^^^^^^^
To build and install rocSHMEM with the hybrid RO (off-node) and IPC (on-node) backends, run:
.. code-block:: bash
git clone git@github.com:ROCm/rocSHMEM.git
cd rocSHMEM
mkdir build
cd build
../scripts/build_configs/ro_ipc
The build script passes configuration options to CMake to set up a canonical build.
.. note::
The only officially supported configuration for the RO backend uses Open MPI and UCX with a CX7 InfiniBand adapter. For more information, see :ref:`install-dependencies`. Other configurations, such as MPI implementations that are thread-safe and support GPU buffers, might work but are considered experimental.
IPC only backend build
^^^^^^^^^^^^^^^^^^^^^^
To build and install rocSHMEM with the IPC on-node, GPU-to-GPU backend, run:
.. code-block:: bash
git clone git@github.com:ROCm/rocSHMEM.git
cd rocSHMEM
mkdir build
cd build
../scripts/build_configs/ipc_single
The build script passes configuration options to CMake to setup a single-node build.
This is similar to the default build in ROCm 6.4.
.. note::
The default configuration changed from IPC only in ROCm 6.4 (built with the ``ipc_single`` script) to RO and IPC in ROCm 7.0 (built with the ``ro_ipc`` script).
Other experimental configuration scripts are available in ``./scripts/build_configs``, but only ``ipc_single`` and ``ro_ipc``
are officially supported.
Installation prefix
^^^^^^^^^^^^^^^^^^^
By default, the build scripts install the library to ``~/rocshmem``. You can customize the installation path by adding
the desired path as the script parameter. For example, to relocate the default configuration:
.. code-block:: bash
../scripts/build_configs/ro_ipc /path/to/install
projects/rocshmem/docs/introduction.rst
+45
Datei anzeigen
@@ -0,0 +1,45 @@
.. meta::
:description: rocSHMEM intra-kernel networking runtime for AMD GPUs on the ROCm platform.
:keywords: rocSHMEM, API, ROCm, documentation, HIP, Networking, Communication
.. _rocshmem-introduction:
---------------------------
What is rocSHMEM?
---------------------------
The ROCm OpenSHMEM (rocSHMEM) is an intra-kernel networking library that provides GPU-centric networking through an OpenSHMEM-like interface. It simplifies application code complexity and enables finer communication and computation overlap than traditional host-driven networking. rocSHMEM uses a single symmetric heap allocated on GPU memories.
The rocSHMEM programming model
-------------------------------
Defining how OpenSHMEM applications interact with GPUs remains an
ongoing active discussion within the OpenSHMEM community, and the OpenSHMEM
specification has yet to coalesce on this topic.
rocSHMEM extends beyond the OpenSHMEM specification to add semantics that
support GPU kernel communication while maintaining close resemblance to
the original OpenSHMEM specification semantics.
Applications using :doc:`HIP <hip:index>` can interface with rocSHMEM.
Using the HIP programming model,
rocSHMEM provides ``__host__`` APIs for host code,
and ``__device__`` APIs for GPU kernels.
Device APIs without special suffixes or infixes , for example, ``_wg`` or ``_wave``,
must be called by a single thread.
GPU specific ``_wg`` and ``_wave`` APIs are designed to be called by multiple GPU threads
and will block until the calling scope completes.
These APIs can be called in divergent code paths, but this is not recommended.
Wavefront APIs
==============
Wavefront APIs are those with the ``_wave`` suffix.
The parameters in which these routines are called must be
the same for every thread in the wavefront.
The behavior is undefined if any thread calls these routines with different parameters. These APIs will block until the calling wavefront is complete.
Workgroup APIs
==============
The workgroup APIs have the ``_wg`` suffix or ``_wg_`` infix.
The parameters in which these routines are called must be
the same for every thread in the workgroup.
The behavior is undefined if any thread calls these routines with different parameters. These APIs will block until the calling workgroup is complete.
projects/rocshmem/docs/license.md
+4
Datei anzeigen
@@ -0,0 +1,4 @@
# License
```{include} ../LICENSE.md
```
projects/rocshmem/docs/sphinx/_toc.yml.in
+48
Datei anzeigen
@@ -0,0 +1,48 @@
defaults:
numbered: False
root: index
subtrees:
- entries:
- file: introduction.rst
title: What is rocSHMEM?
- caption: Install
entries:
- file: install.rst
title: Install rocSHMEM
- caption: How to
entries:
- file: compile_and_run.rst
title: Compile and run applications
- caption: API reference
entries:
- file: api/init.rst
title: Library setup, exit, and query routines
- file: api/memory_management.rst
title: Memory management routines
- file: api/teams.rst
title: Team management routines
- file: api/ctx.rst
title: Context management routines
- file: api/env_variables.rst
title: Environment variables
- file: api/rma.rst
title: Remote memory access routines
- file: api/amo.rst
title: Atomic memory operations
- file: api/sigops.rst
title: Signaling operations
- file: api/coll.rst
title: Collective routines
- file: api/pt2pt_sync.rst
title: Point-to-point synchronization routines
- file: api/memory_ordering.rst
title: Memory ordering routines
- caption: About
entries:
- file: license.rst
projects/rocshmem/docs/sphinx/requirements.in
+2
Datei anzeigen
@@ -0,0 +1,2 @@
rocm-docs-core==1.31.2
projects/rocshmem/docs/sphinx/requirements.txt
+278
Datei anzeigen
@@ -0,0 +1,278 @@
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# pip-compile requirements.in
#
accessible-pygments==0.0.5
# via pydata-sphinx-theme
alabaster==1.0.0
# via sphinx
asttokens==3.0.0
# via stack-data
attrs==25.3.0
# via
# jsonschema
# jupyter-cache
# referencing
babel==2.17.0
# via
# pydata-sphinx-theme
# sphinx
beautifulsoup4==4.13.4
# via pydata-sphinx-theme
breathe==4.36.0
# via rocm-docs-core
certifi==2025.4.26
# via requests
cffi==2.0.0
# via
# cryptography
# pynacl
charset-normalizer==3.4.2
# via requests
click==8.1.8
# via
# jupyter-cache
# sphinx-external-toc
comm==0.2.2
# via ipykernel
cryptography==44.0.3
# via pyjwt
debugpy==1.8.14
# via ipykernel
decorator==5.2.1
# via ipython
deprecated==1.2.18
# via pygithub
docutils==0.21.2
# via
# myst-parser
# pydata-sphinx-theme
# sphinx
exceptiongroup==1.2.2
# via ipython
executing==2.2.0
# via stack-data
fastjsonschema==2.21.1
# via
# nbformat
# rocm-docs-core
gitdb==4.0.12
# via gitpython
gitpython==3.1.44
# via rocm-docs-core
greenlet==3.2.1
# via sqlalchemy
idna==3.10
# via requests
imagesize==1.4.1
# via sphinx
importlib-metadata==8.7.0
# via
# jupyter-cache
# myst-nb
ipykernel==6.29.5
# via myst-nb
ipython==8.36.0
# via
# ipykernel
# myst-nb
jedi==0.19.2
# via ipython
jinja2==3.1.6
# via
# myst-parser
# sphinx
jsonschema==4.23.0
# via nbformat
jsonschema-specifications==2025.4.1
# via jsonschema
jupyter-cache==1.0.1
# via myst-nb
jupyter-client==8.6.3
# via
# ipykernel
# nbclient
jupyter-core==5.7.2
# via
# ipykernel
# jupyter-client
# nbclient
# nbformat
markdown-it-py==3.0.0
# via
# mdit-py-plugins
# myst-parser
markupsafe==3.0.2
# via jinja2
matplotlib-inline==0.1.7
# via
# ipykernel
# ipython
mdit-py-plugins==0.4.2
# via myst-parser
mdurl==0.1.2
# via markdown-it-py
myst-nb==1.2.0
# via rocm-docs-core
myst-parser==4.0.1
# via myst-nb
nbclient==0.10.2
# via
# jupyter-cache
# myst-nb
nbformat==5.10.4
# via
# jupyter-cache
# myst-nb
# nbclient
nest-asyncio==1.6.0
# via ipykernel
packaging==25.0
# via
# ipykernel
# pydata-sphinx-theme
# sphinx
parso==0.8.4
# via jedi
pexpect==4.9.0
# via ipython
platformdirs==4.3.8
# via jupyter-core
prompt-toolkit==3.0.51
# via ipython
psutil==7.0.0
# via ipykernel
ptyprocess==0.7.0
# via pexpect
pure-eval==0.2.3
# via stack-data
pycparser==2.22
# via cffi
pydata-sphinx-theme==0.15.4
# via
# rocm-docs-core
# sphinx-book-theme
pygithub==2.6.1
# via rocm-docs-core
pygments==2.19.1
# via
# accessible-pygments
# ipython
# pydata-sphinx-theme
# sphinx
pyjwt[crypto]==2.10.1
# via pygithub
pynacl==1.6.2
# via pygithub
python-dateutil==2.9.0.post0
# via jupyter-client
pyyaml==6.0.2
# via
# jupyter-cache
# myst-nb
# myst-parser
# rocm-docs-core
# sphinx-external-toc
pyzmq==26.4.0
# via
# ipykernel
# jupyter-client
referencing==0.36.2
# via
# jsonschema
# jsonschema-specifications
requests==2.32.4
# via
# pygithub
# sphinx
rocm-docs-core==1.31.2
# via -r requirements.in
rpds-py==0.24.0
# via
# jsonschema
# referencing
six==1.17.0
# via python-dateutil
smmap==5.0.2
# via gitdb
snowballstemmer==3.0.0.1
# via sphinx
soupsieve==2.7
# via beautifulsoup4
sphinx==8.1.3
# via
# breathe
# myst-nb
# myst-parser
# pydata-sphinx-theme
# rocm-docs-core
# sphinx-book-theme
# sphinx-copybutton
# sphinx-design
# sphinx-external-toc
# sphinx-notfound-page
sphinx-book-theme==1.1.4
# via rocm-docs-core
sphinx-copybutton==0.5.2
# via rocm-docs-core
sphinx-design==0.6.1
# via rocm-docs-core
sphinx-external-toc==1.0.1
# via rocm-docs-core
sphinx-notfound-page==1.1.0
# via rocm-docs-core
sphinxcontrib-applehelp==2.0.0
# via sphinx
sphinxcontrib-devhelp==2.0.0
# via sphinx
sphinxcontrib-htmlhelp==2.1.0
# via sphinx
sphinxcontrib-jsmath==1.0.1
# via sphinx
sphinxcontrib-qthelp==2.0.0
# via sphinx
sphinxcontrib-serializinghtml==2.0.0
# via sphinx
sqlalchemy==2.0.40
# via jupyter-cache
stack-data==0.6.3
# via ipython
tabulate==0.9.0
# via jupyter-cache
tomli==2.2.1
# via sphinx
tornado==6.5.1
# via
# ipykernel
# jupyter-client
traitlets==5.14.3
# via
# comm
# ipykernel
# ipython
# jupyter-client
# jupyter-core
# matplotlib-inline
# nbclient
# nbformat
typing-extensions==4.13.2
# via
# beautifulsoup4
# ipython
# myst-nb
# pydata-sphinx-theme
# pygithub
# referencing
# sqlalchemy
urllib3==2.6.3
# via
# pygithub
# requests
wcwidth==0.2.13
# via prompt-toolkit
wrapt==1.17.2
# via deprecated
zipp==3.21.0
# via importlib-metadata