Add 'projects/rccl/' from commit '1f2f9f33bac3e8ecfd84c69af6063d7352c362fc'
git-subtree-dir: projects/rccl git-subtree-mainline:3fd8a0d393git-subtree-split:1f2f9f33ba
Bu işleme şunda yer alıyor:
Ameya Keshava Mallya
@@ -0,0 +1,18 @@
|
||||
.. meta::
|
||||
:description: RCCL is a stand-alone library that provides multi-GPU and multi-node collective communication primitives optimized for AMD GPUs
|
||||
:keywords: RCCL, ROCm, library, API
|
||||
|
||||
.. _api-library:
|
||||
|
||||
***********
|
||||
API library
|
||||
***********
|
||||
|
||||
RCCL (pronounced "Rickle") is a stand-alone library of standard collective communication routines for GPUs, implementing all-reduce, all-gather, reduce, broadcast, reduce-scatter, gather, scatter, and all-to-all. There is also initial support for direct GPU-to-GPU send and receive operations. It has been optimized to achieve high bandwidth on platforms using PCIe, xGMI as well as networking using InfiniBand Verbs or TCP/IP sockets. RCCL supports an arbitrary number of GPUs installed in a single node or multiple nodes, and can be used in either single- or multi-process (e.g., MPI) applications.
|
||||
|
||||
The collective operations are implemented using ring and tree algorithms and have been optimized for throughput and latency. For best performance, small operations can be either batched into larger operations or aggregated through the API.
|
||||
|
||||
Operations
|
||||
==========
|
||||
|
||||
.. doxygenindex::
|
||||
@@ -0,0 +1,165 @@
|
||||
.. meta::
|
||||
:description: RCCL is a stand-alone library that provides multi-GPU and multi-node collective communication primitives optimized for AMD GPUs
|
||||
:keywords: RCCL, ROCm, library, API, reference, environment variable, environment
|
||||
|
||||
.. _env-variables:
|
||||
|
||||
********************************************************************
|
||||
RCCL environment variables
|
||||
********************************************************************
|
||||
|
||||
This section describes the most important RCCL environment variables,
|
||||
which are grouped by functionality.
|
||||
|
||||
Configuration and setup
|
||||
========================
|
||||
|
||||
The configuration and setup environment variables for RCCL are collected
|
||||
in the following table.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 70,30
|
||||
|
||||
* - **Environment variable**
|
||||
- **Value**
|
||||
|
||||
* - | ``NCCL_CONF_FILE``
|
||||
| Specifies the path to the RCCL configuration file.
|
||||
- | String path to configuration file
|
||||
| Default: ``~/.rccl.conf`` or ``/etc/rccl.conf``
|
||||
|
||||
* - | ``NCCL_HOSTID``
|
||||
| Sets the host identifier for multi-node communication.
|
||||
- | String value for host identification
|
||||
| Used for host hash generation
|
||||
|
||||
Logging and debugging
|
||||
=====================
|
||||
|
||||
The logging and debugging environment variables for RCCL are collected
|
||||
in the following table.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 70,30
|
||||
|
||||
* - **Environment variable**
|
||||
- **Value**
|
||||
|
||||
* - | ``RCCL_LOG_LEVEL``
|
||||
| Controls RCCL logging verbosity.
|
||||
- | Integer value (default: ``1``)
|
||||
| Higher values increase logging detail
|
||||
|
||||
* - | ``NCCL_DEBUG_SUBSYS``
|
||||
| Controls which subsystems generate debug output.
|
||||
- | Comma-separated list of subsystems (e.g., ``INIT,COLL``)
|
||||
| Prefix with ``^`` to invert selection
|
||||
|
||||
Algorithm and protocol control
|
||||
==============================
|
||||
|
||||
The algorithm and protocol control environment variables for RCCL are
|
||||
collected in the following table.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 70,30
|
||||
|
||||
* - **Environment variable**
|
||||
- **Value**
|
||||
|
||||
* - | ``NCCL_ALGO``
|
||||
| Forces specific algorithm selection for collectives.
|
||||
- | Algorithm name string
|
||||
| Used to override automatic algorithm selection
|
||||
|
||||
* - | ``NCCL_PROTO``
|
||||
| Forces specific protocol selection for communication.
|
||||
- | Protocol name string
|
||||
| Used to override automatic protocol selection
|
||||
|
||||
Network and topology
|
||||
====================
|
||||
|
||||
The network and topology environment variables for RCCL are collected
|
||||
in the following table.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 70,30
|
||||
|
||||
* - **Environment variable**
|
||||
- **Value**
|
||||
|
||||
* - | ``NCCL_IB_HCA``
|
||||
| Specifies InfiniBand device:port to use.
|
||||
- | Device specification string
|
||||
| Prefix with ``^`` for exclusion, ``=`` for exact match
|
||||
|
||||
* - | ``NCCL_IB_GID_INDEX``
|
||||
| Defines the Global ID index used in RoCE mode.
|
||||
- | Integer value (default: ``-1``)
|
||||
| See InfiniBand ``show_gids`` command for valid values
|
||||
|
||||
* - | ``NCCL_SOCKET_IFNAME``
|
||||
| Specifies which IP interfaces to use for communication.
|
||||
- | Interface prefix string or list
|
||||
| Multiple prefixes separated by ``,``
|
||||
| Prefix with ``^`` for exclusion, ``=`` for exact match
|
||||
| Example: ``eth`` (all eth interfaces), ``=eth0`` (exact match)
|
||||
|
||||
* - | ``NCCL_SOCKET_FAMILY``
|
||||
| Forces IPv4/IPv6 interface selection.
|
||||
- | ``AF_INET``: Force IPv4
|
||||
| ``AF_INET6``: Force IPv6
|
||||
| Unset: Use first available
|
||||
|
||||
* - | ``NCCL_NET_MERGE_LEVEL``
|
||||
| Controls network device merging behavior.
|
||||
- | Integer value specifying merge level
|
||||
| Default: ``PATH_PORT``
|
||||
|
||||
* - | ``NCCL_NET_FORCE_MERGE``
|
||||
| Forces merging of network devices.
|
||||
- | String specifying forced merge configuration
|
||||
|
||||
* - | ``NCCL_RINGS``
|
||||
| Defines custom ring topology.
|
||||
- | Ring topology specification string
|
||||
| Overrides automatic topology detection
|
||||
|
||||
* - | ``RCCL_TREES``
|
||||
| Defines custom tree topology.
|
||||
- | Tree topology specification string
|
||||
| Alternative to ring topology
|
||||
|
||||
* - | ``NCCL_RINGS_REMAP``
|
||||
| Controls ring remapping for specific topologies.
|
||||
- | Remapping specification string
|
||||
| Used with Rome 4P2H topology
|
||||
|
||||
Development and testing (advanced)
|
||||
==================================
|
||||
|
||||
The development and testing environment variables for RCCL are
|
||||
collected in the following table. These variables are primarily
|
||||
intended for debugging and development purposes.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 70,30
|
||||
|
||||
* - **Environment variable**
|
||||
- **Value**
|
||||
|
||||
* - | ``CUDA_LAUNCH_BLOCKING``
|
||||
| Controls CUDA kernel launch blocking behavior.
|
||||
- | ``0``: Non-blocking launches
|
||||
| ``1`` or non-zero: Blocking launches
|
||||
|
||||
* - | ``NCCL_COMM_ID``
|
||||
| Enables multi-process mode in test applications.
|
||||
- | Any non-empty value enables multi-process mode
|
||||
| Used with test executables for distributed testing
|
||||
@@ -0,0 +1,114 @@
|
||||
.. meta::
|
||||
:description: RCCL is a stand-alone library that provides multi-GPU and multi-node collective communication primitives optimized for AMD GPUs
|
||||
:keywords: RCCL, ROCm, library, API
|
||||
|
||||
.. _library-specification:
|
||||
|
||||
============================
|
||||
RCCL library specification
|
||||
============================
|
||||
|
||||
This document provides details of the API library.
|
||||
|
||||
Communicator functions
|
||||
----------------------
|
||||
|
||||
.. doxygenfunction:: ncclGetUniqueId
|
||||
|
||||
.. doxygenfunction:: ncclCommInitRank
|
||||
|
||||
.. doxygenfunction:: ncclCommInitAll
|
||||
|
||||
.. doxygenfunction:: ncclCommDestroy
|
||||
|
||||
.. doxygenfunction:: ncclCommAbort
|
||||
|
||||
.. doxygenfunction:: ncclCommCount
|
||||
|
||||
.. doxygenfunction:: ncclCommCuDevice
|
||||
|
||||
.. doxygenfunction:: ncclCommUserRank
|
||||
|
||||
Collective communication operations
|
||||
-----------------------------------
|
||||
|
||||
Collective communication operations must be called separately for each communicator in a communicator clique.
|
||||
|
||||
They return when operations have been enqueued on the hipstream.
|
||||
|
||||
Since they may perform inter-CPU synchronization, each call has to be done from a different thread or process, or need to use Group Semantics (see below).
|
||||
|
||||
.. doxygenfunction:: ncclReduce
|
||||
|
||||
.. doxygenfunction:: ncclBcast
|
||||
|
||||
.. doxygenfunction:: ncclBroadcast
|
||||
|
||||
.. doxygenfunction:: ncclAllReduce
|
||||
|
||||
.. doxygenfunction:: ncclReduceScatter
|
||||
|
||||
.. doxygenfunction:: ncclAllGather
|
||||
|
||||
.. doxygenfunction:: ncclSend
|
||||
|
||||
.. doxygenfunction:: ncclRecv
|
||||
|
||||
.. doxygenfunction:: ncclGather
|
||||
|
||||
.. doxygenfunction:: ncclScatter
|
||||
|
||||
.. doxygenfunction:: ncclAllToAll
|
||||
|
||||
Group semantics
|
||||
---------------
|
||||
When managing multiple GPUs from a single thread, and since NCCL collective
|
||||
calls may perform inter-CPU synchronization, we need to "group" calls for
|
||||
different ranks/devices into a single call.
|
||||
|
||||
Grouping NCCL calls as being part of the same collective operation is done
|
||||
using ncclGroupStart and ncclGroupEnd. ncclGroupStart will enqueue all
|
||||
collective calls until the ncclGroupEnd call, which will wait for all calls
|
||||
to be complete. Note that for collective communication, ncclGroupEnd only
|
||||
guarantees that the operations are enqueued on the streams, not that
|
||||
the operation is effectively done.
|
||||
|
||||
Both collective communication and ncclCommInitRank can be used in conjunction
|
||||
of ncclGroupStart/ncclGroupEnd.
|
||||
|
||||
.. doxygenfunction:: ncclGroupStart
|
||||
|
||||
.. doxygenfunction:: ncclGroupEnd
|
||||
|
||||
Library functions
|
||||
-----------------
|
||||
|
||||
.. doxygenfunction:: ncclGetVersion
|
||||
|
||||
.. doxygenfunction:: ncclGetErrorString
|
||||
|
||||
Types
|
||||
-----
|
||||
|
||||
There are few data structures that are internal to the library. The pointer types to these
|
||||
structures are given below. The user would need to use these types to create handles and pass them
|
||||
between different library functions.
|
||||
|
||||
.. doxygentypedef:: ncclComm_t
|
||||
|
||||
.. doxygenstruct:: ncclUniqueId
|
||||
|
||||
|
||||
|
||||
Enumerations
|
||||
------------
|
||||
|
||||
This section provides all the enumerations used.
|
||||
|
||||
.. doxygenenum:: ncclResult_t
|
||||
|
||||
.. doxygenenum:: ncclRedOp_t
|
||||
|
||||
.. _rccl-supported-data-types:
|
||||
|
||||
.. doxygenenum:: ncclDataType_t
|
||||
Yeni konuda referans
Bir kullanıcı engelle