Wenkai Du
104 lines
2.4 KiB
ReStructuredText
104 lines
2.4 KiB
ReStructuredText
.. toctree::
|
|
:maxdepth: 4
|
|
:caption: Contents:
|
|
|
|
===
|
|
API
|
|
===
|
|
|
|
This section provides details of the library API
|
|
|
|
Communicator Functions
|
|
----------------------
|
|
|
|
.. doxygenfunction:: ncclGetUniqueId
|
|
|
|
.. doxygenfunction:: ncclCommInitRank
|
|
|
|
.. doxygenfunction:: ncclCommInitAll
|
|
|
|
.. doxygenfunction:: ncclCommDestroy
|
|
|
|
.. doxygenfunction:: ncclCommCount
|
|
|
|
.. doxygenfunction:: ncclCommCuDevice
|
|
|
|
.. doxygenfunction:: ncclCommUserRank
|
|
|
|
Collection Communication Operations
|
|
-----------------------------------
|
|
|
|
Collective communication operations must be called separately for each communicator in a communicator clique.
|
|
|
|
They return when operations have been enqueued on the hipstream.
|
|
|
|
Since they may perform inter-CPU synchronization, each call has to be done from a different thread or process, or need to use Group Semantics (see below).
|
|
|
|
.. doxygenfunction:: ncclReduce
|
|
|
|
.. doxygenfunction:: ncclBcast
|
|
|
|
.. doxygenfunction:: ncclBroadcast
|
|
|
|
.. doxygenfunction:: ncclAllReduce
|
|
|
|
.. doxygenfunction:: ncclReduceScatter
|
|
|
|
.. doxygenfunction:: ncclAllGather
|
|
|
|
|
|
Group Semantics
|
|
---------------
|
|
When managing multiple GPUs from a single thread, and since NCCL collective
|
|
calls may perform inter-CPU synchronization, we need to "group" calls for
|
|
different ranks/devices into a single call.
|
|
|
|
Grouping NCCL calls as being part of the same collective operation is done
|
|
using ncclGroupStart and ncclGroupEnd. ncclGroupStart will enqueue all
|
|
collective calls until the ncclGroupEnd call, which will wait for all calls
|
|
to be complete. Note that for collective communication, ncclGroupEnd only
|
|
guarantees that the operations are enqueued on the streams, not that
|
|
the operation is effectively done.
|
|
|
|
Both collective communication and ncclCommInitRank can be used in conjunction
|
|
of ncclGroupStart/ncclGroupEnd.
|
|
|
|
.. doxygenfunction:: ncclGroupStart
|
|
|
|
.. doxygenfunction:: ncclGroupEnd
|
|
|
|
Library Functions
|
|
-----------------
|
|
|
|
.. doxygenfunction:: ncclGetVersion
|
|
|
|
.. doxygenfunction:: ncclGetErrorString
|
|
|
|
Types
|
|
-----
|
|
|
|
There are few data structures that are internal to the library. The pointer types to these
|
|
structures are given below. The user would need to use these types to create handles and pass them
|
|
between different library functions.
|
|
|
|
.. doxygentypedef:: ncclComm_t
|
|
|
|
.. doxygenstruct:: ncclUniqueId
|
|
|
|
|
|
|
|
Enumerations
|
|
------------
|
|
|
|
This section provides all the enumerations used.
|
|
|
|
.. doxygenenum:: ncclResult_t
|
|
|
|
.. doxygenenum:: ncclRedOp_t
|
|
|
|
.. doxygenenum:: ncclDataType_t
|
|
|
|
|
|
|
|
|