diff --git a/LICENSE.txt b/LICENSE.txt index 88889d2f9e..2eebfdb553 100644 --- a/LICENSE.txt +++ b/LICENSE.txt @@ -7,39 +7,40 @@ Copyright (c) 2015-2020, NVIDIA CORPORATION. All rights reserved. Modifications Copyright (c) 2019-2024 Advanced Micro Devices, Inc. All rights reserved. Modifications Copyright (c) Microsoft Corporation. Licensed under the MIT License. - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - * Neither the name of NVIDIA CORPORATION, Lawrence Berkeley National - Laboratory, the U.S. Department of Energy, nor the names of their - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY - EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR - PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR - CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, - EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, - PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR - PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY - OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +* Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +* Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. +* Neither the name of NVIDIA CORPORATION, Lawrence Berkeley National + Laboratory, the U.S. Department of Energy, nor the names of their + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. - The U.S. Department of Energy funded the development of this software - under subcontract 7078610 with Lawrence Berkeley National Laboratory. +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ''AS IS'' AND ANY +EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY +OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +The U.S. Department of Energy funded the development of this software +under subcontract 7078610 with Lawrence Berkeley National Laboratory. This code also includes files from the NVIDIA Tools Extension SDK project. See: - https://github.com/NVIDIA/NVTX +https://github.com/NVIDIA/NVTX for more information and license details. diff --git a/docs/attributions.rst b/docs/attributions.rst index ade22bdfd6..5d763c2d95 100644 --- a/docs/attributions.rst +++ b/docs/attributions.rst @@ -17,17 +17,17 @@ Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - - Redistributions of source code must retain the above copyright +- Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. - - Redistributions in binary form must reproduce the above copyright +- Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. - - Neither the name of NVIDIA CORPORATION, Lawrence Berkeley National +- Neither the name of NVIDIA CORPORATION, Lawrence Berkeley National Laboratory, the U.S. Department of Energy, nor the names of their contributors may be used to endorse or promote products derived from this software without specific prior written permission. -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR diff --git a/docs/how-to/using-nccl.rst b/docs/how-to/using-nccl.rst index 34b641bc7d..100d035fda 100644 --- a/docs/how-to/using-nccl.rst +++ b/docs/how-to/using-nccl.rst @@ -1,77 +1,76 @@ .. meta:: - :description: RCCL is a stand-alone library that provides multi-GPU and multi-node collective communication primitives optimized for AMD GPUs - :keywords: RCCL, ROCm, library, API + :description: How to use the NCCL Net API + :keywords: RCCL, ROCm, library, API, NCCL Net, plugin .. _using-nccl: -********************* -Using NCCL Net Plugin -********************* +***************************** +Using the NCCL Net plugin API +***************************** -NCCL provides a way to use external plugins to let NCCL run on many network types. This -topic describes the NCCL Net plugin API and how to implement a network plugin for NCCL. +NCCL provides a way to use external plugins to let NCCL run on many network types. This +topic describes the NCCL Net plugin API and explains how to implement a network plugin for NCCL. -Plugins implement the NCCL network API, and decouple NCCL binary builds which are built against a -particular version of the GPU stack (i.e. CUDA) from the network code which is built against a -particular version of the networking stack. That way, you can easily integrate any CUDA version +Plugins implement the NCCL network API and decouple NCCL binary builds, which are built against a +particular version of the GPU stack (such as NVIDIA CUDA), from the network code, which is built against a +particular version of the networking stack. Using this method, you can easily integrate any CUDA version with any network stack version. -NCCL network plugins come as a shared library called ``libnccl-net.so``. That shared library -contains one or more implementations of the NCCL NET API, in the form of versioned structs, -filled with pointers to all required functions. +NCCL network plugins are packaged as a shared library called ``libnccl-net.so``. The shared library +contains one or more implementations of the NCCL Net API in the form of versioned structs, +which are filled with pointers to all required functions. Plugin architecture =================== -When NCCL is initialized, it will look for a ``libnccl-net.so`` library and dynamically load it, -then look for symbols inside the library. +When NCCL is initialized, it searches for a ``libnccl-net.so`` library and dynamically loads it, +then searches for symbols inside the library. -The ``NCCL_NET_PLUGIN`` environment variable allows multiple plugins to coexist. If set, NCCL -will look for a library with a name of ``libnccl-net-${NCCL_NET_PLUGIN}.so``. It is therefore -advised to name the library following that pattern, with a symlink pointing ``libnccl-net.so`` -to ``libnccl-net-${NCCL_NET_PLUGIN}.so``. That way, if there are multiple plugins in the path, -setting ``NCCL_NET_PLUGIN`` will allow users to select the right plugin. +The ``NCCL_NET_PLUGIN`` environment variable allows multiple plugins to coexist. If it's set, NCCL +looks for a library named ``libnccl-net-${NCCL_NET_PLUGIN}.so``. It is therefore +recommended that you name the library according to that pattern, with a symlink pointing from ``libnccl-net.so`` +to ``libnccl-net-${NCCL_NET_PLUGIN}.so``. This lets users select the correct plugin +if there are multiple plugins in the path. Struct versioning ----------------- -Once a library is found, NCCL will look for a symbol named ``ncclNet_vX``, with ``X`` increasing -over time. The versioning ensures that the plugin and the NCCL core are compatible. +After a library is found, NCCL looks for a symbol named ``ncclNet_vX``, with ``X`` increasing +over time. This versioning pattern ensures that the plugin and the NCCL core are compatible. -Plugins are encouraged to provide multiple of those symbols, implementing multiple versions -of the NCCL NET API, so that the same plugin can be compiled and support a wide range of NCCL +Plugins are encouraged to provide a number of these symbols, implementing many versions +of the NCCL Net API. This is so the same plugin can be compiled for and support a wide range of NCCL versions. -Conversely, and to ease transition, NCCL can choose to support different plugin versions, looking -for the latest ncclNet struct version, but also looking for older ones so that older plugins -would still work. +Conversely, and to ease transition, NCCL can choose to support different plugin versions. It can look +for the latest ``ncclNet`` struct version but also search for older versions, so that older plugins +still work. -In-network collective operations (``collNet``) +In-network collective operations (collNet) ---------------------------------------------- -Additionally to the ``ncclNet`` structure, network plugins can provide a ``collNet`` structure which -implements in-network collective operations, if supported. That can be used by the NCCL ``collNet`` +In addition to the ``ncclNet`` structure, network plugins can provide a ``collNet`` structure which +implements any supported in-network collective operations. This is an optional +structure provided by the network plugin, +but its versioning is tied to the ``ncclNet`` structure and many functions are common between the two to +ease implementation. The ``collNet`` structure can be used by the NCCL ``collNet`` algorithm to accelerate inter-node reductions in allReduce. -The ``collNet`` struct is a different, optional struct provided by the network plugin, but its -versioning is tied to the ncclNet struct and many functions are common between the two to -ease the implementation. - -Headers management +Header management ------------------ -To help users build plugins effortlessly, plugins should copy the ``ncclNet_vX`` definitions -they support to their internal includes. An example is shown in ``ext-net/example/`` where we keep -all headers in the ``nccl/`` directory and provide thin layers to implement old versions on top +To help users effortlessly build plugins, plugins should copy the ``ncclNet_vX`` definitions +they support to their list of internal includes. An example is shown in ``ext-net/example/``, which stores +all headers in the ``nccl/`` directory and provides thin layers to implement old versions on top of newer ones. -The ``nccl/`` directory is populated with ``net_vX.h`` files extracting all relevant definitions -from old API versions. It also provides error codes in ``err.h``. +The ``nccl/`` directory is populated with ``net_vX.h`` files, which extract all relevant definitions +from the old API versions. It also provides error codes in ``err.h``. API (v6) ========= -Below is the main ``ncclNet_v6`` struct. Each function is explained in later sections. +Here is the main ``ncclNet_v6`` struct. Each function is explained in later sections. .. code:: shell @@ -122,160 +121,202 @@ Below is the main ``ncclNet_v6`` struct. Each function is explained in later sec ncclResult_t (*closeListen)(void* listenComm); } ncclNet_v6_t; - Error codes ----------- -All plugins functions use NCCL error codes as return value. `ncclSuccess` should be returned upon -success. +All plugins functions use NCCL error codes as their return value. ``ncclSuccess`` should be returned upon +success. Otherwise, plugins can return one of the following codes: -Otherwise, plugins can return one of the following: +* ``ncclSystemError`` is the most common error for network plugins. It should be returned when a call to the Linux kernel or a system library fails. This typically includes all network and hardware errors. +* ``ncclInternalError`` is returned when the NCCL core code is using the network plugin in an incorrect way, for example, allocating more requests than it should or passing an invalid argument in API calls. +* ``ncclInvalidUsage`` should be returned when the error is most likely due to user error. This can include misconfiguration, but also size mismatches. +* ``ncclInvalidArgument`` should not typically be used by plugins because arguments should be checked by the NCCL core layer. +* ``ncclUnhandledCudaError`` is returned when an error is received from NVIDIA CUDA. Network plugins should not need to rely on CUDA, so this error should not be common. -* ``ncclSystemError`` is the most common error for network plugins, when a call to the linux kernel or a system library fails. This typically includes all network/hardware errors. -* ``ncclInternalError`` is returned when the NCCL core code is using the network plugin in an incorrect way, for example allocating more requests than it should, or passing an invalid argument to calls. -* ``ncclInvalidUsage`` should be returned when the error is most likely a user error. This can include misconfiguration, but also sizes mismatch. -* ``ncclInvalidArgument`` should usually not be used by plugins since arguments should be checked by the NCCL core layer. -* ``ncclUnhandledCudaError`` is returned when an error comes from CUDA. Since network plugins should not need to rely on CUDA, this should not be common. +Operational overview +-------------------- -Operation overview ------------------- +NCCL first calls the ``init`` function, queries the number of network devices with the +``devices`` function, and retrieves the properties from each network device using ``getProperties``. -NCCL will call the ``init`` function first, then query the number of network devices with the -``devices`` function, getting each network device properties with ``getProperties``. +To establish a connection between two network devices, NCCL first calls ``listen`` on the +receiving side. It passes the returned handle to the sender side of the connection, and uses it to call ``connect``. +Finally, ``accept`` is called on the receiving side to finalize the establishment of the connection. -To establish a connection between two network devices, NCCL will first call ``listen`` on the -receiving side, pass the returned handle to the sender side of the connection, and call ``connect`` -with that handle. Finally, `accept` will be called on the receiving side to finalize the connection -establishment. +After the connection is established, communication is performed using the functions ``isend``, +``irecv``, and ``test``. Prior to calling ``isend`` or ``irecv``, NCCL calls the ``regMr`` function on +all buffers to allow RDMA NICs to prepare the buffers. ``deregMr`` is used to unregister buffers. -Once the connection is established, communication will be done using the functions ``isend``, -``irecv`` and ``test``. Prior to calling ``isend`` or ``irecv``, NCCL will call the ``regMr`` function on -all buffers to allow RDMA NICs to prepare buffers. ``deregMr`` will be used to unregister buffers. +In certain conditions, ``iflush`` is called after a ``receive`` call completes to allow the network +plugin to flush data and ensure the GPU processes the newly written data. -In certain conditions, ``iflush`` will be called after a receive calls completes to allow the network -plugin to flush data and ensure the GPU will observe the newly written data. - -To close the connections NCCL will call ``closeListen`` to close the object returned by ``listen``, -``closeSend`` to close the object returned by ``connect`` and ``closeRecv`` to close the object returned +To close the connections, NCCL calls ``closeListen`` to close the object returned by ``listen``, +``closeSend`` to close the object returned by ``connect``, and ``closeRecv`` to close the object returned by ``accept``. API Functions ------------- +The RCCL Tuner plugin API provides the following interface for initialization, connection management, and +communications. + Initialization ^^^^^^^^^^^^^^ -* ``name`` - The ``name`` field should point to a character string with the name of the network plugin. This will be used for all logging, especially when ``NCCL_DEBUG=INFO`` is set. +* ``name`` - The ``name`` field should point to a character string with the name of the network plugin. This name is used for all logging, especially when ``NCCL_DEBUG=INFO`` is set. -.. note:: - Setting ``NCCL_NET=`` will ensure a specific network implementation is used, with - a matching ``name``. This is not to be confused with ``NCCL_NET_PLUGIN`` which defines a suffix to the - ``libnccl-net.so`` library name to load. + .. note:: -* ``init`` - As soon as NCCL finds the plugin and the correct ncclNet symbol, it will call the ``init`` function. This will allow the plugin to discover network devices and make sure they are usable. If the ``init`` function does not return ``ncclSuccess``, then NCCL will not use the plugin and fall back on internal ones. + Setting ``NCCL_NET=`` ensures a specific network implementation is used, with + a matching ``name``. This is not to be confused with ``NCCL_NET_PLUGIN`` which defines a suffix for the + ``libnccl-net.so`` library name to load. - To allow the plugin logs to integrate into the NCCL logs seemlessly, NCCL provides a logging function to ``init``. This function is typically used to allow for ``INFO`` and ``WARN`` macros within the plugin code adding the following definitions: +* ``init`` - As soon as NCCL finds the plugin and the correct ``ncclNet`` symbol, it calls the ``init`` function. This allows the plugin to discover network devices and ensure they are usable. + If the ``init`` function does not return ``ncclSuccess``, then NCCL does not use the plugin and falls back to internal ones. -.. code:: shell + To allow the plugin logs to seamlessly integrate into the NCCL logs, NCCL provides a logging function to ``init``. This function is typically used to allow ``INFO`` and ``WARN`` macros within the plugin code by adding the following definitions: - #define WARN(...) logFunction(NCCL_LOG_WARN, NCCL_ALL, __FILE__, __LINE__, __VA_ARGS__) - #define INFO(FLAGS, ...) logFunction(NCCL_LOG_INFO, (FLAGS), __func__, __LINE__, __VA_ARGS__) + .. code:: shell -* ``devices`` - Once the plugin is initialized, NCCL will query the number of devices available. It should not be zero, otherwise NCCL initialization will fail. If no device is present or usable, the ``init`` function should not return ``ncclSuccess``. + #define WARN(...) logFunction(NCCL_LOG_WARN, NCCL_ALL, __FILE__, __LINE__, __VA_ARGS__) + #define INFO(FLAGS, ...) logFunction(NCCL_LOG_INFO, (FLAGS), __func__, __LINE__, __VA_ARGS__) -* ``getProperties`` - Right after getting the number of devices, NCCL will query properties for each available network device. These properties are critical when multiple adapters are present to ensure NCCL uses each adapter in the most optimized way. +* ``devices`` - After the plugin is initialized, NCCL queries the number of devices available. + This should not be zero. Otherwise, NCCL initialization will fail. If no device is present or usable, the ``init`` function should not return ``ncclSuccess``. -The ``name`` is only used for logging. +* ``getProperties`` - Right after retrieving the number of devices, NCCL queries the properties for each available network device. + These properties are necessary when multiple adapters are present to ensure NCCL uses each adapter in the optimal way. -The ``pciPath`` is the base for all topology detection and should point to the PCI device directory -in ``/sys``. This is typically the directory pointed by ``/sys/class/net/eth0/device`` or -``/sys/class/infiniband/mlx5_0/device``. If the network interface is virtual, then ``pciPath`` should -be ``NULL``. + * The ``name`` is only used for logging. -The ``guid`` field is used to determine when network adapters are connected to multiple PCI -endpoints. For normal cases, it can be set to the device number. If multiple network devices have -the same guid, then NCCL will consider the are sharing the same network port to the fabric, hence -it will not use the port multiple times. + * The ``pciPath`` is the base for all topology detection and should point to the PCI device directory + in ``/sys``. This is typically the directory pointed to by ``/sys/class/net/eth0/device`` or + ``/sys/class/infiniband/mlx5_0/device``. If the network interface is virtual, then ``pciPath`` should + be ``NULL``. -The ``ptrSupport`` field indicates whether or not CUDA pointers are supported. If so, it should be -set to ``NCCL_PTR_HOST``|``NCCL_PTR_CUDA``, otherwise it should be set to ``NCCL_PTR_HOST``. If the plugin -supports ``dmabuf``, it should set ``ptrSupport`` to ``NCCL_PTR_HOST``|``NCCL_PTR_CUDA``|``NCCL_PTR_DMABUF`` and -provide a ``regMrDmaBuf`` function. + * The ``guid`` field is used to determine whether network adapters are connected to multiple PCI + endpoints. For normal cases, this is set to the device number. If multiple network devices have + the same ``guid``, then NCCL understands them to be sharing the same network port to the fabric. In this case, + it will not use the port multiple times. -The ``speed`` field indicates the speed of the network port in Mbps (10^6 bits per second). This is important to ensure proper optimization of flows within the node. + * The ``ptrSupport`` field indicates whether or not CUDA pointers are supported. If so, it should be + set to ``NCCL_PTR_HOST|NCCL_PTR_CUDA``. Otherwise, it should be set to ``NCCL_PTR_HOST``. If the plugin + supports ``dmabuf``, it should set ``ptrSupport`` to ``NCCL_PTR_HOST|NCCL_PTR_CUDA|NCCL_PTR_DMABUF`` and + provide a ``regMrDmaBuf`` function. -The ``port`` field indicates the port number. This is important again for topology detection and flow optimization within the node when a NIC with a single PCI connection is connected to the fabric with multiple ports. + * The ``speed`` field indicates the speed of the network port in Mbps (10^6 bits per second). + This ensures proper optimization of flows within the node. -The ``latency`` field indicates the network latency in microseconds. This can be useful to improve the NCCL tuning and make sure NCCL switches from tree to ring at the right size. + * The ``port`` field indicates the port number. This is important for topology detection and + flow optimization within the node when a NIC with a single PCI connection is connected to the fabric through multiple ports. -The ``maxComms`` field indicates the maximum number of connections we can create. + * The ``latency`` field indicates the network latency in microseconds. This can be useful to + improve the NCCL tuning and ensure NCCL switches from tree to ring at the correct size. -The ``maxRecvs`` field indicates the maximum number for grouped receive operations (see grouped receive). + * The ``maxComms`` field indicates the maximum number of connections that can be created. + + * The ``maxRecvs`` field indicates the maximum number for grouped receive operations (see grouped receive). Connection establishment ^^^^^^^^^^^^^^^^^^^^^^^^ -Connections are used in an unidirectional manner. There is therefore a sender side and a receiver +Connections are used in an unidirectional manner, with a sender side and a receiver side. -* ``listen`` - To create a connection, NCCL will start by calling ``listen`` on the receiver side. This function takes a device number as input argument, and should return a local ``listenComm`` object, and a ``handle`` to pass to the other side, so that the sender side can connect to the receiver. +* ``listen`` - To create a connection, NCCL calls ``listen`` on the receiver side. + This function accepts a device number as an input argument and returns a local ``listenComm`` object and a ``handle`` + to pass to the other side of the connection, so that the sender can connect to the receiver. + The ``handle`` is a buffer of size ``NCCL_NET_HANDLE_MAXSIZE`` and is provided by NCCL. + This call should never block, but unlike ``connect`` and ``accept``, ``listenComm`` should never be ``NULL`` + if the call succeeds. - The ``handle`` is a buffer of size ``NCCL_NET_HANDLE_MAXSIZE`` and is provided by NCCL. +* ``connect`` - NCCL uses its bootstrap infrastructure to provide the ``handle`` to the sender side, + then calls ``connect`` on the sender side on a given device index ``dev`` and provides the ``handle``. + ``connect`` should not block either. Instead, it sets ``sendComm`` to ``NULL`` and returns ``ncclSuccess``. + In that case, NCCL will keep calling ``accept`` again until it succeeds. - This call should never block, but contrary to ``connect`` and ``accept``, ``listenComm`` should never be ``NULL`` if the call succeeds. +* ``accept`` - To finalize the connection, the receiver side calls ``accept`` on the ``listenComm`` object + previously returned by the ``listen`` call. If the sender did not connect yet, ``accept`` should not block. + It should return ``ncclSuccess``, setting ``recvComm`` to ``NULL``. NCCL will keep calling ``accept`` + again until it succeeds. -* ``connect`` - NCCL will use its bootstrap infrastructure to provide the ``handle`` to the sender side, then call ``connect`` on the sender side on a given device index ``dev``, providing the ``handle``. ``connect`` should not block either, and instead set ``sendComm`` to ``NULL`` and return ``ncclSuccess``. In that case, NCCL will call ``accept`` again until it succeeds. - -* ``accept`` - To finalize the connection, the receiver side will call ``accept`` on the ``listenComm`` returned by the ``listen`` call previously. If the sender did not connect yet, ``accept`` should not block. It should return ``ncclSuccess``, setting ``recvComm`` to ``NULL``. NCCL will call ``accept`` again until it succeeds. - -* ``closeListen``/``closeSend``/``closeRecv`` - Once a ``listenComm``/``sendComm``/``recvComm`` is no longer needed, NCCL will call ``closeListen``/``closeSend``/``closeRecv`` to free the associated resources. +* ``closeListen`` / ``closeSend`` / ``closeRecv`` - When a ``listenComm``, ``sendComm``, or ``recvComm`` object is no longer + needed, NCCL calls ``closeListen``, ``closeSend``, or ``closeRecv`` to free the associated resources. Communication ^^^^^^^^^^^^^ -Communication is done using asynchronous send and receive operations: ``isend``, ``irecv`` and ``test``. +Communication is handled using the asynchronous send and receive operations: ``isend``, ``irecv``, and ``test``. To support RDMA capabilities, buffer registration and flush functions are provided. -To keep track of asynchronous send, receive and flush operations, requests are returned to NCCL, -then queried with ``test``. Each ``sendComm`` or ``recvComm`` must be able to handle +To keep track of asynchronous send, receive, and flush operations, requests are returned to NCCL, +then queried using ``test``. Each ``sendComm`` or ``recvComm`` must be able to handle ``NCCL_NET_MAX_REQUESTS`` requests in parallel. .. note:: - That value should be multiplied by the multi-receive capability of the plugin for the sender - side, so that we can effectively have ``NCCL_NET_MAX_REQUESTS`` multi-receive operations happening - in parallel. So, if we have a `maxRecvs`value of 8 and ``NCCL_NET_MAX_REQUESTS`` is 8, then each - ``sendComm`` must be able to handle up to 8x8=64 concurrent ``isend`` operations. -* ``regMr`` - Prior to sending or receiving data, NCCL will call ``regMr`` with any buffers later used for communication. It will provide a ``sendComm`` or ``recvComm`` as ``comm`` argument, then the buffer pointer ``data``, ``size``, and ``type`` being either ``NCCL_PTR_HOST``, or ``NCCL_PTR_CUDA`` if the network supports CUDA pointers. + This value should be multiplied by the multi-receive capability of the plugin for the sender + side, so the plugin can effectively have ``NCCL_NET_MAX_REQUESTS`` multi-receive operations happening + in parallel. If ``maxRecvs`` is 8 and ``NCCL_NET_MAX_REQUESTS`` is 8, then each + ``sendComm`` must be able to handle up to 64 (8x8) concurrent ``isend`` operations. - The network plugin can use the output argument `mhandle` to keep any reference to that memory registration, as this ``mhandle`` will be passed back for all ``isend``, ``irecv``, ``iflush`` and ``deregMr`` calls. +* ``regMr`` - Prior to sending or receiving data, NCCL calls ``regMr`` with any buffers later used for communication. + It provides a ``sendComm`` or ``recvComm`` object for the ``comm`` argument, + the buffer pointer ``data``, the ``size``, and the ``type``. The type is either ``NCCL_PTR_HOST`` or ``NCCL_PTR_CUDA`` if + the network supports CUDA pointers. -* ``regMrDmaBuf`` - If the plugin has set the ``NCCL_PTR_DMABUF`` property in ``ptrSupport``, NCCL will use ``regMrDmaBuf`` instead of ``regMr``. If the property was not set, ``regMrDmaBuf`` can be set to ``NULL``. + The network plugin can use the output argument ``mhandle`` to store any reference to the memory registration, because + ``mhandle`` is returned for all ``isend``, ``irecv``, ``iflush``, and ``deregMr`` calls. -* ``deregMr`` - When buffers will no longer be used for communication, NCCL will call ``deregMr`` to let the plugin free resources. This function is used to deregister handles returned by both ``regMr`` and ``regMrDmaBuf``. +* ``regMrDmaBuf`` - If the plugin has set the ``NCCL_PTR_DMABUF`` property in ``ptrSupport``, + NCCL uses ``regMrDmaBuf`` instead of ``regMr``. If the property was not set, ``regMrDmaBuf`` can be set to ``NULL``. -* ``isend`` - Data will be sent through the connection using ``isend``, passing the ``sendComm`` previously created by ``connect``, and the buffer described by ``data``, ``size``, and ``mhandle``. A ``tag`` must be used if the network supports multi-receive operations (see ``irecv``) to distinguish between different sends matching the same multi-receive. Otherwise it can be set to 0. +* ``deregMr`` - When buffers are no longer used for communication, NCCL calls ``deregMr`` to let the plugin + free resources. This function is used to deregister handles returned by ``regMr`` and ``regMrDmaBuf``. - The ``isend`` operation returns a handle in the ``request`` argument for further calls to ``test``. If the ``isend`` operation cannot be initiated, ``request`` can be set to ``NULL`` and NCCL will call ``isend`` again later. +* ``isend`` - Data is sent through the connection using ``isend``, passing the ``sendComm`` object previously created + by ``connect``, the buffer described by ``data``, ``size``, and ``mhandle``. A ``tag`` must + be used if the network supports multi-receive operations (see ``irecv``) to distinguish between different send requests + matching the same multi-receive. Otherwise it can be set to ``0``. -* ``irecv`` - To receive data, NCCL will call ``irecv`` with the ``recvComm`` returned by ``accept``. The argument ``n`` will allow NCCL to perform a multi-receive, to allow grouping of multiple sends through a single network connection. Each buffer will be described by the ``data``, ``sizes``, and ``mhandles`` arrays. ``tags`` will specify a tag for each receive so that each of the ``n`` independent ``isend`` operations is received into the right buffer. + The ``isend`` operation returns a handle in the ``request`` argument for further calls to ``test``. + If the ``isend`` operation cannot be initiated, ``request`` is set to ``NULL``. NCCL will call ``isend`` again later. - If all receive operations can be initiated, ``irecv`` will return a handle in the ``request`` pointer, otherwise it will set it to ``NULL``. In the case of multi-receive, all ``n`` receive operations are handled by a single request handle. +* ``irecv`` - To receive data, NCCL calls ``irecv`` with the ``recvComm`` returned by ``accept``. + The argument ``n`` configures NCCL for multi-receive, to allow grouping of multiple sends + through a single network connection. Each buffer can be described by the ``data``, ``sizes``, and ``mhandles`` arrays. + ``tags`` specify a tag for each receive so that each of the ``n`` independent ``isend`` operations is received + into the right buffer. - The sizes provided to ``irecv`` can (and will) be larger than the size of the ``isend`` operation. However, if the receive size is smaller than the send size this is an error. + If all receive operations can be initiated, ``irecv`` returns a handle in the ``request`` pointer. Otherwise, + it sets the pointer to ``NULL``. In the case of multi-receive, all ``n`` receive operations are handled by a single request handle. -.. note:: - For a given connection, send/receive operations should always match in the order they were - posted. Tags provided for receive operations are only used to assign a given send operation to one - of the buffers of the first (multi-)receive in the queue, not to allow for out-of-order tag - matching on any receive operation posted. + The sizes provided to ``irecv`` can (and will) be larger than the size of the ``isend`` operation. + However, it is an error if the receive size is smaller than the send size. -* ``test`` - After an ``isend`` or ``irecv`` operation is initiated, NCCL will call `test` on the request handles until they complete. When that happens, ``done`` will be set to 1 and ``sizes`` will be set to the real size sent or received, the latter being potentially lower than the size passed to ``irecv``. + .. note:: - In the case of a multi-receive, all receives will be considered as done as a single operation (the goal being to allow aggregation), hence they share a single request and a single ``done`` status. However, they can have different sizes, so when ``done`` is non-zero, the ``sizes`` array should contain the ``n`` sizes corresponding to the buffers passed to ``irecv``. + For a given connection, send and receive operations should always match in the order they were + posted. Tags provided for receive operations are only used to assign a given send operation to one + of the buffers of the first (multi-)receive operation in the queue, not to allow for out-of-order tag + matching on any receive operation posted. - Once ``test`` returns 1 in ``done``, the request handle can be freed, meaning that NCCL will never call ``test`` again on that request (until it is reallocated by another call to ``isend`` or ``irecv``). +* ``test`` - After an ``isend`` or ``irecv`` operation is initiated, NCCL calls ``test`` on the request handles until + the operation completes. When that happens, ``done`` is set to ``1`` and ``sizes`` is set to the real size sent or received, + the latter could potentially be lower than the size passed to ``irecv``. -* ``iflush`` - After a receive operation completes, if the operation was targeting GPU memory and received a non-zero number of bytes, NCCL will call ``iflush`` to let the network flush any buffer and ensure the GPU can read it right after without seeing stale data. This flush operation is decoupled from the ``test`` code to improve latency of ``LL*`` protocols, as those are capable of determining when data is valid or not. + In the case of a multi-receive, all receives are considered as part of a single operation, the goal + being to allow aggregation. Therefore, they share a single request and a single ``done`` status. However, + they can have different sizes, so if ``done`` is non-zero, the ``sizes`` array should contain the ``n`` sizes + corresponding to the buffers passed to ``irecv``. - ``iflush`` returns a request which needs to be queried with ``test`` until it completes. + After ``test`` returns ``1`` in ``done``, the request handle can be freed. This means that NCCL will never + call ``test`` again on that request, unless it is reallocated by another call to ``isend`` or ``irecv``. + +* ``iflush`` - After a receive operation completes, if the operation was targeting GPU memory and received + a non-zero number of bytes, NCCL calls ``iflush``. This lets the network flush any buffer to ensure + the GPU can read it immediately without seeing stale data. This flush operation is decoupled from + the ``test`` code to improve the latency of ``LL*`` protocols, because those are capable of determining + when data is valid or not. + + ``iflush`` returns a request which must be queried using ``test`` until it completes. diff --git a/docs/how-to/using-rccl-tuner-plugin-api.rst b/docs/how-to/using-rccl-tuner-plugin-api.rst new file mode 100644 index 0000000000..0fcef43922 --- /dev/null +++ b/docs/how-to/using-rccl-tuner-plugin-api.rst @@ -0,0 +1,135 @@ +.. meta:: + :description: How to use the RCCL Tuner plugin API + :keywords: RCCL, ROCm, library, API, Tuner, plugin + +.. _using-rccl-tuner-plugin: + +******************************* +Using the RCCL Tuner plugin API +******************************* + +An external plugin enables users to hand-tailor the selection of an algorithm, +protocol, and number of channels (thread blocks) based on an input configuration specifying the +message size, number of nodes and GPUs, and link types (for instance, PCIe, XGMI, or NET). +One advantage of this plugin is that each user can create and maintain their own hand-tailored tuner +without relying on RCCL to develop and maintain it. This topic describes the API required to implement +an external tuner plugin for RCCL. + +The following usage notes are relevant when using the RCCL Tuner plugin API: + +* The API allows partial outputs: tuners can set only the algorithm and protocol and let RCCL set the remaining fields, + such as the number of channels. +* If ``getCollInfo()`` fails, RCCL uses its default internal mechanisms to determine the best collective configuration. +* ``getCollInfo`` is called for each collective invocation per communicator, so special care + must be taken to avoid introducing excessive latency. +* The supported RCCL algorithms are ``NCCL_ALGO_TREE``, and ``NCCL_ALGO_RING``. +* The supported RCCL protocols are ``NCCL_PROTO_SIMPLE``, ``NCCL_PROTO_LL`` and ``NCCL_PROTO_LL128``. + + * Until support is present for network collectives, use the example in the ``pluginGetCollInfo`` API implementation + to ignore other algorithms as follows: + + .. code-block:: cpp + + if ((a == NCCL_ALGO_COLLNET_DIRECT || a == NCCL_ALGO_COLLNET_CHAIN) && collNetSupport != 1) continue; + if ((a == NCCL_ALGO_NVLS || a == NCCL_ALGO_NVLS_TREE) && nvlsSupport != 1) continue; + if (a == NCCL_ALGO_NVLS && collNetSupport != 1) continue; + +.. note:: + + The `example plugin `_ + uses math models to approximate the bandwidth and latency of the available selection of algorithms and protocols + and select the one with the lowest calculated latency. It is customized for the AMD Instinct MI300 accelerators and RoCEv2 networks + on a limited number of nodes. This example, which is intended for demonstration purposes only, is not meant to be inclusive of all potential AMD GPUs and network configuration. + +API description +================ + +To build a custom tuner, implement the ``ncclTuner_v1_t`` structure. + +Structure: ncclTuner_v1_t +--------------------------- + +**Fields** + +* ``name`` + + * **Type**: ``const char*`` + * **Description**: The name of the tuner, which can be used for logging purposes when ``NCCL_DEBUG=info`` and ``NCCL_DEBUG_SUBSYS=tune`` are set. + +**Functions** + +* ``init`` (called upon communicator initialization with ``ncclCommInitRank``) + + Initializes the tuner states. Each communicator initializes its tuner. ``nNodes`` x ``nRanks`` = the total number of GPUs participating in the collective communication. + + * **Parameters**: + + * ``nRanks`` (``size_t``): The number of devices (GPUs). + * ``nNodes`` (``size_t``): The number of operating system nodes (physical nodes or VMs). + * ``logFunction`` (``ncclDebugLogger_t``): A log function for certain debugging info. + + * **Return**: + + * **Type**: ``ncclResult_t`` + * **Description**: The result of the initialization. + +* ``getCollInfo`` (called for each collective call per communicator) + + Retrieves information about the collective algorithm, protocol, and number of channels for the given input parameters. + + * **Parameters**: + + * ``collType`` (``ncclFunc_t``): The collective type, for example, ``allreduce``, ``allgather``, etc. + * ``nBytes`` (``size_t``): The size of the collective in bytes. + * ``collNetSupport`` (``int``): Whether ``collNet`` supports this type. + * ``nvlsSupport`` (``int``): Whether NVLink SHARP supports this type. + * ``numPipeOps`` (``int``): The number of operations in the group. + + * **Outputs**: + + * ``algorithm`` (``int*``): The selected algorithm to be used for the given collective. + * ``protocol`` (``int*``): The selected protocol to be used for the given collective. + * ``nChannels`` (``int*``): The number of channels (and SMs) to be used. + + * **Description**: + + If ``getCollInfo()`` does not return ``ncclSuccess``, RCCL falls back to its default tuning for the given collective. + The tuner is allowed to leave fields unset, in which case RCCL automatically sets those fields. + + * **Return**: + + * **Type**: ``ncclResult_t`` + * **Description**: The result of the operation. + +* ``destroy`` (called upon communicator finalization with ``ncclCommFinalize``) + + Terminates the plugin and cleans up any resources allocated by the tuner. + + * **Return**: + + * **Type**: ``ncclResult_t`` + * **Description**: The result of the cleanup process. + +Build and usage instructions +============================ + +To use the external plugin, implement the desired algorithm and protocol selection technique using the API described above. +As a reference, the `following example `_ is based on the +MI300 tuning table by default. + +Building and using the example libnccl-tuner.so file +----------------------------------------------------- + +#. Build the ``libnccl-tuner.so`` file following `the example Makefile `_. + + .. code-block:: shell + + cd $RCCL_HOME/ext-tuner/example/ + make + +#. Tell RCCL to use the custom ``libnccl-tuner.so`` file by setting the following environment variable + to the file path: + + .. code-block:: shell + + export NCCL_TUNER_PLUGIN=$RCCL_HOME/ext-tuner/example/libnccl-tuner.so diff --git a/docs/index.rst b/docs/index.rst index ba3950383f..97838d4641 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -26,6 +26,7 @@ The RCCL public repository is located at ``_. .. grid-item-card:: How to + * :doc:`Using the RCCL Tuner plugin <./how-to/using-rccl-tuner-plugin-api>` * :doc:`Using the NCCL Net plugin <./how-to/using-nccl>` * :doc:`Troubleshoot RCCL <./how-to/troubleshooting-rccl>` * :doc:`RCCL usage tips <./how-to/rccl-usage-tips>` diff --git a/docs/sphinx/_toc.yml.in b/docs/sphinx/_toc.yml.in index 5ced6215f5..09205f84ef 100644 --- a/docs/sphinx/_toc.yml.in +++ b/docs/sphinx/_toc.yml.in @@ -16,6 +16,8 @@ subtrees: - caption: How to entries: + - file: how-to/using-rccl-tuner-plugin-api + title: Using the RCCL Tuner plugin - file: how-to/using-nccl title: Using the NCCL Net plugin - file: how-to/troubleshooting-rccl