From 531476dacf01396c870dc75af863b8dd85b286de Mon Sep 17 00:00:00 2001 From: Jeffrey Novotny Date: Fri, 6 Dec 2024 13:25:58 -0500 Subject: [PATCH] Add RCCL debugging guide (#1420) * Add RCCL debugging guide * Changes from external review * More edits from internal review * Additional edits * Minor correction * More changes after external review * Integrate index and ToC changes with incoming merge changes * Integrate feedback from management review * Minor edits from the internal review [ROCm/rccl commit: 6d34fb76321600d5693b24f1edc875605c5cc638] --- .../rccl/docs/how-to/troubleshooting-rccl.rst | 249 ++++++++++++++++++ projects/rccl/docs/index.rst | 1 + projects/rccl/docs/sphinx/_toc.yml.in | 2 + 3 files changed, 252 insertions(+) create mode 100644 projects/rccl/docs/how-to/troubleshooting-rccl.rst diff --git a/projects/rccl/docs/how-to/troubleshooting-rccl.rst b/projects/rccl/docs/how-to/troubleshooting-rccl.rst new file mode 100644 index 0000000000..c4168da258 --- /dev/null +++ b/projects/rccl/docs/how-to/troubleshooting-rccl.rst @@ -0,0 +1,249 @@ +.. meta:: + :description: A guide to troubleshooting the RCCL library of multi-GPU and multi-node collective communication primitives optimized for AMD GPUs + :keywords: RCCL, ROCm, library, API, debug + +.. _troubleshooting-rccl: + +********************* +Troubleshooting RCCL +********************* + +This topic explains the steps to troubleshoot functional and performance issues with RCCL. +While debugging, collect the output from the commands in this guide. This data +can be used as supporting information when submitting an issue report to AMD. + +.. _debugging-system-info: + +Collecting system information +============================= + +Collect this information about the ROCm version, GPU/accelerator, platform, and configuration. + +* Verify the ROCm version. This might be a release version or a + mainline or staging version. Use this command to display the version: + + .. code:: shell + + cat /opt/rocm/.info/version + + Run the following command and collect the output: + + .. code:: shell + + rocm_agent_enumerator + + Also, collect the name of the GPU or accelerator: + + .. code:: shell + + rocminfo + +* Run these ``rocm-smi`` commands to display the system topology. + + .. code:: shell + + rocm-smi + rocm-smi --showtopo + rocm-smi --showdriverversion + +* Determine the values of the ``PATH`` and ``LD_LIBRARY_PATH`` environment variables. + + .. code:: shell + + echo $PATH + echo $LD_LIBRARY_PATH + +* Collect the HIP configuration. + + .. code:: shell + + /opt/rocm/bin/hipconfig --full + +* Verify the network settings and setup. Use the ``ibv_devinfo`` command + to display information about the available RDMA devices and determine + whether they are installed and functioning properly. Run ``rdma link`` + to print a summary of the network links. + + .. code:: shell + + ibv_devinfo + rdma link + +Isolating the issue +------------------- + +The problem might be a general issue or specific to the architecture or system. +To narrow down the issue, collect information about the GPU or accelerator and other +details about the platform and system. Some issues to consider include: + +* Is ROCm running on: + + * A bare-metal setup + * In a Docker container (determine the name of the Docker image) + * In an SR-IOV virtualized + * Some combination of these configurations + +* Is the problem only seen on a specific GPU architecture? +* Is it only seen on a specific system type? +* Is it happening on a single node or multinode setup? +* Use the following troubleshooting techniques to attempt to isolate the issue. + + * Build or run the develop branch version of RCCL and see if the problem persists. + * Try an earlier RCCL version (minor or major). + * If you recently changed the ROCm runtime configuration, KFD/driver, or compiler, + rerun the test with the previous configuration. + +.. _collecting-rccl-info: + +Collecting RCCL information +============================= + +Collect the following information about the RCCL installation and configuration. + +* Run the ``ldd`` command to list any dynamic dependencies for RCCL. + + .. code:: shell + + ldd + +* Determine the RCCL version. This might be the pre-packaged component in + ``/opt/rocm/lib`` or a version that was built from source. To verify the RCCL version, + enter the following command, then run either rccl-tests or an e2e application. + + .. code:: shell + + export NCCL_DEBUG=VERSION + +* Run rccl-tests and collect the results. For information on how to build and run rccl-tests, see the + `rccl-tests GitHub `_. + +* Collect the RCCL logging information. Enable the debug logs, + then run rccl-tests or any e2e workload to collect the logs. Use the + following command to enable the logs. + + .. code:: shell + + export NCCL_DEBUG=INFO + +.. _use-rccl-replayer: + +Using the RCCL Replayer +------------------------ + +The RCCL Replayer is a debugging tool designed to analyze and replay the collective logs obtained from RCCL runs. +It can be helpful when trying to reproduce problems, because it uses dummy data and doesn't have any dependencies +on non-RCCL calls. For more information, +see `RCCL Replayer GitHub documentation `_. + +You must build the RCCL Replayer before you can use it. To build it, run these commands. Ensure ``MPI_DIR`` is set to +the path where MPI is installed. + +.. code:: shell + + cd rccl/tools/rccl_replayer + MPI_DIR=/path/to/mpi make + +To use the RCCL Replayer, follow these steps: + +#. Collect the per-rank logs from the RCCL run by adding the following environment variables. + This prevents any race conditions that might cause ranks to interrupt the output from other ranks. + + .. code:: shell + + NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=COLL NCCL_DEBUG_FILE=some_name_here.%h.%p.log + +#. Combine all the logs into a single file. This will become the input to the RCCL Replayer. + + .. code:: shell + + cat some_name_here_*.log > some_name_here.log + +#. Run the RCCL Replayer using the following command. Replace ```` with the number of MPI processes to + run, ```` with the path to the collective log file generated during + the RCCL runs, and ```` with the number of GPUs per MPI rank used in the application. + + .. code:: shell + + mpirun -np ./rcclReplayer + + In a multi-node application environment, you can replay the collective logs on multiple nodes + using the following command: + + .. code:: shell + + mpirun --hostfile -np ./rcclReplayer + + .. note:: + + Depending on the MPI library you're using, you might need to modify the ``mpirun`` command. + +.. _analyze-performance-info: + +Analyzing performance issues +============================= + +If the issues involve performance issues in an e2e workload, try the following +microbenchmarks and collect the results. Follow the instructions in the subsequent sections +to run these benchmarks and provide the results to the support team. + +* TransferBench +* RCCL Unit Tests +* rccl-tests + +Collect the TransferBench data +--------------------------------- + +TransferBench allows you to benchmark simultaneous copies between +user-specified devices. For more information, +see the :doc:`TransferBench documentation `. + +To collect the TransferBench data, follow these steps: + +#. Clone the TransferBench Git repository. + + .. code:: shell + + git clone https://github.com/ROCm/TransferBench.git + +#. Change to the new directory and build the component. + + .. code:: shell + + cd TransferBench + make + +#. Run the TransferBench utility with the following parameters and save the results. + + .. code:: shell + + USE_FINE_GRAIN=1 GFX_UNROLL=2 ./TransferBench a2a 64M 8 + +Collect the RCCL microbenchmark data +------------------------------------- + +To use the RCCL tests to collect the RCCL benchmark data, follow these steps: + +#. Disable NUMA auto-balancing using the following command: + + .. code:: shell + + sudo sysctl kernel.numa_balancing=0 + + Run the following command to verify the setting. The expected output is ``0``. + + .. code:: shell + + cat /proc/sys/kernel/numa_balancing + +#. Build MPI, RCCL, and rccl-tests. To download and install MPI, see either + `OpenMPI `_ or `MPICH `_. + To learn how to build and run rccl-tests, see the `rccl-tests GitHub `_. + +#. Run rccl-tests with MPI and collect the performance numbers. + +RCCL and NCCL comparisons +============================= + +If you are also using NVIDIA hardware or NCCL and notice a performance gap between the two systems, +collect the system and performance data on the NVIDIA platform. +Provide both sets of data to the support team. diff --git a/projects/rccl/docs/index.rst b/projects/rccl/docs/index.rst index f679db3033..ba3950383f 100644 --- a/projects/rccl/docs/index.rst +++ b/projects/rccl/docs/index.rst @@ -27,6 +27,7 @@ The RCCL public repository is located at ``_. .. grid-item-card:: How to * :doc:`Using the NCCL Net plugin <./how-to/using-nccl>` + * :doc:`Troubleshoot RCCL <./how-to/troubleshooting-rccl>` * :doc:`RCCL usage tips <./how-to/rccl-usage-tips>` diff --git a/projects/rccl/docs/sphinx/_toc.yml.in b/projects/rccl/docs/sphinx/_toc.yml.in index e95be1fb3d..5ced6215f5 100644 --- a/projects/rccl/docs/sphinx/_toc.yml.in +++ b/projects/rccl/docs/sphinx/_toc.yml.in @@ -18,6 +18,8 @@ subtrees: entries: - file: how-to/using-nccl title: Using the NCCL Net plugin + - file: how-to/troubleshooting-rccl + title: Troubleshoot RCCL - file: how-to/rccl-usage-tips - caption: Examples