* Increased max stack size to 640
* Added new binary for executing unit tests
Added new unit tests for argcheck.cc and alt_rsmi.cc files
Modified the method to execute unit tests to cover static methods
by using a bash script to convert static to non-static functions
and variables on the fly restricted to debug build type.
[ROCm/rccl commit: 275fdd43c1]
We add 3 different issue types issue/question/RFE and add some predefined
questions to speed up the debugging process.
We also add a custom action which will close all issues create mode than 6
months ago which have not been updated for more than a month.
[ROCm/rccl commit: bfedf2629e]
Improve support for DirectNIC (CX8)
* Add support for XDR speed detection.
* When DirectNIC is enabled, report only the RDMA interfaces.
Extend the P2C (PXN over C2C) support to send/receive operations.
Support compilation with GCC 14 (Issues #1743, #1751).
Fix the unloading of network plugins that also provide tuner capability.
Fix the change of the current device across the calls to ncclCommDestroy()
and ncclCommAbort().
A note for users on MNNVL systems: please ensure an adequate stack size for
NCCL threads. While the default Linux stack size limit of 8192 KB is known
to be sufficient, we've seen crashes if the limit is changed to
"unlimited", as it causes the glibc library to unexpectedly *decrease* the
stack size of NCCL's background threads to just 2048 KB. Use "ulimit -s"
in bash to print the current limit; if needed, reset it to 8192 KB using
"ulimit -s 8192" (one also needs to ensure that the new setting is
propagated to other nodes when launching a multi-node NCCL job).
[ROCm/rccl commit: 7c12c627c6]
* Adjust tuning factor impacting more than 2 nodes
* Scale max LL128 size for > 2 nodes
* Retune max LL128 range for N > 2
[ROCm/rccl commit: 058264b3f3]
* removing extra build time by removing the gfx11xx arch from using hip_fp8
---------
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>
[ROCm/rccl commit: 697bee4ee8]
If `find_package()` succeeds to find GTest and `INSTALL_DEPENDENCIES`
is set to OFF, `GTEST_BOTH_LIBRARIES` is not set and thus
`rccl-UnitTests` fails with trying to link unkown symbols.
[ROCm/rccl commit: dac0e528a0]
* Add support for extended fine-grained system memory pool
* Use hipHostRegisterUncached
* Add "sc0 sc1" flags for LL store on gfx950
* Update after HIP flag is changed to hipExtHostRegisterUncached
[ROCm/rccl commit: 4640ab19b3]
This feature tracks the proxy events and status of each send/recv op. ProxyTrace keeps a fixed number of active ops in host mem and dumps the status of each op when the program crashes or hangs.
[ROCm/rccl commit: 020dcf0a7c]
When the `roc-obj-ls` executable fails, it sometimes does not return. Since the `execute_process` command will wait until the executable finishes, this means that in some cases, the build will hang indefinitely. There is no error message, and no indication that anything is wrong. This commit fixes that by introducing timeouts into the code and better error reporting.
[ROCm/rccl commit: 2482d1475f]
* [Azure CI] rccl nightly pipeline that runs on slurm
- Login node will be set up as a self-hosted agent on Azure Pipelines.
- Login node will run this job nightly.
- Login node will checkout the latest develop source, and then run build and test through sbatch calls, and then waiting for the jobs to complete. When the jobs are complete, print out the logs.
[ROCm/rccl commit: 12315c259a]
Improvements for GB200 systems
* Optimize the network performance by alternating the direction of the
rings and the NIC to GPU assignment across communicators to limit
unnecessary sharing.
* Fix the detection of C2C links in case GPU Direct RDMA is disabled
between a GPU and a NIC.
* Fix PXN support on MNNVL systems, where NCCL would try (and fail) to
share regular host memory across multiple nodes.
* Fix P2C (PXN over C2C), which is now preferred over regular PXN. This
support is currently preliminary and is disabled by default; use
NCCL_PXN_C2C=1 to enable.
Further reduce the overheads of CUDA graph capturing, which increased in
NCCL 2.26.2 for large graphs.
Optimize the network performance on DGX B200 systems by adjusting the
bandwidths provided to the graph search algorithm.
Enable fp8 reductions in symmetric kernels on Blackwell with CUDA 12.8.
Restore the plugin name handling logic to make it possible to specify a
path to the plugin (Issue #1732).
Restore the ability to change NCCL_COLLNET_ENABLE during execution
(Issue #1741).
Add an example tuner plugin with CSV-based overrides.
Remove an x86 dependency from the example profiler.
[ROCm/rccl commit: 3ea7eedf3b]
Atul Kulkarni
Stephen Sachs
isaki001
Nilesh M Negi
Wenkai Du
dependabot[bot]
Nikhil-Nunna
Jobbins
Kamil Iskra
Bertan Dogancay
Mustafa Abduljabbar
mberenjk
Marius Brehler
Rakesh Roy
ryanhankins
Arm Patinyasakdikul
gilbertlee-amd
Dingming Wu
corey-derochie-amd
jonatluu
Grant Pinkert
BertanDogancay
Joseph Macaranas
Sarat Kamisetty