Fix data corruption with Tree/LL128 on systems with 1GPU:1NIC.
Fix hang with Collnet on bfloat16 on systems with less than one NIC
per GPU.
Fix long initialization time.
Fix data corruption with Collnet when mixing multi-process and
multi-GPU per process.
Fix crash when shared memory creation fails.
Fix Avg operation with Collnet/Chain.
Fix performance of alltoall at scale with more than one NIC per GPU.
Fix performance for DGX H800.
Fix race condition in connection progress causing a crash.
Fix network flush with Collnet.
Fix performance of aggregated allGather/reduceScatter operations.
Fix PXN operation when CUDA_VISIBLE_DEVICES is set.
Fix NVTX3 compilation issues on Debian 10.
[ROCm/rccl commit: ea38312273]
* Fix build on additional architectures
Instead of directly wrapping a platform-specific operation with a
preprocessor check against a gfx macro, it can be more flexible to
check a macro that can be overriden by the user. The gfx macro can then
just provide the default value for the macro, resulting in the same
default behaviour as if the gfx macro was checked directly but with
more control at build-time.
For example, to build rccl without using buffer_wbinvl1_vol on
gfx902, but still use the default on other archs, a user could
export CXXFLAGS='-Xarch_gfx902 -DRCCL_USE_WBINVL1_VOL=1' before
configuring the build. This flexibility isn't always necessary, but
it's nicer to have it and not need it than to need it and not have it.
* Define WARP_SIZE using warpSize builtin
[ROCm/rccl commit: b1a65afd58]
* add DMA_BUF support
* remove unused libraries in src/init.cc
* change NCCL_ALL to NCCL_INIT
* remove extra pointer functions in transport/net.cc
[ROCm/rccl commit: d52b6c0d24]
The default value of unroll factor is 2. Changing the unroll
factor to 4 provides better performance for most of the collectives.
[ROCm/rccl commit: 4d1cfb17c8]
Added a RCCL_MSCCL_ENABLE_DONE_EVENT env var, set it be 0 by default.
The env var is to control whether to use doneEvent when invoking MSCCL
kernels.
Skipping doneEvent would cause the firmware to skip L2 cache flush,
resulting in overall performance improvement.
[ROCm/rccl commit: 12dba425de]
* Revert "Ensure memory copy integrity during transport setup (#731)"
This reverts commit 0c0c35927a.
Add stream synchronization in ncclStrongStreamRelease.
* Use event record and wait
[ROCm/rccl commit: 4ca7742c61]
Add support for IB SHARP to NVLS (NVLink SHARP algorithm).
Add NVLS+Tree algorithm.
Add support for memory management using cuMem* functions.
Use all NICs for Send/Receive operations on systems with more than
one NIC per GPU (#804).
Add ncclCommSplit primitive, with resource sharing option in config.
Fix alltoallv hang (#788)
Increase number of channels on H100 when we're not limited by NVLink.
Improve error reporting in case of IB failure, printing local and
remote ID (#779).
Add build option to allow compilation against RDMA includes instead
of dynamically loading IB verbs symbols (#802).
Fix context creation for progress thread (#803).
NET/IB: add option to use multiple QPs in round-robin mode.
Fix tree performance issue when NVB is disabled on HCM topologies.
[ROCm/rccl commit: d97a32fac8]
* Fix XGMI detection
* Increase stack size
* Temporarily disable signal hangler in CI
[Process: 17281] Inside handler function signal: Segmentation fault (11)
BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0)
BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0)
[ROCm/rccl commit: 22b81fbaae]
Add new NVLS algorithm for allreduce using NVLink SHARP (intra-node only).
Add new config options: cgaClusterSize, minCTAs, maxCTAs, netName.
Enable LL128 when we use PXN to close rings.
NVTX3 includes update.
Fix crash when one CollNet (SHARP) rail fails to initialize.
[ROCm/rccl commit: 5d3ab08b69]
Add support for 400Gbit NDR network adapters (CX7)
Handle EINTR in socket poll() function
Add NCCL_PROGRESS_APPENDOP_FREQ to control op append overhead
Resource cleanup fixes
Fix double free in case of init failure
Fix crash in ncclCommAbort
Revert AMD speed commit
[ROCm/rccl commit: f3d5166783]
Wenkai Du
Wenkai Du
akolliasAMD
Dmitrii Gabor
arvindcheru
Bertan Dogancay
gilbertlee-amd
Nusrat Islam
Sylvain Jeaugey
Cory Bloor
Ziyue Yang
Wen-Heng (Jack) Chung
Wen-Heng (Jack) Chung
Kaiming Ouyang
Ziyue Yang