Profiler improvements
* Add events for CUDA kernel start and end.
* Allow network plugins to generate profiling events
* Enable profiling on a per-operation basis, rather than per-communicator.
* Add support for graph capturing.
Add implicit launch order
* Allow to prevent deadlocks when using multiple NCCL communicators per
device by implicitly ordering NCCL operations using the host program
order. Disabled by default, set NCCL_LAUNCH_ORDER_IMPLICIT=1 to enable.
* Add a complementary mechanism to detect host threads racing to launch
to the same device. Enabled by default, set NCCL_LAUNCH_RACE_FATAL=0 to
disable.
Optimize the PAT algorithm
* Separate the computation and execution of PAT steps on different warps,
allowing to run up to 16 PAT steps in parallel to significantly
accelerate PAT and reduce its linear part.
Add support for setting QoS per communicator
* Add a new trafficClass field to the communicator configuration, to
allow the application to select a particular traffic class for a
given communicator. The meaning of the traffic class is
network-specific and should be set in accordance with the network
configuration.
* For the IB/RoCE plugin, existing config variables such as NCCL_IB_SL
and NCCL_IB_TC take precedence.
Allow to enable GPU Direct RDMA specifically on C2C platforms
* Disabled by default, set NCCL_NET_GDR_C2C=1 to enable.
Do not disable user buffer registration unless PXN is really used
* Only disable UB when a communicator has more than one rank per
node on any node.
RAS subsystem improvements
* Report operation counts separately for each collective operation type.
* Provide details about missing communicator ranks and reliably
distinguish ranks that are no longer a given communicator's members
(now reported as NOCOMM) from those that failed to respond.
Add support for timestamps to NCCL diagnostic messages
* On by default for WARN messages; NCCL_DEBUG_TIMESTAMP_LEVELS can be
used to enable them for other debug levels as well.
* The format can be changed using the NCCL_DEBUG_TIMESTAMP_FORMAT config
variable.
Reduce the memory usage with NVLink SHARP (NVLS)
* Potentially save hundreds of MBs of device memory, considering the
multicast buffer size granularity separately from the address alignment.
Update performance tuning for recent Intel CPUs
* Improve algorithm/protocol selection on recent CPUs such as Emerald
Rapids and Sapphire Rapids.
Improve channel scheduling when mixing LL and Simple operations.
* Make LL operations account for 4x more traffic to ensure LL and simple
operations complete at the same time.
Refactor the plugin code
* Clean up and harmonize the support code across the network, tuner,
and profiler plugins.
Add support for comment lines (starting with #) in the nccl.conf file
* Issue #1540.
Make user buffer registration problems print an INFO instead of a WARN.
Drop support for network plugin interface version 5.
Fix a race condition with split-shared communicators
* NCCL could hang during connection setup if multiple communicators
were grouped together that share resources.
Fix a performance regression when using NCCL_CROSS_NIC=1
* NCCL would unnecessarily alternate rings, breaking the GPU-NIC
associations.
Make GID index detection code more resilient
* Dynamic GID detection code was giving up too soon if the
detected index was not available (e.g., wasn't mapped to the
container's sysfs).
* Issues #1538, #1573.
Fix a race condition with non-blocking operation
* Fix issue when creating a non-blocking communicator after a non-
blocking collective operation on another communicator.
Fix shared memory usage on recent Blackwell GPUs.
* Issues NVIDIA/nccl-tests#287, NVIDIA/nccl-tests#291, #1637.
Fix an error with NIC fusion and IB SHARP when recreating communicators
* Disable the unloading of network plugins
Make the auto-merge failures in the NIC fusion non-fatal
* This could happen when trying to merge IB and RoCE devices.
Fixes to ncclCommAbort
* Fix hangs due to the progress thread spinning indefinitely on the
network progress.
* Reduce the abort time by up to two orders of magnitude.
Fix a crash when libnccl.so was dynamically unloaded
* The RAS subsystem was missing a clean-up handler.
Fix a hang if the network plugin's test() call returns an error.
Fix a hang on heterogeneous architectures
* Ensure we harmonize the tuning to avoid different tuning choices,
causing a hang.
Fix double-free on failed ncclCommInitRank and ncclCommFinalize.
Fix a potential list traversal bug during a group launch of multiple
communicators
* Issue #1599.
Unify the handling of NCCL configuration variables
* Under rare circumstances, some variables specified in the config file
could be ignored.
[ROCm/rccl commit: f44ac759fe]
Rework core for NVIDIA Trusted Computing
* Compress work structs so that they are shared between channels
* Utilize the full amount of kernel argument space permitted (4k)
before resorting to work fifo.
* Rework the task preprocessing phase.
* Use a separate abortDevFlag which is kept in sync with abortFlag
using cudaMemcpy operations.
* Rename src/include/align.h to src/include/bitops.h
Add lazy connection establishment for collective operations
* Move buffer allocation and connection establishment to the first
collective operation using that algorithm.
* Accelerate init time and reduce memory usage.
* Avoid allocating NVLS buffers if all calls are registered.
* Compute algo/proto in ncclLaunchCollTasksInfo early on.
* Connect peers in ncclCollPreconnectFunc if not connected already.
* Also move shared buffer creation to the first send/recv call.
Accelerate intra-node NVLink detection
* Make each rank only detect NVLinks attached to its GPU.
* Fuse XMLs to reconstruct the full NVLink topology
Add init profiling to report time spend in different init phases.
* Report timings of bootstrap, allgather, search, connect, etc.
* Add new "PROFILE" category for NCCL_DEBUG_SUBSYS.
Add support for PCI p2p on split PCI switches
* Detect split PCI switches through a kernel module exposing
switch information.
* Update the topology XML and graph to add those inter-switch
connections.
Add cost estimation API
* Add a new ncclGroupEndSimulate primitive to return the estimated
time a group would take.
Net/IB: Add separate traffic class for fifo messages
* Add NCCL_IB_FIFO_TC to control the traffic class of fifo messages
independently from NCCL_IB_TC.
Merges PR #1194
Net/IB: Add support for IB router
* Use flid instead of lid if subnets do not match
* Warn if flid is 0
Optimizations and fixes for device network offload (unpack)
* Double the default number of channels
* Cache netDeviceType
* Fix save/increment head logic to enable Tree support.
Support ncclGroupStart/End for ncclCommAbort/Destroy
* Allow Abort/Destroy to be called within a group when managing
multiple GPUs with a single process.
Improve Tuner API
* Provide to the plugin the original cost table so that the plugin
can leave unknown or disabled algo/proto combinations untouched.
* Remove nvlsSupport and collnetSupport.
Do not print version to stdout when using a debug file
* Also print version from all processes with INFO debug level.
Fixes issue #1271
Fix clang warnings in NVTX headers
* Update NVTX headers to the latest version
Fixes issue #1270
Disable port fusion in heterogeneous systems
* Do not fuse ports if a mix of multi-port and single port are detected.
Fix NVLS graphs search for dual NICs.
* Fix NVLS graph search when we have more than one NIC per GPU.
Fix crash with collnetDirect
* Add separate graph search for collnetDirect, testing alltoall paths
and working similarly to the NVLS search.
Fix hang when nodes have different CPU types
* Add the CPU type to the rank peer info.
* Align all ranks on the CPU type after the first allgather.
* Only use the aligned CPU type for all tuning operations.
Fixes issue #1136
Fixes issue #1184
Fix performance of registered send/recv operations
* Allow for single full size operations
* Add INFO to confirm the registration of send/recv buffers.
Move all sync ops to finalize stage
* Ensure ncclCommDestroy is non-blocking if ncclCommFinalize has
been called.
Improve error reporting during SHM segment creation
Improve support of various compilers
Merges PR #1177
Merges PR #1228
Allow net and tuner plugins to be statically linked
* Search for ncclNet or ncclTuner symbols in the main binary.
Merges PR #979
Plugin examples includes cleanup
* Harmonize err.h and common.h usage.
* Add mixed plugin with both net and tuner.
[ROCm/rccl commit: 178b6b7590]
Add local user buffer registration for NVLink SHARP.
Add tuning plugin support.
Increase net API to v7 to allow for device-side packet reordering;
remove support for v4 plugins.
Add support for RoCE ECE.
Add support for C2C links.
Better detect SHM allocation failures to avoid crash with Bus Error.
Fix missing thread unlocks in bootstrap (Fixes#936).
Disable network flush by default on H100.
Move device code from src/collectives/device to src/device.
[ROCm/rccl commit: f9c3dc251e]
Fix data corruption with Tree/LL128 on systems with 1GPU:1NIC.
Fix hang with Collnet on bfloat16 on systems with less than one NIC
per GPU.
Fix long initialization time.
Fix data corruption with Collnet when mixing multi-process and
multi-GPU per process.
Fix crash when shared memory creation fails.
Fix Avg operation with Collnet/Chain.
Fix performance of alltoall at scale with more than one NIC per GPU.
Fix performance for DGX H800.
Fix race condition in connection progress causing a crash.
Fix network flush with Collnet.
Fix performance of aggregated allGather/reduceScatter operations.
Fix PXN operation when CUDA_VISIBLE_DEVICES is set.
Fix NVTX3 compilation issues on Debian 10.
[ROCm/rccl commit: ea38312273]
Add support for IB SHARP to NVLS (NVLink SHARP algorithm).
Add NVLS+Tree algorithm.
Add support for memory management using cuMem* functions.
Use all NICs for Send/Receive operations on systems with more than
one NIC per GPU (#804).
Add ncclCommSplit primitive, with resource sharing option in config.
Fix alltoallv hang (#788)
Increase number of channels on H100 when we're not limited by NVLink.
Improve error reporting in case of IB failure, printing local and
remote ID (#779).
Add build option to allow compilation against RDMA includes instead
of dynamically loading IB verbs symbols (#802).
Fix context creation for progress thread (#803).
NET/IB: add option to use multiple QPs in round-robin mode.
Fix tree performance issue when NVB is disabled on HCM topologies.
[ROCm/rccl commit: d97a32fac8]
* Add MSCCL support
* Add alignment and message size checking
* Fix nRanks checking, in-place and out-of-place tests and group call handling
* Fix hipGraph unit test
* Change MSCCL init warning to INFO
* Revise license info
[ROCm/rccl commit: adafc0f759]
Add support for improved fault tolerance: non-blocking mode, new
init function with config, and ncclCommFinalize function.
Reintroduce collnet+chain algorithm, alongside collnet+direct.
Add LL protocol for intra-node P2P (on by default) and network
communication (off by default).
Use network instead of shared memory when performance is better.
Fix: wait for CUDA graph destroy before destroying comm with linked
graph resources.
Remove aggressive polling during enqueue.
Fix DMABUF fallback on MOFED 5.4 and earlier.
[ROCm/rccl commit: c4e2aa6c79]
Optimize CUDA graph launch; avoid launching a CPU callback for
intra-node operations.
Simplify kernel common code to improve the latency of send/recv
operations.
Strengthen CUDA streams semantics.
Change NET API to v6, to add dmabuf support.
Add ncclGetLastError() function.
Add ncclRemoteError code and use it for remote network errors.
Support the use of a different NCCL_NET parameter per communicator.
Add support for SHM and P2P transfers using cudaMemcpy.
[ROCm/rccl commit: 19ab67d172]
Add LL128 Protocol.
Rewrite the topology detection and tree/ring creation (#179). Improve
tree performance by sending/receiving from different GPUs. Add
model-based tuning to switch between the different algorithms and
protocols.
Rework P2P/SHM detection in containers (#155, #248).
Detect duplicated devices and return an error (#231).
Add tuning for GCP
[ROCm/rccl commit: 299c554dcc]
Add support for inter-node communication using sockets and InfiniBand/RoCE.
Improve latency.
Add support for aggregation.
Improve LL/regular tuning.
Remove tests as those are now at github.com/nvidia/nccl-tests .
[ROCm/rccl commit: f93fe9bfd9]
BertanDogancay
Kamil Iskra
Sylvain Jeaugey
Wenkai Du
Ziyue Yang