Sylvain Jeaugey
68b542363f
Add scalable init API * Add new ncclCommInitRankScalable to allow for passing multiple unique IDs to the init function. * Spreads the load onto multiple bootstrap roots, allowing for constant bootstrap time. * Requires multiple ranks to create a unique ID, and the CPU-side ID exchange code to call allgather[v] instead of broadcast. Accelerate init bootstrap operations * Reduce the number of calls to allgather. * Allow roots to reply early to ranks when information is already available. * Add an option to use ncclNet instead of sockets to perform bootstrap allgather operations. Add PAT algorithms for Allgather and ReduceScatter * Parallel Aggregated Trees, variation of Bruck algorithm. * Logarithmic number of network steps for small sizes at scale. * Only supports one rank per node at the moment. Add support for registered buffers for intra-node communication. * Allow registered user buffers to be accessed directly intra-node * Avoids extra copies in algorithms which permit it, saving memory bandwidth and helping with compute overlap. Add profiler plugin API * New plugin API for profiling * Supports various levels of profiling, with a hierarchy. Asynchronous graph allocation * Make calls to cudaMalloc and cudaMemcpy during graph allocation asynchronous. * Significantly speeds up graph capture. Use fatal IB asynchronous events to stop network operation * Avoids many other error messages * Only fatal errors are affected; potentially transient errors (e.g. port down) do not cause an immediate stop. Set P2P level to PXB on AMD CPUs when using more than 2 GPUs per node * P2P would cause a significant performance degradation when using many GPUs, and therefore many interleaved data flows. * Disable P2P through the CPU when we have 3+ GPUs per node; keep it enabled when we only have 2 GPUs. Improve the init logs to report the real NCCL function. * Make the log report ncclCommInitRank or ncclCommSplit, rather than the generic ncclCommInitRankFunc. Add a parameter to set the location of the user configuration file. * Add NCCL_CONF_FILE environment variable to set where the user's configuration file resides. Increase default IB timeout * Increase IB timeout value from 18 to 20. * Should help avoid fatal errors on large RoCE systems. Add new check for nvidia peermem * On linux kernels 6.6+, /sys/kernel/mm/memory_peers is no longer present; check for /sys/module/nvidia_peermem/version instead. Fix old performance regression when mixing small and large operations. * Improves distribution of work on channels. Fix crash when NUMA IDs are equal to -1. * Can happen when a NIC is a virtual NIC, or when linux doesn't know which NUMA node a device is attached to * Issue NVIDIA/nccl-tests#233 Fix tree graph search when NCCL_CROSS_NIC is set to 1. * Would force NCCL to use the balanced_tree pattern, thereby disabling LL128 on platforms with 1 GPU+1 NIC per PCI switch. * Would also try to use alternate rings even though it was not needed. Compiler tweaks and fixes * PR #1177 * PR #1228 Fix stack smash * PR #1325 Fixes for multi-node NVLink + IB operation Coverity fixes and comments.
64 γραμμές
2.2 KiB
C++
64 γραμμές
2.2 KiB
C++
/*************************************************************************
|
|
* Copyright (c) 2016-2019, NVIDIA CORPORATION. All rights reserved.
|
|
*
|
|
* See LICENSE.txt for license information
|
|
************************************************************************/
|
|
|
|
#include "core.h"
|
|
|
|
void dumpLine(int* values, int nranks, const char* prefix) {
|
|
constexpr int line_length = 128;
|
|
char line[line_length];
|
|
int num_width = snprintf(nullptr, 0, "%d", nranks-1); // safe as per "man snprintf"
|
|
int n = snprintf(line, line_length, "%s", prefix);
|
|
for (int i = 0; i < nranks && n < line_length-1; i++) {
|
|
n += snprintf(line + n, line_length - n, " %*d", num_width, values[i]);
|
|
// At this point n may be more than line_length-1, so don't use it
|
|
// for indexing into "line".
|
|
}
|
|
if (n >= line_length) {
|
|
// Sprintf wanted to write more than would fit in the buffer. Assume
|
|
// line_length is at least 4 and replace the end with "..." to
|
|
// indicate that it was truncated.
|
|
snprintf(line+line_length-4, 4, "...");
|
|
}
|
|
INFO(NCCL_INIT, "%s", line);
|
|
}
|
|
|
|
ncclResult_t ncclBuildRings(int nrings, int* rings, int rank, int nranks, int* prev, int* next) {
|
|
for (int r=0; r<nrings; r++) {
|
|
char prefix[40];
|
|
/*sprintf(prefix, "[%d] Channel %d Prev : ", rank, r);
|
|
dumpLine(prev+r*nranks, nranks, prefix);
|
|
sprintf(prefix, "[%d] Channel %d Next : ", rank, r);
|
|
dumpLine(next+r*nranks, nranks, prefix);*/
|
|
|
|
int current = rank;
|
|
for (int i=0; i<nranks; i++) {
|
|
rings[r*nranks+i] = current;
|
|
current = next[r*nranks+current];
|
|
}
|
|
snprintf(prefix, sizeof(prefix), "Channel %02d/%02d :", r, nrings);
|
|
if (rank == 0) dumpLine(rings+r*nranks, nranks, prefix);
|
|
if (current != rank) {
|
|
WARN("Error : ring %d does not loop back to start (%d != %d)", r, current, rank);
|
|
return ncclInternalError;
|
|
}
|
|
// Check that all ranks are there
|
|
for (int i=0; i<nranks; i++) {
|
|
int found = 0;
|
|
for (int j=0; j<nranks; j++) {
|
|
if (rings[r*nranks+j] == i) {
|
|
found = 1;
|
|
break;
|
|
}
|
|
}
|
|
if (found == 0) {
|
|
WARN("Error : ring %d does not contain rank %d", r, i);
|
|
return ncclInternalError;
|
|
}
|
|
}
|
|
}
|
|
return ncclSuccess;
|
|
}
|