2.18.3-1

Fix data corruption with Tree/LL128 on systems with 1GPU:1NIC.
Fix hang with Collnet on bfloat16 on systems with less than one NIC
per GPU.
Fix long initialization time.
Fix data corruption with Collnet when mixing multi-process and
multi-GPU per process.
Fix crash when shared memory creation fails.
Fix Avg operation with Collnet/Chain.
Fix performance of alltoall at scale with more than one NIC per GPU.
Fix performance for DGX H800.
Fix race condition in connection progress causing a crash.
Fix network flush with Collnet.
Fix performance of aggregated allGather/reduceScatter operations.
Fix PXN operation when CUDA_VISIBLE_DEVICES is set.
Fix NVTX3 compilation issues on Debian 10.


[ROCm/rccl commit: ea38312273]

projects/rccl/src/graph/connect.cc

@@ -169,14 +169,16 @@ static ncclResult_t connectTrees(struct ncclComm* comm, int* treeToParent, int*
 static ncclResult_t connectCollNet(struct ncclComm* comm, struct ncclTopoGraph* collNetGraph) {
   int rank = comm->rank;
   int localRanks = comm->localRanks;
-  int nHeads = collNetGraph->nChannels;
+  int nHeads = 0;
   int *heads;
-  NCCLCHECK(ncclCalloc(&heads, nHeads));
+  NCCLCHECK(ncclCalloc(&heads, localRanks));
   // Find all head ranks
   // Head index is always 0
-  for (int c=0; c<nHeads; c++) {
+  for (int c=0; c<collNetGraph->nChannels; c++) {
     int* collNetIntra = collNetGraph->intra+c*localRanks;
-    heads[c] = collNetIntra[0];
+    int head = collNetIntra[0];
+    for (int h=0; h<nHeads; h++) if (heads[h] == head) head = -1;
+    if (head != -1) heads[nHeads++] = collNetIntra[0];
   }
   // For all channels
   for (int c=0; c<comm->nChannels; c++) {