72ef100050
Fix P2P scheduling
2022-10-31 08:54:34 -07:00
2f4cb874ba
Merge tag 'v2.15.5-1'
2022-10-25 01:15:22 -07:00
cb111f764a
2.15.5-1
...
Fix crash with CollnetChain on some node topologies
Fix hang when interleaving the capture of different graphs
Fix hang during init in multi-threaded mode
Fix potential data corruption with LL128 protocol on unaligned buffers.
Fix CPU usage during preconnect
Fixes double-free in the error path for ncclCommInitAll
Workaround hang on H100 with Ring/LL128 on 2 GPUs.
2022-10-25 00:55:55 -07:00
4f0e223db4
Merge remote-tracking branch 'nccl/master' into develop
2022-10-20 15:41:29 +00:00
bc8ef779df
Fix missing initialization due to merge error ( #640 )
2022-10-19 21:20:11 -07:00
9ddf0e0649
Support P2P with invisible devices ( #636 )
...
* Support P2P with invisible devices
* Update copyright year
2022-10-17 10:24:59 -07:00
9916a09818
Merge pull request #634 from yzygitzh/ziyyang/npkit-fix
...
Apply several fixes to NPKit
2022-10-17 08:01:24 -07:00
ebb8b5bf63
Updating files for missing licenses ( #637 )
2022-10-14 13:49:16 -06:00
7d6bbc19d4
apply npkit
2022-10-14 01:28:17 +00:00
d128d62238
Merge tag 'v2.15.1-1'
2022-10-07 11:00:26 -07:00
2401f4a918
Fixes a double-free in the error path of ncclCommInitAll.
...
Fixes https://github.com/NVIDIA/nccl/issues/726
2022-10-03 17:12:32 -07:00
e645b02cd8
introduce a hw topology aware bintree
...
for hayabusa architecture.
2022-10-03 15:26:21 +00:00
ef71550738
Added new gpu targets ( #631 )
2022-09-29 14:53:55 -06:00
a523b37ac7
Another threadfence and flags rework ( #629 )
2022-09-28 16:49:29 -07:00
021932b3c8
Add LL128 tuning ( #630 )
2022-09-27 09:39:09 -07:00
da8152e57a
2.15.1-1
...
Add support for H100 (sm90).
Make sure NCCL kernel honor user stream priorities.
2022-09-27 02:31:13 -07:00
99c28f2e75
Merge remote-tracking branch 'origin/master'
2022-09-27 02:24:41 -07:00
78313a6d21
Use compatibility shim only with static cudart
...
Closes issue 658
2022-09-27 02:22:48 -07:00
e8af0716c4
Merge pull request #619 from whchung/exp_reduce_code_size
...
Only use split tree algorithm to reduce kernel code size.
2022-09-26 10:06:27 -05:00
ecab28a7c9
Fix potential deadlock during init in multi-thread mode.
...
Make sure all calls calling cudaMalloc (including devCommSetup) are
called before the last bootstrapBarrier. That way, we avoid calls to
cudaMalloc be blocked by a NCCL kernel launched on another GPU by
another thread which completed init faster.
Resolve #623 .
2022-09-26 02:13:10 -07:00
35f1fe3434
Merge pull request #621 from whchung/exp_reduce_sleep_cycles
...
Reduce s_sleep cycles
2022-09-23 15:31:16 -05:00
a80cc7e6e1
Merge pull request #624 from whchung/exp_tweak_unroll_factors
...
Tweak unroll factors.
2022-09-23 11:30:05 -05:00
84054c3b30
Tweak unroll factors.
2022-09-22 13:03:04 -05:00
02929cffb6
Only use split tree algorithm to reduce kernel code size.
2022-09-22 12:01:53 -05:00
a3c8ef8c03
Reduce s_sleep cycles
2022-09-22 12:01:12 -05:00
b9ae02d4ad
Abolish syncthreads and only wait on LDS traffic.
2022-09-22 12:00:37 -05:00
49c811ecf9
Rework threadfence and flag setting ( #627 )
2022-09-22 08:35:42 -07:00
d9216af48b
Revert changes to gfx1030 ( #622 )
2022-09-20 20:06:17 -07:00
9e6c87a2bf
Define ncclShmem as global shared ( #618 )
...
* Use global defined shared memory
* Add --hipcc-func-supp to compile option
* Force inline some device functions
* Add back threadfence
2022-09-20 09:00:20 -07:00
f89fd4777d
address review comments
2022-09-20 11:58:33 +02:00
79fb0326ac
Fix intermittent 11.6 builds: generate unique .cu file for each object file
2022-09-20 11:58:33 +02:00
8f3219dbd4
make binary tree work on 2.13.4
2022-09-15 00:01:54 +00:00
a06e14e39b
Misc fixes and disable binTree
2022-09-14 00:26:19 +00:00
e5d2dfed34
Update init.cc
2022-09-13 17:29:32 -05:00
be935d7ce7
Merge branch 'develop' into 2.13.4
2022-09-13 17:19:04 -05:00
65e2ae20e5
add binary tree
...
In addition, introduce the ability to have 2 trees at the same time.
Only for allreduce at the moment.
2022-09-13 20:52:32 +00:00
009e79623f
Merge branch 'develop' into 2.13.4
2022-09-09 23:07:04 +00:00
dd56135a9a
Updating stream caching ( #614 )
...
- Adding non-captured hipStream for use in setup
2022-09-09 16:30:15 -06:00
a79d9e3586
Merge remote-tracking branch 'nccl/master' into develop
2022-09-09 16:05:38 +00:00
7bbce085cc
Enable LL128 protocol support ( #605 )
...
* Enable LL128 protocol support
* Use shared memory object directly when possible
2022-09-08 14:45:27 -07:00
2b57751abb
Fix compilation issues with buck ( #610 )
...
* Fix compilation warning with -Wmisleading-indentation
When compile with -Wmisleading-indentation, it reports warning:
misleading indentation; statement is not part of the previous 'if'
This patch fixes it
* Avoid relative include file path
We don't need relative include file paths for src/graph/*.h
since src/ is already in CMake include_directories
2022-09-07 09:56:05 -06:00
47b2fc3a30
Adding opt-in hipGraph support for RCCL via RCCL_ENABLE_HIPGRAPH ( #608 )
...
Adding opt-in hipGraph support via RCCL_ENABLE_HIPGRAPH
2022-09-06 10:29:46 -06:00
06bce9d0c9
added stream synch after hipMemset ( #609 )
2022-08-30 16:18:37 -06:00
c9f2fe1f65
Use hipExtLaunchKernel when not using graph and not in group mode ( #606 )
2022-08-26 13:40:37 -07:00
e739c62a53
Merge pull request #598 from edgargabriel/topic/tree-multirank
...
Expand ncclTreeBasePostset for multi-rank
2022-08-24 08:28:34 -05:00
88487a62bb
Use non-temporal access for slow path ( #602 )
2022-08-23 08:21:51 -07:00
4141ec1151
fix channelcount for multi-rank scenario
2022-08-22 19:09:22 +00:00
3c1b1ec8c8
Simple tree changes ( #599 )
...
changed treebase to create basic balanced tree
2022-08-19 13:51:49 -06:00
080fc2d9d6
fix error: use of undeclared identifier 'free'
...
include stdlib.h to fix compilation error in rccl :
[39/58] Building CXX object CMakeFiles/rccl.dir/src/misc/signals.cc.o
FAILED: CMakeFiles/rccl.dir/src/misc/signals.cc.o
/opt/rocm/bin/hipcc -DENABLE_COLLTRACE -DHAVE_BFD -DHAVE_CPLUS_DEMANGLE -DUSE_ROCM_SMI64CONFIG -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -Drccl_EXPORTS -I/home/cosmo/build/flgrwqa/build/include -I/home/cosmo/build/flgrwqa/build/include/rccl -I/home/cosmo/build/flgrwqa/rccl/src -I/home/cosmo/build/flgrwqa/rccl/src/include -I/home/cosmo/build/flgrwqa/rccl/src/collectives -I/home/cosmo/build/flgrwqa/rccl/src/collectives/device -I/opt/hsa/include -fPIC -fvisibility=hidden -fgpu-rdc -parallel-jobs=8 -Wno-format-nonliteral -x hip --offload-arch=gfx803 --offload-arch=gfx900:xnack- --offload-arch=gfx906:xnack- --offload-arch=gfx908:xnack- --offload-arch=gfx90a:xnack- --offload-arch=gfx90a:xnack+ --offload-arch=gfx1030 -std=c++14 -MD -MT CMakeFiles/rccl.dir/src/misc/signals.cc.o -MF CMakeFiles/rccl.dir/src/misc/signals.cc.o.d -o CMakeFiles/rccl.dir/src/misc/signals.cc.o -c /home/cosmo/build/flgrwqa/rccl/src/misc/signals.cc
In file included from /home/cosmo/build/flgrwqa/rccl/src/misc/signals.cc:8:
/home/cosmo/build/flgrwqa/rccl/src/include/BfdBacktrace.hpp:138:9: error: use of undeclared identifier 'free'
free(file->syms);
^
/home/cosmo/build/flgrwqa/rccl/src/include/BfdBacktrace.hpp:155:5: error: use of undeclared identifier 'free'
free(file->syms);
^
2022-08-19 20:25:06 +03:00
14b8ff153f
Repurpose profiling implementation to simple timestamps tracing ( #600 )
2022-08-18 15:34:46 -07:00
Wenkai Du
Sylvain Jeaugey
Wenkai Du
gilbertlee-amd
Ziyue Yang
John Bachan
Edgar Gabriel
akolliasAMD
Cliff Woolley
Wen-Heng (Jack) Chung
Wen-Heng (Jack) Chung
Jane Xu
Edgar Gabriel
Gilbert Lee
Min Si
Cosmic Fusion