2288e9ae80
Switch to hipLaunchHostFunc for HIP graph ( #667 )
2022-12-15 10:16:46 -08:00
adafc0f759
Add MSCCL Support ( #658 )
...
* Add MSCCL support
* Add alignment and message size checking
* Fix nRanks checking, in-place and out-of-place tests and group call handling
* Fix hipGraph unit test
* Change MSCCL init warning to INFO
* Revise license info
2022-12-12 15:51:04 -08:00
b953544a59
Fix typo in detecting Intel platforms ( #661 )
2022-12-07 13:36:11 -08:00
eca623df07
decreased warp size for gfx110x ( #655 )
2022-12-01 12:19:21 -07:00
fb9938cffa
Query DMABuf support through HSA runtime API ( #654 )
2022-11-30 08:53:03 -08:00
9594bbee3b
Adjust P2P channels on Intel platform ( #653 )
2022-11-29 13:57:10 -08:00
57764f8152
Fix incorrect rocm-smi ID conversion ( #648 )
2022-11-21 19:44:39 -08:00
9cb72a3d0f
Fix collective trace timestamp format ( #647 )
2022-11-21 08:11:12 -08:00
cf3c32a626
Fix typo in previous hipify change ( #645 )
2022-11-15 11:51:47 -08:00
562dd87036
Move hipify to cmake stage
...
Add minimal ROCm/HIP version requirements for Graph support
2022-11-14 18:10:45 +00:00
94ad7f6f51
Update tuning table and fix topo_expl
2022-11-07 18:24:24 +00:00
9a077e6947
Merge remote-tracking branch 'nccl/master' into develop
2022-11-03 21:17:42 +00:00
72ef100050
Fix P2P scheduling
2022-10-31 08:54:34 -07:00
2f4cb874ba
Merge tag 'v2.15.5-1'
2022-10-25 01:15:22 -07:00
cb111f764a
2.15.5-1
...
Fix crash with CollnetChain on some node topologies
Fix hang when interleaving the capture of different graphs
Fix hang during init in multi-threaded mode
Fix potential data corruption with LL128 protocol on unaligned buffers.
Fix CPU usage during preconnect
Fixes double-free in the error path for ncclCommInitAll
Workaround hang on H100 with Ring/LL128 on 2 GPUs.
2022-10-25 00:55:55 -07:00
4f0e223db4
Merge remote-tracking branch 'nccl/master' into develop
2022-10-20 15:41:29 +00:00
bc8ef779df
Fix missing initialization due to merge error ( #640 )
2022-10-19 21:20:11 -07:00
9ddf0e0649
Support P2P with invisible devices ( #636 )
...
* Support P2P with invisible devices
* Update copyright year
2022-10-17 10:24:59 -07:00
9916a09818
Merge pull request #634 from yzygitzh/ziyyang/npkit-fix
...
Apply several fixes to NPKit
2022-10-17 08:01:24 -07:00
ebb8b5bf63
Updating files for missing licenses ( #637 )
2022-10-14 13:49:16 -06:00
7d6bbc19d4
apply npkit
2022-10-14 01:28:17 +00:00
d128d62238
Merge tag 'v2.15.1-1'
2022-10-07 11:00:26 -07:00
2401f4a918
Fixes a double-free in the error path of ncclCommInitAll.
...
Fixes https://github.com/NVIDIA/nccl/issues/726
2022-10-03 17:12:32 -07:00
e645b02cd8
introduce a hw topology aware bintree
...
for hayabusa architecture.
2022-10-03 15:26:21 +00:00
ef71550738
Added new gpu targets ( #631 )
2022-09-29 14:53:55 -06:00
a523b37ac7
Another threadfence and flags rework ( #629 )
2022-09-28 16:49:29 -07:00
021932b3c8
Add LL128 tuning ( #630 )
2022-09-27 09:39:09 -07:00
da8152e57a
2.15.1-1
...
Add support for H100 (sm90).
Make sure NCCL kernel honor user stream priorities.
2022-09-27 02:31:13 -07:00
99c28f2e75
Merge remote-tracking branch 'origin/master'
2022-09-27 02:24:41 -07:00
78313a6d21
Use compatibility shim only with static cudart
...
Closes issue 658
2022-09-27 02:22:48 -07:00
e8af0716c4
Merge pull request #619 from whchung/exp_reduce_code_size
...
Only use split tree algorithm to reduce kernel code size.
2022-09-26 10:06:27 -05:00
ecab28a7c9
Fix potential deadlock during init in multi-thread mode.
...
Make sure all calls calling cudaMalloc (including devCommSetup) are
called before the last bootstrapBarrier. That way, we avoid calls to
cudaMalloc be blocked by a NCCL kernel launched on another GPU by
another thread which completed init faster.
Resolve #623 .
2022-09-26 02:13:10 -07:00
35f1fe3434
Merge pull request #621 from whchung/exp_reduce_sleep_cycles
...
Reduce s_sleep cycles
2022-09-23 15:31:16 -05:00
a80cc7e6e1
Merge pull request #624 from whchung/exp_tweak_unroll_factors
...
Tweak unroll factors.
2022-09-23 11:30:05 -05:00
84054c3b30
Tweak unroll factors.
2022-09-22 13:03:04 -05:00
02929cffb6
Only use split tree algorithm to reduce kernel code size.
2022-09-22 12:01:53 -05:00
a3c8ef8c03
Reduce s_sleep cycles
2022-09-22 12:01:12 -05:00
b9ae02d4ad
Abolish syncthreads and only wait on LDS traffic.
2022-09-22 12:00:37 -05:00
49c811ecf9
Rework threadfence and flag setting ( #627 )
2022-09-22 08:35:42 -07:00
d9216af48b
Revert changes to gfx1030 ( #622 )
2022-09-20 20:06:17 -07:00
9e6c87a2bf
Define ncclShmem as global shared ( #618 )
...
* Use global defined shared memory
* Add --hipcc-func-supp to compile option
* Force inline some device functions
* Add back threadfence
2022-09-20 09:00:20 -07:00
f89fd4777d
address review comments
2022-09-20 11:58:33 +02:00
79fb0326ac
Fix intermittent 11.6 builds: generate unique .cu file for each object file
2022-09-20 11:58:33 +02:00
8f3219dbd4
make binary tree work on 2.13.4
2022-09-15 00:01:54 +00:00
a06e14e39b
Misc fixes and disable binTree
2022-09-14 00:26:19 +00:00
e5d2dfed34
Update init.cc
2022-09-13 17:29:32 -05:00
be935d7ce7
Merge branch 'develop' into 2.13.4
2022-09-13 17:19:04 -05:00
65e2ae20e5
add binary tree
...
In addition, introduce the ability to have 2 trees at the same time.
Only for allreduce at the moment.
2022-09-13 20:52:32 +00:00
009e79623f
Merge branch 'develop' into 2.13.4
2022-09-09 23:07:04 +00:00
dd56135a9a
Updating stream caching ( #614 )
...
- Adding non-captured hipStream for use in setup
2022-09-09 16:30:15 -06:00
Wenkai Du
Ziyue Yang
akolliasAMD
Wenkai Du
Sylvain Jeaugey
gilbertlee-amd
John Bachan
Edgar Gabriel
Cliff Woolley
Wen-Heng (Jack) Chung
Wen-Heng (Jack) Chung
Jane Xu
Edgar Gabriel
Gilbert Lee