a80cc7e6e1
Merge pull request #624 from whchung/exp_tweak_unroll_factors
...
Tweak unroll factors.
2022-09-23 11:30:05 -05:00
84054c3b30
Tweak unroll factors.
2022-09-22 13:03:04 -05:00
02929cffb6
Only use split tree algorithm to reduce kernel code size.
2022-09-22 12:01:53 -05:00
a3c8ef8c03
Reduce s_sleep cycles
2022-09-22 12:01:12 -05:00
b9ae02d4ad
Abolish syncthreads and only wait on LDS traffic.
2022-09-22 12:00:37 -05:00
49c811ecf9
Rework threadfence and flag setting ( #627 )
2022-09-22 08:35:42 -07:00
d9216af48b
Revert changes to gfx1030 ( #622 )
2022-09-20 20:06:17 -07:00
9e6c87a2bf
Define ncclShmem as global shared ( #618 )
...
* Use global defined shared memory
* Add --hipcc-func-supp to compile option
* Force inline some device functions
* Add back threadfence
2022-09-20 09:00:20 -07:00
f89fd4777d
address review comments
2022-09-20 11:58:33 +02:00
79fb0326ac
Fix intermittent 11.6 builds: generate unique .cu file for each object file
2022-09-20 11:58:33 +02:00
8f3219dbd4
make binary tree work on 2.13.4
2022-09-15 00:01:54 +00:00
a06e14e39b
Misc fixes and disable binTree
2022-09-14 00:26:19 +00:00
e5d2dfed34
Update init.cc
2022-09-13 17:29:32 -05:00
be935d7ce7
Merge branch 'develop' into 2.13.4
2022-09-13 17:19:04 -05:00
65e2ae20e5
add binary tree
...
In addition, introduce the ability to have 2 trees at the same time.
Only for allreduce at the moment.
2022-09-13 20:52:32 +00:00
009e79623f
Merge branch 'develop' into 2.13.4
2022-09-09 23:07:04 +00:00
dd56135a9a
Updating stream caching ( #614 )
...
- Adding non-captured hipStream for use in setup
2022-09-09 16:30:15 -06:00
a79d9e3586
Merge remote-tracking branch 'nccl/master' into develop
2022-09-09 16:05:38 +00:00
7bbce085cc
Enable LL128 protocol support ( #605 )
...
* Enable LL128 protocol support
* Use shared memory object directly when possible
2022-09-08 14:45:27 -07:00
2b57751abb
Fix compilation issues with buck ( #610 )
...
* Fix compilation warning with -Wmisleading-indentation
When compile with -Wmisleading-indentation, it reports warning:
misleading indentation; statement is not part of the previous 'if'
This patch fixes it
* Avoid relative include file path
We don't need relative include file paths for src/graph/*.h
since src/ is already in CMake include_directories
2022-09-07 09:56:05 -06:00
47b2fc3a30
Adding opt-in hipGraph support for RCCL via RCCL_ENABLE_HIPGRAPH ( #608 )
...
Adding opt-in hipGraph support via RCCL_ENABLE_HIPGRAPH
2022-09-06 10:29:46 -06:00
06bce9d0c9
added stream synch after hipMemset ( #609 )
2022-08-30 16:18:37 -06:00
c9f2fe1f65
Use hipExtLaunchKernel when not using graph and not in group mode ( #606 )
2022-08-26 13:40:37 -07:00
e739c62a53
Merge pull request #598 from edgargabriel/topic/tree-multirank
...
Expand ncclTreeBasePostset for multi-rank
2022-08-24 08:28:34 -05:00
88487a62bb
Use non-temporal access for slow path ( #602 )
2022-08-23 08:21:51 -07:00
4141ec1151
fix channelcount for multi-rank scenario
2022-08-22 19:09:22 +00:00
3c1b1ec8c8
Simple tree changes ( #599 )
...
changed treebase to create basic balanced tree
2022-08-19 13:51:49 -06:00
080fc2d9d6
fix error: use of undeclared identifier 'free'
...
include stdlib.h to fix compilation error in rccl :
[39/58] Building CXX object CMakeFiles/rccl.dir/src/misc/signals.cc.o
FAILED: CMakeFiles/rccl.dir/src/misc/signals.cc.o
/opt/rocm/bin/hipcc -DENABLE_COLLTRACE -DHAVE_BFD -DHAVE_CPLUS_DEMANGLE -DUSE_ROCM_SMI64CONFIG -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -Drccl_EXPORTS -I/home/cosmo/build/flgrwqa/build/include -I/home/cosmo/build/flgrwqa/build/include/rccl -I/home/cosmo/build/flgrwqa/rccl/src -I/home/cosmo/build/flgrwqa/rccl/src/include -I/home/cosmo/build/flgrwqa/rccl/src/collectives -I/home/cosmo/build/flgrwqa/rccl/src/collectives/device -I/opt/hsa/include -fPIC -fvisibility=hidden -fgpu-rdc -parallel-jobs=8 -Wno-format-nonliteral -x hip --offload-arch=gfx803 --offload-arch=gfx900:xnack- --offload-arch=gfx906:xnack- --offload-arch=gfx908:xnack- --offload-arch=gfx90a:xnack- --offload-arch=gfx90a:xnack+ --offload-arch=gfx1030 -std=c++14 -MD -MT CMakeFiles/rccl.dir/src/misc/signals.cc.o -MF CMakeFiles/rccl.dir/src/misc/signals.cc.o.d -o CMakeFiles/rccl.dir/src/misc/signals.cc.o -c /home/cosmo/build/flgrwqa/rccl/src/misc/signals.cc
In file included from /home/cosmo/build/flgrwqa/rccl/src/misc/signals.cc:8:
/home/cosmo/build/flgrwqa/rccl/src/include/BfdBacktrace.hpp:138:9: error: use of undeclared identifier 'free'
free(file->syms);
^
/home/cosmo/build/flgrwqa/rccl/src/include/BfdBacktrace.hpp:155:5: error: use of undeclared identifier 'free'
free(file->syms);
^
2022-08-19 20:25:06 +03:00
14b8ff153f
Repurpose profiling implementation to simple timestamps tracing ( #600 )
2022-08-18 15:34:46 -07:00
c4e2aa6c79
2.14.3-1
...
Add support for improved fault tolerance: non-blocking mode, new
init function with config, and ncclCommFinalize function.
Reintroduce collnet+chain algorithm, alongside collnet+direct.
Add LL protocol for intra-node P2P (on by default) and network
communication (off by default).
Use network instead of shared memory when performance is better.
Fix: wait for CUDA graph destroy before destroying comm with linked
graph resources.
Remove aggressive polling during enqueue.
Fix DMABUF fallback on MOFED 5.4 and earlier.
2022-08-18 02:53:17 -07:00
e1d9b273b0
fix NCCL_DEBUG_FILE
...
Summary: NCCL_DEBUG_FILE does not work properly since the recent v2.13.4 updates (https://github.com/NVIDIA/nccl/pull/682 ) because it nows sets `ncclDebugLevel` after parse `NCCL_DEBUG_FILE`. This patch move parsing `tempNcclDebugLevel` before processing `NCCL_DEBUG_FILE` to ensure `NCCL_DEBUG_FILE` is parsed only when `NCCL_DEBUG > NCCL_LOG_VERSION` (same as previous behavior)
Differential Revision: D38415208
fbshipit-source-id: 5689bbb798e73efb9e8594557666987f07e89a30
2022-08-18 11:50:42 +02:00
f5c0b243a8
Add XGMI sys type and clean up detection code ( #597 )
2022-08-12 09:52:29 -07:00
f6b9686482
Improve alignment and tuning for Pivot A2A algorithm ( #593 )
...
* Improve alignment and tuning for Pivot A2A algorithm
* enable pivot a2a by default
2022-08-05 19:40:19 -07:00
4cecdc9be5
minor latency tuning ( #591 )
...
* minor tuning for tree ll
2022-08-03 15:07:44 -06:00
9089e68a99
Revert "Use nontemporal in slow path and add XGMI sys type ( #575 )" ( #590 )
...
This reverts commit b250c01cbe
2022-08-02 09:31:53 -07:00
254208e7dd
Fixed issue with attomicEXCH creating errors on multi node runs ( #587 )
2022-07-22 11:32:49 -06:00
451c287aa6
Removing redundant LOAD and STORE on primitives plus adding some atomics ( #585 )
2022-07-21 13:04:57 -06:00
6dd090917a
Merge pull request #580 from hubertlu-tw/develop
...
Enhancement of RCCL logging information for topology-aware utilities
2022-07-15 15:16:37 -07:00
2b1d5d3bc1
intercept SIGUSR2 in RCCL
...
add support for intercepting SIGUSR2 in RCCL. This signal will
not terminate the execution of the application, but print the stacktrace
of the process that the signal was sent to instead.
2022-07-15 16:28:46 +00:00
77c2bef952
Remove redundant LOAD/STORE usage in LL initialization
2022-07-14 00:40:36 +00:00
a1842df858
Enhancement of RCCL logging information for topology-aware utilities
2022-07-11 19:01:10 +00:00
19ab67d172
2.13.4-1
...
Optimize CUDA graph launch; avoid launching a CPU callback for
intra-node operations.
Simplify kernel common code to improve the latency of send/recv
operations.
Strengthen CUDA streams semantics.
Change NET API to v6, to add dmabuf support.
Add ncclGetLastError() function.
Add ncclRemoteError code and use it for remote network errors.
Support the use of a different NCCL_NET parameter per communicator.
Add support for SHM and P2P transfers using cudaMemcpy.
2022-07-11 08:10:34 -07:00
8c3c8b78c0
Skip HDP cache flush for gfx90a ( #578 )
...
* Skip HDP cache flush for gfx90a
* Remove extra debug print
2022-07-08 10:13:32 -07:00
aa0d7ca882
Add more constraints to enable GDR ( #579 )
...
* Add more constraints to enable GDR
* Revert deleted line
2022-07-08 09:52:27 -07:00
80f53cc171
Reduce AlltoAll port usage in send/recv proxy ( #577 )
...
* Reduce AlltoAll port usage when connecting proxy
Reuse socket ports when connecting proxies in AlltoAll.
Existing port usage in AlltoAll is O(n) for recv and O(n) for send,
reusing socket ports in server or client side will make one of them
O(1), reusing both will reduce the total port usage to O(1) and enables
AlltoAll in >64 MI200 nodes.
* Update changelog accordingly
Update changelog accordingly.
2022-07-07 16:15:52 -07:00
b250c01cbe
Use nontemporal in slow path and add XGMI sys type ( #575 )
...
* Use nontemporal in slow path and add XGMI sys type
* Clean up XGMI detection
2022-07-06 07:58:41 -07:00
00af1f64e9
Fix GPU to NIC mapping in tree ( #573 )
...
* Fix GPU to NIC mapping in tree
* Update tuning table
2022-07-03 20:52:52 -07:00
a89a9966aa
Adding git hash info to version output line ( #572 )
2022-06-28 16:42:51 -06:00
9a285b5e1d
Do not set NET GDR level automatically ( #571 )
2022-06-23 16:28:28 -07:00
c3bb9e70d0
Use different atomics to check flags in kernel ( #568 )
2022-06-23 09:16:41 -07:00
Wen-Heng (Jack) Chung
Wen-Heng (Jack) Chung
Wenkai Du
Wenkai Du
Jane Xu
Edgar Gabriel
Edgar Gabriel
Gilbert Lee
gilbertlee-amd
Min Si
akolliasAMD
Cosmic Fusion
Sylvain Jeaugey
Ching-Hsiang Chu
Ziyue Yang
Hubert Lu
hubertlu-tw
Yifan Xiong