975642e7ee
Abolish syncthreads and only wait on LDS traffic.
...
[ROCm/rccl commit: b9ae02d4ad
2022-09-22 12:00:37 -05:00
e4d46a0f64
Rework threadfence and flag setting ( #627 )
...
[ROCm/rccl commit: 49c811ecf9
2022-09-22 08:35:42 -07:00
81c71aeb67
Revert changes to gfx1030 ( #622 )
...
[ROCm/rccl commit: d9216af48b
2022-09-20 20:06:17 -07:00
98609a7b92
Define ncclShmem as global shared ( #618 )
...
* Use global defined shared memory
* Add --hipcc-func-supp to compile option
* Force inline some device functions
* Add back threadfence
[ROCm/rccl commit: 9e6c87a2bf
2022-09-20 09:00:20 -07:00
95d6ed2154
make binary tree work on 2.13.4
...
[ROCm/rccl commit: 8f3219dbd4
2022-09-15 00:01:54 +00:00
ba6e2db70d
Misc fixes and disable binTree
...
[ROCm/rccl commit: a06e14e39b
2022-09-14 00:26:19 +00:00
4a86adcaba
Update init.cc
...
[ROCm/rccl commit: e5d2dfed34
2022-09-13 17:29:32 -05:00
4c17f4dcc1
Merge branch 'develop' into 2.13.4
...
[ROCm/rccl commit: be935d7ce7
2022-09-13 17:19:04 -05:00
7148c0aa7b
add binary tree
...
In addition, introduce the ability to have 2 trees at the same time.
Only for allreduce at the moment.
[ROCm/rccl commit: 65e2ae20e5
2022-09-13 20:52:32 +00:00
1d24c476f4
Merge branch 'develop' into 2.13.4
...
[ROCm/rccl commit: 009e79623f
2022-09-09 23:07:04 +00:00
35872115f8
Updating stream caching ( #614 )
...
- Adding non-captured hipStream for use in setup
[ROCm/rccl commit: dd56135a9a
2022-09-09 16:30:15 -06:00
af71be44f1
GraphBench ( #613 )
...
Adding simple GraphBench tool for comparing RCCL hipGraph performance
[ROCm/rccl commit: 65d78e9a1d
2022-09-09 12:12:25 -06:00
7874a99c75
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: a79d9e3586
2022-09-09 16:05:38 +00:00
fe99249cde
Enable LL128 protocol support ( #605 )
...
* Enable LL128 protocol support
* Use shared memory object directly when possible
[ROCm/rccl commit: 7bbce085cc
2022-09-08 14:45:27 -07:00
3da06e4704
Update ubuntu18 to ubuntu20 ( #611 )
...
[ROCm/rccl commit: d700a94918
2022-09-07 16:02:37 -06:00
25ba51fe83
Fix compilation issues with buck ( #610 )
...
* Fix compilation warning with -Wmisleading-indentation
When compile with -Wmisleading-indentation, it reports warning:
misleading indentation; statement is not part of the previous 'if'
This patch fixes it
* Avoid relative include file path
We don't need relative include file paths for src/graph/*.h
since src/ is already in CMake include_directories
[ROCm/rccl commit: 2b57751abb
2022-09-07 09:56:05 -06:00
616cb39a0b
Adding opt-in hipGraph support for RCCL via RCCL_ENABLE_HIPGRAPH ( #608 )
...
Adding opt-in hipGraph support via RCCL_ENABLE_HIPGRAPH
[ROCm/rccl commit: 47b2fc3a30
2022-09-06 10:29:46 -06:00
2cd63dac42
added stream synch after hipMemset ( #609 )
...
[ROCm/rccl commit: 06bce9d0c9
2022-08-30 16:18:37 -06:00
f18868f439
Use hipExtLaunchKernel when not using graph and not in group mode ( #606 )
...
[ROCm/rccl commit: c9f2fe1f65
2022-08-26 13:40:37 -07:00
151a8ef56a
git_version cmake consistency changes ( #604 )
...
* git_version cmake variable consistency changes
[ROCm/rccl commit: 6670dc95ab
2022-08-25 15:11:28 -06:00
22dcbed61b
Merge pull request #603 from edgargabriel/topic/float16_unit_tests
...
introduce support for ncclFloat16/half in UT
[ROCm/rccl commit: 8a311583e0
2022-08-25 07:40:20 -05:00
b32b819151
introduce support for ncclFloat16/half in UT
...
[ROCm/rccl commit: f6e00dec13
2022-08-24 15:28:24 +00:00
6bb871c986
Merge pull request #598 from edgargabriel/topic/tree-multirank
...
Expand ncclTreeBasePostset for multi-rank
[ROCm/rccl commit: e739c62a53
2022-08-24 08:28:34 -05:00
56ea2c4be5
Use non-temporal access for slow path ( #602 )
...
[ROCm/rccl commit: 88487a62bb
2022-08-23 08:21:51 -07:00
aa6d450f35
fix channelcount for multi-rank scenario
...
[ROCm/rccl commit: 4141ec1151
2022-08-22 19:09:22 +00:00
1d55fe756c
Simple tree changes ( #599 )
...
changed treebase to create basic balanced tree
[ROCm/rccl commit: 3c1b1ec8c8
2022-08-19 13:51:49 -06:00
1dcf1da5ca
fix error: use of undeclared identifier 'free'
...
include stdlib.h to fix compilation error in rccl :
[39/58] Building CXX object CMakeFiles/rccl.dir/src/misc/signals.cc.o
FAILED: CMakeFiles/rccl.dir/src/misc/signals.cc.o
/opt/rocm/bin/hipcc -DENABLE_COLLTRACE -DHAVE_BFD -DHAVE_CPLUS_DEMANGLE -DUSE_ROCM_SMI64CONFIG -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -Drccl_EXPORTS -I/home/cosmo/build/flgrwqa/build/include -I/home/cosmo/build/flgrwqa/build/include/rccl -I/home/cosmo/build/flgrwqa/rccl/src -I/home/cosmo/build/flgrwqa/rccl/src/include -I/home/cosmo/build/flgrwqa/rccl/src/collectives -I/home/cosmo/build/flgrwqa/rccl/src/collectives/device -I/opt/hsa/include -fPIC -fvisibility=hidden -fgpu-rdc -parallel-jobs=8 -Wno-format-nonliteral -x hip --offload-arch=gfx803 --offload-arch=gfx900:xnack- --offload-arch=gfx906:xnack- --offload-arch=gfx908:xnack- --offload-arch=gfx90a:xnack- --offload-arch=gfx90a:xnack+ --offload-arch=gfx1030 -std=c++14 -MD -MT CMakeFiles/rccl.dir/src/misc/signals.cc.o -MF CMakeFiles/rccl.dir/src/misc/signals.cc.o.d -o CMakeFiles/rccl.dir/src/misc/signals.cc.o -c /home/cosmo/build/flgrwqa/rccl/src/misc/signals.cc
In file included from /home/cosmo/build/flgrwqa/rccl/src/misc/signals.cc:8:
/home/cosmo/build/flgrwqa/rccl/src/include/BfdBacktrace.hpp:138:9: error: use of undeclared identifier 'free'
free(file->syms);
^
/home/cosmo/build/flgrwqa/rccl/src/include/BfdBacktrace.hpp:155:5: error: use of undeclared identifier 'free'
free(file->syms);
^
[ROCm/rccl commit: 080fc2d9d6
2022-08-19 20:25:06 +03:00
c2e9ada40b
Repurpose profiling implementation to simple timestamps tracing ( #600 )
...
[ROCm/rccl commit: 14b8ff153f
2022-08-18 15:34:46 -07:00
c9a50a9ec5
fix NCCL_DEBUG_FILE
...
Summary: NCCL_DEBUG_FILE does not work properly since the recent v2.13.4 updates (https://github.com/NVIDIA/nccl/pull/682 ) because it nows sets `ncclDebugLevel` after parse `NCCL_DEBUG_FILE`. This patch move parsing `tempNcclDebugLevel` before processing `NCCL_DEBUG_FILE` to ensure `NCCL_DEBUG_FILE` is parsed only when `NCCL_DEBUG > NCCL_LOG_VERSION` (same as previous behavior)
Differential Revision: D38415208
fbshipit-source-id: 5689bbb798e73efb9e8594557666987f07e89a30
[ROCm/rccl commit: e1d9b273b0
2022-08-18 11:50:42 +02:00
6c3f1366e8
Add XGMI sys type and clean up detection code ( #597 )
...
[ROCm/rccl commit: f5c0b243a8
2022-08-12 09:52:29 -07:00
478d8312b8
Improve alignment and tuning for Pivot A2A algorithm ( #593 )
...
* Improve alignment and tuning for Pivot A2A algorithm
* enable pivot a2a by default
[ROCm/rccl commit: f6b9686482
2022-08-05 19:40:19 -07:00
e3b832f4ce
Disable clique AllReduce UnitTest ( #595 )
...
[ROCm/rccl commit: dae11c2aca
2022-08-04 18:30:00 -06:00
b350916a6e
Fixing CMake to avoid unnecessary git_version relinking ( #594 )
...
[ROCm/rccl commit: 9ed9cd0e31
2022-08-04 18:03:59 -06:00
a44be6655d
HIP Path default updated to ROCM_PATH (reorg path) ( #592 )
...
Updated default path for hip to ROCM_PATH (/opt/rocm instead of /opt/rocm/hip) as per new/current structure.
[ROCm/rccl commit: 2cb2f9493a
2022-08-04 13:38:41 -04:00
6fb5c5d5e3
minor latency tuning ( #591 )
...
* minor tuning for tree ll
[ROCm/rccl commit: 4cecdc9be5
2022-08-03 15:07:44 -06:00
f70830d629
Revert "Use nontemporal in slow path and add XGMI sys type ( #575 )" ( #590 )
...
This reverts commit e04bba619a9089e68a99
2022-08-02 09:31:53 -07:00
7e124d5b83
Add nccl_net.h to librccl-dev package ( #589 )
...
[ROCm/rccl commit: e2cb95a390
2022-07-29 13:39:49 -07:00
d5ca0be51f
Fixed issue with attomicEXCH creating errors on multi node runs ( #587 )
...
[ROCm/rccl commit: 254208e7dd
2022-07-22 11:32:49 -06:00
fd99ca19f5
updated alltoallV test to reflect how send counts are done in perf tests ( #586 )
...
[ROCm/rccl commit: 686dbc8bc6
2022-07-21 14:59:34 -06:00
18d9fd1b8f
Removing redundant LOAD and STORE on primitives plus adding some atomics ( #585 )
...
[ROCm/rccl commit: 451c287aa6
2022-07-21 13:04:57 -06:00
088d62ff58
Merge pull request #580 from hubertlu-tw/develop
...
Enhancement of RCCL logging information for topology-aware utilities
[ROCm/rccl commit: 6dd090917a
2022-07-15 15:16:37 -07:00
a9e0333dba
intercept SIGUSR2 in RCCL
...
add support for intercepting SIGUSR2 in RCCL. This signal will
not terminate the execution of the application, but print the stacktrace
of the process that the signal was sent to instead.
[ROCm/rccl commit: 2b1d5d3bc1
2022-07-15 16:28:46 +00:00
e1aae026bf
Remove redundant LOAD/STORE usage in LL initialization
...
[ROCm/rccl commit: 77c2bef952
2022-07-14 00:40:36 +00:00
2a1d472a20
updated readme to reflect the newer tests
...
[ROCm/rccl commit: 5950942738
2022-07-13 16:08:28 +00:00
9a9d9cb29b
README.md: add CMAKE_PREFIX_PATH to build steps ( #581 )
...
[ROCm/rccl commit: 314da5a485
2022-07-12 11:32:07 -07:00
e13eb2eab9
Enhancement of RCCL logging information for topology-aware utilities
...
[ROCm/rccl commit: a1842df858
2022-07-11 19:01:10 +00:00
91154e8df9
2.13.4-1
...
Optimize CUDA graph launch; avoid launching a CPU callback for
intra-node operations.
Simplify kernel common code to improve the latency of send/recv
operations.
Strengthen CUDA streams semantics.
Change NET API to v6, to add dmabuf support.
Add ncclGetLastError() function.
Add ncclRemoteError code and use it for remote network errors.
Support the use of a different NCCL_NET parameter per communicator.
Add support for SHM and P2P transfers using cudaMemcpy.
[ROCm/rccl commit: 19ab67d172
2022-07-11 08:10:34 -07:00
c129677fe0
Skip HDP cache flush for gfx90a ( #578 )
...
* Skip HDP cache flush for gfx90a
* Remove extra debug print
[ROCm/rccl commit: 8c3c8b78c0
2022-07-08 10:13:32 -07:00
659cd52d5c
Add more constraints to enable GDR ( #579 )
...
* Add more constraints to enable GDR
* Revert deleted line
[ROCm/rccl commit: aa0d7ca882
2022-07-08 09:52:27 -07:00
bf15ad1d72
Reduce AlltoAll port usage in send/recv proxy ( #577 )
...
* Reduce AlltoAll port usage when connecting proxy
Reuse socket ports when connecting proxies in AlltoAll.
Existing port usage in AlltoAll is O(n) for recv and O(n) for send,
reusing socket ports in server or client side will make one of them
O(1), reusing both will reduce the total port usage to O(1) and enables
AlltoAll in >64 MI200 nodes.
* Update changelog accordingly
Update changelog accordingly.
[ROCm/rccl commit: 80f53cc171
2022-07-07 16:15:52 -07:00
Wen-Heng (Jack) Chung
Wenkai Du
Edgar Gabriel
Wenkai Du
Edgar Gabriel
Gilbert Lee
gilbertlee-amd
Lauren Wrubleski
Min Si
akolliasAMD
Cosmic Fusion
Ching-Hsiang Chu
Ziyue Yang
arvindcheru
Hubert Lu
akolliasAMD
hubertlu-tw
Sylvain Jeaugey
Yifan Xiong