7148c0aa7b
add binary tree
...
In addition, introduce the ability to have 2 trees at the same time.
Only for allreduce at the moment.
[ROCm/rccl commit: 65e2ae20e5
2022-09-13 20:52:32 +00:00
35872115f8
Updating stream caching ( #614 )
...
- Adding non-captured hipStream for use in setup
[ROCm/rccl commit: dd56135a9a
2022-09-09 16:30:15 -06:00
fe99249cde
Enable LL128 protocol support ( #605 )
...
* Enable LL128 protocol support
* Use shared memory object directly when possible
[ROCm/rccl commit: 7bbce085cc
2022-09-08 14:45:27 -07:00
25ba51fe83
Fix compilation issues with buck ( #610 )
...
* Fix compilation warning with -Wmisleading-indentation
When compile with -Wmisleading-indentation, it reports warning:
misleading indentation; statement is not part of the previous 'if'
This patch fixes it
* Avoid relative include file path
We don't need relative include file paths for src/graph/*.h
since src/ is already in CMake include_directories
[ROCm/rccl commit: 2b57751abb
2022-09-07 09:56:05 -06:00
616cb39a0b
Adding opt-in hipGraph support for RCCL via RCCL_ENABLE_HIPGRAPH ( #608 )
...
Adding opt-in hipGraph support via RCCL_ENABLE_HIPGRAPH
[ROCm/rccl commit: 47b2fc3a30
2022-09-06 10:29:46 -06:00
2cd63dac42
added stream synch after hipMemset ( #609 )
...
[ROCm/rccl commit: 06bce9d0c9
2022-08-30 16:18:37 -06:00
f18868f439
Use hipExtLaunchKernel when not using graph and not in group mode ( #606 )
...
[ROCm/rccl commit: c9f2fe1f65
2022-08-26 13:40:37 -07:00
6bb871c986
Merge pull request #598 from edgargabriel/topic/tree-multirank
...
Expand ncclTreeBasePostset for multi-rank
[ROCm/rccl commit: e739c62a53
2022-08-24 08:28:34 -05:00
56ea2c4be5
Use non-temporal access for slow path ( #602 )
...
[ROCm/rccl commit: 88487a62bb
2022-08-23 08:21:51 -07:00
aa6d450f35
fix channelcount for multi-rank scenario
...
[ROCm/rccl commit: 4141ec1151
2022-08-22 19:09:22 +00:00
1d55fe756c
Simple tree changes ( #599 )
...
changed treebase to create basic balanced tree
[ROCm/rccl commit: 3c1b1ec8c8
2022-08-19 13:51:49 -06:00
1dcf1da5ca
fix error: use of undeclared identifier 'free'
...
include stdlib.h to fix compilation error in rccl :
[39/58] Building CXX object CMakeFiles/rccl.dir/src/misc/signals.cc.o
FAILED: CMakeFiles/rccl.dir/src/misc/signals.cc.o
/opt/rocm/bin/hipcc -DENABLE_COLLTRACE -DHAVE_BFD -DHAVE_CPLUS_DEMANGLE -DUSE_ROCM_SMI64CONFIG -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -Drccl_EXPORTS -I/home/cosmo/build/flgrwqa/build/include -I/home/cosmo/build/flgrwqa/build/include/rccl -I/home/cosmo/build/flgrwqa/rccl/src -I/home/cosmo/build/flgrwqa/rccl/src/include -I/home/cosmo/build/flgrwqa/rccl/src/collectives -I/home/cosmo/build/flgrwqa/rccl/src/collectives/device -I/opt/hsa/include -fPIC -fvisibility=hidden -fgpu-rdc -parallel-jobs=8 -Wno-format-nonliteral -x hip --offload-arch=gfx803 --offload-arch=gfx900:xnack- --offload-arch=gfx906:xnack- --offload-arch=gfx908:xnack- --offload-arch=gfx90a:xnack- --offload-arch=gfx90a:xnack+ --offload-arch=gfx1030 -std=c++14 -MD -MT CMakeFiles/rccl.dir/src/misc/signals.cc.o -MF CMakeFiles/rccl.dir/src/misc/signals.cc.o.d -o CMakeFiles/rccl.dir/src/misc/signals.cc.o -c /home/cosmo/build/flgrwqa/rccl/src/misc/signals.cc
In file included from /home/cosmo/build/flgrwqa/rccl/src/misc/signals.cc:8:
/home/cosmo/build/flgrwqa/rccl/src/include/BfdBacktrace.hpp:138:9: error: use of undeclared identifier 'free'
free(file->syms);
^
/home/cosmo/build/flgrwqa/rccl/src/include/BfdBacktrace.hpp:155:5: error: use of undeclared identifier 'free'
free(file->syms);
^
[ROCm/rccl commit: 080fc2d9d6
2022-08-19 20:25:06 +03:00
c2e9ada40b
Repurpose profiling implementation to simple timestamps tracing ( #600 )
...
[ROCm/rccl commit: 14b8ff153f
2022-08-18 15:34:46 -07:00
6c3f1366e8
Add XGMI sys type and clean up detection code ( #597 )
...
[ROCm/rccl commit: f5c0b243a8
2022-08-12 09:52:29 -07:00
478d8312b8
Improve alignment and tuning for Pivot A2A algorithm ( #593 )
...
* Improve alignment and tuning for Pivot A2A algorithm
* enable pivot a2a by default
[ROCm/rccl commit: f6b9686482
2022-08-05 19:40:19 -07:00
6fb5c5d5e3
minor latency tuning ( #591 )
...
* minor tuning for tree ll
[ROCm/rccl commit: 4cecdc9be5
2022-08-03 15:07:44 -06:00
f70830d629
Revert "Use nontemporal in slow path and add XGMI sys type ( #575 )" ( #590 )
...
This reverts commit e04bba619a9089e68a99
2022-08-02 09:31:53 -07:00
d5ca0be51f
Fixed issue with attomicEXCH creating errors on multi node runs ( #587 )
...
[ROCm/rccl commit: 254208e7dd
2022-07-22 11:32:49 -06:00
18d9fd1b8f
Removing redundant LOAD and STORE on primitives plus adding some atomics ( #585 )
...
[ROCm/rccl commit: 451c287aa6
2022-07-21 13:04:57 -06:00
088d62ff58
Merge pull request #580 from hubertlu-tw/develop
...
Enhancement of RCCL logging information for topology-aware utilities
[ROCm/rccl commit: 6dd090917a
2022-07-15 15:16:37 -07:00
a9e0333dba
intercept SIGUSR2 in RCCL
...
add support for intercepting SIGUSR2 in RCCL. This signal will
not terminate the execution of the application, but print the stacktrace
of the process that the signal was sent to instead.
[ROCm/rccl commit: 2b1d5d3bc1
2022-07-15 16:28:46 +00:00
e1aae026bf
Remove redundant LOAD/STORE usage in LL initialization
...
[ROCm/rccl commit: 77c2bef952
2022-07-14 00:40:36 +00:00
e13eb2eab9
Enhancement of RCCL logging information for topology-aware utilities
...
[ROCm/rccl commit: a1842df858
2022-07-11 19:01:10 +00:00
c129677fe0
Skip HDP cache flush for gfx90a ( #578 )
...
* Skip HDP cache flush for gfx90a
* Remove extra debug print
[ROCm/rccl commit: 8c3c8b78c0
2022-07-08 10:13:32 -07:00
659cd52d5c
Add more constraints to enable GDR ( #579 )
...
* Add more constraints to enable GDR
* Revert deleted line
[ROCm/rccl commit: aa0d7ca882
2022-07-08 09:52:27 -07:00
bf15ad1d72
Reduce AlltoAll port usage in send/recv proxy ( #577 )
...
* Reduce AlltoAll port usage when connecting proxy
Reuse socket ports when connecting proxies in AlltoAll.
Existing port usage in AlltoAll is O(n) for recv and O(n) for send,
reusing socket ports in server or client side will make one of them
O(1), reusing both will reduce the total port usage to O(1) and enables
AlltoAll in >64 MI200 nodes.
* Update changelog accordingly
Update changelog accordingly.
[ROCm/rccl commit: 80f53cc171
2022-07-07 16:15:52 -07:00
e04bba619a
Use nontemporal in slow path and add XGMI sys type ( #575 )
...
* Use nontemporal in slow path and add XGMI sys type
* Clean up XGMI detection
[ROCm/rccl commit: b250c01cbe
2022-07-06 07:58:41 -07:00
2f4aea93e0
Fix GPU to NIC mapping in tree ( #573 )
...
* Fix GPU to NIC mapping in tree
* Update tuning table
[ROCm/rccl commit: 00af1f64e9
2022-07-03 20:52:52 -07:00
cb5ae7224e
Adding git hash info to version output line ( #572 )
...
[ROCm/rccl commit: a89a9966aa
2022-06-28 16:42:51 -06:00
915a9d3934
Do not set NET GDR level automatically ( #571 )
...
[ROCm/rccl commit: 9a285b5e1d
2022-06-23 16:28:28 -07:00
784b12bf75
Use different atomics to check flags in kernel ( #568 )
...
[ROCm/rccl commit: c3bb9e70d0
2022-06-23 09:16:41 -07:00
2b418b5dee
Add Feature - Add NPKit Support in RCCL ( #564 )
...
* apply npkit
* fix bug
* add npkit in readme
[ROCm/rccl commit: 6e93fafdc3
2022-06-20 14:30:19 -07:00
0fb000932f
Change default nchannels per peer ( #563 )
...
[ROCm/rccl commit: f274c865c1
2022-06-13 06:39:05 -07:00
8953f5b5ca
Introduce multi-rank support per device.
...
This is a single commit of the source code changes required to
introduce support for multiple ranks per device.
A new interface (ncclCommRankInitMulti) has to be used to make use of
this new feature.
[ROCm/rccl commit: 0336ffdf70
2022-06-10 14:23:12 +00:00
11a6cdd52f
Fix P2P scheduling ( #560 )
...
[ROCm/rccl commit: 5cb2aca3d9
2022-06-06 13:32:28 -07:00
f2dbc77afe
Enable timing profile option ( #558 )
...
[ROCm/rccl commit: 7a6c6927ae
2022-06-03 07:05:13 -07:00
0b55e01ef3
Merge remote-tracking branch 'ncclRepo/master' into develop
...
[ROCm/rccl commit: e0864e7093
2022-06-02 15:27:24 +00:00
1e36b432f1
Revert chunksteps changes ( #555 )
...
[ROCm/rccl commit: eef812bed7
2022-05-31 14:45:51 -07:00
5becf1669f
Add another Rome model ( #553 )
...
* Add another Rome model
* Add option to force enable intranet on single node
* Limit p2p channels to number of ranks
* Refine p2p channels handling
[ROCm/rccl commit: ef499c4810
2022-05-31 11:31:30 -07:00
a03ab8e752
code cleanup ( #554 )
...
[ROCm/rccl commit: a0a686e74c
2022-05-31 09:59:36 -04:00
2c125ce6ed
Update Rome model ( #552 )
...
[ROCm/rccl commit: c5b77121f0
2022-05-26 09:59:23 -07:00
22dc8bd246
Added creation of new tree and added switch for using treesplit for specific cases ( #551 )
...
[ROCm/rccl commit: 98f0809a39
2022-05-25 18:55:14 -04:00
a2a4888497
Moving opt-in custom signal handler from UnitTests into RCCL ( #550 )
...
* Enable via RCCL_ENABLE_SIGNALHANDLER=1
[ROCm/rccl commit: 700b473211
2022-05-20 09:56:38 -06:00
86e8797602
Add switch for pivot alltoall kernel ( #549 )
...
[ROCm/rccl commit: 6707a270b1
2022-05-17 18:14:04 -07:00
b30b8becea
Refine and add new Rome models ( #548 )
...
[ROCm/rccl commit: 283dc86a73
2022-05-17 08:23:59 -07:00
1c5734046d
2.12.12-1
...
Improve allreduce performance when we have more than one network interface per
GPU and we need to use PXN to close rings.
Add support for PCI Gen5 on 5.4 kernels.
Fix crash when setting NCCL_SET_THREAD_NAME.
Fix random crash in init due to uninitialized struct.
Fix hang on cubemesh topologies.
Add P2P_DIRECT_DISABLE parameter to disable direct access to pointers within a
process.
[ROCm/rccl commit: 7aa1c46fd5
2022-05-13 00:26:57 -07:00
b37180b7ed
Improve LL performance ( #546 )
...
* Improve LL performance
* Add split barriers for LL
[ROCm/rccl commit: c9919e0e35
2022-05-10 13:32:10 -07:00
347ea354c2
Update tuning parameters
...
[ROCm/rccl commit: 83fd4f70e7
2022-04-18 16:04:04 -07:00
67e7e6507e
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: d28e1cb44f
2022-04-18 11:15:25 -07:00
58b2c1ec9c
Fix random segfault ( #537 )
...
[ROCm/rccl commit: fd2f1b3b88
2022-04-15 14:32:11 -07:00
Edgar Gabriel
gilbertlee-amd
Wenkai Du
Min Si
akolliasAMD
Edgar Gabriel
Cosmic Fusion
Ziyue Yang
Hubert Lu
hubertlu-tw
Yifan Xiong
Aristotelis
Sylvain Jeaugey
Wenkai Du