4d979ce13d
Fix PCIe gen detection ( #437 )
...
* Fix PCIe gen detection
* Update profiling support
[ROCm/rccl commit: 2508507d0a
2021-10-15 08:23:50 -07:00
b587b55c2e
Add more Rome models ( #434 )
...
* Add more Rome models
* Update models and tuning
* Update tuning
[ROCm/rccl commit: 2249a1d9d3
2021-10-12 08:23:20 -07:00
d377c4dcc6
Add another Rome model ( #431 )
...
[ROCm/rccl commit: e0053311c0
2021-10-06 08:17:12 -07:00
91eca0d7d2
Trim NICs when all GPUs are connected by XGMI ( #430 )
...
* Trim NICs when all GPUs are connected by XGMI
* Only enable clique with maximum of 2 hops
[ROCm/rccl commit: 29c729d8b6
2021-10-05 18:27:43 -07:00
a44ed86b46
Merge remote-tracking branch 'origin/develop' into 2.10.3
...
[ROCm/rccl commit: 5b72727670
2021-09-15 10:33:25 -07:00
1eaf495391
Remove extra L1 cache invalidate and restore __ATOMIC_SEQ_CST atomics ( #426 )
...
[ROCm/rccl commit: 5ae3f3f954
2021-09-14 18:30:16 -07:00
d6064367f0
Merge remote-tracking branch 'origin/develop' into 2.10.3
...
[ROCm/rccl commit: 3667d308ab
2021-09-13 17:32:48 -07:00
f4387b2954
Use relaxed atomics and add sleep and wakeup in barrier loop ( #425 )
...
* Use relaxed atomics and add sleep and wakeup in barrier loop
* atomicAdd in ROCm 4.3 only support unsigned long long
* Switch to atomicAdd and atomicExch in more places
* Restore LOAD/STORE define to __ATOMIC_SEQ_CST
* Restore atomic for sizes FIFO
[ROCm/rccl commit: 020484bf40
2021-09-13 17:03:49 -07:00
8acdb77cc0
Merge remote-tracking branch 'origin/develop' into 2.10.3
...
[ROCm/rccl commit: 8ee2b7932a
2021-09-13 15:51:53 -07:00
9ffeb41fe1
Update tuning table ( #424 )
...
[ROCm/rccl commit: ef432e48e1
2021-09-13 08:39:01 -07:00
d2580c8cf5
Improve barrier implementation
...
[ROCm/rccl commit: adb8d63352
2021-09-08 16:14:32 -05:00
d75504e9dc
Remove atomic from profiling
...
[ROCm/rccl commit: 31bd4236f1
2021-09-08 14:20:32 -05:00
95b37d97cf
Merge remote-tracking branch 'nccl/master' into 2.10.3
...
[ROCm/rccl commit: 21969d7f89
2021-09-01 16:34:44 -07:00
04553b802a
Fix to https://github.com/NVIDIA/nccl/issues/560
...
ncclGroup's containing operations of mixed datatype, element, or collective
would induce crash.
[ROCm/rccl commit: 5f2f2f670f
2021-08-31 15:50:05 -07:00
b9508a6aba
Implement NIC identification and remapping ( #420 )
...
* Add 1H16P GPU model
* Implement NIC identification and remapping
* Revert "Sort IB devices based on device name (#413 )"
This reverts commit de0c586bad#361 )"
This reverts commit fa690c47a05c8380ff5b
2021-08-24 09:42:04 -07:00
e0fd3c6ba3
Modifying ReduceOrCopyMulti to accept number of preOp source to support clique-based kernels
...
[ROCm/rccl commit: ae13d2a354
2021-08-11 11:00:34 -05:00
eaf54184bf
Improve clique kernel performance by increasing unroll
...
[ROCm/rccl commit: a4929465c5
2021-08-26 18:06:09 -07:00
a5e85a66fd
Fix typo that affects clique kernels
...
[ROCm/rccl commit: 574f0aca53
2021-08-26 11:10:58 -07:00
bd03ec2a45
Unit Test: support ncclAvg
...
[ROCm/rccl commit: 1faff323b4
2021-08-25 14:15:54 -07:00
da7ac55e6f
Fix kernel data trace
...
[ROCm/rccl commit: 60ca7484c0
2021-08-24 14:02:53 -07:00
4fd7a14087
Merge remote-tracking branch 'origin/develop' into 2.10.3
...
[ROCm/rccl commit: d5f93649ff
2021-08-24 09:49:47 -07:00
57518da006
Add gfx908 VM model ( #418 )
...
[ROCm/rccl commit: 5f15ed6e3e
2021-08-10 08:55:11 -07:00
51198b536d
Use noinline for kernel functions
...
[ROCm/rccl commit: 707c687090
2021-08-06 09:15:04 -07:00
79bc1e3fde
Fix incorrect network proxy received bytes reporting
...
[ROCm/rccl commit: 01d3b20a66
2021-08-05 17:45:48 -07:00
6e997354e8
Merge branch 'develop' into 2.10.3
...
[ROCm/rccl commit: babbd1047b
2021-08-04 09:45:22 -07:00
de0c586bad
Sort IB devices based on device name ( #413 )
...
[ROCm/rccl commit: 2d0ed8dff6
2021-08-03 15:32:41 -07:00
4b89e98675
Merge remote-tracking branch 'nccl/master' into 2.10.3
...
[ROCm/rccl commit: bf2339f93e
2021-07-30 16:23:14 -07:00
4b082ceb32
XGMI connection is always prioritized over NET regardless of hops ( #412 )
...
[ROCm/rccl commit: 3e27227562
2021-07-29 11:12:42 -07:00
faea6ead5c
Query XGMI links from xml and adjust gfx906 channel usage ( #410 )
...
[ROCm/rccl commit: 818cdb16a8
2021-07-27 17:32:41 -07:00
8fbeb14175
topo_expl: fix build after switching to rocm-smi-lib ( #405 )
...
* topo_expl: fix build after switching to rocm-smi-lib
* Use minimal of 4 channels for gfx908
[ROCm/rccl commit: 135d47d125
2021-07-27 08:30:08 -07:00
bbfc0c85d2
Skipping unnecessary functions in Doxygen by marking as internal ( #353 ) ( #406 )
...
(cherry picked from commit 198e17608ef40acf6b9515c6831d4a26786aabd6)
Co-authored-by: saadrahim <44449863+saadrahim@users.noreply.github.com >
[ROCm/rccl commit: dfc62d5fbb
2021-07-24 11:04:27 -07:00
2ab26a2ff9
Add more info to RCCL logging for topo-aware optim.
...
[ROCm/rccl commit: bd6dbca8fb
2021-07-22 09:52:39 -07:00
58a9f19dc3
2.10.3-1
...
Add support for bfloat16.
Add ncclAvg reduction operation.
Improve performance for aggregated operations.
Improve performance for tree.
Improve network error reporting.
Add NCCL_NET parameter to force a specific network.
Add NCCL_IB_QPS_PER_CONNECTION parameter to split IB traffic onto multiple queue pairs.
Fix topology detection error in WSL2.
Fix proxy memory elements affinity (improve alltoall performance).
Fix graph search on cubemesh topologies.
Fix hang in cubemesh during NVB connections.
[ROCm/rccl commit: 7e51592129
2021-07-08 14:30:14 -07:00
71dfc3978e
Use rocm_smi_lib for getting topology information ( #402 )
...
* Use rocm_smi_lib for getting topology information
* Add rocm-smi-lib dependency to RCCL package
[ROCm/rccl commit: 56155ff5b6
2021-07-08 13:23:11 -07:00
90ae176437
Fixes for NCCL_MAX_NCHANNELS and topo_expl ( #398 )
...
[ROCm/rccl commit: fa6d7e9a63
2021-06-22 08:41:49 -07:00
f82f99f533
Select sendrecv path based on collective data size ( #391 )
...
* Select sendrecv path based on collective data size
* Add comments on packing and unpacking group field
* Toggling RCCL_P2P_NET_DISABLE in combined calls unit tests
[ROCm/rccl commit: 6dcae8a459
2021-06-10 17:51:04 -07:00
5bebcb0015
Setup collectives threshold for enabling intranet ( #387 )
...
* Setup collectives threshold for enabling intranet
* Use separate operation counters for coll and p2p
[ROCm/rccl commit: b815a2800f
2021-06-09 13:24:26 -07:00
ab9b9151d2
Add support for another Rome model ( #385 )
...
[ROCm/rccl commit: c2064adcc7
2021-06-08 13:58:20 -07:00
c8a432dc25
Allow intranode use of network connection ( #383 )
...
* Allow intranode use of network connection
* Checking for graph for null pointer
[ROCm/rccl commit: a3a8c2d56b
2021-06-08 07:37:59 -07:00
4b31e521e9
Add option to enable multiple SAT in SHARP ( #380 )
...
* Add option to enable multiple SAT in SHARP
* Extend number of NICs to 16
[ROCm/rccl commit: 961922ea02
2021-06-03 19:45:18 -07:00
b154b532a2
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: e3abf1c2ec
2021-05-25 20:52:15 -07:00
aa95cc6102
Update Rome models matching ( #376 )
...
[ROCm/rccl commit: 4c83adb75c
2021-05-25 10:12:40 -07:00
ea17b05518
Tweak clique channel usage for gfx908 ( #374 )
...
[ROCm/rccl commit: 8e817ecd6d
2021-05-21 15:36:21 -06:00
92bcdcf5b0
Correction on max number of groups ( #373 )
...
[ROCm/rccl commit: 50da1b48af
2021-05-20 08:58:45 -07:00
b27490d38d
Use fixed segment size for sendrecv ( #369 )
...
[ROCm/rccl commit: 8cde34be51
2021-05-19 08:25:26 -07:00
87423792fd
Tune clique-based AllReduce for device type 908 ( #372 )
...
* Changing switch-over point for clique-based AllReduce
[ROCm/rccl commit: ddceadc313
2021-05-18 15:36:07 -06:00
e0edd3d5e4
Merge remote-tracking branch 'nccl/master' into 2.9.8
...
[ROCm/rccl commit: 87727383fe
2021-05-17 10:15:16 -07:00
6bce88058a
Multiprocess unit test various fixes ( #367 )
...
* Re-enabling mp unit tests
* Fixing shared memory leak and other bugs related to shared mem for MP unit tests
* Revert 43bfbfc97bf9edbae1f386d461439091618ff8ed
* Further tightening up unlinks
* Moving test check macros to separate header file
* Tightening up shared memory unlinking for clique kernels, add munmap for host barrier for MP unit tests
* Updating new MP unit test
* Fixing mqueue bug
* Fixing memory leak in MP unit tests
[ROCm/rccl commit: 0b2bfdd6d8
2021-05-14 09:38:49 -06:00
fa690c47a0
Allow user to link layer with RCCL_IB_HCA_SKIP_LINK_LAYER ( #361 )
...
To skip Infiniband, set RCCL_IB_HCA_SKIP_LINK_LAYER=1.
To skip Ethernet, RCCL_IB_HCA_SKIP_LINK_LAYER=2.
[ROCm/rccl commit: caf5c9992a
2021-05-12 14:14:53 -07:00
3dd3a1ca66
2.9.9-1
...
Fix crash when setting NCCL_MAX_P2P_NCHANNELS below nchannels.
Fix hang during sendrecv dynamic NVB connection establishment on
cubemesh topologies.
Add environment variable to only use SHARP on communicators beyond
a given number of ranks.
Add debug subsystem to trace memory allocations.
Fix compilation with TRACE=1. (Issue #505 )
[ROCm/rccl commit: 3fec2fa5ee
2021-05-12 11:09:31 -07:00
Wenkai Du
Wenkai Du
John Bachan
Gilbert Lee
Lu
Ke Wen
gilbertlee-amd
Stanley Tsang
Sylvain Jeaugey