AI/rocm-systems
2
0
Форкнуть 0
Код Задачи Запросы на слияние Действия Пакеты Проекты Релизы Вики Активность
76 333 коммитов 3 Ветки 0 Теги
develop
Граф коммитов

1071 Коммитов

Автор SHA1 Сообщение Дата
Wenkai Du df59f64e3f Support different protocols and algorithms in all reduce only build (#455) 
* Support different protocols and algorithms in all reduce only build

* Restore deleted line in error

[ROCm/rccl commit: 29170a8b5f]
 2021-11-02 08:39:08 -07:00
Wenkai Du a11b55a37f Check rocm_smi64Config.h on older ROCm build (#452) 
[ROCm/rccl commit: 4643a17f83]
 2021-10-28 07:26:28 -07:00
Wenkai Du b0b2e27df3 Rework kernel launch code (#449) 
[ROCm/rccl commit: d221fb672a]
 2021-10-28 07:26:11 -07:00
Wenkai Du 747216e2b2 Enable timing profiling mode (#447) 
[ROCm/rccl commit: ec36c4c326]
 2021-10-27 08:21:48 -07:00
Stanley Tsang a6feafd5dc Build AllReduce only mode (#443) 
* Initial commit of all_reduce_only support

* Working AllReduce only build

* Removing printfs and restoring release build

* Restore P2P index

* Updates to build_allreduce_only mode.

* cleaning up macro ifdefs

[ROCm/rccl commit: 7e55b211c5]
 2021-10-26 17:36:46 -06:00
Wenkai Du 15143b1cfb Query XGMI link count through rocm_smi_lib API (#442) 
[ROCm/rccl commit: 14a184eb67]
 2021-10-26 10:30:20 -07:00
Wenkai Du aac9d57633 Fix collnet tuning parameters (#441) 
[ROCm/rccl commit: b4cefc05ed]
 2021-10-20 20:45:36 -07:00
Wenkai Du 4d979ce13d Fix PCIe gen detection (#437) 
* Fix PCIe gen detection

* Update profiling support

[ROCm/rccl commit: 2508507d0a]
 2021-10-15 08:23:50 -07:00
Wenkai Du b587b55c2e Add more Rome models (#434) 
* Add more Rome models

* Update models and tuning

* Update tuning

[ROCm/rccl commit: 2249a1d9d3]
 2021-10-12 08:23:20 -07:00
Wenkai Du d377c4dcc6 Add another Rome model (#431) 
[ROCm/rccl commit: e0053311c0]
 2021-10-06 08:17:12 -07:00
Wenkai Du 91eca0d7d2 Trim NICs when all GPUs are connected by XGMI (#430) 
* Trim NICs when all GPUs are connected by XGMI

* Only enable clique with maximum of 2 hops

[ROCm/rccl commit: 29c729d8b6]
 2021-10-05 18:27:43 -07:00
John Bachan 389213c7e8 Fix compilation failure in "src/enqueue.cc" on older GCC because of 
missing `#include <cstring>`.


[ROCm/rccl commit: 30ca3fcacf]
 2021-09-23 09:55:16 -07:00
Sylvain Jeaugey 761812841b Fix Collnet when GDR is disabled 
[ROCm/rccl commit: 4ec992fab7]
 2021-09-22 05:19:16 -07:00
Ke Wen e51f2a83da 2.11.4-1 
Add new API for creating a reduction operation which multiplies the input by a rank-specific scalar before doing an inter-rank summation (see: ncclRedOpCreatePreMulSum).
Improve CollNet (SHARP) performance of ncclAllReduce when captured in a CUDA Graph via user buffer registration.
Add environment variable NCCL_NET_PLUGIN="<suffix>" to allow user to choose among multiple NCCL net plugins by substituting into "libnccl-net-<suffix>.so".
Fix memory leak of NVB connections.
Fix topology detection of IB Virtual Functions (SR-IOV).


[ROCm/rccl commit: e11238b302]
 2021-09-08 16:06:23 -07:00
John Bachan 04553b802a Fix to https://github.com/NVIDIA/nccl/issues/560 
ncclGroup's containing operations of mixed datatype, element, or collective
would induce crash.


[ROCm/rccl commit: 5f2f2f670f]
 2021-08-31 15:50:05 -07:00
Wenkai Du a44ed86b46 Merge remote-tracking branch 'origin/develop' into 2.10.3 
[ROCm/rccl commit: 5b72727670]
 2021-09-15 10:33:25 -07:00
Wenkai Du 1eaf495391 Remove extra L1 cache invalidate and restore __ATOMIC_SEQ_CST atomics (#426) 
[ROCm/rccl commit: 5ae3f3f954]
 2021-09-14 18:30:16 -07:00
Wenkai Du d6064367f0 Merge remote-tracking branch 'origin/develop' into 2.10.3 
[ROCm/rccl commit: 3667d308ab]
 2021-09-13 17:32:48 -07:00
Wenkai Du f4387b2954 Use relaxed atomics and add sleep and wakeup in barrier loop (#425) 
* Use relaxed atomics and add sleep and wakeup in barrier loop

* atomicAdd in ROCm 4.3 only support unsigned long long

* Switch to atomicAdd and atomicExch in more places

* Restore LOAD/STORE define to __ATOMIC_SEQ_CST

* Restore atomic for sizes FIFO

[ROCm/rccl commit: 020484bf40]
 2021-09-13 17:03:49 -07:00
Wenkai Du 8acdb77cc0 Merge remote-tracking branch 'origin/develop' into 2.10.3 
[ROCm/rccl commit: 8ee2b7932a]
 2021-09-13 15:51:53 -07:00
Wenkai Du 9ffeb41fe1 Update tuning table (#424) 
[ROCm/rccl commit: ef432e48e1]
 2021-09-13 08:39:01 -07:00
Wenkai Du d2580c8cf5 Improve barrier implementation 
[ROCm/rccl commit: adb8d63352]
 2021-09-08 16:14:32 -05:00
Wenkai Du d75504e9dc Remove atomic from profiling 
[ROCm/rccl commit: 31bd4236f1]
 2021-09-08 14:20:32 -05:00
Wenkai Du 95b37d97cf Merge remote-tracking branch 'nccl/master' into 2.10.3 
[ROCm/rccl commit: 21969d7f89]
 2021-09-01 16:34:44 -07:00
Wenkai Du b9508a6aba Implement NIC identification and remapping (#420) 
* Add 1H16P GPU model

* Implement NIC identification and remapping

* Revert "Sort IB devices based on device name (#413)"

This reverts commit de0c586bad.

* Fix permute and check order

* Correction on IB speed reporting

* Revert "Allow user to link layer with RCCL_IB_HCA_SKIP_LINK_LAYER (#361)"

This reverts commit fa690c47a0.

[ROCm/rccl commit: 5c8380ff5b]
 2021-08-24 09:42:04 -07:00
Gilbert Lee e0fd3c6ba3 Modifying ReduceOrCopyMulti to accept number of preOp source to support clique-based kernels 
[ROCm/rccl commit: ae13d2a354]
 2021-08-11 11:00:34 -05:00
Wenkai Du eaf54184bf Improve clique kernel performance by increasing unroll 
[ROCm/rccl commit: a4929465c5]
 2021-08-26 18:06:09 -07:00
Wenkai Du a5e85a66fd Fix typo that affects clique kernels 
[ROCm/rccl commit: 574f0aca53]
 2021-08-26 11:10:58 -07:00
Wenkai Du bd03ec2a45 Unit Test: support ncclAvg 
[ROCm/rccl commit: 1faff323b4]
 2021-08-25 14:15:54 -07:00
Wenkai Du da7ac55e6f Fix kernel data trace 
[ROCm/rccl commit: 60ca7484c0]
 2021-08-24 14:02:53 -07:00
Wenkai Du 4fd7a14087 Merge remote-tracking branch 'origin/develop' into 2.10.3 
[ROCm/rccl commit: d5f93649ff]
 2021-08-24 09:49:47 -07:00
Wenkai Du 57518da006 Add gfx908 VM model (#418) 
[ROCm/rccl commit: 5f15ed6e3e]
 2021-08-10 08:55:11 -07:00
Wenkai Du 51198b536d Use noinline for kernel functions 
[ROCm/rccl commit: 707c687090]
 2021-08-06 09:15:04 -07:00
Wenkai Du 79bc1e3fde Fix incorrect network proxy received bytes reporting 
[ROCm/rccl commit: 01d3b20a66]
 2021-08-05 17:45:48 -07:00
Wenkai Du 6e997354e8 Merge branch 'develop' into 2.10.3 
[ROCm/rccl commit: babbd1047b]
 2021-08-04 09:45:22 -07:00
Wenkai Du de0c586bad Sort IB devices based on device name (#413) 
[ROCm/rccl commit: 2d0ed8dff6]
 2021-08-03 15:32:41 -07:00
Wenkai Du 4b89e98675 Merge remote-tracking branch 'nccl/master' into 2.10.3 
[ROCm/rccl commit: bf2339f93e]
 2021-07-30 16:23:14 -07:00
Wenkai Du 4b082ceb32 XGMI connection is always prioritized over NET regardless of hops (#412) 
[ROCm/rccl commit: 3e27227562]
 2021-07-29 11:12:42 -07:00
Wenkai Du faea6ead5c Query XGMI links from xml and adjust gfx906 channel usage (#410) 
[ROCm/rccl commit: 818cdb16a8]
 2021-07-27 17:32:41 -07:00
Wenkai Du 8fbeb14175 topo_expl: fix build after switching to rocm-smi-lib (#405) 
* topo_expl: fix build after switching to rocm-smi-lib

* Use minimal of 4 channels for gfx908

[ROCm/rccl commit: 135d47d125]
 2021-07-27 08:30:08 -07:00
Wenkai Du bbfc0c85d2 Skipping unnecessary functions in Doxygen by marking as internal (#353) (#406) 
(cherry picked from commit 198e17608ef40acf6b9515c6831d4a26786aabd6)

Co-authored-by: saadrahim <44449863+saadrahim@users.noreply.github.com>

[ROCm/rccl commit: dfc62d5fbb]
 2021-07-24 11:04:27 -07:00
Lu 2ab26a2ff9 Add more info to RCCL logging for topo-aware optim. 
[ROCm/rccl commit: bd6dbca8fb]
 2021-07-22 09:52:39 -07:00
Ke Wen 58a9f19dc3 2.10.3-1 
Add support for bfloat16.
Add ncclAvg reduction operation.
Improve performance for aggregated operations.
Improve performance for tree.
Improve network error reporting.
Add NCCL_NET parameter to force a specific network.
Add NCCL_IB_QPS_PER_CONNECTION parameter to split IB traffic onto multiple queue pairs.
Fix topology detection error in WSL2.
Fix proxy memory elements affinity (improve alltoall performance).
Fix graph search on cubemesh topologies.
Fix hang in cubemesh during NVB connections.


[ROCm/rccl commit: 7e51592129]
 2021-07-08 14:30:14 -07:00
Wenkai Du 71dfc3978e Use rocm_smi_lib for getting topology information (#402) 
* Use rocm_smi_lib for getting topology information

* Add rocm-smi-lib dependency to RCCL package

[ROCm/rccl commit: 56155ff5b6]
 2021-07-08 13:23:11 -07:00
Wenkai Du 90ae176437 Fixes for NCCL_MAX_NCHANNELS and topo_expl (#398) 
[ROCm/rccl commit: fa6d7e9a63]
 2021-06-22 08:41:49 -07:00
Wenkai Du f82f99f533 Select sendrecv path based on collective data size (#391) 
* Select sendrecv path based on collective data size

* Add comments on packing and unpacking group field

* Toggling RCCL_P2P_NET_DISABLE in combined calls unit tests

[ROCm/rccl commit: 6dcae8a459]
 2021-06-10 17:51:04 -07:00
Wenkai Du 5bebcb0015 Setup collectives threshold for enabling intranet (#387) 
* Setup collectives threshold for enabling intranet

* Use separate operation counters for coll and p2p

[ROCm/rccl commit: b815a2800f]
 2021-06-09 13:24:26 -07:00
Wenkai Du ab9b9151d2 Add support for another Rome model (#385) 
[ROCm/rccl commit: c2064adcc7]
 2021-06-08 13:58:20 -07:00
Wenkai Du c8a432dc25 Allow intranode use of network connection (#383) 
* Allow intranode use of network connection

* Checking for graph for null pointer

[ROCm/rccl commit: a3a8c2d56b]
 2021-06-08 07:37:59 -07:00
Wenkai Du 4b31e521e9 Add option to enable multiple SAT in SHARP (#380) 
* Add option to enable multiple SAT in SHARP

* Extend number of NICs to 16

[ROCm/rccl commit: 961922ea02]
 2021-06-03 19:45:18 -07:00