AI/rocm-systems
2
0
Forka 0
Codice Problemi Pull Request Actions Pacchetti Progetti Rilasci Wiki Attività
822 Commit 3 Rami (Branch) 0 Tag
dcf46e84e04d471f7a29ca69f0b59a875fcfcd3c
Grafico dei commit

387 Commit

Autore SHA1 Messaggio Data
Wenkai Du 3f16835565 Tune collectives for 2.11.4 (#486) 
[ROCm/rccl commit: 565fbeb5e9]
 2022-01-10 08:25:47 -08:00
Wenkai Du e5ff692673 Merge pull request #482 from ROCmSoftwarePlatform/2.11.4 
Sync up with 2.11.4

[ROCm/rccl commit: 4234a638b5]
 2022-01-05 09:31:51 -08:00
Wenkai Du 93c7526fdc Add another Rome model (#483) 
[ROCm/rccl commit: f8d0775a6f]
 2022-01-05 09:26:31 -08:00
Wenkai Du 618fbfc644 Merge remote-tracking branch 'origin/develop' into 2.11.4 
[ROCm/rccl commit: 434ecb0e1f]
 2022-01-03 09:54:16 -08:00
Ke Wen 07d9648aec Add env NCCL_NET_DISABLE_INTRA 
Disable NET transport for intra-node communication by setting the env to 1
It provides an option to error out instead of falling back to NET when superior intra-node transports (P2P and SHM) are unavailable


[ROCm/rccl commit: c88c9f873f]
 2021-12-08 16:28:19 +01:00
Wenkai Du 7406aed9c9 Update Rome model (#479) 
[ROCm/rccl commit: a94b953bcc]
 2021-12-03 08:24:51 -08:00
Ke Wen 26c275fe16 Improve warning message about truncated messages 
Display hints of cause so that it would be easier for user to debug.
Also change the error type from InternalError to InvalidUsage as most
of time this is caused by a mismatch in collective size or env settings.


[ROCm/rccl commit: f589932130]
 2021-12-02 16:13:15 -08:00
Wenkai Du dd4a53db37 Update tuning parameters (#478) 
[ROCm/rccl commit: 8a08a2f579]
 2021-11-30 08:51:11 -08:00
gilbertlee-amd 2da8358a74 Minor cppcheck fixes, adding suppression file (#475) 
* Minor cppcheck fixes, adding suppression file

[ROCm/rccl commit: 539de1216f]
 2021-11-24 10:23:59 -07:00
Wenkai Du 864c6a645e Determine fine grained memory availability at RCCL bootstrapping (#471) 
[ROCm/rccl commit: e9bf01fb7e]
 2021-11-19 08:12:53 -08:00
Chris Jones a52ca36a9e Perform busIdToInt64 on the stack. 
I noticed when I enabled `NCCL_DEBUG_SUBSYS=ALLOC` that this function is
called thousands of times, making the log output unintelligible.
Fortunately, this function can be implemented without heap allocations.


[ROCm/rccl commit: 8cf7325d69]
 2021-11-19 09:35:55 +01:00
Wenkai Du f8bd2d0cfa Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: 3a919c1f49]
 2021-11-11 14:22:12 -08:00
Wenkai Du fc5f692f71 Remove extra work element copy (#465) 
[ROCm/rccl commit: e05de8fd26]
 2021-11-09 13:52:03 -08:00
Wenkai Du 680660fcf7 Do not use async stream for memory allocation and transport setup without graph (#460) 
[ROCm/rccl commit: 33bdd557c8]
 2021-11-05 13:39:14 -07:00
Wenkai Du fd98ee84b4 Update Rome model matching (#461) 
* Update Rome model matching

* Add another Rome model

* Automatically setup NET GDR level from model

[ROCm/rccl commit: 0331e39f81]
 2021-11-05 08:53:47 -07:00
Wenkai Du befaa35e12 profiling: fix incorrect print out in timing profile (#457) 
[ROCm/rccl commit: 26fc6b0919]
 2021-11-03 16:22:21 -07:00
Wenkai Du df59f64e3f Support different protocols and algorithms in all reduce only build (#455) 
* Support different protocols and algorithms in all reduce only build

* Restore deleted line in error

[ROCm/rccl commit: 29170a8b5f]
 2021-11-02 08:39:08 -07:00
Wenkai Du a11b55a37f Check rocm_smi64Config.h on older ROCm build (#452) 
[ROCm/rccl commit: 4643a17f83]
 2021-10-28 07:26:28 -07:00
Wenkai Du b0b2e27df3 Rework kernel launch code (#449) 
[ROCm/rccl commit: d221fb672a]
 2021-10-28 07:26:11 -07:00
Wenkai Du 747216e2b2 Enable timing profiling mode (#447) 
[ROCm/rccl commit: ec36c4c326]
 2021-10-27 08:21:48 -07:00
Stanley Tsang a6feafd5dc Build AllReduce only mode (#443) 
* Initial commit of all_reduce_only support

* Working AllReduce only build

* Removing printfs and restoring release build

* Restore P2P index

* Updates to build_allreduce_only mode.

* cleaning up macro ifdefs

[ROCm/rccl commit: 7e55b211c5]
 2021-10-26 17:36:46 -06:00
Wenkai Du 15143b1cfb Query XGMI link count through rocm_smi_lib API (#442) 
[ROCm/rccl commit: 14a184eb67]
 2021-10-26 10:30:20 -07:00
Wenkai Du aac9d57633 Fix collnet tuning parameters (#441) 
[ROCm/rccl commit: b4cefc05ed]
 2021-10-20 20:45:36 -07:00
Wenkai Du 4d979ce13d Fix PCIe gen detection (#437) 
* Fix PCIe gen detection

* Update profiling support

[ROCm/rccl commit: 2508507d0a]
 2021-10-15 08:23:50 -07:00
Wenkai Du b587b55c2e Add more Rome models (#434) 
* Add more Rome models

* Update models and tuning

* Update tuning

[ROCm/rccl commit: 2249a1d9d3]
 2021-10-12 08:23:20 -07:00
Wenkai Du d377c4dcc6 Add another Rome model (#431) 
[ROCm/rccl commit: e0053311c0]
 2021-10-06 08:17:12 -07:00
Wenkai Du 91eca0d7d2 Trim NICs when all GPUs are connected by XGMI (#430) 
* Trim NICs when all GPUs are connected by XGMI

* Only enable clique with maximum of 2 hops

[ROCm/rccl commit: 29c729d8b6]
 2021-10-05 18:27:43 -07:00
John Bachan 389213c7e8 Fix compilation failure in "src/enqueue.cc" on older GCC because of 
missing `#include <cstring>`.


[ROCm/rccl commit: 30ca3fcacf]
 2021-09-23 09:55:16 -07:00
Sylvain Jeaugey 761812841b Fix Collnet when GDR is disabled 
[ROCm/rccl commit: 4ec992fab7]
 2021-09-22 05:19:16 -07:00
Ke Wen e51f2a83da 2.11.4-1 
Add new API for creating a reduction operation which multiplies the input by a rank-specific scalar before doing an inter-rank summation (see: ncclRedOpCreatePreMulSum).
Improve CollNet (SHARP) performance of ncclAllReduce when captured in a CUDA Graph via user buffer registration.
Add environment variable NCCL_NET_PLUGIN="<suffix>" to allow user to choose among multiple NCCL net plugins by substituting into "libnccl-net-<suffix>.so".
Fix memory leak of NVB connections.
Fix topology detection of IB Virtual Functions (SR-IOV).


[ROCm/rccl commit: e11238b302]
 2021-09-08 16:06:23 -07:00
John Bachan 04553b802a Fix to https://github.com/NVIDIA/nccl/issues/560 
ncclGroup's containing operations of mixed datatype, element, or collective
would induce crash.


[ROCm/rccl commit: 5f2f2f670f]
 2021-08-31 15:50:05 -07:00
Wenkai Du a44ed86b46 Merge remote-tracking branch 'origin/develop' into 2.10.3 
[ROCm/rccl commit: 5b72727670]
 2021-09-15 10:33:25 -07:00
Wenkai Du 1eaf495391 Remove extra L1 cache invalidate and restore __ATOMIC_SEQ_CST atomics (#426) 
[ROCm/rccl commit: 5ae3f3f954]
 2021-09-14 18:30:16 -07:00
Wenkai Du d6064367f0 Merge remote-tracking branch 'origin/develop' into 2.10.3 
[ROCm/rccl commit: 3667d308ab]
 2021-09-13 17:32:48 -07:00
Wenkai Du f4387b2954 Use relaxed atomics and add sleep and wakeup in barrier loop (#425) 
* Use relaxed atomics and add sleep and wakeup in barrier loop

* atomicAdd in ROCm 4.3 only support unsigned long long

* Switch to atomicAdd and atomicExch in more places

* Restore LOAD/STORE define to __ATOMIC_SEQ_CST

* Restore atomic for sizes FIFO

[ROCm/rccl commit: 020484bf40]
 2021-09-13 17:03:49 -07:00
Wenkai Du 8acdb77cc0 Merge remote-tracking branch 'origin/develop' into 2.10.3 
[ROCm/rccl commit: 8ee2b7932a]
 2021-09-13 15:51:53 -07:00
Wenkai Du 9ffeb41fe1 Update tuning table (#424) 
[ROCm/rccl commit: ef432e48e1]
 2021-09-13 08:39:01 -07:00
Wenkai Du d2580c8cf5 Improve barrier implementation 
[ROCm/rccl commit: adb8d63352]
 2021-09-08 16:14:32 -05:00
Wenkai Du d75504e9dc Remove atomic from profiling 
[ROCm/rccl commit: 31bd4236f1]
 2021-09-08 14:20:32 -05:00
Wenkai Du 95b37d97cf Merge remote-tracking branch 'nccl/master' into 2.10.3 
[ROCm/rccl commit: 21969d7f89]
 2021-09-01 16:34:44 -07:00
Wenkai Du b9508a6aba Implement NIC identification and remapping (#420) 
* Add 1H16P GPU model

* Implement NIC identification and remapping

* Revert "Sort IB devices based on device name (#413)"

This reverts commit de0c586bad.

* Fix permute and check order

* Correction on IB speed reporting

* Revert "Allow user to link layer with RCCL_IB_HCA_SKIP_LINK_LAYER (#361)"

This reverts commit fa690c47a0.

[ROCm/rccl commit: 5c8380ff5b]
 2021-08-24 09:42:04 -07:00
Gilbert Lee e0fd3c6ba3 Modifying ReduceOrCopyMulti to accept number of preOp source to support clique-based kernels 
[ROCm/rccl commit: ae13d2a354]
 2021-08-11 11:00:34 -05:00
Wenkai Du eaf54184bf Improve clique kernel performance by increasing unroll 
[ROCm/rccl commit: a4929465c5]
 2021-08-26 18:06:09 -07:00
Wenkai Du a5e85a66fd Fix typo that affects clique kernels 
[ROCm/rccl commit: 574f0aca53]
 2021-08-26 11:10:58 -07:00
Wenkai Du bd03ec2a45 Unit Test: support ncclAvg 
[ROCm/rccl commit: 1faff323b4]
 2021-08-25 14:15:54 -07:00
Wenkai Du da7ac55e6f Fix kernel data trace 
[ROCm/rccl commit: 60ca7484c0]
 2021-08-24 14:02:53 -07:00
Wenkai Du 4fd7a14087 Merge remote-tracking branch 'origin/develop' into 2.10.3 
[ROCm/rccl commit: d5f93649ff]
 2021-08-24 09:49:47 -07:00
Wenkai Du 57518da006 Add gfx908 VM model (#418) 
[ROCm/rccl commit: 5f15ed6e3e]
 2021-08-10 08:55:11 -07:00
Wenkai Du 51198b536d Use noinline for kernel functions 
[ROCm/rccl commit: 707c687090]
 2021-08-06 09:15:04 -07:00
Wenkai Du 79bc1e3fde Fix incorrect network proxy received bytes reporting 
[ROCm/rccl commit: 01d3b20a66]
 2021-08-05 17:45:48 -07:00