AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
751 Commits 3 Branches 0 Tags
02096c9936defa018138d42fa8364fb3c91f385c
Commit Graph

342 Commits

Author SHA1 Message Date
Wenkai Du 02096c9936 Add another Rome model (#497) 2022-02-12 10:30:16 -08:00
Wenkai Du 400df49dbe Generate proper b-tree with non-repeating channels (#493) 2022-01-19 15:09:17 -08:00
Wenkai Du 973d0111db Merge remote-tracking branch 'nccl/master' into develop 2022-01-17 13:34:36 -08:00
Wenkai Du 598c6fdded Update Rome models (#491) 2022-01-14 10:03:30 -08:00
Wenkai Du 6ff7690cb5 Use noinline and a few other fixes (#489) 
* Use noinline and a few other fixes

* Tune collectives
 2022-01-11 16:51:06 -08:00
Wenkai Du 3669e12432 Use hipGraph instead of cudaGraph (#487) 2022-01-10 08:26:01 -08:00
Wenkai Du 565fbeb5e9 Tune collectives for 2.11.4 (#486) 2022-01-10 08:25:47 -08:00
Wenkai Du 4234a638b5 Merge pull request #482 from ROCmSoftwarePlatform/2.11.4 
Sync up with 2.11.4
 2022-01-05 09:31:51 -08:00
Wenkai Du f8d0775a6f Add another Rome model (#483) 2022-01-05 09:26:31 -08:00
Wenkai Du 434ecb0e1f Merge remote-tracking branch 'origin/develop' into 2.11.4 2022-01-03 09:54:16 -08:00
Ke Wen c88c9f873f Add env NCCL_NET_DISABLE_INTRA 
Disable NET transport for intra-node communication by setting the env to 1
It provides an option to error out instead of falling back to NET when superior intra-node transports (P2P and SHM) are unavailable
 2021-12-08 16:28:19 +01:00
Wenkai Du a94b953bcc Update Rome model (#479) 2021-12-03 08:24:51 -08:00
Wenkai Du 8a08a2f579 Update tuning parameters (#478) 2021-11-30 08:51:11 -08:00
gilbertlee-amd 539de1216f Minor cppcheck fixes, adding suppression file (#475) 
* Minor cppcheck fixes, adding suppression file
 2021-11-24 10:23:59 -07:00
Wenkai Du e9bf01fb7e Determine fine grained memory availability at RCCL bootstrapping (#471) 2021-11-19 08:12:53 -08:00
Chris Jones 8cf7325d69 Perform busIdToInt64 on the stack. 
I noticed when I enabled `NCCL_DEBUG_SUBSYS=ALLOC` that this function is
called thousands of times, making the log output unintelligible.
Fortunately, this function can be implemented without heap allocations.
 2021-11-19 09:35:55 +01:00
Wenkai Du 3a919c1f49 Merge remote-tracking branch 'nccl/master' into develop 2021-11-11 14:22:12 -08:00
Wenkai Du e05de8fd26 Remove extra work element copy (#465) 2021-11-09 13:52:03 -08:00
Wenkai Du 33bdd557c8 Do not use async stream for memory allocation and transport setup without graph (#460) 2021-11-05 13:39:14 -07:00
Wenkai Du 0331e39f81 Update Rome model matching (#461) 
* Update Rome model matching

* Add another Rome model

* Automatically setup NET GDR level from model
 2021-11-05 08:53:47 -07:00
Wenkai Du 26fc6b0919 profiling: fix incorrect print out in timing profile (#457) 2021-11-03 16:22:21 -07:00
Wenkai Du 29170a8b5f Support different protocols and algorithms in all reduce only build (#455) 
* Support different protocols and algorithms in all reduce only build

* Restore deleted line in error
 2021-11-02 08:39:08 -07:00
Wenkai Du 4643a17f83 Check rocm_smi64Config.h on older ROCm build (#452) 2021-10-28 07:26:28 -07:00
Wenkai Du d221fb672a Rework kernel launch code (#449) 2021-10-28 07:26:11 -07:00
Wenkai Du ec36c4c326 Enable timing profiling mode (#447) 2021-10-27 08:21:48 -07:00
Stanley Tsang 7e55b211c5 Build AllReduce only mode (#443) 
* Initial commit of all_reduce_only support

* Working AllReduce only build

* Removing printfs and restoring release build

* Restore P2P index

* Updates to build_allreduce_only mode.

* cleaning up macro ifdefs
 2021-10-26 17:36:46 -06:00
Wenkai Du 14a184eb67 Query XGMI link count through rocm_smi_lib API (#442) 2021-10-26 10:30:20 -07:00
Wenkai Du b4cefc05ed Fix collnet tuning parameters (#441) 2021-10-20 20:45:36 -07:00
Wenkai Du 2508507d0a Fix PCIe gen detection (#437) 
* Fix PCIe gen detection

* Update profiling support
 2021-10-15 08:23:50 -07:00
Wenkai Du 2249a1d9d3 Add more Rome models (#434) 
* Add more Rome models

* Update models and tuning

* Update tuning
 2021-10-12 08:23:20 -07:00
Wenkai Du e0053311c0 Add another Rome model (#431) 2021-10-06 08:17:12 -07:00
Wenkai Du 29c729d8b6 Trim NICs when all GPUs are connected by XGMI (#430) 
* Trim NICs when all GPUs are connected by XGMI

* Only enable clique with maximum of 2 hops
 2021-10-05 18:27:43 -07:00
John Bachan 30ca3fcacf Fix compilation failure in "src/enqueue.cc" on older GCC because of 
missing `#include <cstring>`.
 2021-09-23 09:55:16 -07:00
Sylvain Jeaugey 4ec992fab7 Fix Collnet when GDR is disabled 2021-09-22 05:19:16 -07:00
Ke Wen e11238b302 2.11.4-1 
Add new API for creating a reduction operation which multiplies the input by a rank-specific scalar before doing an inter-rank summation (see: ncclRedOpCreatePreMulSum).
Improve CollNet (SHARP) performance of ncclAllReduce when captured in a CUDA Graph via user buffer registration.
Add environment variable NCCL_NET_PLUGIN="<suffix>" to allow user to choose among multiple NCCL net plugins by substituting into "libnccl-net-<suffix>.so".
Fix memory leak of NVB connections.
Fix topology detection of IB Virtual Functions (SR-IOV).
 2021-09-08 16:06:23 -07:00
John Bachan 5f2f2f670f Fix to https://github.com/NVIDIA/nccl/issues/560 
ncclGroup's containing operations of mixed datatype, element, or collective
would induce crash.
 2021-08-31 15:50:05 -07:00
Wenkai Du 5b72727670 Merge remote-tracking branch 'origin/develop' into 2.10.3 2021-09-15 10:33:25 -07:00
Wenkai Du 5ae3f3f954 Remove extra L1 cache invalidate and restore __ATOMIC_SEQ_CST atomics (#426) 2021-09-14 18:30:16 -07:00
Wenkai Du 3667d308ab Merge remote-tracking branch 'origin/develop' into 2.10.3 2021-09-13 17:32:48 -07:00
Wenkai Du 020484bf40 Use relaxed atomics and add sleep and wakeup in barrier loop (#425) 
* Use relaxed atomics and add sleep and wakeup in barrier loop

* atomicAdd in ROCm 4.3 only support unsigned long long

* Switch to atomicAdd and atomicExch in more places

* Restore LOAD/STORE define to __ATOMIC_SEQ_CST

* Restore atomic for sizes FIFO
 2021-09-13 17:03:49 -07:00
Wenkai Du 8ee2b7932a Merge remote-tracking branch 'origin/develop' into 2.10.3 2021-09-13 15:51:53 -07:00
Wenkai Du ef432e48e1 Update tuning table (#424) 2021-09-13 08:39:01 -07:00
Wenkai Du adb8d63352 Improve barrier implementation 2021-09-08 16:14:32 -05:00
Wenkai Du 31bd4236f1 Remove atomic from profiling 2021-09-08 14:20:32 -05:00
Wenkai Du 21969d7f89 Merge remote-tracking branch 'nccl/master' into 2.10.3 2021-09-01 16:34:44 -07:00
Wenkai Du 5c8380ff5b Implement NIC identification and remapping (#420) 
* Add 1H16P GPU model

* Implement NIC identification and remapping

* Revert "Sort IB devices based on device name (#413)"

This reverts commit 2d0ed8dff6.

* Fix permute and check order

* Correction on IB speed reporting

* Revert "Allow user to link layer with RCCL_IB_HCA_SKIP_LINK_LAYER (#361)"

This reverts commit caf5c9992a.
 2021-08-24 09:42:04 -07:00
Gilbert Lee ae13d2a354 Modifying ReduceOrCopyMulti to accept number of preOp source to support clique-based kernels 2021-08-11 11:00:34 -05:00
Wenkai Du a4929465c5 Improve clique kernel performance by increasing unroll 2021-08-26 18:06:09 -07:00
Wenkai Du 574f0aca53 Fix typo that affects clique kernels 2021-08-26 11:10:58 -07:00
Wenkai Du 1faff323b4 Unit Test: support ncclAvg 2021-08-25 14:15:54 -07:00