AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
716 Commits 3 Branches 0 Tags
7b8b54955ba09ddd247268f1521df33d8b82c058
Commit Graph

322 Commits

Author SHA1 Message Date
Wenkai Du e05de8fd26 Remove extra work element copy (#465) 2021-11-09 13:52:03 -08:00
Wenkai Du 33bdd557c8 Do not use async stream for memory allocation and transport setup without graph (#460) 2021-11-05 13:39:14 -07:00
Wenkai Du 0331e39f81 Update Rome model matching (#461) 
* Update Rome model matching

* Add another Rome model

* Automatically setup NET GDR level from model
 2021-11-05 08:53:47 -07:00
Wenkai Du 26fc6b0919 profiling: fix incorrect print out in timing profile (#457) 2021-11-03 16:22:21 -07:00
Wenkai Du 29170a8b5f Support different protocols and algorithms in all reduce only build (#455) 
* Support different protocols and algorithms in all reduce only build

* Restore deleted line in error
 2021-11-02 08:39:08 -07:00
Wenkai Du 4643a17f83 Check rocm_smi64Config.h on older ROCm build (#452) 2021-10-28 07:26:28 -07:00
Wenkai Du d221fb672a Rework kernel launch code (#449) 2021-10-28 07:26:11 -07:00
Wenkai Du ec36c4c326 Enable timing profiling mode (#447) 2021-10-27 08:21:48 -07:00
Stanley Tsang 7e55b211c5 Build AllReduce only mode (#443) 
* Initial commit of all_reduce_only support

* Working AllReduce only build

* Removing printfs and restoring release build

* Restore P2P index

* Updates to build_allreduce_only mode.

* cleaning up macro ifdefs
 2021-10-26 17:36:46 -06:00
Wenkai Du 14a184eb67 Query XGMI link count through rocm_smi_lib API (#442) 2021-10-26 10:30:20 -07:00
Wenkai Du b4cefc05ed Fix collnet tuning parameters (#441) 2021-10-20 20:45:36 -07:00
Wenkai Du 2508507d0a Fix PCIe gen detection (#437) 
* Fix PCIe gen detection

* Update profiling support
 2021-10-15 08:23:50 -07:00
Wenkai Du 2249a1d9d3 Add more Rome models (#434) 
* Add more Rome models

* Update models and tuning

* Update tuning
 2021-10-12 08:23:20 -07:00
Wenkai Du e0053311c0 Add another Rome model (#431) 2021-10-06 08:17:12 -07:00
Wenkai Du 29c729d8b6 Trim NICs when all GPUs are connected by XGMI (#430) 
* Trim NICs when all GPUs are connected by XGMI

* Only enable clique with maximum of 2 hops
 2021-10-05 18:27:43 -07:00
Wenkai Du 5b72727670 Merge remote-tracking branch 'origin/develop' into 2.10.3 2021-09-15 10:33:25 -07:00
Wenkai Du 5ae3f3f954 Remove extra L1 cache invalidate and restore __ATOMIC_SEQ_CST atomics (#426) 2021-09-14 18:30:16 -07:00
Wenkai Du 3667d308ab Merge remote-tracking branch 'origin/develop' into 2.10.3 2021-09-13 17:32:48 -07:00
Wenkai Du 020484bf40 Use relaxed atomics and add sleep and wakeup in barrier loop (#425) 
* Use relaxed atomics and add sleep and wakeup in barrier loop

* atomicAdd in ROCm 4.3 only support unsigned long long

* Switch to atomicAdd and atomicExch in more places

* Restore LOAD/STORE define to __ATOMIC_SEQ_CST

* Restore atomic for sizes FIFO
 2021-09-13 17:03:49 -07:00
Wenkai Du 8ee2b7932a Merge remote-tracking branch 'origin/develop' into 2.10.3 2021-09-13 15:51:53 -07:00
Wenkai Du ef432e48e1 Update tuning table (#424) 2021-09-13 08:39:01 -07:00
Wenkai Du adb8d63352 Improve barrier implementation 2021-09-08 16:14:32 -05:00
Wenkai Du 31bd4236f1 Remove atomic from profiling 2021-09-08 14:20:32 -05:00
Wenkai Du 21969d7f89 Merge remote-tracking branch 'nccl/master' into 2.10.3 2021-09-01 16:34:44 -07:00
John Bachan 5f2f2f670f Fix to https://github.com/NVIDIA/nccl/issues/560 
ncclGroup's containing operations of mixed datatype, element, or collective
would induce crash.
 2021-08-31 15:50:05 -07:00
Wenkai Du 5c8380ff5b Implement NIC identification and remapping (#420) 
* Add 1H16P GPU model

* Implement NIC identification and remapping

* Revert "Sort IB devices based on device name (#413)"

This reverts commit 2d0ed8dff6.

* Fix permute and check order

* Correction on IB speed reporting

* Revert "Allow user to link layer with RCCL_IB_HCA_SKIP_LINK_LAYER (#361)"

This reverts commit caf5c9992a.
 2021-08-24 09:42:04 -07:00
Gilbert Lee ae13d2a354 Modifying ReduceOrCopyMulti to accept number of preOp source to support clique-based kernels 2021-08-11 11:00:34 -05:00
Wenkai Du a4929465c5 Improve clique kernel performance by increasing unroll 2021-08-26 18:06:09 -07:00
Wenkai Du 574f0aca53 Fix typo that affects clique kernels 2021-08-26 11:10:58 -07:00
Wenkai Du 1faff323b4 Unit Test: support ncclAvg 2021-08-25 14:15:54 -07:00
Wenkai Du 60ca7484c0 Fix kernel data trace 2021-08-24 14:02:53 -07:00
Wenkai Du d5f93649ff Merge remote-tracking branch 'origin/develop' into 2.10.3 2021-08-24 09:49:47 -07:00
Wenkai Du 5f15ed6e3e Add gfx908 VM model (#418) 2021-08-10 08:55:11 -07:00
Wenkai Du 707c687090 Use noinline for kernel functions 2021-08-06 09:15:04 -07:00
Wenkai Du 01d3b20a66 Fix incorrect network proxy received bytes reporting 2021-08-05 17:45:48 -07:00
Wenkai Du babbd1047b Merge branch 'develop' into 2.10.3 2021-08-04 09:45:22 -07:00
Wenkai Du 2d0ed8dff6 Sort IB devices based on device name (#413) 2021-08-03 15:32:41 -07:00
Wenkai Du bf2339f93e Merge remote-tracking branch 'nccl/master' into 2.10.3 2021-07-30 16:23:14 -07:00
Wenkai Du 3e27227562 XGMI connection is always prioritized over NET regardless of hops (#412) 2021-07-29 11:12:42 -07:00
Wenkai Du 818cdb16a8 Query XGMI links from xml and adjust gfx906 channel usage (#410) 2021-07-27 17:32:41 -07:00
Wenkai Du 135d47d125 topo_expl: fix build after switching to rocm-smi-lib (#405) 
* topo_expl: fix build after switching to rocm-smi-lib

* Use minimal of 4 channels for gfx908
 2021-07-27 08:30:08 -07:00
Wenkai Du dfc62d5fbb Skipping unnecessary functions in Doxygen by marking as internal (#353) (#406) 
(cherry picked from commit 1c982d819d9c7fe0310b80f9a25808e54c71137e)

Co-authored-by: saadrahim <44449863+saadrahim@users.noreply.github.com>
 2021-07-24 11:04:27 -07:00
Lu bd6dbca8fb Add more info to RCCL logging for topo-aware optim. 2021-07-22 09:52:39 -07:00
Ke Wen 7e51592129 2.10.3-1 
Add support for bfloat16.
Add ncclAvg reduction operation.
Improve performance for aggregated operations.
Improve performance for tree.
Improve network error reporting.
Add NCCL_NET parameter to force a specific network.
Add NCCL_IB_QPS_PER_CONNECTION parameter to split IB traffic onto multiple queue pairs.
Fix topology detection error in WSL2.
Fix proxy memory elements affinity (improve alltoall performance).
Fix graph search on cubemesh topologies.
Fix hang in cubemesh during NVB connections.
 2021-07-08 14:30:14 -07:00
Wenkai Du 56155ff5b6 Use rocm_smi_lib for getting topology information (#402) 
* Use rocm_smi_lib for getting topology information

* Add rocm-smi-lib dependency to RCCL package
 2021-07-08 13:23:11 -07:00
Wenkai Du fa6d7e9a63 Fixes for NCCL_MAX_NCHANNELS and topo_expl (#398) 2021-06-22 08:41:49 -07:00
Wenkai Du 6dcae8a459 Select sendrecv path based on collective data size (#391) 
* Select sendrecv path based on collective data size

* Add comments on packing and unpacking group field

* Toggling RCCL_P2P_NET_DISABLE in combined calls unit tests
 2021-06-10 17:51:04 -07:00
Wenkai Du b815a2800f Setup collectives threshold for enabling intranet (#387) 
* Setup collectives threshold for enabling intranet

* Use separate operation counters for coll and p2p
 2021-06-09 13:24:26 -07:00
Wenkai Du c2064adcc7 Add support for another Rome model (#385) 2021-06-08 13:58:20 -07:00
Wenkai Du a3a8c2d56b Allow intranode use of network connection (#383) 
* Allow intranode use of network connection

* Checking for graph for null pointer
 2021-06-08 07:37:59 -07:00