AI/rocm-systems
2
0
Atdalīts 0
Kods Problēmas Izmaiņu pieprasījumi Darbības Pakotnes Projekti Laidieni Vikivietne Aktivitāte
2 005 Revīzijas 3 Atzari 0 Tagi
b14e32c46e7afd66dfd717fa8f020df254e40634
Revīziju grafs

1031 Revīzijas

Autors SHA1 Ziņojums Datums
Ke Wen f589932130 Improve warning message about truncated messages 
Display hints of cause so that it would be easier for user to debug.
Also change the error type from InternalError to InvalidUsage as most
of time this is caused by a mismatch in collective size or env settings.
 2021-12-02 16:13:15 -08:00
Wenkai Du 8a08a2f579 Update tuning parameters (#478) 2021-11-30 08:51:11 -08:00
gilbertlee-amd 539de1216f Minor cppcheck fixes, adding suppression file (#475) 
* Minor cppcheck fixes, adding suppression file
 2021-11-24 10:23:59 -07:00
Wenkai Du e9bf01fb7e Determine fine grained memory availability at RCCL bootstrapping (#471) 2021-11-19 08:12:53 -08:00
Chris Jones 8cf7325d69 Perform busIdToInt64 on the stack. 
I noticed when I enabled `NCCL_DEBUG_SUBSYS=ALLOC` that this function is
called thousands of times, making the log output unintelligible.
Fortunately, this function can be implemented without heap allocations.
 2021-11-19 09:35:55 +01:00
Wenkai Du 3a919c1f49 Merge remote-tracking branch 'nccl/master' into develop 2021-11-11 14:22:12 -08:00
Wenkai Du e05de8fd26 Remove extra work element copy (#465) 2021-11-09 13:52:03 -08:00
Wenkai Du 33bdd557c8 Do not use async stream for memory allocation and transport setup without graph (#460) 2021-11-05 13:39:14 -07:00
Wenkai Du 0331e39f81 Update Rome model matching (#461) 
* Update Rome model matching

* Add another Rome model

* Automatically setup NET GDR level from model
 2021-11-05 08:53:47 -07:00
Wenkai Du 26fc6b0919 profiling: fix incorrect print out in timing profile (#457) 2021-11-03 16:22:21 -07:00
Wenkai Du 29170a8b5f Support different protocols and algorithms in all reduce only build (#455) 
* Support different protocols and algorithms in all reduce only build

* Restore deleted line in error
 2021-11-02 08:39:08 -07:00
Wenkai Du 4643a17f83 Check rocm_smi64Config.h on older ROCm build (#452) 2021-10-28 07:26:28 -07:00
Wenkai Du d221fb672a Rework kernel launch code (#449) 2021-10-28 07:26:11 -07:00
Wenkai Du ec36c4c326 Enable timing profiling mode (#447) 2021-10-27 08:21:48 -07:00
Stanley Tsang 7e55b211c5 Build AllReduce only mode (#443) 
* Initial commit of all_reduce_only support

* Working AllReduce only build

* Removing printfs and restoring release build

* Restore P2P index

* Updates to build_allreduce_only mode.

* cleaning up macro ifdefs
 2021-10-26 17:36:46 -06:00
Wenkai Du 14a184eb67 Query XGMI link count through rocm_smi_lib API (#442) 2021-10-26 10:30:20 -07:00
Wenkai Du b4cefc05ed Fix collnet tuning parameters (#441) 2021-10-20 20:45:36 -07:00
Wenkai Du 2508507d0a Fix PCIe gen detection (#437) 
* Fix PCIe gen detection

* Update profiling support
 2021-10-15 08:23:50 -07:00
Wenkai Du 2249a1d9d3 Add more Rome models (#434) 
* Add more Rome models

* Update models and tuning

* Update tuning
 2021-10-12 08:23:20 -07:00
Wenkai Du e0053311c0 Add another Rome model (#431) 2021-10-06 08:17:12 -07:00
Wenkai Du 29c729d8b6 Trim NICs when all GPUs are connected by XGMI (#430) 
* Trim NICs when all GPUs are connected by XGMI

* Only enable clique with maximum of 2 hops
 2021-10-05 18:27:43 -07:00
John Bachan 30ca3fcacf Fix compilation failure in "src/enqueue.cc" on older GCC because of 
missing `#include <cstring>`.
 2021-09-23 09:55:16 -07:00
Sylvain Jeaugey 4ec992fab7 Fix Collnet when GDR is disabled 2021-09-22 05:19:16 -07:00
Ke Wen e11238b302 2.11.4-1 
Add new API for creating a reduction operation which multiplies the input by a rank-specific scalar before doing an inter-rank summation (see: ncclRedOpCreatePreMulSum).
Improve CollNet (SHARP) performance of ncclAllReduce when captured in a CUDA Graph via user buffer registration.
Add environment variable NCCL_NET_PLUGIN="<suffix>" to allow user to choose among multiple NCCL net plugins by substituting into "libnccl-net-<suffix>.so".
Fix memory leak of NVB connections.
Fix topology detection of IB Virtual Functions (SR-IOV).
 2021-09-08 16:06:23 -07:00
John Bachan 5f2f2f670f Fix to https://github.com/NVIDIA/nccl/issues/560 
ncclGroup's containing operations of mixed datatype, element, or collective
would induce crash.
 2021-08-31 15:50:05 -07:00
Wenkai Du 5b72727670 Merge remote-tracking branch 'origin/develop' into 2.10.3 2021-09-15 10:33:25 -07:00
Wenkai Du 5ae3f3f954 Remove extra L1 cache invalidate and restore __ATOMIC_SEQ_CST atomics (#426) 2021-09-14 18:30:16 -07:00
Wenkai Du 3667d308ab Merge remote-tracking branch 'origin/develop' into 2.10.3 2021-09-13 17:32:48 -07:00
Wenkai Du 020484bf40 Use relaxed atomics and add sleep and wakeup in barrier loop (#425) 
* Use relaxed atomics and add sleep and wakeup in barrier loop

* atomicAdd in ROCm 4.3 only support unsigned long long

* Switch to atomicAdd and atomicExch in more places

* Restore LOAD/STORE define to __ATOMIC_SEQ_CST

* Restore atomic for sizes FIFO
 2021-09-13 17:03:49 -07:00
Wenkai Du 8ee2b7932a Merge remote-tracking branch 'origin/develop' into 2.10.3 2021-09-13 15:51:53 -07:00
Wenkai Du ef432e48e1 Update tuning table (#424) 2021-09-13 08:39:01 -07:00
Wenkai Du adb8d63352 Improve barrier implementation 2021-09-08 16:14:32 -05:00
Wenkai Du 31bd4236f1 Remove atomic from profiling 2021-09-08 14:20:32 -05:00
Wenkai Du 21969d7f89 Merge remote-tracking branch 'nccl/master' into 2.10.3 2021-09-01 16:34:44 -07:00
Wenkai Du 5c8380ff5b Implement NIC identification and remapping (#420) 
* Add 1H16P GPU model

* Implement NIC identification and remapping

* Revert "Sort IB devices based on device name (#413)"

This reverts commit 2d0ed8dff6.

* Fix permute and check order

* Correction on IB speed reporting

* Revert "Allow user to link layer with RCCL_IB_HCA_SKIP_LINK_LAYER (#361)"

This reverts commit caf5c9992a.
 2021-08-24 09:42:04 -07:00
Gilbert Lee ae13d2a354 Modifying ReduceOrCopyMulti to accept number of preOp source to support clique-based kernels 2021-08-11 11:00:34 -05:00
Wenkai Du a4929465c5 Improve clique kernel performance by increasing unroll 2021-08-26 18:06:09 -07:00
Wenkai Du 574f0aca53 Fix typo that affects clique kernels 2021-08-26 11:10:58 -07:00
Wenkai Du 1faff323b4 Unit Test: support ncclAvg 2021-08-25 14:15:54 -07:00
Wenkai Du 60ca7484c0 Fix kernel data trace 2021-08-24 14:02:53 -07:00
Wenkai Du d5f93649ff Merge remote-tracking branch 'origin/develop' into 2.10.3 2021-08-24 09:49:47 -07:00
Wenkai Du 5f15ed6e3e Add gfx908 VM model (#418) 2021-08-10 08:55:11 -07:00
Wenkai Du 707c687090 Use noinline for kernel functions 2021-08-06 09:15:04 -07:00
Wenkai Du 01d3b20a66 Fix incorrect network proxy received bytes reporting 2021-08-05 17:45:48 -07:00
Wenkai Du babbd1047b Merge branch 'develop' into 2.10.3 2021-08-04 09:45:22 -07:00
Wenkai Du 2d0ed8dff6 Sort IB devices based on device name (#413) 2021-08-03 15:32:41 -07:00
Wenkai Du bf2339f93e Merge remote-tracking branch 'nccl/master' into 2.10.3 2021-07-30 16:23:14 -07:00
Wenkai Du 3e27227562 XGMI connection is always prioritized over NET regardless of hops (#412) 2021-07-29 11:12:42 -07:00
Wenkai Du 818cdb16a8 Query XGMI links from xml and adjust gfx906 channel usage (#410) 2021-07-27 17:32:41 -07:00
Wenkai Du 135d47d125 topo_expl: fix build after switching to rocm-smi-lib (#405) 
* topo_expl: fix build after switching to rocm-smi-lib

* Use minimal of 4 channels for gfx908
 2021-07-27 08:30:08 -07:00