2151c79d14
Add new Rome model ( #536 )
2022-04-13 11:45:40 -07:00
ba4c165bf3
Add new Rome model ( #535 )
2022-04-12 13:27:32 -07:00
15b572751e
Increase chunk steps of broadcast and reduce ( #528 )
2022-04-07 13:34:04 -07:00
b2ffcf6d89
Doxygen fix for ncclRecv ( #527 )
...
Changed the Doxygen command for ncclRecv and pncclRecv, to be consistent with other APIs.
2022-04-05 14:07:56 -07:00
5cc0a405c0
Add tuning model ( #523 )
2022-04-04 10:19:57 -07:00
bbe780ca6c
Support multiple tuning tables ( #522 )
...
* Support multiple tuning tables
* [UnitTests] Skip managed memory testing
2022-03-31 17:09:21 -07:00
7cbbca4da1
Update tuning parameters ( #518 )
...
* Update tuning parameters
* Respect user algo and topo selections
2022-03-29 08:15:37 -07:00
cd17cf6dce
Update Rome model matching and add new models ( #516 )
...
* Update Rome model matching and add new models
* Add missing file
* Models update
2022-03-21 10:54:40 -07:00
d6d6af710e
Force ring algorithm on single node ( #509 )
2022-03-04 10:29:02 -08:00
29ad0f5fbe
Unit test refactor ( #500 )
...
Refactoring and consolidating single-process / multi-process unit testing
2022-02-25 08:59:07 -07:00
b569c0a1db
Add Pivot AllToAll algorithm for Rome model ( #503 )
...
* add a2a pivot interface
* remove debug info
* address comments
* fix bug
* remove custom script
* address comments
* fix bug
2022-02-20 21:09:47 -08:00
94e0dc8bfd
Allow additional options to be passed in through model's definition ( #501 )
2022-02-17 08:28:58 -08:00
02096c9936
Add another Rome model ( #497 )
2022-02-12 10:30:16 -08:00
400df49dbe
Generate proper b-tree with non-repeating channels ( #493 )
2022-01-19 15:09:17 -08:00
973d0111db
Merge remote-tracking branch 'nccl/master' into develop
2022-01-17 13:34:36 -08:00
598c6fdded
Update Rome models ( #491 )
2022-01-14 10:03:30 -08:00
6ff7690cb5
Use noinline and a few other fixes ( #489 )
...
* Use noinline and a few other fixes
* Tune collectives
2022-01-11 16:51:06 -08:00
3669e12432
Use hipGraph instead of cudaGraph ( #487 )
2022-01-10 08:26:01 -08:00
565fbeb5e9
Tune collectives for 2.11.4 ( #486 )
2022-01-10 08:25:47 -08:00
4234a638b5
Merge pull request #482 from ROCmSoftwarePlatform/2.11.4
...
Sync up with 2.11.4
2022-01-05 09:31:51 -08:00
f8d0775a6f
Add another Rome model ( #483 )
2022-01-05 09:26:31 -08:00
434ecb0e1f
Merge remote-tracking branch 'origin/develop' into 2.11.4
2022-01-03 09:54:16 -08:00
c88c9f873f
Add env NCCL_NET_DISABLE_INTRA
...
Disable NET transport for intra-node communication by setting the env to 1
It provides an option to error out instead of falling back to NET when superior intra-node transports (P2P and SHM) are unavailable
2021-12-08 16:28:19 +01:00
a94b953bcc
Update Rome model ( #479 )
2021-12-03 08:24:51 -08:00
8a08a2f579
Update tuning parameters ( #478 )
2021-11-30 08:51:11 -08:00
539de1216f
Minor cppcheck fixes, adding suppression file ( #475 )
...
* Minor cppcheck fixes, adding suppression file
2021-11-24 10:23:59 -07:00
e9bf01fb7e
Determine fine grained memory availability at RCCL bootstrapping ( #471 )
2021-11-19 08:12:53 -08:00
8cf7325d69
Perform busIdToInt64 on the stack.
...
I noticed when I enabled `NCCL_DEBUG_SUBSYS=ALLOC` that this function is
called thousands of times, making the log output unintelligible.
Fortunately, this function can be implemented without heap allocations.
2021-11-19 09:35:55 +01:00
3a919c1f49
Merge remote-tracking branch 'nccl/master' into develop
2021-11-11 14:22:12 -08:00
e05de8fd26
Remove extra work element copy ( #465 )
2021-11-09 13:52:03 -08:00
33bdd557c8
Do not use async stream for memory allocation and transport setup without graph ( #460 )
2021-11-05 13:39:14 -07:00
0331e39f81
Update Rome model matching ( #461 )
...
* Update Rome model matching
* Add another Rome model
* Automatically setup NET GDR level from model
2021-11-05 08:53:47 -07:00
26fc6b0919
profiling: fix incorrect print out in timing profile ( #457 )
2021-11-03 16:22:21 -07:00
29170a8b5f
Support different protocols and algorithms in all reduce only build ( #455 )
...
* Support different protocols and algorithms in all reduce only build
* Restore deleted line in error
2021-11-02 08:39:08 -07:00
4643a17f83
Check rocm_smi64Config.h on older ROCm build ( #452 )
2021-10-28 07:26:28 -07:00
d221fb672a
Rework kernel launch code ( #449 )
2021-10-28 07:26:11 -07:00
ec36c4c326
Enable timing profiling mode ( #447 )
2021-10-27 08:21:48 -07:00
7e55b211c5
Build AllReduce only mode ( #443 )
...
* Initial commit of all_reduce_only support
* Working AllReduce only build
* Removing printfs and restoring release build
* Restore P2P index
* Updates to build_allreduce_only mode.
* cleaning up macro ifdefs
2021-10-26 17:36:46 -06:00
14a184eb67
Query XGMI link count through rocm_smi_lib API ( #442 )
2021-10-26 10:30:20 -07:00
b4cefc05ed
Fix collnet tuning parameters ( #441 )
2021-10-20 20:45:36 -07:00
2508507d0a
Fix PCIe gen detection ( #437 )
...
* Fix PCIe gen detection
* Update profiling support
2021-10-15 08:23:50 -07:00
2249a1d9d3
Add more Rome models ( #434 )
...
* Add more Rome models
* Update models and tuning
* Update tuning
2021-10-12 08:23:20 -07:00
e0053311c0
Add another Rome model ( #431 )
2021-10-06 08:17:12 -07:00
29c729d8b6
Trim NICs when all GPUs are connected by XGMI ( #430 )
...
* Trim NICs when all GPUs are connected by XGMI
* Only enable clique with maximum of 2 hops
2021-10-05 18:27:43 -07:00
30ca3fcacf
Fix compilation failure in "src/enqueue.cc" on older GCC because of
...
missing `#include <cstring>`.
2021-09-23 09:55:16 -07:00
4ec992fab7
Fix Collnet when GDR is disabled
2021-09-22 05:19:16 -07:00
e11238b302
2.11.4-1
...
Add new API for creating a reduction operation which multiplies the input by a rank-specific scalar before doing an inter-rank summation (see: ncclRedOpCreatePreMulSum).
Improve CollNet (SHARP) performance of ncclAllReduce when captured in a CUDA Graph via user buffer registration.
Add environment variable NCCL_NET_PLUGIN="<suffix>" to allow user to choose among multiple NCCL net plugins by substituting into "libnccl-net-<suffix>.so".
Fix memory leak of NVB connections.
Fix topology detection of IB Virtual Functions (SR-IOV).
2021-09-08 16:06:23 -07:00
5f2f2f670f
Fix to https://github.com/NVIDIA/nccl/issues/560
...
ncclGroup's containing operations of mixed datatype, element, or collective
would induce crash.
2021-08-31 15:50:05 -07:00
5b72727670
Merge remote-tracking branch 'origin/develop' into 2.10.3
2021-09-15 10:33:25 -07:00
5ae3f3f954
Remove extra L1 cache invalidate and restore __ATOMIC_SEQ_CST atomics ( #426 )
2021-09-14 18:30:16 -07:00
Wenkai Du
Colin Smith
gilbertlee-amd
Ziyue Yang
Wenkai Du
Ke Wen
Chris Jones
Stanley Tsang
John Bachan
Sylvain Jeaugey
Ke Wen