54590464ca
Remove unnecessary newline in plugin logging
...
Signed-off-by: Felix Abecassis <fabecassis@nvidia.com >
[ROCm/rccl commit: 1c7c014ceb
2022-03-30 10:44:49 +02:00
828f3d11a0
Update tuning parameters ( #518 )
...
* Update tuning parameters
* Respect user algo and topo selections
[ROCm/rccl commit: 7cbbca4da1
2022-03-29 08:15:37 -07:00
db1e628ba3
Update Rome model matching and add new models ( #516 )
...
* Update Rome model matching and add new models
* Add missing file
* Models update
[ROCm/rccl commit: cd17cf6dce
2022-03-21 10:54:40 -07:00
7707479804
Add pthread_detach()'s for threads we never pthread_join(). Helps
...
reduce diagnostic noise for ThreadSanitizer.
Fixes https://github.com/NVIDIA/nccl/issues/649
[ROCm/rccl commit: 44eb40da0e
2022-03-15 10:27:59 -07:00
133aed2dfb
Force ring algorithm on single node ( #509 )
...
[ROCm/rccl commit: d6d6af710e
2022-03-04 10:29:02 -08:00
f8886d8687
2.12.7-1
...
Add network communication through another GPU connected with NVLink
(PXN).
Add aggregation of messages coming from different local GPUs through
PXN and going to the same destination.
Add new v5 plugin API with grouped receives and tags.
Add compat for v4 plugins.
Add naming of NCCL threads to help debugging.
Fix NVLink detection and avoid data corruption when some NVLinks are
down.
Add support for Relaxed Ordering for IB.
Add profiling and timing infrastructure.
[ROCm/rccl commit: 3c223c105a
2022-03-02 20:48:56 +01:00
a182076a0e
Unit test refactor ( #500 )
...
Refactoring and consolidating single-process / multi-process unit testing
[ROCm/rccl commit: 29ad0f5fbe
2022-02-25 08:59:07 -07:00
dfa9b9e958
Add Pivot AllToAll algorithm for Rome model ( #503 )
...
* add a2a pivot interface
* remove debug info
* address comments
* fix bug
* remove custom script
* address comments
* fix bug
[ROCm/rccl commit: b569c0a1db
2022-02-20 21:09:47 -08:00
0f0388ba0b
Allow additional options to be passed in through model's definition ( #501 )
...
[ROCm/rccl commit: 94e0dc8bfd
2022-02-17 08:28:58 -08:00
5b697e40db
Add another Rome model ( #497 )
...
[ROCm/rccl commit: 02096c9936
2022-02-12 10:30:16 -08:00
92d6888bdc
Split IB parameter sanity check into two parts
...
First part on collective mismatch, second part on internal errors
[ROCm/rccl commit: fbfb6ac5d7
2022-02-08 15:21:22 -08:00
51df47f9b8
Revert "remove unused basePath"
...
This reverts commit d973ddac8bcc78e9fab8
2022-01-21 12:30:34 +01:00
d973ddac8b
remove unused basePath
...
[ROCm/rccl commit: 445bc19657
2022-01-21 12:12:26 +01:00
635c0bcc01
Generate proper b-tree with non-repeating channels ( #493 )
...
[ROCm/rccl commit: 400df49dbe
2022-01-19 15:09:17 -08:00
663c1db898
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: 973d0111db
2022-01-17 13:34:36 -08:00
4d43d9ce22
Update Rome models ( #491 )
...
[ROCm/rccl commit: 598c6fdded
2022-01-14 10:03:30 -08:00
93ab252d52
Use noinline and a few other fixes ( #489 )
...
* Use noinline and a few other fixes
* Tune collectives
[ROCm/rccl commit: 6ff7690cb5
2022-01-11 16:51:06 -08:00
123de1621f
Use hipGraph instead of cudaGraph ( #487 )
...
[ROCm/rccl commit: 3669e12432
2022-01-10 08:26:01 -08:00
3f16835565
Tune collectives for 2.11.4 ( #486 )
...
[ROCm/rccl commit: 565fbeb5e9
2022-01-10 08:25:47 -08:00
e5ff692673
Merge pull request #482 from ROCmSoftwarePlatform/2.11.4
...
Sync up with 2.11.4
[ROCm/rccl commit: 4234a638b5
2022-01-05 09:31:51 -08:00
93c7526fdc
Add another Rome model ( #483 )
...
[ROCm/rccl commit: f8d0775a6f
2022-01-05 09:26:31 -08:00
618fbfc644
Merge remote-tracking branch 'origin/develop' into 2.11.4
...
[ROCm/rccl commit: 434ecb0e1f
2022-01-03 09:54:16 -08:00
07d9648aec
Add env NCCL_NET_DISABLE_INTRA
...
Disable NET transport for intra-node communication by setting the env to 1
It provides an option to error out instead of falling back to NET when superior intra-node transports (P2P and SHM) are unavailable
[ROCm/rccl commit: c88c9f873f
2021-12-08 16:28:19 +01:00
7406aed9c9
Update Rome model ( #479 )
...
[ROCm/rccl commit: a94b953bcc
2021-12-03 08:24:51 -08:00
26c275fe16
Improve warning message about truncated messages
...
Display hints of cause so that it would be easier for user to debug.
Also change the error type from InternalError to InvalidUsage as most
of time this is caused by a mismatch in collective size or env settings.
[ROCm/rccl commit: f589932130
2021-12-02 16:13:15 -08:00
dd4a53db37
Update tuning parameters ( #478 )
...
[ROCm/rccl commit: 8a08a2f579
2021-11-30 08:51:11 -08:00
2da8358a74
Minor cppcheck fixes, adding suppression file ( #475 )
...
* Minor cppcheck fixes, adding suppression file
[ROCm/rccl commit: 539de1216f
2021-11-24 10:23:59 -07:00
864c6a645e
Determine fine grained memory availability at RCCL bootstrapping ( #471 )
...
[ROCm/rccl commit: e9bf01fb7e
2021-11-19 08:12:53 -08:00
a52ca36a9e
Perform busIdToInt64 on the stack.
...
I noticed when I enabled `NCCL_DEBUG_SUBSYS=ALLOC` that this function is
called thousands of times, making the log output unintelligible.
Fortunately, this function can be implemented without heap allocations.
[ROCm/rccl commit: 8cf7325d69
2021-11-19 09:35:55 +01:00
f8bd2d0cfa
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: 3a919c1f49
2021-11-11 14:22:12 -08:00
fc5f692f71
Remove extra work element copy ( #465 )
...
[ROCm/rccl commit: e05de8fd26
2021-11-09 13:52:03 -08:00
680660fcf7
Do not use async stream for memory allocation and transport setup without graph ( #460 )
...
[ROCm/rccl commit: 33bdd557c8
2021-11-05 13:39:14 -07:00
fd98ee84b4
Update Rome model matching ( #461 )
...
* Update Rome model matching
* Add another Rome model
* Automatically setup NET GDR level from model
[ROCm/rccl commit: 0331e39f81
2021-11-05 08:53:47 -07:00
befaa35e12
profiling: fix incorrect print out in timing profile ( #457 )
...
[ROCm/rccl commit: 26fc6b0919
2021-11-03 16:22:21 -07:00
df59f64e3f
Support different protocols and algorithms in all reduce only build ( #455 )
...
* Support different protocols and algorithms in all reduce only build
* Restore deleted line in error
[ROCm/rccl commit: 29170a8b5f
2021-11-02 08:39:08 -07:00
a11b55a37f
Check rocm_smi64Config.h on older ROCm build ( #452 )
...
[ROCm/rccl commit: 4643a17f83
2021-10-28 07:26:28 -07:00
b0b2e27df3
Rework kernel launch code ( #449 )
...
[ROCm/rccl commit: d221fb672a
2021-10-28 07:26:11 -07:00
747216e2b2
Enable timing profiling mode ( #447 )
...
[ROCm/rccl commit: ec36c4c326
2021-10-27 08:21:48 -07:00
a6feafd5dc
Build AllReduce only mode ( #443 )
...
* Initial commit of all_reduce_only support
* Working AllReduce only build
* Removing printfs and restoring release build
* Restore P2P index
* Updates to build_allreduce_only mode.
* cleaning up macro ifdefs
[ROCm/rccl commit: 7e55b211c5
2021-10-26 17:36:46 -06:00
15143b1cfb
Query XGMI link count through rocm_smi_lib API ( #442 )
...
[ROCm/rccl commit: 14a184eb67
2021-10-26 10:30:20 -07:00
aac9d57633
Fix collnet tuning parameters ( #441 )
...
[ROCm/rccl commit: b4cefc05ed
2021-10-20 20:45:36 -07:00
4d979ce13d
Fix PCIe gen detection ( #437 )
...
* Fix PCIe gen detection
* Update profiling support
[ROCm/rccl commit: 2508507d0a
2021-10-15 08:23:50 -07:00
b587b55c2e
Add more Rome models ( #434 )
...
* Add more Rome models
* Update models and tuning
* Update tuning
[ROCm/rccl commit: 2249a1d9d3
2021-10-12 08:23:20 -07:00
d377c4dcc6
Add another Rome model ( #431 )
...
[ROCm/rccl commit: e0053311c0
2021-10-06 08:17:12 -07:00
91eca0d7d2
Trim NICs when all GPUs are connected by XGMI ( #430 )
...
* Trim NICs when all GPUs are connected by XGMI
* Only enable clique with maximum of 2 hops
[ROCm/rccl commit: 29c729d8b6
2021-10-05 18:27:43 -07:00
389213c7e8
Fix compilation failure in "src/enqueue.cc" on older GCC because of
...
missing `#include <cstring>`.
[ROCm/rccl commit: 30ca3fcacf
2021-09-23 09:55:16 -07:00
761812841b
Fix Collnet when GDR is disabled
...
[ROCm/rccl commit: 4ec992fab7
2021-09-22 05:19:16 -07:00
e51f2a83da
2.11.4-1
...
Add new API for creating a reduction operation which multiplies the input by a rank-specific scalar before doing an inter-rank summation (see: ncclRedOpCreatePreMulSum).
Improve CollNet (SHARP) performance of ncclAllReduce when captured in a CUDA Graph via user buffer registration.
Add environment variable NCCL_NET_PLUGIN="<suffix>" to allow user to choose among multiple NCCL net plugins by substituting into "libnccl-net-<suffix>.so".
Fix memory leak of NVB connections.
Fix topology detection of IB Virtual Functions (SR-IOV).
[ROCm/rccl commit: e11238b302
2021-09-08 16:06:23 -07:00
04553b802a
Fix to https://github.com/NVIDIA/nccl/issues/560
...
ncclGroup's containing operations of mixed datatype, element, or collective
would induce crash.
[ROCm/rccl commit: 5f2f2f670f
2021-08-31 15:50:05 -07:00
a44ed86b46
Merge remote-tracking branch 'origin/develop' into 2.10.3
...
[ROCm/rccl commit: 5b72727670
2021-09-15 10:33:25 -07:00
Felix Abecassis
Wenkai Du
John Bachan
Sylvain Jeaugey
gilbertlee-amd
Ziyue Yang
Ke Wen
void-main
Wenkai Du
Chris Jones
Stanley Tsang
Ke Wen