347ea354c2
Update tuning parameters
...
[ROCm/rccl commit: 83fd4f70e7
2022-04-18 16:04:04 -07:00
67e7e6507e
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: d28e1cb44f
2022-04-18 11:15:25 -07:00
58b2c1ec9c
Fix random segfault ( #537 )
...
[ROCm/rccl commit: fd2f1b3b88
2022-04-15 14:32:11 -07:00
011447e4dc
Add new Rome model ( #536 )
...
[ROCm/rccl commit: 2151c79d14
2022-04-13 11:45:40 -07:00
f8023f2e07
Add new Rome model ( #535 )
...
[ROCm/rccl commit: ba4c165bf3
2022-04-12 13:27:32 -07:00
e89ff21d35
Update Makefile to install static library.
...
Make sure make install also installs the static library.
Fixes #662
[ROCm/rccl commit: 9bfc1c6e35
2022-04-08 14:00:43 +02:00
5ccdd9f5e1
Increase chunk steps of broadcast and reduce ( #528 )
...
[ROCm/rccl commit: 15b572751e
2022-04-07 13:34:04 -07:00
3830310156
Doxygen fix for ncclRecv ( #527 )
...
Changed the Doxygen command for ncclRecv and pncclRecv, to be consistent with other APIs.
[ROCm/rccl commit: b2ffcf6d89
2022-04-05 14:07:56 -07:00
9884e61367
Add tuning model ( #523 )
...
[ROCm/rccl commit: 5cc0a405c0
2022-04-04 10:19:57 -07:00
3332cdff07
Support multiple tuning tables ( #522 )
...
* Support multiple tuning tables
* [UnitTests] Skip managed memory testing
[ROCm/rccl commit: bbe780ca6c
2022-03-31 17:09:21 -07:00
74f8baa0f3
Merge remote-tracking branch 'origin/master'
...
[ROCm/rccl commit: 8133784b32
2022-03-30 02:29:05 -07:00
27130280b2
2.12.10-1
...
Fix bug with CollNet
Fix bug with zero-bytes send/recv operations
Fix NCCL_PARAM implementation to avoid taking a lock on every call
Fix bug when setting NCCL_IB_QPS_PER_CONNECTION to more than one.
Improve error reporting for network errors.
[ROCm/rccl commit: 353e8ba446
2022-03-30 02:27:01 -07:00
a52e328ba4
Fix merging error
...
[ROCm/rccl commit: 2247152a8e
2022-03-30 02:14:32 -07:00
3bc2e34df2
Merge branch 'master' into truncated_msg_warning
...
[ROCm/rccl commit: 2dfd83752c
2022-03-30 10:58:05 +02:00
92bdde35eb
Display host name instead of numeric IP when referring to a peer
...
For easier interpretation of debug messages like "connection closed by
peer", "peer message truncated" and "peer collective mismatch"
[ROCm/rccl commit: 1382a87306
2022-03-30 10:47:10 +02:00
f6d1c7261f
Fix typo in net_ib.cc
...
[ROCm/rccl commit: b895abcdb8
2022-03-30 10:45:01 +02:00
54590464ca
Remove unnecessary newline in plugin logging
...
Signed-off-by: Felix Abecassis <fabecassis@nvidia.com >
[ROCm/rccl commit: 1c7c014ceb
2022-03-30 10:44:49 +02:00
828f3d11a0
Update tuning parameters ( #518 )
...
* Update tuning parameters
* Respect user algo and topo selections
[ROCm/rccl commit: 7cbbca4da1
2022-03-29 08:15:37 -07:00
db1e628ba3
Update Rome model matching and add new models ( #516 )
...
* Update Rome model matching and add new models
* Add missing file
* Models update
[ROCm/rccl commit: cd17cf6dce
2022-03-21 10:54:40 -07:00
7707479804
Add pthread_detach()'s for threads we never pthread_join(). Helps
...
reduce diagnostic noise for ThreadSanitizer.
Fixes https://github.com/NVIDIA/nccl/issues/649
[ROCm/rccl commit: 44eb40da0e
2022-03-15 10:27:59 -07:00
133aed2dfb
Force ring algorithm on single node ( #509 )
...
[ROCm/rccl commit: d6d6af710e
2022-03-04 10:29:02 -08:00
f8886d8687
2.12.7-1
...
Add network communication through another GPU connected with NVLink
(PXN).
Add aggregation of messages coming from different local GPUs through
PXN and going to the same destination.
Add new v5 plugin API with grouped receives and tags.
Add compat for v4 plugins.
Add naming of NCCL threads to help debugging.
Fix NVLink detection and avoid data corruption when some NVLinks are
down.
Add support for Relaxed Ordering for IB.
Add profiling and timing infrastructure.
[ROCm/rccl commit: 3c223c105a
2022-03-02 20:48:56 +01:00
a182076a0e
Unit test refactor ( #500 )
...
Refactoring and consolidating single-process / multi-process unit testing
[ROCm/rccl commit: 29ad0f5fbe
2022-02-25 08:59:07 -07:00
dfa9b9e958
Add Pivot AllToAll algorithm for Rome model ( #503 )
...
* add a2a pivot interface
* remove debug info
* address comments
* fix bug
* remove custom script
* address comments
* fix bug
[ROCm/rccl commit: b569c0a1db
2022-02-20 21:09:47 -08:00
0f0388ba0b
Allow additional options to be passed in through model's definition ( #501 )
...
[ROCm/rccl commit: 94e0dc8bfd
2022-02-17 08:28:58 -08:00
5b697e40db
Add another Rome model ( #497 )
...
[ROCm/rccl commit: 02096c9936
2022-02-12 10:30:16 -08:00
92d6888bdc
Split IB parameter sanity check into two parts
...
First part on collective mismatch, second part on internal errors
[ROCm/rccl commit: fbfb6ac5d7
2022-02-08 15:21:22 -08:00
51df47f9b8
Revert "remove unused basePath"
...
This reverts commit d973ddac8bcc78e9fab8
2022-01-21 12:30:34 +01:00
d973ddac8b
remove unused basePath
...
[ROCm/rccl commit: 445bc19657
2022-01-21 12:12:26 +01:00
635c0bcc01
Generate proper b-tree with non-repeating channels ( #493 )
...
[ROCm/rccl commit: 400df49dbe
2022-01-19 15:09:17 -08:00
663c1db898
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: 973d0111db
2022-01-17 13:34:36 -08:00
4d43d9ce22
Update Rome models ( #491 )
...
[ROCm/rccl commit: 598c6fdded
2022-01-14 10:03:30 -08:00
93ab252d52
Use noinline and a few other fixes ( #489 )
...
* Use noinline and a few other fixes
* Tune collectives
[ROCm/rccl commit: 6ff7690cb5
2022-01-11 16:51:06 -08:00
123de1621f
Use hipGraph instead of cudaGraph ( #487 )
...
[ROCm/rccl commit: 3669e12432
2022-01-10 08:26:01 -08:00
3f16835565
Tune collectives for 2.11.4 ( #486 )
...
[ROCm/rccl commit: 565fbeb5e9
2022-01-10 08:25:47 -08:00
e5ff692673
Merge pull request #482 from ROCmSoftwarePlatform/2.11.4
...
Sync up with 2.11.4
[ROCm/rccl commit: 4234a638b5
2022-01-05 09:31:51 -08:00
93c7526fdc
Add another Rome model ( #483 )
...
[ROCm/rccl commit: f8d0775a6f
2022-01-05 09:26:31 -08:00
618fbfc644
Merge remote-tracking branch 'origin/develop' into 2.11.4
...
[ROCm/rccl commit: 434ecb0e1f
2022-01-03 09:54:16 -08:00
07d9648aec
Add env NCCL_NET_DISABLE_INTRA
...
Disable NET transport for intra-node communication by setting the env to 1
It provides an option to error out instead of falling back to NET when superior intra-node transports (P2P and SHM) are unavailable
[ROCm/rccl commit: c88c9f873f
2021-12-08 16:28:19 +01:00
7406aed9c9
Update Rome model ( #479 )
...
[ROCm/rccl commit: a94b953bcc
2021-12-03 08:24:51 -08:00
26c275fe16
Improve warning message about truncated messages
...
Display hints of cause so that it would be easier for user to debug.
Also change the error type from InternalError to InvalidUsage as most
of time this is caused by a mismatch in collective size or env settings.
[ROCm/rccl commit: f589932130
2021-12-02 16:13:15 -08:00
dd4a53db37
Update tuning parameters ( #478 )
...
[ROCm/rccl commit: 8a08a2f579
2021-11-30 08:51:11 -08:00
2da8358a74
Minor cppcheck fixes, adding suppression file ( #475 )
...
* Minor cppcheck fixes, adding suppression file
[ROCm/rccl commit: 539de1216f
2021-11-24 10:23:59 -07:00
864c6a645e
Determine fine grained memory availability at RCCL bootstrapping ( #471 )
...
[ROCm/rccl commit: e9bf01fb7e
2021-11-19 08:12:53 -08:00
a52ca36a9e
Perform busIdToInt64 on the stack.
...
I noticed when I enabled `NCCL_DEBUG_SUBSYS=ALLOC` that this function is
called thousands of times, making the log output unintelligible.
Fortunately, this function can be implemented without heap allocations.
[ROCm/rccl commit: 8cf7325d69
2021-11-19 09:35:55 +01:00
f8bd2d0cfa
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: 3a919c1f49
2021-11-11 14:22:12 -08:00
fc5f692f71
Remove extra work element copy ( #465 )
...
[ROCm/rccl commit: e05de8fd26
2021-11-09 13:52:03 -08:00
680660fcf7
Do not use async stream for memory allocation and transport setup without graph ( #460 )
...
[ROCm/rccl commit: 33bdd557c8
2021-11-05 13:39:14 -07:00
fd98ee84b4
Update Rome model matching ( #461 )
...
* Update Rome model matching
* Add another Rome model
* Automatically setup NET GDR level from model
[ROCm/rccl commit: 0331e39f81
2021-11-05 08:53:47 -07:00
befaa35e12
profiling: fix incorrect print out in timing profile ( #457 )
...
[ROCm/rccl commit: 26fc6b0919
2021-11-03 16:22:21 -07:00
Wenkai Du
Wenkai Du
Sylvain Jeaugey
Colin Smith
Ke Wen
Christopher Hesse
Felix Abecassis
John Bachan
gilbertlee-amd
Ziyue Yang
void-main
Chris Jones