2dfd83752c
Merge branch 'master' into truncated_msg_warning
2022-03-30 10:58:05 +02:00
1382a87306
Display host name instead of numeric IP when referring to a peer
...
For easier interpretation of debug messages like "connection closed by
peer", "peer message truncated" and "peer collective mismatch"
2022-03-30 10:47:10 +02:00
b895abcdb8
Fix typo in net_ib.cc
2022-03-30 10:45:01 +02:00
1c7c014ceb
Remove unnecessary newline in plugin logging
...
Signed-off-by: Felix Abecassis <fabecassis@nvidia.com >
2022-03-30 10:44:49 +02:00
7cbbca4da1
Update tuning parameters ( #518 )
...
* Update tuning parameters
* Respect user algo and topo selections
2022-03-29 08:15:37 -07:00
cd17cf6dce
Update Rome model matching and add new models ( #516 )
...
* Update Rome model matching and add new models
* Add missing file
* Models update
2022-03-21 10:54:40 -07:00
44eb40da0e
Add pthread_detach()'s for threads we never pthread_join(). Helps
...
reduce diagnostic noise for ThreadSanitizer.
Fixes https://github.com/NVIDIA/nccl/issues/649
2022-03-15 10:27:59 -07:00
d6d6af710e
Force ring algorithm on single node ( #509 )
2022-03-04 10:29:02 -08:00
3c223c105a
2.12.7-1
...
Add network communication through another GPU connected with NVLink
(PXN).
Add aggregation of messages coming from different local GPUs through
PXN and going to the same destination.
Add new v5 plugin API with grouped receives and tags.
Add compat for v4 plugins.
Add naming of NCCL threads to help debugging.
Fix NVLink detection and avoid data corruption when some NVLinks are
down.
Add support for Relaxed Ordering for IB.
Add profiling and timing infrastructure.
2022-03-02 20:48:56 +01:00
29ad0f5fbe
Unit test refactor ( #500 )
...
Refactoring and consolidating single-process / multi-process unit testing
2022-02-25 08:59:07 -07:00
b569c0a1db
Add Pivot AllToAll algorithm for Rome model ( #503 )
...
* add a2a pivot interface
* remove debug info
* address comments
* fix bug
* remove custom script
* address comments
* fix bug
2022-02-20 21:09:47 -08:00
94e0dc8bfd
Allow additional options to be passed in through model's definition ( #501 )
2022-02-17 08:28:58 -08:00
02096c9936
Add another Rome model ( #497 )
2022-02-12 10:30:16 -08:00
fbfb6ac5d7
Split IB parameter sanity check into two parts
...
First part on collective mismatch, second part on internal errors
2022-02-08 15:21:22 -08:00
cc78e9fab8
Revert "remove unused basePath"
...
This reverts commit 445bc19657
2022-01-21 12:30:34 +01:00
445bc19657
remove unused basePath
2022-01-21 12:12:26 +01:00
400df49dbe
Generate proper b-tree with non-repeating channels ( #493 )
2022-01-19 15:09:17 -08:00
973d0111db
Merge remote-tracking branch 'nccl/master' into develop
2022-01-17 13:34:36 -08:00
598c6fdded
Update Rome models ( #491 )
2022-01-14 10:03:30 -08:00
6ff7690cb5
Use noinline and a few other fixes ( #489 )
...
* Use noinline and a few other fixes
* Tune collectives
2022-01-11 16:51:06 -08:00
3669e12432
Use hipGraph instead of cudaGraph ( #487 )
2022-01-10 08:26:01 -08:00
565fbeb5e9
Tune collectives for 2.11.4 ( #486 )
2022-01-10 08:25:47 -08:00
4234a638b5
Merge pull request #482 from ROCmSoftwarePlatform/2.11.4
...
Sync up with 2.11.4
2022-01-05 09:31:51 -08:00
f8d0775a6f
Add another Rome model ( #483 )
2022-01-05 09:26:31 -08:00
434ecb0e1f
Merge remote-tracking branch 'origin/develop' into 2.11.4
2022-01-03 09:54:16 -08:00
c88c9f873f
Add env NCCL_NET_DISABLE_INTRA
...
Disable NET transport for intra-node communication by setting the env to 1
It provides an option to error out instead of falling back to NET when superior intra-node transports (P2P and SHM) are unavailable
2021-12-08 16:28:19 +01:00
a94b953bcc
Update Rome model ( #479 )
2021-12-03 08:24:51 -08:00
f589932130
Improve warning message about truncated messages
...
Display hints of cause so that it would be easier for user to debug.
Also change the error type from InternalError to InvalidUsage as most
of time this is caused by a mismatch in collective size or env settings.
2021-12-02 16:13:15 -08:00
8a08a2f579
Update tuning parameters ( #478 )
2021-11-30 08:51:11 -08:00
539de1216f
Minor cppcheck fixes, adding suppression file ( #475 )
...
* Minor cppcheck fixes, adding suppression file
2021-11-24 10:23:59 -07:00
e9bf01fb7e
Determine fine grained memory availability at RCCL bootstrapping ( #471 )
2021-11-19 08:12:53 -08:00
8cf7325d69
Perform busIdToInt64 on the stack.
...
I noticed when I enabled `NCCL_DEBUG_SUBSYS=ALLOC` that this function is
called thousands of times, making the log output unintelligible.
Fortunately, this function can be implemented without heap allocations.
2021-11-19 09:35:55 +01:00
3a919c1f49
Merge remote-tracking branch 'nccl/master' into develop
2021-11-11 14:22:12 -08:00
e05de8fd26
Remove extra work element copy ( #465 )
2021-11-09 13:52:03 -08:00
33bdd557c8
Do not use async stream for memory allocation and transport setup without graph ( #460 )
2021-11-05 13:39:14 -07:00
0331e39f81
Update Rome model matching ( #461 )
...
* Update Rome model matching
* Add another Rome model
* Automatically setup NET GDR level from model
2021-11-05 08:53:47 -07:00
26fc6b0919
profiling: fix incorrect print out in timing profile ( #457 )
2021-11-03 16:22:21 -07:00
29170a8b5f
Support different protocols and algorithms in all reduce only build ( #455 )
...
* Support different protocols and algorithms in all reduce only build
* Restore deleted line in error
2021-11-02 08:39:08 -07:00
4643a17f83
Check rocm_smi64Config.h on older ROCm build ( #452 )
2021-10-28 07:26:28 -07:00
d221fb672a
Rework kernel launch code ( #449 )
2021-10-28 07:26:11 -07:00
ec36c4c326
Enable timing profiling mode ( #447 )
2021-10-27 08:21:48 -07:00
7e55b211c5
Build AllReduce only mode ( #443 )
...
* Initial commit of all_reduce_only support
* Working AllReduce only build
* Removing printfs and restoring release build
* Restore P2P index
* Updates to build_allreduce_only mode.
* cleaning up macro ifdefs
2021-10-26 17:36:46 -06:00
14a184eb67
Query XGMI link count through rocm_smi_lib API ( #442 )
2021-10-26 10:30:20 -07:00
b4cefc05ed
Fix collnet tuning parameters ( #441 )
2021-10-20 20:45:36 -07:00
2508507d0a
Fix PCIe gen detection ( #437 )
...
* Fix PCIe gen detection
* Update profiling support
2021-10-15 08:23:50 -07:00
2249a1d9d3
Add more Rome models ( #434 )
...
* Add more Rome models
* Update models and tuning
* Update tuning
2021-10-12 08:23:20 -07:00
e0053311c0
Add another Rome model ( #431 )
2021-10-06 08:17:12 -07:00
29c729d8b6
Trim NICs when all GPUs are connected by XGMI ( #430 )
...
* Trim NICs when all GPUs are connected by XGMI
* Only enable clique with maximum of 2 hops
2021-10-05 18:27:43 -07:00
30ca3fcacf
Fix compilation failure in "src/enqueue.cc" on older GCC because of
...
missing `#include <cstring>`.
2021-09-23 09:55:16 -07:00
4ec992fab7
Fix Collnet when GDR is disabled
2021-09-22 05:19:16 -07:00
Sylvain Jeaugey
Ke Wen
Christopher Hesse
Felix Abecassis
Wenkai Du
John Bachan
gilbertlee-amd
Ziyue Yang
void-main
Wenkai Du
Chris Jones
Stanley Tsang