6e93fafdc3
Add Feature - Add NPKit Support in RCCL ( #564 )
...
* apply npkit
* fix bug
* add npkit in readme
2022-06-20 14:30:19 -07:00
f274c865c1
Change default nchannels per peer ( #563 )
2022-06-13 06:39:05 -07:00
0336ffdf70
Introduce multi-rank support per device.
...
This is a single commit of the source code changes required to
introduce support for multiple ranks per device.
A new interface (ncclCommRankInitMulti) has to be used to make use of
this new feature.
2022-06-10 14:23:12 +00:00
5cb2aca3d9
Fix P2P scheduling ( #560 )
2022-06-06 13:32:28 -07:00
7a6c6927ae
Enable timing profile option ( #558 )
2022-06-03 07:05:13 -07:00
e0864e7093
Merge remote-tracking branch 'ncclRepo/master' into develop
2022-06-02 15:27:24 +00:00
eef812bed7
Revert chunksteps changes ( #555 )
2022-05-31 14:45:51 -07:00
ef499c4810
Add another Rome model ( #553 )
...
* Add another Rome model
* Add option to force enable intranet on single node
* Limit p2p channels to number of ranks
* Refine p2p channels handling
2022-05-31 11:31:30 -07:00
a0a686e74c
code cleanup ( #554 )
2022-05-31 09:59:36 -04:00
c5b77121f0
Update Rome model ( #552 )
2022-05-26 09:59:23 -07:00
98f0809a39
Added creation of new tree and added switch for using treesplit for specific cases ( #551 )
2022-05-25 18:55:14 -04:00
700b473211
Moving opt-in custom signal handler from UnitTests into RCCL ( #550 )
...
* Enable via RCCL_ENABLE_SIGNALHANDLER=1
2022-05-20 09:56:38 -06:00
6707a270b1
Add switch for pivot alltoall kernel ( #549 )
2022-05-17 18:14:04 -07:00
283dc86a73
Refine and add new Rome models ( #548 )
2022-05-17 08:23:59 -07:00
7aa1c46fd5
2.12.12-1
...
Improve allreduce performance when we have more than one network interface per
GPU and we need to use PXN to close rings.
Add support for PCI Gen5 on 5.4 kernels.
Fix crash when setting NCCL_SET_THREAD_NAME.
Fix random crash in init due to uninitialized struct.
Fix hang on cubemesh topologies.
Add P2P_DIRECT_DISABLE parameter to disable direct access to pointers within a
process.
2022-05-13 00:26:57 -07:00
c9919e0e35
Improve LL performance ( #546 )
...
* Improve LL performance
* Add split barriers for LL
2022-05-10 13:32:10 -07:00
83fd4f70e7
Update tuning parameters
2022-04-18 16:04:04 -07:00
d28e1cb44f
Merge remote-tracking branch 'nccl/master' into develop
2022-04-18 11:15:25 -07:00
fd2f1b3b88
Fix random segfault ( #537 )
2022-04-15 14:32:11 -07:00
2151c79d14
Add new Rome model ( #536 )
2022-04-13 11:45:40 -07:00
ba4c165bf3
Add new Rome model ( #535 )
2022-04-12 13:27:32 -07:00
9bfc1c6e35
Update Makefile to install static library.
...
Make sure make install also installs the static library.
Fixes #662
2022-04-08 14:00:43 +02:00
15b572751e
Increase chunk steps of broadcast and reduce ( #528 )
2022-04-07 13:34:04 -07:00
b2ffcf6d89
Doxygen fix for ncclRecv ( #527 )
...
Changed the Doxygen command for ncclRecv and pncclRecv, to be consistent with other APIs.
2022-04-05 14:07:56 -07:00
5cc0a405c0
Add tuning model ( #523 )
2022-04-04 10:19:57 -07:00
bbe780ca6c
Support multiple tuning tables ( #522 )
...
* Support multiple tuning tables
* [UnitTests] Skip managed memory testing
2022-03-31 17:09:21 -07:00
8133784b32
Merge remote-tracking branch 'origin/master'
2022-03-30 02:29:05 -07:00
353e8ba446
2.12.10-1
...
Fix bug with CollNet
Fix bug with zero-bytes send/recv operations
Fix NCCL_PARAM implementation to avoid taking a lock on every call
Fix bug when setting NCCL_IB_QPS_PER_CONNECTION to more than one.
Improve error reporting for network errors.
2022-03-30 02:27:01 -07:00
2247152a8e
Fix merging error
2022-03-30 02:14:32 -07:00
2dfd83752c
Merge branch 'master' into truncated_msg_warning
2022-03-30 10:58:05 +02:00
1382a87306
Display host name instead of numeric IP when referring to a peer
...
For easier interpretation of debug messages like "connection closed by
peer", "peer message truncated" and "peer collective mismatch"
2022-03-30 10:47:10 +02:00
b895abcdb8
Fix typo in net_ib.cc
2022-03-30 10:45:01 +02:00
1c7c014ceb
Remove unnecessary newline in plugin logging
...
Signed-off-by: Felix Abecassis <fabecassis@nvidia.com >
2022-03-30 10:44:49 +02:00
7cbbca4da1
Update tuning parameters ( #518 )
...
* Update tuning parameters
* Respect user algo and topo selections
2022-03-29 08:15:37 -07:00
cd17cf6dce
Update Rome model matching and add new models ( #516 )
...
* Update Rome model matching and add new models
* Add missing file
* Models update
2022-03-21 10:54:40 -07:00
44eb40da0e
Add pthread_detach()'s for threads we never pthread_join(). Helps
...
reduce diagnostic noise for ThreadSanitizer.
Fixes https://github.com/NVIDIA/nccl/issues/649
2022-03-15 10:27:59 -07:00
d6d6af710e
Force ring algorithm on single node ( #509 )
2022-03-04 10:29:02 -08:00
3c223c105a
2.12.7-1
...
Add network communication through another GPU connected with NVLink
(PXN).
Add aggregation of messages coming from different local GPUs through
PXN and going to the same destination.
Add new v5 plugin API with grouped receives and tags.
Add compat for v4 plugins.
Add naming of NCCL threads to help debugging.
Fix NVLink detection and avoid data corruption when some NVLinks are
down.
Add support for Relaxed Ordering for IB.
Add profiling and timing infrastructure.
2022-03-02 20:48:56 +01:00
29ad0f5fbe
Unit test refactor ( #500 )
...
Refactoring and consolidating single-process / multi-process unit testing
2022-02-25 08:59:07 -07:00
b569c0a1db
Add Pivot AllToAll algorithm for Rome model ( #503 )
...
* add a2a pivot interface
* remove debug info
* address comments
* fix bug
* remove custom script
* address comments
* fix bug
2022-02-20 21:09:47 -08:00
94e0dc8bfd
Allow additional options to be passed in through model's definition ( #501 )
2022-02-17 08:28:58 -08:00
02096c9936
Add another Rome model ( #497 )
2022-02-12 10:30:16 -08:00
fbfb6ac5d7
Split IB parameter sanity check into two parts
...
First part on collective mismatch, second part on internal errors
2022-02-08 15:21:22 -08:00
cc78e9fab8
Revert "remove unused basePath"
...
This reverts commit 445bc19657
2022-01-21 12:30:34 +01:00
445bc19657
remove unused basePath
2022-01-21 12:12:26 +01:00
400df49dbe
Generate proper b-tree with non-repeating channels ( #493 )
2022-01-19 15:09:17 -08:00
973d0111db
Merge remote-tracking branch 'nccl/master' into develop
2022-01-17 13:34:36 -08:00
598c6fdded
Update Rome models ( #491 )
2022-01-14 10:03:30 -08:00
6ff7690cb5
Use noinline and a few other fixes ( #489 )
...
* Use noinline and a few other fixes
* Tune collectives
2022-01-11 16:51:06 -08:00
3669e12432
Use hipGraph instead of cudaGraph ( #487 )
2022-01-10 08:26:01 -08:00
Ziyue Yang
Wenkai Du
Edgar
Aristotelis
akolliasAMD
gilbertlee-amd
Sylvain Jeaugey
Wenkai Du
Colin Smith
Ke Wen
Christopher Hesse
Felix Abecassis
John Bachan
Ziyue Yang
void-main