98f0809a39
Added creation of new tree and added switch for using treesplit for specific cases ( #551 )
2022-05-25 18:55:14 -04:00
700b473211
Moving opt-in custom signal handler from UnitTests into RCCL ( #550 )
...
* Enable via RCCL_ENABLE_SIGNALHANDLER=1
2022-05-20 09:56:38 -06:00
6707a270b1
Add switch for pivot alltoall kernel ( #549 )
2022-05-17 18:14:04 -07:00
283dc86a73
Refine and add new Rome models ( #548 )
2022-05-17 08:23:59 -07:00
c9919e0e35
Improve LL performance ( #546 )
...
* Improve LL performance
* Add split barriers for LL
2022-05-10 13:32:10 -07:00
46b30c5f9b
Merge pull request #544 from edgargabriel/topic/header-file-include
...
fix cmake logic to handle old and new include dirs
2022-04-28 16:29:08 -05:00
4c4a7cb696
fix cmake logic to handle old and new include dirs
...
Starting from rocm 5.2 there is a reorganization of the
include directories. This pr allows to compile
rccl on both the old and the new directory layout.
This solution is using find_package() for identifying correct
settings for rocm_smi starting from rocm-5.2, and the original (manual)
settings for all previous releases.
Tested with rocm-5.2, 5.1.1, 5.0.2, and 4.5.2.
2022-04-28 14:33:46 -04:00
685bcea127
[TransferBench] Syncing with TransferBench v1.02 ( #541 )
2022-04-27 20:43:24 -06:00
063da25563
topo_expl: fix build and add tuning support ( #539 )
2022-04-26 15:40:07 -07:00
379940dfac
Merge pull request #533 from ROCmSoftwarePlatform/2.12.10
...
Sync up with NCCL 2.12.10
2022-04-26 10:09:37 -07:00
39e3002e19
Merge pull request #530 from edgargabriel/topic/signal-intercept
...
Topic/signal intercept
2022-04-25 10:44:26 -05:00
2bf6d254b6
add a signal handler and backtrace
...
Tweak the signal handler and force non-release build
Increase ulimit locked memory value
Update the singal handler to use bfd symbol resolution.
Include configure logic to find bfd functions.
Add optionally c++ function name demangling
2022-04-25 10:48:17 -04:00
83fd4f70e7
Update tuning parameters
2022-04-18 16:04:04 -07:00
d28e1cb44f
Merge remote-tracking branch 'nccl/master' into develop
2022-04-18 11:15:25 -07:00
fd2f1b3b88
Fix random segfault ( #537 )
2022-04-15 14:32:11 -07:00
2151c79d14
Add new Rome model ( #536 )
2022-04-13 11:45:40 -07:00
ba4c165bf3
Add new Rome model ( #535 )
2022-04-12 13:27:32 -07:00
b83efe9c5c
Installing rccl.h wrapper to /opt/rocm-xxx/include path ( #532 )
...
* Fixing the broken library soft link
* Installing rccl.h wrapper to /opt/rocm-xxx/include path.
This missing wrapper was causing compilation errors with pytorch. Fixing it
2022-04-09 07:55:39 -07:00
def6832287
Transfer bench single stream mode ( #531 )
...
- Adding single stream mode
- Removing some unused env vars
- Adding output to CSV mode for p2p benchmark, topology listing modes
2022-04-08 15:20:55 -06:00
9bfc1c6e35
Update Makefile to install static library.
...
Make sure make install also installs the static library.
Fixes #662
2022-04-08 14:00:43 +02:00
acfb0210ea
Fixing the broken library soft link ( #529 )
2022-04-07 15:19:33 -07:00
15b572751e
Increase chunk steps of broadcast and reduce ( #528 )
2022-04-07 13:34:04 -07:00
b2ffcf6d89
Doxygen fix for ncclRecv ( #527 )
...
Changed the Doxygen command for ncclRecv and pncclRecv, to be consistent with other APIs.
2022-04-05 14:07:56 -07:00
5cc0a405c0
Add tuning model ( #523 )
2022-04-04 10:19:57 -07:00
bbe780ca6c
Support multiple tuning tables ( #522 )
...
* Support multiple tuning tables
* [UnitTests] Skip managed memory testing
2022-03-31 17:09:21 -07:00
8133784b32
Merge remote-tracking branch 'origin/master'
2022-03-30 02:29:05 -07:00
353e8ba446
2.12.10-1
...
Fix bug with CollNet
Fix bug with zero-bytes send/recv operations
Fix NCCL_PARAM implementation to avoid taking a lock on every call
Fix bug when setting NCCL_IB_QPS_PER_CONNECTION to more than one.
Improve error reporting for network errors.
2022-03-30 02:27:01 -07:00
2247152a8e
Fix merging error
2022-03-30 02:14:32 -07:00
2dfd83752c
Merge branch 'master' into truncated_msg_warning
2022-03-30 10:58:05 +02:00
1382a87306
Display host name instead of numeric IP when referring to a peer
...
For easier interpretation of debug messages like "connection closed by
peer", "peer message truncated" and "peer collective mismatch"
2022-03-30 10:47:10 +02:00
b895abcdb8
Fix typo in net_ib.cc
2022-03-30 10:45:01 +02:00
1c7c014ceb
Remove unnecessary newline in plugin logging
...
Signed-off-by: Felix Abecassis <fabecassis@nvidia.com >
2022-03-30 10:44:49 +02:00
7cbbca4da1
Update tuning parameters ( #518 )
...
* Update tuning parameters
* Respect user algo and topo selections
2022-03-29 08:15:37 -07:00
2d558c9abc
Adding explicit request for coarse-grained host memory due to changes in HipHostMalloc ( #517 )
2022-03-25 13:05:07 -06:00
a8f1e61f48
Packages for test and benchmark executables on all supported OSes using CPack. ( #512 )
2022-03-21 15:04:14 -06:00
cd17cf6dce
Update Rome model matching and add new models ( #516 )
...
* Update Rome model matching and add new models
* Add missing file
* Models update
2022-03-21 10:54:40 -07:00
65ea3d80db
Added alltoallv test and optional args variable on collective args ( #514 )
...
* Added alltoallv test and optional args variable on collective args
2022-03-18 13:55:11 -04:00
44eb40da0e
Add pthread_detach()'s for threads we never pthread_join(). Helps
...
reduce diagnostic noise for ThreadSanitizer.
Fixes https://github.com/NVIDIA/nccl/issues/649
2022-03-15 10:27:59 -07:00
a04da71647
Merge pull request #511 from nunnikri/develop
...
File reorganization as per the new defined standard
2022-03-10 08:39:29 -08:00
115461cc04
File reorganization with backward compatibility
...
Updated the header file location and export path
2022-03-10 01:28:41 +00:00
676a4737c1
File reorganization as per the new defined standard
...
The header files will in /opt/rocm-xxx/include/rccl
Libraries and cmake will be in /opt/rocm-xxx/lib folder.
Added wrappers for header files using rocm-cmake functions for backward compatibility.
2022-03-08 17:32:02 +00:00
0687940b84
Changing initialization method for UnitTests ( #510 )
2022-03-07 09:22:55 -07:00
d6d6af710e
Force ring algorithm on single node ( #509 )
2022-03-04 10:29:02 -08:00
b634b2f1c2
Adding NCCL_DEBUG=INFO for CI runs ( #508 )
2022-03-03 18:04:28 -07:00
699dc30f05
[UnitTests] Check process mask for custom tests ( #507 )
2022-03-02 17:24:14 -07:00
ff54e79799
Added Unit test for nccl send recv ( #506 )
...
Added Send Receive test that tests through all pairs
2022-03-02 15:50:16 -05:00
3c223c105a
2.12.7-1
...
Add network communication through another GPU connected with NVLink
(PXN).
Add aggregation of messages coming from different local GPUs through
PXN and going to the same destination.
Add new v5 plugin API with grouped receives and tags.
Add compat for v4 plugins.
Add naming of NCCL threads to help debugging.
Fix NVLink detection and avoid data corruption when some NVLinks are
down.
Add support for Relaxed Ordering for IB.
Add profiling and timing infrastructure.
2022-03-02 20:48:56 +01:00
29ad0f5fbe
Unit test refactor ( #500 )
...
Refactoring and consolidating single-process / multi-process unit testing
2022-02-25 08:59:07 -07:00
b569c0a1db
Add Pivot AllToAll algorithm for Rome model ( #503 )
...
* add a2a pivot interface
* remove debug info
* address comments
* fix bug
* remove custom script
* address comments
* fix bug
2022-02-20 21:09:47 -08:00
94e0dc8bfd
Allow additional options to be passed in through model's definition ( #501 )
2022-02-17 08:28:58 -08:00
akolliasAMD
gilbertlee-amd
Wenkai Du
Edgar Gabriel
Edgar
Wenkai Du
nunnikri
Sylvain Jeaugey
Colin Smith
Ke Wen
Christopher Hesse
Felix Abecassis
Liam Wrubleski
John Bachan
Nirmal Unnikrishnan
Ziyue Yang