56155ff5b6
Use rocm_smi_lib for getting topology information ( #402 )
...
* Use rocm_smi_lib for getting topology information
* Add rocm-smi-lib dependency to RCCL package
2021-07-08 13:23:11 -07:00
fa6d7e9a63
Fixes for NCCL_MAX_NCHANNELS and topo_expl ( #398 )
2021-06-22 08:41:49 -07:00
6dcae8a459
Select sendrecv path based on collective data size ( #391 )
...
* Select sendrecv path based on collective data size
* Add comments on packing and unpacking group field
* Toggling RCCL_P2P_NET_DISABLE in combined calls unit tests
2021-06-10 17:51:04 -07:00
b815a2800f
Setup collectives threshold for enabling intranet ( #387 )
...
* Setup collectives threshold for enabling intranet
* Use separate operation counters for coll and p2p
2021-06-09 13:24:26 -07:00
c2064adcc7
Add support for another Rome model ( #385 )
2021-06-08 13:58:20 -07:00
a3a8c2d56b
Allow intranode use of network connection ( #383 )
...
* Allow intranode use of network connection
* Checking for graph for null pointer
2021-06-08 07:37:59 -07:00
961922ea02
Add option to enable multiple SAT in SHARP ( #380 )
...
* Add option to enable multiple SAT in SHARP
* Extend number of NICs to 16
2021-06-03 19:45:18 -07:00
e3abf1c2ec
Merge remote-tracking branch 'nccl/master' into develop
2021-05-25 20:52:15 -07:00
4c83adb75c
Update Rome models matching ( #376 )
2021-05-25 10:12:40 -07:00
8e817ecd6d
Tweak clique channel usage for gfx908 ( #374 )
2021-05-21 15:36:21 -06:00
50da1b48af
Correction on max number of groups ( #373 )
2021-05-20 08:58:45 -07:00
8cde34be51
Use fixed segment size for sendrecv ( #369 )
2021-05-19 08:25:26 -07:00
ddceadc313
Tune clique-based AllReduce for device type 908 ( #372 )
...
* Changing switch-over point for clique-based AllReduce
2021-05-18 15:36:07 -06:00
87727383fe
Merge remote-tracking branch 'nccl/master' into 2.9.8
2021-05-17 10:15:16 -07:00
0b2bfdd6d8
Multiprocess unit test various fixes ( #367 )
...
* Re-enabling mp unit tests
* Fixing shared memory leak and other bugs related to shared mem for MP unit tests
* Revert 43bfbfc97bf9edbae1f386d461439091618ff8ed
* Further tightening up unlinks
* Moving test check macros to separate header file
* Tightening up shared memory unlinking for clique kernels, add munmap for host barrier for MP unit tests
* Updating new MP unit test
* Fixing mqueue bug
* Fixing memory leak in MP unit tests
2021-05-14 09:38:49 -06:00
caf5c9992a
Allow user to link layer with RCCL_IB_HCA_SKIP_LINK_LAYER ( #361 )
...
To skip Infiniband, set RCCL_IB_HCA_SKIP_LINK_LAYER=1.
To skip Ethernet, RCCL_IB_HCA_SKIP_LINK_LAYER=2.
2021-05-12 14:14:53 -07:00
3fec2fa5ee
2.9.9-1
...
Fix crash when setting NCCL_MAX_P2P_NCHANNELS below nchannels.
Fix hang during sendrecv dynamic NVB connection establishment on
cubemesh topologies.
Add environment variable to only use SHARP on communicators beyond
a given number of ranks.
Add debug subsystem to trace memory allocations.
Fix compilation with TRACE=1. (Issue #505 )
2021-05-12 11:09:31 -07:00
e796b1645c
Clique tuning upgrade ( #352 ) ( #19 )
...
* Enabling clique for any XGMI-connected topology, adding tuning
* Updating CHANGELOG for clique tuning
* Re-working clique barrier system to work on multi-process / multi-gpu
2021-05-11 08:44:59 -06:00
ca8485b0d0
2.9.8-1
...
Fix memory leaks.
Fix crash in bootstrap error case.
Fix Collnet clean-up issue.
Make PCI switch vendor/device optional for XML injection.
Add support for nvidia-peermem module.
2021-05-10 14:00:03 -07:00
a4ea1fed5b
Merge remote-tracking branch 'nccl/master' into develop
2021-05-05 16:01:01 -07:00
4f8e788a61
Fixing potential race-condition in env var parameter macro ( #359 )
2021-04-28 12:04:41 -06:00
ed237dcaa7
Use better name for kernel collective trace enable ( #357 )
...
"NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,COLL" enables collectives API
trace. Adding "RCCL_KERNEL_COLL_TRACE_ENABLE=1" enables kernel traces.
2021-04-26 08:35:53 -07:00
9cc9c3360b
Control collective trace from kernel separately ( #356 )
2021-04-23 16:36:19 -07:00
70597789d0
Message queue refactor to POSIX implementation and leak fix ( #355 )
...
* Fixing message queue leak.
* Using POSIX implementation of Message Queues
* Adding unlink to msgqueue
* MsgQueue update
* Adding timeout check to msgqueue broadcast; tightening up system checks
* Removing unnecessary code
* Removing extra argument from print
* Adding explicit msg queue close call to all other ranks
2021-04-23 11:33:20 -06:00
415c7cd3d1
Tune number of channels for gfx90a ( #349 )
2021-04-19 15:27:01 -07:00
9c718ce6d6
Use correct WARP_SIZE for gfx1030 ( #348 )
2021-04-14 14:09:52 -07:00
a79f74082e
Limit max channels for ring graph on single node Rome ( #347 )
...
* Limit max channels for ring graph on single node Rome
* Partially revert "Use non-temporal access for streaming data (#341 )"
2021-04-14 10:14:54 -07:00
1fe031402a
Add gfx90a target ( #344 )
...
* Add gfx90a target
* Support gfx90a topology
Co-authored-by: Eiden Yoshida <eiden.yoshida@amd.com >
2021-04-14 09:29:00 -06:00
a46ea10583
2.9.6-1
...
Add support for CUDA graphs.
Fuse BCM Gen4 switches to avoid suboptimal performance on some platforms. Issue #439 .
Fix bootstrap issue caused by connection reordering.
Fix CPU locking block.
Improve CollNet algorithm.
Improve performance on DGX A100 for communicators with only one GPU per node.
2021-04-12 16:00:46 -07:00
87f12cbb86
Add detection of cooperative multi device launch attribute ( #345 )
2021-04-11 13:29:24 -07:00
9dfc2c183e
Use non-temporal access for streaming data ( #341 )
...
* Use non-temporal access for streaming data
* Revert to ulong2 after fixing compiling issue
2021-04-07 17:34:35 -07:00
caba0a63d2
Fixing clique-topology detection ( #342 )
...
* Fixing clique-topology detection
* Fix to enable multi-process clique-based kernels
2021-04-07 11:29:44 -06:00
e26ad2995e
Cleanup number of channels calculation ( #340 )
2021-04-05 17:51:56 -07:00
17491c918e
Fix incorrect net counting ( #339 )
...
* Fix incorrect net counting
* Add comments
2021-04-05 12:21:57 -07:00
1d2946ee4b
Rework network port trimming code ( #338 )
...
* Rework network port trimming code
* Move Rome related changes to separate source files
2021-03-31 10:25:59 -07:00
0c78553ee0
Check fine grained memory on peer GPU before enabling P2P ( #337 )
2021-03-30 09:06:39 -07:00
d87dc7c2e8
collnet: support multiple NICs ( #335 )
2021-03-25 20:59:32 -07:00
289db2a636
Fixing message queue leak. ( #331 )
2021-03-25 19:11:43 -06:00
0fbb9510a5
Remove HDP workaround for ROCm 4.2 HIP ( #334 )
2021-03-23 20:11:37 -07:00
1d6244b18d
Enable collnet in RCCL ( #333 )
...
* Enable CollNet and use different number of channels
* topo_expl: enable collnet
2021-03-19 12:58:13 -07:00
b46260260a
Sort GPUs by HIP device ID ( #329 )
...
* Sort GPUs by HIP device ID
* Remove extra space
2021-03-16 16:51:32 -07:00
f60b76c67a
Add GPU memory usage tracker ( #326 )
2021-03-06 20:32:30 -08:00
8e180cf087
Revert "Port alltoall[v]" ( #325 )
...
This reverts commit f4d5d3d620
2021-03-06 13:59:31 -08:00
c018edf0f2
Enable local sendrecv over network if GDR is available on all GPUs ( #324 )
2021-03-05 19:59:41 -08:00
f4a9b9acba
Adding pthread_join / pthread_detach to clean up pthreads to avoid leaks ( #322 )
2021-02-26 16:29:55 -07:00
e820a943e9
Update tuning parameters for XGMI and NET
2021-02-23 21:41:26 +00:00
ec8d89b1dd
Match NBIO only when GPUs and NICs are directly connected to CPU
2021-02-22 18:52:29 -05:00
45f5255f7c
Fixing cache deletion for CliqueManager; updating copyright
2021-02-19 22:22:46 +00:00
95f178324c
Add support to another Rome model
2021-02-18 02:00:31 +00:00
c985358e11
Merge remote-tracking branch 'nccl/master' into 2.8.3
2021-02-15 18:44:47 -05:00
Wenkai Du
Wenkai Du
gilbertlee-amd
Stanley Tsang
Sylvain Jeaugey
TomSang