287ed0f18a
Enable collnet in RCCL ( #333 )
...
* Enable CollNet and use different number of channels
* topo_expl: enable collnet
[ROCm/rccl commit: 1d6244b18d
2021-03-19 12:58:13 -07:00
052119c20a
Sort GPUs by HIP device ID ( #329 )
...
* Sort GPUs by HIP device ID
* Remove extra space
[ROCm/rccl commit: b46260260a
2021-03-16 16:51:32 -07:00
bcf4ecb0e3
Enable local sendrecv over network if GDR is available on all GPUs ( #324 )
...
[ROCm/rccl commit: c018edf0f2
2021-03-05 19:59:41 -08:00
020ccb6730
Update tuning parameters for XGMI and NET
...
[ROCm/rccl commit: e820a943e9
2021-02-23 21:41:26 +00:00
d24df90882
Match NBIO only when GPUs and NICs are directly connected to CPU
...
[ROCm/rccl commit: ec8d89b1dd
2021-02-22 18:52:29 -05:00
6c3ccc2192
Add support to another Rome model
...
[ROCm/rccl commit: 95f178324c
2021-02-18 02:00:31 +00:00
ab71643c99
Merge remote-tracking branch 'nccl/master' into 2.8.3
...
[ROCm/rccl commit: c985358e11
2021-02-15 18:44:47 -05:00
fc7bdb38a5
2.8.4-1
...
Fix hang in corner cases of alltoallv using point to point send/recv.
Harmonize error messages.
Fix missing NVTX section in the license.
Update README.
[ROCm/rccl commit: 911d61f214
2021-02-09 15:36:48 -08:00
ae5779702a
Merge remote-tracking branch 'origin/develop' into 2.8.3
...
[ROCm/rccl commit: ab1e7a0318
2021-02-04 20:02:34 -05:00
57abf599b2
Enable GPU direct RDMA read from GPU
...
[ROCm/rccl commit: 5f97122442
2021-02-03 02:48:30 +00:00
2c49121171
Port alltoall[v]
...
[ROCm/rccl commit: f4d5d3d620
2021-01-14 19:28:01 -05:00
41bead5a4e
Do not allow GPU as intermediate
...
[ROCm/rccl commit: 105db19a11
2021-01-14 19:28:01 -05:00
34c6013299
Revert "Changes to topology based on XGMI ( #272 )"
...
This reverts commit 0a9adc16f4e055229e56
2021-01-14 19:28:01 -05:00
adff98765c
Merge remote-tracking branch 'nccl/master' into no-target-id
...
[ROCm/rccl commit: d469947641
2021-01-14 19:27:53 -05:00
1db601566f
x86: Add CPU detection for Zhaoxin processors
...
Signed-off-by: Jonas Zhou <JonasZhou@zhaoxin.com >
[ROCm/rccl commit: 3996562690
2020-12-17 11:15:18 -08:00
4ea285c527
Fix Rome PCIe 2 node topology generation ( #310 )
...
[ROCm/rccl commit: 373a108516
2020-12-15 17:16:17 -08:00
b68ff1ebba
Add Rome model and improve search ( #305 )
...
[ROCm/rccl commit: 975b14dffa
2020-11-17 14:55:06 -08:00
a8908b34ee
2.8.3-1
...
Optimization for Tree allreduce on A100.
Improve aggregation performance.
Use shared buffers for inter-node send/recv.
Add NVTX profiling hooks.
Accelerate alltoall connections by merging communication for all
channels.
Add support for one hop communication through NVLink, for faster
send/recv communication on cubemesh topologies like DGX-1.
Improve alltoall scheduling to better balance intra/inter node
communication.
Increase send/recv parallelism by 8x, each warp sending or
receiving to a different peer.
Net: move to v4.
Net: make flush operation asynchronous to accelerate alltoall.
Net: define maximum number of requests.
Fix hang when using LL128 protocol after 2^31 steps.
Fix #379 : topology injection failing when using less GPUs than
described in the XML.
Fix #394 : protocol mismatch causing hangs or crashes when using
one GPU per node.
[ROCm/rccl commit: 920dbe5b35
2020-11-17 11:08:52 -08:00
f19cbc8e51
Use device's link width and speed if port doesn't report ( #304 )
...
[ROCm/rccl commit: 554729079d
2020-11-13 17:58:04 -08:00
a4dd1a9548
Improve GPU direct RDMA handling on Rome ( #294 )
...
[ROCm/rccl commit: 709b7e4880
2020-11-03 14:29:08 -08:00
c0c64d970a
Add more Rome models ( #292 )
...
[ROCm/rccl commit: dfa3c41ede
2020-10-30 21:26:04 -07:00
8b120c0508
Update Rome single node models ( #277 )
...
[ROCm/rccl commit: 33babcb5e2
2020-10-13 13:33:09 -07:00
41260bb948
Rework Rome detection and add multiple network ports models ( #274 )
...
* Rework Rome detection and add multiple network ports models
* Remove unused opCount in p2p transport
[ROCm/rccl commit: ae008fd2db
2020-10-07 13:37:36 -07:00
dbde26e681
Add Alltoallv RCCL kernel implementation ( #269 )
...
* Add alltoallv API and implementation
* Extend Rome P2P channel limit to multinode and alltoall kernels
* topo_expl: fix compilation and sync up with main
* gtest: use RCCL alltoallv API
* Code review changes
[ROCm/rccl commit: b871ea3c0c
2020-09-30 16:25:36 -07:00
0a9adc16f4
Changes to topology based on XGMI ( #272 )
...
* Alterations to topology search to improve XGMI-enabled nodes
[ROCm/rccl commit: 01bd2573db
2020-09-25 12:20:09 -06:00
f0a303664e
Limit P2P channels on Rome
...
[ROCm/rccl commit: 42955f5f4f
2020-09-17 17:20:32 -07:00
03bb6bcb54
Increase minimal channels for gfx908 ( #259 )
...
[ROCm/rccl commit: c5cbece6d0
2020-08-26 11:40:11 -07:00
5f49a0e088
Add NPS4 support on some models ( #256 )
...
* Add NPS4 support on some models
* Add XML models
[ROCm/rccl commit: 391bbf3f1e
2020-08-19 11:03:20 -07:00
3d5fb8142e
Add another Rome model ( #249 )
...
* Add another Rome model
* Add gfx908 4P3L models and support
* Revert "Use cached value for detecting GDR support only once"
This reverts commit 0108a1219da51e4071e3
2020-08-17 10:51:02 -07:00
f242a2f0b0
Collect gcnArch and hipDeviceArch_t in XML ( #252 )
...
[ROCm/rccl commit: 7e3d8a31cc
2020-08-12 15:48:38 -07:00
e5ec2d94d5
Merge pull request #248 from wenkaidu/2.7.8
...
2.7.8
[ROCm/rccl commit: 066223333d
2020-08-11 08:20:37 -07:00
14ad6ff3b4
Merge remote-tracking branch 'nccl/master' into 2.7.8
...
[ROCm/rccl commit: 7e3f841fab
2020-08-10 16:11:00 +00:00
c9815aaa36
Add more Rome 4P2H models
...
[ROCm/rccl commit: 09ef75656a
2020-08-06 18:20:02 +00:00
dca19952fd
Setting type when gpu sub node is discovered
...
[ROCm/rccl commit: de49a77074
2020-08-05 13:39:23 -07:00
d6a78cb1c7
Don't require NIC devices to have specific PCI class
...
If a PCI node is the parent of a NIC, treat it as such, regardless of
the PCI class code for the device. This allows non-traditional devices
to act as NICs via the net plugin mechanism.
For consistency, treat GPUs similarly.
[ROCm/rccl commit: 700c0e0f24
2020-08-05 12:46:29 -07:00
5f96f13e75
Allow setup ring through NCCL_RINGS to facilitate testing
...
[ROCm/rccl commit: 5b03132ace
2020-08-04 21:07:00 +00:00
22a6211eaf
Improve 4P2H topology on Rome ( #243 )
...
1. Use bi-directional rings
2. GPU search is sorted by PCI device ID to get consistent results
[ROCm/rccl commit: d1e20b4c5e
2020-07-28 14:21:44 -07:00
487f93b83f
Topology tuning for 4P2H on Rome ( #242 )
...
* Topology tuning for 4P2H on Rome
* Use ncclTopoIdToIndex
[ROCm/rccl commit: e7a10aa0e4
2020-07-27 11:53:57 -07:00
f604fc774e
Add 8P6L multi-node models ( #239 )
...
[ROCm/rccl commit: d5f90e19b5
2020-07-21 14:10:36 -07:00
3e2c9054cd
Change default channels duplication for chordal ring ( #233 )
...
[ROCm/rccl commit: ab787c767e
2020-07-14 15:16:50 -07:00
c7805f224e
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: da3b197d6c
2020-07-01 16:51:25 -07:00
5432eb8e9d
2.7.6-1
...
Fix crash when NVswitch is not visible inside a VM.
[ROCm/rccl commit: 1952325569
2020-06-26 16:35:54 -07:00
1f6b75ca8b
2.7.5-1
...
Minor fixes for A100 platforms.
Add a WARN for invalid GroupEnd call.
[ROCm/rccl commit: 01afd20a77
2020-06-26 14:39:49 -07:00
e7d664200f
Use posix_memalign for network buffer allocation on host memory ( #221 )
...
* Use posix_memalign for network buffer allocation on host memory
* ib-test: add ability to specify run iterations
* ib-test: define iterations as multiple of default cycles
* Add checking to posix_memalign return value
[ROCm/rccl commit: 0eb19a563a
2020-06-22 13:06:25 -07:00
67fb4f6cdb
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: 26a0fd2517
2020-06-09 17:40:11 -07:00
89072c82e5
2.7.3-1
...
Add support for A100 GPU and related platforms.
Add support for CUDA 11.
Add support for send/receive operations (beta).
[ROCm/rccl commit: 5949d96f36
2020-06-08 09:31:44 -07:00
27519fd019
Give preference to path with more XGMI connections
...
[ROCm/rccl commit: b3c9852634
2020-05-14 15:33:16 -07:00
9813d67cd1
Merge remote-tracking branch 'nccl/master' into HEAD
...
[ROCm/rccl commit: c4edc257b0
2020-04-27 17:16:54 +00:00
7b7f781658
Fix incorrect next device ID in PCI ordered search
...
[ROCm/rccl commit: edb49ed2d5
2020-04-25 01:01:13 +00:00
c43022b9d8
Fix crash when only a subset of GPUs are visible within a container.
...
Fixes #326 .
[ROCm/rccl commit: f36540f55a
2020-04-17 10:03:14 -07:00
Wenkai Du
Wenkai Du
Sylvain Jeaugey
Jonas Zhou
gilbertlee-amd
Jack Snyder
Eric Badger