e820a943e9
Update tuning parameters for XGMI and NET
2021-02-23 21:41:26 +00:00
ec8d89b1dd
Match NBIO only when GPUs and NICs are directly connected to CPU
2021-02-22 18:52:29 -05:00
95f178324c
Add support to another Rome model
2021-02-18 02:00:31 +00:00
c985358e11
Merge remote-tracking branch 'nccl/master' into 2.8.3
2021-02-15 18:44:47 -05:00
911d61f214
2.8.4-1
...
Fix hang in corner cases of alltoallv using point to point send/recv.
Harmonize error messages.
Fix missing NVTX section in the license.
Update README.
2021-02-09 15:36:48 -08:00
ab1e7a0318
Merge remote-tracking branch 'origin/develop' into 2.8.3
2021-02-04 20:02:34 -05:00
5f97122442
Enable GPU direct RDMA read from GPU
2021-02-03 02:48:30 +00:00
f4d5d3d620
Port alltoall[v]
2021-01-14 19:28:01 -05:00
105db19a11
Do not allow GPU as intermediate
2021-01-14 19:28:01 -05:00
e055229e56
Revert "Changes to topology based on XGMI ( #272 )"
...
This reverts commit 01bd2573db
2021-01-14 19:28:01 -05:00
d469947641
Merge remote-tracking branch 'nccl/master' into no-target-id
2021-01-14 19:27:53 -05:00
3996562690
x86: Add CPU detection for Zhaoxin processors
...
Signed-off-by: Jonas Zhou <JonasZhou@zhaoxin.com >
2020-12-17 11:15:18 -08:00
373a108516
Fix Rome PCIe 2 node topology generation ( #310 )
2020-12-15 17:16:17 -08:00
975b14dffa
Add Rome model and improve search ( #305 )
2020-11-17 14:55:06 -08:00
920dbe5b35
2.8.3-1
...
Optimization for Tree allreduce on A100.
Improve aggregation performance.
Use shared buffers for inter-node send/recv.
Add NVTX profiling hooks.
Accelerate alltoall connections by merging communication for all
channels.
Add support for one hop communication through NVLink, for faster
send/recv communication on cubemesh topologies like DGX-1.
Improve alltoall scheduling to better balance intra/inter node
communication.
Increase send/recv parallelism by 8x, each warp sending or
receiving to a different peer.
Net: move to v4.
Net: make flush operation asynchronous to accelerate alltoall.
Net: define maximum number of requests.
Fix hang when using LL128 protocol after 2^31 steps.
Fix #379 : topology injection failing when using less GPUs than
described in the XML.
Fix #394 : protocol mismatch causing hangs or crashes when using
one GPU per node.
2020-11-17 11:08:52 -08:00
554729079d
Use device's link width and speed if port doesn't report ( #304 )
2020-11-13 17:58:04 -08:00
709b7e4880
Improve GPU direct RDMA handling on Rome ( #294 )
2020-11-03 14:29:08 -08:00
dfa3c41ede
Add more Rome models ( #292 )
2020-10-30 21:26:04 -07:00
33babcb5e2
Update Rome single node models ( #277 )
2020-10-13 13:33:09 -07:00
ae008fd2db
Rework Rome detection and add multiple network ports models ( #274 )
...
* Rework Rome detection and add multiple network ports models
* Remove unused opCount in p2p transport
2020-10-07 13:37:36 -07:00
b871ea3c0c
Add Alltoallv RCCL kernel implementation ( #269 )
...
* Add alltoallv API and implementation
* Extend Rome P2P channel limit to multinode and alltoall kernels
* topo_expl: fix compilation and sync up with main
* gtest: use RCCL alltoallv API
* Code review changes
2020-09-30 16:25:36 -07:00
01bd2573db
Changes to topology based on XGMI ( #272 )
...
* Alterations to topology search to improve XGMI-enabled nodes
2020-09-25 12:20:09 -06:00
42955f5f4f
Limit P2P channels on Rome
2020-09-17 17:20:32 -07:00
c5cbece6d0
Increase minimal channels for gfx908 ( #259 )
2020-08-26 11:40:11 -07:00
391bbf3f1e
Add NPS4 support on some models ( #256 )
...
* Add NPS4 support on some models
* Add XML models
2020-08-19 11:03:20 -07:00
a51e4071e3
Add another Rome model ( #249 )
...
* Add another Rome model
* Add gfx908 4P3L models and support
* Revert "Use cached value for detecting GDR support only once"
This reverts commit 67c8e72ce3
2020-08-17 10:51:02 -07:00
7e3d8a31cc
Collect gcnArch and hipDeviceArch_t in XML ( #252 )
2020-08-12 15:48:38 -07:00
066223333d
Merge pull request #248 from wenkaidu/2.7.8
...
2.7.8
2020-08-11 08:20:37 -07:00
7e3f841fab
Merge remote-tracking branch 'nccl/master' into 2.7.8
2020-08-10 16:11:00 +00:00
09ef75656a
Add more Rome 4P2H models
2020-08-06 18:20:02 +00:00
de49a77074
Setting type when gpu sub node is discovered
2020-08-05 13:39:23 -07:00
700c0e0f24
Don't require NIC devices to have specific PCI class
...
If a PCI node is the parent of a NIC, treat it as such, regardless of
the PCI class code for the device. This allows non-traditional devices
to act as NICs via the net plugin mechanism.
For consistency, treat GPUs similarly.
2020-08-05 12:46:29 -07:00
5b03132ace
Allow setup ring through NCCL_RINGS to facilitate testing
2020-08-04 21:07:00 +00:00
d1e20b4c5e
Improve 4P2H topology on Rome ( #243 )
...
1. Use bi-directional rings
2. GPU search is sorted by PCI device ID to get consistent results
2020-07-28 14:21:44 -07:00
e7a10aa0e4
Topology tuning for 4P2H on Rome ( #242 )
...
* Topology tuning for 4P2H on Rome
* Use ncclTopoIdToIndex
2020-07-27 11:53:57 -07:00
d5f90e19b5
Add 8P6L multi-node models ( #239 )
2020-07-21 14:10:36 -07:00
ab787c767e
Change default channels duplication for chordal ring ( #233 )
2020-07-14 15:16:50 -07:00
da3b197d6c
Merge remote-tracking branch 'nccl/master' into develop
2020-07-01 16:51:25 -07:00
1952325569
2.7.6-1
...
Fix crash when NVswitch is not visible inside a VM.
2020-06-26 16:35:54 -07:00
01afd20a77
2.7.5-1
...
Minor fixes for A100 platforms.
Add a WARN for invalid GroupEnd call.
2020-06-26 14:39:49 -07:00
0eb19a563a
Use posix_memalign for network buffer allocation on host memory ( #221 )
...
* Use posix_memalign for network buffer allocation on host memory
* ib-test: add ability to specify run iterations
* ib-test: define iterations as multiple of default cycles
* Add checking to posix_memalign return value
2020-06-22 13:06:25 -07:00
26a0fd2517
Merge remote-tracking branch 'nccl/master' into develop
2020-06-09 17:40:11 -07:00
5949d96f36
2.7.3-1
...
Add support for A100 GPU and related platforms.
Add support for CUDA 11.
Add support for send/receive operations (beta).
2020-06-08 09:31:44 -07:00
b3c9852634
Give preference to path with more XGMI connections
2020-05-14 15:33:16 -07:00
c4edc257b0
Merge remote-tracking branch 'nccl/master' into HEAD
2020-04-27 17:16:54 +00:00
edb49ed2d5
Fix incorrect next device ID in PCI ordered search
2020-04-25 01:01:13 +00:00
f36540f55a
Fix crash when only a subset of GPUs are visible within a container.
...
Fixes #326 .
2020-04-17 10:03:14 -07:00
23a9fbb788
Improve robustness of PCI detection
...
Fallback to default values when class/speed is unknown.
2020-04-16 14:27:50 -07:00
b5b6c6acdd
Fix bug #307 : wrong NIC selection on the reduction tree.
...
The reduction tree (tree up) was inverting the NICs to use,
causing performance issue in cases where we are using different
NICs on a given channel.
2020-04-09 17:14:07 -07:00
a95090d981
Fix HIP-Clang build with HSA headers
...
HIP-Clang does not include these HSA headers, and they need to be explicitly added in RCCL.
2020-04-03 17:58:23 -04:00
Wenkai Du
Sylvain Jeaugey
Jonas Zhou
Wenkai Du
gilbertlee-amd
Jack Snyder
Eric Badger
Aaron Enye Shi