3e62ceddc5
Clique kernel support ( #295 ) ( #15 )
...
* Adding experimental clique-based kernels (opt-in only)
Co-authored-by: Stanley Tsang <stanley.tsang@amd.com >
Co-authored-by: Gilbert Lee <gilbert.lee@amd.com >
Co-authored-by: Wenkai Du <43822138+wenkaidu@users.noreply.github.com >
Co-authored-by: Stanley Tsang <stanley.tsang@amd.com >
Co-authored-by: Wenkai Du <43822138+wenkaidu@users.noreply.github.com >
2021-01-28 09:45:01 -07:00
2ddbe6646b
Improve collective trace
2021-01-14 19:28:01 -05:00
f4d5d3d620
Port alltoall[v]
2021-01-14 19:28:01 -05:00
105db19a11
Do not allow GPU as intermediate
2021-01-14 19:28:01 -05:00
e055229e56
Revert "Changes to topology based on XGMI ( #272 )"
...
This reverts commit 01bd2573db
2021-01-14 19:28:01 -05:00
d469947641
Merge remote-tracking branch 'nccl/master' into no-target-id
2021-01-14 19:27:53 -05:00
920dbe5b35
2.8.3-1
...
Optimization for Tree allreduce on A100.
Improve aggregation performance.
Use shared buffers for inter-node send/recv.
Add NVTX profiling hooks.
Accelerate alltoall connections by merging communication for all
channels.
Add support for one hop communication through NVLink, for faster
send/recv communication on cubemesh topologies like DGX-1.
Improve alltoall scheduling to better balance intra/inter node
communication.
Increase send/recv parallelism by 8x, each warp sending or
receiving to a different peer.
Net: move to v4.
Net: make flush operation asynchronous to accelerate alltoall.
Net: define maximum number of requests.
Fix hang when using LL128 protocol after 2^31 steps.
Fix #379 : topology injection failing when using less GPUs than
described in the XML.
Fix #394 : protocol mismatch causing hangs or crashes when using
one GPU per node.
2020-11-17 11:08:52 -08:00
2e8b3a0857
Use ncclSend/ncclRecv for alltoall type of collectives as default ( #297 )
2020-11-09 11:23:17 -08:00
709b7e4880
Improve GPU direct RDMA handling on Rome ( #294 )
2020-11-03 14:29:08 -08:00
dfa3c41ede
Add more Rome models ( #292 )
2020-10-30 21:26:04 -07:00
084207e685
fix proxyArgs for trace log
2020-10-21 09:18:40 -07:00
dcad0ef7cb
Fix incorrect pointer checking for scatter and gather ( #285 )
2020-10-19 13:27:09 -07:00
c835d8263a
Merge remote-tracking branch 'nccl/master' into nccl_sync
2020-10-15 18:42:38 -04:00
84a2541e01
Revert "Initial support for clique-based kernels ( #276 )" ( #280 )
...
This reverts commit 2b8184808d
2020-10-15 11:30:18 -07:00
0e14394c5f
Fix affinity move
2020-10-13 16:58:05 -07:00
c6dbdb0084
Make sure proxy threads inherit the CPU affinity.
2020-10-13 16:37:52 -07:00
33babcb5e2
Update Rome single node models ( #277 )
2020-10-13 13:33:09 -07:00
2b8184808d
Initial support for clique-based kernels ( #276 )
...
* Initial support for clique-based kernels
2020-10-13 11:22:04 -06:00
ae008fd2db
Rework Rome detection and add multiple network ports models ( #274 )
...
* Rework Rome detection and add multiple network ports models
* Remove unused opCount in p2p transport
2020-10-07 13:37:36 -07:00
b871ea3c0c
Add Alltoallv RCCL kernel implementation ( #269 )
...
* Add alltoallv API and implementation
* Extend Rome P2P channel limit to multinode and alltoall kernels
* topo_expl: fix compilation and sync up with main
* gtest: use RCCL alltoallv API
* Code review changes
2020-09-30 16:25:36 -07:00
acca2ae20a
Updating inline asm to not require explicit L1 cache invalidation ( #270 )
2020-09-25 13:46:26 -06:00
01bd2573db
Changes to topology based on XGMI ( #272 )
...
* Alterations to topology search to improve XGMI-enabled nodes
2020-09-25 12:20:09 -06:00
44fcde7835
Ensure all ranks on same send/receive or alltoall kernel path ( #271 )
2020-09-24 08:25:04 -07:00
d871fceb54
Change network plugin name to librccl-net.so ( #266 )
2020-09-18 13:23:30 -07:00
42955f5f4f
Limit P2P channels on Rome
2020-09-17 17:20:32 -07:00
60819dcf8d
Merge pull request #262 from wenkaidu/alignment
...
Make data alignment requirements matching ISA manual
2020-09-08 10:40:42 -07:00
e2042ccf8a
Fix broken profiling build ( #263 )
2020-09-02 15:39:52 -07:00
4751992231
Make data alignment requirements matching ISA manual
...
From https://developer.amd.com/wp-content/resources/Vega_Shader_ISA.pdf
8.1.7. Alignment
For Dword or larger reads or writes, the two LSBs of the byte-address
are ignored, thus forcing Dword alignment.
2020-09-01 21:21:58 +00:00
4180e6409e
Fix incorrect threads split in sendrecv ( #261 )
2020-08-31 17:33:22 -07:00
c5cbece6d0
Increase minimal channels for gfx908 ( #259 )
2020-08-26 11:40:11 -07:00
b0919dc46c
Only use software barrier for synchronization ( #258 )
2020-08-25 13:16:34 -07:00
391bbf3f1e
Add NPS4 support on some models ( #256 )
...
* Add NPS4 support on some models
* Add XML models
2020-08-19 11:03:20 -07:00
a51e4071e3
Add another Rome model ( #249 )
...
* Add another Rome model
* Add gfx908 4P3L models and support
* Revert "Use cached value for detecting GDR support only once"
This reverts commit 67c8e72ce3
2020-08-17 10:51:02 -07:00
7e3d8a31cc
Collect gcnArch and hipDeviceArch_t in XML ( #252 )
2020-08-12 15:48:38 -07:00
066223333d
Merge pull request #248 from wenkaidu/2.7.8
...
2.7.8
2020-08-11 08:20:37 -07:00
7e3f841fab
Merge remote-tracking branch 'nccl/master' into 2.7.8
2020-08-10 16:11:00 +00:00
09ef75656a
Add more Rome 4P2H models
2020-08-06 18:20:02 +00:00
de49a77074
Setting type when gpu sub node is discovered
2020-08-05 13:39:23 -07:00
700c0e0f24
Don't require NIC devices to have specific PCI class
...
If a PCI node is the parent of a NIC, treat it as such, regardless of
the PCI class code for the device. This allows non-traditional devices
to act as NICs via the net plugin mechanism.
For consistency, treat GPUs similarly.
2020-08-05 12:46:29 -07:00
5b03132ace
Allow setup ring through NCCL_RINGS to facilitate testing
2020-08-04 21:07:00 +00:00
d1e20b4c5e
Improve 4P2H topology on Rome ( #243 )
...
1. Use bi-directional rings
2. GPU search is sorted by PCI device ID to get consistent results
2020-07-28 14:21:44 -07:00
033d799524
2.7.8-1
...
Fix collective mismatch error when using ncclSend/ncclRecv
2020-07-27 16:34:09 -07:00
e7a10aa0e4
Topology tuning for 4P2H on Rome ( #242 )
...
* Topology tuning for 4P2H on Rome
* Use ncclTopoIdToIndex
2020-07-27 11:53:57 -07:00
8d5fb920b6
ib-test: support multiple channels ( #241 )
2020-07-27 11:03:12 -07:00
d5f90e19b5
Add 8P6L multi-node models ( #239 )
2020-07-21 14:10:36 -07:00
ab787c767e
Change default channels duplication for chordal ring ( #233 )
2020-07-14 15:16:50 -07:00
5215130168
Revert "Split primitive class to smaller structures" ( #230 )
...
This reverts commit 486fd436af
2020-07-08 11:06:50 -07:00
2d8601701d
Fix build action order
...
Add $(INCTARGETS) to build dependencies of %.o and $(DEVICELIB).
As there were no dep files during the first build, Make may kick off source
compilation before nccl.h got generated, which leads to occasional build
failures on systems with high core count. The build failure could be
reproduced reliably with a `sleep 5` in $(INCDIR)/nccl.h rule.
2020-07-07 10:20:51 -07:00
da3b197d6c
Merge remote-tracking branch 'nccl/master' into develop
2020-07-01 16:51:25 -07:00
964c4c2061
Merge sendrecv kernel from NCCL 2.7.3
...
This commit was cherry-picked and modified from
https://github.com/NVIDIA/nccl/commit/5949d96f36d050e59d05872f8bbffd2549318e95
2020-06-29 08:47:46 -07:00
gilbertlee-amd
Wenkai Du
Sylvain Jeaugey
Wenkai Du
xietingwew
Stanley Tsang
Jack Snyder
Eric Badger
David Addison
Riatre Foo