Commit Graph

180 Commits

Author SHA1 Message Date
Wenkai Du a51e4071e3 Add another Rome model (#249)
* Add another Rome model

* Add gfx908 4P3L models and support

* Revert "Use cached value for detecting GDR support only once"

This reverts commit 67c8e72ce3.

* Skip using ibverb for GPU direct RDMA detection

* Fine tune one Rome model
2020-08-17 10:51:02 -07:00
Wenkai Du 7e3d8a31cc Collect gcnArch and hipDeviceArch_t in XML (#252) 2020-08-12 15:48:38 -07:00
Wenkai Du 066223333d Merge pull request #248 from wenkaidu/2.7.8
2.7.8
2020-08-11 08:20:37 -07:00
Wenkai Du 7e3f841fab Merge remote-tracking branch 'nccl/master' into 2.7.8 2020-08-10 16:11:00 +00:00
Wenkai Du 09ef75656a Add more Rome 4P2H models 2020-08-06 18:20:02 +00:00
Jack Snyder de49a77074 Setting type when gpu sub node is discovered 2020-08-05 13:39:23 -07:00
Eric Badger 700c0e0f24 Don't require NIC devices to have specific PCI class
If a PCI node is the parent of a NIC, treat it as such, regardless of
the PCI class code for the device. This allows non-traditional devices
to act as NICs via the net plugin mechanism.

For consistency, treat GPUs similarly.
2020-08-05 12:46:29 -07:00
Wenkai Du 5b03132ace Allow setup ring through NCCL_RINGS to facilitate testing 2020-08-04 21:07:00 +00:00
Wenkai Du d1e20b4c5e Improve 4P2H topology on Rome (#243)
1. Use bi-directional rings
2. GPU search is sorted by PCI device ID to get consistent results
2020-07-28 14:21:44 -07:00
David Addison 033d799524 2.7.8-1
Fix collective mismatch error when using ncclSend/ncclRecv
2020-07-27 16:34:09 -07:00
Wenkai Du e7a10aa0e4 Topology tuning for 4P2H on Rome (#242)
* Topology tuning for 4P2H on Rome

* Use ncclTopoIdToIndex
2020-07-27 11:53:57 -07:00
Wenkai Du 8d5fb920b6 ib-test: support multiple channels (#241) 2020-07-27 11:03:12 -07:00
Wenkai Du d5f90e19b5 Add 8P6L multi-node models (#239) 2020-07-21 14:10:36 -07:00
Wenkai Du ab787c767e Change default channels duplication for chordal ring (#233) 2020-07-14 15:16:50 -07:00
Wenkai Du 5215130168 Revert "Split primitive class to smaller structures" (#230)
This reverts commit 486fd436af.
2020-07-08 11:06:50 -07:00
Riatre Foo 2d8601701d Fix build action order
Add $(INCTARGETS) to build dependencies of %.o and $(DEVICELIB).
As there were no dep files during the first build, Make may kick off source
compilation before nccl.h got generated, which leads to occasional build
failures on systems with high core count. The build failure could be
reproduced reliably with a `sleep 5` in $(INCDIR)/nccl.h rule.
2020-07-07 10:20:51 -07:00
Wenkai Du da3b197d6c Merge remote-tracking branch 'nccl/master' into develop 2020-07-01 16:51:25 -07:00
Wenkai Du 964c4c2061 Merge sendrecv kernel from NCCL 2.7.3
This commit was cherry-picked and modified from
https://github.com/NVIDIA/nccl/commit/5949d96f36d050e59d05872f8bbffd2549318e95
2020-06-29 08:47:46 -07:00
Wenkai Du b90735c935 Use separate threads for send and receive 2020-06-29 08:47:15 -07:00
Sylvain Jeaugey 1952325569 2.7.6-1
Fix crash when NVswitch is not visible inside a VM.
2020-06-26 16:35:54 -07:00
Sylvain Jeaugey 01afd20a77 2.7.5-1
Minor fixes for A100 platforms.
Add a WARN for invalid GroupEnd call.
2020-06-26 14:39:49 -07:00
Wenkai Du 84f8ba3bb0 Revert use posix_memalign for network buffer allocation on host memory (#222) 2020-06-24 11:25:55 -07:00
Wenkai Du 0eb19a563a Use posix_memalign for network buffer allocation on host memory (#221)
* Use posix_memalign for network buffer allocation on host memory

* ib-test: add ability to specify run iterations

* ib-test: define iterations as multiple of default cycles

* Add checking to posix_memalign return value
2020-06-22 13:06:25 -07:00
Stanley Tsang 8d21adb5e3 Documentation updates for NCCL 2.7.0 (#219)
* Making hip-clang the default compiler; documentation update

* Adding back --hip-clang to install.sh as a silent option for CI

* Documentation updates for NCCL 2.7

* Restoring deleted line in install script
2020-06-16 16:48:11 -06:00
Wenkai Du cfa97eccd3 Add IB/RDMA unit test 2020-06-16 18:29:17 +00:00
Wenkai Du 95b8f70d15 Limit network profiling support to simple protocol and avoid overflow 2020-06-15 20:51:36 +00:00
Wenkai Du 7484e53ff7 Rework network proxy profiling 2020-06-13 03:13:58 +00:00
Wenkai Du b257676f30 Reduce RCCL kernel count as we don't pass first coll in argument 2020-06-12 21:30:04 +00:00
Wenkai Du a6d621176c Sender rank's opCount maybe ahead by one if it finishes earlier 2020-06-12 03:39:45 +00:00
Wenkai Du e98891d039 Log NUMA node of RDMA host buffer allocation 2020-06-09 17:44:15 -07:00
Wenkai Du 812543104d Add network proxy profiling support 2020-06-09 17:44:15 -07:00
Wenkai Du c9aa11928a Calculate and use total wait cycles for RCCL profiling 2020-06-09 17:44:15 -07:00
Wenkai Du e80e29573c Add gather, scatter and alltoall collectives
Introducing 3 new APIs:
ncclResult_t  ncclGather(const void* sendbuff, void* recvbuff, size_t sendcount,
    ncclDataType_t datatype, int root, ncclComm_t comm, hipStream_t stream);
ncclResult_t  ncclScatter(const void* sendbuff, void* recvbuff,
    size_t recvcount, ncclDataType_t datatype, int root, ncclComm_t comm,
    hipStream_t stream);
ncclResult_t  ncclAllToAll(const void* sendbuff, void* recvbuff, size_t count,
    ncclDataType_t datatype, ncclComm_t comm, hipStream_t stream);

Only out of place operation is supported.
Preprocessor symbol RCCL_GATHER_SCATTER=1 indicates API availibility.
By default the APIs launche RCCL kernel implementation, which can be disabled by
RCCL_ALLTOALL_KERNEL_DISABLE=1. Then the APIs use wrapper around ncclSend and ncclRecv.
2020-06-09 17:44:08 -07:00
Wenkai Du 26a0fd2517 Merge remote-tracking branch 'nccl/master' into develop 2020-06-09 17:40:11 -07:00
Sylvain Jeaugey 5949d96f36 2.7.3-1
Add support for A100 GPU and related platforms.
Add support for CUDA 11.
Add support for send/receive operations (beta).
2020-06-08 09:31:44 -07:00
Wenkai Du 67c8e72ce3 Use cached value for detecting GDR support only once 2020-05-22 17:19:10 +00:00
Wenkai Du e41ab173cf Report HIP version in logs 2020-05-20 18:15:32 +00:00
Wenkai Du ca493a6b51 Revert "Tuning the inline and unroll to reduce the scratch usage"
This reverts commit eec319038e.
2020-05-15 14:15:40 -07:00
Wenkai Du c245f1507e Merge pull request #209 from wenkaidu/hip-clang
Rename files which only diffs in extension
2020-05-15 13:51:12 -07:00
Wenkai Du 706de76046 Merge pull request #208 from wenkaidu/perf_xgmi
Give preference to path with more XGMI connections
2020-05-15 10:07:22 -07:00
Wenkai Du e7b36304c8 Rename files which only diffs in extension 2020-05-15 09:16:32 -07:00
Wenkai Du b3c9852634 Give preference to path with more XGMI connections 2020-05-14 15:33:16 -07:00
Stanley Tsang 787ac13486 Restoring doxygen documentation to nccl.h.in. 2020-05-12 22:03:31 +00:00
Wenkai Du d5a07a7b5c Update rccl_bfloat16.h to match rocBLAS 2020-05-08 22:48:07 +00:00
Wenkai Du 24ea2ef6dd Set flags when calling hipExtLaunchMultiKernelMultiDevice in hip-clang 2020-05-08 15:57:14 +00:00
Wenkai Du c4edc257b0 Merge remote-tracking branch 'nccl/master' into HEAD 2020-04-27 17:16:54 +00:00
Wenkai Du edb49ed2d5 Fix incorrect next device ID in PCI ordered search 2020-04-25 01:01:13 +00:00
Sylvain Jeaugey f36540f55a Fix crash when only a subset of GPUs are visible within a container.
Fixes #326.
2020-04-17 10:03:14 -07:00
Sylvain Jeaugey 23a9fbb788 Improve robustness of PCI detection
Fallback to default values when class/speed is unknown.
2020-04-16 14:27:50 -07:00
aokomoriuta a783484ab5 Fix wrong variable name "slice" to "chunk"
https://github.com/NVIDIA/nccl/issues/287
2020-04-14 19:00:51 -07:00