7e3f841fab
Merge remote-tracking branch 'nccl/master' into 2.7.8
2020-08-10 16:11:00 +00:00
09ef75656a
Add more Rome 4P2H models
2020-08-06 18:20:02 +00:00
de49a77074
Setting type when gpu sub node is discovered
2020-08-05 13:39:23 -07:00
700c0e0f24
Don't require NIC devices to have specific PCI class
...
If a PCI node is the parent of a NIC, treat it as such, regardless of
the PCI class code for the device. This allows non-traditional devices
to act as NICs via the net plugin mechanism.
For consistency, treat GPUs similarly.
2020-08-05 12:46:29 -07:00
5b03132ace
Allow setup ring through NCCL_RINGS to facilitate testing
2020-08-04 21:07:00 +00:00
d1e20b4c5e
Improve 4P2H topology on Rome ( #243 )
...
1. Use bi-directional rings
2. GPU search is sorted by PCI device ID to get consistent results
2020-07-28 14:21:44 -07:00
033d799524
2.7.8-1
...
Fix collective mismatch error when using ncclSend/ncclRecv
2020-07-27 16:34:09 -07:00
e7a10aa0e4
Topology tuning for 4P2H on Rome ( #242 )
...
* Topology tuning for 4P2H on Rome
* Use ncclTopoIdToIndex
2020-07-27 11:53:57 -07:00
8d5fb920b6
ib-test: support multiple channels ( #241 )
2020-07-27 11:03:12 -07:00
d5f90e19b5
Add 8P6L multi-node models ( #239 )
2020-07-21 14:10:36 -07:00
ab787c767e
Change default channels duplication for chordal ring ( #233 )
2020-07-14 15:16:50 -07:00
5215130168
Revert "Split primitive class to smaller structures" ( #230 )
...
This reverts commit 486fd436af
2020-07-08 11:06:50 -07:00
2d8601701d
Fix build action order
...
Add $(INCTARGETS) to build dependencies of %.o and $(DEVICELIB).
As there were no dep files during the first build, Make may kick off source
compilation before nccl.h got generated, which leads to occasional build
failures on systems with high core count. The build failure could be
reproduced reliably with a `sleep 5` in $(INCDIR)/nccl.h rule.
2020-07-07 10:20:51 -07:00
da3b197d6c
Merge remote-tracking branch 'nccl/master' into develop
2020-07-01 16:51:25 -07:00
964c4c2061
Merge sendrecv kernel from NCCL 2.7.3
...
This commit was cherry-picked and modified from
https://github.com/NVIDIA/nccl/commit/5949d96f36d050e59d05872f8bbffd2549318e95
2020-06-29 08:47:46 -07:00
b90735c935
Use separate threads for send and receive
2020-06-29 08:47:15 -07:00
1952325569
2.7.6-1
...
Fix crash when NVswitch is not visible inside a VM.
2020-06-26 16:35:54 -07:00
01afd20a77
2.7.5-1
...
Minor fixes for A100 platforms.
Add a WARN for invalid GroupEnd call.
2020-06-26 14:39:49 -07:00
84f8ba3bb0
Revert use posix_memalign for network buffer allocation on host memory ( #222 )
2020-06-24 11:25:55 -07:00
0eb19a563a
Use posix_memalign for network buffer allocation on host memory ( #221 )
...
* Use posix_memalign for network buffer allocation on host memory
* ib-test: add ability to specify run iterations
* ib-test: define iterations as multiple of default cycles
* Add checking to posix_memalign return value
2020-06-22 13:06:25 -07:00
8d21adb5e3
Documentation updates for NCCL 2.7.0 ( #219 )
...
* Making hip-clang the default compiler; documentation update
* Adding back --hip-clang to install.sh as a silent option for CI
* Documentation updates for NCCL 2.7
* Restoring deleted line in install script
2020-06-16 16:48:11 -06:00
cfa97eccd3
Add IB/RDMA unit test
2020-06-16 18:29:17 +00:00
95b8f70d15
Limit network profiling support to simple protocol and avoid overflow
2020-06-15 20:51:36 +00:00
7484e53ff7
Rework network proxy profiling
2020-06-13 03:13:58 +00:00
b257676f30
Reduce RCCL kernel count as we don't pass first coll in argument
2020-06-12 21:30:04 +00:00
a6d621176c
Sender rank's opCount maybe ahead by one if it finishes earlier
2020-06-12 03:39:45 +00:00
e98891d039
Log NUMA node of RDMA host buffer allocation
2020-06-09 17:44:15 -07:00
812543104d
Add network proxy profiling support
2020-06-09 17:44:15 -07:00
c9aa11928a
Calculate and use total wait cycles for RCCL profiling
2020-06-09 17:44:15 -07:00
e80e29573c
Add gather, scatter and alltoall collectives
...
Introducing 3 new APIs:
ncclResult_t ncclGather(const void* sendbuff, void* recvbuff, size_t sendcount,
ncclDataType_t datatype, int root, ncclComm_t comm, hipStream_t stream);
ncclResult_t ncclScatter(const void* sendbuff, void* recvbuff,
size_t recvcount, ncclDataType_t datatype, int root, ncclComm_t comm,
hipStream_t stream);
ncclResult_t ncclAllToAll(const void* sendbuff, void* recvbuff, size_t count,
ncclDataType_t datatype, ncclComm_t comm, hipStream_t stream);
Only out of place operation is supported.
Preprocessor symbol RCCL_GATHER_SCATTER=1 indicates API availibility.
By default the APIs launche RCCL kernel implementation, which can be disabled by
RCCL_ALLTOALL_KERNEL_DISABLE=1. Then the APIs use wrapper around ncclSend and ncclRecv.
2020-06-09 17:44:08 -07:00
26a0fd2517
Merge remote-tracking branch 'nccl/master' into develop
2020-06-09 17:40:11 -07:00
5949d96f36
2.7.3-1
...
Add support for A100 GPU and related platforms.
Add support for CUDA 11.
Add support for send/receive operations (beta).
2020-06-08 09:31:44 -07:00
67c8e72ce3
Use cached value for detecting GDR support only once
2020-05-22 17:19:10 +00:00
e41ab173cf
Report HIP version in logs
2020-05-20 18:15:32 +00:00
ca493a6b51
Revert "Tuning the inline and unroll to reduce the scratch usage"
...
This reverts commit eec319038e
2020-05-15 14:15:40 -07:00
c245f1507e
Merge pull request #209 from wenkaidu/hip-clang
...
Rename files which only diffs in extension
2020-05-15 13:51:12 -07:00
706de76046
Merge pull request #208 from wenkaidu/perf_xgmi
...
Give preference to path with more XGMI connections
2020-05-15 10:07:22 -07:00
e7b36304c8
Rename files which only diffs in extension
2020-05-15 09:16:32 -07:00
b3c9852634
Give preference to path with more XGMI connections
2020-05-14 15:33:16 -07:00
787ac13486
Restoring doxygen documentation to nccl.h.in.
2020-05-12 22:03:31 +00:00
d5a07a7b5c
Update rccl_bfloat16.h to match rocBLAS
2020-05-08 22:48:07 +00:00
24ea2ef6dd
Set flags when calling hipExtLaunchMultiKernelMultiDevice in hip-clang
2020-05-08 15:57:14 +00:00
c4edc257b0
Merge remote-tracking branch 'nccl/master' into HEAD
2020-04-27 17:16:54 +00:00
edb49ed2d5
Fix incorrect next device ID in PCI ordered search
2020-04-25 01:01:13 +00:00
f36540f55a
Fix crash when only a subset of GPUs are visible within a container.
...
Fixes #326 .
2020-04-17 10:03:14 -07:00
23a9fbb788
Improve robustness of PCI detection
...
Fallback to default values when class/speed is unknown.
2020-04-16 14:27:50 -07:00
a783484ab5
Fix wrong variable name "slice" to "chunk"
...
https://github.com/NVIDIA/nccl/issues/287
2020-04-14 19:00:51 -07:00
b5b6c6acdd
Fix bug #307 : wrong NIC selection on the reduction tree.
...
The reduction tree (tree up) was inverting the NICs to use,
causing performance issue in cases where we are using different
NICs on a given channel.
2020-04-09 17:14:07 -07:00
a95090d981
Fix HIP-Clang build with HSA headers
...
HIP-Clang does not include these HSA headers, and they need to be explicitly added in RCCL.
2020-04-03 17:58:23 -04:00
6f54b23503
topo_expl: update to 2.6
2020-04-01 13:37:08 -07:00
Wenkai Du
Jack Snyder
Eric Badger
Wenkai Du
David Addison
Riatre Foo
Sylvain Jeaugey
Stanley Tsang
aokomoriuta
Aaron Enye Shi