da3b197d6c
Merge remote-tracking branch 'nccl/master' into develop
2020-07-01 16:51:25 -07:00
964c4c2061
Merge sendrecv kernel from NCCL 2.7.3
...
This commit was cherry-picked and modified from
https://github.com/NVIDIA/nccl/commit/5949d96f36d050e59d05872f8bbffd2549318e95
2020-06-29 08:47:46 -07:00
b90735c935
Use separate threads for send and receive
2020-06-29 08:47:15 -07:00
1952325569
2.7.6-1
...
Fix crash when NVswitch is not visible inside a VM.
2020-06-26 16:35:54 -07:00
01afd20a77
2.7.5-1
...
Minor fixes for A100 platforms.
Add a WARN for invalid GroupEnd call.
2020-06-26 14:39:49 -07:00
84f8ba3bb0
Revert use posix_memalign for network buffer allocation on host memory ( #222 )
2020-06-24 11:25:55 -07:00
0eb19a563a
Use posix_memalign for network buffer allocation on host memory ( #221 )
...
* Use posix_memalign for network buffer allocation on host memory
* ib-test: add ability to specify run iterations
* ib-test: define iterations as multiple of default cycles
* Add checking to posix_memalign return value
2020-06-22 13:06:25 -07:00
8d21adb5e3
Documentation updates for NCCL 2.7.0 ( #219 )
...
* Making hip-clang the default compiler; documentation update
* Adding back --hip-clang to install.sh as a silent option for CI
* Documentation updates for NCCL 2.7
* Restoring deleted line in install script
2020-06-16 16:48:11 -06:00
cfa97eccd3
Add IB/RDMA unit test
2020-06-16 18:29:17 +00:00
95b8f70d15
Limit network profiling support to simple protocol and avoid overflow
2020-06-15 20:51:36 +00:00
7484e53ff7
Rework network proxy profiling
2020-06-13 03:13:58 +00:00
b257676f30
Reduce RCCL kernel count as we don't pass first coll in argument
2020-06-12 21:30:04 +00:00
a6d621176c
Sender rank's opCount maybe ahead by one if it finishes earlier
2020-06-12 03:39:45 +00:00
e98891d039
Log NUMA node of RDMA host buffer allocation
2020-06-09 17:44:15 -07:00
812543104d
Add network proxy profiling support
2020-06-09 17:44:15 -07:00
c9aa11928a
Calculate and use total wait cycles for RCCL profiling
2020-06-09 17:44:15 -07:00
e80e29573c
Add gather, scatter and alltoall collectives
...
Introducing 3 new APIs:
ncclResult_t ncclGather(const void* sendbuff, void* recvbuff, size_t sendcount,
ncclDataType_t datatype, int root, ncclComm_t comm, hipStream_t stream);
ncclResult_t ncclScatter(const void* sendbuff, void* recvbuff,
size_t recvcount, ncclDataType_t datatype, int root, ncclComm_t comm,
hipStream_t stream);
ncclResult_t ncclAllToAll(const void* sendbuff, void* recvbuff, size_t count,
ncclDataType_t datatype, ncclComm_t comm, hipStream_t stream);
Only out of place operation is supported.
Preprocessor symbol RCCL_GATHER_SCATTER=1 indicates API availibility.
By default the APIs launche RCCL kernel implementation, which can be disabled by
RCCL_ALLTOALL_KERNEL_DISABLE=1. Then the APIs use wrapper around ncclSend and ncclRecv.
2020-06-09 17:44:08 -07:00
26a0fd2517
Merge remote-tracking branch 'nccl/master' into develop
2020-06-09 17:40:11 -07:00
5949d96f36
2.7.3-1
...
Add support for A100 GPU and related platforms.
Add support for CUDA 11.
Add support for send/receive operations (beta).
2020-06-08 09:31:44 -07:00
67c8e72ce3
Use cached value for detecting GDR support only once
2020-05-22 17:19:10 +00:00
e41ab173cf
Report HIP version in logs
2020-05-20 18:15:32 +00:00
ca493a6b51
Revert "Tuning the inline and unroll to reduce the scratch usage"
...
This reverts commit eec319038e
2020-05-15 14:15:40 -07:00
c245f1507e
Merge pull request #209 from wenkaidu/hip-clang
...
Rename files which only diffs in extension
2020-05-15 13:51:12 -07:00
706de76046
Merge pull request #208 from wenkaidu/perf_xgmi
...
Give preference to path with more XGMI connections
2020-05-15 10:07:22 -07:00
e7b36304c8
Rename files which only diffs in extension
2020-05-15 09:16:32 -07:00
b3c9852634
Give preference to path with more XGMI connections
2020-05-14 15:33:16 -07:00
787ac13486
Restoring doxygen documentation to nccl.h.in.
2020-05-12 22:03:31 +00:00
d5a07a7b5c
Update rccl_bfloat16.h to match rocBLAS
2020-05-08 22:48:07 +00:00
24ea2ef6dd
Set flags when calling hipExtLaunchMultiKernelMultiDevice in hip-clang
2020-05-08 15:57:14 +00:00
c4edc257b0
Merge remote-tracking branch 'nccl/master' into HEAD
2020-04-27 17:16:54 +00:00
edb49ed2d5
Fix incorrect next device ID in PCI ordered search
2020-04-25 01:01:13 +00:00
f36540f55a
Fix crash when only a subset of GPUs are visible within a container.
...
Fixes #326 .
2020-04-17 10:03:14 -07:00
23a9fbb788
Improve robustness of PCI detection
...
Fallback to default values when class/speed is unknown.
2020-04-16 14:27:50 -07:00
a783484ab5
Fix wrong variable name "slice" to "chunk"
...
https://github.com/NVIDIA/nccl/issues/287
2020-04-14 19:00:51 -07:00
b5b6c6acdd
Fix bug #307 : wrong NIC selection on the reduction tree.
...
The reduction tree (tree up) was inverting the NICs to use,
causing performance issue in cases where we are using different
NICs on a given channel.
2020-04-09 17:14:07 -07:00
a95090d981
Fix HIP-Clang build with HSA headers
...
HIP-Clang does not include these HSA headers, and they need to be explicitly added in RCCL.
2020-04-03 17:58:23 -04:00
6f54b23503
topo_expl: update to 2.6
2020-04-01 13:37:08 -07:00
fa36fd9ef9
Merge remote-tracking branch 'nccl/master' into v2.6.4_merge
2020-04-01 13:35:12 -07:00
533e3702cf
Merge pull request #314 from NVIDIA/v2.6
...
2.6.4-1
2020-03-26 17:31:24 -07:00
b221128eca
2.6.4-1
...
Add support for network collectives.
Add support for XML topology dump/injection.
Add text values for GDR and P2P Levels, including "NVL".
Add speed detection for PCI, Infiniband and Ethernet cards.
Add CPU detection for ARM and AMD CPUs.
Add support for adaptive routing on Infiniband.
Change NET plugin API to v3 : merge PCI path and GPU pointer
capability into a single structure and add other properties.
2020-03-20 14:58:36 -07:00
6c61492eba
Check return code for Flush operation
...
Current NCCL code does not abort for failed Flush operations by
underlying network. This may compromise data integrity.
Signed-off-by: Rashika Kheria <rashika@amazon.com >
2020-03-16 20:40:59 -07:00
0976e47b06
Merge pull request #183 from wenkaidu/dup_rings
...
Remove condition for ring duplication
2020-03-02 17:12:42 -08:00
62dc28bd2e
Remove condition for ring duplication
...
Fix insufficent number of rings on single node after pull #179
2020-03-02 12:55:06 -08:00
fb59328a7b
Check fine grained memory before enabling RDMA
...
Adding back the check which was lost from 2.5 merge.
2020-03-02 11:18:27 -08:00
8b5bc8bca2
Merge pull request #179 from wenkaidu/search
...
Use fraction of system maxWidth as steps for searching
2020-02-28 11:05:46 -08:00
8e73a2ad60
Merge remote-tracking branch 'remotes/nccl/master'
2020-02-27 12:53:03 -08:00
d2adc61bf6
Revise PCI BW numbers on Rome
2020-02-26 13:17:49 -08:00
8391637613
Use fraction of system maxWidth as steps for searching
...
This reverts previous workaround of deducting only half of width
from paths.
2020-02-26 09:14:35 -08:00
077c3cda74
Fix abort handling in LL primitives
2020-02-25 13:42:54 -08:00
9b80b3633f
Fix system maxSpeed and maxWidth calculation
2020-02-24 15:18:57 -08:00
Wenkai Du
Sylvain Jeaugey
Wenkai Du
Stanley Tsang
aokomoriuta
Aaron Enye Shi
Rashika Kheria