26a0fd2517
Merge remote-tracking branch 'nccl/master' into develop
2020-06-09 17:40:11 -07:00
5949d96f36
2.7.3-1
...
Add support for A100 GPU and related platforms.
Add support for CUDA 11.
Add support for send/receive operations (beta).
2020-06-08 09:31:44 -07:00
67c8e72ce3
Use cached value for detecting GDR support only once
2020-05-22 17:19:10 +00:00
e41ab173cf
Report HIP version in logs
2020-05-20 18:15:32 +00:00
ca493a6b51
Revert "Tuning the inline and unroll to reduce the scratch usage"
...
This reverts commit eec319038e
2020-05-15 14:15:40 -07:00
c245f1507e
Merge pull request #209 from wenkaidu/hip-clang
...
Rename files which only diffs in extension
2020-05-15 13:51:12 -07:00
706de76046
Merge pull request #208 from wenkaidu/perf_xgmi
...
Give preference to path with more XGMI connections
2020-05-15 10:07:22 -07:00
e7b36304c8
Rename files which only diffs in extension
2020-05-15 09:16:32 -07:00
b3c9852634
Give preference to path with more XGMI connections
2020-05-14 15:33:16 -07:00
787ac13486
Restoring doxygen documentation to nccl.h.in.
2020-05-12 22:03:31 +00:00
d5a07a7b5c
Update rccl_bfloat16.h to match rocBLAS
2020-05-08 22:48:07 +00:00
24ea2ef6dd
Set flags when calling hipExtLaunchMultiKernelMultiDevice in hip-clang
2020-05-08 15:57:14 +00:00
c4edc257b0
Merge remote-tracking branch 'nccl/master' into HEAD
2020-04-27 17:16:54 +00:00
edb49ed2d5
Fix incorrect next device ID in PCI ordered search
2020-04-25 01:01:13 +00:00
f36540f55a
Fix crash when only a subset of GPUs are visible within a container.
...
Fixes #326 .
2020-04-17 10:03:14 -07:00
23a9fbb788
Improve robustness of PCI detection
...
Fallback to default values when class/speed is unknown.
2020-04-16 14:27:50 -07:00
a783484ab5
Fix wrong variable name "slice" to "chunk"
...
https://github.com/NVIDIA/nccl/issues/287
2020-04-14 19:00:51 -07:00
b5b6c6acdd
Fix bug #307 : wrong NIC selection on the reduction tree.
...
The reduction tree (tree up) was inverting the NICs to use,
causing performance issue in cases where we are using different
NICs on a given channel.
2020-04-09 17:14:07 -07:00
a95090d981
Fix HIP-Clang build with HSA headers
...
HIP-Clang does not include these HSA headers, and they need to be explicitly added in RCCL.
2020-04-03 17:58:23 -04:00
6f54b23503
topo_expl: update to 2.6
2020-04-01 13:37:08 -07:00
fa36fd9ef9
Merge remote-tracking branch 'nccl/master' into v2.6.4_merge
2020-04-01 13:35:12 -07:00
533e3702cf
Merge pull request #314 from NVIDIA/v2.6
...
2.6.4-1
2020-03-26 17:31:24 -07:00
b221128eca
2.6.4-1
...
Add support for network collectives.
Add support for XML topology dump/injection.
Add text values for GDR and P2P Levels, including "NVL".
Add speed detection for PCI, Infiniband and Ethernet cards.
Add CPU detection for ARM and AMD CPUs.
Add support for adaptive routing on Infiniband.
Change NET plugin API to v3 : merge PCI path and GPU pointer
capability into a single structure and add other properties.
2020-03-20 14:58:36 -07:00
6c61492eba
Check return code for Flush operation
...
Current NCCL code does not abort for failed Flush operations by
underlying network. This may compromise data integrity.
Signed-off-by: Rashika Kheria <rashika@amazon.com >
2020-03-16 20:40:59 -07:00
0976e47b06
Merge pull request #183 from wenkaidu/dup_rings
...
Remove condition for ring duplication
2020-03-02 17:12:42 -08:00
62dc28bd2e
Remove condition for ring duplication
...
Fix insufficent number of rings on single node after pull #179
2020-03-02 12:55:06 -08:00
fb59328a7b
Check fine grained memory before enabling RDMA
...
Adding back the check which was lost from 2.5 merge.
2020-03-02 11:18:27 -08:00
8b5bc8bca2
Merge pull request #179 from wenkaidu/search
...
Use fraction of system maxWidth as steps for searching
2020-02-28 11:05:46 -08:00
8e73a2ad60
Merge remote-tracking branch 'remotes/nccl/master'
2020-02-27 12:53:03 -08:00
d2adc61bf6
Revise PCI BW numbers on Rome
2020-02-26 13:17:49 -08:00
8391637613
Use fraction of system maxWidth as steps for searching
...
This reverts previous workaround of deducting only half of width
from paths.
2020-02-26 09:14:35 -08:00
077c3cda74
Fix abort handling in LL primitives
2020-02-25 13:42:54 -08:00
9b80b3633f
Fix system maxSpeed and maxWidth calculation
2020-02-24 15:18:57 -08:00
f54dc58113
Fix incorrect CR8 detection
...
Also change level of ring graph print to help debugging
2020-02-21 10:09:49 -08:00
5b3856f2ed
Merge pull request #172 from wenkaidu/topo_expl
...
Add topology explorer
2020-02-20 15:16:55 -08:00
55f8e2dec7
Add topology explorer
2020-02-19 14:42:06 -08:00
c38f174bd4
Fix Allgather operations above 4G with multiple GPUs per process.
...
Fixes nccl-tests#37.
Direct offsets were still on 32 bits in the low-level primitives.
2020-02-12 11:11:55 -08:00
abcfbf1231
Generate 8G6L chordal ring from reference
2020-02-11 22:01:12 +00:00
d1dae2721d
Add ring bandwidth correction factor
2020-01-30 09:52:27 -08:00
20fa04d9b6
Updating copyright notices for 2020.
2020-01-29 15:28:08 -08:00
486fd436af
Split primitive class to smaller structures
2020-01-29 15:27:23 -08:00
1e55645d97
Misc fixes and improvements for 2.5.6
...
1. Fix RCCL unit test
2. Add ROME detection and tuning
3. Change default P2P level
4. Fix search algorithm for XGMI
5. Remove explicit channel duplication with implicit by using half of link speed
6. Add collective trace support
7. Correct Intel Skylake CPU detection and bandwidth
8. Fix topo connect function
9. Disable GDR read and remove unreachable code
10. Disable LL128 kernels
11. Add tuning parameters
12. Use original clock64() implementation which returns RTC counter value
13. Print out timestamp of collective trace
14. Do not use struct ncclColl in kernel launch parameter
15. Fix abort handling and add tracing
17. Add __launch_bounds__ to kernel functions
18. Remove unused abortCount
19. Unset default MIN_NRINGS and MIN_NCHANNELS
20. Do not allocate shared memory when not using LL128 kernels
21. Correct time print out in tuning log
2020-01-29 15:27:05 -08:00
7a18fe0784
[topology] remove NET links when trimming system
...
This fixes a memory leak.
2020-01-07 13:29:57 -08:00
3899f6e0f2
Fix clang build ( #274 )
...
The attribute is called `optnone`, not `noopt`.
2019-12-09 09:31:13 -08:00
aa15dfb29c
Fix clang compilation
2019-12-06 09:55:54 -08:00
8c564e9b57
Fix clang build ( #271 )
...
Clang doesn't understand `optimize("O0")`. It has `noopt`, which GCC doesn't understand. Wrap the difference in a macro.
2019-12-06 09:14:55 -08:00
6648c81dc6
Merge remote-tracking branch 'remotes/nccl/master' into rccl_2.5.6
2019-12-03 15:42:04 -08:00
a0be2b8812
Disable direct buffers to reduce scratch memory size
2019-11-20 13:03:16 -08:00
299c554dcc
2.5.6-1 ( #255 )
...
Add LL128 Protocol.
Rewrite the topology detection and tree/ring creation (#179 ). Improve
tree performance by sending/receiving from different GPUs. Add
model-based tuning to switch between the different algorithms and
protocols.
Rework P2P/SHM detection in containers (#155 , #248 ).
Detect duplicated devices and return an error (#231 ).
Add tuning for GCP
2019-11-19 14:57:39 -08:00
5e109ed400
Add bfloat16 support in RCCL
...
Preprocessor symbol RCCL_BFLOAT16 is used as feature indicator
2019-11-18 13:45:53 -08:00
Wenkai Du
Sylvain Jeaugey
Wenkai Du
Stanley Tsang
aokomoriuta
Aaron Enye Shi
Rashika Kheria
Luke Yeager
Christian Sigg