5ba51914f1
Update rccl_bfloat16.h to match rocBLAS
...
[ROCm/rccl commit: d5a07a7b5c
2020-05-08 22:48:07 +00:00
069322d05a
Set flags when calling hipExtLaunchMultiKernelMultiDevice in hip-clang
...
[ROCm/rccl commit: 24ea2ef6dd
2020-05-08 15:57:14 +00:00
9813d67cd1
Merge remote-tracking branch 'nccl/master' into HEAD
...
[ROCm/rccl commit: c4edc257b0
2020-04-27 17:16:54 +00:00
7b7f781658
Fix incorrect next device ID in PCI ordered search
...
[ROCm/rccl commit: edb49ed2d5
2020-04-25 01:01:13 +00:00
c43022b9d8
Fix crash when only a subset of GPUs are visible within a container.
...
Fixes #326 .
[ROCm/rccl commit: f36540f55a
2020-04-17 10:03:14 -07:00
5df2502deb
Improve robustness of PCI detection
...
Fallback to default values when class/speed is unknown.
[ROCm/rccl commit: 23a9fbb788
2020-04-16 14:27:50 -07:00
acd868dfcb
Fix wrong variable name "slice" to "chunk"
...
https://github.com/NVIDIA/nccl/issues/287
[ROCm/rccl commit: a783484ab5
2020-04-14 19:00:51 -07:00
627e1a06d0
Fix bug #307 : wrong NIC selection on the reduction tree.
...
The reduction tree (tree up) was inverting the NICs to use,
causing performance issue in cases where we are using different
NICs on a given channel.
[ROCm/rccl commit: b5b6c6acdd
2020-04-09 17:14:07 -07:00
bfbfe370c3
Fix HIP-Clang build with HSA headers
...
HIP-Clang does not include these HSA headers, and they need to be explicitly added in RCCL.
[ROCm/rccl commit: a95090d981
2020-04-03 17:58:23 -04:00
8852e54181
topo_expl: update to 2.6
...
[ROCm/rccl commit: 6f54b23503
2020-04-01 13:37:08 -07:00
4aeb7f041e
Merge remote-tracking branch 'nccl/master' into v2.6.4_merge
...
[ROCm/rccl commit: fa36fd9ef9
2020-04-01 13:35:12 -07:00
b996c2ca00
Merge pull request #314 from NVIDIA/v2.6
...
2.6.4-1
[ROCm/rccl commit: 533e3702cf
2020-03-26 17:31:24 -07:00
40adc74496
2.6.4-1
...
Add support for network collectives.
Add support for XML topology dump/injection.
Add text values for GDR and P2P Levels, including "NVL".
Add speed detection for PCI, Infiniband and Ethernet cards.
Add CPU detection for ARM and AMD CPUs.
Add support for adaptive routing on Infiniband.
Change NET plugin API to v3 : merge PCI path and GPU pointer
capability into a single structure and add other properties.
[ROCm/rccl commit: b221128eca
2020-03-20 14:58:36 -07:00
38b445c94f
Check return code for Flush operation
...
Current NCCL code does not abort for failed Flush operations by
underlying network. This may compromise data integrity.
Signed-off-by: Rashika Kheria <rashika@amazon.com >
[ROCm/rccl commit: 6c61492eba
2020-03-16 20:40:59 -07:00
cb19bce4e0
Merge pull request #183 from wenkaidu/dup_rings
...
Remove condition for ring duplication
[ROCm/rccl commit: 0976e47b06
2020-03-02 17:12:42 -08:00
85fd51a06f
Remove condition for ring duplication
...
Fix insufficent number of rings on single node after pull #179
[ROCm/rccl commit: 62dc28bd2e
2020-03-02 12:55:06 -08:00
593d99d9a9
Check fine grained memory before enabling RDMA
...
Adding back the check which was lost from 2.5 merge.
[ROCm/rccl commit: fb59328a7b
2020-03-02 11:18:27 -08:00
2a66deb694
Merge pull request #179 from wenkaidu/search
...
Use fraction of system maxWidth as steps for searching
[ROCm/rccl commit: 8b5bc8bca2
2020-02-28 11:05:46 -08:00
b750defc28
Merge remote-tracking branch 'remotes/nccl/master'
...
[ROCm/rccl commit: 8e73a2ad60
2020-02-27 12:53:03 -08:00
45a7541582
Revise PCI BW numbers on Rome
...
[ROCm/rccl commit: d2adc61bf6
2020-02-26 13:17:49 -08:00
b4be0ff3b8
Use fraction of system maxWidth as steps for searching
...
This reverts previous workaround of deducting only half of width
from paths.
[ROCm/rccl commit: 8391637613
2020-02-26 09:14:35 -08:00
5747c3cac1
Fix abort handling in LL primitives
...
[ROCm/rccl commit: 077c3cda74
2020-02-25 13:42:54 -08:00
d640f38d56
Fix system maxSpeed and maxWidth calculation
...
[ROCm/rccl commit: 9b80b3633f
2020-02-24 15:18:57 -08:00
93d448e2fe
Fix incorrect CR8 detection
...
Also change level of ring graph print to help debugging
[ROCm/rccl commit: f54dc58113
2020-02-21 10:09:49 -08:00
cf4bce4ad3
Merge pull request #172 from wenkaidu/topo_expl
...
Add topology explorer
[ROCm/rccl commit: 5b3856f2ed
2020-02-20 15:16:55 -08:00
00f421ccbd
Add topology explorer
...
[ROCm/rccl commit: 55f8e2dec7
2020-02-19 14:42:06 -08:00
6034c27655
Fix Allgather operations above 4G with multiple GPUs per process.
...
Fixes nccl-tests#37.
Direct offsets were still on 32 bits in the low-level primitives.
[ROCm/rccl commit: c38f174bd4
2020-02-12 11:11:55 -08:00
8432e8a921
Generate 8G6L chordal ring from reference
...
[ROCm/rccl commit: abcfbf1231
2020-02-11 22:01:12 +00:00
6b2d7de200
Add ring bandwidth correction factor
...
[ROCm/rccl commit: d1dae2721d
2020-01-30 09:52:27 -08:00
e5419407c4
Updating copyright notices for 2020.
...
[ROCm/rccl commit: 20fa04d9b6
2020-01-29 15:28:08 -08:00
622b49e80a
Split primitive class to smaller structures
...
[ROCm/rccl commit: 486fd436af
2020-01-29 15:27:23 -08:00
d2fbcfea02
Misc fixes and improvements for 2.5.6
...
1. Fix RCCL unit test
2. Add ROME detection and tuning
3. Change default P2P level
4. Fix search algorithm for XGMI
5. Remove explicit channel duplication with implicit by using half of link speed
6. Add collective trace support
7. Correct Intel Skylake CPU detection and bandwidth
8. Fix topo connect function
9. Disable GDR read and remove unreachable code
10. Disable LL128 kernels
11. Add tuning parameters
12. Use original clock64() implementation which returns RTC counter value
13. Print out timestamp of collective trace
14. Do not use struct ncclColl in kernel launch parameter
15. Fix abort handling and add tracing
17. Add __launch_bounds__ to kernel functions
18. Remove unused abortCount
19. Unset default MIN_NRINGS and MIN_NCHANNELS
20. Do not allocate shared memory when not using LL128 kernels
21. Correct time print out in tuning log
[ROCm/rccl commit: 1e55645d97
2020-01-29 15:27:05 -08:00
d91217b16f
[topology] remove NET links when trimming system
...
This fixes a memory leak.
[ROCm/rccl commit: 7a18fe0784
2020-01-07 13:29:57 -08:00
ff74ebdcea
Fix clang build ( #274 )
...
The attribute is called `optnone`, not `noopt`.
[ROCm/rccl commit: 3899f6e0f2
2019-12-09 09:31:13 -08:00
e5a17ee58d
Fix clang compilation
...
[ROCm/rccl commit: aa15dfb29c
2019-12-06 09:55:54 -08:00
4984d5ce0b
Fix clang build ( #271 )
...
Clang doesn't understand `optimize("O0")`. It has `noopt`, which GCC doesn't understand. Wrap the difference in a macro.
[ROCm/rccl commit: 8c564e9b57
2019-12-06 09:14:55 -08:00
d7d4175df0
Merge remote-tracking branch 'remotes/nccl/master' into rccl_2.5.6
...
[ROCm/rccl commit: 6648c81dc6
2019-12-03 15:42:04 -08:00
4cb52294d5
Disable direct buffers to reduce scratch memory size
...
[ROCm/rccl commit: a0be2b8812
2019-11-20 13:03:16 -08:00
71560fd67b
2.5.6-1 ( #255 )
...
Add LL128 Protocol.
Rewrite the topology detection and tree/ring creation (#179 ). Improve
tree performance by sending/receiving from different GPUs. Add
model-based tuning to switch between the different algorithms and
protocols.
Rework P2P/SHM detection in containers (#155 , #248 ).
Detect duplicated devices and return an error (#231 ).
Add tuning for GCP
[ROCm/rccl commit: 299c554dcc
2019-11-19 14:57:39 -08:00
1e182391ad
Add bfloat16 support in RCCL
...
Preprocessor symbol RCCL_BFLOAT16 is used as feature indicator
[ROCm/rccl commit: 5e109ed400
2019-11-18 13:45:53 -08:00
62042e47bc
Correct RTC frequencies for profiling purpose
...
[ROCm/rccl commit: 8995047830
2019-11-05 11:36:45 -08:00
41f6319b33
Check for fine grain support using memory allocation
...
[ROCm/rccl commit: 669f1951a4
2019-11-01 15:58:49 -07:00
e43e1f1b3d
additional check for fine grain support in p2pCanConnect ( #146 )
...
[ROCm/rccl commit: 5a502955c9
2019-10-31 08:58:38 -07:00
d3f399f619
Disable HDP flush for RDMA
...
[ROCm/rccl commit: 296176a4fd
2019-10-23 14:40:17 -07:00
21bc1ef493
Revert collective chunk and slice steps to avoid drop in throughput
...
[ROCm/rccl commit: df74d12946
2019-10-18 12:54:00 -07:00
7560929bd7
Reverting GenericOp bug workaround modifications to slice/chunk steps
...
[ROCm/rccl commit: 37603ae6cb
2019-10-11 09:20:10 -07:00
cf597ff257
Performing __threadfence_system() with only first thread
...
[ROCm/rccl commit: 1392dd2997
2019-10-11 09:16:19 -07:00
d257970ad1
Fix for GenericOp device primitive bug
...
[ROCm/rccl commit: 8ae1bce3bb
2019-10-10 22:39:45 -07:00
fbcdfd8348
Merge pull request #136 from wenkaidu/tree
...
Enable tree kernels in build
[ROCm/rccl commit: 062c798c86
2019-10-09 10:58:52 -07:00
f86ee41415
Enable tree kernels in build
...
Need to tune and specify NCCL_TREE_THRESHOLD to allow usage
[ROCm/rccl commit: 76976c9e2e
2019-10-08 23:20:11 +00:00
Wenkai Du
Sylvain Jeaugey
aokomoriuta
Aaron Enye Shi
Rashika Kheria
Wenkai Du
Stanley Tsang
Luke Yeager
Christian Sigg
Jeff Daily
Gilbert Lee