Commit Graph

29 Commits

Author SHA1 Message Date
Wenkai Du a4dd1a9548 Improve GPU direct RDMA handling on Rome (#294)
[ROCm/rccl commit: 709b7e4880]
2020-11-03 14:29:08 -08:00
Wenkai Du 8b120c0508 Update Rome single node models (#277)
[ROCm/rccl commit: 33babcb5e2]
2020-10-13 13:33:09 -07:00
Wenkai Du 41260bb948 Rework Rome detection and add multiple network ports models (#274)
* Rework Rome detection and add multiple network ports models

* Remove unused opCount in p2p transport

[ROCm/rccl commit: ae008fd2db]
2020-10-07 13:37:36 -07:00
gilbertlee-amd 0a9adc16f4 Changes to topology based on XGMI (#272)
* Alterations to topology search to improve XGMI-enabled nodes

[ROCm/rccl commit: 01bd2573db]
2020-09-25 12:20:09 -06:00
Wenkai Du f0a303664e Limit P2P channels on Rome
[ROCm/rccl commit: 42955f5f4f]
2020-09-17 17:20:32 -07:00
Wenkai Du 5f49a0e088 Add NPS4 support on some models (#256)
* Add NPS4 support on some models

* Add XML models

[ROCm/rccl commit: 391bbf3f1e]
2020-08-19 11:03:20 -07:00
Wenkai Du 3d5fb8142e Add another Rome model (#249)
* Add another Rome model

* Add gfx908 4P3L models and support

* Revert "Use cached value for detecting GDR support only once"

This reverts commit 0108a1219d.

* Skip using ibverb for GPU direct RDMA detection

* Fine tune one Rome model

[ROCm/rccl commit: a51e4071e3]
2020-08-17 10:51:02 -07:00
Wenkai Du c9815aaa36 Add more Rome 4P2H models
[ROCm/rccl commit: 09ef75656a]
2020-08-06 18:20:02 +00:00
Wenkai Du 5f96f13e75 Allow setup ring through NCCL_RINGS to facilitate testing
[ROCm/rccl commit: 5b03132ace]
2020-08-04 21:07:00 +00:00
Wenkai Du 22a6211eaf Improve 4P2H topology on Rome (#243)
1. Use bi-directional rings
2. GPU search is sorted by PCI device ID to get consistent results

[ROCm/rccl commit: d1e20b4c5e]
2020-07-28 14:21:44 -07:00
Wenkai Du 487f93b83f Topology tuning for 4P2H on Rome (#242)
* Topology tuning for 4P2H on Rome

* Use ncclTopoIdToIndex

[ROCm/rccl commit: e7a10aa0e4]
2020-07-27 11:53:57 -07:00
Wenkai Du f604fc774e Add 8P6L multi-node models (#239)
[ROCm/rccl commit: d5f90e19b5]
2020-07-21 14:10:36 -07:00
Wenkai Du 3e2c9054cd Change default channels duplication for chordal ring (#233)
[ROCm/rccl commit: ab787c767e]
2020-07-14 15:16:50 -07:00
Wenkai Du c7805f224e Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: da3b197d6c]
2020-07-01 16:51:25 -07:00
Sylvain Jeaugey 1f6b75ca8b 2.7.5-1
Minor fixes for A100 platforms.
Add a WARN for invalid GroupEnd call.


[ROCm/rccl commit: 01afd20a77]
2020-06-26 14:39:49 -07:00
Wenkai Du 67fb4f6cdb Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: 26a0fd2517]
2020-06-09 17:40:11 -07:00
Sylvain Jeaugey 89072c82e5 2.7.3-1
Add support for A100 GPU and related platforms.
Add support for CUDA 11.
Add support for send/receive operations (beta).


[ROCm/rccl commit: 5949d96f36]
2020-06-08 09:31:44 -07:00
Wenkai Du 27519fd019 Give preference to path with more XGMI connections
[ROCm/rccl commit: b3c9852634]
2020-05-14 15:33:16 -07:00
Wenkai Du 9813d67cd1 Merge remote-tracking branch 'nccl/master' into HEAD
[ROCm/rccl commit: c4edc257b0]
2020-04-27 17:16:54 +00:00
Wenkai Du 7b7f781658 Fix incorrect next device ID in PCI ordered search
[ROCm/rccl commit: edb49ed2d5]
2020-04-25 01:01:13 +00:00
Sylvain Jeaugey 627e1a06d0 Fix bug #307 : wrong NIC selection on the reduction tree.
The reduction tree (tree up) was inverting the NICs to use,
causing performance issue in cases where we are using different
NICs on a given channel.


[ROCm/rccl commit: b5b6c6acdd]
2020-04-09 17:14:07 -07:00
Wenkai Du 4aeb7f041e Merge remote-tracking branch 'nccl/master' into v2.6.4_merge
[ROCm/rccl commit: fa36fd9ef9]
2020-04-01 13:35:12 -07:00
Sylvain Jeaugey 40adc74496 2.6.4-1
Add support for network collectives.
Add support for XML topology dump/injection.
Add text values for GDR and P2P Levels, including "NVL".
Add speed detection for PCI, Infiniband and Ethernet cards.
Add CPU detection for ARM and AMD CPUs.
Add support for adaptive routing on Infiniband.
Change NET plugin API to v3 : merge PCI path and GPU pointer
  capability into a single structure and add other properties.


[ROCm/rccl commit: b221128eca]
2020-03-20 14:58:36 -07:00
Wenkai Du b4be0ff3b8 Use fraction of system maxWidth as steps for searching
This reverts previous workaround of deducting only half of width
from paths.


[ROCm/rccl commit: 8391637613]
2020-02-26 09:14:35 -08:00
Wenkai Du 93d448e2fe Fix incorrect CR8 detection
Also change level of ring graph print to help debugging


[ROCm/rccl commit: f54dc58113]
2020-02-21 10:09:49 -08:00
Wenkai Du 8432e8a921 Generate 8G6L chordal ring from reference
[ROCm/rccl commit: abcfbf1231]
2020-02-11 22:01:12 +00:00
Stanley Tsang e5419407c4 Updating copyright notices for 2020.
[ROCm/rccl commit: 20fa04d9b6]
2020-01-29 15:28:08 -08:00
Wenkai Du d2fbcfea02 Misc fixes and improvements for 2.5.6
1. Fix RCCL unit test
2. Add ROME detection and tuning
3. Change default P2P level
4. Fix search algorithm for XGMI
5. Remove explicit channel duplication with implicit by using half of link speed
6. Add collective trace support
7. Correct Intel Skylake CPU detection and bandwidth
8. Fix topo connect function
9. Disable GDR read and remove unreachable code
10. Disable LL128 kernels
11. Add tuning parameters
12. Use original clock64() implementation which returns RTC counter value
13. Print out timestamp of collective trace
14. Do not use struct ncclColl in kernel launch parameter
15. Fix abort handling and add tracing
17. Add __launch_bounds__ to kernel functions
18. Remove unused abortCount
19. Unset default MIN_NRINGS and MIN_NCHANNELS
20. Do not allocate shared memory when not using LL128 kernels
21. Correct time print out in tuning log


[ROCm/rccl commit: 1e55645d97]
2020-01-29 15:27:05 -08:00
Sylvain Jeaugey 71560fd67b 2.5.6-1 (#255)
Add LL128 Protocol.

Rewrite the topology detection and tree/ring creation (#179). Improve
tree performance by sending/receiving from different GPUs. Add
model-based tuning to switch between the different algorithms and
protocols.

Rework P2P/SHM detection in containers (#155, #248).

Detect duplicated devices and return an error (#231).

Add tuning for GCP

[ROCm/rccl commit: 299c554dcc]
2019-11-19 14:57:39 -08:00