Commit Graph

92 Commitit

Tekijä SHA1 Viesti Päivämäärä
Aristotelis 0b55e01ef3 Merge remote-tracking branch 'ncclRepo/master' into develop
[ROCm/rccl commit: e0864e7093]
2022-06-02 15:27:24 +00:00
Wenkai Du 5becf1669f Add another Rome model (#553)
* Add another Rome model

* Add option to force enable intranet on single node

* Limit p2p channels to number of ranks

* Refine p2p channels handling

[ROCm/rccl commit: ef499c4810]
2022-05-31 11:31:30 -07:00
Wenkai Du b30b8becea Refine and add new Rome models (#548)
[ROCm/rccl commit: 283dc86a73]
2022-05-17 08:23:59 -07:00
Sylvain Jeaugey 1c5734046d 2.12.12-1
Improve allreduce performance when we have more than one network interface per
GPU and we need to use PXN to close rings.
Add support for PCI Gen5 on 5.4 kernels.
Fix crash when setting NCCL_SET_THREAD_NAME.
Fix random crash in init due to uninitialized struct.
Fix hang on cubemesh topologies.
Add P2P_DIRECT_DISABLE parameter to disable direct access to pointers within a
process.


[ROCm/rccl commit: 7aa1c46fd5]
2022-05-13 00:26:57 -07:00
Wenkai Du 67e7e6507e Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: d28e1cb44f]
2022-04-18 11:15:25 -07:00
Sylvain Jeaugey f8886d8687 2.12.7-1
Add network communication through another GPU connected with NVLink
(PXN).
Add aggregation of messages coming from different local GPUs through
PXN and going to the same destination.
Add new v5 plugin API with grouped receives and tags.
Add compat for v4 plugins.
Add naming of NCCL threads to help debugging.
Fix NVLink detection and avoid data corruption when some NVLinks are
down.
Add support for Relaxed Ordering for IB.
Add profiling and timing infrastructure.


[ROCm/rccl commit: 3c223c105a]
2022-03-02 20:48:56 +01:00
Sylvain Jeaugey 51df47f9b8 Revert "remove unused basePath"
This reverts commit d973ddac8b.


[ROCm/rccl commit: cc78e9fab8]
2022-01-21 12:30:34 +01:00
void-main d973ddac8b remove unused basePath
[ROCm/rccl commit: 445bc19657]
2022-01-21 12:12:26 +01:00
Wenkai Du 93c7526fdc Add another Rome model (#483)
[ROCm/rccl commit: f8d0775a6f]
2022-01-05 09:26:31 -08:00
Wenkai Du fd98ee84b4 Update Rome model matching (#461)
* Update Rome model matching

* Add another Rome model

* Automatically setup NET GDR level from model

[ROCm/rccl commit: 0331e39f81]
2021-11-05 08:53:47 -07:00
Wenkai Du 91eca0d7d2 Trim NICs when all GPUs are connected by XGMI (#430)
* Trim NICs when all GPUs are connected by XGMI

* Only enable clique with maximum of 2 hops

[ROCm/rccl commit: 29c729d8b6]
2021-10-05 18:27:43 -07:00
Wenkai Du 4b89e98675 Merge remote-tracking branch 'nccl/master' into 2.10.3
[ROCm/rccl commit: bf2339f93e]
2021-07-30 16:23:14 -07:00
Ke Wen 58a9f19dc3 2.10.3-1
Add support for bfloat16.
Add ncclAvg reduction operation.
Improve performance for aggregated operations.
Improve performance for tree.
Improve network error reporting.
Add NCCL_NET parameter to force a specific network.
Add NCCL_IB_QPS_PER_CONNECTION parameter to split IB traffic onto multiple queue pairs.
Fix topology detection error in WSL2.
Fix proxy memory elements affinity (improve alltoall performance).
Fix graph search on cubemesh topologies.
Fix hang in cubemesh during NVB connections.


[ROCm/rccl commit: 7e51592129]
2021-07-08 14:30:14 -07:00
Wenkai Du b154b532a2 Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: e3abf1c2ec]
2021-05-25 20:52:15 -07:00
Sylvain Jeaugey 3dd3a1ca66 2.9.9-1
Fix crash when setting NCCL_MAX_P2P_NCHANNELS below nchannels.
Fix hang during sendrecv dynamic NVB connection establishment on
cubemesh topologies.
Add environment variable to only use SHARP on communicators beyond
a given number of ranks.
Add debug subsystem to trace memory allocations.
Fix compilation with TRACE=1. (Issue #505)


[ROCm/rccl commit: 3fec2fa5ee]
2021-05-12 11:09:31 -07:00
Wenkai Du a76bebf8b6 Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: a4ea1fed5b]
2021-05-05 16:01:01 -07:00
Wenkai Du e28bac31aa Tune number of channels for gfx90a (#349)
[ROCm/rccl commit: 415c7cd3d1]
2021-04-19 15:27:01 -07:00
Sylvain Jeaugey 20da390b96 2.9.6-1
Add support for CUDA graphs.
Fuse BCM Gen4 switches to avoid suboptimal performance on some platforms. Issue #439.
Fix bootstrap issue caused by connection reordering.
Fix CPU locking block.
Improve CollNet algorithm.
Improve performance on DGX A100 for communicators with only one GPU per node.


[ROCm/rccl commit: a46ea10583]
2021-04-12 16:00:46 -07:00
Wenkai Du 018a31877c Rework network port trimming code (#338)
* Rework network port trimming code

* Move Rome related changes to separate source files

[ROCm/rccl commit: 1d2946ee4b]
2021-03-31 10:25:59 -07:00
Wenkai Du 287ed0f18a Enable collnet in RCCL (#333)
* Enable CollNet and use different number of channels

* topo_expl: enable collnet

[ROCm/rccl commit: 1d6244b18d]
2021-03-19 12:58:13 -07:00
Wenkai Du bcf4ecb0e3 Enable local sendrecv over network if GDR is available on all GPUs (#324)
[ROCm/rccl commit: c018edf0f2]
2021-03-05 19:59:41 -08:00
Wenkai Du d24df90882 Match NBIO only when GPUs and NICs are directly connected to CPU
[ROCm/rccl commit: ec8d89b1dd]
2021-02-22 18:52:29 -05:00
Wenkai Du ab71643c99 Merge remote-tracking branch 'nccl/master' into 2.8.3
[ROCm/rccl commit: c985358e11]
2021-02-15 18:44:47 -05:00
Sylvain Jeaugey fc7bdb38a5 2.8.4-1
Fix hang in corner cases of alltoallv using point to point send/recv.
Harmonize error messages.
Fix missing NVTX section in the license.
Update README.


[ROCm/rccl commit: 911d61f214]
2021-02-09 15:36:48 -08:00
Wenkai Du 57abf599b2 Enable GPU direct RDMA read from GPU
[ROCm/rccl commit: 5f97122442]
2021-02-03 02:48:30 +00:00
Wenkai Du 2c49121171 Port alltoall[v]
[ROCm/rccl commit: f4d5d3d620]
2021-01-14 19:28:01 -05:00
Wenkai Du 41bead5a4e Do not allow GPU as intermediate
[ROCm/rccl commit: 105db19a11]
2021-01-14 19:28:01 -05:00
Wenkai Du 34c6013299 Revert "Changes to topology based on XGMI (#272)"
This reverts commit 0a9adc16f4.


[ROCm/rccl commit: e055229e56]
2021-01-14 19:28:01 -05:00
Wenkai Du adff98765c Merge remote-tracking branch 'nccl/master' into no-target-id
[ROCm/rccl commit: d469947641]
2021-01-14 19:27:53 -05:00
Jonas Zhou 1db601566f x86: Add CPU detection for Zhaoxin processors
Signed-off-by: Jonas Zhou <JonasZhou@zhaoxin.com>


[ROCm/rccl commit: 3996562690]
2020-12-17 11:15:18 -08:00
Sylvain Jeaugey a8908b34ee 2.8.3-1
Optimization for Tree allreduce on A100.
Improve aggregation performance.
Use shared buffers for inter-node send/recv.
Add NVTX profiling hooks.
Accelerate alltoall connections by merging communication for all
channels.
Add support for one hop communication through NVLink, for faster
send/recv communication on cubemesh topologies like DGX-1.
Improve alltoall scheduling to better balance intra/inter node
communication.
Increase send/recv parallelism by 8x, each warp sending or
receiving to a different peer.
Net: move to v4.
Net: make flush operation asynchronous to accelerate alltoall.
Net: define maximum number of requests.
Fix hang when using LL128 protocol after 2^31 steps.
Fix #379 : topology injection failing when using less GPUs than
described in the XML.
Fix #394 : protocol mismatch causing hangs or crashes when using
one GPU per node.


[ROCm/rccl commit: 920dbe5b35]
2020-11-17 11:08:52 -08:00
Wenkai Du a4dd1a9548 Improve GPU direct RDMA handling on Rome (#294)
[ROCm/rccl commit: 709b7e4880]
2020-11-03 14:29:08 -08:00
Wenkai Du dbde26e681 Add Alltoallv RCCL kernel implementation (#269)
* Add alltoallv API and implementation

* Extend Rome P2P channel limit to multinode and alltoall kernels

* topo_expl: fix compilation and sync up with main

* gtest: use RCCL alltoallv API

* Code review changes

[ROCm/rccl commit: b871ea3c0c]
2020-09-30 16:25:36 -07:00
gilbertlee-amd 0a9adc16f4 Changes to topology based on XGMI (#272)
* Alterations to topology search to improve XGMI-enabled nodes

[ROCm/rccl commit: 01bd2573db]
2020-09-25 12:20:09 -06:00
Wenkai Du f0a303664e Limit P2P channels on Rome
[ROCm/rccl commit: 42955f5f4f]
2020-09-17 17:20:32 -07:00
Wenkai Du f604fc774e Add 8P6L multi-node models (#239)
[ROCm/rccl commit: d5f90e19b5]
2020-07-21 14:10:36 -07:00
Wenkai Du 67fb4f6cdb Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: 26a0fd2517]
2020-06-09 17:40:11 -07:00
Sylvain Jeaugey 89072c82e5 2.7.3-1
Add support for A100 GPU and related platforms.
Add support for CUDA 11.
Add support for send/receive operations (beta).


[ROCm/rccl commit: 5949d96f36]
2020-06-08 09:31:44 -07:00
Wenkai Du 4aeb7f041e Merge remote-tracking branch 'nccl/master' into v2.6.4_merge
[ROCm/rccl commit: fa36fd9ef9]
2020-04-01 13:35:12 -07:00
Sylvain Jeaugey 40adc74496 2.6.4-1
Add support for network collectives.
Add support for XML topology dump/injection.
Add text values for GDR and P2P Levels, including "NVL".
Add speed detection for PCI, Infiniband and Ethernet cards.
Add CPU detection for ARM and AMD CPUs.
Add support for adaptive routing on Infiniband.
Change NET plugin API to v3 : merge PCI path and GPU pointer
  capability into a single structure and add other properties.


[ROCm/rccl commit: b221128eca]
2020-03-20 14:58:36 -07:00
Luke Yeager d91217b16f [topology] remove NET links when trimming system
This fixes a memory leak.


[ROCm/rccl commit: 7a18fe0784]
2020-01-07 13:29:57 -08:00
Sylvain Jeaugey 71560fd67b 2.5.6-1 (#255)
Add LL128 Protocol.

Rewrite the topology detection and tree/ring creation (#179). Improve
tree performance by sending/receiving from different GPUs. Add
model-based tuning to switch between the different algorithms and
protocols.

Rework P2P/SHM detection in containers (#155, #248).

Detect duplicated devices and return an error (#231).

Add tuning for GCP

[ROCm/rccl commit: 299c554dcc]
2019-11-19 14:57:39 -08:00