304 Commit-ok

Szerző SHA1 Üzenet Dátum
Wenkai Du 5becf1669f Add another Rome model (#553)
* Add another Rome model

* Add option to force enable intranet on single node

* Limit p2p channels to number of ranks

* Refine p2p channels handling

[ROCm/rccl commit: ef499c4810]
2022-05-31 11:31:30 -07:00
akolliasAMD 22dc8bd246 Added creation of new tree and added switch for using treesplit for specific cases (#551)
[ROCm/rccl commit: 98f0809a39]
2022-05-25 18:55:14 -04:00
Wenkai Du 86e8797602 Add switch for pivot alltoall kernel (#549)
[ROCm/rccl commit: 6707a270b1]
2022-05-17 18:14:04 -07:00
Sylvain Jeaugey 1c5734046d 2.12.12-1
Improve allreduce performance when we have more than one network interface per
GPU and we need to use PXN to close rings.
Add support for PCI Gen5 on 5.4 kernels.
Fix crash when setting NCCL_SET_THREAD_NAME.
Fix random crash in init due to uninitialized struct.
Fix hang on cubemesh topologies.
Add P2P_DIRECT_DISABLE parameter to disable direct access to pointers within a
process.


[ROCm/rccl commit: 7aa1c46fd5]
2022-05-13 00:26:57 -07:00
Wenkai Du 67e7e6507e Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: d28e1cb44f]
2022-04-18 11:15:25 -07:00
Wenkai Du 3332cdff07 Support multiple tuning tables (#522)
* Support multiple tuning tables

* [UnitTests] Skip managed memory testing

[ROCm/rccl commit: bbe780ca6c]
2022-03-31 17:09:21 -07:00
Sylvain Jeaugey 27130280b2 2.12.10-1
Fix bug with CollNet
Fix bug with zero-bytes send/recv operations
Fix NCCL_PARAM implementation to avoid taking a lock on every call
Fix bug when setting NCCL_IB_QPS_PER_CONNECTION to more than one.
Improve error reporting for network errors.


[ROCm/rccl commit: 353e8ba446]
2022-03-30 02:27:01 -07:00
Sylvain Jeaugey f8886d8687 2.12.7-1
Add network communication through another GPU connected with NVLink
(PXN).
Add aggregation of messages coming from different local GPUs through
PXN and going to the same destination.
Add new v5 plugin API with grouped receives and tags.
Add compat for v4 plugins.
Add naming of NCCL threads to help debugging.
Fix NVLink detection and avoid data corruption when some NVLinks are
down.
Add support for Relaxed Ordering for IB.
Add profiling and timing infrastructure.


[ROCm/rccl commit: 3c223c105a]
2022-03-02 20:48:56 +01:00
Ziyue Yang dfa9b9e958 Add Pivot AllToAll algorithm for Rome model (#503)
* add a2a pivot interface

* remove debug info

* address comments

* fix bug

* remove custom script

* address comments

* fix bug

[ROCm/rccl commit: b569c0a1db]
2022-02-20 21:09:47 -08:00
Wenkai Du 4d43d9ce22 Update Rome models (#491)
[ROCm/rccl commit: 598c6fdded]
2022-01-14 10:03:30 -08:00
Wenkai Du 618fbfc644 Merge remote-tracking branch 'origin/develop' into 2.11.4
[ROCm/rccl commit: 434ecb0e1f]
2022-01-03 09:54:16 -08:00
Wenkai Du 864c6a645e Determine fine grained memory availability at RCCL bootstrapping (#471)
[ROCm/rccl commit: e9bf01fb7e]
2021-11-19 08:12:53 -08:00
Wenkai Du f8bd2d0cfa Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: 3a919c1f49]
2021-11-11 14:22:12 -08:00
Wenkai Du fd98ee84b4 Update Rome model matching (#461)
* Update Rome model matching

* Add another Rome model

* Automatically setup NET GDR level from model

[ROCm/rccl commit: 0331e39f81]
2021-11-05 08:53:47 -07:00
Wenkai Du befaa35e12 profiling: fix incorrect print out in timing profile (#457)
[ROCm/rccl commit: 26fc6b0919]
2021-11-03 16:22:21 -07:00
Wenkai Du 747216e2b2 Enable timing profiling mode (#447)
[ROCm/rccl commit: ec36c4c326]
2021-10-27 08:21:48 -07:00
Wenkai Du 4d979ce13d Fix PCIe gen detection (#437)
* Fix PCIe gen detection

* Update profiling support

[ROCm/rccl commit: 2508507d0a]
2021-10-15 08:23:50 -07:00
Wenkai Du b587b55c2e Add more Rome models (#434)
* Add more Rome models

* Update models and tuning

* Update tuning

[ROCm/rccl commit: 2249a1d9d3]
2021-10-12 08:23:20 -07:00
Wenkai Du 91eca0d7d2 Trim NICs when all GPUs are connected by XGMI (#430)
* Trim NICs when all GPUs are connected by XGMI

* Only enable clique with maximum of 2 hops

[ROCm/rccl commit: 29c729d8b6]
2021-10-05 18:27:43 -07:00
Ke Wen e51f2a83da 2.11.4-1
Add new API for creating a reduction operation which multiplies the input by a rank-specific scalar before doing an inter-rank summation (see: ncclRedOpCreatePreMulSum).
Improve CollNet (SHARP) performance of ncclAllReduce when captured in a CUDA Graph via user buffer registration.
Add environment variable NCCL_NET_PLUGIN="<suffix>" to allow user to choose among multiple NCCL net plugins by substituting into "libnccl-net-<suffix>.so".
Fix memory leak of NVB connections.
Fix topology detection of IB Virtual Functions (SR-IOV).


[ROCm/rccl commit: e11238b302]
2021-09-08 16:06:23 -07:00
Wenkai Du 8acdb77cc0 Merge remote-tracking branch 'origin/develop' into 2.10.3
[ROCm/rccl commit: 8ee2b7932a]
2021-09-13 15:51:53 -07:00
Wenkai Du 9ffeb41fe1 Update tuning table (#424)
[ROCm/rccl commit: ef432e48e1]
2021-09-13 08:39:01 -07:00
Wenkai Du d75504e9dc Remove atomic from profiling
[ROCm/rccl commit: 31bd4236f1]
2021-09-08 14:20:32 -05:00
Wenkai Du b9508a6aba Implement NIC identification and remapping (#420)
* Add 1H16P GPU model

* Implement NIC identification and remapping

* Revert "Sort IB devices based on device name (#413)"

This reverts commit de0c586bad.

* Fix permute and check order

* Correction on IB speed reporting

* Revert "Allow user to link layer with RCCL_IB_HCA_SKIP_LINK_LAYER (#361)"

This reverts commit fa690c47a0.

[ROCm/rccl commit: 5c8380ff5b]
2021-08-24 09:42:04 -07:00
Wenkai Du 4fd7a14087 Merge remote-tracking branch 'origin/develop' into 2.10.3
[ROCm/rccl commit: d5f93649ff]
2021-08-24 09:49:47 -07:00
Wenkai Du 4b89e98675 Merge remote-tracking branch 'nccl/master' into 2.10.3
[ROCm/rccl commit: bf2339f93e]
2021-07-30 16:23:14 -07:00
Wenkai Du faea6ead5c Query XGMI links from xml and adjust gfx906 channel usage (#410)
[ROCm/rccl commit: 818cdb16a8]
2021-07-27 17:32:41 -07:00
Wenkai Du 8fbeb14175 topo_expl: fix build after switching to rocm-smi-lib (#405)
* topo_expl: fix build after switching to rocm-smi-lib

* Use minimal of 4 channels for gfx908

[ROCm/rccl commit: 135d47d125]
2021-07-27 08:30:08 -07:00
Lu 2ab26a2ff9 Add more info to RCCL logging for topo-aware optim.
[ROCm/rccl commit: bd6dbca8fb]
2021-07-22 09:52:39 -07:00
Ke Wen 58a9f19dc3 2.10.3-1
Add support for bfloat16.
Add ncclAvg reduction operation.
Improve performance for aggregated operations.
Improve performance for tree.
Improve network error reporting.
Add NCCL_NET parameter to force a specific network.
Add NCCL_IB_QPS_PER_CONNECTION parameter to split IB traffic onto multiple queue pairs.
Fix topology detection error in WSL2.
Fix proxy memory elements affinity (improve alltoall performance).
Fix graph search on cubemesh topologies.
Fix hang in cubemesh during NVB connections.


[ROCm/rccl commit: 7e51592129]
2021-07-08 14:30:14 -07:00
Wenkai Du 71dfc3978e Use rocm_smi_lib for getting topology information (#402)
* Use rocm_smi_lib for getting topology information

* Add rocm-smi-lib dependency to RCCL package

[ROCm/rccl commit: 56155ff5b6]
2021-07-08 13:23:11 -07:00
Wenkai Du f82f99f533 Select sendrecv path based on collective data size (#391)
* Select sendrecv path based on collective data size

* Add comments on packing and unpacking group field

* Toggling RCCL_P2P_NET_DISABLE in combined calls unit tests

[ROCm/rccl commit: 6dcae8a459]
2021-06-10 17:51:04 -07:00
Wenkai Du 5bebcb0015 Setup collectives threshold for enabling intranet (#387)
* Setup collectives threshold for enabling intranet

* Use separate operation counters for coll and p2p

[ROCm/rccl commit: b815a2800f]
2021-06-09 13:24:26 -07:00
Wenkai Du b154b532a2 Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: e3abf1c2ec]
2021-05-25 20:52:15 -07:00
Wenkai Du e0edd3d5e4 Merge remote-tracking branch 'nccl/master' into 2.9.8
[ROCm/rccl commit: 87727383fe]
2021-05-17 10:15:16 -07:00
Sylvain Jeaugey 3dd3a1ca66 2.9.9-1
Fix crash when setting NCCL_MAX_P2P_NCHANNELS below nchannels.
Fix hang during sendrecv dynamic NVB connection establishment on
cubemesh topologies.
Add environment variable to only use SHARP on communicators beyond
a given number of ranks.
Add debug subsystem to trace memory allocations.
Fix compilation with TRACE=1. (Issue #505)


[ROCm/rccl commit: 3fec2fa5ee]
2021-05-12 11:09:31 -07:00
gilbertlee-amd 071150a1b4 Clique tuning upgrade (#352) (#19)
* Enabling clique for any XGMI-connected topology, adding tuning
* Updating CHANGELOG for clique tuning
* Re-working clique barrier system to work on multi-process / multi-gpu

[ROCm/rccl commit: e796b1645c]
2021-05-11 08:44:59 -06:00
Sylvain Jeaugey 780273774a 2.9.8-1
Fix memory leaks.
Fix crash in bootstrap error case.
Fix Collnet clean-up issue.
Make PCI switch vendor/device optional for XML injection.
Add support for nvidia-peermem module.


[ROCm/rccl commit: ca8485b0d0]
2021-05-10 14:00:03 -07:00
Wenkai Du a76bebf8b6 Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: a4ea1fed5b]
2021-05-05 16:01:01 -07:00
Wenkai Du 185ad0deab Use better name for kernel collective trace enable (#357)
"NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,COLL" enables collectives API
trace. Adding "RCCL_KERNEL_COLL_TRACE_ENABLE=1" enables kernel traces.

[ROCm/rccl commit: ed237dcaa7]
2021-04-26 08:35:53 -07:00
Wenkai Du 60a24aa4db Control collective trace from kernel separately (#356)
[ROCm/rccl commit: 9cc9c3360b]
2021-04-23 16:36:19 -07:00
Wenkai Du e28bac31aa Tune number of channels for gfx90a (#349)
[ROCm/rccl commit: 415c7cd3d1]
2021-04-19 15:27:01 -07:00
Sylvain Jeaugey 20da390b96 2.9.6-1
Add support for CUDA graphs.
Fuse BCM Gen4 switches to avoid suboptimal performance on some platforms. Issue #439.
Fix bootstrap issue caused by connection reordering.
Fix CPU locking block.
Improve CollNet algorithm.
Improve performance on DGX A100 for communicators with only one GPU per node.


[ROCm/rccl commit: a46ea10583]
2021-04-12 16:00:46 -07:00
TomSang 6105af2dfc Add detection of cooperative multi device launch attribute (#345)
[ROCm/rccl commit: 87f12cbb86]
2021-04-11 13:29:24 -07:00
gilbertlee-amd a7e99734e8 Fixing clique-topology detection (#342)
* Fixing clique-topology detection
* Fix to enable multi-process clique-based kernels

[ROCm/rccl commit: caba0a63d2]
2021-04-07 11:29:44 -06:00
Wenkai Du b4a7fa7011 Cleanup number of channels calculation (#340)
[ROCm/rccl commit: e26ad2995e]
2021-04-05 17:51:56 -07:00
Wenkai Du 8927d8bf17 Fix incorrect net counting (#339)
* Fix incorrect net counting

* Add comments

[ROCm/rccl commit: 17491c918e]
2021-04-05 12:21:57 -07:00
Wenkai Du 065bde98d8 collnet: support multiple NICs (#335)
[ROCm/rccl commit: d87dc7c2e8]
2021-03-25 20:59:32 -07:00
Wenkai Du 287ed0f18a Enable collnet in RCCL (#333)
* Enable CollNet and use different number of channels

* topo_expl: enable collnet

[ROCm/rccl commit: 1d6244b18d]
2021-03-19 12:58:13 -07:00
Wenkai Du 7374c512d9 Add GPU memory usage tracker (#326)
[ROCm/rccl commit: f60b76c67a]
2021-03-06 20:32:30 -08:00