Wykres commitów

77 Commity

Autor SHA1 Wiadomość Data
Wenkai Du faea6ead5c Query XGMI links from xml and adjust gfx906 channel usage (#410)
[ROCm/rccl commit: 818cdb16a8]
2021-07-27 17:32:41 -07:00
Wenkai Du 8fbeb14175 topo_expl: fix build after switching to rocm-smi-lib (#405)
* topo_expl: fix build after switching to rocm-smi-lib

* Use minimal of 4 channels for gfx908

[ROCm/rccl commit: 135d47d125]
2021-07-27 08:30:08 -07:00
Lu 2ab26a2ff9 Add more info to RCCL logging for topo-aware optim.
[ROCm/rccl commit: bd6dbca8fb]
2021-07-22 09:52:39 -07:00
Wenkai Du 71dfc3978e Use rocm_smi_lib for getting topology information (#402)
* Use rocm_smi_lib for getting topology information

* Add rocm-smi-lib dependency to RCCL package

[ROCm/rccl commit: 56155ff5b6]
2021-07-08 13:23:11 -07:00
Wenkai Du f82f99f533 Select sendrecv path based on collective data size (#391)
* Select sendrecv path based on collective data size

* Add comments on packing and unpacking group field

* Toggling RCCL_P2P_NET_DISABLE in combined calls unit tests

[ROCm/rccl commit: 6dcae8a459]
2021-06-10 17:51:04 -07:00
Wenkai Du 5bebcb0015 Setup collectives threshold for enabling intranet (#387)
* Setup collectives threshold for enabling intranet

* Use separate operation counters for coll and p2p

[ROCm/rccl commit: b815a2800f]
2021-06-09 13:24:26 -07:00
Wenkai Du b154b532a2 Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: e3abf1c2ec]
2021-05-25 20:52:15 -07:00
Wenkai Du e0edd3d5e4 Merge remote-tracking branch 'nccl/master' into 2.9.8
[ROCm/rccl commit: 87727383fe]
2021-05-17 10:15:16 -07:00
Sylvain Jeaugey 3dd3a1ca66 2.9.9-1
Fix crash when setting NCCL_MAX_P2P_NCHANNELS below nchannels.
Fix hang during sendrecv dynamic NVB connection establishment on
cubemesh topologies.
Add environment variable to only use SHARP on communicators beyond
a given number of ranks.
Add debug subsystem to trace memory allocations.
Fix compilation with TRACE=1. (Issue #505)


[ROCm/rccl commit: 3fec2fa5ee]
2021-05-12 11:09:31 -07:00
gilbertlee-amd 071150a1b4 Clique tuning upgrade (#352) (#19)
* Enabling clique for any XGMI-connected topology, adding tuning
* Updating CHANGELOG for clique tuning
* Re-working clique barrier system to work on multi-process / multi-gpu

[ROCm/rccl commit: e796b1645c]
2021-05-11 08:44:59 -06:00
Sylvain Jeaugey 780273774a 2.9.8-1
Fix memory leaks.
Fix crash in bootstrap error case.
Fix Collnet clean-up issue.
Make PCI switch vendor/device optional for XML injection.
Add support for nvidia-peermem module.


[ROCm/rccl commit: ca8485b0d0]
2021-05-10 14:00:03 -07:00
Wenkai Du a76bebf8b6 Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: a4ea1fed5b]
2021-05-05 16:01:01 -07:00
Wenkai Du 185ad0deab Use better name for kernel collective trace enable (#357)
"NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,COLL" enables collectives API
trace. Adding "RCCL_KERNEL_COLL_TRACE_ENABLE=1" enables kernel traces.

[ROCm/rccl commit: ed237dcaa7]
2021-04-26 08:35:53 -07:00
Wenkai Du 60a24aa4db Control collective trace from kernel separately (#356)
[ROCm/rccl commit: 9cc9c3360b]
2021-04-23 16:36:19 -07:00
Wenkai Du e28bac31aa Tune number of channels for gfx90a (#349)
[ROCm/rccl commit: 415c7cd3d1]
2021-04-19 15:27:01 -07:00
Sylvain Jeaugey 20da390b96 2.9.6-1
Add support for CUDA graphs.
Fuse BCM Gen4 switches to avoid suboptimal performance on some platforms. Issue #439.
Fix bootstrap issue caused by connection reordering.
Fix CPU locking block.
Improve CollNet algorithm.
Improve performance on DGX A100 for communicators with only one GPU per node.


[ROCm/rccl commit: a46ea10583]
2021-04-12 16:00:46 -07:00
TomSang 6105af2dfc Add detection of cooperative multi device launch attribute (#345)
[ROCm/rccl commit: 87f12cbb86]
2021-04-11 13:29:24 -07:00
gilbertlee-amd a7e99734e8 Fixing clique-topology detection (#342)
* Fixing clique-topology detection
* Fix to enable multi-process clique-based kernels

[ROCm/rccl commit: caba0a63d2]
2021-04-07 11:29:44 -06:00
Wenkai Du b4a7fa7011 Cleanup number of channels calculation (#340)
[ROCm/rccl commit: e26ad2995e]
2021-04-05 17:51:56 -07:00
Wenkai Du 8927d8bf17 Fix incorrect net counting (#339)
* Fix incorrect net counting

* Add comments

[ROCm/rccl commit: 17491c918e]
2021-04-05 12:21:57 -07:00
Wenkai Du 065bde98d8 collnet: support multiple NICs (#335)
[ROCm/rccl commit: d87dc7c2e8]
2021-03-25 20:59:32 -07:00
Wenkai Du 287ed0f18a Enable collnet in RCCL (#333)
* Enable CollNet and use different number of channels

* topo_expl: enable collnet

[ROCm/rccl commit: 1d6244b18d]
2021-03-19 12:58:13 -07:00
Wenkai Du 7374c512d9 Add GPU memory usage tracker (#326)
[ROCm/rccl commit: f60b76c67a]
2021-03-06 20:32:30 -08:00
Wenkai Du b7253710ca Revert "Port alltoall[v]" (#325)
This reverts commit 2c49121171.

[ROCm/rccl commit: 8e180cf087]
2021-03-06 13:59:31 -08:00
Wenkai Du bcf4ecb0e3 Enable local sendrecv over network if GDR is available on all GPUs (#324)
[ROCm/rccl commit: c018edf0f2]
2021-03-05 19:59:41 -08:00
Wenkai Du ab71643c99 Merge remote-tracking branch 'nccl/master' into 2.8.3
[ROCm/rccl commit: c985358e11]
2021-02-15 18:44:47 -05:00
Sylvain Jeaugey fc7bdb38a5 2.8.4-1
Fix hang in corner cases of alltoallv using point to point send/recv.
Harmonize error messages.
Fix missing NVTX section in the license.
Update README.


[ROCm/rccl commit: 911d61f214]
2021-02-09 15:36:48 -08:00
gilbertlee-amd c981e76efe Clique kernel support (#295) (#15)
* Adding experimental clique-based kernels (opt-in only)

Co-authored-by: Stanley Tsang <stanley.tsang@amd.com>
Co-authored-by: Gilbert Lee <gilbert.lee@amd.com>
Co-authored-by: Wenkai Du <43822138+wenkaidu@users.noreply.github.com>

Co-authored-by: Stanley Tsang <stanley.tsang@amd.com>
Co-authored-by: Wenkai Du <43822138+wenkaidu@users.noreply.github.com>

[ROCm/rccl commit: 3e62ceddc5]
2021-01-28 09:45:01 -07:00
Wenkai Du d4382de267 Improve collective trace
[ROCm/rccl commit: 2ddbe6646b]
2021-01-14 19:28:01 -05:00
Wenkai Du 2c49121171 Port alltoall[v]
[ROCm/rccl commit: f4d5d3d620]
2021-01-14 19:28:01 -05:00
Wenkai Du adff98765c Merge remote-tracking branch 'nccl/master' into no-target-id
[ROCm/rccl commit: d469947641]
2021-01-14 19:27:53 -05:00
Sylvain Jeaugey a8908b34ee 2.8.3-1
Optimization for Tree allreduce on A100.
Improve aggregation performance.
Use shared buffers for inter-node send/recv.
Add NVTX profiling hooks.
Accelerate alltoall connections by merging communication for all
channels.
Add support for one hop communication through NVLink, for faster
send/recv communication on cubemesh topologies like DGX-1.
Improve alltoall scheduling to better balance intra/inter node
communication.
Increase send/recv parallelism by 8x, each warp sending or
receiving to a different peer.
Net: move to v4.
Net: make flush operation asynchronous to accelerate alltoall.
Net: define maximum number of requests.
Fix hang when using LL128 protocol after 2^31 steps.
Fix #379 : topology injection failing when using less GPUs than
described in the XML.
Fix #394 : protocol mismatch causing hangs or crashes when using
one GPU per node.


[ROCm/rccl commit: 920dbe5b35]
2020-11-17 11:08:52 -08:00
Wenkai Du a0109b2eae Use ncclSend/ncclRecv for alltoall type of collectives as default (#297)
[ROCm/rccl commit: 2e8b3a0857]
2020-11-09 11:23:17 -08:00
Wenkai Du 194135a40c Merge remote-tracking branch 'nccl/master' into nccl_sync
[ROCm/rccl commit: c835d8263a]
2020-10-15 18:42:38 -04:00
gilbertlee-amd 94437eef28 Revert "Initial support for clique-based kernels (#276)" (#280)
This reverts commit d68a532bc6.

[ROCm/rccl commit: 84a2541e01]
2020-10-15 11:30:18 -07:00
Sylvain Jeaugey 591ffd32fe Fix affinity move
[ROCm/rccl commit: 0e14394c5f]
2020-10-13 16:58:05 -07:00
Sylvain Jeaugey 5de6b6681d Make sure proxy threads inherit the CPU affinity.
[ROCm/rccl commit: c6dbdb0084]
2020-10-13 16:37:52 -07:00
gilbertlee-amd d68a532bc6 Initial support for clique-based kernels (#276)
* Initial support for clique-based kernels

[ROCm/rccl commit: 2b8184808d]
2020-10-13 11:22:04 -06:00
Wenkai Du 41260bb948 Rework Rome detection and add multiple network ports models (#274)
* Rework Rome detection and add multiple network ports models

* Remove unused opCount in p2p transport

[ROCm/rccl commit: ae008fd2db]
2020-10-07 13:37:36 -07:00
Wenkai Du dbde26e681 Add Alltoallv RCCL kernel implementation (#269)
* Add alltoallv API and implementation

* Extend Rome P2P channel limit to multinode and alltoall kernels

* topo_expl: fix compilation and sync up with main

* gtest: use RCCL alltoallv API

* Code review changes

[ROCm/rccl commit: b871ea3c0c]
2020-09-30 16:25:36 -07:00
Wenkai Du 7ba087e069 Ensure all ranks on same send/receive or alltoall kernel path (#271)
[ROCm/rccl commit: 44fcde7835]
2020-09-24 08:25:04 -07:00
Wenkai Du 37f7eec6b7 Change network plugin name to librccl-net.so (#266)
[ROCm/rccl commit: d871fceb54]
2020-09-18 13:23:30 -07:00
Wenkai Du f0a303664e Limit P2P channels on Rome
[ROCm/rccl commit: 42955f5f4f]
2020-09-17 17:20:32 -07:00
Wenkai Du 03bb6bcb54 Increase minimal channels for gfx908 (#259)
[ROCm/rccl commit: c5cbece6d0]
2020-08-26 11:40:11 -07:00
Wenkai Du 14ad6ff3b4 Merge remote-tracking branch 'nccl/master' into 2.7.8
[ROCm/rccl commit: 7e3f841fab]
2020-08-10 16:11:00 +00:00
David Addison 4acaffebb3 2.7.8-1
Fix collective mismatch error when using ncclSend/ncclRecv


[ROCm/rccl commit: 033d799524]
2020-07-27 16:34:09 -07:00
Wenkai Du 3e2c9054cd Change default channels duplication for chordal ring (#233)
[ROCm/rccl commit: ab787c767e]
2020-07-14 15:16:50 -07:00
Wenkai Du 3389838def Rework network proxy profiling
[ROCm/rccl commit: 7484e53ff7]
2020-06-13 03:13:58 +00:00
Wenkai Du 846e1a4427 Add network proxy profiling support
[ROCm/rccl commit: 812543104d]
2020-06-09 17:44:15 -07:00
Wenkai Du bc6840f937 Calculate and use total wait cycles for RCCL profiling
[ROCm/rccl commit: c9aa11928a]
2020-06-09 17:44:15 -07:00