AI/rocm-systems
2
0
Fork 0
Código Incidencias Pull Requests Acciones Paquetes Proyectos Lanzamientos Wiki Actividad
1.326 Commits 3 Ramas 0 Etiquetas
64e4e20da5f06d5ca044f3d98d394952146b39c0
Gráfico de commits

227 Commits

Autor SHA1 Mensaje Fecha
Wenkai Du 058886cb20 Add another Rome model (#1095) 
[ROCm/rccl commit: df98a6957d]
 2024-02-28 10:46:05 -08:00
Wenkai Du 874998033f Add new GPU model (#1080) 
[ROCm/rccl commit: 74f9e5db64]
 2024-02-23 12:19:42 -08:00
Bertan Dogancay 32e1c8cba0 Merge pull request #1079 from BertanDogancay/2.19.4-sync 
2.19.4 Sync

[ROCm/rccl commit: 2fb12a9358]
 2024-02-16 09:50:11 -07:00
akolliasAMD e0dd21028f Allow bus id to be null (#1085) 
* Allow bus id to be null


[ROCm/rccl commit: bac57421c7]
 2024-02-15 16:36:51 -07:00
Wenkai Du d5f5091e5d Merge remote-tracking branch 'rccl/develop' into 2.19.4 
[ROCm/rccl commit: d999d9ad21]
 2024-02-09 11:31:03 -06:00
Wenkai Du 6775a75906 2.18.5 fix (#1077) 
* Revert "Revert "2.18.5-1""

This reverts commit 7cc572ecf9.

* Fix initial net device value

[ROCm/rccl commit: 5669b0d7b6]
 2024-02-09 09:18:38 -08:00
Wenkai Du ce39eefe65 Doubling P2P channels per peer on single node gfx94x only (#1074) 
[ROCm/rccl commit: 704c9ef0d1]
 2024-02-07 14:05:57 -08:00
Wenkai Du 57e508f2e4 Doubling P2P channels per peer on single node only (#1069) 
[ROCm/rccl commit: 1d989f6524]
 2024-02-02 12:41:00 -08:00
Wenkai Du 4f4e52aafe Merge remote-tracking branch 'rccl/develop' into 2.19.4 
[ROCm/rccl commit: 1a134b283b]
 2024-01-31 11:53:10 -06:00
BertanDogancay c2c9ed2acb Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: 9ff53eeeae]
 2024-01-30 14:43:43 -08:00
Pedram Alizadeh a184aabda5 modifying the tuning table to improve the performance of allreduce for 8MB and 16MB for single-node MI300X (#1063) 
[ROCm/rccl commit: ccfb35fa6d]
 2024-01-26 09:05:53 -05:00
Wenkai Du 71a356670f Increase P2P channels per peer (#1060) 
[ROCm/rccl commit: ffde530af5]
 2024-01-25 11:21:58 -08:00
BertanDogancay 404d398bac Merge remote-tracking branch 'nccl/v2.19' into develop 
[ROCm/rccl commit: 81ddf9de89]
 2024-01-24 15:25:33 -08:00
Wenkai Du dbf906d8fa Only use full MAXCHANNELS for gfx94x (#1050) 
[ROCm/rccl commit: 3325f96c56]
 2024-01-17 09:00:49 -08:00
Pedram Alizadeh 5c349cd729 adding rccl tuning parameters for MI300X gfx942 with 8 GPUs single and multi-node (#1047) 
[ROCm/rccl commit: b08124c85d]
 2024-01-16 13:44:32 -05:00
Wenkai Du 8d38747c65 Add option to force enable network transport on single node (#1046) 
[ROCm/rccl commit: 261707d90a]
 2024-01-16 07:54:18 -08:00
PedramAlizadeh 7cc572ecf9 Revert "2.18.5-1" 
This reverts commit 26b91b9dbb.


[ROCm/rccl commit: 767fde8210]
 2024-01-12 16:54:19 +00:00
Hossein Pourreza df0dbb887d cover more gpu/nic mapping cases (#1037) 
[ROCm/rccl commit: 735178c1fe]
 2024-01-10 08:01:37 -08:00
PedramAlizadeh 401a104c5a resolved conflicts, fixed the localNetCount/0 bug 
[ROCm/rccl commit: 0d515f9388]
 2023-12-18 08:11:34 +00:00
Ziyue Yang e4b63a8ba0 Fully disable MSCCL when machine is not matched (#1017) 
* Disable MSCCL algorithm meta loading when machine is not matched

* fully disable init

* fix potential segfault

[ROCm/rccl commit: 655742a3a6]
 2023-12-13 08:36:21 -08:00
Wenkai Du 918ce6c2e2 msccl: disable on multi-node (#1018) 
[ROCm/rccl commit: 53d807a5b9]
 2023-12-13 07:41:40 -08:00
Wenkai Du b38b7fa3a2 Increase max channles to 64 (#993) 
[ROCm/rccl commit: 4ba65d1d6a]
 2023-12-01 16:01:11 -08:00
pradeep-ramanna bf57487384 Fix GPU to NIC mapping for peertopeer (#994) 
[ROCm/rccl commit: 0b53f79196]
 2023-12-01 08:00:17 -08:00
Ziyue Yang f0c47d085e Move MSCCL algorithm loading to initialization to workaround HIP graph conflict (#982) 
* MSCCL: pre-specify channels and pre-load algorithms

* add mutex

* fix bug

* clean include

* disable all-gathers temporarily

[ROCm/rccl commit: 4bb0b4a380]
 2023-11-30 09:47:20 -08:00
Wenkai Du dcf623f2ec Add special handling of gfx940 (#976) 
* Add special handling of gfx940

* Update ring base

[ROCm/rccl commit: 50b2dd9fd7]
 2023-11-22 15:07:36 -08:00
Sylvain Jeaugey 382ae08419 Merge remote-tracking branch 'origin/master' 
[ROCm/rccl commit: b6d7438d31]
 2023-11-20 05:07:23 -08:00
Alexander Grund c78787e089 Fix use of CPUID overwriting registers in use. 
CPUID writes to EAX, EBX, ECX, and EDX so the inline-asm must state that.
Otherwise currently in-use register might get overwritten which may
cause all kinds of failures like segfaults or wrong results.

Alternatively `__cpuid` can be used which avoids this and related issues.
So do that as suggested in the GCC issue https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112513


[ROCm/rccl commit: cece6415b0]
 2023-11-14 12:38:02 +01:00
Sylvain Jeaugey b16882a024 2.19.4-1 
Split transport connect phase into multiple steps to avoid port
exhaustion when connecting alltoall at large scale. Defaults to 128
peers per round.
Fix memory leaks on CUDA graph capture.
Fix alltoallv crash on self-sendrecv.
Make topology detection more deterministic when PCI speeds are not
available (fix issue #1020).
Properly close shared memory in NVLS resources.
Revert proxy detach after 5 seconds.
Add option to print progress during transport connect.
Add option to set NCCL_DEBUG to INFO on first WARN.


[ROCm/rccl commit: 88d44d777f]
 2023-11-13 10:36:12 -08:00
Wenkai Du aa02d2b675 Use parallel init of LDS and adjust P2P channels for gfx94x (#943) 
* Use parallel init of LDS and adjust P2P channels for gfx94x

* Move another init to parallel

* Fix NCCL_NCHANNELS_PER_PEER setting

[ROCm/rccl commit: dbcba2923b]
 2023-11-03 16:06:49 -07:00
Nilesh M Negi 8b1254a4f1 Fix gcnArchName bug in topology dump (#937) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 1e5ca6820b]
 2023-10-28 12:30:36 -05:00
Nilesh M Negi 706750597c remove gcnArch support (#920) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: f22df90e5c]
 2023-10-26 12:09:15 -05:00
Wenkai Du cc4de02a86 Add missing gfx942 support (#927) 
[ROCm/rccl commit: c4e65fd382]
 2023-10-23 12:04:37 -07:00
Wenkai Du edeea499b5 Update rome models (#922) 
[ROCm/rccl commit: 4278a9918b]
 2023-10-18 17:28:01 -07:00
Bertan Dogancay 6f7965796f Revert "Remove 2H4P condition from P2P channels adjustment (#890)" (#904) 
This reverts commit 057e30e705.

[ROCm/rccl commit: a6ff4618c7]
 2023-10-04 09:46:11 -06:00
Sylvain Jeaugey 69ee68b6d3 2.19.3-1 
H800/H100 fixes and tuning.
Re-enable intra-process direct pointer buffer access when CUMEM is
enabled.


[ROCm/rccl commit: 8c6c595185]
 2023-09-26 05:57:15 -07:00
Sylvain Jeaugey 506d6c332c 2.19.1-1 
Add local user buffer registration for NVLink SHARP.
Add tuning plugin support.
Increase net API to v7 to allow for device-side packet reordering;
remove support for v4 plugins.
Add support for RoCE ECE.
Add support for C2C links.
Better detect SHM allocation failures to avoid crash with Bus Error.
Fix missing thread unlocks in bootstrap (Fixes #936).
Disable network flush by default on H100.
Move device code from src/collectives/device to src/device.


[ROCm/rccl commit: f9c3dc251e]
 2023-09-26 05:50:33 -07:00
akolliasAMD 6f7eb65308 changed the form that RCCL_TREE uses (#888) 
* changed the form that RCCL_TREE uses

[ROCm/rccl commit: b85d73c02e]
 2023-09-15 15:01:33 -06:00
Wenkai Du 057e30e705 Remove 2H4P condition from P2P channels adjustment (#890) 
[ROCm/rccl commit: 16dd05a58a]
 2023-09-13 12:54:21 -07:00
Ziyue Yang 6d593761dc Add single-node MI300X topology (#889) 
[ROCm/rccl commit: c1bfd5f0d8]
 2023-09-13 11:07:17 -07:00
Audrey MP 2e3d45a53a Gcn arch name (#886) 
We use CMake to determine if we're compiling against a version of ROCm that supports gcnArchName and handles architecture checking appropriately. It includes a few helper functions as drop ins for the functionality we used gcnArch for before; sometimes to enable flags, and sometimes to set frequencies.

[ROCm/rccl commit: e58ec78d35]
 2023-09-12 15:34:40 -04:00
Wenkai Du af04103d72 Add MSCCL xml files (#861) 
[ROCm/rccl commit: aeca1af374]
 2023-08-23 14:12:34 -07:00
Sylvain Jeaugey 26b91b9dbb 2.18.5-1 
Fix NVLS search (issue #931).
Increase max IB NICs to 32.
Fix inconsistent device ordering (issue #820).
Try to use different devices for different GPUs in systems with
more than one NIC per GFU.


[ROCm/rccl commit: 559b70f86c]
 2023-08-23 06:32:36 -07:00
Wenkai Du 5983f0e371 Use relaxed atomics for LL on GFX11 (#859) 
[ROCm/rccl commit: 6a0a6a37d9]
 2023-08-21 16:28:39 -07:00
akolliasAMD 56129830a6 NCCL_TREES variable and rome model fixes (#856) 
[ROCm/rccl commit: d33cd5a233]
 2023-08-21 10:35:37 -06:00
Wenkai Du 0c31452135 Add new model support (#847) 
* Add new model support

* Update new rings

[ROCm/rccl commit: 7044599575]
 2023-08-10 17:14:51 -07:00
Wenkai Du c328294820 gfx11xx: disable LL protocol to workaround mtype issue (#840) 
[ROCm/rccl commit: 8e58b65873]
 2023-08-04 07:53:07 -07:00
Sylvain Jeaugey 020800d7b7 Fix inter-node NVLS graph search 
We were passing a net ID instead of a gpu index, which could cause
crashes if those were unrelated (and they usually are).

Issue #931


[ROCm/rccl commit: 8ed014bae9]
 2023-08-02 07:06:35 -07:00
Wenkai Du b0255eb9fb Revert "Enable Ll128 on gfx90a (#823)" (#829) 
This reverts commit d80fba94e9.

Also increase number of parallel jobs for linking

[ROCm/rccl commit: 3db371c9a5]
 2023-07-27 20:25:18 -07:00
Wenkai Du dfda1d6fab Enable gfx94x (#808) (#816) 
(cherry picked from commit 94da229a7788d74685d1591a4e75a8341de64f41)

[ROCm/rccl commit: a7fcd58a97]
 2023-07-21 07:31:27 -07:00
Wenkai Du f98715baea Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: abd0615351]
 2023-06-26 22:51:56 +00:00