AI/rocm-systems
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,416 Commits 3 Branches 0 Tags
7c029672a808d79c7abbfd1f64ca46a37142de37
Commit Graph
249 Commits
Author SHA1 Message Date
Nusrat Islam 7c029672a8 Merge pull request #1200 from nusislam/multi-node-256-fix 
graph: fix multi-node channel count

[ROCm/rccl commit: 9660e2e2dc]
 2024-06-07 14:34:20 -05:00
gilbertlee-amd efe851cfff Disabling NUMA maching for model 79 for some VM configs (#1204) 
[ROCm/rccl commit: 9b94a1052f]
 2024-06-06 17:15:04 -06:00
Nusrat Islam 3cbf67715f graph: restrict maxChannels to 64 for multi-node and RCCL_ENABLE_INTRANET=1 
[ROCm/rccl commit: 526cce9bf4]
 2024-06-06 10:58:41 -05:00
Nusrat Islam bf35178250 graph: fix multi-node minChannel count 
[ROCm/rccl commit: 6ab20a7c6b]
 2024-06-06 10:56:39 -05:00
Nusrat Islam 62cbb3bcdd set MIN_NCHANNEL limit to 64 for multi-node 
[ROCm/rccl commit: 9746d8ca3f]
 2024-06-03 13:05:05 -05:00
Nusrat Islam 48821ad0d7 set MAXCHANNELS to 128 
[ROCm/rccl commit: ef442f8f92]
 2024-06-03 13:05:05 -05:00
Nusrat Islam 2447161fb4 graph: restrict MAXCHANNELS for certain platforms 
[ROCm/rccl commit: 9f654f6cf5]
 2024-06-03 13:05:01 -05:00
gilbertlee-amd a009e43f3c Changing channel stride for MI300X multinode (#1196) 
* Shuffling MI300X multi-node channels
* Updating tree channel logic

[ROCm/rccl commit: 0948eecbba]
 2024-06-03 10:00:55 -06:00
gilbertlee-amd b35e2b8c4b Addressing possible out-of-bounds mem access during channel duplication (#1193) 
[ROCm/rccl commit: 354e0b29a6]
 2024-05-30 14:02:14 -06:00
Wenkai DuandPedramAlizadeh a724f1ebb7 Add ring simple chunk size tuning (#1180) 
* Add ring simple chunk size tuning

* modifying the tuning table to improve the performance of broadcast for 8MB to 32MB for single-node MI300X after ring simple chunk size tuning

* modifying the tuning table to improve the performance of reduce for 1MB to 4MB for single-node MI300X after ring simple chunk size tuning

---------

Co-authored-by: PedramAlizadeh <pmohamma@amd.com>

[ROCm/rccl commit: 73221b4230]
 2024-05-29 07:59:47 -07:00
Pedram Alizadeh 43d875b422 modifying the tuning table to improve the performance of broadcast for 1MB to 64MB for single-node MI300X (#1172) 
[ROCm/rccl commit: 73acf3eeec]
 2024-05-08 15:49:33 -04:00
mberenjkandakolliasAMD 00c0f3d67f Adding ASAN changes to address memory leak issue" (#1170) 
Co-authored-by: akolliasAMD <akollias@amd.com>

[ROCm/rccl commit: 408278209d]
 2024-05-08 09:16:00 -05:00
Wenkai Du 63d8de3bd3 Add compiler warning for uninitialized variable and fix (#1163) 
* Add compiler warning for uninitialized variable and fix

* Add -Wsometimes-uninitialized

* Convert warning to error

[ROCm/rccl commit: b18784d8b8]
 2024-05-08 07:00:25 -07:00
Wenkai Du eab9978e48 Use normal permute path when one NIC per GPU (#1171) 
[ROCm/rccl commit: f679db6ff6]
 2024-05-08 06:59:57 -07:00
Wenkai Du c782aba364 Bypass NVIDIA Ampere related tuning (#1165) 
[ROCm/rccl commit: b513c3970a]
 2024-05-03 17:57:16 -07:00
Wenkai Du 7c811a7582 Fix ignore NUMA not being observed for NICs during model matching (#1164) 
[ROCm/rccl commit: bb58b1c258]
 2024-05-03 16:42:07 -07:00
Wenkai Du e494f29235 Replace __HIP_PLATFORM_HCC__ with __HIP_PLATFORM_AMD__ (#1154) 
[ROCm/rccl commit: 9e0c9b4ed8]
 2024-04-25 07:19:18 -07:00
BertanDogancay 36f9492cda Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: e1a835910e]
 2024-04-23 13:34:00 -07:00
Wenkai Du 35f8d269f8 Use hipExtMallocWithFlags to allocate host memory on APU (#1149) 
Also use SM60 as CUDA compatibility level.

[ROCm/rccl commit: 220066197a]
 2024-04-17 16:56:38 -07:00
gilbertlee-amd 422a7ffcbb Rail optimization for rings (#1140) 
- Modifies the ring creation algorithm to be friendlier to rail-optimized topologies (should not affect classic fabric topologies)

[ROCm/rccl commit: 4cb62f999a]
 2024-04-15 12:03:57 -06:00
Wenkai Du 058886cb20 Add another Rome model (#1095) 
[ROCm/rccl commit: df98a6957d]
 2024-02-28 10:46:05 -08:00
Sylvain Jeaugey 5fa796af0c 2.20.5-1 
Fix UDS connection failure when using ncclCommSplit. Issue #1185


[ROCm/rccl commit: 48bb7fec79]
 2024-02-26 02:52:39 -08:00
Wenkai Du 874998033f Add new GPU model (#1080) 
[ROCm/rccl commit: 74f9e5db64]
 2024-02-23 12:19:42 -08:00
Bertan Dogancay 32e1c8cba0 Merge pull request #1079 from BertanDogancay/2.19.4-sync 
2.19.4 Sync

[ROCm/rccl commit: 2fb12a9358]
 2024-02-16 09:50:11 -07:00
akolliasAMD e0dd21028f Allow bus id to be null (#1085) 
* Allow bus id to be null


[ROCm/rccl commit: bac57421c7]
 2024-02-15 16:36:51 -07:00
Sylvain Jeaugey 2ab8a3a750 2.20.3-1 
Add support for alternating rings, allow for cross-nic rings without
cross-rail communication.
Add support for user buffer registration for network send/recv.
Optimize aggregated operations to better utilize all channels.
Add flattening for BCM PCI gen5 switches.
Add support for inter-node NVLink communication
Add support for port fusion in NET/IB.
Add support for ReduceScatter and AllGather using Collnet.
Update net API to v8.
Fix hang during A2A connection.


[ROCm/rccl commit: b6475625fb]
 2024-02-13 04:22:38 -08:00
Wenkai Du d5f5091e5d Merge remote-tracking branch 'rccl/develop' into 2.19.4 
[ROCm/rccl commit: d999d9ad21]
 2024-02-09 11:31:03 -06:00
Wenkai Du 6775a75906 2.18.5 fix (#1077) 
* Revert "Revert "2.18.5-1""

This reverts commit 7cc572ecf9.

* Fix initial net device value

[ROCm/rccl commit: 5669b0d7b6]
 2024-02-09 09:18:38 -08:00
Wenkai Du ce39eefe65 Doubling P2P channels per peer on single node gfx94x only (#1074) 
[ROCm/rccl commit: 704c9ef0d1]
 2024-02-07 14:05:57 -08:00
Wenkai Du 57e508f2e4 Doubling P2P channels per peer on single node only (#1069) 
[ROCm/rccl commit: 1d989f6524]
 2024-02-02 12:41:00 -08:00
Wenkai Du 4f4e52aafe Merge remote-tracking branch 'rccl/develop' into 2.19.4 
[ROCm/rccl commit: 1a134b283b]
 2024-01-31 11:53:10 -06:00
BertanDogancay c2c9ed2acb Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: 9ff53eeeae]
 2024-01-30 14:43:43 -08:00
Pedram Alizadeh a184aabda5 modifying the tuning table to improve the performance of allreduce for 8MB and 16MB for single-node MI300X (#1063) 
[ROCm/rccl commit: ccfb35fa6d]
 2024-01-26 09:05:53 -05:00
Wenkai Du 71a356670f Increase P2P channels per peer (#1060) 
[ROCm/rccl commit: ffde530af5]
 2024-01-25 11:21:58 -08:00
BertanDogancay 404d398bac Merge remote-tracking branch 'nccl/v2.19' into develop 
[ROCm/rccl commit: 81ddf9de89]
 2024-01-24 15:25:33 -08:00
Wenkai Du dbf906d8fa Only use full MAXCHANNELS for gfx94x (#1050) 
[ROCm/rccl commit: 3325f96c56]
 2024-01-17 09:00:49 -08:00
Pedram Alizadeh 5c349cd729 adding rccl tuning parameters for MI300X gfx942 with 8 GPUs single and multi-node (#1047) 
[ROCm/rccl commit: b08124c85d]
 2024-01-16 13:44:32 -05:00
Wenkai Du 8d38747c65 Add option to force enable network transport on single node (#1046) 
[ROCm/rccl commit: 261707d90a]
 2024-01-16 07:54:18 -08:00
PedramAlizadeh 7cc572ecf9 Revert "2.18.5-1" 
This reverts commit 26b91b9dbb.


[ROCm/rccl commit: 767fde8210]
 2024-01-12 16:54:19 +00:00
Hossein Pourreza df0dbb887d cover more gpu/nic mapping cases (#1037) 
[ROCm/rccl commit: 735178c1fe]
 2024-01-10 08:01:37 -08:00
PedramAlizadeh 401a104c5a resolved conflicts, fixed the localNetCount/0 bug 
[ROCm/rccl commit: 0d515f9388]
 2023-12-18 08:11:34 +00:00
Ziyue Yang e4b63a8ba0 Fully disable MSCCL when machine is not matched (#1017) 
* Disable MSCCL algorithm meta loading when machine is not matched

* fully disable init

* fix potential segfault

[ROCm/rccl commit: 655742a3a6]
 2023-12-13 08:36:21 -08:00
Wenkai Du 918ce6c2e2 msccl: disable on multi-node (#1018) 
[ROCm/rccl commit: 53d807a5b9]
 2023-12-13 07:41:40 -08:00
Wenkai Du b38b7fa3a2 Increase max channles to 64 (#993) 
[ROCm/rccl commit: 4ba65d1d6a]
 2023-12-01 16:01:11 -08:00
pradeep-ramanna bf57487384 Fix GPU to NIC mapping for peertopeer (#994) 
[ROCm/rccl commit: 0b53f79196]
 2023-12-01 08:00:17 -08:00
Ziyue Yang f0c47d085e Move MSCCL algorithm loading to initialization to workaround HIP graph conflict (#982) 
* MSCCL: pre-specify channels and pre-load algorithms

* add mutex

* fix bug

* clean include

* disable all-gathers temporarily

[ROCm/rccl commit: 4bb0b4a380]
 2023-11-30 09:47:20 -08:00
Wenkai Du dcf623f2ec Add special handling of gfx940 (#976) 
* Add special handling of gfx940

* Update ring base

[ROCm/rccl commit: 50b2dd9fd7]
 2023-11-22 15:07:36 -08:00
Sylvain Jeaugey 382ae08419 Merge remote-tracking branch 'origin/master' 
[ROCm/rccl commit: b6d7438d31]
 2023-11-20 05:07:23 -08:00
Alexander Grund c78787e089 Fix use of CPUID overwriting registers in use. 
CPUID writes to EAX, EBX, ECX, and EDX so the inline-asm must state that.
Otherwise currently in-use register might get overwritten which may
cause all kinds of failures like segfaults or wrong results.

Alternatively `__cpuid` can be used which avoids this and related issues.
So do that as suggested in the GCC issue https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112513


[ROCm/rccl commit: cece6415b0]
 2023-11-14 12:38:02 +01:00
Sylvain Jeaugey b16882a024 2.19.4-1 
Split transport connect phase into multiple steps to avoid port
exhaustion when connecting alltoall at large scale. Defaults to 128
peers per round.
Fix memory leaks on CUDA graph capture.
Fix alltoallv crash on self-sendrecv.
Make topology detection more deterministic when PCI speeds are not
available (fix issue #1020).
Properly close shared memory in NVLS resources.
Revert proxy detach after 5 seconds.
Add option to print progress during transport connect.
Add option to set NCCL_DEBUG to INFO on first WARN.


[ROCm/rccl commit: 88d44d777f]
 2023-11-13 10:36:12 -08:00