b09ea29d66
graph: fix minNchannels for multi-node overwrite ( #1230 )
2024-06-26 16:56:10 -05:00
ad31d93f3d
Revert "Changing channel stride for MI300X multinode ( #1196 )" ( #1224 )
...
This reverts channel stride change in commit
0948eecbba
2024-06-25 14:03:30 -07:00
e170f41ddd
Unit Tests for testing channels ( #1222 )
2024-06-25 10:10:10 -05:00
05df0f8cea
graph: fix minNchannels for multi-node
...
Multi-node rccl was not correctly setting the minNchannels value. This
PR fixes the bug.
2024-06-24 16:42:44 -05:00
9660e2e2dc
Merge pull request #1200 from nusislam/multi-node-256-fix
...
graph: fix multi-node channel count
2024-06-07 14:34:20 -05:00
9b94a1052f
Disabling NUMA maching for model 79 for some VM configs ( #1204 )
2024-06-06 17:15:04 -06:00
526cce9bf4
graph: restrict maxChannels to 64 for multi-node and RCCL_ENABLE_INTRANET=1
2024-06-06 10:58:41 -05:00
6ab20a7c6b
graph: fix multi-node minChannel count
2024-06-06 10:56:39 -05:00
9746d8ca3f
set MIN_NCHANNEL limit to 64 for multi-node
2024-06-03 13:05:05 -05:00
ef442f8f92
set MAXCHANNELS to 128
2024-06-03 13:05:05 -05:00
9f654f6cf5
graph: restrict MAXCHANNELS for certain platforms
2024-06-03 13:05:01 -05:00
0948eecbba
Changing channel stride for MI300X multinode ( #1196 )
...
* Shuffling MI300X multi-node channels
* Updating tree channel logic
2024-06-03 10:00:55 -06:00
354e0b29a6
Addressing possible out-of-bounds mem access during channel duplication ( #1193 )
2024-05-30 14:02:14 -06:00
73221b4230
Add ring simple chunk size tuning ( #1180 )
...
* Add ring simple chunk size tuning
* modifying the tuning table to improve the performance of broadcast for 8MB to 32MB for single-node MI300X after ring simple chunk size tuning
* modifying the tuning table to improve the performance of reduce for 1MB to 4MB for single-node MI300X after ring simple chunk size tuning
---------
Co-authored-by: PedramAlizadeh <pmohamma@amd.com >
2024-05-29 07:59:47 -07:00
73acf3eeec
modifying the tuning table to improve the performance of broadcast for 1MB to 64MB for single-node MI300X ( #1172 )
2024-05-08 15:49:33 -04:00
408278209d
Adding ASAN changes to address memory leak issue" ( #1170 )
...
Co-authored-by: akolliasAMD <akollias@amd.com >
2024-05-08 09:16:00 -05:00
b18784d8b8
Add compiler warning for uninitialized variable and fix ( #1163 )
...
* Add compiler warning for uninitialized variable and fix
* Add -Wsometimes-uninitialized
* Convert warning to error
2024-05-08 07:00:25 -07:00
f679db6ff6
Use normal permute path when one NIC per GPU ( #1171 )
2024-05-08 06:59:57 -07:00
b513c3970a
Bypass NVIDIA Ampere related tuning ( #1165 )
2024-05-03 17:57:16 -07:00
bb58b1c258
Fix ignore NUMA not being observed for NICs during model matching ( #1164 )
2024-05-03 16:42:07 -07:00
9e0c9b4ed8
Replace __HIP_PLATFORM_HCC__ with __HIP_PLATFORM_AMD__ ( #1154 )
2024-04-25 07:19:18 -07:00
e1a835910e
Merge remote-tracking branch 'nccl/master' into develop
2024-04-23 13:34:00 -07:00
220066197a
Use hipExtMallocWithFlags to allocate host memory on APU ( #1149 )
...
Also use SM60 as CUDA compatibility level.
2024-04-17 16:56:38 -07:00
4cb62f999a
Rail optimization for rings ( #1140 )
...
- Modifies the ring creation algorithm to be friendlier to rail-optimized topologies (should not affect classic fabric topologies)
2024-04-15 12:03:57 -06:00
df98a6957d
Add another Rome model ( #1095 )
2024-02-28 10:46:05 -08:00
48bb7fec79
2.20.5-1
...
Fix UDS connection failure when using ncclCommSplit. Issue #1185
2024-02-26 02:52:39 -08:00
74f9e5db64
Add new GPU model ( #1080 )
2024-02-23 12:19:42 -08:00
2fb12a9358
Merge pull request #1079 from BertanDogancay/2.19.4-sync
...
2.19.4 Sync
2024-02-16 09:50:11 -07:00
bac57421c7
Allow bus id to be null ( #1085 )
...
* Allow bus id to be null
2024-02-15 16:36:51 -07:00
b6475625fb
2.20.3-1
...
Add support for alternating rings, allow for cross-nic rings without
cross-rail communication.
Add support for user buffer registration for network send/recv.
Optimize aggregated operations to better utilize all channels.
Add flattening for BCM PCI gen5 switches.
Add support for inter-node NVLink communication
Add support for port fusion in NET/IB.
Add support for ReduceScatter and AllGather using Collnet.
Update net API to v8.
Fix hang during A2A connection.
2024-02-13 04:22:38 -08:00
d999d9ad21
Merge remote-tracking branch 'rccl/develop' into 2.19.4
2024-02-09 11:31:03 -06:00
5669b0d7b6
2.18.5 fix ( #1077 )
...
* Revert "Revert "2.18.5-1""
This reverts commit 767fde8210
2024-02-09 09:18:38 -08:00
704c9ef0d1
Doubling P2P channels per peer on single node gfx94x only ( #1074 )
2024-02-07 14:05:57 -08:00
1d989f6524
Doubling P2P channels per peer on single node only ( #1069 )
2024-02-02 12:41:00 -08:00
1a134b283b
Merge remote-tracking branch 'rccl/develop' into 2.19.4
2024-01-31 11:53:10 -06:00
9ff53eeeae
Merge remote-tracking branch 'nccl/master' into develop
2024-01-30 14:43:43 -08:00
ccfb35fa6d
modifying the tuning table to improve the performance of allreduce for 8MB and 16MB for single-node MI300X ( #1063 )
2024-01-26 09:05:53 -05:00
ffde530af5
Increase P2P channels per peer ( #1060 )
2024-01-25 11:21:58 -08:00
81ddf9de89
Merge remote-tracking branch 'nccl/v2.19' into develop
2024-01-24 15:25:33 -08:00
3325f96c56
Only use full MAXCHANNELS for gfx94x ( #1050 )
2024-01-17 09:00:49 -08:00
b08124c85d
adding rccl tuning parameters for MI300X gfx942 with 8 GPUs single and multi-node ( #1047 )
2024-01-16 13:44:32 -05:00
261707d90a
Add option to force enable network transport on single node ( #1046 )
2024-01-16 07:54:18 -08:00
767fde8210
Revert "2.18.5-1"
...
This reverts commit 559b70f86c
2024-01-12 16:54:19 +00:00
735178c1fe
cover more gpu/nic mapping cases ( #1037 )
2024-01-10 08:01:37 -08:00
0d515f9388
resolved conflicts, fixed the localNetCount/0 bug
2023-12-18 08:11:34 +00:00
655742a3a6
Fully disable MSCCL when machine is not matched ( #1017 )
...
* Disable MSCCL algorithm meta loading when machine is not matched
* fully disable init
* fix potential segfault
2023-12-13 08:36:21 -08:00
53d807a5b9
msccl: disable on multi-node ( #1018 )
2023-12-13 07:41:40 -08:00
4ba65d1d6a
Increase max channles to 64 ( #993 )
2023-12-01 16:01:11 -08:00
0b53f79196
Fix GPU to NIC mapping for peertopeer ( #994 )
2023-12-01 08:00:17 -08:00
4bb0b4a380
Move MSCCL algorithm loading to initialization to workaround HIP graph conflict ( #982 )
...
* MSCCL: pre-specify channels and pre-load algorithms
* add mutex
* fix bug
* clean include
* disable all-gathers temporarily
2023-11-30 09:47:20 -08:00
Nusrat Islam
Wenkai Du
saurabhAMD
gilbertlee-amd
Pedram Alizadeh
mberenjk
BertanDogancay
Sylvain Jeaugey
Bertan Dogancay
akolliasAMD
Wenkai Du
Hossein Pourreza
Ziyue Yang
pradeep-ramanna