Граф коммитов
280 Коммитов
Автор SHA1 Сообщение Дата
qiwei_ji 2b9394f08a Check nvlink_node instead of xgmi_node in xml.cc (#1407)
It seems like here wants to check xgmi_node instead. If checks node for "nvlink", it will verify the link_info everytime.
If checks node for "xgmi", when get yes answer, it won't need check vsmi topo interface.

[ROCm/rccl commit: f2ee8d9132]
2025-01-06 17:09:27 -08:00
Hujingboиhujingbo bea86f6248 increase p2p channels for Intel platform (#1448)
Co-authored-by: hujingbo <hujingbo@kuaishou.com>

[ROCm/rccl commit: ad4c36dc34]
2024-12-10 07:33:37 -08:00
Benjamin Kitor fe806d5427 Add Topologies for 16-GPU gfx942 SuperNode (#1417)
* Add Topologies for 16-GPU gfx942 SuperNode

- Add GigaIO topologies to tools/topo_expl for dev and testing
- Add GigaIO Columba 16 GPU romeModel and adjust topology
  matching algorithm in rome_models for 16 GPU system
- Fix bug which failed to match Rome Model when using subsets
  of system resources (i.e. ROCR_VISIBLE_DEVICES is set)
- Fixes for topo_expl

* Fix bug w/ 1H16P

[ROCm/rccl commit: a05329bd0d]
2024-12-03 13:12:03 -08:00
gilbertlee-amd be0865f335 Adding RCCL_MODEL_REVERSAL_DISABLE env var to disable model reversal (#1431)
* Adding RCCL_MODEL_REVERSAL_DISABLE env var to disable model reversal

[ROCm/rccl commit: 000575867c]
2024-11-25 11:24:54 -07:00
corey-derochie-amd ad1384bea1 Hide or fix all build warnings (#1331)
* Changing C-strings to be const.

* Changed variable-length arrays to std::vector to avoid warnings. VLA is a compiler extension.

* Changed `#define` inside functions into `constexpr int` to preserve scoping and avoid macro redefinition warnings.

* Disabled warnings for modifying `CMAKE_CXX_FLAGS` caused by `check_symbol_exists`, which temporarily modifies the flag to do a compile check.

* Fixed VLA in rccl UT.

[ROCm/rccl commit: 1c45962273]
2024-11-04 09:46:42 -07:00
Avinash da3887bafb Memory leak fixes in hostside functions (#1388)
memory leak fixes for parseRome4P2H and ncclTopoAddGPU

[ROCm/rccl commit: d6006f0425]
2024-10-30 14:25:56 -05:00
gilbertlee-amd 02bf3a3bf8 Adding support for odd nodes for model_87 (#1309)
[ROCm/rccl commit: 0cbce2a757]
2024-10-24 08:38:12 -06:00
Arm Patinyasakdikul 928414ac06 Increased maximum number of XML nodes to support CPX mode. (#1386)
[ROCm/rccl commit: 29f87c7191]
2024-10-23 11:15:11 -05:00
Wenkai Du bd0cdf5a50 Add back missing net flush (#1376)
[ROCm/rccl commit: c8d3543d3f]
2024-10-15 08:12:26 -07:00
Wenkai Du 4cd1b3a9f5 Improve model matching for GPUs with alltoall XGMI connection (#1372)
[ROCm/rccl commit: 5c367a21d0]
2024-10-11 09:53:14 -07:00
Wenkai Du 1b988c1b31 Fix crash when PXN is enabled on some platforms (#1369)
[ROCm/rccl commit: b55b6be0cb]
2024-10-11 09:02:59 -07:00
corey-derochie-amd 35d98330f2 Only set minNchannels if we are actually using MSCCL, checked using comm->mscclCompatible. (#1337)
[ROCm/rccl commit: c11f6b1531]
2024-10-08 10:20:55 -06:00
BertanDogancay 9059445acb Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: 84081064a0]
2024-10-02 09:31:25 -05:00
Wenkai Du 74aa13afbe Add another Rome model (#1354)
[ROCm/rccl commit: e453f1ced9]
2024-10-01 17:41:27 -05:00
Nusrat Islam 7676d49260 graph: fix for MI300X 64 GPU case (#1308)
PR #1290 introduced a failure for 64 GPU case on MI300X. This PR
fixes the failure.

[ROCm/rccl commit: 833435be18]
2024-08-26 18:37:58 -05:00
Wenkai Du 157cc5f6ba Add new Rome model (#1304)
* Add another rome model and override

* Fix bug

* Fix typo

* Add ring

* Update ring

* Fix model matching

* Clean up

* Clean up

* Reverse rings for NCCL_RINGS input

* Only reverse NCCL_RINGS for ring graph

* Fix mapping issue when using  NCCL_RINGS

* Add NCCL_RINGS_REMAP to handle inconsistant net names

[ROCm/rccl commit: 532b70afb6]
2024-08-23 08:45:43 +08:00
Wenkai Du d5a33e7107 Fix gfx940 CPX mode (#1290)
[ROCm/rccl commit: d3171b51b7]
2024-08-16 08:46:06 +08:00
Wenkai Du e770054b7f Fix model matching with PXN enable (#1295)
[ROCm/rccl commit: eff56735b0]
2024-08-16 06:16:00 +08:00
akolliasAMD 38e189bb1e removed hcc mentions (#1291)
[ROCm/rccl commit: d6c317d6ae]
2024-08-14 15:04:13 -06:00
Pedram Alizadeh e40a0939f1 adding new tunning table for very large number of nodes (#1288)
[ROCm/rccl commit: a25ca9bb90]
2024-08-09 10:47:42 -04:00
akolliasAMD 37c44d531b gfx12 Disable ll protocol (#1268)
[ROCm/rccl commit: c246e25f8e]
2024-07-26 08:59:55 -06:00
Nusrat Islam df63c9772f Enable CPX mode for MI300X (#1259)
* graph: enable cpx mode for MI300X

* graph: tune limits for cpx and cleanup

[ROCm/rccl commit: 6f331b0d43]
2024-07-19 11:30:37 -05:00
Wenkai Duиgilbertlee-amd 54e4899607 Template unroll for RCCL kernels (#1250)
* Template unroll for RCCL kernels

* Adding unroll template arg during CMake hipification

* Reduce linking parallel jobs to avoid OOM in CI

* Workaround issues with UT tests

SWDEV-469533: register spill fix is needed for mainline build
LWPCOMMLIBS-369: cannot enable 112 channels with 80 CUs
Use -parallel-jobs=8 for linking

* CI: do not use -j 16 when building

* CI: use -j 8 when building

* Only reduce parallel linking job for CI extended

* Restore original jenkins command. Change parallel linking jobs in cmake

* Disable MSCCLPP

---------

Co-authored-by: gilbertlee-amd <gilbert.lee@amd.com>

[ROCm/rccl commit: 89349f2ce4]
2024-07-19 08:15:59 -07:00
Nilesh M Negi 73e17b3e70 Consistent channel shuffling for MI300X multi-node (#1255)
* Revert "[GRAPH] Use channel shuffling only for IB systems (#1228)"

This reverts commit 8a7dd0e590.

* Revert "Revert "Changing channel stride for MI300X multinode (#1196)" (#1224)"

This reverts commit bb6fab3d8e.

[ROCm/rccl commit: a1ef217b32]
2024-07-18 10:18:09 -05:00
Nilesh M Negi 13134c6c64 [GRAPH] Disable MSCCL override of no. of channels (#1187)
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 67e867271f]
2024-07-15 10:45:21 -05:00
Nilesh M Negi 8a7dd0e590 [GRAPH] Use channel shuffling only for IB systems (#1228)
* [GRAPH] Use channel shuffling only for IB systems

Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

* [GRAPH] Define channels=48 for gfx94 RoCE systems

Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

* [GRAPH] Increase channels for RoCE gfx94 systems

Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

---------

Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 5be3b713ef]
2024-07-02 12:20:40 -05:00
Nusrat Islam c24b1ff14f graph: fix minNchannels for multi-node overwrite (#1230)
[ROCm/rccl commit: b09ea29d66]
2024-06-26 16:56:10 -05:00
Wenkai Du bb6fab3d8e Revert "Changing channel stride for MI300X multinode (#1196)" (#1224)
This reverts channel stride change in commit
a009e43f3c

[ROCm/rccl commit: ad31d93f3d]
2024-06-25 14:03:30 -07:00
saurabhAMD de7ea612d7 Unit Tests for testing channels (#1222)
[ROCm/rccl commit: e170f41ddd]
2024-06-25 10:10:10 -05:00
Nusrat Islam bf164da244 graph: fix minNchannels for multi-node
Multi-node rccl was not correctly setting the minNchannels value. This
PR fixes the bug.


[ROCm/rccl commit: 05df0f8cea]
2024-06-24 16:42:44 -05:00
Nusrat Islam 7c029672a8 Merge pull request #1200 from nusislam/multi-node-256-fix
graph: fix multi-node channel count

[ROCm/rccl commit: 9660e2e2dc]
2024-06-07 14:34:20 -05:00
gilbertlee-amd efe851cfff Disabling NUMA maching for model 79 for some VM configs (#1204)
[ROCm/rccl commit: 9b94a1052f]
2024-06-06 17:15:04 -06:00
Nusrat Islam 3cbf67715f graph: restrict maxChannels to 64 for multi-node and RCCL_ENABLE_INTRANET=1
[ROCm/rccl commit: 526cce9bf4]
2024-06-06 10:58:41 -05:00
Nusrat Islam bf35178250 graph: fix multi-node minChannel count
[ROCm/rccl commit: 6ab20a7c6b]
2024-06-06 10:56:39 -05:00
Nusrat Islam 62cbb3bcdd set MIN_NCHANNEL limit to 64 for multi-node
[ROCm/rccl commit: 9746d8ca3f]
2024-06-03 13:05:05 -05:00
Nusrat Islam 48821ad0d7 set MAXCHANNELS to 128
[ROCm/rccl commit: ef442f8f92]
2024-06-03 13:05:05 -05:00
Nusrat Islam 2447161fb4 graph: restrict MAXCHANNELS for certain platforms
[ROCm/rccl commit: 9f654f6cf5]
2024-06-03 13:05:01 -05:00
gilbertlee-amd a009e43f3c Changing channel stride for MI300X multinode (#1196)
* Shuffling MI300X multi-node channels
* Updating tree channel logic

[ROCm/rccl commit: 0948eecbba]
2024-06-03 10:00:55 -06:00
gilbertlee-amd b35e2b8c4b Addressing possible out-of-bounds mem access during channel duplication (#1193)
[ROCm/rccl commit: 354e0b29a6]
2024-05-30 14:02:14 -06:00
Wenkai DuиPedramAlizadeh a724f1ebb7 Add ring simple chunk size tuning (#1180)
* Add ring simple chunk size tuning

* modifying the tuning table to improve the performance of broadcast for 8MB to 32MB for single-node MI300X after ring simple chunk size tuning

* modifying the tuning table to improve the performance of reduce for 1MB to 4MB for single-node MI300X after ring simple chunk size tuning

---------

Co-authored-by: PedramAlizadeh <pmohamma@amd.com>

[ROCm/rccl commit: 73221b4230]
2024-05-29 07:59:47 -07:00
Pedram Alizadeh 43d875b422 modifying the tuning table to improve the performance of broadcast for 1MB to 64MB for single-node MI300X (#1172)
[ROCm/rccl commit: 73acf3eeec]
2024-05-08 15:49:33 -04:00
mberenjkиakolliasAMD 00c0f3d67f Adding ASAN changes to address memory leak issue" (#1170)
Co-authored-by: akolliasAMD <akollias@amd.com>

[ROCm/rccl commit: 408278209d]
2024-05-08 09:16:00 -05:00
Wenkai Du 63d8de3bd3 Add compiler warning for uninitialized variable and fix (#1163)
* Add compiler warning for uninitialized variable and fix

* Add -Wsometimes-uninitialized

* Convert warning to error

[ROCm/rccl commit: b18784d8b8]
2024-05-08 07:00:25 -07:00
Wenkai Du eab9978e48 Use normal permute path when one NIC per GPU (#1171)
[ROCm/rccl commit: f679db6ff6]
2024-05-08 06:59:57 -07:00
Wenkai Du c782aba364 Bypass NVIDIA Ampere related tuning (#1165)
[ROCm/rccl commit: b513c3970a]
2024-05-03 17:57:16 -07:00
Wenkai Du 7c811a7582 Fix ignore NUMA not being observed for NICs during model matching (#1164)
[ROCm/rccl commit: bb58b1c258]
2024-05-03 16:42:07 -07:00
Wenkai Du e494f29235 Replace __HIP_PLATFORM_HCC__ with __HIP_PLATFORM_AMD__ (#1154)
[ROCm/rccl commit: 9e0c9b4ed8]
2024-04-25 07:19:18 -07:00
BertanDogancay 36f9492cda Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: e1a835910e]
2024-04-23 13:34:00 -07:00
Wenkai Du 35f8d269f8 Use hipExtMallocWithFlags to allocate host memory on APU (#1149)
Also use SM60 as CUDA compatibility level.

[ROCm/rccl commit: 220066197a]
2024-04-17 16:56:38 -07:00
gilbertlee-amd 422a7ffcbb Rail optimization for rings (#1140)
- Modifies the ring creation algorithm to be friendlier to rail-optimized topologies (should not affect classic fabric topologies)

[ROCm/rccl commit: 4cb62f999a]
2024-04-15 12:03:57 -06:00