AI/rocm-systems
2
0
Форкнуть 0
Код Задачи Запросы на слияние Действия Пакеты Проекты Релизы Вики Активность
1 728 коммитов 3 Ветки 0 Теги
c81ea25407eb4a79ea30e7828e253cdaaedc1ca9
Граф коммитов

878 Коммитов

Автор SHA1 Сообщение Дата
Avinash c81ea25407 collective trace improvements for debugging (#1661) 
[ROCm/rccl commit: c54a0c085a]
 2025-05-07 13:37:31 -05:00
Bertan Dogancay c75ebd9147 Merge pull request #1662 from BertanDogancay/2.25 
[SYNC] 2.25.1-1

[ROCm/rccl commit: 590ad6acc2]
 2025-05-06 09:39:09 -04:00
Mustafa Abduljabbar ab4a3eb0c1 Fix topo explorer's compatibility with NCCL 2.24 (#1671) 
* Fix build issues

* Fix failure to find path remote rank


[ROCm/rccl commit: f3f3336468]
 2025-05-05 15:26:29 -04:00
Bertan Dogancay b435c75068 [Graph] Try using P2P by default (#1670) 
[ROCm/rccl commit: acfac55516]
 2025-05-02 11:54:30 -04:00
Nilesh M Negi a6972c0d09 Revert "[SRC] Enable unroll=1 for gfx950 (#1602)" (#1667) 
* Revert "[SRC] Enable unroll=1 for gfx950 (#1602)"
This reverts commit 210f90ae0f.

* Update Changelog

---------

Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 329e13efff]
 2025-04-30 23:33:08 -05:00
BertanDogancay 064062ef70 Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: cb6e23ae67]
 2025-04-30 13:31:41 -05:00
Tim 33c390b36e minor fix for empty scope (group) (#1666) 
[ROCm/rccl commit: dc0c5f9153]
 2025-04-30 13:29:13 -04:00
BertanDogancay d045d0ca23 Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: a6bf9bfc9e]
 2025-04-23 20:47:43 -07:00
Mustafa Abduljabbar 07620c7efd Expose production tuning table in topo_explorer using internal RCCL/NCCL logic (#1628) 
* Internal RCCL/NCCL functionality exposed when RCCL_EXPOSE_STATIC is enabled
* Algo/protocol/max channels can be obtained with the new RCCL API
* Introduce rccl_static and rccl_static_inline macros to work around invisible functions in core source files like enqueue.cc
* Add usage example in topo-explorer tool

[ROCm/rccl commit: 82afb2bcfe]
 2025-04-23 15:44:56 -04:00
Bertan Dogancay aac829125c Fix NPKit for SendRecv (#1651) 
[ROCm/rccl commit: ac8ec4c08c]
 2025-04-21 12:34:47 -04:00
Tim 58ee618194 RCCL Replayer update (#1603) 
RCCL recorder w/ suggested change and UT



[ROCm/rccl commit: 9a55ff60a9]
 2025-04-19 00:21:27 -04:00
Mustafa Abduljabbar de2b66921a Address nested designator compiler warning issue (#1633) 
[ROCm/rccl commit: 52bfdf05dc]
 2025-04-18 17:09:50 -04:00
Nusrat Islam 691e98940c Fix MSCCLPP accuracy issue for allreduce7 (#1634) 
* ext-src: fix a graph-mode bug in allreduce7

* change MSCCLPP threshold to 16MB

* ext-src: change message size threshold for allreduce7

* ext-src: address review comments

[ROCm/rccl commit: f20c33effd]
 2025-04-18 08:54:32 -05:00
AbandiGa acf0bc1c6e added copyright (#1635) 
[ROCm/rccl commit: 7a84c5dbb0]
 2025-04-14 09:46:18 -05:00
Dingming Wu 63c6180130 Adding #include <dlfcn.h> in profiler.cc to pass build (#1632) 
w/o the header, experience following errors:
```
error: use of undeclared identifier 'RTLD_NOW'
error: use of undeclared identifier 'RTLD_LOCAL'
error: use of undeclared identifier 'dlerror'
error: use of undeclared identifier 'dlsym'
error: use of undeclared identifier 'dlclose'
```

[ROCm/rccl commit: 1786c0268b]
 2025-04-10 08:48:18 -07:00
Arm Patinyasakdikul f29d59aa00 Add device synchronization before destroying proxy thread. (#1631) 
This commit ensures that GPU finishes all kernel before destroying
communicator thread.

[ROCm/rccl commit: 52654e2301]
 2025-04-10 10:44:16 -05:00
Pedram Alizadeh 93ac2ea61e all_reduce LL/LL128 and Ring/Tree multi-node tuning for MI300 (#1627) 
* Enabling LL128 by default on MI300

* Add missing CUDACHECK

* Adjust BW correction factors to fix the Tree->Ring switching point

* Refactor and add ll128 AR logarithmic factor to tuning models

* Move RCCL tuning changes to a separate file 

* Use enum for tunable indexing

* Use explicit indexing in tuning models to avoid mismatch issues

* Place rcclGetSizePerRank in a function

* Remove HIP ifdef for rccl-only call

---------

Co-authored-by: Mustafa Abduljabbar <mustafa.abduljabbar@amd.com>

[ROCm/rccl commit: e40ff4f84a]
 2025-04-10 11:43:54 -04:00
Mustafa Abduljabbar 2f4cd5718e Add AllGather LL128 multi-node tuning and include LL cutoff points in tuning models (#1618) 
* Enable LL/LL128 cutoff points in tuning models

* Initializing ll/ll128 model cutoffs for MI300

* Use RCCL_LL_LIMITS_UNDEFINED

---------

Co-authored-by: PedramAlizadeh <pmohamma@amd.com>

[ROCm/rccl commit: 4be06f04d8]
 2025-04-02 16:26:23 -04:00
Nilesh M Negi 1a2eca1756 Revert "[GRAPH] Increase default nChannels to 112 for gfx950 (#1596)" (#1620) 
* Revert "[GRAPH] Increase default nChannels to 112 for gfx950 (#1596)"

This reverts commit cf17cff5b6.

* [DOC] Update Changelog

* [DOC] Update CHANGELOG

[ROCm/rccl commit: b17338d164]
 2025-03-28 17:57:06 -05:00
Bertan Dogancay b737d8c222 Merge pull request #1559 from BertanDogancay/2.23 
[SYNC] 2.23.4-1

[ROCm/rccl commit: 532f54c244]
 2025-03-28 17:06:56 -04:00
Nilesh M Negi 210f90ae0f [SRC] Enable unroll=1 for gfx950 (#1602) 
* [SRC] Enable unroll=1 for gfx950

* Fix typo from rebase in generate.py

* Support for unroll=1 and gfx90a when building for all GPU targets

---------

Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 307bc10781]
 2025-03-27 18:21:35 -05:00
BertanDogancay 8ed27fde74 Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: 0b2062c560]
 2025-03-27 12:53:04 -05:00
isaki001 f0c853438c Disable mscclpp (#1614) 
* disable mscclpp by default

[ROCm/rccl commit: 9dc23d9265]
 2025-03-25 15:21:16 -05:00
gilbertlee-amd 4f67522420 Removing the experimental clique kernel files (#1610) 
[ROCm/rccl commit: 626dc50ab5]
 2025-03-20 18:10:01 -06:00
Wenkai Du e86b217182 Add fault injection of starting warps with random variations (#1593) 
* Add fault injection of starting warps with random variations

This is done by inserting randomly delays after __syncthreads().
The feature can be turned off by FAULT_INJECTION=OFF in cmake.

* Remove manually introduced bug for demo purpose

* Use only one thread per warp for checking wall clock

[ROCm/rccl commit: 90ad586d94]
 2025-03-20 16:11:43 -07:00
corey-derochie-amd e95578ef4c removed gfx940 and gfx941 (#1606) 
* removed gfx940 and gfx941

* removed gfx940 and gfx941

* Update "gfx94" to "gfx942" in init.cc

* Updated remaining "gfx94" updates to "gfx942"

* Update filenames and variables from gfx940 to gfx942

---------

Co-authored-by: akolliasAMD <akollias@amd.com>

[ROCm/rccl commit: 6505639cf4]
 2025-03-20 09:34:53 -06:00
Wenkai Du c6f4c8d17a GDRCOPY support: Off by default (#1605) 
[ROCm/rccl commit: bd0092e8f1]
 2025-03-18 08:17:01 -07:00
Avinash 5a25b110af Memory leak fix when numIBDevices = 0 (#1429) 
* Initial commit for testing

* Fix memory leak in checkOptions

* Fix memory leak in checkOption

* x

* Delete cmake-3.28.2-linux-x86_64.sh

* gcn changes

* gcn memleak fixes

* gcn leak fix

* memory leak fixes for parseRome4P2H and ncclTopoAddGPU

* Keeping only necessary file for fixes

Deleting temporary scripts I created for debugging and testing

* changing to GCN_ARCH_NAME_LEN

* Added sanity check directory

* refactoring scripts

* Updated to sanity checks folder

* Initial fixes

* changes in tools

* pointing RCCL lib build to debug version

* Removed second pthread_detach

* Removing sanity checks

* Keeping only code changes

* addressing memory leaks in ncclIbinit

---------

Co-authored-by: Chao Chen <cchen104@amd.com>

[ROCm/rccl commit: ccb0820743]
 2025-03-17 11:21:19 -05:00
Mustafa Abduljabbar 74d6537141 Multi-node reduce_scatter improved auto-selection for LL and LL128 on gfx942 (#1604) 
* Add reduce_scatter LL and LL128 thresholds

* Always honor user choice for protocol

[ROCm/rccl commit: f67b2cc908]
 2025-03-17 11:21:01 -04:00
Wenkai Du 0bd40f5a87 Enable LL128 on gfx942 (#1549) 
[ROCm/rccl commit: 245c2de909]
 2025-03-16 15:10:05 -07:00
Nilesh M Negi cf17cff5b6 [GRAPH] Increase default nChannels to 112 for gfx950 (#1596) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 1df73e209e]
 2025-03-14 14:47:03 -07:00
Wenkai Du afd04a5117 Limit P2P channels per peer to not exceeding max channels (#1594) 
* Limit P2P channels per peer to not exceeding max channels

* [UT] test single GPU cases for all collectives

* [UT] fix out of range root value

[ROCm/rccl commit: 4237caad69]
 2025-03-11 09:32:09 -07:00
Tim a79fa36b77 mscclpp compatibility check for ubr (#1573) 
[ROCm/rccl commit: f6c6d451a9]
 2025-03-09 22:10:47 -04:00
Nusrat Islam e7c90e0a46 misc/msccl: force use of mscclpp (#1581) 
[ROCm/rccl commit: ac823818aa]
 2025-03-04 12:48:59 -06:00
Bertan Dogancay d1247bbf2a [Transport] Fix IntraNet (#1582) 
[ROCm/rccl commit: d88cca3098]
 2025-03-04 13:30:36 -05:00
Wenkai Du 086fa823db NPKit: enable reduce scatter profiling (#1580) 
[ROCm/rccl commit: f957c4fe22]
 2025-03-04 10:03:56 -08:00
Nusrat Islam f70f406463 misc/msccl: Read graph capture status for every collective call (#1576) 
* misc/msccl: read graphCaptureStatus for every collective call

* fix a bug in checking whether UBR is enabled in MSCCLPP

* cmake: Fix patch reversal order

* misc/msccl: add logging

[ROCm/rccl commit: 23c0b7bd84]
 2025-02-28 17:16:07 -06:00
Wenkai Du 3be905ca83 Improve RDMA flushing by write dummy payload with RO=0 (#1570) 
* Improve RDMA flushing by write dummy payload with RO=0

* Rename env var for disabling this change to RCCL_GDR_FLUSH_GPU_MEM_NO_RELAXED_ORDERING

[ROCm/rccl commit: 60c1264d27]
 2025-02-27 16:20:32 -08:00
Nilesh M Negi a3f44b4a10 [BUILD] Fix generate.py for gfx950 (#1575) 
* [BUILD] Fix generate.py for gfx950

Signed-off-by: nilenegi <Nilesh.Negi@amd.com>

* [BUILD] Cleaner way to check for gfx targets

Signed-off-by: nilenegi <Nilesh.Negi@amd.com>

---------

Signed-off-by: nilenegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 02b8e504f0]
 2025-02-26 18:43:37 -06:00
Bertan Dogancay a9d09c6551 Use bit reversal based mapping for multi-node (#1572) 
[ROCm/rccl commit: 85eb1f16bc]
 2025-02-26 09:48:03 -05:00
Pedram Alizadeh acf5822a6c enable building rccl for gfx950 (#1571) 
[ROCm/rccl commit: f268553ee4]
 2025-02-25 16:13:48 -05:00
gilbertlee-amd 4ca7e6873e Rail optimized trees (#1540) 
* Allow disabling rail-optimized trees via RCCL_DISABLE_RAIL_TREES, Graphviz-friendly output via RCCL_OUTPUT_TREES


[ROCm/rccl commit: ddc5d58b93]
 2025-02-20 15:18:29 -07:00
akolliasAMD ae0b3c19d2 reverted the syncLDS back to syncthreads (#1554) 
[ROCm/rccl commit: aedbc95735]
 2025-02-19 10:44:32 -07:00
Wenkai Du 7eff149ceb Insert barrier after loading work items to LDS (#1551) 
[ROCm/rccl commit: baaa2ac64d]
 2025-02-18 10:17:27 -08:00
Wenkai Du 35987b9170 Enable GDRCopy only on gfx94x (#1550) 
* Enable GDRCopy only on gfx94x

* Use cudaFree instead of hipFree

* Add warning if failed to get device property

* Remove extra return

[ROCm/rccl commit: 32dc7ef47c]
 2025-02-17 13:28:19 -08:00
Pedram Alizadeh 37f46f1669 reverting the (Reduce NPKit latency overhead in MSCCL kernel) PR #893 (#1525) 
[ROCm/rccl commit: 0e5f4d0662]
 2025-02-14 11:03:43 -05:00
corey-derochie-amd 30eecfdb25 Revert "replacing rccl_float8 with hip_fp8 and address compatibility issue (#…" (#1545) 
This reverts commit 9463a79dd9.

[ROCm/rccl commit: 824b81c034]
 2025-02-13 10:00:22 -07:00
mberenjk 9463a79dd9 replacing rccl_float8 with hip_fp8 and address compatibility issue (#1538) 
* replacing rccl_float8 with hip_fp8 and address compatibility issue with gfx942
---------

Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>

[ROCm/rccl commit: d437d6e41c]
 2025-02-13 10:34:17 -06:00
Wenkai Du 170cc1afde Print KL/CL/KE events for all warps (#1544) 
* Print KL/CL/KE events for all warps

* Fix count off-by-one issue

* Fix opCount in KE and restore CPU thread option

* Simplify count calculation

[ROCm/rccl commit: ebf7e2305e]
 2025-02-12 13:36:31 -08:00
Wenkai Du 7fdbcdfdec Move collective trace to HBM and fix log issue (#1542) 
[ROCm/rccl commit: f5b15f27a9]
 2025-02-11 11:40:14 -08:00