Commit Graph

922 Commits

Author SHA1 Message Date
mberenjk 1623fcc7a1 Improving build time by removing the gfx11xx and host code from rccl_float8.h (#1789)
* removing extra build time by removing the gfx11xx arch from using hip_fp8

---------

Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>

[ROCm/rccl commit: 697bee4ee8]
2025-07-09 14:03:47 -05:00
Bertan Dogancay b1470b4e50 [GRAPH] Pass rank instead of busId due to a change in an internal function signature (#1792)
[ROCm/rccl commit: 9c89573580]
2025-07-08 08:45:54 -04:00
Bertan Dogancay 471fc6bff2 [DEVICE] Enable PAT algo for RCCL 1ppn (#1756)
* Enable PAT algo for RCCL 1ppn


[ROCm/rccl commit: e96c8473a1]
2025-07-04 13:45:18 -04:00
Wenkai Du e031a4a2f4 msccl: use special send for LL on gfx950 (#1788)
[ROCm/rccl commit: ae9642d4bc]
2025-07-03 04:16:18 -05:00
ryanhankins f910d25563 Adding #include <dlfcn.h> in nccl_net.h to pass build (#1786)
[ROCm/rccl commit: 9d35581d5e]
2025-07-02 19:21:53 -05:00
Wenkai Du 6db3b4cd4f Add support for extended fine grained system memory pool (#1770)
* Add support for extended fine-grained system memory pool
* Use hipHostRegisterUncached
* Add "sc0 sc1" flags for LL store on gfx950
* Update after HIP flag is changed to hipExtHostRegisterUncached

[ROCm/rccl commit: 4640ab19b3]
2025-07-01 16:38:49 -05:00
Nilesh M Negi d88d033aba [RAS] Add support for RAS client (#1748)
Enable RAS client binary `rcclras`

[ROCm/rccl commit: 8d3a5542fb]
2025-06-29 18:53:16 -05:00
isaki001 79473681e5 added tuning table for gfx950 (#1779)
* added tuning table for mi350

* remove erroneous string

[ROCm/rccl commit: 75d22b47cb]
2025-06-29 15:45:39 -05:00
Bertan Dogancay ac5dad287a Switch to linear channel mapping for 2 nodes (#1777)
[ROCm/rccl commit: 358dc1bc84]
2025-06-28 09:10:18 -05:00
Arm Patinyasakdikul c3b110f9e9 add warning if workFIFO is not available after multiple retries. (#1772)
* add warning if workFIFO is not available after multiple retries.

* Update src/enqueue.cc

Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>

---------

Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>

[ROCm/rccl commit: 265b1b3775]
2025-06-26 19:49:52 -05:00
Arm Patinyasakdikul 32e80aedc0 Update plugin to look for librccl-net.so. (#1768)
[ROCm/rccl commit: 71c788d4d7]
2025-06-26 16:59:38 -05:00
mberenjk 2c02ee0a99 changing the HIP-VERSION to 6.3 to avoid using hip_fp8 for older ROCm versions (#1764)
Co-authored-by: Marzieh Berenjkoub <mberenjk@.amd.com>

[ROCm/rccl commit: 5fb9d8f828]
2025-06-26 11:15:01 -05:00
Mustafa Abduljabbar ffb17bd9d7 Revert LL64 cutoff points based on internal tuning (#1771)
[ROCm/rccl commit: 7e2ac00980]
2025-06-26 11:59:42 -04:00
Dingming Wu d34a38ccfc Add proxyTrace (#1732)
This feature tracks the proxy events and status of each send/recv op. ProxyTrace keeps a fixed number of active ops in host mem and dumps the status of each op when the program crashes or hangs.

[ROCm/rccl commit: 020dcf0a7c]
2025-06-25 23:01:34 -05:00
Bertan Dogancay eaa770a017 [NPKit] Create dump dir regardless of default or user provided path (#1757)
[ROCm/rccl commit: 675b495a00]
2025-06-21 21:18:20 -05:00
BertanDogancay c0c9312e38 Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: aaf023976a]
2025-06-20 07:54:49 -05:00
Sarat Kamisetty e359e834f5 generic net plugin ctxt that is extensible for use in multiple APIs (#1735)
Co-authored-by: Sarat Kamisetty <sakamiset@amd.com>

[ROCm/rccl commit: fa0422f174]
2025-06-16 14:48:08 -07:00
Bertan Dogancay 9fa53cc454 [NPKit] Use default output directory when env var is not set (#1747)
[ROCm/rccl commit: 39211c6b41]
2025-06-16 15:26:53 -04:00
Tim 7051f217a7 replayer update v0 (#1733)
* First version of new replayer, with comments on future TODOs

* plus minor fixes for UT

* Updated format of recorder, especially in binary department, according to replayer's need

[ROCm/rccl commit: ba97c9c18b]
2025-06-13 15:05:34 -04:00
Richard Barnes 2c0cc20a76 Enable -Wdeprecated-copy-with-user-provided-copy (#1643)
[ROCm/rccl commit: 4486d091b8]
2025-06-13 08:23:31 -07:00
Arm Patinyasakdikul 7f7f1cede3 Added missing copyright message. (#1742)
* Added missing copyright message.

* addressed comments.

[ROCm/rccl commit: 6c37ae9470]
2025-06-12 09:58:01 -05:00
corey-derochie-amd 2e7aa3556e Deprecated MSCCL API functions (#1740)
[ROCm/rccl commit: 03fba66e71]
2025-06-11 17:52:09 -06:00
Nusrat Islam 99813a3288 msccl: adjust msccl threshold for bf16 (#1736)
* msccl: adjust msccl threshold for bf16

* Update src/collectives.cc

Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>

---------

Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>

[ROCm/rccl commit: 75c3c8215c]
2025-06-11 09:09:57 -05:00
Arm Patinyasakdikul 69f7167b74 Fixed errorneous parenthesis. (#1739)
[ROCm/rccl commit: 600ace7f19]
2025-06-11 09:08:00 -05:00
Nilesh M Negi 4cadf3597c [DEVICE] Adding ability to choose unroll factor at runtime (#1734)
* Adding runtime unroll factor selection via RCCL_UNROLL_FACTOR
* [BUILD] Add support for user-defined UNROLL for debugging
* Update CHANGELOG.md
* Fix COLLTRACE errors in CI
* Add debug statements for unroll and resolve warnings
* Incorporate UNROLL into ONLY_FUNCS for debugging

---------

Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
Co-authored-by: gilbertlee-amd <44450918+gilbertlee-amd@users.noreply.github.com>
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

[ROCm/rccl commit: 9d72be7b2f]
2025-06-11 00:07:59 -05:00
Nilesh M Negi b797b62f6b [DEVICE] Use threadfence on gfx950 for LL protocol (#1686)
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: b926203c05]
2025-06-09 01:26:07 -05:00
Nilesh M Negi 7abc3160e7 [BUILD] Enable LL128 on gfx950 (#1731)
* [BUILD] Enable LL128 on gfx950
* Modify comment in src/rccl_wrap.cc
* Update CHANGELOG

Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>

[ROCm/rccl commit: ef5b4ff630]
2025-06-09 00:25:54 -05:00
Arm Patinyasakdikul f65777536f Remove 'warpSize' compiler constant as it is deprecated in ROCm 7.0. (#1720)
* Remove 'warpSize' compiler constant as it is deprecated in ROCm 7.0.

* Create ncclShmemScratchWarpSize on host side for enqueue.cc.

* Update src/enqueue.cc

Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>

* address comments

* fix number of threads

---------

Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>

[ROCm/rccl commit: ec6efa9b26]
2025-06-06 07:34:43 -05:00
Arm Patinyasakdikul 8dd9747504 Increase default WORK_FIFO size to accommodate larger alltoall. (#1722)
[ROCm/rccl commit: d5b5f6b159]
2025-06-05 09:02:45 -05:00
Pedram Alizadeh 1ace5d05ed Reapplying PR #1641 [AG and RS channel tuning] Add thread work threshold to tuning models and precompute reg index in LL128 (#1713)
* Reapply "[AG and RS channel tuning] Add thread work threshold to tuning models and precompute reg index in LL128 (#1641)"

This reverts commit 943ad6f7820739385a0b54e81f823d0df1dbf71c.

* Decreasing NCCL_LL128_SHMEM_ELEMS_PER_THREAD from 16 to 8

[ROCm/rccl commit: 3f7c08648f]
2025-06-04 13:22:11 -04:00
Avinash a50ff2c3d3 SPLITCOMM design fix in src/misc/msccl (#1715)
* Fix TOC-TOU in mcclInit

* Improving vector resize thread safety

* Initial commit rank to comm change

* Removing unwanted include header changes

* Updated CHANGELOG.md

* Update CHANGELOG.md

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

---------

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

[ROCm/rccl commit: e94b360246]
2025-06-01 21:00:38 -05:00
alex-breslow-amd 4277b5aa88 Use One Slice per Basic Primitive for AllReduce, ReduceScatter, AllGather (#1681) for Single Node on Some GFX9 Systems
Using a single slice rather than the typical two provides about 5% speedup (sometimes more or less) on some GFX9 systems for single node.

[ROCm/rccl commit: 2f6b20c00a]
2025-05-29 16:17:35 -07:00
Nilesh M Negi 19ed482121 Re-apply unroll=1 and 112 channels for gfx950 (#1706)
* Reapply "[SRC] Enable unroll=1 for gfx950 (#1602)" (#1667)
This reverts commit a6972c0d09.

* Reapply "[GRAPH] Increase default nChannels to 112 for gfx950 (#1596)" (#1620)
This reverts commit 1a2eca1756.

[ROCm/rccl commit: 12517a957e]
2025-05-28 14:58:10 -05:00
PedramAlizadeh a99f960742 Revert "[AG and RS channel tuning] Add thread work threshold to tuning models and precompute reg index in LL128 (#1641)"
This reverts commit 951ed9cde1.


[ROCm/rccl commit: 7f878baef0]
2025-05-21 20:21:27 -05:00
Giuseppe Congiu 285e2e41c8 NCCL 2.26.6-1
Fix profiler_v2 compatibility layer
 * Removing trafficBytes in profiler_v3 breaks casting to ncclProfilerEventDescr_v2_t
   in the compatibility layer for profiler_v2 interface. This patch fixes the issue
   by making the conversion between the two descriptors explicit.


[ROCm/rccl commit: 8171af656b]
2025-05-20 04:04:41 -07:00
corey-derochie-amd 65d67dce7a Switched to using the hip_fp8 header instead of rccl_float8, resolving compatibility issues. (#1546)
* Revert "Revert "replacing rccl_float8 with hip_fp8 and address compatibility …"

This reverts commit 30eecfdb25.

* [UT] Modify max stack size to 496

* adding a check for OCP type and replacing ROCM_VERSION with HIP_VERSION

* addressing the ci failure

* Adding the device tag

---------

Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>

[ROCm/rccl commit: 170acf3bda]
2025-05-14 15:33:03 -05:00
Mustafa Abduljabbar 951ed9cde1 [AG and RS channel tuning] Add thread work threshold to tuning models and precompute reg index in LL128 (#1641)
* Update LL128 elems per thread

* Precompute ix[g] in LL128 prim

* Make Threadthreshold part of tuning models

* Ignore channel tuning when channels are env controlled

* Tune LL128 max limit for AG

* Tune LL128 max limit for RS

* Retune AR LL128 limits due to changes

* Update CHANGELOG.md

---------

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

[ROCm/rccl commit: 00c1eb098c]
2025-05-14 14:35:54 -05:00
Dingming Wu 3731cae1b7 Detect if HSA_NO_SCRATCH_RECLAIM is set after initEnv() (#1683)
* Detect if HSA_NO_SCRATCH_RECLAIM is set after initEnv()

 For rocm older than 6.4, we need to set HSA_NO_SCRATCH_RECLAIM=1 to use LL128 protocol.
This Env is set outside of RCCL, add the logging to detect whether its set during runtime.

* check hip runtime ver via hipRuntimeGetVersion

* move the detection to ncclinit func

* correct rocm version integer

* update warning message

* avoid unnecessary info msg on hsa_no_scratch_reclaim detection

[ROCm/rccl commit: 51f87fbb43]
2025-05-14 10:12:45 -05:00
mberenjk 08c0b8b0fc moving the thread_fence to apply before atomic fetch (#1672)
* applying thread_fence only on warp 0 before atomic fetch

---------

Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>

[ROCm/rccl commit: 1cefcee51f]
2025-05-14 10:10:05 -05:00
gilbertlee-amd 6e57154001 Fix when more than 64 channels are used for multi-collective group calls (#1688)
* Fix when more than 64 channels are used for multi-collective group calls

* Update CHANGELOG.md

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

---------

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

[ROCm/rccl commit: 9ef45df8f7]
2025-05-12 18:05:57 -05:00
Avinash 6d6dd8434a RCCL Multinode DMA Buffer crash fix (#1682)
This commit handles DMABUF initialization and call appropriate handling function. This fixes crash in OS with no peermem support and relying on only DMABUF.

* Initial test commit
* Handling Dmabuf_fd opening and closing
* Cleanup
* Use DMABuff or Peermem as needed
* Using user input for ibDmaBufSupportInitOnce
* Revert all changes to rocmwrap.cc
* Revert all changes to rocmwrap.cc
* Changing to func definition braces
* Reverting line removal in utils.h
* useDmaBuf to calculate  flushEnabled

[ROCm/rccl commit: 5f6805b4f4]
2025-05-08 19:17:39 -05:00
Avinash c81ea25407 collective trace improvements for debugging (#1661)
[ROCm/rccl commit: c54a0c085a]
2025-05-07 13:37:31 -05:00
Bertan Dogancay c75ebd9147 Merge pull request #1662 from BertanDogancay/2.25
[SYNC] 2.25.1-1

[ROCm/rccl commit: 590ad6acc2]
2025-05-06 09:39:09 -04:00
Mustafa Abduljabbar ab4a3eb0c1 Fix topo explorer's compatibility with NCCL 2.24 (#1671)
* Fix build issues

* Fix failure to find path remote rank


[ROCm/rccl commit: f3f3336468]
2025-05-05 15:26:29 -04:00
Bertan Dogancay b435c75068 [Graph] Try using P2P by default (#1670)
[ROCm/rccl commit: acfac55516]
2025-05-02 11:54:30 -04:00
Nilesh M Negi a6972c0d09 Revert "[SRC] Enable unroll=1 for gfx950 (#1602)" (#1667)
* Revert "[SRC] Enable unroll=1 for gfx950 (#1602)"
This reverts commit 210f90ae0f.

* Update Changelog

---------

Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 329e13efff]
2025-04-30 23:33:08 -05:00
BertanDogancay 064062ef70 Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: cb6e23ae67]
2025-04-30 13:31:41 -05:00
Tim 33c390b36e minor fix for empty scope (group) (#1666)
[ROCm/rccl commit: dc0c5f9153]
2025-04-30 13:29:13 -04:00
BertanDogancay d045d0ca23 Merge remote-tracking branch 'nccl/master' into develop
[ROCm/rccl commit: a6bf9bfc9e]
2025-04-23 20:47:43 -07:00
Mustafa Abduljabbar 07620c7efd Expose production tuning table in topo_explorer using internal RCCL/NCCL logic (#1628)
* Internal RCCL/NCCL functionality exposed when RCCL_EXPOSE_STATIC is enabled
* Algo/protocol/max channels can be obtained with the new RCCL API
* Introduce rccl_static and rccl_static_inline macros to work around invisible functions in core source files like enqueue.cc
* Add usage example in topo-explorer tool

[ROCm/rccl commit: 82afb2bcfe]
2025-04-23 15:44:56 -04:00