7158adb57f
[GEN] Fix typo in IFC code gen ( #1796 )
2025-07-11 09:19:39 -04:00
6b4ad0fd74
[BUILD] Use fmt-header instead of libfmt ( #1791 )
2025-07-10 17:19:53 -05:00
2c099fe29a
[INIT] Fix fallback for unsupported user-specified runtime unroll factor ( #1780 )
...
* [INIT] Fix fallback for unsupported user-specified runtime unroll factor
* Add CollTrace guard
* Move `commSetUnrollFactor()` to rccl_wrap.cc
* Modify comments in the device-code generator script
2025-07-10 10:56:18 -05:00
68d6f99e0f
[DEVICE] Fix validation errors for multi-node LL with gfx950 non-coherent system memory ( #1795 )
2025-07-10 09:05:46 -05:00
058264b3f3
Fix AllReduce regression due to previous max range increase for LL64/LL128 ( #1787 )
...
* Adjust tuning factor impacting more than 2 nodes
* Scale max LL128 size for > 2 nodes
* Retune max LL128 range for N > 2
2025-07-09 19:17:10 -05:00
697bee4ee8
Improving build time by removing the gfx11xx and host code from rccl_float8.h ( #1789 )
...
* removing extra build time by removing the gfx11xx arch from using hip_fp8
---------
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com >
2025-07-09 14:03:47 -05:00
9c89573580
[GRAPH] Pass rank instead of busId due to a change in an internal function signature ( #1792 )
2025-07-08 08:45:54 -04:00
e96c8473a1
[DEVICE] Enable PAT algo for RCCL 1ppn ( #1756 )
...
* Enable PAT algo for RCCL 1ppn
2025-07-04 13:45:18 -04:00
ae9642d4bc
msccl: use special send for LL on gfx950 ( #1788 )
2025-07-03 04:16:18 -05:00
9d35581d5e
Adding #include <dlfcn.h> in nccl_net.h to pass build ( #1786 )
2025-07-02 19:21:53 -05:00
4640ab19b3
Add support for extended fine grained system memory pool ( #1770 )
...
* Add support for extended fine-grained system memory pool
* Use hipHostRegisterUncached
* Add "sc0 sc1" flags for LL store on gfx950
* Update after HIP flag is changed to hipExtHostRegisterUncached
2025-07-01 16:38:49 -05:00
8d3a5542fb
[RAS] Add support for RAS client ( #1748 )
...
Enable RAS client binary `rcclras`
2025-06-29 18:53:16 -05:00
75d22b47cb
added tuning table for gfx950 ( #1779 )
...
* added tuning table for mi350
* remove erroneous string
2025-06-29 15:45:39 -05:00
358dc1bc84
Switch to linear channel mapping for 2 nodes ( #1777 )
2025-06-28 09:10:18 -05:00
265b1b3775
add warning if workFIFO is not available after multiple retries. ( #1772 )
...
* add warning if workFIFO is not available after multiple retries.
* Update src/enqueue.cc
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com >
---------
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com >
2025-06-26 19:49:52 -05:00
71c788d4d7
Update plugin to look for librccl-net.so. ( #1768 )
2025-06-26 16:59:38 -05:00
5fb9d8f828
changing the HIP-VERSION to 6.3 to avoid using hip_fp8 for older ROCm versions ( #1764 )
...
Co-authored-by: Marzieh Berenjkoub <mberenjk@.amd.com>
2025-06-26 11:15:01 -05:00
7e2ac00980
Revert LL64 cutoff points based on internal tuning ( #1771 )
2025-06-26 11:59:42 -04:00
020dcf0a7c
Add proxyTrace ( #1732 )
...
This feature tracks the proxy events and status of each send/recv op. ProxyTrace keeps a fixed number of active ops in host mem and dumps the status of each op when the program crashes or hangs.
2025-06-25 23:01:34 -05:00
675b495a00
[NPKit] Create dump dir regardless of default or user provided path ( #1757 )
2025-06-21 21:18:20 -05:00
aaf023976a
Merge remote-tracking branch 'nccl/master' into develop
2025-06-20 07:54:49 -05:00
fa0422f174
generic net plugin ctxt that is extensible for use in multiple APIs ( #1735 )
...
Co-authored-by: Sarat Kamisetty <sakamiset@amd.com >
2025-06-16 14:48:08 -07:00
39211c6b41
[NPKit] Use default output directory when env var is not set ( #1747 )
2025-06-16 15:26:53 -04:00
ba97c9c18b
replayer update v0 ( #1733 )
...
* First version of new replayer, with comments on future TODOs
* plus minor fixes for UT
* Updated format of recorder, especially in binary department, according to replayer's need
2025-06-13 15:05:34 -04:00
4486d091b8
Enable -Wdeprecated-copy-with-user-provided-copy ( #1643 )
2025-06-13 08:23:31 -07:00
6c37ae9470
Added missing copyright message. ( #1742 )
...
* Added missing copyright message.
* addressed comments.
2025-06-12 09:58:01 -05:00
03fba66e71
Deprecated MSCCL API functions ( #1740 )
2025-06-11 17:52:09 -06:00
75c3c8215c
msccl: adjust msccl threshold for bf16 ( #1736 )
...
* msccl: adjust msccl threshold for bf16
* Update src/collectives.cc
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com >
---------
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com >
2025-06-11 09:09:57 -05:00
600ace7f19
Fixed errorneous parenthesis. ( #1739 )
2025-06-11 09:08:00 -05:00
9d72be7b2f
[DEVICE] Adding ability to choose unroll factor at runtime ( #1734 )
...
* Adding runtime unroll factor selection via RCCL_UNROLL_FACTOR
* [BUILD] Add support for user-defined UNROLL for debugging
* Update CHANGELOG.md
* Fix COLLTRACE errors in CI
* Add debug statements for unroll and resolve warnings
* Incorporate UNROLL into ONLY_FUNCS for debugging
---------
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
Co-authored-by: gilbertlee-amd <44450918+gilbertlee-amd@users.noreply.github.com >
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com >
2025-06-11 00:07:59 -05:00
b926203c05
[DEVICE] Use threadfence on gfx950 for LL protocol ( #1686 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2025-06-09 01:26:07 -05:00
ef5b4ff630
[BUILD] Enable LL128 on gfx950 ( #1731 )
...
* [BUILD] Enable LL128 on gfx950
* Modify comment in src/rccl_wrap.cc
* Update CHANGELOG
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com >
2025-06-09 00:25:54 -05:00
ec6efa9b26
Remove 'warpSize' compiler constant as it is deprecated in ROCm 7.0. ( #1720 )
...
* Remove 'warpSize' compiler constant as it is deprecated in ROCm 7.0.
* Create ncclShmemScratchWarpSize on host side for enqueue.cc.
* Update src/enqueue.cc
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com >
* address comments
* fix number of threads
---------
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com >
2025-06-06 07:34:43 -05:00
d5b5f6b159
Increase default WORK_FIFO size to accommodate larger alltoall. ( #1722 )
2025-06-05 09:02:45 -05:00
3f7c08648f
Reapplying PR #1641 [AG and RS channel tuning] Add thread work threshold to tuning models and precompute reg index in LL128 ( #1713 )
...
* Reapply "[AG and RS channel tuning] Add thread work threshold to tuning models and precompute reg index in LL128 (#1641 )"
This reverts commit 943ad6f7820739385a0b54e81f823d0df1dbf71c.
* Decreasing NCCL_LL128_SHMEM_ELEMS_PER_THREAD from 16 to 8
2025-06-04 13:22:11 -04:00
e94b360246
SPLITCOMM design fix in src/misc/msccl ( #1715 )
...
* Fix TOC-TOU in mcclInit
* Improving vector resize thread safety
* Initial commit rank to comm change
* Removing unwanted include header changes
* Updated CHANGELOG.md
* Update CHANGELOG.md
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com >
---------
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com >
2025-06-01 21:00:38 -05:00
2f6b20c00a
Use One Slice per Basic Primitive for AllReduce, ReduceScatter, AllGather ( #1681 ) for Single Node on Some GFX9 Systems
...
Using a single slice rather than the typical two provides about 5% speedup (sometimes more or less) on some GFX9 systems for single node.
2025-05-29 16:17:35 -07:00
12517a957e
Re-apply unroll=1 and 112 channels for gfx950 ( #1706 )
...
* Reapply "[SRC] Enable unroll=1 for gfx950 (#1602 )" (#1667 )
This reverts commit 329e13efff#1596 )" (#1620 )
This reverts commit b17338d164
2025-05-28 14:58:10 -05:00
7f878baef0
Revert "[AG and RS channel tuning] Add thread work threshold to tuning models and precompute reg index in LL128 ( #1641 )"
...
This reverts commit 00c1eb098c
2025-05-21 20:21:27 -05:00
8171af656b
NCCL 2.26.6-1
...
Fix profiler_v2 compatibility layer
* Removing trafficBytes in profiler_v3 breaks casting to ncclProfilerEventDescr_v2_t
in the compatibility layer for profiler_v2 interface. This patch fixes the issue
by making the conversion between the two descriptors explicit.
2025-05-20 04:04:41 -07:00
170acf3bda
Switched to using the hip_fp8 header instead of rccl_float8, resolving compatibility issues. ( #1546 )
...
* Revert "Revert "replacing rccl_float8 with hip_fp8 and address compatibility …"
This reverts commit 824b81c034mberenjk@amd.com >
2025-05-14 15:33:03 -05:00
00c1eb098c
[AG and RS channel tuning] Add thread work threshold to tuning models and precompute reg index in LL128 ( #1641 )
...
* Update LL128 elems per thread
* Precompute ix[g] in LL128 prim
* Make Threadthreshold part of tuning models
* Ignore channel tuning when channels are env controlled
* Tune LL128 max limit for AG
* Tune LL128 max limit for RS
* Retune AR LL128 limits due to changes
* Update CHANGELOG.md
---------
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com >
2025-05-14 14:35:54 -05:00
51f87fbb43
Detect if HSA_NO_SCRATCH_RECLAIM is set after initEnv() ( #1683 )
...
* Detect if HSA_NO_SCRATCH_RECLAIM is set after initEnv()
For rocm older than 6.4, we need to set HSA_NO_SCRATCH_RECLAIM=1 to use LL128 protocol.
This Env is set outside of RCCL, add the logging to detect whether its set during runtime.
* check hip runtime ver via hipRuntimeGetVersion
* move the detection to ncclinit func
* correct rocm version integer
* update warning message
* avoid unnecessary info msg on hsa_no_scratch_reclaim detection
2025-05-14 10:12:45 -05:00
1cefcee51f
moving the thread_fence to apply before atomic fetch ( #1672 )
...
* applying thread_fence only on warp 0 before atomic fetch
---------
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com >
2025-05-14 10:10:05 -05:00
9ef45df8f7
Fix when more than 64 channels are used for multi-collective group calls ( #1688 )
...
* Fix when more than 64 channels are used for multi-collective group calls
* Update CHANGELOG.md
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com >
---------
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com >
2025-05-12 18:05:57 -05:00
5f6805b4f4
RCCL Multinode DMA Buffer crash fix ( #1682 )
...
This commit handles DMABUF initialization and call appropriate handling function. This fixes crash in OS with no peermem support and relying on only DMABUF.
* Initial test commit
* Handling Dmabuf_fd opening and closing
* Cleanup
* Use DMABuff or Peermem as needed
* Using user input for ibDmaBufSupportInitOnce
* Revert all changes to rocmwrap.cc
* Revert all changes to rocmwrap.cc
* Changing to func definition braces
* Reverting line removal in utils.h
* useDmaBuf to calculate flushEnabled
2025-05-08 19:17:39 -05:00
c54a0c085a
collective trace improvements for debugging ( #1661 )
2025-05-07 13:37:31 -05:00
590ad6acc2
Merge pull request #1662 from BertanDogancay/2.25
...
[SYNC] 2.25.1-1
2025-05-06 09:39:09 -04:00
f3f3336468
Fix topo explorer's compatibility with NCCL 2.24 ( #1671 )
...
* Fix build issues
* Fix failure to find path remote rank
2025-05-05 15:26:29 -04:00
acfac55516
[Graph] Try using P2P by default ( #1670 )
2025-05-02 11:54:30 -04:00
Bertan Dogancay
Nilesh M Negi
Mustafa Abduljabbar
mberenjk
Wenkai Du
ryanhankins
isaki001
Arm Patinyasakdikul
Dingming Wu
BertanDogancay
Sarat Kamisetty
Tim
Richard Barnes
corey-derochie-amd
Nusrat Islam
Pedram Alizadeh
Avinash
alex-breslow-amd
Giuseppe Congiu
Dingming Wu
gilbertlee-amd