307bc10781
[SRC] Enable unroll=1 for gfx950 ( #1602 )
...
* [SRC] Enable unroll=1 for gfx950
* Fix typo from rebase in generate.py
* Support for unroll=1 and gfx90a when building for all GPU targets
---------
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2025-03-27 18:21:35 -05:00
9dc23d9265
Disable mscclpp ( #1614 )
...
* disable mscclpp by default
2025-03-25 15:21:16 -05:00
565f0ade60
[CI] Fix warnings from pytest in Azure CI ( #1617 )
...
* [CI] Fix warnings from pytest
* [CI] Append to LD_LIBRARY_PATH instead of overwrite
---------
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2025-03-22 23:22:53 -05:00
d6b987a53f
[UT] Increase stack size for StandaloneTests to 480 ( #1616 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2025-03-21 21:33:32 -05:00
fe3f3aae17
Bump jinja2 from 3.1.5 to 3.1.6 in /docs/sphinx ( #1591 )
...
Bumps [jinja2](https://github.com/pallets/jinja ) from 3.1.5 to 3.1.6.
- [Release notes](https://github.com/pallets/jinja/releases )
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst )
- [Commits](https://github.com/pallets/jinja/compare/3.1.5...3.1.6 )
---
updated-dependencies:
- dependency-name: jinja2
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-03-21 17:17:32 -06:00
dd61df75ab
Bump cryptography from 43.0.1 to 44.0.1 in /docs/sphinx ( #1543 )
...
Bumps [cryptography](https://github.com/pyca/cryptography ) from 43.0.1 to 44.0.1.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pyca/cryptography/compare/43.0.1...44.0.1 )
---
updated-dependencies:
- dependency-name: cryptography
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-03-21 17:17:19 -06:00
626dc50ab5
Removing the experimental clique kernel files ( #1610 )
2025-03-20 18:10:01 -06:00
90ad586d94
Add fault injection of starting warps with random variations ( #1593 )
...
* Add fault injection of starting warps with random variations
This is done by inserting randomly delays after __syncthreads().
The feature can be turned off by FAULT_INJECTION=OFF in cmake.
* Remove manually introduced bug for demo purpose
* Use only one thread per warp for checking wall clock
2025-03-20 16:11:43 -07:00
9a4e49ff1a
Psuedo-randomly adding zero-byte sends in AllToAllv unit test ( #1597 )
2025-03-20 17:00:48 -06:00
805448261d
[Azure CI] Minor environment setup fixes
...
- Add extra deletion to ensure source workspace is clean for the job.
- pytest expects function names to start with test_
2025-03-20 15:36:10 -04:00
6505639cf4
removed gfx940 and gfx941 ( #1606 )
...
* removed gfx940 and gfx941
* removed gfx940 and gfx941
* Update "gfx94" to "gfx942" in init.cc
* Updated remaining "gfx94" updates to "gfx942"
* Update filenames and variables from gfx940 to gfx942
---------
Co-authored-by: akolliasAMD <akollias@amd.com >
2025-03-20 09:34:53 -06:00
6403afff4a
Introduce framework to run Azure Pipeline jobs ( #1608 )
...
- PR runs targeting develop will require approval from at least one person in a predefined list.
- Nightly runs are separated and will not require approval.
- Sample pytest script is provided for expanding test coverage.
2025-03-18 17:29:44 -04:00
bd0092e8f1
GDRCOPY support: Off by default ( #1605 )
2025-03-18 08:17:01 -07:00
ccb0820743
Memory leak fix when numIBDevices = 0 ( #1429 )
...
* Initial commit for testing
* Fix memory leak in checkOptions
* Fix memory leak in checkOption
* x
* Delete cmake-3.28.2-linux-x86_64.sh
* gcn changes
* gcn memleak fixes
* gcn leak fix
* memory leak fixes for parseRome4P2H and ncclTopoAddGPU
* Keeping only necessary file for fixes
Deleting temporary scripts I created for debugging and testing
* changing to GCN_ARCH_NAME_LEN
* Added sanity check directory
* refactoring scripts
* Updated to sanity checks folder
* Initial fixes
* changes in tools
* pointing RCCL lib build to debug version
* Removed second pthread_detach
* Removing sanity checks
* Keeping only code changes
* addressing memory leaks in ncclIbinit
---------
Co-authored-by: Chao Chen <cchen104@amd.com >
2025-03-17 11:21:19 -05:00
5f691aaf65
Skipping AllReduce test on more than 8 ranks for FP8 type on Hyabusa ( #1598 )
...
* Skipping AllReduce FP8 test on 9 to 16 ranks (gfx90a) as it's using Tree algorithm not RING
---------
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com >
2025-03-17 10:22:49 -05:00
f67b2cc908
Multi-node reduce_scatter improved auto-selection for LL and LL128 on gfx942 ( #1604 )
...
* Add reduce_scatter LL and LL128 thresholds
* Always honor user choice for protocol
2025-03-17 11:21:01 -04:00
245c2de909
Enable LL128 on gfx942 ( #1549 )
2025-03-16 15:10:05 -07:00
1df73e209e
[GRAPH] Increase default nChannels to 112 for gfx950 ( #1596 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2025-03-14 14:47:03 -07:00
4237caad69
Limit P2P channels per peer to not exceeding max channels ( #1594 )
...
* Limit P2P channels per peer to not exceeding max channels
* [UT] test single GPU cases for all collectives
* [UT] fix out of range root value
2025-03-11 09:32:09 -07:00
f6c6d451a9
mscclpp compatibility check for ubr ( #1573 )
2025-03-09 22:10:47 -04:00
59c55842f1
fix the size of the recv buffer in AllGather UBR test ( #1564 )
2025-03-05 11:42:15 -06:00
ac823818aa
misc/msccl: force use of mscclpp ( #1581 )
2025-03-04 12:48:59 -06:00
d88cca3098
[Transport] Fix IntraNet ( #1582 )
2025-03-04 13:30:36 -05:00
f957c4fe22
NPKit: enable reduce scatter profiling ( #1580 )
2025-03-04 10:03:56 -08:00
063c6cfc11
[BUILD] Enable multiple GPU targets in MSCCLPP ( #1574 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2025-03-01 22:28:42 -06:00
a6b2ca224e
Bump rocm-docs-core from 1.15.0 to 1.17.0 in /docs/sphinx ( #1558 )
...
Bumps [rocm-docs-core](https://github.com/ROCm/rocm-docs-core ) from 1.15.0 to 1.17.0.
- [Release notes](https://github.com/ROCm/rocm-docs-core/releases )
- [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md )
- [Commits](https://github.com/ROCm/rocm-docs-core/compare/v1.15.0...v1.17.0 )
---
updated-dependencies:
- dependency-name: rocm-docs-core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-28 16:32:38 -07:00
23c0b7bd84
misc/msccl: Read graph capture status for every collective call ( #1576 )
...
* misc/msccl: read graphCaptureStatus for every collective call
* fix a bug in checking whether UBR is enabled in MSCCLPP
* cmake: Fix patch reversal order
* misc/msccl: add logging
2025-02-28 17:16:07 -06:00
60c1264d27
Improve RDMA flushing by write dummy payload with RO=0 ( #1570 )
...
* Improve RDMA flushing by write dummy payload with RO=0
* Rename env var for disabling this change to RCCL_GDR_FLUSH_GPU_MEM_NO_RELAXED_ORDERING
2025-02-27 16:20:32 -08:00
02b8e504f0
[BUILD] Fix generate.py for gfx950 ( #1575 )
...
* [BUILD] Fix generate.py for gfx950
Signed-off-by: nilenegi <Nilesh.Negi@amd.com >
* [BUILD] Cleaner way to check for gfx targets
Signed-off-by: nilenegi <Nilesh.Negi@amd.com >
---------
Signed-off-by: nilenegi <Nilesh.Negi@amd.com >
2025-02-26 18:43:37 -06:00
85eb1f16bc
Use bit reversal based mapping for multi-node ( #1572 )
2025-02-26 09:48:03 -05:00
f268553ee4
enable building rccl for gfx950 ( #1571 )
2025-02-25 16:13:48 -05:00
daaa6e155f
[BUILD] MSCCLPP: Fix OS check for CentOS ( #1568 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
Co-authored-by: corey-derochie-amd <corey.derochie@amd.com >
2025-02-25 13:03:04 -06:00
fdf75fd2c1
ext-src: tuning for allreduce8 kernel ( #1560 )
...
This PR tunes the number of threadblocks used for larger (>1MB)
message sizes.
2025-02-21 19:34:38 -06:00
83f8b191ff
ext-src: fix mscclpp allreduce for non-multiple of 128 message sizes ( #1556 )
2025-02-21 11:58:10 -06:00
ddc5d58b93
Rail optimized trees ( #1540 )
...
* Allow disabling rail-optimized trees via RCCL_DISABLE_RAIL_TREES, Graphviz-friendly output via RCCL_OUTPUT_TREES
2025-02-20 15:18:29 -07:00
159587be5c
[TOOLS] Update rcclDiagnostics script ( #1557 )
...
* [TOOLS] Update rcclDiagnostics script
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
* Fix typo in valid_marketing_names list
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com >
---------
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com >
2025-02-20 16:11:05 -06:00
aedbc95735
reverted the syncLDS back to syncthreads ( #1554 )
2025-02-19 10:44:32 -07:00
baaa2ac64d
Insert barrier after loading work items to LDS ( #1551 )
2025-02-18 10:17:27 -08:00
32dc7ef47c
Enable GDRCopy only on gfx94x ( #1550 )
...
* Enable GDRCopy only on gfx94x
* Use cudaFree instead of hipFree
* Add warning if failed to get device property
* Remove extra return
2025-02-17 13:28:19 -08:00
2f1c0bb213
Remove COMPILING_TARGETS from CMakeLists.txt ( #1533 )
...
COMPILING_TARGETS is not actually used for --offload-arch option,
instead GPU_TARGETS is being used implicitly when we call
find_package(hip REQUIRED) (See hip-config-amd.cmake).
2025-02-16 21:46:37 -06:00
4ba94d6662
Env conf debug ( #1534 )
...
* Initial Script ready for review
* Added RCCL-tests and RCCL versions
* Added output folder and README
* Base format built
* Added ROCm version
* Added function to center titles and Vram information
* Added HIP version
* Cleaned formatting
* UCX version and MPI version
* Added NUMA balancing
* Added rocminfo
* Removed notes
* Changed regex for broadcom Nic
* Removed note by the ACS info
* Added Hostname to summary and details
* Print summary to terminal
* Added argparse
* Added flags and readme
* Added GPU ID
* fixed spelling
* renamed script again
* Added file descriptor and locked mem checks
* Added file descriptor and locked mem checks
* Removed extra spaces from summary table
* printing output file location
* Removed sudo in code and ACS flag
2025-02-14 17:31:18 -06:00
0e5f4d0662
reverting the (Reduce NPKit latency overhead in MSCCL kernel) PR #893 ( #1525 )
2025-02-14 11:03:43 -05:00
824b81c034
Revert "replacing rccl_float8 with hip_fp8 and address compatibility issue (#…" ( #1545 )
...
This reverts commit d437d6e41c
2025-02-13 10:00:22 -07:00
d437d6e41c
replacing rccl_float8 with hip_fp8 and address compatibility issue ( #1538 )
...
* replacing rccl_float8 with hip_fp8 and address compatibility issue with gfx942
---------
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com >
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com >
2025-02-13 10:34:17 -06:00
ebf7e2305e
Print KL/CL/KE events for all warps ( #1544 )
...
* Print KL/CL/KE events for all warps
* Fix count off-by-one issue
* Fix opCount in KE and restore CPU thread option
* Simplify count calculation
2025-02-12 13:36:31 -08:00
f5b15f27a9
Move collective trace to HBM and fix log issue ( #1542 )
2025-02-11 11:40:14 -08:00
92ac136db5
Make rccl version detection robust. ( #1517 )
...
* Accept an EXPLICIT_ROCM_VERSION and use that vs inspecting the environment if provided.
* Use CMake's built in file reading support vs execute_process (without error checking) to avoid silent but deadly later failures.
* Properly quote some comparisons to avoid syntax errors if they happen to have an empty string.
* Guard against ROCM_PATH being an empty string, avoiding stray path extensions to root directories, etc.
Co-authored-by: Stella Laurenzo <stellaraccident@gmail.com >
2025-02-11 10:48:22 -07:00
42ab425037
Switched from cmake_host_system_information feature to a manual parse ( #1518 )
...
* Switched cmake_host_system_information feature to a manual parse to remain cmake 3.5 compliant.
* Updating minimum cmake to 3.16 to conform with the rest of ROCm. This change still applies.
2025-02-11 08:51:39 -07:00
4e406acc43
[UT] Include iomanip if not defined ( #1510 )
...
* [UT] Include iomanip if not defined
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
* Remove include guards
`iomanip.h` has pre-defined include guards. These are not needed.
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com >
---------
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com >
2025-02-11 08:48:47 -07:00
e8fb1335fd
Replace atomicAdd with _hip_atmoc_fetch_add in getting colltrace tail position ( #1539 )
2025-02-10 08:53:25 -08:00
Nilesh M Negi
isaki001
dependabot[bot]
gilbertlee-amd
Wenkai Du
amd-jmacaran
corey-derochie-amd
Joseph Macaranas
Avinash
mberenjk
Mustafa Abduljabbar
Tim
Nusrat Islam
Bertan Dogancay
Pedram Alizadeh
akolliasAMD
Sohaib Nadeem
Nikhil-Nunna
rahulc1984
Dingming Wu