Commit Graph

1664 Commits

Author SHA1 Message Date
Avinash ccb0820743 Memory leak fix when numIBDevices = 0 (#1429)
* Initial commit for testing

* Fix memory leak in checkOptions

* Fix memory leak in checkOption

* x

* Delete cmake-3.28.2-linux-x86_64.sh

* gcn changes

* gcn memleak fixes

* gcn leak fix

* memory leak fixes for parseRome4P2H and ncclTopoAddGPU

* Keeping only necessary file for fixes

Deleting temporary scripts I created for debugging and testing

* changing to GCN_ARCH_NAME_LEN

* Added sanity check directory

* refactoring scripts

* Updated to sanity checks folder

* Initial fixes

* changes in tools

* pointing RCCL lib build to debug version

* Removed second pthread_detach

* Removing sanity checks

* Keeping only code changes

* addressing memory leaks in ncclIbinit

---------

Co-authored-by: Chao Chen <cchen104@amd.com>
2025-03-17 11:21:19 -05:00
mberenjk 5f691aaf65 Skipping AllReduce test on more than 8 ranks for FP8 type on Hyabusa (#1598)
* Skipping AllReduce FP8 test on 9 to 16 ranks (gfx90a) as it's using Tree algorithm not RING

---------

Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>
2025-03-17 10:22:49 -05:00
Mustafa Abduljabbar f67b2cc908 Multi-node reduce_scatter improved auto-selection for LL and LL128 on gfx942 (#1604)
* Add reduce_scatter LL and LL128 thresholds

* Always honor user choice for protocol
2025-03-17 11:21:01 -04:00
Wenkai Du 245c2de909 Enable LL128 on gfx942 (#1549) 2025-03-16 15:10:05 -07:00
Nilesh M Negi 1df73e209e [GRAPH] Increase default nChannels to 112 for gfx950 (#1596)
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
2025-03-14 14:47:03 -07:00
Wenkai Du 4237caad69 Limit P2P channels per peer to not exceeding max channels (#1594)
* Limit P2P channels per peer to not exceeding max channels

* [UT] test single GPU cases for all collectives

* [UT] fix out of range root value
2025-03-11 09:32:09 -07:00
Tim f6c6d451a9 mscclpp compatibility check for ubr (#1573) 2025-03-09 22:10:47 -04:00
isaki001 59c55842f1 fix the size of the recv buffer in AllGather UBR test (#1564) 2025-03-05 11:42:15 -06:00
Nusrat Islam ac823818aa misc/msccl: force use of mscclpp (#1581) 2025-03-04 12:48:59 -06:00
Bertan Dogancay d88cca3098 [Transport] Fix IntraNet (#1582) 2025-03-04 13:30:36 -05:00
Wenkai Du f957c4fe22 NPKit: enable reduce scatter profiling (#1580) 2025-03-04 10:03:56 -08:00
Nilesh M Negi 063c6cfc11 [BUILD] Enable multiple GPU targets in MSCCLPP (#1574)
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
2025-03-01 22:28:42 -06:00
dependabot[bot] a6b2ca224e Bump rocm-docs-core from 1.15.0 to 1.17.0 in /docs/sphinx (#1558)
Bumps [rocm-docs-core](https://github.com/ROCm/rocm-docs-core) from 1.15.0 to 1.17.0.
- [Release notes](https://github.com/ROCm/rocm-docs-core/releases)
- [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/ROCm/rocm-docs-core/compare/v1.15.0...v1.17.0)

---
updated-dependencies:
- dependency-name: rocm-docs-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-28 16:32:38 -07:00
Nusrat Islam 23c0b7bd84 misc/msccl: Read graph capture status for every collective call (#1576)
* misc/msccl: read graphCaptureStatus for every collective call

* fix a bug in checking whether UBR is enabled in MSCCLPP

* cmake: Fix patch reversal order

* misc/msccl: add logging
2025-02-28 17:16:07 -06:00
Wenkai Du 60c1264d27 Improve RDMA flushing by write dummy payload with RO=0 (#1570)
* Improve RDMA flushing by write dummy payload with RO=0

* Rename env var for disabling this change to RCCL_GDR_FLUSH_GPU_MEM_NO_RELAXED_ORDERING
2025-02-27 16:20:32 -08:00
Nilesh M Negi 02b8e504f0 [BUILD] Fix generate.py for gfx950 (#1575)
* [BUILD] Fix generate.py for gfx950

Signed-off-by: nilenegi <Nilesh.Negi@amd.com>

* [BUILD] Cleaner way to check for gfx targets

Signed-off-by: nilenegi <Nilesh.Negi@amd.com>

---------

Signed-off-by: nilenegi <Nilesh.Negi@amd.com>
2025-02-26 18:43:37 -06:00
Bertan Dogancay 85eb1f16bc Use bit reversal based mapping for multi-node (#1572) 2025-02-26 09:48:03 -05:00
Pedram Alizadeh f268553ee4 enable building rccl for gfx950 (#1571) 2025-02-25 16:13:48 -05:00
Nilesh M Negi daaa6e155f [BUILD] MSCCLPP: Fix OS check for CentOS (#1568)
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
Co-authored-by: corey-derochie-amd <corey.derochie@amd.com>
2025-02-25 13:03:04 -06:00
Nusrat Islam fdf75fd2c1 ext-src: tuning for allreduce8 kernel (#1560)
This PR tunes the number of threadblocks used for larger (>1MB)
message sizes.
2025-02-21 19:34:38 -06:00
Nusrat Islam 83f8b191ff ext-src: fix mscclpp allreduce for non-multiple of 128 message sizes (#1556) 2025-02-21 11:58:10 -06:00
gilbertlee-amd ddc5d58b93 Rail optimized trees (#1540)
* Allow disabling rail-optimized trees via RCCL_DISABLE_RAIL_TREES, Graphviz-friendly output via RCCL_OUTPUT_TREES
2025-02-20 15:18:29 -07:00
Nilesh M Negi 159587be5c [TOOLS] Update rcclDiagnostics script (#1557)
* [TOOLS] Update rcclDiagnostics script

Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

* Fix typo in valid_marketing_names list

Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>

---------

Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>
2025-02-20 16:11:05 -06:00
akolliasAMD aedbc95735 reverted the syncLDS back to syncthreads (#1554) 2025-02-19 10:44:32 -07:00
Wenkai Du baaa2ac64d Insert barrier after loading work items to LDS (#1551) 2025-02-18 10:17:27 -08:00
Wenkai Du 32dc7ef47c Enable GDRCopy only on gfx94x (#1550)
* Enable GDRCopy only on gfx94x

* Use cudaFree instead of hipFree

* Add warning if failed to get device property

* Remove extra return
2025-02-17 13:28:19 -08:00
Sohaib Nadeem 2f1c0bb213 Remove COMPILING_TARGETS from CMakeLists.txt (#1533)
COMPILING_TARGETS is not actually used for --offload-arch option,
instead GPU_TARGETS is being used implicitly when we call
find_package(hip REQUIRED) (See hip-config-amd.cmake).
2025-02-16 21:46:37 -06:00
Nikhil-Nunna 4ba94d6662 Env conf debug (#1534)
* Initial Script ready for review

* Added RCCL-tests and RCCL versions

* Added output folder and README

* Base format built

* Added ROCm version

* Added function to center titles and Vram information

* Added HIP version

* Cleaned formatting

* UCX version and MPI version

* Added NUMA balancing

* Added rocminfo

* Removed notes

* Changed regex for broadcom Nic

* Removed note by the ACS info

* Added Hostname to summary and details

* Print summary to terminal

* Added argparse

* Added flags and readme

* Added GPU ID

* fixed spelling

* renamed script again

* Added file descriptor and locked mem checks

* Added file descriptor and locked mem checks

* Removed extra spaces from summary table

* printing output file location

* Removed sudo in code and ACS flag
2025-02-14 17:31:18 -06:00
Pedram Alizadeh 0e5f4d0662 reverting the (Reduce NPKit latency overhead in MSCCL kernel) PR #893 (#1525) 2025-02-14 11:03:43 -05:00
corey-derochie-amd 824b81c034 Revert "replacing rccl_float8 with hip_fp8 and address compatibility issue (#…" (#1545)
This reverts commit d437d6e41c.
2025-02-13 10:00:22 -07:00
mberenjk d437d6e41c replacing rccl_float8 with hip_fp8 and address compatibility issue (#1538)
* replacing rccl_float8 with hip_fp8 and address compatibility issue with gfx942
---------

Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>
2025-02-13 10:34:17 -06:00
Wenkai Du ebf7e2305e Print KL/CL/KE events for all warps (#1544)
* Print KL/CL/KE events for all warps

* Fix count off-by-one issue

* Fix opCount in KE and restore CPU thread option

* Simplify count calculation
2025-02-12 13:36:31 -08:00
Wenkai Du f5b15f27a9 Move collective trace to HBM and fix log issue (#1542) 2025-02-11 11:40:14 -08:00
rahulc1984 92ac136db5 Make rccl version detection robust. (#1517)
* Accept an EXPLICIT_ROCM_VERSION and use that vs inspecting the environment if provided.
* Use CMake's built in file reading support vs execute_process (without error checking) to avoid silent but deadly later failures.
* Properly quote some comparisons to avoid syntax errors if they happen to have an empty string.
* Guard against ROCM_PATH being an empty string, avoiding stray path extensions to root directories, etc.

Co-authored-by: Stella Laurenzo <stellaraccident@gmail.com>
2025-02-11 10:48:22 -07:00
corey-derochie-amd 42ab425037 Switched from cmake_host_system_information feature to a manual parse (#1518)
* Switched cmake_host_system_information feature to a manual parse to remain cmake 3.5 compliant.

* Updating minimum cmake to 3.16 to conform with the rest of ROCm. This change still applies.
2025-02-11 08:51:39 -07:00
Nilesh M Negi 4e406acc43 [UT] Include iomanip if not defined (#1510)
* [UT] Include iomanip if not defined

Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

* Remove include guards

`iomanip.h` has pre-defined include guards. These are not needed.

Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>

---------

Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>
2025-02-11 08:48:47 -07:00
Dingming Wu e8fb1335fd Replace atomicAdd with _hip_atmoc_fetch_add in getting colltrace tail position (#1539) 2025-02-10 08:53:25 -08:00
gilbertlee-amd 6cb0599e38 Updating topology explorer (#1536) 2025-02-07 08:44:04 -07:00
Vijay Srinivasan 3494f52d40 Adding AINIC Network Plugin check (#1528)
- Adding AINIC network plugin check to pass unused parameter to pass the channelId to the network plugin layer
2025-02-06 23:37:53 -06:00
Nikhil-Nunna 8abc729f9e Merge pull request #1535 from Nikhil-Nunna/add-codeowner
Added @Nikhil-Nunna as a code owner
2025-02-05 14:41:46 -06:00
Nikhil-Nunna fd3422afdb Added Nikhil-Nunna to codeowners 2025-02-05 14:28:00 -06:00
AbandiGa e92a103bad Adding @AbandiGa (myself) as code owner (#1532)
Signed-off-by: AbandiGa <galaband@amd.com>
2025-02-05 13:23:25 -06:00
Wenkai Du a12bf32475 Reset barrier and make barrier_next thread local (#1531) 2025-02-05 09:06:48 -08:00
Wenkai Du d00e903d72 Revert "Remove unused code path (#1527)" (#1530)
This reverts commit 091bf899a1.
2025-02-04 13:14:43 -08:00
Edgar Gabriel 3646b1de43 update CODEOWNERS (#1529)
* update CODEOWNERS
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>
2025-02-04 11:54:42 -07:00
Wenkai Du 091bf899a1 Remove unused code path (#1527) 2025-02-04 10:24:56 -08:00
Bertan Dogancay 387c973b5d [P2P] Have connIdx for both send and recv (#1524) 2025-02-04 11:53:20 -05:00
isaki001 3398fa78fe non-hipGraph MSCCL++ tests for allReduce and allGather (#1503)
* working tests for a single message size

* move call_RCCL routine StandaloneUtils, create .cpp file for StandaloneUtils so that it can be included in several tests

* simplify test invocation

* remove unecessary logs and exit from ncclCommRegister

* set expected results for allGather

* skip test if nranks doesn't match number of gpus, call getAndDistributeNCCLid only from parent process

* fix improper size of expected-results vector

* Removing unused changes.

* Refactored to create a new file for the forked collectives call, as StandaloneUtils is for the Standalone tests. Renamed the functions to be slightly more accurate and follow existing naming conventions.

* Apply suggestions from code review

Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>

---------

Co-authored-by: isaki001 <isakioti@banff-pla-r27-38.pla.dcgpu>
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>
Co-authored-by: Corey Derochie <corey.derochie@amd.com>
2025-02-04 09:11:32 -06:00
isaki001 19105206f6 Update MSCCL++ register/deregister (#1523)
* erase handle key from mscclpp communicator during deregistration

* remove check on buffer size being a multiple of 32 from registration/deregistration routines since these checks are applied during enqueue

* add check for greater than zero buffer size in mscclpp registration
2025-02-04 09:09:56 -06:00
Bertan Dogancay 5804603632 [BUILD] Fix unsupported arguments in generator (#1519)
* Fix unsupported arguments in generator

* Get ROCM_PATH as env variable
2025-02-03 14:51:55 -05:00