* Added `RCCL_MSCCL_ENABLE_SINGLE_PROCESS` runtime flag to return to the original MSCCL enablement behaviour except when explicitly enabling for multi-thread.
* Added documentation for the new `RCCL_MSCCL_ENABLE_SINGLE_PROCESS` runtime env var.
[ROCm/rccl commit: e056fe8f7e]
* Add another rome model and override
* Fix bug
* Fix typo
* Add ring
* Update ring
* Fix model matching
* Clean up
* Clean up
* Reverse rings for NCCL_RINGS input
* Only reverse NCCL_RINGS for ring graph
* Fix mapping issue when using NCCL_RINGS
* Add NCCL_RINGS_REMAP to handle inconsistant net names
[ROCm/rccl commit: 532b70afb6]
* adding all nccl apis to api_support to enable rccl tracing by rocprofv3
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>
Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>
[ROCm/rccl commit: db840f024e]
remove the MultiRank examples, the features was never released (because
it didn't work reliably), and it might just cause confusion if somebody
sees it. In additional, the locdation in tools was suboptimal.
[ROCm/rccl commit: 8953a26bcd]
* Restore number of parallel linking jobs
* Dynamically adjust number of linker jobs with limit of 16 jobs max
* Fix typo
* Add cgroup v1 support
[ROCm/rccl commit: ca5341d419]
* Adding performance collection feature in rccl_replayer, and updating MSCCL logging and replayer parsing
* Performance collection feature in rccl_replayer, and updating MSCCL logging and replayer parsing
[ROCm/rccl commit: cf311b71ee]
* adding rocprof parser script
* adding the support for multiple json files
* adding pytorch profiler script
* remove filtering from pytorch log
* adding the addressing the comments and add the feature to parse all kernels
* completing the report for torch profiler
---------
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>
[ROCm/rccl commit: 519843d2cf]
* Template unroll for RCCL kernels
* Adding unroll template arg during CMake hipification
* Reduce linking parallel jobs to avoid OOM in CI
* Workaround issues with UT tests
SWDEV-469533: register spill fix is needed for mainline build
LWPCOMMLIBS-369: cannot enable 112 channels with 80 CUs
Use -parallel-jobs=8 for linking
* CI: do not use -j 16 when building
* CI: use -j 8 when building
* Only reduce parallel linking job for CI extended
* Restore original jenkins command. Change parallel linking jobs in cmake
* Disable MSCCLPP
---------
Co-authored-by: gilbertlee-amd <gilbert.lee@amd.com>
[ROCm/rccl commit: 89349f2ce4]
* Revert "[GRAPH] Use channel shuffling only for IB systems (#1228)"
This reverts commit 8a7dd0e590.
* Revert "Revert "Changing channel stride for MI300X multinode (#1196)" (#1224)"
This reverts commit bb6fab3d8e.
[ROCm/rccl commit: a1ef217b32]
* Only enable MSCCL++ AllReduce for message sizes that are multiples of 32. MSCCL++ does not handle these other sizes.
* Sanitized MSCCL++ logging.
[ROCm/rccl commit: 9cbb3da224]
MSCCL can now run in a multi-threaded configuration. To test in the unit tests, added the ENABLE_OPENMP compile definition flag and the --openmp-test-enable flag to the unit test build script. To activate, set the environment variables UT_MULTITHREADED=1 and UT_PROCESS_MASK=1. Set Jenkins to use this mode.
[ROCm/rccl commit: 0c36d571ea]
* [GRAPH] Use channel shuffling only for IB systems
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
* [GRAPH] Define channels=48 for gfx94 RoCE systems
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
* [GRAPH] Increase channels for RoCE gfx94 systems
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
---------
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
[ROCm/rccl commit: 5be3b713ef]
Tim
corey-derochie-amd
Wenkai Du
Nusrat Islam
Nilesh M Negi
Edgar Gabriel
mberenjk
dependabot[bot]
Edgar Gabriel
akolliasAMD
Pedram Alizadeh
Ziyue Yang
Richard Barnes
Sam Wu
Benjamin Kitor
Joseph Macaranas
saurabhAMD
amd-jmacaran
Rahul Vaidya