AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,205 Commits 3 Branches 0 Tags
e6e99a1ae9785f5ccf14ee2232c812fc2ddec4e7
Commit Graph

598 Commits

Author SHA1 Message Date
Wen-Heng (Jack) Chung 33aa8b67be Use a map to host scratch buffers (#1004) 
* Use a map to host scratch buffers

* Address review feedbacks. Deliberately keep mscclSetupScratch function.

[ROCm/rccl commit: 293f0fb752]
 2023-12-05 13:15:28 -06:00
Nilesh M Negi 403a91137c Fix gcnArch bug in IFC mix build (#998) (#1002) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: bc44e3faa7]
 2023-12-04 16:20:22 -06:00
Bertan Dogancay ae0bdad45c IFC mix build (#998) 
[ROCm/rccl commit: 7c0f49a878]
 2023-12-02 18:49:52 -07:00
Wenkai Du b38b7fa3a2 Increase max channles to 64 (#993) 
[ROCm/rccl commit: 4ba65d1d6a]
 2023-12-01 16:01:11 -08:00
pradeep-ramanna bf57487384 Fix GPU to NIC mapping for peertopeer (#994) 
[ROCm/rccl commit: 0b53f79196]
 2023-12-01 08:00:17 -08:00
Ziyue Yang cef45b8311 Fix mscclAlgoHandle not initialized issue (#995) 
[ROCm/rccl commit: e44e112a17]
 2023-12-01 07:58:01 -08:00
Ziyue Yang f0c47d085e Move MSCCL algorithm loading to initialization to workaround HIP graph conflict (#982) 
* MSCCL: pre-specify channels and pre-load algorithms

* add mutex

* fix bug

* clean include

* disable all-gathers temporarily

[ROCm/rccl commit: 4bb0b4a380]
 2023-11-30 09:47:20 -08:00
akolliasAMD bd982864d5 recreated pr 914 to work with current develop branch (#979) 
[ROCm/rccl commit: 56ce9ef05f]
 2023-11-28 16:33:47 -07:00
Wenkai Du dcf623f2ec Add special handling of gfx940 (#976) 
* Add special handling of gfx940

* Update ring base

[ROCm/rccl commit: 50b2dd9fd7]
 2023-11-22 15:07:36 -08:00
Wenkai Du 4c2fa05a23 msccl: allocate scratch as ext-scope fine-grained (#968) 
[ROCm/rccl commit: 569d3f7d59]
 2023-11-16 09:57:25 -06:00
Wenkai Du 7c0920cd62 Fix kernel command line warnings (#961) 
* Fix kernel command line warnings

* Remove while loop

[ROCm/rccl commit: bc8661f092]
 2023-11-15 18:01:12 -08:00
Ziyue Yang 6ce074d92d Fix MSCCL work FIFO allocation with HIP graph enabled (#967) 
[ROCm/rccl commit: 7fc891bc8d]
 2023-11-15 16:43:28 -08:00
Bertan Dogancay 9e8eb41337 Check to support older ROCm versions (#963) 
[ROCm/rccl commit: 198f14923b]
 2023-11-15 12:36:31 -07:00
Ziyue Yang 2351578d5b Optimize MSCCL all-gather algorithms for gfx942 (#964) 
[ROCm/rccl commit: 7ae95db5b8]
 2023-11-15 08:18:59 -08:00
Ziyue Yang 2c6eededec Optimize MSCCL reduce primitive switching for gfx942 (#962) 
* Optimize reduce primitive switching for gfx942

* address comment

[ROCm/rccl commit: df128879a6]
 2023-11-15 08:18:44 -08:00
Wenkai Du 534af85d0f msccl: enable basic collective trace (#959) 
To avoid increasing number of kernels, colltrace is only enabled with
RCCL_MSCCL_FORCE_FULLOPS=1

[ROCm/rccl commit: 5a800e00cd]
 2023-11-08 20:14:28 -08:00
Wen-Heng (Jack) Chung 270aa41f6b Use send instead of sendWithBarrier. (#727) 
[ROCm/rccl commit: efc42d9045]
 2023-11-07 13:47:24 -06:00
Nusrat Islam 83a36c65c1 Merge pull request #950 from nusislam/msccl-red2 
msccl: remove cases from numReduction switch statement

[ROCm/rccl commit: 022735d208]
 2023-11-04 02:48:03 -05:00
Wenkai Du aa02d2b675 Use parallel init of LDS and adjust P2P channels for gfx94x (#943) 
* Use parallel init of LDS and adjust P2P channels for gfx94x

* Move another init to parallel

* Fix NCCL_NCHANNELS_PER_PEER setting

[ROCm/rccl commit: dbcba2923b]
 2023-11-03 16:06:49 -07:00
Nusrat Islam b6f47bad7c msccl: remove cases from numReduction switch statement 
[ROCm/rccl commit: f545b94d4b]
 2023-11-03 16:56:51 -05:00
Wenkai Du 1557a1f258 msccl: use 32-bit LDS access and add RCCL_MSCCL_FORCE_FULLOPS (#953) 
[ROCm/rccl commit: bb84345943]
 2023-11-03 10:38:02 -07:00
akolliasAMD 4cd86b185c MSCCL stream fix (#948) 
[ROCm/rccl commit: 988efe605a]
 2023-11-03 09:10:52 -06:00
Wenkai Du 297023a7a6 msccl: add templated kernel (#945) 
* msccl: add templated kernel

* Use defines to improve code readability

* Fix kernel indexing and review feedback

[ROCm/rccl commit: f484ff17b9]
 2023-11-02 17:21:53 -07:00
Nusrat Islam 88c8bb1495 msccl: remove dereference of reduce args 
It can be removed because the msccl kernel will never execute this code
according to the current msccl setup.


[ROCm/rccl commit: 6b80a0d0d4]
 2023-11-02 13:20:00 -05:00
Wenkai Du 3eeaea3f00 msccl: use atomic to set dependency flags (#941) 
[ROCm/rccl commit: a7400218a2]
 2023-10-31 14:46:57 -07:00
Wenkai Du b736b506c0 NPkit: misc fixes for MSCCL (#936) 
* msccl: add xcc_id to timestamp sync

* NPKit: add timestamp for rrc operator

* NPKit: add timestamp for MSCCL init

[ROCm/rccl commit: a497722894]
 2023-10-30 10:00:12 -07:00
Nilesh M Negi 8b1254a4f1 Fix gcnArchName bug in topology dump (#937) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 1e5ca6820b]
 2023-10-28 12:30:36 -05:00
Ziyue Yang e1dfb82023 Fix MSCCL work FIFO out-of-bound issue (#935) 
[ROCm/rccl commit: 4c117e5335]
 2023-10-27 11:24:52 -07:00
Nilesh M Negi b4ba7cc79d SRC/INIT: fix typo for ENABLE_PROFILING (#934) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 96ec3ffe2e]
 2023-10-26 23:52:46 -05:00
Nilesh M Negi 706750597c remove gcnArch support (#920) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: f22df90e5c]
 2023-10-26 12:09:15 -05:00
Wenkai Du 446c8cbf66 msccl: reduce debug output when using NCCL_DEBUG=INFO (#932) 
[ROCm/rccl commit: fb0eccb57b]
 2023-10-25 08:05:19 -07:00
Wenkai Du cc4de02a86 Add missing gfx942 support (#927) 
[ROCm/rccl commit: c4e65fd382]
 2023-10-23 12:04:37 -07:00
Wenkai Du 6f0f614d0b Remove LDS based software barriers from MSCCL (#923) 
[ROCm/rccl commit: dbb5611a3a]
 2023-10-19 16:39:41 -05:00
Wenkai Du edeea499b5 Update rome models (#922) 
[ROCm/rccl commit: 4278a9918b]
 2023-10-18 17:28:01 -07:00
Wenkai Du e0cc7de446 NPKit: add xcc_id field (#918) 
[ROCm/rccl commit: 39812ce757]
 2023-10-13 15:24:59 -07:00
Wenkai Du c0bd012e6c Fix incorrect arch name parsing (#916) 
[ROCm/rccl commit: 1b80d041cb]
 2023-10-13 10:01:11 -07:00
Wenkai Du 102f0165d6 Port init_once fix from NCCL (#915) 
[ROCm/rccl commit: 6d0b5c1e89]
 2023-10-13 08:01:12 -07:00
Wen-Heng (Jack) Chung dfa0d98f9e Change MSCCL kernel signature to allow kernel arguments be preloaded via SGPR (#911) 
* Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895)

Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu>

* Only build gfx941

* demo

* fine tune malloc

* Fix merge errors

* Fix merge errors

* Disable parallel build

* Adopt --amdgpu-kernarg-preload-count

* Revert "Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895)"

This reverts commit f5e252dddf02a41b4d1bc512f306f45f97166304.

* Revert CMake changes.

* NPKIT changes.

* Remove some license declarations.

* Address code review feedbacks on msccl_kernel_impl.h

* Update CMakeLists.txt

* Add CMake logic to check the existence of --amdgpu-kernarg-preload-count

* Fix NPKIT trace logic.

---------

Co-authored-by: Pedram Alizadeh <pmohamma@amd.com>
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu>
Co-authored-by: Ziyue Yang <ziyyang@microsoft.com>

[ROCm/rccl commit: 7ee5c1c28b]
 2023-10-12 20:17:08 -05:00
Bertan Dogancay 6f7965796f Revert "Remove 2H4P condition from P2P channels adjustment (#890)" (#904) 
This reverts commit 057e30e705.

[ROCm/rccl commit: a6ff4618c7]
 2023-10-04 09:46:11 -06:00
akolliasAMD 1ffd3eff31 Dma buf support optin (#905) 
* dmaBufSupport Optin added on every part of the code that should invoke it

[ROCm/rccl commit: 28d7fe5629]
 2023-10-03 03:17:48 -06:00
Bertan Dogancay b35ea4bd78 Modify All-To-All doc (#896) 
* Modify All-To-All doc

* Update nccl.h.in

* update unit-tests

---------

Co-authored-by: gilbertlee-amd <44450918+gilbertlee-amd@users.noreply.github.com>

[ROCm/rccl commit: c1f57a7041]
 2023-09-27 12:45:21 -04:00
akolliasAMD 6f7eb65308 changed the form that RCCL_TREE uses (#888) 
* changed the form that RCCL_TREE uses

[ROCm/rccl commit: b85d73c02e]
 2023-09-15 15:01:33 -06:00
Wenkai Du 3cc41809dd Reduce NPKit latency overhead in MSCCL kernel (#893) 
* Reduce NPKit latency overhead in MSCCL kernel

* Fix build error without NPKit enable

[ROCm/rccl commit: 26e982d913]
 2023-09-15 13:28:26 -07:00
Wenkai Du 057e30e705 Remove 2H4P condition from P2P channels adjustment (#890) 
[ROCm/rccl commit: 16dd05a58a]
 2023-09-13 12:54:21 -07:00
Ziyue Yang 6d593761dc Add single-node MI300X topology (#889) 
[ROCm/rccl commit: c1bfd5f0d8]
 2023-09-13 11:07:17 -07:00
Wenkai Du b0a16d80ff Fix crash when NPKit is enabled (#887) 
[ROCm/rccl commit: 6a4d5ec089]
 2023-09-13 11:00:12 -07:00
Audrey MP 2e3d45a53a Gcn arch name (#886) 
We use CMake to determine if we're compiling against a version of ROCm that supports gcnArchName and handles architecture checking appropriately. It includes a few helper functions as drop ins for the functionality we used gcnArch for before; sometimes to enable flags, and sometimes to set frequencies.

[ROCm/rccl commit: e58ec78d35]
 2023-09-12 15:34:40 -04:00
Andy li 43a9fd00ee enable hip graph on multi-node (#884) 
* initial checkin

* enable msccl when hip graph is on

* remove the commented out code of msccl enable check

* clean up the code

* remove the msccl HighestTransportType check logic

[ROCm/rccl commit: e1dc4d5e42]
 2023-09-11 15:30:04 -07:00
Nusrat Islam ffbfe43500 msccl: add NPKIT profiling for MSCCL send-recv 
[ROCm/rccl commit: a283f55f12]
 2023-09-08 13:11:16 -05:00
Bertan Dogancay 2aa31c89df RCCL should use hipPointerAttribute_t.type (#872) 
[ROCm/rccl commit: 6230b5f6b3]
 2023-09-05 09:44:12 -06:00