AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,277 Commits 3 Branches 0 Tags
45ed3ef4e78e5d779cfc66d0fd78a0c3a5ee323f
Commit Graph

640 Commits

Author SHA1 Message Date
Ziyue Yang 6ce074d92d Fix MSCCL work FIFO allocation with HIP graph enabled (#967) 
[ROCm/rccl commit: 7fc891bc8d]
 2023-11-15 16:43:28 -08:00
Bertan Dogancay 9e8eb41337 Check to support older ROCm versions (#963) 
[ROCm/rccl commit: 198f14923b]
 2023-11-15 12:36:31 -07:00
Ziyue Yang 2351578d5b Optimize MSCCL all-gather algorithms for gfx942 (#964) 
[ROCm/rccl commit: 7ae95db5b8]
 2023-11-15 08:18:59 -08:00
Ziyue Yang 2c6eededec Optimize MSCCL reduce primitive switching for gfx942 (#962) 
* Optimize reduce primitive switching for gfx942

* address comment

[ROCm/rccl commit: df128879a6]
 2023-11-15 08:18:44 -08:00
Wenkai Du 534af85d0f msccl: enable basic collective trace (#959) 
To avoid increasing number of kernels, colltrace is only enabled with
RCCL_MSCCL_FORCE_FULLOPS=1

[ROCm/rccl commit: 5a800e00cd]
 2023-11-08 20:14:28 -08:00
Wen-Heng (Jack) Chung 270aa41f6b Use send instead of sendWithBarrier. (#727) 
[ROCm/rccl commit: efc42d9045]
 2023-11-07 13:47:24 -06:00
Nusrat Islam 83a36c65c1 Merge pull request #950 from nusislam/msccl-red2 
msccl: remove cases from numReduction switch statement

[ROCm/rccl commit: 022735d208]
 2023-11-04 02:48:03 -05:00
Wenkai Du aa02d2b675 Use parallel init of LDS and adjust P2P channels for gfx94x (#943) 
* Use parallel init of LDS and adjust P2P channels for gfx94x

* Move another init to parallel

* Fix NCCL_NCHANNELS_PER_PEER setting

[ROCm/rccl commit: dbcba2923b]
 2023-11-03 16:06:49 -07:00
Nusrat Islam b6f47bad7c msccl: remove cases from numReduction switch statement 
[ROCm/rccl commit: f545b94d4b]
 2023-11-03 16:56:51 -05:00
Wenkai Du 1557a1f258 msccl: use 32-bit LDS access and add RCCL_MSCCL_FORCE_FULLOPS (#953) 
[ROCm/rccl commit: bb84345943]
 2023-11-03 10:38:02 -07:00
akolliasAMD 4cd86b185c MSCCL stream fix (#948) 
[ROCm/rccl commit: 988efe605a]
 2023-11-03 09:10:52 -06:00
Wenkai Du 297023a7a6 msccl: add templated kernel (#945) 
* msccl: add templated kernel

* Use defines to improve code readability

* Fix kernel indexing and review feedback

[ROCm/rccl commit: f484ff17b9]
 2023-11-02 17:21:53 -07:00
Nusrat Islam 88c8bb1495 msccl: remove dereference of reduce args 
It can be removed because the msccl kernel will never execute this code
according to the current msccl setup.


[ROCm/rccl commit: 6b80a0d0d4]
 2023-11-02 13:20:00 -05:00
Wenkai Du 3eeaea3f00 msccl: use atomic to set dependency flags (#941) 
[ROCm/rccl commit: a7400218a2]
 2023-10-31 14:46:57 -07:00
Wenkai Du b736b506c0 NPkit: misc fixes for MSCCL (#936) 
* msccl: add xcc_id to timestamp sync

* NPKit: add timestamp for rrc operator

* NPKit: add timestamp for MSCCL init

[ROCm/rccl commit: a497722894]
 2023-10-30 10:00:12 -07:00
Nilesh M Negi 8b1254a4f1 Fix gcnArchName bug in topology dump (#937) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 1e5ca6820b]
 2023-10-28 12:30:36 -05:00
Ziyue Yang e1dfb82023 Fix MSCCL work FIFO out-of-bound issue (#935) 
[ROCm/rccl commit: 4c117e5335]
 2023-10-27 11:24:52 -07:00
Nilesh M Negi b4ba7cc79d SRC/INIT: fix typo for ENABLE_PROFILING (#934) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 96ec3ffe2e]
 2023-10-26 23:52:46 -05:00
Nilesh M Negi 706750597c remove gcnArch support (#920) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: f22df90e5c]
 2023-10-26 12:09:15 -05:00
Wenkai Du 446c8cbf66 msccl: reduce debug output when using NCCL_DEBUG=INFO (#932) 
[ROCm/rccl commit: fb0eccb57b]
 2023-10-25 08:05:19 -07:00
Wenkai Du cc4de02a86 Add missing gfx942 support (#927) 
[ROCm/rccl commit: c4e65fd382]
 2023-10-23 12:04:37 -07:00
Wenkai Du 6f0f614d0b Remove LDS based software barriers from MSCCL (#923) 
[ROCm/rccl commit: dbb5611a3a]
 2023-10-19 16:39:41 -05:00
Wenkai Du edeea499b5 Update rome models (#922) 
[ROCm/rccl commit: 4278a9918b]
 2023-10-18 17:28:01 -07:00
Wenkai Du e0cc7de446 NPKit: add xcc_id field (#918) 
[ROCm/rccl commit: 39812ce757]
 2023-10-13 15:24:59 -07:00
Wenkai Du c0bd012e6c Fix incorrect arch name parsing (#916) 
[ROCm/rccl commit: 1b80d041cb]
 2023-10-13 10:01:11 -07:00
Wenkai Du 102f0165d6 Port init_once fix from NCCL (#915) 
[ROCm/rccl commit: 6d0b5c1e89]
 2023-10-13 08:01:12 -07:00
Wen-Heng (Jack) Chung dfa0d98f9e Change MSCCL kernel signature to allow kernel arguments be preloaded via SGPR (#911) 
* Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895)

Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu>

* Only build gfx941

* demo

* fine tune malloc

* Fix merge errors

* Fix merge errors

* Disable parallel build

* Adopt --amdgpu-kernarg-preload-count

* Revert "Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895)"

This reverts commit f5e252dddf02a41b4d1bc512f306f45f97166304.

* Revert CMake changes.

* NPKIT changes.

* Remove some license declarations.

* Address code review feedbacks on msccl_kernel_impl.h

* Update CMakeLists.txt

* Add CMake logic to check the existence of --amdgpu-kernarg-preload-count

* Fix NPKIT trace logic.

---------

Co-authored-by: Pedram Alizadeh <pmohamma@amd.com>
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu>
Co-authored-by: Ziyue Yang <ziyyang@microsoft.com>

[ROCm/rccl commit: 7ee5c1c28b]
 2023-10-12 20:17:08 -05:00
Bertan Dogancay 6f7965796f Revert "Remove 2H4P condition from P2P channels adjustment (#890)" (#904) 
This reverts commit 057e30e705.

[ROCm/rccl commit: a6ff4618c7]
 2023-10-04 09:46:11 -06:00
akolliasAMD 1ffd3eff31 Dma buf support optin (#905) 
* dmaBufSupport Optin added on every part of the code that should invoke it

[ROCm/rccl commit: 28d7fe5629]
 2023-10-03 03:17:48 -06:00
Bertan Dogancay b35ea4bd78 Modify All-To-All doc (#896) 
* Modify All-To-All doc

* Update nccl.h.in

* update unit-tests

---------

Co-authored-by: gilbertlee-amd <44450918+gilbertlee-amd@users.noreply.github.com>

[ROCm/rccl commit: c1f57a7041]
 2023-09-27 12:45:21 -04:00
Kaiming Ouyang d03ae00bac Fix cudaMemcpyAsync bug 
We are trying to use the copy result of first cudaMemcpyAsync in the
second cudaMemcpyAsync without sync in between. This patch fixes it
by allocating a CPU side array to cache device side addr so that we
can avoid this consecutive cuda mem copy.

Fixes #957


[ROCm/rccl commit: 4365458757]
 2023-09-20 05:51:14 -07:00
akolliasAMD 6f7eb65308 changed the form that RCCL_TREE uses (#888) 
* changed the form that RCCL_TREE uses

[ROCm/rccl commit: b85d73c02e]
 2023-09-15 15:01:33 -06:00
Wenkai Du 3cc41809dd Reduce NPKit latency overhead in MSCCL kernel (#893) 
* Reduce NPKit latency overhead in MSCCL kernel

* Fix build error without NPKit enable

[ROCm/rccl commit: 26e982d913]
 2023-09-15 13:28:26 -07:00
Wenkai Du 057e30e705 Remove 2H4P condition from P2P channels adjustment (#890) 
[ROCm/rccl commit: 16dd05a58a]
 2023-09-13 12:54:21 -07:00
Ziyue Yang 6d593761dc Add single-node MI300X topology (#889) 
[ROCm/rccl commit: c1bfd5f0d8]
 2023-09-13 11:07:17 -07:00
Wenkai Du b0a16d80ff Fix crash when NPKit is enabled (#887) 
[ROCm/rccl commit: 6a4d5ec089]
 2023-09-13 11:00:12 -07:00
Audrey MP 2e3d45a53a Gcn arch name (#886) 
We use CMake to determine if we're compiling against a version of ROCm that supports gcnArchName and handles architecture checking appropriately. It includes a few helper functions as drop ins for the functionality we used gcnArch for before; sometimes to enable flags, and sometimes to set frequencies.

[ROCm/rccl commit: e58ec78d35]
 2023-09-12 15:34:40 -04:00
Andy li 43a9fd00ee enable hip graph on multi-node (#884) 
* initial checkin

* enable msccl when hip graph is on

* remove the commented out code of msccl enable check

* clean up the code

* remove the msccl HighestTransportType check logic

[ROCm/rccl commit: e1dc4d5e42]
 2023-09-11 15:30:04 -07:00
Nusrat Islam ffbfe43500 msccl: add NPKIT profiling for MSCCL send-recv 
[ROCm/rccl commit: a283f55f12]
 2023-09-08 13:11:16 -05:00
Bertan Dogancay 2aa31c89df RCCL should use hipPointerAttribute_t.type (#872) 
[ROCm/rccl commit: 6230b5f6b3]
 2023-09-05 09:44:12 -06:00
gilbertlee-amd 5fe857c562 More robust msccl shared directory location discovery (#868) 
[ROCm/rccl commit: 4297315de7]
 2023-08-30 08:10:14 -06:00
Pedram Alizadeh b4f96a23e6 optimizing COLL_UNROLL for MI100 machines (#863) 
[ROCm/rccl commit: e7f27c66e0]
 2023-08-29 12:49:06 -04:00
Wenkai Du af04103d72 Add MSCCL xml files (#861) 
[ROCm/rccl commit: aeca1af374]
 2023-08-23 14:12:34 -07:00
gilbertlee-amd 3dd880fe74 Minor fix for some msccl installations (#862) 
[ROCm/rccl commit: 5bcd3768cc]
 2023-08-23 13:48:58 -06:00
Sylvain Jeaugey 26b91b9dbb 2.18.5-1 
Fix NVLS search (issue #931).
Increase max IB NICs to 32.
Fix inconsistent device ordering (issue #820).
Try to use different devices for different GPUs in systems with
more than one NIC per GFU.


[ROCm/rccl commit: 559b70f86c]
 2023-08-23 06:32:36 -07:00
Wenkai Du 5983f0e371 Use relaxed atomics for LL on GFX11 (#859) 
[ROCm/rccl commit: 6a0a6a37d9]
 2023-08-21 16:28:39 -07:00
David Pagan 75e3927f50 Fix static_assert string literal that contains a "\%". This is no longer (#860) 
valid. They can only be simple escape sequences. Removing '\' fixes
issue. Assert message now compiles and emits the '%' as expected.

[ROCm/rccl commit: 2ec2648247]
 2023-08-21 16:19:59 -07:00
akolliasAMD 56129830a6 NCCL_TREES variable and rome model fixes (#856) 
[ROCm/rccl commit: d33cd5a233]
 2023-08-21 10:35:37 -06:00
Wenkai Du 6fdb4103b7 gfx11: don't use LL for sendrecv (#853) 
* gfx11: don't use LL for sendrecv

* Use builtin instead of inline asm

[ROCm/rccl commit: f70e3e569b]
 2023-08-17 08:50:51 -07:00
Wenkai Du 0c31452135 Add new model support (#847) 
* Add new model support

* Update new rings

[ROCm/rccl commit: 7044599575]
 2023-08-10 17:14:51 -07:00