AI/rocm-systems
2
0 Форкнуть 0
Код Задачи Запросы на слияние Действия Пакеты Проекты Релизы Вики Активность
1 224 коммитов 3 Ветки 0 Теги
cd7a3462970ecfe32f04b0a5b64e4f09c1701c06
Граф коммитов
611 Коммитов
Автор SHA1 Сообщение Дата
Wenkai Du cd7a346297 Doubling buffer size to fix NCCL INFO corruption with increased channels (#1035) 
[ROCm/rccl commit: f7e39fced2]
 2024-01-08 08:14:33 -08:00
Wenkai Du 626608c172 Increase stack size for gfx906 (#1034) 
Occationally "Memory access fault by GPU node-8 (Agent handle: 0x23a5640) on address 0x7f461ec00000. Reason: Page not present or supervisor privilege" can be seen from gfx906 CI

[ROCm/rccl commit: e5bf56c6d8]
 2024-01-07 20:25:02 -08:00
Ziyue Yang 1b39fef32a Fix MSCCL multi-node (#1032) 
1) Move needsProxy initialization before mscclSetupConnections since the latter
will revise it later.
2) Remove mscclAvailable check in net.cc since it's no more required and caused
non-shared buffer allocated for MSCCL which is not expected.

[ROCm/rccl commit: 70bbeb4773]
 2024-01-05 17:03:43 -08:00
Wenkai Du 13791d7ee3 Rework barriers and adjust scope of atomics (#1019) 
[ROCm/rccl commit: abf265a911]
 2024-01-04 08:18:48 -08:00
Ziyue Yang e3d45f9de4 Improve MSCCL algorithms (#1023) 
[ROCm/rccl commit: 0a53077c9c]
 2024-01-03 14:51:34 -08:00
akolliasAMD 0c1f773021 rearranged how the min and max functions are part of msccl (#1025) 
* rearranged how the min and max functions are part of msccl

* added more coverage on in place graph tests

[ROCm/rccl commit: f4858e14b2]
 2023-12-21 08:58:33 -07:00
Ziyue Yang e4b63a8ba0 Fully disable MSCCL when machine is not matched (#1017) 
* Disable MSCCL algorithm meta loading when machine is not matched

* fully disable init

* fix potential segfault

[ROCm/rccl commit: 655742a3a6]
 2023-12-13 08:36:21 -08:00
Wenkai Du 918ce6c2e2 msccl: disable on multi-node (#1018) 
[ROCm/rccl commit: 53d807a5b9]
 2023-12-13 07:41:40 -08:00
Wenkai Du 48107b18c9 msccl: fix data corruption with MTYPE_RW (#1014) 
[ROCm/rccl commit: 81602814a7]
 2023-12-11 20:33:15 -08:00
Wenkai Du 481a35bc59 Fix memory fence and use non-temporal store (#1007) 
* Fix memory fence and use non-temporal store

* Use amdgcn builtin instead of inline asm

* Move threadfence location

* Revert changes to gfx90a

* Rework gfx90a change

* Apply changes to gfx94x

[ROCm/rccl commit: 7965c8b53c]
 2023-12-09 12:16:08 -08:00
Ziyue Yang fe250212bf Fix MSCCL scratch allocation (#1010) 
[ROCm/rccl commit: c002f20029]
 2023-12-08 17:47:10 -06:00
Wen-Heng (Jack) Chung 529f72ceaa Relax workgroup barrier implementation for MSCCL send/recv ops. (#997) 
* Trim logic.

* Revert "Trim logic."

This reverts commit 8f2dba6c764108acf2bf5428366b9f41d4d206b9.

* Introduce MSCCL template parameters to send / recv.

* Address review feedbacks.

[ROCm/rccl commit: baadda4bd8]
 2023-12-08 17:46:53 -06:00
Wenkai Du 22730bdc4d msccl: build same number of kernels as in ROCm 5.7 (#1005) 
Removed fullOps kernels from build

[ROCm/rccl commit: 12c08fc52a]
 2023-12-07 13:36:04 -06:00
Wen-Heng (Jack) Chung 33aa8b67be Use a map to host scratch buffers (#1004) 
* Use a map to host scratch buffers

* Address review feedbacks. Deliberately keep mscclSetupScratch function.

[ROCm/rccl commit: 293f0fb752]
 2023-12-05 13:15:28 -06:00
Nilesh M Negi 403a91137c Fix gcnArch bug in IFC mix build (#998) (#1002) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: bc44e3faa7]
 2023-12-04 16:20:22 -06:00
Bertan Dogancay ae0bdad45c IFC mix build (#998) 
[ROCm/rccl commit: 7c0f49a878]
 2023-12-02 18:49:52 -07:00
Wenkai Du b38b7fa3a2 Increase max channles to 64 (#993) 
[ROCm/rccl commit: 4ba65d1d6a]
 2023-12-01 16:01:11 -08:00
pradeep-ramanna bf57487384 Fix GPU to NIC mapping for peertopeer (#994) 
[ROCm/rccl commit: 0b53f79196]
 2023-12-01 08:00:17 -08:00
Ziyue Yang cef45b8311 Fix mscclAlgoHandle not initialized issue (#995) 
[ROCm/rccl commit: e44e112a17]
 2023-12-01 07:58:01 -08:00
Ziyue Yang f0c47d085e Move MSCCL algorithm loading to initialization to workaround HIP graph conflict (#982) 
* MSCCL: pre-specify channels and pre-load algorithms

* add mutex

* fix bug

* clean include

* disable all-gathers temporarily

[ROCm/rccl commit: 4bb0b4a380]
 2023-11-30 09:47:20 -08:00
akolliasAMD bd982864d5 recreated pr 914 to work with current develop branch (#979) 
[ROCm/rccl commit: 56ce9ef05f]
 2023-11-28 16:33:47 -07:00
Wenkai Du dcf623f2ec Add special handling of gfx940 (#976) 
* Add special handling of gfx940

* Update ring base

[ROCm/rccl commit: 50b2dd9fd7]
 2023-11-22 15:07:36 -08:00
Wenkai Du 4c2fa05a23 msccl: allocate scratch as ext-scope fine-grained (#968) 
[ROCm/rccl commit: 569d3f7d59]
 2023-11-16 09:57:25 -06:00
Wenkai Du 7c0920cd62 Fix kernel command line warnings (#961) 
* Fix kernel command line warnings

* Remove while loop

[ROCm/rccl commit: bc8661f092]
 2023-11-15 18:01:12 -08:00
Ziyue Yang 6ce074d92d Fix MSCCL work FIFO allocation with HIP graph enabled (#967) 
[ROCm/rccl commit: 7fc891bc8d]
 2023-11-15 16:43:28 -08:00
Bertan Dogancay 9e8eb41337 Check to support older ROCm versions (#963) 
[ROCm/rccl commit: 198f14923b]
 2023-11-15 12:36:31 -07:00
Ziyue Yang 2351578d5b Optimize MSCCL all-gather algorithms for gfx942 (#964) 
[ROCm/rccl commit: 7ae95db5b8]
 2023-11-15 08:18:59 -08:00
Ziyue Yang 2c6eededec Optimize MSCCL reduce primitive switching for gfx942 (#962) 
* Optimize reduce primitive switching for gfx942

* address comment

[ROCm/rccl commit: df128879a6]
 2023-11-15 08:18:44 -08:00
Wenkai Du 534af85d0f msccl: enable basic collective trace (#959) 
To avoid increasing number of kernels, colltrace is only enabled with
RCCL_MSCCL_FORCE_FULLOPS=1

[ROCm/rccl commit: 5a800e00cd]
 2023-11-08 20:14:28 -08:00
Wen-Heng (Jack) Chung 270aa41f6b Use send instead of sendWithBarrier. (#727) 
[ROCm/rccl commit: efc42d9045]
 2023-11-07 13:47:24 -06:00
Nusrat Islam 83a36c65c1 Merge pull request #950 from nusislam/msccl-red2 
msccl: remove cases from numReduction switch statement

[ROCm/rccl commit: 022735d208]
 2023-11-04 02:48:03 -05:00
Wenkai Du aa02d2b675 Use parallel init of LDS and adjust P2P channels for gfx94x (#943) 
* Use parallel init of LDS and adjust P2P channels for gfx94x

* Move another init to parallel

* Fix NCCL_NCHANNELS_PER_PEER setting

[ROCm/rccl commit: dbcba2923b]
 2023-11-03 16:06:49 -07:00
Nusrat Islam b6f47bad7c msccl: remove cases from numReduction switch statement 
[ROCm/rccl commit: f545b94d4b]
 2023-11-03 16:56:51 -05:00
Wenkai Du 1557a1f258 msccl: use 32-bit LDS access and add RCCL_MSCCL_FORCE_FULLOPS (#953) 
[ROCm/rccl commit: bb84345943]
 2023-11-03 10:38:02 -07:00
akolliasAMD 4cd86b185c MSCCL stream fix (#948) 
[ROCm/rccl commit: 988efe605a]
 2023-11-03 09:10:52 -06:00
Wenkai Du 297023a7a6 msccl: add templated kernel (#945) 
* msccl: add templated kernel

* Use defines to improve code readability

* Fix kernel indexing and review feedback

[ROCm/rccl commit: f484ff17b9]
 2023-11-02 17:21:53 -07:00
Nusrat Islam 88c8bb1495 msccl: remove dereference of reduce args 
It can be removed because the msccl kernel will never execute this code
according to the current msccl setup.


[ROCm/rccl commit: 6b80a0d0d4]
 2023-11-02 13:20:00 -05:00
Wenkai Du 3eeaea3f00 msccl: use atomic to set dependency flags (#941) 
[ROCm/rccl commit: a7400218a2]
 2023-10-31 14:46:57 -07:00
Wenkai Du b736b506c0 NPkit: misc fixes for MSCCL (#936) 
* msccl: add xcc_id to timestamp sync

* NPKit: add timestamp for rrc operator

* NPKit: add timestamp for MSCCL init

[ROCm/rccl commit: a497722894]
 2023-10-30 10:00:12 -07:00
Nilesh M Negi 8b1254a4f1 Fix gcnArchName bug in topology dump (#937) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 1e5ca6820b]
 2023-10-28 12:30:36 -05:00
Ziyue Yang e1dfb82023 Fix MSCCL work FIFO out-of-bound issue (#935) 
[ROCm/rccl commit: 4c117e5335]
 2023-10-27 11:24:52 -07:00
Nilesh M Negi b4ba7cc79d SRC/INIT: fix typo for ENABLE_PROFILING (#934) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 96ec3ffe2e]
 2023-10-26 23:52:46 -05:00
Nilesh M Negi 706750597c remove gcnArch support (#920) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: f22df90e5c]
 2023-10-26 12:09:15 -05:00
Wenkai Du 446c8cbf66 msccl: reduce debug output when using NCCL_DEBUG=INFO (#932) 
[ROCm/rccl commit: fb0eccb57b]
 2023-10-25 08:05:19 -07:00
Wenkai Du cc4de02a86 Add missing gfx942 support (#927) 
[ROCm/rccl commit: c4e65fd382]
 2023-10-23 12:04:37 -07:00
Wenkai Du 6f0f614d0b Remove LDS based software barriers from MSCCL (#923) 
[ROCm/rccl commit: dbb5611a3a]
 2023-10-19 16:39:41 -05:00
Wenkai Du edeea499b5 Update rome models (#922) 
[ROCm/rccl commit: 4278a9918b]
 2023-10-18 17:28:01 -07:00
Wenkai Du e0cc7de446 NPKit: add xcc_id field (#918) 
[ROCm/rccl commit: 39812ce757]
 2023-10-13 15:24:59 -07:00
Wenkai Du c0bd012e6c Fix incorrect arch name parsing (#916) 
[ROCm/rccl commit: 1b80d041cb]
 2023-10-13 10:01:11 -07:00
Wenkai Du 102f0165d6 Port init_once fix from NCCL (#915) 
[ROCm/rccl commit: 6d0b5c1e89]
 2023-10-13 08:01:12 -07:00