3e66d42d95
msccl: ensure memory coherence after data receive ( #1062 )
...
[ROCm/rccl commit: f7550d83b8
2024-01-30 08:22:50 -08:00
17f70df058
Revert "msccl: build same number of kernels as in ROCm 5.7" ( #1058 )
...
This reverts commit f960174d03be7e5174baa83b256526d388a38842.
[ROCm/rccl commit: 7987015a19
2024-01-24 08:43:50 -08:00
30f744dc35
msccl: use custom reduce function ( #1033 )
...
[ROCm/rccl commit: d9871d171b
2024-01-08 14:53:12 -08:00
13791d7ee3
Rework barriers and adjust scope of atomics ( #1019 )
...
[ROCm/rccl commit: abf265a911
2024-01-04 08:18:48 -08:00
e3d45f9de4
Improve MSCCL algorithms ( #1023 )
...
[ROCm/rccl commit: 0a53077c9c
2024-01-03 14:51:34 -08:00
0c1f773021
rearranged how the min and max functions are part of msccl ( #1025 )
...
* rearranged how the min and max functions are part of msccl
* added more coverage on in place graph tests
[ROCm/rccl commit: f4858e14b2
2023-12-21 08:58:33 -07:00
48107b18c9
msccl: fix data corruption with MTYPE_RW ( #1014 )
...
[ROCm/rccl commit: 81602814a7
2023-12-11 20:33:15 -08:00
529f72ceaa
Relax workgroup barrier implementation for MSCCL send/recv ops. ( #997 )
...
* Trim logic.
* Revert "Trim logic."
This reverts commit 8f2dba6c764108acf2bf5428366b9f41d4d206b9.
* Introduce MSCCL template parameters to send / recv.
* Address review feedbacks.
[ROCm/rccl commit: baadda4bd8
2023-12-08 17:46:53 -06:00
22730bdc4d
msccl: build same number of kernels as in ROCm 5.7 ( #1005 )
...
Removed fullOps kernels from build
[ROCm/rccl commit: 12c08fc52a
2023-12-07 13:36:04 -06:00
bd982864d5
recreated pr 914 to work with current develop branch ( #979 )
...
[ROCm/rccl commit: 56ce9ef05f
2023-11-28 16:33:47 -07:00
4c2fa05a23
msccl: allocate scratch as ext-scope fine-grained ( #968 )
...
[ROCm/rccl commit: 569d3f7d59
2023-11-16 09:57:25 -06:00
2c6eededec
Optimize MSCCL reduce primitive switching for gfx942 ( #962 )
...
* Optimize reduce primitive switching for gfx942
* address comment
[ROCm/rccl commit: df128879a6
2023-11-15 08:18:44 -08:00
534af85d0f
msccl: enable basic collective trace ( #959 )
...
To avoid increasing number of kernels, colltrace is only enabled with
RCCL_MSCCL_FORCE_FULLOPS=1
[ROCm/rccl commit: 5a800e00cd
2023-11-08 20:14:28 -08:00
270aa41f6b
Use send instead of sendWithBarrier. ( #727 )
...
[ROCm/rccl commit: efc42d9045
2023-11-07 13:47:24 -06:00
83a36c65c1
Merge pull request #950 from nusislam/msccl-red2
...
msccl: remove cases from numReduction switch statement
[ROCm/rccl commit: 022735d208
2023-11-04 02:48:03 -05:00
b6f47bad7c
msccl: remove cases from numReduction switch statement
...
[ROCm/rccl commit: f545b94d4b
2023-11-03 16:56:51 -05:00
1557a1f258
msccl: use 32-bit LDS access and add RCCL_MSCCL_FORCE_FULLOPS ( #953 )
...
[ROCm/rccl commit: bb84345943
2023-11-03 10:38:02 -07:00
297023a7a6
msccl: add templated kernel ( #945 )
...
* msccl: add templated kernel
* Use defines to improve code readability
* Fix kernel indexing and review feedback
[ROCm/rccl commit: f484ff17b9
2023-11-02 17:21:53 -07:00
88c8bb1495
msccl: remove dereference of reduce args
...
It can be removed because the msccl kernel will never execute this code
according to the current msccl setup.
[ROCm/rccl commit: 6b80a0d0d4
2023-11-02 13:20:00 -05:00
3eeaea3f00
msccl: use atomic to set dependency flags ( #941 )
...
[ROCm/rccl commit: a7400218a2
2023-10-31 14:46:57 -07:00
b736b506c0
NPkit: misc fixes for MSCCL ( #936 )
...
* msccl: add xcc_id to timestamp sync
* NPKit: add timestamp for rrc operator
* NPKit: add timestamp for MSCCL init
[ROCm/rccl commit: a497722894
2023-10-30 10:00:12 -07:00
6f0f614d0b
Remove LDS based software barriers from MSCCL ( #923 )
...
[ROCm/rccl commit: dbb5611a3a
2023-10-19 16:39:41 -05:00
e0cc7de446
NPKit: add xcc_id field ( #918 )
...
[ROCm/rccl commit: 39812ce757
2023-10-13 15:24:59 -07:00
dfa0d98f9e
Change MSCCL kernel signature to allow kernel arguments be preloaded via SGPR ( #911 )
...
* Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
* Only build gfx941
* demo
* fine tune malloc
* Fix merge errors
* Fix merge errors
* Disable parallel build
* Adopt --amdgpu-kernarg-preload-count
* Revert "Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )"
This reverts commit f5e252dddf02a41b4d1bc512f306f45f97166304.
* Revert CMake changes.
* NPKIT changes.
* Remove some license declarations.
* Address code review feedbacks on msccl_kernel_impl.h
* Update CMakeLists.txt
* Add CMake logic to check the existence of --amdgpu-kernarg-preload-count
* Fix NPKIT trace logic.
---------
Co-authored-by: Pedram Alizadeh <pmohamma@amd.com >
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
Co-authored-by: Ziyue Yang <ziyyang@microsoft.com >
[ROCm/rccl commit: 7ee5c1c28b
2023-10-12 20:17:08 -05:00
3cc41809dd
Reduce NPKit latency overhead in MSCCL kernel ( #893 )
...
* Reduce NPKit latency overhead in MSCCL kernel
* Fix build error without NPKit enable
[ROCm/rccl commit: 26e982d913
2023-09-15 13:28:26 -07:00
43a9fd00ee
enable hip graph on multi-node ( #884 )
...
* initial checkin
* enable msccl when hip graph is on
* remove the commented out code of msccl enable check
* clean up the code
* remove the msccl HighestTransportType check logic
[ROCm/rccl commit: e1dc4d5e42
2023-09-11 15:30:04 -07:00
ffbfe43500
msccl: add NPKIT profiling for MSCCL send-recv
...
[ROCm/rccl commit: a283f55f12
2023-09-08 13:11:16 -05:00
18811f6159
NPKit update ( #844 )
...
* NPKit update
1. Enable NPKit for MSCCL kernels
2. Fix NPKit context index calculation for sendrecv kernels
* Update build script for npkit
[ROCm/rccl commit: d33a70e620
2023-08-08 17:30:40 -07:00
1ae071944e
improve compilation time and create timetrace plot ( #773 )
...
* improve compilation time and create time-trace plot
* set default value for nproc
[ROCm/rccl commit: f35777e9b0
2023-06-14 09:17:51 -06:00
Wenkai Du
Ziyue Yang
akolliasAMD
Wen-Heng (Jack) Chung
Nusrat Islam
Andy li
Bertan Dogancay