f0c47d085e
Move MSCCL algorithm loading to initialization to workaround HIP graph conflict ( #982 )
...
* MSCCL: pre-specify channels and pre-load algorithms
* add mutex
* fix bug
* clean include
* disable all-gathers temporarily
[ROCm/rccl commit: 4bb0b4a380
2023-11-30 09:47:20 -08:00
bd982864d5
recreated pr 914 to work with current develop branch ( #979 )
...
[ROCm/rccl commit: 56ce9ef05f
2023-11-28 16:33:47 -07:00
6ce074d92d
Fix MSCCL work FIFO allocation with HIP graph enabled ( #967 )
...
[ROCm/rccl commit: 7fc891bc8d
2023-11-15 16:43:28 -08:00
534af85d0f
msccl: enable basic collective trace ( #959 )
...
To avoid increasing number of kernels, colltrace is only enabled with
RCCL_MSCCL_FORCE_FULLOPS=1
[ROCm/rccl commit: 5a800e00cd
2023-11-08 20:14:28 -08:00
297023a7a6
msccl: add templated kernel ( #945 )
...
* msccl: add templated kernel
* Use defines to improve code readability
* Fix kernel indexing and review feedback
[ROCm/rccl commit: f484ff17b9
2023-11-02 17:21:53 -07:00
e1dfb82023
Fix MSCCL work FIFO out-of-bound issue ( #935 )
...
[ROCm/rccl commit: 4c117e5335
2023-10-27 11:24:52 -07:00
dfa0d98f9e
Change MSCCL kernel signature to allow kernel arguments be preloaded via SGPR ( #911 )
...
* Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
* Only build gfx941
* demo
* fine tune malloc
* Fix merge errors
* Fix merge errors
* Disable parallel build
* Adopt --amdgpu-kernarg-preload-count
* Revert "Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )"
This reverts commit f5e252dddf02a41b4d1bc512f306f45f97166304.
* Revert CMake changes.
* NPKIT changes.
* Remove some license declarations.
* Address code review feedbacks on msccl_kernel_impl.h
* Update CMakeLists.txt
* Add CMake logic to check the existence of --amdgpu-kernarg-preload-count
* Fix NPKIT trace logic.
---------
Co-authored-by: Pedram Alizadeh <pmohamma@amd.com >
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
Co-authored-by: Ziyue Yang <ziyyang@microsoft.com >
[ROCm/rccl commit: 7ee5c1c28b
2023-10-12 20:17:08 -05:00
43a9fd00ee
enable hip graph on multi-node ( #884 )
...
* initial checkin
* enable msccl when hip graph is on
* remove the commented out code of msccl enable check
* clean up the code
* remove the msccl HighestTransportType check logic
[ROCm/rccl commit: e1dc4d5e42
2023-09-11 15:30:04 -07:00
75e3927f50
Fix static_assert string literal that contains a "\%". This is no longer ( #860 )
...
valid. They can only be simple escape sequences. Removing '\' fixes
issue. Assert message now compiles and emits the '%' as expected.
[ROCm/rccl commit: 2ec2648247
2023-08-21 16:19:59 -07:00
73c6d51454
fix min, max and avg ( #745 )
...
[ROCm/rccl commit: 11676267b5
2023-05-18 11:02:59 -07:00
1af52edd6a
msccl: print stack and memory usage ( #723 )
...
* msccl: print stack and memory usage
* Update number of kernels calculation
[ROCm/rccl commit: 4b09ffba43
2023-04-14 14:59:03 -07:00
f7f669e7f0
MSCCL: Improve executor and integrate scheduler ( #694 )
...
* MSCCL: improve executor and add scheduler for testing
* Use external scheduler
* Fix cmake error
* Address comments
* Fix thread safe issue
* Make MSCCL lifecycle APIs thread safe
* Make MSCCL internal scheduler aware of topology hint
* Revise error message
[ROCm/rccl commit: e3b2342f39
2023-03-14 14:34:25 -07:00
431c7d2cb9
Add MSCCL Support ( #658 )
...
* Add MSCCL support
* Add alignment and message size checking
* Fix nRanks checking, in-place and out-of-place tests and group call handling
* Fix hipGraph unit test
* Change MSCCL init warning to INFO
* Revise license info
[ROCm/rccl commit: adafc0f759
2022-12-12 15:51:04 -08:00
Ziyue Yang
akolliasAMD
Wenkai Du
Wen-Heng (Jack) Chung
Andy li
David Pagan