7fc891bc8d
Fix MSCCL work FIFO allocation with HIP graph enabled ( #967 )
2023-11-15 16:43:28 -08:00
5a800e00cd
msccl: enable basic collective trace ( #959 )
...
To avoid increasing number of kernels, colltrace is only enabled with
RCCL_MSCCL_FORCE_FULLOPS=1
2023-11-08 20:14:28 -08:00
4c117e5335
Fix MSCCL work FIFO out-of-bound issue ( #935 )
2023-10-27 11:24:52 -07:00
7ee5c1c28b
Change MSCCL kernel signature to allow kernel arguments be preloaded via SGPR ( #911 )
...
* Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
* Only build gfx941
* demo
* fine tune malloc
* Fix merge errors
* Fix merge errors
* Disable parallel build
* Adopt --amdgpu-kernarg-preload-count
* Revert "Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )"
This reverts commit f5e252dddf02a41b4d1bc512f306f45f97166304.
* Revert CMake changes.
* NPKIT changes.
* Remove some license declarations.
* Address code review feedbacks on msccl_kernel_impl.h
* Update CMakeLists.txt
* Add CMake logic to check the existence of --amdgpu-kernarg-preload-count
* Fix NPKIT trace logic.
---------
Co-authored-by: Pedram Alizadeh <pmohamma@amd.com >
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
Co-authored-by: Ziyue Yang <ziyyang@microsoft.com >
2023-10-12 20:17:08 -05:00
e1dc4d5e42
enable hip graph on multi-node ( #884 )
...
* initial checkin
* enable msccl when hip graph is on
* remove the commented out code of msccl enable check
* clean up the code
* remove the msccl HighestTransportType check logic
2023-09-11 15:30:04 -07:00
4297315de7
More robust msccl shared directory location discovery ( #868 )
2023-08-30 08:10:14 -06:00
5bcd3768cc
Minor fix for some msccl installations ( #862 )
2023-08-23 13:48:58 -06:00
64c32d1c5b
Disable MSCCL kernels at compile time ( #834 )
...
* Disable MSCCL kernels at compile time
2023-08-02 09:45:18 -06:00
a7fcd58a97
Enable gfx94x ( #808 ) ( #816 )
...
(cherry picked from commit 94da229a7788d74685d1591a4e75a8341de64f41)
2023-07-21 07:31:27 -07:00
848e60b00c
Fix path finding in msccl internal scheduler ( #794 )
2023-07-12 13:31:28 -07:00
bd14ac8b59
ASAN build excluding additional files, Algodir support for share folder
...
* ASAN build excluding additional files, Algodir support for share folder (#786 )
* Algodir support for share folder
2023-06-23 10:57:20 -04:00
11676267b5
fix min, max and avg ( #745 )
2023-05-18 11:02:59 -07:00
4b09ffba43
msccl: print stack and memory usage ( #723 )
...
* msccl: print stack and memory usage
* Update number of kernels calculation
2023-04-14 14:59:03 -07:00
c8e33b1232
fix msccl stream usage ( #717 )
2023-03-24 10:59:36 -07:00
e3b2342f39
MSCCL: Improve executor and integrate scheduler ( #694 )
...
* MSCCL: improve executor and add scheduler for testing
* Use external scheduler
* Fix cmake error
* Address comments
* Fix thread safe issue
* Make MSCCL lifecycle APIs thread safe
* Make MSCCL internal scheduler aware of topology hint
* Revise error message
2023-03-14 14:34:25 -07:00
adafc0f759
Add MSCCL Support ( #658 )
...
* Add MSCCL support
* Add alignment and message size checking
* Fix nRanks checking, in-place and out-of-place tests and group call handling
* Fix hipGraph unit test
* Change MSCCL init warning to INFO
* Revise license info
2022-12-12 15:51:04 -08:00
Ziyue Yang
Wenkai Du
Wen-Heng (Jack) Chung
Andy li
gilbertlee-amd
Bertan Dogancay
arvindcheru
Ziyue Yang