94910b8f80
Fix bug where the first collective call was using MSCCL instead of MSCCL++ ( #1260 )
...
[ROCm/rccl commit: 69135976d6
2024-07-22 15:46:47 -06:00
37bf54b8f8
Enable multi-threading for MSCCL ( #1203 )
...
MSCCL can now run in a multi-threaded configuration. To test in the unit tests, added the ENABLE_OPENMP compile definition flag and the --openmp-test-enable flag to the unit test build script. To activate, set the environment variables UT_MULTITHREADED=1 and UT_PROCESS_MASK=1. Set Jenkins to use this mode.
[ROCm/rccl commit: 0c36d571ea
2024-07-04 09:34:38 -06:00
da835cff9c
replacing rccl_bfloat16 with hip_bfloat16 ( #1126 )
...
Co-authored-by: mberenjk <mberenjk@amd.com >
[ROCm/rccl commit: 428837ffe4
2024-04-11 11:30:37 -05:00
e373bd44bf
Enable fp8 support ( #1101 )
...
* initial checkin
* resolve cr comments
* resolve the build issue
* fix the data correctless issue
* update fp8 header file and update the unit test for fp8 support
* remove fp16 from fp8 headers
* fix ut issue and catch up the latest code from develop
* udate according to cr comments
* update ut according to cr comments
* update num floats for each SumPostDiv from 4 to 6
* update fp8 header file name
* fix the typo
[ROCm/rccl commit: 6777e65c1d
2024-03-08 15:17:53 -08:00
4f4e52aafe
Merge remote-tracking branch 'rccl/develop' into 2.19.4
...
[ROCm/rccl commit: 1a134b283b
2024-01-31 11:53:10 -06:00
404d398bac
Merge remote-tracking branch 'nccl/v2.19' into develop
...
[ROCm/rccl commit: 81ddf9de89
2024-01-24 15:25:33 -08:00
17f70df058
Revert "msccl: build same number of kernels as in ROCm 5.7" ( #1058 )
...
This reverts commit f960174d03be7e5174baa83b256526d388a38842.
[ROCm/rccl commit: 7987015a19
2024-01-24 08:43:50 -08:00
cec06d59d1
Un-escaped character causes error with address sanitizer builds ( #992 )
...
Signed-off-by: Nilesh M Negi <Nilesh.Negi@amd.com >
Co-authored-by: Jenkins <jenkins-compute@amd.com >
[ROCm/rccl commit: 249e9f7f65
2024-01-09 13:28:32 -06:00
0c1f773021
rearranged how the min and max functions are part of msccl ( #1025 )
...
* rearranged how the min and max functions are part of msccl
* added more coverage on in place graph tests
[ROCm/rccl commit: f4858e14b2
2023-12-21 08:58:33 -07:00
e4b63a8ba0
Fully disable MSCCL when machine is not matched ( #1017 )
...
* Disable MSCCL algorithm meta loading when machine is not matched
* fully disable init
* fix potential segfault
[ROCm/rccl commit: 655742a3a6
2023-12-13 08:36:21 -08:00
22730bdc4d
msccl: build same number of kernels as in ROCm 5.7 ( #1005 )
...
Removed fullOps kernels from build
[ROCm/rccl commit: 12c08fc52a
2023-12-07 13:36:04 -06:00
33aa8b67be
Use a map to host scratch buffers ( #1004 )
...
* Use a map to host scratch buffers
* Address review feedbacks. Deliberately keep mscclSetupScratch function.
[ROCm/rccl commit: 293f0fb752
2023-12-05 13:15:28 -06:00
f0c47d085e
Move MSCCL algorithm loading to initialization to workaround HIP graph conflict ( #982 )
...
* MSCCL: pre-specify channels and pre-load algorithms
* add mutex
* fix bug
* clean include
* disable all-gathers temporarily
[ROCm/rccl commit: 4bb0b4a380
2023-11-30 09:47:20 -08:00
bd982864d5
recreated pr 914 to work with current develop branch ( #979 )
...
[ROCm/rccl commit: 56ce9ef05f
2023-11-28 16:33:47 -07:00
6ce074d92d
Fix MSCCL work FIFO allocation with HIP graph enabled ( #967 )
...
[ROCm/rccl commit: 7fc891bc8d
2023-11-15 16:43:28 -08:00
534af85d0f
msccl: enable basic collective trace ( #959 )
...
To avoid increasing number of kernels, colltrace is only enabled with
RCCL_MSCCL_FORCE_FULLOPS=1
[ROCm/rccl commit: 5a800e00cd
2023-11-08 20:14:28 -08:00
297023a7a6
msccl: add templated kernel ( #945 )
...
* msccl: add templated kernel
* Use defines to improve code readability
* Fix kernel indexing and review feedback
[ROCm/rccl commit: f484ff17b9
2023-11-02 17:21:53 -07:00
e1dfb82023
Fix MSCCL work FIFO out-of-bound issue ( #935 )
...
[ROCm/rccl commit: 4c117e5335
2023-10-27 11:24:52 -07:00
dfa0d98f9e
Change MSCCL kernel signature to allow kernel arguments be preloaded via SGPR ( #911 )
...
* Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
* Only build gfx941
* demo
* fine tune malloc
* Fix merge errors
* Fix merge errors
* Disable parallel build
* Adopt --amdgpu-kernarg-preload-count
* Revert "Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )"
This reverts commit f5e252dddf02a41b4d1bc512f306f45f97166304.
* Revert CMake changes.
* NPKIT changes.
* Remove some license declarations.
* Address code review feedbacks on msccl_kernel_impl.h
* Update CMakeLists.txt
* Add CMake logic to check the existence of --amdgpu-kernarg-preload-count
* Fix NPKIT trace logic.
---------
Co-authored-by: Pedram Alizadeh <pmohamma@amd.com >
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
Co-authored-by: Ziyue Yang <ziyyang@microsoft.com >
[ROCm/rccl commit: 7ee5c1c28b
2023-10-12 20:17:08 -05:00
43a9fd00ee
enable hip graph on multi-node ( #884 )
...
* initial checkin
* enable msccl when hip graph is on
* remove the commented out code of msccl enable check
* clean up the code
* remove the msccl HighestTransportType check logic
[ROCm/rccl commit: e1dc4d5e42
2023-09-11 15:30:04 -07:00
75e3927f50
Fix static_assert string literal that contains a "\%". This is no longer ( #860 )
...
valid. They can only be simple escape sequences. Removing '\' fixes
issue. Assert message now compiles and emits the '%' as expected.
[ROCm/rccl commit: 2ec2648247
2023-08-21 16:19:59 -07:00
73c6d51454
fix min, max and avg ( #745 )
...
[ROCm/rccl commit: 11676267b5
2023-05-18 11:02:59 -07:00
1af52edd6a
msccl: print stack and memory usage ( #723 )
...
* msccl: print stack and memory usage
* Update number of kernels calculation
[ROCm/rccl commit: 4b09ffba43
2023-04-14 14:59:03 -07:00
f7f669e7f0
MSCCL: Improve executor and integrate scheduler ( #694 )
...
* MSCCL: improve executor and add scheduler for testing
* Use external scheduler
* Fix cmake error
* Address comments
* Fix thread safe issue
* Make MSCCL lifecycle APIs thread safe
* Make MSCCL internal scheduler aware of topology hint
* Revise error message
[ROCm/rccl commit: e3b2342f39
2023-03-14 14:34:25 -07:00
431c7d2cb9
Add MSCCL Support ( #658 )
...
* Add MSCCL support
* Add alignment and message size checking
* Fix nRanks checking, in-place and out-of-place tests and group call handling
* Fix hipGraph unit test
* Change MSCCL init warning to INFO
* Revise license info
[ROCm/rccl commit: adafc0f759
2022-12-12 15:51:04 -08:00
corey-derochie-amd
mberenjk
Andy li
Wenkai Du
BertanDogancay
Wenkai Du
Nilesh M Negi
akolliasAMD
Ziyue Yang
Wen-Heng (Jack) Chung
David Pagan