569d3f7d59
msccl: allocate scratch as ext-scope fine-grained ( #968 )
2023-11-16 09:57:25 -06:00
15fa77bb57
Update README.md ( #955 )
...
Remove references to HCC, which was removed from ROCm ~2yrs ago
2023-11-15 18:01:45 -08:00
bc8661f092
Fix kernel command line warnings ( #961 )
...
* Fix kernel command line warnings
* Remove while loop
2023-11-15 18:01:12 -08:00
7fc891bc8d
Fix MSCCL work FIFO allocation with HIP graph enabled ( #967 )
2023-11-15 16:43:28 -08:00
198f14923b
Check to support older ROCm versions ( #963 )
2023-11-15 12:36:31 -07:00
7ae95db5b8
Optimize MSCCL all-gather algorithms for gfx942 ( #964 )
2023-11-15 08:18:59 -08:00
df128879a6
Optimize MSCCL reduce primitive switching for gfx942 ( #962 )
...
* Optimize reduce primitive switching for gfx942
* address comment
2023-11-15 08:18:44 -08:00
cece6415b0
Fix use of CPUID overwriting registers in use.
...
CPUID writes to EAX, EBX, ECX, and EDX so the inline-asm must state that.
Otherwise currently in-use register might get overwritten which may
cause all kinds of failures like segfaults or wrong results.
Alternatively `__cpuid` can be used which avoids this and related issues.
So do that as suggested in the GCC issue https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112513
2023-11-14 12:38:02 +01:00
88d44d777f
2.19.4-1
...
Split transport connect phase into multiple steps to avoid port
exhaustion when connecting alltoall at large scale. Defaults to 128
peers per round.
Fix memory leaks on CUDA graph capture.
Fix alltoallv crash on self-sendrecv.
Make topology detection more deterministic when PCI speeds are not
available (fix issue #1020 ).
Properly close shared memory in NVLS resources.
Revert proxy detach after 5 seconds.
Add option to print progress during transport connect.
Add option to set NCCL_DEBUG to INFO on first WARN.
2023-11-13 10:36:12 -08:00
5a800e00cd
msccl: enable basic collective trace ( #959 )
...
To avoid increasing number of kernels, colltrace is only enabled with
RCCL_MSCCL_FORCE_FULLOPS=1
2023-11-08 20:14:28 -08:00
8e0258a73d
Revert "Remove hip::device ( #954 )" ( #956 )
...
This reverts commit 7edb486154
2023-11-07 13:40:41 -07:00
efc42d9045
Use send instead of sendWithBarrier. ( #727 )
2023-11-07 13:47:24 -06:00
7291144c94
Bump rocm-docs-core from 0.26.0 to 0.27.0 in /docs/sphinx ( #951 )
...
Bumps [rocm-docs-core](https://github.com/RadeonOpenCompute/rocm-docs-core ) from 0.26.0 to 0.27.0.
- [Release notes](https://github.com/RadeonOpenCompute/rocm-docs-core/releases )
- [Changelog](https://github.com/RadeonOpenCompute/rocm-docs-core/blob/develop/CHANGELOG.md )
- [Commits](https://github.com/RadeonOpenCompute/rocm-docs-core/compare/v0.26.0...v0.27.0 )
---
updated-dependencies:
- dependency-name: rocm-docs-core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-07 10:01:16 -07:00
022735d208
Merge pull request #950 from nusislam/msccl-red2
...
msccl: remove cases from numReduction switch statement
2023-11-04 02:48:03 -05:00
7edb486154
Remove hip::device ( #954 )
2023-11-03 19:31:00 -06:00
dbcba2923b
Use parallel init of LDS and adjust P2P channels for gfx94x ( #943 )
...
* Use parallel init of LDS and adjust P2P channels for gfx94x
* Move another init to parallel
* Fix NCCL_NCHANNELS_PER_PEER setting
2023-11-03 16:06:49 -07:00
f545b94d4b
msccl: remove cases from numReduction switch statement
2023-11-03 16:56:51 -05:00
d50bab28bf
Adding LaunchBench tool ( #952 )
2023-11-03 12:04:52 -06:00
bb84345943
msccl: use 32-bit LDS access and add RCCL_MSCCL_FORCE_FULLOPS ( #953 )
2023-11-03 10:38:02 -07:00
988efe605a
MSCCL stream fix ( #948 )
2023-11-03 09:10:52 -06:00
f484ff17b9
msccl: add templated kernel ( #945 )
...
* msccl: add templated kernel
* Use defines to improve code readability
* Fix kernel indexing and review feedback
2023-11-02 17:21:53 -07:00
61aed56ca7
Merge pull request #946 from nusislam/msccl-redop
...
msccl: remove dereference of reduce args
2023-11-02 16:22:24 -05:00
6b80a0d0d4
msccl: remove dereference of reduce args
...
It can be removed because the msccl kernel will never execute this code
according to the current msccl setup.
2023-11-02 13:20:00 -05:00
a7400218a2
msccl: use atomic to set dependency flags ( #941 )
2023-10-31 14:46:57 -07:00
9f02ee8dea
Revert "Introduce allgather for MSCCL on 8 sockets up to 320KB. ( #931 )" ( #939 )
...
This reverts commit bfb8642450
2023-10-30 23:52:58 -06:00
a497722894
NPkit: misc fixes for MSCCL ( #936 )
...
* msccl: add xcc_id to timestamp sync
* NPKit: add timestamp for rrc operator
* NPKit: add timestamp for MSCCL init
2023-10-30 10:00:12 -07:00
1e5ca6820b
Fix gcnArchName bug in topology dump ( #937 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2023-10-28 12:30:36 -05:00
4c117e5335
Fix MSCCL work FIFO out-of-bound issue ( #935 )
2023-10-27 11:24:52 -07:00
96ec3ffe2e
SRC/INIT: fix typo for ENABLE_PROFILING ( #934 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2023-10-26 23:52:46 -05:00
f22df90e5c
remove gcnArch support ( #920 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2023-10-26 12:09:15 -05:00
fb0eccb57b
msccl: reduce debug output when using NCCL_DEBUG=INFO ( #932 )
2023-10-25 08:05:19 -07:00
0e35f5d390
Merge tag 'v2.19.3-1'
2023-10-25 06:51:36 -07:00
bfb8642450
Introduce allgather for MSCCL on 8 sockets up to 320KB. ( #931 )
2023-10-24 18:41:12 -05:00
3f9ffe4788
Introduce allgather MSCCL XML specification for MI250X up to 320KB. ( #930 )
2023-10-24 18:35:55 -05:00
72d5fbddfd
Introduce 1-shot allreduce for MI250X Hayabusa. ( #929 )
2023-10-24 16:31:18 -05:00
c4e65fd382
Add missing gfx942 support ( #927 )
2023-10-23 12:04:37 -07:00
d8dc282eeb
AllReduceTests,fixed the number of roots ( #925 )
2023-10-20 10:25:11 -06:00
b173c13831
Bump urllib3 from 1.26.17 to 1.26.18 in /docs/sphinx ( #921 )
...
Bumps [urllib3](https://github.com/urllib3/urllib3 ) from 1.26.17 to 1.26.18.
- [Release notes](https://github.com/urllib3/urllib3/releases )
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst )
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.17...1.26.18 )
---
updated-dependencies:
- dependency-name: urllib3
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-20 10:15:42 -06:00
dd5f01aeaf
Merge pull request #926 from ROCmSoftwarePlatform/searlmc1-patch-1
...
Remove quotes causing asan build breakage
2023-10-20 07:52:39 -07:00
f59de10524
Remove quotes causing asan build breakage
...
The quotes around "-fsanitize=address -shared-libasan" cause the BUILD_ADDRESS_SANITIZER build to fail; remove the quotes
2023-10-19 16:13:39 -07:00
3807c203fc
Update install.sh --fast and README ( #924 )
2023-10-19 16:35:10 -06:00
dbb5611a3a
Remove LDS based software barriers from MSCCL ( #923 )
2023-10-19 16:39:41 -05:00
4278a9918b
Update rome models ( #922 )
2023-10-18 17:28:01 -07:00
341926c60a
Introduce 1pass allreduce. Tailor it for very small message sizes <= 20KB. ( #919 )
2023-10-16 16:31:08 -05:00
39812ce757
NPKit: add xcc_id field ( #918 )
2023-10-13 15:24:59 -07:00
1b80d041cb
Fix incorrect arch name parsing ( #916 )
2023-10-13 10:01:11 -07:00
6d0b5c1e89
Port init_once fix from NCCL ( #915 )
2023-10-13 08:01:12 -07:00
f7e530259d
Bump rocm-docs-core from 0.25.0 to 0.26.0 in /docs/sphinx ( #917 )
...
Bumps [rocm-docs-core](https://github.com/RadeonOpenCompute/rocm-docs-core ) from 0.25.0 to 0.26.0.
- [Release notes](https://github.com/RadeonOpenCompute/rocm-docs-core/releases )
- [Changelog](https://github.com/RadeonOpenCompute/rocm-docs-core/blob/develop/CHANGELOG.md )
- [Commits](https://github.com/RadeonOpenCompute/rocm-docs-core/compare/v0.25.0...v0.26.0 )
---
updated-dependencies:
- dependency-name: rocm-docs-core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-13 08:27:04 -06:00
7ee5c1c28b
Change MSCCL kernel signature to allow kernel arguments be preloaded via SGPR ( #911 )
...
* Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
* Only build gfx941
* demo
* fine tune malloc
* Fix merge errors
* Fix merge errors
* Disable parallel build
* Adopt --amdgpu-kernarg-preload-count
* Revert "Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )"
This reverts commit f5e252dddf02a41b4d1bc512f306f45f97166304.
* Revert CMake changes.
* NPKIT changes.
* Remove some license declarations.
* Address code review feedbacks on msccl_kernel_impl.h
* Update CMakeLists.txt
* Add CMake logic to check the existence of --amdgpu-kernarg-preload-count
* Fix NPKIT trace logic.
---------
Co-authored-by: Pedram Alizadeh <pmohamma@amd.com >
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
Co-authored-by: Ziyue Yang <ziyyang@microsoft.com >
2023-10-12 20:17:08 -05:00
7e2d905376
adding cuda support for EmptyKernelTest ( #913 )
2023-10-11 14:11:12 -05:00
Wenkai Du
searlmc1
Ziyue Yang
Bertan Dogancay
Alexander Grund
Sylvain Jeaugey
Wen-Heng (Jack) Chung
dependabot[bot]
Nusrat Islam
gilbertlee-amd
akolliasAMD
Nilesh M Negi
mberenjk