6dd45024f8
msccl: fix scratch memory allocation after API change ( #1103 )
...
[ROCm/rccl commit: 77615cce28
2024-03-06 11:11:04 -08:00
cee279fd99
Implement ROCTX ( #1094 )
...
* Implement roctx
[ROCm/rccl commit: b617aecc31
2024-02-27 15:46:15 -07:00
4b31894d70
Update RCCL/MSCCL work FIFO depth to 256K ( #1091 )
...
[ROCm/rccl commit: c5ab37211b
2024-02-21 17:15:11 -08:00
b17f5947ba
Revert re-usage of connect and listen ports
...
[ROCm/rccl commit: 12ac20ade5
2024-02-01 10:03:13 -08:00
4f4e52aafe
Merge remote-tracking branch 'rccl/develop' into 2.19.4
...
[ROCm/rccl commit: 1a134b283b
2024-01-31 11:53:10 -06:00
404d398bac
Merge remote-tracking branch 'nccl/v2.19' into develop
...
[ROCm/rccl commit: 81ddf9de89
2024-01-24 15:25:33 -08:00
17f70df058
Revert "msccl: build same number of kernels as in ROCm 5.7" ( #1058 )
...
This reverts commit f960174d03be7e5174baa83b256526d388a38842.
[ROCm/rccl commit: 7987015a19
2024-01-24 08:43:50 -08:00
8b8179a689
Use new HIP graph API compatible with CUDA 11030 ( #991 )
...
* Use new HIP graph API compatible with CUDA 11030
* Update dependency to ROCm 6.1
* Fix single stream use case
[ROCm/rccl commit: 7e25d5bc55
2024-01-21 19:00:50 -08:00
1b39fef32a
Fix MSCCL multi-node ( #1032 )
...
1) Move needsProxy initialization before mscclSetupConnections since the latter
will revise it later.
2) Remove mscclAvailable check in net.cc since it's no more required and caused
non-shared buffer allocated for MSCCL which is not expected.
[ROCm/rccl commit: 70bbeb4773
2024-01-05 17:03:43 -08:00
e3d45f9de4
Improve MSCCL algorithms ( #1023 )
...
[ROCm/rccl commit: 0a53077c9c
2024-01-03 14:51:34 -08:00
0c1f773021
rearranged how the min and max functions are part of msccl ( #1025 )
...
* rearranged how the min and max functions are part of msccl
* added more coverage on in place graph tests
[ROCm/rccl commit: f4858e14b2
2023-12-21 08:58:33 -07:00
e4b63a8ba0
Fully disable MSCCL when machine is not matched ( #1017 )
...
* Disable MSCCL algorithm meta loading when machine is not matched
* fully disable init
* fix potential segfault
[ROCm/rccl commit: 655742a3a6
2023-12-13 08:36:21 -08:00
fe250212bf
Fix MSCCL scratch allocation ( #1010 )
...
[ROCm/rccl commit: c002f20029
2023-12-08 17:47:10 -06:00
22730bdc4d
msccl: build same number of kernels as in ROCm 5.7 ( #1005 )
...
Removed fullOps kernels from build
[ROCm/rccl commit: 12c08fc52a
2023-12-07 13:36:04 -06:00
33aa8b67be
Use a map to host scratch buffers ( #1004 )
...
* Use a map to host scratch buffers
* Address review feedbacks. Deliberately keep mscclSetupScratch function.
[ROCm/rccl commit: 293f0fb752
2023-12-05 13:15:28 -06:00
cef45b8311
Fix mscclAlgoHandle not initialized issue ( #995 )
...
[ROCm/rccl commit: e44e112a17
2023-12-01 07:58:01 -08:00
f0c47d085e
Move MSCCL algorithm loading to initialization to workaround HIP graph conflict ( #982 )
...
* MSCCL: pre-specify channels and pre-load algorithms
* add mutex
* fix bug
* clean include
* disable all-gathers temporarily
[ROCm/rccl commit: 4bb0b4a380
2023-11-30 09:47:20 -08:00
bd982864d5
recreated pr 914 to work with current develop branch ( #979 )
...
[ROCm/rccl commit: 56ce9ef05f
2023-11-28 16:33:47 -07:00
4c2fa05a23
msccl: allocate scratch as ext-scope fine-grained ( #968 )
...
[ROCm/rccl commit: 569d3f7d59
2023-11-16 09:57:25 -06:00
6ce074d92d
Fix MSCCL work FIFO allocation with HIP graph enabled ( #967 )
...
[ROCm/rccl commit: 7fc891bc8d
2023-11-15 16:43:28 -08:00
9e8eb41337
Check to support older ROCm versions ( #963 )
...
[ROCm/rccl commit: 198f14923b
2023-11-15 12:36:31 -07:00
2351578d5b
Optimize MSCCL all-gather algorithms for gfx942 ( #964 )
...
[ROCm/rccl commit: 7ae95db5b8
2023-11-15 08:18:59 -08:00
534af85d0f
msccl: enable basic collective trace ( #959 )
...
To avoid increasing number of kernels, colltrace is only enabled with
RCCL_MSCCL_FORCE_FULLOPS=1
[ROCm/rccl commit: 5a800e00cd
2023-11-08 20:14:28 -08:00
1557a1f258
msccl: use 32-bit LDS access and add RCCL_MSCCL_FORCE_FULLOPS ( #953 )
...
[ROCm/rccl commit: bb84345943
2023-11-03 10:38:02 -07:00
4cd86b185c
MSCCL stream fix ( #948 )
...
[ROCm/rccl commit: 988efe605a
2023-11-03 09:10:52 -06:00
297023a7a6
msccl: add templated kernel ( #945 )
...
* msccl: add templated kernel
* Use defines to improve code readability
* Fix kernel indexing and review feedback
[ROCm/rccl commit: f484ff17b9
2023-11-02 17:21:53 -07:00
8b1254a4f1
Fix gcnArchName bug in topology dump ( #937 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
[ROCm/rccl commit: 1e5ca6820b
2023-10-28 12:30:36 -05:00
e1dfb82023
Fix MSCCL work FIFO out-of-bound issue ( #935 )
...
[ROCm/rccl commit: 4c117e5335
2023-10-27 11:24:52 -07:00
706750597c
remove gcnArch support ( #920 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
[ROCm/rccl commit: f22df90e5c
2023-10-26 12:09:15 -05:00
446c8cbf66
msccl: reduce debug output when using NCCL_DEBUG=INFO ( #932 )
...
[ROCm/rccl commit: fb0eccb57b
2023-10-25 08:05:19 -07:00
c0bd012e6c
Fix incorrect arch name parsing ( #916 )
...
[ROCm/rccl commit: 1b80d041cb
2023-10-13 10:01:11 -07:00
102f0165d6
Port init_once fix from NCCL ( #915 )
...
[ROCm/rccl commit: 6d0b5c1e89
2023-10-13 08:01:12 -07:00
dfa0d98f9e
Change MSCCL kernel signature to allow kernel arguments be preloaded via SGPR ( #911 )
...
* Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
* Only build gfx941
* demo
* fine tune malloc
* Fix merge errors
* Fix merge errors
* Disable parallel build
* Adopt --amdgpu-kernarg-preload-count
* Revert "Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )"
This reverts commit f5e252dddf02a41b4d1bc512f306f45f97166304.
* Revert CMake changes.
* NPKIT changes.
* Remove some license declarations.
* Address code review feedbacks on msccl_kernel_impl.h
* Update CMakeLists.txt
* Add CMake logic to check the existence of --amdgpu-kernarg-preload-count
* Fix NPKIT trace logic.
---------
Co-authored-by: Pedram Alizadeh <pmohamma@amd.com >
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
Co-authored-by: Ziyue Yang <ziyyang@microsoft.com >
[ROCm/rccl commit: 7ee5c1c28b
2023-10-12 20:17:08 -05:00
1ffd3eff31
Dma buf support optin ( #905 )
...
* dmaBufSupport Optin added on every part of the code that should invoke it
[ROCm/rccl commit: 28d7fe5629
2023-10-03 03:17:48 -06:00
506d6c332c
2.19.1-1
...
Add local user buffer registration for NVLink SHARP.
Add tuning plugin support.
Increase net API to v7 to allow for device-side packet reordering;
remove support for v4 plugins.
Add support for RoCE ECE.
Add support for C2C links.
Better detect SHM allocation failures to avoid crash with Bus Error.
Fix missing thread unlocks in bootstrap (Fixes #936 ).
Disable network flush by default on H100.
Move device code from src/collectives/device to src/device.
[ROCm/rccl commit: f9c3dc251e
2023-09-26 05:50:33 -07:00
b0a16d80ff
Fix crash when NPKit is enabled ( #887 )
...
[ROCm/rccl commit: 6a4d5ec089
2023-09-13 11:00:12 -07:00
2e3d45a53a
Gcn arch name ( #886 )
...
We use CMake to determine if we're compiling against a version of ROCm that supports gcnArchName and handles architecture checking appropriately. It includes a few helper functions as drop ins for the functionality we used gcnArch for before; sometimes to enable flags, and sometimes to set frequencies.
[ROCm/rccl commit: e58ec78d35
2023-09-12 15:34:40 -04:00
43a9fd00ee
enable hip graph on multi-node ( #884 )
...
* initial checkin
* enable msccl when hip graph is on
* remove the commented out code of msccl enable check
* clean up the code
* remove the msccl HighestTransportType check logic
[ROCm/rccl commit: e1dc4d5e42
2023-09-11 15:30:04 -07:00
2aa31c89df
RCCL should use hipPointerAttribute_t.type ( #872 )
...
[ROCm/rccl commit: 6230b5f6b3
2023-09-05 09:44:12 -06:00
5fe857c562
More robust msccl shared directory location discovery ( #868 )
...
[ROCm/rccl commit: 4297315de7
2023-08-30 08:10:14 -06:00
3dd880fe74
Minor fix for some msccl installations ( #862 )
...
[ROCm/rccl commit: 5bcd3768cc
2023-08-23 13:48:58 -06:00
74dd9c4807
Disable MSCCL kernels at compile time ( #834 )
...
* Disable MSCCL kernels at compile time
[ROCm/rccl commit: 64c32d1c5b
2023-08-02 09:45:18 -06:00
9d9dcd23ce
Fix MSCCL proxy number of chunks calculation ( #821 )
...
Current number of transmissions parsed from MSCCL algorithm is 1-based value,
but when calculating proxy number of chunks, it's taken as 0-based value.
This commit fixes this issue.
[ROCm/rccl commit: f7dc7b7e6a
2023-07-26 13:24:49 -07:00
dfda1d6fab
Enable gfx94x ( #808 ) ( #816 )
...
(cherry picked from commit 94da229a7788d74685d1591a4e75a8341de64f41)
[ROCm/rccl commit: a7fcd58a97
2023-07-21 07:31:27 -07:00
17926777ff
Fix path finding in msccl internal scheduler ( #794 )
...
[ROCm/rccl commit: 848e60b00c
2023-07-12 13:31:28 -07:00
9b2475ef06
Merge pull request #782 from ROCmSoftwarePlatform/2.18.3
...
Sync up with NCCL 2.18.3
[ROCm/rccl commit: ce6a2ffac8
2023-06-29 15:04:16 -07:00
f98715baea
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: abd0615351
2023-06-26 22:51:56 +00:00
1607567e42
ASAN build excluding additional files, Algodir support for share folder
...
* ASAN build excluding additional files, Algodir support for share folder (#786 )
* Algodir support for share folder
[ROCm/rccl commit: bd14ac8b59
2023-06-23 10:57:20 -04:00
225e871de2
fixed npkit size to never be a negative number ( #779 )
...
[ROCm/rccl commit: 9bdf6797a5
2023-06-21 08:26:40 -06:00
2dc2c86ec1
2.18.3-1
...
Fix data corruption with Tree/LL128 on systems with 1GPU:1NIC.
Fix hang with Collnet on bfloat16 on systems with less than one NIC
per GPU.
Fix long initialization time.
Fix data corruption with Collnet when mixing multi-process and
multi-GPU per process.
Fix crash when shared memory creation fails.
Fix Avg operation with Collnet/Chain.
Fix performance of alltoall at scale with more than one NIC per GPU.
Fix performance for DGX H800.
Fix race condition in connection progress causing a crash.
Fix network flush with Collnet.
Fix performance of aggregated allGather/reduceScatter operations.
Fix PXN operation when CUDA_VISIBLE_DEVICES is set.
Fix NVTX3 compilation issues on Debian 10.
[ROCm/rccl commit: ea38312273
2023-06-14 01:29:17 -07:00
Wenkai Du
Bertan Dogancay
BertanDogancay
Wenkai Du
Ziyue Yang
akolliasAMD
Wen-Heng (Jack) Chung
Nilesh M Negi
Sylvain Jeaugey
Audrey MP
Andy li
gilbertlee-amd
arvindcheru