dd5d66d01a
Use rocm-smi thread only mutex when available ( #1169 )
...
[ROCm/rccl commit: a64aab5f63
2024-05-08 14:32:24 -07:00
dd8e937948
fix typo for DMABUF_ENABLE
...
[ROCm/rccl commit: 140b7dd40f
2024-05-06 13:27:50 -07:00
36f9492cda
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: e1a835910e
2024-04-23 13:34:00 -07:00
da835cff9c
replacing rccl_bfloat16 with hip_bfloat16 ( #1126 )
...
Co-authored-by: mberenjk <mberenjk@amd.com >
[ROCm/rccl commit: 428837ffe4
2024-04-11 11:30:37 -05:00
43bbee4dcc
Remove hipEventDisableSystemFence ( #1122 )
...
There is no indication that disabling system fence has any latency improvement.
Removing it per recommendation from HIP.
[ROCm/rccl commit: 5976f757dd
2024-03-25 08:01:57 -07:00
e373bd44bf
Enable fp8 support ( #1101 )
...
* initial checkin
* resolve cr comments
* resolve the build issue
* fix the data correctless issue
* update fp8 header file and update the unit test for fp8 support
* remove fp16 from fp8 headers
* fix ut issue and catch up the latest code from develop
* udate according to cr comments
* update ut according to cr comments
* update num floats for each SumPostDiv from 4 to 6
* update fp8 header file name
* fix the typo
[ROCm/rccl commit: 6777e65c1d
2024-03-08 15:17:53 -08:00
6dd45024f8
msccl: fix scratch memory allocation after API change ( #1103 )
...
[ROCm/rccl commit: 77615cce28
2024-03-06 11:11:04 -08:00
cee279fd99
Implement ROCTX ( #1094 )
...
* Implement roctx
[ROCm/rccl commit: b617aecc31
2024-02-27 15:46:15 -07:00
4b31894d70
Update RCCL/MSCCL work FIFO depth to 256K ( #1091 )
...
[ROCm/rccl commit: c5ab37211b
2024-02-21 17:15:11 -08:00
2ab8a3a750
2.20.3-1
...
Add support for alternating rings, allow for cross-nic rings without
cross-rail communication.
Add support for user buffer registration for network send/recv.
Optimize aggregated operations to better utilize all channels.
Add flattening for BCM PCI gen5 switches.
Add support for inter-node NVLink communication
Add support for port fusion in NET/IB.
Add support for ReduceScatter and AllGather using Collnet.
Update net API to v8.
Fix hang during A2A connection.
[ROCm/rccl commit: b6475625fb
2024-02-13 04:22:38 -08:00
b17f5947ba
Revert re-usage of connect and listen ports
...
[ROCm/rccl commit: 12ac20ade5
2024-02-01 10:03:13 -08:00
4f4e52aafe
Merge remote-tracking branch 'rccl/develop' into 2.19.4
...
[ROCm/rccl commit: 1a134b283b
2024-01-31 11:53:10 -06:00
404d398bac
Merge remote-tracking branch 'nccl/v2.19' into develop
...
[ROCm/rccl commit: 81ddf9de89
2024-01-24 15:25:33 -08:00
17f70df058
Revert "msccl: build same number of kernels as in ROCm 5.7" ( #1058 )
...
This reverts commit f960174d03be7e5174baa83b256526d388a38842.
[ROCm/rccl commit: 7987015a19
2024-01-24 08:43:50 -08:00
8b8179a689
Use new HIP graph API compatible with CUDA 11030 ( #991 )
...
* Use new HIP graph API compatible with CUDA 11030
* Update dependency to ROCm 6.1
* Fix single stream use case
[ROCm/rccl commit: 7e25d5bc55
2024-01-21 19:00:50 -08:00
1b39fef32a
Fix MSCCL multi-node ( #1032 )
...
1) Move needsProxy initialization before mscclSetupConnections since the latter
will revise it later.
2) Remove mscclAvailable check in net.cc since it's no more required and caused
non-shared buffer allocated for MSCCL which is not expected.
[ROCm/rccl commit: 70bbeb4773
2024-01-05 17:03:43 -08:00
e3d45f9de4
Improve MSCCL algorithms ( #1023 )
...
[ROCm/rccl commit: 0a53077c9c
2024-01-03 14:51:34 -08:00
0c1f773021
rearranged how the min and max functions are part of msccl ( #1025 )
...
* rearranged how the min and max functions are part of msccl
* added more coverage on in place graph tests
[ROCm/rccl commit: f4858e14b2
2023-12-21 08:58:33 -07:00
e4b63a8ba0
Fully disable MSCCL when machine is not matched ( #1017 )
...
* Disable MSCCL algorithm meta loading when machine is not matched
* fully disable init
* fix potential segfault
[ROCm/rccl commit: 655742a3a6
2023-12-13 08:36:21 -08:00
fe250212bf
Fix MSCCL scratch allocation ( #1010 )
...
[ROCm/rccl commit: c002f20029
2023-12-08 17:47:10 -06:00
22730bdc4d
msccl: build same number of kernels as in ROCm 5.7 ( #1005 )
...
Removed fullOps kernels from build
[ROCm/rccl commit: 12c08fc52a
2023-12-07 13:36:04 -06:00
33aa8b67be
Use a map to host scratch buffers ( #1004 )
...
* Use a map to host scratch buffers
* Address review feedbacks. Deliberately keep mscclSetupScratch function.
[ROCm/rccl commit: 293f0fb752
2023-12-05 13:15:28 -06:00
cef45b8311
Fix mscclAlgoHandle not initialized issue ( #995 )
...
[ROCm/rccl commit: e44e112a17
2023-12-01 07:58:01 -08:00
f0c47d085e
Move MSCCL algorithm loading to initialization to workaround HIP graph conflict ( #982 )
...
* MSCCL: pre-specify channels and pre-load algorithms
* add mutex
* fix bug
* clean include
* disable all-gathers temporarily
[ROCm/rccl commit: 4bb0b4a380
2023-11-30 09:47:20 -08:00
bd982864d5
recreated pr 914 to work with current develop branch ( #979 )
...
[ROCm/rccl commit: 56ce9ef05f
2023-11-28 16:33:47 -07:00
4c2fa05a23
msccl: allocate scratch as ext-scope fine-grained ( #968 )
...
[ROCm/rccl commit: 569d3f7d59
2023-11-16 09:57:25 -06:00
6ce074d92d
Fix MSCCL work FIFO allocation with HIP graph enabled ( #967 )
...
[ROCm/rccl commit: 7fc891bc8d
2023-11-15 16:43:28 -08:00
9e8eb41337
Check to support older ROCm versions ( #963 )
...
[ROCm/rccl commit: 198f14923b
2023-11-15 12:36:31 -07:00
2351578d5b
Optimize MSCCL all-gather algorithms for gfx942 ( #964 )
...
[ROCm/rccl commit: 7ae95db5b8
2023-11-15 08:18:59 -08:00
534af85d0f
msccl: enable basic collective trace ( #959 )
...
To avoid increasing number of kernels, colltrace is only enabled with
RCCL_MSCCL_FORCE_FULLOPS=1
[ROCm/rccl commit: 5a800e00cd
2023-11-08 20:14:28 -08:00
1557a1f258
msccl: use 32-bit LDS access and add RCCL_MSCCL_FORCE_FULLOPS ( #953 )
...
[ROCm/rccl commit: bb84345943
2023-11-03 10:38:02 -07:00
4cd86b185c
MSCCL stream fix ( #948 )
...
[ROCm/rccl commit: 988efe605a
2023-11-03 09:10:52 -06:00
297023a7a6
msccl: add templated kernel ( #945 )
...
* msccl: add templated kernel
* Use defines to improve code readability
* Fix kernel indexing and review feedback
[ROCm/rccl commit: f484ff17b9
2023-11-02 17:21:53 -07:00
8b1254a4f1
Fix gcnArchName bug in topology dump ( #937 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
[ROCm/rccl commit: 1e5ca6820b
2023-10-28 12:30:36 -05:00
e1dfb82023
Fix MSCCL work FIFO out-of-bound issue ( #935 )
...
[ROCm/rccl commit: 4c117e5335
2023-10-27 11:24:52 -07:00
706750597c
remove gcnArch support ( #920 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
[ROCm/rccl commit: f22df90e5c
2023-10-26 12:09:15 -05:00
446c8cbf66
msccl: reduce debug output when using NCCL_DEBUG=INFO ( #932 )
...
[ROCm/rccl commit: fb0eccb57b
2023-10-25 08:05:19 -07:00
c0bd012e6c
Fix incorrect arch name parsing ( #916 )
...
[ROCm/rccl commit: 1b80d041cb
2023-10-13 10:01:11 -07:00
102f0165d6
Port init_once fix from NCCL ( #915 )
...
[ROCm/rccl commit: 6d0b5c1e89
2023-10-13 08:01:12 -07:00
dfa0d98f9e
Change MSCCL kernel signature to allow kernel arguments be preloaded via SGPR ( #911 )
...
* Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
* Only build gfx941
* demo
* fine tune malloc
* Fix merge errors
* Fix merge errors
* Disable parallel build
* Adopt --amdgpu-kernarg-preload-count
* Revert "Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )"
This reverts commit f5e252dddf02a41b4d1bc512f306f45f97166304.
* Revert CMake changes.
* NPKIT changes.
* Remove some license declarations.
* Address code review feedbacks on msccl_kernel_impl.h
* Update CMakeLists.txt
* Add CMake logic to check the existence of --amdgpu-kernarg-preload-count
* Fix NPKIT trace logic.
---------
Co-authored-by: Pedram Alizadeh <pmohamma@amd.com >
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
Co-authored-by: Ziyue Yang <ziyyang@microsoft.com >
[ROCm/rccl commit: 7ee5c1c28b
2023-10-12 20:17:08 -05:00
1ffd3eff31
Dma buf support optin ( #905 )
...
* dmaBufSupport Optin added on every part of the code that should invoke it
[ROCm/rccl commit: 28d7fe5629
2023-10-03 03:17:48 -06:00
506d6c332c
2.19.1-1
...
Add local user buffer registration for NVLink SHARP.
Add tuning plugin support.
Increase net API to v7 to allow for device-side packet reordering;
remove support for v4 plugins.
Add support for RoCE ECE.
Add support for C2C links.
Better detect SHM allocation failures to avoid crash with Bus Error.
Fix missing thread unlocks in bootstrap (Fixes #936 ).
Disable network flush by default on H100.
Move device code from src/collectives/device to src/device.
[ROCm/rccl commit: f9c3dc251e
2023-09-26 05:50:33 -07:00
b0a16d80ff
Fix crash when NPKit is enabled ( #887 )
...
[ROCm/rccl commit: 6a4d5ec089
2023-09-13 11:00:12 -07:00
2e3d45a53a
Gcn arch name ( #886 )
...
We use CMake to determine if we're compiling against a version of ROCm that supports gcnArchName and handles architecture checking appropriately. It includes a few helper functions as drop ins for the functionality we used gcnArch for before; sometimes to enable flags, and sometimes to set frequencies.
[ROCm/rccl commit: e58ec78d35
2023-09-12 15:34:40 -04:00
43a9fd00ee
enable hip graph on multi-node ( #884 )
...
* initial checkin
* enable msccl when hip graph is on
* remove the commented out code of msccl enable check
* clean up the code
* remove the msccl HighestTransportType check logic
[ROCm/rccl commit: e1dc4d5e42
2023-09-11 15:30:04 -07:00
2aa31c89df
RCCL should use hipPointerAttribute_t.type ( #872 )
...
[ROCm/rccl commit: 6230b5f6b3
2023-09-05 09:44:12 -06:00
5fe857c562
More robust msccl shared directory location discovery ( #868 )
...
[ROCm/rccl commit: 4297315de7
2023-08-30 08:10:14 -06:00
3dd880fe74
Minor fix for some msccl installations ( #862 )
...
[ROCm/rccl commit: 5bcd3768cc
2023-08-23 13:48:58 -06:00
74dd9c4807
Disable MSCCL kernels at compile time ( #834 )
...
* Disable MSCCL kernels at compile time
[ROCm/rccl commit: 64c32d1c5b
2023-08-02 09:45:18 -06:00
9d9dcd23ce
Fix MSCCL proxy number of chunks calculation ( #821 )
...
Current number of transmissions parsed from MSCCL algorithm is 1-based value,
but when calculating proxy number of chunks, it's taken as 0-based value.
This commit fixes this issue.
[ROCm/rccl commit: f7dc7b7e6a
2023-07-26 13:24:49 -07:00
Wenkai Du
paklui
BertanDogancay
mberenjk
Andy li
Bertan Dogancay
Sylvain Jeaugey
Wenkai Du
Ziyue Yang
akolliasAMD
Wen-Heng (Jack) Chung
Nilesh M Negi
Audrey MP
gilbertlee-amd