edeea499b5
Update rome models ( #922 )
...
[ROCm/rccl commit: 4278a9918b
2023-10-18 17:28:01 -07:00
e0cc7de446
NPKit: add xcc_id field ( #918 )
...
[ROCm/rccl commit: 39812ce757
2023-10-13 15:24:59 -07:00
c0bd012e6c
Fix incorrect arch name parsing ( #916 )
...
[ROCm/rccl commit: 1b80d041cb
2023-10-13 10:01:11 -07:00
102f0165d6
Port init_once fix from NCCL ( #915 )
...
[ROCm/rccl commit: 6d0b5c1e89
2023-10-13 08:01:12 -07:00
dfa0d98f9e
Change MSCCL kernel signature to allow kernel arguments be preloaded via SGPR ( #911 )
...
* Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
* Only build gfx941
* demo
* fine tune malloc
* Fix merge errors
* Fix merge errors
* Disable parallel build
* Adopt --amdgpu-kernarg-preload-count
* Revert "Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )"
This reverts commit f5e252dddf02a41b4d1bc512f306f45f97166304.
* Revert CMake changes.
* NPKIT changes.
* Remove some license declarations.
* Address code review feedbacks on msccl_kernel_impl.h
* Update CMakeLists.txt
* Add CMake logic to check the existence of --amdgpu-kernarg-preload-count
* Fix NPKIT trace logic.
---------
Co-authored-by: Pedram Alizadeh <pmohamma@amd.com >
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
Co-authored-by: Ziyue Yang <ziyyang@microsoft.com >
[ROCm/rccl commit: 7ee5c1c28b
2023-10-12 20:17:08 -05:00
6f7965796f
Revert "Remove 2H4P condition from P2P channels adjustment ( #890 )" ( #904 )
...
This reverts commit 057e30e705a6ff4618c7
2023-10-04 09:46:11 -06:00
1ffd3eff31
Dma buf support optin ( #905 )
...
* dmaBufSupport Optin added on every part of the code that should invoke it
[ROCm/rccl commit: 28d7fe5629
2023-10-03 03:17:48 -06:00
b35ea4bd78
Modify All-To-All doc ( #896 )
...
* Modify All-To-All doc
* Update nccl.h.in
* update unit-tests
---------
Co-authored-by: gilbertlee-amd <44450918+gilbertlee-amd@users.noreply.github.com >
[ROCm/rccl commit: c1f57a7041
2023-09-27 12:45:21 -04:00
69ee68b6d3
2.19.3-1
...
H800/H100 fixes and tuning.
Re-enable intra-process direct pointer buffer access when CUMEM is
enabled.
[ROCm/rccl commit: 8c6c595185
2023-09-26 05:57:15 -07:00
216dfe3d55
Merge remote-tracking branch 'origin/master' into v2.19
...
[ROCm/rccl commit: 3435178b6c
2023-09-26 05:55:56 -07:00
506d6c332c
2.19.1-1
...
Add local user buffer registration for NVLink SHARP.
Add tuning plugin support.
Increase net API to v7 to allow for device-side packet reordering;
remove support for v4 plugins.
Add support for RoCE ECE.
Add support for C2C links.
Better detect SHM allocation failures to avoid crash with Bus Error.
Fix missing thread unlocks in bootstrap (Fixes #936 ).
Disable network flush by default on H100.
Move device code from src/collectives/device to src/device.
[ROCm/rccl commit: f9c3dc251e
2023-09-26 05:50:33 -07:00
d03ae00bac
Fix cudaMemcpyAsync bug
...
We are trying to use the copy result of first cudaMemcpyAsync in the
second cudaMemcpyAsync without sync in between. This patch fixes it
by allocating a CPU side array to cache device side addr so that we
can avoid this consecutive cuda mem copy.
Fixes #957
[ROCm/rccl commit: 4365458757
2023-09-20 05:51:14 -07:00
6f7eb65308
changed the form that RCCL_TREE uses ( #888 )
...
* changed the form that RCCL_TREE uses
[ROCm/rccl commit: b85d73c02e
2023-09-15 15:01:33 -06:00
3cc41809dd
Reduce NPKit latency overhead in MSCCL kernel ( #893 )
...
* Reduce NPKit latency overhead in MSCCL kernel
* Fix build error without NPKit enable
[ROCm/rccl commit: 26e982d913
2023-09-15 13:28:26 -07:00
057e30e705
Remove 2H4P condition from P2P channels adjustment ( #890 )
...
[ROCm/rccl commit: 16dd05a58a
2023-09-13 12:54:21 -07:00
6d593761dc
Add single-node MI300X topology ( #889 )
...
[ROCm/rccl commit: c1bfd5f0d8
2023-09-13 11:07:17 -07:00
b0a16d80ff
Fix crash when NPKit is enabled ( #887 )
...
[ROCm/rccl commit: 6a4d5ec089
2023-09-13 11:00:12 -07:00
2e3d45a53a
Gcn arch name ( #886 )
...
We use CMake to determine if we're compiling against a version of ROCm that supports gcnArchName and handles architecture checking appropriately. It includes a few helper functions as drop ins for the functionality we used gcnArch for before; sometimes to enable flags, and sometimes to set frequencies.
[ROCm/rccl commit: e58ec78d35
2023-09-12 15:34:40 -04:00
43a9fd00ee
enable hip graph on multi-node ( #884 )
...
* initial checkin
* enable msccl when hip graph is on
* remove the commented out code of msccl enable check
* clean up the code
* remove the msccl HighestTransportType check logic
[ROCm/rccl commit: e1dc4d5e42
2023-09-11 15:30:04 -07:00
ffbfe43500
msccl: add NPKIT profiling for MSCCL send-recv
...
[ROCm/rccl commit: a283f55f12
2023-09-08 13:11:16 -05:00
2aa31c89df
RCCL should use hipPointerAttribute_t.type ( #872 )
...
[ROCm/rccl commit: 6230b5f6b3
2023-09-05 09:44:12 -06:00
5fe857c562
More robust msccl shared directory location discovery ( #868 )
...
[ROCm/rccl commit: 4297315de7
2023-08-30 08:10:14 -06:00
b4f96a23e6
optimizing COLL_UNROLL for MI100 machines ( #863 )
...
[ROCm/rccl commit: e7f27c66e0
2023-08-29 12:49:06 -04:00
af04103d72
Add MSCCL xml files ( #861 )
...
[ROCm/rccl commit: aeca1af374
2023-08-23 14:12:34 -07:00
3dd880fe74
Minor fix for some msccl installations ( #862 )
...
[ROCm/rccl commit: 5bcd3768cc
2023-08-23 13:48:58 -06:00
26b91b9dbb
2.18.5-1
...
Fix NVLS search (issue #931 ).
Increase max IB NICs to 32.
Fix inconsistent device ordering (issue #820 ).
Try to use different devices for different GPUs in systems with
more than one NIC per GFU.
[ROCm/rccl commit: 559b70f86c
2023-08-23 06:32:36 -07:00
5983f0e371
Use relaxed atomics for LL on GFX11 ( #859 )
...
[ROCm/rccl commit: 6a0a6a37d9
2023-08-21 16:28:39 -07:00
75e3927f50
Fix static_assert string literal that contains a "\%". This is no longer ( #860 )
...
valid. They can only be simple escape sequences. Removing '\' fixes
issue. Assert message now compiles and emits the '%' as expected.
[ROCm/rccl commit: 2ec2648247
2023-08-21 16:19:59 -07:00
56129830a6
NCCL_TREES variable and rome model fixes ( #856 )
...
[ROCm/rccl commit: d33cd5a233
2023-08-21 10:35:37 -06:00
6fdb4103b7
gfx11: don't use LL for sendrecv ( #853 )
...
* gfx11: don't use LL for sendrecv
* Use builtin instead of inline asm
[ROCm/rccl commit: f70e3e569b
2023-08-17 08:50:51 -07:00
0c31452135
Add new model support ( #847 )
...
* Add new model support
* Update new rings
[ROCm/rccl commit: 7044599575
2023-08-10 17:14:51 -07:00
18811f6159
NPKit update ( #844 )
...
* NPKit update
1. Enable NPKit for MSCCL kernels
2. Fix NPKit context index calculation for sendrecv kernels
* Update build script for npkit
[ROCm/rccl commit: d33a70e620
2023-08-08 17:30:40 -07:00
cb291157f4
Detect HIP_UNCACHED_MEMORY support from HIP version ( #842 )
...
[ROCm/rccl commit: d65c0830c6
2023-08-04 10:17:04 -07:00
c328294820
gfx11xx: disable LL protocol to workaround mtype issue ( #840 )
...
[ROCm/rccl commit: 8e58b65873
2023-08-04 07:53:07 -07:00
c419755576
Fix merge error and replace inline asm ( #838 )
...
[ROCm/rccl commit: 60efe26549
2023-08-03 13:46:40 -07:00
c0729f28c8
Improve collective trace ( #835 )
...
[ROCm/rccl commit: c8085eb704
2023-08-03 07:16:12 -07:00
74dd9c4807
Disable MSCCL kernels at compile time ( #834 )
...
* Disable MSCCL kernels at compile time
[ROCm/rccl commit: 64c32d1c5b
2023-08-02 09:45:18 -06:00
020800d7b7
Fix inter-node NVLS graph search
...
We were passing a net ID instead of a gpu index, which could cause
crashes if those were unrelated (and they usually are).
Issue #931
[ROCm/rccl commit: 8ed014bae9
2023-08-02 07:06:35 -07:00
1066eb08c5
Updating Doxygen documentation ( #831 )
...
[ROCm/rccl commit: afa4a5ecf8
2023-07-28 16:09:06 -06:00
b0255eb9fb
Revert "Enable Ll128 on gfx90a ( #823 )" ( #829 )
...
This reverts commit d80fba94e93db371c9a5
2023-07-27 20:25:18 -07:00
9d9dcd23ce
Fix MSCCL proxy number of chunks calculation ( #821 )
...
Current number of transmissions parsed from MSCCL algorithm is 1-based value,
but when calculating proxy number of chunks, it's taken as 0-based value.
This commit fixes this issue.
[ROCm/rccl commit: f7dc7b7e6a
2023-07-26 13:24:49 -07:00
d80fba94e9
Enable Ll128 on gfx90a ( #823 )
...
[ROCm/rccl commit: 420f8af6a0
2023-07-26 11:44:15 -07:00
c64fd04c11
Replace atomicExch with __atomic_store_n ( #818 )
...
* Replace atomicExch with __atomic_store_n
* Remove extra semicolon
[ROCm/rccl commit: 4d20b4b758
2023-07-25 11:15:21 -07:00
435a055c97
Merge pull request #810 from nusislam/tune-send-recv
...
device: fine-tune RCCL send-recv on MI250/MI200
[ROCm/rccl commit: 47f754e6f5
2023-07-25 10:18:12 -05:00
0508fa569a
Implement RCCL Replayer ( #817 )
...
* Implement RCCL Replayer
[ROCm/rccl commit: 8bab4f04b7
2023-07-24 16:26:22 -06:00
dfda1d6fab
Enable gfx94x ( #808 ) ( #816 )
...
(cherry picked from commit 94da229a7788d74685d1591a4e75a8341de64f41)
[ROCm/rccl commit: a7fcd58a97
2023-07-21 07:31:27 -07:00
c02f5de7e7
device: fine-tune RCCL send-recv
...
[ROCm/rccl commit: b2a0a0bd3a
2023-07-17 21:45:52 -05:00
17926777ff
Fix path finding in msccl internal scheduler ( #794 )
...
[ROCm/rccl commit: 848e60b00c
2023-07-12 13:31:28 -07:00
3ce8711153
npkit: separate network timing between send and test ( #798 )
...
[ROCm/rccl commit: 0f14e5a640
2023-07-10 09:31:49 -07:00
aa8a5f3b2a
device: fine tune MI200/MI250 simple protocol performance
...
With Simple protocol, unroll factor of 4 offers better
performance for most of the collectives (on MI200. MI250, and
MI300) except large message allreduce with Ring algorithm
on MI250 and MI200). This PR changes the default unroll factor
to 4 while adding fine tuning for reduction operations.
[ROCm/rccl commit: 58e53dfd37
2023-07-08 20:21:18 -05:00
Wenkai Du
Wen-Heng (Jack) Chung
Bertan Dogancay
akolliasAMD
Sylvain Jeaugey
Kaiming Ouyang
Ziyue Yang
Audrey MP
Andy li
Nusrat Islam
gilbertlee-amd
Pedram Alizadeh
David Pagan