c4e65fd382
Add missing gfx942 support ( #927 )
2023-10-23 12:04:37 -07:00
dbb5611a3a
Remove LDS based software barriers from MSCCL ( #923 )
2023-10-19 16:39:41 -05:00
4278a9918b
Update rome models ( #922 )
2023-10-18 17:28:01 -07:00
39812ce757
NPKit: add xcc_id field ( #918 )
2023-10-13 15:24:59 -07:00
1b80d041cb
Fix incorrect arch name parsing ( #916 )
2023-10-13 10:01:11 -07:00
6d0b5c1e89
Port init_once fix from NCCL ( #915 )
2023-10-13 08:01:12 -07:00
7ee5c1c28b
Change MSCCL kernel signature to allow kernel arguments be preloaded via SGPR ( #911 )
...
* Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
* Only build gfx941
* demo
* fine tune malloc
* Fix merge errors
* Fix merge errors
* Disable parallel build
* Adopt --amdgpu-kernarg-preload-count
* Revert "Adding a script that will download/compile/run TransferBench/RCCL/UCX/RCCL-tests/RCCL-Unittests/hip-mpi-testsuite (#895 )"
This reverts commit f5e252dddf02a41b4d1bc512f306f45f97166304.
* Revert CMake changes.
* NPKIT changes.
* Remove some license declarations.
* Address code review feedbacks on msccl_kernel_impl.h
* Update CMakeLists.txt
* Add CMake logic to check the existence of --amdgpu-kernarg-preload-count
* Fix NPKIT trace logic.
---------
Co-authored-by: Pedram Alizadeh <pmohamma@amd.com >
Co-authored-by: Pedram Alizadeh <pmohamma@banff-pla-r27-05.pla.dcgpu >
Co-authored-by: Ziyue Yang <ziyyang@microsoft.com >
2023-10-12 20:17:08 -05:00
a6ff4618c7
Revert "Remove 2H4P condition from P2P channels adjustment ( #890 )" ( #904 )
...
This reverts commit 16dd05a58a
2023-10-04 09:46:11 -06:00
28d7fe5629
Dma buf support optin ( #905 )
...
* dmaBufSupport Optin added on every part of the code that should invoke it
2023-10-03 03:17:48 -06:00
c1f57a7041
Modify All-To-All doc ( #896 )
...
* Modify All-To-All doc
* Update nccl.h.in
* update unit-tests
---------
Co-authored-by: gilbertlee-amd <44450918+gilbertlee-amd@users.noreply.github.com >
2023-09-27 12:45:21 -04:00
8c6c595185
2.19.3-1
...
H800/H100 fixes and tuning.
Re-enable intra-process direct pointer buffer access when CUMEM is
enabled.
2023-09-26 05:57:15 -07:00
3435178b6c
Merge remote-tracking branch 'origin/master' into v2.19
2023-09-26 05:55:56 -07:00
f9c3dc251e
2.19.1-1
...
Add local user buffer registration for NVLink SHARP.
Add tuning plugin support.
Increase net API to v7 to allow for device-side packet reordering;
remove support for v4 plugins.
Add support for RoCE ECE.
Add support for C2C links.
Better detect SHM allocation failures to avoid crash with Bus Error.
Fix missing thread unlocks in bootstrap (Fixes #936 ).
Disable network flush by default on H100.
Move device code from src/collectives/device to src/device.
2023-09-26 05:50:33 -07:00
4365458757
Fix cudaMemcpyAsync bug
...
We are trying to use the copy result of first cudaMemcpyAsync in the
second cudaMemcpyAsync without sync in between. This patch fixes it
by allocating a CPU side array to cache device side addr so that we
can avoid this consecutive cuda mem copy.
Fixes #957
2023-09-20 05:51:14 -07:00
b85d73c02e
changed the form that RCCL_TREE uses ( #888 )
...
* changed the form that RCCL_TREE uses
2023-09-15 15:01:33 -06:00
26e982d913
Reduce NPKit latency overhead in MSCCL kernel ( #893 )
...
* Reduce NPKit latency overhead in MSCCL kernel
* Fix build error without NPKit enable
2023-09-15 13:28:26 -07:00
16dd05a58a
Remove 2H4P condition from P2P channels adjustment ( #890 )
2023-09-13 12:54:21 -07:00
c1bfd5f0d8
Add single-node MI300X topology ( #889 )
2023-09-13 11:07:17 -07:00
6a4d5ec089
Fix crash when NPKit is enabled ( #887 )
2023-09-13 11:00:12 -07:00
e58ec78d35
Gcn arch name ( #886 )
...
We use CMake to determine if we're compiling against a version of ROCm that supports gcnArchName and handles architecture checking appropriately. It includes a few helper functions as drop ins for the functionality we used gcnArch for before; sometimes to enable flags, and sometimes to set frequencies.
2023-09-12 15:34:40 -04:00
e1dc4d5e42
enable hip graph on multi-node ( #884 )
...
* initial checkin
* enable msccl when hip graph is on
* remove the commented out code of msccl enable check
* clean up the code
* remove the msccl HighestTransportType check logic
2023-09-11 15:30:04 -07:00
a283f55f12
msccl: add NPKIT profiling for MSCCL send-recv
2023-09-08 13:11:16 -05:00
6230b5f6b3
RCCL should use hipPointerAttribute_t.type ( #872 )
2023-09-05 09:44:12 -06:00
4297315de7
More robust msccl shared directory location discovery ( #868 )
2023-08-30 08:10:14 -06:00
e7f27c66e0
optimizing COLL_UNROLL for MI100 machines ( #863 )
2023-08-29 12:49:06 -04:00
aeca1af374
Add MSCCL xml files ( #861 )
2023-08-23 14:12:34 -07:00
5bcd3768cc
Minor fix for some msccl installations ( #862 )
2023-08-23 13:48:58 -06:00
559b70f86c
2.18.5-1
...
Fix NVLS search (issue #931 ).
Increase max IB NICs to 32.
Fix inconsistent device ordering (issue #820 ).
Try to use different devices for different GPUs in systems with
more than one NIC per GFU.
2023-08-23 06:32:36 -07:00
6a0a6a37d9
Use relaxed atomics for LL on GFX11 ( #859 )
2023-08-21 16:28:39 -07:00
2ec2648247
Fix static_assert string literal that contains a "\%". This is no longer ( #860 )
...
valid. They can only be simple escape sequences. Removing '\' fixes
issue. Assert message now compiles and emits the '%' as expected.
2023-08-21 16:19:59 -07:00
d33cd5a233
NCCL_TREES variable and rome model fixes ( #856 )
2023-08-21 10:35:37 -06:00
f70e3e569b
gfx11: don't use LL for sendrecv ( #853 )
...
* gfx11: don't use LL for sendrecv
* Use builtin instead of inline asm
2023-08-17 08:50:51 -07:00
7044599575
Add new model support ( #847 )
...
* Add new model support
* Update new rings
2023-08-10 17:14:51 -07:00
d33a70e620
NPKit update ( #844 )
...
* NPKit update
1. Enable NPKit for MSCCL kernels
2. Fix NPKit context index calculation for sendrecv kernels
* Update build script for npkit
2023-08-08 17:30:40 -07:00
d65c0830c6
Detect HIP_UNCACHED_MEMORY support from HIP version ( #842 )
2023-08-04 10:17:04 -07:00
8e58b65873
gfx11xx: disable LL protocol to workaround mtype issue ( #840 )
2023-08-04 07:53:07 -07:00
60efe26549
Fix merge error and replace inline asm ( #838 )
2023-08-03 13:46:40 -07:00
c8085eb704
Improve collective trace ( #835 )
2023-08-03 07:16:12 -07:00
64c32d1c5b
Disable MSCCL kernels at compile time ( #834 )
...
* Disable MSCCL kernels at compile time
2023-08-02 09:45:18 -06:00
8ed014bae9
Fix inter-node NVLS graph search
...
We were passing a net ID instead of a gpu index, which could cause
crashes if those were unrelated (and they usually are).
Issue #931
2023-08-02 07:06:35 -07:00
afa4a5ecf8
Updating Doxygen documentation ( #831 )
2023-07-28 16:09:06 -06:00
3db371c9a5
Revert "Enable Ll128 on gfx90a ( #823 )" ( #829 )
...
This reverts commit 420f8af6a0
2023-07-27 20:25:18 -07:00
f7dc7b7e6a
Fix MSCCL proxy number of chunks calculation ( #821 )
...
Current number of transmissions parsed from MSCCL algorithm is 1-based value,
but when calculating proxy number of chunks, it's taken as 0-based value.
This commit fixes this issue.
2023-07-26 13:24:49 -07:00
420f8af6a0
Enable Ll128 on gfx90a ( #823 )
2023-07-26 11:44:15 -07:00
4d20b4b758
Replace atomicExch with __atomic_store_n ( #818 )
...
* Replace atomicExch with __atomic_store_n
* Remove extra semicolon
2023-07-25 11:15:21 -07:00
47f754e6f5
Merge pull request #810 from nusislam/tune-send-recv
...
device: fine-tune RCCL send-recv on MI250/MI200
2023-07-25 10:18:12 -05:00
8bab4f04b7
Implement RCCL Replayer ( #817 )
...
* Implement RCCL Replayer
2023-07-24 16:26:22 -06:00
a7fcd58a97
Enable gfx94x ( #808 ) ( #816 )
...
(cherry picked from commit 94da229a7788d74685d1591a4e75a8341de64f41)
2023-07-21 07:31:27 -07:00
b2a0a0bd3a
device: fine-tune RCCL send-recv
2023-07-17 21:45:52 -05:00
848e60b00c
Fix path finding in msccl internal scheduler ( #794 )
2023-07-12 13:31:28 -07:00
Wenkai Du
Wen-Heng (Jack) Chung
Bertan Dogancay
akolliasAMD
Sylvain Jeaugey
Kaiming Ouyang
Ziyue Yang
Audrey MP
Andy li
Nusrat Islam
gilbertlee-amd
Pedram Alizadeh
David Pagan