c1f57a7041
Modify All-To-All doc ( #896 )
...
* Modify All-To-All doc
* Update nccl.h.in
* update unit-tests
---------
Co-authored-by: gilbertlee-amd <44450918+gilbertlee-amd@users.noreply.github.com >
2023-09-27 12:45:21 -04:00
b85d73c02e
changed the form that RCCL_TREE uses ( #888 )
...
* changed the form that RCCL_TREE uses
2023-09-15 15:01:33 -06:00
26e982d913
Reduce NPKit latency overhead in MSCCL kernel ( #893 )
...
* Reduce NPKit latency overhead in MSCCL kernel
* Fix build error without NPKit enable
2023-09-15 13:28:26 -07:00
16dd05a58a
Remove 2H4P condition from P2P channels adjustment ( #890 )
2023-09-13 12:54:21 -07:00
c1bfd5f0d8
Add single-node MI300X topology ( #889 )
2023-09-13 11:07:17 -07:00
6a4d5ec089
Fix crash when NPKit is enabled ( #887 )
2023-09-13 11:00:12 -07:00
e58ec78d35
Gcn arch name ( #886 )
...
We use CMake to determine if we're compiling against a version of ROCm that supports gcnArchName and handles architecture checking appropriately. It includes a few helper functions as drop ins for the functionality we used gcnArch for before; sometimes to enable flags, and sometimes to set frequencies.
2023-09-12 15:34:40 -04:00
e1dc4d5e42
enable hip graph on multi-node ( #884 )
...
* initial checkin
* enable msccl when hip graph is on
* remove the commented out code of msccl enable check
* clean up the code
* remove the msccl HighestTransportType check logic
2023-09-11 15:30:04 -07:00
a283f55f12
msccl: add NPKIT profiling for MSCCL send-recv
2023-09-08 13:11:16 -05:00
6230b5f6b3
RCCL should use hipPointerAttribute_t.type ( #872 )
2023-09-05 09:44:12 -06:00
4297315de7
More robust msccl shared directory location discovery ( #868 )
2023-08-30 08:10:14 -06:00
e7f27c66e0
optimizing COLL_UNROLL for MI100 machines ( #863 )
2023-08-29 12:49:06 -04:00
aeca1af374
Add MSCCL xml files ( #861 )
2023-08-23 14:12:34 -07:00
5bcd3768cc
Minor fix for some msccl installations ( #862 )
2023-08-23 13:48:58 -06:00
6a0a6a37d9
Use relaxed atomics for LL on GFX11 ( #859 )
2023-08-21 16:28:39 -07:00
2ec2648247
Fix static_assert string literal that contains a "\%". This is no longer ( #860 )
...
valid. They can only be simple escape sequences. Removing '\' fixes
issue. Assert message now compiles and emits the '%' as expected.
2023-08-21 16:19:59 -07:00
d33cd5a233
NCCL_TREES variable and rome model fixes ( #856 )
2023-08-21 10:35:37 -06:00
f70e3e569b
gfx11: don't use LL for sendrecv ( #853 )
...
* gfx11: don't use LL for sendrecv
* Use builtin instead of inline asm
2023-08-17 08:50:51 -07:00
7044599575
Add new model support ( #847 )
...
* Add new model support
* Update new rings
2023-08-10 17:14:51 -07:00
d33a70e620
NPKit update ( #844 )
...
* NPKit update
1. Enable NPKit for MSCCL kernels
2. Fix NPKit context index calculation for sendrecv kernels
* Update build script for npkit
2023-08-08 17:30:40 -07:00
d65c0830c6
Detect HIP_UNCACHED_MEMORY support from HIP version ( #842 )
2023-08-04 10:17:04 -07:00
8e58b65873
gfx11xx: disable LL protocol to workaround mtype issue ( #840 )
2023-08-04 07:53:07 -07:00
60efe26549
Fix merge error and replace inline asm ( #838 )
2023-08-03 13:46:40 -07:00
c8085eb704
Improve collective trace ( #835 )
2023-08-03 07:16:12 -07:00
64c32d1c5b
Disable MSCCL kernels at compile time ( #834 )
...
* Disable MSCCL kernels at compile time
2023-08-02 09:45:18 -06:00
afa4a5ecf8
Updating Doxygen documentation ( #831 )
2023-07-28 16:09:06 -06:00
3db371c9a5
Revert "Enable Ll128 on gfx90a ( #823 )" ( #829 )
...
This reverts commit 420f8af6a0
2023-07-27 20:25:18 -07:00
f7dc7b7e6a
Fix MSCCL proxy number of chunks calculation ( #821 )
...
Current number of transmissions parsed from MSCCL algorithm is 1-based value,
but when calculating proxy number of chunks, it's taken as 0-based value.
This commit fixes this issue.
2023-07-26 13:24:49 -07:00
420f8af6a0
Enable Ll128 on gfx90a ( #823 )
2023-07-26 11:44:15 -07:00
4d20b4b758
Replace atomicExch with __atomic_store_n ( #818 )
...
* Replace atomicExch with __atomic_store_n
* Remove extra semicolon
2023-07-25 11:15:21 -07:00
47f754e6f5
Merge pull request #810 from nusislam/tune-send-recv
...
device: fine-tune RCCL send-recv on MI250/MI200
2023-07-25 10:18:12 -05:00
8bab4f04b7
Implement RCCL Replayer ( #817 )
...
* Implement RCCL Replayer
2023-07-24 16:26:22 -06:00
a7fcd58a97
Enable gfx94x ( #808 ) ( #816 )
...
(cherry picked from commit 94da229a7788d74685d1591a4e75a8341de64f41)
2023-07-21 07:31:27 -07:00
b2a0a0bd3a
device: fine-tune RCCL send-recv
2023-07-17 21:45:52 -05:00
848e60b00c
Fix path finding in msccl internal scheduler ( #794 )
2023-07-12 13:31:28 -07:00
0f14e5a640
npkit: separate network timing between send and test ( #798 )
2023-07-10 09:31:49 -07:00
58e53dfd37
device: fine tune MI200/MI250 simple protocol performance
...
With Simple protocol, unroll factor of 4 offers better
performance for most of the collectives (on MI200. MI250, and
MI300) except large message allreduce with Ring algorithm
on MI250 and MI200). This PR changes the default unroll factor
to 4 while adding fine tuning for reduction operations.
2023-07-08 20:21:18 -05:00
e0c70af46b
Merge remote-tracking branch 'nccl/master' into nccl_sync
2023-07-05 07:53:53 -07:00
ce6a2ffac8
Merge pull request #782 from ROCmSoftwarePlatform/2.18.3
...
Sync up with NCCL 2.18.3
2023-06-29 15:04:16 -07:00
9bba4a2f2a
added npkit support into the all_gather run ring algorithm ( #790 )
2023-06-29 13:59:54 -06:00
6e24ef4e1f
Prevent WR index truncation in the InfiniBand transport plugin
2023-06-28 11:39:19 +02:00
abd0615351
Merge remote-tracking branch 'nccl/master' into develop
2023-06-26 22:51:56 +00:00
bd14ac8b59
ASAN build excluding additional files, Algodir support for share folder
...
* ASAN build excluding additional files, Algodir support for share folder (#786 )
* Algodir support for share folder
2023-06-23 10:57:20 -04:00
0c77c66221
Disable Colltrace for --fast option ( #778 )
...
* Disable Colltrace for --fast option
* Limit nprocs for CI
2023-06-21 14:16:09 -06:00
9bdf6797a5
fixed npkit size to never be a negative number ( #779 )
2023-06-21 08:26:40 -06:00
52a28ff2fc
Switching to using atomicAdd_system within kernel for collective trace ( #780 )
2023-06-20 17:49:52 -06:00
3a741787bf
device: use unroll factor based on platforms
2023-06-14 13:36:15 -05:00
f35777e9b0
improve compilation time and create timetrace plot ( #773 )
...
* improve compilation time and create time-trace plot
* set default value for nproc
2023-06-14 09:17:51 -06:00
ea38312273
2.18.3-1
...
Fix data corruption with Tree/LL128 on systems with 1GPU:1NIC.
Fix hang with Collnet on bfloat16 on systems with less than one NIC
per GPU.
Fix long initialization time.
Fix data corruption with Collnet when mixing multi-process and
multi-GPU per process.
Fix crash when shared memory creation fails.
Fix Avg operation with Collnet/Chain.
Fix performance of alltoall at scale with more than one NIC per GPU.
Fix performance for DGX H800.
Fix race condition in connection progress causing a crash.
Fix network flush with Collnet.
Fix performance of aggregated allGather/reduceScatter operations.
Fix PXN operation when CUDA_VISIBLE_DEVICES is set.
Fix NVTX3 compilation issues on Debian 10.
2023-06-14 01:29:17 -07:00
9cdac774ea
Wall clock update and npkit trace script Update ( #771 )
...
* changed builtin clock to wall_clock64
* updated npkit_Trace_generator to the new version of npkit
2023-06-07 17:47:10 -06:00
Bertan Dogancay
akolliasAMD
Wenkai Du
Ziyue Yang
Audrey MP
Andy li
Nusrat Islam
gilbertlee-amd
Pedram Alizadeh
David Pagan
Wenkai Du
Dmitrii Gabor
arvindcheru
Sylvain Jeaugey