b8542c2477
Integrated RCCL with MSCCL++ for small message sizes ( #1231 )
...
[ROCm/rccl commit: 6dc47eecd7
2024-07-12 15:32:58 -06:00
f60367f1c3
Improved version reporting in NCCL_DEBUG=VERSION ( #1232 )
...
* Improved version reporting in NCCL_DEBUG=VERSION.
Signed-off-by: rahulvaidya20 <ravaidya@amd.com >
* Version reporting changes
Signed-off-by: rahulvaidya20 <ravaidya@amd.com >
* Versioning changes: Initialized char arrays to null and fixed typo.
---------
Signed-off-by: rahulvaidya20 <ravaidya@amd.com >
[ROCm/rccl commit: c755b9cf93
2024-07-12 08:14:29 -05:00
37bf54b8f8
Enable multi-threading for MSCCL ( #1203 )
...
MSCCL can now run in a multi-threaded configuration. To test in the unit tests, added the ENABLE_OPENMP compile definition flag and the --openmp-test-enable flag to the unit test build script. To activate, set the environment variables UT_MULTITHREADED=1 and UT_PROCESS_MASK=1. Set Jenkins to use this mode.
[ROCm/rccl commit: 0c36d571ea
2024-07-04 09:34:38 -06:00
b3e9d2d61b
NPKit: separate time stamps for GPU access from different blocks ( #1229 )
...
To avoid races in memory access in GPU
[ROCm/rccl commit: 9d8f68b4ee
2024-06-28 08:00:22 -07:00
48821ad0d7
set MAXCHANNELS to 128
...
[ROCm/rccl commit: ef442f8f92
2024-06-03 13:05:05 -05:00
2447161fb4
graph: restrict MAXCHANNELS for certain platforms
...
[ROCm/rccl commit: 9f654f6cf5
2024-06-03 13:05:01 -05:00
b0b2aa1166
add 256 channels support
...
[ROCm/rccl commit: 506f16c506
2024-06-03 13:03:18 -05:00
c31ac0d17b
add alternative to rocm_smi_lib
...
[ROCm/rccl commit: 9ad913bfa8
2024-05-14 13:51:41 -07:00
0ff5fc0bad
npkit: add broadcast trace ( #1166 )
...
[ROCm/rccl commit: a0cef69110
2024-05-07 14:00:16 -07:00
9638535690
Fix build error when roctracer-dev package is not installed ( #1161 )
...
[ROCm/rccl commit: 6f5a8ce1fb
2024-05-01 13:55:09 -07:00
b99b89e7a2
[GRAPH] Reduce NCCL_TOPO_MAX_NODES to 64 ( #1153 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
[ROCm/rccl commit: b90436d292
2024-04-27 23:41:11 -05:00
e494f29235
Replace __HIP_PLATFORM_HCC__ with __HIP_PLATFORM_AMD__ ( #1154 )
...
[ROCm/rccl commit: 9e0c9b4ed8
2024-04-25 07:19:18 -07:00
36f9492cda
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: e1a835910e
2024-04-23 13:34:00 -07:00
35f8d269f8
Use hipExtMallocWithFlags to allocate host memory on APU ( #1149 )
...
Also use SM60 as CUDA compatibility level.
[ROCm/rccl commit: 220066197a
2024-04-17 16:56:38 -07:00
da835cff9c
replacing rccl_bfloat16 with hip_bfloat16 ( #1126 )
...
Co-authored-by: mberenjk <mberenjk@amd.com >
[ROCm/rccl commit: 428837ffe4
2024-04-11 11:30:37 -05:00
99c7fc29ba
NPKit: doubling size of event buffers following MAXCHANNELS change ( #1135 )
...
[ROCm/rccl commit: 0ce68f21d4
2024-04-09 08:02:58 -07:00
e373bd44bf
Enable fp8 support ( #1101 )
...
* initial checkin
* resolve cr comments
* resolve the build issue
* fix the data correctless issue
* update fp8 header file and update the unit test for fp8 support
* remove fp16 from fp8 headers
* fix ut issue and catch up the latest code from develop
* udate according to cr comments
* update ut according to cr comments
* update num floats for each SumPostDiv from 4 to 6
* update fp8 header file name
* fix the typo
[ROCm/rccl commit: 6777e65c1d
2024-03-08 15:17:53 -08:00
2354601589
Improve debug messages of memory allocations ( #1107 )
...
[ROCm/rccl commit: ff951e607d
2024-03-08 10:55:10 -08:00
e5aedb153e
Add support for using contiguous for GPU direct RDMA ( #1096 )
...
Enabled by env var RCCL_NET_CONTIGUOUS_MEM=1
[ROCm/rccl commit: cbd955627e
2024-02-29 10:06:43 -08:00
cee279fd99
Implement ROCTX ( #1094 )
...
* Implement roctx
[ROCm/rccl commit: b617aecc31
2024-02-27 15:46:15 -07:00
2ab8a3a750
2.20.3-1
...
Add support for alternating rings, allow for cross-nic rings without
cross-rail communication.
Add support for user buffer registration for network send/recv.
Optimize aggregated operations to better utilize all channels.
Add flattening for BCM PCI gen5 switches.
Add support for inter-node NVLink communication
Add support for port fusion in NET/IB.
Add support for ReduceScatter and AllGather using Collnet.
Update net API to v8.
Fix hang during A2A connection.
[ROCm/rccl commit: b6475625fb
2024-02-13 04:22:38 -08:00
d5f5091e5d
Merge remote-tracking branch 'rccl/develop' into 2.19.4
...
[ROCm/rccl commit: d999d9ad21
2024-02-09 11:31:03 -06:00
45ed3ef4e7
Nvtx support ( #1076 )
...
* NVTX support
[ROCm/rccl commit: 8a442faa12
2024-02-08 14:08:24 -07:00
b17f5947ba
Revert re-usage of connect and listen ports
...
[ROCm/rccl commit: 12ac20ade5
2024-02-01 10:03:13 -08:00
4f4e52aafe
Merge remote-tracking branch 'rccl/develop' into 2.19.4
...
[ROCm/rccl commit: 1a134b283b
2024-01-31 11:53:10 -06:00
c2c9ed2acb
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: 9ff53eeeae
2024-01-30 14:43:43 -08:00
96479caeda
Include common.h in enqueue.cc instead ( #1067 )
...
[ROCm/rccl commit: 01b359027b
2024-01-30 08:24:22 -08:00
404d398bac
Merge remote-tracking branch 'nccl/v2.19' into develop
...
[ROCm/rccl commit: 81ddf9de89
2024-01-24 15:25:33 -08:00
17f70df058
Revert "msccl: build same number of kernels as in ROCm 5.7" ( #1058 )
...
This reverts commit f960174d03be7e5174baa83b256526d388a38842.
[ROCm/rccl commit: 7987015a19
2024-01-24 08:43:50 -08:00
56482a8be8
Fix collective trace when rccl is configured ( #1056 )
...
* Fix collective trace when rccl is configured
[ROCm/rccl commit: c4dbf8a914
2024-01-22 09:26:44 -07:00
8b8179a689
Use new HIP graph API compatible with CUDA 11030 ( #991 )
...
* Use new HIP graph API compatible with CUDA 11030
* Update dependency to ROCm 6.1
* Fix single stream use case
[ROCm/rccl commit: 7e25d5bc55
2024-01-21 19:00:50 -08:00
11674674fc
[DEV] Configure functions in RCCL ( #986 )
...
* configure functions in rccl
[ROCm/rccl commit: 28d9b170c9
2024-01-18 15:07:16 -07:00
cec06d59d1
Un-escaped character causes error with address sanitizer builds ( #992 )
...
Signed-off-by: Nilesh M Negi <Nilesh.Negi@amd.com >
Co-authored-by: Jenkins <jenkins-compute@amd.com >
[ROCm/rccl commit: 249e9f7f65
2024-01-09 13:28:32 -06:00
d3a47bb387
Merge pull request #1022 from PedramAlizadeh/sync_nccl_2.18.6
...
Sync to nccl 2.18.6
[ROCm/rccl commit: aa5c84c997
2024-01-09 13:29:29 -05:00
0c1f773021
rearranged how the min and max functions are part of msccl ( #1025 )
...
* rearranged how the min and max functions are part of msccl
* added more coverage on in place graph tests
[ROCm/rccl commit: f4858e14b2
2023-12-21 08:58:33 -07:00
401a104c5a
resolved conflicts, fixed the localNetCount/0 bug
...
[ROCm/rccl commit: 0d515f9388
2023-12-18 08:11:34 +00:00
e4b63a8ba0
Fully disable MSCCL when machine is not matched ( #1017 )
...
* Disable MSCCL algorithm meta loading when machine is not matched
* fully disable init
* fix potential segfault
[ROCm/rccl commit: 655742a3a6
2023-12-13 08:36:21 -08:00
22730bdc4d
msccl: build same number of kernels as in ROCm 5.7 ( #1005 )
...
Removed fullOps kernels from build
[ROCm/rccl commit: 12c08fc52a
2023-12-07 13:36:04 -06:00
33aa8b67be
Use a map to host scratch buffers ( #1004 )
...
* Use a map to host scratch buffers
* Address review feedbacks. Deliberately keep mscclSetupScratch function.
[ROCm/rccl commit: 293f0fb752
2023-12-05 13:15:28 -06:00
ae0bdad45c
IFC mix build ( #998 )
...
[ROCm/rccl commit: 7c0f49a878
2023-12-02 18:49:52 -07:00
b38b7fa3a2
Increase max channles to 64 ( #993 )
...
[ROCm/rccl commit: 4ba65d1d6a
2023-12-01 16:01:11 -08:00
f0c47d085e
Move MSCCL algorithm loading to initialization to workaround HIP graph conflict ( #982 )
...
* MSCCL: pre-specify channels and pre-load algorithms
* add mutex
* fix bug
* clean include
* disable all-gathers temporarily
[ROCm/rccl commit: 4bb0b4a380
2023-11-30 09:47:20 -08:00
bd982864d5
recreated pr 914 to work with current develop branch ( #979 )
...
[ROCm/rccl commit: 56ce9ef05f
2023-11-28 16:33:47 -07:00
6ce074d92d
Fix MSCCL work FIFO allocation with HIP graph enabled ( #967 )
...
[ROCm/rccl commit: 7fc891bc8d
2023-11-15 16:43:28 -08:00
b16882a024
2.19.4-1
...
Split transport connect phase into multiple steps to avoid port
exhaustion when connecting alltoall at large scale. Defaults to 128
peers per round.
Fix memory leaks on CUDA graph capture.
Fix alltoallv crash on self-sendrecv.
Make topology detection more deterministic when PCI speeds are not
available (fix issue #1020 ).
Properly close shared memory in NVLS resources.
Revert proxy detach after 5 seconds.
Add option to print progress during transport connect.
Add option to set NCCL_DEBUG to INFO on first WARN.
[ROCm/rccl commit: 88d44d777f
2023-11-13 10:36:12 -08:00
534af85d0f
msccl: enable basic collective trace ( #959 )
...
To avoid increasing number of kernels, colltrace is only enabled with
RCCL_MSCCL_FORCE_FULLOPS=1
[ROCm/rccl commit: 5a800e00cd
2023-11-08 20:14:28 -08:00
297023a7a6
msccl: add templated kernel ( #945 )
...
* msccl: add templated kernel
* Use defines to improve code readability
* Fix kernel indexing and review feedback
[ROCm/rccl commit: f484ff17b9
2023-11-02 17:21:53 -07:00
b736b506c0
NPkit: misc fixes for MSCCL ( #936 )
...
* msccl: add xcc_id to timestamp sync
* NPKit: add timestamp for rrc operator
* NPKit: add timestamp for MSCCL init
[ROCm/rccl commit: a497722894
2023-10-30 10:00:12 -07:00
8b1254a4f1
Fix gcnArchName bug in topology dump ( #937 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
[ROCm/rccl commit: 1e5ca6820b
2023-10-28 12:30:36 -05:00
e1dfb82023
Fix MSCCL work FIFO out-of-bound issue ( #935 )
...
[ROCm/rccl commit: 4c117e5335
2023-10-27 11:24:52 -07:00
corey-derochie-amd
Rahul Vaidya
Wenkai Du
Nusrat Islam
Edgar Gabriel
Nilesh M Negi
BertanDogancay
mberenjk
Andy li
Bertan Dogancay
Sylvain Jeaugey
Wenkai Du
Pedram Alizadeh
akolliasAMD
Ziyue Yang
Wen-Heng (Jack) Chung