bac57421c7
Allow bus id to be null ( #1085 )
...
* Allow bus id to be null
2024-02-15 16:36:51 -07:00
6f3310605c
Disable unsupported ld/st instructions
2024-02-15 13:58:16 -08:00
76f83f95ab
Merge remote-tracking branch 'rccl/develop' into 2.19.4
2024-02-15 13:37:14 -08:00
51003c9980
Use native half without conversion ( #1083 )
2024-02-13 16:57:34 -08:00
1f0af90206
Fix undefined symbol when nvtx is not enabled ( #1082 )
2024-02-13 14:03:43 -08:00
b6475625fb
2.20.3-1
...
Add support for alternating rings, allow for cross-nic rings without
cross-rail communication.
Add support for user buffer registration for network send/recv.
Optimize aggregated operations to better utilize all channels.
Add flattening for BCM PCI gen5 switches.
Add support for inter-node NVLink communication
Add support for port fusion in NET/IB.
Add support for ReduceScatter and AllGather using Collnet.
Update net API to v8.
Fix hang during A2A connection.
2024-02-13 04:22:38 -08:00
32cca51894
Fix docs
2024-02-11 22:32:55 -08:00
d999d9ad21
Merge remote-tracking branch 'rccl/develop' into 2.19.4
2024-02-09 11:31:03 -06:00
5669b0d7b6
2.18.5 fix ( #1077 )
...
* Revert "Revert "2.18.5-1""
This reverts commit 767fde8210
2024-02-09 09:18:38 -08:00
8a442faa12
Nvtx support ( #1076 )
...
* NVTX support
2024-02-08 14:08:24 -07:00
5257c753c5
msccl: use relaxed atomics on scratch buffer ( #1075 )
2024-02-08 12:09:56 -08:00
704c9ef0d1
Doubling P2P channels per peer on single node gfx94x only ( #1074 )
2024-02-07 14:05:57 -08:00
1d989f6524
Doubling P2P channels per peer on single node only ( #1069 )
2024-02-02 12:41:00 -08:00
12ac20ade5
Revert re-usage of connect and listen ports
2024-02-01 10:03:13 -08:00
00fdb1ef51
Clean up
2024-01-31 17:27:15 -08:00
da85abab54
Fix stack size
2024-01-31 17:09:07 -08:00
95f87232c4
Fix transport merge
2024-01-31 17:35:12 -06:00
1a134b283b
Merge remote-tracking branch 'rccl/develop' into 2.19.4
2024-01-31 11:53:10 -06:00
9ff53eeeae
Merge remote-tracking branch 'nccl/master' into develop
2024-01-30 14:43:43 -08:00
01b359027b
Include common.h in enqueue.cc instead ( #1067 )
2024-01-30 08:24:22 -08:00
f7550d83b8
msccl: ensure memory coherence after data receive ( #1062 )
2024-01-30 08:22:50 -08:00
31ec5d5cb0
correct data type
2024-01-28 19:55:19 -08:00
ccfb35fa6d
modifying the tuning table to improve the performance of allreduce for 8MB and 16MB for single-node MI300X ( #1063 )
2024-01-26 09:05:53 -05:00
be8ef4367f
colltrace: fix dropped trace messages ( #1059 )
...
* colltrace: fix dropped trace messages
* Remove extra space
2024-01-25 13:31:53 -08:00
ffde530af5
Increase P2P channels per peer ( #1060 )
2024-01-25 11:21:58 -08:00
4aafb2a3c5
Fix sendrecv merge
2024-01-24 16:23:53 -08:00
81ddf9de89
Merge remote-tracking branch 'nccl/v2.19' into develop
2024-01-24 15:25:33 -08:00
7987015a19
Revert "msccl: build same number of kernels as in ROCm 5.7" ( #1058 )
...
This reverts commit f960174d03be7e5174baa83b256526d388a38842.
2024-01-24 08:43:50 -08:00
5564d65e71
Use binary search for direct function calls ( #1057 )
...
* Use binary search for direct function calls
* fix scratch mem issue on MI300
2024-01-22 17:37:56 -07:00
c4dbf8a914
Fix collective trace when rccl is configured ( #1056 )
...
* Fix collective trace when rccl is configured
2024-01-22 09:26:44 -07:00
7e25d5bc55
Use new HIP graph API compatible with CUDA 11030 ( #991 )
...
* Use new HIP graph API compatible with CUDA 11030
* Update dependency to ROCm 6.1
* Fix single stream use case
2024-01-21 19:00:50 -08:00
8b97a20943
COLLECTIVES: Switch to unroll 2 for MI300 ( #1051 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2024-01-19 12:16:05 -06:00
28d9b170c9
[DEV] Configure functions in RCCL ( #986 )
...
* configure functions in rccl
2024-01-18 15:07:16 -07:00
3325f96c56
Only use full MAXCHANNELS for gfx94x ( #1050 )
2024-01-17 09:00:49 -08:00
b08124c85d
adding rccl tuning parameters for MI300X gfx942 with 8 GPUs single and multi-node ( #1047 )
2024-01-16 13:44:32 -05:00
261707d90a
Add option to force enable network transport on single node ( #1046 )
2024-01-16 07:54:18 -08:00
767fde8210
Revert "2.18.5-1"
...
This reverts commit 559b70f86c
2024-01-12 16:54:19 +00:00
cf248d9402
Addressing the compiler warning ( #988 )
2024-01-10 14:59:40 -07:00
735178c1fe
cover more gpu/nic mapping cases ( #1037 )
2024-01-10 08:01:37 -08:00
5851ae5974
Re-enable L128 on gfx90a of compiler supports it ( #1036 )
2024-01-10 08:01:11 -08:00
249e9f7f65
Un-escaped character causes error with address sanitizer builds ( #992 )
...
Signed-off-by: Nilesh M Negi <Nilesh.Negi@amd.com >
Co-authored-by: Jenkins <jenkins-compute@amd.com >
2024-01-09 13:28:32 -06:00
aa5c84c997
Merge pull request #1022 from PedramAlizadeh/sync_nccl_2.18.6
...
Sync to nccl 2.18.6
2024-01-09 13:29:29 -05:00
d9871d171b
msccl: use custom reduce function ( #1033 )
2024-01-08 14:53:12 -08:00
f7e39fced2
Doubling buffer size to fix NCCL INFO corruption with increased channels ( #1035 )
2024-01-08 08:14:33 -08:00
e5bf56c6d8
Increase stack size for gfx906 ( #1034 )
...
Occationally "Memory access fault by GPU node-8 (Agent handle: 0x23a5640) on address 0x7f461ec00000. Reason: Page not present or supervisor privilege" can be seen from gfx906 CI
2024-01-07 20:25:02 -08:00
70bbeb4773
Fix MSCCL multi-node ( #1032 )
...
1) Move needsProxy initialization before mscclSetupConnections since the latter
will revise it later.
2) Remove mscclAvailable check in net.cc since it's no more required and caused
non-shared buffer allocated for MSCCL which is not expected.
2024-01-05 17:03:43 -08:00
abf265a911
Rework barriers and adjust scope of atomics ( #1019 )
2024-01-04 08:18:48 -08:00
0a53077c9c
Improve MSCCL algorithms ( #1023 )
2024-01-03 14:51:34 -08:00
f4858e14b2
rearranged how the min and max functions are part of msccl ( #1025 )
...
* rearranged how the min and max functions are part of msccl
* added more coverage on in place graph tests
2023-12-21 08:58:33 -07:00
0d515f9388
resolved conflicts, fixed the localNetCount/0 bug
2023-12-18 08:11:34 +00:00
akolliasAMD
BertanDogancay
Wenkai Du
Sylvain Jeaugey
Wenkai Du
Bertan Dogancay
Pedram Alizadeh
Nilesh M Negi
Hossein Pourreza
Ziyue Yang