45ed3ef4e7
Nvtx support ( #1076 )
...
* NVTX support
[ROCm/rccl commit: 8a442faa12
2024-02-08 14:08:24 -07:00
1538b908ac
msccl: use relaxed atomics on scratch buffer ( #1075 )
...
[ROCm/rccl commit: 5257c753c5
2024-02-08 12:09:56 -08:00
ce39eefe65
Doubling P2P channels per peer on single node gfx94x only ( #1074 )
...
[ROCm/rccl commit: 704c9ef0d1
2024-02-07 14:05:57 -08:00
57e508f2e4
Doubling P2P channels per peer on single node only ( #1069 )
...
[ROCm/rccl commit: 1d989f6524
2024-02-02 12:41:00 -08:00
96479caeda
Include common.h in enqueue.cc instead ( #1067 )
...
[ROCm/rccl commit: 01b359027b
2024-01-30 08:24:22 -08:00
3e66d42d95
msccl: ensure memory coherence after data receive ( #1062 )
...
[ROCm/rccl commit: f7550d83b8
2024-01-30 08:22:50 -08:00
a184aabda5
modifying the tuning table to improve the performance of allreduce for 8MB and 16MB for single-node MI300X ( #1063 )
...
[ROCm/rccl commit: ccfb35fa6d
2024-01-26 09:05:53 -05:00
cbc6aa13e7
colltrace: fix dropped trace messages ( #1059 )
...
* colltrace: fix dropped trace messages
* Remove extra space
[ROCm/rccl commit: be8ef4367f
2024-01-25 13:31:53 -08:00
71a356670f
Increase P2P channels per peer ( #1060 )
...
[ROCm/rccl commit: ffde530af5
2024-01-25 11:21:58 -08:00
17f70df058
Revert "msccl: build same number of kernels as in ROCm 5.7" ( #1058 )
...
This reverts commit f960174d03be7e5174baa83b256526d388a38842.
[ROCm/rccl commit: 7987015a19
2024-01-24 08:43:50 -08:00
645411f8f4
Use binary search for direct function calls ( #1057 )
...
* Use binary search for direct function calls
* fix scratch mem issue on MI300
[ROCm/rccl commit: 5564d65e71
2024-01-22 17:37:56 -07:00
56482a8be8
Fix collective trace when rccl is configured ( #1056 )
...
* Fix collective trace when rccl is configured
[ROCm/rccl commit: c4dbf8a914
2024-01-22 09:26:44 -07:00
8b8179a689
Use new HIP graph API compatible with CUDA 11030 ( #991 )
...
* Use new HIP graph API compatible with CUDA 11030
* Update dependency to ROCm 6.1
* Fix single stream use case
[ROCm/rccl commit: 7e25d5bc55
2024-01-21 19:00:50 -08:00
1c9b4fab39
COLLECTIVES: Switch to unroll 2 for MI300 ( #1051 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
[ROCm/rccl commit: 8b97a20943
2024-01-19 12:16:05 -06:00
11674674fc
[DEV] Configure functions in RCCL ( #986 )
...
* configure functions in rccl
[ROCm/rccl commit: 28d9b170c9
2024-01-18 15:07:16 -07:00
dbf906d8fa
Only use full MAXCHANNELS for gfx94x ( #1050 )
...
[ROCm/rccl commit: 3325f96c56
2024-01-17 09:00:49 -08:00
5c349cd729
adding rccl tuning parameters for MI300X gfx942 with 8 GPUs single and multi-node ( #1047 )
...
[ROCm/rccl commit: b08124c85d
2024-01-16 13:44:32 -05:00
8d38747c65
Add option to force enable network transport on single node ( #1046 )
...
[ROCm/rccl commit: 261707d90a
2024-01-16 07:54:18 -08:00
7cc572ecf9
Revert "2.18.5-1"
...
This reverts commit 26b91b9dbb767fde8210
2024-01-12 16:54:19 +00:00
a056463d4d
Addressing the compiler warning ( #988 )
...
[ROCm/rccl commit: cf248d9402
2024-01-10 14:59:40 -07:00
df0dbb887d
cover more gpu/nic mapping cases ( #1037 )
...
[ROCm/rccl commit: 735178c1fe
2024-01-10 08:01:37 -08:00
64cf812da0
Re-enable L128 on gfx90a of compiler supports it ( #1036 )
...
[ROCm/rccl commit: 5851ae5974
2024-01-10 08:01:11 -08:00
cec06d59d1
Un-escaped character causes error with address sanitizer builds ( #992 )
...
Signed-off-by: Nilesh M Negi <Nilesh.Negi@amd.com >
Co-authored-by: Jenkins <jenkins-compute@amd.com >
[ROCm/rccl commit: 249e9f7f65
2024-01-09 13:28:32 -06:00
d3a47bb387
Merge pull request #1022 from PedramAlizadeh/sync_nccl_2.18.6
...
Sync to nccl 2.18.6
[ROCm/rccl commit: aa5c84c997
2024-01-09 13:29:29 -05:00
30f744dc35
msccl: use custom reduce function ( #1033 )
...
[ROCm/rccl commit: d9871d171b
2024-01-08 14:53:12 -08:00
cd7a346297
Doubling buffer size to fix NCCL INFO corruption with increased channels ( #1035 )
...
[ROCm/rccl commit: f7e39fced2
2024-01-08 08:14:33 -08:00
626608c172
Increase stack size for gfx906 ( #1034 )
...
Occationally "Memory access fault by GPU node-8 (Agent handle: 0x23a5640) on address 0x7f461ec00000. Reason: Page not present or supervisor privilege" can be seen from gfx906 CI
[ROCm/rccl commit: e5bf56c6d8
2024-01-07 20:25:02 -08:00
1b39fef32a
Fix MSCCL multi-node ( #1032 )
...
1) Move needsProxy initialization before mscclSetupConnections since the latter
will revise it later.
2) Remove mscclAvailable check in net.cc since it's no more required and caused
non-shared buffer allocated for MSCCL which is not expected.
[ROCm/rccl commit: 70bbeb4773
2024-01-05 17:03:43 -08:00
13791d7ee3
Rework barriers and adjust scope of atomics ( #1019 )
...
[ROCm/rccl commit: abf265a911
2024-01-04 08:18:48 -08:00
e3d45f9de4
Improve MSCCL algorithms ( #1023 )
...
[ROCm/rccl commit: 0a53077c9c
2024-01-03 14:51:34 -08:00
0c1f773021
rearranged how the min and max functions are part of msccl ( #1025 )
...
* rearranged how the min and max functions are part of msccl
* added more coverage on in place graph tests
[ROCm/rccl commit: f4858e14b2
2023-12-21 08:58:33 -07:00
401a104c5a
resolved conflicts, fixed the localNetCount/0 bug
...
[ROCm/rccl commit: 0d515f9388
2023-12-18 08:11:34 +00:00
e4b63a8ba0
Fully disable MSCCL when machine is not matched ( #1017 )
...
* Disable MSCCL algorithm meta loading when machine is not matched
* fully disable init
* fix potential segfault
[ROCm/rccl commit: 655742a3a6
2023-12-13 08:36:21 -08:00
918ce6c2e2
msccl: disable on multi-node ( #1018 )
...
[ROCm/rccl commit: 53d807a5b9
2023-12-13 07:41:40 -08:00
48107b18c9
msccl: fix data corruption with MTYPE_RW ( #1014 )
...
[ROCm/rccl commit: 81602814a7
2023-12-11 20:33:15 -08:00
481a35bc59
Fix memory fence and use non-temporal store ( #1007 )
...
* Fix memory fence and use non-temporal store
* Use amdgcn builtin instead of inline asm
* Move threadfence location
* Revert changes to gfx90a
* Rework gfx90a change
* Apply changes to gfx94x
[ROCm/rccl commit: 7965c8b53c
2023-12-09 12:16:08 -08:00
fe250212bf
Fix MSCCL scratch allocation ( #1010 )
...
[ROCm/rccl commit: c002f20029
2023-12-08 17:47:10 -06:00
529f72ceaa
Relax workgroup barrier implementation for MSCCL send/recv ops. ( #997 )
...
* Trim logic.
* Revert "Trim logic."
This reverts commit 8f2dba6c764108acf2bf5428366b9f41d4d206b9.
* Introduce MSCCL template parameters to send / recv.
* Address review feedbacks.
[ROCm/rccl commit: baadda4bd8
2023-12-08 17:46:53 -06:00
22730bdc4d
msccl: build same number of kernels as in ROCm 5.7 ( #1005 )
...
Removed fullOps kernels from build
[ROCm/rccl commit: 12c08fc52a
2023-12-07 13:36:04 -06:00
33aa8b67be
Use a map to host scratch buffers ( #1004 )
...
* Use a map to host scratch buffers
* Address review feedbacks. Deliberately keep mscclSetupScratch function.
[ROCm/rccl commit: 293f0fb752
2023-12-05 13:15:28 -06:00
403a91137c
Fix gcnArch bug in IFC mix build ( #998 ) ( #1002 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
[ROCm/rccl commit: bc44e3faa7
2023-12-04 16:20:22 -06:00
ae0bdad45c
IFC mix build ( #998 )
...
[ROCm/rccl commit: 7c0f49a878
2023-12-02 18:49:52 -07:00
b38b7fa3a2
Increase max channles to 64 ( #993 )
...
[ROCm/rccl commit: 4ba65d1d6a
2023-12-01 16:01:11 -08:00
bf57487384
Fix GPU to NIC mapping for peertopeer ( #994 )
...
[ROCm/rccl commit: 0b53f79196
2023-12-01 08:00:17 -08:00
cef45b8311
Fix mscclAlgoHandle not initialized issue ( #995 )
...
[ROCm/rccl commit: e44e112a17
2023-12-01 07:58:01 -08:00
f0c47d085e
Move MSCCL algorithm loading to initialization to workaround HIP graph conflict ( #982 )
...
* MSCCL: pre-specify channels and pre-load algorithms
* add mutex
* fix bug
* clean include
* disable all-gathers temporarily
[ROCm/rccl commit: 4bb0b4a380
2023-11-30 09:47:20 -08:00
bd982864d5
recreated pr 914 to work with current develop branch ( #979 )
...
[ROCm/rccl commit: 56ce9ef05f
2023-11-28 16:33:47 -07:00
dcf623f2ec
Add special handling of gfx940 ( #976 )
...
* Add special handling of gfx940
* Update ring base
[ROCm/rccl commit: 50b2dd9fd7
2023-11-22 15:07:36 -08:00
4c2fa05a23
msccl: allocate scratch as ext-scope fine-grained ( #968 )
...
[ROCm/rccl commit: 569d3f7d59
2023-11-16 09:57:25 -06:00
7c0920cd62
Fix kernel command line warnings ( #961 )
...
* Fix kernel command line warnings
* Remove while loop
[ROCm/rccl commit: bc8661f092
2023-11-15 18:01:12 -08:00
Bertan Dogancay
Wenkai Du
Pedram Alizadeh
Nilesh M Negi
Hossein Pourreza
Ziyue Yang
akolliasAMD
Wen-Heng (Jack) Chung
pradeep-ramanna