81ddf9de89
Merge remote-tracking branch 'nccl/v2.19' into develop
2024-01-24 15:25:33 -08:00
18a57bac10
Turned on RCCL signal handler in CI tests ( #1039 )
2024-01-23 17:07:45 -05:00
5564d65e71
Use binary search for direct function calls ( #1057 )
...
* Use binary search for direct function calls
* fix scratch mem issue on MI300
2024-01-22 17:37:56 -07:00
c4dbf8a914
Fix collective trace when rccl is configured ( #1056 )
...
* Fix collective trace when rccl is configured
2024-01-22 09:26:44 -07:00
7e25d5bc55
Use new HIP graph API compatible with CUDA 11030 ( #991 )
...
* Use new HIP graph API compatible with CUDA 11030
* Update dependency to ROCm 6.1
* Fix single stream use case
2024-01-21 19:00:50 -08:00
8b97a20943
COLLECTIVES: Switch to unroll 2 for MI300 ( #1051 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2024-01-19 12:16:05 -06:00
1ac800f9fc
Disable LL128 for gfx90a ( #1054 )
2024-01-18 18:34:19 -07:00
5f365a9957
Turn IFC off ( #1053 )
2024-01-18 15:29:36 -07:00
28d9b170c9
[DEV] Configure functions in RCCL ( #986 )
...
* configure functions in rccl
2024-01-18 15:07:16 -07:00
05850e89f2
Relaxing default timeout limit, add error log ( #1052 )
...
Signed-off-by: Tim Hu <timhu102@amd.com >
2024-01-18 15:09:08 -05:00
c2a073a97d
Adding regression test ( #1045 )
...
* adding regression test
Signed-off-by: Tim Hu <timhu102@amd.com >
* modifying regression test
Signed-off-by: Tim Hu <timhu102@amd.com >
* Update StandaloneTests.cpp
---------
Signed-off-by: Tim Hu <timhu102@amd.com >
2024-01-18 10:46:16 -05:00
3325f96c56
Only use full MAXCHANNELS for gfx94x ( #1050 )
2024-01-17 09:00:49 -08:00
600b44fee5
topo-expl: fix broken build ( #1048 )
2024-01-17 08:59:03 -08:00
9c0ef11ac7
Adding timeout functionality/EnvVar to TestBed ( #1044 )
...
* Adding timeout functionality/EnvVar to TestBed
* updating timeout unit to microseconds
Signed-off-by: Tim Hu <timhu102@amd.com >
2024-01-17 11:33:01 -05:00
15f0ccaec7
Bump rocm-docs-core from 0.30.3 to 0.31.0 in /docs/sphinx ( #1049 )
...
Bumps [rocm-docs-core](https://github.com/RadeonOpenCompute/rocm-docs-core ) from 0.30.3 to 0.31.0.
- [Release notes](https://github.com/RadeonOpenCompute/rocm-docs-core/releases )
- [Changelog](https://github.com/RadeonOpenCompute/rocm-docs-core/blob/develop/CHANGELOG.md )
- [Commits](https://github.com/RadeonOpenCompute/rocm-docs-core/compare/v0.30.3...v0.31.0 )
---
updated-dependencies:
- dependency-name: rocm-docs-core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-16 11:53:14 -07:00
b08124c85d
adding rccl tuning parameters for MI300X gfx942 with 8 GPUs single and multi-node ( #1047 )
2024-01-16 13:44:32 -05:00
261707d90a
Add option to force enable network transport on single node ( #1046 )
2024-01-16 07:54:18 -08:00
246dbd16d7
Standardize documentation for ReadtheDocs ( #1027 )
2024-01-15 09:26:18 -07:00
2a8c632516
Bump rocm-docs-core from 0.30.3 to 0.31.0 in /docs/sphinx ( #1043 )
...
Bumps [rocm-docs-core](https://github.com/RadeonOpenCompute/rocm-docs-core ) from 0.30.3 to 0.31.0.
- [Release notes](https://github.com/RadeonOpenCompute/rocm-docs-core/releases )
- [Changelog](https://github.com/RadeonOpenCompute/rocm-docs-core/blob/develop/CHANGELOG.md )
- [Commits](https://github.com/RadeonOpenCompute/rocm-docs-core/compare/v0.30.3...v0.31.0 )
---
updated-dependencies:
- dependency-name: rocm-docs-core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-15 09:10:58 -07:00
80726361b9
Merge pull request #1042 from PedramAlizadeh/revert_2.18.5
...
Revert nccl "2.18.5-1" from 2.18.6
2024-01-12 12:00:12 -05:00
767fde8210
Revert "2.18.5-1"
...
This reverts commit 559b70f86c
2024-01-12 16:54:19 +00:00
a1ee3e1ba9
Bump jinja2 from 3.1.2 to 3.1.3 in /docs/sphinx ( #1040 )
...
Bumps [jinja2](https://github.com/pallets/jinja ) from 3.1.2 to 3.1.3.
- [Release notes](https://github.com/pallets/jinja/releases )
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst )
- [Commits](https://github.com/pallets/jinja/compare/3.1.2...3.1.3 )
---
updated-dependencies:
- dependency-name: jinja2
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-11 17:02:03 -07:00
ff7c9c4050
Add codeowners ( #1041 )
2024-01-11 15:41:08 -07:00
cf248d9402
Addressing the compiler warning ( #988 )
2024-01-10 14:59:40 -07:00
ccba6f7a74
Bump gitpython from 3.1.37 to 3.1.41 in /docs/sphinx ( #1038 )
...
Bumps [gitpython](https://github.com/gitpython-developers/GitPython ) from 3.1.37 to 3.1.41.
- [Release notes](https://github.com/gitpython-developers/GitPython/releases )
- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES )
- [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.37...3.1.41 )
---
updated-dependencies:
- dependency-name: gitpython
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-10 11:30:09 -07:00
735178c1fe
cover more gpu/nic mapping cases ( #1037 )
2024-01-10 08:01:37 -08:00
5851ae5974
Re-enable L128 on gfx90a of compiler supports it ( #1036 )
2024-01-10 08:01:11 -08:00
414884c6cb
Remove FORCE from AMDGPU_TARGETS and add support in install script ( #989 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2024-01-09 13:29:47 -06:00
249e9f7f65
Un-escaped character causes error with address sanitizer builds ( #992 )
...
Signed-off-by: Nilesh M Negi <Nilesh.Negi@amd.com >
Co-authored-by: Jenkins <jenkins-compute@amd.com >
2024-01-09 13:28:32 -06:00
aa5c84c997
Merge pull request #1022 from PedramAlizadeh/sync_nccl_2.18.6
...
Sync to nccl 2.18.6
2024-01-09 13:29:29 -05:00
d9871d171b
msccl: use custom reduce function ( #1033 )
2024-01-08 14:53:12 -08:00
f7e39fced2
Doubling buffer size to fix NCCL INFO corruption with increased channels ( #1035 )
2024-01-08 08:14:33 -08:00
e5bf56c6d8
Increase stack size for gfx906 ( #1034 )
...
Occationally "Memory access fault by GPU node-8 (Agent handle: 0x23a5640) on address 0x7f461ec00000. Reason: Page not present or supervisor privilege" can be seen from gfx906 CI
2024-01-07 20:25:02 -08:00
70bbeb4773
Fix MSCCL multi-node ( #1032 )
...
1) Move needsProxy initialization before mscclSetupConnections since the latter
will revise it later.
2) Remove mscclAvailable check in net.cc since it's no more required and caused
non-shared buffer allocated for MSCCL which is not expected.
2024-01-05 17:03:43 -08:00
cfc04a8aef
p2p-latency-tests: fix build by switching to gcnArchName ( #1030 )
...
* p2p-latency-tests: fix build by switching to gcnArchName
* rccl-prim-test: switch to gcnArchName
2024-01-04 13:36:48 -08:00
abf265a911
Rework barriers and adjust scope of atomics ( #1019 )
2024-01-04 08:18:48 -08:00
0a53077c9c
Improve MSCCL algorithms ( #1023 )
2024-01-03 14:51:34 -08:00
f4858e14b2
rearranged how the min and max functions are part of msccl ( #1025 )
...
* rearranged how the min and max functions are part of msccl
* added more coverage on in place graph tests
2023-12-21 08:58:33 -07:00
7e1cbb440d
Bump rocm-docs-core from 0.30.2 to 0.30.3 in /docs/sphinx ( #1024 )
...
Bumps [rocm-docs-core](https://github.com/RadeonOpenCompute/rocm-docs-core ) from 0.30.2 to 0.30.3.
- [Release notes](https://github.com/RadeonOpenCompute/rocm-docs-core/releases )
- [Changelog](https://github.com/RadeonOpenCompute/rocm-docs-core/blob/develop/CHANGELOG.md )
- [Commits](https://github.com/RadeonOpenCompute/rocm-docs-core/compare/v0.30.2...v0.30.3 )
---
updated-dependencies:
- dependency-name: rocm-docs-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-20 10:37:13 -07:00
d8c53e90d7
Bump rocm-docs-core from 0.30.1 to 0.30.2 in /docs/sphinx ( #1021 )
...
Bumps [rocm-docs-core](https://github.com/RadeonOpenCompute/rocm-docs-core ) from 0.30.1 to 0.30.2.
- [Release notes](https://github.com/RadeonOpenCompute/rocm-docs-core/releases )
- [Changelog](https://github.com/RadeonOpenCompute/rocm-docs-core/blob/develop/CHANGELOG.md )
- [Commits](https://github.com/RadeonOpenCompute/rocm-docs-core/compare/v0.30.1...v0.30.2 )
---
updated-dependencies:
- dependency-name: rocm-docs-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-19 13:34:37 -07:00
0d515f9388
resolved conflicts, fixed the localNetCount/0 bug
2023-12-18 08:11:34 +00:00
a924454f0f
CMake does not allow for capital letters been used in package names ( #1020 )
2023-12-15 12:39:17 -07:00
655742a3a6
Fully disable MSCCL when machine is not matched ( #1017 )
...
* Disable MSCCL algorithm meta loading when machine is not matched
* fully disable init
* fix potential segfault
2023-12-13 08:36:21 -08:00
53d807a5b9
msccl: disable on multi-node ( #1018 )
2023-12-13 07:41:40 -08:00
81602814a7
msccl: fix data corruption with MTYPE_RW ( #1014 )
2023-12-11 20:33:15 -08:00
fca459baaf
correct package name ( #1012 )
2023-12-11 09:40:29 -07:00
7965c8b53c
Fix memory fence and use non-temporal store ( #1007 )
...
* Fix memory fence and use non-temporal store
* Use amdgcn builtin instead of inline asm
* Move threadfence location
* Revert changes to gfx90a
* Rework gfx90a change
* Apply changes to gfx94x
2023-12-09 12:16:08 -08:00
c002f20029
Fix MSCCL scratch allocation ( #1010 )
2023-12-08 17:47:10 -06:00
bb144dcd50
Tune MSCCL all-reduce algorithm ( #1009 )
2023-12-08 17:47:02 -06:00
baadda4bd8
Relax workgroup barrier implementation for MSCCL send/recv ops. ( #997 )
...
* Trim logic.
* Revert "Trim logic."
This reverts commit 8f2dba6c764108acf2bf5428366b9f41d4d206b9.
* Introduce MSCCL template parameters to send / recv.
* Address review feedbacks.
2023-12-08 17:46:53 -06:00
BertanDogancay
Tim
Bertan Dogancay
Wenkai Du
Nilesh M Negi
dependabot[bot]
Pedram Alizadeh
Sam Wu
Hossein Pourreza
Ziyue Yang
akolliasAMD
Wen-Heng (Jack) Chung