AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,235 Commits 3 Branches 0 Tags
df0dbb887dd0ff64417482ac8a2e983de5467f22
Commit Graph

1126 Commits

Author SHA1 Message Date
Hossein Pourreza df0dbb887d cover more gpu/nic mapping cases (#1037) 
[ROCm/rccl commit: 735178c1fe]
 2024-01-10 08:01:37 -08:00
Wenkai Du 64cf812da0 Re-enable L128 on gfx90a of compiler supports it (#1036) 
[ROCm/rccl commit: 5851ae5974]
 2024-01-10 08:01:11 -08:00
Nilesh M Negi c1acf97c05 Remove FORCE from AMDGPU_TARGETS and add support in install script (#989) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 414884c6cb]
 2024-01-09 13:29:47 -06:00
Nilesh M Negi cec06d59d1 Un-escaped character causes error with address sanitizer builds (#992) 
Signed-off-by: Nilesh M Negi <Nilesh.Negi@amd.com>
Co-authored-by: Jenkins <jenkins-compute@amd.com>

[ROCm/rccl commit: 249e9f7f65]
 2024-01-09 13:28:32 -06:00
Pedram Alizadeh d3a47bb387 Merge pull request #1022 from PedramAlizadeh/sync_nccl_2.18.6 
Sync to nccl 2.18.6

[ROCm/rccl commit: aa5c84c997]
 2024-01-09 13:29:29 -05:00
Wenkai Du 30f744dc35 msccl: use custom reduce function (#1033) 
[ROCm/rccl commit: d9871d171b]
 2024-01-08 14:53:12 -08:00
Wenkai Du cd7a346297 Doubling buffer size to fix NCCL INFO corruption with increased channels (#1035) 
[ROCm/rccl commit: f7e39fced2]
 2024-01-08 08:14:33 -08:00
Wenkai Du 626608c172 Increase stack size for gfx906 (#1034) 
Occationally "Memory access fault by GPU node-8 (Agent handle: 0x23a5640) on address 0x7f461ec00000. Reason: Page not present or supervisor privilege" can be seen from gfx906 CI

[ROCm/rccl commit: e5bf56c6d8]
 2024-01-07 20:25:02 -08:00
Ziyue Yang 1b39fef32a Fix MSCCL multi-node (#1032) 
1) Move needsProxy initialization before mscclSetupConnections since the latter
will revise it later.
2) Remove mscclAvailable check in net.cc since it's no more required and caused
non-shared buffer allocated for MSCCL which is not expected.

[ROCm/rccl commit: 70bbeb4773]
 2024-01-05 17:03:43 -08:00
Wenkai Du 4eaf90f84c p2p-latency-tests: fix build by switching to gcnArchName (#1030) 
* p2p-latency-tests: fix build by switching to gcnArchName

* rccl-prim-test: switch to gcnArchName

[ROCm/rccl commit: cfc04a8aef]
 2024-01-04 13:36:48 -08:00
Wenkai Du 13791d7ee3 Rework barriers and adjust scope of atomics (#1019) 
[ROCm/rccl commit: abf265a911]
 2024-01-04 08:18:48 -08:00
Ziyue Yang e3d45f9de4 Improve MSCCL algorithms (#1023) 
[ROCm/rccl commit: 0a53077c9c]
 2024-01-03 14:51:34 -08:00
akolliasAMD 0c1f773021 rearranged how the min and max functions are part of msccl (#1025) 
* rearranged how the min and max functions are part of msccl

* added more coverage on in place graph tests

[ROCm/rccl commit: f4858e14b2]
 2023-12-21 08:58:33 -07:00
dependabot[bot] 08b097d6ae Bump rocm-docs-core from 0.30.2 to 0.30.3 in /docs/sphinx (#1024) 
Bumps [rocm-docs-core](https://github.com/RadeonOpenCompute/rocm-docs-core) from 0.30.2 to 0.30.3.
- [Release notes](https://github.com/RadeonOpenCompute/rocm-docs-core/releases)
- [Changelog](https://github.com/RadeonOpenCompute/rocm-docs-core/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/RadeonOpenCompute/rocm-docs-core/compare/v0.30.2...v0.30.3)

---
updated-dependencies:
- dependency-name: rocm-docs-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

[ROCm/rccl commit: 7e1cbb440d]
 2023-12-20 10:37:13 -07:00
dependabot[bot] b94284f000 Bump rocm-docs-core from 0.30.1 to 0.30.2 in /docs/sphinx (#1021) 
Bumps [rocm-docs-core](https://github.com/RadeonOpenCompute/rocm-docs-core) from 0.30.1 to 0.30.2.
- [Release notes](https://github.com/RadeonOpenCompute/rocm-docs-core/releases)
- [Changelog](https://github.com/RadeonOpenCompute/rocm-docs-core/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/RadeonOpenCompute/rocm-docs-core/compare/v0.30.1...v0.30.2)

---
updated-dependencies:
- dependency-name: rocm-docs-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

[ROCm/rccl commit: d8c53e90d7]
 2023-12-19 13:34:37 -07:00
PedramAlizadeh 401a104c5a resolved conflicts, fixed the localNetCount/0 bug 
[ROCm/rccl commit: 0d515f9388]
 2023-12-18 08:11:34 +00:00
akolliasAMD 9cb0f98e73 CMake does not allow for capital letters been used in package names (#1020) 
[ROCm/rccl commit: a924454f0f]
 2023-12-15 12:39:17 -07:00
Ziyue Yang e4b63a8ba0 Fully disable MSCCL when machine is not matched (#1017) 
* Disable MSCCL algorithm meta loading when machine is not matched

* fully disable init

* fix potential segfault

[ROCm/rccl commit: 655742a3a6]
 2023-12-13 08:36:21 -08:00
Wenkai Du 918ce6c2e2 msccl: disable on multi-node (#1018) 
[ROCm/rccl commit: 53d807a5b9]
 2023-12-13 07:41:40 -08:00
Wenkai Du 48107b18c9 msccl: fix data corruption with MTYPE_RW (#1014) 
[ROCm/rccl commit: 81602814a7]
 2023-12-11 20:33:15 -08:00
Bertan Dogancay fe5a902f97 correct package name (#1012) 
[ROCm/rccl commit: fca459baaf]
 2023-12-11 09:40:29 -07:00
Wenkai Du 481a35bc59 Fix memory fence and use non-temporal store (#1007) 
* Fix memory fence and use non-temporal store

* Use amdgcn builtin instead of inline asm

* Move threadfence location

* Revert changes to gfx90a

* Rework gfx90a change

* Apply changes to gfx94x

[ROCm/rccl commit: 7965c8b53c]
 2023-12-09 12:16:08 -08:00
Ziyue Yang fe250212bf Fix MSCCL scratch allocation (#1010) 
[ROCm/rccl commit: c002f20029]
 2023-12-08 17:47:10 -06:00
Ziyue Yang 62299668bd Tune MSCCL all-reduce algorithm (#1009) 
[ROCm/rccl commit: bb144dcd50]
 2023-12-08 17:47:02 -06:00
Wen-Heng (Jack) Chung 529f72ceaa Relax workgroup barrier implementation for MSCCL send/recv ops. (#997) 
* Trim logic.

* Revert "Trim logic."

This reverts commit 8f2dba6c764108acf2bf5428366b9f41d4d206b9.

* Introduce MSCCL template parameters to send / recv.

* Address review feedbacks.

[ROCm/rccl commit: baadda4bd8]
 2023-12-08 17:46:53 -06:00
Wenkai Du 22730bdc4d msccl: build same number of kernels as in ROCm 5.7 (#1005) 
Removed fullOps kernels from build

[ROCm/rccl commit: 12c08fc52a]
 2023-12-07 13:36:04 -06:00
dependabot[bot] e6e99a1ae9 Bump rocm-docs-core from 0.29.0 to 0.30.1 in /docs/sphinx (#1008) 
Bumps [rocm-docs-core](https://github.com/RadeonOpenCompute/rocm-docs-core) from 0.29.0 to 0.30.1.
- [Release notes](https://github.com/RadeonOpenCompute/rocm-docs-core/releases)
- [Changelog](https://github.com/RadeonOpenCompute/rocm-docs-core/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/RadeonOpenCompute/rocm-docs-core/compare/v0.29.0...v0.30.1)

---
updated-dependencies:
- dependency-name: rocm-docs-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

[ROCm/rccl commit: 9c3fea1751]
 2023-12-07 10:32:06 -07:00
Wen-Heng (Jack) Chung 0266febb31 Let 320KB message size uses LL protocol. (#1006) 
[ROCm/rccl commit: 8e8323252a]
 2023-12-06 18:14:31 -06:00
Wen-Heng (Jack) Chung 33aa8b67be Use a map to host scratch buffers (#1004) 
* Use a map to host scratch buffers

* Address review feedbacks. Deliberately keep mscclSetupScratch function.

[ROCm/rccl commit: 293f0fb752]
 2023-12-05 13:15:28 -06:00
Nilesh M Negi 403a91137c Fix gcnArch bug in IFC mix build (#998) (#1002) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: bc44e3faa7]
 2023-12-04 16:20:22 -06:00
Bertan Dogancay ae0bdad45c IFC mix build (#998) 
[ROCm/rccl commit: 7c0f49a878]
 2023-12-02 18:49:52 -07:00
Wenkai Du b38b7fa3a2 Increase max channles to 64 (#993) 
[ROCm/rccl commit: 4ba65d1d6a]
 2023-12-01 16:01:11 -08:00
pradeep-ramanna bf57487384 Fix GPU to NIC mapping for peertopeer (#994) 
[ROCm/rccl commit: 0b53f79196]
 2023-12-01 08:00:17 -08:00
Ziyue Yang cef45b8311 Fix mscclAlgoHandle not initialized issue (#995) 
[ROCm/rccl commit: e44e112a17]
 2023-12-01 07:58:01 -08:00
dependabot[bot] d237322b8d Bump cryptography from 41.0.4 to 41.0.6 in /docs/sphinx (#985) 
Bumps [cryptography](https://github.com/pyca/cryptography) from 41.0.4 to 41.0.6.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/41.0.4...41.0.6)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

[ROCm/rccl commit: ddd5c07b56]
 2023-11-30 15:05:50 -07:00
Ziyue Yang f0c47d085e Move MSCCL algorithm loading to initialization to workaround HIP graph conflict (#982) 
* MSCCL: pre-specify channels and pre-load algorithms

* add mutex

* fix bug

* clean include

* disable all-gathers temporarily

[ROCm/rccl commit: 4bb0b4a380]
 2023-11-30 09:47:20 -08:00
Bertan Dogancay 5efe13655d Renaming unit-tests package (#987) 
[ROCm/rccl commit: 20b02af19b]
 2023-11-29 15:05:32 -07:00
dependabot[bot] 51e0dd2ab8 Bump rocm-docs-core from 0.28.0 to 0.29.0 in /docs/sphinx (#980) 
Bumps [rocm-docs-core](https://github.com/RadeonOpenCompute/rocm-docs-core) from 0.28.0 to 0.29.0.
- [Release notes](https://github.com/RadeonOpenCompute/rocm-docs-core/releases)
- [Changelog](https://github.com/RadeonOpenCompute/rocm-docs-core/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/RadeonOpenCompute/rocm-docs-core/compare/v0.28.0...v0.29.0)

---
updated-dependencies:
- dependency-name: rocm-docs-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

[ROCm/rccl commit: 10a7cb7556]
 2023-11-29 09:39:33 -07:00
akolliasAMD bd982864d5 recreated pr 914 to work with current develop branch (#979) 
[ROCm/rccl commit: 56ce9ef05f]
 2023-11-28 16:33:47 -07:00
akolliasAMD 81cc39899b npkit trace script now syncs the on average difference per rank (#981) 
[ROCm/rccl commit: c71bae1608]
 2023-11-28 11:03:55 -07:00
gilbertlee-amd d0a194ec16 JitterBench (#975) 
[ROCm/rccl commit: 213869a6b4]
 2023-11-23 11:14:11 -07:00
Wenkai Du dcf623f2ec Add special handling of gfx940 (#976) 
* Add special handling of gfx940

* Update ring base

[ROCm/rccl commit: 50b2dd9fd7]
 2023-11-22 15:07:36 -08:00
dependabot[bot] 68ffd1e90d Bump rocm-docs-core from 0.27.0 to 0.28.0 in /docs/sphinx (#969) 
Bumps [rocm-docs-core](https://github.com/RadeonOpenCompute/rocm-docs-core) from 0.27.0 to 0.28.0.
- [Release notes](https://github.com/RadeonOpenCompute/rocm-docs-core/releases)
- [Changelog](https://github.com/RadeonOpenCompute/rocm-docs-core/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/RadeonOpenCompute/rocm-docs-core/compare/v0.27.0...v0.28.0)

---
updated-dependencies:
- dependency-name: rocm-docs-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

[ROCm/rccl commit: b1c746e7b5]
 2023-11-22 08:58:26 -07:00
Wenkai Du 4c2fa05a23 msccl: allocate scratch as ext-scope fine-grained (#968) 
[ROCm/rccl commit: 569d3f7d59]
 2023-11-16 09:57:25 -06:00
searlmc1 b5642f39ed Update README.md (#955) 
Remove references to HCC, which was removed from ROCm ~2yrs ago

[ROCm/rccl commit: 15fa77bb57]
 2023-11-15 18:01:45 -08:00
Wenkai Du 7c0920cd62 Fix kernel command line warnings (#961) 
* Fix kernel command line warnings

* Remove while loop

[ROCm/rccl commit: bc8661f092]
 2023-11-15 18:01:12 -08:00
Ziyue Yang 6ce074d92d Fix MSCCL work FIFO allocation with HIP graph enabled (#967) 
[ROCm/rccl commit: 7fc891bc8d]
 2023-11-15 16:43:28 -08:00
Bertan Dogancay 9e8eb41337 Check to support older ROCm versions (#963) 
[ROCm/rccl commit: 198f14923b]
 2023-11-15 12:36:31 -07:00
Ziyue Yang 2351578d5b Optimize MSCCL all-gather algorithms for gfx942 (#964) 
[ROCm/rccl commit: 7ae95db5b8]
 2023-11-15 08:18:59 -08:00
Ziyue Yang 2c6eededec Optimize MSCCL reduce primitive switching for gfx942 (#962) 
* Optimize reduce primitive switching for gfx942

* address comment

[ROCm/rccl commit: df128879a6]
 2023-11-15 08:18:44 -08:00