AI/rocm-systems
2
0
Fork 0
Koodi Ongelmat Pull-pyynnöt Actions Paketit Projektit Julkaisut Wiki Toiminta
1 426 Commitit 3 Branchit 0 Tagit
5f2b88bc28c13f187fc97beb4fa346440802ce11
Commit Graph

1426 Commitit

This Branch
This Branch
Kaikki haarat
Tekijä SHA1 Viesti Päivämäärä
Jack Taylor 5f2b88bc28 Add pytorch rccl/intra node all-reduce benchmark (#1221) 
* Add gpt-fast pytorch all reduce benchmark script

* Update readme instructions

* Minor changes
 2024-06-25 08:04:38 -07:00
Nusrat Islam 9f2514e5c8 Merge pull request #1223 from nusislam/minNchannels-multinode 
graph: fix minNchannels for multi-node
 2024-06-25 10:03:35 -05:00
Wenkai Du 5d7078e383 Fix DMABUF support (#1218) 
* Fix DMABUF support

* Reduce log output by moving dmabuf allocation details to TRACE

* Enable peer memory GDR support if ib_umem_get_peer is in kernel
 2024-06-25 08:00:15 -07:00
Nusrat Islam 05df0f8cea graph: fix minNchannels for multi-node 
Multi-node rccl was not correctly setting the minNchannels value. This
PR fixes the bug.
 2024-06-24 16:42:44 -05:00
dependabot[bot] 1ddb02c010 Bump urllib3 from 2.2.1 to 2.2.2 in /docs/sphinx (#1215) 
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.2.1 to 2.2.2.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.2.1...2.2.2)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-18 14:52:24 -06:00
dependabot[bot] 53dcfcc5e0 Bump rocm-docs-core from 1.1.3 to 1.4.0 in /docs/sphinx (#1213) 
Bumps [rocm-docs-core](https://github.com/ROCm/rocm-docs-core) from 1.1.3 to 1.4.0.
- [Release notes](https://github.com/ROCm/rocm-docs-core/releases)
- [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/ROCm/rocm-docs-core/compare/v1.1.3...v1.4.0)

---
updated-dependencies:
- dependency-name: rocm-docs-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-17 09:32:27 -06:00
Sam Wu 9f01acc030 Update Read the Docs configuration to use Python 3.10 and latest rocm-docs-core (#1190) 
* Add doc team as owners of RTD config

* Update Read the Docs configuration to use Python 3.10 and latest rocm-docs-core
 2024-06-14 12:12:22 -06:00
saurabhAMD 959545dce2 Merge pull request #1211 from saurabhAMD/channel 
enable UT to test with channels greater than 64
 2024-06-13 14:38:38 -05:00
saurabhAMD 392a73fdef enable UT to test with channels greater than 64 2024-06-13 13:54:08 -05:00
Paul Emberson 435756af02 fix initOnceFunc setting incorrect result code (#1205) 
Addresses DMA-BUF support check unexpectedly failing
 2024-06-07 16:47:19 -07:00
Nusrat Islam 9660e2e2dc Merge pull request #1200 from nusislam/multi-node-256-fix 
graph: fix multi-node channel count
 2024-06-07 14:34:20 -05:00
Nilesh M Negi d9661c17e6 Fix min_nchannels bug for gfx94* nranks=4 (#1202) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
 2024-06-07 14:31:28 -05:00
gilbertlee-amd 9b94a1052f Disabling NUMA maching for model 79 for some VM configs (#1204) 2024-06-06 17:15:04 -06:00
Nusrat Islam 526cce9bf4 graph: restrict maxChannels to 64 for multi-node and RCCL_ENABLE_INTRANET=1 2024-06-06 10:58:41 -05:00
Nusrat Islam 6ab20a7c6b graph: fix multi-node minChannel count 2024-06-06 10:56:39 -05:00
Wenkai Du 9fcd7b55e1 Allow multiple parameters during selective function generation (#1201) 
* Allow multiple parameters during selective function generation

* Remove debug print

* Add examples into Generator.cmake
 2024-06-06 07:07:24 -07:00
Nusrat Islam 955347bab4 Merge pull request #1184 from nusislam/256-channel-2 
add 256 channels support
 2024-06-04 08:25:34 -05:00
Nusrat Islam 9746d8ca3f set MIN_NCHANNEL limit to 64 for multi-node 2024-06-03 13:05:05 -05:00
Nusrat Islam 0634c5c8e1 doubling debug buffer size with increased channels 2024-06-03 13:05:05 -05:00
Nusrat Islam ef442f8f92 set MAXCHANNELS to 128 2024-06-03 13:05:05 -05:00
Nusrat Islam 9f654f6cf5 graph: restrict MAXCHANNELS for certain platforms 2024-06-03 13:05:01 -05:00
Nusrat Islam 48859a97b1 device: update the logic for channelId assignment 2024-06-03 13:03:18 -05:00
Nusrat Islam 506f16c506 add 256 channels support 2024-06-03 13:03:18 -05:00
akolliasAMD 6475da2ed9 fixed typo on BFD linkage (#1192) 2024-06-03 10:05:47 -06:00
gilbertlee-amd 0948eecbba Changing channel stride for MI300X multinode (#1196) 
* Shuffling MI300X multi-node channels
* Updating tree channel logic
 2024-06-03 10:00:55 -06:00
srawat 3301cdf59a doc organization (#1197) 
* doc organization

* removing what is rccl file

* Update index.rst
 2024-06-03 18:38:45 +05:30
ClementLinCF cab25f919e Optimize NCHANNELS and MSCCL config for gfx942 80CUs (#1195) 
* Optimize NCHANNELS and MSCCL config for gfx942 80CUs

Set appropriately for different NCCL_MIN_NCHANNELS and MSCCL config,
potentially improving communication perf on the MI300x 80CUs

* Delete tools/msccl-algorithms/allreduce_1step_mccl_8_2_16777216_LL.xml

* Change the factor of gfx94 and update msccl config
 2024-06-01 07:07:46 -07:00
Nilesh M Negi 5aaf7121d9 [BUILD] Update install.sh for RCCL build (#1191) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
 2024-05-31 17:58:34 -05:00
Nilesh M Negi 1249a6c3fd [MSCCL]: Move scratch buffer debug msgs to TRACE (#1189) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
 2024-05-31 17:54:23 -05:00
gilbertlee-amd 354e0b29a6 Addressing possible out-of-bounds mem access during channel duplication (#1193) 2024-05-30 14:02:14 -06:00
Wenkai Du 73221b4230 Add ring simple chunk size tuning (#1180) 
* Add ring simple chunk size tuning

* modifying the tuning table to improve the performance of broadcast for 8MB to 32MB for single-node MI300X after ring simple chunk size tuning

* modifying the tuning table to improve the performance of reduce for 1MB to 4MB for single-node MI300X after ring simple chunk size tuning

---------

Co-authored-by: PedramAlizadeh <pmohamma@amd.com>
 2024-05-29 07:59:47 -07:00
Wenkai Du 8f099b1adb Make WSL detection thread safe (#1178) 
* Make WSL detection thread safe

* Change static to beginning

* Switch to use atomics
 2024-05-28 17:23:50 -07:00
Edgar Gabriel a78c4f5e88 Merge pull request #1175 from edgargabriel/topic/alt_rsmi 
add alternative to rocm_smi_lib
 2024-05-28 09:36:55 -05:00
Joseph Macaranas e4c10e4438 Merge pull request #1188 from ROCm/amd/jmacaran/externalCIEnablement 
Enable external CI pipeline triggers
 2024-05-27 15:52:49 -04:00
amd-jmacaran 125f841c5f Enable external CI pipeline triggers 2024-05-23 16:29:05 -04:00
dependabot[bot] b06e617bea --- (#1183) 
updated-dependencies:
- dependency-name: requests
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-05-21 09:34:41 -06:00
Wenkai Du eeea3b693b Report error when collective is not enabled in build (#1177) 
* Report error when collective is not enabled in build

* Fix typo
 2024-05-16 10:11:12 -07:00
Edgar Gabriel 9ad913bfa8 add alternative to rocm_smi_lib 2024-05-14 13:51:41 -07:00
gilbertlee-amd 52fa5d1178 RCCL Replayer - multi communicator support (#1176) 2024-05-13 10:56:32 -06:00
Wenkai Du ecafc1969c Support WSL2 (#1173) 2024-05-10 07:31:12 -07:00
Tim f078db5998 Upload npkit_trace_analysis.py (#1152) 
script for parsing json trace, generating heatmap, throughput series, etc.
 2024-05-09 16:27:49 -04:00
mberenjk 87d01e6bf5 Changing the failure print to reflect a bad installation of MSCCL (#1160) 
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>
 2024-05-08 16:56:26 -05:00
Wenkai Du a64aab5f63 Use rocm-smi thread only mutex when available (#1169) 2024-05-08 14:32:24 -07:00
Pedram Alizadeh 73acf3eeec modifying the tuning table to improve the performance of broadcast for 1MB to 64MB for single-node MI300X (#1172) 2024-05-08 15:49:33 -04:00
mberenjk 408278209d Adding ASAN changes to address memory leak issue" (#1170) 
Co-authored-by: akolliasAMD <akollias@amd.com>
 2024-05-08 09:16:00 -05:00
Wenkai Du b18784d8b8 Add compiler warning for uninitialized variable and fix (#1163) 
* Add compiler warning for uninitialized variable and fix

* Add -Wsometimes-uninitialized

* Convert warning to error
 2024-05-08 07:00:25 -07:00
Wenkai Du f679db6ff6 Use normal permute path when one NIC per GPU (#1171) 2024-05-08 06:59:57 -07:00
Wenkai Du a0cef69110 npkit: add broadcast trace (#1166) 2024-05-07 14:00:16 -07:00
Pak Nin Lui 92a4fc6204 Merge pull request #1167 from paklui/dmabuf 
fix typo for DMABUF_ENABLE
 2024-05-07 08:48:44 -07:00
dependabot[bot] eb562e7b22 Bump jinja2 from 3.1.3 to 3.1.4 in /docs/sphinx (#1168) 
Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.4.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/jinja/compare/3.1.3...3.1.4)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-05-06 15:35:34 -06:00