519843d2cf
adding rocprof and pytorch parser scripts ( #1214 )
...
* adding rocprof parser script
* adding the support for multiple json files
* adding pytorch profiler script
* remove filtering from pytorch log
* adding the addressing the comments and add the feature to parse all kernels
* completing the report for torch profiler
---------
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com >
2024-07-19 14:51:28 -05:00
6f331b0d43
Enable CPX mode for MI300X ( #1259 )
...
* graph: enable cpx mode for MI300X
* graph: tune limits for cpx and cleanup
2024-07-19 11:30:37 -05:00
89349f2ce4
Template unroll for RCCL kernels ( #1250 )
...
* Template unroll for RCCL kernels
* Adding unroll template arg during CMake hipification
* Reduce linking parallel jobs to avoid OOM in CI
* Workaround issues with UT tests
SWDEV-469533: register spill fix is needed for mainline build
LWPCOMMLIBS-369: cannot enable 112 channels with 80 CUs
Use -parallel-jobs=8 for linking
* CI: do not use -j 16 when building
* CI: use -j 8 when building
* Only reduce parallel linking job for CI extended
* Restore original jenkins command. Change parallel linking jobs in cmake
* Disable MSCCLPP
---------
Co-authored-by: gilbertlee-amd <gilbert.lee@amd.com >
2024-07-19 08:15:59 -07:00
a1ef217b32
Consistent channel shuffling for MI300X multi-node ( #1255 )
...
* Revert "[GRAPH] Use channel shuffling only for IB systems (#1228 )"
This reverts commit 5be3b713ef#1196 )" (#1224 )"
This reverts commit ad31d93f3d
2024-07-18 10:18:09 -05:00
67e867271f
[GRAPH] Disable MSCCL override of no. of channels ( #1187 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2024-07-15 10:45:21 -05:00
9cbb3da224
Only enable MSCCL++ AllReduce for message sizes that are multiples 32 ( #1253 )
...
* Only enable MSCCL++ AllReduce for message sizes that are multiples of 32. MSCCL++ does not handle these other sizes.
* Sanitized MSCCL++ logging.
2024-07-12 17:04:23 -07:00
6dc47eecd7
Integrated RCCL with MSCCL++ for small message sizes ( #1231 )
2024-07-12 15:32:58 -06:00
c755b9cf93
Improved version reporting in NCCL_DEBUG=VERSION ( #1232 )
...
* Improved version reporting in NCCL_DEBUG=VERSION.
Signed-off-by: rahulvaidya20 <ravaidya@amd.com >
* Version reporting changes
Signed-off-by: rahulvaidya20 <ravaidya@amd.com >
* Versioning changes: Initialized char arrays to null and fixed typo.
---------
Signed-off-by: rahulvaidya20 <ravaidya@amd.com >
2024-07-12 08:14:29 -05:00
63e4d76e23
gfx12 initial enablement ( #1219 )
2024-07-10 13:32:09 -06:00
7e78641dc1
cleaned codeowners file ( #1247 )
2024-07-09 10:31:23 -06:00
71e0f551e7
Bump certifi from 2024.2.2 to 2024.7.4 in /docs/sphinx ( #1241 )
...
Bumps [certifi](https://github.com/certifi/python-certifi ) from 2024.2.2 to 2024.7.4.
- [Commits](https://github.com/certifi/python-certifi/compare/2024.02.02...2024.07.04 )
---
updated-dependencies:
- dependency-name: certifi
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-08 11:49:35 -06:00
799a8b5e59
Bump rocm-docs-core from 1.4.1 to 1.5.0 in /docs/sphinx ( #1240 )
...
Bumps [rocm-docs-core](https://github.com/ROCm/rocm-docs-core ) from 1.4.1 to 1.5.0.
- [Release notes](https://github.com/ROCm/rocm-docs-core/releases )
- [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md )
- [Commits](https://github.com/ROCm/rocm-docs-core/compare/v1.4.1...v1.5.0 )
---
updated-dependencies:
- dependency-name: rocm-docs-core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-08 10:59:32 -06:00
0c36d571ea
Enable multi-threading for MSCCL ( #1203 )
...
MSCCL can now run in a multi-threaded configuration. To test in the unit tests, added the ENABLE_OPENMP compile definition flag and the --openmp-test-enable flag to the unit test build script. To activate, set the environment variables UT_MULTITHREADED=1 and UT_PROCESS_MASK=1. Set Jenkins to use this mode.
2024-07-04 09:34:38 -06:00
45f3fbc52f
Checking kernel header files only when missing sysfs entry ( #1239 )
2024-07-03 15:53:15 -07:00
aeaaacad26
Bump rocm-docs-core from 1.4.0 to 1.4.1 in /docs/sphinx ( #1233 )
...
Bumps [rocm-docs-core](https://github.com/ROCm/rocm-docs-core ) from 1.4.0 to 1.4.1.
- [Release notes](https://github.com/ROCm/rocm-docs-core/releases )
- [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md )
- [Commits](https://github.com/ROCm/rocm-docs-core/compare/v1.4.0...v1.4.1 )
---
updated-dependencies:
- dependency-name: rocm-docs-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-03 09:32:49 -06:00
5be3b713ef
[GRAPH] Use channel shuffling only for IB systems ( #1228 )
...
* [GRAPH] Use channel shuffling only for IB systems
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
* [GRAPH] Define channels=48 for gfx94 RoCE systems
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
* [GRAPH] Increase channels for RoCE gfx94 systems
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
---------
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2024-07-02 12:20:40 -05:00
9d8f68b4ee
NPKit: separate time stamps for GPU access from different blocks ( #1229 )
...
To avoid races in memory access in GPU
2024-06-28 08:00:22 -07:00
b09ea29d66
graph: fix minNchannels for multi-node overwrite ( #1230 )
2024-06-26 16:56:10 -05:00
ad31d93f3d
Revert "Changing channel stride for MI300X multinode ( #1196 )" ( #1224 )
...
This reverts channel stride change in commit
0948eecbba
2024-06-25 14:03:30 -07:00
e170f41ddd
Unit Tests for testing channels ( #1222 )
2024-06-25 10:10:10 -05:00
5f2b88bc28
Add pytorch rccl/intra node all-reduce benchmark ( #1221 )
...
* Add gpt-fast pytorch all reduce benchmark script
* Update readme instructions
* Minor changes
2024-06-25 08:04:38 -07:00
9f2514e5c8
Merge pull request #1223 from nusislam/minNchannels-multinode
...
graph: fix minNchannels for multi-node
2024-06-25 10:03:35 -05:00
5d7078e383
Fix DMABUF support ( #1218 )
...
* Fix DMABUF support
* Reduce log output by moving dmabuf allocation details to TRACE
* Enable peer memory GDR support if ib_umem_get_peer is in kernel
2024-06-25 08:00:15 -07:00
05df0f8cea
graph: fix minNchannels for multi-node
...
Multi-node rccl was not correctly setting the minNchannels value. This
PR fixes the bug.
2024-06-24 16:42:44 -05:00
1ddb02c010
Bump urllib3 from 2.2.1 to 2.2.2 in /docs/sphinx ( #1215 )
...
Bumps [urllib3](https://github.com/urllib3/urllib3 ) from 2.2.1 to 2.2.2.
- [Release notes](https://github.com/urllib3/urllib3/releases )
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst )
- [Commits](https://github.com/urllib3/urllib3/compare/2.2.1...2.2.2 )
---
updated-dependencies:
- dependency-name: urllib3
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-18 14:52:24 -06:00
53dcfcc5e0
Bump rocm-docs-core from 1.1.3 to 1.4.0 in /docs/sphinx ( #1213 )
...
Bumps [rocm-docs-core](https://github.com/ROCm/rocm-docs-core ) from 1.1.3 to 1.4.0.
- [Release notes](https://github.com/ROCm/rocm-docs-core/releases )
- [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md )
- [Commits](https://github.com/ROCm/rocm-docs-core/compare/v1.1.3...v1.4.0 )
---
updated-dependencies:
- dependency-name: rocm-docs-core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-17 09:32:27 -06:00
9f01acc030
Update Read the Docs configuration to use Python 3.10 and latest rocm-docs-core ( #1190 )
...
* Add doc team as owners of RTD config
* Update Read the Docs configuration to use Python 3.10 and latest rocm-docs-core
2024-06-14 12:12:22 -06:00
959545dce2
Merge pull request #1211 from saurabhAMD/channel
...
enable UT to test with channels greater than 64
2024-06-13 14:38:38 -05:00
392a73fdef
enable UT to test with channels greater than 64
2024-06-13 13:54:08 -05:00
435756af02
fix initOnceFunc setting incorrect result code ( #1205 )
...
Addresses DMA-BUF support check unexpectedly failing
2024-06-07 16:47:19 -07:00
9660e2e2dc
Merge pull request #1200 from nusislam/multi-node-256-fix
...
graph: fix multi-node channel count
2024-06-07 14:34:20 -05:00
d9661c17e6
Fix min_nchannels bug for gfx94* nranks=4 ( #1202 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2024-06-07 14:31:28 -05:00
9b94a1052f
Disabling NUMA maching for model 79 for some VM configs ( #1204 )
2024-06-06 17:15:04 -06:00
526cce9bf4
graph: restrict maxChannels to 64 for multi-node and RCCL_ENABLE_INTRANET=1
2024-06-06 10:58:41 -05:00
6ab20a7c6b
graph: fix multi-node minChannel count
2024-06-06 10:56:39 -05:00
9fcd7b55e1
Allow multiple parameters during selective function generation ( #1201 )
...
* Allow multiple parameters during selective function generation
* Remove debug print
* Add examples into Generator.cmake
2024-06-06 07:07:24 -07:00
955347bab4
Merge pull request #1184 from nusislam/256-channel-2
...
add 256 channels support
2024-06-04 08:25:34 -05:00
9746d8ca3f
set MIN_NCHANNEL limit to 64 for multi-node
2024-06-03 13:05:05 -05:00
0634c5c8e1
doubling debug buffer size with increased channels
2024-06-03 13:05:05 -05:00
ef442f8f92
set MAXCHANNELS to 128
2024-06-03 13:05:05 -05:00
9f654f6cf5
graph: restrict MAXCHANNELS for certain platforms
2024-06-03 13:05:01 -05:00
48859a97b1
device: update the logic for channelId assignment
2024-06-03 13:03:18 -05:00
506f16c506
add 256 channels support
2024-06-03 13:03:18 -05:00
6475da2ed9
fixed typo on BFD linkage ( #1192 )
2024-06-03 10:05:47 -06:00
0948eecbba
Changing channel stride for MI300X multinode ( #1196 )
...
* Shuffling MI300X multi-node channels
* Updating tree channel logic
2024-06-03 10:00:55 -06:00
3301cdf59a
doc organization ( #1197 )
...
* doc organization
* removing what is rccl file
* Update index.rst
2024-06-03 18:38:45 +05:30
cab25f919e
Optimize NCHANNELS and MSCCL config for gfx942 80CUs ( #1195 )
...
* Optimize NCHANNELS and MSCCL config for gfx942 80CUs
Set appropriately for different NCCL_MIN_NCHANNELS and MSCCL config,
potentially improving communication perf on the MI300x 80CUs
* Delete tools/msccl-algorithms/allreduce_1step_mccl_8_2_16777216_LL.xml
* Change the factor of gfx94 and update msccl config
2024-06-01 07:07:46 -07:00
5aaf7121d9
[BUILD] Update install.sh for RCCL build ( #1191 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2024-05-31 17:58:34 -05:00
1249a6c3fd
[MSCCL]: Move scratch buffer debug msgs to TRACE ( #1189 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2024-05-31 17:54:23 -05:00
354e0b29a6
Addressing possible out-of-bounds mem access during channel duplication ( #1193 )
2024-05-30 14:02:14 -06:00
mberenjk
Nusrat Islam
Wenkai Du
Nilesh M Negi
corey-derochie-amd
Rahul Vaidya
akolliasAMD
dependabot[bot]
saurabhAMD
Jack Taylor
Sam Wu
saurabhAMD
Paul Emberson
gilbertlee-amd
srawat
ClementLinCF