71809e8bc0
Updating RCCL Replayer README ( #1408 )
...
[ROCm/rccl commit: cb1027de97
2024-11-05 08:06:11 -07:00
fcb0b2da3f
[Replayer] Add validation ( #1387 )
...
* Add validation to rccl_replayer
[ROCm/rccl commit: cfecce790f
2024-10-22 10:41:08 -04:00
5f8571dcbc
msccl: disable 1-shot xmls ( #1375 )
...
MSCCL 1-shot xmls may cause different output values on different ranks.
Disabling them for now to avoid undefined behavior in applications.
[ROCm/rccl commit: 62d10fdc25
2024-10-14 15:10:53 -07:00
9ad1fe571b
Temporarily disable MSCCL all gather XMLs due to UT failure ( #1373 )
...
[ROCm/rccl commit: a680e329e6
2024-10-12 08:43:16 -07:00
9059445acb
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: 84081064a0
2024-10-02 09:31:25 -05:00
74aa13afbe
Add another Rome model ( #1354 )
...
[ROCm/rccl commit: e453f1ced9
2024-10-01 17:41:27 -05:00
27e0569eed
topo_expl: update sm fields in topo xml files ( #1310 )
...
[ROCm/rccl commit: 1a48e19b18
2024-08-29 12:03:51 -07:00
5368ea024a
Merge pull request #1299 from edgargabriel/topic/remove-multirank-examples
...
Remove MultiRank examples
[ROCm/rccl commit: bba3559334
2024-08-23 08:32:16 -05:00
157cc5f6ba
Add new Rome model ( #1304 )
...
* Add another rome model and override
* Fix bug
* Fix typo
* Add ring
* Update ring
* Fix model matching
* Clean up
* Clean up
* Reverse rings for NCCL_RINGS input
* Only reverse NCCL_RINGS for ring graph
* Fix mapping issue when using NCCL_RINGS
* Add NCCL_RINGS_REMAP to handle inconsistant net names
[ROCm/rccl commit: 532b70afb6
2024-08-23 08:45:43 +08:00
3d44929d08
Remove MultiRank examples
...
remove the MultiRank examples, the features was never released (because
it didn't work reliably), and it might just cause confusion if somebody
sees it. In additional, the locdation in tools was suboptimal.
[ROCm/rccl commit: 8953a26bcd
2024-08-14 14:11:16 -07:00
30e3db969f
Fix number of loops in p2p-latency-test ( #1286 )
...
[ROCm/rccl commit: 145a13235a
2024-08-05 13:35:56 -07:00
d2df042c36
topo_expl: Update channel masks for >64 channels ( #1279 )
...
[ROCm/rccl commit: 4bc118336a
2024-07-25 17:27:34 -07:00
24e1ed5288
Adding performance collection feature in rccl_replayer, and updating MSCCL logging and replayer parsing ( #1265 )
...
* Adding performance collection feature in rccl_replayer, and updating MSCCL logging and replayer parsing
* Performance collection feature in rccl_replayer, and updating MSCCL logging and replayer parsing
[ROCm/rccl commit: cf311b71ee
2024-07-22 10:21:29 -05:00
863b213fd2
adding rocprof and pytorch parser scripts ( #1214 )
...
* adding rocprof parser script
* adding the support for multiple json files
* adding pytorch profiler script
* remove filtering from pytorch log
* adding the addressing the comments and add the feature to parse all kernels
* completing the report for torch profiler
---------
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com >
[ROCm/rccl commit: 519843d2cf
2024-07-19 14:51:28 -05:00
ebcac26530
Add pytorch rccl/intra node all-reduce benchmark ( #1221 )
...
* Add gpt-fast pytorch all reduce benchmark script
* Update readme instructions
* Minor changes
[ROCm/rccl commit: 5f2b88bc28
2024-06-25 08:04:38 -07:00
b34fd115a1
doubling debug buffer size with increased channels
...
[ROCm/rccl commit: 0634c5c8e1
2024-06-03 13:05:05 -05:00
4f56aa5f8c
Optimize NCHANNELS and MSCCL config for gfx942 80CUs ( #1195 )
...
* Optimize NCHANNELS and MSCCL config for gfx942 80CUs
Set appropriately for different NCCL_MIN_NCHANNELS and MSCCL config,
potentially improving communication perf on the MI300x 80CUs
* Delete tools/msccl-algorithms/allreduce_1step_mccl_8_2_16777216_LL.xml
* Change the factor of gfx94 and update msccl config
[ROCm/rccl commit: cab25f919e
2024-06-01 07:07:46 -07:00
33a9e1c29f
RCCL Replayer - multi communicator support ( #1176 )
...
[ROCm/rccl commit: 52fa5d1178
2024-05-13 10:56:32 -06:00
b7c743b3a0
Upload npkit_trace_analysis.py ( #1152 )
...
script for parsing json trace, generating heatmap, throughput series, etc.
[ROCm/rccl commit: f078db5998
2024-05-09 16:27:49 -04:00
3906e992f8
MSCCL: add support for out-of-place all reduce ( #1156 )
...
[ROCm/rccl commit: 4e1b8c1cbb
2024-04-28 19:49:09 -07:00
e494f29235
Replace __HIP_PLATFORM_HCC__ with __HIP_PLATFORM_AMD__ ( #1154 )
...
[ROCm/rccl commit: 9e0c9b4ed8
2024-04-25 07:19:18 -07:00
422a7ffcbb
Rail optimization for rings ( #1140 )
...
- Modifies the ring creation algorithm to be friendlier to rail-optimized topologies (should not affect classic fabric topologies)
[ROCm/rccl commit: 4cb62f999a
2024-04-15 12:03:57 -06:00
62b9f0d3a7
[topo_expl] Adding -n option to override number of nodes ( #1134 )
...
[ROCm/rccl commit: 93982533d7
2024-04-04 15:11:47 -06:00
890fafc2f7
rccl_prim_test: increase max number of workgroups and test iterations ( #1132 )
...
[ROCm/rccl commit: e8c76fd806
2024-04-03 11:29:21 -07:00
62a6a07d49
Replaced ROCmSoftwarePlatform and RadeonOpenCompute links with ROCm links. ( #1125 )
...
[ROCm/rccl commit: 503a472a25
2024-03-25 16:29:13 -06:00
19897f8d90
Fixes the copyright comment block on each of topo_expl/models/*.xml. The format was not valid XML. ( #1124 )
...
[ROCm/rccl commit: 9eefc68cb5
2024-03-25 16:21:17 -06:00
61f89d680d
msccl algorithms tuning for alltoall on MI300 ( #1120 )
...
Co-authored-by: PedramAlizadeh <amd@pmohamma.com >
[ROCm/rccl commit: c2fc1d6809
2024-03-21 20:35:29 -04:00
17b9546da9
msccl algorithms tuning for allgather on MI300 ( #1110 )
...
[ROCm/rccl commit: 50f22e8317
2024-03-14 12:18:26 -04:00
e373bd44bf
Enable fp8 support ( #1101 )
...
* initial checkin
* resolve cr comments
* resolve the build issue
* fix the data correctless issue
* update fp8 header file and update the unit test for fp8 support
* remove fp16 from fp8 headers
* fix ut issue and catch up the latest code from develop
* udate according to cr comments
* update ut according to cr comments
* update num floats for each SumPostDiv from 4 to 6
* update fp8 header file name
* fix the typo
[ROCm/rccl commit: 6777e65c1d
2024-03-08 15:17:53 -08:00
c2eff3ecd9
topo_expl: 2.19.4 update and fix build error ( #1098 )
...
[ROCm/rccl commit: d2224fd3e1
2024-03-07 08:52:50 -08:00
058886cb20
Add another Rome model ( #1095 )
...
[ROCm/rccl commit: df98a6957d
2024-02-28 10:46:05 -08:00
874998033f
Add new GPU model ( #1080 )
...
[ROCm/rccl commit: 74f9e5db64
2024-02-23 12:19:42 -08:00
bf48d1bc4d
msccl algorithms tuning for allreduce on MI300 ( #1088 )
...
[ROCm/rccl commit: 5a0f9990a9
2024-02-21 11:31:56 -05:00
7842411fb3
Merge remote-tracking branch 'rccl/develop' into 2.19.4
...
[ROCm/rccl commit: 76f83f95ab
2024-02-15 13:37:14 -08:00
5d44815d95
Npkit updates ( #1084 )
...
* removed warmup runs to be an opt in
[ROCm/rccl commit: 16d7f372b7
2024-02-15 07:48:45 -07:00
df1d9b2415
topo_expl: 2.19 update
...
[ROCm/rccl commit: d1575a1622
2024-01-31 16:11:14 -06:00
366cd12bed
topo-expl: fix broken build ( #1048 )
...
[ROCm/rccl commit: 600b44fee5
2024-01-17 08:59:03 -08:00
cd7a346297
Doubling buffer size to fix NCCL INFO corruption with increased channels ( #1035 )
...
[ROCm/rccl commit: f7e39fced2
2024-01-08 08:14:33 -08:00
4eaf90f84c
p2p-latency-tests: fix build by switching to gcnArchName ( #1030 )
...
* p2p-latency-tests: fix build by switching to gcnArchName
* rccl-prim-test: switch to gcnArchName
[ROCm/rccl commit: cfc04a8aef
2024-01-04 13:36:48 -08:00
e3d45f9de4
Improve MSCCL algorithms ( #1023 )
...
[ROCm/rccl commit: 0a53077c9c
2024-01-03 14:51:34 -08:00
62299668bd
Tune MSCCL all-reduce algorithm ( #1009 )
...
[ROCm/rccl commit: bb144dcd50
2023-12-08 17:47:02 -06:00
0266febb31
Let 320KB message size uses LL protocol. ( #1006 )
...
[ROCm/rccl commit: 8e8323252a
2023-12-06 18:14:31 -06:00
cef45b8311
Fix mscclAlgoHandle not initialized issue ( #995 )
...
[ROCm/rccl commit: e44e112a17
2023-12-01 07:58:01 -08:00
f0c47d085e
Move MSCCL algorithm loading to initialization to workaround HIP graph conflict ( #982 )
...
* MSCCL: pre-specify channels and pre-load algorithms
* add mutex
* fix bug
* clean include
* disable all-gathers temporarily
[ROCm/rccl commit: 4bb0b4a380
2023-11-30 09:47:20 -08:00
81cc39899b
npkit trace script now syncs the on average difference per rank ( #981 )
...
[ROCm/rccl commit: c71bae1608
2023-11-28 11:03:55 -07:00
d0a194ec16
JitterBench ( #975 )
...
[ROCm/rccl commit: 213869a6b4
2023-11-23 11:14:11 -07:00
dcf623f2ec
Add special handling of gfx940 ( #976 )
...
* Add special handling of gfx940
* Update ring base
[ROCm/rccl commit: 50b2dd9fd7
2023-11-22 15:07:36 -08:00
2351578d5b
Optimize MSCCL all-gather algorithms for gfx942 ( #964 )
...
[ROCm/rccl commit: 7ae95db5b8
2023-11-15 08:18:59 -08:00
d8471eaddf
Adding LaunchBench tool ( #952 )
...
[ROCm/rccl commit: d50bab28bf
2023-11-03 12:04:52 -06:00
691df735a3
Revert "Introduce allgather for MSCCL on 8 sockets up to 320KB. ( #931 )" ( #939 )
...
This reverts commit 769f00db5c9f02ee8dea
2023-10-30 23:52:58 -06:00
gilbertlee-amd
Bertan Dogancay
Wenkai Du
BertanDogancay
Edgar Gabriel
Edgar Gabriel
Ziyue Yang
Benjamin Kitor
saurabhAMD
mberenjk
Jack Taylor
Nusrat Islam
ClementLinCF
Tim
corey-derochie-amd
Pedram Alizadeh
Andy li
akolliasAMD
Wenkai Du
Wen-Heng (Jack) Chung