84081064a0
Merge remote-tracking branch 'nccl/master' into develop
2024-10-02 09:31:25 -05:00
e453f1ced9
Add another Rome model ( #1354 )
2024-10-01 17:41:27 -05:00
1a48e19b18
topo_expl: update sm fields in topo xml files ( #1310 )
2024-08-29 12:03:51 -07:00
bba3559334
Merge pull request #1299 from edgargabriel/topic/remove-multirank-examples
...
Remove MultiRank examples
2024-08-23 08:32:16 -05:00
532b70afb6
Add new Rome model ( #1304 )
...
* Add another rome model and override
* Fix bug
* Fix typo
* Add ring
* Update ring
* Fix model matching
* Clean up
* Clean up
* Reverse rings for NCCL_RINGS input
* Only reverse NCCL_RINGS for ring graph
* Fix mapping issue when using NCCL_RINGS
* Add NCCL_RINGS_REMAP to handle inconsistant net names
2024-08-23 08:45:43 +08:00
8953a26bcd
Remove MultiRank examples
...
remove the MultiRank examples, the features was never released (because
it didn't work reliably), and it might just cause confusion if somebody
sees it. In additional, the locdation in tools was suboptimal.
2024-08-14 14:11:16 -07:00
145a13235a
Fix number of loops in p2p-latency-test ( #1286 )
2024-08-05 13:35:56 -07:00
4bc118336a
topo_expl: Update channel masks for >64 channels ( #1279 )
2024-07-25 17:27:34 -07:00
cf311b71ee
Adding performance collection feature in rccl_replayer, and updating MSCCL logging and replayer parsing ( #1265 )
...
* Adding performance collection feature in rccl_replayer, and updating MSCCL logging and replayer parsing
* Performance collection feature in rccl_replayer, and updating MSCCL logging and replayer parsing
2024-07-22 10:21:29 -05:00
519843d2cf
adding rocprof and pytorch parser scripts ( #1214 )
...
* adding rocprof parser script
* adding the support for multiple json files
* adding pytorch profiler script
* remove filtering from pytorch log
* adding the addressing the comments and add the feature to parse all kernels
* completing the report for torch profiler
---------
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com >
2024-07-19 14:51:28 -05:00
5f2b88bc28
Add pytorch rccl/intra node all-reduce benchmark ( #1221 )
...
* Add gpt-fast pytorch all reduce benchmark script
* Update readme instructions
* Minor changes
2024-06-25 08:04:38 -07:00
0634c5c8e1
doubling debug buffer size with increased channels
2024-06-03 13:05:05 -05:00
cab25f919e
Optimize NCHANNELS and MSCCL config for gfx942 80CUs ( #1195 )
...
* Optimize NCHANNELS and MSCCL config for gfx942 80CUs
Set appropriately for different NCCL_MIN_NCHANNELS and MSCCL config,
potentially improving communication perf on the MI300x 80CUs
* Delete tools/msccl-algorithms/allreduce_1step_mccl_8_2_16777216_LL.xml
* Change the factor of gfx94 and update msccl config
2024-06-01 07:07:46 -07:00
52fa5d1178
RCCL Replayer - multi communicator support ( #1176 )
2024-05-13 10:56:32 -06:00
f078db5998
Upload npkit_trace_analysis.py ( #1152 )
...
script for parsing json trace, generating heatmap, throughput series, etc.
2024-05-09 16:27:49 -04:00
4e1b8c1cbb
MSCCL: add support for out-of-place all reduce ( #1156 )
2024-04-28 19:49:09 -07:00
9e0c9b4ed8
Replace __HIP_PLATFORM_HCC__ with __HIP_PLATFORM_AMD__ ( #1154 )
2024-04-25 07:19:18 -07:00
4cb62f999a
Rail optimization for rings ( #1140 )
...
- Modifies the ring creation algorithm to be friendlier to rail-optimized topologies (should not affect classic fabric topologies)
2024-04-15 12:03:57 -06:00
93982533d7
[topo_expl] Adding -n option to override number of nodes ( #1134 )
2024-04-04 15:11:47 -06:00
e8c76fd806
rccl_prim_test: increase max number of workgroups and test iterations ( #1132 )
2024-04-03 11:29:21 -07:00
503a472a25
Replaced ROCmSoftwarePlatform and RadeonOpenCompute links with ROCm links. ( #1125 )
2024-03-25 16:29:13 -06:00
9eefc68cb5
Fixes the copyright comment block on each of topo_expl/models/*.xml. The format was not valid XML. ( #1124 )
2024-03-25 16:21:17 -06:00
c2fc1d6809
msccl algorithms tuning for alltoall on MI300 ( #1120 )
...
Co-authored-by: PedramAlizadeh <amd@pmohamma.com >
2024-03-21 20:35:29 -04:00
50f22e8317
msccl algorithms tuning for allgather on MI300 ( #1110 )
2024-03-14 12:18:26 -04:00
6777e65c1d
Enable fp8 support ( #1101 )
...
* initial checkin
* resolve cr comments
* resolve the build issue
* fix the data correctless issue
* update fp8 header file and update the unit test for fp8 support
* remove fp16 from fp8 headers
* fix ut issue and catch up the latest code from develop
* udate according to cr comments
* update ut according to cr comments
* update num floats for each SumPostDiv from 4 to 6
* update fp8 header file name
* fix the typo
2024-03-08 15:17:53 -08:00
d2224fd3e1
topo_expl: 2.19.4 update and fix build error ( #1098 )
2024-03-07 08:52:50 -08:00
df98a6957d
Add another Rome model ( #1095 )
2024-02-28 10:46:05 -08:00
74f9e5db64
Add new GPU model ( #1080 )
2024-02-23 12:19:42 -08:00
5a0f9990a9
msccl algorithms tuning for allreduce on MI300 ( #1088 )
2024-02-21 11:31:56 -05:00
76f83f95ab
Merge remote-tracking branch 'rccl/develop' into 2.19.4
2024-02-15 13:37:14 -08:00
16d7f372b7
Npkit updates ( #1084 )
...
* removed warmup runs to be an opt in
2024-02-15 07:48:45 -07:00
d1575a1622
topo_expl: 2.19 update
2024-01-31 16:11:14 -06:00
600b44fee5
topo-expl: fix broken build ( #1048 )
2024-01-17 08:59:03 -08:00
f7e39fced2
Doubling buffer size to fix NCCL INFO corruption with increased channels ( #1035 )
2024-01-08 08:14:33 -08:00
cfc04a8aef
p2p-latency-tests: fix build by switching to gcnArchName ( #1030 )
...
* p2p-latency-tests: fix build by switching to gcnArchName
* rccl-prim-test: switch to gcnArchName
2024-01-04 13:36:48 -08:00
0a53077c9c
Improve MSCCL algorithms ( #1023 )
2024-01-03 14:51:34 -08:00
bb144dcd50
Tune MSCCL all-reduce algorithm ( #1009 )
2023-12-08 17:47:02 -06:00
8e8323252a
Let 320KB message size uses LL protocol. ( #1006 )
2023-12-06 18:14:31 -06:00
e44e112a17
Fix mscclAlgoHandle not initialized issue ( #995 )
2023-12-01 07:58:01 -08:00
4bb0b4a380
Move MSCCL algorithm loading to initialization to workaround HIP graph conflict ( #982 )
...
* MSCCL: pre-specify channels and pre-load algorithms
* add mutex
* fix bug
* clean include
* disable all-gathers temporarily
2023-11-30 09:47:20 -08:00
c71bae1608
npkit trace script now syncs the on average difference per rank ( #981 )
2023-11-28 11:03:55 -07:00
213869a6b4
JitterBench ( #975 )
2023-11-23 11:14:11 -07:00
50b2dd9fd7
Add special handling of gfx940 ( #976 )
...
* Add special handling of gfx940
* Update ring base
2023-11-22 15:07:36 -08:00
7ae95db5b8
Optimize MSCCL all-gather algorithms for gfx942 ( #964 )
2023-11-15 08:18:59 -08:00
d50bab28bf
Adding LaunchBench tool ( #952 )
2023-11-03 12:04:52 -06:00
9f02ee8dea
Revert "Introduce allgather for MSCCL on 8 sockets up to 320KB. ( #931 )" ( #939 )
...
This reverts commit bfb8642450
2023-10-30 23:52:58 -06:00
bfb8642450
Introduce allgather for MSCCL on 8 sockets up to 320KB. ( #931 )
2023-10-24 18:41:12 -05:00
3f9ffe4788
Introduce allgather MSCCL XML specification for MI250X up to 320KB. ( #930 )
2023-10-24 18:35:55 -05:00
72d5fbddfd
Introduce 1-shot allreduce for MI250X Hayabusa. ( #929 )
2023-10-24 16:31:18 -05:00
341926c60a
Introduce 1pass allreduce. Tailor it for very small message sizes <= 20KB. ( #919 )
2023-10-16 16:31:08 -05:00
BertanDogancay
Wenkai Du
Edgar Gabriel
Edgar Gabriel
Ziyue Yang
Benjamin Kitor
saurabhAMD
mberenjk
Jack Taylor
Nusrat Islam
ClementLinCF
gilbertlee-amd
Tim
corey-derochie-amd
Pedram Alizadeh
Andy li
akolliasAMD
Wenkai Du
Wen-Heng (Jack) Chung