ddc5d58b93
Rail optimized trees ( #1540 )
...
* Allow disabling rail-optimized trees via RCCL_DISABLE_RAIL_TREES, Graphviz-friendly output via RCCL_OUTPUT_TREES
2025-02-20 15:18:29 -07:00
6cb0599e38
Updating topology explorer ( #1536 )
2025-02-07 08:44:04 -07:00
a05329bd0d
Add Topologies for 16-GPU gfx942 SuperNode ( #1417 )
...
* Add Topologies for 16-GPU gfx942 SuperNode
- Add GigaIO topologies to tools/topo_expl for dev and testing
- Add GigaIO Columba 16 GPU romeModel and adjust topology
matching algorithm in rome_models for 16 GPU system
- Fix bug which failed to match Rome Model when using subsets
of system resources (i.e. ROCR_VISIBLE_DEVICES is set)
- Fixes for topo_expl
* Fix bug w/ 1H16P
2024-12-03 13:12:03 -08:00
84081064a0
Merge remote-tracking branch 'nccl/master' into develop
2024-10-02 09:31:25 -05:00
e453f1ced9
Add another Rome model ( #1354 )
2024-10-01 17:41:27 -05:00
1a48e19b18
topo_expl: update sm fields in topo xml files ( #1310 )
2024-08-29 12:03:51 -07:00
532b70afb6
Add new Rome model ( #1304 )
...
* Add another rome model and override
* Fix bug
* Fix typo
* Add ring
* Update ring
* Fix model matching
* Clean up
* Clean up
* Reverse rings for NCCL_RINGS input
* Only reverse NCCL_RINGS for ring graph
* Fix mapping issue when using NCCL_RINGS
* Add NCCL_RINGS_REMAP to handle inconsistant net names
2024-08-23 08:45:43 +08:00
4bc118336a
topo_expl: Update channel masks for >64 channels ( #1279 )
2024-07-25 17:27:34 -07:00
0634c5c8e1
doubling debug buffer size with increased channels
2024-06-03 13:05:05 -05:00
4cb62f999a
Rail optimization for rings ( #1140 )
...
- Modifies the ring creation algorithm to be friendlier to rail-optimized topologies (should not affect classic fabric topologies)
2024-04-15 12:03:57 -06:00
93982533d7
[topo_expl] Adding -n option to override number of nodes ( #1134 )
2024-04-04 15:11:47 -06:00
9eefc68cb5
Fixes the copyright comment block on each of topo_expl/models/*.xml. The format was not valid XML. ( #1124 )
2024-03-25 16:21:17 -06:00
6777e65c1d
Enable fp8 support ( #1101 )
...
* initial checkin
* resolve cr comments
* resolve the build issue
* fix the data correctless issue
* update fp8 header file and update the unit test for fp8 support
* remove fp16 from fp8 headers
* fix ut issue and catch up the latest code from develop
* udate according to cr comments
* update ut according to cr comments
* update num floats for each SumPostDiv from 4 to 6
* update fp8 header file name
* fix the typo
2024-03-08 15:17:53 -08:00
d2224fd3e1
topo_expl: 2.19.4 update and fix build error ( #1098 )
2024-03-07 08:52:50 -08:00
df98a6957d
Add another Rome model ( #1095 )
2024-02-28 10:46:05 -08:00
74f9e5db64
Add new GPU model ( #1080 )
2024-02-23 12:19:42 -08:00
d1575a1622
topo_expl: 2.19 update
2024-01-31 16:11:14 -06:00
600b44fee5
topo-expl: fix broken build ( #1048 )
2024-01-17 08:59:03 -08:00
f7e39fced2
Doubling buffer size to fix NCCL INFO corruption with increased channels ( #1035 )
2024-01-08 08:14:33 -08:00
50b2dd9fd7
Add special handling of gfx940 ( #976 )
...
* Add special handling of gfx940
* Update ring base
2023-11-22 15:07:36 -08:00
762a42859e
Fixed topo_expl ( #891 )
2023-09-13 12:05:35 -06:00
e58ec78d35
Gcn arch name ( #886 )
...
We use CMake to determine if we're compiling against a version of ROCm that supports gcnArchName and handles architecture checking appropriately. It includes a few helper functions as drop ins for the functionality we used gcnArch for before; sometimes to enable flags, and sometimes to set frequencies.
2023-09-12 15:34:40 -04:00
d33cd5a233
NCCL_TREES variable and rome model fixes ( #856 )
2023-08-21 10:35:37 -06:00
7044599575
Add new model support ( #847 )
...
* Add new model support
* Update new rings
2023-08-10 17:14:51 -07:00
a7fcd58a97
Enable gfx94x ( #808 ) ( #816 )
...
(cherry picked from commit 94da229a7788d74685d1591a4e75a8341de64f41)
2023-07-21 07:31:27 -07:00
abd0615351
Merge remote-tracking branch 'nccl/master' into develop
2023-06-26 22:51:56 +00:00
3af90902c8
Add NCCL_NCHANNELS_PER_PEER override ( #767 )
...
Also fix topol_expl build issue
2023-06-06 08:41:38 -07:00
e3b2342f39
MSCCL: Improve executor and integrate scheduler ( #694 )
...
* MSCCL: improve executor and add scheduler for testing
* Use external scheduler
* Fix cmake error
* Address comments
* Fix thread safe issue
* Make MSCCL lifecycle APIs thread safe
* Make MSCCL internal scheduler aware of topology hint
* Revise error message
2023-03-14 14:34:25 -07:00
e1cb45ff22
Merge remote-tracking branch 'nccl/master' into HEAD
2023-02-04 01:44:43 +00:00
a0dd8e0b84
topo_expl: fix broken build by adding hipify steps ( #670 )
2023-01-06 07:29:40 -08:00
94ad7f6f51
Update tuning table and fix topo_expl
2022-11-07 18:24:24 +00:00
4f0e223db4
Merge remote-tracking branch 'nccl/master' into develop
2022-10-20 15:41:29 +00:00
fc554a2428
topo_expl: fix compilation error ( #639 )
2022-10-19 14:19:50 -07:00
ebb8b5bf63
Updating files for missing licenses ( #637 )
2022-10-14 13:49:16 -06:00
a79d9e3586
Merge remote-tracking branch 'nccl/master' into develop
2022-09-09 16:05:38 +00:00
2cb2f9493a
HIP Path default updated to ROCM_PATH (reorg path) ( #592 )
...
Updated default path for hip to ROCM_PATH (/opt/rocm instead of /opt/rocm/hip) as per new/current structure.
2022-08-04 13:38:41 -04:00
0336ffdf70
Introduce multi-rank support per device.
...
This is a single commit of the source code changes required to
introduce support for multiple ranks per device.
A new interface (ncclCommRankInitMulti) has to be used to make use of
this new feature.
2022-06-10 14:23:12 +00:00
ef499c4810
Add another Rome model ( #553 )
...
* Add another Rome model
* Add option to force enable intranet on single node
* Limit p2p channels to number of ranks
* Refine p2p channels handling
2022-05-31 11:31:30 -07:00
c5b77121f0
Update Rome model ( #552 )
2022-05-26 09:59:23 -07:00
98f0809a39
Added creation of new tree and added switch for using treesplit for specific cases ( #551 )
2022-05-25 18:55:14 -04:00
283dc86a73
Refine and add new Rome models ( #548 )
2022-05-17 08:23:59 -07:00
063da25563
topo_expl: fix build and add tuning support ( #539 )
2022-04-26 15:40:07 -07:00
d28e1cb44f
Merge remote-tracking branch 'nccl/master' into develop
2022-04-18 11:15:25 -07:00
2151c79d14
Add new Rome model ( #536 )
2022-04-13 11:45:40 -07:00
ba4c165bf3
Add new Rome model ( #535 )
2022-04-12 13:27:32 -07:00
bbe780ca6c
Support multiple tuning tables ( #522 )
...
* Support multiple tuning tables
* [UnitTests] Skip managed memory testing
2022-03-31 17:09:21 -07:00
cd17cf6dce
Update Rome model matching and add new models ( #516 )
...
* Update Rome model matching and add new models
* Add missing file
* Models update
2022-03-21 10:54:40 -07:00
b569c0a1db
Add Pivot AllToAll algorithm for Rome model ( #503 )
...
* add a2a pivot interface
* remove debug info
* address comments
* fix bug
* remove custom script
* address comments
* fix bug
2022-02-20 21:09:47 -08:00
598c6fdded
Update Rome models ( #491 )
2022-01-14 10:03:30 -08:00
369c021992
topo_expl: update for 2.11.4 ( #490 )
...
* topo_expl: update for 2.11.4
* topo_expl: revert a few logging changes
2022-01-13 13:33:07 -08:00
gilbertlee-amd
Benjamin Kitor
BertanDogancay
Wenkai Du
Benjamin Kitor
Nusrat Islam
corey-derochie-amd
Andy li
Wenkai Du
akolliasAMD
Audrey MP
Ziyue Yang
arvindcheru
Edgar
Ziyue Yang