Commit Graph

52 Commits

Author SHA1 Message Date
Wenkai Du df98a6957d Add another Rome model (#1095) 2024-02-28 10:46:05 -08:00
Wenkai Du 74f9e5db64 Add new GPU model (#1080) 2024-02-23 12:19:42 -08:00
Pedram Alizadeh b08124c85d adding rccl tuning parameters for MI300X gfx942 with 8 GPUs single and multi-node (#1047) 2024-01-16 13:44:32 -05:00
Wenkai Du 53d807a5b9 msccl: disable on multi-node (#1018) 2023-12-13 07:41:40 -08:00
Wenkai Du 4ba65d1d6a Increase max channles to 64 (#993) 2023-12-01 16:01:11 -08:00
Wenkai Du 50b2dd9fd7 Add special handling of gfx940 (#976)
* Add special handling of gfx940

* Update ring base
2023-11-22 15:07:36 -08:00
Wenkai Du 4278a9918b Update rome models (#922) 2023-10-18 17:28:01 -07:00
akolliasAMD b85d73c02e changed the form that RCCL_TREE uses (#888)
* changed the form that RCCL_TREE uses
2023-09-15 15:01:33 -06:00
Ziyue Yang c1bfd5f0d8 Add single-node MI300X topology (#889) 2023-09-13 11:07:17 -07:00
Wenkai Du aeca1af374 Add MSCCL xml files (#861) 2023-08-23 14:12:34 -07:00
akolliasAMD d33cd5a233 NCCL_TREES variable and rome model fixes (#856) 2023-08-21 10:35:37 -06:00
Wenkai Du 7044599575 Add new model support (#847)
* Add new model support

* Update new rings
2023-08-10 17:14:51 -07:00
Wenkai Du a7fcd58a97 Enable gfx94x (#808) (#816)
(cherry picked from commit 94da229a7788d74685d1591a4e75a8341de64f41)
2023-07-21 07:31:27 -07:00
Wenkai Du abd0615351 Merge remote-tracking branch 'nccl/master' into develop 2023-06-26 22:51:56 +00:00
Ziyue Yang e3b2342f39 MSCCL: Improve executor and integrate scheduler (#694)
* MSCCL: improve executor and add scheduler for testing

* Use external scheduler

* Fix cmake error

* Address comments

* Fix thread safe issue

* Make MSCCL lifecycle APIs thread safe

* Make MSCCL internal scheduler aware of topology hint

* Revise error message
2023-03-14 14:34:25 -07:00
Wenkai Du 94ad7f6f51 Update tuning table and fix topo_expl 2022-11-07 18:24:24 +00:00
Wenkai Du 4f0e223db4 Merge remote-tracking branch 'nccl/master' into develop 2022-10-20 15:41:29 +00:00
Wenkai Du 021932b3c8 Add LL128 tuning (#630) 2022-09-27 09:39:09 -07:00
Wenkai Du 7bbce085cc Enable LL128 protocol support (#605)
* Enable LL128 protocol support

* Use shared memory object directly when possible
2022-09-08 14:45:27 -07:00
Wenkai Du 00af1f64e9 Fix GPU to NIC mapping in tree (#573)
* Fix GPU to NIC mapping in tree

* Update tuning table
2022-07-03 20:52:52 -07:00
Wenkai Du 9a285b5e1d Do not set NET GDR level automatically (#571) 2022-06-23 16:28:28 -07:00
Edgar 0336ffdf70 Introduce multi-rank support per device.
This is a single commit of the source code changes required to
introduce support for multiple ranks per device.
A new interface (ncclCommRankInitMulti) has to be used to make use of
this new feature.
2022-06-10 14:23:12 +00:00
Wenkai Du ef499c4810 Add another Rome model (#553)
* Add another Rome model

* Add option to force enable intranet on single node

* Limit p2p channels to number of ranks

* Refine p2p channels handling
2022-05-31 11:31:30 -07:00
akolliasAMD a0a686e74c code cleanup (#554) 2022-05-31 09:59:36 -04:00
Wenkai Du c5b77121f0 Update Rome model (#552) 2022-05-26 09:59:23 -07:00
akolliasAMD 98f0809a39 Added creation of new tree and added switch for using treesplit for specific cases (#551) 2022-05-25 18:55:14 -04:00
Wenkai Du 283dc86a73 Refine and add new Rome models (#548) 2022-05-17 08:23:59 -07:00
Wenkai Du 83fd4f70e7 Update tuning parameters 2022-04-18 16:04:04 -07:00
Wenkai Du fd2f1b3b88 Fix random segfault (#537) 2022-04-15 14:32:11 -07:00
Wenkai Du 2151c79d14 Add new Rome model (#536) 2022-04-13 11:45:40 -07:00
Wenkai Du ba4c165bf3 Add new Rome model (#535) 2022-04-12 13:27:32 -07:00
Wenkai Du 5cc0a405c0 Add tuning model (#523) 2022-04-04 10:19:57 -07:00
Wenkai Du bbe780ca6c Support multiple tuning tables (#522)
* Support multiple tuning tables

* [UnitTests] Skip managed memory testing
2022-03-31 17:09:21 -07:00
Wenkai Du cd17cf6dce Update Rome model matching and add new models (#516)
* Update Rome model matching and add new models

* Add missing file

* Models update
2022-03-21 10:54:40 -07:00
Ziyue Yang b569c0a1db Add Pivot AllToAll algorithm for Rome model (#503)
* add a2a pivot interface

* remove debug info

* address comments

* fix bug

* remove custom script

* address comments

* fix bug
2022-02-20 21:09:47 -08:00
Wenkai Du 94e0dc8bfd Allow additional options to be passed in through model's definition (#501) 2022-02-17 08:28:58 -08:00
Wenkai Du 02096c9936 Add another Rome model (#497) 2022-02-12 10:30:16 -08:00
Wenkai Du 598c6fdded Update Rome models (#491) 2022-01-14 10:03:30 -08:00
Wenkai Du f8d0775a6f Add another Rome model (#483) 2022-01-05 09:26:31 -08:00
Wenkai Du a94b953bcc Update Rome model (#479) 2021-12-03 08:24:51 -08:00
Wenkai Du 0331e39f81 Update Rome model matching (#461)
* Update Rome model matching

* Add another Rome model

* Automatically setup NET GDR level from model
2021-11-05 08:53:47 -07:00
Wenkai Du 2249a1d9d3 Add more Rome models (#434)
* Add more Rome models

* Update models and tuning

* Update tuning
2021-10-12 08:23:20 -07:00
Wenkai Du e0053311c0 Add another Rome model (#431) 2021-10-06 08:17:12 -07:00
Wenkai Du 5c8380ff5b Implement NIC identification and remapping (#420)
* Add 1H16P GPU model

* Implement NIC identification and remapping

* Revert "Sort IB devices based on device name (#413)"

This reverts commit 2d0ed8dff6.

* Fix permute and check order

* Correction on IB speed reporting

* Revert "Allow user to link layer with RCCL_IB_HCA_SKIP_LINK_LAYER (#361)"

This reverts commit caf5c9992a.
2021-08-24 09:42:04 -07:00
Wenkai Du 5f15ed6e3e Add gfx908 VM model (#418) 2021-08-10 08:55:11 -07:00
Wenkai Du 56155ff5b6 Use rocm_smi_lib for getting topology information (#402)
* Use rocm_smi_lib for getting topology information

* Add rocm-smi-lib dependency to RCCL package
2021-07-08 13:23:11 -07:00
Wenkai Du c2064adcc7 Add support for another Rome model (#385) 2021-06-08 13:58:20 -07:00
Wenkai Du a3a8c2d56b Allow intranode use of network connection (#383)
* Allow intranode use of network connection

* Checking for graph for null pointer
2021-06-08 07:37:59 -07:00
Wenkai Du 961922ea02 Add option to enable multiple SAT in SHARP (#380)
* Add option to enable multiple SAT in SHARP

* Extend number of NICs to 16
2021-06-03 19:45:18 -07:00
Wenkai Du 4c83adb75c Update Rome models matching (#376) 2021-05-25 10:12:40 -07:00