AI/rocm-systems
2
0
Форкнуть 0
Код Задачи Запросы на слияние Действия Пакеты Проекты Релизы Вики Активность
708 коммитов 3 Ветки 0 Теги
0331e39f8178d79f4df22fb9f667d1688cf1d861
Граф коммитов

106 Коммитов

Автор SHA1 Сообщение Дата
Wenkai Du 0331e39f81 Update Rome model matching (#461) 
* Update Rome model matching

* Add another Rome model

* Automatically setup NET GDR level from model
 2021-11-05 08:53:47 -07:00
Wenkai Du 14a184eb67 Query XGMI link count through rocm_smi_lib API (#442) 2021-10-26 10:30:20 -07:00
Wenkai Du b4cefc05ed Fix collnet tuning parameters (#441) 2021-10-20 20:45:36 -07:00
Wenkai Du 2508507d0a Fix PCIe gen detection (#437) 
* Fix PCIe gen detection

* Update profiling support
 2021-10-15 08:23:50 -07:00
Wenkai Du 2249a1d9d3 Add more Rome models (#434) 
* Add more Rome models

* Update models and tuning

* Update tuning
 2021-10-12 08:23:20 -07:00
Wenkai Du e0053311c0 Add another Rome model (#431) 2021-10-06 08:17:12 -07:00
Wenkai Du 29c729d8b6 Trim NICs when all GPUs are connected by XGMI (#430) 
* Trim NICs when all GPUs are connected by XGMI

* Only enable clique with maximum of 2 hops
 2021-10-05 18:27:43 -07:00
Wenkai Du 8ee2b7932a Merge remote-tracking branch 'origin/develop' into 2.10.3 2021-09-13 15:51:53 -07:00
Wenkai Du ef432e48e1 Update tuning table (#424) 2021-09-13 08:39:01 -07:00
Wenkai Du 5c8380ff5b Implement NIC identification and remapping (#420) 
* Add 1H16P GPU model

* Implement NIC identification and remapping

* Revert "Sort IB devices based on device name (#413)"

This reverts commit 2d0ed8dff6.

* Fix permute and check order

* Correction on IB speed reporting

* Revert "Allow user to link layer with RCCL_IB_HCA_SKIP_LINK_LAYER (#361)"

This reverts commit caf5c9992a.
 2021-08-24 09:42:04 -07:00
Wenkai Du d5f93649ff Merge remote-tracking branch 'origin/develop' into 2.10.3 2021-08-24 09:49:47 -07:00
Wenkai Du 5f15ed6e3e Add gfx908 VM model (#418) 2021-08-10 08:55:11 -07:00
Wenkai Du bf2339f93e Merge remote-tracking branch 'nccl/master' into 2.10.3 2021-07-30 16:23:14 -07:00
Wenkai Du 818cdb16a8 Query XGMI links from xml and adjust gfx906 channel usage (#410) 2021-07-27 17:32:41 -07:00
Ke Wen 7e51592129 2.10.3-1 
Add support for bfloat16.
Add ncclAvg reduction operation.
Improve performance for aggregated operations.
Improve performance for tree.
Improve network error reporting.
Add NCCL_NET parameter to force a specific network.
Add NCCL_IB_QPS_PER_CONNECTION parameter to split IB traffic onto multiple queue pairs.
Fix topology detection error in WSL2.
Fix proxy memory elements affinity (improve alltoall performance).
Fix graph search on cubemesh topologies.
Fix hang in cubemesh during NVB connections.
 2021-07-08 14:30:14 -07:00
Wenkai Du 56155ff5b6 Use rocm_smi_lib for getting topology information (#402) 
* Use rocm_smi_lib for getting topology information

* Add rocm-smi-lib dependency to RCCL package
 2021-07-08 13:23:11 -07:00
Wenkai Du fa6d7e9a63 Fixes for NCCL_MAX_NCHANNELS and topo_expl (#398) 2021-06-22 08:41:49 -07:00
Wenkai Du b815a2800f Setup collectives threshold for enabling intranet (#387) 
* Setup collectives threshold for enabling intranet

* Use separate operation counters for coll and p2p
 2021-06-09 13:24:26 -07:00
Wenkai Du c2064adcc7 Add support for another Rome model (#385) 2021-06-08 13:58:20 -07:00
Wenkai Du a3a8c2d56b Allow intranode use of network connection (#383) 
* Allow intranode use of network connection

* Checking for graph for null pointer
 2021-06-08 07:37:59 -07:00
Wenkai Du 961922ea02 Add option to enable multiple SAT in SHARP (#380) 
* Add option to enable multiple SAT in SHARP

* Extend number of NICs to 16
 2021-06-03 19:45:18 -07:00
Wenkai Du e3abf1c2ec Merge remote-tracking branch 'nccl/master' into develop 2021-05-25 20:52:15 -07:00
Wenkai Du 4c83adb75c Update Rome models matching (#376) 2021-05-25 10:12:40 -07:00
Wenkai Du 87727383fe Merge remote-tracking branch 'nccl/master' into 2.9.8 2021-05-17 10:15:16 -07:00
Sylvain Jeaugey 3fec2fa5ee 2.9.9-1 
Fix crash when setting NCCL_MAX_P2P_NCHANNELS below nchannels.
Fix hang during sendrecv dynamic NVB connection establishment on
cubemesh topologies.
Add environment variable to only use SHARP on communicators beyond
a given number of ranks.
Add debug subsystem to trace memory allocations.
Fix compilation with TRACE=1. (Issue #505)
 2021-05-12 11:09:31 -07:00
Sylvain Jeaugey ca8485b0d0 2.9.8-1 
Fix memory leaks.
Fix crash in bootstrap error case.
Fix Collnet clean-up issue.
Make PCI switch vendor/device optional for XML injection.
Add support for nvidia-peermem module.
 2021-05-10 14:00:03 -07:00
Wenkai Du a4ea1fed5b Merge remote-tracking branch 'nccl/master' into develop 2021-05-05 16:01:01 -07:00
Wenkai Du 415c7cd3d1 Tune number of channels for gfx90a (#349) 2021-04-19 15:27:01 -07:00
Wenkai Du a79f74082e Limit max channels for ring graph on single node Rome (#347) 
* Limit max channels for ring graph on single node Rome
* Partially revert "Use non-temporal access for streaming data (#341)"
 2021-04-14 10:14:54 -07:00
Wenkai Du 1fe031402a Add gfx90a target (#344) 
* Add gfx90a target

* Support gfx90a topology

Co-authored-by: Eiden Yoshida <eiden.yoshida@amd.com>
 2021-04-14 09:29:00 -06:00
Sylvain Jeaugey a46ea10583 2.9.6-1 
Add support for CUDA graphs.
Fuse BCM Gen4 switches to avoid suboptimal performance on some platforms. Issue #439.
Fix bootstrap issue caused by connection reordering.
Fix CPU locking block.
Improve CollNet algorithm.
Improve performance on DGX A100 for communicators with only one GPU per node.
 2021-04-12 16:00:46 -07:00
Wenkai Du e26ad2995e Cleanup number of channels calculation (#340) 2021-04-05 17:51:56 -07:00
Wenkai Du 1d2946ee4b Rework network port trimming code (#338) 
* Rework network port trimming code

* Move Rome related changes to separate source files
 2021-03-31 10:25:59 -07:00
Wenkai Du d87dc7c2e8 collnet: support multiple NICs (#335) 2021-03-25 20:59:32 -07:00
Wenkai Du 1d6244b18d Enable collnet in RCCL (#333) 
* Enable CollNet and use different number of channels

* topo_expl: enable collnet
 2021-03-19 12:58:13 -07:00
Wenkai Du b46260260a Sort GPUs by HIP device ID (#329) 
* Sort GPUs by HIP device ID

* Remove extra space
 2021-03-16 16:51:32 -07:00
Wenkai Du c018edf0f2 Enable local sendrecv over network if GDR is available on all GPUs (#324) 2021-03-05 19:59:41 -08:00
Wenkai Du e820a943e9 Update tuning parameters for XGMI and NET 2021-02-23 21:41:26 +00:00
Wenkai Du ec8d89b1dd Match NBIO only when GPUs and NICs are directly connected to CPU 2021-02-22 18:52:29 -05:00
Wenkai Du 95f178324c Add support to another Rome model 2021-02-18 02:00:31 +00:00
Wenkai Du c985358e11 Merge remote-tracking branch 'nccl/master' into 2.8.3 2021-02-15 18:44:47 -05:00
Sylvain Jeaugey 911d61f214 2.8.4-1 
Fix hang in corner cases of alltoallv using point to point send/recv.
Harmonize error messages.
Fix missing NVTX section in the license.
Update README.
 2021-02-09 15:36:48 -08:00
Wenkai Du ab1e7a0318 Merge remote-tracking branch 'origin/develop' into 2.8.3 2021-02-04 20:02:34 -05:00
Wenkai Du 5f97122442 Enable GPU direct RDMA read from GPU 2021-02-03 02:48:30 +00:00
Wenkai Du f4d5d3d620 Port alltoall[v] 2021-01-14 19:28:01 -05:00
Wenkai Du 105db19a11 Do not allow GPU as intermediate 2021-01-14 19:28:01 -05:00
Wenkai Du e055229e56 Revert "Changes to topology based on XGMI (#272)" 
This reverts commit 01bd2573db.
 2021-01-14 19:28:01 -05:00
Wenkai Du d469947641 Merge remote-tracking branch 'nccl/master' into no-target-id 2021-01-14 19:27:53 -05:00
Jonas Zhou 3996562690 x86: Add CPU detection for Zhaoxin processors 
Signed-off-by: Jonas Zhou <JonasZhou@zhaoxin.com>
 2020-12-17 11:15:18 -08:00
Wenkai Du 373a108516 Fix Rome PCIe 2 node topology generation (#310) 2020-12-15 17:16:17 -08:00