AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
722 Commits 3 Branches 0 Tags
8d6cfcbf58fd3a52d66093bd00478efd8b0b7927
Commit Graph

108 Commits

Author SHA1 Message Date
Wenkai Du 7406aed9c9 Update Rome model (#479) 
[ROCm/rccl commit: a94b953bcc]
 2021-12-03 08:24:51 -08:00
Wenkai Du dd4a53db37 Update tuning parameters (#478) 
[ROCm/rccl commit: 8a08a2f579]
 2021-11-30 08:51:11 -08:00
Wenkai Du fd98ee84b4 Update Rome model matching (#461) 
* Update Rome model matching

* Add another Rome model

* Automatically setup NET GDR level from model

[ROCm/rccl commit: 0331e39f81]
 2021-11-05 08:53:47 -07:00
Wenkai Du 15143b1cfb Query XGMI link count through rocm_smi_lib API (#442) 
[ROCm/rccl commit: 14a184eb67]
 2021-10-26 10:30:20 -07:00
Wenkai Du aac9d57633 Fix collnet tuning parameters (#441) 
[ROCm/rccl commit: b4cefc05ed]
 2021-10-20 20:45:36 -07:00
Wenkai Du 4d979ce13d Fix PCIe gen detection (#437) 
* Fix PCIe gen detection

* Update profiling support

[ROCm/rccl commit: 2508507d0a]
 2021-10-15 08:23:50 -07:00
Wenkai Du b587b55c2e Add more Rome models (#434) 
* Add more Rome models

* Update models and tuning

* Update tuning

[ROCm/rccl commit: 2249a1d9d3]
 2021-10-12 08:23:20 -07:00
Wenkai Du d377c4dcc6 Add another Rome model (#431) 
[ROCm/rccl commit: e0053311c0]
 2021-10-06 08:17:12 -07:00
Wenkai Du 91eca0d7d2 Trim NICs when all GPUs are connected by XGMI (#430) 
* Trim NICs when all GPUs are connected by XGMI

* Only enable clique with maximum of 2 hops

[ROCm/rccl commit: 29c729d8b6]
 2021-10-05 18:27:43 -07:00
Wenkai Du 8acdb77cc0 Merge remote-tracking branch 'origin/develop' into 2.10.3 
[ROCm/rccl commit: 8ee2b7932a]
 2021-09-13 15:51:53 -07:00
Wenkai Du 9ffeb41fe1 Update tuning table (#424) 
[ROCm/rccl commit: ef432e48e1]
 2021-09-13 08:39:01 -07:00
Wenkai Du b9508a6aba Implement NIC identification and remapping (#420) 
* Add 1H16P GPU model

* Implement NIC identification and remapping

* Revert "Sort IB devices based on device name (#413)"

This reverts commit de0c586bad.

* Fix permute and check order

* Correction on IB speed reporting

* Revert "Allow user to link layer with RCCL_IB_HCA_SKIP_LINK_LAYER (#361)"

This reverts commit fa690c47a0.

[ROCm/rccl commit: 5c8380ff5b]
 2021-08-24 09:42:04 -07:00
Wenkai Du 4fd7a14087 Merge remote-tracking branch 'origin/develop' into 2.10.3 
[ROCm/rccl commit: d5f93649ff]
 2021-08-24 09:49:47 -07:00
Wenkai Du 57518da006 Add gfx908 VM model (#418) 
[ROCm/rccl commit: 5f15ed6e3e]
 2021-08-10 08:55:11 -07:00
Wenkai Du 4b89e98675 Merge remote-tracking branch 'nccl/master' into 2.10.3 
[ROCm/rccl commit: bf2339f93e]
 2021-07-30 16:23:14 -07:00
Wenkai Du faea6ead5c Query XGMI links from xml and adjust gfx906 channel usage (#410) 
[ROCm/rccl commit: 818cdb16a8]
 2021-07-27 17:32:41 -07:00
Ke Wen 58a9f19dc3 2.10.3-1 
Add support for bfloat16.
Add ncclAvg reduction operation.
Improve performance for aggregated operations.
Improve performance for tree.
Improve network error reporting.
Add NCCL_NET parameter to force a specific network.
Add NCCL_IB_QPS_PER_CONNECTION parameter to split IB traffic onto multiple queue pairs.
Fix topology detection error in WSL2.
Fix proxy memory elements affinity (improve alltoall performance).
Fix graph search on cubemesh topologies.
Fix hang in cubemesh during NVB connections.


[ROCm/rccl commit: 7e51592129]
 2021-07-08 14:30:14 -07:00
Wenkai Du 71dfc3978e Use rocm_smi_lib for getting topology information (#402) 
* Use rocm_smi_lib for getting topology information

* Add rocm-smi-lib dependency to RCCL package

[ROCm/rccl commit: 56155ff5b6]
 2021-07-08 13:23:11 -07:00
Wenkai Du 90ae176437 Fixes for NCCL_MAX_NCHANNELS and topo_expl (#398) 
[ROCm/rccl commit: fa6d7e9a63]
 2021-06-22 08:41:49 -07:00
Wenkai Du 5bebcb0015 Setup collectives threshold for enabling intranet (#387) 
* Setup collectives threshold for enabling intranet

* Use separate operation counters for coll and p2p

[ROCm/rccl commit: b815a2800f]
 2021-06-09 13:24:26 -07:00
Wenkai Du ab9b9151d2 Add support for another Rome model (#385) 
[ROCm/rccl commit: c2064adcc7]
 2021-06-08 13:58:20 -07:00
Wenkai Du c8a432dc25 Allow intranode use of network connection (#383) 
* Allow intranode use of network connection

* Checking for graph for null pointer

[ROCm/rccl commit: a3a8c2d56b]
 2021-06-08 07:37:59 -07:00
Wenkai Du 4b31e521e9 Add option to enable multiple SAT in SHARP (#380) 
* Add option to enable multiple SAT in SHARP

* Extend number of NICs to 16

[ROCm/rccl commit: 961922ea02]
 2021-06-03 19:45:18 -07:00
Wenkai Du b154b532a2 Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: e3abf1c2ec]
 2021-05-25 20:52:15 -07:00
Wenkai Du aa95cc6102 Update Rome models matching (#376) 
[ROCm/rccl commit: 4c83adb75c]
 2021-05-25 10:12:40 -07:00
Wenkai Du e0edd3d5e4 Merge remote-tracking branch 'nccl/master' into 2.9.8 
[ROCm/rccl commit: 87727383fe]
 2021-05-17 10:15:16 -07:00
Sylvain Jeaugey 3dd3a1ca66 2.9.9-1 
Fix crash when setting NCCL_MAX_P2P_NCHANNELS below nchannels.
Fix hang during sendrecv dynamic NVB connection establishment on
cubemesh topologies.
Add environment variable to only use SHARP on communicators beyond
a given number of ranks.
Add debug subsystem to trace memory allocations.
Fix compilation with TRACE=1. (Issue #505)


[ROCm/rccl commit: 3fec2fa5ee]
 2021-05-12 11:09:31 -07:00
Sylvain Jeaugey 780273774a 2.9.8-1 
Fix memory leaks.
Fix crash in bootstrap error case.
Fix Collnet clean-up issue.
Make PCI switch vendor/device optional for XML injection.
Add support for nvidia-peermem module.


[ROCm/rccl commit: ca8485b0d0]
 2021-05-10 14:00:03 -07:00
Wenkai Du a76bebf8b6 Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: a4ea1fed5b]
 2021-05-05 16:01:01 -07:00
Wenkai Du e28bac31aa Tune number of channels for gfx90a (#349) 
[ROCm/rccl commit: 415c7cd3d1]
 2021-04-19 15:27:01 -07:00
Wenkai Du 661b1351a3 Limit max channels for ring graph on single node Rome (#347) 
* Limit max channels for ring graph on single node Rome
* Partially revert "Use non-temporal access for streaming data (#341)"

[ROCm/rccl commit: a79f74082e]
 2021-04-14 10:14:54 -07:00
Wenkai Du 0f4d497edc Add gfx90a target (#344) 
* Add gfx90a target

* Support gfx90a topology

Co-authored-by: Eiden Yoshida <eiden.yoshida@amd.com>

[ROCm/rccl commit: 1fe031402a]
 2021-04-14 09:29:00 -06:00
Sylvain Jeaugey 20da390b96 2.9.6-1 
Add support for CUDA graphs.
Fuse BCM Gen4 switches to avoid suboptimal performance on some platforms. Issue #439.
Fix bootstrap issue caused by connection reordering.
Fix CPU locking block.
Improve CollNet algorithm.
Improve performance on DGX A100 for communicators with only one GPU per node.


[ROCm/rccl commit: a46ea10583]
 2021-04-12 16:00:46 -07:00
Wenkai Du b4a7fa7011 Cleanup number of channels calculation (#340) 
[ROCm/rccl commit: e26ad2995e]
 2021-04-05 17:51:56 -07:00
Wenkai Du 018a31877c Rework network port trimming code (#338) 
* Rework network port trimming code

* Move Rome related changes to separate source files

[ROCm/rccl commit: 1d2946ee4b]
 2021-03-31 10:25:59 -07:00
Wenkai Du 065bde98d8 collnet: support multiple NICs (#335) 
[ROCm/rccl commit: d87dc7c2e8]
 2021-03-25 20:59:32 -07:00
Wenkai Du 287ed0f18a Enable collnet in RCCL (#333) 
* Enable CollNet and use different number of channels

* topo_expl: enable collnet

[ROCm/rccl commit: 1d6244b18d]
 2021-03-19 12:58:13 -07:00
Wenkai Du 052119c20a Sort GPUs by HIP device ID (#329) 
* Sort GPUs by HIP device ID

* Remove extra space

[ROCm/rccl commit: b46260260a]
 2021-03-16 16:51:32 -07:00
Wenkai Du bcf4ecb0e3 Enable local sendrecv over network if GDR is available on all GPUs (#324) 
[ROCm/rccl commit: c018edf0f2]
 2021-03-05 19:59:41 -08:00
Wenkai Du 020ccb6730 Update tuning parameters for XGMI and NET 
[ROCm/rccl commit: e820a943e9]
 2021-02-23 21:41:26 +00:00
Wenkai Du d24df90882 Match NBIO only when GPUs and NICs are directly connected to CPU 
[ROCm/rccl commit: ec8d89b1dd]
 2021-02-22 18:52:29 -05:00
Wenkai Du 6c3ccc2192 Add support to another Rome model 
[ROCm/rccl commit: 95f178324c]
 2021-02-18 02:00:31 +00:00
Wenkai Du ab71643c99 Merge remote-tracking branch 'nccl/master' into 2.8.3 
[ROCm/rccl commit: c985358e11]
 2021-02-15 18:44:47 -05:00
Sylvain Jeaugey fc7bdb38a5 2.8.4-1 
Fix hang in corner cases of alltoallv using point to point send/recv.
Harmonize error messages.
Fix missing NVTX section in the license.
Update README.


[ROCm/rccl commit: 911d61f214]
 2021-02-09 15:36:48 -08:00
Wenkai Du ae5779702a Merge remote-tracking branch 'origin/develop' into 2.8.3 
[ROCm/rccl commit: ab1e7a0318]
 2021-02-04 20:02:34 -05:00
Wenkai Du 57abf599b2 Enable GPU direct RDMA read from GPU 
[ROCm/rccl commit: 5f97122442]
 2021-02-03 02:48:30 +00:00
Wenkai Du 2c49121171 Port alltoall[v] 
[ROCm/rccl commit: f4d5d3d620]
 2021-01-14 19:28:01 -05:00
Wenkai Du 41bead5a4e Do not allow GPU as intermediate 
[ROCm/rccl commit: 105db19a11]
 2021-01-14 19:28:01 -05:00
Wenkai Du 34c6013299 Revert "Changes to topology based on XGMI (#272)" 
This reverts commit 0a9adc16f4.


[ROCm/rccl commit: e055229e56]
 2021-01-14 19:28:01 -05:00
Wenkai Du adff98765c Merge remote-tracking branch 'nccl/master' into no-target-id 
[ROCm/rccl commit: d469947641]
 2021-01-14 19:27:53 -05:00