AI/rocm-systems
2
0 Форкнуть 0
Код Задачи Запросы на слияние Действия Пакеты Проекты Релизы Вики Активность
431 коммит 3 Ветки 0 Теги
b8a7fc694729cdb23c4b695a93b0f8f1fb807415
Граф коммитов
164 Коммитов
Автор SHA1 Сообщение Дата
Wenkai Du c7805f224e Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: da3b197d6c]
 2020-07-01 16:51:25 -07:00
Wenkai Du 77d94493c3 Merge sendrecv kernel from NCCL 2.7.3 
This commit was cherry-picked and modified from
https://github.com/NVIDIA/nccl/commit/89072c82e551d7c8bf7bd5ca77780ef540b7611e


[ROCm/rccl commit: 964c4c2061]
 2020-06-29 08:47:46 -07:00
Wenkai Du 3c674c9c1f Use separate threads for send and receive 
[ROCm/rccl commit: b90735c935]
 2020-06-29 08:47:15 -07:00
Sylvain Jeaugey 5432eb8e9d 2.7.6-1 
Fix crash when NVswitch is not visible inside a VM.


[ROCm/rccl commit: 1952325569]
 2020-06-26 16:35:54 -07:00
Sylvain Jeaugey 1f6b75ca8b 2.7.5-1 
Minor fixes for A100 platforms.
Add a WARN for invalid GroupEnd call.


[ROCm/rccl commit: 01afd20a77]
 2020-06-26 14:39:49 -07:00
Wenkai Du 117f0f6e6e Revert use posix_memalign for network buffer allocation on host memory (#222) 
[ROCm/rccl commit: 84f8ba3bb0]
 2020-06-24 11:25:55 -07:00
Wenkai Du e7d664200f Use posix_memalign for network buffer allocation on host memory (#221) 
* Use posix_memalign for network buffer allocation on host memory

* ib-test: add ability to specify run iterations

* ib-test: define iterations as multiple of default cycles

* Add checking to posix_memalign return value

[ROCm/rccl commit: 0eb19a563a]
 2020-06-22 13:06:25 -07:00
Stanley Tsang dafe176570 Documentation updates for NCCL 2.7.0 (#219) 
* Making hip-clang the default compiler; documentation update

* Adding back --hip-clang to install.sh as a silent option for CI

* Documentation updates for NCCL 2.7

* Restoring deleted line in install script

[ROCm/rccl commit: 8d21adb5e3]
 2020-06-16 16:48:11 -06:00
Wenkai Du 9ca3176346 Add IB/RDMA unit test 
[ROCm/rccl commit: cfa97eccd3]
 2020-06-16 18:29:17 +00:00
Wenkai Du 5d93f83ae3 Limit network profiling support to simple protocol and avoid overflow 
[ROCm/rccl commit: 95b8f70d15]
 2020-06-15 20:51:36 +00:00
Wenkai Du 3389838def Rework network proxy profiling 
[ROCm/rccl commit: 7484e53ff7]
 2020-06-13 03:13:58 +00:00
Wenkai Du a8952356d1 Reduce RCCL kernel count as we don't pass first coll in argument 
[ROCm/rccl commit: b257676f30]
 2020-06-12 21:30:04 +00:00
Wenkai Du a90fac8579 Sender rank's opCount maybe ahead by one if it finishes earlier 
[ROCm/rccl commit: a6d621176c]
 2020-06-12 03:39:45 +00:00
Wenkai Du e7dd92dc87 Log NUMA node of RDMA host buffer allocation 
[ROCm/rccl commit: e98891d039]
 2020-06-09 17:44:15 -07:00
Wenkai Du 846e1a4427 Add network proxy profiling support 
[ROCm/rccl commit: 812543104d]
 2020-06-09 17:44:15 -07:00
Wenkai Du bc6840f937 Calculate and use total wait cycles for RCCL profiling 
[ROCm/rccl commit: c9aa11928a]
 2020-06-09 17:44:15 -07:00
Wenkai Du 15cd2aff3c Add gather, scatter and alltoall collectives 
Introducing 3 new APIs:
ncclResult_t  ncclGather(const void* sendbuff, void* recvbuff, size_t sendcount,
    ncclDataType_t datatype, int root, ncclComm_t comm, hipStream_t stream);
ncclResult_t  ncclScatter(const void* sendbuff, void* recvbuff,
    size_t recvcount, ncclDataType_t datatype, int root, ncclComm_t comm,
    hipStream_t stream);
ncclResult_t  ncclAllToAll(const void* sendbuff, void* recvbuff, size_t count,
    ncclDataType_t datatype, ncclComm_t comm, hipStream_t stream);

Only out of place operation is supported.
Preprocessor symbol RCCL_GATHER_SCATTER=1 indicates API availibility.
By default the APIs launche RCCL kernel implementation, which can be disabled by
RCCL_ALLTOALL_KERNEL_DISABLE=1. Then the APIs use wrapper around ncclSend and ncclRecv.


[ROCm/rccl commit: e80e29573c]
 2020-06-09 17:44:08 -07:00
Wenkai Du 67fb4f6cdb Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: 26a0fd2517]
 2020-06-09 17:40:11 -07:00
Sylvain Jeaugey 89072c82e5 2.7.3-1 
Add support for A100 GPU and related platforms.
Add support for CUDA 11.
Add support for send/receive operations (beta).


[ROCm/rccl commit: 5949d96f36]
 2020-06-08 09:31:44 -07:00
Wenkai Du 0108a1219d Use cached value for detecting GDR support only once 
[ROCm/rccl commit: 67c8e72ce3]
 2020-05-22 17:19:10 +00:00
Wenkai Du b449a3b36d Report HIP version in logs 
[ROCm/rccl commit: e41ab173cf]
 2020-05-20 18:15:32 +00:00
Wenkai Du e7eff47be4 Revert "Tuning the inline and unroll to reduce the scratch usage" 
This reverts commit d8a06589c9.


[ROCm/rccl commit: ca493a6b51]
 2020-05-15 14:15:40 -07:00
Wenkai Du 5780637fff Merge pull request #209 from wenkaidu/hip-clang 
Rename files which only diffs in extension

[ROCm/rccl commit: c245f1507e]
 2020-05-15 13:51:12 -07:00
Wenkai Du 61a82ff572 Merge pull request #208 from wenkaidu/perf_xgmi 
Give preference to path with more XGMI connections

[ROCm/rccl commit: 706de76046]
 2020-05-15 10:07:22 -07:00
Wenkai Du c2afb1f4ca Rename files which only diffs in extension 
[ROCm/rccl commit: e7b36304c8]
 2020-05-15 09:16:32 -07:00
Wenkai Du 27519fd019 Give preference to path with more XGMI connections 
[ROCm/rccl commit: b3c9852634]
 2020-05-14 15:33:16 -07:00
Stanley Tsang abba9d5504 Restoring doxygen documentation to nccl.h.in. 
[ROCm/rccl commit: 787ac13486]
 2020-05-12 22:03:31 +00:00
Wenkai Du 5ba51914f1 Update rccl_bfloat16.h to match rocBLAS 
[ROCm/rccl commit: d5a07a7b5c]
 2020-05-08 22:48:07 +00:00
Wenkai Du 069322d05a Set flags when calling hipExtLaunchMultiKernelMultiDevice in hip-clang 
[ROCm/rccl commit: 24ea2ef6dd]
 2020-05-08 15:57:14 +00:00
Wenkai Du 9813d67cd1 Merge remote-tracking branch 'nccl/master' into HEAD 
[ROCm/rccl commit: c4edc257b0]
 2020-04-27 17:16:54 +00:00
Wenkai Du 7b7f781658 Fix incorrect next device ID in PCI ordered search 
[ROCm/rccl commit: edb49ed2d5]
 2020-04-25 01:01:13 +00:00
Sylvain Jeaugey c43022b9d8 Fix crash when only a subset of GPUs are visible within a container. 
Fixes #326.


[ROCm/rccl commit: f36540f55a]
 2020-04-17 10:03:14 -07:00
Sylvain Jeaugey 5df2502deb Improve robustness of PCI detection 
Fallback to default values when class/speed is unknown.


[ROCm/rccl commit: 23a9fbb788]
 2020-04-16 14:27:50 -07:00
aokomoriuta acd868dfcb Fix wrong variable name "slice" to "chunk" 
https://github.com/NVIDIA/nccl/issues/287


[ROCm/rccl commit: a783484ab5]
 2020-04-14 19:00:51 -07:00
Sylvain Jeaugey 627e1a06d0 Fix bug #307 : wrong NIC selection on the reduction tree. 
The reduction tree (tree up) was inverting the NICs to use,
causing performance issue in cases where we are using different
NICs on a given channel.


[ROCm/rccl commit: b5b6c6acdd]
 2020-04-09 17:14:07 -07:00
Aaron Enye Shi bfbfe370c3 Fix HIP-Clang build with HSA headers 
HIP-Clang does not include these HSA headers, and they need to be explicitly added in RCCL.


[ROCm/rccl commit: a95090d981]
 2020-04-03 17:58:23 -04:00
Wenkai Du 8852e54181 topo_expl: update to 2.6 
[ROCm/rccl commit: 6f54b23503]
 2020-04-01 13:37:08 -07:00
Wenkai Du 4aeb7f041e Merge remote-tracking branch 'nccl/master' into v2.6.4_merge 
[ROCm/rccl commit: fa36fd9ef9]
 2020-04-01 13:35:12 -07:00
Sylvain Jeaugey b996c2ca00 Merge pull request #314 from NVIDIA/v2.6 
2.6.4-1

[ROCm/rccl commit: 533e3702cf]
 2020-03-26 17:31:24 -07:00
Sylvain Jeaugey 40adc74496 2.6.4-1 
Add support for network collectives.
Add support for XML topology dump/injection.
Add text values for GDR and P2P Levels, including "NVL".
Add speed detection for PCI, Infiniband and Ethernet cards.
Add CPU detection for ARM and AMD CPUs.
Add support for adaptive routing on Infiniband.
Change NET plugin API to v3 : merge PCI path and GPU pointer
  capability into a single structure and add other properties.


[ROCm/rccl commit: b221128eca]
 2020-03-20 14:58:36 -07:00
Rashika Kheria 38b445c94f Check return code for Flush operation 
Current NCCL code does not abort for failed Flush operations by
underlying network. This may compromise data integrity.

Signed-off-by: Rashika Kheria <rashika@amazon.com>


[ROCm/rccl commit: 6c61492eba]
 2020-03-16 20:40:59 -07:00
Wenkai Du cb19bce4e0 Merge pull request #183 from wenkaidu/dup_rings 
Remove condition for ring duplication

[ROCm/rccl commit: 0976e47b06]
 2020-03-02 17:12:42 -08:00
Wenkai Du 85fd51a06f Remove condition for ring duplication 
Fix insufficent number of rings on single node after pull #179


[ROCm/rccl commit: 62dc28bd2e]
 2020-03-02 12:55:06 -08:00
Wenkai Du 593d99d9a9 Check fine grained memory before enabling RDMA 
Adding back the check which was lost from 2.5 merge.


[ROCm/rccl commit: fb59328a7b]
 2020-03-02 11:18:27 -08:00
Wenkai Du 2a66deb694 Merge pull request #179 from wenkaidu/search 
Use fraction of system maxWidth as steps for searching

[ROCm/rccl commit: 8b5bc8bca2]
 2020-02-28 11:05:46 -08:00
Wenkai Du b750defc28 Merge remote-tracking branch 'remotes/nccl/master' 
[ROCm/rccl commit: 8e73a2ad60]
 2020-02-27 12:53:03 -08:00
Wenkai Du 45a7541582 Revise PCI BW numbers on Rome 
[ROCm/rccl commit: d2adc61bf6]
 2020-02-26 13:17:49 -08:00
Wenkai Du b4be0ff3b8 Use fraction of system maxWidth as steps for searching 
This reverts previous workaround of deducting only half of width
from paths.


[ROCm/rccl commit: 8391637613]
 2020-02-26 09:14:35 -08:00
Wenkai Du 5747c3cac1 Fix abort handling in LL primitives 
[ROCm/rccl commit: 077c3cda74]
 2020-02-25 13:42:54 -08:00
Wenkai Du d640f38d56 Fix system maxSpeed and maxWidth calculation 
[ROCm/rccl commit: 9b80b3633f]
 2020-02-24 15:18:57 -08:00