AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
455 Commits 3 Branches 0 Tags
14ad6ff3b4124c9dffd69eeb275bb60c7e0a43e2
Commit Graph

382 Commits

Author SHA1 Message Date
Wenkai Du 14ad6ff3b4 Merge remote-tracking branch 'nccl/master' into 2.7.8 
[ROCm/rccl commit: 7e3f841fab]
 2020-08-10 16:11:00 +00:00
Stanley Tsang bbc4b72ebe Adding static library building option. (#244) 
* Adding static library building option.

* Disabling running tests for static build

* Removing static packaging in CI

Co-authored-by: Saad Rahim <saad.rahim@amd.com>

[ROCm/rccl commit: c5d4d9eb76]
 2020-08-06 11:19:43 -06:00
saadrahim e5432857db Download GTest if not found in system (#237) 
Co-authored-by: Stanley Tsang <stanley.tsang@amd.com>

[ROCm/rccl commit: 0dc019e35f]
 2020-08-06 09:36:58 -06:00
Jack Snyder dca19952fd Setting type when gpu sub node is discovered 
[ROCm/rccl commit: de49a77074]
 2020-08-05 13:39:23 -07:00
Eric Badger d6a78cb1c7 Don't require NIC devices to have specific PCI class 
If a PCI node is the parent of a NIC, treat it as such, regardless of
the PCI class code for the device. This allows non-traditional devices
to act as NICs via the net plugin mechanism.

For consistency, treat GPUs similarly.


[ROCm/rccl commit: 700c0e0f24]
 2020-08-05 12:46:29 -07:00
Wenkai Du 22a6211eaf Improve 4P2H topology on Rome (#243) 
1. Use bi-directional rings
2. GPU search is sorted by PCI device ID to get consistent results

[ROCm/rccl commit: d1e20b4c5e]
 2020-07-28 14:21:44 -07:00
David Addison 4acaffebb3 2.7.8-1 
Fix collective mismatch error when using ncclSend/ncclRecv


[ROCm/rccl commit: 033d799524]
 2020-07-27 16:34:09 -07:00
Wenkai Du 487f93b83f Topology tuning for 4P2H on Rome (#242) 
* Topology tuning for 4P2H on Rome

* Use ncclTopoIdToIndex

[ROCm/rccl commit: e7a10aa0e4]
 2020-07-27 11:53:57 -07:00
Wenkai Du 49c667ac8a ib-test: support multiple channels (#241) 
[ROCm/rccl commit: 8d5fb920b6]
 2020-07-27 11:03:12 -07:00
Sourav Chakraborty e55a3f20ba add 4 node 8P6L 1 NIC 2nd Hive model 
[ROCm/rccl commit: 2475daafee]
 2020-07-22 16:27:15 +00:00
Sourav Chakraborty b3306d1c13 simplify model definitions in topo expl 
[ROCm/rccl commit: db55afb014]
 2020-07-22 16:05:53 +00:00
Wenkai Du f604fc774e Add 8P6L multi-node models (#239) 
[ROCm/rccl commit: d5f90e19b5]
 2020-07-21 14:10:36 -07:00
Stanley Tsang 56d8c7c893 Adding better naming to unit tests for filtering; adding short and full unit test suites (#235) 
[ROCm/rccl commit: 684f3e6af4]
 2020-07-21 12:19:47 -06:00
Wenkai Du cb80d93b11 Fix RCCL build package name (#236) 
[ROCm/rccl commit: 35c5a7fe45]
 2020-07-20 14:43:00 -07:00
saadrahim f80136bbba Changing GTest inclusion in cmake to use find_package (#234) 
* GTest is used via find_package. No longer downloaded in cmake.

* Adding error handling

[ROCm/rccl commit: 99a491273f]
 2020-07-15 20:51:48 -06:00
saadrahim b600c00292 Changing dependency to hip-rocclr (#228) 
[ROCm/rccl commit: 7f93aa7e53]
 2020-07-14 17:49:56 -06:00
Wenkai Du 3e2c9054cd Change default channels duplication for chordal ring (#233) 
[ROCm/rccl commit: ab787c767e]
 2020-07-14 15:16:50 -07:00
gilbertlee-amd b83acc8032 Removing UnitTest as install, removing unused env var (#231) 
[ROCm/rccl commit: f87ba17737]
 2020-07-10 09:30:28 -06:00
Wenkai Du 47dfed5f01 Revert "Split primitive class to smaller structures" (#230) 
This reverts commit 622b49e80a.

[ROCm/rccl commit: 5215130168]
 2020-07-08 11:06:50 -07:00
Wenkai Du 4a3b58ac3a Match RCCL package name to API version (#229) 
[ROCm/rccl commit: 1addf4f196]
 2020-07-07 13:30:39 -07:00
Riatre Foo e791f8c8c7 Fix build action order 
Add $(INCTARGETS) to build dependencies of %.o and $(DEVICELIB).
As there were no dep files during the first build, Make may kick off source
compilation before nccl.h got generated, which leads to occasional build
failures on systems with high core count. The build failure could be
reproduced reliably with a `sleep 5` in $(INCDIR)/nccl.h rule.


[ROCm/rccl commit: 2d8601701d]
 2020-07-07 10:20:51 -07:00
Stanley Tsang 3dd54437fe Adding appropriate references in rccl-prim-test (#227) 
Adding appropriate references to rccl-prim-test.

[ROCm/rccl commit: 9bd4c14603]
 2020-07-06 10:15:03 -06:00
Wenkai Du c7805f224e Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: da3b197d6c]
 2020-07-01 16:51:25 -07:00
Wenkai Du 2f99c7bbad topo_expl: each rank needs to have its own memory for graphs (#225) 
[ROCm/rccl commit: d3548cc474]
 2020-07-01 15:11:02 -07:00
Wenkai Du e8da2a0da6 topo_expl: fix broken build (#224) 
[ROCm/rccl commit: a6be82f5ab]
 2020-06-30 11:11:23 -07:00
Wenkai Du ac87c9db37 gtest: extend testing up to 8 GPUs 
[ROCm/rccl commit: 8db0aa8f4c]
 2020-06-29 09:32:31 -07:00
Wenkai Du 77d94493c3 Merge sendrecv kernel from NCCL 2.7.3 
This commit was cherry-picked and modified from
https://github.com/NVIDIA/nccl/commit/89072c82e551d7c8bf7bd5ca77780ef540b7611e


[ROCm/rccl commit: 964c4c2061]
 2020-06-29 08:47:46 -07:00
Wenkai Du 3c674c9c1f Use separate threads for send and receive 
[ROCm/rccl commit: b90735c935]
 2020-06-29 08:47:15 -07:00
Sylvain Jeaugey 5432eb8e9d 2.7.6-1 
Fix crash when NVswitch is not visible inside a VM.


[ROCm/rccl commit: 1952325569]
 2020-06-26 16:35:54 -07:00
Sylvain Jeaugey 1f6b75ca8b 2.7.5-1 
Minor fixes for A100 platforms.
Add a WARN for invalid GroupEnd call.


[ROCm/rccl commit: 01afd20a77]
 2020-06-26 14:39:49 -07:00
Wenkai Du 117f0f6e6e Revert use posix_memalign for network buffer allocation on host memory (#222) 
[ROCm/rccl commit: 84f8ba3bb0]
 2020-06-24 11:25:55 -07:00
Wenkai Du e7d664200f Use posix_memalign for network buffer allocation on host memory (#221) 
* Use posix_memalign for network buffer allocation on host memory

* ib-test: add ability to specify run iterations

* ib-test: define iterations as multiple of default cycles

* Add checking to posix_memalign return value

[ROCm/rccl commit: 0eb19a563a]
 2020-06-22 13:06:25 -07:00
Wenkai Du 2dc0cfcbf5 ib-test: support host memory allocation through posix_memalign (#220) 
* ib-test: support host memory allocation through posix_memalign

* ib-test: add missing CUDACHECK to hip calls

[ROCm/rccl commit: dc739c4e70]
 2020-06-17 16:16:54 -07:00
Stanley Tsang dafe176570 Documentation updates for NCCL 2.7.0 (#219) 
* Making hip-clang the default compiler; documentation update

* Adding back --hip-clang to install.sh as a silent option for CI

* Documentation updates for NCCL 2.7

* Restoring deleted line in install script

[ROCm/rccl commit: 8d21adb5e3]
 2020-06-16 16:48:11 -06:00
Wenkai Du 9ca3176346 Add IB/RDMA unit test 
[ROCm/rccl commit: cfa97eccd3]
 2020-06-16 18:29:17 +00:00
Wenkai Du 5d93f83ae3 Limit network profiling support to simple protocol and avoid overflow 
[ROCm/rccl commit: 95b8f70d15]
 2020-06-15 20:51:36 +00:00
Wenkai Du 3389838def Rework network proxy profiling 
[ROCm/rccl commit: 7484e53ff7]
 2020-06-13 03:13:58 +00:00
Wenkai Du a8952356d1 Reduce RCCL kernel count as we don't pass first coll in argument 
[ROCm/rccl commit: b257676f30]
 2020-06-12 21:30:04 +00:00
Wenkai Du a90fac8579 Sender rank's opCount maybe ahead by one if it finishes earlier 
[ROCm/rccl commit: a6d621176c]
 2020-06-12 03:39:45 +00:00
Wenkai Du d1f8fdc3a8 gtest: add scatter, gather and all to all unit tests 
[ROCm/rccl commit: fee1a20b74]
 2020-06-09 17:44:15 -07:00
Wenkai Du e7dd92dc87 Log NUMA node of RDMA host buffer allocation 
[ROCm/rccl commit: e98891d039]
 2020-06-09 17:44:15 -07:00
Wenkai Du 846e1a4427 Add network proxy profiling support 
[ROCm/rccl commit: 812543104d]
 2020-06-09 17:44:15 -07:00
Wenkai Du bc6840f937 Calculate and use total wait cycles for RCCL profiling 
[ROCm/rccl commit: c9aa11928a]
 2020-06-09 17:44:15 -07:00
Wenkai Du 15cd2aff3c Add gather, scatter and alltoall collectives 
Introducing 3 new APIs:
ncclResult_t  ncclGather(const void* sendbuff, void* recvbuff, size_t sendcount,
    ncclDataType_t datatype, int root, ncclComm_t comm, hipStream_t stream);
ncclResult_t  ncclScatter(const void* sendbuff, void* recvbuff,
    size_t recvcount, ncclDataType_t datatype, int root, ncclComm_t comm,
    hipStream_t stream);
ncclResult_t  ncclAllToAll(const void* sendbuff, void* recvbuff, size_t count,
    ncclDataType_t datatype, ncclComm_t comm, hipStream_t stream);

Only out of place operation is supported.
Preprocessor symbol RCCL_GATHER_SCATTER=1 indicates API availibility.
By default the APIs launche RCCL kernel implementation, which can be disabled by
RCCL_ALLTOALL_KERNEL_DISABLE=1. Then the APIs use wrapper around ncclSend and ncclRecv.


[ROCm/rccl commit: e80e29573c]
 2020-06-09 17:44:08 -07:00
Wenkai Du 69eb70ce43 tpol_expl: update to 2.7 
[ROCm/rccl commit: 71ec3e09df]
 2020-06-09 17:40:24 -07:00
Wenkai Du 67fb4f6cdb Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: 26a0fd2517]
 2020-06-09 17:40:11 -07:00
Sylvain Jeaugey 89072c82e5 2.7.3-1 
Add support for A100 GPU and related platforms.
Add support for CUDA 11.
Add support for send/receive operations (beta).


[ROCm/rccl commit: 5949d96f36]
 2020-06-08 09:31:44 -07:00
saadrahim 3077902de6 Fixing CI as install.sh script should not install dependencies without user request (#217) 
[ROCm/rccl commit: 87db65f22d]
 2020-06-05 11:04:03 -06:00
Stanley Tsang 8e43b854f1 Making hip-clang the default compiler; documentation update (#216) 
* Making hip-clang the default compiler; documentation update

* Adding back --hip-clang to install.sh as a silent option for CI

[ROCm/rccl commit: dc403e0ca2]
 2020-06-04 11:58:27 -06:00
Wenkai Du 0108a1219d Use cached value for detecting GDR support only once 
[ROCm/rccl commit: 67c8e72ce3]
 2020-05-22 17:19:10 +00:00