Commit Graph

448 Commits

Author SHA1 Message Date
saadrahim 0dc019e35f Download GTest if not found in system (#237)
Co-authored-by: Stanley Tsang <stanley.tsang@amd.com>
2020-08-06 09:36:58 -06:00
Wenkai Du d1e20b4c5e Improve 4P2H topology on Rome (#243)
1. Use bi-directional rings
2. GPU search is sorted by PCI device ID to get consistent results
2020-07-28 14:21:44 -07:00
Wenkai Du e7a10aa0e4 Topology tuning for 4P2H on Rome (#242)
* Topology tuning for 4P2H on Rome

* Use ncclTopoIdToIndex
2020-07-27 11:53:57 -07:00
Wenkai Du 8d5fb920b6 ib-test: support multiple channels (#241) 2020-07-27 11:03:12 -07:00
Sourav Chakraborty fe3d520601 Merge pull request #240 from ROCmSoftwarePlatform/sourav/topo-expl-1
simplify model definitions in topo expl
2020-07-22 12:35:17 -05:00
Sourav Chakraborty 2475daafee add 4 node 8P6L 1 NIC 2nd Hive model 2020-07-22 16:27:15 +00:00
Sourav Chakraborty db55afb014 simplify model definitions in topo expl 2020-07-22 16:05:53 +00:00
Wenkai Du d5f90e19b5 Add 8P6L multi-node models (#239) 2020-07-21 14:10:36 -07:00
Stanley Tsang 684f3e6af4 Adding better naming to unit tests for filtering; adding short and full unit test suites (#235) 2020-07-21 12:19:47 -06:00
Wenkai Du 35c5a7fe45 Fix RCCL build package name (#236) 2020-07-20 14:43:00 -07:00
saadrahim 99a491273f Changing GTest inclusion in cmake to use find_package (#234)
* GTest is used via find_package. No longer downloaded in cmake.

* Adding error handling
2020-07-15 20:51:48 -06:00
saadrahim 7f93aa7e53 Changing dependency to hip-rocclr (#228) 2020-07-14 17:49:56 -06:00
Wenkai Du ab787c767e Change default channels duplication for chordal ring (#233) 2020-07-14 15:16:50 -07:00
gilbertlee-amd f87ba17737 Removing UnitTest as install, removing unused env var (#231) 2020-07-10 09:30:28 -06:00
Wenkai Du 5215130168 Revert "Split primitive class to smaller structures" (#230)
This reverts commit 486fd436af.
2020-07-08 11:06:50 -07:00
Wenkai Du 1addf4f196 Match RCCL package name to API version (#229) 2020-07-07 13:30:39 -07:00
Stanley Tsang 9bd4c14603 Adding appropriate references in rccl-prim-test (#227)
Adding appropriate references to rccl-prim-test.
2020-07-06 10:15:03 -06:00
Wenkai Du ecae1cd76a Merge pull request #226 from wenkaidu/develop
Sync up to NCCL 2.7.6
2020-07-06 09:10:09 -07:00
Wenkai Du da3b197d6c Merge remote-tracking branch 'nccl/master' into develop 2020-07-01 16:51:25 -07:00
Wenkai Du d3548cc474 topo_expl: each rank needs to have its own memory for graphs (#225) 2020-07-01 15:11:02 -07:00
Wenkai Du a6be82f5ab topo_expl: fix broken build (#224) 2020-06-30 11:11:23 -07:00
Wenkai Du a144a85465 Merge pull request #223 from wenkaidu/sendrecv
Use separate threads for send and receive
2020-06-30 10:50:06 -07:00
Wenkai Du 8db0aa8f4c gtest: extend testing up to 8 GPUs 2020-06-29 09:32:31 -07:00
Wenkai Du 964c4c2061 Merge sendrecv kernel from NCCL 2.7.3
This commit was cherry-picked and modified from
https://github.com/NVIDIA/nccl/commit/5949d96f36d050e59d05872f8bbffd2549318e95
2020-06-29 08:47:46 -07:00
Wenkai Du b90735c935 Use separate threads for send and receive 2020-06-29 08:47:15 -07:00
Sylvain Jeaugey 1952325569 2.7.6-1
Fix crash when NVswitch is not visible inside a VM.
2020-06-26 16:35:54 -07:00
Sylvain Jeaugey 01afd20a77 2.7.5-1
Minor fixes for A100 platforms.
Add a WARN for invalid GroupEnd call.
2020-06-26 14:39:49 -07:00
Wenkai Du 84f8ba3bb0 Revert use posix_memalign for network buffer allocation on host memory (#222) 2020-06-24 11:25:55 -07:00
Wenkai Du 0eb19a563a Use posix_memalign for network buffer allocation on host memory (#221)
* Use posix_memalign for network buffer allocation on host memory

* ib-test: add ability to specify run iterations

* ib-test: define iterations as multiple of default cycles

* Add checking to posix_memalign return value
2020-06-22 13:06:25 -07:00
Wenkai Du dc739c4e70 ib-test: support host memory allocation through posix_memalign (#220)
* ib-test: support host memory allocation through posix_memalign

* ib-test: add missing CUDACHECK to hip calls
2020-06-17 16:16:54 -07:00
Stanley Tsang 8d21adb5e3 Documentation updates for NCCL 2.7.0 (#219)
* Making hip-clang the default compiler; documentation update

* Adding back --hip-clang to install.sh as a silent option for CI

* Documentation updates for NCCL 2.7

* Restoring deleted line in install script
2020-06-16 16:48:11 -06:00
Wenkai Du 0023b9b081 Merge pull request #218 from wenkaidu/develop
Develop
2020-06-16 13:30:50 -07:00
Wenkai Du cfa97eccd3 Add IB/RDMA unit test 2020-06-16 18:29:17 +00:00
Wenkai Du 95b8f70d15 Limit network profiling support to simple protocol and avoid overflow 2020-06-15 20:51:36 +00:00
Wenkai Du 7484e53ff7 Rework network proxy profiling 2020-06-13 03:13:58 +00:00
Wenkai Du b257676f30 Reduce RCCL kernel count as we don't pass first coll in argument 2020-06-12 21:30:04 +00:00
Wenkai Du a6d621176c Sender rank's opCount maybe ahead by one if it finishes earlier 2020-06-12 03:39:45 +00:00
Wenkai Du fee1a20b74 gtest: add scatter, gather and all to all unit tests 2020-06-09 17:44:15 -07:00
Wenkai Du e98891d039 Log NUMA node of RDMA host buffer allocation 2020-06-09 17:44:15 -07:00
Wenkai Du 812543104d Add network proxy profiling support 2020-06-09 17:44:15 -07:00
Wenkai Du c9aa11928a Calculate and use total wait cycles for RCCL profiling 2020-06-09 17:44:15 -07:00
Wenkai Du e80e29573c Add gather, scatter and alltoall collectives
Introducing 3 new APIs:
ncclResult_t  ncclGather(const void* sendbuff, void* recvbuff, size_t sendcount,
    ncclDataType_t datatype, int root, ncclComm_t comm, hipStream_t stream);
ncclResult_t  ncclScatter(const void* sendbuff, void* recvbuff,
    size_t recvcount, ncclDataType_t datatype, int root, ncclComm_t comm,
    hipStream_t stream);
ncclResult_t  ncclAllToAll(const void* sendbuff, void* recvbuff, size_t count,
    ncclDataType_t datatype, ncclComm_t comm, hipStream_t stream);

Only out of place operation is supported.
Preprocessor symbol RCCL_GATHER_SCATTER=1 indicates API availibility.
By default the APIs launche RCCL kernel implementation, which can be disabled by
RCCL_ALLTOALL_KERNEL_DISABLE=1. Then the APIs use wrapper around ncclSend and ncclRecv.
2020-06-09 17:44:08 -07:00
Wenkai Du 71ec3e09df tpol_expl: update to 2.7 2020-06-09 17:40:24 -07:00
Wenkai Du 26a0fd2517 Merge remote-tracking branch 'nccl/master' into develop 2020-06-09 17:40:11 -07:00
Sylvain Jeaugey 5949d96f36 2.7.3-1
Add support for A100 GPU and related platforms.
Add support for CUDA 11.
Add support for send/receive operations (beta).
2020-06-08 09:31:44 -07:00
saadrahim 87db65f22d Fixing CI as install.sh script should not install dependencies without user request (#217) 2020-06-05 11:04:03 -06:00
Stanley Tsang dc403e0ca2 Making hip-clang the default compiler; documentation update (#216)
* Making hip-clang the default compiler; documentation update

* Adding back --hip-clang to install.sh as a silent option for CI
2020-06-04 11:58:27 -06:00
Wenkai Du 2a4514772c Merge pull request #214 from wenkaidu/gdr
Use cached value for detecting GDR support only once
2020-05-22 13:36:23 -07:00
Wenkai Du 67c8e72ce3 Use cached value for detecting GDR support only once 2020-05-22 17:19:10 +00:00
Wenkai Du 957be85944 Merge pull request #212 from wenkaidu/version
Report HIP version in logs
2020-05-20 16:25:54 -07:00