AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
149 Commits 3 Branches 0 Tags
08a65f22012c80d2230757740e3a5b18d2f0290b
Commit Graph

56 Commits

Author SHA1 Message Date
Gilbert Lee 60f91f645d Updating RCCL based on NCCL 2.3.7 
- Contains modifications to support AMD hardware
- Adds unit tests


[ROCm/rccl commit: 55a4b22ad7]
 2019-05-16 16:16:18 +00:00
Christian Sigg 3a7974d4c4 Fix memory leak in bootstrapRoot() 
[ROCm/rccl commit: 4861e197fd]
 2019-01-07 14:18:46 -08:00
Sylvain Jeaugey 4e32c7a19a Replace CUDA_VERSION by CUDART_VERSION 
[ROCm/rccl commit: c244b51ae7]
 2018-12-13 15:22:17 -08:00
Christian Sigg 6a93dca51b Qualify nullptr_t with std:: 
[ROCm/rccl commit: 3e6afef473]
 2018-12-13 14:18:09 -08:00
Christian Sigg fcf027b42d Two temporary workarounds for cuda-clang issues. 
[ROCm/rccl commit: 346fc49514]
 2018-12-13 14:17:58 -08:00
Christian Sigg 457c2ab5ae Change __CUDACC_VER_*__ preprocessor directives to CUDA_VERSION because clang doesn't define the former. 
[ROCm/rccl commit: d08e9b5279]
 2018-12-13 14:17:46 -08:00
Sylvain Jeaugey 7fa81a40a1 Fix #163 : remove warnings 
[ROCm/rccl commit: 469b69a5d0]
 2018-12-11 09:19:16 -08:00
Sylvain Jeaugey e2f73fea64 Remove error logging from a normal path 
When initNet fails, we should not print the backtrace as it is
supposed to be normal operation (falling back to sockets)


[ROCm/rccl commit: 57368189e1]
 2018-12-04 14:47:41 -08:00
Sylvain Jeaugey 49819bcb7c Fix GPU Direct RDMA detection. 
Whether the network supported GPU Direct RDMA or not was ignored,
causing sockets to break when cards were local enough that NCCL
tried to use it.


[ROCm/rccl commit: 4b39a4cf91]
 2018-12-04 14:42:28 -08:00
Sylvain Jeaugey 274fd3756a Add NCCL_NET flag to many debug lines. 
[ROCm/rccl commit: b8a9a32ccb]
 2018-12-04 13:10:19 -08:00
Sylvain Jeaugey 245786e9a5 Improve INFO message when external network is not found. 
Fix #162


[ROCm/rccl commit: cdae05b277]
 2018-12-04 12:10:58 -08:00
David Addison f842c81503 Fixed some compilation errors when TRACE=1 set 
[ROCm/rccl commit: 5fe2618c0e]
 2018-11-29 14:12:14 -08:00
Sylvain Jeaugey 5dffba1287 Rework shared memory code to use SYSCHECK macros. 
This is to handle EINTR/EGAIN properly (issue #137), and also
make the code consistent with the rest.

Unfortunately posix_fallocate and mmap do not follow the classic
return code/errno pattern, so we need to write wrappers around those
functions.


[ROCm/rccl commit: eed8218e17]
 2018-11-29 12:52:13 -08:00
Sylvain Jeaugey 57bff87e1d Rework SYSCHECK macros to better handle retries. 
SYSCHECKVAL was not retrying when a retry was needed. Since not all
calls are inside a loop, that means we could silently miss an
EINTR/EAGAIN return code.

Also rework the socket connection code and improve error reporting.


[ROCm/rccl commit: 302d538b73]
 2018-11-29 12:52:13 -08:00
Sylvain Jeaugey f0e160728f Improve net API description 
[ROCm/rccl commit: 61b50a63ef]
 2018-11-26 16:24:31 -08:00
Sylvain Jeaugey ec1247221e Make network isend/irecv non blocking 
[ROCm/rccl commit: 98adf2fe11]
 2018-11-26 16:24:31 -08:00
Sylvain Jeaugey 378a103248 Add support for external network. 
Dynamically load external network from libnccl-net.so.
Add init function in networks.
Move PCI scoring to net.cu, only ask transport to provide a path.
Simplify CUDA PCI path detection.
Add dummy external network


[ROCm/rccl commit: 0d3a20f96d]
 2018-11-26 16:24:31 -08:00
Alex Sergeev b458b3b337 Generate host-hash for P2P and SHM based on $(readlink /proc/self/ns/uts) + $(readlink /proc/self/ns/mnt) (#156) 
[ROCm/rccl commit: d7a58cfa58]
 2018-11-19 17:39:44 -08:00
Sylvain Jeaugey 726c60336d Generate nccl.h in build instead of src 
Generating nccl.h in src makes source directories dirty after builds.

[ROCm/rccl commit: 3c6e25210b]
 2018-11-09 14:00:41 -08:00
David Addison 8c945929af 2.3.7-1 
Improved LL tuning for multi-node jobs.
Improved bootstrap for large job scaling.
Fixed a hang during bootstrap due to socket reuse.
Added operation name to the COLL INFO logging.


[ROCm/rccl commit: b56650c7f5]
 2018-10-24 14:44:59 -07:00
Sylvain Jeaugey 8ffcfac437 2.3.5-5 
Add support for inter-node communication using sockets and InfiniBand/RoCE.
Improve latency.
Add support for aggregation.
Improve LL/regular tuning.
Remove tests as those are now at github.com/nvidia/nccl-tests .


[ROCm/rccl commit: f93fe9bfd9]
 2018-09-25 14:12:01 -07:00
Sylvain Jeaugey 338f6eaae1 Add support for CUDA9 half semantics 
[ROCm/rccl commit: 29a1a916dc]
 2017-06-14 11:20:24 -07:00
Ilya Biryukov fa9d609d5d Fix compilation error when compiling with 'clang -x cuda'. 
Functions vFetch and vStore are not found by ADL with clang,
so they need to be declared before usage in ReduceCopy.


[ROCm/rccl commit: 8241cd7b6e]
 2017-03-16 12:01:11 +01:00
Nathan Luehr 72ab3de6f6 Only enable peer access for ring neighbors. 
This enables support for systems with more than 9 GPUs attached to a single PCIe root complex.


[ROCm/rccl commit: 8996811936]
 2017-03-01 16:42:38 -08:00
Sylvain Jeaugey f93f7b5d52 Fix copy/paste typo in error message 
[ROCm/rccl commit: c219a183d0]
 2017-03-01 16:42:38 -08:00
Sylvain Jeaugey d2a9c5d52e Fix crash in Reduce when non-root ranks have invalid recvbuff 
[ROCm/rccl commit: 8e1d6f9b60]
 2017-03-01 16:42:38 -08:00
Chad Whipkey 030edd0507 Qualify nullptr_t with std::. 
[ROCm/rccl commit: 5eab428294]
 2017-02-08 07:06:31 -08:00
Sylvain Jeaugey 851663654d Fix 1.3.2 compilation 
[ROCm/rccl commit: 2a974f5ca2]
 2016-12-08 09:11:43 -08:00
Sylvain Jeaugey 56e1a845e9 Adding missing file 
[ROCm/rccl commit: 648e9fbb58]
 2016-12-05 18:06:24 -08:00
Sylvain Jeaugey db3103d96a 1.3.2 release 
Broadcast tuning
Better checking of inputs
Copy/reduce code simplification


[ROCm/rccl commit: 34d27771c6]
 2016-12-01 15:17:50 -08:00
Sylvain Jeaugey ea6feab31f Fix primitives function prototype 
[ROCm/rccl commit: b2781d0501]
 2016-10-13 10:32:42 -07:00
Sylvain Jeaugey ad50fded63 NVML (libwrap) : import the needed definitions 
[ROCm/rccl commit: bf7d1514f7]
 2016-10-13 10:28:59 -07:00
Sylvain Jeaugey 827119c43e Improved allreduce segmentation for small sizes 
[ROCm/rccl commit: 8bb06c94be]
 2016-10-07 12:42:23 -07:00
Sylvain Jeaugey b540c2b5cc Heavy code refactoring to remove a lot of code in collectives (~1000 lines). 
Have all collectives use the same args, the same ring, and the same primitives for synchronization between threads with the same pattern.


[ROCm/rccl commit: cabd6848e4]
 2016-09-22 11:57:56 -07:00
Sylvain Jeaugey 93f5211c22 Add profiling API 
[ROCm/rccl commit: e3dbc6110e]
 2016-09-22 11:56:51 -07:00
Sylvain Jeaugey 996a743a44 Merge pull request #41 from jia-kai/master 
Some minor fixes for compile/usage

[ROCm/rccl commit: 9ee6189bf9]
 2016-09-15 09:45:52 -07:00
Sylvain Jeaugey a59d93cf7a Updated LICENCE.txt 
[ROCm/rccl commit: 75bad643bd]
 2016-08-26 15:08:20 -07:00
jiakai 1cd55c5e03 pass devlist as const int* rather than int* in ncclCommInitAll 
[ROCm/rccl commit: 47b0797fe1]
 2016-08-19 19:00:14 +08:00
Sylvain Jeaugey 43d2570611 Merge remote-tracking branch 'github/master' into public 
[ROCm/rccl commit: 428ec5b2a3]
 2016-07-25 10:53:01 -07:00
Nathan Luehr 159408c5d5 Fixed redundant contexts in multi-process apps 
Change-Id: If787014450fd281304f0c7baf01d25963e40905d


[ROCm/rccl commit: 55c42ad681]
 2016-07-25 10:10:30 -07:00
Sylvain Jeaugey 2a9f6574a5 Add a debug level to NCCL and CUDA versions at init 
[ROCm/rccl commit: e51e922924]
 2016-06-16 17:04:41 -07:00
Sylvain Jeaugey db542e763d Only call the CUDA runtime. That may fix #27. 
[ROCm/rccl commit: d5e507fc7f]
 2016-06-07 16:27:51 -07:00
Sylvain Jeaugey c0c210d2d2 Make NCCL collectives work on communicators with only one rank 
[ROCm/rccl commit: 7edfc57228]
 2016-06-06 14:35:00 -07:00
Sylvain Jeaugey d4fbd0464e Removing unneeded includes 
[ROCm/rccl commit: acb93d1aed]
 2016-06-02 17:33:43 -07:00
Sylvain Jeaugey 50903c8ce0 Fix random deadlock during ncclCommInitRank. 
[ROCm/rccl commit: dba3ec9428]
 2016-04-19 10:47:27 -07:00
Nathan Luehr b029c954f1 Fixed useRemoteRecv consistency issue. 
Change-Id: Ib093a8dc3bb093eddc89dad81d3fffa53c03a6a2
Reviewed-on: http://git-master/r/1013543
Reviewed-by: Cliff Woolley <jwoolley@nvidia.com>
Tested-by: Przemek Tredak <ptredak@nvidia.com>


[ROCm/rccl commit: 5554a4c9f0]
 2016-02-18 13:45:42 -08:00
Nathan Luehr b5a2ca138d Fixed buffer overflow in ReduceOrCopy 
Bug caused AllGathers and ReduceScatters of less than
8 bytes to fail in certain cases.

Change-Id: I33e1beb50805bfdb457ae16a90e3f91c1b283b9b
Reviewed-on: http://git-master/r/1011505
Reviewed-by: Przemek Tredak <ptredak@nvidia.com>
Tested-by: Przemek Tredak <ptredak@nvidia.com>


[ROCm/rccl commit: 9442285526]
 2016-02-12 15:13:56 -08:00
Nathan Luehr 60799f9427 Libwrap checks for LIB.so.1 if LIB.so not found 
Change-Id: I6f07f887f828cb2259dcfd496a2ad707db898cf5
Reviewed-on: http://git-master/r/1000162
Reviewed-by: Przemek Tredak <ptredak@nvidia.com>
Tested-by: Przemek Tredak <ptredak@nvidia.com>


[ROCm/rccl commit: caa40b8dd3]
 2016-01-29 12:36:42 -08:00
Nathan Luehr c019d205a9 Enabled support for char type to be unsigned. 
GCC on POWER arch defines char type as unsigned.

Change-Id: Ic143cb058fe42414b1f6f1f45b02132c837726ae
Reviewed-on: http://git-master/r/999614
Reviewed-by: Przemek Tredak <ptredak@nvidia.com>
Tested-by: Przemek Tredak <ptredak@nvidia.com>


[ROCm/rccl commit: fe1a956715]
 2016-01-28 13:38:18 -08:00
Sylvain Jeaugey 42c3edc35a Moved tests to separate dir and improved MPI test 
test sources moved to test/ directory.
MPI test displays PASS/FAIL and returns code accordingly.

Change-Id: I058ebd1bd5202d8f38cc9787898b2480100c102b
Reviewed-on: http://git-master/r/936086
Reviewed-by: Przemek Tredak <ptredak@nvidia.com>
Tested-by: Przemek Tredak <ptredak@nvidia.com>


[ROCm/rccl commit: c05312f151]
 2016-01-28 12:56:36 -08:00