754ed213cc
do not use internal stream ( #79 )
2019-06-12 16:26:59 -06:00
b7a6307371
Implement HDP flush when transfer data over PCIe P2P ( #75 )
...
* Implement HDP flush when transfer data over PCIe P2P
* Add some descriptions to HDP flushing
* Fix for review comments
2019-06-03 16:29:55 -07:00
5827a4f616
Make ncclFuncs static
...
This is necessary to constant propagate the function pointers
to eliminate the indirect function call.
2019-05-29 10:50:13 -04:00
f45566a8bd
Increase number of rings with XGMI connection
...
Improve throughput for about 20%. Also remove P2P over PCIe which was
left enabled at initial release.
Signed-off-by: Wenkai Du <wenkai.du@amd.com >
2019-05-24 20:58:51 +00:00
b921279a21
Fix build failure for hip-clang ( #69 )
2019-05-23 16:53:25 -06:00
55a4b22ad7
Updating RCCL based on NCCL 2.3.7
...
- Contains modifications to support AMD hardware
- Adds unit tests
2019-05-16 16:16:18 +00:00
4861e197fd
Fix memory leak in bootstrapRoot()
2019-01-07 14:18:46 -08:00
c244b51ae7
Replace CUDA_VERSION by CUDART_VERSION
2018-12-13 15:22:17 -08:00
3e6afef473
Qualify nullptr_t with std::
2018-12-13 14:18:09 -08:00
346fc49514
Two temporary workarounds for cuda-clang issues.
2018-12-13 14:17:58 -08:00
d08e9b5279
Change __CUDACC_VER_*__ preprocessor directives to CUDA_VERSION because clang doesn't define the former.
2018-12-13 14:17:46 -08:00
469b69a5d0
Fix #163 : remove warnings
2018-12-11 09:19:16 -08:00
57368189e1
Remove error logging from a normal path
...
When initNet fails, we should not print the backtrace as it is
supposed to be normal operation (falling back to sockets)
2018-12-04 14:47:41 -08:00
4b39a4cf91
Fix GPU Direct RDMA detection.
...
Whether the network supported GPU Direct RDMA or not was ignored,
causing sockets to break when cards were local enough that NCCL
tried to use it.
2018-12-04 14:42:28 -08:00
b8a9a32ccb
Add NCCL_NET flag to many debug lines.
2018-12-04 13:10:19 -08:00
cdae05b277
Improve INFO message when external network is not found.
...
Fix #162
2018-12-04 12:10:58 -08:00
5fe2618c0e
Fixed some compilation errors when TRACE=1 set
2018-11-29 14:12:14 -08:00
eed8218e17
Rework shared memory code to use SYSCHECK macros.
...
This is to handle EINTR/EGAIN properly (issue #137 ), and also
make the code consistent with the rest.
Unfortunately posix_fallocate and mmap do not follow the classic
return code/errno pattern, so we need to write wrappers around those
functions.
2018-11-29 12:52:13 -08:00
302d538b73
Rework SYSCHECK macros to better handle retries.
...
SYSCHECKVAL was not retrying when a retry was needed. Since not all
calls are inside a loop, that means we could silently miss an
EINTR/EAGAIN return code.
Also rework the socket connection code and improve error reporting.
2018-11-29 12:52:13 -08:00
61b50a63ef
Improve net API description
2018-11-26 16:24:31 -08:00
98adf2fe11
Make network isend/irecv non blocking
2018-11-26 16:24:31 -08:00
0d3a20f96d
Add support for external network.
...
Dynamically load external network from libnccl-net.so.
Add init function in networks.
Move PCI scoring to net.cu, only ask transport to provide a path.
Simplify CUDA PCI path detection.
Add dummy external network
2018-11-26 16:24:31 -08:00
d7a58cfa58
Generate host-hash for P2P and SHM based on $(readlink /proc/self/ns/uts) + $(readlink /proc/self/ns/mnt) ( #156 )
2018-11-19 17:39:44 -08:00
3c6e25210b
Generate nccl.h in build instead of src
...
Generating nccl.h in src makes source directories dirty after builds.
2018-11-09 14:00:41 -08:00
b56650c7f5
2.3.7-1
...
Improved LL tuning for multi-node jobs.
Improved bootstrap for large job scaling.
Fixed a hang during bootstrap due to socket reuse.
Added operation name to the COLL INFO logging.
2018-10-24 14:44:59 -07:00
f93fe9bfd9
2.3.5-5
...
Add support for inter-node communication using sockets and InfiniBand/RoCE.
Improve latency.
Add support for aggregation.
Improve LL/regular tuning.
Remove tests as those are now at github.com/nvidia/nccl-tests .
2018-09-25 14:12:01 -07:00
29a1a916dc
Add support for CUDA9 half semantics
2017-06-14 11:20:24 -07:00
8241cd7b6e
Fix compilation error when compiling with 'clang -x cuda'.
...
Functions vFetch and vStore are not found by ADL with clang,
so they need to be declared before usage in ReduceCopy.
2017-03-16 12:01:11 +01:00
8996811936
Only enable peer access for ring neighbors.
...
This enables support for systems with more than 9 GPUs attached to a single PCIe root complex.
2017-03-01 16:42:38 -08:00
c219a183d0
Fix copy/paste typo in error message
2017-03-01 16:42:38 -08:00
8e1d6f9b60
Fix crash in Reduce when non-root ranks have invalid recvbuff
2017-03-01 16:42:38 -08:00
5eab428294
Qualify nullptr_t with std::.
2017-02-08 07:06:31 -08:00
2a974f5ca2
Fix 1.3.2 compilation
2016-12-08 09:11:43 -08:00
648e9fbb58
Adding missing file
2016-12-05 18:06:24 -08:00
34d27771c6
1.3.2 release
...
Broadcast tuning
Better checking of inputs
Copy/reduce code simplification
2016-12-01 15:17:50 -08:00
b2781d0501
Fix primitives function prototype
2016-10-13 10:32:42 -07:00
bf7d1514f7
NVML (libwrap) : import the needed definitions
2016-10-13 10:28:59 -07:00
8bb06c94be
Improved allreduce segmentation for small sizes
2016-10-07 12:42:23 -07:00
cabd6848e4
Heavy code refactoring to remove a lot of code in collectives (~1000 lines).
...
Have all collectives use the same args, the same ring, and the same primitives for synchronization between threads with the same pattern.
2016-09-22 11:57:56 -07:00
e3dbc6110e
Add profiling API
2016-09-22 11:56:51 -07:00
9ee6189bf9
Merge pull request #41 from jia-kai/master
...
Some minor fixes for compile/usage
2016-09-15 09:45:52 -07:00
75bad643bd
Updated LICENCE.txt
2016-08-26 15:08:20 -07:00
47b0797fe1
pass devlist as const int* rather than int* in ncclCommInitAll
2016-08-19 19:00:14 +08:00
428ec5b2a3
Merge remote-tracking branch 'github/master' into public
2016-07-25 10:53:01 -07:00
55c42ad681
Fixed redundant contexts in multi-process apps
...
Change-Id: If787014450fd281304f0c7baf01d25963e40905d
2016-07-25 10:10:30 -07:00
e51e922924
Add a debug level to NCCL and CUDA versions at init
2016-06-16 17:04:41 -07:00
d5e507fc7f
Only call the CUDA runtime. That may fix #27 .
2016-06-07 16:27:51 -07:00
7edfc57228
Make NCCL collectives work on communicators with only one rank
2016-06-06 14:35:00 -07:00
acb93d1aed
Removing unneeded includes
2016-06-02 17:33:43 -07:00
dba3ec9428
Fix random deadlock during ncclCommInitRank.
2016-04-19 10:47:27 -07:00
Jeff Daily
Wenkai Du
Yaxun Sam Liu
Wenkai Du
Gilbert Lee
Christian Sigg
Sylvain Jeaugey
Christian Sigg
David Addison
Alex Sergeev
Ilya Biryukov
Nathan Luehr
Chad Whipkey
jiakai