e7c8825b0b
Wrap ncclCommWindowRegister() calls within ncclGroup
2025-06-03 10:36:53 -07:00
5290298ab6
Reinstate Pascal suppport for CUDA 12.8+ builds
2025-06-02 09:29:52 -07:00
8bc16f4e01
Need to drop Volta (sm_70) support from CUDA 13.0
2025-05-30 18:04:25 -07:00
a5c539e68b
Add support for Symmetric Memory Registration
...
From NCCL 2.27.x we can now use the Symmetric Memory APIs (-R 2)
2025-05-30 17:31:34 -07:00
e041d901e6
Re-add sm_70 support for CUDA 12.8+ and 13.0 builds
2025-05-07 10:30:59 -07:00
1021260ca9
Make verifiable a DSO and add NAME_SUFFIX support
...
Build option DSO=1 generates libverifiable.so which can be
used to reduce the combined binary size.
Build option NAME_SUFFIX can be used to a add suffix to all
generated binaries. e.g. NAME_SUFFIX=_mpi
Added new make target: clean_intermediates
2025-04-23 17:07:24 -07:00
501a149d57
Add support for FP8 datatypes
...
Added new datatypes: f8e4m3, f8e5m2
Only supported on H100+ architectures and NCCL versions >= 2.24.0
2025-04-18 19:20:59 -07:00
b4300cc79d
Add PCI domain and device ID for GPU device BDF display
2025-02-28 13:25:51 -08:00
a89cf07fe8
Perftests: Introduce NCCL_TESTS_SPLIT env
...
`NCCL_TESTS_SPLIT` serves as new way of computing the color for splitting communicators.
Will be overrided by `NCCL_TESTS_SPLIT_MASK`.
Examples:
NCCL_TESTS_SPLIT_MASK="0x7" # color = rank & 0x7. What we do today to run on a DGX with one GPU per node.
NCCL_TESTS_SPLIT="AND 0x7" # color = rank & 0x7. New way to run on one GPU per node on a DGX, equivalent to NCCL_TESTS_SPLIT_MASK=0x7
NCCL_TESTS_SPLIT="MOD 72" # color = rank % 72. One GPU per NVLink domain on an NVL72 system.
NCCL_TESTS_SPLIT="DIV 72" # color = rank / 72. Intra NVLink domain on NVL72.
You can also use: "%" "&" "|" "/" for short.
Extra spaces in the middle will be automatically ignored.
Not case sensitive.
The followings are all equivalent:
NCCL_TESTS_SPLIT="%0x7"
NCCL_TESTS_SPLIT="%0b111"
NCCL_TESTS_SPLIT="AND 7"
NCCL_TESTS_SPLIT="and 0x7"
2025-02-04 15:18:09 -08:00
cb6a46fdd6
Update CUDA gencodes
...
Add support for Blackwell sm100 and sm120 from CUDA 12.8
Add support for Hopper sm90 from CUDA 12.0
2025-01-25 17:32:16 -08:00
29f4114f02
Fixes to all tests that divide buffers by nranks so that they trim buffer sizes to be multiples of 16 bytes.
...
This ensures non-pow2 ranks have buffer addresses aligned suitably for performance.
2024-12-18 11:20:28 -08:00
34d6d53910
Future-proof ncclstringtotype
...
Ensure that ncclstringtotype iterates only over data types known to
nccl-tests (as indicated by test_typenum), not over a potentially larger
set of all NCCL types.
2024-10-24 09:21:37 -07:00
9d26b8422b
Merge pull request #226 from netgroup/master
...
improve parsing of stepbytes (increment size) argument
2024-07-30 14:58:54 -07:00
d2d40cc824
Added -N,--run_cycles option
2024-07-25 22:00:23 -07:00
746549b28d
improve parsing of stepbytes (increment size) argument
2024-06-14 11:28:55 +02:00
d028efcf35
Change ncclCommRegister size to maxBytes in serial comm init
2024-06-06 06:54:48 -07:00
a1efb427e7
Add -R option to register user buffers
2024-06-03 01:04:58 -07:00
c6afef0b6f
Added missing MPI_Comm_free() call before MPI_Finalize()
2024-02-05 08:53:54 -08:00
1292b25553
Added an MPI_Barrier() call after MPI_Bcast() for HCOLL issue
2023-10-12 16:53:32 -07:00
6c46206a47
Make the -c option be a datacheck iteration count parameter
...
Default is 1
2023-09-13 14:03:38 -07:00
7ccda3c97b
fix handling of variable NVCC. Permit overriding the variable using environment variables
2023-03-25 16:56:16 -03:00
0b4c4cb99f
Add boot_id to the hostname hash due to collisions on Azure
...
Fixes #60
2022-12-12 01:16:46 -08:00
0aeba157db
Use DJB2a hash algorithm in getHostHash()
2022-12-12 01:16:38 -08:00
24fcf64ed1
Call cudaFreeHost() on wrongPerGpu not cudaFree()
2022-11-22 11:18:37 -08:00
3bd2bd292b
Add fflush(stdout) before perf output
2022-11-22 11:16:47 -08:00
365b92a1ea
Fix build on RHEL7 with GCC 4.8
...
Add -std=c++11 to CXXFLAGS.
Fixes #116 .
2022-10-12 01:24:14 -07:00
d313d20a26
Update NCCL tests
2022-09-23 01:13:29 -07:00
749573f2d6
Fix preprocessor version check for ncclGetLastError()
...
ncclGetLastError() was added in NCCL 2.13.0
2022-09-07 16:10:41 -07:00
afa4c56b6a
Fix an issue with the last commit when data checking is disabled
2022-09-07 11:23:49 -07:00
a0a14911ee
Display N/A for error count in AlltoAll in-place test
...
AlltoAll does not support in-place buffers
2022-09-06 13:17:15 -07:00
51af5572bf
Resync with NCCL 2.13
...
* Added "verifiable", a suite of kernels for generating and verifying reduction
input and output arrays in a bit-precise way.
* Data corruption errors now reported in number of wrong elements instead of max
deviation.
* Use ncclGetLastError.
* Don't run hypercube on non-powers of 2 ranks.
* Fix to hypercube data verification.
* Use "thread local" as the defaut CUDA capture mode.
* Replaced pthread_yield -> sched_yield()
* Bugfix to the cpu-side barrier/allreduce implementations.
2022-08-22 17:51:06 -07:00
de3ddbe261
Add option to statically link cudart
...
Build with CUDARTLIB=cudart_static to remove dynamic linkage
Also removed unused curand and nvToolsExt dependencies
BUG 95
2021-11-10 10:02:41 -08:00
7130fa6096
Add MPI_IBM build option
2021-10-25 16:30:57 -07:00
f773748b46
Resync with NCCL 2.11
...
New operator: mulsum
New test: gather
2021-09-17 09:02:45 -07:00
1f8f541686
Add CUDA graph support only for CUDA 11.3 and later builds
...
Fixes #90
2021-07-13 10:47:47 -07:00
b9f90d12a9
Removed MPI_SUPPORT conditional compilation of average flag
2021-07-12 11:43:57 -07:00
547e119d35
Fix issues with MPI_Allreduce and multi-threaded tests
2021-07-08 16:42:40 -07:00
f476f4a17a
Merge branch 'bfloat16'
2021-07-06 10:20:32 -07:00
1dfc76eccc
Added new option to report average iteration time
2021-06-30 19:36:07 -07:00
1ae8cdc315
Resync with changes in gitilab-master code
2021-06-30 13:16:04 -07:00
9dae3d3a37
Added new tests: scatter, sendrecv, hypercube
2021-06-28 16:49:10 -07:00
e55ad3796d
Added support for CUDA graph capture/replay (-G)
2021-06-28 14:19:45 -07:00
526eacadf7
Fixed formatting for bfloat16 support
2021-06-28 10:12:34 -07:00
cde7e769c1
Add support for ncclAvg operation
2021-06-28 09:41:58 -07:00
c4de829d91
Cleanup argument error handling and messages
...
Add error checking for minbytes and maxbytes arguments
Also accept lowercase literals when parsing size arguments and print errors and usage on stderr.
2021-06-04 21:47:40 +00:00
e37545e491
Add support for new datatype: bfloat16
2021-03-15 17:13:35 -07:00
7677f3f608
Do not allocate memory for expected buffer if checking disabled
...
This allows the tests to be run with larger buffers
2021-01-20 17:08:40 -08:00
ae1ce98e69
Add boot_id to the hostname hash due to collisions on Azure
...
Fixes #60
2021-01-04 11:38:45 -08:00
da67a81c8e
Use DJB2a hash algorithm in getHostHash()
2020-12-18 10:12:54 -08:00
afdaf59b3b
Fix typo in src/Makefile
2020-06-24 14:39:22 -07:00
David Addison
Junyu Ma
John Bachan
Kamil Iskra
David Addison
Stefano Salsano
Kaiming Ouyang
Giuseppe Congiu
alan.souza
Jithin Jose
Sylvain Jeaugey
Greg Inozemtsev
Luke Yeager