e2a9cbb362
Perftests: Introduce NCCL_TESTS_SPLIT env
...
`NCCL_TESTS_SPLIT` serves as new way of computing the color for splitting communicators.
Will be overrided by `NCCL_TESTS_SPLIT_MASK`.
Examples:
NCCL_TESTS_SPLIT_MASK="0x7" # color = rank & 0x7. What we do today to run on a DGX with one GPU per node.
NCCL_TESTS_SPLIT="AND 0x7" # color = rank & 0x7. New way to run on one GPU per node on a DGX, equivalent to NCCL_TESTS_SPLIT_MASK=0x7
NCCL_TESTS_SPLIT="MOD 72" # color = rank % 72. One GPU per NVLink domain on an NVL72 system.
NCCL_TESTS_SPLIT="DIV 72" # color = rank / 72. Intra NVLink domain on NVL72.
You can also use: "%" "&" "|" "/" for short.
Extra spaces in the middle will be automatically ignored.
Not case sensitive.
The followings are all equivalent:
NCCL_TESTS_SPLIT="%0x7"
NCCL_TESTS_SPLIT="%0b111"
NCCL_TESTS_SPLIT="AND 7"
NCCL_TESTS_SPLIT="and 0x7"
[ROCm/rccl-tests commit: a89cf07fe8
2025-02-04 15:18:09 -08:00
6f2e0f8a21
Update CUDA gencodes
...
Add support for Blackwell sm100 and sm120 from CUDA 12.8
Add support for Hopper sm90 from CUDA 12.0
[ROCm/rccl-tests commit: cb6a46fdd6
2025-01-25 17:32:16 -08:00
69b9a05e71
Fixes to all tests that divide buffers by nranks so that they trim buffer sizes to be multiples of 16 bytes.
...
This ensures non-pow2 ranks have buffer addresses aligned suitably for performance.
[ROCm/rccl-tests commit: 29f4114f02
2024-12-18 11:20:28 -08:00
637acfba05
Merge pull request #259 from NVIDIA/fix-ncclstringtotype
...
Future-proof ncclstringtotype
[ROCm/rccl-tests commit: 8dfeab9eb9
2024-10-24 10:28:02 -07:00
0a2f08311b
Future-proof ncclstringtotype
...
Ensure that ncclstringtotype iterates only over data types known to
nccl-tests (as indicated by test_typenum), not over a potentially larger
set of all NCCL types.
[ROCm/rccl-tests commit: 34d6d53910
2024-10-24 09:21:37 -07:00
58349a42b2
Merge pull request #226 from netgroup/master
...
improve parsing of stepbytes (increment size) argument
[ROCm/rccl-tests commit: 9d26b8422b
2024-07-30 14:58:54 -07:00
98b958afbd
Added some missing command line options to README.md
...
Also updated single and multi-node examples.
[ROCm/rccl-tests commit: 0d86b5a6e7
2024-07-30 14:50:45 -07:00
cf3ffb2f5f
Added -N,--run_cycles option
...
[ROCm/rccl-tests commit: d2d40cc824
2024-07-25 22:00:23 -07:00
95419a2f47
Merge pull request #240 from OrenLeung/patch-1
...
doc: add all2all factor
[ROCm/rccl-tests commit: 3a3f790efd
2024-07-25 22:00:06 -07:00
5061074d09
doc: add all2all factor
...
[ROCm/rccl-tests commit: c6eb15875f
2024-07-24 22:55:00 -04:00
a0e06f2133
improve parsing of stepbytes (increment size) argument
...
[ROCm/rccl-tests commit: 746549b28d
2024-06-14 11:28:55 +02:00
1922bd71cb
Change ncclCommRegister size to maxBytes in serial comm init
...
[ROCm/rccl-tests commit: d028efcf35
2024-06-06 06:54:48 -07:00
411a7912f8
Add -R option to register user buffers
...
[ROCm/rccl-tests commit: a1efb427e7
2024-06-03 01:04:58 -07:00
5d52f0285c
Added missing MPI_Comm_free() call before MPI_Finalize()
...
[ROCm/rccl-tests commit: c6afef0b6f
2024-02-05 08:53:54 -08:00
459b52158f
Added an MPI_Barrier() call after MPI_Bcast() for HCOLL issue
...
[ROCm/rccl-tests commit: 1292b25553
2023-10-12 16:53:32 -07:00
e1f13fac90
Make the -c option be a datacheck iteration count parameter
...
Default is 1
[ROCm/rccl-tests commit: 6c46206a47
2023-09-13 14:03:38 -07:00
876c86fcdb
Merge pull request #146 from yangxingwu/master
...
makefile: remove extra space
[ROCm/rccl-tests commit: 1a5f551ffd
2023-06-06 11:58:24 +02:00
cba8bfd093
makefile: remove extra space
...
[ROCm/rccl-tests commit: 52ea1b2148
2023-06-06 09:47:50 +00:00
5244aae891
Merge pull request #135 from aavbsouza/fix_nvcc_variable_handling
...
fix handling of variable NVCC.
[ROCm/rccl-tests commit: e98ef24bc0
2023-03-27 11:14:10 +02:00
4fd5ceeba8
fix handling of variable NVCC. Permit overriding the variable using environment variables
...
[ROCm/rccl-tests commit: 7ccda3c97b
2023-03-25 16:56:16 -03:00
f5512c6a2e
Merge pull request #134 from flx42/patch-1
...
Update README.md to fix -i default increment value.
[ROCm/rccl-tests commit: e76e36e9a9
2023-03-23 09:53:15 -07:00
b3db782c3f
Update README.md
...
[ROCm/rccl-tests commit: 17d0a42d5a
2023-03-23 09:05:41 -07:00
b70cac2b33
Update README.md
...
Improve MPI example to avoid confusion of number of processes / total number of GPUs.
https://github.com/NVIDIA/nccl-tests/issues/54#issuecomment-1212023369
[ROCm/rccl-tests commit: 2cbb968101
2023-01-03 08:47:43 +01:00
129a1b4b78
Add boot_id to the hostname hash due to collisions on Azure
...
Fixes #60
[ROCm/rccl-tests commit: 0b4c4cb99f
2022-12-12 01:16:46 -08:00
5ba670d551
Use DJB2a hash algorithm in getHostHash()
...
[ROCm/rccl-tests commit: 0aeba157db
2022-12-12 01:16:38 -08:00
6313530fcc
Call cudaFreeHost() on wrongPerGpu not cudaFree()
...
[ROCm/rccl-tests commit: 24fcf64ed1
2022-11-22 11:18:37 -08:00
04b5c40b1c
Add fflush(stdout) before perf output
...
[ROCm/rccl-tests commit: 3bd2bd292b
2022-11-22 11:16:47 -08:00
c0e3f4d443
Fix build on RHEL7 with GCC 4.8
...
Add -std=c++11 to CXXFLAGS.
Fixes #116 .
[ROCm/rccl-tests commit: 365b92a1ea
2022-10-12 01:24:14 -07:00
fdaa88710b
Update NCCL tests
...
[ROCm/rccl-tests commit: d313d20a26
2022-09-23 01:13:29 -07:00
35ee4ec3eb
Fix preprocessor version check for ncclGetLastError()
...
ncclGetLastError() was added in NCCL 2.13.0
[ROCm/rccl-tests commit: 749573f2d6
2022-09-07 16:10:41 -07:00
a43863e1a7
Fix an issue with the last commit when data checking is disabled
...
[ROCm/rccl-tests commit: afa4c56b6a
2022-09-07 11:23:49 -07:00
59ed17798f
Display N/A for error count in AlltoAll in-place test
...
AlltoAll does not support in-place buffers
[ROCm/rccl-tests commit: a0a14911ee
2022-09-06 13:17:15 -07:00
70b6c0f5e5
Changed top-level Makefile behavior so that BUILDDIR is interpreted
...
as relative to top-level directory. This done is by abspath'ing it before
passing it to subdirectory Makefile's.
The old behavior had two cases: with and without BUILDDIR being set by
the user. With BUILDDIR not set, the build dir would be named "build"
in the top-level directory. If BUILDDIR was set, then the build dir
would be placed at "src/${BUILDDIR}".
The new behavior is simpler, if BUILDDIR is not set then it defaults
to "build", and the directory holding the final build is always at just
"${BUILDDIR}" in the top level.
[ROCm/rccl-tests commit: bc5f7cfb0a
2022-08-23 10:08:49 -07:00
b5d746b58e
Resync with NCCL 2.13
...
* Added "verifiable", a suite of kernels for generating and verifying reduction
input and output arrays in a bit-precise way.
* Data corruption errors now reported in number of wrong elements instead of max
deviation.
* Use ncclGetLastError.
* Don't run hypercube on non-powers of 2 ranks.
* Fix to hypercube data verification.
* Use "thread local" as the defaut CUDA capture mode.
* Replaced pthread_yield -> sched_yield()
* Bugfix to the cpu-side barrier/allreduce implementations.
[ROCm/rccl-tests commit: 51af5572bf
2022-08-22 17:51:06 -07:00
22ebb430a6
Merge pull request #96 from NVIDIA/nersc-linkage-fix
...
Add option to statically link cudart
[ROCm/rccl-tests commit: 8274cb47b6
2022-05-26 16:54:44 -07:00
dd8563b279
Add option to statically link cudart
...
Build with CUDARTLIB=cudart_static to remove dynamic linkage
Also removed unused curand and nvToolsExt dependencies
BUG 95
[ROCm/rccl-tests commit: de3ddbe261
2021-11-10 10:02:41 -08:00
ad9aac78df
Add MPI_IBM build option
...
[ROCm/rccl-tests commit: 7130fa6096
2021-10-25 16:30:57 -07:00
56ff821802
Resync with NCCL 2.11
...
New operator: mulsum
New test: gather
[ROCm/rccl-tests commit: f773748b46
2021-09-17 09:02:45 -07:00
f81f5baaed
Add CUDA graph support only for CUDA 11.3 and later builds
...
Fixes #90
[ROCm/rccl-tests commit: 1f8f541686
2021-07-13 10:47:47 -07:00
6719794fc8
Removed MPI_SUPPORT conditional compilation of average flag
...
[ROCm/rccl-tests commit: b9f90d12a9
2021-07-12 11:43:57 -07:00
d3061dc2a9
Fix issues with MPI_Allreduce and multi-threaded tests
...
[ROCm/rccl-tests commit: 547e119d35
2021-07-08 16:42:40 -07:00
ea6eec9e80
Updated with new command line arguments
...
[ROCm/rccl-tests commit: 11cff17a04
2021-07-06 16:27:45 -07:00
230983c84e
Merge branch 'bfloat16'
...
[ROCm/rccl-tests commit: f476f4a17a
2021-07-06 10:20:32 -07:00
a23cffe28a
Added new option to report average iteration time
...
[ROCm/rccl-tests commit: 1dfc76eccc
2021-06-30 19:36:07 -07:00
1044cd1f32
Resync with changes in gitilab-master code
...
[ROCm/rccl-tests commit: 1ae8cdc315
2021-06-30 13:16:04 -07:00
efaaf56199
Merge pull request #88 from nzmsv/master
...
Cleanup argument error handling and messages
[ROCm/rccl-tests commit: 44df0bf010
2021-06-30 12:35:47 -07:00
d30e35f150
Added new tests: scatter, sendrecv, hypercube
...
[ROCm/rccl-tests commit: 9dae3d3a37
2021-06-28 16:49:10 -07:00
e73e5a239b
Added support for CUDA graph capture/replay (-G)
...
[ROCm/rccl-tests commit: e55ad3796d
2021-06-28 14:19:45 -07:00
20b63cf465
Fixed formatting for bfloat16 support
...
[ROCm/rccl-tests commit: 526eacadf7
2021-06-28 10:12:34 -07:00
a41268e26e
Add support for ncclAvg operation
...
[ROCm/rccl-tests commit: cde7e769c1
2021-06-28 09:41:58 -07:00
Junyu Ma
David Addison
John Bachan
Sylvain Jeaugey
Kamil Iskra
David Addison
Oren
Stefano Salsano
Kaiming Ouyang
Giuseppe Congiu
yangxingwu
alan.souza
Felix Abecassis
Jithin Jose