f611dbd49a
Fix message size logging ( #115 )
...
Previously, the logger was logging the number of expected bytes a node was to recieve.
This differs from the stdout logging, where the reported message size is the total size of a message.
Signed-off-by: Grant Pinkert <gpinkert@amd.com >
2025-04-25 11:05:21 -05:00
5625599dda
Merge remote-tracking branch 'nccl-tests/master' into develop
2025-04-21 19:46:10 -05:00
b4300cc79d
Add PCI domain and device ID for GPU device BDF display
2025-02-28 13:25:51 -08:00
a89cf07fe8
Perftests: Introduce NCCL_TESTS_SPLIT env
...
`NCCL_TESTS_SPLIT` serves as new way of computing the color for splitting communicators.
Will be overrided by `NCCL_TESTS_SPLIT_MASK`.
Examples:
NCCL_TESTS_SPLIT_MASK="0x7" # color = rank & 0x7. What we do today to run on a DGX with one GPU per node.
NCCL_TESTS_SPLIT="AND 0x7" # color = rank & 0x7. New way to run on one GPU per node on a DGX, equivalent to NCCL_TESTS_SPLIT_MASK=0x7
NCCL_TESTS_SPLIT="MOD 72" # color = rank % 72. One GPU per NVLink domain on an NVL72 system.
NCCL_TESTS_SPLIT="DIV 72" # color = rank / 72. Intra NVLink domain on NVL72.
You can also use: "%" "&" "|" "/" for short.
Extra spaces in the middle will be automatically ignored.
Not case sensitive.
The followings are all equivalent:
NCCL_TESTS_SPLIT="%0x7"
NCCL_TESTS_SPLIT="%0b111"
NCCL_TESTS_SPLIT="AND 7"
NCCL_TESTS_SPLIT="and 0x7"
2025-02-04 15:18:09 -08:00
959cc19920
Add option to output results to a file ( #93 )
...
* Use find_package for MPI
* Add functionality to output results to file
* fix compilation
* report num gpus
* Revert "Use find_package for MPI"
This reverts commit c8fa253724ef4d0beac0d9c72f968062fbc6908e.
* Change inplace key
* remove dependency on json library
* Print "ranks, ranksPerNode, gpusPerRank"
* Add "nodes" field
---------
Co-authored-by: nileshnegi <Nilesh.Negi@amd.com >
2025-01-13 17:28:29 -06:00
fc9917e0da
Updating to use hipDeviceMallocUncached ( #95 )
...
Use hipDeviceMallocUncached instead of hipDeviceMallocFinegrained on newer ROCm versions.
2025-01-11 23:25:24 -06:00
f2a48983ae
Memset to fix inflated performance when GPU is reset ( #94 )
...
* Memset to fix inflated performance when GPU is reset
* use hipMemset for both memsets
2025-01-11 23:25:24 -06:00
f7a5df7fc4
hot fixing ncclMemFree for mscclpp ( #100 )
2025-01-09 12:03:52 -05:00
29f4114f02
Fixes to all tests that divide buffers by nranks so that they trim buffer sizes to be multiples of 16 bytes.
...
This ensures non-pow2 ranks have buffer addresses aligned suitably for performance.
2024-12-18 11:20:28 -08:00
afd5ca10ae
Merge -R option for memory allocation
...
Signed-off-by: AtlantaPepsi <timhu102@amd.com >
2024-07-31 14:57:20 +00:00
9d26b8422b
Merge pull request #226 from netgroup/master
...
improve parsing of stepbytes (increment size) argument
2024-07-30 14:58:54 -07:00
d2d40cc824
Added -N,--run_cycles option
2024-07-25 22:00:23 -07:00
c5cae38bb8
Fix --root all issue. ( #83 )
...
Signed-off-by: rahulvaidya20 <ravaidya@amd.com >
2024-06-14 11:46:08 -05:00
746549b28d
improve parsing of stepbytes (increment size) argument
2024-06-14 11:28:55 +02:00
d028efcf35
Change ncclCommRegister size to maxBytes in serial comm init
2024-06-06 06:54:48 -07:00
36a2c372ac
Rotating tensor -R (default:off)
2024-06-04 11:35:39 -05:00
a1efb427e7
Add -R option to register user buffers
2024-06-03 01:04:58 -07:00
74c4177f58
updating cache flush on functionality
2024-05-10 08:46:13 -07:00
699478dadf
Enable cache flush after every -F iteration. Default : 0 (No cache flush)
2024-05-07 11:32:30 -05:00
3c0728e8eb
Cache flush
2024-05-07 11:09:32 -05:00
16dfeaf89b
Fix incorrect device ordinal with limited device visibility ( #74 )
2024-05-02 11:14:57 -07:00
f74c04b686
Fixed spelling
2024-05-02 09:18:25 -06:00
0c762d210c
Wrapped the warmup iters in captures when doing graph mode to do a proper warmup.
2024-05-01 20:41:12 -05:00
eb65dadfc5
replacing rccl_bfloat16 with hip_bfloat16 ( #70 )
...
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com >
2024-04-23 17:00:20 -05:00
3f7f7859bf
adding git version to rccl-tests ( #69 )
...
Co-authored-by: mberenjk <mberenjk@amd.com >
2024-03-28 14:03:59 -05:00
91609be0ef
Revert "adding git version to rccl-test ( #66 )"
...
This reverts commit a31679775c
2024-03-22 10:21:37 -06:00
a31679775c
adding git version to rccl-test ( #66 )
...
* adding git version to rccl-test
---------
Co-authored-by: mberenjk <mberenjk@banff-cyxtera-s74-2.ctr.dcgpu >
2024-03-20 10:04:12 -05:00
e447c17382
update the fp8 header file name ( #65 )
...
* update the fp8 header name
2024-03-08 10:02:40 -08:00
21e59fb283
Enable fp8 support ( #63 )
...
* initial checkin
* rename the fp8 datatype name
* update based on cr comments
* resolve the build issue
* resolve fp8 campability issue
* fix minior bug and catch up to reflex latest develop branch change
* add fp8 + operatior support
* update fp8 header file
* resolve merge issue from develop branch
2024-03-07 16:54:41 -08:00
88cf7dbf45
Add hipify steps prior to build ( #62 )
...
* Add hipify steps prior to build
2024-03-05 09:47:18 -07:00
621dde544d
Merge remote-tracking branch 'nccl-tests/master' into HEAD
2024-03-01 18:34:44 +00:00
7715a0cf1f
Fix typo in rank assignment ( #59 )
2024-02-15 12:04:38 -08:00
c6afef0b6f
Added missing MPI_Comm_free() call before MPI_Finalize()
2024-02-05 08:53:54 -08:00
a2bec5d2f6
Add option to disable out-of-place
2024-01-04 16:43:50 -06:00
5ee7a08994
Warm up both out-of-place and in-place collectives ( #51 )
2023-10-16 12:13:50 -07:00
1292b25553
Added an MPI_Barrier() call after MPI_Bcast() for HCOLL issue
2023-10-12 16:53:32 -07:00
6c46206a47
Make the -c option be a datacheck iteration count parameter
...
Default is 1
2023-09-13 14:03:38 -07:00
652a24d38d
Fix merge error
2023-06-14 20:26:33 +00:00
bb0f15d407
Merge remote-tracking branch 'nccl/master' into develop
2023-06-14 08:21:02 -07:00
469225bcaf
Merge remote-tracking branch 'origin/master' into develop
2023-06-14 08:01:50 -07:00
d16d1fb16b
fixing the error message for mpirun when number of requested GPUs exceeds the limits ( #37 )
2023-04-27 14:06:17 -04:00
e856fa720f
Revert "fixing the error message for mpirun when number of requested GPUs exceeds the limits ( #33 )" ( #36 )
...
This reverts commit e146460810
2023-04-25 13:44:43 -04:00
e146460810
fixing the error message for mpirun when number of requested GPUs exceeds the limits ( #33 )
2023-04-03 11:37:13 -04:00
24fcf64ed1
Call cudaFreeHost() on wrongPerGpu not cudaFree()
2022-11-22 11:18:37 -08:00
3bd2bd292b
Add fflush(stdout) before perf output
2022-11-22 11:16:47 -08:00
9d3a53dfa3
added std::max to avoid buffer overflow in printing ( #25 )
2022-11-01 11:34:55 -06:00
8a754f15ad
fix a messing endif statement
...
error introduced with the web merger-resolution tool :-(
2022-10-25 16:31:57 +00:00
4d7cd871c1
Merge branch 'develop' into topic/v2.13.4-sync
2022-10-21 17:12:45 -05:00
9a89c300b6
Allow more precise measurements of single operation ( #20 )
2022-10-21 22:07:41 +00:00
641e93e99c
make rccl-test compile again.
...
all files compile now.
mpi tests also pass
2022-10-21 22:07:33 +00:00
Grant Pinkert
nileshnegi
David Addison
Junyu Ma
David Sidler
saurabhAMD
Mustafa Abduljabbar
Tim
John Bachan
AtlantaPepsi
David Addison
Rahul Vaidya
Stefano Salsano
Kaiming Ouyang
saurabhAMD
Giuseppe Congiu
Wenkai Du
corey-derochie-amd
Corey Derochie
mberenjk
akolliasAMD
Andy li
Bertan Dogancay
Wenkai Du
Nusrat Islam
Pedram Alizadeh
Edgar Gabriel
Edgar Gabriel