AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
257 Commits 3 Branches 0 Tags
e3b9d785cc88a64bcbf9ebf941cce6d7fcb1f67d
Commit Graph

109 Commits

Author SHA1 Message Date
Grant Pinkert 3f962f5d58 Fix message size logging (#115) 
Previously, the logger was logging the number of expected bytes a node was to recieve.
This differs from the stdout logging, where the reported message size is the total size of a message.

Signed-off-by: Grant Pinkert <gpinkert@amd.com>

[ROCm/rccl-tests commit: f611dbd49a]
 2025-04-25 11:05:21 -05:00
nileshnegi 8d887aad0d Merge remote-tracking branch 'nccl-tests/master' into develop 
[ROCm/rccl-tests commit: 5625599dda]
 2025-04-21 19:46:10 -05:00
David Addison d516392fac Add PCI domain and device ID for GPU device BDF display 
[ROCm/rccl-tests commit: b4300cc79d]
 2025-02-28 13:25:51 -08:00
Junyu Ma e2a9cbb362 Perftests: Introduce NCCL_TESTS_SPLIT env 
`NCCL_TESTS_SPLIT` serves as new way of computing the color for splitting communicators.

Will be overrided by `NCCL_TESTS_SPLIT_MASK`.

Examples:

NCCL_TESTS_SPLIT_MASK="0x7" # color = rank & 0x7. What we do today to run on a DGX with one GPU per node.
NCCL_TESTS_SPLIT="AND 0x7"  # color = rank & 0x7. New way to run on one GPU per node on a DGX, equivalent to NCCL_TESTS_SPLIT_MASK=0x7
NCCL_TESTS_SPLIT="MOD 72"   # color = rank % 72.  One GPU per NVLink domain on an NVL72 system.
NCCL_TESTS_SPLIT="DIV 72"   # color = rank / 72.  Intra NVLink domain on NVL72.

You can also use: "%" "&" "|" "/" for short.
Extra spaces in the middle will be automatically ignored.
Not case sensitive.

The followings are all equivalent:

NCCL_TESTS_SPLIT="%0x7"
NCCL_TESTS_SPLIT="%0b111"
NCCL_TESTS_SPLIT="AND 7"
NCCL_TESTS_SPLIT="and 0x7"


[ROCm/rccl-tests commit: a89cf07fe8]
 2025-02-04 15:18:09 -08:00
David Sidler 93977b8866 Add option to output results to a file (#93) 
* Use find_package for MPI
* Add functionality to output results to file
* fix compilation
* report num gpus
* Revert "Use find_package for MPI"
This reverts commit c8fa253724ef4d0beac0d9c72f968062fbc6908e.
* Change inplace key
* remove dependency on json library
* Print "ranks, ranksPerNode, gpusPerRank"
* Add "nodes" field
---------

Co-authored-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl-tests commit: 959cc19920]
 2025-01-13 17:28:29 -06:00
saurabhAMD a2a3515e33 Updating to use hipDeviceMallocUncached (#95) 
Use hipDeviceMallocUncached instead of hipDeviceMallocFinegrained on newer ROCm versions.

[ROCm/rccl-tests commit: fc9917e0da]
 2025-01-11 23:25:24 -06:00
Mustafa Abduljabbar 7fbad10633 Memset to fix inflated performance when GPU is reset (#94) 
* Memset to fix inflated performance when GPU is reset
* use hipMemset for both memsets

[ROCm/rccl-tests commit: f2a48983ae]
 2025-01-11 23:25:24 -06:00
Tim 549f58f2a1 hot fixing ncclMemFree for mscclpp (#100) 
[ROCm/rccl-tests commit: f7a5df7fc4]
 2025-01-09 12:03:52 -05:00
John Bachan 69b9a05e71 Fixes to all tests that divide buffers by nranks so that they trim buffer sizes to be multiples of 16 bytes. 
This ensures non-pow2 ranks have buffer addresses aligned suitably for performance.


[ROCm/rccl-tests commit: 29f4114f02]
 2024-12-18 11:20:28 -08:00
AtlantaPepsi ccf92dcea2 Merge -R option for memory allocation 
Signed-off-by: AtlantaPepsi <timhu102@amd.com>


[ROCm/rccl-tests commit: afd5ca10ae]
 2024-07-31 14:57:20 +00:00
David Addison 58349a42b2 Merge pull request #226 from netgroup/master 
improve parsing of stepbytes (increment size) argument

[ROCm/rccl-tests commit: 9d26b8422b]
 2024-07-30 14:58:54 -07:00
David Addison cf3ffb2f5f Added -N,--run_cycles option 
[ROCm/rccl-tests commit: d2d40cc824]
 2024-07-25 22:00:23 -07:00
Rahul Vaidya 77524bffd0 Fix --root all issue. (#83) 
Signed-off-by: rahulvaidya20 <ravaidya@amd.com>

[ROCm/rccl-tests commit: c5cae38bb8]
 2024-06-14 11:46:08 -05:00
Stefano Salsano a0e06f2133 improve parsing of stepbytes (increment size) argument 
[ROCm/rccl-tests commit: 746549b28d]
 2024-06-14 11:28:55 +02:00
Kaiming Ouyang 1922bd71cb Change ncclCommRegister size to maxBytes in serial comm init 
[ROCm/rccl-tests commit: d028efcf35]
 2024-06-06 06:54:48 -07:00
saurabhAMD 351a047627 Rotating tensor -R (default:off) 
[ROCm/rccl-tests commit: 36a2c372ac]
 2024-06-04 11:35:39 -05:00
Giuseppe Congiu 411a7912f8 Add -R option to register user buffers 
[ROCm/rccl-tests commit: a1efb427e7]
 2024-06-03 01:04:58 -07:00
saurabhAMD 5700751b7e updating cache flush on functionality 
[ROCm/rccl-tests commit: 74c4177f58]
 2024-05-10 08:46:13 -07:00
saurabhAMD ce8e61cc3b Enable cache flush after every -F iteration. Default : 0 (No cache flush) 
[ROCm/rccl-tests commit: 699478dadf]
 2024-05-07 11:32:30 -05:00
saurabhAMD e1a3c5a6dc Cache flush 
[ROCm/rccl-tests commit: 3c0728e8eb]
 2024-05-07 11:09:32 -05:00
Wenkai Du baf9242e07 Fix incorrect device ordinal with limited device visibility (#74) 
[ROCm/rccl-tests commit: 16dfeaf89b]
 2024-05-02 11:14:57 -07:00
corey-derochie-amd fe151f517b Fixed spelling 
[ROCm/rccl-tests commit: f74c04b686]
 2024-05-02 09:18:25 -06:00
Corey Derochie 85bdda3812 Wrapped the warmup iters in captures when doing graph mode to do a proper warmup. 
[ROCm/rccl-tests commit: 0c762d210c]
 2024-05-01 20:41:12 -05:00
mberenjk 9cfb0745c0 replacing rccl_bfloat16 with hip_bfloat16 (#70) 
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>

[ROCm/rccl-tests commit: eb65dadfc5]
 2024-04-23 17:00:20 -05:00
mberenjk ca4ba933a3 adding git version to rccl-tests (#69) 
Co-authored-by: mberenjk <mberenjk@amd.com>

[ROCm/rccl-tests commit: 3f7f7859bf]
 2024-03-28 14:03:59 -05:00
akolliasAMD f8c62f64c3 Revert "adding git version to rccl-test (#66)" 
This reverts commit 82c71f1838.


[ROCm/rccl-tests commit: 91609be0ef]
 2024-03-22 10:21:37 -06:00
mberenjk 82c71f1838 adding git version to rccl-test (#66) 
* adding git version to rccl-test

---------

Co-authored-by: mberenjk <mberenjk@banff-cyxtera-s74-2.ctr.dcgpu>

[ROCm/rccl-tests commit: a31679775c]
 2024-03-20 10:04:12 -05:00
Andy li aaf1e27af2 update the fp8 header file name (#65) 
* update the fp8 header name

[ROCm/rccl-tests commit: e447c17382]
 2024-03-08 10:02:40 -08:00
Andy li c128f0422d Enable fp8 support (#63) 
* initial checkin

* rename the fp8 datatype name

* update based on cr comments

* resolve the build issue

* resolve fp8 campability issue

* fix minior bug and catch up to reflex latest develop branch change

* add fp8 + operatior support

* update fp8 header file

* resolve merge issue from develop branch

[ROCm/rccl-tests commit: 21e59fb283]
 2024-03-07 16:54:41 -08:00
Bertan Dogancay 882a96f5cb Add hipify steps prior to build (#62) 
* Add hipify steps prior to build

[ROCm/rccl-tests commit: 88cf7dbf45]
 2024-03-05 09:47:18 -07:00
Wenkai Du b49f6da1ec Merge remote-tracking branch 'nccl-tests/master' into HEAD 
[ROCm/rccl-tests commit: 621dde544d]
 2024-03-01 18:34:44 +00:00
Wenkai Du ff97af6529 Fix typo in rank assignment (#59) 
[ROCm/rccl-tests commit: 7715a0cf1f]
 2024-02-15 12:04:38 -08:00
David Addison 5d52f0285c Added missing MPI_Comm_free() call before MPI_Finalize() 
[ROCm/rccl-tests commit: c6afef0b6f]
 2024-02-05 08:53:54 -08:00
Nusrat Islam 26b1b0b822 Add option to disable out-of-place 
[ROCm/rccl-tests commit: a2bec5d2f6]
 2024-01-04 16:43:50 -06:00
Wenkai Du ccad358bc9 Warm up both out-of-place and in-place collectives (#51) 
[ROCm/rccl-tests commit: 5ee7a08994]
 2023-10-16 12:13:50 -07:00
David Addison 459b52158f Added an MPI_Barrier() call after MPI_Bcast() for HCOLL issue 
[ROCm/rccl-tests commit: 1292b25553]
 2023-10-12 16:53:32 -07:00
David Addison e1f13fac90 Make the -c option be a datacheck iteration count parameter 
Default is 1


[ROCm/rccl-tests commit: 6c46206a47]
 2023-09-13 14:03:38 -07:00
Wenkai Du a87b64d7c0 Fix merge error 
[ROCm/rccl-tests commit: 652a24d38d]
 2023-06-14 20:26:33 +00:00
Wenkai Du a2e8a44adb Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl-tests commit: bb0f15d407]
 2023-06-14 08:21:02 -07:00
Wenkai Du 7820b9f63f Merge remote-tracking branch 'origin/master' into develop 
[ROCm/rccl-tests commit: 469225bcaf]
 2023-06-14 08:01:50 -07:00
Pedram Alizadeh aa5f75aa9c fixing the error message for mpirun when number of requested GPUs exceeds the limits (#37) 
[ROCm/rccl-tests commit: d16d1fb16b]
 2023-04-27 14:06:17 -04:00
Pedram Alizadeh b7054b500a Revert "fixing the error message for mpirun when number of requested GPUs exceeds the limits (#33)" (#36) 
This reverts commit 39f83d5bb7.

[ROCm/rccl-tests commit: e856fa720f]
 2023-04-25 13:44:43 -04:00
Pedram Alizadeh 39f83d5bb7 fixing the error message for mpirun when number of requested GPUs exceeds the limits (#33) 
[ROCm/rccl-tests commit: e146460810]
 2023-04-03 11:37:13 -04:00
David Addison 6313530fcc Call cudaFreeHost() on wrongPerGpu not cudaFree() 
[ROCm/rccl-tests commit: 24fcf64ed1]
 2022-11-22 11:18:37 -08:00
David Addison 04b5c40b1c Add fflush(stdout) before perf output 
[ROCm/rccl-tests commit: 3bd2bd292b]
 2022-11-22 11:16:47 -08:00
akolliasAMD 1db5cf9986 added std::max to avoid buffer overflow in printing (#25) 
[ROCm/rccl-tests commit: 9d3a53dfa3]
 2022-11-01 11:34:55 -06:00
Edgar Gabriel 9b06fe6fd5 fix a messing endif statement 
error introduced with the web merger-resolution tool :-(


[ROCm/rccl-tests commit: 8a754f15ad]
 2022-10-25 16:31:57 +00:00
Edgar Gabriel d76b2db507 Merge branch 'develop' into topic/v2.13.4-sync 
[ROCm/rccl-tests commit: 4d7cd871c1]
 2022-10-21 17:12:45 -05:00
Wenkai Du 036145c75f Allow more precise measurements of single operation (#20) 
[ROCm/rccl-tests commit: 9a89c300b6]
 2022-10-21 22:07:41 +00:00
Edgar Gabriel fab39367ae make rccl-test compile again. 
all files compile now.
mpi tests also pass


[ROCm/rccl-tests commit: 641e93e99c]
 2022-10-21 22:07:33 +00:00