Add host API for alltoallmem_on_stream collective operation (#333)

* Add host-side rocshmem_alltoallmem_on_stream function

Function signature:
  rocshmem_alltoallmem_on_stream(rocshmem_team_t team, void *dest,
                                 const void *source, size_t size,
                                 hipStream_t stream)

- The function launches rocshmem_alltoallmem_kernel which calls
device-side alltoall<char> workgroup collective through default context.
- Uses dynamic block size determination via occupancy API.
- Implemented for all backends.

* Fix incorrect sync buffer size allocation for alltoall in GDA and IPC backends

When allocating memory for alltoall_pSync_pool in setup_teams() and
teams_init() functions, the code incorrectly used ROCSHMEM_BCAST_SYNC_SIZE
instead of ROCSHMEM_ALLTOALL_SYNC_SIZE.

* Add functional test for team_alltoallmem_on_stream

This commit adds a new functional test to verify the correctness of
the host-side rocshmem_team_alltoallmem_on_stream API.

* Add documentation for rocshmem_alltoallmem_on_stream

This commit adds API documentation for the host-side
rocshmem_alltoallmem_on_stream function in the collective routines
section. The documentation includes:

[ROCm/rocshmem commit: 5577feb70d]
This commit is contained in:
Anatolii Rozanov
2025-12-03 14:40:24 +01:00
committed by GitHub
parent 0f32739b52
commit 4b04b540bf
24 changed files with 479 additions and 6 deletions
+30
View File
@@ -333,6 +333,36 @@ __host__ void HostInterface::barrier_all_on_stream(hipStream_t stream) {
rocshmem_barrier_all_kernel<<<1, 1, 0, stream>>>();
}
__host__ void HostInterface::alltoallmem_on_stream(rocshmem_team_t team,
void *dest,
const void *source,
size_t size,
hipStream_t stream) {
// launch kernel to do alltoall with given stream, if none, use default stream
if (stream == nullptr) {
stream = hipStreamDefault;
}
// Use dynamic block size determination:
// - Query optimal block size using occupancy API
// - Limit block size to size (number of bytes) to avoid over-subscription
// - Always use 1 block (single workgroup collective)
int optimal_block_size = 0;
int grid_size = 0;
CHECK_HIP(hipOccupancyMaxPotentialBlockSize(&grid_size, &optimal_block_size,
rocshmem_alltoallmem_kernel, 0,
0));
// Limit block size to size (bytes) to avoid over-subscription
int num_threads_per_block = (optimal_block_size > static_cast<int>(size))
? static_cast<int>(size)
: optimal_block_size;
dim3 gridSize(1);
dim3 blockSize(num_threads_per_block);
rocshmem_alltoallmem_kernel<<<gridSize, blockSize, 0, stream>>>(team, dest,
source, size);
}
__host__ void HostInterface::barrier_for_sync() {
if (host_comm_world_ != MPI_COMM_NULL) {
+4
View File
@@ -196,6 +196,10 @@ class HostInterface {
__host__ void barrier_all_on_stream(hipStream_t stream);
__host__ void alltoallmem_on_stream(rocshmem_team_t team, void *dest,
const void *source, size_t size,
hipStream_t stream);
__host__ void barrier_for_sync();
__host__ void sync_all(WindowInfo* window_info);