rocm-systems

Files

T

Edgar 8953f5b5ca Introduce multi-rank support per device.

This is a single commit of the source code changes required to
introduce support for multiple ranks per device.
A new interface (ncclCommRankInitMulti) has to be used to make use of
this new feature.


[ROCm/rccl commit: 0336ffdf70]

2022-06-10 14:23:12 +00:00

Makefile

Introduce multi-rank support per device.

2022-06-10 14:23:12 +00:00

rccl-allreduce-mpi-multirank.cc

Introduce multi-rank support per device.

2022-06-10 14:23:12 +00:00

rccl-allreduce-multirank.cc

Introduce multi-rank support per device.

2022-06-10 14:23:12 +00:00

rccl-reducescatter-mpi-multirank.cc

Introduce multi-rank support per device.

2022-06-10 14:23:12 +00:00

rccl-reducescatter-multirank.cc

Introduce multi-rank support per device.

2022-06-10 14:23:12 +00:00

README.mpi

Introduce multi-rank support per device.

2022-06-10 14:23:12 +00:00

README.mpi

#!/bin/bash
export MPI_INSTALL_PATH=
export RCCL_INSTALL_PATH=
export ROCM_INSTALL_PATH=

$MPI_INSTALL_PATH/bin/mpiCC -o rccl-allreduce-mpi-multirank rccl-allreduce-mpi-multirank.cc -I$ROCM_INSTALL_PATH/include -I$RCCL_INSTALL_PATH/include -D__HIP_PLATFORM_AMD__ -L$ROCM_INSTALL_PATH/lib -lamdhip64 -L$RCCL_INSTALL_PATH/lib -lrccl

$MPI_INSTALL_PATH/bin/mpiCC -o rccl-reducescatter-mpi-multirank rccl-reducescatter-mpi-multirank.cc -I$ROCM_INSTALL_PATH/include -I$RCCL_INSTALL_PATH/include -D__HIP_PLATFORM_AMD__ -L$ROCM_INSTALL_PATH/lib -lamdhip64 -L$RCCL_INSTALL_PATH/lib -lrccl


export GPU_MAX_HW_QUEUES=16
$MPI_INSTALL_PATH/bin/mpirun --mca pml ucx -np 4 ./rccl-allreduce-mpi-multirank  0 0 2
$MPI_INSTALL_PATH/bin/mpirun --mca pml ucx -np 4 ./rccl-reducescatter-mpi-multirank  0 0 2