update README documentation for RO (#63)
* README: update documentation for RO support
update the README and the install_dependencies script to match the
requirements of the RO conduit.
* add CODEOWNERS file
[ROCm/rocshmem commit: 4e48c9748e]
This commit is contained in:
+1
@@ -0,0 +1 @@
|
|||||||
|
@avinashkethineedi @Yiltan @BKP @abouteiller @edgargabriel @gaoikawa
|
||||||
@@ -10,9 +10,6 @@ rocSHMEM uses a single symmetric heap (SHEAP) that is allocated on GPU memories.
|
|||||||
There are currently two backends for rocSHMEM;
|
There are currently two backends for rocSHMEM;
|
||||||
IPC and Reverse Offload (RO).
|
IPC and Reverse Offload (RO).
|
||||||
The backends primarily differ in their implementations of intra-kernel networking.
|
The backends primarily differ in their implementations of intra-kernel networking.
|
||||||
Currently, only the IPC backend is supported.
|
|
||||||
The RO backend is provided as-is with
|
|
||||||
no guarantees of support from AMD or AMD Research.
|
|
||||||
|
|
||||||
The IPC backend implements communication primitives using load/store operations issued from the GPU.
|
The IPC backend implements communication primitives using load/store operations issued from the GPU.
|
||||||
|
|
||||||
@@ -21,6 +18,8 @@ to the host-side runtime, which calls into a traditional MPI or OpenSHMEM
|
|||||||
implementation. This forwarding of requests is transparent to the
|
implementation. This forwarding of requests is transparent to the
|
||||||
programmer, who only sees the GPU-side interface.
|
programmer, who only sees the GPU-side interface.
|
||||||
|
|
||||||
|
The RO backend is provided as-is with limited support from AMD or AMD Research.
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
rocSHMEM base requirements:
|
rocSHMEM base requirements:
|
||||||
@@ -40,7 +39,7 @@ OpenCL.
|
|||||||
rocSHMEM uses the CMake build system. The CMakeLists file contains
|
rocSHMEM uses the CMake build system. The CMakeLists file contains
|
||||||
additional details about library options.
|
additional details about library options.
|
||||||
|
|
||||||
To create an out-of-source build for the IPC backend:
|
To create an out-of-source build for the IPC backend for single-node use-cases:
|
||||||
|
|
||||||
```
|
```
|
||||||
mkdir build
|
mkdir build
|
||||||
@@ -48,6 +47,14 @@ cd build
|
|||||||
../scripts/build_configs/ipc_single
|
../scripts/build_configs/ipc_single
|
||||||
```
|
```
|
||||||
|
|
||||||
|
To create an out-of-source build for the RO backend for multi-node use-cases that can also utilize the IPC mechanisms for certain intra-node operations:
|
||||||
|
|
||||||
|
```
|
||||||
|
mkdir build
|
||||||
|
cd build
|
||||||
|
../scripts/build_configs/ro_ipc
|
||||||
|
```
|
||||||
|
|
||||||
The build script passes configuration options to CMake to setup canonical builds.
|
The build script passes configuration options to CMake to setup canonical builds.
|
||||||
There are other scripts in `./scripts/build_configs`
|
There are other scripts in `./scripts/build_configs`
|
||||||
directory but currently, only `ipc_single` is supported.
|
directory but currently, only `ipc_single` is supported.
|
||||||
@@ -111,6 +118,12 @@ rocSHMEM has the following enviroment variables:
|
|||||||
|
|
||||||
ROCSHMEM_RO_DISABLE_IPC (default : 0)
|
ROCSHMEM_RO_DISABLE_IPC (default : 0)
|
||||||
Disables IPC support for the reverse offload backend.
|
Disables IPC support for the reverse offload backend.
|
||||||
|
|
||||||
|
ROCSHMEM_MAX_NUM_CONTEXTS (default : 1024)
|
||||||
|
Maximum number of contexts used in library
|
||||||
|
|
||||||
|
ROCSHMEM_MAX_NUM_TEAMS (default : 40)
|
||||||
|
Maximum number of teams supported by the library
|
||||||
```
|
```
|
||||||
|
|
||||||
## Examples
|
## Examples
|
||||||
@@ -124,7 +137,7 @@ or to look at the provided example code in the `./example/` directory.
|
|||||||
The examples can be run like so:
|
The examples can be run like so:
|
||||||
|
|
||||||
```
|
```
|
||||||
mpirun -np 2 ./build/examples/rocshmem_getmem_test
|
mpirun --map-by numa --mca pml ucx --mca osc ucx -np 2 ./build/examples/rocshmem_getmem_test
|
||||||
```
|
```
|
||||||
|
|
||||||
## Tests
|
## Tests
|
||||||
@@ -135,7 +148,7 @@ To run the tests, you may use the driver scripts provided in the `./scripts/` di
|
|||||||
|
|
||||||
```
|
```
|
||||||
# Run Functional Tests
|
# Run Functional Tests
|
||||||
./scripts/functional_tests/driver.sh ./build/tests/functional_tests/rocshmem_example_driver short <log_directory>
|
./scripts/functional_tests/driver.sh ./build/tests/functional_tests/rocshmem_example_driver all <log_directory>
|
||||||
|
|
||||||
# Run Unit Tests
|
# Run Unit Tests
|
||||||
./scripts/unit_tests/driver.sh ./build/tests/unit_tests/rocshmem_unit_tests all
|
./scripts/unit_tests/driver.sh ./build/tests/unit_tests/rocshmem_unit_tests all
|
||||||
@@ -153,7 +166,7 @@ To build and configure ROCm-Aware UCX (1.17.0 or later), you need to:
|
|||||||
git clone https://github.com/openucx/ucx.git -b v1.17.x
|
git clone https://github.com/openucx/ucx.git -b v1.17.x
|
||||||
cd ucx
|
cd ucx
|
||||||
./autogen.sh
|
./autogen.sh
|
||||||
./configure --prefix=<prefix_dir> --with-rocm=<rocm_path> --enable-mt
|
./configure --prefix=<ucx_install_dir> --with-rocm=<rocm_path> --enable-mt
|
||||||
make -j 8
|
make -j 8
|
||||||
make -j 8 install
|
make -j 8 install
|
||||||
```
|
```
|
||||||
@@ -164,13 +177,21 @@ Then, you need to build Open MPI (5.0.6 or later) with UCX support.
|
|||||||
git clone --recursive https://github.com/open-mpi/ompi.git -b v5.0.x
|
git clone --recursive https://github.com/open-mpi/ompi.git -b v5.0.x
|
||||||
cd ompi
|
cd ompi
|
||||||
./autogen.pl
|
./autogen.pl
|
||||||
./configure --prefix=<prefix_dir> --with-rocm=<rocm_path> --with-ucx=<ucx_path>
|
./configure --prefix=<ompi_install_dir> --with-rocm=<rocm_path> --with-ucx=<ucx_install_dir>
|
||||||
make -j 8
|
make -j 8
|
||||||
make -j 8 install
|
make -j 8 install
|
||||||
```
|
```
|
||||||
|
|
||||||
|
After compiling and installing UCX and Open MPI, please update your PATH and LD_LIBRARY_PATH to point to the installation locations, e.g.
|
||||||
|
|
||||||
|
```
|
||||||
|
export PATH=<ompi_install_dir>/bin:$PATH
|
||||||
|
export LD_LIBRARY_PATH=<ompi_install_dir>/lib:<ucx_install_dir>/lib:$LD_LIBRARY_PATH
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
Alternatively, we have script to install dependencies.
|
Alternatively, we have script to install dependencies.
|
||||||
However, it is not gauranteed to work and perform optimally on all platforms.
|
However, it is not guaranteed to work and perform optimally on all platforms.
|
||||||
Configuration options are platform dependent.
|
Configuration options are platform dependent.
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -35,7 +35,6 @@ git checkout $_UCX_COMMIT_HASH
|
|||||||
--without-go \
|
--without-go \
|
||||||
--without-java \
|
--without-java \
|
||||||
--without-cuda \
|
--without-cuda \
|
||||||
--without-verbs \
|
|
||||||
--without-knem
|
--without-knem
|
||||||
make -j
|
make -j
|
||||||
make install
|
make install
|
||||||
@@ -56,7 +55,6 @@ git submodule update --init --recursive
|
|||||||
--with-hwloc=internal \
|
--with-hwloc=internal \
|
||||||
--with-libevent=internal \
|
--with-libevent=internal \
|
||||||
--without-cuda \
|
--without-cuda \
|
||||||
--disable-sphinx \
|
|
||||||
--disable-mpi-fortran \
|
--disable-mpi-fortran \
|
||||||
--without-ofi
|
--without-ofi
|
||||||
make -j
|
make -j
|
||||||
|
|||||||
Reference in New Issue
Block a user