Edgar Gabriel
e167f50803
* relax MPI dependency from code
This commit (series) removes the strict dependency on MPI in code base.
rocSHMEM will still be compiled with MPI, but the goal is to make the
code work even if MPI_Init_thread has not been invoked, at least for
certain, well-defined scenarios. Hence, the goal is not remove any
mentioning of MPI from rocSHMEM, but to ensure correct execution of the
ipc conduit even if the library has been initialized using other means.
Details:
- add non-MPI version of remote_heap and WindowInfo classes
- host interfaces work on WindowInfoMPI, they will not work with the
non-MPI code path. Since it is unclear whether we plan to support the
host interfaces at all, this is probably not a major limitation.
* update symmetric_heap structures and backend
* first cut on initialization
and enabling non-MPI initialization of the IPCBackend
* add non-MPI hostInterface methods
at the moment, only barrier_all and sync_all are explicitely supported.
* add non-mpi version of ipc_policy
and a number of smaller fixes required in other files.
A small init/finalize test already passes now with the branch.
* add non-mpi team_split_strided code
* minor fixes for non-MPI use-case
* disable symmetric-heap-window-ionfo test
disable this test for now just to make the compilation pass. Will have
to rework it.
* make no-mpi great again
after rebasing on top of the MPI singleton changes.
* enable running functional tests with uuid init
to run the functional tests using rocshmem_init_attr and the uuid
mechanism requires
a) a PMIx installation on the system
b) setting the environment variable ROCSHMEM_TEST_UUID=1
* fix multi-team creation bug
fix a bug occuring when creating many teams, which was the result of
incorrectly applying two indices in our own implementation of Allreduce.
* make unit tests pass again
* reverse offload was impacted by code change
fix the RO conduit to cope wioth the non-MPI path introduced for the IPC
conduit.
* update to cmake logic to find pmix
* Update src/memory/window_info.hpp
Co-authored-by: Yiltan <ytemucin@amd.com>
* Update CMakeLists.txt
Co-authored-by: Yiltan <ytemucin@amd.com>
* document ROCSHMEM_UNIQUEID_NO_MPI
* rename env. variable to UNIQUEID_WITH_MPI
* update host.cpp to use USE_HDP_FLUSH macro
instead of the deprecated USE_COHERENT_HEAP.
* add note for running example with RO conduit
add a note clarifying that running init_attr_test from the example
directory requires setting an additional environment variable with the
RO conduit.
* Find PMIx in more cases, only apply pmix build options to the test that
needs it, if OMPI_COMM_WORLD_LOCA_RANK is not setenv, abort
---------
Co-authored-by: Yiltan <ytemucin@amd.com>
Co-authored-by: Aurelien Bouteiller <abouteil@amd.com>
[ROCm/rocshmem commit: 6ea5edc951]
87 γραμμές
3.4 KiB
ReStructuredText
87 γραμμές
3.4 KiB
ReStructuredText
.. meta::
|
|
:description: Information on how to compile and run rocSHMEM applications.
|
|
:keywords: rocSHMEM, ROCm, library, API, compile, link, hipcc
|
|
|
|
.. _running-applications:
|
|
|
|
--------------------------------------------------
|
|
Compiling and running rocSHMEM applications
|
|
--------------------------------------------------
|
|
|
|
This topic explains how to compile and run rocSHMEM applications.
|
|
|
|
Compiling and linking with rocSHMEM
|
|
-----------------------------------
|
|
|
|
rocSHMEM is a library that can be statically linked to your application during compilation with ``hipcc``. For more information, see :doc:`HIPCC <hipcc:index>`.
|
|
|
|
When compiling your application with ``hipcc``, you must include the rocSHMEM header files and the rocSHMEM library.
|
|
Because rocSHMEM depends on MPI (Message Passing Interface), you must manually add the arguments for MPI linkage instead of using ``mpicc``.
|
|
|
|
When using ``hipcc`` directly without a build system, it's recommended to perform the compilation and linking steps separately.
|
|
|
|
Example compile and link commands are provided at the top of the example files in the ``examples`` directory:
|
|
|
|
.. code-block:: bash
|
|
|
|
# Compile
|
|
hipcc -c -fgpu-rdc -x hip rocshmem_allreduce_test.cc \
|
|
-I/opt/rocm/include \
|
|
-I$ROCSHMEM_INSTALL_DIR/include \
|
|
-I$OPENMPI_UCX_INSTALL_DIR/include/
|
|
|
|
# Link
|
|
hipcc -fgpu-rdc --hip-link rocshmem_allreduce_test.o -o rocshmem_allreduce_test \
|
|
$ROCSHMEM_INSTALL_DIR/lib/librocshmem.a \
|
|
$OPENMPI_UCX_INSTALL_DIR/lib/libmpi.so \
|
|
-L/opt/rocm/lib -lamdhip64 -lhsa-runtime64
|
|
|
|
If your project uses CMake, see
|
|
`Using CMake with AMD ROCm <https://rocmdocs.amd.com/en/latest/conceptual/cmake-packages.html>`_.
|
|
|
|
Running a rocSHMEM application
|
|
--------------------------
|
|
|
|
Applications using rocSHMEM typically deploy multiple processes, usually one per GPU.
|
|
The MPI launcher, for example, ``mpiexec`` with Open MPI, is used to start the required number
|
|
of processes. For example, to launch two ``getmem`` example processes (available when compiled from source):
|
|
|
|
.. code-block:: bash
|
|
|
|
mpiexec --map-by numa --mca pml ucx --mca osc ucx -np 2 ./build/examples/rocshmem_getmem_test
|
|
|
|
See the `Open MPI documentation <https://docs.open-mpi.org/en/main/>`_ for more information about ``mpiexec`` command line parameters.
|
|
|
|
.. note::
|
|
|
|
Some systems may have multiple MPI installations, some of which do not
|
|
have GPU support enabled. You must use the ``mpiexec`` from the expected
|
|
MPI library, especially when using the MPI built by yourself
|
|
as part of :ref:`install-dependencies`.
|
|
|
|
Environment variables
|
|
---------------------
|
|
|
|
You can control the behavior of rocSHMEM by using the following environment variables:
|
|
|
|
.. list-table:: Environment Variables
|
|
:widths: 30 10 20
|
|
:header-rows: 1
|
|
|
|
* - Name
|
|
- Default Value
|
|
- Description
|
|
* - ROCSHMEM_HEAP_SIZE
|
|
- 1
|
|
- Defines the size of the rocSHMEM symmetric heap in GB.
|
|
Note the heap is on the GPU memory.
|
|
* - ROCSHMEM_MAX_NUM_CONTEXTS
|
|
- 1024
|
|
- Defines the number of contexts an application can use.
|
|
* - ROCSHMEM_MAX_NUM_TEAMS
|
|
- 40
|
|
- Defines the number of teams an application can use.
|
|
* - ROCSHMEM_UNIQUEID_WITH_MPI
|
|
- 0
|
|
- Defines whether rocSHMEM is expected to use MPI when using the uniqueId based initialization.
|