* Let functional tests build without external MPI
* Fix error conditions when using uuid startup with internal MPI
* Do not abort if libibverbs is not found but not using GDA
* Enabled RO functional test initialized with TEST_UUID
* Reduce load time for ro backend_can_run and prevent mpilib_dlclose
crashing
* Fix case TEST_UUID=1, ROCSHMEM_BACKEND='' (autoloading gda)
* reenable gfx1100
use the modified version of the flat_store_short assembly instruction as suggested by the compiler team (32bit input value instead of 16bit)
* add fix for gfx1201
add the same fix for gfx1201 that was introduced for gfx1100
* Add host-side rocshmem_alltoallmem_on_stream function
Function signature:
rocshmem_alltoallmem_on_stream(rocshmem_team_t team, void *dest,
const void *source, size_t size,
hipStream_t stream)
- The function launches rocshmem_alltoallmem_kernel which calls
device-side alltoall<char> workgroup collective through default context.
- Uses dynamic block size determination via occupancy API.
- Implemented for all backends.
* Fix incorrect sync buffer size allocation for alltoall in GDA and IPC backends
When allocating memory for alltoall_pSync_pool in setup_teams() and
teams_init() functions, the code incorrectly used ROCSHMEM_BCAST_SYNC_SIZE
instead of ROCSHMEM_ALLTOALL_SYNC_SIZE.
* Add functional test for team_alltoallmem_on_stream
This commit adds a new functional test to verify the correctness of
the host-side rocshmem_team_alltoallmem_on_stream API.
* Add documentation for rocshmem_alltoallmem_on_stream
This commit adds API documentation for the host-side
rocshmem_alltoallmem_on_stream function in the collective routines
section. The documentation includes:
* gda: add check for active interfaces when selecting the GDA backend
* fix __func__ maco in rocshmem_ctx_pe_quiet
* gda: switch to more generic RDMA NIC term in has_active_ib_interface
* gda: add active MLX5 and Pensando vendor ID checks for backend selection
* add relaxed_ordering option
add an environment variable that allows to control setting the
IBV_ACCESS_RELAXED_ORDERING flag when registering memory with the
ibv_reg_mr* functions.
* missed a spot
* Remove testing of data types
As the collective is templated, we are just testing if sizeof(T) works
* Added single threaded varients
* Applied thread puts optimization to barrier
* Apply single threaded optimization to alltoall
* This optimization only works on bnxt, so place a switch to protect it
* Handle the edge case where the thread count is smaller than the number of PEs
disable fine-grain and coarse-grain memory testst until a fix is
available in ROCm 7.1 and/or our CI image. Otherwise we might miss other
errors due to constant CI failures.
A copy paste mistake in a previous commit caused source and dest to
be reversed. Correct the source and dest params.
Fixes: 6de67d5d7c
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
Use all available threads for polling the cq to increase the maximum
message rate. Even when posting a single wqe in the wave, use all
available theads for polling the cq to reserve space in the sq.
Changes were needed in the rocshmem abstraction to avoid disabling gpu
threads, like taking turns or using only the first thread in a wave or
wavefront. To avoid breaking other gda implementations, reimplement
turn-based or single thread strategy in post_wqe_rma_turn and
post_wqe_rma_single.
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
When creating a python venv during the install_dependencies script, we try to use ensurepip if it is installed, as it deals better with cases where multiple venvs are active simultaneously. (as seen in CI buildbot)
* The install_dependencies script would fail on ubuntu 24.04
they changed how pip works so we need to create a venv first now
* Fix install_dependencies for ubuntu 22
* Make sure we build in the builddir and install in the installdir
combine installdir for ucx and ompi when user-provided by INSTALL_DIR
retain prior behavior if not overridden to avoid breaking CI scripts
To make the functional tests more useful for benchmarking, allow user to
specify the number of loops and related parameters via command options.
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
somehow the test whether we requested MPI support or not stopped
working, although no obvious code change can be located.
Make the if-statement more stringent by explicitely testing whether
USE_MPI_SUPPORT is "ON".
* SyncAll test case would run Sync
* Despecialized name for argument reader
* Rename sync-test to team-sync-test as it uses teams
* Another stab at probing NUM_GPUS
* add gfx1100 support
Add support for Radeon 7900 GPUs (RX and PRO), and 7800 PRO.
I was contemplating to add gfx1101 and gfx1102 GPUs as well, but those are the lower end models that are more unlikely to be used for compute intensive jobs. In addition, I do not have access to them to test the support.
* update WF_SIZe for different options
Radeon systems use a WarpSize of 32, unlike current Instinct systems,
which use a warp size of 64. For the device side, a gfx specific ifdef
is sufficient. For the host side, we need to query the device
properties.
* adjust functional tests to wf_size of 32
* update unit tests to handle wf_size of 32
* address reviewer comments
* Add `ROCSHMEM_CTX_INVALID` for invalid context handling
- Define `ROCSHMEM_CTX_INVALID` as {nullptr, nullptr}
- Add == and != operators to rocshmem_ctx_t
- Use `ROCSHMEM_CTX_INVALID` on failed context creation
- Skip ctx destroy if context is invalid
* Update docs for context create and destroy APIs usage and behavior
update the tester script to only tests the amo functions on RO that are
expected to pass. We can revisit the non-passing tests later, but this
prevents us from having passing CIs at the moment, while RO is simply
lower priority than other asks.
- make adding PMIx library to compile time based on the result of
finding PMIx support. This is required eg if compiling rocSHMEM with ompi
4.0/4.1, which do not have a built-in PMIx version.
- when setting USE_EXTERNAL_MPI=OFF which ensures that we do not
check for external MPI libraries (even if one would be available).
Aurelien Bouteiller
Yiltan
Avinash Kethineedi
Edgar Gabriel
Anatolii Rozanov
Kutovoi, Vadim
Adel Johar
Yiltan
Dana Robinson
Allen Hubbe
erieaton-amd
akolliasAMD