Remove MPI compile-time dependency (#264)

* use dlsym for MPI functions

to allow compiling without MPI support, convert the usage of MPI functions and symbols to be based on a dlopen/dlsym based mechanism. Turns out this cannot be done entirely vendor neutral, slightly different solutions might be required for Open MPI, MPICH and the new MPI ABI.

* checkpoint

more work to be done.

* checkpoint 2

* checkpoint 3

* checkpoint 4

examples compile and link correctly

* checkpoitn 5 (I think)

* Checkpoitn 6

* dyld-mpi: adapt GDA

* dyldmpi: tests that depend on MPI need to link with it themselves

* do not ../mpi_instance.h

* dyldmpi: make the symetricHeapTestFixture compile

* dyldmpi: Change cmakery, compiles and run gda w/o external MPI

* Make it also compile in external MPI mode

* dyldmpi: ipc unit tests compile but do not link

* dyldmpi: new approach, if external mpi required, link with mpi,
otherwise use ompi5 abi

* C-style comments in cmakelist..

* dyldmpi: examples: do not fail compiling if MPI not found at build time,
instead do not compile the MPI required examples

* more updates to CMake logic

* convert RO backend

and a few other cleanups

* update some unit tests

to work with the dlopen MPI environment correctly.

---------

Co-authored-by: Aurelien Bouteiller <abouteil@amd.com>

src/host/host.hpp

@@ -34,8 +34,6 @@
  * any backend type.
  */
 
-#include <mpi.h>
-
 #include <map>
 
 #include "rocshmem/rocshmem.hpp"
@@ -43,6 +41,7 @@
 #include "memory/symmetric_heap.hpp"
 #include "memory/window_info.hpp"
 #include "bootstrap/bootstrap.hpp"
+#include "mpi_instance.hpp"
 
 namespace rocshmem {
 
@@ -268,17 +267,17 @@ class HostInterface {
       if (i == my_pe_) {
         continue;
       }
-      MPI_Put(&flush_val, 1, MPI_UNSIGNED, i, 0, 1, MPI_UNSIGNED, hdp_win);
+      mpilib_ftable_.Put(&flush_val, 1, MPI_UNSIGNED, i, 0, 1, MPI_UNSIGNED, hdp_win);
     }
-    MPI_Win_flush_all(hdp_win);
+    mpilib_ftable_.Win_flush_all(hdp_win);
 #endif // USE_HDP_FLUSH
   }
 
   __host__ void flush_remote_hdp(int pe) {
 #if defined USE_HDP_FLUSH
     unsigned flush_val{HdpPolicy::HDP_FLUSH_VAL};
-    MPI_Put(&flush_val, 1, MPI_UNSIGNED, pe, 0, 1, MPI_UNSIGNED, hdp_win);
-    MPI_Win_flush(pe, hdp_win);
+    mpilib_ftable_.Put(&flush_val, 1, MPI_UNSIGNED, pe, 0, 1, MPI_UNSIGNED, hdp_win);
+    mpilib_ftable_.Win_flush(pe, hdp_win);
 #endif // USE_HDP_FLUSH
   }