Sampling support + testing + omnitrace namespace (#19)

* omnitrace namespace

* Kokkos + Lulesh example/tests

* Sampling support + more

- OMNITRACE_BUILD_TESTING option
- sampling support
- pthread_gotcha
- fixes to labels for mpi_gotcha, fork_gotcha, omnitrace_component
- tasking::block_signals, tasking::unblock_signals
- instrumentation mode option in omnitrace exe
- argument option groups in omnitrace exe
- categories in omnitrace settings
- remove TIMEMORY_ prefixed options

* Release workflow updates

* Updated settings printing

* Fixed defaults in README

* Tweak setting defaults in README

* CMake fixes

* cmake-format

* clang-format

* LULESH_USE_MPI OFF

* LULESH_USE_MPI fix

* timemory add_secondary fix

* timemory ambiguous internal namespace fix

* Update timemory submodule

* Handle output path/prefix in omnitrace

- updated timemory
- updated test environment

* sampling + papi fix

* Fix to sampling without PAPI

* Fix for using too many processors in CI

* formatting

* Updated CI

- minor cmake tweaks
- updated timemory submodule

* Updated CI

* Updated CI

* CI + timemory updates

- data race fixes

* CI updates + debug for sampling

* Sampling updates

- moved tasking::{block,unblock}_signals to sampling namespace
- improvements to sampling w.r.t. thread-locality

* Minimum OMNITRACE_THREAD_COUNT of 128

* Handle multiple dims in sampler data

* Configure libunwind support for timemory

* Improved safeguards for sampling

- updated CI
- lulesh runtime-instrument test tweak

* formatting

* CI updates + sampler updates + misc

- fixed stack-buffer-overflow in omnitrace (get_*file_line_info)
- test labels
- steady_clock instead of system_clock in sampler
- update dyninst submodule with upgradePlaceholder fix
- disable OMNITRACE_BUILD_TESTING by default

* Updated timemory submodule

- hidden visibility for timemory
- storage finalizers do not capture this

* Update timemory submodule

- component visibility updates

* Reworked header includes

- use <...> for timemory headers
- always include <library/defines.hpp>

* Rename some config options

* Update PTL submodule

* Update kokkos submodule

* Updated sampling

* Updated CI

* Reworked instrumentation exe

- lowered min-address-range threshold to 256
- extended whole function exclude

* CI fix + timemory submodule update

- TIMEMORY_VISIBLE on component base
- RelWithDebugInfo -> RelWithDebInfo
- Info output for parallel-overhead

* Sampling flags + transpose update + CI update

- disable critical trace for parallel-overhead in CI
- SA_RESTART only in sampler
- reworked transpose example to use fewer threads

* CI update

- removed ubuntu-focal-external-debug
- reduced data artifacts upload

* CI timeouts

- updated timemory submodule
- minor tweaks to omnitrace exe logging

* LICENSE updates (partial)

* CI Test stage timeout extension

* Docker and Packaging updates

* Miscellaneous fixes/tweaks

- gpu.hpp / gpu.cpp
- disable roctracer component if no devices
- re-enable InstrStackFrames by default
- disable sampling by default
- pthread_gotcha::m_enable_sampling is false by default
- timemory submodule update w/ sampler and pop(tid) updates
- fix minor bug in sampler logic
- CMake: OMNITRACE_USE_HIP option
- roctracer + timemory fix

* Replaced OMNITRACE_USE_ROCTRACER with OMNITRACE_USE_HIP where appropriate

* cmake format

* Sampler deadlock fixes

* Removed debug messages from sampler

* Fix for MPI detection + test tweaks + misc

* Sampler deadlock fixes + misc

- removed papi_tot_ins
- pthread_gotcha blocks signals globally until sampler is setup
- metadata specialization for sampling components
- OMNITRACE_INSTRUMENTATION_MODE -> OMNITRACE_MODE
- default sampling delay increased to 0.05 from 1.0e-6
- removed {block,unblock}_signals from critical_trace and ptl
    - no longer necessary to use
- sampling delay minimum is 1.0e-3
- OMNITRACE_BUILD_HIDDEN_VISIBILITY

* omnitrace-avail + libunwind update + restructure

- restructured omnitrace components
- build custom omnitrace-avail executable
- updated libunwind to avoid malloc in get_unw_backtrace

* Fix remaining reorganization issues

- removed some duplicate code
- fixed some trait specializations after implicit instatiation
- formatting

* ensure_storage fix + avail improvements

- fix ensure_storage when component not avail
- suppress irrelevant info in omnitrace-avail

* Delay settings initialization

- slight tweak to tests w/ MPI

* Disable OpenMPI testing w/ ubuntu-bionic

- MPI testing is hanging bc of network interface issue on system:

> [[20462,1],0]: A high-performance Open MPI point-to-point messaging module
> was unable to find any relevant network interfaces:
> Module: OpenFabrics (openib)
>   Host: fv-az19-371
> Another transport will be used instead, although this may result in
> lower performance.
> NOTE: You can disable this warning by setting the MCA parameter
> btl_base_warn_component_unused to 0.

examples/transpose/transpose.cpp

@@ -21,6 +21,7 @@ THE SOFTWARE.
 */
 
 #include "hip/hip_runtime.h"
+
 #include <cfloat>
 #include <chrono>
 #include <cmath>
@@ -29,14 +30,19 @@ THE SOFTWARE.
 #include <fstream>
 #include <iomanip>
 #include <iostream>
+#include <mutex>
 #include <thread>
 #include <vector>
 
+static std::mutex print_lock{};
+using auto_lock_t = std::unique_lock<std::mutex>;
+
 #define HIP_API_CALL(CALL)                                                               \
     {                                                                                    \
         hipError_t error_ = (CALL);                                                      \
         if(error_ != hipSuccess)                                                         \
         {                                                                                \
+            auto_lock_t _lk{ print_lock };                                               \
             fprintf(stderr, "%s:%d :: HIP error : %s\n", __FILE__, __LINE__,             \
                     hipGetErrorString(error_));                                          \
             exit(EXIT_FAILURE);                                                          \
@@ -49,6 +55,7 @@ check_hip_error(void)
     hipError_t err = hipGetLastError();
     if(err != hipSuccess)
     {
+        auto_lock_t _lk{ print_lock };
         std::cerr << "Error: " << hipGetErrorString(err) << std::endl;
         exit(err);
     }
@@ -63,6 +70,7 @@ verify(int* in, int* out, int M, int N)
         int col = rand() % N;
         if(in[row * N + col] != out[col * M + row])
         {
+            auto_lock_t _lk{ print_lock };
             std::cout << "mismatch: " << row << ", " << col << " : " << in[row * N + col]
                       << " | " << out[col * M + row] << "\n";
         }
@@ -85,19 +93,23 @@ transpose_a(int* in, int* out, int M, int N)
 }
 
 void
-run(int rank, int argc, char** argv)
+run(int rank, int tid, hipStream_t stream, int argc, char** argv)
 {
-    (void) argc;
-    (void) argv;
-    unsigned int M = 4960 * 2;
-    unsigned int N = 4960 * 2;
+    size_t       nitr = 5000;
+    unsigned int M    = 4960 * 2;
+    unsigned int N    = 4960 * 2;
+    if(argc > 2) nitr = atoll(argv[2]);
+
+    auto_lock_t _lk{ print_lock };
+    std::cout << "[" << rank << "][" << tid << "] M: " << M << " N: " << N << std::endl;
+    _lk.unlock();
 
-    std::cout << "[" << rank << "] M: " << M << " N: " << N << std::endl;
     size_t size   = sizeof(int) * M * N;
-    int*   matrix = (int*) malloc(size);
+    int*   matrix = new int[size];
     for(size_t i = 0; i < M * N; i++)
         matrix[i] = rand() % 1002;
-    int *in, *out;
+    int* in  = nullptr;
+    int* out = nullptr;
 
     std::chrono::high_resolution_clock::time_point t1, t2;
 
@@ -106,37 +118,36 @@ run(int rank, int argc, char** argv)
     HIP_API_CALL(hipMemset(in, 0, size));
     HIP_API_CALL(hipMemset(out, 0, size));
     HIP_API_CALL(hipMemcpy(in, matrix, size, hipMemcpyHostToDevice));
+    HIP_API_CALL(hipDeviceSynchronize());
+
     hipDeviceProp_t props;
     HIP_API_CALL(hipGetDeviceProperties(&props, 0));
 
     dim3 grid(M / 32, N / 32, 1);
     dim3 block(32, 32, 1);  // transpose_a
 
-    t1                   = std::chrono::high_resolution_clock::now();
-    const unsigned times = 10000;
-    auto           _func = [&](hipStream_t stream) {
-        for(size_t i = 0; i < times / 2; i++)
-        {
-            transpose_a<<<grid, block, 0, stream>>>(in, out, M, N);
-            check_hip_error();
-        }
-        HIP_API_CALL(hipStreamSynchronize(stream));
-    };
-    hipStream_t _stream{};
-    HIP_API_CALL(hipStreamCreate(&_stream));
-    std::thread _t{ _func, _stream };
-    _t.join();
-    _func(0);
-    HIP_API_CALL(hipDeviceSynchronize());
+    t1 = std::chrono::high_resolution_clock::now();
+    for(size_t i = 0; i < nitr; i++)
+    {
+        transpose_a<<<grid, block, 0, stream>>>(in, out, M, N);
+        check_hip_error();
+    }
+    HIP_API_CALL(hipStreamSynchronize(stream));
     t2 = std::chrono::high_resolution_clock::now();
     double time =
         std::chrono::duration_cast<std::chrono::duration<double>>(t2 - t1).count();
-    float GB = (float) size * times * 2 / (1 << 30);
-    std::cout << "[" << rank << "] Runtime of transpose is " << time << " sec\n"
+    float GB = (float) size * nitr * 2 / (1 << 30);
+
+    print_lock.lock();
+    std::cout << "[" << rank << "][" << tid << "] Runtime of transpose is " << time
+              << " sec\n"
               << "The average performance of transpose is " << GB / time << " GBytes/sec"
               << std::endl;
+    print_lock.unlock();
 
-    int* out_matrix = (int*) malloc(size);
+    HIP_API_CALL(hipDeviceSynchronize());
+
+    int* out_matrix = new int[size];
     HIP_API_CALL(hipMemcpy(out_matrix, out, size, hipMemcpyDeviceToHost));
 
     // cpu_transpose(matrix, out_matrix, M, N);
@@ -145,8 +156,8 @@ run(int rank, int argc, char** argv)
     HIP_API_CALL(hipFree(in));
     HIP_API_CALL(hipFree(out));
 
-    free(matrix);
-    free(out_matrix);
+    delete[] matrix;
+    delete[] out_matrix;
 }
 
 #if defined(USE_MPI)
@@ -174,12 +185,16 @@ main(int argc, char** argv)
     int rank     = 0;
     int size     = 1;
     int nthreads = 2;
+    int nitr     = 5000;
     if(argc > 1) nthreads = atoi(argv[1]);
+    if(argc > 2) nitr = atoi(argv[2]);
 
 #if defined(USE_MPI)
     MPI_Init(&argc, &argv);
     MPI_Comm_rank(MPI_COMM_WORLD, &rank);
     MPI_Comm_size(MPI_COMM_WORLD, &size);
+#else
+    (void) size;
 #endif
     // this is a temporary workaround in omnitrace when HIP + MPI is enabled
     int ndevice = 0;
@@ -193,16 +208,24 @@ main(int argc, char** argv)
     if(rank == devid && rank < ndevice)
     {
         std::vector<std::thread> _threads{};
+        std::vector<hipStream_t> _streams(nthreads);
+        for(int i = 0; i < nthreads; ++i)
+            HIP_API_CALL(hipStreamCreate(&_streams.at(i)));
         for(int i = 1; i < nthreads; ++i)
-            _threads.emplace_back(run, rank, argc, argv);
-        run(rank, argc, argv);
+            _threads.emplace_back(run, rank, i, _streams.at(i), argc, argv);
+        run(rank, 0, _streams.at(0), argc, argv);
         for(auto& itr : _threads)
             itr.join();
+        for(int i = 0; i < nthreads; ++i)
+            HIP_API_CALL(hipStreamDestroy(_streams.at(i)));
     }
 #if defined(USE_MPI)
     MPI_Barrier(MPI_COMM_WORLD);
     do_a2a(rank);
     MPI_Finalize();
 #endif
+    HIP_API_CALL(hipDeviceSynchronize());
+    HIP_API_CALL(hipDeviceReset());
+
     return 0;
 }