Use finegrain allocator by default (#140)

* Use FineGrained allocator for heap by default, consolidate all types of
allocators under saner cmake controls

Co-authored-by: Yiltan <ytemucin@amd.com>

* Uncached may not be only for debug

Need to include the rocshmem config otherwise produce an inconsistent
build with different allocators used in different files

* Undo this pr adding presumably useless hip_host_allocator_noncoherent

* Rename HEAP_IS_COHERENT/USE_COHERENT_HEAP to USE_HDP_FLUSH as the former
was misleading

* Remove unused __roc_inv()

---------

Co-authored-by: Yiltan <ytemucin@amd.com>

[ROCm/rocshmem commit: 41fd9e2d57]

projects/rocshmem/scripts/build_configs/ro_net_debug

@@ -42,11 +42,10 @@ cmake \
     -DDEBUG=OFF \
     -DPROFILE=OFF \
     -DUSE_IPC=OFF \
-    -DUSE_COHERENT_HEAP=ON \
     -DUSE_THREADS=OFF \
     -DUSE_WF_COAL=OFF \
-    -DUSE_HOST_SIDE_HDP_FLUSH=OFF\
-    -DUSE_MANAGED_HEAP=OFF \
+    -DUSE_HDP_FLUSH=OFF \
+    -DUSE_HDP_FLUSH_HOST_SIDE=OFF \
     -DUSE_RO=ON \
     -DBUILD_FUNCTIONAL_TESTS=ON \
     -DBUILD_UNIT_TESTS=ON \