Make host memory allocations coherent by default.

Associated change is to optimize event recording so it uses
agent-scope releaes (since it was only using system-scope release
to support non-coherent host mem).

Flags and environment variables exist to obtain previous behavior
if desired.  Options are documented in new performance guide.


[ROCm/hip commit: 8b8e97ff28]

projects/hip/src/hip_memory.cpp

@@ -293,7 +293,7 @@ hipError_t hipHostMalloc(void** ptr, size_t sizeBytes, unsigned int flags)
                 amFlags = amHostNonCoherent;
             } else {
                 // depends on env variables:
-                amFlags = HIP_COHERENT_HOST_ALLOC ? amHostCoherent : amHostNonCoherent;
+                amFlags = HIP_HOST_COHERENT ? amHostCoherent : amHostNonCoherent;
             }