Make host memory allocations coherent by default.

Associated change is to optimize event recording so it uses
agent-scope releaes (since it was only using system-scope release
to support non-coherent host mem).

Flags and environment variables exist to obtain previous behavior
if desired.  Options are documented in new performance guide.


[ROCm/hip commit: 8b8e97ff28]

projects/hip/src/hip_device.cpp

@@ -180,6 +180,7 @@ hipError_t hipDeviceReset(void)
     return ihipLogStatus(hipSuccess);
 }
 
+
 hipError_t ihipDeviceSetState(void)
 {
     hipError_t e = hipErrorInvalidContext;