SWDEV-193956

[hipclang-vdi-rocm][perf]~45% to 50% of Performance drop on
rocBLAS_int8 test

- Enable AMD_OPT_FLUSH optimization by default to match HCC
- Disable CPU writes to GPU memory on boards with large bar,
because it requires HDP flush tracking.
- Enable L2 cache on kernel arguments, because L2 will be
invalidated on memory reuse .

Change-Id: I124cf250bdd4d19c523ce542c163813828f8fbdc

rocclr/device/rocm/rocmemory.cpp

@@ -707,7 +707,9 @@ bool Buffer::create() {
     if (dev().forceFineGrain(owner()) ||
         dev().isFineGrainedSystem(true)) {
       memFlags |= CL_MEM_SVM_FINE_GRAIN_BUFFER;
-      flags_ |= HostMemoryDirectAccess;
+      // Don't enable direct access to GPU memory with large bar, because
+      // there is no tracking of HDP flush after CPU writes
+      // flags_ |= HostMemoryDirectAccess;
     }
     const bool isFineGrain = memFlags & CL_MEM_SVM_FINE_GRAIN_BUFFER;