SWDEV-305016 - Improve MGPU scaling in Tensorflow

Add a threshold for ROCR/SDMA P2P transfers. ROCR copy path
requires extra barriers in compute for synchronization. That costs
extra performance with tiny transfers.
Reduce active wait time to 10us. Tensorflow uses extra thread
per GPU with constant hipEventQuery() calls. Longer active waits
in ROCr affect CPU performance.

Change-Id: I9020358438615fa2d4617f862f00a562f0a588e7

rocclr/device/rocm/rocblit.cpp

@@ -2056,7 +2056,8 @@ bool KernelBlitManager::copyBuffer(device::Memory& srcMemory, device::Memory& ds
                                    const amd::Coord3D& sizeIn, bool entire) const {
   amd::ScopedLock k(lockXferOps_);
   bool result = false;
-  bool p2p = (&gpuMem(srcMemory).dev() != &gpuMem(dstMemory).dev());
+  bool p2p = (&gpuMem(srcMemory).dev() != &gpuMem(dstMemory).dev()) &&
+             (sizeIn[0] > ROC_P2P_SDMA_SIZE * Ki);
   bool asan = false;
 #if defined(__clang__)
 #if __has_feature(address_sanitizer)