kfdtest: Improve the stablility of SignalHandling test

On gfx1012, allocating 1/4 of the system memory on a 32G RAM machine
could fail, resulting in this test to fail. Limit the maximum buffer
to allocate to be smaller than 3G to accommodate this situation.

Change-Id: I38b0a0f7da1f0b9ca851e04d2d0a51767858c801
Signed-off-by: Yong Zhao <Yong.Zhao@amd.com>


[ROCm/ROCR-Runtime commit: 2b70d73f68]

projects/rocr-runtime/tests/kfdtest/src/KFDMemoryTest.cpp

@@ -1569,6 +1569,13 @@ TEST_F(KFDMemoryTest, SignalHandling) {
      * Try to allocate 1/4th System RAM
      */
     size = (sysMemSize >> 2) & ~(HSAuint64)(PAGE_SIZE - 1);
+
+    /* We don't need a too large buffer for this test. If it is too large,
+     * on some platform, the upcoming hsaKmtAllocMemory() might fail. In
+     * order to avoid this flaky behavior, limit the size to 3G.
+     */
+    size = size > (3ULL << 30) ? (3ULL << 30) : size;
+
     m_MemoryFlags.ui32.NoNUMABind = 1;
     ASSERT_SUCCESS(hsaKmtAllocMemory(0 /* system */, size, m_MemoryFlags, reinterpret_cast<void**>(&pDb)));
     // Verify that pDb is not null before it's being used