Update MP UT to support arbitrary # of GPUs; multiple bugfixes (#16)

* Fixing temp file creation/deletion for Clique kernel mode.

* Refactoring of MP unit tests; include bugfixes and general support for any number of GPUs

* GroupCall MP UT properly quits when too many devices specified

* MP UT will programmatically set NCCL_COMM_ID if not specified; updated install script
Этот коммит содержится в:
Stanley Tsang
2021-02-05 17:49:25 -07:00
коммит произвёл GitHub
родитель 6dfdfef98f
Коммит d00b7d17bd
23 изменённых файлов: 538 добавлений и 716 удалений
+7 -3
Просмотреть файл
@@ -19,11 +19,15 @@ namespace CorrectnessTests
dataset.NumBytes(), hipMemcpyDeviceToHost));
}
void TestGather(int rank, Dataset& dataset)
void TestGather(int rank, Dataset& dataset, bool& pass)
{
SetUpPerProcess(rank, ncclCollGather, comms[rank], streams[rank], dataset);
if (numDevices > numDevicesAvailable) return;
if (numDevices > numDevicesAvailable)
{
pass = true;
return;
}
Barrier barrier(rank, numDevices, std::atoi(getenv("NCCL_COMM_ID")));
@@ -44,7 +48,7 @@ namespace CorrectnessTests
HIP_CALL(hipStreamSynchronize(streams[rank]));
// Check results
ValidateResults(dataset, rank, root);
pass = ValidateResults(dataset, rank, root);
// Ensure all processes have finished current iteration before proceeding
barrier.Wait();