Check return code for Flush operation
Current NCCL code does not abort for failed Flush operations by
underlying network. This may compromise data integrity.
Signed-off-by: Rashika Kheria <rashika@amazon.com>
[ROCm/rccl commit: 6c61492eba]
Этот коммит содержится в:
коммит произвёл
Sylvain Jeaugey
родитель
6034c27655
Коммит
38b445c94f
@@ -378,7 +378,7 @@ ncclResult_t netRecvProxy(struct ncclProxyArgs* args) {
|
||||
if (done) {
|
||||
args->head += args->sliceSteps;
|
||||
if (args->protocol == NCCL_PROTO_SIMPLE) {
|
||||
if (resources->useGdr) ncclNetFlush(resources->netRecvComm, localBuff+buffSlot*stepSize, size, mhandle);
|
||||
if (resources->useGdr) NCCLCHECK(ncclNetFlush(resources->netRecvComm, localBuff+buffSlot*stepSize, size, mhandle));
|
||||
resources->hostRecvMem->tail = args->head;
|
||||
}
|
||||
args->idle = 0;
|
||||
|
||||
Ссылка в новой задаче
Block a user