AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add warning if workFIFO is not available after multiple retries. (#1772)

Browse Source 
* add warning if workFIFO is not available after multiple retries.

* Update src/enqueue.cc

Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>

---------

Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>

[ROCm/rccl commit: 265b1b3775]
This commit is contained in:
Arm Patinyasakdikul
2025-06-26 19:49:52 -05:00
committed by GitHub
parent 32e80aedc0
commit c3b110f9e9
1 changed files with 13 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
projects/rccl/src/enqueue.cc
+13
View File
@@ -1197,6 +1197,8 @@ static ncclResult_t scheduleP2pTasksToPlan(
// Spin until its safe to increase comm->workFifoProduced to desiredProduced.
static void waitWorkFifoAvailable(struct ncclComm* comm, uint32_t desiredProduced) {
bool hasRoom = (desiredProduced - comm->workFifoConsumedLeast) <= comm->workFifoBytes;
uint64_t count = 0;
int warned = 0;
if (hasRoom) return;
while (true) {
// We have to poll for notifications from device.
@@ -1235,6 +1237,17 @@ static void waitWorkFifoAvailable(struct ncclComm* comm, uint32_t desiredProduce
hasRoom = (desiredProduced - comm->workFifoConsumedLeast) <= comm->workFifoBytes;
if (hasRoom) break;
sched_yield();
/* Warn if we get stuck waiting for workFifo. */
count++;
if (warned == 0 && count == 100000 && comm->rank == 0) {
warned = 1;
WARN("Waiting for work FIFO to become available. "
"Work fifo exhaustion can happen in large scale/high iteration count of alltoall. "
"In order to increase work FIFO size, set NCCL_WORK_FIFO_BYTES to higher number (current: %ld).\n\n"
"RCCL continues to retry...", comm->workFifoBytes);
}
}
}