From 1cabd1d6389af2f9e41a61fed6b8b21c491758e9 Mon Sep 17 00:00:00 2001 From: Sylvain Jeaugey Date: Fri, 14 Jun 2024 01:54:43 -0700 Subject: [PATCH] Add decription for regIsGlobal in the NET API documentation [ROCm/rccl commit: 529ee691c36c5a43656b80c32c049b9ae976d5c0] --- projects/rccl/ext-net/README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/projects/rccl/ext-net/README.md b/projects/rccl/ext-net/README.md index 5361b44e89..781fd904a4 100644 --- a/projects/rccl/ext-net/README.md +++ b/projects/rccl/ext-net/README.md @@ -209,6 +209,16 @@ set to `NCCL_PTR_HOST|NCCL_PTR_CUDA`, otherwise it should be set to `NCCL_PTR_HO supports `dmabuf`, it should set `ptrSupport` to `NCCL_PTR_HOST|NCCL_PTR_CUDA|NCCL_PTR_DMABUF` and provide a `regMrDmaBuf` function. +The `regIsGlobal` field allows NCCL to register buffers in advance using e.g. a loopback connection +and later on, expect that another registration on a buffer contained within a previous registration +will be nearly immediate, as the buffer is already known by the network adapter. A typical +implementation would maintain a registration cache; the call to ncclCommRegister will create the +initial entry in the cache using regMr() on a loopback connection. Any later call to NCCL +operations will call regMr() again on the real connection, with the real buffer (could be at a +different offset within the original buffer, with a smaller size, etc), then deregMr() right after. +The call to ncclCommDeregister should call the final deregMr() and effectively remove the mapping +on the network adapter. + The `speed` field indicates the speed of the network port in Mbps (10^6 bits per second). This is important to ensure proper optimization of flows within the node.