From e12c35d84b026acc5fb573f7ac5a732430eedd32 Mon Sep 17 00:00:00 2001 From: Sylvain Jeaugey Date: Thu, 27 May 2021 09:12:52 -0700 Subject: [PATCH] Update PERFORMANCE.md --- doc/PERFORMANCE.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/PERFORMANCE.md b/doc/PERFORMANCE.md index 7cc6ecee66..21fef609af 100644 --- a/doc/PERFORMANCE.md +++ b/doc/PERFORMANCE.md @@ -46,7 +46,7 @@ A tree would do it hierarchically : `(((((i_{n-1} + i_{n-2}) + (i_{n-3} + i_{n-4})) + ... + (i_1 + i_0))))) -> o_0 -> (o_{n/2} -> (o_{3n/4} ...))` -In all cases, we need n-1 additions and n assignations for each element. Since every step is on a different rank except potentially one (the last input and the first output), +In all cases, we need n-1 additions and n assignments for each element. Since every step is on a different rank except potentially one (the last input and the first output), we need 2(n-1) data transfers (x number of elements) to perform an allReduce operation. Considering that each rank has a bandwidth to the outside world of _B_, the time to perform an allReduce operation of _S_ elements is at best : @@ -82,7 +82,7 @@ Note that here, S is the size in bytes of the total array, which for NCCL is equ ### AllGather -The AllGather operation requires only to perform the assignation part of the allReduce operation : +The AllGather operation requires only to perform the assignment part of the allReduce operation : `o_0 = o_1 = o_2 = ... = o_{n-1} = i_K`