* misc/msccl: read graphCaptureStatus for every collective call
* fix a bug in checking whether UBR is enabled in MSCCLPP
* cmake: Fix patch reversal order
* misc/msccl: add logging
[ROCm/rccl commit: 23c0b7bd84]
* Improve RDMA flushing by write dummy payload with RO=0
* Rename env var for disabling this change to RCCL_GDR_FLUSH_GPU_MEM_NO_RELAXED_ORDERING
[ROCm/rccl commit: 60c1264d27]
* Enable GDRCopy only on gfx94x
* Use cudaFree instead of hipFree
* Add warning if failed to get device property
* Remove extra return
[ROCm/rccl commit: 32dc7ef47c]
COMPILING_TARGETS is not actually used for --offload-arch option,
instead GPU_TARGETS is being used implicitly when we call
find_package(hip REQUIRED) (See hip-config-amd.cmake).
[ROCm/rccl commit: 2f1c0bb213]
* Initial Script ready for review
* Added RCCL-tests and RCCL versions
* Added output folder and README
* Base format built
* Added ROCm version
* Added function to center titles and Vram information
* Added HIP version
* Cleaned formatting
* UCX version and MPI version
* Added NUMA balancing
* Added rocminfo
* Removed notes
* Changed regex for broadcom Nic
* Removed note by the ACS info
* Added Hostname to summary and details
* Print summary to terminal
* Added argparse
* Added flags and readme
* Added GPU ID
* fixed spelling
* renamed script again
* Added file descriptor and locked mem checks
* Added file descriptor and locked mem checks
* Removed extra spaces from summary table
* printing output file location
* Removed sudo in code and ACS flag
[ROCm/rccl commit: 4ba94d6662]
* Print KL/CL/KE events for all warps
* Fix count off-by-one issue
* Fix opCount in KE and restore CPU thread option
* Simplify count calculation
[ROCm/rccl commit: ebf7e2305e]
* Accept an EXPLICIT_ROCM_VERSION and use that vs inspecting the environment if provided.
* Use CMake's built in file reading support vs execute_process (without error checking) to avoid silent but deadly later failures.
* Properly quote some comparisons to avoid syntax errors if they happen to have an empty string.
* Guard against ROCM_PATH being an empty string, avoiding stray path extensions to root directories, etc.
Co-authored-by: Stella Laurenzo <stellaraccident@gmail.com>
[ROCm/rccl commit: 92ac136db5]
* Switched cmake_host_system_information feature to a manual parse to remain cmake 3.5 compliant.
* Updating minimum cmake to 3.16 to conform with the rest of ROCm. This change still applies.
[ROCm/rccl commit: 42ab425037]
* working tests for a single message size
* move call_RCCL routine StandaloneUtils, create .cpp file for StandaloneUtils so that it can be included in several tests
* simplify test invocation
* remove unecessary logs and exit from ncclCommRegister
* set expected results for allGather
* skip test if nranks doesn't match number of gpus, call getAndDistributeNCCLid only from parent process
* fix improper size of expected-results vector
* Removing unused changes.
* Refactored to create a new file for the forked collectives call, as StandaloneUtils is for the Standalone tests. Renamed the functions to be slightly more accurate and follow existing naming conventions.
* Apply suggestions from code review
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>
---------
Co-authored-by: isaki001 <isakioti@banff-pla-r27-38.pla.dcgpu>
Co-authored-by: corey-derochie-amd <161367113+corey-derochie-amd@users.noreply.github.com>
Co-authored-by: Corey Derochie <corey.derochie@amd.com>
[ROCm/rccl commit: 3398fa78fe]
* erase handle key from mscclpp communicator during deregistration
* remove check on buffer size being a multiple of 32 from registration/deregistration routines since these checks are applied during enqueue
* add check for greater than zero buffer size in mscclpp registration
[ROCm/rccl commit: 19105206f6]
* Fix collective trace
* Use nontemporal for st_global
* Fix previous commit
* Add HDP flush to data receive path
* Fix previous commit
* Control flushing by NCCL_NET_FORCE_FLUSH and RCCL_NET_HDP_FLUSH
* Introduce RCCL_NET_HDP_FLUSH and RCCL_NET_GDR_FLUSH
Both are on by default. Turn both off will skip all flush will likely
result in data error.
* Enable GDR copy by default
* Remove GDR flush env var because it is disabled by GDC flush
* Output kernel collective trace at comm destroy by default
* Limit kernel timeout messages to 100
* Use system relaxed atomic for loadInt
* Refine timeout messages and use atomic for setting offset from CPU
* Add kernel trace for barrier timeout
* Add backup barrier to avoid race in atomicAdd
* Use different counters for different warps
* Rework barrier implementation
* Fix for other GFX
* Use __hip_atomic_store and __hip_atomic_load
* Fix bug in previous commit
* Don't reset barrier values in running kernel
* Update trace format
* Fix typo
* Switch back to hip_atomic_fetch_add
* Use same barrier implementation for all GFX
* Remove extra threadfence
* Turn off HDP flush by default
Please use RCCL_NET_HDP_FLUSH=1 to switch on HDP flush
* Remove unnecessary changes from alterative barrier implementation
* Added back __threadfence_block
* Revert back to threadfence for gfx other than gfx94x
[ROCm/rccl commit: caba0bc049]
Wenkai Du
Nilesh M Negi
dependabot[bot]
Nusrat Islam
Bertan Dogancay
Pedram Alizadeh
gilbertlee-amd
akolliasAMD
Sohaib Nadeem
Nikhil-Nunna
corey-derochie-amd
mberenjk
rahulc1984
Dingming Wu
Vijay Srinivasan
Nikhil-Nunna
AbandiGa
Edgar Gabriel
isaki001
Jeffrey E Erickson
Jeffrey Novotny
Mustafa Abduljabbar
Arm Patinyasakdikul