AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
906 Commits 3 Branches 0 Tags
9ddf0e06498d4b85082f767e616a8ca4d3aaec9b
Commit Graph

444 Commits

Author SHA1 Message Date
Wenkai Du 9ddf0e0649 Support P2P with invisible devices (#636) 
* Support P2P with invisible devices

* Update copyright year
 2022-10-17 10:24:59 -07:00
Wenkai Du 9916a09818 Merge pull request #634 from yzygitzh/ziyyang/npkit-fix 
Apply several fixes to NPKit
 2022-10-17 08:01:24 -07:00
gilbertlee-amd ebb8b5bf63 Updating files for missing licenses (#637) 2022-10-14 13:49:16 -06:00
Ziyue Yang 7d6bbc19d4 apply npkit 2022-10-14 01:28:17 +00:00
Edgar Gabriel e645b02cd8 introduce a hw topology aware bintree 
for hayabusa architecture.
 2022-10-03 15:26:21 +00:00
akolliasAMD ef71550738 Added new gpu targets (#631) 2022-09-29 14:53:55 -06:00
Wenkai Du a523b37ac7 Another threadfence and flags rework (#629) 2022-09-28 16:49:29 -07:00
Wenkai Du 021932b3c8 Add LL128 tuning (#630) 2022-09-27 09:39:09 -07:00
Wen-Heng (Jack) Chung e8af0716c4 Merge pull request #619 from whchung/exp_reduce_code_size 
Only use split tree algorithm to reduce kernel code size.
 2022-09-26 10:06:27 -05:00
Wen-Heng (Jack) Chung 35f1fe3434 Merge pull request #621 from whchung/exp_reduce_sleep_cycles 
Reduce s_sleep cycles
 2022-09-23 15:31:16 -05:00
Wen-Heng (Jack) Chung a80cc7e6e1 Merge pull request #624 from whchung/exp_tweak_unroll_factors 
Tweak unroll factors.
 2022-09-23 11:30:05 -05:00
Wen-Heng (Jack) Chung 84054c3b30 Tweak unroll factors. 2022-09-22 13:03:04 -05:00
Wenkai Du 02929cffb6 Only use split tree algorithm to reduce kernel code size. 2022-09-22 12:01:53 -05:00
Wenkai Du a3c8ef8c03 Reduce s_sleep cycles 2022-09-22 12:01:12 -05:00
Wen-Heng (Jack) Chung b9ae02d4ad Abolish syncthreads and only wait on LDS traffic. 2022-09-22 12:00:37 -05:00
Wenkai Du 49c811ecf9 Rework threadfence and flag setting (#627) 2022-09-22 08:35:42 -07:00
Wenkai Du d9216af48b Revert changes to gfx1030 (#622) 2022-09-20 20:06:17 -07:00
Wenkai Du 9e6c87a2bf Define ncclShmem as global shared (#618) 
* Use global defined shared memory

* Add --hipcc-func-supp to compile option

* Force inline some device functions

* Add back threadfence
 2022-09-20 09:00:20 -07:00
Edgar Gabriel 8f3219dbd4 make binary tree work on 2.13.4 2022-09-15 00:01:54 +00:00
Wenkai Du a06e14e39b Misc fixes and disable binTree 2022-09-14 00:26:19 +00:00
Edgar Gabriel e5d2dfed34 Update init.cc 2022-09-13 17:29:32 -05:00
Edgar Gabriel be935d7ce7 Merge branch 'develop' into 2.13.4 2022-09-13 17:19:04 -05:00
Edgar Gabriel 65e2ae20e5 add binary tree 
In addition, introduce the ability to have 2 trees at the same time.
Only for allreduce at the moment.
 2022-09-13 20:52:32 +00:00
Gilbert Lee 009e79623f Merge branch 'develop' into 2.13.4 2022-09-09 23:07:04 +00:00
gilbertlee-amd dd56135a9a Updating stream caching (#614) 
- Adding non-captured hipStream for use in setup
 2022-09-09 16:30:15 -06:00
Wenkai Du a79d9e3586 Merge remote-tracking branch 'nccl/master' into develop 2022-09-09 16:05:38 +00:00
Wenkai Du 7bbce085cc Enable LL128 protocol support (#605) 
* Enable LL128 protocol support

* Use shared memory object directly when possible
 2022-09-08 14:45:27 -07:00
Min Si 2b57751abb Fix compilation issues with buck (#610) 
* Fix compilation warning with -Wmisleading-indentation

When compile with -Wmisleading-indentation, it reports warning:
misleading indentation; statement is not part of the previous 'if'

This patch fixes it

* Avoid relative include file path

We don't need relative include file paths for src/graph/*.h
since src/ is already in CMake include_directories
 2022-09-07 09:56:05 -06:00
gilbertlee-amd 47b2fc3a30 Adding opt-in hipGraph support for RCCL via RCCL_ENABLE_HIPGRAPH (#608) 
Adding opt-in hipGraph support via RCCL_ENABLE_HIPGRAPH
 2022-09-06 10:29:46 -06:00
akolliasAMD 06bce9d0c9 added stream synch after hipMemset (#609) 2022-08-30 16:18:37 -06:00
Wenkai Du c9f2fe1f65 Use hipExtLaunchKernel when not using graph and not in group mode (#606) 2022-08-26 13:40:37 -07:00
Edgar Gabriel e739c62a53 Merge pull request #598 from edgargabriel/topic/tree-multirank 
Expand ncclTreeBasePostset for multi-rank
 2022-08-24 08:28:34 -05:00
Wenkai Du 88487a62bb Use non-temporal access for slow path (#602) 2022-08-23 08:21:51 -07:00
Edgar Gabriel 4141ec1151 fix channelcount for multi-rank scenario 2022-08-22 19:09:22 +00:00
akolliasAMD 3c1b1ec8c8 Simple tree changes (#599) 
changed treebase to create basic balanced tree
 2022-08-19 13:51:49 -06:00
Cosmic Fusion 080fc2d9d6 fix error: use of undeclared identifier 'free' 
include stdlib.h to fix compilation error in rccl :

[39/58] Building CXX object CMakeFiles/rccl.dir/src/misc/signals.cc.o
FAILED: CMakeFiles/rccl.dir/src/misc/signals.cc.o 
/opt/rocm/bin/hipcc -DENABLE_COLLTRACE -DHAVE_BFD -DHAVE_CPLUS_DEMANGLE -DUSE_ROCM_SMI64CONFIG -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -Drccl_EXPORTS -I/home/cosmo/build/flgrwqa/build/include -I/home/cosmo/build/flgrwqa/build/include/rccl -I/home/cosmo/build/flgrwqa/rccl/src -I/home/cosmo/build/flgrwqa/rccl/src/include -I/home/cosmo/build/flgrwqa/rccl/src/collectives -I/home/cosmo/build/flgrwqa/rccl/src/collectives/device -I/opt/hsa/include -fPIC -fvisibility=hidden -fgpu-rdc -parallel-jobs=8 -Wno-format-nonliteral -x hip --offload-arch=gfx803 --offload-arch=gfx900:xnack- --offload-arch=gfx906:xnack- --offload-arch=gfx908:xnack- --offload-arch=gfx90a:xnack- --offload-arch=gfx90a:xnack+ --offload-arch=gfx1030 -std=c++14 -MD -MT CMakeFiles/rccl.dir/src/misc/signals.cc.o -MF CMakeFiles/rccl.dir/src/misc/signals.cc.o.d -o CMakeFiles/rccl.dir/src/misc/signals.cc.o -c /home/cosmo/build/flgrwqa/rccl/src/misc/signals.cc
In file included from /home/cosmo/build/flgrwqa/rccl/src/misc/signals.cc:8:
/home/cosmo/build/flgrwqa/rccl/src/include/BfdBacktrace.hpp:138:9: error: use of undeclared identifier 'free'
        free(file->syms);
        ^
/home/cosmo/build/flgrwqa/rccl/src/include/BfdBacktrace.hpp:155:5: error: use of undeclared identifier 'free'
    free(file->syms);
    ^
 2022-08-19 20:25:06 +03:00
Wenkai Du 14b8ff153f Repurpose profiling implementation to simple timestamps tracing (#600) 2022-08-18 15:34:46 -07:00
Ching-Hsiang Chu e1d9b273b0 fix NCCL_DEBUG_FILE 
Summary: NCCL_DEBUG_FILE does not work properly since the recent v2.13.4 updates (https://github.com/NVIDIA/nccl/pull/682) because it nows sets `ncclDebugLevel` after parse `NCCL_DEBUG_FILE`. This patch move parsing `tempNcclDebugLevel` before processing `NCCL_DEBUG_FILE` to ensure `NCCL_DEBUG_FILE` is parsed only when `NCCL_DEBUG > NCCL_LOG_VERSION` (same as previous behavior)

Differential Revision: D38415208

fbshipit-source-id: 5689bbb798e73efb9e8594557666987f07e89a30
 2022-08-18 11:50:42 +02:00
Wenkai Du f5c0b243a8 Add XGMI sys type and clean up detection code (#597) 2022-08-12 09:52:29 -07:00
Ziyue Yang f6b9686482 Improve alignment and tuning for Pivot A2A algorithm (#593) 
* Improve alignment and tuning for Pivot A2A algorithm

* enable pivot a2a by default
 2022-08-05 19:40:19 -07:00
akolliasAMD 4cecdc9be5 minor latency tuning (#591) 
* minor tuning for tree ll
 2022-08-03 15:07:44 -06:00
Wenkai Du 9089e68a99 Revert "Use nontemporal in slow path and add XGMI sys type (#575)" (#590) 
This reverts commit b250c01cbe.
 2022-08-02 09:31:53 -07:00
akolliasAMD 254208e7dd Fixed issue with attomicEXCH creating errors on multi node runs (#587) 2022-07-22 11:32:49 -06:00
akolliasAMD 451c287aa6 Removing redundant LOAD and STORE on primitives plus adding some atomics (#585) 2022-07-21 13:04:57 -06:00
Hubert Lu 6dd090917a Merge pull request #580 from hubertlu-tw/develop 
Enhancement of RCCL logging information for topology-aware utilities
 2022-07-15 15:16:37 -07:00
Edgar Gabriel 2b1d5d3bc1 intercept SIGUSR2 in RCCL 
add support for intercepting SIGUSR2 in RCCL. This signal will
not terminate the execution of the application, but print the stacktrace
of the process that the signal was sent to instead.
 2022-07-15 16:28:46 +00:00
Ziyue Yang 77c2bef952 Remove redundant LOAD/STORE usage in LL initialization 2022-07-14 00:40:36 +00:00
hubertlu-tw a1842df858 Enhancement of RCCL logging information for topology-aware utilities 2022-07-11 19:01:10 +00:00
Sylvain Jeaugey 19ab67d172 2.13.4-1 
Optimize CUDA graph launch; avoid launching a CPU callback for
intra-node operations.
Simplify kernel common code to improve the latency of send/recv
operations.
Strengthen CUDA streams semantics.
Change NET API to v6, to add dmabuf support.
Add ncclGetLastError() function.
Add ncclRemoteError code and use it for remote network errors.
Support the use of a different NCCL_NET parameter per communicator.
Add support for SHM and P2P transfers using cudaMemcpy.
 2022-07-11 08:10:34 -07:00
Wenkai Du 8c3c8b78c0 Skip HDP cache flush for gfx90a (#578) 
* Skip HDP cache flush for gfx90a

* Remove extra debug print
 2022-07-08 10:13:32 -07:00