Cosmic Fusion
1dcf1da5ca
fix error: use of undeclared identifier 'free'
...
include stdlib.h to fix compilation error in rccl :
[39/58] Building CXX object CMakeFiles/rccl.dir/src/misc/signals.cc.o
FAILED: CMakeFiles/rccl.dir/src/misc/signals.cc.o
/opt/rocm/bin/hipcc -DENABLE_COLLTRACE -DHAVE_BFD -DHAVE_CPLUS_DEMANGLE -DUSE_ROCM_SMI64CONFIG -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -Drccl_EXPORTS -I/home/cosmo/build/flgrwqa/build/include -I/home/cosmo/build/flgrwqa/build/include/rccl -I/home/cosmo/build/flgrwqa/rccl/src -I/home/cosmo/build/flgrwqa/rccl/src/include -I/home/cosmo/build/flgrwqa/rccl/src/collectives -I/home/cosmo/build/flgrwqa/rccl/src/collectives/device -I/opt/hsa/include -fPIC -fvisibility=hidden -fgpu-rdc -parallel-jobs=8 -Wno-format-nonliteral -x hip --offload-arch=gfx803 --offload-arch=gfx900:xnack- --offload-arch=gfx906:xnack- --offload-arch=gfx908:xnack- --offload-arch=gfx90a:xnack- --offload-arch=gfx90a:xnack+ --offload-arch=gfx1030 -std=c++14 -MD -MT CMakeFiles/rccl.dir/src/misc/signals.cc.o -MF CMakeFiles/rccl.dir/src/misc/signals.cc.o.d -o CMakeFiles/rccl.dir/src/misc/signals.cc.o -c /home/cosmo/build/flgrwqa/rccl/src/misc/signals.cc
In file included from /home/cosmo/build/flgrwqa/rccl/src/misc/signals.cc:8:
/home/cosmo/build/flgrwqa/rccl/src/include/BfdBacktrace.hpp:138:9: error: use of undeclared identifier 'free'
free(file->syms);
^
/home/cosmo/build/flgrwqa/rccl/src/include/BfdBacktrace.hpp:155:5: error: use of undeclared identifier 'free'
free(file->syms);
^
[ROCm/rccl commit: 080fc2d9d6 ]
2022-08-19 20:25:06 +03:00
Wenkai Du
c2e9ada40b
Repurpose profiling implementation to simple timestamps tracing ( #600 )
...
[ROCm/rccl commit: 14b8ff153f ]
2022-08-18 15:34:46 -07:00
Wenkai Du
6c3f1366e8
Add XGMI sys type and clean up detection code ( #597 )
...
[ROCm/rccl commit: f5c0b243a8 ]
2022-08-12 09:52:29 -07:00
Ziyue Yang
478d8312b8
Improve alignment and tuning for Pivot A2A algorithm ( #593 )
...
* Improve alignment and tuning for Pivot A2A algorithm
* enable pivot a2a by default
[ROCm/rccl commit: f6b9686482 ]
2022-08-05 19:40:19 -07:00
akolliasAMD
6fb5c5d5e3
minor latency tuning ( #591 )
...
* minor tuning for tree ll
[ROCm/rccl commit: 4cecdc9be5 ]
2022-08-03 15:07:44 -06:00
Wenkai Du
f70830d629
Revert "Use nontemporal in slow path and add XGMI sys type ( #575 )" ( #590 )
...
This reverts commit e04bba619a .
[ROCm/rccl commit: 9089e68a99 ]
2022-08-02 09:31:53 -07:00
akolliasAMD
d5ca0be51f
Fixed issue with attomicEXCH creating errors on multi node runs ( #587 )
...
[ROCm/rccl commit: 254208e7dd ]
2022-07-22 11:32:49 -06:00
akolliasAMD
18d9fd1b8f
Removing redundant LOAD and STORE on primitives plus adding some atomics ( #585 )
...
[ROCm/rccl commit: 451c287aa6 ]
2022-07-21 13:04:57 -06:00
Hubert Lu
088d62ff58
Merge pull request #580 from hubertlu-tw/develop
...
Enhancement of RCCL logging information for topology-aware utilities
[ROCm/rccl commit: 6dd090917a ]
2022-07-15 15:16:37 -07:00
Edgar Gabriel
a9e0333dba
intercept SIGUSR2 in RCCL
...
add support for intercepting SIGUSR2 in RCCL. This signal will
not terminate the execution of the application, but print the stacktrace
of the process that the signal was sent to instead.
[ROCm/rccl commit: 2b1d5d3bc1 ]
2022-07-15 16:28:46 +00:00
Ziyue Yang
e1aae026bf
Remove redundant LOAD/STORE usage in LL initialization
...
[ROCm/rccl commit: 77c2bef952 ]
2022-07-14 00:40:36 +00:00
hubertlu-tw
e13eb2eab9
Enhancement of RCCL logging information for topology-aware utilities
...
[ROCm/rccl commit: a1842df858 ]
2022-07-11 19:01:10 +00:00
Wenkai Du
c129677fe0
Skip HDP cache flush for gfx90a ( #578 )
...
* Skip HDP cache flush for gfx90a
* Remove extra debug print
[ROCm/rccl commit: 8c3c8b78c0 ]
2022-07-08 10:13:32 -07:00
Wenkai Du
659cd52d5c
Add more constraints to enable GDR ( #579 )
...
* Add more constraints to enable GDR
* Revert deleted line
[ROCm/rccl commit: aa0d7ca882 ]
2022-07-08 09:52:27 -07:00
Yifan Xiong
bf15ad1d72
Reduce AlltoAll port usage in send/recv proxy ( #577 )
...
* Reduce AlltoAll port usage when connecting proxy
Reuse socket ports when connecting proxies in AlltoAll.
Existing port usage in AlltoAll is O(n) for recv and O(n) for send,
reusing socket ports in server or client side will make one of them
O(1), reusing both will reduce the total port usage to O(1) and enables
AlltoAll in >64 MI200 nodes.
* Update changelog accordingly
Update changelog accordingly.
[ROCm/rccl commit: 80f53cc171 ]
2022-07-07 16:15:52 -07:00
Wenkai Du
e04bba619a
Use nontemporal in slow path and add XGMI sys type ( #575 )
...
* Use nontemporal in slow path and add XGMI sys type
* Clean up XGMI detection
[ROCm/rccl commit: b250c01cbe ]
2022-07-06 07:58:41 -07:00
Wenkai Du
2f4aea93e0
Fix GPU to NIC mapping in tree ( #573 )
...
* Fix GPU to NIC mapping in tree
* Update tuning table
[ROCm/rccl commit: 00af1f64e9 ]
2022-07-03 20:52:52 -07:00
gilbertlee-amd
cb5ae7224e
Adding git hash info to version output line ( #572 )
...
[ROCm/rccl commit: a89a9966aa ]
2022-06-28 16:42:51 -06:00
Wenkai Du
915a9d3934
Do not set NET GDR level automatically ( #571 )
...
[ROCm/rccl commit: 9a285b5e1d ]
2022-06-23 16:28:28 -07:00
Wenkai Du
784b12bf75
Use different atomics to check flags in kernel ( #568 )
...
[ROCm/rccl commit: c3bb9e70d0 ]
2022-06-23 09:16:41 -07:00
Ziyue Yang
2b418b5dee
Add Feature - Add NPKit Support in RCCL ( #564 )
...
* apply npkit
* fix bug
* add npkit in readme
[ROCm/rccl commit: 6e93fafdc3 ]
2022-06-20 14:30:19 -07:00
Wenkai Du
0fb000932f
Change default nchannels per peer ( #563 )
...
[ROCm/rccl commit: f274c865c1 ]
2022-06-13 06:39:05 -07:00
Edgar
8953f5b5ca
Introduce multi-rank support per device.
...
This is a single commit of the source code changes required to
introduce support for multiple ranks per device.
A new interface (ncclCommRankInitMulti) has to be used to make use of
this new feature.
[ROCm/rccl commit: 0336ffdf70 ]
2022-06-10 14:23:12 +00:00
Wenkai Du
11a6cdd52f
Fix P2P scheduling ( #560 )
...
[ROCm/rccl commit: 5cb2aca3d9 ]
2022-06-06 13:32:28 -07:00
Wenkai Du
f2dbc77afe
Enable timing profile option ( #558 )
...
[ROCm/rccl commit: 7a6c6927ae ]
2022-06-03 07:05:13 -07:00
Aristotelis
0b55e01ef3
Merge remote-tracking branch 'ncclRepo/master' into develop
...
[ROCm/rccl commit: e0864e7093 ]
2022-06-02 15:27:24 +00:00
Wenkai Du
1e36b432f1
Revert chunksteps changes ( #555 )
...
[ROCm/rccl commit: eef812bed7 ]
2022-05-31 14:45:51 -07:00
Wenkai Du
5becf1669f
Add another Rome model ( #553 )
...
* Add another Rome model
* Add option to force enable intranet on single node
* Limit p2p channels to number of ranks
* Refine p2p channels handling
[ROCm/rccl commit: ef499c4810 ]
2022-05-31 11:31:30 -07:00
akolliasAMD
a03ab8e752
code cleanup ( #554 )
...
[ROCm/rccl commit: a0a686e74c ]
2022-05-31 09:59:36 -04:00
Wenkai Du
2c125ce6ed
Update Rome model ( #552 )
...
[ROCm/rccl commit: c5b77121f0 ]
2022-05-26 09:59:23 -07:00
akolliasAMD
22dc8bd246
Added creation of new tree and added switch for using treesplit for specific cases ( #551 )
...
[ROCm/rccl commit: 98f0809a39 ]
2022-05-25 18:55:14 -04:00
gilbertlee-amd
a2a4888497
Moving opt-in custom signal handler from UnitTests into RCCL ( #550 )
...
* Enable via RCCL_ENABLE_SIGNALHANDLER=1
[ROCm/rccl commit: 700b473211 ]
2022-05-20 09:56:38 -06:00
Wenkai Du
86e8797602
Add switch for pivot alltoall kernel ( #549 )
...
[ROCm/rccl commit: 6707a270b1 ]
2022-05-17 18:14:04 -07:00
Wenkai Du
b30b8becea
Refine and add new Rome models ( #548 )
...
[ROCm/rccl commit: 283dc86a73 ]
2022-05-17 08:23:59 -07:00
Sylvain Jeaugey
1c5734046d
2.12.12-1
...
Improve allreduce performance when we have more than one network interface per
GPU and we need to use PXN to close rings.
Add support for PCI Gen5 on 5.4 kernels.
Fix crash when setting NCCL_SET_THREAD_NAME.
Fix random crash in init due to uninitialized struct.
Fix hang on cubemesh topologies.
Add P2P_DIRECT_DISABLE parameter to disable direct access to pointers within a
process.
[ROCm/rccl commit: 7aa1c46fd5 ]
2022-05-13 00:26:57 -07:00
Wenkai Du
b37180b7ed
Improve LL performance ( #546 )
...
* Improve LL performance
* Add split barriers for LL
[ROCm/rccl commit: c9919e0e35 ]
2022-05-10 13:32:10 -07:00
Wenkai Du
347ea354c2
Update tuning parameters
...
[ROCm/rccl commit: 83fd4f70e7 ]
2022-04-18 16:04:04 -07:00
Wenkai Du
67e7e6507e
Merge remote-tracking branch 'nccl/master' into develop
...
[ROCm/rccl commit: d28e1cb44f ]
2022-04-18 11:15:25 -07:00
Wenkai Du
58b2c1ec9c
Fix random segfault ( #537 )
...
[ROCm/rccl commit: fd2f1b3b88 ]
2022-04-15 14:32:11 -07:00
Wenkai Du
011447e4dc
Add new Rome model ( #536 )
...
[ROCm/rccl commit: 2151c79d14 ]
2022-04-13 11:45:40 -07:00
Wenkai Du
f8023f2e07
Add new Rome model ( #535 )
...
[ROCm/rccl commit: ba4c165bf3 ]
2022-04-12 13:27:32 -07:00
Sylvain Jeaugey
e89ff21d35
Update Makefile to install static library.
...
Make sure make install also installs the static library.
Fixes #662
[ROCm/rccl commit: 9bfc1c6e35 ]
2022-04-08 14:00:43 +02:00
Wenkai Du
5ccdd9f5e1
Increase chunk steps of broadcast and reduce ( #528 )
...
[ROCm/rccl commit: 15b572751e ]
2022-04-07 13:34:04 -07:00
Colin Smith
3830310156
Doxygen fix for ncclRecv ( #527 )
...
Changed the Doxygen command for ncclRecv and pncclRecv, to be consistent with other APIs.
[ROCm/rccl commit: b2ffcf6d89 ]
2022-04-05 14:07:56 -07:00
Wenkai Du
9884e61367
Add tuning model ( #523 )
...
[ROCm/rccl commit: 5cc0a405c0 ]
2022-04-04 10:19:57 -07:00
Wenkai Du
3332cdff07
Support multiple tuning tables ( #522 )
...
* Support multiple tuning tables
* [UnitTests] Skip managed memory testing
[ROCm/rccl commit: bbe780ca6c ]
2022-03-31 17:09:21 -07:00
Sylvain Jeaugey
74f8baa0f3
Merge remote-tracking branch 'origin/master'
...
[ROCm/rccl commit: 8133784b32 ]
2022-03-30 02:29:05 -07:00
Sylvain Jeaugey
27130280b2
2.12.10-1
...
Fix bug with CollNet
Fix bug with zero-bytes send/recv operations
Fix NCCL_PARAM implementation to avoid taking a lock on every call
Fix bug when setting NCCL_IB_QPS_PER_CONNECTION to more than one.
Improve error reporting for network errors.
[ROCm/rccl commit: 353e8ba446 ]
2022-03-30 02:27:01 -07:00
Sylvain Jeaugey
a52e328ba4
Fix merging error
...
[ROCm/rccl commit: 2247152a8e ]
2022-03-30 02:14:32 -07:00
Sylvain Jeaugey
3bc2e34df2
Merge branch 'master' into truncated_msg_warning
...
[ROCm/rccl commit: 2dfd83752c ]
2022-03-30 10:58:05 +02:00