eba4e9e100
Merge pull request #742 from whchung/skip_done_event_msccl
...
Allow skipping doneEvent inside MSCCL.
2023-05-18 10:17:20 -05:00
403cda6322
Fix merge error ( #744 )
2023-05-18 08:09:27 -07:00
ca4a1dfd67
Address review feedbacks and make the flag be disabled by default.
2023-05-17 17:50:25 +00:00
12dba425de
Skip doneEvent inside MSCCL by default.
...
Added a RCCL_MSCCL_ENABLE_DONE_EVENT env var, set it be 0 by default.
The env var is to control whether to use doneEvent when invoking MSCCL
kernels.
Skipping doneEvent would cause the firmware to skip L2 cache flush,
resulting in overall performance improvement.
2023-05-17 16:49:42 +00:00
4ca7742c61
Revert "Ensure memory copy integrity during transport setup ( #731 )" ( #741 )
...
* Revert "Ensure memory copy integrity during transport setup (#731 )"
This reverts commit 36e453c61e
2023-05-16 10:34:47 -07:00
c88475462b
added modified npkit_trace_generator.py to scripts ( #738 )
...
* added modified npkit_trace_generator.py to scripts
2023-05-09 10:11:35 -06:00
8bb3340fcb
Skip checking of some settings in Cray OS ( #739 )
2023-05-09 07:59:56 -07:00
b6542c9b82
Merge pull request #735 from wenkaidu/nvls
...
Remove references to NVLS functions
2023-05-05 15:09:58 -07:00
897745a266
Remove references to NVLS functions
2023-05-05 07:55:20 -07:00
e21be4acaf
Merge pull request #732 from ROCmSoftwarePlatform/2.17.1
...
Sync up to NCCL 2.17.1
2023-05-02 08:30:36 -07:00
53a1f91857
Merge remote-tracking branch 'nccl/master' into develop
2023-04-25 15:38:32 -07:00
36e453c61e
Ensure memory copy integrity during transport setup ( #731 )
2023-04-25 14:41:43 -07:00
79df139f0b
Bump rocm-docs-core from 0.2.0 to 0.5.0 in /docs/.sphinx ( #728 )
...
Bumps [rocm-docs-core](https://github.com/RadeonOpenCompute/rocm-docs-core ) from 0.2.0 to 0.5.0.
- [Release notes](https://github.com/RadeonOpenCompute/rocm-docs-core/releases )
- [Changelog](https://github.com/RadeonOpenCompute/rocm-docs-core/blob/develop/CHANGELOG.md )
- [Commits](https://github.com/RadeonOpenCompute/rocm-docs-core/commits/v0.5.0 )
---
updated-dependencies:
- dependency-name: rocm-docs-core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-16 18:20:17 -06:00
a78ff46861
Standardizing documentation homepage message ( #726 )
2023-04-16 18:14:56 -06:00
9b7d5edbfc
Merge pull request #822 from KaimingOuyang/github/pytorch-hang-fix
...
Shutdown socket before close in ncclSocketClose()
2023-04-14 19:52:45 -07:00
4b09ffba43
msccl: print stack and memory usage ( #723 )
...
* msccl: print stack and memory usage
* Update number of kernels calculation
2023-04-14 14:59:03 -07:00
53c1c38f0e
Disabled hipgraph tests! ( #725 )
2023-04-13 17:42:05 -04:00
006b6bc7dc
Add a comment to shutdown() in ncclSocketClose
2023-04-13 09:13:44 -07:00
367e9b61c3
Shutdown socket before close in ncclSocketClose()
2023-04-13 09:11:52 -07:00
7289c05146
MSCCL: Fix memcpy bug ( #721 )
2023-04-11 14:46:53 -07:00
dc149a9fbd
pin rocm-docs-core and add dependabot config ( #722 )
2023-04-11 10:01:24 -06:00
2ce7d971e5
lessened the amount of child processes to active ones ( #720 )
2023-04-11 08:59:56 -06:00
27e0cb43c2
Unit test performance refactor ( #700 )
...
* Refactoring unit tests to improve performance
* Spawning child processes during InitComms instead of on TestBed construction
* Temporarily disabling graph unit tests
2023-04-06 12:28:53 -06:00
9fe5a349f1
added npkit_enable on CI tests ( #698 )
2023-04-05 08:05:23 -06:00
addbf4bd90
rccl-prim-test: minor update ( #718 )
2023-04-03 07:30:04 -07:00
c8e33b1232
fix msccl stream usage ( #717 )
2023-03-24 10:59:36 -07:00
caf7a3d47d
fixed jenkins LD_LIBRARY_PATH ( #714 )
2023-03-21 11:05:43 -06:00
00c3d8d850
Adding interactive mode for unit tests (UT_INTERACTIVE) ( #715 )
2023-03-21 10:58:24 -06:00
9a0d4a07a6
Test Fixes ( #710 )
...
* splitting CI tests in running SP first and MP second
* set device before hipStreamSynchronize on tests
2023-03-21 08:48:39 -06:00
b02fd04165
Fix unit test HIP graph error ( #712 )
2023-03-20 15:34:09 -07:00
8c56e6c892
Fix Docs static analysis ( #708 )
...
* remove ref to old script in static analysis
* update README with doc build instructions
update gitignore with doc artifacts
2023-03-16 13:12:43 -06:00
e3b2342f39
MSCCL: Improve executor and integrate scheduler ( #694 )
...
* MSCCL: improve executor and add scheduler for testing
* Use external scheduler
* Fix cmake error
* Address comments
* Fix thread safe issue
* Make MSCCL lifecycle APIs thread safe
* Make MSCCL internal scheduler aware of topology hint
* Revise error message
2023-03-14 14:34:25 -07:00
6e48e518d9
Standard template implementation ( #703 )
2023-03-13 11:00:57 -06:00
22b81fbaae
Fix XGMI detection ( #699 )
...
* Fix XGMI detection
* Increase stack size
* Temporarily disable signal hangler in CI
[Process: 17281] Inside handler function signal: Segmentation fault (11)
BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0)
BFD: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0)
2023-03-08 14:08:07 -08:00
79a2031951
Warn user on incorrect system settings ( #696 )
...
* Warn user on incorrect system settings
* Fix typo
* Add possible impact
* Ignore iommu settings in VM
2023-03-06 08:17:06 -08:00
5d3ab08b69
2.17.1-1
...
Add new NVLS algorithm for allreduce using NVLink SHARP (intra-node only).
Add new config options: cgaClusterSize, minCTAs, maxCTAs, netName.
Enable LL128 when we use PXN to close rings.
NVTX3 includes update.
Fix crash when one CollNet (SHARP) rail fails to initialize.
2023-03-01 00:39:04 -08:00
80ed608a9d
Multi stream unit test ( #693 )
...
* Adding multi-stream support to unit tests
2023-02-23 13:28:50 -07:00
d601c4909c
Merge pull request #685 from ROCmSoftwarePlatform/2.16.5
...
Sync up to NCCL 2.16.5
2023-02-22 10:29:02 -08:00
f63d3b1978
Adding UnitTest timing summary (UT_SHOW_TIMING) ( #692 )
2023-02-22 08:57:13 -07:00
d119c0886e
UnitTests: made reduceScatter run a smaller amount of tests ( #691 )
2023-02-21 16:21:24 -07:00
86e7b71234
Fix P2P scheduling ( #690 )
2023-02-21 07:49:54 -08:00
a640c6983f
Unit test fail check ( #689 )
...
* Adding fall-through on unit test failure
* Workaround for hipGraph validity check issue
2023-02-18 08:50:46 -08:00
1c166046a2
Add back __syncthreads() in barrier and adjust stack size ( #688 )
2023-02-18 08:50:31 -08:00
f4bf47f325
NPKit: improve clock calibration and fix GPU clock API ( #683 )
...
* Improve clock calibration in NPKit
* Improve gfx macro
* Fix macro
2023-02-17 12:26:57 -07:00
aee7b42bb8
Merge remote-tracking branch 'nccl/master' into HEAD
2023-02-14 17:14:13 -08:00
f7a456122c
Remove workaround and use indirect function call ( #684 )
2023-02-14 13:59:48 -08:00
9461a43168
Merge pull request #681 from wenkaidu/gfx9
...
Add HIP event optimization and remove special code for gfx90a
2023-02-13 08:04:59 -08:00
f525b8e1e6
Adding -pthread flag into CMakeLists.txt ( #682 )
...
Adding -pthread flag for linking issues into CMakeLists.txt
2023-02-10 17:22:30 -05:00
e05560ea82
Updated RCCL introduction at docs/source/library.rst ( #680 )
...
* Updated RCCL introduction at docs/source/library.rst
* space after period
---------
Co-authored-by: Saad Rahim <44449863+saadrahim@users.noreply.github.com >
2023-02-10 17:20:54 -05:00
39534e8724
Add HIP event optimization and remove special code for gfx90a
2023-02-10 16:46:01 +00:00
Wen-Heng (Jack) Chung
Wenkai Du
Wen-Heng (Jack) Chung
akolliasAMD
Wenkai Du
dependabot[bot]
Saad Rahim
David Addison
Pedram Alizadeh
Kaiming Ouyang
Ziyue Yang
Sam Wu
gilbertlee-amd
Ziyue Yang
Sylvain Jeaugey