Граф коммитов

580 Коммитов

Автор SHA1 Сообщение Дата
Wenkai Du 17491c918e Fix incorrect net counting (#339)
* Fix incorrect net counting

* Add comments
2021-04-05 12:21:57 -07:00
Wenkai Du 1d2946ee4b Rework network port trimming code (#338)
* Rework network port trimming code

* Move Rome related changes to separate source files
2021-03-31 10:25:59 -07:00
Wenkai Du 0c78553ee0 Check fine grained memory on peer GPU before enabling P2P (#337) 2021-03-30 09:06:39 -07:00
Wenkai Du d87dc7c2e8 collnet: support multiple NICs (#335) 2021-03-25 20:59:32 -07:00
gilbertlee-amd 69c0b44553 Updating CHANGELOG.md for ROCm 4.1.0 (#336) 2021-03-25 21:44:24 -06:00
Stanley Tsang 289db2a636 Fixing message queue leak. (#331) 2021-03-25 19:11:43 -06:00
Wenkai Du 0fbb9510a5 Remove HDP workaround for ROCm 4.2 HIP (#334) 2021-03-23 20:11:37 -07:00
Wenkai Du 1d6244b18d Enable collnet in RCCL (#333)
* Enable CollNet and use different number of channels

* topo_expl: enable collnet
2021-03-19 12:58:13 -07:00
Wenkai Du 244e25d980 Hide non-public symbols from library (#332)
* Hide non-public symbols from library

* Move flag outside of parallel-jobs check
2021-03-18 18:08:08 -07:00
Wenkai Du b46260260a Sort GPUs by HIP device ID (#329)
* Sort GPUs by HIP device ID

* Remove extra space
2021-03-16 16:51:32 -07:00
Wenkai Du f60b76c67a Add GPU memory usage tracker (#326) 2021-03-06 20:32:30 -08:00
Wenkai Du 8e180cf087 Revert "Port alltoall[v]" (#325)
This reverts commit f4d5d3d620.
2021-03-06 13:59:31 -08:00
Wenkai Du c018edf0f2 Enable local sendrecv over network if GDR is available on all GPUs (#324) 2021-03-05 19:59:41 -08:00
gilbertlee-amd f4a9b9acba Adding pthread_join / pthread_detach to clean up pthreads to avoid leaks (#322) 2021-02-26 16:29:55 -07:00
Wenkai Du 7191d0959b Merge pull request #320 from wenkaidu/nbio
Match NBIO only when GPUs and NICs are directly connected to CPU
2021-02-25 17:49:09 -08:00
Stanley Tsang 5d3b1fa4d0 Temporarily disabling multiprocess unit tests 2021-02-25 23:59:44 +00:00
Eiden Yoshida 25a7dad765 Add ubuntu gfx908 target to CI 2021-02-24 19:01:56 -07:00
Wenkai Du e820a943e9 Update tuning parameters for XGMI and NET 2021-02-23 21:41:26 +00:00
Wenkai Du ec8d89b1dd Match NBIO only when GPUs and NICs are directly connected to CPU 2021-02-22 18:52:29 -05:00
Stanley Tsang 61d66c8e18 Do not search for GTest if we're force installing it into build directory (#318) 2021-02-22 15:22:01 -07:00
Wenkai Du a058d5639e Merge pull request #319 from wenkaidu/2.8.3
Update to 2.8.4
2021-02-22 10:17:38 -08:00
Stanley Tsang 45f5255f7c Fixing cache deletion for CliqueManager; updating copyright 2021-02-19 22:22:46 +00:00
Stanley Tsang 6a568a8652 Make CI only run subset of unit tests. 2021-02-19 21:37:51 +00:00
Cory Bloor 8aea5edb29 Add Jenkins docs build (#18)
* Fix typo in copyright

* Minor README improvements

- Prevent underscores from being interpreted as italics in test name format.
- Switch URL to HTTPS.

* Update docs scripts config

- Allow run_doc.sh and run_doxygen.sh to be called from any directory.

* Add docs build to Jenkins
2021-02-18 16:37:37 -07:00
Wenkai Du 95f178324c Add support to another Rome model 2021-02-18 02:00:31 +00:00
Wenkai Du c985358e11 Merge remote-tracking branch 'nccl/master' into 2.8.3 2021-02-15 18:44:47 -05:00
Wenkai Du 3a1aebd742 Merge remote-tracking branch 'rccl/develop' into 2.8.3 2021-02-15 13:17:38 -05:00
Wenkai Du bf8eb40705 Move HDP flush to CPU 2021-02-12 18:06:19 +00:00
pramenku e9f7908592 Update install.sh (#317)
* Update install.sh

Install.sh having hard code like /opt/rocm/bin/hipcc for rocm_path and default_path=/opt/rocm
This will work only when we have standalone rocm installed. If anyone has installed, side-by-side, they will face below error.

Can we keep like ROCM_PATH=$ROCM_PATH  instead of “default_path” as variable name and 
ROCM_BIN_PATH=$ROCM_PATH/bin ,rocm_path can be replaced with ROCM_BIN_PATH.

This way, we will have option to export ROCM_PATH as env variable as per need and use the script. 
I have also tried locally, it’s working.  ROCM_PATH is common variable name, we are having.

If you are ok, I can also submit the PR for the same.


Error when side-by-side install is done for driver.
# ./install.sh -dtr 2>&1 | tee /dockerx/6519_rccl-test.log
CMake Error at /usr/share/cmake/Modules/CMakeDetermineCXXCompiler.cmake:48 (message):
Could not find compiler set in environment variable CXX:
/opt/rocm/bin/hipcc.
Call Stack (most recent call first):
CMakeLists.txt:12 (project)

CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
See also "/root/driver/rccl/build/release/CMakeFiles/CMakeOutput.log".

* Update install.sh

Removed ROCM_PATH=$ROCM_PATH

* Update install.sh

Set default value if external value is not supplied.
2021-02-12 08:44:30 -08:00
Stanley Tsang 6b7b312fb9 Fixed temp file creation/deletion with clique mode (#316) 2021-02-12 08:44:10 -08:00
Sylvain Jeaugey 911d61f214 2.8.4-1
Fix hang in corner cases of alltoallv using point to point send/recv.
Harmonize error messages.
Fix missing NVTX section in the license.
Update README.
2021-02-09 15:36:48 -08:00
Gilbert Lee f1a9ce3fa5 Using GTEST_SKIP() to skip unit tests that have insufficient devices. Skipping out earlier 2021-02-09 03:54:04 +00:00
Wenkai Du 9cc3b56166 Fix GDRDMA read and remove unused files 2021-02-09 01:34:39 +00:00
Stanley Tsang d00b7d17bd Update MP UT to support arbitrary # of GPUs; multiple bugfixes (#16)
* Fixing temp file creation/deletion for Clique kernel mode.

* Refactoring of MP unit tests; include bugfixes and general support for any number of GPUs

* GroupCall MP UT properly quits when too many devices specified

* MP UT will programmatically set NCCL_COMM_ID if not specified; updated install script
2021-02-05 16:49:25 -08:00
Wenkai Du 6dfdfef98f Add gfx908 Rome 4 NICs model 2021-02-06 00:19:47 +00:00
Gilbert Lee f372c53d52 [TransferBench] Fixing some merge issues 2021-02-05 16:46:20 +00:00
Wenkai Du ab1e7a0318 Merge remote-tracking branch 'origin/develop' into 2.8.3 2021-02-04 20:02:34 -05:00
Gilbert Lee 2f541508c5 [topo_expl] Updating for 2.8.3 2021-02-04 19:08:42 +00:00
Gilbert Lee 9aac1ed38f [ib-test] Update for 2.8.3] 2021-02-04 19:05:03 +00:00
Gilbert Lee 9ce203dd0a [TransferBench] Updating for 2.8.3 2021-02-04 18:58:25 +00:00
gilbertlee-amd 1990ffd76a Tuning some clique-based kernel parameters (#315) 2021-02-03 20:00:08 -07:00
Wenkai Du 5f97122442 Enable GPU direct RDMA read from GPU 2021-02-03 02:48:30 +00:00
gilbertlee-amd 62e0447e9a [TransferBench] Restore some previous fixes - memory leak, PCIe address (#314) 2021-02-01 09:48:09 -07:00
Gilbert Lee 01a998b17c Removing in-place tests from Combined calls (no support for send/recv) 2021-01-28 20:09:03 +00:00
gilbertlee-amd 3e62ceddc5 Clique kernel support (#295) (#15)
* Adding experimental clique-based kernels (opt-in only)

Co-authored-by: Stanley Tsang <stanley.tsang@amd.com>
Co-authored-by: Gilbert Lee <gilbert.lee@amd.com>
Co-authored-by: Wenkai Du <43822138+wenkaidu@users.noreply.github.com>

Co-authored-by: Stanley Tsang <stanley.tsang@amd.com>
Co-authored-by: Wenkai Du <43822138+wenkaidu@users.noreply.github.com>
2021-01-28 09:45:01 -07:00
Wenkai Du 41e47a36e7 Use less unroll for clique kernels (#313) 2021-01-15 17:48:10 -08:00
Stanley Tsang d3fa257682 Adding multiprocess unit tests (#312)
Adding multiprocess unit tests for collectives.  

To run, NCCL_COMM_ID=$HOSTNAME:12345 build/release/test/UnitTestsMultiProcess
2021-01-15 16:34:36 -07:00
Wenkai Du 2ddbe6646b Improve collective trace 2021-01-14 19:28:01 -05:00
Wenkai Du b33a2cac8b gtest: add scatter to combined calls and use loops (#303)
* gtest: add scatter to combined calls and use loops

* gtest: run validation inside loop

* gtest: revert small element count to 2520

* gtest: fix memory leak in validation

(cherry picked from commit b0853ccd51)

* Fix combined call UT

* Fix memory leak

* Fix alltoallv test
2021-01-14 19:28:01 -05:00
Wenkai Du f4d5d3d620 Port alltoall[v] 2021-01-14 19:28:01 -05:00