AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
324 Commits 3 Branches 0 Tags
5747c3cac172a7ce6a481f189fc53f3ee28681a8
Commit Graph

111 Commits

Author SHA1 Message Date
Wenkai Du 5747c3cac1 Fix abort handling in LL primitives 
[ROCm/rccl commit: 077c3cda74]
 2020-02-25 13:42:54 -08:00
Wenkai Du d640f38d56 Fix system maxSpeed and maxWidth calculation 
[ROCm/rccl commit: 9b80b3633f]
 2020-02-24 15:18:57 -08:00
Wenkai Du 93d448e2fe Fix incorrect CR8 detection 
Also change level of ring graph print to help debugging


[ROCm/rccl commit: f54dc58113]
 2020-02-21 10:09:49 -08:00
Wenkai Du cf4bce4ad3 Merge pull request #172 from wenkaidu/topo_expl 
Add topology explorer

[ROCm/rccl commit: 5b3856f2ed]
 2020-02-20 15:16:55 -08:00
Wenkai Du 00f421ccbd Add topology explorer 
[ROCm/rccl commit: 55f8e2dec7]
 2020-02-19 14:42:06 -08:00
Wenkai Du 8432e8a921 Generate 8G6L chordal ring from reference 
[ROCm/rccl commit: abcfbf1231]
 2020-02-11 22:01:12 +00:00
Wenkai Du 6b2d7de200 Add ring bandwidth correction factor 
[ROCm/rccl commit: d1dae2721d]
 2020-01-30 09:52:27 -08:00
Stanley Tsang e5419407c4 Updating copyright notices for 2020. 
[ROCm/rccl commit: 20fa04d9b6]
 2020-01-29 15:28:08 -08:00
Wenkai Du 622b49e80a Split primitive class to smaller structures 
[ROCm/rccl commit: 486fd436af]
 2020-01-29 15:27:23 -08:00
Wenkai Du d2fbcfea02 Misc fixes and improvements for 2.5.6 
1. Fix RCCL unit test
2. Add ROME detection and tuning
3. Change default P2P level
4. Fix search algorithm for XGMI
5. Remove explicit channel duplication with implicit by using half of link speed
6. Add collective trace support
7. Correct Intel Skylake CPU detection and bandwidth
8. Fix topo connect function
9. Disable GDR read and remove unreachable code
10. Disable LL128 kernels
11. Add tuning parameters
12. Use original clock64() implementation which returns RTC counter value
13. Print out timestamp of collective trace
14. Do not use struct ncclColl in kernel launch parameter
15. Fix abort handling and add tracing
17. Add __launch_bounds__ to kernel functions
18. Remove unused abortCount
19. Unset default MIN_NRINGS and MIN_NCHANNELS
20. Do not allocate shared memory when not using LL128 kernels
21. Correct time print out in tuning log


[ROCm/rccl commit: 1e55645d97]
 2020-01-29 15:27:05 -08:00
Wenkai Du d7d4175df0 Merge remote-tracking branch 'remotes/nccl/master' into rccl_2.5.6 
[ROCm/rccl commit: 6648c81dc6]
 2019-12-03 15:42:04 -08:00
Wenkai Du 4cb52294d5 Disable direct buffers to reduce scratch memory size 
[ROCm/rccl commit: a0be2b8812]
 2019-11-20 13:03:16 -08:00
Sylvain Jeaugey 71560fd67b 2.5.6-1 (#255) 
Add LL128 Protocol.

Rewrite the topology detection and tree/ring creation (#179). Improve
tree performance by sending/receiving from different GPUs. Add
model-based tuning to switch between the different algorithms and
protocols.

Rework P2P/SHM detection in containers (#155, #248).

Detect duplicated devices and return an error (#231).

Add tuning for GCP

[ROCm/rccl commit: 299c554dcc]
 2019-11-19 14:57:39 -08:00
Wenkai Du 1e182391ad Add bfloat16 support in RCCL 
Preprocessor symbol RCCL_BFLOAT16 is used as feature indicator


[ROCm/rccl commit: 5e109ed400]
 2019-11-18 13:45:53 -08:00
Wenkai Du 62042e47bc Correct RTC frequencies for profiling purpose 
[ROCm/rccl commit: 8995047830]
 2019-11-05 11:36:45 -08:00
Wenkai Du 41f6319b33 Check for fine grain support using memory allocation 
[ROCm/rccl commit: 669f1951a4]
 2019-11-01 15:58:49 -07:00
Jeff Daily e43e1f1b3d additional check for fine grain support in p2pCanConnect (#146) 
[ROCm/rccl commit: 5a502955c9]
 2019-10-31 08:58:38 -07:00
Wenkai Du d3f399f619 Disable HDP flush for RDMA 
[ROCm/rccl commit: 296176a4fd]
 2019-10-23 14:40:17 -07:00
Wenkai Du 21bc1ef493 Revert collective chunk and slice steps to avoid drop in throughput 
[ROCm/rccl commit: df74d12946]
 2019-10-18 12:54:00 -07:00
Gilbert Lee 7560929bd7 Reverting GenericOp bug workaround modifications to slice/chunk steps 
[ROCm/rccl commit: 37603ae6cb]
 2019-10-11 09:20:10 -07:00
Gilbert Lee cf597ff257 Performing __threadfence_system() with only first thread 
[ROCm/rccl commit: 1392dd2997]
 2019-10-11 09:16:19 -07:00
Gilbert Lee d257970ad1 Fix for GenericOp device primitive bug 
[ROCm/rccl commit: 8ae1bce3bb]
 2019-10-10 22:39:45 -07:00
Wenkai Du fbcdfd8348 Merge pull request #136 from wenkaidu/tree 
Enable tree kernels in build

[ROCm/rccl commit: 062c798c86]
 2019-10-09 10:58:52 -07:00
Wenkai Du f86ee41415 Enable tree kernels in build 
Need to tune and specify NCCL_TREE_THRESHOLD to allow usage


[ROCm/rccl commit: 76976c9e2e]
 2019-10-08 23:20:11 +00:00
Changpeng Fang d8a06589c9 Tuning the inline and unroll to reduce the scratch usage 
Summary:
 1. remove the noinline attribute for AllReduceThreeKernel;
 2. change AUTPUNROLL for tree functions to 1 or 2;
 Combining 1 and 2 will reduce the scratch usage from 1256 to 952


[ROCm/rccl commit: eec319038e]
 2019-10-08 14:02:25 -07:00
Wenkai Du 57dcac6afe Only generate kernels for sum and copy 
[ROCm/rccl commit: 61ef1dcad5]
 2019-09-24 17:01:12 -07:00
Gilbert Lee dbb7500fd1 RDMA HDP flush fix 
[ROCm/rccl commit: 86ce0a93b5]
 2019-09-06 16:35:55 +00:00
Gilbert Lee 2847ad9576 Revert "Set RDMA default to off state" 
This reverts commit 4afd6818ba.


[ROCm/rccl commit: 3e6b326a19]
 2019-09-05 18:16:53 +00:00
Wenkai Du daf2c4b200 Allocate opCount in pinned host memory for P2P transport 
To avoid remote P2P read access when checking remote GPU's opCount


[ROCm/rccl commit: 8c975353ed]
 2019-08-29 10:22:09 -07:00
Wenkai Du 4afd6818ba Set RDMA default to off state 
[ROCm/rccl commit: 0f16ad966a]
 2019-08-26 10:59:33 -07:00
Wenkai Du 4df1defc3b Merge pull request #125 from wenkaidu/fix_nvml_id 
Assign unused nmvlDev to avoid random number

[ROCm/rccl commit: 6759660529]
 2019-08-19 09:08:13 -07:00
Wenkai Du 04cd446d89 Assign unused nmvlDev to avoid random number 
[ROCm/rccl commit: 86efdfc3b5]
 2019-08-16 16:34:14 -07:00
Wenkai Du 60989a3fc9 Merge remote-tracking branch 'remotes/nccl/master' into HEAD 
[ROCm/rccl commit: 7c38da0939]
 2019-08-16 16:13:34 -07:00
Wenkai Du 7396d5c3ba Tune AUTOUNROLL for better performance 
Also remove all unused UNROLL defines


[ROCm/rccl commit: 1faededc03]
 2019-08-16 10:34:53 -07:00
Michael LIAO f4a240065f Fix build with hip-clang. 
- Add necessary function attribute for HIP programming model.
- Explicitly include hsa headers.


[ROCm/rccl commit: 9369f8d75d]
 2019-08-15 14:56:04 -04:00
Wenkai Du d4862fa605 Tune LL threshold for VEGA 
Also move abort check after SPINS_BEFORE_CHECK_ABORT as NCCL


[ROCm/rccl commit: 2223cccf15]
 2019-08-15 09:16:11 -07:00
Wenkai Du 93c44e96cb Default to minimal 2 rings and improve LL loop 
[ROCm/rccl commit: 4b77a16f3f]
 2019-08-14 14:12:56 -07:00
Wenkai Du 1feef99e7d Remove duplicate line 
[ROCm/rccl commit: 5782a8d857]
 2019-08-14 13:22:43 -07:00
Wenkai Du 6047487815 RCCL 2.4 update 
[ROCm/rccl commit: f11c8f60cd]
 2019-08-14 10:42:35 -07:00
David Addison d57c0b0f92 Updated PR#196 to use a common hash function 
[ROCm/rccl commit: fad079a8ae]
 2019-08-14 10:08:39 -07:00
David Addison bb5b11fa23 Merge branch 'shm' of git://github.com/lowintelligence/nccl into lowintelligence-shm 
[ROCm/rccl commit: 01d1836668]
 2019-08-14 09:45:45 -07:00
David Addison c7957daee3 Make use of SO_REUSEPORT conditional 
Fixes: #244

SO_RESUEPORT was introduced in Linux 3.9 and later.
This change allows NCCL to compile against older releases.

The functionality is only required if the user is specifying
a NCCL bootstrap address via an environment variable.


[ROCm/rccl commit: 7f2b337e70]
 2019-08-13 16:32:07 -07:00
Ke Wen a66ab68630 Fix NIC distances for 11+ NICs 
[ROCm/rccl commit: 4d579e51cc]
 2019-07-17 06:32:33 -07:00
Ke Wen 5c5c58c73b Fix #224: prevent number of IB devices from going out of bound 
[ROCm/rccl commit: 920ae57c14]
 2019-07-17 06:32:33 -07:00
Ke Wen 4211da6d29 Size up IPC buffers to multiples of 2MB 
Avoid potential CUDA error in concurrent communicator initialization


[ROCm/rccl commit: c8c68fb5f7]
 2019-07-12 09:50:17 -07:00
Hirochika Asai ee08e8b421 Add the exact matching modifier support "=" to the NCCL_IB_HCA variable (#236) 
Perform exact matching when the prefix "=" is specified in the NCCL_IB_HCA variable to exclude HCAs mlx5_X[0-9]+ when mlx5_X is specified.

[ROCm/rccl commit: 0b192d2299]
 2019-07-09 14:45:41 -07:00
Ke Wen 3c13a4d1bb Merge branch 'master' into HEAD 
[ROCm/rccl commit: 8e04d80382]
 2019-06-25 13:39:08 -07:00
Ke Wen b91d8170f8 2.4.8-1 
Fix #209: improve socket transport performance
  Split transfers over multiple sockets
  Launch multiple threads to drive sockets
  Detect AWS NICs and set nsockets/nthreads accordingly


[ROCm/rccl commit: 7c72dee660]
 2019-06-25 13:22:47 -07:00
Felix Abecassis d2f579ba8b Fix out-of-bounds read in ncclStrToCpuset (#233) 
The affinityStr string was not null-terminated but was passed to strlen(3).

Signed-off-by: Felix Abecassis <fabecassis@nvidia.com>

[ROCm/rccl commit: 37e4f8729e]
 2019-06-21 10:25:08 +02:00
David Addison 00bc3a724a NCCL 2.4.7-1 
Performance tweaks for PowerPC builds only;
      Set default NCCL_MIN_NRINGS to 4
      Disable PCI-E NUMA distance detection


[ROCm/rccl commit: 0ceaec9cee]
 2019-05-10 13:52:16 -07:00