354e0b29a6
Addressing possible out-of-bounds mem access during channel duplication ( #1193 )
2024-05-30 14:02:14 -06:00
73221b4230
Add ring simple chunk size tuning ( #1180 )
...
* Add ring simple chunk size tuning
* modifying the tuning table to improve the performance of broadcast for 8MB to 32MB for single-node MI300X after ring simple chunk size tuning
* modifying the tuning table to improve the performance of reduce for 1MB to 4MB for single-node MI300X after ring simple chunk size tuning
---------
Co-authored-by: PedramAlizadeh <pmohamma@amd.com >
2024-05-29 07:59:47 -07:00
8f099b1adb
Make WSL detection thread safe ( #1178 )
...
* Make WSL detection thread safe
* Change static to beginning
* Switch to use atomics
2024-05-28 17:23:50 -07:00
a78c4f5e88
Merge pull request #1175 from edgargabriel/topic/alt_rsmi
...
add alternative to rocm_smi_lib
2024-05-28 09:36:55 -05:00
eeea3b693b
Report error when collective is not enabled in build ( #1177 )
...
* Report error when collective is not enabled in build
* Fix typo
2024-05-16 10:11:12 -07:00
9ad913bfa8
add alternative to rocm_smi_lib
2024-05-14 13:51:41 -07:00
ecafc1969c
Support WSL2 ( #1173 )
2024-05-10 07:31:12 -07:00
87d01e6bf5
Changing the failure print to reflect a bad installation of MSCCL ( #1160 )
...
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com >
2024-05-08 16:56:26 -05:00
a64aab5f63
Use rocm-smi thread only mutex when available ( #1169 )
2024-05-08 14:32:24 -07:00
73acf3eeec
modifying the tuning table to improve the performance of broadcast for 1MB to 64MB for single-node MI300X ( #1172 )
2024-05-08 15:49:33 -04:00
408278209d
Adding ASAN changes to address memory leak issue" ( #1170 )
...
Co-authored-by: akolliasAMD <akollias@amd.com >
2024-05-08 09:16:00 -05:00
b18784d8b8
Add compiler warning for uninitialized variable and fix ( #1163 )
...
* Add compiler warning for uninitialized variable and fix
* Add -Wsometimes-uninitialized
* Convert warning to error
2024-05-08 07:00:25 -07:00
f679db6ff6
Use normal permute path when one NIC per GPU ( #1171 )
2024-05-08 06:59:57 -07:00
a0cef69110
npkit: add broadcast trace ( #1166 )
2024-05-07 14:00:16 -07:00
140b7dd40f
fix typo for DMABUF_ENABLE
2024-05-06 13:27:50 -07:00
b513c3970a
Bypass NVIDIA Ampere related tuning ( #1165 )
2024-05-03 17:57:16 -07:00
bb58b1c258
Fix ignore NUMA not being observed for NICs during model matching ( #1164 )
2024-05-03 16:42:07 -07:00
6f5a8ce1fb
Fix build error when roctracer-dev package is not installed ( #1161 )
2024-05-01 13:55:09 -07:00
4e1b8c1cbb
MSCCL: add support for out-of-place all reduce ( #1156 )
2024-04-28 19:49:09 -07:00
cd6e840e0b
Add back tree simple chunk size tuning ( #1157 )
2024-04-28 19:48:53 -07:00
b90436d292
[GRAPH] Reduce NCCL_TOPO_MAX_NODES to 64 ( #1153 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2024-04-27 23:41:11 -05:00
cc39e91c6f
Merge pull request #1158 from AtlantaPepsi/NPKit_fix
...
Prevent segfault from npkit-enabled rccl build
2024-04-26 12:44:04 -04:00
67246649ac
prevent segfault from npkit-enabled rccl build
...
Signed-off-by: AtlantaPepsi <timhu102@amd.com >
2024-04-26 10:54:27 -05:00
f330b82985
Revert "Use relaxed atomics for LL on GFX11 ( #859 )" ( #1148 )
...
This reverts commit 6a0a6a37d9
2024-04-26 07:49:55 -07:00
efe99057b0
SWDEV-455705: Fix an UB that could lead to miscompilation ( #1155 )
2024-04-25 10:10:01 -07:00
9e0c9b4ed8
Replace __HIP_PLATFORM_HCC__ with __HIP_PLATFORM_AMD__ ( #1154 )
2024-04-25 07:19:18 -07:00
e1a835910e
Merge remote-tracking branch 'nccl/master' into develop
2024-04-23 13:34:00 -07:00
220066197a
Use hipExtMallocWithFlags to allocate host memory on APU ( #1149 )
...
Also use SM60 as CUDA compatibility level.
2024-04-17 16:56:38 -07:00
4cb62f999a
Rail optimization for rings ( #1140 )
...
- Modifies the ring creation algorithm to be friendlier to rail-optimized topologies (should not affect classic fabric topologies)
2024-04-15 12:03:57 -06:00
428837ffe4
replacing rccl_bfloat16 with hip_bfloat16 ( #1126 )
...
Co-authored-by: mberenjk <mberenjk@amd.com >
2024-04-11 11:30:37 -05:00
0ce68f21d4
NPKit: doubling size of event buffers following MAXCHANNELS change ( #1135 )
2024-04-09 08:02:58 -07:00
137571fa01
Fix buffer overflow when parsing kernel cmdline ( #1133 )
2024-04-08 11:12:20 -07:00
5976f757dd
Remove hipEventDisableSystemFence ( #1122 )
...
There is no indication that disabling system fence has any latency improvement.
Removing it per recommendation from HIP.
2024-03-25 08:01:57 -07:00
9ef920a77b
[bugfix]save undo plans in some case
2024-03-12 00:00:16 +08:00
6777e65c1d
Enable fp8 support ( #1101 )
...
* initial checkin
* resolve cr comments
* resolve the build issue
* fix the data correctless issue
* update fp8 header file and update the unit test for fp8 support
* remove fp16 from fp8 headers
* fix ut issue and catch up the latest code from develop
* udate according to cr comments
* update ut according to cr comments
* update num floats for each SumPostDiv from 4 to 6
* update fp8 header file name
* fix the typo
2024-03-08 15:17:53 -08:00
ff951e607d
Improve debug messages of memory allocations ( #1107 )
2024-03-08 10:55:10 -08:00
77615cce28
msccl: fix scratch memory allocation after API change ( #1103 )
2024-03-06 11:11:04 -08:00
cbd955627e
Add support for using contiguous for GPU direct RDMA ( #1096 )
...
Enabled by env var RCCL_NET_CONTIGUOUS_MEM=1
2024-02-29 10:06:43 -08:00
df98a6957d
Add another Rome model ( #1095 )
2024-02-28 10:46:05 -08:00
b617aecc31
Implement ROCTX ( #1094 )
...
* Implement roctx
2024-02-27 15:46:15 -07:00
48bb7fec79
2.20.5-1
...
Fix UDS connection failure when using ncclCommSplit. Issue #1185
2024-02-26 02:52:39 -08:00
74f9e5db64
Add new GPU model ( #1080 )
2024-02-23 12:19:42 -08:00
c5ab37211b
Update RCCL/MSCCL work FIFO depth to 256K ( #1091 )
2024-02-21 17:15:11 -08:00
b275ed0b56
LL128 check if all XGMI ( #1089 )
2024-02-21 09:41:40 -07:00
2fb12a9358
Merge pull request #1079 from BertanDogancay/2.19.4-sync
...
2.19.4 Sync
2024-02-16 09:50:11 -07:00
bac57421c7
Allow bus id to be null ( #1085 )
...
* Allow bus id to be null
2024-02-15 16:36:51 -07:00
6f3310605c
Disable unsupported ld/st instructions
2024-02-15 13:58:16 -08:00
76f83f95ab
Merge remote-tracking branch 'rccl/develop' into 2.19.4
2024-02-15 13:37:14 -08:00
51003c9980
Use native half without conversion ( #1083 )
2024-02-13 16:57:34 -08:00
1f0af90206
Fix undefined symbol when nvtx is not enabled ( #1082 )
2024-02-13 14:03:43 -08:00
gilbertlee-amd
Wenkai Du
Edgar Gabriel
Edgar Gabriel
mberenjk
Pedram Alizadeh
paklui
Nilesh M Negi
Tim
AtlantaPepsi
Shilei Tian
BertanDogancay
FrankJ
Andy li
Bertan Dogancay
Sylvain Jeaugey
akolliasAMD