0c36d571ea
Enable multi-threading for MSCCL ( #1203 )
...
MSCCL can now run in a multi-threaded configuration. To test in the unit tests, added the ENABLE_OPENMP compile definition flag and the --openmp-test-enable flag to the unit test build script. To activate, set the environment variables UT_MULTITHREADED=1 and UT_PROCESS_MASK=1. Set Jenkins to use this mode.
2024-07-04 09:34:38 -06:00
45f3fbc52f
Checking kernel header files only when missing sysfs entry ( #1239 )
2024-07-03 15:53:15 -07:00
5be3b713ef
[GRAPH] Use channel shuffling only for IB systems ( #1228 )
...
* [GRAPH] Use channel shuffling only for IB systems
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
* [GRAPH] Define channels=48 for gfx94 RoCE systems
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
* [GRAPH] Increase channels for RoCE gfx94 systems
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
---------
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2024-07-02 12:20:40 -05:00
9d8f68b4ee
NPKit: separate time stamps for GPU access from different blocks ( #1229 )
...
To avoid races in memory access in GPU
2024-06-28 08:00:22 -07:00
b09ea29d66
graph: fix minNchannels for multi-node overwrite ( #1230 )
2024-06-26 16:56:10 -05:00
ad31d93f3d
Revert "Changing channel stride for MI300X multinode ( #1196 )" ( #1224 )
...
This reverts channel stride change in commit
0948eecbba
2024-06-25 14:03:30 -07:00
e170f41ddd
Unit Tests for testing channels ( #1222 )
2024-06-25 10:10:10 -05:00
9f2514e5c8
Merge pull request #1223 from nusislam/minNchannels-multinode
...
graph: fix minNchannels for multi-node
2024-06-25 10:03:35 -05:00
5d7078e383
Fix DMABUF support ( #1218 )
...
* Fix DMABUF support
* Reduce log output by moving dmabuf allocation details to TRACE
* Enable peer memory GDR support if ib_umem_get_peer is in kernel
2024-06-25 08:00:15 -07:00
05df0f8cea
graph: fix minNchannels for multi-node
...
Multi-node rccl was not correctly setting the minNchannels value. This
PR fixes the bug.
2024-06-24 16:42:44 -05:00
435756af02
fix initOnceFunc setting incorrect result code ( #1205 )
...
Addresses DMA-BUF support check unexpectedly failing
2024-06-07 16:47:19 -07:00
9660e2e2dc
Merge pull request #1200 from nusislam/multi-node-256-fix
...
graph: fix multi-node channel count
2024-06-07 14:34:20 -05:00
d9661c17e6
Fix min_nchannels bug for gfx94* nranks=4 ( #1202 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2024-06-07 14:31:28 -05:00
9b94a1052f
Disabling NUMA maching for model 79 for some VM configs ( #1204 )
2024-06-06 17:15:04 -06:00
526cce9bf4
graph: restrict maxChannels to 64 for multi-node and RCCL_ENABLE_INTRANET=1
2024-06-06 10:58:41 -05:00
6ab20a7c6b
graph: fix multi-node minChannel count
2024-06-06 10:56:39 -05:00
9746d8ca3f
set MIN_NCHANNEL limit to 64 for multi-node
2024-06-03 13:05:05 -05:00
0634c5c8e1
doubling debug buffer size with increased channels
2024-06-03 13:05:05 -05:00
ef442f8f92
set MAXCHANNELS to 128
2024-06-03 13:05:05 -05:00
9f654f6cf5
graph: restrict MAXCHANNELS for certain platforms
2024-06-03 13:05:01 -05:00
48859a97b1
device: update the logic for channelId assignment
2024-06-03 13:03:18 -05:00
506f16c506
add 256 channels support
2024-06-03 13:03:18 -05:00
0948eecbba
Changing channel stride for MI300X multinode ( #1196 )
...
* Shuffling MI300X multi-node channels
* Updating tree channel logic
2024-06-03 10:00:55 -06:00
cab25f919e
Optimize NCHANNELS and MSCCL config for gfx942 80CUs ( #1195 )
...
* Optimize NCHANNELS and MSCCL config for gfx942 80CUs
Set appropriately for different NCCL_MIN_NCHANNELS and MSCCL config,
potentially improving communication perf on the MI300x 80CUs
* Delete tools/msccl-algorithms/allreduce_1step_mccl_8_2_16777216_LL.xml
* Change the factor of gfx94 and update msccl config
2024-06-01 07:07:46 -07:00
1249a6c3fd
[MSCCL]: Move scratch buffer debug msgs to TRACE ( #1189 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2024-05-31 17:54:23 -05:00
354e0b29a6
Addressing possible out-of-bounds mem access during channel duplication ( #1193 )
2024-05-30 14:02:14 -06:00
73221b4230
Add ring simple chunk size tuning ( #1180 )
...
* Add ring simple chunk size tuning
* modifying the tuning table to improve the performance of broadcast for 8MB to 32MB for single-node MI300X after ring simple chunk size tuning
* modifying the tuning table to improve the performance of reduce for 1MB to 4MB for single-node MI300X after ring simple chunk size tuning
---------
Co-authored-by: PedramAlizadeh <pmohamma@amd.com >
2024-05-29 07:59:47 -07:00
8f099b1adb
Make WSL detection thread safe ( #1178 )
...
* Make WSL detection thread safe
* Change static to beginning
* Switch to use atomics
2024-05-28 17:23:50 -07:00
a78c4f5e88
Merge pull request #1175 from edgargabriel/topic/alt_rsmi
...
add alternative to rocm_smi_lib
2024-05-28 09:36:55 -05:00
eeea3b693b
Report error when collective is not enabled in build ( #1177 )
...
* Report error when collective is not enabled in build
* Fix typo
2024-05-16 10:11:12 -07:00
9ad913bfa8
add alternative to rocm_smi_lib
2024-05-14 13:51:41 -07:00
ecafc1969c
Support WSL2 ( #1173 )
2024-05-10 07:31:12 -07:00
87d01e6bf5
Changing the failure print to reflect a bad installation of MSCCL ( #1160 )
...
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com >
2024-05-08 16:56:26 -05:00
a64aab5f63
Use rocm-smi thread only mutex when available ( #1169 )
2024-05-08 14:32:24 -07:00
73acf3eeec
modifying the tuning table to improve the performance of broadcast for 1MB to 64MB for single-node MI300X ( #1172 )
2024-05-08 15:49:33 -04:00
408278209d
Adding ASAN changes to address memory leak issue" ( #1170 )
...
Co-authored-by: akolliasAMD <akollias@amd.com >
2024-05-08 09:16:00 -05:00
b18784d8b8
Add compiler warning for uninitialized variable and fix ( #1163 )
...
* Add compiler warning for uninitialized variable and fix
* Add -Wsometimes-uninitialized
* Convert warning to error
2024-05-08 07:00:25 -07:00
f679db6ff6
Use normal permute path when one NIC per GPU ( #1171 )
2024-05-08 06:59:57 -07:00
a0cef69110
npkit: add broadcast trace ( #1166 )
2024-05-07 14:00:16 -07:00
140b7dd40f
fix typo for DMABUF_ENABLE
2024-05-06 13:27:50 -07:00
b513c3970a
Bypass NVIDIA Ampere related tuning ( #1165 )
2024-05-03 17:57:16 -07:00
bb58b1c258
Fix ignore NUMA not being observed for NICs during model matching ( #1164 )
2024-05-03 16:42:07 -07:00
6f5a8ce1fb
Fix build error when roctracer-dev package is not installed ( #1161 )
2024-05-01 13:55:09 -07:00
4e1b8c1cbb
MSCCL: add support for out-of-place all reduce ( #1156 )
2024-04-28 19:49:09 -07:00
cd6e840e0b
Add back tree simple chunk size tuning ( #1157 )
2024-04-28 19:48:53 -07:00
b90436d292
[GRAPH] Reduce NCCL_TOPO_MAX_NODES to 64 ( #1153 )
...
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com >
2024-04-27 23:41:11 -05:00
cc39e91c6f
Merge pull request #1158 from AtlantaPepsi/NPKit_fix
...
Prevent segfault from npkit-enabled rccl build
2024-04-26 12:44:04 -04:00
67246649ac
prevent segfault from npkit-enabled rccl build
...
Signed-off-by: AtlantaPepsi <timhu102@amd.com >
2024-04-26 10:54:27 -05:00
f330b82985
Revert "Use relaxed atomics for LL on GFX11 ( #859 )" ( #1148 )
...
This reverts commit 6a0a6a37d9
2024-04-26 07:49:55 -07:00
efe99057b0
SWDEV-455705: Fix an UB that could lead to miscompilation ( #1155 )
2024-04-25 10:10:01 -07:00
corey-derochie-amd
Wenkai Du
Nilesh M Negi
Nusrat Islam
saurabhAMD
Paul Emberson
gilbertlee-amd
ClementLinCF
Edgar Gabriel
Edgar Gabriel
mberenjk
Pedram Alizadeh
paklui
Tim
AtlantaPepsi
Shilei Tian