AI/rocm-systems
2
0 Fork 0
Κώδικας Ζητήματα Pull Requests Δράσεις Πακέτα Έργα Κυκλοφορίες Wiki Δραστηριότητα
1.423 Υποβολές 3 Κλάδοι 0 Ετικέτες
5d7078e383d6c40a08d1a9f430a19c88baf7149c
Γράφημα Υποβολών
725 Υποβολές
Συγγραφέας SHA1 Μήνυμα Ημερομηνία
Wenkai Du 5d7078e383 Fix DMABUF support (#1218) 
* Fix DMABUF support

* Reduce log output by moving dmabuf allocation details to TRACE

* Enable peer memory GDR support if ib_umem_get_peer is in kernel
 2024-06-25 08:00:15 -07:00
Paul Emberson 435756af02 fix initOnceFunc setting incorrect result code (#1205) 
Addresses DMA-BUF support check unexpectedly failing
 2024-06-07 16:47:19 -07:00
Nusrat Islam 9660e2e2dc Merge pull request #1200 from nusislam/multi-node-256-fix 
graph: fix multi-node channel count
 2024-06-07 14:34:20 -05:00
Nilesh M Negi d9661c17e6 Fix min_nchannels bug for gfx94* nranks=4 (#1202) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
 2024-06-07 14:31:28 -05:00
gilbertlee-amd 9b94a1052f Disabling NUMA maching for model 79 for some VM configs (#1204) 2024-06-06 17:15:04 -06:00
Nusrat Islam 526cce9bf4 graph: restrict maxChannels to 64 for multi-node and RCCL_ENABLE_INTRANET=1 2024-06-06 10:58:41 -05:00
Nusrat Islam 6ab20a7c6b graph: fix multi-node minChannel count 2024-06-06 10:56:39 -05:00
Nusrat Islam 9746d8ca3f set MIN_NCHANNEL limit to 64 for multi-node 2024-06-03 13:05:05 -05:00
Nusrat Islam 0634c5c8e1 doubling debug buffer size with increased channels 2024-06-03 13:05:05 -05:00
Nusrat Islam ef442f8f92 set MAXCHANNELS to 128 2024-06-03 13:05:05 -05:00
Nusrat Islam 9f654f6cf5 graph: restrict MAXCHANNELS for certain platforms 2024-06-03 13:05:01 -05:00
Nusrat Islam 48859a97b1 device: update the logic for channelId assignment 2024-06-03 13:03:18 -05:00
Nusrat Islam 506f16c506 add 256 channels support 2024-06-03 13:03:18 -05:00
gilbertlee-amd 0948eecbba Changing channel stride for MI300X multinode (#1196) 
* Shuffling MI300X multi-node channels
* Updating tree channel logic
 2024-06-03 10:00:55 -06:00
ClementLinCF cab25f919e Optimize NCHANNELS and MSCCL config for gfx942 80CUs (#1195) 
* Optimize NCHANNELS and MSCCL config for gfx942 80CUs

Set appropriately for different NCCL_MIN_NCHANNELS and MSCCL config,
potentially improving communication perf on the MI300x 80CUs

* Delete tools/msccl-algorithms/allreduce_1step_mccl_8_2_16777216_LL.xml

* Change the factor of gfx94 and update msccl config
 2024-06-01 07:07:46 -07:00
Nilesh M Negi 1249a6c3fd [MSCCL]: Move scratch buffer debug msgs to TRACE (#1189) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
 2024-05-31 17:54:23 -05:00
gilbertlee-amd 354e0b29a6 Addressing possible out-of-bounds mem access during channel duplication (#1193) 2024-05-30 14:02:14 -06:00
Wenkai DuκαιPedramAlizadeh 73221b4230 Add ring simple chunk size tuning (#1180) 
* Add ring simple chunk size tuning

* modifying the tuning table to improve the performance of broadcast for 8MB to 32MB for single-node MI300X after ring simple chunk size tuning

* modifying the tuning table to improve the performance of reduce for 1MB to 4MB for single-node MI300X after ring simple chunk size tuning

---------

Co-authored-by: PedramAlizadeh <pmohamma@amd.com>
 2024-05-29 07:59:47 -07:00
Wenkai Du 8f099b1adb Make WSL detection thread safe (#1178) 
* Make WSL detection thread safe

* Change static to beginning

* Switch to use atomics
 2024-05-28 17:23:50 -07:00
Edgar Gabriel a78c4f5e88 Merge pull request #1175 from edgargabriel/topic/alt_rsmi 
add alternative to rocm_smi_lib
 2024-05-28 09:36:55 -05:00
Wenkai Du eeea3b693b Report error when collective is not enabled in build (#1177) 
* Report error when collective is not enabled in build

* Fix typo
 2024-05-16 10:11:12 -07:00
Edgar Gabriel 9ad913bfa8 add alternative to rocm_smi_lib 2024-05-14 13:51:41 -07:00
Wenkai Du ecafc1969c Support WSL2 (#1173) 2024-05-10 07:31:12 -07:00
mberenjkκαιMarzieh Berenjkoub 87d01e6bf5 Changing the failure print to reflect a bad installation of MSCCL (#1160) 
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>
 2024-05-08 16:56:26 -05:00
Wenkai Du a64aab5f63 Use rocm-smi thread only mutex when available (#1169) 2024-05-08 14:32:24 -07:00
Pedram Alizadeh 73acf3eeec modifying the tuning table to improve the performance of broadcast for 1MB to 64MB for single-node MI300X (#1172) 2024-05-08 15:49:33 -04:00
mberenjkκαιakolliasAMD 408278209d Adding ASAN changes to address memory leak issue" (#1170) 
Co-authored-by: akolliasAMD <akollias@amd.com>
 2024-05-08 09:16:00 -05:00
Wenkai Du b18784d8b8 Add compiler warning for uninitialized variable and fix (#1163) 
* Add compiler warning for uninitialized variable and fix

* Add -Wsometimes-uninitialized

* Convert warning to error
 2024-05-08 07:00:25 -07:00
Wenkai Du f679db6ff6 Use normal permute path when one NIC per GPU (#1171) 2024-05-08 06:59:57 -07:00
Wenkai Du a0cef69110 npkit: add broadcast trace (#1166) 2024-05-07 14:00:16 -07:00
paklui 140b7dd40f fix typo for DMABUF_ENABLE 2024-05-06 13:27:50 -07:00
Wenkai Du b513c3970a Bypass NVIDIA Ampere related tuning (#1165) 2024-05-03 17:57:16 -07:00
Wenkai Du bb58b1c258 Fix ignore NUMA not being observed for NICs during model matching (#1164) 2024-05-03 16:42:07 -07:00
Wenkai Du 6f5a8ce1fb Fix build error when roctracer-dev package is not installed (#1161) 2024-05-01 13:55:09 -07:00
Wenkai Du 4e1b8c1cbb MSCCL: add support for out-of-place all reduce (#1156) 2024-04-28 19:49:09 -07:00
Wenkai Du cd6e840e0b Add back tree simple chunk size tuning (#1157) 2024-04-28 19:48:53 -07:00
Nilesh M Negi b90436d292 [GRAPH] Reduce NCCL_TOPO_MAX_NODES to 64 (#1153) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
 2024-04-27 23:41:11 -05:00
Tim cc39e91c6f Merge pull request #1158 from AtlantaPepsi/NPKit_fix 
Prevent segfault from npkit-enabled rccl build
 2024-04-26 12:44:04 -04:00
AtlantaPepsi 67246649ac prevent segfault from npkit-enabled rccl build 
Signed-off-by: AtlantaPepsi <timhu102@amd.com>
 2024-04-26 10:54:27 -05:00
Wenkai Du f330b82985 Revert "Use relaxed atomics for LL on GFX11 (#859)" (#1148) 
This reverts commit 6a0a6a37d9.

Use inline asm for 128b load on GFX11 for better peformance.
 2024-04-26 07:49:55 -07:00
Shilei Tian efe99057b0 SWDEV-455705: Fix an UB that could lead to miscompilation (#1155) 2024-04-25 10:10:01 -07:00
Wenkai Du 9e0c9b4ed8 Replace __HIP_PLATFORM_HCC__ with __HIP_PLATFORM_AMD__ (#1154) 2024-04-25 07:19:18 -07:00
BertanDogancay e1a835910e Merge remote-tracking branch 'nccl/master' into develop 2024-04-23 13:34:00 -07:00
Wenkai Du 220066197a Use hipExtMallocWithFlags to allocate host memory on APU (#1149) 
Also use SM60 as CUDA compatibility level.
 2024-04-17 16:56:38 -07:00
gilbertlee-amd 4cb62f999a Rail optimization for rings (#1140) 
- Modifies the ring creation algorithm to be friendlier to rail-optimized topologies (should not affect classic fabric topologies)
 2024-04-15 12:03:57 -06:00
mberenjkκαιmberenjk 428837ffe4 replacing rccl_bfloat16 with hip_bfloat16 (#1126) 
Co-authored-by: mberenjk <mberenjk@amd.com>
 2024-04-11 11:30:37 -05:00
Wenkai Du 0ce68f21d4 NPKit: doubling size of event buffers following MAXCHANNELS change (#1135) 2024-04-09 08:02:58 -07:00
Wenkai Du 137571fa01 Fix buffer overflow when parsing kernel cmdline (#1133) 2024-04-08 11:12:20 -07:00
Wenkai Du 5976f757dd Remove hipEventDisableSystemFence (#1122) 
There is no indication that disabling system fence has any latency improvement.
Removing it per recommendation from HIP.
 2024-03-25 08:01:57 -07:00
FrankJ 9ef920a77b [bugfix]save undo plans in some case 2024-03-12 00:00:16 +08:00