AI/rocm-systems
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,399 Commits 3 Branches 0 Tags
7ca67f1cb9c0068547b99a8e99a4919c47a635ff
Commit Graph
710 Commits
Author SHA1 Message Date
Nilesh M Negi 8518ef4afb [MSCCL]: Move scratch buffer debug msgs to TRACE (#1189) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: 1249a6c3fd]
 2024-05-31 17:54:23 -05:00
gilbertlee-amd b35e2b8c4b Addressing possible out-of-bounds mem access during channel duplication (#1193) 
[ROCm/rccl commit: 354e0b29a6]
 2024-05-30 14:02:14 -06:00
Wenkai DuandPedramAlizadeh a724f1ebb7 Add ring simple chunk size tuning (#1180) 
* Add ring simple chunk size tuning

* modifying the tuning table to improve the performance of broadcast for 8MB to 32MB for single-node MI300X after ring simple chunk size tuning

* modifying the tuning table to improve the performance of reduce for 1MB to 4MB for single-node MI300X after ring simple chunk size tuning

---------

Co-authored-by: PedramAlizadeh <pmohamma@amd.com>

[ROCm/rccl commit: 73221b4230]
 2024-05-29 07:59:47 -07:00
Wenkai Du ae6c372406 Make WSL detection thread safe (#1178) 
* Make WSL detection thread safe

* Change static to beginning

* Switch to use atomics

[ROCm/rccl commit: 8f099b1adb]
 2024-05-28 17:23:50 -07:00
Edgar Gabriel 7f4bec7682 Merge pull request #1175 from edgargabriel/topic/alt_rsmi 
add alternative to rocm_smi_lib

[ROCm/rccl commit: a78c4f5e88]
 2024-05-28 09:36:55 -05:00
Wenkai Du bc29c89d9d Report error when collective is not enabled in build (#1177) 
* Report error when collective is not enabled in build

* Fix typo

[ROCm/rccl commit: eeea3b693b]
 2024-05-16 10:11:12 -07:00
Edgar Gabriel c31ac0d17b add alternative to rocm_smi_lib 
[ROCm/rccl commit: 9ad913bfa8]
 2024-05-14 13:51:41 -07:00
Wenkai Du 984f87a5d5 Support WSL2 (#1173) 
[ROCm/rccl commit: ecafc1969c]
 2024-05-10 07:31:12 -07:00
mberenjkandMarzieh Berenjkoub 42d9f2458a Changing the failure print to reflect a bad installation of MSCCL (#1160) 
Co-authored-by: Marzieh Berenjkoub <mberenjk@amd.com>

[ROCm/rccl commit: 87d01e6bf5]
 2024-05-08 16:56:26 -05:00
Wenkai Du dd5d66d01a Use rocm-smi thread only mutex when available (#1169) 
[ROCm/rccl commit: a64aab5f63]
 2024-05-08 14:32:24 -07:00
Pedram Alizadeh 43d875b422 modifying the tuning table to improve the performance of broadcast for 1MB to 64MB for single-node MI300X (#1172) 
[ROCm/rccl commit: 73acf3eeec]
 2024-05-08 15:49:33 -04:00
mberenjkandakolliasAMD 00c0f3d67f Adding ASAN changes to address memory leak issue" (#1170) 
Co-authored-by: akolliasAMD <akollias@amd.com>

[ROCm/rccl commit: 408278209d]
 2024-05-08 09:16:00 -05:00
Wenkai Du 63d8de3bd3 Add compiler warning for uninitialized variable and fix (#1163) 
* Add compiler warning for uninitialized variable and fix

* Add -Wsometimes-uninitialized

* Convert warning to error

[ROCm/rccl commit: b18784d8b8]
 2024-05-08 07:00:25 -07:00
Wenkai Du eab9978e48 Use normal permute path when one NIC per GPU (#1171) 
[ROCm/rccl commit: f679db6ff6]
 2024-05-08 06:59:57 -07:00
Wenkai Du 0ff5fc0bad npkit: add broadcast trace (#1166) 
[ROCm/rccl commit: a0cef69110]
 2024-05-07 14:00:16 -07:00
paklui dd8e937948 fix typo for DMABUF_ENABLE 
[ROCm/rccl commit: 140b7dd40f]
 2024-05-06 13:27:50 -07:00
Wenkai Du c782aba364 Bypass NVIDIA Ampere related tuning (#1165) 
[ROCm/rccl commit: b513c3970a]
 2024-05-03 17:57:16 -07:00
Wenkai Du 7c811a7582 Fix ignore NUMA not being observed for NICs during model matching (#1164) 
[ROCm/rccl commit: bb58b1c258]
 2024-05-03 16:42:07 -07:00
Wenkai Du 9638535690 Fix build error when roctracer-dev package is not installed (#1161) 
[ROCm/rccl commit: 6f5a8ce1fb]
 2024-05-01 13:55:09 -07:00
Wenkai Du 3906e992f8 MSCCL: add support for out-of-place all reduce (#1156) 
[ROCm/rccl commit: 4e1b8c1cbb]
 2024-04-28 19:49:09 -07:00
Wenkai Du 703014e960 Add back tree simple chunk size tuning (#1157) 
[ROCm/rccl commit: cd6e840e0b]
 2024-04-28 19:48:53 -07:00
Nilesh M Negi b99b89e7a2 [GRAPH] Reduce NCCL_TOPO_MAX_NODES to 64 (#1153) 
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>

[ROCm/rccl commit: b90436d292]
 2024-04-27 23:41:11 -05:00
Tim afeaa17475 Merge pull request #1158 from AtlantaPepsi/NPKit_fix 
Prevent segfault from npkit-enabled rccl build

[ROCm/rccl commit: cc39e91c6f]
 2024-04-26 12:44:04 -04:00
AtlantaPepsi 8cf28704ce prevent segfault from npkit-enabled rccl build 
Signed-off-by: AtlantaPepsi <timhu102@amd.com>


[ROCm/rccl commit: 67246649ac]
 2024-04-26 10:54:27 -05:00
Wenkai Du 3c94f98688 Revert "Use relaxed atomics for LL on GFX11 (#859)" (#1148) 
This reverts commit 5983f0e371.

Use inline asm for 128b load on GFX11 for better peformance.

[ROCm/rccl commit: f330b82985]
 2024-04-26 07:49:55 -07:00
Shilei Tian 9a203f439c SWDEV-455705: Fix an UB that could lead to miscompilation (#1155) 
[ROCm/rccl commit: efe99057b0]
 2024-04-25 10:10:01 -07:00
Wenkai Du e494f29235 Replace __HIP_PLATFORM_HCC__ with __HIP_PLATFORM_AMD__ (#1154) 
[ROCm/rccl commit: 9e0c9b4ed8]
 2024-04-25 07:19:18 -07:00
BertanDogancay 36f9492cda Merge remote-tracking branch 'nccl/master' into develop 
[ROCm/rccl commit: e1a835910e]
 2024-04-23 13:34:00 -07:00
Wenkai Du 35f8d269f8 Use hipExtMallocWithFlags to allocate host memory on APU (#1149) 
Also use SM60 as CUDA compatibility level.

[ROCm/rccl commit: 220066197a]
 2024-04-17 16:56:38 -07:00
gilbertlee-amd 422a7ffcbb Rail optimization for rings (#1140) 
- Modifies the ring creation algorithm to be friendlier to rail-optimized topologies (should not affect classic fabric topologies)

[ROCm/rccl commit: 4cb62f999a]
 2024-04-15 12:03:57 -06:00
mberenjkandmberenjk da835cff9c replacing rccl_bfloat16 with hip_bfloat16 (#1126) 
Co-authored-by: mberenjk <mberenjk@amd.com>

[ROCm/rccl commit: 428837ffe4]
 2024-04-11 11:30:37 -05:00
Wenkai Du 99c7fc29ba NPKit: doubling size of event buffers following MAXCHANNELS change (#1135) 
[ROCm/rccl commit: 0ce68f21d4]
 2024-04-09 08:02:58 -07:00
Wenkai Du 0941d6bc6e Fix buffer overflow when parsing kernel cmdline (#1133) 
[ROCm/rccl commit: 137571fa01]
 2024-04-08 11:12:20 -07:00
Wenkai Du 43bbee4dcc Remove hipEventDisableSystemFence (#1122) 
There is no indication that disabling system fence has any latency improvement.
Removing it per recommendation from HIP.

[ROCm/rccl commit: 5976f757dd]
 2024-03-25 08:01:57 -07:00
FrankJ 894d3459a5 [bugfix]save undo plans in some case 
[ROCm/rccl commit: 9ef920a77b]
 2024-03-12 00:00:16 +08:00
Andy li e373bd44bf Enable fp8 support (#1101) 
* initial checkin

* resolve cr comments

* resolve the build issue

* fix the data correctless issue

* update fp8 header file and update the unit test for fp8 support

* remove fp16 from fp8 headers

* fix ut issue and catch up the latest code from develop

* udate according to cr comments

* update ut according to cr comments

* update num floats for each SumPostDiv from 4 to 6

* update fp8 header file name

* fix the typo

[ROCm/rccl commit: 6777e65c1d]
 2024-03-08 15:17:53 -08:00
Wenkai Du 2354601589 Improve debug messages of memory allocations (#1107) 
[ROCm/rccl commit: ff951e607d]
 2024-03-08 10:55:10 -08:00
Wenkai Du 6dd45024f8 msccl: fix scratch memory allocation after API change (#1103) 
[ROCm/rccl commit: 77615cce28]
 2024-03-06 11:11:04 -08:00
Wenkai Du e5aedb153e Add support for using contiguous for GPU direct RDMA (#1096) 
Enabled by env var RCCL_NET_CONTIGUOUS_MEM=1

[ROCm/rccl commit: cbd955627e]
 2024-02-29 10:06:43 -08:00
Wenkai Du 058886cb20 Add another Rome model (#1095) 
[ROCm/rccl commit: df98a6957d]
 2024-02-28 10:46:05 -08:00
Bertan Dogancay cee279fd99 Implement ROCTX (#1094) 
* Implement roctx

[ROCm/rccl commit: b617aecc31]
 2024-02-27 15:46:15 -07:00
Sylvain Jeaugey 5fa796af0c 2.20.5-1 
Fix UDS connection failure when using ncclCommSplit. Issue #1185


[ROCm/rccl commit: 48bb7fec79]
 2024-02-26 02:52:39 -08:00
Wenkai Du 874998033f Add new GPU model (#1080) 
[ROCm/rccl commit: 74f9e5db64]
 2024-02-23 12:19:42 -08:00
Wenkai Du 4b31894d70 Update RCCL/MSCCL work FIFO depth to 256K (#1091) 
[ROCm/rccl commit: c5ab37211b]
 2024-02-21 17:15:11 -08:00
Bertan Dogancay 4b4bdd904e LL128 check if all XGMI (#1089) 
[ROCm/rccl commit: b275ed0b56]
 2024-02-21 09:41:40 -07:00
Bertan Dogancay 32e1c8cba0 Merge pull request #1079 from BertanDogancay/2.19.4-sync 
2.19.4 Sync

[ROCm/rccl commit: 2fb12a9358]
 2024-02-16 09:50:11 -07:00
akolliasAMD e0dd21028f Allow bus id to be null (#1085) 
* Allow bus id to be null


[ROCm/rccl commit: bac57421c7]
 2024-02-15 16:36:51 -07:00
BertanDogancay ef72944015 Disable unsupported ld/st instructions 
[ROCm/rccl commit: 6f3310605c]
 2024-02-15 13:58:16 -08:00
BertanDogancay 7842411fb3 Merge remote-tracking branch 'rccl/develop' into 2.19.4 
[ROCm/rccl commit: 76f83f95ab]
 2024-02-15 13:37:14 -08:00
Wenkai Du c4e9e2b18a Use native half without conversion (#1083) 
[ROCm/rccl commit: 51003c9980]
 2024-02-13 16:57:34 -08:00