SaleelK
6b28faa532
Problem:
The existing SDMA engine selection logic had several issues:
1. Same VirtualGPU/stream could use different SDMA engines for consecutive
async copies since copy_engine_status may report engines as busy
2. Busy and Preferred engine check for every copy
3. No global tracking of which VirtualGPU uses which engine, leading to
suboptimal resource allocation
Solution:
Implemented a global SDMA engine allocator with per-stream affinity:
- Added Device::SdmaEngineAllocator to manage VirtualGPU → engine assignments
* Maintains global map of active assignments
* Enforces exclusivity: different streams use different engines (except
inter-GPU copies where preferred engines are prioritized for optimal
hardware paths like XGMI links)
* Thread-safe allocation/release with Monitor lock
- Modified VirtualGPU to cache assigned engine locally (assigned_sdma_engine_)
for fast lookup without map access on hot path
- Refactored rocrCopyBuffer() to:
1. Check local cached engine first → use if assigned
2. Call AllocateSdmaEngine() if not assigned → cache result
- Moved HSA API queries (memory_copy_engine_status, memory_get_preferred_copy_engine)
into AllocateEngine() for cleaner separation of concerns
- Engine release on HostQueue::finish() instead of only VirtualGPU destruction
* Improves engine utilization by releasing earlier
* Added virtual ReleaseSdmaEngines() method to device::VirtualDevice
- Added future path for simple round-robin allocation (kUseSimpleRR) for
next-gen GPUs with uniform SDMA bandwidth (disabled by default)
Cleanup:
- Removed selectSdmaEngine() helper (logic moved to allocator)
- Removed getSdmaRWMasks() (allocator accesses maxSdmaReadMask_/WriteMask_ directly)
- Removed unused sdmaEngineReadMask_/WriteMask_ member variables from DmaBlitManager
Benefits:
- Ensures consistent per-stream SDMA engine usage
- Prevents cross-stream contention and engine thrashing
- Prioritizes hardware-optimal paths for inter-GPU transfers
- Better resource utilization through earlier release
- Cleaner, more maintainable code structure
ROCm Systems
Welcome to the ROCm Systems super-repo. This repository consolidates multiple ROCm systems projects into a single repository to streamline development, CI, and integration. The first set of projects focuses on requirements for building PyTorch.
Super-repo Status and CI Health
This table provides the current status of the migration of specific ROCm systems projects as well as a pointer to their current CI health.
Key:
- Completed: Fully migrated and integrated. This super-repo should be considered the source of truth for this project. The old repo may still be used for release activities.
- In Progress: Ongoing migration, tests, or integration. Please refrain from submitting new pull requests on the individual repo of the project, and develop on the super-repo.
- Pending: Not yet started or in the early planning stages. The individual repo should be considered the source of truth for this project.
| Component | Source of Truth | Migration Status | Azure CI Status | Component CI Status |
|---|---|---|---|---|
amdsmi |
EMU | Pending | ||
aqlprofile |
Public | Completed |   |
|
clr |
Public | Completed | ||
hip |
Public | Completed | ||
hipother |
Public | Completed | ||
hip-tests |
Public | Completed | ||
rdc |
Public | Completed | ||
rocm-core |
Public | Completed | ||
rocminfo |
Public | Completed | ||
rocm-smi-lib |
Public | Completed | ||
rocprofiler |
Public | Completed | ||
rocprofiler-compute |
Public | Completed |     |
|
rocprofiler-register |
Public | Completed |  |
|
rocprofiler-sdk |
Public | Completed |         |
|
rocprofiler-systems |
Public | Completed |          |
|
rocr-runtime |
Public | Completed | ||
roctracer |
Public | Completed |
Tentative migration schedule
| Component | Tentative Date |
|---|
*Remaining schedule to be determined.
TheRock CI Status
Note TheRock CI performs multi-component testing on top of builds leveraging TheRock build system.
Nomenclature
Project names have been standardized to match the casing and punctuation of released packages. This removes inconsistent camel-casing and underscores used in legacy repositories.
Structure
The repository is organized as follows:
projects/
amdsmi/
aqlprofile/
clr/
hip/
hipother/
hip-tests/
rccl/
rdc/
rocm-core
rocminfo/
rocmsmilib/
rocprofiler/
rocprofiler-compute/
rocprofiler-register/
rocprofiler-sdk/
rocprofiler-systems/
rocrruntime/
rocshmem/
roctracer/
- Each folder under
projects/corresponds to a ROCm systems project that was previously maintained in a standalone GitHub repository and released as distinct packages. - Each folder under
shared/contains code that existed in its own repository and is used as a dependency by multiple projects, but does not produce its own distinct packages in previous ROCm releases.
Goals
- Enable unified build and test workflows across ROCm libraries.
- Facilitate shared tooling, CI, and contributor experience.
- Improve integration, visibility, and collaboration across ROCm library teams.
Getting Started
To begin contributing or building, see the CONTRIBUTING.md guide. It includes setup instructions, sparse-checkout configuration, development workflow, and pull request guidelines.
License
This super-repo contains multiple subprojects, each of which retains the license under which it was originally published.
📁 Refer to the LICENSE, LICENSE.md, or LICENSE.txt file within each projects/ or shared/ directory for specific license terms.
📄 Refer to the header notice in individual files outside projects/ or shared/ folders for their specific license terms.
Note
: The root of this repository does not define a unified license across all components.
Questions or Feedback?
We're happy to help!