Terminology change: monorepo -> super-repo

此提交包含在:
Joseph Macaranas
2025-08-02 22:23:24 -04:00
父節點 153f71fb0d
當前提交 870dc948b1
共有 20 個檔案被更改,包括 109 行新增184 行删除
+4 -4
查看文件
@@ -1,14 +1,14 @@
import os
from pathlib import Path
# Determine monorepo root and output CODEOWNERS path
monorepo_root = Path(__file__).resolve().parents[2]
output_path = monorepo_root / ".github" / "CODEOWNERS"
# Determine super-repo root and output CODEOWNERS path
super_repo_root = Path(__file__).resolve().parents[2]
output_path = super_repo_root / ".github" / "CODEOWNERS"
merged_entries = []
# Walk top-level directories (excluding .github/.git/etc.)
for subdir in monorepo_root.iterdir():
for subdir in super_repo_root.iterdir():
if subdir.name.startswith(".") or not subdir.is_dir():
continue
+7 -7
查看文件
@@ -4,14 +4,14 @@
Sync Patches to Subrepositories
-------------------------------
This script is part of the monorepo synchronization system. It runs after a monorepo pull request
This script is part of the super-repo synchronization system. It runs after a super-repo pull request
is merged and applies relevant changes to the corresponding sub-repositories using Git patches.
- Uses the merge commit of the monorepo PR to extract subtree changes.
- Uses the merge commit of the super-repo PR to extract subtree changes.
- Generates patch files per changed subtree.
- Applies each patch to its respective sub-repository, adjusting for subtree prefix.
- Uses the repos-config.json file to map subtrees to sub-repos.
- Assumes this script is run from the root of the monorepo.
- Assumes this script is run from the root of the super-repo.
Arguments:
--repo : Full repository name (e.g., org/repo)
@@ -127,9 +127,9 @@ def _extract_commit_message_from_patch(patch_path: Path) -> str:
commit_msg_lines.append(line)
return "".join(commit_msg_lines).strip()
def _format_commit_message(monorepo_url: str, pr_number: int, merge_sha: str, original_msg: str) -> str:
def _format_commit_message(super_repo_url: str, pr_number: int, merge_sha: str, original_msg: str) -> str:
"""Append a sync annotation to the original commit message."""
annotation = f"\n[rocm-systems] {monorepo_url}#{pr_number} (commit {merge_sha[:7]})\n"
annotation = f"\n[rocm-systems] {super_repo_url}#{pr_number} (commit {merge_sha[:7]})\n"
return original_msg + annotation
def _commit_changes(repo_path: Path, message: str, author_name: str, author_email: str) -> None:
@@ -175,7 +175,7 @@ def resolve_patch_author(client: GitHubCLIClient, repo: str, pr: int) -> tuple[s
name, email = client.get_user(username)
return name or username, email
def apply_patch_to_subrepo(entry: RepoEntry, monorepo_url: str, monorepo_pr: int,
def apply_patch_to_subrepo(entry: RepoEntry, super_repo_url: str, super_repo_pr: int,
patch_path: Path, author_name: str, author_email: str,
merge_sha: str, dry_run: bool = False) -> None:
"""Clone the subrepo, apply the patch, and attribute to the original author with commit message annotations."""
@@ -189,7 +189,7 @@ def apply_patch_to_subrepo(entry: RepoEntry, monorepo_url: str, monorepo_pr: int
_apply_patch(subrepo_path, patch_path)
_stage_changes(subrepo_path)
original_commit_msg = _extract_commit_message_from_patch(patch_path)
commit_msg = _format_commit_message(monorepo_url, monorepo_pr, merge_sha, original_commit_msg)
commit_msg = _format_commit_message(super_repo_url, super_repo_pr, merge_sha, original_commit_msg)
_commit_changes(subrepo_path, commit_msg, author_name, author_email)
_set_authenticated_remote(subrepo_path, entry.url)
_push_changes(subrepo_path, entry.branch)
+1 -1
查看文件
@@ -32,7 +32,7 @@ class RepoEntry(BaseModel):
name : Name of the project matching packaging file names. Lower-cased and no underscores. (e.g., "rocblas")
url : Individual GitHub org plus repo names in matching case and punctuation. (e.g., "ROCm/rocBLAS")
branch : The base branch of the sub-repo to target (e.g., "develop").
category : Directory category in the monorepo (e.g., "projects" or "shared").
category : Directory category in the super-repo (e.g., "projects" or "shared").
"""
name: str
url: str
+2 -2
查看文件
@@ -13,7 +13,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout monorepo
- name: Checkout super-repo
uses: actions/checkout@v4
- name: Set up Python
@@ -24,7 +24,7 @@ jobs:
- name: Install dependencies
run: pip install PyYAML requests
- name: Apply labels to monorepo
- name: Apply labels to super-repo
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_REPO: ${{ github.repository }}
+1 -1
查看文件
@@ -1,7 +1,7 @@
# Azure CI Dispatcher
# ------------------
# This workflow allows Azure CI to be centralized in a single PR check.
# It detects which subtrees (from a monorepo structure) were changed in a
# It detects which subtrees (from a super-repo structure) were changed in a
# pull request, and automatically requests Azure CI runs for the corresponding
# subtrees.
#
+1 -1
查看文件
@@ -13,7 +13,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout monorepo
- name: Checkout super-repo
uses: actions/checkout@v4
- name: Set up Python
+8 -8
查看文件
@@ -1,14 +1,14 @@
name: Setup Monorepo
name: Setup super-repo
on:
workflow_dispatch:
env:
MONOREPO_URL: github.com/ROCm/rocm-systems.git
MONOREPO_BRANCH: develop
SUPER_REPO_URL: github.com/ROCm/rocm-systems.git
SUPER_REPO_BRANCH: develop
jobs:
setup-monorepo:
setup-super-repo:
runs-on: ubuntu-24.04
steps:
- name: Generate a token
@@ -19,7 +19,7 @@ jobs:
private-key: ${{ secrets.APP_PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
- name: Checkout the Monorepo
- name: Checkout the Super-repo
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
token: ${{ steps.generate-token.outputs.token }}
@@ -29,7 +29,7 @@ jobs:
git config user.name "systems-assistant[bot]"
git config user.email "systems-assistant[bot]@users.noreply.github.com"
- name: Add Repositories to the Monorepo
- name: Add Repositories to the Super-repo
run: |
for repo in $(cat .github/repos-config.json | jq -r '.repositories[].name'); do
repo_name=$(cat .github/repos-config.json | jq -r ".repositories[] | select(.name == \"$repo\") | .reponame")
@@ -40,5 +40,5 @@ jobs:
git subtree add --prefix "${category}/${repo}" https://github.com/${url}.git "$branch"
done
- name: Push changes to Monorepo
run: git push https://${{ env.MONOREPO_URL }} ${{ env.MONOREPO_BRANCH }}
- name: Push changes to Super-repo
run: git push https://${{ env.SUPER_REPO_URL }} ${{ env.SUPER_REPO_BRANCH }}
+2 -2
查看文件
@@ -1,6 +1,6 @@
# Import Issue Workflow
# ---------------------
# This workflow imports a single issue from a source repository into this monorepo repository,
# This workflow imports a single issue from a source repository into this super-repo repository,
# as part of a migration. It preserves key metadata including title, body, labels, and comment
# history with timestamps and authorship.
#
@@ -9,7 +9,7 @@
# - issue_number: Issue number in the source repository to import
#
# Steps:
# 1. Validate that the caller has 'admin' or 'maintain' permission on the monorepo
# 1. Validate that the caller has 'admin' or 'maintain' permission on the super-repo
# 2. Fetch the issue body, title, labels, and comments using the GitHub CLI (`gh api`)
# 3. Create a new issue in the target repo, preserving:
# - Original title
+1 -1
查看文件
@@ -7,7 +7,7 @@ jobs:
combine-gitmodules:
runs-on: ubuntu-latest
steps:
- name: Checkout Monorepo
- name: Checkout super-repo
uses: actions/checkout@v4
- name: Set up Python
+15 -15
查看文件
@@ -1,26 +1,26 @@
# Import Subrepo PR Workflow
# --------------------------
# This workflow imports an open PR from a sub-repository into the monorepo.
# This workflow imports an open PR from a sub-repository into the super-repo.
# It is intended for maintainers to migrate pending pull requests from
# the subrepos, to help developers onboard to the monorepo sooner.
# the subrepos, to help developers onboard to the super-repo sooner.
#
# Trigger: Manually via `workflow_dispatch` with the following inputs:
# - subrepo-prefix: Path within the monorepo where the subrepo lives (e.g., "projects/rocblas")
# - subrepo-prefix: Path within the super-repo where the subrepo lives (e.g., "projects/rocblas")
# - subrepo-pr-number: PR number in the subrepo to import
# - subrepo-repo: Full subrepo or fork name (e.g., "ROCm/rocBLAS" or "user/rocBLAS-fork")
# - subrepo-upstream: Canonical subrepo repo name (e.g., "ROCm/rocBLAS")
#
# Steps:
# 1. Validate that the caller has 'admin' or 'maintain' permission on the monorepo
# 1. Validate that the caller has 'admin' or 'maintain' permission on the super-repo
# 2. Generate a GitHub App token for authenticated API access
# 3. Checkout only the necessary parts of the monorepo (sparse checkout)
# 3. Checkout only the necessary parts of the super-repo (sparse checkout)
# 4. Fetch PR metadata from the subrepo using GitHub CLI
# 5. Create a new branch and import the PR using `git subtree pull`
# 6. Create a corresponding draft PR in the monorepo with metadata preserved
# 6. Create a corresponding draft PR in the super-repo with metadata preserved
#
# Notes:
# - This workflow requires GitHub App credentials (APP_ID and APP_PRIVATE_KEY)
# - The resulting monorepo PR is a draft and includes attribution to the original author
# - The resulting super-repo PR is a draft and includes attribution to the original author
name: Import Subrepo PR
@@ -28,7 +28,7 @@ on:
workflow_dispatch:
inputs:
subrepo-prefix:
description: "Monorepo path prefix (e.g., projects/rocblas)"
description: "super-repo path prefix (e.g., projects/rocblas)"
required: true
subrepo-pr-number:
description: "Subrepo PR number to import"
@@ -39,8 +39,8 @@ on:
subrepo-upstream:
description: "Canonical subrepo repo (e.g., ROCm/rocBLAS)"
required: true
monorepo-target-branch:
description: "Target branch in the monorepo (default: develop)"
super-repo-target-branch:
description: "Target branch in the super-repo (default: develop)"
required: false
default: "develop"
@@ -113,9 +113,9 @@ jobs:
- name: Create new branch for import
id: import-branch
run: |
git fetch origin ${{ github.event.inputs.monorepo-target-branch }}
git checkout ${{ github.event.inputs.monorepo-target-branch }}
SANITIZED_BASE=$(echo "${{ github.event.inputs.monorepo-target-branch }}" | sed 's|/|_|g')
git fetch origin ${{ github.event.inputs.super-repo-target-branch }}
git checkout ${{ github.event.inputs.super-repo-target-branch }}
SANITIZED_BASE=$(echo "${{ github.event.inputs.super-repo-target-branch }}" | sed 's|/|_|g')
SANITIZED_REPO=$(echo "${{ github.event.inputs.subrepo-repo }}" | sed 's|/|_|g')
SANITIZED_REF=$(echo "${{ steps.prdata.outputs.head_ref }}" | sed 's|/|_|g')
IMPORT_BRANCH="import/${SANITIZED_BASE}/${SANITIZED_REPO}/${SANITIZED_REF}"
@@ -127,7 +127,7 @@ jobs:
git subtree pull --prefix=${{ github.event.inputs.subrepo-prefix }} https://github.com/${{ github.event.inputs.subrepo-repo }} ${{ steps.prdata.outputs.head_ref }}
git push origin ${{ steps.import-branch.outputs.import_branch }}
- name: Create monorepo PR
- name: Create super-repo PR
env:
GH_TOKEN: ${{ steps.generate-token.outputs.token }}
run: |
@@ -153,7 +153,7 @@ jobs:
fi
gh pr create \
--base "${{ github.event.inputs.monorepo-target-branch }}" \
--base "${{ github.event.inputs.super-repo-target-branch }}" \
--head "${{ steps.import-branch.outputs.import_branch }}" \
--title "$PR_TITLE" \
--label "imported pr" \
+4 -4
查看文件
@@ -1,13 +1,13 @@
# Apply Patch to Subrepos After Merge (Manual Run)
# --------------------------------------------------
# This GitHub Actions workflow lets you manually run the patching logic for
# previously merged pull requests in the monorepo.
# previously merged pull requests in the super-repo.
#
# It is useful for cases where the automatic patch workflow failed (e.g., due
# to fork PRs lacking secrets), or if configuration or credentials have changed.
#
# Key Steps:
# 1. Validate that the caller has 'admin' or 'maintain' permission on the monorepo.
# 1. Validate that the caller has 'admin' or 'maintain' permission on the super-repo.
# 2. Accept PR number as input.
# 3. Generate a GitHub App token for authentication.
# 4. Use a Python script to detect which subtrees were modified.
@@ -15,11 +15,11 @@
# - Generate a patch from the merge commit for that subtree.
# - Determine the appropriate author (based on PR metadata or fallback).
# - Clone the target sub-repo and apply the patch.
# - Amend the commit message to include links to the monorepo PR and commit.
# - Amend the commit message to include links to the super-repo PR and commit.
# - Push the commit directly to the sub-repo.
#
# This ensures downstream sub-repositories are updated to reflect changes
# made in the monorepo, even if the original automated job failed.
# made in the super-repo, even if the original automated job failed.
name: Manual Patch Rerun
+4 -4
查看文件
@@ -1,24 +1,24 @@
# Apply Patch to Subrepos After Merge
# -----------------------------------
# This GitHub Actions workflow runs on push commits. If it detects the push is from
# a pull request merged into the monorepo, then it continues to run the workflow.
# a pull request merged into the super-repo, then it continues to run the workflow.
# It identifies which subtrees (defined in .github/repos-config.json) were affected,
# generates a patch from the merge commit, and applies that patch to the corresponding
# sub-repositories by cloning them and committing the patch directly.
#
# Key Steps:
# 1. Generate a GitHub App token for authentication.
# 2. Checkout the monorepo at the merge commit.
# 2. Checkout the super-repo at the merge commit.
# 3. Use a Python script to detect which subtrees were modified.
# 4. For each changed subtree:
# - Generate a patch from the merge commit for that subtree.
# - Determine the appropriate author (based on PR metadata or fallback).
# - Clone the target sub-repo and apply the patch.
# - Amend the commit message to include links to the monorepo PR and commit.
# - Amend the commit message to include links to the super-repo PR and commit.
# - Push the commit directly to the sub-repo.
#
# This ensures downstream sub-repositories are updated to reflect changes
# made in the monorepo immediately after merge.
# made in the super-repo immediately after merge.
name: Merged PR to Patch Subrepos
-75
查看文件
@@ -1,75 +0,0 @@
name: Synchronize Release Staging Subtrees
on:
workflow_dispatch:
schedule:
- cron: '0 * * * *'
env:
MONOREPO_URL: github.com/ROCm/rocm-systems.git
MONOREPO_BRANCH: release-staging/rocm-rel-7.0
concurrency:
group: pr-update-subtrees-develop
cancel-in-progress: false
jobs:
synchronize-subtrees:
runs-on: ubuntu-24.04
steps:
- name: Generate a token
id: generate-token
uses: actions/create-github-app-token@df432ceedc7162793a195dd1713ff69aefc7379e # v2.0.6
with:
app-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
- name: Checkout the Monorepo
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0 # needed for git subtree pull/push
token: ${{ steps.generate-token.outputs.token }}
ref: ${{ env.MONOREPO_BRANCH }}
- name: Set up Git user
run: |
git config user.name "systems-assistant[bot]"
git config user.email "systems-assistant[bot]@users.noreply.github.com"
- name: Switch to the Monorepo branch
run: |
git checkout -B "${{ env.MONOREPO_BRANCH }}" "origin/${{ env.MONOREPO_BRANCH }}"
- name: Update Repositories in the Monorepo
run: |
has_errors=false
for repo in $(cat .github/repos-config.json | jq -r '.repositories[].name'); do
url=$(cat .github/repos-config.json | jq -r ".repositories[] | select(.name == \"$repo\") | .url")
branch=$(cat .github/repos-config.json | jq -r ".repositories[] | select(.name == \"$repo\") | .branch")
category=$(cat .github/repos-config.json | jq -r ".repositories[] | select(.name == \"$repo\") | .category")
enable_pull=$(cat .github/repos-config.json | jq -r ".repositories[] | select(.name == \"$repo\") | .auto_subtree_pull")
enable_push=$(cat .github/repos-config.json | jq -r ".repositories[] | select(.name == \"$repo\") | .auto_subtree_push")
if [ "$enable_pull" = true ]; then
git subtree pull --prefix "${category}/${repo}" https://github.com/${url}.git $branch || {
has_errors=true
}
fi
# if [ "$enable_push" = true ]; then
# git fetch origin subtrees/${repo}/${branch}
# git branch -f subtrees/${repo}/${branch} origin/subtrees/${repo}/${branch}
# git subtree split --prefix="${category}/${repo}" -b subtrees/${repo}/${branch} --quiet --rejoin || {
# has_errors=true
# }
# git push origin subtrees/${repo}/${branch}
# git push https://github.com/${url}.git subtrees/${repo}/${branch}:${branch}
# fi
done
if [ "$has_errors" = true ]; then
echo "One or more errors occurred during the repository update."
exit 1
else
git push https://${{ env.MONOREPO_URL }} ${{ env.MONOREPO_BRANCH }}
echo "All repositories updated successfully!"
fi
+7 -7
查看文件
@@ -6,8 +6,8 @@ on:
- cron: '0 * * * *'
env:
MONOREPO_URL: github.com/ROCm/rocm-systems.git
MONOREPO_BRANCH: develop
SUPER_REPO_URL: github.com/ROCm/rocm-systems.git
SUPER_REPO_BRANCH: develop
concurrency:
group: pr-update-subtrees-develop
@@ -25,7 +25,7 @@ jobs:
private-key: ${{ secrets.APP_PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
- name: Checkout the Monorepo
- name: Checkout the Super-repo
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0 # needed for git subtree pull/push
@@ -36,11 +36,11 @@ jobs:
git config user.name "systems-assistant[bot]"
git config user.email "systems-assistant[bot]@users.noreply.github.com"
- name: Switch to the Monorepo branch
- name: Switch to the Super-repo branch
run: |
git checkout -B "${{ env.MONOREPO_BRANCH }}" "origin/${{ env.MONOREPO_BRANCH }}"
git checkout -B "${{ env.SUPER_REPO_BRANCH }}" "origin/${{ env.SUPER_REPO_BRANCH }}"
- name: Update Repositories in the Monorepo
- name: Update Repositories in the Super-repo
run: |
has_errors=false
for repo in $(cat .github/repos-config.json | jq -r '.repositories[].name'); do
@@ -69,6 +69,6 @@ jobs:
echo "One or more errors occurred during the repository update."
exit 1
else
git push https://${{ env.MONOREPO_URL }} ${{ env.MONOREPO_BRANCH }}
git push https://${{ env.SUPER_REPO_URL }} ${{ env.SUPER_REPO_BRANCH }}
echo "All repositories updated successfully!"
fi
+11 -11
查看文件
@@ -1,17 +1,17 @@
# Contributing to the ROCm Libraries
Thank you for contributing! This guide outlines the development workflow, contribution standards, and best practices when working in the monorepo.
Thank you for contributing! This guide outlines the development workflow, contribution standards, and best practices when working in the super-repo.
## Getting Started
### Option A: Clone the Monorepo
### Option A: Clone the super-repo
```bash
git clone https://github.com/ROCm/rocm-systems.git
cd rocm-systems
```
### Option B: Clone the Monorepo with Sparse-Checkout
### Option B: Clone the super-repo with Sparse-Checkout
To limit your local checkout to only the project(s) you work on and improve performance with a large codebase, you can configure sparse-checkout prior to cloning:
@@ -36,7 +36,7 @@ If your work involves changing projects or introducing new projects, you can upd
git sparse-checkout set projects/hipsparse projects/rocsparse
```
This keeps your working directory clean and fast, as you won't need to clone the entire monorepo.
This keeps your working directory clean and fast, as you won't need to clone the entire super-repo.
---
@@ -55,7 +55,7 @@ Further changes to the structure may be made to improve development efficiency a
### From a Developer's Perspective
You can continue working inside your project's folder as you did before the monorepo migration.
You can continue working inside your project's folder as you did before the super-repo migration.
This process is intended to remain as familiar as possible, though some adjustments may be made to improve efficiency based on feedback.
#### Example: hipblaslt Developer
@@ -69,7 +69,7 @@ cd projects/hipblaslt
## Keeping Your Branch in Sync
To stay up to date with the latest changes in the monorepo:
To stay up to date with the latest changes in the super-repo:
```bash
git fetch origin
@@ -82,7 +82,7 @@ Avoid using git merge to keep history clean and maintain a linear progression.
## New Product Introduction (NPI) and New Technology Introduction (NTI) Development
A mirror of this monorepo will be on GitHub Enterprise Managed User (EMU) and available only on the AMD intranet.
A mirror of this super-repo will be on GitHub Enterprise Managed User (EMU) and available only on the AMD intranet.
Please reach out within the AMD intranet if you need the link and permissions.
A primary development branch will be created for a new product or new technology.
@@ -119,11 +119,11 @@ git push origin branch-name-like-above
### 3. Auto-Labeling and Review Routing
The monorepo uses automation to assign labels and reviewers based on the changed files. Reviewers are designated via the top-level CODEOWNERS file.
The super-repo uses automation to assign labels and reviewers based on the changed files. Reviewers are designated via the top-level CODEOWNERS file.
### 4. Tests and CI
Existing testing and CI infrastructure will be updated to directly point to the monorepo.
Existing testing and CI infrastructure will be updated to directly point to the super-repo.
Specific checks will become mandatory for pull requests before merging. Initially, these will be limited to compilation, but will expand to correctness tests and eventually performance tests.
Hardware and operating system coverage will also expand for these checks over time.
Please refer to [this documentation](/docs/continuous-integration.md) for further details on the current signals that will be provided through CI for pull requests and commits.
@@ -142,9 +142,9 @@ See [docs/gardening.md](docs/gardening.md) for more information.
## Developer Communications
As this monorepo continues to evolve, weekly office hour sessions with a wide audience of ROCm engineers and managers will occur.
As this super-repo continues to evolve, weekly office hour sessions with a wide audience of ROCm engineers and managers will occur.
Focused meetings with smaller project teams will be also be scheduled regularly.
These discussions can go over any topic of the monorepo important to the different teams.
These discussions can go over any topic of the super-repo important to the different teams.
If you want to be looped into these syncs, please reach out to project leadership.
---
+5 -5
查看文件
@@ -1,14 +1,14 @@
# ROCm Systems
Welcome to the ROCm Systems monorepo. This repository consolidates multiple ROCm systems projects into a single repository to streamline development, CI, and integration. The first set of projects focuses on requirements for building PyTorch.
Welcome to the ROCm Systems super-repo. This repository consolidates multiple ROCm systems projects into a single repository to streamline development, CI, and integration. The first set of projects focuses on requirements for building PyTorch.
# Monorepo Status and CI Health
# Super-repo Status and CI Health
This table provides the current status of the migration of specific ROCm systems projects as well as a pointer to their current CI health.
**Key:**
- **Completed**: Fully migrated and integrated. This monorepo should be considered the source of truth for this project. The old repo may still be used for release activities.
- **In Progress**: Ongoing migration, tests, or integration. Please refrain from submitting new pull requests on the individual repo of the project, and develop on the monorepo.
- **Completed**: Fully migrated and integrated. This super-repo should be considered the source of truth for this project. The old repo may still be used for release activities.
- **In Progress**: Ongoing migration, tests, or integration. Please refrain from submitting new pull requests on the individual repo of the project, and develop on the super-repo.
- **Pending**: Not yet started or in the early planning stages. The individual repo should be considered the source of truth for this project.
| Component | Source of Truth | Migration Status | Azure CI Status | Component CI Status |
@@ -111,7 +111,7 @@ To begin contributing or building, see the [CONTRIBUTING.md](./CONTRIBUTING.md)
## License
This monorepo contains multiple subprojects, each of which retains the license under which it was originally published.
This super-repo contains multiple subprojects, each of which retains the license under which it was originally published.
📁 Refer to the `LICENSE`, `LICENSE.md`, or `LICENSE.txt` file within each `projects/` or `shared/` directory for specific license terms.
+3 -3
查看文件
@@ -3,7 +3,7 @@
> [!IMPORTANT]
> This document is currently in **draft** and may be subject to change.
This document is to detail the various continuous integration (CI) systems that are run on the rocm-systems monorepo.
This document is to detail the various continuous integration (CI) systems that are run on the rocm-systems super-repo.
## Table of Contents
1. [Azure Pipelines](#azure-pipelines)
@@ -25,7 +25,7 @@ This document is to detail the various continuous integration (CI) systems that
The ROCm Azure Pipelines CI (also known as External CI) is a public-facing CI system that builds and tests against latest public source code. It encompasses almost all of the ROCm stack, typically pulling source code from the `develop` or `amd-staging` branch on a component's GitHub repository. The CI's main source is publically available at [ROCm/ROCm/.azuredevops](https://github.com/ROCm/ROCm/tree/develop/.azuredevops).
See the [Azure monorepo dashboard](https://dev.azure.com/ROCm-CI/ROCm-CI/_build?definitionScope=%5Cmonorepo) for a full list of pipelines running in the monorepo.
See the [Azure super-repo dashboard](https://dev.azure.com/ROCm-CI/ROCm-CI/_build?definitionScope=%5Csuper-repo) for a full list of pipelines running in the super-repo.
For commits, the pipelines will run based on the conditions defined in the trigger files under [/.azuredevops](https://github.com/ROCm/rocm-systems/tree/develop/.azuredevops).
@@ -85,7 +85,7 @@ For testing, the majority of components use `ctest` or `gtest`. Component-specif
### Downstream Job Triggers <a id="az-downstream"></a>
Azure CI runs for a component will trigger runs for downstream components (provided that they are fully migrated onto the monorepo). The end goal is to catch upstream breaking changes before they are merged and to ensure the monorepo is always in a valid state.
Azure CI runs for a component will trigger runs for downstream components (provided that they are fully migrated onto the super-repo). The end goal is to catch upstream breaking changes before they are merged and to ensure the super-repo is always in a valid state.
For example: a rocPRIM PR will trigger a rocPRIM job. If successful, it will then continue to run hipCUB and rocThrust jobs.
+2 -2
查看文件
@@ -21,7 +21,7 @@ For anyone who wants to reach a gardener please email:
Your primary job is to keep the mono-repo shippable. In order to facilitate this we've made
status badges for all relevant CI available here:
https://github.com/ROCm/rocm-systems?tab=readme-ov-file#monorepo-status-and-ci-health.
https://github.com/ROCm/rocm-systems?tab=readme-ov-file#super-repo-status-and-ci-health.
Effectively your job is to ensure all status badges are green. All of these status
badges are clickable which will allow you to deep-dive on any failures quickly. If any
CI is missing, please file an issue leveraging the "gardener" tag, ping on the teams chat,
@@ -52,7 +52,7 @@ In scope:
- Gardeners are responsible for ensuring develop (post-submit) checks remain green.
- If a post-submit check is red, the gardeners should review the failing CI system and triage the issue.
- No matter the issue, gardeners should notify the larger gardening team at least once per day about any post-submit failures.
- If the issue is related to a failure in the CI system (not a code change), the gardener should note the issue,
- If the issue is related to a failure in the CI system (not a code change), the gardener should note the issue,
verify whether existing PRs are facing the same problem, and notify the appropriate CI team, escalating the issue if required.
- If the issue is related to a code change, the gardener should isolate the error message, and notify the
appropriate component owners with a link to the log (reference the [CODEOWNERS](../.github/CODEOWNERS) file).
+12 -12
查看文件
@@ -1,13 +1,13 @@
# How to cherry-pick monorepo changes into release-staging branches
# How to cherry-pick super-repo changes into release-staging branches
> [!IMPORTANT]
> This document is currently in **draft** and may be subject to change.
When a project has been migrated into the ROCm monorepo, day-to-day work happens on the monorepo’s `develop` branch.
Down-stream teams, however, still consume the original (pre-monorepo) repositories, particularly their `release-staging/rocm-rel-x.y` branches, through a variety of mechanisms.
This document explains how to move a change from the monorepo into those release-staging branches while guaranteeing that every commit on a release-staging branch also exists in the monorepo.
When a project has been migrated into the ROCm super-repo, day-to-day work happens on the super-repo’s `develop` branch.
Down-stream teams, however, still consume the original (pre-super-repo) repositories, particularly their `release-staging/rocm-rel-x.y` branches, through a variety of mechanisms.
This document explains how to move a change from the super-repo into those release-staging branches while guaranteeing that every commit on a release-staging branch also exists in the super-repo.
## 1. Land the change in the monorepo's develop branch
## 1. Land the change in the super-repo's develop branch
1. Create a pull request in `ROCm/rocm-systems` that targets `develop`.
2. When merging, choose **Squash & Merge** (if the change can be represented as a single logical commit).
@@ -15,7 +15,7 @@ This document explains how to move a change from the monorepo into those release
Result: The commit is now on `ROCm/rocm-systems:develop`.
## 2. Cherry-pick into the monorepo's release-staging branch
## 2. Cherry-pick into the super-repo's release-staging branch
1. Create a local branch based on the release-staging branch:
@@ -35,17 +35,17 @@ $ git cherry-pick abcd1234
## 3. Wait for the automatic “fan-out” sync
Every ~15 minutes, a CI job copies new commits from the monorepo back into the corresponding standalone repositories.
Every ~15 minutes, a CI job copies new commits from the super-repo back into the corresponding standalone repositories.
After merging your PR:
1. Monitor the CI job or simply wait ~15 minutes.
2. Go to the original (pre-monorepo) repository and verify the commits have been reflected onto the `develop` and `release-staging/rocm-rel-x.y` branches.
2. Go to the original (pre-super-repo) repository and verify the commits have been reflected onto the `develop` and `release-staging/rocm-rel-x.y` branches.
## FAQ
Q : Can I cherry-pick multiple commits at once?
A : Yes, but prefer a squash merge in the monorepo so you only need to pick one.
A : Yes, but prefer a squash merge in the super-repo so you only need to pick one.
Q : What if the auto-sync hasn’t copied the commit?
A : Verify the CI status in `rocm-systems`. If failed, ask the infra team; the commit will re-sync after a successful run.
@@ -60,8 +60,8 @@ A : It's likely that this fix/change will also be landed in develop at some poin
In short:
1. Merge change to monorepo `develop`.
2. Cherry-pick to monorepo `release-staging/rocm-rel-x.y`.
1. Merge change to super-repo `develop`.
2. Cherry-pick to super-repo `release-staging/rocm-rel-x.y`.
3. Wait for the fan-out sync and verify the changes are reflected in the original repository.
Following this process keeps release branches in sync with the monorepo while allowing critical fixes to flow to down-stream consumers.
Following this process keeps release branches in sync with the super-repo while allowing critical fixes to flow to down-stream consumers.
+19 -19
查看文件
@@ -1,7 +1,7 @@
# Migration from Single Repo to Monorepo
# Migration from Single Repo to super-repo
## Introduction
This document outlines the process for migrating from a single library repository to this monorepo. It covers the necessary steps to ensure a smooth transition, including pre-conditions, conflict resolution, and changes to repository management.
This document outlines the process for migrating from a single library repository to this super-repo. It covers the necessary steps to ensure a smooth transition, including pre-conditions, conflict resolution, and changes to repository management.
## Pre-conditions
To ensure consistency and maintainability during the migration, the following pre-conditions must be satisfied:
@@ -16,7 +16,7 @@ To ensure consistency and maintainability during the migration, the following pr
- Any point-fixes for previous releases will not be migrated over.
3. **Pause Merges:**
- There are GitHub Actions that automatically synchronize changes from the individual repos to the monorepo.
- There are GitHub Actions that automatically synchronize changes from the individual repos to the super-repo.
- These automated actions need to be paused by disabling the workflow on the GitHub UI.
- develop branch workflow: https://github.com/ROCm/rocm-systems/actions/workflows/update-subtrees.yml
- release-staging branch workflow: https://github.com/ROCm/rocm-systems/actions/workflows/update-release-staging-subtree.yml
@@ -28,19 +28,19 @@ To ensure consistency and maintainability during the migration, the following pr
1. **Automated Import of Pull Requests:**
- Pull requests without merge conflicts will be automatically imported with a GitHub Action, only executable by maintainers and admins.
- This GitHub action will create a feature branch on the monorepo, pulling in the changes from the PR on the original repo using `git subtree`.
- This GitHub action will create a feature branch on the super-repo, pulling in the changes from the PR on the original repo using `git subtree`.
- These imported pull requests will have the `imported pr` label applied.
- After running the action successfully, close the PR on the original repo.
- GitHub Action: https://github.com/ROCm/rocm-systems/actions/workflows/pr-import.yml
- Example Imported Pull Request on Monorepo: https://github.com/ROCm/rocm-systems/pull/206
- Example Imported Pull Request on super-repo: https://github.com/ROCm/rocm-systems/pull/206
- Corresponding Pull Request on Original Repo: https://github.com/ROCm/Tensile/pull/2135
2. **Conflict Resolution:**
- For pull requests with merge conflicts, add a comment explaining the merge conflict and blocking issue preventing import.
- Collaborate with contributors to import these PRs after the migration period, or the contributor can reopen the pull request themselves on the monorepo.
- Collaborate with contributors to import these PRs after the migration period, or the contributor can reopen the pull request themselves on the super-repo.
3. **NPI Development:**
- Repeat this import process for the monorepo on GitHub EMU for npi work.
- Repeat this import process for the super-repo on GitHub EMU for npi work.
### Step 2: Issue and Comment Import
@@ -51,7 +51,7 @@ To ensure consistency and maintainability during the migration, the following pr
- Ensure issue status and labels are preserved during migration.
- Look for any weird unicode characters that get mangled during the automated import.
- After running the action successfully, close the issue with a comment on the original repo.
- Example Imported Issue on Monorepo: https://github.com/ROCm/rocm-systems/issues/100
- Example Imported Issue on super-repo: https://github.com/ROCm/rocm-systems/issues/100
- Corresponding Issue on Original Repo: https://github.com/ROCm/rocThrust/issues/501
### Step 3: Path-Based Commit History
@@ -75,8 +75,8 @@ git checkout develop
git pull origin
git checkout -b filtered/hipblas-common
git filter-repo --path-rename '':'projects/hipblas-common/' --commit-callback "original_hash = commit.original_id.decode(); original_message = commit.message.decode(); new_message = f'{original_message}\\n\\n[ROCm/hipBLAS-common commit: {original_hash}]' if original_message.strip() else f'[ROCm/hipBLAS-common commit: {original_hash}]'; commit.message = new_message.encode()" --force
git remote add monorepo git@github.com:ROCm/rocm-systems.git
git push monorepo filtered/hipblas-common
git remote add super-repo git@github.com:ROCm/rocm-systems.git
git push super-repo filtered/hipblas-common
popd
git clone git@github.com:ROCm/rocm-systems.git
git checkout develop
@@ -85,7 +85,7 @@ git branch backup/develop-hipblas-common
git checkout filtered/hipblas-common
git checkout -b preserved/hipblas-common
git merge origin/develop --allow-unrelated-histories
# Set merge commit message to "Import path-preserved history of hipblas-common into the monorepo."
# Set merge commit message to "Import path-preserved history of hipblas-common into the super-repo."
git push --set-upstream origin preserved/hipblas-common
git checkout develop
git reset --hard preserved/hipblas-common
@@ -97,7 +97,7 @@ git push origin develop
### Step 4: CI/CD Triggers
1. **CI/CD Trigger Points:**
- Modify the existing CI/CD systems to be triggered off changes to this project in the monorepo.
- Modify the existing CI/CD systems to be triggered off changes to this project in the super-repo.
### Step 5: Repository Adjustments
@@ -106,7 +106,7 @@ git push origin develop
- Example: https://github.com/ROCm/rocPRIM/tree/develop_deprecated
2. **Disable Dependabot Updates:**
- Cease automatic dependency updates in the old repository to streamline the focus on the monorepo.
- Cease automatic dependency updates in the old repository to streamline the focus on the super-repo.
- Clear the contents in this file on the original repo: https://github.com/ROCm/rocPRIM/blob/develop_deprecated/.github/dependabot.yml
- In the original repo settings, go to Security -> Advanced Security and disable all the Dependabot settings.
@@ -123,8 +123,8 @@ git push origin develop
- https://github.com/ROCm/rocm-systems/blob/develop/.github/repos-config.json
- https://github.com/ROCm/rocm-systems/blob/release-staging/rocm-rel-7.0/.github/repos-config.json
2. **Update the monorepo README.md:**
- Update the migration status on the monorepo's main readme to indicate the migration has been completed.
2. **Update the super-repo README.md:**
- Update the migration status on the super-repo's main readme to indicate the migration has been completed.
- https://github.com/ROCm/rocm-systems/blob/develop/README.md
## Post-Migration Activities
@@ -137,9 +137,9 @@ git push origin develop
- Continue daily meetings and active written communications to offer support for any issues that arise.
3. **Automated Patching of Original Repos:**
- During the migration period, when a pull request is merged on the monorepo, the contents of the pull request will be split into patches to be pushed onto the original repos.
- During the migration period, when a pull request is merged on the super-repo, the contents of the pull request will be split into patches to be pushed onto the original repos.
- This supports potential pull requests that touch multiple projects.
- Example pull request on the monorepo: https://github.com/ROCm/rocm-systems/pull/230
- Example pull request on the super-repo: https://github.com/ROCm/rocm-systems/pull/230
- Corresponding patches on the original repos:
- https://github.com/ROCm/hipCUB/commit/50438ec4971def627729ea3d9dc1485e52b09e48
- https://github.com/ROCm/hipRAND/commit/74afe303def580290a8e5b149ea13ae739bc4c61
@@ -148,9 +148,9 @@ git push origin develop
- https://github.com/ROCm/rocThrust/commit/cdcc666a4c42770fcb7d9fde7c71c243b53c476e
4. **Monitoring:**
- Monitor the monorepo for any issues or discrepancies.
- Monitor the super-repo for any issues or discrepancies.
- If the automated patching for a PR failed to make it to the original repo, use this GitHub Action: https://github.com/ROCm/rocm-systems/actions/workflows/pr-merge-sync-patches-manual.yml
## Conclusion
This migration process aims to assist the ROCm development teams transition from many repos to a monorepo by addressing the topics above. By following these outlined steps, we aim to maintain and improve the quality of our development workflow post-migration.
This migration process aims to assist the ROCm development teams transition from many repos to a super-repo by addressing the topics above. By following these outlined steps, we aim to maintain and improve the quality of our development workflow post-migration.