Terminology change: monorepo -> super-repo

Šī revīzija ir iekļauta:
Joseph Macaranas
2025-08-02 22:23:24 -04:00
vecāks 153f71fb0d
revīzija 870dc948b1
20 mainīti faili ar 109 papildinājumiem un 184 dzēšanām
+3 -3
Parādīt failu
@@ -3,7 +3,7 @@
> [!IMPORTANT]
> This document is currently in **draft** and may be subject to change.
This document is to detail the various continuous integration (CI) systems that are run on the rocm-systems monorepo.
This document is to detail the various continuous integration (CI) systems that are run on the rocm-systems super-repo.
## Table of Contents
1. [Azure Pipelines](#azure-pipelines)
@@ -25,7 +25,7 @@ This document is to detail the various continuous integration (CI) systems that
The ROCm Azure Pipelines CI (also known as External CI) is a public-facing CI system that builds and tests against latest public source code. It encompasses almost all of the ROCm stack, typically pulling source code from the `develop` or `amd-staging` branch on a component's GitHub repository. The CI's main source is publically available at [ROCm/ROCm/.azuredevops](https://github.com/ROCm/ROCm/tree/develop/.azuredevops).
See the [Azure monorepo dashboard](https://dev.azure.com/ROCm-CI/ROCm-CI/_build?definitionScope=%5Cmonorepo) for a full list of pipelines running in the monorepo.
See the [Azure super-repo dashboard](https://dev.azure.com/ROCm-CI/ROCm-CI/_build?definitionScope=%5Csuper-repo) for a full list of pipelines running in the super-repo.
For commits, the pipelines will run based on the conditions defined in the trigger files under [/.azuredevops](https://github.com/ROCm/rocm-systems/tree/develop/.azuredevops).
@@ -85,7 +85,7 @@ For testing, the majority of components use `ctest` or `gtest`. Component-specif
### Downstream Job Triggers <a id="az-downstream"></a>
Azure CI runs for a component will trigger runs for downstream components (provided that they are fully migrated onto the monorepo). The end goal is to catch upstream breaking changes before they are merged and to ensure the monorepo is always in a valid state.
Azure CI runs for a component will trigger runs for downstream components (provided that they are fully migrated onto the super-repo). The end goal is to catch upstream breaking changes before they are merged and to ensure the super-repo is always in a valid state.
For example: a rocPRIM PR will trigger a rocPRIM job. If successful, it will then continue to run hipCUB and rocThrust jobs.
+2 -2
Parādīt failu
@@ -21,7 +21,7 @@ For anyone who wants to reach a gardener please email:
Your primary job is to keep the mono-repo shippable. In order to facilitate this we've made
status badges for all relevant CI available here:
https://github.com/ROCm/rocm-systems?tab=readme-ov-file#monorepo-status-and-ci-health.
https://github.com/ROCm/rocm-systems?tab=readme-ov-file#super-repo-status-and-ci-health.
Effectively your job is to ensure all status badges are green. All of these status
badges are clickable which will allow you to deep-dive on any failures quickly. If any
CI is missing, please file an issue leveraging the "gardener" tag, ping on the teams chat,
@@ -52,7 +52,7 @@ In scope:
- Gardeners are responsible for ensuring develop (post-submit) checks remain green.
- If a post-submit check is red, the gardeners should review the failing CI system and triage the issue.
- No matter the issue, gardeners should notify the larger gardening team at least once per day about any post-submit failures.
- If the issue is related to a failure in the CI system (not a code change), the gardener should note the issue,
- If the issue is related to a failure in the CI system (not a code change), the gardener should note the issue,
verify whether existing PRs are facing the same problem, and notify the appropriate CI team, escalating the issue if required.
- If the issue is related to a code change, the gardener should isolate the error message, and notify the
appropriate component owners with a link to the log (reference the [CODEOWNERS](../.github/CODEOWNERS) file).
@@ -1,13 +1,13 @@
# How to cherry-pick monorepo changes into release-staging branches
# How to cherry-pick super-repo changes into release-staging branches
> [!IMPORTANT]
> This document is currently in **draft** and may be subject to change.
When a project has been migrated into the ROCm monorepo, day-to-day work happens on the monorepos `develop` branch.
Down-stream teams, however, still consume the original (pre-monorepo) repositories, particularly their `release-staging/rocm-rel-x.y` branches, through a variety of mechanisms.
This document explains how to move a change from the monorepo into those release-staging branches while guaranteeing that every commit on a release-staging branch also exists in the monorepo.
When a project has been migrated into the ROCm super-repo, day-to-day work happens on the super-repos `develop` branch.
Down-stream teams, however, still consume the original (pre-super-repo) repositories, particularly their `release-staging/rocm-rel-x.y` branches, through a variety of mechanisms.
This document explains how to move a change from the super-repo into those release-staging branches while guaranteeing that every commit on a release-staging branch also exists in the super-repo.
## 1. Land the change in the monorepo's develop branch
## 1. Land the change in the super-repo's develop branch
1. Create a pull request in `ROCm/rocm-systems` that targets `develop`.
2. When merging, choose **Squash & Merge** (if the change can be represented as a single logical commit).
@@ -15,7 +15,7 @@ This document explains how to move a change from the monorepo into those release
Result: The commit is now on `ROCm/rocm-systems:develop`.
## 2. Cherry-pick into the monorepo's release-staging branch
## 2. Cherry-pick into the super-repo's release-staging branch
1. Create a local branch based on the release-staging branch:
@@ -35,17 +35,17 @@ $ git cherry-pick abcd1234
## 3. Wait for the automatic “fan-out” sync
Every ~15 minutes, a CI job copies new commits from the monorepo back into the corresponding standalone repositories.
Every ~15 minutes, a CI job copies new commits from the super-repo back into the corresponding standalone repositories.
After merging your PR:
1. Monitor the CI job or simply wait ~15 minutes.
2. Go to the original (pre-monorepo) repository and verify the commits have been reflected onto the `develop` and `release-staging/rocm-rel-x.y` branches.
2. Go to the original (pre-super-repo) repository and verify the commits have been reflected onto the `develop` and `release-staging/rocm-rel-x.y` branches.
## FAQ
Q : Can I cherry-pick multiple commits at once?
A : Yes, but prefer a squash merge in the monorepo so you only need to pick one.
A : Yes, but prefer a squash merge in the super-repo so you only need to pick one.
Q : What if the auto-sync hasnt copied the commit?
A : Verify the CI status in `rocm-systems`. If failed, ask the infra team; the commit will re-sync after a successful run.
@@ -60,8 +60,8 @@ A : It's likely that this fix/change will also be landed in develop at some poin
In short:
1. Merge change to monorepo `develop`.
2. Cherry-pick to monorepo `release-staging/rocm-rel-x.y`.
1. Merge change to super-repo `develop`.
2. Cherry-pick to super-repo `release-staging/rocm-rel-x.y`.
3. Wait for the fan-out sync and verify the changes are reflected in the original repository.
Following this process keeps release branches in sync with the monorepo while allowing critical fixes to flow to down-stream consumers.
Following this process keeps release branches in sync with the super-repo while allowing critical fixes to flow to down-stream consumers.
+19 -19
Parādīt failu
@@ -1,7 +1,7 @@
# Migration from Single Repo to Monorepo
# Migration from Single Repo to super-repo
## Introduction
This document outlines the process for migrating from a single library repository to this monorepo. It covers the necessary steps to ensure a smooth transition, including pre-conditions, conflict resolution, and changes to repository management.
This document outlines the process for migrating from a single library repository to this super-repo. It covers the necessary steps to ensure a smooth transition, including pre-conditions, conflict resolution, and changes to repository management.
## Pre-conditions
To ensure consistency and maintainability during the migration, the following pre-conditions must be satisfied:
@@ -16,7 +16,7 @@ To ensure consistency and maintainability during the migration, the following pr
- Any point-fixes for previous releases will not be migrated over.
3. **Pause Merges:**
- There are GitHub Actions that automatically synchronize changes from the individual repos to the monorepo.
- There are GitHub Actions that automatically synchronize changes from the individual repos to the super-repo.
- These automated actions need to be paused by disabling the workflow on the GitHub UI.
- develop branch workflow: https://github.com/ROCm/rocm-systems/actions/workflows/update-subtrees.yml
- release-staging branch workflow: https://github.com/ROCm/rocm-systems/actions/workflows/update-release-staging-subtree.yml
@@ -28,19 +28,19 @@ To ensure consistency and maintainability during the migration, the following pr
1. **Automated Import of Pull Requests:**
- Pull requests without merge conflicts will be automatically imported with a GitHub Action, only executable by maintainers and admins.
- This GitHub action will create a feature branch on the monorepo, pulling in the changes from the PR on the original repo using `git subtree`.
- This GitHub action will create a feature branch on the super-repo, pulling in the changes from the PR on the original repo using `git subtree`.
- These imported pull requests will have the `imported pr` label applied.
- After running the action successfully, close the PR on the original repo.
- GitHub Action: https://github.com/ROCm/rocm-systems/actions/workflows/pr-import.yml
- Example Imported Pull Request on Monorepo: https://github.com/ROCm/rocm-systems/pull/206
- Example Imported Pull Request on super-repo: https://github.com/ROCm/rocm-systems/pull/206
- Corresponding Pull Request on Original Repo: https://github.com/ROCm/Tensile/pull/2135
2. **Conflict Resolution:**
- For pull requests with merge conflicts, add a comment explaining the merge conflict and blocking issue preventing import.
- Collaborate with contributors to import these PRs after the migration period, or the contributor can reopen the pull request themselves on the monorepo.
- Collaborate with contributors to import these PRs after the migration period, or the contributor can reopen the pull request themselves on the super-repo.
3. **NPI Development:**
- Repeat this import process for the monorepo on GitHub EMU for npi work.
- Repeat this import process for the super-repo on GitHub EMU for npi work.
### Step 2: Issue and Comment Import
@@ -51,7 +51,7 @@ To ensure consistency and maintainability during the migration, the following pr
- Ensure issue status and labels are preserved during migration.
- Look for any weird unicode characters that get mangled during the automated import.
- After running the action successfully, close the issue with a comment on the original repo.
- Example Imported Issue on Monorepo: https://github.com/ROCm/rocm-systems/issues/100
- Example Imported Issue on super-repo: https://github.com/ROCm/rocm-systems/issues/100
- Corresponding Issue on Original Repo: https://github.com/ROCm/rocThrust/issues/501
### Step 3: Path-Based Commit History
@@ -75,8 +75,8 @@ git checkout develop
git pull origin
git checkout -b filtered/hipblas-common
git filter-repo --path-rename '':'projects/hipblas-common/' --commit-callback "original_hash = commit.original_id.decode(); original_message = commit.message.decode(); new_message = f'{original_message}\\n\\n[ROCm/hipBLAS-common commit: {original_hash}]' if original_message.strip() else f'[ROCm/hipBLAS-common commit: {original_hash}]'; commit.message = new_message.encode()" --force
git remote add monorepo git@github.com:ROCm/rocm-systems.git
git push monorepo filtered/hipblas-common
git remote add super-repo git@github.com:ROCm/rocm-systems.git
git push super-repo filtered/hipblas-common
popd
git clone git@github.com:ROCm/rocm-systems.git
git checkout develop
@@ -85,7 +85,7 @@ git branch backup/develop-hipblas-common
git checkout filtered/hipblas-common
git checkout -b preserved/hipblas-common
git merge origin/develop --allow-unrelated-histories
# Set merge commit message to "Import path-preserved history of hipblas-common into the monorepo."
# Set merge commit message to "Import path-preserved history of hipblas-common into the super-repo."
git push --set-upstream origin preserved/hipblas-common
git checkout develop
git reset --hard preserved/hipblas-common
@@ -97,7 +97,7 @@ git push origin develop
### Step 4: CI/CD Triggers
1. **CI/CD Trigger Points:**
- Modify the existing CI/CD systems to be triggered off changes to this project in the monorepo.
- Modify the existing CI/CD systems to be triggered off changes to this project in the super-repo.
### Step 5: Repository Adjustments
@@ -106,7 +106,7 @@ git push origin develop
- Example: https://github.com/ROCm/rocPRIM/tree/develop_deprecated
2. **Disable Dependabot Updates:**
- Cease automatic dependency updates in the old repository to streamline the focus on the monorepo.
- Cease automatic dependency updates in the old repository to streamline the focus on the super-repo.
- Clear the contents in this file on the original repo: https://github.com/ROCm/rocPRIM/blob/develop_deprecated/.github/dependabot.yml
- In the original repo settings, go to Security -> Advanced Security and disable all the Dependabot settings.
@@ -123,8 +123,8 @@ git push origin develop
- https://github.com/ROCm/rocm-systems/blob/develop/.github/repos-config.json
- https://github.com/ROCm/rocm-systems/blob/release-staging/rocm-rel-7.0/.github/repos-config.json
2. **Update the monorepo README.md:**
- Update the migration status on the monorepo's main readme to indicate the migration has been completed.
2. **Update the super-repo README.md:**
- Update the migration status on the super-repo's main readme to indicate the migration has been completed.
- https://github.com/ROCm/rocm-systems/blob/develop/README.md
## Post-Migration Activities
@@ -137,9 +137,9 @@ git push origin develop
- Continue daily meetings and active written communications to offer support for any issues that arise.
3. **Automated Patching of Original Repos:**
- During the migration period, when a pull request is merged on the monorepo, the contents of the pull request will be split into patches to be pushed onto the original repos.
- During the migration period, when a pull request is merged on the super-repo, the contents of the pull request will be split into patches to be pushed onto the original repos.
- This supports potential pull requests that touch multiple projects.
- Example pull request on the monorepo: https://github.com/ROCm/rocm-systems/pull/230
- Example pull request on the super-repo: https://github.com/ROCm/rocm-systems/pull/230
- Corresponding patches on the original repos:
- https://github.com/ROCm/hipCUB/commit/50438ec4971def627729ea3d9dc1485e52b09e48
- https://github.com/ROCm/hipRAND/commit/74afe303def580290a8e5b149ea13ae739bc4c61
@@ -148,9 +148,9 @@ git push origin develop
- https://github.com/ROCm/rocThrust/commit/cdcc666a4c42770fcb7d9fde7c71c243b53c476e
4. **Monitoring:**
- Monitor the monorepo for any issues or discrepancies.
- Monitor the super-repo for any issues or discrepancies.
- If the automated patching for a PR failed to make it to the original repo, use this GitHub Action: https://github.com/ROCm/rocm-systems/actions/workflows/pr-merge-sync-patches-manual.yml
## Conclusion
This migration process aims to assist the ROCm development teams transition from many repos to a monorepo by addressing the topics above. By following these outlined steps, we aim to maintain and improve the quality of our development workflow post-migration.
This migration process aims to assist the ROCm development teams transition from many repos to a super-repo by addressing the topics above. By following these outlined steps, we aim to maintain and improve the quality of our development workflow post-migration.