mariatuplayground/ Morph/ repositories

Using git repositories with Baserock

  • There is a git repository corresponding to each upstream project. This repository gets mirrored (possibly with conversion) from upstream version control systems or release tarballs.

    • exact details of the mirroring are not yet specified, but are irrelevant for this discussion, as long as the result is a set of git repositories
    • no non-upstream branches get automatically changed by the mirroring
    • if upstream does not contain the morphologies, we add them in a feature branch, baserock/morphologies, which has no other changes; see below how this gets integrated during build
    • if the morphologies need to be changed, we create a feature branch baserock/foo off baserock/morphologies, merge in changes from upstream's master branch , make and test any fixes, and then merge the changes to baserock/morphologies

  • Additionally, there is at least one more git repository for stratum and system morphologies. For simplicity, we will assume they're all in one repo and call this the morphs repo.

    • in reality, it may be more convenient or necessary to have several repositories, e.g., GNOME might have their own, and NDAs may prevent some custom hardware support projects from being kept on a public git server until the hardware is released
    • the master branch is where main development happens
    • in master, the morphologies refer to the upstream master branches in upstream project repos, so that when we build from master, we get the current development version of everything
    • they also refer to any baserock/morphologies branches if they exist

  • When something needs to be changed, it always affects at least the morphs repository, and may affect one or more upstream repositories. The changes will be done in dedicated feature branches.

    • in all affected repos, create new branches baserock/foo (using the same branch name in all repos)
    • in the morphs repo, the new branch is based on whatever was the current branch
    • in other repositories, the new branch is based on whatever main branch the morphologies refer to (i.e., the first branch)
    • in the morphs repo, change the morphologies to refer to the baserock/foo branch in the other affected repositories; this allows you to build binaries that use the changes you make in the other repositories
    • however, repositories that are not affected by the change will continue to refer to their original repositories and branches; this avoids creating a lot of unnecessary branches all over the place
    • changes will remain in the new branches; see below for integration

  • Feature branches are integrated by specifying several branches for an upstream project in a morphology. The morph tool will merge the changes into a unified branch for building.

    • a stratum morphology may specify, e.g., that busybox needs to be built from the baserock/master, baserock/bug-12765, and baserock/mkfs.btrfs branches
    • the merging of feature branches into an integration branch is thus specified in morphologies, rather than in an external configuration file for a continuous integration tool; this keeps everything nicely explicit and version controlled
    • if the merging fails, e.g., because the various feature branches conflict, the build fails

  • When it is time to prepare for a release, it is necessary to branch all constituent projects in such a way that normal development changes are no longer automatically included in the release candidate, but changes are still comfortable to make manually.

    • morphs is branched into baserock/petrifying/foo, where foo is a name or version number for the upcoming release, picked by the user
    • the new branch in morphs may be based on master or any other suitable branch
    • for every upstream project referred to from any morphology, a similar branch is made: baserock/petrifying/foo; these branches are based on whatever main (first) branches are referred to from the morphologies
    • it is an error for the morphologies to refer to different main branches of the same upstream project, since this causes a conflict of which branch to base baserock/petrifying/foo on
    • the morphologies are updated to refer to the new branches
    • the release candidate can be updated by making changes to the baserock/petrifying/foo branches manually; no upstream changes will automatically flow into the release candidate, they must all be merged in by hand

  • When a release is ready, it is petrified. At this point, the versions (commits) of all constituent projects are nailed down in such a way that they cannot be (easily) changed again.

    • morphs is branched from baserock/petrifying/foo to baserock/petrified/foo
    • in all morphologies the reference to the upstream projects' baserock/petrifying/foo branch is changed to the commit id of the HEAD of the branch.
    • this allows the release to be reproduced exactly at any later time, regardless of what happens to the repositories, as long as the specific commit id still exists
    • the baserock/petrifying/foo branch is kept so that future releases can be based on the work done so far
    • baserock/petrifying/foo is tagged with the exact release number (e.g, baserock/petrifying/1.2-series would be tagged version-1.2.0), so that it's easier to find where the release happened

  • A continuous integration tool builds and tests system images from the morphs repository.

    • master is always tested whenever it or any constituent project changes
    • baserock/petrifying/* are similarly always tested when something in them changes
    • baserock/petrified/* as well, though nothing in them should ever actually change (any change in a release would result in a new release being done)
    • other branches may also be configured manually, when necessary
    • for each branch, the CI tool builds the system image, and runs automatic tests, reporting results in some suitable manner; the tests will include upgrading from previous releases, and probably should include installation and upgrading and removal of specific strata on some of the systems, but that's getting beyond the scope of this

What a lotta branches

Keeping things in feature branches sounds like a good idea so that:

  • upstream can more easily see what we've changed, and can easily incorporate every change separately
  • it's easy to back out of a set of changes (just drop it from the integration)
  • it's easy to combine the same fix into many deliverables (e.g., an eglibc bug fix might be backported into several releases)

I hate this

The above sounds like a lot of bureaucratic hassle that will get really annoying and horrifically error prone very quickly. However, fear not, the morph tool will be make it all painless. In fact, the above is an outline for a design for how morph will work behind the scenes. What the user actually sees will be much simpler.