usd-import beta2!

Nish Aravamudan nish.aravamudan at
Fri Jun 3 01:32:05 UTC 2016

[Adapted from an e-mail I sent earlier today to explain the latest
version of the tooling, which felt like something I should broadcast
more publicly.]

Couple of quick links first:

The importer and helper tools live at:

git-dsc-commit, git-reconstruct-changelog and git-merge-changelogs (as
well as xgit) are at:
[I sent a PR today to pull in the --tree-only mode for git-dsc-commit
which is needed for the importer]

The imported trees temporarily are at:

I say temporarily because once we feel confident in the utility and
correctness of the importer, we'll move them to

On to some details:

The first problem we were trying to solve was how to make merges
consistent across the server team, and, ideally easy to review for
sponsors/uploaders. Robie Basak and others came up with a git-based
workflow (documented roughly at the above github URL, and I plan on
making a entry tomorrow for the same with details and
examples, including much of this e-mail). This workflow uses git (the
only addition to 'stock' git are a few commands (git-dsc-commit,
git-merge-changelogs, git-reconstruct-changelog) from
git:// to effectively rebase the
ubuntu tip onto the latest debian tip. That presumed you spent the time
yourself to create a repository with commits representing those tips.
The goal for the importer is to have canonical (little c) location for
such commits to live, per-package.

Scott Moser then asked if we could import the entire history for a
package, because he wanted to `git-blame` a give file and get useful
data. So we extended the algorithm (that Robie designed) to be more
flexible. After hitting many corner-cases during implementation, we
scrapped our complicated algorithm that produced clean trees for a clean
algorithm that produces complicated trees.

In essence, the algorithm looks at Launchpad's publishing history for
versions it hasn't seen before (which if an empty or no local repository
is specified and you aren't cloning an existing repository will be all
of them). For each such version, it uses
`pull-lp-source`/`pull-debian-source` equivalents and `git-dsc-commit`
to import them into the git repository. Technically it uses a
lower-level command then a proper commit (`git write-tree` and `git
commit-tree`), so that we can get the imported tree, examine it and find
it's parents and then commit it. The parents for an imported tree are at
most 2:
    1) The last version imported into the same series/pocket, with some
    knowledge of how to establish a new series/pocket.
    2) The last debian/changelog entry (using debian/changelog from the
    just-imported tree) that was successfully imported.
We call these the 'publishing parent' and 'changelog parent'
respectively. Now, if we can't find either of these, we will orphan the
import and if we only find one, we'll use what we have. But the
resulting tree looks like many git-merges, even where there are not
ubuntu-merges. This is correct by the algorithm but can be confusing to
the original `git-rebase` workflow, because rebase will try to replay
the git-merges and that doesn't work.

So we have created a simple helper script (also in the same repositry as
the importer) entitled (for now) 'usd-import-reconstruct-merge' which
can take a usd-import'd tree and a commitish and attempts to give you a
reconstructed sequence of linear commits that represent the same state
as the commitish. It does this by using merge-base to figure out the
common ancestor (it assumes onto is debian/sid but it also accepts a
parameter) and then playing back the debian/changelog looking for
imported tags. It then cherry-picks those tags from oldest to newest
against the merge-base and does a quick sanity check that the resulting
commit does not differ from the original one.

It tags that as reconstruct/<version> which sort of conflicts with the
git-based workflow some have been used. In that workflow,
reconstruct/<version> is the broken-down sequence of changes including
changelog and metadata, where each commit is a single change from the
changelog. I have taken to now calling that pointer
deconstruct/<version> because it's clever.

So you would break down reconstruct/<version> into its constituent
changes (per debian/changelog for each version) and tag the resulting
end-commit as deconstruct/<version>.

That can then be broken up by our workflow into a logical/<version> tag
and then rebased onto debian/sid (or the specified onto).

Some last thoughts:

a) SRU/maintenance might become easier because you can cherry-pick
commits around to the right head, build the source package
(dpkg-buildpackage -S) from the tree directly, etc.

b) The resulting gitk graph really demonstrates what Ubuntu is, for a
specific package. And it clearly indicates when a package has not been
merged recently and when it has.

Feel free to respond with any questions or feedback!
Nishanth Aravamudan
Ubuntu Server
Canonical Ltd

More information about the ubuntu-server mailing list