About plainbox-provider-certification-{client,server}

[Zygmunt Krynicki]

Hey

tl;dr; plainbox-provider-certification-{client,server} now live in
https://code.launchpad.net/plainbox-provider-canonical-certification
and releases are using git-dpm. Read on for details.

I wanted to explain my actions affecting both certification providers
that occurred last week. As you know, the last maintenance release did
happen, but it was made outside of lp:checkbox. This happened due to a
combination of several factors. You can read about those below. Now
that it is done, with [1] being merged, let me tell how how the two
providers are going to live on.

First of all, this is the first time we're going to use git on a
plainbox provider natively. There might be some learning curve though
it should be related only to git-on-launchpad and UI issues rather to
git itself.

The workflow is conceptually the same as before, where we propose a
merge request to a given branch an after it lands, it's done. The only
visible difference here now is that there are two branches
(master-client and master-server), each holding exactly one of the two
providers. This is strictly related to simplicity, tagging and
management of releases. I wanted to move to a model where each
provider can essentially live in a small repository and everything
related to the life cycle of that provider would be no different than
any other provider that follows the same pattern.

The reason there are two branches is very simple, I didn't want to
split the Launchpad project in two, given how complex and time
consuming that is. With git it's equally easy to track and work on one
or one hundred branches, as long as they share one repository. So in
practice, everything relating to the *client* provider is in the
master-client branch while the *server* provider is in the
master-server branch. If you find that confusing we can trivially
rename that to just client and server, there's no problem in doing
that.

Next thing is merge requests. Apart from the slight UI change (you
have to target a branch _in_ a repository rather than branch alone)
everything works as before. I'm sure there will be a few WTF moments
as we all learn the new Launchpad user interface but we'll get over
that soon and the knowledge is useful for all the other projects we
have, or are moving to git. After the merge request is approved a CI
automaton will merge it. This is still a bit up in the air. I'll work
with a few people to enhance our CI system to work with git and bzr
together (i.e. switch to the new "tarmac" prototype). This will happen
over the next few weeks. We need this for a number of other projects
that started off with git. This will be no different and I will send
updates as we progress on this effort.

The only thing that *is* significantly different is how any kind of
"transition" is made. In the past we could have easily made a job
rename in one commit, that touches all providers at the same time
(well, except for those that are already out-of-tree, like CDTS). Now
that will need to be coordinated. This also leads us to the greater
importance of "API" stability. For a provider, the API is the set of
jobs it contains. I don't think this is a major issue, we already have
that problem with our downstream project/commercial derivatives. We
will just have to tread ourselves the same way as we treat those
projects. With planning, consideration, tooling and communication.

The last thing, and also the biggest change, is how releases work. The
key reason I moved away from lp:checkbox was the broken release
support for this pair providers. Rather than add more complex tooling
I wanted to removed the excessive complexity that made the tools
needed in the first place. Releases are simple: they are based on
signed tags and tarballs (generated with ./manage.py sdist). Tarballs
are uploaded to Launchpad (to the
plainbox-provider-canonical-certification project). This is exactly
how we already treat all of the other providers. We now have less
divergence as as result of this. This tarball is then combined with a
packaging branch, like we do in all the Debian and Ubuntu releases
(though not as we do in our PPAs) and the resulting source package is
uploaded to a PPA for building.

The major thing here, is that we have now adopted git-dpm. Git DPM is
the preferred form of modification of all the packages in Debian and
in Ubuntu. All our Debian packaging will be automatically converted to
git-dpm in the next few days (or week). This is already done in
testing but the final trigger will be pulled at around after Debconf.
This is a Debian-wide change and we are just following the transition.
Git DPM is *fantastic*. If you ever plan to, or need to, work on a
release you need to learn this tool. The manual page is a great start.
There are also several online tutorials but I think there are
overcomplicated for what this tool is doing. Your mileage may vary.
Git DPM lets us work, in one tree, with our upstream branches, our PPA
releases and our Debian releases. I'm sure that we're just getting
started and as we explore and try we will simplify our release process
greatly.

There is a downside though, as Launchpad git auto-builder is not yet
functional we have lost the ability to do daily builds straight from
the repository. This is not a major change, because this is a project
that sees less updates than others, but it is a loss non the less. As
Launchpad evolves this functionality will be restored. For the moment,
and for our other projects that are also on Git I will be working on
simple tooling (thanks to git DPM) that lets us upload daily source
packages from a cron job somewhere.

The one thing that you must remember from this, is that our packaging
is no longer based on the tree export from the bzr repository. This is
a *big* change as now packaging for all kinds of releases (Ubuntu,
Debian, PPA, daily PPA, etc) for this and all projects that follow the
same pattern is _identical_. This is a major improvement over our
prior model, where packaging was totally different an never kept
up-to-date. The certification provider does not have Debian or Ubuntu
releases (and likely won't ever have) but the same release tooling
(git-dpm) applies equally to all of those. I strongly encourage you to
look at git-dpm, at the packaging branch [2]. I recommend that you
clone it, and follow the git-dpm manual page to build it. There are
helpful instructions in the debian/README.Debian file.

If you have any questions about this please don't hesitate to contact me.

Best regards
ZK

[1] https://code.launchpad.net/~zyga/checkbox/move-cert-providers-to-git/+merge/268197
[2] https://code.launchpad.net/~checkbox-dev/plainbox-provider-canonical-certification/+git/packaging-client