[braindump] ppa update discontents

[Max Bowsher]

On 20/08/10 09:47, Martin Pool wrote:
> Along with maxb and garyvdm I've been updating the bzr beta ppa for
> our 2.2 release.  This is not absolutely the most fun thing I've ever
> done; in fact it's pretty tedious.

Hi Martin :-)

> It's the kind of tedious work that doesn't require much creative input
> but is not very automated, seems to have a bunch of snags that make it
> not trivially automatable, and with long latency, so you can't just
> sit down and do it then know you're done.  It's about 10 minutes of
> actual thinking spread over a couple of days of spade work.

I spent 3 hours last night in which I made 29 uploads to the
bzr/proposed PPA - I didn't find it particularly tedious, and the most
time consuming parts were wrestling the dulwich testsuite into working
with older git versions in older distroseries, and familiarizing myself
with the packaging branches in use.

I think the critical reason I have a different viewpoint on the
tedium-level involved is that I already _have_ automated the repetitive
bits, and I'm strongly familiar with the general packaging process.

> We have about ten packages to rebuild, and we want to do do that
> across four distroseries (hardy, jaunty, karmic, lucid, maverick).  So
> anything that happens once per package build is quite a lot of work;
> and there's a fair amount of human-in-the-loop latency per package.
> Doing them all seems to be multiple days work, even with some amount
> of automation.  Ideally we would update on every release but to be
> able to do that we have to scale better.
> 
> Most of what we do is just add a rebuild line into the debian
> changelog, push, and upload, but things occasionally go wrong per
> package.  This should be very automatable.

A lot of the packages I touched weren't using packaging branches at all
- ironically, this actually made my life easier, when I could simply
take the package from maverick/sid, apply boilerplate debian/changelog
tweaks, and upload.

Indeed, I scripted this (which was easy for me since it was a trivial
derivation of a script I use for other PPAs). The script is attached,
for the curious, but basically it does the following:

Take a .dsc file as input.
Unpack the .dsc.
Use the dch tool to tweak the changelog.
Pack up the modified source, and generate a .changes file.
Optionally sign it.
Optionally upload it.

So with all that, I actually had a single-line invocation to take the
debian sid version of a package sitting on my local disk, and upload it
to the PPA, for a given series.

Of course, I swiftly ran into the problem of source format 3.0 - so I
enhanced the script to automatically transform source format 3.0 to
source format 1.0 when targeting jaunty or below.

Now, certainly, this isn't a magic bullet - it's highly dependent on the
Debian sid source package building without modifications in all the
target series - but for a lot of bzr plugins, that's the case.

> It can be a bit hard to work out which packaging branch is used for
> the version in our ppa.  Launchpad has a concept of series branches
> for We can have a convention for plugins that we package, but it's
> just a convention and because it's evolved over time it's not always
> consistent.  For instance if Gary has uploaded bzr-gtk -
> 0.99.0-1~hardy4 but it's failing, I can get the source for that
> package and try to fix it, but I can't easily work out what branch
> contains that source.  (Well, I can guess it's one of the recently
> updated branches in <https://code.edge.launchpad.net/bzr-gtk> but it's
> not great.)

I think this just requires that the project set a convention and use it.
I have a thread on the bazaar ML proposing we do so.

> Some external dependencies of bzr (like subvertpy, testtools, and
> subunit) are packaged into other ppas.  To make the install experience
> better we copy all the dependency packages into our ppa.  This works
> pretty well.  It does introduce another possible level of
> inconsistency or confusion though: not everything in our ppa will have
> a branch in the same place, and not everything is meant to be updated
> in the same way.  (bzr packages you should directly upload; subvertpy
> you should prefer to copy from somewhere else.)

Well, interesting that you should mention that. For subvertpy and
dulwich, we're now going with the solution of building them directly in
the bazaar PPAs, as Jelmer has advised that he doesn't intend that those
PPAs aim to package the latest release across all distroseries with any
promptness.

I'm inclined to suggest we do the same for testtools and subunit,
especially given that the versioning of those other PPAs seems to be
violating the best practice of not making PPA version strings that look
like ones that would be in the main archive.

> Maverick's bzr has its copy of python-configobj cut out, but that
> library is not shipped on hardy.  Dealing with things like this make
> me wonder a lot whether it's a good use of time to keep supporting
> many old platforms.

Definitely a concern, but in this case a simple backport of the
python-configobj package ought to sort matters.

> The lag seems to come in mostly through Soyuz delays: there's a delay
> before your package is accepted, then a delay that was running at
> several hours yesterday before it's actually built or rejected.  I do
> realize that there's a lot of demand for Soyuz build machines and that
> people are working on improving scheduling and performance.  But still
> it does mean there's this long period where you have mental
> work-in-progress, and can't wholeheartedly move on to something else.
> If there's a mistake either in your work, or the scripts, you have to
> page in the state you had a while ago.  The long lag is pretty
> unreasonable when bzr has barely any compilation steps and builds in
> well under a minute on a laptop.

This is absolutely and definitely a problem. For example, last night I
uploaded dulwich packages, which FTBFS due to a silly limitation in the
bit of sbuild that processes build-deps. And I had to wait until the
next day to find out about this, and then wait another day for my fixed
packages to build.

> One approach to the lag in soyuz builds is to test build every package
> locally, before uploading.  This seems like a bit of an unnecessary
> kludge, but perhaps it's the most sensible thing for now.  Since we
> want to build on various different distroseries we have to use some
> kind of virtual environment.  There are various options including
> pbuilder, but none that I've found are really plug-and-play.  I have a
> wrapper script passed on by Robert that gives it some better defaults.
>  I think this doesn't yet take into account that some build
> dependencies might need to come from either the destination or other
> ppas, but we could possibly add that.

I'm not sure it's a kludge, so much as a totally sensible part of
preparing an upload. However, given I keep a full set of cowbuilder
chroots updated for all active distroseries anyway, what could be of
great annoyance to many, is fairly trivial to me.

Furthermore, there's the gotcha of what to do when you want to build
something - say bzrtools - which depends on another package in a ppa -
like the bzr of the right version.

In my case my .pbuilderrc contains magic so that I pass a
ppas=bzr/proposed envvar, pass --override-config to the cowbuilder
invocation, and it Just Does It.

Again, a case of having the tooling already available saving the day.

> It seems like we have to keep track of which builds have passed or
> failed by collecting a set of nearly-identical mails.  There's a
> "latest updates" portlet in eg
> <https://edge.launchpad.net/~bzr/+archive/proposed> but it only shows
> the last 5 which is not really enough (http://pad.lv/620903).

https://edge.launchpad.net/~bzr/+archive/proposed/+packages is fit for
the purpose.

> Perhaps we could standardize a way to do local test builds in a way as
> close as possible to what Launchpad does, including how to get
> bzr-builddeb to use it for a a test build before uploading.

pbuilder (well, cowbuilder, which makes pbuilder fast enough to be
pleasant) does a very nice job most of the time.

sbuild (what's actually used on the buildds) is also an option, if a
little trickier to set up, and requiring LVM for the kind of
copy-on-write behaviour that cowbuilder gives. (Though I hear sbuild has
recently added union-fs support.)

> Our process at the moment is to upload everything into
> <https://launchpad.net/~bzr/+archive/+proposed> before then promoting
> it to the regular archive, so that we don't break dependencies between
> packages.  (Uploading a bzr newer than the plugins support might make
> them uninstallable, and unlike the main archive(?) ppas don't
> themselves check for this.)

Nope, the main archive has no more magic here.

> I think the basic approach of using a
> staging archive is ok.  We ought to be able to script something that
> does both the check and the promotion if it succeeds.

edos-debcheck would be something to look into for the check part.
launchpadlib scripts can easily accomplish the promotion part.

> I discovered that you can copy binaries from maverick into lucid,
> assuming it's a pure python package that doesn't need to be rebuilt.
> This makes things a little easier.  We could add a tool to hydrazine
> to do it.

This always makes me a little nervous. Yes, it'll often work for the
development release to the next-older release. But, it's quite hard to
reliably spot when it'll stop working. For this reason, I tend to favour
a rebuild anyway.



Max.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: bzrppa-backport
Url: https://lists.ubuntu.com/archives/ubuntu-distributed-devel/attachments/20100821/14fe484a/attachment-0001.txt 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/ubuntu-distributed-devel/attachments/20100821/14fe484a/attachment-0001.pgp