stable+1 team December report/handover

Martin Pitt martin.pitt at ubuntu.com
Tue Dec 20 10:47:57 UTC 2011


Hello Ubuntu developers,

the second monthly cycle of our new Stable+1 maintenance team [1]
rotation is almost over. In December, the team consisted of James
Page, Allison Randal, and myself.

James and Allison are already on vacation, and yesterday was my last
day, so time to summarize our stable+1 activity for December and
describe the current status.

Daily ISO
---------
https://jenkins.qa.ubuntu.com/view/Precise%20ISO%20Testing%20Dashboard/view/Daily/?

This was our topmost priority, Martin was responsible for watching and
fixing these, with help from James and the server team for non-trivial
server specific issues. We succeeded in getting working daily ISOs for
desktop, alternate, and server every workday, and on most weekends.
There was a weekend following a large swamp of syncs/Haskell
transition/etc. uploads which caused a huge buildd lag and some
uninstallability, but at that point we could not do anything about
that any more except for making sure that the packages on the ISOs
build first.

The situation on ports images has been a bit worse. We often
experienced large armel backlogs because a lot of the armel builders
were borrowed for the armhf bootstrap. But as this is over now, we
should have enough armel builders again, so I don't think this is an
ongoing problem.

powerpc is worse, though, the buildds almost always have a backlog of
15 hours or more, so we often saw ISO build failures there. But as
nobody really cares much about these any more, we did not waste any
time on those and just let it sort itself out eventually.

We retired the manual tracking of the ISO status [2] and are
exclusively using Jenkins now.

Testing
-------
https://jenkins.qa.ubuntu.com/view/Precise/

The QA team has done tremendous work on keeping the automatic ISO and
upgrade testing working and made a lot of improvements there.

From the stable+1 team side this was James' work area in December.

James added a number of interesting test cases: 

 * LVM multi-partition mode with separate /usr/, /var/, /tmp/ etc.
 * RAID-1 tests
 * EC2 tests: the "precise-server-ec2" one is known-broken right now,
   but -adhoc and -daily work
 * started working on Openstack tests

Now all the tests pass, except:

 * precise-server-ec2 (see above)
 * precise-server-*-virtual: it's "yellow", i. e. install succeeds,
   but post-install tests fail. That's because they are currently
   oversized by ~ 50 MB, the documented limit is 500 MB. Either the
   server team needs to downsize the footprint, or raise the
   documented limit.
 * upgrades (see below)

The daily tests were tremendously helpful for discovering regressions
which we otherwise wouldn't have noticed until the next (i. e.
alpha-2) milestone testing. Now we could fix them right away, which is
both a lot easier (as the offending change was very recent) and also
takes the pressure out of the alpha-2 release engineering.

Uninstallable packages
----------------------
http://people.canonical.com/~ubuntu-archive/testing/precise_probs.html

At the start of December there still were a number of long-standing
uninstallable packages. Martin took over the responsibility for this
area.

All uninstallable packages were fixed in the first days, and since
then our #1 priority was to keep this at zero. The only prolonged time
period where it wasn't zero for about three days was around that huge
buildd lag weekend, where armel was uninstallable.

Martin also worked on improving the report to actually say _why_ a
package is not installable. This was a snapshot when the script was
improved:

  http://people.canonical.com/~pitti/tmp/precise_probs.html

On the ports side [3] there are still some problems. Most of it right
now is just buildd lag again (seriously, we either need to kill
powerpc or give it more buildds..), but the only "real" issue here is
that libreoffice fails on armhf. This is allegedly being worked on
between Matthias, Bjoern, and upstream, but I'm not sure of the
current status.

Upgrades
--------
https://jenkins.qa.ubuntu.com/view/Precise/job/precise-upgrade/

This was mostly Martin's area during that stable+1 cycle.

We now have a fairly large coverage of upgrade scenarios, including
an upgrade of the entire lucid main to precise (which is certainly the
hardest one to get right, but at the same time is also the most useful
one to discover problems). As you can see from the history we never
managed to get this fully working yet, but Martin spent about half of
his time on upgrade bugs (e. g. #900004, #897880, #901572, #902077,
and lots more).

Just today all the upgrade scenarios actually succeeded without
failures in apt and during the upgrade, we now "just" fail some
post-upgrade tests. Apparently some Python modules stop working after
the upgrade, there was a dpkg conffile prompt in mailman (we used to
have lots more, but most of them got fixed), and powernap crashed [4].

However, they currently cheat a bit: The main-all test currently
blacklists the "tdsodbc" package, which is LP #901638. We already
uploaded a fix for this, but had to revert it because it breaks
K/Edubuntu, so we need to convert soprano-daemon from the old
unmaintained iodbc2 to unixodbc.  Martin has a first attempt in a
branch/PPA, but it doesn't work yet.

So this is definitively an area where the January team should continue
on, to iron out the remaining issues and then keep them working.

NBS
---
http://people.canonical.com/~ubuntu-archive/nbs.html

This was the second-biggest chunk of work for us. Both James and I
worked on that pretty much every day. The most important and
time-consuming one so far was the poppler transition, which included
some nontrivial API changes/drops, but is fully finished now. But
these days new issues come up about as fast as we are able to fix
them (mostly through autosyncs, but some also through Ubuntu uploads),
so this is an area which needs constant attention and a huge amount of
work.

Outstanding issues:

 * libcegui-mk2-1 - still two packages outstanding with non-trivial fixes -
   bugs raised in LP/Debian:
   https://launchpad.net/bugs/903121
   https://launchpad.net/bugs/903126

 * libhpdf-2.1.0 - One outstanding (witty) - needs work upstream;
   other two are just because of waiting for Godot^Wpowerpc buildds.

 * simgear2.0.0 - non-trivial fix required for transition - bug
   reported in debian and upstream working on new release:

   https://launchpad.net/bugs/903225

 * libahven17.0 - this has crept through from Debian pre-transition
   completion of gnat-4.4 to gnat-4.6 into testing as it does not
   comply with the Debian Ada Library Policy in that it only depends
   on gnat-4.6 (not gnat and gnat-4.6).  Ubuntu already builds gnat
   from gnat-4.6 so we can't actually re-build any of the Ada
   libraries which are policy compliant ATM.

 * gnutls-bin - This is now built by gnutls28 in Debian, which we
   don't have in Ubuntu yet.

   Suggested plan of action: Verify that we can build all of main
   against gnutls28 with reasonable porting effort, sync gnutls28 into
   main, do the transition, drop gnutls26. Debian is undergoing this
   transition now, I think, so we can cooperate with them and send
   patches. If porting to 28 involves too big hurdles, the fallback is
   to build gnutls-bin from gnutls26 again (i. e. partial revert).

 * kalgebra-dev, libanalitza4, libanalitza4-dev: New kalgebra was
   uploaded to Ubuntu instead of the PPA. Kubuntu team asked us to
   leave it that way for the time being, though, they promised to sort
   it out.

 * libkdb5-5: New krb5 came through autosyncs with new build
   dependencies libverto and libev. libverto needs a MIR (LP#906281)
   and be changed to build against libevent instead of libev.

 * vtk: This is another bigger one which came in through autosyncs. We
   could not work on this yesterday yet, because it still needed
   building on some architectures, but hopefully will just be mostly
   no-change rebuilds.

priority/architecture mismatches
--------------------------------
http://people.canonical.com/~ubuntu-archive/priority-mismatches.txt
http://people.canonical.com/~ubuntu-archive/architecture-mismatches.txt

This was a rather easy topic. We fixed all outstanding issues,
blacklisted the one that we cannot technically fix (hfsutils on
powerpc optional → standard, Launchpad cannot do per-architecture
priority overrides), and since then kept it at zero.

As these reports are not usually looked at very often, we integrated
them into Jenkins, so that we see the red dot popping up when problems
come up.

component-mismatches
--------------------
http://people.canonical.com/~ubuntu-archive/component-mismatches.txt
http://people.canonical.com/~ubuntu-archive/component-mismatches.svg

Martin worked on improving the textual report to point to MIR bugs and
also mark packages with (MAIN) which are already in main, i. e. the
root of the chain. He also added an SVG report to give a more easily
readable way of the source/binary universe → main promotions with
clickable nodes (-> MIR bugs).

We resolved most of the issues and kep the list as small as possible

Outstanding issues:

 * openstack seed wants "keystone", but this still needs a MIR
   completion. This does not break any image builds (they will just
   ignore unavailable seeded packages), the server team asked us to
   leave it seeded for now.

 * krb5 -- see above in NBS

 * koffice-l10n demotion: Jonathan asked to leave it in main until he
   tested that updates correctly move over to the new "calligra" which
   replaces KOffice.

FTBFS
-----
http://qa.ubuntuwire.org/ftbfs/

Allison started looking into this and fixed a couple of long-standing
ones like magic++. 

James fixed ~25 bugs relating to source file
encoding errors with Java 7 and one relating to public API changes for JDBC4.1.
Still lots more work to be done: https://wiki.ubuntu.com/JavaTeam/Java7Default.

Otherwise we did not explicitly work on FTBFS a lot, mostly because
our time got fully eaten up by fixing installability, ISO failures,
upgrade bugs, NBS, and component-mismatches. This is a very active
time of development, so apparently two full time engineers are still
not enough with keeping up with all these and fix FTBFS.

So this remains to be a major area of work.

Other infrastructure
--------------------
http://conflictchecker.ubuntu.com/possible-conflicts/
http://harvest.ubuntu.com/

As already described above, both Jenkins and the archive consistency
reports got some improvements. We did not start fixing the
possible-conflicts reports to ignore all the false positives, or
look into improving Harvest, so these also remain outstanding.

Summary
-------
I am very happy with how this new a stable+1 team works. It it a huge
amount of work, but the benefits (timely feedback, keeping everything
installable, quickly fixing regressions) are well worth it. I have
heard quite a lot of positive feedback on IRC or elsewhere, in the
tone of "how do you guys make precise so stable at that early time?",
and I think we also have a lot more Canonical folks running precise
now than we used to have in earlier development releases.

I certainly enjoyed playing firefighter for that month, keeping
precise working, and also working on some report improvements.

Thanks, and enjoy the end-of-year holidays!

Martin

[1] https://wiki.ubuntu.com/PlusOneMaintenanceTeam
[2] https://wiki.ubuntu.com/PlusOneMaintenanceTeam/Status 
[3] http://people.canonical.com/~ubuntu-archive/testing-ports/precise_probs.html
[4] https://jenkins.qa.ubuntu.com/view/Precise/job/precise-upgrade/PROFILE=main-all,label=upgrade-test/80/

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20111220/9fa09c52/attachment.pgp>


More information about the ubuntu-devel mailing list