+1 Maintenance report
christian.ehrhardt at canonical.com
Fri Aug 14 12:46:16 UTC 2020
this is my +1 duty report of this week. I was glad to see several others
work on migrations and overall archive health as well - even with a massive
update_excuses it can feel good if you see progress.
Here is a little summary of what I worked on, but I beg your pardon as I'm
I documented everything, especially some cases that were rather trivial
touch-and-go might be missing.
1. chromium-browser not upgraded to snap in Focal
- This is less a +1 task than a generic archive dependency issue that
I quickly looked at after being asked. But I found it is known and
control at bug LP: #1889106
- This later became a ubuntu-devel discussion about the use of epoch's in
package versions for deb-to-snap transitions.
2. groovy rsync was one of the many packages waiting for
armhf test backlog to resolve - it just needed some help with test
and then migrated.
3. Haskell was blocked last time I was on +1, I revisited the current state.
It seems the transition I linked last time  is still going on in
A few current things one could wonder about being real problems:
- dependencies libghc-haxr-dev-3000.11.4.1-$hash are odd as the same
source packages in Debian/Ubuntu 3000.11.4.1-1 built different
$provides and hence there might be further mismatches.
- lack of build dependencies, there are many cases waiting for the
same thing when tracing down the tree - maybe with a bit of work we can
at least reduce the number blocked in excuses.
haskell-gtk-strut  (and others) -B-depend-> haskell-gi-gdk 
haskell-gi-gdk  (and others) ->B-depend-> haskell-gi-pango 
haskell-gi-pango (and others) -B-depend-> haskell-haskell-gi-base
- The latter 'haskell-haskell-gi-base' is not auto-synced v23
from Debian, but neither did I find a removal
- unfortunately  tells us that it would (even in Debian) not
ready yet anyway.
- But since it could unblock so many things in update-excuses I gave
it a try and it built fine  - it worked.
- Furthermore I checked but there was no removal filed  that would
explain the current lack of the package (someone might have wanted to
hold all dependent things back)
- we still have to wait for this to be fully ready as discussed in 
but we could make some steps forward by syncing in
Yet I feel it might be removed/blocked-syncing for a reason - hence I
wanted to ask upfront.
- After talking to the AAs I learned that this information is another gem
that can be found in the package publishing history. Yes there was no
nor a sync blocker, but  shows "Temporary removal to downgrade to a
version compatible with available reverse-dependencies"
- This might (as I have assumed) interact with the ICU transition so we
should not touch it right now.
- But as soon as ICU is done one should syncpackage 0.23* of
haskell-haskell-gi-base to get the haskell transition rolling again.
- While I'm unsure if anyone actually reads it, I updated the +1 wikie
- The other day Vorlon pinged me (I have updated the wiki page):
<vorlon> cpaelzer: fyi after yesterday's conversation (and britney
nagging me about a number of packages that I "uploaded" being stuck in
-proposed too long ;), I am going ahead and rolling forward the
haskell packages that had been rolled back, after confirming they're
entangled with any current transitions. (e.g. the haskell-gi-* stuff
can't be since it was all removed from release pocket)
- This unblocked some, but not the whole haskell* yet and as one would
expect there also are test issues on needs to look after.
Seems like this will still be with us a while.
4. missing dependencies all over the place
Checking from the bottom I found multiple packages with unsatisfiable
dependencies for ~100 days. Worth trying to clean up.
- icingaweb2-module-x509 depends on icingaweb2-module-reactbundle
- state is the same in Debian 
- we should remove it before 20.10 closes since it is broken as-is
- removal filed  (TBH I'm not 100% sure it needs to be removed in
this state, but personally I'd prefer it clean and gone - the AAs
will tell me)
- gspiceui depends on geda-gschem & geda-gnetlist which are unavailable
- missing packages are from src:geda-gaf which we should have in
- this source it blocks on is missing in testing as well as >=Focal
- turns out this was related to the guile-2.0 removal and the new
just came in as auto-sync - but has no chance to work as the bug 
to get geda-gaf updated stalled, found no package owner and got
- the scheduled removal was filed against gspiceui  and if there is
a fix it will auto-sync then, but for now we should remove it.
- Removal filed at 
- The list above would go on, but I started to wonder if it is worth the
- I started to wonder if those are worth all the effort
Pro: removal cleans up excuses
Con: won't end up in -release and causes work
- I started a discussion about it and I learned that while RM bugs
work for those they are plenty of overhead. It would be better for
those cases to just ping an Archive Admin and ask him to remove
have a chance to later bring back things when the dependencies are
available there is .
- A requirement for that also is that the package maintainer is
e.g. have a Debian bug filed about it to increase the chance it can
come back later.
- The AAs realized this isn't documented well and want to do that,
a mail to ubuntu-devel at some time.
Thanks Steve, Matthias and Sebastian for the discussion.
- After learning the above I provided further cases of this category
directly as an MP asking for MP-merge and removal of the packages in
- After learning the above I analyzed way more cases and created  out
of that for the Archive admins to remove packages but also to track
they can come back and/or prevent further syncs.
5. missing builds all over the place
- Further there are many packages that show "has no binaries on any arch"
- not 100% the same as the dependency issue, but similar enough (would
- I found that we have quite a lot of these cases (61). But all of
that I checked are only in -proposed and removed in debian.
- As of today, ignoring those that belong to another known bucket like
rust, haskell or desktop I've seen 25 of those packages:
node-jsdom, crystal, dropwatch, libzypp, taffybar, iitii,
singularity-container, siconos, simple-ccsm, ants, bombono-dvd,
jinja2-time, ledger-autosync, pytest-services, q2-cutadapt, bslizr,
bchoppr, hgsubversion, freezer, hdevtools, biometric-authentication,
- In this case it would not be as easy to track if chances are high to
re-introduce them, probably worth another discussion, for now I did
list but not tackle these.
6. ICU - the elephant in the room that feels entangled with everything else
ICU always is massive and painful, no different this time it seems.
I was tracking down further dependencies it blocks (as did many others
- ros-rosconsole was just fixed to build on riscv64 when I checked it
No need to action on this anymore.
- boost1.71 has is a dependency of ICU to migrate and blocked by a few
things on its own.
- mrs is known to FTFBS with gcc-10 for quite a while 
- this also is one of the gcc-10 FTFBS reported by doko
- mrs got removed in Debian  due to that
- we need to fix the FTBFS or remove it as well
- I pinged other people likely to work on it but no one seems to do
to make sure it can be found by others I filed 
- Tracking down the issue revealed a rather easy fix that after a
arch test build  reported to Debian and Upstream and uploaded
- dart FTFBS
- this also is one of the gcc-10 FTFBS reported by doko
- it also had a faky test on gazebo/amd64 that worked on a re-trigger
- FTFBFS on s390x and riscv64 only
- riscv64 segfault in Test #29: test_ContactConstraint
- s390x segfault in test #17 test_ForwardKinematics
- s390x segfault in test #47 test_SdfParser
- s390x segfault in test #50 test_DartLoader
- s390x segfault in test #51 test_IkFast
- Same s390x issue happened to the last build some weeks ago
https://launchpad.net/ubuntu/+source/dart/6.9.2-3 but there riscv
was still ok
- riscv64 => seems to be gcc-10 breakage
- s390x => just NEVER built
- for migration right now we don't have to fix s390x here
we should try to see if is easy/trivial)
- there are a bunch of slightly newer releases at
- no issue references the problems we see
- no recent commits in that regard
- builds slow even on s390x, will be much worse on emulated riscv64
- I reported it upstream  on LP  and Debian  so
is at least aware
- getting a riscv64 build env that I can debug in was quite an effort
gladly I had at least https://people.ubuntu.com/~wgrant/riscv64/
- I was working on a small upload that will skip just the failing
on riscv64, that worked in a test build.
- I was debugging this for a while in riscv64 qemu and found that on
function return it seems to break the instruction pointer - at that
point backtraces as well as everything else is broken. Almost more
like a gcc bug than the dart code - I added a gcc-10 task to the
- I also added that insight to the upstream bug to help them
understand the issue as well.
- since there was no fix in reach to unblock proposed I uploaded the
test skip for now.
- libreoffice 6.4.5 mostly fine but armhf autopkgtests
From the test logs it seemed flaky in the past and ken-vandine seems
to be already on it (he was triggering restarts). Test queues were
today, I retriggered it as well after the last fail to a) increase the
chance for a flaky-good run and b) in case it is a reproducible fail to
get more test logs on it.
Over the next few days I saw more and combined retries which means +1
doesn't have to look after it right now since the desktop team does.
Eventually it seems no one had a fix and it ended up in 
- dee (missing build)
- FTFBS on no change rebuild for ICU
- this is an odd one - all architectures started & finished build two
- except riscv64 which was running still - seemed someone restarted
the build recently
- but that build (probably again) hangs at the after-build stage for
- 7 days ago on +1 mwhudson said "dee's tests are timing out on
This isn't as it looks right now, but I need to wait until it
to get a full build log
- I pinged people that might have or actively do look into this for
a discussion but got no response
- it hung for two hours more in that state
- I asked if there was some post mortem we could do and cjwatson was so
kind to help. We identified issues of a hanging process that needed
to be debugged and I filed  for it.
- Further debugging showed that this is really a problem on it's own.
<wgrant> Laney, cpaelzer: dbus-test-runner is what generally gets
stuck, I haven't worked out why. But there are a couple of packages
that were consistently hanging because it didn't die for a while.
they seemed to eventually fix themselves by around maybe September.
- I added a dbus-test-runner task to  and spun a riscv64 emulator
to see if we can catch it live for further debugging
- for dee itself I tested if a timeout bump and/or test skip are needed
and uploaded that fix to get things unblocked in -proposed
- to my disappointment only the test skip got things working well, so
I uploaded that as it is part of the ICU transition block.
-  built fine then, it depends on ICU but otherwise is ready to go
- this is a big chunk on it's own and I wanted to look at it as well
- gladly before I did there was a devel post  that get sorted
who was working on which sub-problem of it for the last weeks.
- to get ICU closer to migrate Steve rolled back nodejs for the time
After ICU transitions it can migrate non-entangled.
- As one would expect this filled the test queues again, but there
enough results yet to spot which remaining issues need to be worked
This likely should be a priority of +1 members next week when the
of the tests for this downgrade are in.
7. procenv (blocking sbuild/dpkg)
- There is a GCC-10 FTFBS in procenv
- reported as
- the bug had a patch to sponsor which I reviewed and sponsored
- procenv is the package built in the sbuild tests and thereby
was blocking sbuild/dpkg
- after migration test retriggers were needed
- I even found others doing combined triggers while it was in proposed,
- since this isn't a normal test dep it only resolved once procenv was
- with that tests of sbuild itself unblocked
- next dpkg tests of sbuild unblocked
Those also needed 'sbuild/0.80.0ubuntu1' trigger due to
::get_build_env_whitelist() in there.
8. systemd flaky tests
- this is a usual suspect, it just needs to test so much and still is a
- IMHO important is to check the test logs, if we see different tests
every time then it is most likely flakiness. It also is worth to check
the same tests cross arch for that.
If instead the same tests fail every time chances are high we see a
- with the check above I identified 8 cases of flaky tests and
but also tracked their results in case one turns out to look more like
genuine issue on retry.
- I got six packages to no more be blocked on systemd \o/ and spotted one
unique issue around the new plymouth vs systemd tests. That was already
mentioned in  quite a while ago, so I got in touch with rbalint to
check the state. He will solve that remaining bit on the next systemd
- I Updated the bug  with that information and left the remaining
now that it was clear it was a real issue
9. gubbins hangin on build dependencies
- Debian never built non-x86, we did
- Recent Debian change added an x86 only build dependency
- This migrated fine in Debian but is stuck for u
- I filed  for an archive admin to resolve
- once the non x86 binary packages are removed from groovy 2.4.x should
able to migrate fine
- when scanning for broken build-depends for #4 I also found 
- I updated the bug accordingly and just half a day later we got a fix
- Essentially this is a py2->py3 transition that has fallen through the
- I reviewed and sponsored the fix which will remove this finally
from proposed migration
- half a day later builds and tests are good and this has migrated
11. More failing autpkgtests
- Looking at the head of excuses instead of the tail this time I found
some more tests that failed, a few of them just are without hope
given the result history, so I filed an MP to reset them 
- this also includes try-to-test-i386-when-there-is-no-build cases
that seem to be missed in the past
- A few others just needed some more re-triggers (e.g. we had a flurry of
failing apt access to the archive on arm again - I've seen those a few
times, but still don't know what they are about)
- As always (and I can recommend that) keep a log or tab on what you
restarted and track if it works out
- This is another old proposed candidate, the autopkgtest is broken.
- this was found in the past by bdmurray and filed as 
- given that it is more than 100 day without any response we need to do
something about it.
- Some debugging on the case revealed that the tracing seems really
in groovy missing what feels like an arbitrary amount of events
- Ubuntu is more exposed having newer kernels in the build env but I was
able to confirm that the bug is present in old/new fatrace version as
as in Debian & Ubuntu
- I updated Ubuntu and Debian bugs with the insights so far, it might
to get things going again.
: MP for removal of b-dep issues
Staff Engineer, Ubuntu Server
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the ubuntu-devel