<div dir="ltr">Hi everyone, this is my +1 duty report of this week. I was glad to see several others work on migrations and overall archive health as well - even with a massive update_excuses it can feel good if you see progress. Here is a little summary of what I worked on, but I beg your pardon as I'm unsure if I documented everything, especially some cases that were rather trivial touch-and-go might be missing. 1. chromium-browser not upgraded to snap in Focal - This is less a +1 task than a generic archive dependency issue that I quickly looked at after being asked. But I found it is known and under control at bug LP: #1889106 - This later became a ubuntu-devel discussion about the use of epoch's in package versions for deb-to-snap transitions. 2. groovy rsync was one of the many packages waiting for armhf test backlog to resolve - it just needed some help with test triggers and then migrated. 3. Haskell was blocked last time I was on +1, I revisited the current state. It seems the transition I linked last time [1] is still going on in Debian. A few current things one could wonder about being real problems: - dependencies libghc-haxr-dev-3000.11.4.1-$hash are odd as the same source packages in Debian/Ubuntu 3000.11.4.1-1 built different $provides and hence there might be further mismatches. - lack of build dependencies, there are many cases waiting for the same thing when tracing down the tree - maybe with a bit of work we can at least reduce the number blocked in excuses. haskell-gtk-strut [2] (and others) -B-depend-> haskell-gi-gdk [3] haskell-gi-gdk [3] (and others) ->B-depend-> haskell-gi-pango [4] haskell-gi-pango [4](and others) -B-depend-> haskell-haskell-gi-base - The latter 'haskell-haskell-gi-base' is not auto-synced v23 from Debian, but neither did I find a removal - unfortunately [5] tells us that it would (even in Debian) not overall be ready yet anyway. - But since it could unblock so many things in update-excuses I gave it a try and it built fine [6] - it worked. - Furthermore I checked but there was no removal filed [6] that would explain the current lack of the package (someone might have wanted to hold all dependent things back) - we still have to wait for this to be fully ready as discussed in [1] but we could make some steps forward by syncing in haskell-haskell-gi-base. Yet I feel it might be removed/blocked-syncing for a reason - hence I wanted to ask upfront. - After talking to the AAs I learned that this information is another gem that can be found in the package publishing history. Yes there was no bug nor a sync blocker, but [17] shows "Temporary removal to downgrade to a version compatible with available reverse-dependencies" - This might (as I have assumed) interact with the ICU transition so we should not touch it right now. - But as soon as ICU is done one should syncpackage 0.23* of haskell-haskell-gi-base to get the haskell transition rolling again. - While I'm unsure if anyone actually reads it, I updated the +1 wikie [18] about that. - The other day Vorlon pinged me (I have updated the wiki page): <vorlon> cpaelzer: fyi after yesterday's conversation (and britney nagging me about a number of packages that I "uploaded" being stuck in -proposed too long ;), I am going ahead and rolling forward the various haskell packages that had been rolled back, after confirming they're not entangled with any current transitions. (e.g. the haskell-gi-* stuff can't be since it was all removed from release pocket) - This unblocked some, but not the whole haskell* yet and as one would expect there also are test issues on needs to look after. Seems like this will still be with us a while. 4. missing dependencies all over the place Checking from the bottom I found multiple packages with unsatisfiable dependencies for ~100 days. Worth trying to clean up. - icingaweb2-module-x509 depends on icingaweb2-module-reactbundle - state is the same in Debian [7] - we should remove it before 20.10 closes since it is broken as-is - removal filed [8] (TBH I'm not 100% sure it needs to be removed in this state, but personally I'd prefer it clean and gone - the AAs will tell me) - gspiceui depends on geda-gschem & geda-gnetlist which are unavailable - missing packages are from src:geda-gaf which we should have in universe - this source it blocks on is missing in testing as well as >=Focal - turns out this was related to the guile-2.0 removal and the new package just came in as auto-sync - but has no chance to work as the bug [9] to get geda-gaf updated stalled, found no package owner and got removed [10]. - the scheduled removal was filed against gspiceui [11] and if there is a fix it will auto-sync then, but for now we should remove it. - Removal filed at [12] - The list above would go on, but I started to wonder if it is worth the effort - I started to wonder if those are worth all the effort Pro: removal cleans up excuses Con: won't end up in -release and causes work - I started a discussion about it and I learned that while RM bugs would work for those they are plenty of overhead. It would be better for those cases to just ping an Archive Admin and ask him to remove it. To have a chance to later bring back things when the dependencies are available there is [15]. - A requirement for that also is that the package maintainer is aware, e.g. have a Debian bug filed about it to increase the chance it can come back later. - The AAs realized this isn't documented well and want to do that, expect a mail to ubuntu-devel at some time. Thanks Steve, Matthias and Sebastian for the discussion. - After learning the above I provided further cases of this category directly as an MP asking for MP-merge and removal of the packages in one step [16]. - After learning the above I analyzed way more cases and created [23] out of that for the Archive admins to remove packages but also to track when they can come back and/or prevent further syncs. 5. missing builds all over the place - Further there are many packages that show "has no binaries on any arch" - not 100% the same as the dependency issue, but similar enough (would need removals). - I found that we have quite a lot of these cases (61). But all of those that I checked are only in -proposed and removed in debian. - As of today, ignoring those that belong to another known bucket like rust, haskell or desktop I've seen 25 of those packages: lomiri-download-manager, qtfeedback-opensource-src, qtpim-opensource-src, node-jsdom, crystal, dropwatch, libzypp, taffybar, iitii, singularity-container, siconos, simple-ccsm, ants, bombono-dvd, jinja2-time, ledger-autosync, pytest-services, q2-cutadapt, bslizr, bchoppr, hgsubversion, freezer, hdevtools, biometric-authentication, xenium - In this case it would not be as easy to track if chances are high to re-introduce them, probably worth another discussion, for now I did only list but not tackle these. 6. ICU - the elephant in the room that feels entangled with everything else ICU always is massive and painful, no different this time it seems. I was tracking down further dependencies it blocks (as did many others this week). - ros-rosconsole was just fixed to build on riscv64 when I checked it No need to action on this anymore. - boost1.71 has is a dependency of ICU to migrate and blocked by a few things on its own. - mrs is known to FTFBS with gcc-10 for quite a while [13] - this also is one of the gcc-10 FTFBS reported by doko - mrs got removed in Debian [14] due to that - we need to fix the FTBFS or remove it as well - I pinged other people likely to work on it but no one seems to do so to make sure it can be found by others I filed [19] - Tracking down the issue revealed a rather easy fix that after a cross arch test build [20] reported to Debian and Upstream and uploaded to groovy. - dart FTFBS - this also is one of the gcc-10 FTFBS reported by doko - it also had a faky test on gazebo/amd64 that worked on a re-trigger - FTFBFS on s390x and riscv64 only - riscv64 segfault in Test #29: test_ContactConstraint - s390x segfault in test #17 test_ForwardKinematics - s390x segfault in test #47 test_SdfParser - s390x segfault in test #50 test_DartLoader - s390x segfault in test #51 test_IkFast - Same s390x issue happened to the last build some weeks ago <a href="https://launchpad.net/ubuntu/+source/dart/6.9.2-3">https://launchpad.net/ubuntu/+source/dart/6.9.2-3</a> but there riscv was still ok - riscv64 => seems to be gcc-10 breakage - s390x => just NEVER built - for migration right now we don't have to fix s390x here (obviously we should try to see if is easy/trivial) - there are a bunch of slightly newer releases at <a href="https://github.com/dartsim/dart/milestones">https://github.com/dartsim/dart/milestones</a> - no issue references the problems we see - no recent commits in that regard - builds slow even on s390x, will be much worse on emulated riscv64 - I reported it upstream [27][28] on LP [29] and Debian [30] so everyone is at least aware - getting a riscv64 build env that I can debug in was quite an effort gladly I had at least <a href="https://people.ubuntu.com/~wgrant/riscv64/">https://people.ubuntu.com/~wgrant/riscv64/</a> to start. - I was working on a small upload that will skip just the failing test on riscv64, that worked in a test build. - I was debugging this for a while in riscv64 qemu and found that on function return it seems to break the instruction pointer - at that point backtraces as well as everything else is broken. Almost more like a gcc bug than the dart code - I added a gcc-10 task to the bug. - I also added that insight to the upstream bug to help them understand the issue as well. - since there was no fix in reach to unblock proposed I uploaded the test skip for now. - libreoffice 6.4.5 mostly fine but armhf autopkgtests From the test logs it seemed flaky in the past and ken-vandine seems to be already on it (he was triggering restarts). Test queues were better today, I retriggered it as well after the last fail to a) increase the chance for a flaky-good run and b) in case it is a reproducible fail to get more test logs on it. Over the next few days I saw more and combined retries which means +1 doesn't have to look after it right now since the desktop team does. Eventually it seems no one had a fix and it ended up in [33] - dee (missing build) - FTFBS on no change rebuild for ICU - this is an odd one - all architectures started & finished build two weeks ago - except riscv64 which was running still - seemed someone restarted the build recently - but that build (probably again) hangs at the after-build stage for hours now. - 7 days ago on +1 mwhudson said "dee's tests are timing out on riscv64 :(" This isn't as it looks right now, but I need to wait until it completes to get a full build log - I pinged people that might have or actively do look into this for a discussion but got no response - it hung for two hours more in that state - I asked if there was some post mortem we could do and cjwatson was so kind to help. We identified issues of a hanging process that needed to be debugged and I filed [21] for it. - Further debugging showed that this is really a problem on it's own. Wgrant reported: <wgrant> Laney, cpaelzer: dbus-test-runner is what generally gets stuck, I haven't worked out why. But there are a couple of packages that were consistently hanging because it didn't die for a while. But they seemed to eventually fix themselves by around maybe September. - I added a dbus-test-runner task to [21] and spun a riscv64 emulator to see if we can catch it live for further debugging - for dee itself I tested if a timeout bump and/or test skip are needed and uploaded that fix to get things unblocked in -proposed - to my disappointment only the test skip got things working well, so I uploaded that as it is part of the ICU transition block. - [26] built fine then, it depends on ICU but otherwise is ready to go now - nodejs - this is a big chunk on it's own and I wanted to look at it as well - gladly before I did there was a devel post [24] that get sorted who was working on which sub-problem of it for the last weeks. - to get ICU closer to migrate Steve rolled back nodejs for the time being. After ICU transitions it can migrate non-entangled. - As one would expect this filled the test queues again, but there aren't enough results yet to spot which remaining issues need to be worked on. This likely should be a priority of +1 members next week when the results of the tests for this downgrade are in. 7. procenv (blocking sbuild/dpkg) - There is a GCC-10 FTFBS in procenv - reported as <a href="https://bugs.launchpad.net/ubuntu/+source/procenv/+bug/1889138">https://bugs.launchpad.net/ubuntu/+source/procenv/+bug/1889138</a> - the bug had a patch to sponsor which I reviewed and sponsored - procenv is the package built in the sbuild tests and thereby was blocking sbuild/dpkg - after migration test retriggers were needed - I even found others doing combined triggers while it was in proposed, - since this isn't a normal test dep it only resolved once procenv was in -release - with that tests of sbuild itself unblocked - next dpkg tests of sbuild unblocked Those also needed 'sbuild/0.80.0ubuntu1' trigger due to Dpkg::Build::Info ::get_build_env_whitelist() in there. 8. systemd flaky tests - this is a usual suspect, it just needs to test so much and still is a bit flaky sometimes - IMHO important is to check the test logs, if we see different tests fail every time then it is most likely flakiness. It also is worth to check the same tests cross arch for that. If instead the same tests fail every time chances are high we see a genuine issue. - with the check above I identified 8 cases of flaky tests and retriggered, but also tracked their results in case one turns out to look more like a genuine issue on retry. - I got six packages to no more be blocked on systemd \o/ and spotted one unique issue around the new plymouth vs systemd tests. That was already mentioned in [25] quite a while ago, so I got in touch with rbalint to check the state. He will solve that remaining bit on the next systemd upload. - I Updated the bug [25] with that information and left the remaining tests now that it was clear it was a real issue 9. gubbins hangin on build dependencies - Debian never built non-x86, we did - Recent Debian change added an x86 only build dependency - This migrated fine in Debian but is stuck for u - I filed [22] for an archive admin to resolve - once the non x86 binary packages are removed from groovy 2.4.x should be able to migrate fine 10. autopilot-gtk - when scanning for broken build-depends for #4 I also found [31] - I updated the bug accordingly and just half a day later we got a fix (Thanks jibel) - Essentially this is a py2->py3 transition that has fallen through the cracks - I reviewed and sponsored the fix which will remove this finally from proposed migration - half a day later builds and tests are good and this has migrated 11. More failing autpkgtests - Looking at the head of excuses instead of the tail this time I found some more tests that failed, a few of them just are without hope given the result history, so I filed an MP to reset them [32] - this also includes try-to-test-i386-when-there-is-no-build cases that seem to be missed in the past - A few others just needed some more re-triggers (e.g. we had a flurry of failing apt access to the archive on arm again - I've seen those a few times, but still don't know what they are about) - As always (and I can recommend that) keep a log or tab on what you restarted and track if it works out 12. fatrace - This is another old proposed candidate, the autopkgtest is broken. - this was found in the past by bdmurray and filed as [34][35] - given that it is more than 100 day without any response we need to do something about it. - Some debugging on the case revealed that the tracing seems really broken in groovy missing what feels like an arbitrary amount of events - Ubuntu is more exposed having newer kernels in the build env but I was able to confirm that the bug is present in old/new fatrace version as well as in Debian & Ubuntu - I updated Ubuntu and Debian bugs with the insights so far, it might help to get things going again. [1]: <a href="https://lists.debian.org/debian-haskell/2020/06/msg00003.html">https://lists.debian.org/debian-haskell/2020/06/msg00003.html</a> [2]: <a href="https://launchpad.net/ubuntu/+source/haskell-gtk-strut/0.1.3.0-2build1">https://launchpad.net/ubuntu/+source/haskell-gtk-strut/0.1.3.0-2build1</a> [3]: <a href="https://launchpad.net/ubuntu/+source/haskell-gi-gdk/3.0.22-1build1">https://launchpad.net/ubuntu/+source/haskell-gi-gdk/3.0.22-1build1</a> [4]: <a href="https://launchpad.net/ubuntu/+source/haskell-gi-pango/1.0.22-1build1">https://launchpad.net/ubuntu/+source/haskell-gi-pango/1.0.22-1build1</a> [5]: <a href="https://tracker.debian.org/pkg/haskell-haskell-gi-base">https://tracker.debian.org/pkg/haskell-haskell-gi-base</a> [6]: <a href="https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4184/+packages">https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4184/+packages</a> [7]: <a href="https://tracker.debian.org/pkg/icingaweb2-module-x509">https://tracker.debian.org/pkg/icingaweb2-module-x509</a> [8]: <a href="https://bugs.launchpad.net/ubuntu/+source/icingaweb2-module-x509/+bug/1891011">https://bugs.launchpad.net/ubuntu/+source/icingaweb2-module-x509/+bug/1891011</a> [9]: <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=885195">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=885195</a> [10]: <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965098">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965098</a> [11]: <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=967915">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=967915</a> [12]: <a href="https://bugs.launchpad.net/ubuntu/+source/gspiceui/+bug/1891017">https://bugs.launchpad.net/ubuntu/+source/gspiceui/+bug/1891017</a> [13]: <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=957567">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=957567</a> [14]: <a href="https://tracker.debian.org/news/1166233/mrs-removed-from-testing/">https://tracker.debian.org/news/1166233/mrs-removed-from-testing/</a> [15]: <a href="https://bazaar.launchpad.net/~ubuntu-archive/+junk/sync-blacklist/view/head:/extra-removals.txt">https://bazaar.launchpad.net/~ubuntu-archive/+junk/sync-blacklist/view/head:/extra-removals.txt</a> [16]: MP for removal of b-dep issues [17]: <a href="https://launchpad.net/ubuntu/+source/haskell-haskell-gi-base/+publishinghistory">https://launchpad.net/ubuntu/+source/haskell-haskell-gi-base/+publishinghistory</a> [18]: <a href="https://wiki.ubuntu.com/PlusOneMaintenanceTeam/Status">https://wiki.ubuntu.com/PlusOneMaintenanceTeam/Status</a> [19]: <a href="https://bugs.launchpad.net/ubuntu/+source/mrs/+bug/1891023">https://bugs.launchpad.net/ubuntu/+source/mrs/+bug/1891023</a> [20]: <a href="https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4189/+packages">https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4189/+packages</a> [21]: <a href="https://bugs.launchpad.net/ubuntu/+source/dee/+bug/1891158">https://bugs.launchpad.net/ubuntu/+source/dee/+bug/1891158</a> [22]: <a href="https://bugs.launchpad.net/ubuntu/+source/gubbins/+bug/1891340">https://bugs.launchpad.net/ubuntu/+source/gubbins/+bug/1891340</a> [23]: <a href="https://code.launchpad.net/~paelzer/junk/sync-blacklist-plus-one-clean-groovy">https://code.launchpad.net/~paelzer/junk/sync-blacklist-plus-one-clean-groovy</a> [24]: <a href="https://lists.ubuntu.com/archives/ubuntu-devel/2020-August/041121.html">https://lists.ubuntu.com/archives/ubuntu-devel/2020-August/041121.html</a> [25]: <a href="https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1886886">https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1886886</a> [26]: <a href="https://launchpad.net/ubuntu/+source/dee/1.2.7+17.10.20170616-6ubuntu1">https://launchpad.net/ubuntu/+source/dee/1.2.7+17.10.20170616-6ubuntu1</a> [27]: <a href="https://github.com/dartsim/dart/issues/1483">https://github.com/dartsim/dart/issues/1483</a> [28]: <a href="https://github.com/dartsim/dart/issues/1482">https://github.com/dartsim/dart/issues/1482</a> [29]: <a href="https://bugs.launchpad.net/ubuntu/+source/dart/+bug/1891440">https://bugs.launchpad.net/ubuntu/+source/dart/+bug/1891440</a> [30]: <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=968332">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=968332</a> [31]: <a href="https://bugs.launchpad.net/ubuntu/+source/autopilot-legacy/+bug/1856574">https://bugs.launchpad.net/ubuntu/+source/autopilot-legacy/+bug/1856574</a> [32]: <a href="https://code.launchpad.net/~paelzer/britney/hints-ubuntu-clean-groovy-proposed2/+merge/389239">https://code.launchpad.net/~paelzer/britney/hints-ubuntu-clean-groovy-proposed2/+merge/389239</a> [33]: <a href="https://code.launchpad.net/~seb128/britney/ignore-libreoffice-armhf/+merge/389227">https://code.launchpad.net/~seb128/britney/ignore-libreoffice-armhf/+merge/389227</a> [34]: <a href="https://bugs.launchpad.net/ubuntu/+source/fatrace/+bug/1885188">https://bugs.launchpad.net/ubuntu/+source/fatrace/+bug/1885188</a> [35]: <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963714">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963714</a> <div> </div>-- <div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">Christian Ehrhardt Staff Engineer, Ubuntu Server Canonical Ltd</div></div>