+1 Maintenance Report

[Bryce Harrington]

[Shorter week than usual due to holiday.]

An unusual amount of tmpfail and timeouts this week.  This may be due to
an infrastructure problem, as discussed in this bug:

    https://bugs.launchpad.net/auto-package-testing/+bug/2067074

It could be specific to autopkgtest, or more general, but that needs
analysis, and probably by someone with access to the machinery.

I retriggered a boatload of failed tests that appear to be due to this
problem, and 99% of the time that worked; this was the bulk of my time
spent this stint.  Some of these tmpfails were associated with
transitions, and I have a few additional details to note, below.


### Perl Transition ###

There is currently a minor -4 to -5 update to Perl, which afaict is just
Debian tweaking some architecture settings, but it's kicked off a ton of
tests (the Perl ecosystem is a big one), many of which failed due to the
tmpfail issue above.  I've retriggered all the affected tests but there
may be some residual work.  However, Debian has tended to do multiple
Perl updates per cycle, so I didn't want to spend too much time on that.
There may be more substantial updates in coming months.

Anyway, got all these passing ⚠️  -> ✅ for perl/5.38.2-5:  alice,
aptitude-robot, carton, chemonomatopist, config-package-dev, dh-octave,
dh-sysuser, distro-info, freecontact, hamlib, inspircd, io-stringy,
libacme-eyedrops-perl, libalgorithm-combinatorics-perl,
libalgorithm-munkres-perl, libapache-session-perl,
libcompress-raw-zlib-perl, libconfig-augeas-perl,
libconfig-identity-perl, ,libconfig-json-perl,
libconfig-model-itself-perl, libfile-remove-perl, libfile-util-perl,
libfile-which-perl, libfile-zglob-perl, libfilesys-statvfs-perl,
libnumber-recordlocator-perl, libredis-fast-perl


### Pytest Transition ###

This is messier, with pytest moving from 7.x to 8.x bringing breaking
API changes that have busted a lot of packages.  The upstream changelog
lists a number of "Breaking Changes", so presumably syncs/merges will be
needed.  But there are actually several situations going on:

  * tmpfails.  As mentioned above.  I've retriggered most of these, and
    some of them then passed:
    - h5py [oracular/s390x]
    - ipyparallel [oracular/armhf]
    - ipyparallel [oracular/s390x]

  * i386 missing deps.  Not sure if these should all just be
    force-badtest hinted or if something more involved is needed, but I
    didn't dig into this.  The packages affected here are:
    - asdf-astropy/0.6.1-1 (fails on i386 only)
    - asdf-standard/1.1.1-1
    - asdf-transform-schemas/0.5.0-1
    - astroml/1.0.2-4
    - ccdproc/2.4.2-1
    - feature-check/2.1.0-2build1
    - ginga/5.0.1-1
    - gwcs/0.21.0-1
    - gwcs/0.21.0-
    - ndcube/2.2.0-1
    - pytest-arraydiff/0.6.1-3
    - pyvo/1.5.1-1

  * dcmstack: Needs merged; debian's update should fix the issue.

  * jupyter-client: There is a new 8.x upstream, which Debian is
    packaging experimentally in git, but no updates in recent months.
    Probably worth discussion with Doko / Debian.
    https://qa.debian.org/cgi-bin/vcswatch?package=jupyter-client

  * Other API breaks.  Remaining packages don't have ubuntu delta so
    autosync, and same errors occur in Debian.


### httpx ###

This is the source of blockage for a number of python modules,
flask-sqlalchemy, etc.  The errors tend to be proxy errors and timeouts
on socket connections.  Retriggering them hasn't seemed to resolve
them.  I poked around upstream and in debian but whatever is going on
seems to be unique to us.

My guess it is some conflict with how the httpx proxy's tests are
interacting with our infrastructure squid.  Near as I can tell this
worked with httpx/0.26.0-2 and regressed with 0.27, so it seems likely
to be a change in this release.  The changelog for
0.27 (https://github.com/encode/httpx/releases/tag/0.27.0) has only 3
entries, one of which is a fix to make http1 work, which appears to have
been an issue with squid in the distant past.  

I think this may need someone more conversant in autopkgtest
infrastructure to examine.


### ubuntu-release-upgrader ###

The i386 failure was marked version=blacklisted, result=denylisted, and
the log error text suggested contacting Ubuntu QA.  I did so, and Brian
Murray removed it from never_run and requeued the test which then passed.


### Recipes ###

Below are some commands I've collected that can sometimes produce
easily actionable tasks (retriggers, etc.)

retry-autopkgtest-regressions \
    --log-regex 'Temporary failure resolving' \
    --min-age 2
retry-autopkgtest-regressions \
    --force-cached --min-age 2 \
    --log-regex 'FAIL timed out'
retry-autopkgtest-regressions \
    --force-cached --min-age 2 \
    --log-regex 'Unable to connect to ftpmaster'

The above commands generally don't produce anything, but when they do
the item invariably can be simply retriggered and will pass.


retry-autopkgtest-regressions \
    --force-cached \
    --only-unknown

This is the power tool this week, for these tmpfail bugs.  In
particular, it's looking for migration issues listing version as
'unknown', which seems to be a common tell-tale.  For what we're seeing
this week, simple retriggers seem to be the way to go, but in general
they may be identifying a deeper breakage and/or infra issues.


retry-autopkgtest-regressions \
    --force-cached --min-age 2 \
    --log-regex 'Cannot allocate memory'

These may be suggesting they need added to `big_packages` in
autopkgtest.  The packages should be verified to pass autopkgtest run
manually in a VM of the given architecture, first.

Credit for these go to retry-autopkgtest-regressions developers, and
other packagers that have shared them in their +1 maintenance reports.
Brian Murray first introduced me to them, I believe.  HTH


### Stats ###

  2024-05-28   855  update excuse records found
  2024-05-29   768
  2024-05-30   665
  2024-05-31   652