+1 maintenance report

[Bryce Harrington]

On Fri, Feb 18, 2022 at 05:14:46PM +0000, Robie Basak wrote:
> I was on +1 maintenance this week.
> 
> It's been a long time (years) since I last did this. I'm not sure I had
> a good feel for what would be useful to work on, so rather than spending
> time changing my mind constantly I just focused on what I saw
> regardless.
> 
> There seem to be a lot of delays waiting on amd64 dep8 tests. Near the
> start of the week there were 6705 tests in the queue. At the end of the
> week there are 4643. But still the queueing time is generally over a
> week. It would be nice if I could see what entries in the queue were
> submitted manually by others. This doesn't seem to be presented at
> https://autopkgtest.ubuntu.com/running. Christian thought this
> information was available in the associated json output, but I haven't
> looked into this.
> 
> On the topic of delays, I feel like I've only just gotten into a lot of
> this. Because of the lag between taking some action and seeing the
> result, I ended up with many pending items at any given time, because
> most of the time all other items would be waiting on something before I
> could proceed further with them. So I'd look for something else. So, as
> I am writing up my week, I am finding many loose ends that I did not
> resolve because I had started them but then previously blocked items
> needed attention again.

That does seem to be a pretty good description of the +1 maintenance
experience.  :-)

I imagine every +1'er has their own strategy for dealing with that
chaotic workflow.  What I do is create a bulleted list of every package
I touch, noting the action I take, next steps depending on outcome,
theories on what's wrong, and the ultimate resolution (if any).

> In general I'd have liked to have seen more coordination with others on
> what is going on. Sometimes I might spend an hour tracking something
> down, coming to a conclusion about what action is necessary to take, and
> then finding that somebody had already figured this out, done it and the
> appropriate rebuild or dep8 retry was in progress or in a queue
> somewhere. This is frustrating. I'd prefer to spend my efforts on
> something nobody else is looking at. But there isn't any clear process
> to determine what that is.

A process I've seen others use, and that I've adopted myself, is to file
bugs tagged update-excuse, and use the bug report for tracking findings
and next-actions and such.  This seems to work well for the more
challenging transition items, but is overkill for smaller stuff where it
just needs a customized retriggering or a straightforward patch.

But unfortunately, for that smaller stuff we have poor transparency into
what's currently running.  Since britney can sometimes take a few hours
in its processing cycle even for simple things, that creates a big
window for people to submit redundant retriggers.

Sometimes I wonder if we directed some of the +1 effort towards tool
improvements if we could get a better workflow, or maybe even automate
away some of the tasks.


> I found a common use case was that I'd know the binary or source package
> names for extra things in proposed to retry failing dep8 tests against.
> But looking up all the versions, converting to source package names and
> URL-encoding manually was tedious. retry-autopkgtest-regressions in
> ubuntu-archive-tools didn't seem suitable for this, and nor could I find
> anything in ubuntu-helpers, so I knocked something up for this use case:
> https://git.launchpad.net/~ubuntu-server/+git/ubuntu-helpers/tree/rbasak/autopkgtest-urls
> I just give it the source or binary package names I want, and it looks
> up the details from a chdist and outputs the retry URLs.

Yeah I also made a tool for that (I'm sure others have as well).  Mine
is at:

    https://launchpad.net/~bryce/+git/excuses-kicker

It uses a downloaded autopkgtest.db to identify other packages that the
given package has been triggered with before, looks up if those packages
have versions in -proposed, and then constructs URLs including triggers
for those packages.

  $ excuses-kicker -a amd64 php-doctrine-dbal
  https://autopkgtest.ubuntu.com/request.cgi?release=jammy&arch=amd64&package=php-doctrine-dbal&trigger=php-doctrine-dbal%2F3.3.2%2Bdfsg-1ubuntu1&trigger=php-doctrine-persistence%2F2.3.0-2ubuntu2&trigger=symfony%2F5.4.4%2Bdfsg-1ubuntu2&trigger=orafce%2F3.18.1-1&trigger=doctrine%2F2.11.1%2Bdfsg-1ubuntu1&trigger=php-nesbot-carbon%2F2.55.2-1&trigger=php-twig%2F3.3.8-2&trigger=php-doctrine-data-fixtures%2F1.5.2-1&trigger=gtk4%2F4.6.1%2Bds-1&trigger=php8.1%2F8.1.2-1build1&trigger=php-symfony-security-acl%2F3.3.0-1&trigger=postgresql-14%2F14.2-1&trigger=link-grammar%2F5.10.2~dfsg-2ubuntu1&trigger=php-psr-log%2F3.0.0-1&trigger=php-doctrine-cache%2F2.1.1-3ubuntu1

There's also options to add/remove triggers to customize things a bit.
It's still missing a few features and the docs are poor so I hesistate
to recommend it broadly but I've found this tool quite effective at
sorting out a lot of the run-of-the-mill migration problems I run into.

Bryce