+1 maintenance report

[Julian Andres Klode]

On Mon, Feb 15, 2021 at 05:36:15PM +0100, Jan Ceuleers wrote:
> On 13/02/2021 04:49, Seth Arnold wrote:
> > Could we build a retriggerbot that smashes the retry button three times
> > before bothering any humans about failed tests?
> >
> > Hitting retry is often the first troubleshooting step people take; I've
> > heard tests may be retried something like ten times by different people,
> > each of whom was taking a reasonable enough "first debugging step"
> > without noticing that other people have also done the same.
> 
> Not an Ubuntu developer but I do work as a quality manager. Not sure
> whether my list post will be accepted, so I'm copying you.
> 
> The assumption underlying your suggestion is that tests that
> intermittently fail do so because of intermittent failures in the test
> environment rather than due to actual bugs that manifest themselves only
> intermittently (such as race conditions).
> 
> This is fine if you have evidence that the assumption holds in a
> sufficiently large majority of cases.

Analysing the root cause of intermittent failures is in practice wishful
thinking though. There are too many tests to fix all those race
conditions. The only thing we can do is mindlessly retry stuff.

The only two cases we care about are stable tests becoming flaky, and
flaky tests becoming consistent failures, and both of these will be
flagged by not migrating to release pocket.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en