Considering using --needs-internet=try in autopkgtest

[Iain Lane]

As background, our autopkgtest workers can only access the wider 
internet via a http/s proxy. DNS resolution also works, but other 
services most likely will not.

A while ago I advised some folks to make a test 'skippable', where it 
was trying to access resources that couldn't be accessed due to this.

This patch was duly forwarded to the Debian maintainer - thanks - as

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=969057

you can read that log for the full discussion, but the tl;dr is that 
they feel that the "needs-internet" restriction which is already present 
in the package should cover this case.

I've got to say that I am kind of convinced by that.  
README.package-tests.rst says:

  The test needs unrestricted internet access

and also

  If a test does use the internet outside of the pre-configured apt 
  domain, the test must be marked with the needs-internet restriction.  
  Using the internet usually makes tests less reliable, so this should 
  be kept to a minimum. But for many packages their main purpose is to 
  interact with remote web services and thus their testing should 
  actually cover those too, to ensure that the distribution package 
  keeps working with their corresponding web service.

Given this, I think we're on dodgy ground asking the maintainer to make 
changes. We have two choices that I can see:

  * Accept that some tests which need to access external services will 
    fail, and we might have to hint (ignore) those failures.
  * Run autopkgtest with "--needs-internet=try", which means that we run 
    the test but treat it as flaky if it fails.

I prefer the latter since it will result in less work for the release 
team, but it does come with a caveat which is that needs-internet tests 
will essentially no longer be able to fail. Given that we don't have a 
restriction which expresses our particular flavour of internet access, 
tests which only use http/s, and so are totally fine for us to run fully 
currently, will regress in coverage. We won't be able to detect legit 
failures any more.

In the medium term, if scheduled, we could try to work on a way to 
express our type of internet access and then have packages use that*, 
but here I'd like to limit the discussion to solutions we can implement 
now, please.

Would appreciate opinions.

Cheers,

-- 
Iain Lane                                  [ iain at orangesquash.org.uk ]
Debian Developer                                   [ laney at debian.org ]
Ubuntu Developer                                   [ laney at ubuntu.com ]

* To avoid restriction proliferation, I am thinking that parameterised 
  restrictions like "needs-internet=http-https-only" might be a neat way 
  to solve this. Just an idea right now, not discussed with autopkgtest 
  maintainers.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-release/attachments/20210114/b323788a/attachment.sig>