[Bug 1837871] Re: Add retry logic to snap-tool to make downloads more resilient

[Adam Conrad]

Hello Tobias, or anyone else affected,

Accepted livecd-rootfs into disco-proposed. The package will build now
and be available at https://launchpad.net/ubuntu/+source/livecd-
rootfs/2.578.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-disco to verification-done-disco. If it does not fix
the bug for you, please add a comment stating that, and change the tag
to verification-failed-disco. In either case, without details of your
testing we will not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: livecd-rootfs (Ubuntu Disco)
       Status: New => Fix Committed

** Tags added: verification-needed verification-needed-disco

** Changed in: livecd-rootfs (Ubuntu)
       Status: In Progress => Fix Released

** Changed in: livecd-rootfs (Ubuntu Bionic)
       Status: New => Fix Committed

** Tags added: verification-needed-bionic

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1837871

Title:
  Add retry logic to snap-tool to make downloads more resilient

Status in livecd-rootfs package in Ubuntu:
  Fix Released
Status in livecd-rootfs source package in Bionic:
  Fix Committed
Status in livecd-rootfs source package in Disco:
  Fix Committed

Bug description:
  [Impact]

  * livecd-rootfs builds will fail immediately when a snap-tool
  invocation fails to contact the snapstore because of ephemeral
  connection problems or a transient error on the server side.

  * The snap-tool script included with livecd-rootfs in Eoan has been
  enhanced to retry on connection errors and 5xx server errors reducing
  the likelihood of image builds breaking due to a flaky connection or a
  server hiccup.

  [Test Cases]

  * Download core, core18 snaps using both `snap download <snap-name>`
  and `snap-tool download <snap-name>` and make sure the downloads are
  identical.

  * Invoke `snap-tool info <snap-name>` for a few snaps, e.g. review-
  gator, lpshipit, azure-cli and verify that all fields carry correct
  information.

  * Test the backoff/retry logic using the following procedure:

  Make netcat listen on port 12345

      netcat -l -p 12345

  Create a symlink from snaptool.py to snap-tool and import the
  ExpBackoffHTTPClient class from a Python session:

      ln -s snap-tool snaptool.py

      python3
      from snaptool import ExpBackoffHTTPClient
      http_client = ExpBackoffHTTPClient()
      request = http_client.get("http://127.0.0.1:12345/")
      request.text()

  Go back to the terminal where you invoked netcat and stop it. snap-
  tool should print the following and then fail:

  WARNING: failed to open URL 'http://127.0.0.1:12345/': Remote end closed connection without response
  Retrying HTTP request in 2 seconds...
  WARNING: failed to open URL 'http://127.0.0.1:12345/': <urlopen error [Errno 111] Connection refused>
  Retrying HTTP request in 4 seconds...

  Repeat the procedure above but instead of stopping netcat, paste the
  following response:

  HTTP/1.1 503 Error

  and hit enter twice. You should see

  WARNING: failed to open URL 'http://127.0.0.1:12345/': HTTP Error 503: Error
  Retrying HTTP request in 2 seconds...

  Repeat the above pasting "HTTP/1.1 404 Not found" instead. The snap-
  tool should fail immediately.

  [Regression Potential]

   * Tool logic and http request headers/body are unchanged, only the
  way connections are built has been modified. The expectation is that
  this will be more robust and testing in devel hasn't surfaced any
  bugs, but there is a slight risk that the tool's behavior has changed
  in unobvious corner cases that were missed during testing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1837871/+subscriptions