[Bug 1837871] Re: Add retry logic to snap-tool to make downloads more resilient

[Łukasz Zemczak]

** Changed in: livecd-rootfs (Ubuntu)
   Importance: Low => High

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1837871

Title:
  Add retry logic to snap-tool to make downloads more resilient

Status in livecd-rootfs package in Ubuntu:
  In Progress

Bug description:
  [Impact]

  * livecd-rootfs builds will fail immediately when a snap-tool
  invocation fails to contact the snapstore because of ephemeral
  connection problems or a transient error on the server side.

  * The snap-tool script included with livecd-rootfs in Eoan has been
  enhanced to retry on connection errors and 5xx server errors reducing
  the likelihood of image builds breaking due to a flaky connection or a
  server hiccup.

  [Test Cases]

  * Download core, core18 snaps using both `snap download <snap-name>`
  and `snap-tool download <snap-name>` and make sure the downloads are
  identical.

  * Invoke `snap-tool info <snap-name>` for a few snaps, e.g. review-
  gator, lpshipit, azure-cli and verify that all fields carry correct
  information.

  * Test the backoff/retry logic using the following procedure:

  Make netcat listen on port 12345

      netcat -l -p 12345

  Create a symlink from snaptool.py to snap-tool and import the
  ExpBackoffHTTPClient class from a Python session:

      ln -s snap-tool snaptool.py

      python3
      from snaptool import ExpBackoffHTTPClient
      http_client = ExpBackoffHTTPClient()
      request = http_client.get("http://127.0.0.1:12345/")
      request.text()

  Go back to the terminal where you invoked netcat and stop it. snap-
  tool should print the following and then fail:

  WARNING: failed to open URL 'http://127.0.0.1:12345/': Remote end closed connection without response
  Retrying HTTP request in 2 seconds...
  WARNING: failed to open URL 'http://127.0.0.1:12345/': <urlopen error [Errno 111] Connection refused>
  Retrying HTTP request in 4 seconds...

  Repeat the procedure above but instead of stopping netcat, paste the
  following response:

  HTTP/1.1 503 Error

  and hit enter twice. You should see

  WARNING: failed to open URL 'http://127.0.0.1:12345/': HTTP Error 503: Error
  Retrying HTTP request in 2 seconds...

  Repeat the above pasting "HTTP/1.1 404 Not found" instead. The snap-
  tool should fail immediately.

  [Regression Potential]

   * Tool logic and http request headers/body are unchanged, only the
  way connections are built has been modified. The expectation is that
  this will be more robust and testing in devel hasn't surfaced any
  bugs, but there is a slight risk that the tool's behavior has changed
  in unobvious corner cases that were missed during testing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1837871/+subscriptions