[Bug 1899994] Re: do-release-upgrade fails with TLS inspecting proxy (if CA is not installed system wide)

[DaJo]

The legal clearance is in the works.
I expect this to be ready in the next weeks. Sorry for the delays.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to update-manager in Ubuntu.
https://bugs.launchpad.net/bugs/1899994

Title:
  do-release-upgrade fails with TLS inspecting proxy (if CA is not
  installed system wide)

Status in update-manager package in Ubuntu:
  New

Bug description:
  Problem: do-release-upgrade fails with TLS inspecting proxy (if CA is not installed system wide)
  Solution: patch provided below (at least for the detection)
            additional error reporting: see below for demonstration of added error messages.

  Related bug that might be fixed by the attached patch:
  https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/1821034

  Affected package (Bionic and package main branch also does not fix it):
  Package: python3-update-manager
  Architecture: all
  Version: 1:18.04.11.13
  Priority: standard
  Section: python
  Source: update-manager
  Origin: Ubuntu
  Maintainer: Ubuntu Developers <ubuntu-devel-discuss at lists.ubuntu.com>
  Bugs: https://bugs.launchpad.net/ubuntu/+filebug

  
  Problem description details:
    Company environment with HTTP-proxy required to connect to the internet.
    The proxy inspects HTTPS traffic and changes HTTPS server's certificate.
    The proxy's CA is not installed/trusted system wide.
    APT is configured to use additional CA certificate file via "Acquire::https::CAInfo "/etc/ssl/company/proxyCA.pem" to trust the proxy when downloading updates via HTTPS.

    After I created the patch I learned about:
  ```
  /usr/lib/apt/apt-helper auto-detect-proxy "https://www.ubuntu.com"
  Using proxy '' for URL 'https://www.ubuntu.com/'
  ```
    but that command does not output proxy or CA information for me. This might be another bug?
    
    I've provided a patch that applies to python3-update-manager AND python3-distupgrade (with changed paths - not sure why there are redundant copies).
    And allows do-release-upgrade to detect / use the correct certificate while not breaking existing setups (as far as I can tell).

  System details:
  1) lsb_release -a
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 18.04.5 LTS
  Release:        18.04
  Codename:       bionic
  2) apt-cache policy python3-update-manager
   python3-update-manager:
    Installed: 1:20.04.10.1
    Candidate: 1:20.04.10.1
    Version table:
   *** 1:20.04.10.1 500
          500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
          100 /var/lib/dpkg/status
       1:20.04.9 500
          500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
  3) Expectation: do-release-upgrade works and picks up working apt configuration
  4) Actual: do-release-upgrade reports no updates (stuck on Bionic)

  Solution:
  Note that an apt miss-configuration does only output an error/warning, but then the attempt to check for upgrades is continued (without TLS-config or proxy).
  So the proxy and default trust store is used to access the https URL.

  Additionally if that fails due to certificate mismatch that error is now reported.
  As well as timeouts or BadStatusLine errors - just to understand the root cause of the problem.
  ```
  $ do-release-upgrade
  Error failed to read '/etc/ssl/company/company_proxy.pem2' from apt conf: [Errno 2] No such file or directory
  Checking for a new Ubuntu release
  Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings
          Reason: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)                                                                                   There is no development version of an LTS available.
  To upgrade to the latest non-LTS development release
  set Prompt=normal in /etc/update-manager/release-upgrades.
  ```

  Example /etc/apt/apt.conf.d/proxy.conf:
  ```
  Acquire::http::Proxy "http://proxy.example.org:8080";
  Acquire::https::Proxy "http://proxy.example.org:8080";
  Acquire::https::CAInfo "/etc/ssl/company/proxyCA.pem";
  ```

  The successful update with debug information now looks like this (while it failed before):
  ```
  $ DEBUG_UPDATE_MANAGER="yes" do-release-upgrade
  Checking for a new Ubuntu release
  MetaRelease.__init__() useDevel=False useProposed=False
  /etc/update-manager/meta-release: https://changelogs.ubuntu.com/meta-release
  /etc/update-manager/meta-release: https://changelogs.ubuntu.com/meta-release-lts
  /etc/update-manager/meta-release: -development
  /etc/update-manager/meta-release: -proposed
  metarelease-uri: https://changelogs.ubuntu.com/meta-release-lts
  MetaRelease.download()
  have self.metarelease_information
  MetaRelease.parse()
  current dist name: 'bionic'
  found distro name: 'dapper'
  found distro name: 'hardy'
  found distro name: 'lucid'
  found distro name: 'precise'
  found distro name: 'trusty'
  found distro name: 'xenial'
  found distro name: 'bionic'
  found distro name: 'focal'
  new dist: <UpdateManager.Core.MetaRelease.Dist object at 0x7f5ac7022e80>
  Please install all available updates for your release before upgrading.
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/1899994/+subscriptions