[Bug 1946520] Re: magic-proxy does not gracefully handle error conditions

[Utkarsh Gupta]

** Changed in: livecd-rootfs (Ubuntu)
     Assignee: (unassigned) => Thomas Bechtold (toabctl)

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1946520

Title:
  magic-proxy does not gracefully handle error conditions

Status in livecd-rootfs package in Ubuntu:
  In Progress

Bug description:
  During an impish build of ubuntu-cpc images with magic-proxy enabled,
  the installation of certain packages tried to reach an internet
  endpoint (specifically http://169.254.169.254/latest/meta-
  data/instance-id - which appears to be used for EC2 metadata)

  The magic-proxy log indicated :

  10.10.10.2 - - [01/Oct/2021 13:21:26] urlopen() failed for http://169.254.169.254/2009-04-04/meta-data/instance-id with [Errno 110] Connection timed out
  10.10.10.2 - - [01/Oct/2021 13:21:26] code 501, message [Errno 110] Connection timed out
  10.10.10.2 - - [01/Oct/2021 13:21:26] "GET /2009-04-04/meta-data/instance-id HTTP/1.1" 501 -
  ----------------------------------------
  Exception occurred during processing of request from ('10.10.10.2', 49672)
  Traceback (most recent call last):
    File "/usr/lib/python3.9/urllib/request.py", line 1346, in do_open
      h.request(req.get_method(), req.selector, req.data, headers,
    File "/usr/lib/python3.9/http/client.py", line 1279, in request
      self._send_request(method, url, body, headers, encode_chunked)
    File "/usr/lib/python3.9/http/client.py", line 1325, in _send_request
      self.endheaders(body, encode_chunked=encode_chunked)
    File "/usr/lib/python3.9/http/client.py", line 1274, in endheaders
      self._send_output(message_body, encode_chunked=encode_chunked)
    File "/usr/lib/python3.9/http/client.py", line 1034, in _send_output
      self.send(msg)
    File "/usr/lib/python3.9/http/client.py", line 974, in send
      self.connect()
    File "/usr/lib/python3.9/http/client.py", line 945, in connect
      self.sock = self._create_connection(
    File "/usr/lib/python3.9/socket.py", line 844, in create_connection
      raise err
    File "/usr/lib/python3.9/socket.py", line 832, in create_connection
      sock.connect(sa)
  TimeoutError: [Errno 110] Connection timed out

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/share/livecd-rootfs/magic-proxy", line 873, in __get_request
      with urllib.request.urlopen(
    File "/usr/lib/python3.9/urllib/request.py", line 214, in urlopen
      return opener.open(url, data, timeout)
    File "/usr/lib/python3.9/urllib/request.py", line 517, in open
      response = self._open(req, data)
    File "/usr/lib/python3.9/urllib/request.py", line 534, in _open
      result = self._call_chain(self.handle_open, protocol, protocol +
    File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
      result = func(*args)
    File "/usr/lib/python3.9/urllib/request.py", line 1375, in http_open
      return self.do_open(http.client.HTTPConnection, req)
    File "/usr/lib/python3.9/urllib/request.py", line 1349, in do_open
      raise URLError(err)
  urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/lib/python3.9/socketserver.py", line 683, in process_request_thread
      self.finish_request(request, client_address)
    File "/usr/lib/python3.9/socketserver.py", line 360, in finish_request
      self.RequestHandlerClass(request, client_address, self)
    File "/usr/lib/python3.9/socketserver.py", line 747, in __init__
      self.handle()
    File "/usr/lib/python3.9/http/server.py", line 427, in handle
      self.handle_one_request()
    File "/usr/lib/python3.9/http/server.py", line 415, in handle_one_request
      method()
    File "/usr/share/livecd-rootfs/magic-proxy", line 787, in do_GET
      self.__get_request()
    File "/usr/share/livecd-rootfs/magic-proxy", line 887, in __get_request
      self.send_error(501, e.reason)
    File "/usr/lib/python3.9/http/server.py", line 473, in send_error
      'message': html.escape(message, quote=False),
    File "/usr/lib/python3.9/html/__init__.py", line 19, in escape
      s = s.replace("&", "&") # Must be done first!
  AttributeError: 'TimeoutError' object has no attribute 'replace'

  Because magic-proxy is proxying all traffic to port 80, it will
  intercept these sort of messages. The TimeoutError ends up bubbling up
  to the serversocket handling code, which will then cause a broken pipe
  for all future communications, effectively breaking future apt calls
  from within the build environment.

  The concrete problem here is that inside magic-proxy, self.send_error
  is called with the argument of e.reason, which can be a string (no
  error) or an exception (in case of a nested exception). If the nested
  exception is passed into self.send_error, html.replace will try to be
  called on the exception, hence the above exception.

  In addition to fixing this specific problem, magic-proxy's request
  handling should catch all exceptions to not interfere with higher up
  socketserver handling.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1946520/+subscriptions