[Bug 1501772] Re: Metadata proxy process errors with binary user_data

Fri Oct 2 19:52:56 UTC 2015

Reviewed:  https://review.openstack.org/230513
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=13b6d76da361178ea0b84baf4831a4c7f53e29c7
Submitter: Jenkins
Branch:    stable/liberty

commit 13b6d76da361178ea0b84baf4831a4c7f53e29c7
Author: Ihar Hrachyshka <ihrachys at redhat.com>
Date:   Thu Oct 1 17:13:25 2015 +0200

    metadata: don't crash proxy on non-unicode user data

    We attempt to log every successful metadata response with LOG.debug. But
    as per oslo.log docs [1], we should make sure that what we pass into the
    library is unicode.

    Http.request returns a tuple of Response object and a string, which is
    bytes in Python 2.x [2].

    That's why we need to convert the response content to unicode before
    passing it into oslo.log.

    To achieve it, we utilize encodeutils.safe_decode with 'replace' errors
    handling strategy, so that we don't get exceptions on input that does
    not conform unicode.

    For the unit test case, we pass a string that is not expected to convert
    to unicode with errors='strict' strategy or similar, and check that we
    still don't crash.

    While at it, we remove a check for the number of log calls being
    triggered, because it's something that we should avoid validating in
    test cases, and it cannot trigger a real bug. The mock that was used to
    count the number would also hide the bug that we try to reproduce.

    Note that the bug does not require debug to be set because the crash
    occurs before oslo.log machinery decides it should not log the message.

    [1]: http://docs.openstack.org/developer/oslo.log/usage.html#no-more-implicit-conversion-to-unicode-str
    [2]: http://bitworking.org/projects/httplib2/doc/html/libhttplib2.html#httplib2.Http.request

    Closes-Bug: #1501772
    Change-Id: I6a32c40ff117fae43913386134c8981539697ce8
    (cherry picked from commit 80e3d9be4923ecad17377b1d15c8392b8a43dac6)

** Tags added: in-stable-liberty

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1501772

Title:
  Metadata proxy process errors with binary user_data

Status in neutron:
  Fix Committed
Status in neutron package in Ubuntu:
  In Progress

Bug description:
  Boot instances with binary user data content (rather than simple text)
  is not happy right now:

  2015-10-01 13:19:39.109 10854 DEBUG neutron.agent.metadata.namespace_proxy [-] {'date': 'Thu, 01 Oct 2015 13:19:39 GMT', 'status': '200', 'content-length': '979', 'content-type': 'text/plain; charset=UTF-8', 'content-location': u'http://169.254.169.254/openstack/2013-10-17/user_data'} _proxy_request /usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py:90
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy [-] Unexpected error.
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy Traceback (most recent call last):
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py", line 55, in __call__
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy     req.body)
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py", line 91, in _proxy_request
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy     LOG.debug(content)
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/logging/__init__.py", line 1437, in debug
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy     msg, kwargs = self.process(msg, kwargs)
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/oslo_log/log.py", line 139, in process
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy     msg = _ensure_unicode(msg)
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/oslo_log/log.py", line 113, in _ensure_unicode
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy     errors='xmlcharrefreplace',
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/oslo_utils/encodeutils.py", line 43, in safe_decode
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy     return text.decode(incoming, errors)
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy     return codecs.utf_8_decode(input, errors, True)
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy TypeError: don't know how to handle UnicodeDecodeError in error callback
  2015-10-01 13:19:39.109 10854 ERROR neutron.agent.metadata.namespace_proxy
  2015-10-01 13:19:39.112 10854 INFO neutron.wsgi [-] 192.168.21.15 - - [01/Oct/2015 13:19:39] "GET /openstack/2013-10-17/user_data HTTP/1.1" 500 343 0.014536

  This is thrown be the log call just prior to it being served back to
  the instance.

  ProblemType: Bug
  DistroRelease: Ubuntu 15.10
  Package: neutron-metadata-agent 2:7.0.0~b3-0ubuntu3
  ProcVersionSignature: Ubuntu 4.2.0-11.13-generic 4.2.1
  Uname: Linux 4.2.0-11-generic x86_64
  ApportVersion: 2.19-0ubuntu1
  Architecture: amd64
  Date: Thu Oct  1 13:38:21 2015
  Ec2AMI: ami-000005ce
  Ec2AMIManifest: FIXME
  Ec2AvailabilityZone: nova
  Ec2InstanceType: m1.small.osci
  Ec2Kernel: None
  Ec2Ramdisk: None
  JournalErrors: -- No entries --
  PackageArchitecture: all
  SourcePackage: neutron
  UpgradeStatus: No upgrade log present (probably fresh install)
  mtime.conffile..etc.neutron.metadata.agent.ini: 2015-10-01T13:18:25.075633

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1501772/+subscriptions