[Bug 1282861] Related fix merged to python-swiftclient (master)

[OpenStack Infra]

Reviewed:  https://review.openstack.org/184956
Committed: https://git.openstack.org/cgit/openstack/python-swiftclient/commit/?id=f728027bed59d08a6491ae9c14d2f5968f8d6fa3
Submitter: Jenkins
Branch:    master

commit f728027bed59d08a6491ae9c14d2f5968f8d6fa3
Author: Tim Burke <tim.burke at gmail.com>
Date:   Thu May 21 22:44:36 2015 -0700

    Accept gzip-encoded API responses
    
    Previously, we would accept gzip-encoded responses, but only because we
    were letting requests decode *all* responses (even object data). This
    restores the previous capability, but with tighter controls about which
    requests will accept gzipped responses and where the decoding happens.
    
    Change-Id: I4fd8b97207b9ab01b1bcf825cc16efd8ad46344a
    Related-Bug: 1282861
    Related-Bug: 1338464

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to python-swiftclient in Ubuntu.
https://bugs.launchpad.net/bugs/1282861

Title:
  crash on content-encoding:gzip http responses

Status in python-swiftclient:
  Fix Released
Status in python-swiftclient package in Ubuntu:
  Fix Released
Status in python-swiftclient source package in Trusty:
  Confirmed
Status in python-swiftclient source package in Utopic:
  Fix Released

Bug description:
  
  Versions:
  python-swiftclient==2.0.2
  requests==2.2.1

  
  python-swiftclient cannot seem to handle gziped http response.


  DEBUG:swiftclient:RESP HEADERS: [('x-container-object-count', '352'), ('content-encoding', 'gzip'), ('transfer-encoding', 'chunked'), ('accept-ranges', 'bytes'), ('date', 'Fri, 21 Feb 2014 02:51:24 GMT'), ('x-timestamp', '1363448617.07881'), ('x-trans-id', 'tx7adcd463316f4393a81ec046dcc7e585'), ('x-container-bytes-used', '4023589531'), ('content-type', 'application/json; charset=utf-8')]
  DEBUG:swiftclient:RESP BODY:[A LOT OF BINARY (GZIPPED) DATA]
  Traceback (most recent call last):
    File "./bin/swift", line 1488, in <module>
      globals()['st_%s' % args[0]](parser, argv[1:], thread_manager)
    File "./bin/swift", line 592, in st_list
      prefix=options.prefix, delimiter=options.delimiter)[1]
    File "/dev/python/env/lib/python2.7/site-packages/swiftclient/client.py", line 1263, in get_container
      full_listing=full_listing)
    File "/dev/python/env/lib/python2.7/site-packages/swiftclient/client.py", line 1192, in _retry
      rv = func(self.url, self.token, *args, **kwargs)
    File "/dev/python/env/lib/python2.7/site-packages/swiftclient/client.py", line 592, in get_container
      return resp_headers, json_loads(body)
    File "/dev/python/env/lib/python2.7/site-packages/simplejson/__init__.py", line 488, in loads
      return _default_decoder.decode(s)
    File "/dev/python/env/lib/python2.7/site-packages/simplejson/decoder.py", line 370, in decode
      obj, end = self.raw_decode(s)
    File "/dev/python/lib/python2.7/site-packages/simplejson/decoder.py", line 389, in raw_decode
      return self.scan_once(s, idx=_w(s, idx).end())
  simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

  
  HTTP requests are made through the HTTPConnection class, which is a wrapper around python-requests mimicking the httplib API ?..

  A way to fix this would be to let urllib do the decoding by passing
  decode_content=True in HTTPConnection.getresponse()

           self.resp.getheaders = getheaders
           self.resp.getheader = getheader
  -        self.resp.read = self.resp.raw.read
  +        self.resp.read = functools.partial(self.resp.raw.read, decode_content=True)
           return self.resp

  The other way would be to let every function using
  HTTPConnection.getresponse() handle gzip decoding manually...

To manage notifications about this bug go to:
https://bugs.launchpad.net/python-swiftclient/+bug/1282861/+subscriptions