[Bug 1493303] Re: [OSSA 2016-004] Swift proxy memory leak on unfinished read (CVE-2016-0738)

Fri Jan 22 03:40:00 UTC 2016

Reviewed:  https://review.openstack.org/270234
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=a4c1825a026655b7ed21d779824ae7c25318fd52
Submitter: Jenkins
Branch:    stable/kilo

commit a4c1825a026655b7ed21d779824ae7c25318fd52
Author: Samuel Merritt <sam at swiftstack.com>
Date:   Tue Dec 8 16:36:05 2015 -0800

    Fix memory/socket leak in proxy on truncated SLO/DLO GET

    When a client disconnected while consuming an SLO or DLO GET response,
    the proxy would leak a socket. This could be observed via strace as a
    socket that had shutdown() called on it, but was never closed. It
    could also be observed by counting entries in /proc/<pid>/fd, where
    <pid> is the pid of a proxy server worker process.

    This is due to a memory leak in SegmentedIterable. A SegmentedIterable
    has an 'app_iter' attribute, which is a generator. That generator
    references 'self' (the SegmentedIterable object). This creates a
    cyclic reference: the generator refers to the SegmentedIterable, and
    the SegmentedIterable refers to the generator.

    Python can normally handle cyclic garbage; reference counting won't
    reclaim it, but the garbage collector will. However, objects with
    finalizers will stop the garbage collector from collecting them* and
    the cycle of which they are part.

    For most objects, "has finalizer" is synonymous with "has a __del__
    method". However, a generator has a finalizer once it's started
    running and before it finishes: basically, while it has stack frames
    associated with it**.

    When a client disconnects mid-stream, we get a memory leak. We have
    our SegmentedIterable object (call it "si"), and its associated
    generator. si.app_iter is the generator, and the generator closes over
    si, so we have a cycle; and the generator has started but not yet
    finished, so the generator needs finalization; hence, the garbage
    collector won't ever clean it up.

    The socket leak comes in because the generator *also* refers to the
    request's WSGI environment, which contains wsgi.input, which
    ultimately refers to a _socket object from the standard
    library. Python's _socket objects only close their underlying file
    descriptor when their reference counts fall to 0***.

    This commit makes SegmentedIterable.close() call
    self.app_iter.close(), thereby unwinding its generator's stack and
    making it eligible for garbage collection.

    * in Python < 3.4, at least. See PEP 442.

    ** see PyGen_NeedsFinalizing() in Objects/genobject.c and also
       has_finalizer() in Modules/gcmodule.c in Python.

    *** see sock_dealloc() in Modules/socketmodule.c in Python. See
        sock_close() in the same file for the other half of the sad story.

    This closes CVE-2016-0738.

    Closes-Bug: 1493303

    Change-Id: I9b617bfc152dca40d1750131d1d814d85c0a88dd
    Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota at lab.ntt.co.jp>

** Tags added: in-stable-kilo

** Tags added: in-stable-liberty

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to swift in Ubuntu.
https://bugs.launchpad.net/bugs/1493303

Title:
  [OSSA 2016-004] Swift proxy memory leak on unfinished read
  (CVE-2016-0738)

Status in Ubuntu Cloud Archive:
  New
Status in OpenStack Security Advisory:
  Fix Committed
Status in OpenStack Object Storage (swift):
  Fix Released
Status in swift package in Ubuntu:
  Confirmed

Bug description:
  It looks like the Swift proxy will leak memory if the connection is
  closed and the full response is not read. This opens for a potential
  DoS attacks.

  Reproduce:

  $ swift -A http://localhost:8888/auth/v1.0 -U .. -K .. upload --use-slo --segment-size 1048576 <container> <big-file>
  $ curl -H'X-Auth-Token: AUTH_...' "http://localhost:8888/v1/AUTH_../<container>/<big-file>" -m 0.001 > /dev/null

  Repeat the curl command a couple of times and you will have more
  information in netstat and sockstat. The important part is the -m
  which sets the max time curl spends at downloading. After that point,
  it'll close the connection.

  $ sudo netstat -ant -p | grep :6000
  $ cat /proc/net/sockstat

  tcp        0      0 127.0.0.1:6000          0.0.0.0:*               LISTEN      1358/python
  tcp        0  43221 127.0.0.1:6000          127.0.0.1:48350         FIN_WAIT1   -
  tcp        0  43221 127.0.0.1:6000          127.0.0.1:48882         FIN_WAIT1   -
  tcp   939820      0 127.0.0.1:48350         127.0.0.1:6000          ESTABLISHED 17897/python
  tcp   939820      0 127.0.0.1:48882         127.0.0.1:6000          ESTABLISHED 17890/python
  tcp   983041      0 127.0.0.1:48191         127.0.0.1:6000          CLOSE_WAIT  17897/python
  tcp   983041      0 127.0.0.1:48948         127.0.0.1:6000          CLOSE_WAIT  17892/python

  Restarting the proxy frees up the lingering memory.

  This problem did not exist in 2.2.0.

  ProblemType: Bug
  DistroRelease: Ubuntu 14.04
  Package: swift 2.2.2-0ubuntu1~cloud0 [origin: Canonical]
  ProcVersionSignature: Ubuntu 3.16.0-48.64~14.04.1-generic 3.16.7-ckt15
  Uname: Linux 3.16.0-48-generic x86_64
  ApportVersion: 2.14.1-0ubuntu3.12
  Architecture: amd64
  CrashDB:
   {
                  "impl": "launchpad",
                  "project": "cloud-archive",
                  "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
               }
  Date: Tue Sep  8 09:55:05 2015
  InstallationDate: Installed on 2015-06-22 (77 days ago)
  InstallationMedia: Ubuntu-Server 14.04.2 LTS "Trusty Tahr" - Release amd64 (20150218.1)
  PackageArchitecture: all
  SourcePackage: swift
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1493303/+subscriptions