[Bug 1724342] Re: test_drop_privileges AssertionError groups mismatch

Fri Oct 20 16:00:29 UTC 2017

Reviewed:  https://review.openstack.org/513730
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=bb064f0b30ec083cec85e8371fe12824804779cf
Submitter: Zuul
Branch:    feature/s3api

commit 0bca0d4d2bf95a37e4a403457ee5c7db4c0b6290
Author: Pete Zaitcev <zaitcev at kotori.zaitcev.us>
Date:   Wed Oct 18 18:15:11 2017 -0500

    Be more tolerant of exception messages from sqlite

    Parsing the text of error messages was always perilous, perhaps.
    Either way, this should fix the problem of test_get() failing.

    Change-Id: I929c2f6e60624241075a292d020c5707ea0ff1c3

commit e001c02ff938d9aea560970fa70f3d8884a8ea33
Author: Tim Burke <tim.burke at gmail.com>
Date:   Tue Oct 17 22:49:39 2017 +0000

    Stop logging tracebacks on bad xLOs

    The error messages alone should provide plenty enough information.
    Plus, running functional tests really shouldn't cause tracebacks.

    Also, tighten up tests for log messages.

    Change-Id: I55136484d342af756fa153d971dcb9159a435f13

commit 356b110229ee5e5be93ce1ec60ed2de8f75090bf
Author: Samuel Merritt <sam at swiftstack.com>
Date:   Wed Oct 18 15:09:12 2017 -0700

    Remove swift-temp-url man page.

    Commit 250da37a removed bin/swift-temp-url, but left the man page.

    Change-Id: Ic518f23678e3c3134f02a46a51a2bcb90d92bdc2

commit 1aecd1dfc61988f12d4cdfda7110dd648133e4cf
Author: Alistair Coles <alistairncoles at gmail.com>
Date:   Wed Oct 18 10:52:52 2017 +0100

    tighten up drop_privileges unit tests

    add more assertions about args that are passed to
    os module functions

    Related-Change: Ida15e72ae4ecdc2d6ce0d37bd99c2d86bd4e5ddc
    Change-Id: Iee483274aff37fc9930cd54008533de2917157f4

commit e6cf9b4758612f38f7317c52423f317b9a11bbc9
Author: Samuel Merritt <sam at swiftstack.com>
Date:   Tue Oct 17 15:16:43 2017 -0700

    Fix some spelling in a docstring

    Change-Id: I6b32238da3381848ae56aed1c570d299be72473e

commit 646f7507a1f5cf479b8d2023645009de0a28cd79
Author: Tim Burke <tim.burke at gmail.com>
Date:   Tue Oct 17 21:17:57 2017 +0000

    Quiet test output when running test_utils.py in isolation

    Change-Id: I4cf85d3cd5a20424e9bbbdf0213120b3c3d4b837

commit 0f91c862e19bb8d81c20d03e65e6e6657f7265dc
Author: Corey Bryant <corey.bryant at canonical.com>
Date:   Tue Oct 17 15:04:45 2017 -0400

    Drop group comparison from drop_privileges test

    Drop the group comparison from drop_privileges test as it isn't
    valid since os.setgroups() is mocked.

    Change-Id: Ida15e72ae4ecdc2d6ce0d37bd99c2d86bd4e5ddc
    Closes-Bug: #1724342

commit 5438fe7d9bcb7dc99118c12bdf1ccfd663390b08
Author: Samuel Merritt <sam at swiftstack.com>
Date:   Fri Oct 13 17:03:18 2017 -0700

    Clean up some leftover Python 2.6-isms.

    Change-Id: I24f0aa3bf7314a669c4cf44eadc46e874d6803f1

commit 250da37a7b279b05a1ac8954b506add1661f194d
Author: Tim Burke <tim.burke at gmail.com>
Date:   Fri Oct 13 23:27:11 2017 +0000

    Remove swift-temp-url script

    This has been deprecated since Swift 2.10.0 (Newton) including a
    message that it would go away. Let's actually remove it.

    Change-Id: I7d3659761c71119363ff2c0c750e37b4c6374a39
    Related-Change: Ifa8bf636f20f82db4845b02d1b58699edaa39356

commit c118059719ded85381e9b134b091d04a7b5a3955
Author: Tim Burke <tim.burke at gmail.com>
Date:   Mon Sep 11 22:08:12 2017 +0000

    Respond 400 Bad Request when Accept headers fail to parse

    Change-Id: I6eb4e4bca95e2ee4fecdb703394cb2419737922d
    Closes-Bug: 1716509

commit 03e8ab7171580f3fb93131e9ccfb69fd35364672
Author: Samuel Merritt <sam at swiftstack.com>
Date:   Thu Oct 12 17:15:54 2017 -0700

    functests: don't crash if no second account

    In test.functional.test_object.TestObject.setUp, we create a container
    in account 2. However, if we've only got one account, we don't skip
    this class, resulting in a TypeError down in requests somewhere and a
    stack trace. Since we're using account 2 in setup, we should skip the
    tests if account 2 is not configured.

    Change-Id: I569d98baf071d2dce7cf34a9538070f00afda388

commit eaea0c49332273085b17852e22582940ab059006
Author: Samuel Merritt <sam at swiftstack.com>
Date:   Thu Oct 12 16:58:41 2017 -0700

    Skip cross-account-copy functest if only one account

    This looks like a case of copy-paste-itis. The cross-account-copy
    functest is skipped if we have no test accounts configured, but not if
    we have only one.

    Change-Id: Ifbefdd9aeb98e3d02c536e9d29759f86ec9af6a1

commit dc8da5bb194d0a544fb8065aa9a4c7484f605715
Author: Samuel Merritt <sam at swiftstack.com>
Date:   Thu Oct 12 10:45:12 2017 -0700

    Use "poll" or "selects" Eventlet hub for all Swift daemons.

    Previously, Swift's WSGI servers, the object replicator, and the
    object reconstructor were setting Eventlet's hub to either "poll" or
    "selects", depending on availability. Other daemons were letting
    Eventlet use its default hub, which is "epoll".

    In any daemons that fork, we really don't want to use epoll. Epoll
    instances end up shared between the parent and all children, and you
    get some awful messes when file descriptors are shared.

    Here's an example where two processes are trying to wait on the same
    file descriptor using the same epoll instance, and everything goes
    wrong:

    [proc A] epoll_ctl(6, EPOLL_CTL_ADD, 3, ...) = 0

    [proc B] epoll_ctl(6, EPOLL_CTL_ADD, 3, ...) = -1 EEXIST (File exists)
    [proc B] epoll_wait(6, ...) = 1
    [proc B] epoll_ctl(6, EPOLL_CTL_DEL, 3, ...) = 0

    [proc A] epoll_wait(6, ...)

    This primarily affects the container updater and object updater since
    they fork. I've decided to change the hub for all Swift daemons so
    that we don't add multiprocessing support to some other daemon someday
    and suffer through this same bug again.

    This problem was made more apparent by commit 6d16079, which made our
    logging mutex use file descriptors. However, it could have struck on
    any shared file descriptor on which a read or write returned EAGAIN.

    Change-Id: Ic2c1178ac918c88b0b901e581eb4fab3b2666cfe
    Closes-Bug: 1722951

commit 37f23b072ee7a655ab9072f1d0bece0919dec61b
Author: Samuel Merritt <sam at swiftstack.com>
Date:   Mon Oct 9 17:31:30 2017 -0700

    Allow SLOs to have zero-byte last segments.

    Since we used to allow zero-byte last segments but now we don't, it
    can be difficult to deal with some old SLO manifests.

    Imagine you're writing some code to sync objects from Swift cluster A
    to Swift cluster B. You start off with just a GET from A piped into a
    PUT to B, and that works great until you hit a SLO manifest and B
    won't accept a 500GB object. So, you write some code to detect SLO
    manifests, sync their segments, then take the JSON manifest
    (?multipart-manifest=get) and sync *that* over.

    Now, life is good... until one day you get an exception notification
    that there's this manifest on cluster A that cluster B won't
    accept. Turns out that, back when Swift would take zero-byte final
    segments on SLOs (before commit 7f636a5), someone uploaded such a SLO
    to cluster A. Now, however, zero-byte final segments are invalid, so
    that SLO that exists over in cluster A can't just be copied to cluster
    B.

    A little coding later, your sync tool detects zero-byte final segments
    and removes them when copying a manifest. But now your ETags don't
    match between clusters, so you have to figure out some way to deal
    with that, and so you put it in metadata, but then you realize that
    your syncer might encounter a SLO which contains a sub-SLO which has a
    zero-byte final segment, and it's right about then that you start
    thinking about giving up on programming and getting a job as an
    elevator mechanic.

    This commit makes life easier for developers of such applications by
    allowing SLOs to have zero-byte segments again.

    Change-Id: Ia37880bbb435e269ec53b2963eb1b9121696d479

commit a959d24bf5733650c6005412382fe24c65897c8b
Author: Alistair Coles <alistairncoles at gmail.com>
Date:   Tue Aug 15 12:37:33 2017 +0100

    Document keystone role element in container ACL

    The use of a keystone role name in container ACLs is supported
    and tested. This patch adds documentation.

    [1] https://github.com/openstack/swift/blob/fb3d01a974fb7df8cfadc56ff15bdc04b3c90759/swift/common/middleware/keystoneauth.py#L491-L497
    [2] test.unit.common.middleware.test_keystoneauth.TestAuthorize.test_authorize_succeeds_for_user_role_in_roles

    Change-Id: I77df27393a10f1d8c5a43161fdd4eb08be632566
    Closes-Bug: #1705300

** Tags added: in-feature-s3api

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to swift in Ubuntu.
https://bugs.launchpad.net/bugs/1724342

Title:
  test_drop_privileges AssertionError groups mismatch

Status in Ubuntu Cloud Archive:
  Fix Committed
Status in Ubuntu Cloud Archive pike series:
  Fix Committed
Status in OpenStack Object Storage (swift):
  Fix Released
Status in swift package in Ubuntu:
  Fix Released

Bug description:
  This test is failing in Ubuntu builds on Artful. However I think it
  generally just gets lucky when it is successful.

  ======================================================================
  FAIL: test_drop_privileges (test.unit.common.test_utils.TestUtils)
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "/<<PKGBUILDDIR>>/test/unit/common/test_utils.py", line 2151, in test_drop_privileges
      self.assertEqual(set(groups), set(os.getgroups()))
  AssertionError: Items in the second set but not the first:
  110
      """Fail immediately, with the given message."""
  >>  raise self.failureException('Items in the second set but not the first:\n110')

  It seems that setgroups() is mocked out. Part of what drop_privileges() does is:

      groups = [g.gr_gid for g in grp.getgrall() if user in g.gr_mem]
      os.setgroups(groups)

  The failing part of test_drop_privileges() is:

      groups = [g.gr_gid for g in grp.getgrall() if user in g.gr_mem]
      groups.append(pwd.getpwnam(user).pw_gid)
      self.assertEqual(set(groups), set(os.getgroups()))

  The test is trying to assert that os.getgroups() was set in
  drop_privileges. However os.getgroups() can't change since
  os.setgroups() is mocked out, and os.getgroups() isn't mocked out.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1724342/+subscriptions