[Bug 1843085] Re: Backport of zero-length gc chain fixes to Luminous

[James Page]

** Description changed:

  [Impact]
  Cancelling large S3/Swift object puts may result in garbage collection entries with zero-length chains. Rados gateway garbage collection does not efficiently process and clean up these zero-length chains.
  
  A large number of zero-length chains will result in rgw processes
  quickly spinning through the garbage collection lists doing very little
  work. This can result in abnormally high cpu utilization and op
  workloads.
  
  [Test Case]
  Disable garbage collection:
  `juju config ceph-radosgw config-flags='{"rgw": {"rgw enable gc threads": "false"}}'`
  
  Repeatedly kill 256MB object put requests for randomized object names.
  `for i in {0.. 1000}; do f=$(mktemp); fallocate -l 256M $f; s3cmd put $f s3://test_bucket &; pid=$!; sleep $((RANDOM % 3)); kill $pid; rm $f; done`
  
  Capture omap detail. Verify zero-length chains were created:
  `for i in $(seq 0 ${RGW_GC_MAX_OBJS:-32}); do rados -p default.rgw.log --namespace gc listomapvals gc.$i; done`
  
  Raise radosgw debug levels, and enable garbage collection:
  `juju config ceph-radosgw config-flags='{"rgw": {"rgw enable gc threads": "false"}}' loglevel=20`
  
  Verify zero-lenth chains are processed correctly by inspecting radosgw
  logs.
  
  [Regression Potential]
- {Pending} Back-port still needs to be accepted upstream. Need complete fix to assess regression potential.
+ Backport has been accepted into the Luminous release stable branch upstream.
  
  [Other Information]
  This issue has been reported upstream [0] and was fixed in Nautilus alongside a number of other garbage collection issues/enhancements in pr#26601 [1]:
  * adds additional logging to make future debugging easier.
  * resolves bug where the truncated flag was not always set correctly in gc_iterate_entries
  * resolves bug where marker in RGWGC::process was not advanced
  * resolves bug in which gc entries with a zero-length chain were not trimmed
  * resolves bug where same gc entry tag was added to list for deletion multiple times
  
  These fixes were slated for back-port into Luminous and Mimic, but the
  Luminous work was not completed because of a required dependency: AIO GC
  [2]. This dependency has been resolved upstream, and is pending SRU
  verification in Ubuntu packages [3].
  
  [0] https://tracker.ceph.com/issues/38454
  [1] https://github.com/ceph/ceph/pull/26601
  [2] https://tracker.ceph.com/issues/23223
  [3] https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1838858

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1843085

Title:
  Backport of zero-length gc chain fixes to Luminous

Status in Ubuntu Cloud Archive:
  Invalid
Status in Ubuntu Cloud Archive queens series:
  Triaged
Status in Ubuntu Cloud Archive rocky series:
  Fix Released
Status in ceph package in Ubuntu:
  Invalid
Status in ceph source package in Bionic:
  In Progress

Bug description:
  [Impact]
  Cancelling large S3/Swift object puts may result in garbage collection entries with zero-length chains. Rados gateway garbage collection does not efficiently process and clean up these zero-length chains.

  A large number of zero-length chains will result in rgw processes
  quickly spinning through the garbage collection lists doing very
  little work. This can result in abnormally high cpu utilization and op
  workloads.

  [Test Case]
  Disable garbage collection:
  `juju config ceph-radosgw config-flags='{"rgw": {"rgw enable gc threads": "false"}}'`

  Repeatedly kill 256MB object put requests for randomized object names.
  `for i in {0.. 1000}; do f=$(mktemp); fallocate -l 256M $f; s3cmd put $f s3://test_bucket &; pid=$!; sleep $((RANDOM % 3)); kill $pid; rm $f; done`

  Capture omap detail. Verify zero-length chains were created:
  `for i in $(seq 0 ${RGW_GC_MAX_OBJS:-32}); do rados -p default.rgw.log --namespace gc listomapvals gc.$i; done`

  Raise radosgw debug levels, and enable garbage collection:
  `juju config ceph-radosgw config-flags='{"rgw": {"rgw enable gc threads": "false"}}' loglevel=20`

  Verify zero-lenth chains are processed correctly by inspecting radosgw
  logs.

  [Regression Potential]
  Backport has been accepted into the Luminous release stable branch upstream.

  [Other Information]
  This issue has been reported upstream [0] and was fixed in Nautilus alongside a number of other garbage collection issues/enhancements in pr#26601 [1]:
  * adds additional logging to make future debugging easier.
  * resolves bug where the truncated flag was not always set correctly in gc_iterate_entries
  * resolves bug where marker in RGWGC::process was not advanced
  * resolves bug in which gc entries with a zero-length chain were not trimmed
  * resolves bug where same gc entry tag was added to list for deletion multiple times

  These fixes were slated for back-port into Luminous and Mimic, but the
  Luminous work was not completed because of a required dependency: AIO
  GC [2]. This dependency has been resolved upstream, and is pending SRU
  verification in Ubuntu packages [3].

  [0] https://tracker.ceph.com/issues/38454
  [1] https://github.com/ceph/ceph/pull/26601
  [2] https://tracker.ceph.com/issues/23223
  [3] https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1838858

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1843085/+subscriptions