[Bug 1840348] Re: Sharded OpWQ drops suicide_grace after waiting for work
Dan Hill
1840348 at bugs.launchpad.net
Fri Apr 10 19:15:35 UTC 2020
There are two edge-cases in 12.2.11 where a worker thread's suicide_grace value gets dropped:
[0] In the Threadpool context, Threadpool:worker() drops suicide_grace while waiting on an empty work queue.
[1] In the ShardedThreadpool context, OSD::ShardedOpWQ::_process() drops suicide_grace while opportunistically waiting for more work (to prevent additional lock contention).
The Threadpool context always re-assigns suicide_grace before driving
any work. The ShardedThreadpool context does not follow this pattern.
After delaying to find additional work, the default sharded work queue
timeouts are not re-applied.
This oversight exists in Luminous on-wards. Mimic, and Nautilus have
each reworked the ShardedOpWQ code path, but did not address the
problem.
[0] https://github.com/ceph/ceph/blob/v12.2.11/src/common/WorkQueue.cc#L137
[1] https://github.com/ceph/ceph/blob/v12.2.11/src/osd/OSD.cc#L10476
** Description changed:
- Multiple incidents have been seen where ops were blocked for various
- reasons and the suicide_grace timeout was not observed, meaning that the
- OSD failed to suicide as expected.
+ [Impact]
+ The Sharded OpWQ will opportunistically wait for more work when processing an
+ empty queue. While waiting, the heartbeat timeout and suicide_grace values are
+ modified. On Luminous, the `threadpool_default_timeout` grace is left applied
+ and suicide_grace is left disabled. On later releases both the grace and
+ suicide_grace are left disabled.
+
+ After finding work, the original work queue grace/suicide_grace values are
+ not re-applied. This can result in hung operations that do not trigger an OSD
+ suicide recovery.
+
+ The missing suicide recovery was observed on Luminous 12.2.11. The environment
+ was consistently hitting a known authentication race condition (issue#37778
+ [0]) due to repeated OSD service restarts on a node exhibiting MCEs from a
+ faulty DIMM.
+
+ The auth race condition would stall pg operations. In some cases the hung ops
+ would persist for hours without suicide recovery.
+
+ [Test Case]
+ - In-Progress -
+ Haven't landed on a reliable reproducer. Currently testing the fix by
+ exercising I/O. Since the fix applies to all version of Ceph, the plan is to
+ let this bake in the latest release before considering a back-port.
+
+ [Regression Potential]
+ This fix improves suicide_grace coverage of the Sharded OpWq.
+
+ This change is made in a critical code path that drives client I/O. An OSD
+ suicide will trigger a service restart and repeated restarts (flapping) will
+ adversely impact cluster performance.
+
+ The fix mitigates risk by keeping the applied suicide_grace value consistent
+ with the value applied before entering `OSD::ShardedOpWQ::_process()`. The fix
+ is also restricted to the empty queue edge-case that drops the suicide_grace
+ timeout. The suicide_grace value is only re-applied when work is found after
+ waiting on an empty queue.
+
+ - In-Progress -
+ The fix will bake upstream on later levels before back-port consideration.
** Description changed:
[Impact]
- The Sharded OpWQ will opportunistically wait for more work when processing an
- empty queue. While waiting, the heartbeat timeout and suicide_grace values are
- modified. On Luminous, the `threadpool_default_timeout` grace is left applied
- and suicide_grace is left disabled. On later releases both the grace and
- suicide_grace are left disabled.
+ The Sharded OpWQ will opportunistically wait for more work when processing an empty queue. While waiting, the heartbeat timeout and suicide_grace values are modified. On Luminous, the `threadpool_default_timeout` grace is left applied and suicide_grace is left disabled. On later releases both the grace and suicide_grace are left disabled.
- After finding work, the original work queue grace/suicide_grace values are
- not re-applied. This can result in hung operations that do not trigger an OSD
- suicide recovery.
+ After finding work, the original work queue grace/suicide_grace values
+ are not re-applied. This can result in hung operations that do not
+ trigger an OSD suicide recovery.
- The missing suicide recovery was observed on Luminous 12.2.11. The environment
- was consistently hitting a known authentication race condition (issue#37778
- [0]) due to repeated OSD service restarts on a node exhibiting MCEs from a
- faulty DIMM.
+ The missing suicide recovery was observed on Luminous 12.2.11. The
+ environment was consistently hitting a known authentication race
+ condition (issue#37778 [0]) due to repeated OSD service restarts on a
+ node exhibiting MCEs from a faulty DIMM.
- The auth race condition would stall pg operations. In some cases the hung ops
- would persist for hours without suicide recovery.
+ The auth race condition would stall pg operations. In some cases, the
+ hung ops would persist for hours without suicide recovery.
[Test Case]
- In-Progress -
- Haven't landed on a reliable reproducer. Currently testing the fix by
- exercising I/O. Since the fix applies to all version of Ceph, the plan is to
- let this bake in the latest release before considering a back-port.
+ Haven't landed on a reliable reproducer. Currently testing the fix by exercising I/O. Since the fix applies to all version of Ceph, the plan is to let this bake in the latest release before considering a back-port.
[Regression Potential]
- This fix improves suicide_grace coverage of the Sharded OpWq.
+ This fix improves suicide_grace coverage of the Sharded OpWq.
- This change is made in a critical code path that drives client I/O. An OSD
- suicide will trigger a service restart and repeated restarts (flapping) will
- adversely impact cluster performance.
+ This change is made in a critical code path that drives client I/O. An
+ OSD suicide will trigger a service restart and repeated restarts
+ (flapping) will adversely impact cluster performance.
- The fix mitigates risk by keeping the applied suicide_grace value consistent
- with the value applied before entering `OSD::ShardedOpWQ::_process()`. The fix
- is also restricted to the empty queue edge-case that drops the suicide_grace
- timeout. The suicide_grace value is only re-applied when work is found after
- waiting on an empty queue.
+ The fix mitigates risk by keeping the applied suicide_grace value
+ consistent with the value applied before entering
+ `OSD::ShardedOpWQ::_process()`. The fix is also restricted to the empty
+ queue edge-case that drops the suicide_grace timeout. The suicide_grace
+ value is only re-applied when work is found after waiting on an empty
+ queue.
- In-Progress -
The fix will bake upstream on later levels before back-port consideration.
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1840348
Title:
Sharded OpWQ drops suicide_grace after waiting for work
Status in ceph package in Ubuntu:
Triaged
Bug description:
[Impact]
The Sharded OpWQ will opportunistically wait for more work when processing an empty queue. While waiting, the heartbeat timeout and suicide_grace values are modified. On Luminous, the `threadpool_default_timeout` grace is left applied and suicide_grace is left disabled. On later releases both the grace and suicide_grace are left disabled.
After finding work, the original work queue grace/suicide_grace values
are not re-applied. This can result in hung operations that do not
trigger an OSD suicide recovery.
The missing suicide recovery was observed on Luminous 12.2.11. The
environment was consistently hitting a known authentication race
condition (issue#37778 [0]) due to repeated OSD service restarts on a
node exhibiting MCEs from a faulty DIMM.
The auth race condition would stall pg operations. In some cases, the
hung ops would persist for hours without suicide recovery.
[Test Case]
- In-Progress -
Haven't landed on a reliable reproducer. Currently testing the fix by exercising I/O. Since the fix applies to all version of Ceph, the plan is to let this bake in the latest release before considering a back-port.
[Regression Potential]
This fix improves suicide_grace coverage of the Sharded OpWq.
This change is made in a critical code path that drives client I/O. An
OSD suicide will trigger a service restart and repeated restarts
(flapping) will adversely impact cluster performance.
The fix mitigates risk by keeping the applied suicide_grace value
consistent with the value applied before entering
`OSD::ShardedOpWQ::_process()`. The fix is also restricted to the
empty queue edge-case that drops the suicide_grace timeout. The
suicide_grace value is only re-applied when work is found after
waiting on an empty queue.
- In-Progress -
The fix will bake upstream on later levels before back-port consideration.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1840348/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list