systemd-oomd issues on desktop

[Dan Streetman]

On Thu, Jun 9, 2022 at 3:03 PM Nick Rosbrook
<nick.rosbrook at canonical.com> wrote:
>
> Hi,
>
> During the 22.04 cycle, we enabled systemd-oomd [1] by default on
> desktop. Since then, there have been reports of systemd-oomd killing
> user applications too frequently (e.g. browsers, IDEs, and gnome-shell
> in some cases). In addition to a couple of LPs [2][3], I have heard
> these reports by word-of-mouth, and there have been discussions on
> internal Mattermost. A common theme in these reports is that e.g.
> Chrome is killed "suddenly" without any other observable symptoms of
> the system nearing OOM.
>
> For more context, systemd-oomd basically has two methods for deciding
> a unit's cgroup is a candidate for OOM kill:
>
> 1. When total system memory usage _and_ swap usage both exceed
> SwapUsedLimit (90% by default, and on Ubuntu) [4], monitored cgoups
> with greater than 5% swap usage become OOM kill candidates, and
> cgroups with the highest swap usage are acted on first.
>
> 2. When a unit's cgroup memory pressure exceeds MemoryPressureLimit
> [5] for at least MemoryPressureDurationSec [6], monitored descendant
> cgroups will be acted on starting from the ones with the most reclaim
> activity to the least reclaim activity.
>
> In the reports I refer to above, applications are being killed due to
> (1). In practice, the SwapUsedLimit might be too easy to reach on
> Ubuntu, largely because Ubuntu provides just 1GB of swap. Since we
> follow the suggestion of setting ManagedOOMSwap=kill on the root slice
> [7], every cgroup is eligible for swap kill. When this condition is
> met, user applications like browsers are going to be killed first.
>
> While investigating [2], we patched upstream systemd-oomd to fix how
> "used memory" was calculated, and we brought the patch into Jammy.
> This may have helped the situation a bit, but it does not appear this
> was enough to fix the issue entirely.
>
> Given the current situation, I think we should re-consider how
> systemd-oomd is configured on Ubuntu. These are the options that come
> to mind:
>
> 1. Increase SwapUsedLimit (again, currently at 90%). I think this is
> probably the safest change, but it is not clear to me how significant
> of an impact this would have.
>
> 2. Set ManagedOOMSwap more selectively. Again, we currently follow the
> recommendation of setting ManagedOOMSwap=kill on the root slice
> (-.slice), so every descendant cgroup is a candidate for swap kill. It
> _might_ be effective to say "do not swap kill cgroups descendant of
> user's app.slice". The downsides of this approach would be that the
> configuration does not scale well (i.e. a lot more configuration
> needed to get the proper swap kill "coverage"), and this may just
> place the problem onto a different class of processes.
>
> 3. Do not enable swap kill at all. This would mean systemd-oomd would
> only act when memory pressure limits are reached. Given Ubuntu's swap
> configuration, does it make sense for systemd-oomd to act on high swap
> usage?
>
> 4. Increase swap on Ubuntu. I am adding this for completeness, but I
> doubt this is a viable option.

Personally, I think this is the correct option. 1GB is not a good
default swap size.

>
> I think that either option (1) or (3) would be the most reasonable --
> maybe trying (1) first and falling back to (3) if necessary. If anyone
> has an opinion on this, or can think of other options, I would
> appreciate the input.

Was systemd-oomd enabled by default for a specific reason? The kernel
is quite able to handle oom situations itself, and has been for years,
so while I'm not trying to suggest systemd-oomd is without any use
case, I'm skeptical that systemd-oomd should be enabled *by default*.
I think it's more likely to behave better when enabled to address a
specific system use case, and leave the default behavior of handling
oom to the kernel.

>
> Thanks,
>
> Nick 'enr0n' Rosbrook
>
> [1] https://www.freedesktop.org/software/systemd/man/systemd-oomd.service.html
> [2] https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1966381
> [3] https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1972159
> [4] https://www.freedesktop.org/software/systemd/man/oomd.conf.html#SwapUsedLimit=
> [5] https://www.freedesktop.org/software/systemd/man/oomd.conf.html#DefaultMemoryPressureLimit=
> [6] https://www.freedesktop.org/software/systemd/man/oomd.conf.html#DefaultMemoryPressureDurationSec=
> [7] https://www.freedesktop.org/software/systemd/man/systemd-oomd.service.html#Usage%20Recommendations
>
> --
> ubuntu-devel mailing list
> ubuntu-devel at lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel