systemd-oomd issues on desktop

[Nick Rosbrook]

Hi,

During the 22.04 cycle, we enabled systemd-oomd [1] by default on
desktop. Since then, there have been reports of systemd-oomd killing
user applications too frequently (e.g. browsers, IDEs, and gnome-shell
in some cases). In addition to a couple of LPs [2][3], I have heard
these reports by word-of-mouth, and there have been discussions on
internal Mattermost. A common theme in these reports is that e.g.
Chrome is killed "suddenly" without any other observable symptoms of
the system nearing OOM.

For more context, systemd-oomd basically has two methods for deciding
a unit's cgroup is a candidate for OOM kill:

1. When total system memory usage _and_ swap usage both exceed
SwapUsedLimit (90% by default, and on Ubuntu) [4], monitored cgoups
with greater than 5% swap usage become OOM kill candidates, and
cgroups with the highest swap usage are acted on first.

2. When a unit's cgroup memory pressure exceeds MemoryPressureLimit
[5] for at least MemoryPressureDurationSec [6], monitored descendant
cgroups will be acted on starting from the ones with the most reclaim
activity to the least reclaim activity.

In the reports I refer to above, applications are being killed due to
(1). In practice, the SwapUsedLimit might be too easy to reach on
Ubuntu, largely because Ubuntu provides just 1GB of swap. Since we
follow the suggestion of setting ManagedOOMSwap=kill on the root slice
[7], every cgroup is eligible for swap kill. When this condition is
met, user applications like browsers are going to be killed first.

While investigating [2], we patched upstream systemd-oomd to fix how
"used memory" was calculated, and we brought the patch into Jammy.
This may have helped the situation a bit, but it does not appear this
was enough to fix the issue entirely.

Given the current situation, I think we should re-consider how
systemd-oomd is configured on Ubuntu. These are the options that come
to mind:

1. Increase SwapUsedLimit (again, currently at 90%). I think this is
probably the safest change, but it is not clear to me how significant
of an impact this would have.

2. Set ManagedOOMSwap more selectively. Again, we currently follow the
recommendation of setting ManagedOOMSwap=kill on the root slice
(-.slice), so every descendant cgroup is a candidate for swap kill. It
_might_ be effective to say "do not swap kill cgroups descendant of
user's app.slice". The downsides of this approach would be that the
configuration does not scale well (i.e. a lot more configuration
needed to get the proper swap kill "coverage"), and this may just
place the problem onto a different class of processes.

3. Do not enable swap kill at all. This would mean systemd-oomd would
only act when memory pressure limits are reached. Given Ubuntu's swap
configuration, does it make sense for systemd-oomd to act on high swap
usage?

4. Increase swap on Ubuntu. I am adding this for completeness, but I
doubt this is a viable option.

I think that either option (1) or (3) would be the most reasonable --
maybe trying (1) first and falling back to (3) if necessary. If anyone
has an opinion on this, or can think of other options, I would
appreciate the input.

Thanks,

Nick 'enr0n' Rosbrook

[1] https://www.freedesktop.org/software/systemd/man/systemd-oomd.service.html
[2] https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1966381
[3] https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1972159
[4] https://www.freedesktop.org/software/systemd/man/oomd.conf.html#SwapUsedLimit=
[5] https://www.freedesktop.org/software/systemd/man/oomd.conf.html#DefaultMemoryPressureLimit=
[6] https://www.freedesktop.org/software/systemd/man/oomd.conf.html#DefaultMemoryPressureDurationSec=
[7] https://www.freedesktop.org/software/systemd/man/systemd-oomd.service.html#Usage%20Recommendations