systemd-oomd issues on desktop

[Nick Rosbrook]

On Tue, Jun 14, 2022 at 4:54 AM Lukas Märdian <slyon at ubuntu.com> wrote:
>
> Am 14.06.22 um 01:22 schrieb Nick Rosbrook:
> > On Mon, Jun 13, 2022 at 8:19 AM Lukas Märdian <slyon at ubuntu.com> wrote:
> >> I wonder if we could use a more selective approach, though, using
> >> "OOMScoreAdjust=" in the systemd.exec environment (i.e. Gnome-Shell
> >> launcher in Ubuntu's context, as sd-oomd is currently only enabled on
> >> Ubuntu Desktop) [2], to reduce the probability of certain "important"
> >> apps being killed first, e.g. by maintaining an allow-list of such apps.
> >
> > IIUC, OOMScoreAdjust only influences the behavior of the kernel OOM
> > killer, not systemd-oomd. In any case, I do agree that a "selective
> > approach" might improve the current situation (this is what I tried to
> > convey in option (2)).
>
> Indeed. I misread the man page "This setting also applies to
> systemd-oomd" => This only applies to "OOMPolicy=" and only as of v251
> so it won't help us here. [1]
>
> > The ManagedOOMPreference[1] option exists, but this property is
> > ignored if the unit's cgroup is not owned by root. The rationale for
> > the limitation is described in a link that Michel shared [2]. In
> > short, this option is reserved for "critical services" and should be
> > used sparingly. But, as Steve said earlier, it is likely that the
> > browser is the most important process to many desktop users. Maybe
> > this limitation on setting ManagedOOMPreference could be removed, or
> > eased to at least allow setting `ManagedOOMPreference=avoid` on
> > non-root owned cgroups? For this to be effective, there would need to
> > be other viable kill candidates (i.e. cgroups with greater than 5%
> > swap usage) when the SwapUsedLimit is met.
>
> Yes, I feel like allowing a "ManagedOOMPreference=avoid" for critical UI
> applications, would be a sensible change to make. This should allow
> starting applications like this (possibly integrated with the
> gnome-shell launcher for allow-listed applications):
> $ systemd-run --user -p ManagedOOMPreference=avoid firefox

I have opened an upstream PR to implement this [1], and it seems
upstream is OK with the idea in principle, but some more thinking
needs to be done before it can be merged.

Assuming we can push that change through upstream, service units will
immediately benefit because .service files can configure the
ManagedOOMPreference property. However, applications which are
launched by gnome-shell or snapd run as transient scope units, which
means the ManagedOOMPreference property needs to be set when e.g.
systemd-run is invoked, as demonstrated in the example above. This
means that a bit of integration work will be needed from snapd,
gnome-shell, etc. to set ManagedOOMPreference=avoid on _some_
applications. This immediately raises new questions:

1. Which services and applications should be given a setting of
ManagedOOMPreference=avoid by default?
2. What is the interface to designate such applications? It seems to
me that we would want to have a "single source of truth" from which
gnome-shell, snapd, etc. can determine when ManagedOOMPreference=avoid
should be set.

> Testing option (3) should be as easy as adding a new file in /etc to
> override the "ManagedOOMSwap=" setting, using the current systemd build
> available in Ubuntu Jammy:
>
> $ cat /etc/systemd/system/-.slice.d/10-oomd-root-slice-defaults.conf
>
> [Slice]
>
> ManagedOOMSwap=auto
>

I have been running this configuration on my machine for the last week
or so, and I reported some of my findings on the LP [2]. For anyone
else that has been testing this configuration, please report your
findings on the same LP.

In the longer term, I think that selectively setting
ManagedOOMPreference=avoid on critical applications and services is
desirable. But, given the need to improve the current situation ahead
of 22.04.1, I think our next step should be to disable swap kill by
default (i.e. set ManagedOOMSwap=auto on -.slice). I will continue to
work with upstream on the ManagedOOMPreference changes so that we can
investigate that option in future releases.

Thanks,
Nick

[1] https://github.com/systemd/systemd/pull/23764
[2] https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1972159/comments/18