[Bug 1732803] [NEW] systemd-journald RateLimitBurst is sometimes divided by 4

[Launchpad Bug Tracker]

You have been subscribed to a public bug by David Glasser (glasser):

[Impact]

systemd-journald allows you to configure a per-service journal rate limit in /etc/systemd/journald.conf via the RateLimitBurst parameter. systemd-journald has
code that effectively increases the rate limit when there is a lot of disk space available.
However, all versions of systemd before v232 had a bug which would shrink the rate limit
when there is between 1 and 16 MB available on disk.

If you designed a service to log at a rate R and configured
RateLimitBurst to a little above R, this can lead to loss of logs when
free disk is between 1 and 16 MB, as your service will be surprisingly
rate limited at lower than your configured rate.

This bug was fixed upstream in https://github.com/systemd/systemd/pull/4218
It is a straightforward one-line change that makes the code match the comments under it.

[Test Case]

Run a systemd service that prints lots of logs (eg `yes`). Fill your
disk to have only 1MB full. Use journalctl to see how many log lines are
between "Suppressed" lines. Note that it is 1/4 of what you'd expect.
(Admittedly this test case is a little hard to achieve since journald
itself is writing to disk. I did run into this in production.)

[Regression Potential]

This does mean that journald can write slightly more to disk than it did
before when free disk is between 1 and 16MB, but given that the full
burst rate is available below 1MB it seems unlikely that any systems are
depending on this change in order to not break.

The fix has been in systemd since v232 (shipped in Zesty). I would like
to see it in Xenial.


[Other Info]
 
I am seeing this on:
ubuntu at ip-10-0-2-135[i-0b196ce4b8dc3fc55] 1 ~/systemd-229$ lsb_release -rd
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
ubuntu at ip-10-0-2-135[i-0b196ce4b8dc3fc55] 0 ~/systemd-229$ apt-cache policy systemd
systemd:
  Installed: 229-4ubuntu19
  Candidate: 229-4ubuntu21
  Version table:
     229-4ubuntu21 500
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
 *** 229-4ubuntu19 100
        100 /var/lib/dpkg/status
     229-4ubuntu10 500
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
     229-4ubuntu4 500
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

This bug seems to date back to the original implementation of rate
limiting (https://github.com/systemd/systemd/commit/6e409ce10d).

** Affects: systemd (Ubuntu)
     Importance: Undecided
         Status: New

-- 
systemd-journald RateLimitBurst is sometimes divided by 4
https://bugs.launchpad.net/bugs/1732803
You received this bug notification because you are a member of Ubuntu Sponsors Team, which is subscribed to the bug report.