[Bug 1732803] Re: systemd-journald RateLimitBurst is sometimes divided by 4

[Brian Murray]

Hello David, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.2 in a few
hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-xenial to verification-done-xenial. If it does not
fix the bug for you, please add a comment stating that, and change the
tag to verification-failed-xenial. In either case, without details of
your testing we will not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: systemd (Ubuntu Xenial)
       Status: In Progress => Fix Committed

** Tags added: verification-needed verification-needed-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1732803

Title:
  systemd-journald RateLimitBurst is sometimes divided by 4

Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Xenial:
  Fix Committed

Bug description:
  [Impact]

  systemd-journald allows you to configure a per-service journal rate limit in /etc/systemd/journald.conf via the RateLimitBurst parameter. systemd-journald has
  code that effectively increases the rate limit when there is a lot of disk space available.
  However, all versions of systemd before v232 had a bug which would shrink the rate limit
  when there is between 1 and 16 MB available on disk.

  If you designed a service to log at a rate R and configured
  RateLimitBurst to a little above R, this can lead to loss of logs when
  free disk is between 1 and 16 MB, as your service will be surprisingly
  rate limited at lower than your configured rate.

  This bug was fixed upstream in https://github.com/systemd/systemd/pull/4218
  It is a straightforward one-line change that makes the code match the comments under it.

  [Test Case]

  Run a systemd service that prints lots of logs (eg `yes`). Fill your
  disk to have only 1MB full. Use journalctl to see how many log lines
  are between "Suppressed" lines. Note that it is 1/4 of what you'd
  expect.  (Admittedly this test case is a little hard to achieve since
  journald itself is writing to disk. I did run into this in
  production.)

  [Regression Potential]

  This does mean that journald can write slightly more to disk than it
  did before when free disk is between 1 and 16MB, but given that the
  full burst rate is available below 1MB it seems unlikely that any
  systems are depending on this change in order to not break.

  The fix has been in systemd since v232 (shipped in Zesty). I would
  like to see it in Xenial.

  
  [Other Info]
   
  I am seeing this on:
  ubuntu at ip-10-0-2-135[i-0b196ce4b8dc3fc55] 1 ~/systemd-229$ lsb_release -rd
  Description:	Ubuntu 16.04.3 LTS
  Release:	16.04
  ubuntu at ip-10-0-2-135[i-0b196ce4b8dc3fc55] 0 ~/systemd-229$ apt-cache policy systemd
  systemd:
    Installed: 229-4ubuntu19
    Candidate: 229-4ubuntu21
    Version table:
       229-4ubuntu21 500
          500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
   *** 229-4ubuntu19 100
          100 /var/lib/dpkg/status
       229-4ubuntu10 500
          500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
       229-4ubuntu4 500
          500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

  This bug seems to date back to the original implementation of rate
  limiting (https://github.com/systemd/systemd/commit/6e409ce10d).

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1732803/+subscriptions