[Bug 1875493] Re: [core] log rotation doesn't properly restart rsyslogd

[Tony Espy]

@ogra If the component in question was removed and/or deprecated and no
longer used in the current development release, then the policy
shouldn't apply. In this case I don't think it would be much of a
stretch to get a waiver for this. That said, as I'm pretty sure this bug
is specific to Core, so fixing the issue in the core snap directly is
probably the better approach. We can't just leave things "as is", as
this bug can lead to Core devices running out of storage.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/1875493

Title:
  [core] log rotation doesn't properly restart rsyslogd

Status in Snappy:
  New
Status in rsyslog package in Ubuntu:
  New

Bug description:
  One of our commercial partners is developing a device agent snap for
  UC16. One of their engineers just shared the following bug report:

  I wanted to give you a heads up on an issue that we encountered with
  one of our gateways (a Dell 3000) regarding a full disk (/writable).
  It seems as though rsyslog had a bad logrotate config and was holding
  on to deleted files as described here:

   - https://www.claudiokuenzler.com/blog/861/after-ubuntu-update-trusty-xenial-disk-filled-syslog-logrotate
   - https://bugzilla.redhat.com/show_bug.cgi?id=1713358

  Both df and du where showing different results for the “/writable”
  path on the Dell gateway.

  We were able to copy the lsof binary over and found this:

  $ sudo /tmp/lsof | grep deleted
  ...
  rsyslogd 1765 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted)
  rsyslogd 1765 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
  rsyslogd 1765 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted)
  in:imuxso 1765 1776 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted)
  in:imuxso 1765 1776 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
  in:imuxso 1765 1776 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted)
  in:imklog 1765 1777 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted)
  in:imklog 1765 1777 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
  in:imklog 1765 1777 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted)
  rs:main 1765 1778 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted)
  rs:main 1765 1778 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
  rs:main 1765 1778 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted)
  ...

  Restarting the service freed up the deleted files / disk space:

  sudo service rsyslog restart

  The rsyslog config (/etc/logrotate.d/rsyslog) calls this script post
  rotate:

  /usr/lib/rsyslog/rsyslog-rotate

  Which does not actually kill the rsyslog process:

  admin at GR0GP42:~$ ps aux | grep rsyslog
  syslog 2305 0.7 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n
  admin 2464 0.0 0.0 12984 932 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog
  admin at GR0GP42:~$ sudo /usr/lib/rsyslog/rsyslog-rotate
  admin at GR0GP42:~$ ps aux | grep rsyslog
  syslog 2305 0.6 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n
  admin 2469 0.0 0.0 12984 972 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog

  The fix appears to be using the following command: sudo systemctl
  restart rsyslog >/dev/null 2>&1 || true

  admin at GR0GP42:~$ ps aux | grep rsyslog
  syslog 2305 0.6 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n
  admin 2482 0.0 0.0 12984 972 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog
  admin at GR0GP42:~$ sudo systemctl restart rsyslog >/dev/null 2>&1 || true
  admin at GR0GP42:~$ ps aux | grep rsyslog
  syslog 2487 1.0 0.1 262692 3432 ? Ssl 20:36 0:00 /usr/sbin/rsyslogd -n
  admin 2496 0.0 0.0 12984 980 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog

To manage notifications about this bug go to:
https://bugs.launchpad.net/snappy/+bug/1875493/+subscriptions