[Bug 1875493] Re: [core] log rotation doesn't properly restart rsyslogd

[Tony Espy]

@ogra This is a bug in the core[16] snap. I'm not sure why we'd even
consider SRU'ing to UC18 or UC20. On both of the those systems, core
wouldn't be the booting snap, so any fix would be a no-op.

Possible ways to fix include:

 - releasing an rsyslog SRU for 16.04, but as Michael points out this
could impact classic

OR

 - patch rsyslog in the core snap to do the right thing

I'll spend more time on this as soon as I have some more cycles and will
try to determine the exact cause of the restart failure.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/1875493

Title:
  [core] log rotation doesn't properly restart rsyslogd

Status in Snappy:
  New
Status in rsyslog package in Ubuntu:
  New

Bug description:
  One of our commercial partners is developing a device agent snap for
  UC16. One of their engineers just shared the following bug report:

  I wanted to give you a heads up on an issue that we encountered with
  one of our gateways (a Dell 3000) regarding a full disk (/writable).
  It seems as though rsyslog had a bad logrotate config and was holding
  on to deleted files as described here:

   - https://www.claudiokuenzler.com/blog/861/after-ubuntu-update-trusty-xenial-disk-filled-syslog-logrotate
   - https://bugzilla.redhat.com/show_bug.cgi?id=1713358

  Both df and du where showing different results for the “/writable”
  path on the Dell gateway.

  We were able to copy the lsof binary over and found this:

  $ sudo /tmp/lsof | grep deleted
  ...
  rsyslogd 1765 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted)
  rsyslogd 1765 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
  rsyslogd 1765 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted)
  in:imuxso 1765 1776 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted)
  in:imuxso 1765 1776 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
  in:imuxso 1765 1776 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted)
  in:imklog 1765 1777 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted)
  in:imklog 1765 1777 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
  in:imklog 1765 1777 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted)
  rs:main 1765 1778 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted)
  rs:main 1765 1778 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
  rs:main 1765 1778 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted)
  ...

  Restarting the service freed up the deleted files / disk space:

  sudo service rsyslog restart

  The rsyslog config (/etc/logrotate.d/rsyslog) calls this script post
  rotate:

  /usr/lib/rsyslog/rsyslog-rotate

  Which does not actually kill the rsyslog process:

  admin at GR0GP42:~$ ps aux | grep rsyslog
  syslog 2305 0.7 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n
  admin 2464 0.0 0.0 12984 932 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog
  admin at GR0GP42:~$ sudo /usr/lib/rsyslog/rsyslog-rotate
  admin at GR0GP42:~$ ps aux | grep rsyslog
  syslog 2305 0.6 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n
  admin 2469 0.0 0.0 12984 972 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog

  The fix appears to be using the following command: sudo systemctl
  restart rsyslog >/dev/null 2>&1 || true

  admin at GR0GP42:~$ ps aux | grep rsyslog
  syslog 2305 0.6 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n
  admin 2482 0.0 0.0 12984 972 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog
  admin at GR0GP42:~$ sudo systemctl restart rsyslog >/dev/null 2>&1 || true
  admin at GR0GP42:~$ ps aux | grep rsyslog
  syslog 2487 1.0 0.1 262692 3432 ? Ssl 20:36 0:00 /usr/sbin/rsyslogd -n
  admin 2496 0.0 0.0 12984 980 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog

To manage notifications about this bug go to:
https://bugs.launchpad.net/snappy/+bug/1875493/+subscriptions