[Bug 1875493] Re: [core] log rotation doesn't properly restart rsyslogd
Michael Vogt
michael.vogt at canonical.com
Fri May 8 16:08:10 UTC 2020
Thanks for your bugreport.
What happens is that rsyslog:
- has a rsyslog.logrotate config file that has "postrotate" /usr/lib/rsyslog/rsyslog-rotate
- the rsyslog-rotate has "systemctl kill -s HUP rsyslog.service" in it
It looks like neither rsyslog.logrotate nor rsyslog-rotate have changed between both xenial (UC16) and focal (UC20). So this needs some deeper investigation, I don't think we can change the postrotate line for UC16 only without understanding if:
a) does this affect ubuntu classic as well?
b) if not, why is it only affecting core?
We will also then SRU it (and maybe put in the ppa:snappy-dev/image to
get testing in edge). But we need to stay in sync with xenial here.
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/1875493
Title:
[core] log rotation doesn't properly restart rsyslogd
Status in Snappy:
New
Status in rsyslog package in Ubuntu:
New
Bug description:
One of our commercial partners is developing a device agent snap for
UC16. One of their engineers just shared the following bug report:
I wanted to give you a heads up on an issue that we encountered with
one of our gateways (a Dell 3000) regarding a full disk (/writable).
It seems as though rsyslog had a bad logrotate config and was holding
on to deleted files as described here:
- https://www.claudiokuenzler.com/blog/861/after-ubuntu-update-trusty-xenial-disk-filled-syslog-logrotate
- https://bugzilla.redhat.com/show_bug.cgi?id=1713358
Both df and du where showing different results for the “/writable”
path on the Dell gateway.
We were able to copy the lsof binary over and found this:
$ sudo /tmp/lsof | grep deleted
...
rsyslogd 1765 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted)
rsyslogd 1765 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
rsyslogd 1765 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted)
in:imuxso 1765 1776 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted)
in:imuxso 1765 1776 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
in:imuxso 1765 1776 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted)
in:imklog 1765 1777 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted)
in:imklog 1765 1777 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
in:imklog 1765 1777 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted)
rs:main 1765 1778 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted)
rs:main 1765 1778 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted)
rs:main 1765 1778 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted)
...
Restarting the service freed up the deleted files / disk space:
sudo service rsyslog restart
The rsyslog config (/etc/logrotate.d/rsyslog) calls this script post
rotate:
/usr/lib/rsyslog/rsyslog-rotate
Which does not actually kill the rsyslog process:
admin at GR0GP42:~$ ps aux | grep rsyslog
syslog 2305 0.7 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n
admin 2464 0.0 0.0 12984 932 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog
admin at GR0GP42:~$ sudo /usr/lib/rsyslog/rsyslog-rotate
admin at GR0GP42:~$ ps aux | grep rsyslog
syslog 2305 0.6 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n
admin 2469 0.0 0.0 12984 972 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog
The fix appears to be using the following command: sudo systemctl
restart rsyslog >/dev/null 2>&1 || true
admin at GR0GP42:~$ ps aux | grep rsyslog
syslog 2305 0.6 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n
admin 2482 0.0 0.0 12984 972 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog
admin at GR0GP42:~$ sudo systemctl restart rsyslog >/dev/null 2>&1 || true
admin at GR0GP42:~$ ps aux | grep rsyslog
syslog 2487 1.0 0.1 262692 3432 ? Ssl 20:36 0:00 /usr/sbin/rsyslogd -n
admin 2496 0.0 0.0 12984 980 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog
To manage notifications about this bug go to:
https://bugs.launchpad.net/snappy/+bug/1875493/+subscriptions
More information about the foundations-bugs
mailing list