[Bug 1630516] [NEW] Logrotate doesn't clean old system logs, allowing them to fill the disk
Launchpad Bug Tracker
1630516 at bugs.launchpad.net
Wed Apr 5 07:47:32 UTC 2017
You have been subscribed to a public bug by ChristianEhrhardt (paelzer):
[Impact]
Logrotate fails to rotate a log and then will continue to fail to rotate
it until manual intervention takes place. If messaging has not been
configured on the system there will be no warning issued to the user.
The log will grow day by day until a user intervenes or the drive that
the log is stored on is full.
Very large log files can make it more difficult to find useful data.
Full drives make the rest of the system fail to operate. Backporting a
fix would prevent drives filling up on stable releases.
[Test Case]
Go to your logs area (/var/logs) and create a file with a name ending
.2, as would be created part way through the logrotate process. So if
you have /var/log/syslog, /var/log/syslog.1, /var/log/syslog.2.gz,
/var/log/syslog.3.gz; create a new file named /var/log/syslog.2. Your
subsequent log rotate runs will fail.
[Regression Potential]
- I'd hope the potential is low as it only triggers under certain conditions that are special (target filename in the way).
- So far in those conditions it failed to rotate
- Yet If despite my hope there still manifests an issue I'd expect it could be renaming files it should not, so people would end up "missing" their logs - the good thing is that this is a rename, so they should find it at different names.
- Another thing I consider possible is that some unexpected conditions cause e.g. a crash in the changed code, in that case the logs are not rotated, but since there is no unlink the logs will still exist.
- Therefore I consider the Potential low enough to consider the fix.
[Other Info]
n/a
---
Good afternoon.
I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have
$ logrotate --version
logrotate 3.8.7
bundled in it. A few weeks ago I started getting root emails such as this:
> Subject: Cron <root at warden> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
>
> /etc/cron.daily/logrotate:
> error: error creating output file /var/log/munin/munin-node.log.1: File exists
> run-parts: /etc/cron.daily/logrotate exited with return code 1
When I inspected the area of concern I was able to see that there was an existing .1 file.
manager at warden:/var/log/munin$ ll
total 580
drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./
drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../
-rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log
-rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log
-rw-r--r-- 1 root root 56598 Sep 21 02:01 munin-node.log.1
-rw-r--r-- 1 root root 24576 Aug 31 02:01 munin-node.log.2
-rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz
The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed.
I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week:
> Subject: Cron <root at trac> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
>
> /etc/cron.daily/logrotate:
> error: error creating output file /var/log/syslog.1.gz: File exists
> run-parts: /etc/cron.daily/logrotate exited with return code 1
Different file, different machine but a very similar error message.
Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file.
manager at trac:/var/log$ ll syslog*
-rw-r----- 1 syslog adm 918492 Oct 5 10:30 syslog
-rw-r----- 1 syslog adm 125819 Sep 30 06:25 syslog.1
-rw-r----- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz
-rw-r----- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz
-rw-r----- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz
-rw-r----- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz
-rw-r----- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz
-rw-r----- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz
-rw-r----- 1 syslog adm 39275 Sep 21 06:35 syslog.7.gz
There has been some speculation that a full or nearly full /var
partition would cause this issue. I can confirm that /var is part of /
on my systems and that presently both of them have several gigabytes of
space. I run Munin an Icinga to monitor system state. Neither of these
show / being completely full in the past month. They have both had
/boot fill significantly. Trac had a highest use value of / being
99.28% full in the past year but warden has only had a peak of 33% in
the past year.
A quick search of the internet suggests a couple of other people reporting similar issues:
https://github.com/gitlabhq/gitlabhq/issues/6894
http://raspberrypi.stackexchange.com/questions/22545/why-are-system-logs-not-rotating
I do not believe that I have altered by logrotate configuration files but here is a copy of the ones I know about:
$ cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly
# use the syslog group by default, since this is the owning group
# of /var/log/syslog.
su root syslog
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# uncomment this if you want your log files compressed
#compress
# packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
missingok
monthly
create 0664 root utmp
rotate 1
}
/var/log/btmp {
missingok
monthly
create 0660 root utmp
rotate 1
}
# system-specific logs may be configured here
$ cat /etc/logrotate.d/rsyslog
/var/log/syslog
{
rotate 7
daily
missingok
notifempty
delaycompress
compress
postrotate
reload rsyslog >/dev/null 2>&1 || true
endscript
}
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
reload rsyslog >/dev/null 2>&1 || true
endscript
}
manager at trac:/var/log$ cat /etc/logrotate.d/munin-node
/var/log/munin/munin-node.log {
daily
missingok
rotate 7
compress
copytruncate
notifempty
create 640 root root
}
I am confident that is all the files which are relative to the configuration:
manager at trac:/var/log$ rgrep syslog /etc/logrotate.*
/etc/logrotate.conf:# use the syslog group by default, since this is the owning group
/etc/logrotate.conf:# of /var/log/syslog.
/etc/logrotate.conf:su root syslog
/etc/logrotate.d/ufw: invoke-rc.d rsyslog reload >/dev/null 2>&1 || true
/etc/logrotate.d/rsyslog:/var/log/syslog
/etc/logrotate.d/rsyslog: reload rsyslog >/dev/null 2>&1 || true
/etc/logrotate.d/rsyslog: reload rsyslog >/dev/null 2>&1 || true
manager at trac:/var/log$ rgrep munin /etc/logrotate.*
/etc/logrotate.d/munin-node:/var/log/munin/munin-node.log
If there is anything else I can do to help diagnose the issue please let
me know.
Usual boiler plate bits:
"trac" machine:
$ lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
$ apt-cache policy logrotate
logrotate:
Installed: 3.8.7-1ubuntu1
Candidate: 3.8.7-1ubuntu1
Version table:
*** 3.8.7-1ubuntu1 0
500 http://gb.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
100 /var/lib/dpkg/status
"warden" machine:
$ lsb_release -rd
Description: Ubuntu 16.04.1 LTS
Release: 16.04
$ apt-cache policy logrotate
logrotate:
Installed: 3.8.7-2ubuntu2
Candidate: 3.8.7-2ubuntu2
Version table:
*** 3.8.7-2ubuntu2 500
500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
N: Ignoring file '50unattended-upgrades.ucf-dist' in directory '/etc/apt/apt.conf.d/' as it has an invalid filename extension
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: logrotate 3.8.7-1ubuntu1
ProcVersionSignature: Ubuntu 4.2.0-42.49~14.04.1-generic 4.2.8-ckt12
Uname: Linux 4.2.0-42-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.21
Architecture: amd64
Date: Wed Oct 5 11:14:53 2016
InstallationDate: Installed on 2015-12-14 (295 days ago)
InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805)
SourcePackage: logrotate
UpgradeStatus: No upgrade log present (probably fresh install)
** Affects: hundredpapercuts
Importance: Critical
Status: Triaged
** Affects: logrotate (Ubuntu)
Importance: Critical
Status: Triaged
** Affects: logrotate (Ubuntu Zesty)
Importance: Critical
Status: Triaged
** Affects: logrotate (Debian)
Importance: Unknown
Status: Fix Released
** Tags: amd64 apport-bug trusty xenial yakkety zesty
--
Logrotate doesn't clean old system logs, allowing them to fill the disk
https://bugs.launchpad.net/bugs/1630516
You received this bug notification because you are a member of Ubuntu Sponsors Team, which is subscribed to the bug report.
More information about the Ubuntu-sponsors
mailing list