[Bug 1964881] [NEW] Logging/Log rotation does not work for catalina.out
Launchpad Bug Tracker
1964881 at bugs.launchpad.net
Tue Aug 1 20:13:39 UTC 2023
You have been subscribed to a public bug by Ubuntu Foundations Team Bug Bot (crichton):
[Impact]
Log handling in tomcat9 is broken in several ways:
a) logrotate fails to rotate the catalina.out log file
b) rsyslog is configured to chown the catalina.out log file to the
tomcat user, but lacks the privileges to do so (in Ubuntu, rsyslog runs
unprivileged)
c) even though on a fresh install tomcat9 is able to log to
/var/log/tomcat9/catalina.out via rsyslog, a simple upgrade or reinstall
of the tomcat9 package will break that logging by changing the ownership
of catalina.out to the "tomcat9" user, in which case rsyslog won't be
able to write to it anymore (as soon as it closes the fd and tries to
reopen it)
[Test Plan]
Create a container or VM for the ubuntu release under test. Here we will
use lxc, and the commands and outputs below will be shown for jammy:
lxc launch ubuntu:jammy j-tomcat9-logging
lxc shell j-tomcat9-logging
apt update && apt install tomcat9
Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm:
root at j-tomcat9-logging:~# ls -la /var/log/tomcat9
total 12
drwxrws--- 1 tomcat adm 188 Jul 20 18:32 .
drwxrwxr-x 1 root syslog 314 Jul 20 18:32 ..
-rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log
-rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out
-rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log
-rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt
But here the problems start, and these are the ones fixed by this SRU:
a) rsyslog is complaining that it can't change the ownership of
catalina.out:
root at j-tomcat9-logging:~# grep -E "chown.*catalina\.out.*not permitted" /var/log/syslog
Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ]
b) logrotate fails:
root at j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf
error: error opening /var/log/tomcat9/catalina.out: Permission denied
And catalina.out remains unrotated:
root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/
total 12
drwxrws--- 1 tomcat adm 188 Jul 20 18:32 .
drwxrwxr-x 1 root syslog 430 Jul 20 18:33 ..
-rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log
-rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out
-rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log
-rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt
c) if the package is reinstalled, or an update without this fix becomes
available and is applied, the catalina.out file will have incorrect
ownership and rsyslog won't be able to write to it anymore:
before reinstall:
root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
-rw-r----- 1 syslog adm 3523 Jul 20 18:49 catalina.out
after reinstall:
root at j-tomcat9-logging:~# apt install --reinstall tomcat9 -y
Reading package lists... Done
(...)
Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ...
root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
-rw-r----- 1 tomcat adm 3797 Jul 20 18:49 catalina.out
And logging is broken:
root at j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog
Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ]
Now install the tomcat9 package from proposed.
a) rsyslog won't complain anymore about failing to open or chown the file:
root at j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket
root at j-tomcat9-logging:~# > /var/log/syslog
root at j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket
root at j-tomcat9-logging:~# grep rsyslogd /var/log/syslog
Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0]
Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied.
Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ]
Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111
Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104
Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start
b) This time logrotate works, and the catalina.out file will be rotated:
root at j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf
root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/
total 24
drwxrws--- 1 tomcat adm 216 Jul 20 18:39 .
drwxrwxr-x 1 root syslog 612 Jul 20 18:39 ..
-rw-r----- 1 tomcat adm 12487 Jul 20 18:37 catalina.2022-07-20.log
-rw-r----- 1 syslog adm 0 Jul 20 18:39 catalina.out
-rw-r----- 1 syslog adm 7699 Jul 20 18:39 catalina.out.1
-rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log
-rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt
c) reinstalling the package won't break logging again:
root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
-rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out
root at j-tomcat9-logging:~# apt install tomcat9 -y --reinstall
Reading package lists... Done
(...)
Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ...
root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
-rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out
[Where problems could occur]
These logging problems have been ongoing for quite some time, at least
since Focal (20.04), so it's quite possible that users have made local
configuration changes to avoid it. Part of the fix in this SRU is in the
tomcat9.postinst maintainer script, which is difficult for local users
to override, so it's possible that this update will undo, or conflict,
with whatever local fixes were made.
It's hard to predict what it could be, and trying to be smart about it
carries its own set of risks and complexities. I didn't go down that
road, trying to keep the change simple and easy to understand.
[Other Info]
Older logging bug: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881
MP proposing this fix for Kinetic, with some discussion and considerations: https://code.launchpad.net/~ahasenack/ubuntu/+source/tomcat9/+git/tomcat9/+merge/425340
For focal, this SRU is also changing the permissions of /var/log/tomcat9
from 02750 to 02770. In jammy and later, it's 02770 already.
[Original Description]
In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out`
This could be fixed in a newer package but was not backported:
https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881
In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated`
is not able to rotate `/var/log/tomcat/catalina.out`
Because the `catalina.out` is created with `syslog:adm` ownerships.
`syslog` user does not have enough permissions to change this.
This causes following error:
rsyslogd: error during config processing: omfile: chown for file
'/var/log/tomcat9/catalina.out' failed: Operation not permitted
[v8.2112.0 try https://www.rsyslog.com/e/2207 ]
At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm`
directive. Therefore the `logrotated` is not able to truncate the
`/var/log/tomcat/catalina.out`
This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done.
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu79
Architecture: amd64
CasperMD5CheckResult: pass
DistroRelease: Ubuntu 22.04
InstallationDate: Installed on 2022-02-27 (18 days ago)
InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121)
Package: tomcat9 9.0.58-1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12
RebootRequiredPkgs: Error: path contained symlinks.
Tags: jammy
Uname: Linux 5.15.0-18-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
** Affects: tomcat10 (Ubuntu)
Importance: Undecided
Status: New
** Affects: tomcat9 (Ubuntu)
Importance: Undecided
Assignee: Andreas Hasenack (ahasenack)
Status: Fix Released
** Affects: tomcat9 (Ubuntu Focal)
Importance: Undecided
Assignee: Andreas Hasenack (ahasenack)
Status: Fix Released
** Affects: tomcat9 (Ubuntu Jammy)
Importance: Undecided
Assignee: Andreas Hasenack (ahasenack)
Status: Fix Released
** Affects: tomcat9 (Debian)
Importance: Unknown
Status: New
** Tags: apport-collected focal jammy patch server-todo verification-done verification-done-focal verification-done-jammy
--
Logging/Log rotation does not work for catalina.out
https://bugs.launchpad.net/bugs/1964881
You received this bug notification because you are a member of Ubuntu Sponsors, which is subscribed to the bug report.
More information about the Ubuntu-sponsors
mailing list