[Bug 1964881] Re: Logging/Log rotation does not work for catalina.out

Andreas Hasenack 1964881 at bugs.launchpad.net
Thu Jul 21 13:31:29 UTC 2022


** Description changed:

  [Impact]
  
  Log handling in tomcat9 is broken in several ways:
  
  a) logrotate fails to rotate the catalina.out log file
  
  b) rsyslog is configured to chown the catalina.out log file to the
  tomcat user, but lacks the privileges to do so (in Ubuntu, rsyslog runs
  unprivileged)
  
  c) even though on a fresh install tomcat9 is able to log to
  /var/log/tomcat9/catalina.out via rsyslog, a simple upgrade or reinstall
  of the tomcat9 package will break that logging by changing the ownership
  of catalina.out to the "tomcat9" user, in which case rsyslog won't be
  able to write to it anymore (as soon as it closes the fd and tries to
  reopen it)
  
  [Test Plan]
  
  Create a container or VM for the ubuntu release under test. Here we will
  use lxc, and the commands and outputs below will be shown for jammy:
  
  lxc launch ubuntu:jammy j-tomcat9-logging
  
  lxc shell j-tomcat9-logging
  
  apt update && apt install tomcat9
  
  Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm:
  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9
  total 12
  drwxrws--- 1 tomcat adm     188 Jul 20 18:32 .
  drwxrwxr-x 1 root   syslog  314 Jul 20 18:32 ..
  -rw-r----- 1 tomcat adm    5994 Jul 20 18:32 catalina.2022-07-20.log
  -rw-r----- 1 syslog adm    3522 Jul 20 18:32 catalina.out
  -rw-r----- 1 tomcat adm       0 Jul 20 18:32 localhost.2022-07-20.log
  -rw-r----- 1 tomcat adm       0 Jul 20 18:32 localhost_access_log.2022-07-20.txt
  
  But here the problems start, and these are the ones fixed by this SRU:
  
  a) rsyslog is complaining that it can't change the ownership of
  catalina.out:
  
- root at j-tomcat9-logging:~# grep catalina\\.out /var/log/syslog
+ root at j-tomcat9-logging:~# grep -E "chown.*catalina\.out.*not permitted" /var/log/syslog
  Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ]
  
  b) logrotate fails:
  
  root at j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf
  error: error opening /var/log/tomcat9/catalina.out: Permission denied
  
  And catalina.out remains unrotated:
  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/
  total 12
  drwxrws--- 1 tomcat adm     188 Jul 20 18:32 .
  drwxrwxr-x 1 root   syslog  430 Jul 20 18:33 ..
  -rw-r----- 1 tomcat adm    5994 Jul 20 18:32 catalina.2022-07-20.log
  -rw-r----- 1 syslog adm    3522 Jul 20 18:32 catalina.out
  -rw-r----- 1 tomcat adm       0 Jul 20 18:32 localhost.2022-07-20.log
  -rw-r----- 1 tomcat adm       0 Jul 20 18:32 localhost_access_log.2022-07-20.txt
  
  c) if the package is reinstalled, or an update without this fix becomes
  available and is applied, the catalina.out file will have incorrect
  ownership and rsyslog won't be able to write to it anymore:
  
  before reinstall:
  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
  -rw-r----- 1 syslog adm    3523 Jul 20 18:49 catalina.out
  
  after reinstall:
  root at j-tomcat9-logging:~# apt install --reinstall tomcat9 -y
  Reading package lists... Done
  (...)
  Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ...
  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
  -rw-r----- 1 tomcat adm     3797 Jul 20 18:49 catalina.out
  
  And logging is broken:
  root at j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog
  Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ]
  
  Now install the tomcat9 package from proposed.
  
  a) rsyslog won't complain anymore about failing to open or chown the file:
  root at j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket
  root at j-tomcat9-logging:~# > /var/log/syslog
  root at j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket
  root at j-tomcat9-logging:~# grep rsyslogd /var/log/syslog
  Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.2112.0]
  Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied.
  Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ]
  Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111
  Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104
  Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start
  
  b) This time logrotate works, and the catalina.out file will be rotated:
  
  root at j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf
  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/
  total 24
  drwxrws--- 1 tomcat adm      216 Jul 20 18:39 .
  drwxrwxr-x 1 root   syslog   612 Jul 20 18:39 ..
  -rw-r----- 1 tomcat adm    12487 Jul 20 18:37 catalina.2022-07-20.log
  -rw-r----- 1 syslog adm        0 Jul 20 18:39 catalina.out
  -rw-r----- 1 syslog adm     7699 Jul 20 18:39 catalina.out.1
  -rw-r----- 1 tomcat adm        0 Jul 20 18:32 localhost.2022-07-20.log
  -rw-r----- 1 tomcat adm        0 Jul 20 18:32 localhost_access_log.2022-07-20.txt
  
  c) reinstalling the package won't break logging again:
  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
  -rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out
  
  root at j-tomcat9-logging:~# apt install tomcat9 -y --reinstall
  Reading package lists... Done
  (...)
  Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ...
  
  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
  -rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out
  
  [Where problems could occur]
  
  These logging problems have been ongoing for quite some time, at least
  since Focal (20.04), so it's quite possible that users have made local
  configuration changes to avoid it. Part of the fix in this SRU is in the
  tomcat9.postinst maintainer script, which is difficult for local users
  to override, so it's possible that this update will undo, or conflict,
  with whatever local fixes were made.
  
  It's hard to predict what it could be, and trying to be smart about it
  carries its own set of risks and complexities. I didn't go down that
  road, trying to keep the change simple and easy to understand.
  
  [Other Info]
  Older logging bug: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881
  MP proposing this fix for Kinetic, with some discussion and considerations: https://code.launchpad.net/~ahasenack/ubuntu/+source/tomcat9/+git/tomcat9/+merge/425340
  
  [Original Description]
  
  In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out`
  This could be fixed in a newer package but was not backported:
  https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881
  
  In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest)  package, `logrotated`
  is not able to rotate `/var/log/tomcat/catalina.out`
  
  Because the `catalina.out` is created with `syslog:adm` ownerships.
  `syslog` user does not have enough permissions to change this.
  
  This causes following error:
  
  rsyslogd: error during config processing: omfile: chown for file
  '/var/log/tomcat9/catalina.out' failed: Operation not permitted
  [v8.2112.0 try https://www.rsyslog.com/e/2207 ]
  
  At the same time, the  `/etc/logrotate.d/tomcat9` has `su tomcat adm`
  directive. Therefore the `logrotated` is not able to truncate the
  `/var/log/tomcat/catalina.out`
  
  This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done.
  ---
  ProblemType: Bug
  ApportVersion: 2.20.11-0ubuntu79
  Architecture: amd64
  CasperMD5CheckResult: pass
  DistroRelease: Ubuntu 22.04
  InstallationDate: Installed on 2022-02-27 (18 days ago)
  InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121)
  Package: tomcat9 9.0.58-1
  PackageArchitecture: all
  ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12
  RebootRequiredPkgs: Error: path contained symlinks.
  Tags:  jammy
  Uname: Linux 5.15.0-18-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups: N/A
  _MarkForUpload: True

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1964881

Title:
  Logging/Log rotation does not work for catalina.out

Status in tomcat9 package in Ubuntu:
  Fix Released
Status in tomcat9 source package in Focal:
  In Progress
Status in tomcat9 source package in Jammy:
  In Progress
Status in tomcat9 package in Debian:
  New

Bug description:
  [Impact]

  Log handling in tomcat9 is broken in several ways:

  a) logrotate fails to rotate the catalina.out log file

  b) rsyslog is configured to chown the catalina.out log file to the
  tomcat user, but lacks the privileges to do so (in Ubuntu, rsyslog
  runs unprivileged)

  c) even though on a fresh install tomcat9 is able to log to
  /var/log/tomcat9/catalina.out via rsyslog, a simple upgrade or
  reinstall of the tomcat9 package will break that logging by changing
  the ownership of catalina.out to the "tomcat9" user, in which case
  rsyslog won't be able to write to it anymore (as soon as it closes the
  fd and tries to reopen it)

  [Test Plan]

  Create a container or VM for the ubuntu release under test. Here we
  will use lxc, and the commands and outputs below will be shown for
  jammy:

  lxc launch ubuntu:jammy j-tomcat9-logging

  lxc shell j-tomcat9-logging

  apt update && apt install tomcat9

  Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm:
  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9
  total 12
  drwxrws--- 1 tomcat adm     188 Jul 20 18:32 .
  drwxrwxr-x 1 root   syslog  314 Jul 20 18:32 ..
  -rw-r----- 1 tomcat adm    5994 Jul 20 18:32 catalina.2022-07-20.log
  -rw-r----- 1 syslog adm    3522 Jul 20 18:32 catalina.out
  -rw-r----- 1 tomcat adm       0 Jul 20 18:32 localhost.2022-07-20.log
  -rw-r----- 1 tomcat adm       0 Jul 20 18:32 localhost_access_log.2022-07-20.txt

  But here the problems start, and these are the ones fixed by this SRU:

  a) rsyslog is complaining that it can't change the ownership of
  catalina.out:

  root at j-tomcat9-logging:~# grep -E "chown.*catalina\.out.*not permitted" /var/log/syslog
  Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ]

  b) logrotate fails:

  root at j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf
  error: error opening /var/log/tomcat9/catalina.out: Permission denied

  And catalina.out remains unrotated:
  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/
  total 12
  drwxrws--- 1 tomcat adm     188 Jul 20 18:32 .
  drwxrwxr-x 1 root   syslog  430 Jul 20 18:33 ..
  -rw-r----- 1 tomcat adm    5994 Jul 20 18:32 catalina.2022-07-20.log
  -rw-r----- 1 syslog adm    3522 Jul 20 18:32 catalina.out
  -rw-r----- 1 tomcat adm       0 Jul 20 18:32 localhost.2022-07-20.log
  -rw-r----- 1 tomcat adm       0 Jul 20 18:32 localhost_access_log.2022-07-20.txt

  c) if the package is reinstalled, or an update without this fix
  becomes available and is applied, the catalina.out file will have
  incorrect ownership and rsyslog won't be able to write to it anymore:

  before reinstall:
  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
  -rw-r----- 1 syslog adm    3523 Jul 20 18:49 catalina.out

  after reinstall:
  root at j-tomcat9-logging:~# apt install --reinstall tomcat9 -y
  Reading package lists... Done
  (...)
  Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ...
  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
  -rw-r----- 1 tomcat adm     3797 Jul 20 18:49 catalina.out

  And logging is broken:
  root at j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog
  Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ]

  Now install the tomcat9 package from proposed.

  a) rsyslog won't complain anymore about failing to open or chown the file:
  root at j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket
  root at j-tomcat9-logging:~# > /var/log/syslog
  root at j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket
  root at j-tomcat9-logging:~# grep rsyslogd /var/log/syslog
  Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.2112.0]
  Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied.
  Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ]
  Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111
  Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104
  Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start

  b) This time logrotate works, and the catalina.out file will be
  rotated:

  root at j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf
  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/
  total 24
  drwxrws--- 1 tomcat adm      216 Jul 20 18:39 .
  drwxrwxr-x 1 root   syslog   612 Jul 20 18:39 ..
  -rw-r----- 1 tomcat adm    12487 Jul 20 18:37 catalina.2022-07-20.log
  -rw-r----- 1 syslog adm        0 Jul 20 18:39 catalina.out
  -rw-r----- 1 syslog adm     7699 Jul 20 18:39 catalina.out.1
  -rw-r----- 1 tomcat adm        0 Jul 20 18:32 localhost.2022-07-20.log
  -rw-r----- 1 tomcat adm        0 Jul 20 18:32 localhost_access_log.2022-07-20.txt

  c) reinstalling the package won't break logging again:
  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
  -rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out

  root at j-tomcat9-logging:~# apt install tomcat9 -y --reinstall
  Reading package lists... Done
  (...)
  Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ...

  root at j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
  -rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out

  [Where problems could occur]

  These logging problems have been ongoing for quite some time, at least
  since Focal (20.04), so it's quite possible that users have made local
  configuration changes to avoid it. Part of the fix in this SRU is in
  the tomcat9.postinst maintainer script, which is difficult for local
  users to override, so it's possible that this update will undo, or
  conflict, with whatever local fixes were made.

  It's hard to predict what it could be, and trying to be smart about it
  carries its own set of risks and complexities. I didn't go down that
  road, trying to keep the change simple and easy to understand.

  [Other Info]
  Older logging bug: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881
  MP proposing this fix for Kinetic, with some discussion and considerations: https://code.launchpad.net/~ahasenack/ubuntu/+source/tomcat9/+git/tomcat9/+merge/425340

  [Original Description]

  In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out`
  This could be fixed in a newer package but was not backported:
  https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881

  In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest)  package,
  `logrotated` is not able to rotate `/var/log/tomcat/catalina.out`

  Because the `catalina.out` is created with `syslog:adm` ownerships.
  `syslog` user does not have enough permissions to change this.

  This causes following error:

  rsyslogd: error during config processing: omfile: chown for file
  '/var/log/tomcat9/catalina.out' failed: Operation not permitted
  [v8.2112.0 try https://www.rsyslog.com/e/2207 ]

  At the same time, the  `/etc/logrotate.d/tomcat9` has `su tomcat adm`
  directive. Therefore the `logrotated` is not able to truncate the
  `/var/log/tomcat/catalina.out`

  This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done.
  ---
  ProblemType: Bug
  ApportVersion: 2.20.11-0ubuntu79
  Architecture: amd64
  CasperMD5CheckResult: pass
  DistroRelease: Ubuntu 22.04
  InstallationDate: Installed on 2022-02-27 (18 days ago)
  InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121)
  Package: tomcat9 9.0.58-1
  PackageArchitecture: all
  ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12
  RebootRequiredPkgs: Error: path contained symlinks.
  Tags:  jammy
  Uname: Linux 5.15.0-18-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups: N/A
  _MarkForUpload: True

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1964881/+subscriptions




More information about the Ubuntu-sponsors mailing list