[Bug 2064096] Re: rsyslog service timeout on noble numbat

[Sergio Durigan Junior]

Andreas and I spent some time this afternoon investigating this issue.
Here are our findings.

First, we noticed that the paths being reported by apparmor on dmesg
appear to be relative to /run.  This is just an impression, though: I
believe that, for some reason, apparmor/systemd/something-else is
actually seeing the paths as "/systemd/notify" instead of
"/run/systemd/notify".  Therefore, we decided to try to list those paths
inside the apparmor profile, like:

  /systemd/journal/dev-log     rwkl,
  /systemd/notify      rwkl,

Note that we're using "rwkl" just because we don't want to deal with
limiting the scope of each access.

After adding these paths to /etc/apparmor.d/usr.sbin.rsyslogd and
reloading the profile, the service can finally be (re)started.  This
indicates that there's a discrepancy between the paths seeing by
apparmor/systemd/Linux and those seeing by the userspace application.

With that in mind, our next idea was to try to use "systemd-run" to
mimic what's happening with rsyslogd.  This could help us determine
which component is problematic, but unfortunately we were unable to make
the failure happen.  We tried many combinations of commands; some of
them are listed below:

# Try to "ls" the notify socket using different paths
systemd-run -p AppArmorProfile=rsyslogd ls /run/systemd/notify
systemd-run -p AppArmorProfile=rsyslogd ls /systemd/notify

# Likewise, but running the command using the syslog user
systemd-run --uid 102 -p AppArmorProfile=rsyslogd ls /run/systemd/notify
systemd-run --uid 102 -p AppArmorProfile=rsyslogd ls /systemd/notify

Strangely, "ls" was able to properly list the contents of
/run/systemd/notify on both cases (which it shouldn't, because the
apparmor profile doesn't allow it).  It also reported that
"/systemd/notify", which is correct but unexpected (because we thought
that systemd might be the problematic component which doesn't use "/run"
in the paths).  We also double checked and confirmed that the processes
started by "systemd-run" have "systemd" as their parent, so in theory we
should have seen the same problem here.

There is also the fact that these file accesses are being denied even
when the apparmor profile is running in complain mode.  AFAIU, this
shouldn't happen.  Unless apparmor is affecting the path resolution that
happens when the service tries to connect to the socket, effectively
mangling the final path...  but that would be very weird, I believe.

Either way, it is unclear:

1) Why we're seeing these "partial" paths in the logs.

2) Why these accesses are being denied even when the apparmor profile is
in complain mode.

3) Why "systemd-run" can't seem to reproduce the problem.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/2064096

Title:
  rsyslog service timeout on noble numbat

Status in rsyslog package in Ubuntu:
  Confirmed

Bug description:
  This might be related to #2064088

  The rsyslog service is continually timing out and restarting. If I use
  a service drop-in file and change the 'Type' from 'notify' to
  'simple', the service starts and appears to work normally.

  In the journal, I can see the attached apparmor errors. I can't make
  sense of them, but if it's a similar issue to #2064088, then I suspect
  apparmor is preventing the systemd notify function from alerting
  systemd that the service is up and running.

  ProblemType: Bug
  DistroRelease: Ubuntu 24.04
  Package: rsyslog 8.2312.0-3ubuntu9
  ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1
  Uname: Linux 6.8.0-31-generic x86_64
  ApportVersion: 2.28.1-0ubuntu2
  Architecture: amd64
  CasperMD5CheckMismatches: ./boot/grub/grub.cfg
  CasperMD5CheckResult: fail
  CurrentDesktop: ubuntu:GNOME
  Date: Mon Apr 29 10:37:46 2024
  ProcEnviron:
   LANG=en_GB.UTF-8
   PATH=(custom, no user)
   SHELL=/bin/bash
   TERM=xterm-256color
   XDG_RUNTIME_DIR=<set>
  SourcePackage: rsyslog
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/2064096/+subscriptions