[Bug 1761630] Re: journald takes over /dev/log in bionic, impacts usability of syslog querying

[Dimitri John Ledkov]

Looking at systemd-journald-dev-log.socket, it creates
/run/systemd/journal/dev-log socket, and that unit specifies Symlink to
take over /dev/log. And has been doing so since at least xenial (check
in a LXD container).

In journald.conf it has set #ForwardToSyslog=yes, which forwards to this
socket:

$ sudo fuser -v /run/systemd/journal/syslog
                     USER        PID ACCESS COMMAND
/run/systemd/journal/syslog:
                     root          1 F.... systemd
                     syslog     2023 F.... rsyslogd

Which is owned by rsyslogd. Hence, e.g. my /var/log/syslog is populated.

I see that most things configured in /etc/rsyslog.d/50-default.conf, are
not getting populated... which is not good:

/var/log/auth.log
/var/log/kern.log
/var/log/mail.log
/var/log/mail.err

All appear to be empty. I guess i need to compare this with Trusty,
figure out what's wrong, add an autopkgtest which tests for this, and
tests things.

All entries have PRIORITY, SYSLOG_FACILITY, SYSLOG_IDENTIFIER set...
since journal must set that upon message arrival, in case it will
forward these to the real syslog. To view thing that arrived via syslog
protocol, one should filter on the _TRANSPORT field instead.

E.g. The below command shows the syslog stuff
$ journalctl _TRANSPORT=syslog

BTW. All options for _TRANSPORT field are shown below
$ journalctl -F _TRANSPORT
stdout
syslog
journal
driver
audit
kernel

One can filter on severity levels, using syslog keywords for them, e.g.
'crit' using the -p option.

$ journalctl -p crit

On my machine that prints a lot of stuff from consolekit.... no idea why
i still have consolekit installed. Maybe we should be force purging it
these days on upgrades?

It is sad that one cannot specify logical facility names, as listed in
syslog(3) under Values for facility LOG_AUTH..LOG_UUCP.

I'm not sure if we are allowed to forward audit & kernel messages.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/1761630

Title:
  journald takes over /dev/log in bionic, impacts usability of syslog
  querying

Status in rsyslog package in Ubuntu:
  Confirmed
Status in systemd package in Ubuntu:
  Confirmed
Status in rsyslog source package in Bionic:
  Confirmed
Status in systemd source package in Bionic:
  Confirmed

Bug description:
  In bionic, /dev/log is now owned by systemd-journald.

  $ sudo fuser -v /dev/log 
                       USER        PID ACCESS COMMAND
  /run/systemd/journal/dev-log:
                       root          1 F.... systemd
                       root        628 F.... systemd-journal
  $

  This is ok; there are good reasons to want to do this.  (Timestamp
  precision that's not dependent on the syslog protocol; logging
  available earlier in boot and later at shutdown; unified queriability
  of logs around services.)

  The logfiles previously populated by rsyslog under /var/log are now
  empty (no longer being updated, and the previous files logrotated
  away):

  $ ls -l /var/log/auth.log
  -rw-r----- 1 syslog adm 0 Mar 25 00:06 /var/log/auth.log
  $

  This is also ok, we don't really want log data duplicated in two
  places on the system; so since we now have persistent journal under
  /var/log/journal, we don't want to also create these plaintext files
  by default.

  However, it's not clear that this is by design, rather than by
  accident.  rsyslog is still /configured/ to log to these files; it
  just isn't receiving any data because it no longer controls the
  socket.

  $ sudo lsof -p 852|grep DGRAM
  rsyslogd 852 syslog    3u  unix 0xffff8e5680435000      0t0        351 /run/systemd/journal/syslog type=DGRAM
  $

  Dimitri and I have discussed that rsyslog should continue to function,
  so that users who have remote syslogging configured can still make use
  of this.  It looks like currently this is not the case, because
  journald is not forwarding to rsyslog.

  That is not ok.

  What is also not ok is that, now that logs are only being written to
  the journal by default instead of to syslog files, querying these logs
  by syslog priority only works if you know the syslog facility by
  number (and, afaics, you can only filter by one syslog facility at a
  time).  So whereas before, you could find mail logs by looking in
  /var/log/mail.log by default, you now have to know to run 'journalctl
  SYSLOG_FACILITY=2'; and if you want a view of what was previously in
  /var/log/syslog (i.e. filtering out non-syslog systemd logs, it seems
  the most compact way to express this is 'journalctl --system
  SYSLOG_FACILITY={0,1,2,3,5,6,7,8,9,11,12,13,14,15,16,17,18,19,20,21,22,23}'.

  This is really not good.  Admins have never needed to know the mapping
  of symbolic names to facility numbers to work with syslog; this throws
  away 30 years of convention with log handling.

  If we are going to store the logs in the journal by default, I think
  there needs to be a way to query the logs using symbolic names
  consistent with syslog(3) and /etc/rsyslog.d/50-default.conf.

  I don't think we can release with things in the current state.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/1761630/+subscriptions