[Bug 1618188] Re: systemd journal should be persistent by default: /var/log/journal should be created; remove rsyslog from default installs

[Sean Ford]

I understand it is not desirable to have duplicate logging, but there is
a corner case where logging that is done during systemd shutdown is lost
because rsyslog is killed. This makes shutdown look broken due to it
being non-deterministic exactly when rsyslog is killed.

Currently, the easiest way to get accurate logging is creating
/var/log/journal.

## Sometimes shutdown logs are brief:

Oct 13 18:51:07 HOST systemd[1]: Stopped target Cloud-init target.
Oct 13 18:51:07 HOST systemd[1]: Starting Unattended Upgrades Shutdown...
Oct 13 18:51:07 HOST systemd[1]: Stopping Session 1 of user ubuntu.
Oct 13 18:51:07 HOST systemd[1]: Stopped target Graphical Interface.
Oct 13 18:51:07 HOST systemd[1]: Stopping Accounts Service...
Oct 13 18:51:07 HOST rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="759" x-info="http://www.rsyslog.com"] exiting on signal 15.

## Sometimes more detailed

Oct 13 18:57:33 HOST systemd[1]: Stopping Session 1 of user ubuntu.
Oct 13 18:57:33 HOST systemd[1]: Stopping User Manager for UID 1000...
Oct 13 18:57:33 HOST systemd[1]: Stopped target Timers.
Oct 13 18:57:33 HOST systemd[1]: Stopped Daily apt activities.
Oct 13 18:57:33 HOST systemd[1277]: Reached target Shutdown.
Oct 13 18:57:33 HOST systemd[1277]: Stopped target Default.
Oct 13 18:57:33 HOST systemd[1277]: Stopped target Basic System.
Oct 13 18:57:33 HOST systemd[1277]: Stopped target Timers.
Oct 13 18:57:33 HOST systemd[1277]: Stopped target Sockets.
Oct 13 18:57:33 HOST systemd[1277]: Starting Exit the Session...
Oct 13 18:57:33 HOST systemd[1]: Stopped Daily Cleanup of Temporary Directories.
Oct 13 18:57:33 HOST systemd[1277]: Stopped target Paths.
Oct 13 18:57:33 HOST systemd[1]: Stopped target Graphical Interface.
Oct 13 18:57:33 HOST systemd[1]: Stopped Timer to automatically refresh installed snaps.
Oct 13 18:57:33 HOST systemd[1]: Stopping ACPI event daemon...
Oct 13 18:57:33 HOST systemd[1]: Starting Unattended Upgrades Shutdown...
Oct 13 18:57:33 HOST systemd[1]: Stopping Accounts Service...
Oct 13 18:57:33 HOST systemd[1]: Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.
Oct 13 18:57:33 HOST systemd[1]: Stopping Virtual machine log manager...
Oct 13 18:57:33 HOST systemd[1]: Stopped target Cloud-init target.
Oct 13 18:57:33 HOST systemd[1277]: Received SIGRTMIN+24 from PID 1592 (kill).
Oct 13 18:57:33 HOST systemd[1]: Stopped Execute cloud user/final scripts.
Oct 13 18:57:33 HOST systemd[1]: Stopped Apply the settings specified in cloud-config.
Oct 13 18:57:33 HOST systemd[1]: Stopped target Cloud-config availability.
Oct 13 18:57:33 HOST rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="737" x-info="http://www.rsyslog.com"] exiting on signal 15.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ubuntu-meta in Ubuntu.
https://bugs.launchpad.net/bugs/1618188

Title:
  systemd journal should be persistent by default: /var/log/journal
  should be created; remove rsyslog from default installs

Status in systemd package in Ubuntu:
  Triaged
Status in ubuntu-meta package in Ubuntu:
  Triaged

Bug description:
  After upgrading 14.04 -> 16.04, key services are now running on
  systemd and using the systemd journal for logging. In 14.04, key
  system logs like /var/log/messages and /var/log/syslog were
  persistent, but after the upgrade to 16.04 there has a been a
  regression of sorts: Logs sent to systemd's journald are now being
  thrown away during reboots.

  This behavior is controlled by the `Storage=` option in
  `/etc/systemd/journald.conf`. The default setting is `Storage=auto`
  which will persist logs in `/var/log/journal/`, *only if the directory
  already exists*. But the directory was not created as part of the
  14.04 -> 16.04 upgrade, so logging was being lost for a while before I
  realized what was happening.

  This issue could be solved by either creating /var/log/journal or
  changing the default Storage behavior to `Storage=persistent`, which
  would create the directory if need be.

  ## Related reference

   * `systemd` currently compounds the issue by having ["journal --disk-usage" report memory usage as disk usage](https://github.com/systemd/systemd/issues/4059), giving the impression that the disk is being used for logging when it isn't. 
   * [User wonders where to find logs from previous boots, unaware that the logs were thrown away](http://askubuntu.com/questions/765315/how-to-find-previous-boot-log-after-ubuntu-16-04-restarts)

  ## Recommended fix

  Restoring persistent logging as the default is recommended.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1618188/+subscriptions