[Bug 1726930] Re: System fails to start (boot) on battery due to read-only root file-system

Tue Oct 24 17:40:48 UTC 2017

** Also affects: laptop-mode-tools
   Importance: Undecided
       Status: New

** No longer affects: laptop-mode-tools

** Bug watch added: Debian Bug tracker #762018
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762018

** Also affects: systemd (Debian) via
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762018
   Importance: Unknown
       Status: Unknown

** Description changed:

  This report is to capture the extended diagnostic efforts done on IRC
  #ubuntu for user maszlo, who has a Lenovo T450 with SSD that was
  upgraded from 17.04 to 17.10 and since fails to complete start-up of
  services when powered on battery.

  We'd previously applied the "acpi=os=! 'acpi_osi=Windows 2015'"
  workaround in case this was an ACPI DSDT issue.

  This appears to be due to a read-only root file-system. What is not
  clear is how the rootfs goes read-only.

  The file-system (sda6, ext4) is fsck-ed successfully in the initrd
  according to the /run/initramfs/fsck.log.

  From the start-up logs (with "systemd.log_level=debug") we see the ext4
  driver report a remount operation not once but five times.

  $ grep 'EXT4-fs.*sda6' systemd-debug.log
  Oct 23 18:10:44 T450s kernel: EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
  Oct 23 18:10:44 T450s kernel: EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
  Oct 23 18:10:46 T450s kernel: EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro,data=ordered,commit=600
  Oct 23 18:10:47 T450s kernel: EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro,data=ordered,commit=600
  Oct 23 18:10:48 T450s kernel: EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro,data=ordered,commit=600
  Oct 23 18:10:48 T450s kernel: EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro,data=ordered,commit=600
  Oct 23 18:10:49 T450s kernel: EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro,data=ordered,commit=600

  Those last five appear to be carrying options generated by laptop-mode-
  tools.

  User has disabled laptop-mode.service and laptop-mode.timer, and lmt-
  poll.service, but the issue remains.

  We see several reports of "read-only file-system":

  $ grep 'Read-only' systemd-debug.log
  Oct 23 18:10:46 T450s grub-common[1792]: grub-editenv: error: cannot open `/boot/grub/grubenv': Read-only file system.
  Oct 23 18:10:46 T450s systemd[1]: colord.service: Failed to run 'start' task: Read-only file system
  Oct 23 18:10:46 T450s gpu-manager[1797]: Warning: writing to /var/log/gpu-manager.log failed (Read-only file system)
  Oct 23 18:10:46 T450s systemd[1]: systemd-resolved.service: Failed to run 'start' task: Read-only file system
  Oct 23 18:10:46 T450s systemd[1]: systemd-resolved.service: Failed to run 'start' task: Read-only file system
  Oct 23 18:10:46 T450s gdm3[1932]: Failed to create LogDir /var/log/gdm3: Read-only file system
  ...
  Oct 23 18:11:26 T450s cupsd[1461]: Unable to change ownership of "/var/log/cups" - Read-only file system
  Oct 23 18:11:26 T450s cupsd[1461]: Unable to open log file "/var/log/cups/access_log" - Read-only file system
  Oct 23 18:12:52 T450s login[9248]: pam_lastlog(login:session): unable to open /var/log/lastlog: Read-only file system

  We also see several problems with systemd's connection to Dbus:

  $ grep 'Transport endpoint' systemd-debug.log
  Oct 23 18:10:46 T450s systemd[1]: Failed to send job change signal for 773: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: systemd-logind.service: Failed to send unit change signal for systemd-logind.service: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: Failed to send job remove signal for 156: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: lmt-poll.service: Failed to send unit change signal for lmt-poll.service: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: Failed to send job remove signal for 182: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: Failed to send job remove signal for 95: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: polkit.service: Failed to send unit change signal for polkit.service: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: cups-browsed.service: Failed to send unit change signal for cups-browsed.service: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: Failed to send job change signal for 773: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: Failed to send job remove signal for 161: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: gdomap.service: Failed to send unit change signal for gdomap.service: Transport endpoint is not connected

  When the user purged laptop-mode-tools the system starts successfully on
  battery.

  The user installed a fresh instance of 17.10 on a 2nd M2 SSD. When
  laptop-mode-tools was added to that instance it also exhibited the
  failure.

  # apt-cache rdepends laptop-mode-tools
  laptop-mode-tools
  Reverse Depends:
-   systemd
-   tlp
-   powertop-1.13
-   powertop
+   systemd
+   tlp
+   powertop-1.13
+   powertop

- # apt-cache depends systemd
- systemd
-   PreDepends: libc6
-   Depends: libacl1
- ...
-   Breaks: apparmor
-   Breaks: ifupdown
-   Breaks: laptop-mode-tools
-   Breaks: <systemd-shim>
-   Breaks: udev
- ...
+ # apt show systemd | grep Breaks
+ Breaks: apparmor (<< 2.9.2-1), ifupdown (<< 0.8.5~), laptop-mode-tools (<< 1.68~), systemd-shim (<< 10-3~), udev (<< 228-5)

- So it seems that systemd 'knows' it'll break laptop-mode-tools but
- during a release-upgrade it doesn't cause that package to be removed.
+ # apt-cache policy laptop-mode-tools
+ laptop-mode-tools:
+   Installed: (none)
+   Candidate: 1.71-2ubuntu1

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1726930

Title:
  System fails to start (boot) on battery due to read-only root file-
  system

Status in laptop-mode-tools package in Ubuntu:
  New
Status in systemd package in Ubuntu:
  New
Status in systemd package in Debian:
  Unknown

Bug description:
  This report is to capture the extended diagnostic efforts done on IRC
  #ubuntu for user maszlo, who has a Lenovo T450 with SSD that was
  upgraded from 17.04 to 17.10 and since fails to complete start-up of
  services when powered on battery.

  We'd previously applied the "acpi=os=! 'acpi_osi=Windows 2015'"
  workaround in case this was an ACPI DSDT issue.

  This appears to be due to a read-only root file-system. What is not
  clear is how the rootfs goes read-only.

  The file-system (sda6, ext4) is fsck-ed successfully in the initrd
  according to the /run/initramfs/fsck.log.

  From the start-up logs (with "systemd.log_level=debug") we see the
  ext4 driver report a remount operation not once but five times.

  $ grep 'EXT4-fs.*sda6' systemd-debug.log
  Oct 23 18:10:44 T450s kernel: EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
  Oct 23 18:10:44 T450s kernel: EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro
  Oct 23 18:10:46 T450s kernel: EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro,data=ordered,commit=600
  Oct 23 18:10:47 T450s kernel: EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro,data=ordered,commit=600
  Oct 23 18:10:48 T450s kernel: EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro,data=ordered,commit=600
  Oct 23 18:10:48 T450s kernel: EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro,data=ordered,commit=600
  Oct 23 18:10:49 T450s kernel: EXT4-fs (sda6): re-mounted. Opts: errors=remount-ro,data=ordered,commit=600

  Those last five appear to be carrying options generated by laptop-
  mode-tools.

  User has disabled laptop-mode.service and laptop-mode.timer, and lmt-
  poll.service, but the issue remains.

  We see several reports of "read-only file-system":

  $ grep 'Read-only' systemd-debug.log
  Oct 23 18:10:46 T450s grub-common[1792]: grub-editenv: error: cannot open `/boot/grub/grubenv': Read-only file system.
  Oct 23 18:10:46 T450s systemd[1]: colord.service: Failed to run 'start' task: Read-only file system
  Oct 23 18:10:46 T450s gpu-manager[1797]: Warning: writing to /var/log/gpu-manager.log failed (Read-only file system)
  Oct 23 18:10:46 T450s systemd[1]: systemd-resolved.service: Failed to run 'start' task: Read-only file system
  Oct 23 18:10:46 T450s systemd[1]: systemd-resolved.service: Failed to run 'start' task: Read-only file system
  Oct 23 18:10:46 T450s gdm3[1932]: Failed to create LogDir /var/log/gdm3: Read-only file system
  ...
  Oct 23 18:11:26 T450s cupsd[1461]: Unable to change ownership of "/var/log/cups" - Read-only file system
  Oct 23 18:11:26 T450s cupsd[1461]: Unable to open log file "/var/log/cups/access_log" - Read-only file system
  Oct 23 18:12:52 T450s login[9248]: pam_lastlog(login:session): unable to open /var/log/lastlog: Read-only file system

  We also see several problems with systemd's connection to Dbus:

  $ grep 'Transport endpoint' systemd-debug.log
  Oct 23 18:10:46 T450s systemd[1]: Failed to send job change signal for 773: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: systemd-logind.service: Failed to send unit change signal for systemd-logind.service: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: Failed to send job remove signal for 156: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: lmt-poll.service: Failed to send unit change signal for lmt-poll.service: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: Failed to send job remove signal for 182: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: Failed to send job remove signal for 95: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: polkit.service: Failed to send unit change signal for polkit.service: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: cups-browsed.service: Failed to send unit change signal for cups-browsed.service: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: Failed to send job change signal for 773: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: Failed to send job remove signal for 161: Transport endpoint is not connected
  Oct 23 18:10:46 T450s systemd[1]: gdomap.service: Failed to send unit change signal for gdomap.service: Transport endpoint is not connected

  When the user purged laptop-mode-tools the system starts successfully
  on battery.

  The user installed a fresh instance of 17.10 on a 2nd M2 SSD. When
  laptop-mode-tools was added to that instance it also exhibited the
  failure.

  # apt-cache rdepends laptop-mode-tools
  laptop-mode-tools
  Reverse Depends:
    systemd
    tlp
    powertop-1.13
    powertop

  # apt show systemd | grep Breaks
  Breaks: apparmor (<< 2.9.2-1), ifupdown (<< 0.8.5~), laptop-mode-tools (<< 1.68~), systemd-shim (<< 10-3~), udev (<< 228-5)

  # apt-cache policy laptop-mode-tools
  laptop-mode-tools:
    Installed: (none)
    Candidate: 1.71-2ubuntu1

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/laptop-mode-tools/+bug/1726930/+subscriptions