[Bug 1978079] Re: EFI pstore not cleared on boot

[Alexander Graf]

Systemd pull request: https://github.com/systemd/systemd/pull/23682

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1978079

Title:
  EFI pstore not cleared on boot

Status in systemd package in Ubuntu:
  New

Bug description:
  Systemd has a systemd-pstore component that scans the pstore on boot
  and if non-empty, takes all previously created dumps, transfers them
  into its journal and removes the pstore elements. This is very
  important on UEFI systems, which only have a limited amount of space
  for variables.

  In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m
  which means the EFI pstore support gets loaded dynamically. In all of
  my boots, this dynamic module loading happened *after* systemd tried
  to check for pstore variables. So systemd-pstore never starts and
  never clears the UEFI variable store. I see this happening in AWS on
  Graviton instances, which eventually run out of space to store the
  dumps. On real hardware, this behavior may lead to unbootable systems.

  ```
  $ systemctl status systemd-pstore
  ○ systemd-pstore.service - Platform Persistent Storage Archival
       Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled)
       Active: inactive (dead)
    Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago
               └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met
         Docs: man:systemd-pstore(8)

  Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in
  Platform Persistent Storage Archival being skipped.

  $ ls -la /sys/fs/pstore
  total 0
  drwxr-x--- 2 root root    0 Jun  9 09:11 .
  drwxr-xr-x 8 root root    0 Jun  9 09:11 ..
  -r--r--r-- 1 root root 1803 Jun  9 09:07 dmesg-efi-165476562001001
  -r--r--r-- 1 root root 1777 Jun  9 09:07 dmesg-efi-165476562002001
  -r--r--r-- 1 root root 1773 Jun  9 09:07 dmesg-efi-165476562003001
  -r--r--r-- 1 root root 1815 Jun  9 09:07 dmesg-efi-165476562004001
  -r--r--r-- 1 root root 1826 Jun  9 09:07 dmesg-efi-165476562005001
  -r--r--r-- 1 root root 1754 Jun  9 09:07 dmesg-efi-165476562006001
  -r--r--r-- 1 root root 1821 Jun  9 09:07 dmesg-efi-165476562007001
  -r--r--r-- 1 root root 1767 Jun  9 09:07 dmesg-efi-165476562008001
  -r--r--r-- 1 root root 1729 Jun  9 09:07 dmesg-efi-165476562009001
  -r--r--r-- 1 root root 1819 Jun  9 09:07 dmesg-efi-165476562010001
  -r--r--r-- 1 root root 1767 Jun  9 09:07 dmesg-efi-165476562011001
  -r--r--r-- 1 root root 1775 Jun  9 09:07 dmesg-efi-165476562012001
  -r--r--r-- 1 root root 1802 Jun  9 09:07 dmesg-efi-165476562013001
  -r--r--r-- 1 root root 1812 Jun  9 09:07 dmesg-efi-165476562014001
  -r--r--r-- 1 root root 1764 Jun  9 09:07 dmesg-efi-165476562015001
  -r--r--r-- 1 root root 1795 Jun  9 09:11 dmesg-efi-165476589801001
  -r--r--r-- 1 root root 1785 Jun  9 09:11 dmesg-efi-165476589802001
  -r--r--r-- 1 root root 1683 Jun  9 09:11 dmesg-efi-165476589803001
  -r--r--r-- 1 root root 1785 Jun  9 09:11 dmesg-efi-165476589804001
  -r--r--r-- 1 root root 1771 Jun  9 09:11 dmesg-efi-165476589805001
  -r--r--r-- 1 root root 1797 Jun  9 09:11 dmesg-efi-165476589806001
  -r--r--r-- 1 root root 1805 Jun  9 09:11 dmesg-efi-165476589807001
  -r--r--r-- 1 root root 1781 Jun  9 09:11 dmesg-efi-165476589808001
  -r--r--r-- 1 root root 1806 Jun  9 09:11 dmesg-efi-165476589809001
  -r--r--r-- 1 root root 1821 Jun  9 09:11 dmesg-efi-165476589810001
  -r--r--r-- 1 root root 1763 Jun  9 09:11 dmesg-efi-165476589811001
  -r--r--r-- 1 root root 1783 Jun  9 09:11 dmesg-efi-165476589812001
  -r--r--r-- 1 root root 1788 Jun  9 09:11 dmesg-efi-165476589813001
  -r--r--r-- 1 root root 1788 Jun  9 09:11 dmesg-efi-165476589814001
  -r--r--r-- 1 root root 1786 Jun  9 09:11 dmesg-efi-165476589815001
  ```

  This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix
  would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always
  available. A long term fix would make systemd rescan the directory
  after all module probing settled.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1978079/+subscriptions