[Bug 2036467] [NEW] Resizing cloud-images occasionally fails due to superblock checksum mismatch in resize2fs

[Launchpad Bug Tracker]

*** This bug is a security vulnerability ***

You have been subscribed to a public security bug by Heitor Alves de Siqueira (halves):

[Impact]

This is a long running bug plaguing cloud-images, where on a rare
occasion resize2fs would fail and the image would not resize to fit the
entire disk.

Online resizes would fail due to a superblock checksum mismatch, where
the superblock in memory differs from what is currently on disk due to
changes made to the image.

$ resize2fs /dev/nvme1n1p1
resize2fs 1.47.0 (5-Feb-2023)
resize2fs: Superblock checksum does not match superblock while trying to open /dev/nvme1n1p1
Couldn't find valid filesystem superblock.

Changing the read of the superblock to Direct I/O solves the issue.

[Testcase]

Start an c5.large instance on AWS, and attach a 60gb gp3 volume for use
as a scratch disk.

Run the following script, courtesy of Krister Johansen and his team:

   #!/usr/bin/bash
   set -euxo pipefail

   while true
   do
           parted /dev/nvme1n1 mklabel gpt mkpart primary 2048s 2099200s
           sleep .5
           mkfs.ext4 /dev/nvme1n1p1
           mount -t ext4 /dev/nvme1n1p1 /mnt
           stress-ng --temp-path /mnt -D 4 &
           STRESS_PID=$!
           sleep 1
           growpart /dev/nvme1n1 1
           resize2fs /dev/nvme1n1p1
           kill $STRESS_PID
           wait $STRESS_PID
           umount /mnt
           wipefs -a /dev/nvme1n1p1
           wipefs -a /dev/nvme1n1
   done

Test packages are available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/lp2036467-test

If you install the test packages, the race no longer occurs.

[Where problems could occur]

We are changing how resize2fs reads the superblock from underlying
disks.

If a regression were to occur, resize2fs could fail to resize offline or
online volumes. As all cloud-images are online resized during their
initial boot, this could have a large impact to public and private
clouds should a regression occur.

[Other info]

Upstream mailing list discussion:
https://lore.kernel.org/linux-ext4/20230605225221.GA5737@templeofstupid.com/
https://lore.kernel.org/linux-ext4/20230609042239.GA1436857@mit.edu/

This was fixed in the below commit upstream:

commit 43a498e938887956f393b5e45ea6ac79cc5f4b84
Author: Theodore Ts'o <tytso at mit.edu>
Date: Thu, 15 Jun 2023 00:17:01 -0400
Subject: resize2fs: use Direct I/O when reading the superblock for
 online resizes
Link: https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/commit/?id=43a498e938887956f393b5e45ea6ac79cc5f4b84

The commit has not been tagged to any release. All supported Ubuntu
releases require this fix, and need to be published in standard non-ESM
archives to be picked up in cloud images.

** Affects: cloud-images
     Importance: Critical
         Status: New

** Affects: e2fsprogs (Ubuntu)
     Importance: Critical
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress

** Affects: e2fsprogs (Ubuntu Trusty)
     Importance: Critical
     Assignee: Matthew Ruffell (mruffell)
         Status: Won't Fix

** Affects: e2fsprogs (Ubuntu Xenial)
     Importance: Critical
     Assignee: Matthew Ruffell (mruffell)
         Status: Won't Fix

** Affects: e2fsprogs (Ubuntu Bionic)
     Importance: Critical
     Assignee: Matthew Ruffell (mruffell)
         Status: Won't Fix

** Affects: e2fsprogs (Ubuntu Focal)
     Importance: Critical
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress

** Affects: e2fsprogs (Ubuntu Jammy)
     Importance: Critical
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress

** Affects: e2fsprogs (Ubuntu Lunar)
     Importance: Critical
     Assignee: Matthew Ruffell (mruffell)
         Status: Won't Fix

** Affects: e2fsprogs (Ubuntu Mantic)
     Importance: Critical
     Assignee: Matthew Ruffell (mruffell)
         Status: In Progress


** Tags: foundations-todo patch patch-accepted-upstream sts
-- 
Resizing cloud-images occasionally fails due to superblock checksum mismatch in resize2fs
https://bugs.launchpad.net/bugs/2036467
You received this bug notification because you are a member of Ubuntu Sponsors, which is subscribed to the bug report.