[Bug 1871454] [NEW] I/O error while writing superblock on persistent partition with Ubuntu 20.04
Akeo
pete at akeo.ie
Tue Apr 7 17:25:38 UTC 2020
Public bug reported:
The current (2020.04.07) daily build of Ubuntu 20.04 (focal-desktop-
amd64.iso) appears to consistently produce errors on reboot/powerdown
when a partition is in use. Occasionally, power down appears to fail
after these errors are being displayed altogether.
This was tested on 4 different UEFI machines, with a live persistent
media containing the Ubuntu ISO content on a FAT32 first partition, and
a 'casper-rw' or 'writable' labelled ext3 second partition (the label
used made no difference), and the problem (stream of error messages
about trying to access the media beyond the end of the device and
inability to write the persistent partition's superblock) manifested
itself every single time.
Here's an example of the error messages being displayed from a failed
power down, i.e. on a target where the system just freezes during power
down, forcing the user to perform a hard reset:
------------------------------------------------------
[ 34.613315] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 34.613332] blk_update_request: I/O error, dev sda, sector 31277056 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 34.613378] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 34.613391] blk_update_request: I/O error, dev sda, sector 31277056 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 34.613411] Buffer I/O error on dev sda, logical block 3909632, async page read
[ 40.209025] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 40.209051] blk_update_request: I/O error, dev sda, sector 18703953 op 0x1:(WRITE) flags 0x800 phys_seg 4 prio class 0
[ 40.209078] Aborting journal on device sda2-8.
[ 40.209099] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 40.209114] blk_update_request: I/O error, dev sda, sector 18695433 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 40.209137] Buffer I/O error on dev sda2, logical block 262, lost sync page write
[ 40.209200] JBD2: Error -5 detected when updating journal superblock for sda2-8.
[ 40.212828] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 40.212851] blk_update_request: I/O error, dev sda, sector 18694337 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 40.212874] Buffer I/O error on dev sda2, logical block 0, lost sync page write
[ 40.212898] EXT4-fs (sda2): I/O error while writing superblock
[ 40.212912] EXT4-fs error (device sda2): ext4_journal_check_start:61: Detected aborted journal
[ 40.212930] EXT4-fs (sda2): remounting file system read-only
[ 40.212946] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 40.212960] blk_update_request: I/O error, dev sda, sector 18694337 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 40.213025] Buffer I/O error on dev sda2, logical block 0, lost sync page write
[ 40.213042] EXT4-fs (sda2): I/O error while writing superblock
------------------------------------------------------
Is it possible that the devices providing access to the bootable media
and persistent partition have been deleted or replaced on the system
before all accesses have been finished, with the end result being that
/dev/sda[#] *regular files* are being used? The errors above would
certainly be consistent with /dev/sda[#] no longer mapping to an actual
block device, especially as all of the block numbers allegedly residing
beyond the end of the device above are inside the highest possible block
for the device (31277198 in this case).
To replicate the issue for UEFI boot, you can create a bootable media
(e.g. USB Flash Drive) in the following fashion:
------------------------------------------------------
root at nano:/# ## Make sure to change the following disk to your USB media
root at nano:/# export TARGET_DISK=/dev/sda
root at nano:/# ## The following two commands erase the partition tables
root at nano:/# dd if=/dev/zero of=$TARGET_DISK bs=512 count=34
34+0 records in
34+0 records out
17408 bytes (17 kB, 17 KiB) copied, 0.00602524 s, 2.9 MB/s
root at nano:/# dd if=/dev/zero of=$TARGET_DISK bs=512 count=34 seek=$((`blockdev --getsz $TARGET_DISK` - 34))
34+0 records in
34+0 records out
17408 bytes (17 kB, 17 KiB) copied, 0.0249334 s, 698 kB/s
root at nano:/# gdisk $TARGET_DISK
GPT fdisk (gdisk) version 1.0.5
Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present
Creating new GPT entries in memory.
Command (? for help): n
Partition number (1-128, default 1):
First sector (34-31277198, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-31277198, default = 31277198) or {+-}size{KMGTP}: +10G
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 0700
Changed type of partition to 'Microsoft basic data'
Command (? for help): n
Partition number (2-128, default 2):
First sector (34-31277198, default = 20973568) or {+-}size{KMGTP}:
Last sector (20973568-31277198, default = 31277198) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'
Command (? for help): p
Disk /dev/sda: 31277232 sectors, 14.9 GiB
Model: Extreme
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 0E42CF4F-3EB9-420E-A5A1-6AE9E3D746F0
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 31277198
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)
Number Start (sector) End (sector) Size Code Name
1 2048 20973567 10.0 GiB 0700 Microsoft basic data
2 20973568 31277198 4.9 GiB 8300 Linux filesystem
Command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): Y
OK; writing new GUID partition table (GPT) to /dev/sda.
The operation has completed successfully.
root at nano:/# mkfs.vfat ${TARGET_DISK}1
mkfs.fat 4.1 (2017-01-24)
root at nano:/# mkfs.ext3 -L casper-rw ${TARGET_DISK}2
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 1287953 4k blocks and 322560 inodes
Filesystem UUID: b8f3305d-17ab-4344-a611-b2817fe8fcb3
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
root at nano:/# mount -o loop /mnt/ssd/focal-desktop-amd64.iso /mnt/iso
mount: /mnt/iso: WARNING: device write-protected, mounted read-only.
root at nano:/# mount ${TARGET_DISK}1 /mnt/hd
root at nano:/# cp -rT /mnt/iso /mnt/hd
cp: cannot create symbolic link '/mnt/hd/ubuntu': Operation not permitted
cp: cannot create symbolic link '/mnt/hd/dists/stable': Operation not permitted
cp: cannot create symbolic link '/mnt/hd/dists/unstable': Operation not permitted
root at nano:/# ## The following adds the 'persistent' option to 'grub.cfg'
root at nano:/# sed -i 's/\/casper\/vmlinuz\$casper_flavour /\/casper\/vmlinuz\$casper_flavour persistent /g' /mnt/hd/boot/grub/grub.cfg
root at nano:/# umount /mnt/hd
root at nano:/# umount /mnt/iso
root at nano:/# sync
------------------------------------------------------
Booting the media above, verifying that the persistent partition is
mounted and then shutting down or rebooting the platform should produce
the errors.
** Affects: casper (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to casper in Ubuntu.
https://bugs.launchpad.net/bugs/1871454
Title:
I/O error while writing superblock on persistent partition with Ubuntu
20.04
Status in casper package in Ubuntu:
New
Bug description:
The current (2020.04.07) daily build of Ubuntu 20.04 (focal-desktop-
amd64.iso) appears to consistently produce errors on reboot/powerdown
when a partition is in use. Occasionally, power down appears to fail
after these errors are being displayed altogether.
This was tested on 4 different UEFI machines, with a live persistent
media containing the Ubuntu ISO content on a FAT32 first partition,
and a 'casper-rw' or 'writable' labelled ext3 second partition (the
label used made no difference), and the problem (stream of error
messages about trying to access the media beyond the end of the device
and inability to write the persistent partition's superblock)
manifested itself every single time.
Here's an example of the error messages being displayed from a failed
power down, i.e. on a target where the system just freezes during
power down, forcing the user to perform a hard reset:
------------------------------------------------------
[ 34.613315] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 34.613332] blk_update_request: I/O error, dev sda, sector 31277056 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 34.613378] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 34.613391] blk_update_request: I/O error, dev sda, sector 31277056 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 34.613411] Buffer I/O error on dev sda, logical block 3909632, async page read
[ 40.209025] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 40.209051] blk_update_request: I/O error, dev sda, sector 18703953 op 0x1:(WRITE) flags 0x800 phys_seg 4 prio class 0
[ 40.209078] Aborting journal on device sda2-8.
[ 40.209099] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 40.209114] blk_update_request: I/O error, dev sda, sector 18695433 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 40.209137] Buffer I/O error on dev sda2, logical block 262, lost sync page write
[ 40.209200] JBD2: Error -5 detected when updating journal superblock for sda2-8.
[ 40.212828] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 40.212851] blk_update_request: I/O error, dev sda, sector 18694337 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 40.212874] Buffer I/O error on dev sda2, logical block 0, lost sync page write
[ 40.212898] EXT4-fs (sda2): I/O error while writing superblock
[ 40.212912] EXT4-fs error (device sda2): ext4_journal_check_start:61: Detected aborted journal
[ 40.212930] EXT4-fs (sda2): remounting file system read-only
[ 40.212946] sd 3:0:0:0: [sda] tag#0 access beyond end of device
[ 40.212960] blk_update_request: I/O error, dev sda, sector 18694337 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 40.213025] Buffer I/O error on dev sda2, logical block 0, lost sync page write
[ 40.213042] EXT4-fs (sda2): I/O error while writing superblock
------------------------------------------------------
Is it possible that the devices providing access to the bootable media
and persistent partition have been deleted or replaced on the system
before all accesses have been finished, with the end result being that
/dev/sda[#] *regular files* are being used? The errors above would
certainly be consistent with /dev/sda[#] no longer mapping to an
actual block device, especially as all of the block numbers allegedly
residing beyond the end of the device above are inside the highest
possible block for the device (31277198 in this case).
To replicate the issue for UEFI boot, you can create a bootable media
(e.g. USB Flash Drive) in the following fashion:
------------------------------------------------------
root at nano:/# ## Make sure to change the following disk to your USB media
root at nano:/# export TARGET_DISK=/dev/sda
root at nano:/# ## The following two commands erase the partition tables
root at nano:/# dd if=/dev/zero of=$TARGET_DISK bs=512 count=34
34+0 records in
34+0 records out
17408 bytes (17 kB, 17 KiB) copied, 0.00602524 s, 2.9 MB/s
root at nano:/# dd if=/dev/zero of=$TARGET_DISK bs=512 count=34 seek=$((`blockdev --getsz $TARGET_DISK` - 34))
34+0 records in
34+0 records out
17408 bytes (17 kB, 17 KiB) copied, 0.0249334 s, 698 kB/s
root at nano:/# gdisk $TARGET_DISK
GPT fdisk (gdisk) version 1.0.5
Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present
Creating new GPT entries in memory.
Command (? for help): n
Partition number (1-128, default 1):
First sector (34-31277198, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-31277198, default = 31277198) or {+-}size{KMGTP}: +10G
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 0700
Changed type of partition to 'Microsoft basic data'
Command (? for help): n
Partition number (2-128, default 2):
First sector (34-31277198, default = 20973568) or {+-}size{KMGTP}:
Last sector (20973568-31277198, default = 31277198) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'
Command (? for help): p
Disk /dev/sda: 31277232 sectors, 14.9 GiB
Model: Extreme
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 0E42CF4F-3EB9-420E-A5A1-6AE9E3D746F0
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 31277198
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)
Number Start (sector) End (sector) Size Code Name
1 2048 20973567 10.0 GiB 0700 Microsoft basic data
2 20973568 31277198 4.9 GiB 8300 Linux filesystem
Command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): Y
OK; writing new GUID partition table (GPT) to /dev/sda.
The operation has completed successfully.
root at nano:/# mkfs.vfat ${TARGET_DISK}1
mkfs.fat 4.1 (2017-01-24)
root at nano:/# mkfs.ext3 -L casper-rw ${TARGET_DISK}2
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 1287953 4k blocks and 322560 inodes
Filesystem UUID: b8f3305d-17ab-4344-a611-b2817fe8fcb3
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
root at nano:/# mount -o loop /mnt/ssd/focal-desktop-amd64.iso /mnt/iso
mount: /mnt/iso: WARNING: device write-protected, mounted read-only.
root at nano:/# mount ${TARGET_DISK}1 /mnt/hd
root at nano:/# cp -rT /mnt/iso /mnt/hd
cp: cannot create symbolic link '/mnt/hd/ubuntu': Operation not permitted
cp: cannot create symbolic link '/mnt/hd/dists/stable': Operation not permitted
cp: cannot create symbolic link '/mnt/hd/dists/unstable': Operation not permitted
root at nano:/# ## The following adds the 'persistent' option to 'grub.cfg'
root at nano:/# sed -i 's/\/casper\/vmlinuz\$casper_flavour /\/casper\/vmlinuz\$casper_flavour persistent /g' /mnt/hd/boot/grub/grub.cfg
root at nano:/# umount /mnt/hd
root at nano:/# umount /mnt/iso
root at nano:/# sync
------------------------------------------------------
Booting the media above, verifying that the persistent partition is
mounted and then shutting down or rebooting the platform should
produce the errors.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1871454/+subscriptions
More information about the foundations-bugs
mailing list