[Bug 1992785] [NEW] BTRFS scrub doesn't seem to rewrite bad sectors

Thu Oct 13 12:23:21 UTC 2022

Public bug reported:

BTRFS, unlike ext4, doesn't accept badblocks lists because modern hardware should remap to a spare block all by itself [1].
The theory is that in a RAID1* configuration "btrfs will rewrite the metadata block from the surviving mirror copy, which will trigger write remapping in the drive".

On Linux pi 5.15.0-1016-raspi #18-Ubuntu SMP PREEMPT Wed Sep 28 12:15:55 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
With a BTRFS filesystem with Data, Metadata, and System in RAID 10 on (mechanical) hard drives, during a scrub, I recently hit bad blocks:

sd 1:0:0:3: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s
sd 1:0:0:3: [sde] tag#0 Sense Key : Medium Error [current] 
sd 1:0:0:3: [sde] tag#0 Add. Sense: Unrecovered read error
sd 1:0:0:3: [sde] tag#0 CDB: Read(16) 88 00 00 00 00 00 13 db 57 00 00 00 02 00 00 00
blk_update_request: critical medium error, dev sde, sector 333141760 op 0x0:(READ) flags 0x4000 phys_seg 64 prio class 0
sd 1:0:0:3: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s
sd 1:0:0:3: [sde] tag#0 Sense Key : Medium Error [current] 
sd 1:0:0:3: [sde] tag#0 Add. Sense: Unrecovered read error
sd 1:0:0:3: [sde] tag#0 CDB: Read(16) 88 00 00 00 00 00 13 db 57 20 00 00 00 08 00 00
blk_update_request: critical medium error, dev sde, sector 333141792 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
BTRFS warning (device sde): i/o error at logical 36578097381376 on dev /dev/sde, physical 170568597504, root 1557, inode 108062, offset 23859200, length 4096, links 1 (path: TIFFs/TIF-ZIP_vol2-JONATHAN/cap_15076.tif)
BTRFS warning (device sde): i/o error at logical 36578097381376 on dev /dev/sde, physical 170568597504, root 3255, inode 108062, offset 23859200, length 4096, links 1 (path: TIFFs/TIF-ZIP_vol2-JONATHAN/cap_15076.tif)
BTRFS error (device sde): bdev /dev/sde errs: wr 0, rd 3, flush 0, corrupt 0, gen 0
BTRFS error (device sde): fixed up error at logical 36578097381376 on dev /dev/sde

Considering the above statement and the logs "fixed up error at logical
xxx on dev /dev/sde", I was expecting that the magic would work and that
the bad blocks would be remapped. But this did not happen, I re-ran the
scrub twice and the errors would still occur at the same place.

So I ended up using hdparm --write-sector on both bad sectors, I ran
scrub again, this time it didn't complain about 'i/o error', only
'checksum error' and it 'fixed up error' error again, but this time for
good, subsequent scrubs pass without problems.

So I think there is an issue with scrub not really rewriting bad blocks.

[1] https://github.com/kdave/btrfs-
progs/issues/293#issuecomment-685198083

** Affects: btrfs-progs (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to btrfs-progs in Ubuntu.
https://bugs.launchpad.net/bugs/1992785

Title:
  BTRFS scrub doesn't seem to rewrite bad sectors

Status in btrfs-progs package in Ubuntu:
  New

Bug description:
  BTRFS, unlike ext4, doesn't accept badblocks lists because modern hardware should remap to a spare block all by itself [1].
  The theory is that in a RAID1* configuration "btrfs will rewrite the metadata block from the surviving mirror copy, which will trigger write remapping in the drive".

  On Linux pi 5.15.0-1016-raspi #18-Ubuntu SMP PREEMPT Wed Sep 28 12:15:55 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
  With a BTRFS filesystem with Data, Metadata, and System in RAID 10 on (mechanical) hard drives, during a scrub, I recently hit bad blocks:

  sd 1:0:0:3: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s
  sd 1:0:0:3: [sde] tag#0 Sense Key : Medium Error [current] 
  sd 1:0:0:3: [sde] tag#0 Add. Sense: Unrecovered read error
  sd 1:0:0:3: [sde] tag#0 CDB: Read(16) 88 00 00 00 00 00 13 db 57 00 00 00 02 00 00 00
  blk_update_request: critical medium error, dev sde, sector 333141760 op 0x0:(READ) flags 0x4000 phys_seg 64 prio class 0
  sd 1:0:0:3: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s
  sd 1:0:0:3: [sde] tag#0 Sense Key : Medium Error [current] 
  sd 1:0:0:3: [sde] tag#0 Add. Sense: Unrecovered read error
  sd 1:0:0:3: [sde] tag#0 CDB: Read(16) 88 00 00 00 00 00 13 db 57 20 00 00 00 08 00 00
  blk_update_request: critical medium error, dev sde, sector 333141792 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
  BTRFS warning (device sde): i/o error at logical 36578097381376 on dev /dev/sde, physical 170568597504, root 1557, inode 108062, offset 23859200, length 4096, links 1 (path: TIFFs/TIF-ZIP_vol2-JONATHAN/cap_15076.tif)
  BTRFS warning (device sde): i/o error at logical 36578097381376 on dev /dev/sde, physical 170568597504, root 3255, inode 108062, offset 23859200, length 4096, links 1 (path: TIFFs/TIF-ZIP_vol2-JONATHAN/cap_15076.tif)
  BTRFS error (device sde): bdev /dev/sde errs: wr 0, rd 3, flush 0, corrupt 0, gen 0
  BTRFS error (device sde): fixed up error at logical 36578097381376 on dev /dev/sde

  Considering the above statement and the logs "fixed up error at
  logical xxx on dev /dev/sde", I was expecting that the magic would
  work and that the bad blocks would be remapped. But this did not
  happen, I re-ran the scrub twice and the errors would still occur at
  the same place.

  So I ended up using hdparm --write-sector on both bad sectors, I ran
  scrub again, this time it didn't complain about 'i/o error', only
  'checksum error' and it 'fixed up error' error again, but this time
  for good, subsequent scrubs pass without problems.

  So I think there is an issue with scrub not really rewriting bad
  blocks.

  [1] https://github.com/kdave/btrfs-
  progs/issues/293#issuecomment-685198083

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/btrfs-progs/+bug/1992785/+subscriptions