ACK/Cmnt: [SRU][J][PATCH 0/3] bcache: fix deadlock

[Stefan Bader]

On 29.07.22 13:58, Nikhil Kshirsagar wrote:
> BugLink: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980925

Beside this ^ being needed in all 3 patches (and it should use the standard 
format of https://bugs.launchpad.net/bugs/1980925) all 3 SHA1 references are 
incorrect. Those *should* be:

622536443b6731ec82c563aae7807165adbe9178
  "bcache: improve multithreaded bch_btree_check()"
4dc34ae1b45fe26e772a44379f936c72623dd407
  "bcache: improve multithreaded bch_sectors_dirty_init()"
7d6b902ea0e02b2a25c480edf471cbaa4ebe6b3c
  "bcache: memset on stack variables in bch_btree_check() and
   bch_sectors_dirty_init()"

Also this is for now submitted to jammy only, but the report has a nomination 
for Focal. But both patch #1 and #2 refer to v5.7 patches for introducing the 
issue they fix and they did not get backported into v5.4[.y] as of 5.4.208. The 
fixes are included in 5.19 so Kinetic is neither affected. I marked both as invalid.

For Jammy, provided the SHA1 and BugLink will be fixed when applying:

Acked-by: Stefan Bader <stefan.bader at canonical.com>

> 
> [Impact]
> 
> When Random Read I/O is started with a test like -
> 
> fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0
> 
> or
> 
> random read-writes with a test like,
> 
> fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0
> 
> traces are seen in the kernel log,
> 
> [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds.
> [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445
> [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 4474.731391] task:bcache_writebac state:D stack: 0 pid: 1835 ppid: 2 flags:0x00004000
> [ 4474.731408] Call Trace:
> [ 4474.731411] <TASK>
> [ 4474.731413] __schedule+0x23d/0x5a0
> [ 4474.731433] schedule+0x4e/0xb0
> [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0
> [ 4474.731441] down_write+0x43/0x50
> [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache]
> [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache]
> [ 4474.731487] kthread+0x12a/0x150
> [ 4474.731491] ? set_kthread_struct+0x50/0x50
> [ 4474.731494] ret_from_fork+0x22/0x30
> [ 4474.731499] </TASK>
> 
> The bug exists till kernel 5.15.50-051550-generic
> 
> The reproducer is pasted below:
> 
> # uname -a
> Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
> 
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
> sdd 8:48 0 279.4G 0 disk
> └─sdd1 8:49 0 60G 0 part
>    └─bcache0 252:0 0 60G 0 disk /home/ubuntu/bcache_mount
> nvme0n1 259:0 0 372.6G 0 disk
> └─nvme0n1p1 259:2 0 15G 0 part
>    └─bcache0 252:0 0 60G 0 disk /home/ubuntu/bcache_mount
> 
> fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1
> read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
> fio-3.28
> Starting 1 process
> read_iops: Laying out IO file (1 file / 12288MiB)
> 
> The test does not progress beyond a few minutes, and this trace is then seen in the kernel log,
> 
> [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds.
> [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445
> [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 4474.731391] task:bcache_writebac state:D stack: 0 pid: 1835 ppid: 2 flags:0x00004000
> [ 4474.731408] Call Trace:
> [ 4474.731411] <TASK>
> [ 4474.731413] __schedule+0x23d/0x5a0
> [ 4474.731433] schedule+0x4e/0xb0
> [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0
> [ 4474.731441] down_write+0x43/0x50
> [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache]
> [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache]
> [ 4474.731487] kthread+0x12a/0x150
> [ 4474.731491] ? set_kthread_struct+0x50/0x50
> [ 4474.731494] ret_from_fork+0x22/0x30
> [ 4474.731499] </TASK>
> 
> [Fix]
> These 3 fixes are needed for the SRU.
> 
> dea3560e5f31965165bcf34ecf0b47af28bfd155, 6445ec3df23f24677064a327dce437ef3e02dc6,
> dc60301fb408e06e0b718c0980cdd31d2b238bee
> 
> I have built these fixes into kernel 5.15.0-39-generic (jammy) and tested to verify the problem is fixed.
> 
> [Regression Potential]
> 
> Regression potential should be minimal. I have not seen any potential drawbacks or harmful effects of this fix in my testing.
> 
> Coly Li (3):
>    bcache: improve multithreaded bch_btree_check()
>    bcache: improve multithreaded bch_sectors_dirty_init()
>    bcache: memset on stack variables in bch_btree_check() and
>      bch_sectors_dirty_init()
> 
>   drivers/md/bcache/btree.c     | 59 +++++++++++++++-----------------
>   drivers/md/bcache/btree.h     |  2 +-
>   drivers/md/bcache/writeback.c | 63 +++++++++++++++--------------------
>   drivers/md/bcache/writeback.h |  2 +-
>   4 files changed, 55 insertions(+), 71 deletions(-)
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20220801/2b4b5c09/attachment.sig>