ACK+Cmnt: [Patch 0/2] [impish/linux-aws] Xen: Issues with detaching volume

Thadeu Lima de Souza Cascardo cascardo at canonical.com
Wed Mar 30 17:15:49 UTC 2022


On Wed, Mar 30, 2022 at 09:13:57AM -0600, Tim Gardner wrote:
> BugLink: https://bugs.launchpad.net/bugs/1966969
> 
> SRU Justification
> 
> [Impact]
> 
> We are observing issue with the secondary volume stuck in detaching. This is observed
> with the latest Canonical, Ubuntu EKS Node OS (k8s_1.19), 20.04 LTS, amd64 focal
> image build on 2022-03-08 and Xen instance type( for eg : m4, t2 instance type )
> 
> AMI in eu-west-1 : ami-0f4ffbcba23a6c434
> AMI in us-east-1 : ami-021feb4aa3b3c59c3
> 
> When terminating an instance with a stuck volume the shutdown process is interrupted by a xen task hanging:
> [ 847.895334] INFO: task xenwatch:188 blocked for more than 483 seconds.
> [ 847.901573] Not tainted 5.13.0-1017-aws #19~20.04.1-Ubuntu
> [ 847.907144] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 847.914462] task:xenwatch state:D stack: 0 pid: 188 ppid: 2 flags:0x00004000
> [ 847.914467] Call Trace:
> [ 847.914469] <TASK>
> [ 847.914472] __schedule+0x2ee/0x900
> [ 847.914478] schedule+0x4f/0xc0
> [ 847.914479] schedule_preempt_disabled+0xe/0x10
> [ 847.914482] __mutex_lock.isra.0+0x183/0x4d0
> [ 847.914486] __mutex_lock_slowpath+0x13/0x20
> [ 847.914487] mutex_lock+0x32/0x40
> [ 847.914489] del_gendisk+0x90/0x200
> [ 847.914493] xlvbd_release_gendisk+0x72/0xc0
> [ 847.914499] blkback_changed+0x101/0x210
> [ 847.914502] xenbus_otherend_changed+0x8f/0x130
> [ 847.914507] backend_changed+0x13/0x20
> [ 847.914510] xenwatch_thread+0xa6/0x180
> [ 847.914513] ? wait_woken+0x80/0x80
> [ 847.914517] ? test_reply.isra.0+0x40/0x40
> [ 847.914520] kthread+0x12b/0x150
> [ 847.914523] ? set_kthread_struct+0x40/0x40
> [ 847.914525] ret_from_fork+0x22/0x30
> [ 847.914531] </TASK>
> 
> this looks like it's waiting on a xen block device to be released.
> 
> Following steps used to reproduce the issue:
> * Created a m4,t2(xen) instance with the latest ami for latest Canonical, Ubuntu EKS Node OS (k8s_1.19), 20.04 LTS, amd64
> * Created a filesystem on volume
> * Mounted volume through OS
> * Unmounted volume in OS
> * Detached volume from AWS console
> * Volume gets stuck.
> 
> We at the internal team observed this commit upstream: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=05d69d950d9d84218fc9beafd02dea1f6a70e09e
> 
> [...] and a del_gendisk from the block device release
> method, which will deadlock.
> 
> It has a Fixes: tag referring to a commit from 5.13 so this could be the root-cause.
> While testing, we observed this commit is fixing the issue.
> 
> [Test Plan]
> 
> Amazon tested. Good results.
> 
> [Where things could go wrong]
> 
> Detaching volumes could fail in new and bizarre ways
> 
> [Other Info]
> 
> SF: #00331175
> 
> 

Just wondering if commit 3b62c140e93d32c825ed028faca45dee58dbe37f
("xen-blkfront: use blk_mq_alloc_disk and blk_cleanup_disk") is useful or
needed here as well.

But given the testing:

Acked-by: Thadeu Lima de Souza Cascardo <cascardo at canonical.com>



More information about the kernel-team mailing list