[3.13.y-ckt stable] Patch "nfs: don't call blocking operations while !TASK_RUNNING" has been added to staging queue
Kamal Mostafa
kamal at canonical.com
Tue Mar 31 19:23:25 UTC 2015
On Tue, 2015-03-31 at 14:57 -0400, Jeff Layton wrote:
> On Tue, 31 Mar 2015 11:45:48 -0700
> Kamal Mostafa <kamal at canonical.com> wrote:
>
> > This is a note to let you know that I have just added a patch titled
> >
> > nfs: don't call blocking operations while !TASK_RUNNING
> >
> > to the linux-3.13.y-queue branch of the 3.13.y-ckt extended stable tree
> > which can be found at:
> >
> > http://kernel.ubuntu.com/git?p=ubuntu/linux.git;a=shortlog;h=refs/heads/linux-3.13.y-queue
> >
> > This patch is scheduled to be released in version 3.13.11-ckt18.
> >
> > If you, or anyone else, feels it should not be added to this tree, please
> > reply to this email.
> >
>
> You may want to wait until the follow up patch to this makes its way to mainline:
>
> [PATCH] nfs: fix high load average due to callback thread sleeping
Okay, I'll drop "don't call blocking operations" from 3.13-stable until
that follow-up lands. Thanks very much, Jeff!
-Kamal
> This patch pins the load average at 1 due to the fact that Linux counts
> TASK_UNINTERRUPTIBLE tasks against the load average. The follow-on
> patch has also been marked for stable so once it goes in you'll
> probably want to take both.
>
> > For more information about the 3.13.y-ckt tree, see
> > https://wiki.ubuntu.com/Kernel/Dev/ExtendedStable
> >
> > Thanks.
> > -Kamal
> >
> > ------
> >
> > From 2cc6ed1b42411ceb375410f0685ad00a2eb77632 Mon Sep 17 00:00:00 2001
> > From: Jeff Layton <jlayton at primarydata.com>
> > Date: Wed, 14 Jan 2015 13:08:57 -0500
> > Subject: nfs: don't call blocking operations while !TASK_RUNNING
> >
> > commit 6ffa30d3f734d4f6b478081dfc09592021028f90 upstream.
> >
> > Bruce reported seeing this warning pop when mounting using v4.1:
> >
> > ------------[ cut here ]------------
> > WARNING: CPU: 1 PID: 1121 at kernel/sched/core.c:7300 __might_sleep+0xbd/0xd0()
> > do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff810ff58f>] prepare_to_wait+0x2f/0x90
> > Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer ppdev joydev snd virtio_console virtio_balloon pcspkr serio_raw parport_pc parport pvpanic floppy soundcore i2c_piix4 virtio_blk virtio_net qxl drm_kms_helper ttm drm virtio_pci virtio_ring ata_generic virtio pata_acpi
> > CPU: 1 PID: 1121 Comm: nfsv4.1-svc Not tainted 3.19.0-rc4+ #25
> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140709_153950- 04/01/2014
> > 0000000000000000 000000004e5e3f73 ffff8800b998fb48 ffffffff8186ac78
> > 0000000000000000 ffff8800b998fba0 ffff8800b998fb88 ffffffff810ac9da
> > ffff8800b998fb68 ffffffff81c923e7 00000000000004d9 0000000000000000
> > Call Trace:
> > [<ffffffff8186ac78>] dump_stack+0x4c/0x65
> > [<ffffffff810ac9da>] warn_slowpath_common+0x8a/0xc0
> > [<ffffffff810aca65>] warn_slowpath_fmt+0x55/0x70
> > [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
> > [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
> > [<ffffffff810dd2ad>] __might_sleep+0xbd/0xd0
> > [<ffffffff8124c973>] kmem_cache_alloc_trace+0x243/0x430
> > [<ffffffff810d941e>] ? groups_alloc+0x3e/0x130
> > [<ffffffff810d941e>] groups_alloc+0x3e/0x130
> > [<ffffffffa0301b1e>] svcauth_unix_accept+0x16e/0x290 [sunrpc]
> > [<ffffffffa0300571>] svc_authenticate+0xe1/0xf0 [sunrpc]
> > [<ffffffffa02fc564>] svc_process_common+0x244/0x6a0 [sunrpc]
> > [<ffffffffa02fd044>] bc_svc_process+0x1c4/0x260 [sunrpc]
> > [<ffffffffa03d5478>] nfs41_callback_svc+0x128/0x1f0 [nfsv4]
> > [<ffffffff810ff970>] ? wait_woken+0xc0/0xc0
> > [<ffffffffa03d5350>] ? nfs4_callback_svc+0x60/0x60 [nfsv4]
> > [<ffffffff810d45bf>] kthread+0x11f/0x140
> > [<ffffffff810ea815>] ? local_clock+0x15/0x30
> > [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
> > [<ffffffff81874bfc>] ret_from_fork+0x7c/0xb0
> > [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
> > ---[ end trace 675220a11e30f4f2 ]---
> >
> > nfs41_callback_svc does most of its work while in TASK_INTERRUPTIBLE,
> > which is just wrong. Fix that by finishing the wait immediately if we've
> > found that the list has something on it.
> >
> > Also, we don't expect this kthread to accept signals, so we should be
> > using a TASK_UNINTERRUPTIBLE sleep instead. That however, opens us up
> > hung task warnings from the watchdog, so have the schedule_timeout
> > wake up every 60s if there's no callback activity.
> >
> > Reported-by: "J. Bruce Fields" <bfields at fieldses.org>
> > Signed-off-by: Jeff Layton <jlayton at primarydata.com>
> > Signed-off-by: Trond Myklebust <trond.myklebust at primarydata.com>
> > Signed-off-by: Kamal Mostafa <kamal at canonical.com>
> > ---
> > fs/nfs/callback.c | 8 +++++---
> > 1 file changed, 5 insertions(+), 3 deletions(-)
> >
> > diff --git a/fs/nfs/callback.c b/fs/nfs/callback.c
> > index 073b4cf..0a2016b 100644
> > --- a/fs/nfs/callback.c
> > +++ b/fs/nfs/callback.c
> > @@ -128,22 +128,24 @@ nfs41_callback_svc(void *vrqstp)
> > if (try_to_freeze())
> > continue;
> >
> > - prepare_to_wait(&serv->sv_cb_waitq, &wq, TASK_INTERRUPTIBLE);
> > + prepare_to_wait(&serv->sv_cb_waitq, &wq, TASK_UNINTERRUPTIBLE);
> > spin_lock_bh(&serv->sv_cb_lock);
> > if (!list_empty(&serv->sv_cb_list)) {
> > req = list_first_entry(&serv->sv_cb_list,
> > struct rpc_rqst, rq_bc_list);
> > list_del(&req->rq_bc_list);
> > spin_unlock_bh(&serv->sv_cb_lock);
> > + finish_wait(&serv->sv_cb_waitq, &wq);
> > dprintk("Invoking bc_svc_process()\n");
> > error = bc_svc_process(serv, req, rqstp);
> > dprintk("bc_svc_process() returned w/ error code= %d\n",
> > error);
> > } else {
> > spin_unlock_bh(&serv->sv_cb_lock);
> > - schedule();
> > + /* schedule_timeout to game the hung task watchdog */
> > + schedule_timeout(60 * HZ);
> > + finish_wait(&serv->sv_cb_waitq, &wq);
> > }
> > - finish_wait(&serv->sv_cb_waitq, &wq);
> > }
> > return 0;
> > }
> > --
> > 1.9.1
> >
>
>
More information about the kernel-team
mailing list