[Bug 1071910] Re: lxc stop will hang forever
Tim
iceczd at gmail.com
Tue Oct 30 20:14:48 UTC 2012
*** This bug is a duplicate of bug 1073238 ***
https://bugs.launchpad.net/bugs/1073238
** Changed in: lxc (Ubuntu)
Status: Fix Released => New
** Changed in: lxc (Ubuntu)
Status: New => Incomplete
** Description changed:
Background:
This is issue occurs during an automated process and occurs with a 1/20 chance per iteration
I have one lxc-container on the machine
It is backed with an lvm2 snapshot
Running on ubuntu 12.10 on ec2 small instance - upgraded from 12.04 fresh instance
This is a new issue that has occurred after migrating my code from 11.10
Process:
create snapshot "lvcreate"
mount snapshot "mount"
lxc-start
do actions in container
lxc-stop
unmount snapshot "umount"
remove snapshot "lvremove"
-repeat
The issue can occur at either lxc-stop or lvremove.
when it occurs with lxc-stop:
ps -A reveals that lxc-start is still running along with kdmflush, kjournald, and init that appears to be the init process for the container
kdmflush, kjournald, init or it's sub-processes cannot be killed with "kill -9 pid" but lxc-start can
when it occurs with lvremove it occurs after lvremove is called again after failing the first time with stderr:
Using logical volume(s) on command line
Archiving volume group "vmg1" metadata (seqno 272).
Removing snapshot snap
Found volume group "vmg1"
Found volume group "vmg1"
Loading vmg1-vm table (252:0)
Loading vmg1-snap table (252:1)
/sbin/dmeventd: stat failed: No such file or directory
vmg1/snapshot0 already not monitored.
Suspending vmg1-vm (252:0) with device flush
Suspending vmg1-snap (252:1) with device flush
Suspending vmg1-vm-real (252:2) with device flush
Suspending vmg1-snap-cow (252:3) with device flush
Found volume group "vmg1"
Resuming vmg1-snap-cow (252:3)
Resuming vmg1-vm-real (252:2)
Resuming vmg1-snap (252:1)
Removing vmg1-snap-cow (252:3)
device-mapper: remove ioctl on failed: Device or resource busy
Unable to deactivate vmg1-snap-cow (252:3)
Failed to resume snap.
libdevmapper exiting with 1 device(s) still suspended.
lvremove spawns the lvm process and neither can be killed with "kill -9
pid" which indicates to me that they are waiting for something from the
kernel, and I am guessing this happens because of the same reason why
lxc-stop also hangs, and the containers processes can not be killed.
Here is an excerpt from the syslog - lxc-stop hangs because of this
kernel error, and the error doesn't always occur on the cat command, it
can happen on others as well.
--START
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.406366] kjournald starting. Commit interval 5 seconds
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.406929] EXT3-fs (dm-1): using internal journal
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.406931] EXT3-fs (dm-1): mounted filesystem with ordered data mode
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.435218] device vethyurrCc entered promiscuous mode
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.435613] IPv6: ADDRCONF(NETDEV_UP): vethyurrCc: link is not ready
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.534518] IPv6: ADDRCONF(NETDEV_CHANGE): vethyurrCc: link becomes ready
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.534543] br0: port 1(vethyurrCc) entered forwarding state
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.534547] br0: port 1(vethyurrCc) entered forwarding state
--STOP
Oct 30 14:43:17 domU-12-31-39-14-64-79 kernel: [ 1099.112881] br0: port 1(vethyurrCc) entered disabled state
Oct 30 14:43:17 domU-12-31-39-14-64-79 kernel: [ 1099.115187] device vethyurrCc left promiscuous mode
Oct 30 14:43:17 domU-12-31-39-14-64-79 kernel: [ 1099.115190] br0: port 1(vethyurrCc) entered disabled state
--START
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.188337] kjournald starting. Commit interval 5 seconds
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.188852] EXT3-fs (dm-1): using internal journal
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.188859] EXT3-fs (dm-1): mounted filesystem with ordered data mode
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.310142] device vethfsh25j entered promiscuous mode
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.310539] IPv6: ADDRCONF(NETDEV_UP): vethfsh25j: link is not ready
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.319210] IPv6: ADDRCONF(NETDEV_CHANGE): vethfsh25j: link becomes ready
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.319240] br0: port 1(vethfsh25j) entered forwarding state
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.319244] br0: port 1(vethfsh25j) entered forwarding state
--STOP
Oct 30 14:43:23 domU-12-31-39-14-64-79 kernel: [ 1105.073237] br0: port 1(vethfsh25j) entered disabled state
Oct 30 14:43:23 domU-12-31-39-14-64-79 kernel: [ 1105.075541] device vethfsh25j left promiscuous mode
Oct 30 14:43:23 domU-12-31-39-14-64-79 kernel: [ 1105.075544] br0: port 1(vethfsh25j) entered disabled state
--START
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.091653] kjournald starting. Commit interval 5 seconds
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.092173] EXT3-fs (dm-1): using internal journal
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.092176] EXT3-fs (dm-1): mounted filesystem with ordered data mode
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.119867] device vethYI2DWn entered promiscuous mode
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.120382] IPv6: ADDRCONF(NETDEV_UP): vethYI2DWn: link is not ready
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.128936] IPv6: ADDRCONF(NETDEV_CHANGE): vethYI2DWn: link becomes ready
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.128964] br0: port 1(vethYI2DWn) entered forwarding state
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.128968] br0: port 1(vethYI2DWn) entered forwarding state
--STOP
Oct 30 14:43:28 domU-12-31-39-14-64-79 kernel: [ 1110.816859] br0: port 1(vethYI2DWn) entered disabled state
Oct 30 14:43:28 domU-12-31-39-14-64-79 kernel: [ 1110.819087] device vethYI2DWn left promiscuous mode
Oct 30 14:43:28 domU-12-31-39-14-64-79 kernel: [ 1110.819090] br0: port 1(vethYI2DWn) entered disabled state
--Why is this happening occasionally?
Oct 30 14:43:29 domU-12-31-39-14-64-79 udevd[2811]: inotify_add_watch(6, /dev/dm-1, 10) failed: No such file or directory
--START
Oct 30 14:43:29 domU-12-31-39-14-64-79 kernel: [ 1111.748495] kjournald starting. Commit interval 5 seconds
Oct 30 14:43:29 domU-12-31-39-14-64-79 kernel: [ 1111.748933] EXT3-fs (dm-1): using internal journal
Oct 30 14:43:29 domU-12-31-39-14-64-79 kernel: [ 1111.748936] EXT3-fs (dm-1): mounted filesystem with ordered data mode
Oct 30 14:43:29 domU-12-31-39-14-64-79 kernel: [ 1111.868572] device vethSaApSo entered promiscuous mode
Oct 30 14:43:29 domU-12-31-39-14-64-79 kernel: [ 1111.869304] IPv6: ADDRCONF(NETDEV_UP): vethSaApSo: link is not ready
Oct 30 14:43:30 domU-12-31-39-14-64-79 kernel: [ 1111.874370] IPv6: ADDRCONF(NETDEV_CHANGE): vethSaApSo: link becomes ready
Oct 30 14:43:30 domU-12-31-39-14-64-79 kernel: [ 1111.874394] br0: port 1(vethSaApSo) entered forwarding state
Oct 30 14:43:30 domU-12-31-39-14-64-79 kernel: [ 1111.874398] br0: port 1(vethSaApSo) entered forwarding state
--STOP
Oct 30 14:43:34 domU-12-31-39-14-64-79 kernel: [ 1116.749280] br0: port 1(vethSaApSo) entered disabled state
Oct 30 14:43:34 domU-12-31-39-14-64-79 kernel: [ 1116.751502] device vethSaApSo left promiscuous mode
Oct 30 14:43:34 domU-12-31-39-14-64-79 kernel: [ 1116.751505] br0: port 1(vethSaApSo) entered disabled state
--START
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.774270] kjournald starting. Commit interval 5 seconds
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.774709] EXT3-fs (dm-1): using internal journal
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.774711] EXT3-fs (dm-1): mounted filesystem with ordered data mode
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.803322] device vethC8ic4K entered promiscuous mode
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.803718] IPv6: ADDRCONF(NETDEV_UP): vethC8ic4K: link is not ready
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.812401] IPv6: ADDRCONF(NETDEV_CHANGE): vethC8ic4K: link becomes ready
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.812458] br0: port 1(vethC8ic4K) entered forwarding state
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.812464] br0: port 1(vethC8ic4K) entered forwarding state
--KERNEL ERROR
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252907] ------------[ cut here ]------------
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252921] kernel BUG at /build/buildd/linux-3.5.0/arch/x86/mm/fault.c:396!
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252926] invalid opcode: 0000 [#1] SMP
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252932] CPU 0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252934] Modules linked in: veth dm_snapshot xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc isofs microcode acpiphp
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252958]
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252960] Pid: 8140, comm: cat Not tainted 3.5.0-17-generic #28-Ubuntu
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252966] RIP: e030:[<ffffffff8168533f>] [<ffffffff8168533f>] vmalloc_fault+0x11f/0x208
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252979] RSP: e02b:ffff880002f1d9b8 EFLAGS: 00010046
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252983] RAX: ffff880026caeff8 RBX: ffffe8ffffc00ac8 RCX: 0000000000000000
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252988] RDX: 00003ffffffff000 RSI: ffff880000000ff8 RDI: 0000000000000000
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252993] RBP: ffff880002f1d9d8 R08: ffff880017c6ae70 R09: 00007f7b4d46e000
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252998] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880066231e88
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253003] R13: ffff880026caeff8 R14: ffff880000000ff8 R15: 0000000000000002
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253012] FS: 00007f7b4d68c700(0000) GS:ffff88006a000000(0000) knlGS:0000000000000000
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253017] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253021] CR2: ffffe8ffffc00ac8 CR3: 0000000066231000 CR4: 0000000000002660
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253027] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253033] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253038] Process cat (pid: 8140, threadinfo ffff880002f1c000, task ffff88002470dc00)
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253044] Stack:
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253046] ffffe8ffffc00ac8 0000000000000029 ffff880002f1daf8 0000000000000000
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253055] ffff880002f1dae8 ffffffff816858f9 0000000000000657 ffffffff812e79e1
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253064] ffff88002470dc00 0000000000000060 ffff880055ecdd1c ffff88005636b540
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253072] Call Trace:
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253078] [<ffffffff816858f9>] do_page_fault+0x3b9/0x4e0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253087] [<ffffffff812e79e1>] ? aa_path_name+0x71/0x440
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253094] [<ffffffff8107e86a>] ? lg_local_unlock+0x1a/0x20
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253100] [<ffffffff8168b14b>] ? xen_hypervisor_callback+0x1b/0x20
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253106] [<ffffffff81004eec>] ? xen_mc_extend_args+0xec/0x110
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253112] [<ffffffff810046c0>] ? load_TLS_descriptor+0x40/0xc0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253118] [<ffffffff81004bd2>] ? xen_mc_flush+0xb2/0x1b0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253123] [<ffffffff816821e5>] page_fault+0x25/0x30
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253130] [<ffffffff81176e54>] ? mem_cgroup_charge_statistics.isra.15+0x14/0x50
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253137] [<ffffffff81178ebc>] __mem_cgroup_uncharge_common+0xcc/0x2c0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253143] [<ffffffff8100761d>] ? xen_pte_val+0x1d/0x40
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253149] [<ffffffff8117c242>] mem_cgroup_uncharge_page+0x22/0x30
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253155] [<ffffffff81153c97>] page_remove_rmap+0xb7/0x140
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253162] [<ffffffff8114797a>] ? vm_normal_page+0x1a/0x80
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253168] [<ffffffff81148c31>] unmap_page_range+0x4b1/0x740
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253173] [<ffffffff81148f4b>] unmap_single_vma+0x8b/0xd0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253179] [<ffffffff81149762>] unmap_vmas+0x52/0xa0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253184] [<ffffffff81150cf2>] exit_mmap+0x92/0x150
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253191] [<ffffffff81681dbe>] ? _raw_spin_lock_irqsave+0x2e/0x40
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253198] [<ffffffff8104ef24>] mmput+0x74/0x110
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253204] [<ffffffff810577ba>] exit_mm+0x10a/0x130
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253208] [<ffffffff81057939>] do_exit+0x159/0x8e0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253213] [<ffffffff8105841f>] do_group_exit+0x3f/0xa0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253218] [<ffffffff81058497>] sys_exit_group+0x17/0x20
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253224] [<ffffffff81689d29>] system_call_fastpath+0x16/0x1b
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253229] Code: 4c 89 e7 e8 71 e4 fe ff 4c 89 ef 48 89 de 49 89 c6 e8 63 e4 fe ff 48 83 38 00 49 89 c5 0f 84 e5 00 00 00 49 8b 3e 48 85 ff 75 02 <0f> 0b ff 14 25 e0 dd c1 81 48 89 c2 4$
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253285] RIP [<ffffffff8168533f>] vmalloc_fault+0x11f/0x208
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253291] RSP <ffff880002f1d9b8>
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253307] ---[ end trace 0e83c1ffecd3a6f4 ]---
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253312] Fixing recursive fault but reboot is needed!
+
+
+
+ --- ERROR - now using kernel v3.5.7
+
+ And then I rebooted, and it occurred on the first iteration
+
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992470] ------------[ cut here ]------------
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992483] kernel BUG at /home/apw/COD/linux/arch/x86/mm/fault.c:396!
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992489] invalid opcode: 0000 [#1] SMP
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992495] CPU 0
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992497] Modules linked in: veth dm_snapshot xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc isofs microcode acpiphp
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992524]
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992527] Pid: 1233, comm: telnet Not tainted 3.5.7-030507-generic #201210130556
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992534] RIP: e030:[<ffffffff8169d8e4>] [<ffffffff8169d8e4>] vmalloc_fault+0x114/0x1cf
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992550] RSP: e02b:ffff880045525978 EFLAGS: 00010046
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992554] RAX: ffff88006639cff8 RBX: ffffe8ffffc00a98 RCX: 0000000000000000
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992559] RDX: ffff880000000000 RSI: ffffe8ffffc00a98 RDI: 000000065fb22067
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992564] RBP: ffff880045525998 R08: ffff8800028fb370 R09: 0000000000000001
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992569] R10: 000000000098967f R11: 0000000000000001 R12: ffff88006639cff8
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992574] R13: ffffffff81c0be88 R14: ffff880000000ff8 R15: ffff880045525ab8
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992585] FS: 00007f7203243740(0000) GS:ffff88006a000000(0000) knlGS:0000000000000000
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992591] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992595] CR2: ffffe8ffffc00a98 CR3: 000000004551e000 CR4: 0000000000002660
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992600] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992606] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992612] Process telnet (pid: 1233, threadinfo ffff880045524000, task ffff8800327e8000)
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992617] Stack:
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992620] 0000000000000029 0000000000000000 0000000000000001 0000000000000060
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992629] ffff880045525aa8 ffffffff8169dee0 ffff8800455259e8 ffff8800327e8000
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992637] 0000000000000002 ffffe8ffffc00a98 0000000000000002 0000000000000004
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992645] Call Trace:
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992652] [<ffffffff8169dee0>] do_page_fault+0x3c0/0x520
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992661] [<ffffffff8119a387>] ? do_select+0x537/0x5c0
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992670] [<ffffffff81005131>] ? xen_mc_extend_args+0x111/0x150
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992676] [<ffffffff8169a5e5>] page_fault+0x25/0x30
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992714] [<ffffffff8117cea3>] ? mem_cgroup_charge_statistics.isra.16+0x13/0x50
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992729] [<ffffffff8117ee2b>] __mem_cgroup_uncharge_common+0xcb/0x1f0
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992737] [<ffffffff810064ed>] ? xen_pte_val+0x1d/0x40
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992743] [<ffffffff81181b82>] mem_cgroup_uncharge_page+0x22/0x30
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992751] [<ffffffff8115921b>] page_remove_rmap+0x5b/0x150
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992759] [<ffffffff8114d25b>] ? vm_normal_page+0x1b/0x80
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992765] [<ffffffff8114e201>] zap_pte_range+0x1e1/0x430
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992771] [<ffffffff81005b10>] ? xen_make_pte+0x20/0x70
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992776] [<ffffffff8114e602>] unmap_page_range+0x1b2/0x310
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992782] [<ffffffff8114e7e8>] unmap_single_vma+0x88/0xd0
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992788] [<ffffffff8114eff4>] unmap_vmas+0x54/0xa0
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992793] [<ffffffff81156042>] exit_mmap+0x92/0x150
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992798] [<ffffffff81004d92>] ? xen_mc_flush+0xb2/0x1c0
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992807] [<ffffffff8105098a>] mmput.part.20+0x4a/0x120
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992812] [<ffffffff81050a89>] mmput+0x29/0x30
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992819] [<ffffffff81058ffa>] exit_mm+0x10a/0x130
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992826] [<ffffffff810eeae5>] ? taskstats_exit+0x45/0x240
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992833] [<ffffffff8169a165>] ? _raw_spin_lock_irq+0x15/0x20
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992839] [<ffffffff81059191>] do_exit+0x171/0x480
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992844] [<ffffffff81698904>] ? __schedule+0x3c4/0x700
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992849] [<ffffffff81059644>] do_group_exit+0x44/0xa0
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992855] [<ffffffff810596b7>] sys_exit_group+0x17/0x20
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992862] [<ffffffff816a26e9>] system_call_fastpath+0x16/0x1b
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992867] Code: 83 38 00 49 89 c4 0f 84 ce 00 00 00 49 83 3e 00 74 19 49 8b 3e e8 45 22 fe ff 49 8b 3c 24 49 89 c5 e8 39 22 fe ff 49 39 c5 74 02 <0f> 0b 48 89 de 4c 89 f7 e8 4e 22 fe ff 48$
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992922] RIP [<ffffffff8169d8e4>] vmalloc_fault+0x114/0x1cf
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992930] RSP <ffff880045525978>
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992935] ---[ end trace 03270cf1ab1c4452 ]---
+ Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992939] Fixing recursive fault but reboot is needed!
** Description changed:
Background:
This is issue occurs during an automated process and occurs with a 1/20 chance per iteration
I have one lxc-container on the machine
It is backed with an lvm2 snapshot
Running on ubuntu 12.10 on ec2 small instance - upgraded from 12.04 fresh instance
This is a new issue that has occurred after migrating my code from 11.10
Process:
create snapshot "lvcreate"
mount snapshot "mount"
lxc-start
do actions in container
lxc-stop
unmount snapshot "umount"
remove snapshot "lvremove"
-repeat
The issue can occur at either lxc-stop or lvremove.
when it occurs with lxc-stop:
ps -A reveals that lxc-start is still running along with kdmflush, kjournald, and init that appears to be the init process for the container
kdmflush, kjournald, init or it's sub-processes cannot be killed with "kill -9 pid" but lxc-start can
when it occurs with lvremove it occurs after lvremove is called again after failing the first time with stderr:
Using logical volume(s) on command line
Archiving volume group "vmg1" metadata (seqno 272).
Removing snapshot snap
Found volume group "vmg1"
Found volume group "vmg1"
Loading vmg1-vm table (252:0)
Loading vmg1-snap table (252:1)
/sbin/dmeventd: stat failed: No such file or directory
vmg1/snapshot0 already not monitored.
Suspending vmg1-vm (252:0) with device flush
Suspending vmg1-snap (252:1) with device flush
Suspending vmg1-vm-real (252:2) with device flush
Suspending vmg1-snap-cow (252:3) with device flush
Found volume group "vmg1"
Resuming vmg1-snap-cow (252:3)
Resuming vmg1-vm-real (252:2)
Resuming vmg1-snap (252:1)
Removing vmg1-snap-cow (252:3)
device-mapper: remove ioctl on failed: Device or resource busy
Unable to deactivate vmg1-snap-cow (252:3)
Failed to resume snap.
libdevmapper exiting with 1 device(s) still suspended.
lvremove spawns the lvm process and neither can be killed with "kill -9
pid" which indicates to me that they are waiting for something from the
kernel, and I am guessing this happens because of the same reason why
lxc-stop also hangs, and the containers processes can not be killed.
Here is an excerpt from the syslog - lxc-stop hangs because of this
kernel error, and the error doesn't always occur on the cat command, it
can happen on others as well.
--START
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.406366] kjournald starting. Commit interval 5 seconds
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.406929] EXT3-fs (dm-1): using internal journal
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.406931] EXT3-fs (dm-1): mounted filesystem with ordered data mode
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.435218] device vethyurrCc entered promiscuous mode
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.435613] IPv6: ADDRCONF(NETDEV_UP): vethyurrCc: link is not ready
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.534518] IPv6: ADDRCONF(NETDEV_CHANGE): vethyurrCc: link becomes ready
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.534543] br0: port 1(vethyurrCc) entered forwarding state
Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.534547] br0: port 1(vethyurrCc) entered forwarding state
--STOP
Oct 30 14:43:17 domU-12-31-39-14-64-79 kernel: [ 1099.112881] br0: port 1(vethyurrCc) entered disabled state
Oct 30 14:43:17 domU-12-31-39-14-64-79 kernel: [ 1099.115187] device vethyurrCc left promiscuous mode
Oct 30 14:43:17 domU-12-31-39-14-64-79 kernel: [ 1099.115190] br0: port 1(vethyurrCc) entered disabled state
--START
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.188337] kjournald starting. Commit interval 5 seconds
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.188852] EXT3-fs (dm-1): using internal journal
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.188859] EXT3-fs (dm-1): mounted filesystem with ordered data mode
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.310142] device vethfsh25j entered promiscuous mode
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.310539] IPv6: ADDRCONF(NETDEV_UP): vethfsh25j: link is not ready
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.319210] IPv6: ADDRCONF(NETDEV_CHANGE): vethfsh25j: link becomes ready
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.319240] br0: port 1(vethfsh25j) entered forwarding state
Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.319244] br0: port 1(vethfsh25j) entered forwarding state
--STOP
Oct 30 14:43:23 domU-12-31-39-14-64-79 kernel: [ 1105.073237] br0: port 1(vethfsh25j) entered disabled state
Oct 30 14:43:23 domU-12-31-39-14-64-79 kernel: [ 1105.075541] device vethfsh25j left promiscuous mode
Oct 30 14:43:23 domU-12-31-39-14-64-79 kernel: [ 1105.075544] br0: port 1(vethfsh25j) entered disabled state
--START
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.091653] kjournald starting. Commit interval 5 seconds
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.092173] EXT3-fs (dm-1): using internal journal
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.092176] EXT3-fs (dm-1): mounted filesystem with ordered data mode
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.119867] device vethYI2DWn entered promiscuous mode
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.120382] IPv6: ADDRCONF(NETDEV_UP): vethYI2DWn: link is not ready
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.128936] IPv6: ADDRCONF(NETDEV_CHANGE): vethYI2DWn: link becomes ready
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.128964] br0: port 1(vethYI2DWn) entered forwarding state
Oct 30 14:43:24 domU-12-31-39-14-64-79 kernel: [ 1106.128968] br0: port 1(vethYI2DWn) entered forwarding state
--STOP
Oct 30 14:43:28 domU-12-31-39-14-64-79 kernel: [ 1110.816859] br0: port 1(vethYI2DWn) entered disabled state
Oct 30 14:43:28 domU-12-31-39-14-64-79 kernel: [ 1110.819087] device vethYI2DWn left promiscuous mode
Oct 30 14:43:28 domU-12-31-39-14-64-79 kernel: [ 1110.819090] br0: port 1(vethYI2DWn) entered disabled state
--Why is this happening occasionally?
Oct 30 14:43:29 domU-12-31-39-14-64-79 udevd[2811]: inotify_add_watch(6, /dev/dm-1, 10) failed: No such file or directory
--START
Oct 30 14:43:29 domU-12-31-39-14-64-79 kernel: [ 1111.748495] kjournald starting. Commit interval 5 seconds
Oct 30 14:43:29 domU-12-31-39-14-64-79 kernel: [ 1111.748933] EXT3-fs (dm-1): using internal journal
Oct 30 14:43:29 domU-12-31-39-14-64-79 kernel: [ 1111.748936] EXT3-fs (dm-1): mounted filesystem with ordered data mode
Oct 30 14:43:29 domU-12-31-39-14-64-79 kernel: [ 1111.868572] device vethSaApSo entered promiscuous mode
Oct 30 14:43:29 domU-12-31-39-14-64-79 kernel: [ 1111.869304] IPv6: ADDRCONF(NETDEV_UP): vethSaApSo: link is not ready
Oct 30 14:43:30 domU-12-31-39-14-64-79 kernel: [ 1111.874370] IPv6: ADDRCONF(NETDEV_CHANGE): vethSaApSo: link becomes ready
Oct 30 14:43:30 domU-12-31-39-14-64-79 kernel: [ 1111.874394] br0: port 1(vethSaApSo) entered forwarding state
Oct 30 14:43:30 domU-12-31-39-14-64-79 kernel: [ 1111.874398] br0: port 1(vethSaApSo) entered forwarding state
--STOP
Oct 30 14:43:34 domU-12-31-39-14-64-79 kernel: [ 1116.749280] br0: port 1(vethSaApSo) entered disabled state
Oct 30 14:43:34 domU-12-31-39-14-64-79 kernel: [ 1116.751502] device vethSaApSo left promiscuous mode
Oct 30 14:43:34 domU-12-31-39-14-64-79 kernel: [ 1116.751505] br0: port 1(vethSaApSo) entered disabled state
--START
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.774270] kjournald starting. Commit interval 5 seconds
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.774709] EXT3-fs (dm-1): using internal journal
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.774711] EXT3-fs (dm-1): mounted filesystem with ordered data mode
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.803322] device vethC8ic4K entered promiscuous mode
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.803718] IPv6: ADDRCONF(NETDEV_UP): vethC8ic4K: link is not ready
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.812401] IPv6: ADDRCONF(NETDEV_CHANGE): vethC8ic4K: link becomes ready
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.812458] br0: port 1(vethC8ic4K) entered forwarding state
Oct 30 14:43:36 domU-12-31-39-14-64-79 kernel: [ 1118.812464] br0: port 1(vethC8ic4K) entered forwarding state
--KERNEL ERROR
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252907] ------------[ cut here ]------------
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252921] kernel BUG at /build/buildd/linux-3.5.0/arch/x86/mm/fault.c:396!
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252926] invalid opcode: 0000 [#1] SMP
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252932] CPU 0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252934] Modules linked in: veth dm_snapshot xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc isofs microcode acpiphp
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252958]
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252960] Pid: 8140, comm: cat Not tainted 3.5.0-17-generic #28-Ubuntu
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252966] RIP: e030:[<ffffffff8168533f>] [<ffffffff8168533f>] vmalloc_fault+0x11f/0x208
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252979] RSP: e02b:ffff880002f1d9b8 EFLAGS: 00010046
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252983] RAX: ffff880026caeff8 RBX: ffffe8ffffc00ac8 RCX: 0000000000000000
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252988] RDX: 00003ffffffff000 RSI: ffff880000000ff8 RDI: 0000000000000000
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252993] RBP: ffff880002f1d9d8 R08: ffff880017c6ae70 R09: 00007f7b4d46e000
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.252998] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880066231e88
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253003] R13: ffff880026caeff8 R14: ffff880000000ff8 R15: 0000000000000002
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253012] FS: 00007f7b4d68c700(0000) GS:ffff88006a000000(0000) knlGS:0000000000000000
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253017] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253021] CR2: ffffe8ffffc00ac8 CR3: 0000000066231000 CR4: 0000000000002660
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253027] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253033] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253038] Process cat (pid: 8140, threadinfo ffff880002f1c000, task ffff88002470dc00)
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253044] Stack:
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253046] ffffe8ffffc00ac8 0000000000000029 ffff880002f1daf8 0000000000000000
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253055] ffff880002f1dae8 ffffffff816858f9 0000000000000657 ffffffff812e79e1
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253064] ffff88002470dc00 0000000000000060 ffff880055ecdd1c ffff88005636b540
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253072] Call Trace:
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253078] [<ffffffff816858f9>] do_page_fault+0x3b9/0x4e0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253087] [<ffffffff812e79e1>] ? aa_path_name+0x71/0x440
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253094] [<ffffffff8107e86a>] ? lg_local_unlock+0x1a/0x20
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253100] [<ffffffff8168b14b>] ? xen_hypervisor_callback+0x1b/0x20
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253106] [<ffffffff81004eec>] ? xen_mc_extend_args+0xec/0x110
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253112] [<ffffffff810046c0>] ? load_TLS_descriptor+0x40/0xc0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253118] [<ffffffff81004bd2>] ? xen_mc_flush+0xb2/0x1b0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253123] [<ffffffff816821e5>] page_fault+0x25/0x30
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253130] [<ffffffff81176e54>] ? mem_cgroup_charge_statistics.isra.15+0x14/0x50
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253137] [<ffffffff81178ebc>] __mem_cgroup_uncharge_common+0xcc/0x2c0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253143] [<ffffffff8100761d>] ? xen_pte_val+0x1d/0x40
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253149] [<ffffffff8117c242>] mem_cgroup_uncharge_page+0x22/0x30
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253155] [<ffffffff81153c97>] page_remove_rmap+0xb7/0x140
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253162] [<ffffffff8114797a>] ? vm_normal_page+0x1a/0x80
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253168] [<ffffffff81148c31>] unmap_page_range+0x4b1/0x740
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253173] [<ffffffff81148f4b>] unmap_single_vma+0x8b/0xd0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253179] [<ffffffff81149762>] unmap_vmas+0x52/0xa0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253184] [<ffffffff81150cf2>] exit_mmap+0x92/0x150
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253191] [<ffffffff81681dbe>] ? _raw_spin_lock_irqsave+0x2e/0x40
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253198] [<ffffffff8104ef24>] mmput+0x74/0x110
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253204] [<ffffffff810577ba>] exit_mm+0x10a/0x130
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253208] [<ffffffff81057939>] do_exit+0x159/0x8e0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253213] [<ffffffff8105841f>] do_group_exit+0x3f/0xa0
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253218] [<ffffffff81058497>] sys_exit_group+0x17/0x20
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253224] [<ffffffff81689d29>] system_call_fastpath+0x16/0x1b
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253229] Code: 4c 89 e7 e8 71 e4 fe ff 4c 89 ef 48 89 de 49 89 c6 e8 63 e4 fe ff 48 83 38 00 49 89 c5 0f 84 e5 00 00 00 49 8b 3e 48 85 ff 75 02 <0f> 0b ff 14 25 e0 dd c1 81 48 89 c2 4$
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253285] RIP [<ffffffff8168533f>] vmalloc_fault+0x11f/0x208
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253291] RSP <ffff880002f1d9b8>
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253307] ---[ end trace 0e83c1ffecd3a6f4 ]---
Oct 30 14:43:39 domU-12-31-39-14-64-79 kernel: [ 1121.253312] Fixing recursive fault but reboot is needed!
-
-
--- ERROR - now using kernel v3.5.7
-
- And then I rebooted, and it occurred on the first iteration
-
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992470] ------------[ cut here ]------------
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992483] kernel BUG at /home/apw/COD/linux/arch/x86/mm/fault.c:396!
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992489] invalid opcode: 0000 [#1] SMP
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992495] CPU 0
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992497] Modules linked in: veth dm_snapshot xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc isofs microcode acpiphp
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992524]
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992527] Pid: 1233, comm: telnet Not tainted 3.5.7-030507-generic #201210130556
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992534] RIP: e030:[<ffffffff8169d8e4>] [<ffffffff8169d8e4>] vmalloc_fault+0x114/0x1cf
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992550] RSP: e02b:ffff880045525978 EFLAGS: 00010046
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992554] RAX: ffff88006639cff8 RBX: ffffe8ffffc00a98 RCX: 0000000000000000
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992559] RDX: ffff880000000000 RSI: ffffe8ffffc00a98 RDI: 000000065fb22067
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992564] RBP: ffff880045525998 R08: ffff8800028fb370 R09: 0000000000000001
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992569] R10: 000000000098967f R11: 0000000000000001 R12: ffff88006639cff8
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992574] R13: ffffffff81c0be88 R14: ffff880000000ff8 R15: ffff880045525ab8
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992585] FS: 00007f7203243740(0000) GS:ffff88006a000000(0000) knlGS:0000000000000000
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992591] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992595] CR2: ffffe8ffffc00a98 CR3: 000000004551e000 CR4: 0000000000002660
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992600] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992606] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992612] Process telnet (pid: 1233, threadinfo ffff880045524000, task ffff8800327e8000)
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992617] Stack:
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992620] 0000000000000029 0000000000000000 0000000000000001 0000000000000060
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992629] ffff880045525aa8 ffffffff8169dee0 ffff8800455259e8 ffff8800327e8000
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992637] 0000000000000002 ffffe8ffffc00a98 0000000000000002 0000000000000004
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992645] Call Trace:
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992652] [<ffffffff8169dee0>] do_page_fault+0x3c0/0x520
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992661] [<ffffffff8119a387>] ? do_select+0x537/0x5c0
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992670] [<ffffffff81005131>] ? xen_mc_extend_args+0x111/0x150
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992676] [<ffffffff8169a5e5>] page_fault+0x25/0x30
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992714] [<ffffffff8117cea3>] ? mem_cgroup_charge_statistics.isra.16+0x13/0x50
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992729] [<ffffffff8117ee2b>] __mem_cgroup_uncharge_common+0xcb/0x1f0
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992737] [<ffffffff810064ed>] ? xen_pte_val+0x1d/0x40
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992743] [<ffffffff81181b82>] mem_cgroup_uncharge_page+0x22/0x30
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992751] [<ffffffff8115921b>] page_remove_rmap+0x5b/0x150
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992759] [<ffffffff8114d25b>] ? vm_normal_page+0x1b/0x80
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992765] [<ffffffff8114e201>] zap_pte_range+0x1e1/0x430
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992771] [<ffffffff81005b10>] ? xen_make_pte+0x20/0x70
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992776] [<ffffffff8114e602>] unmap_page_range+0x1b2/0x310
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992782] [<ffffffff8114e7e8>] unmap_single_vma+0x88/0xd0
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992788] [<ffffffff8114eff4>] unmap_vmas+0x54/0xa0
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992793] [<ffffffff81156042>] exit_mmap+0x92/0x150
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992798] [<ffffffff81004d92>] ? xen_mc_flush+0xb2/0x1c0
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992807] [<ffffffff8105098a>] mmput.part.20+0x4a/0x120
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992812] [<ffffffff81050a89>] mmput+0x29/0x30
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992819] [<ffffffff81058ffa>] exit_mm+0x10a/0x130
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992826] [<ffffffff810eeae5>] ? taskstats_exit+0x45/0x240
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992833] [<ffffffff8169a165>] ? _raw_spin_lock_irq+0x15/0x20
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992839] [<ffffffff81059191>] do_exit+0x171/0x480
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992844] [<ffffffff81698904>] ? __schedule+0x3c4/0x700
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992849] [<ffffffff81059644>] do_group_exit+0x44/0xa0
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992855] [<ffffffff810596b7>] sys_exit_group+0x17/0x20
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992862] [<ffffffff816a26e9>] system_call_fastpath+0x16/0x1b
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992867] Code: 83 38 00 49 89 c4 0f 84 ce 00 00 00 49 83 3e 00 74 19 49 8b 3e e8 45 22 fe ff 49 8b 3c 24 49 89 c5 e8 39 22 fe ff 49 39 c5 74 02 <0f> 0b 48 89 de 4c 89 f7 e8 4e 22 fe ff 48$
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992922] RIP [<ffffffff8169d8e4>] vmalloc_fault+0x114/0x1cf
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992930] RSP <ffff880045525978>
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992935] ---[ end trace 03270cf1ab1c4452 ]---
Oct 30 19:57:31 ip-10-72-206-25 kernel: [40516179.992939] Fixing recursive fault but reboot is needed!
** This bug has been marked a duplicate of bug 1073238
Occasional kernel error while running commands in LXC
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to lxc in Ubuntu.
https://bugs.launchpad.net/bugs/1071910
Title:
lxc stop will hang forever
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions
More information about the Ubuntu-server-bugs
mailing list