[Bug 1736390] Re: openvswitch: kernel oops destroying interfaces on i386
Christian Ehrhardt
1736390 at bugs.launchpad.net
Thu Sep 13 09:16:47 UTC 2018
While the tests that run on autopkgtest infra suggest it still is an
issue, I first I tried to reproduce as-is to be sure the trigger is good
(new release, new kernel, new OVS):
$ autopkgtest --apt-upgrade --shell --no-built-binaries openvswitch_2.9.0-0ubuntu1.dsc -- qemu --qemu-command=qemu-system-i386 --cpus 4 --ram-size=4096 ~/autopkgtest-bionic-i386.img
$ autopkgtest --apt-upgrade --shell --no-built-binaries openvswitch_2.9.0-0ubuntu1.dsc -- qemu --cpus 4 --ram-size=4096 ~/autopkgtest-bionic-i386.img
While faster in the crash with more CPUs I eventually reduced to 1 to
have better (more clear) stack traces.
It hangs (tests POV) and crashes (main console running dmesg -w).
[ 56.320025] BUG: unable to handle kernel NULL pointer dereference at 00000000
[ 56.320760] IP: add_grec+0x28/0x450
[ 56.321137] *pdpt = 000000001ebe7001 *pde = 0000000000000000
[ 56.321699] Oops: 0000 [#1] SMP
[ 56.322009] Modules linked in: veth openvswitch nsh nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack libcrc32c 9p fscache kvm_intel ppdev kvm irqbypass joydev 9pnet_virtio input_leds parport_pc serio_raw 9pnet parport qemu_fw_cfg mac_hid sch_fq_codel ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq psmouse virtio_blk virtio_net i2c_piix4 pata_acpi floppy
[ 56.325571] CPU: 0 PID: 240 Comm: systemd-journal Tainted: G W 4.15.0-34-generic #37-Ubuntu
[ 56.326485] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 56.327356] EIP: add_grec+0x28/0x450
[ 56.327712] EFLAGS: 00010202 CPU: 0
[ 56.328052] EAX: 00000000 EBX: dda65420 ECX: 00000006 EDX: dda65420
[ 56.328651] ESI: dc489a00 EDI: dc489a00 EBP: d94c9f34 ESP: d94c9ef4
[ 56.329259] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 56.329774] CR0: 80050033 CR2: 00000000 CR3: 1e9adba0 CR4: 000006f0
[ 56.330379] Call Trace:
[ 56.330623] <SOFTIRQ>
[ 56.330864] mld_ifc_timer_expire+0x10e/0x260
[ 56.331285] ? igmp6_timer_handler+0x60/0x60
[ 56.331699] call_timer_fn+0x2f/0x120
[ 56.332066] ? igmp6_timer_handler+0x60/0x60
[ 56.332489] run_timer_softirq+0x3b5/0x410
[ 56.332899] ? rcu_process_callbacks+0xc8/0x470
[ 56.333353] ? __softirqentry_text_start+0x8/0x8
[ 56.333808] __do_softirq+0xae/0x255
[ 56.334163] ? __softirqentry_text_start+0x8/0x8
[ 56.334617] call_on_stack+0x45/0x50
[ 56.334971] </SOFTIRQ>
[ 56.335219] ? irq_exit+0xb5/0xc0
[ 56.335549] ? smp_apic_timer_interrupt+0x6c/0x120
[ 56.336022] ? apic_timer_interrupt+0x3c/0x44
[ 56.336451] Code: 74 26 00 3e 8d 74 26 00 55 89 e5 57 56 53 89 c6 83 ec 34 89 4d e8 65 a1 14 00 00 00 89 45 f0 31 c0 f6 42 44 08 8b 42 10 89 45 cc <8b> 00 c7 45 ec 00 00 00 00 0f 85 f1 01 00 00 8b 80 54 01 00 00
[ 56.338295] EIP: add_grec+0x28/0x450 SS:ESP: 0068:d94c9ef4
[ 56.338832] CR2: 0000000000000000
[ 56.339163] ---[ end trace 6b06ace1457ab251 ]---
[ 56.339616] Kernel panic - not syncing: Fatal exception in interrupt
[ 56.340448] Kernel Offset: 0x9000000 from 0xc1000000 (relocation range: 0xc0000000-0xdf7fdfff)
[ 56.341293] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
With that, try the new kernel.
Note: console after starting the test
$ sudo nc -U /tmp/autopkgtest-qemu*/ttyS0
Umm, I was stopped in my tracks realizing this is an amd64 kernel.
@Jsalisbury - I'll need i386 kernels to do this.
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to openvswitch in Ubuntu.
https://bugs.launchpad.net/bugs/1736390
Title:
openvswitch: kernel oops destroying interfaces on i386
Status in linux package in Ubuntu:
In Progress
Status in openvswitch package in Ubuntu:
Invalid
Status in linux source package in Artful:
Won't Fix
Status in openvswitch source package in Artful:
Invalid
Status in linux source package in Bionic:
In Progress
Status in openvswitch source package in Bionic:
Invalid
Status in linux source package in Cosmic:
In Progress
Status in openvswitch source package in Cosmic:
Invalid
Bug description:
Reproducable on bionic using the autopkgtest's from openvswitch on
i386:
[ 41.420568] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 41.421000] IP: igmp_group_dropped+0x21/0x220
[ 41.421246] *pdpt = 000000001d62c001 *pde = 0000000000000000
[ 41.421659] Oops: 0000 [#1] SMP
[ 41.421852] Modules linked in: veth openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack libcrc32c 9p fscache ppdev kvm_intel kvm 9pnet_virtio irqbypass input_leds joydev 9pnet parport_pc serio_raw parport i2c_piix4 qemu_fw_cfg mac_hid sch_fq_codel ip_tables x_tables autofs4 btrfs xor raid6_pq psmouse virtio_blk virtio_net pata_acpi floppy
[ 41.423855] CPU: 0 PID: 5 Comm: kworker/u2:0 Tainted: G W 4.13.0-18-generic #21-Ubuntu
[ 41.424355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 41.424849] Workqueue: netns cleanup_net
[ 41.425071] task: db8fba80 task.stack: dba10000
[ 41.425346] EIP: igmp_group_dropped+0x21/0x220
[ 41.425656] EFLAGS: 00010202 CPU: 0
[ 41.425864] EAX: 00000000 EBX: dd726360 ECX: dba11e6c EDX: 00000002
[ 41.426335] ESI: 00000000 EDI: dd4db500 EBP: dba11dcc ESP: dba11d94
[ 41.426687] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 41.426990] CR0: 80050033 CR2: 00000000 CR3: 1e6d6d60 CR4: 000006f0
[ 41.427340] Call Trace:
[ 41.427485] ? __wake_up+0x36/0x40
[ 41.427680] ip_mc_down+0x27/0x90
[ 41.427869] inetdev_event+0x398/0x4e0
[ 41.428082] ? skb_dequeue+0x5b/0x70
[ 41.428286] ? wireless_nlevent_flush+0x4c/0x90
[ 41.428541] notifier_call_chain+0x4e/0x70
[ 41.428772] raw_notifier_call_chain+0x11/0x20
[ 41.429023] call_netdevice_notifiers_info+0x2a/0x60
[ 41.429301] dev_close_many+0x9d/0xe0
[ 41.429509] rollback_registered_many+0xd7/0x380
[ 41.429768] unregister_netdevice_many.part.102+0x10/0x80
[ 41.430075] default_device_exit_batch+0x134/0x160
[ 41.430344] ? do_wait_intr_irq+0x80/0x80
[ 41.430650] ops_exit_list.isra.8+0x4d/0x60
[ 41.430886] cleanup_net+0x18e/0x260
[ 41.431090] process_one_work+0x1a0/0x390
[ 41.431317] worker_thread+0x37/0x450
[ 41.431525] kthread+0xf3/0x110
[ 41.431714] ? process_one_work+0x390/0x390
[ 41.431941] ? kthread_create_on_node+0x20/0x20
[ 41.432187] ret_from_fork+0x19/0x24
[ 41.432382] Code: 90 90 90 90 90 90 90 90 90 90 3e 8d 74 26 00 55 89 e5 57 56 53 89 c3 83 ec 2c 8b 33 65 a1 14 00 00 00 89 45 f0 31 c0 80 7b 4b 00 <8b> 06 8b b8 20 03 00 00 8b 43 04 0f 85 5e 01 00 00 3d e0 00 00
[ 41.433405] EIP: igmp_group_dropped+0x21/0x220 SS:ESP: 0068:dba11d94
[ 41.433750] CR2: 0000000000000000
[ 41.433961] ---[ end trace 595db54cab84070c ]---
system then becomes unresponsive; no further interfaces can be created.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1736390/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list