[Bug 1829829] Re: Ubuntu CI has been flaky for a week
Evgeny Vereshchagin
evvers at ya.ru
Tue May 21 20:15:44 UTC 2019
To judge from https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic-upstream-systemd-ci-systemd-ci/bionic/i386/s/systemd-upstream/20190521_185445_b37c3@/log.gz, VMs seems to also be throwing kernel panics:
```
[ 0.894903] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[ 0.895921] EDD information not available.
[ 0.985183] isapnp: No Plug & Play device found
[ 0.986013] md: Waiting for all devices to be available before autodetect
[ 0.987166] md: If you don't use raid, use raid=noautodetect
[ 0.988330] md: Autodetecting RAID arrays.
[ 0.989038] md: autorun ...
[ 0.989525] md: ... autorun DONE.
[ 0.990139] VFS: Cannot open root device "UUID=961ce78b-c06e-4632-9aa0-56a8356b081b" or unknown-block(0,0): error -6
[ 0.991923] Please append a correct "root=" boot option; here are the available partitions:
[ 0.993406] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 0.994818] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.15.0-50-generic #54-Ubuntu
[ 0.996121] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.10.1-1ubuntu1~cloud0 04/01/2014
[ 0.997704] Call Trace:
[ 0.998138] dump_stack+0x58/0x81
[ 0.998713] panic+0x94/0x1e6
[ 0.999233] mount_block_root+0x199/0x234
[ 1.000111] ? calibrate_delay+0x190/0x4e0
[ 1.000833] ? create_dev.constprop.7+0x3a/0x53
[ 1.001611] mount_root+0x33/0x35
[ 1.002182] prepare_namespace+0x11b/0x167
[ 1.002885] kernel_init_freeable+0x1b3/0x1c5
[ 1.003634] ? rest_init+0xa0/0xa0
[ 1.004238] kernel_init+0xd/0x100
[ 1.004830] ret_from_fork+0x1c/0x38
[ 1.005892] Kernel Offset: 0x10000000 from 0xc1000000 (relocation range: 0xc0000000-0xdf5fdfff)
[ 1.007376] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
```
And there is another one at https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic-upstream-systemd-ci-systemd-ci/bionic/i386/s/systemd-upstream/20190521_100015_69099@/log.gz
```
[ 41.006515] IP: rollback_registered_many+0x141/0x390
[ 41.007872] *pdpt = 000000001da0b001 *pde = 0000000000000000
[ 41.009424] Oops: 0002 [#1] SMP PTI
[ 41.010396] Modules linked in: kvm_intel kvm irqbypass crc32_pclmul pcbc aesni_intel aes_i586 crypto_simd cryptd joydev input_leds serio_raw mac_hid qemu_fw_cfg xt_TCPMSS xt_tcpudp iptable_mangle ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq psmouse pata_acpi virtio_blk virtio_net i2c_piix4 floppy
[ 41.017754] CPU: 0 PID: 85 Comm: kworker/u2:2 Not tainted 4.15.0-50-generic #54-Ubuntu
[ 41.019883] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.10.1-1ubuntu1~cloud0 04/01/2014
[ 41.022385] Workqueue: netns cleanup_net
[ 41.023459] EIP: rollback_registered_many+0x141/0x390
[ 41.024825] EFLAGS: 00010202 CPU: 0
[ 41.025783] EAX: 00000008 EBX: 00000008 ECX: 00000020 EDX: 00000000
[ 41.027477] ESI: dae31ec4 EDI: 00ffffff EBP: dae31eac ESP: dae31e88
[ 41.029162] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 41.030641] CR0: 80050033 CR2: dfc746c0 CR3: 1da28000 CR4: 001406f0
[ 41.032333] Call Trace:
[ 41.033030] ? netdev_run_todo+0x4a/0x300
[ 41.034136] unregister_netdevice_many.part.104+0x10/0x80
[ 41.035603] default_device_exit_batch+0x13c/0x160
[ 41.036933] ? do_wait_intr_irq+0x80/0x80
[ 41.038034] ops_exit_list.isra.8+0x53/0x60
[ 41.039169] cleanup_net+0x18e/0x260
[ 41.040152] process_one_work+0x1b9/0x3d0
[ 41.041246] worker_thread+0x37/0x420
[ 41.042254] kthread+0xf0/0x110
[ 41.043115] ? process_one_work+0x3d0/0x3d0
[ 41.044256] ? kthread_create_worker_on_cpu+0x20/0x20
[ 41.045640] ret_from_fork+0x2e/0x38
[ 41.046629] Code: ce d6 89 d8 03 0c 9d c0 34 bc d6 e8 5a d5 92 ff 89 d8 ba a0 e2 ce d6 e8 de de 11 00 3b 05 fc e3 ce d6 89 c3 72 d4 bf ff ff ff 00 <00> 00 00 00 00 00 00 00 00 01 00 ff ff ff ff ff ff fa 16 3e bb
[ 41.051663] EIP: rollback_registered_many+0x141/0x390 SS:ESP: 0068:dae31e88
[ 41.053543] CR2: 0000000000000008
[ 41.054464] ---[ end trace 06d0534c6301b91d ]---
[ 41.057078] BUG: unable to handle kernel NULL pointer dereference at 00000000
[ 41.059059] IP: flush_backlog+0x3f/0x120
[ 41.060132] *pdpt = 000000001da0b001 *pde = 0000000000000000
[ 41.061682] Oops: 0000 [#2] SMP PTI
[ 41.062649] Modules linked in: kvm_intel kvm irqbypass crc32_pclmul pcbc aesni_intel aes_i586 crypto_simd cryptd joydev input_leds serio_raw mac_hid qemu_fw_cfg xt_TCPMSS xt_tcpudp iptable_mangle ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq psmouse pata_acpi virtio_blk virtio_net i2c_piix4 floppy
[ 41.069990] CPU: 0 PID: 149 Comm: kworker/0:1H Tainted: G D 4.15.0-50-generic #54-Ubuntu
[ 41.072469] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.10.1-1ubuntu1~cloud0 04/01/2014
[ 41.074954] Workqueue: events_highpri flush_backlog
[ 41.076266] EIP: flush_backlog+0x3f/0x120
[ 41.077346] EFLAGS: 00010046 CPU: 0
[ 41.078609] EAX: 00000000 EBX: dbc98a00 ECX: dbc98aac EDX: 00000000
[ 41.080285] ESI: dbc97dc0 EDI: dbc98aa0 EBP: dafeff38 ESP: dafeff28
[ 41.081992] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 41.083447] CR0: 80050033 CR2: 00000000 CR3: 1da28000 CR4: 001406f0
[ 41.085131] Call Trace:
[ 41.085817] process_one_work+0x1b9/0x3d0
[ 41.086904] worker_thread+0x37/0x420
[ 41.087901] kthread+0xf0/0x110
[ 41.088758] ? process_one_work+0x3d0/0x3d0
[ 41.089908] ? kthread_create_worker_on_cpu+0x20/0x20
[ 41.091274] ret_from_fork+0x2e/0x38
[ 41.092245] Code: d6 00 02 00 00 bb 00 9a e1 d6 64 03 1d 28 91 e0 d6 fa 90 8d 74 26 00 8d 83 ac 00 00 00 8d bb a0 00 00 00 89 45 f0 e8 21 cb 13 00 <8b> 00 00 00 00 00 00 00 00 00 00 01 00 ff ff ff ff ff ff fa 16
[ 41.101806] EIP: flush_backlog+0x3f/0x120 SS:ESP: 0068:dafeff28
[ 41.103393] CR2: 0000000000000000
[ 41.104296] ---[ end trace 06d0534c6301b91e ]---
[ 41.105543] Kernel panic - not syncing: Fatal exception in interrupt
[ 41.107486] Kernel Offset: 0x15000000 from 0xc1000000 (relocation range: 0xc0000000-0xdf5fdfff)
[ 41.109805] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
```
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1829829
Title:
Ubuntu CI has been flaky for a week
Status in systemd package in Ubuntu:
New
Bug description:
It was originally reported in https://github.com/systemd/systemd/pull/12583#issuecomment-492949206 5 days ago. To judge from the logs VMs can't be rebooted there:
```
Ubuntu 18.04.2 LTS autopkgtest ttyS0
autopkgtest login:
---------------------------------------------------
------- nova show 91e76a78-d05c-412a-b383-55a26010ae69 (adt-bionic-amd64-systemd-upstream-20190516-051604) ------
+--------------------------------------+------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | euler |
| OS-EXT-SRV-ATTR:hypervisor_hostname | euler.lcy01.scalingstack |
| OS-EXT-SRV-ATTR:instance_name | instance-003d216a |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-05-16T07:00:42.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2019-05-16T07:00:33Z |
| flavor | autopkgtest (f878e70e-9991-46e0-ba02-8ea159a71656) |
| hostId | 1722c5f2face86c3fc9f338ae96835924721512372342f664e6941bd |
| id | 91e76a78-d05c-412a-b383-55a26010ae69 |
| image | adt/ubuntu-bionic-amd64-server-20190516.img (d00bf12c-467e-433f-a4f5-15720f13bff1) |
| key_name | testbed-juju-prod-ues-proposed-migration-machine-11 |
| metadata | {} |
| name | adt-bionic-amd64-systemd-upstream-20190516-051604 |
| net_ues_proposed_migration network | 10.42.40.13 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | autopkgtest at lcy01-27.secgroup |
| status | ACTIVE |
| tenant_id | afaef86b96dd4828a1ed5ee395ea1421 |
| updated | 2019-05-16T07:00:42Z |
| user_id | 8524250971084851b3792a68fbc398dd |
+--------------------------------------+------------------------------------------------------------------------------------+
---------------------------------------------------
<VirtSubproc>: failure: Timed out on waiting for ssh connection
autopkgtest [07:07:45]: ERROR: testbed failure: cannot send to testbed: [Errno 32] Broken pipe
```
Though judging by https://github.com/systemd/systemd/pull/12626, it
appears that sometimes the tests pass.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1829829/+subscriptions
More information about the foundations-bugs
mailing list