[Bug 1882519] Re: Network partition - lost connectivity - ovs staleness?
Dan Ackerson
1882519 at bugs.launchpad.net
Mon Jun 8 11:40:47 UTC 2020
** Description changed:
Short description
====================================
- We have intermittent re-accurance of what appears to be OVS being dead.
- Clients loose connection to the networks (vxlan,gre) with no root cause and any visible sign of issue. "Lights are on but non-one is home"
+ We have intermittent reoccurance of what appears to be OVS being dead.
+ Clients lose connection to the networks (vxlan,gre) with no root cause and any visible sign of issue. "Lights are on but noone is home"
Steps to reproduce
===================
unknown
- Steps to reproduce
+ Steps to resolve
=================
restart the VM
- openstack server restart
+ `openstack server restart <id>`
- creating new vms are not impacted and work immediately
-
+ Newly created vms are not impacted and work immediately.
Sospreorts
===========
ubuntu at files.support.canonical.com
- sosreport-..............-00282744-2020-06-08-typrlwu.tar.xz
- sosreport-...........-00282744-2020-06-08-qdvfplm.tar.xz
+ sosreport-..............-00282744-2020-06-08-typrlwu.tar.xz
+ sosreport-...........-00282744-2020-06-08-qdvfplm.tar.xz
openvswitch
===========
dpkg --list | grep -i openv | awk '{print $2 " " $3 }'
neutron-openvswitch-agent 2:12.1.0-0ubuntu1~cloud0
neutron-plugin-openvswitch-agent 2:12.1.0-0ubuntu1~cloud0
openvswitch-common 2.9.5-0ubuntu0.18.04.1~cloud0
openvswitch-switch 2.9.5-0ubuntu0.18.04.1~cloud0
python-openvswitch 2.9.5-0ubuntu0.18.04.1~cloud0
qemu
====
dpkg --list | grep -i qemu | awk '{print $2 " " $3 }'
ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2.2~cloud0
ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2~cloud0
qemu 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-block-extra:amd64 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-kvm 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-slof 20151103+dfsg-1ubuntu1.1
qemu-system 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-arm 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-common 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-mips 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-misc 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-ppc 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-s390x 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-sparc 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-x86 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-user 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-user-binfmt 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-utils 1:2.11+dfsg-1ubuntu7.23~cloud0
-
Details
=======
openstack server show f0c5b261-73ed-43cc-bd5d-82d181680d87
+-------------------------------------+------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | hidden to protect customer |
| OS-EXT-SRV-ATTR:host | hidden to protect customer |
| OS-EXT-SRV-ATTR:hypervisor_hostname | hidden to protect customer |
| OS-EXT-SRV-ATTR:instance_name | instance-0000b9fc |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2020-01-28T13:16:46.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | DCS1-BO-SERVICES=172.16.15.22, xxxxxxxxxxxxxx |
-
VM interface 172.16.15.22
===============================
virsh domiflist instance-0000b9fc
Interface Type Source Model MAC
-------------------------------------------------------
tapba67f7c2-bc bridge br-int virtio fa:16:3e:b5:56:26
-
OVS stat info
=============
icmp, dns, dhcp, flows are visible
/snap/bin/ovs-stat -p results --host dcs1-clp-nod23 --tree
https://pastebin.canonical.com/p/NMZSRCqdPh/
-
Dead icmp
=========
ping 172.16.15.22
PING 172.16.15.22 (172.16.15.22) 56(84) bytes of data.
^C
--- 172.16.15.22 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1031ms
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to openvswitch in Ubuntu.
https://bugs.launchpad.net/bugs/1882519
Title:
Network partition - lost connectivity - ovs staleness?
Status in openvswitch package in Ubuntu:
New
Bug description:
Short description
====================================
We have intermittent reoccurance of what appears to be OVS being dead.
Clients lose connection to the networks (vxlan,gre) with no root cause and any visible sign of issue. "Lights are on but noone is home"
Steps to reproduce
===================
unknown
Steps to resolve
=================
restart the VM
`openstack server restart <id>`
Newly created vms are not impacted and work immediately.
Sospreorts
===========
ubuntu at files.support.canonical.com
sosreport-..............-00282744-2020-06-08-typrlwu.tar.xz
sosreport-...........-00282744-2020-06-08-qdvfplm.tar.xz
openvswitch
===========
dpkg --list | grep -i openv | awk '{print $2 " " $3 }'
neutron-openvswitch-agent 2:12.1.0-0ubuntu1~cloud0
neutron-plugin-openvswitch-agent 2:12.1.0-0ubuntu1~cloud0
openvswitch-common 2.9.5-0ubuntu0.18.04.1~cloud0
openvswitch-switch 2.9.5-0ubuntu0.18.04.1~cloud0
python-openvswitch 2.9.5-0ubuntu0.18.04.1~cloud0
qemu
====
dpkg --list | grep -i qemu | awk '{print $2 " " $3 }'
ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2.2~cloud0
ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2~cloud0
qemu 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-block-extra:amd64 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-kvm 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-slof 20151103+dfsg-1ubuntu1.1
qemu-system 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-arm 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-common 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-mips 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-misc 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-ppc 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-s390x 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-sparc 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-x86 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-user 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-user-binfmt 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-utils 1:2.11+dfsg-1ubuntu7.23~cloud0
Details
=======
openstack server show f0c5b261-73ed-43cc-bd5d-82d181680d87
+-------------------------------------+------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | hidden to protect customer |
| OS-EXT-SRV-ATTR:host | hidden to protect customer |
| OS-EXT-SRV-ATTR:hypervisor_hostname | hidden to protect customer |
| OS-EXT-SRV-ATTR:instance_name | instance-0000b9fc |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2020-01-28T13:16:46.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | DCS1-BO-SERVICES=172.16.15.22, xxxxxxxxxxxxxx |
VM interface 172.16.15.22
===============================
virsh domiflist instance-0000b9fc
Interface Type Source Model MAC
-------------------------------------------------------
tapba67f7c2-bc bridge br-int virtio fa:16:3e:b5:56:26
OVS stat info
=============
icmp, dns, dhcp, flows are visible
/snap/bin/ovs-stat -p results --host dcs1-clp-nod23 --tree
https://pastebin.canonical.com/p/NMZSRCqdPh/
Dead icmp
=========
ping 172.16.15.22
PING 172.16.15.22 (172.16.15.22) 56(84) bytes of data.
^C
--- 172.16.15.22 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1031ms
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1882519/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list