[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state

[Mohammed Naser]

I have done some extensive research on this and I am able to confirm
that while Bug #1864864 does solve a class of issues, it does not solve
the one affecting Open vSwitch, as it does rely on
PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP:

https://github.com/openvswitch/ovs/blob/0de1b425962db073ebbaa3ddbde445580afda840/lib
/ovs-thread.c#L247-L248

And when running the reproducer here:

https://sourceware.org/bugzilla/show_bug.cgi?id=23861

By doing the following:

```
curl https://sourceware.org/bugzilla/attachment.cgi?id=11382 -o bug23861.c
sed -i 's/do_exit = 0/do_exit(0)/' bug23861.c 
g++ bug23861.c -lpthread -o bug23861
for ((x=1;x<100;x++)) ; do echo $x;date;./bug23861 --prefer-writer-nonrecursive;done
```

I will add that in my initial tests of doing this inside a small VM with
a Docker container, it did not hang after running for 10-15 minutes, but
it immediately hung on a much more powerful machine.  I will file a bug
to try and get the patch above SRU'd into glibc as well and hopefully
that will be the end of it.

** Bug watch added: Sourceware.org Bugzilla #23861
   https://sourceware.org/bugzilla/show_bug.cgi?id=23861

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to openvswitch in Ubuntu.
https://bugs.launchpad.net/bugs/1839592

Title:
  Open vSwitch (Version 2.9.2) goes into deadlocked state

Status in glibc package in Ubuntu:
  Confirmed
Status in openvswitch package in Ubuntu:
  Invalid

Bug description:
  Description:	Ubuntu 18.04.2 LTS
  Release:	18.04

  root at kv02:~# apt-cache policy openvswitch-common
  openvswitch-common:
    Installed: 2.9.2-0ubuntu0.18.04.3
    Candidate: 2.9.2-0ubuntu0.18.04.3
    Version table:
   *** 2.9.2-0ubuntu0.18.04.3 500

  
  Randomly ovs-vswitchd service is locked waiting for handler thread to quiesce. Here is the log tail:

  ovs-vswitchd.log:
  2019-08-07T04:25:01.687Z|00101|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
  2019-08-07T08:48:23.885Z|00012|ovs_rcu(urcu5)|WARN|blocked 1000 ms waiting for revalidator127 to quiesce
  2019-08-07T08:48:24.884Z|00102|ovs_rcu|WARN|blocked 1000 ms waiting for revalidator127 to quiesce
  2019-08-07T08:48:24.885Z|00013|ovs_rcu(urcu5)|WARN|blocked 2000 ms waiting for revalidator127 to quiesce
  2019-08-07T08:48:25.883Z|00103|ovs_rcu|WARN|blocked 2000 ms waiting for revalidator127 to quiesce
  2019-08-07T08:48:26.886Z|00014|ovs_rcu(urcu5)|WARN|blocked 4001 ms waiting for revalidator127 to quiesce
  2019-08-07T08:48:27.884Z|00104|ovs_rcu|WARN|blocked 4000 ms waiting for revalidator127 to quiesce
  2019-08-07T08:48:30.886Z|00015|ovs_rcu(urcu5)|WARN|blocked 8001 ms waiting for revalidator127 to quiesce
  2019-08-07T08:48:31.883Z|00105|ovs_rcu|WARN|blocked 8000 ms waiting for revalidator127 to quiesce
  2019-08-07T08:48:38.886Z|00016|ovs_rcu(urcu5)|WARN|blocked 16001 ms waiting for revalidator127 to quiesce
  2019-08-07T08:48:39.883Z|00106|ovs_rcu|WARN|blocked 16000 ms waiting for revalidator127 to quiesce
  2019-08-07T08:48:54.885Z|00017|ovs_rcu(urcu5)|WARN|blocked 32000 ms waiting for revalidator127 to quiesce
  2019-08-07T08:48:55.883Z|00107|ovs_rcu|WARN|blocked 32000 ms waiting for revalidator127 to quiesce
  2019-08-07T08:49:26.885Z|00018|ovs_rcu(urcu5)|WARN|blocked 64000 ms waiting for revalidator127 to quiesce
  2019-08-07T08:49:27.883Z|00108|ovs_rcu|WARN|blocked 64000 ms waiting for revalidator127 to quiesce
  2019-08-07T08:50:30.885Z|00019|ovs_rcu(urcu5)|WARN|blocked 128000 ms waiting for revalidator127 to quiesce
  2019-08-07T08:50:31.883Z|00109|ovs_rcu|WARN|blocked 128000 ms waiting for revalidator127 to quiesce
  2019-08-07T08:52:38.885Z|00020|ovs_rcu(urcu5)|WARN|blocked 256000 ms waiting for revalidator127 to quiesce

  This cause the commands blocks and does not generate any output.

  ovs-ofctl show sw0

  ovs-ofctl dump-flows sw0

  Is necesary restart ovs-vswitchd service, to recover it.

  Related https://github.com/openvswitch/ovs-issues/issues/153

  ¿You are planning updating openvswitch service to 2.9.5 version?

  https://www.openvswitch.org/releases/NEWS-2.9.5.txt

  Ubuntu 18.04 is stuck in 2.9.2.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions