[Bug 1645324] Re: ebtables: Lock file handling has races
Eric Desrochers
eric.desrochers at canonical.com
Tue May 16 13:17:50 UTC 2017
** Description changed:
[Impact]
- * ebtables uses creation of a file with an exclusive flag
- as a lock to synchronize table modification when used
- with --concurrent parameter.
+ * ebtables uses creation of a file with an exclusive flag
+ as a lock to synchronize table modification when used
+ with --concurrent parameter.
- * If ebtables crashes it will leave a stale lock file.
- This will prevent another copy of ebtables from running,
- and indirectly any other service that depends on ebtables
- will also be affected.
+ * If ebtables crashes it will leave a stale lock file.
+ This will prevent another copy of ebtables from running,
+ and indirectly any other service that depends on ebtables
+ will also be affected.
- * This change adds support for real locks that get
- cleaned up if a process exits or crashes.
+ * This change adds support for real locks that get
+ cleaned up if a process exits or crashes.
[Test Case]
- * Test Case1:
- 1. $ sudo touch /var/lib/ebtables/lock"
- 2. $ sudo ebtables --concurrent -L
- 3. ebtables can't acquire a lock
+ * Test Case1:
+ 1. $ sudo touch /var/lib/ebtables/lock"
+ 2. $ sudo ebtables --concurrent -L
+ 3. ebtables can't acquire a lock
- * Test Case 2:
- 1. $ while true; do /usr/sbin/ebtables --concurrent -L; done
- 2. hard reboot VM
- 3. likely that the lock file is present under /var/lib/ebtables
- 4. libvird hanging, try to connect to qemu:///system
+ * Test Case 2:
+ 1. $ while true; do /usr/sbin/ebtables --concurrent -L; done
+ 2. hard reboot VM
+ 3. likely that the lock file is present under /var/lib/ebtables
+ 4. libvird hanging, try to connect to qemu:///system
+ [Regression Potential]
- [Regression Potential]
+ * Normal Use:
+ There is no regression potential during normal use and
+ operation of ebtables.
- * Normal Use:
- There is no regression potential during normal use and
- operation of ebtables.
+ * Package Upgrade:
+ There is a very very small regression potential during the package
+ upgrade to the latest version. Once the package is upgraded that
+ potential is gone. It is a very small potential because several
+ things have to happen in a very small time frame and in an exact
+ order since ebtables is not a resident program like a daemon:
+ 1. ebtables is launched during package upgrade AND
+ 2. new ebtables binary has not yet been written to disk AND
+ 3. it is launched with --concurent switch AND
+ 4. another ebtables with new binary is launched AND
+ 5. it is launched with --concurent switch AND
+ 6. the first ebtables copy hasn't exited yet AND
+ 7. both copies of ebtables are launched with a WRITE command AND
+ 8. both copies of ebtables are manipulating the same resource.
+ Then one of the binaries could potentially fail, but once the old
+ binary exits the potential is gone so subsequent re-runs of
+ ebtables will succeed.
- * Package Upgrade:
- There is a very very small regression potential during the package
- upgrade to the latest version. Once the package is upgraded that
- potential is gone. It is a very small potential because several
- things have to happen in a very small time frame and in an exact
- order since ebtables is not a resident program like a daemon:
- 1. ebtables is launched during package upgrade AND
- 2. new ebtables binary has not yet been written to disk AND
- 3. it is launched with --concurent switch AND
- 4. another ebtables with new binary is launched AND
- 5. it is launched with --concurent switch AND
- 6. the first ebtables copy hasn't exited yet AND
- 7. both copies of ebtables are launched with a WRITE command AND
- 8. both copies of ebtables are manipulating the same resource.
- Then one of the binaries could potentially fail, but once the old
- binary exits the potential is gone so subsequent re-runs of
- ebtables will succeed.
+ * Dragan's patch has been submitted to Debian via :
+ https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=860590
+
+ * Note that the ebtables upstream project is nearly dead. Nowadays, all
+ the development is now happening in nft project which is intended to be
+ replacement.
+
[Original Text]
libvirtd is hanging after startup due to ebtables lock file -from an earlier run- remains intact when the system reboots.
Same issue is happening than it is reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1290327 when the system boots.
After booting the system, It's not possible connect to the qemu-service.
- libvirt daemon tried to obtain a lock:
[pid 20966] read(24, "Trying to obtain lock /var/lib/e"..., 1024) = 45
[pid 20966] poll([{fd=22, events=POLLIN}, {fd=24, events=POLLIN}], 2, 4294967295) = 1 ([{fd=24, revents=POLLIN}])
[pid 20966] read(24, "Trying to obtain lock /var/lib/e"..., 1024) = 45
[pid 20966] poll([{fd=22, events=POLLIN}, {fd=24, events=POLLIN}], 2, 4294967295) = 1 ([{fd=24, revents=POLLIN}])
[pid 20966] read(24, "Trying to obtain lock /var/lib/e"..., 1024) = 45
[pid 20966] poll([{fd=22, events=POLLIN}, {fd=24, events=POLLIN}], 2, 4294967295^CProcess 20916 detached
- there was a file named 'lock' in /var/lib/ebtables directory with timestamp 14:54
- ebtables was configured:
* Ebtables support available, number of installed rules [ OK ]
(other nodes appeared to be in the same state from ebtables point of view, but without the lock file)
- I removed the lock file and libvirt started to work instantly - the lock obtain messages have disappeared from the trace and virsh commands are working
- at 14:54 the host was booting up. According to the logs, there were other reboots after that one, but the lock file remained intact (at least the timestamp was not updated).
Could you please suggest a solution to be sure that ebtables lock file
is removed during boot?
--
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1645324
Title:
ebtables: Lock file handling has races
Status in ebtables package in Ubuntu:
In Progress
Status in ebtables source package in Trusty:
In Progress
Status in ebtables source package in Zesty:
Triaged
Status in ebtables source package in Artful:
In Progress
Status in ebtables package in Debian:
New
Bug description:
[Impact]
* ebtables uses creation of a file with an exclusive flag
as a lock to synchronize table modification when used
with --concurrent parameter.
* If ebtables crashes it will leave a stale lock file.
This will prevent another copy of ebtables from running,
and indirectly any other service that depends on ebtables
will also be affected.
* This change adds support for real locks that get
cleaned up if a process exits or crashes.
[Test Case]
* Test Case1:
1. $ sudo touch /var/lib/ebtables/lock"
2. $ sudo ebtables --concurrent -L
3. ebtables can't acquire a lock
* Test Case 2:
1. $ while true; do /usr/sbin/ebtables --concurrent -L; done
2. hard reboot VM
3. likely that the lock file is present under /var/lib/ebtables
4. libvird hanging, try to connect to qemu:///system
[Regression Potential]
* Normal Use:
There is no regression potential during normal use and
operation of ebtables.
* Package Upgrade:
There is a very very small regression potential during the package
upgrade to the latest version. Once the package is upgraded that
potential is gone. It is a very small potential because several
things have to happen in a very small time frame and in an exact
order since ebtables is not a resident program like a daemon:
1. ebtables is launched during package upgrade AND
2. new ebtables binary has not yet been written to disk AND
3. it is launched with --concurent switch AND
4. another ebtables with new binary is launched AND
5. it is launched with --concurent switch AND
6. the first ebtables copy hasn't exited yet AND
7. both copies of ebtables are launched with a WRITE command AND
8. both copies of ebtables are manipulating the same resource.
Then one of the binaries could potentially fail, but once the old
binary exits the potential is gone so subsequent re-runs of
ebtables will succeed.
* Dragan's patch has been submitted to Debian via :
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=860590
* Note that the ebtables upstream project is nearly dead. Nowadays,
all the development is now happening in nft project which is intended
to be replacement.
[Original Text]
libvirtd is hanging after startup due to ebtables lock file -from an earlier run- remains intact when the system reboots.
Same issue is happening than it is reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1290327 when the system boots.
After booting the system, It's not possible connect to the qemu-service.
- libvirt daemon tried to obtain a lock:
[pid 20966] read(24, "Trying to obtain lock /var/lib/e"..., 1024) = 45
[pid 20966] poll([{fd=22, events=POLLIN}, {fd=24, events=POLLIN}], 2, 4294967295) = 1 ([{fd=24, revents=POLLIN}])
[pid 20966] read(24, "Trying to obtain lock /var/lib/e"..., 1024) = 45
[pid 20966] poll([{fd=22, events=POLLIN}, {fd=24, events=POLLIN}], 2, 4294967295) = 1 ([{fd=24, revents=POLLIN}])
[pid 20966] read(24, "Trying to obtain lock /var/lib/e"..., 1024) = 45
[pid 20966] poll([{fd=22, events=POLLIN}, {fd=24, events=POLLIN}], 2, 4294967295^CProcess 20916 detached
- there was a file named 'lock' in /var/lib/ebtables directory with timestamp 14:54
- ebtables was configured:
* Ebtables support available, number of installed rules [ OK ]
(other nodes appeared to be in the same state from ebtables point of view, but without the lock file)
- I removed the lock file and libvirt started to work instantly - the lock obtain messages have disappeared from the trace and virsh commands are working
- at 14:54 the host was booting up. According to the logs, there were other reboots after that one, but the lock file remained intact (at least the timestamp was not updated).
Could you please suggest a solution to be sure that ebtables lock file
is removed during boot?
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ebtables/+bug/1645324/+subscriptions
More information about the Ubuntu-sponsors
mailing list