[Bug 1277157] Re: libvirtd restart due to assertion failure in libnl
Chris J Arges
1277157 at bugs.launchpad.net
Fri Feb 14 15:21:58 UTC 2014
This bug is related to: https://bugzilla.redhat.com/show_bug.cgi?id=886454
The patch that addresses this issue is 9aadccd57ef2ce3769475f52bc1c30cd689c8085 in netcf. This patch is present in saucy versions of netcf and beyond.
** Also affects: libnl3 (Ubuntu Precise)
Importance: Undecided
Status: New
** Also affects: libnl3 (Ubuntu Quantal)
Importance: Undecided
Status: New
** Changed in: libnl3 (Ubuntu Precise)
Assignee: (unassigned) => Chris J Arges (arges)
** Changed in: libnl3 (Ubuntu Quantal)
Assignee: (unassigned) => Chris J Arges (arges)
** Changed in: libnl3 (Ubuntu Precise)
Importance: Undecided => High
** Changed in: libnl3 (Ubuntu Quantal)
Importance: Undecided => High
** Changed in: libnl3 (Ubuntu Precise)
Status: New => In Progress
** Changed in: libnl3 (Ubuntu Quantal)
Status: New => In Progress
** Changed in: libnl3 (Ubuntu)
Status: In Progress => Fix Released
** Changed in: libnl3 (Ubuntu)
Importance: High => Undecided
** Changed in: libnl3 (Ubuntu)
Assignee: Chris J Arges (arges) => (unassigned)
** Description changed:
Running a multiple-domain start/destroy loop, libvirtd 1.1.1 restarts
occasionally due to assertion failure in libnl.
+ This affects both precise/quantal versions of netcf.
+
Distibution version: Ubuntu 12.04.2 LTS
- Kernel: 3.5.0-41-generic #64~lp1241764v201312091434 SMP Mon Dec 9 20:35:04 UTC 2013 x86_64
+ Kernel: 3.5.0-41-generic #64 SMP Mon Dec 9 20:35:04 UTC 2013 x86_64
Libvirt: libvirt_1.1.1-0ubuntu8.2~cloud1 from cloud repository.
Qemu-kvm: qemu-kvm_1.0+noroms-0ubuntu14.12 [qemu 1.5 not yet tested]
libnl: libnl-3-200_3.2.3-2ubuntu2
Platform: HP ProLiant SL390s G7 x86_64; 2 socket x 6 cores/socket x 2 HT/core; 96GB
How to reproduce:
1. Define a number of test domains. E.g., test-nn, nn from 01 .. NN. I
use 32
Test domains have 4 vcpu, 8G memory, running an Ubuntu 11.04 image.
Network is libvirt default virtual network. Domain mac addresses based
on test number nn and dhcp serves fixed IP addresses based on mac
address.
2. In a loop:
2a. start the NN domains serially -- waiting for the "virsh start" command to complete before starting next domain.
2b. sleep 20sec -- give domains some time to boot. Test doesn't check that domains have completed booting.
2c. destroy the NN domains "in parallel" -- with "virsh destroy test-$nn &"
2d. sleep 15secs.
sleep times are more or less arbitrary. Next pass of starts does
usually begin before previous pass of destroys completes, but I've never
seen it say that "domain <name> already running" or such.
What I expected to happen: start-destroy loop runs indefinitely without
error.
What happened instead: Eventually, I start seeing errors like:
error: Failed to destroy domain lnvtest-31
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the hypervisor
and
error: failed to get domain 'lnvtest-12'
error: Domain not found: no domain with matching name 'lnvtest-12'
Checking the libvirt debug log, I see that libvirtd restarted during
this iteration, dumping its internal log buffer in the process.
gdb traceback shows:
- #3 0x00007f845cafb192 in __GI___assert_fail (assertion=0x7f845c4b89bd "0", file=0x7f845c4b8538 "/build/buildd/libnl3-3.2.3/./lib/object.c",
- line=185, function=0x7f845c4b8668 "nl_object_put") at assert.c:103
+ #3 0x00007f845cafb192 in __GI___assert_fail (assertion=0x7f845c4b89bd "0", file=0x7f845c4b8538 "/build/buildd/libnl3-3.2.3/./lib/object.c",
+ line=185, function=0x7f845c4b8668 "nl_object_put") at assert.c:103
#4 0x00007f845c4b4dea in nl_object_put () from /lib/libnl-3.so.200
#5 0x00007f845c4afb92 in nl_cache_remove () from /lib/libnl-3.so.200
#6 0x00007f845c4b4b07 in nl_object_free () from /lib/libnl-3.so.200
#7 0x00007f845c4b4b15 in nl_object_free () from /lib/libnl-3.so.200
#8 0x00007f845c4afb92 in nl_cache_remove () from /lib/libnl-3.so.200
#9 0x00007f845c4afd0b in nl_cache_clear () from /lib/libnl-3.so.200
#10 0x00007f845c4afd3e in nl_cache_free () from /lib/libnl-3.so.200
#11 0x00007f84517f3096 in ?? () from /usr/lib/libnetcf.so.1
#12 0x00007f84517f4220 in ?? () from /usr/lib/libnetcf.so.1
#13 0x00007f84517ef53f in ncf_close () from /usr/lib/libnetcf.so.1
#14 0x00007f8451a0be6f in netcfInterfaceClose (conn=0x7f83f00e7ad0) at /tmp/buildd/libvirt-1.1.1/./src/interface/interface_backend_netcf.c:197
#15 0x00007f845d19d224 in virConnectDispose (obj=0x7f83f00e7ad0) at /tmp/buildd/libvirt-1.1.1/./src/datatypes.c:149
#16 0x00007f845d1246bb in virObjectUnref (anyobj=<optimized out>) at /tmp/buildd/libvirt-1.1.1/./src/util/virobject.c:262
#17 0x00007f845d1a6e2f in virConnectClose (conn=0x7f83f00e7ad0) at /tmp/buildd/libvirt-1.1.1/./src/libvirt.c:1510
#18 0x00007f845db46581 in remoteClientFreeFunc (data=<optimized out>) at /tmp/buildd/libvirt-1.1.1/./daemon/remote.c:683
#19 0x00007f845d20f362 in virNetServerClientDispose (obj=<optimized out>) at /tmp/buildd/libvirt-1.1.1/./src/rpc/virnetserverclient.c:911
#20 0x00007f845d1246bb in virObjectUnref (anyobj=<optimized out>) at /tmp/buildd/libvirt-1.1.1/./src/util/virobject.c:262
#21 0x00007f845d21781d in virNetSocketEventFree (opaque=<optimized out>) at /tmp/buildd/libvirt-1.1.1/./src/rpc/virnetsocket.c:1714
#22 0x00007f845d108b29 in virEventPollCleanupHandles () at /tmp/buildd/libvirt-1.1.1/./src/util/vireventpoll.c:580
#23 0x00007f845d1096e3 in virEventPollRunOnce () at /tmp/buildd/libvirt-1.1.1/./src/util/vireventpoll.c:616
#24 0x00007f845d1086ad in virEventRunDefaultImpl () at /tmp/buildd/libvirt-1.1.1/./src/util/virevent.c:273
#25 0x00007f845d20ecfd in virNetServerRun (srv=0x7f845e818510) at /tmp/buildd/libvirt-1.1.1/./src/rpc/virnetserver.c:1096
#26 0x00007f845db2445e in main (argc=<optimized out>, argv=<optimized out>) at /tmp/buildd/libvirt-1.1.1/./daemon/libvirtd.c
BUG() at line 185 in nl_object_put(); reference count going negative:
171 /**
172 * Release a reference from an object
173 * @arg obj object to release reference from
174 */
175 void nl_object_put(struct nl_object *obj)
176 {
177 if (!obj)
178 return;
- 179
+ 179
180 obj->ce_refcnt--;
181 NL_DBG(4, "Returned object reference %p, %d remaining\n",
182 obj, obj->ce_refcnt);
- 183
+ 183
184 if (obj->ce_refcnt < 0)
185 BUG();
- 186
+ 186
187 if (obj->ce_refcnt <= 0)
188 nl_object_free(obj);
189 }
I hope this is sufficient information.
** Also affects: netcf (Ubuntu)
Importance: Undecided
Status: New
** No longer affects: libnl3 (Ubuntu)
** Changed in: netcf (Ubuntu)
Status: New => Fix Released
** Changed in: netcf (Ubuntu Precise)
Assignee: (unassigned) => Chris J Arges (arges)
** Changed in: netcf (Ubuntu Quantal)
Assignee: (unassigned) => Chris J Arges (arges)
** No longer affects: libnl3 (Ubuntu Precise)
** No longer affects: libnl3 (Ubuntu Quantal)
** Changed in: netcf (Ubuntu Precise)
Status: New => In Progress
** Changed in: netcf (Ubuntu Quantal)
Status: New => In Progress
** Changed in: netcf (Ubuntu Precise)
Importance: Undecided => High
** Changed in: netcf (Ubuntu Quantal)
Importance: Undecided => High
** Bug watch added: Red Hat Bugzilla #886454
https://bugzilla.redhat.com/show_bug.cgi?id=886454
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to libnl3 in Ubuntu.
https://bugs.launchpad.net/bugs/1277157
Title:
libvirtd restart due to assertion failure in libnl
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/netcf/+bug/1277157/+subscriptions
More information about the Ubuntu-server-bugs
mailing list