[Bug 1789659] Re: libvirt-daemon error "virHashSearch:727 : Hash operation not allowed during iteration"
Christian Ehrhardt
1789659 at bugs.launchpad.net
Fri Sep 28 06:44:36 UTC 2018
@Anish - it fixes the first of your messages in comment #8
The second is yet unknown/undebugged to me - in case you hit that still regularly I'd need an extra bug with much more information. The info in this bug here was about the hash issue that is now fixed.
@Michael - I had no intention to bring that to 3.6 as that was in Ubuntu Artful which is now out of service. Are you referring to the 3.6 version that is in UCA [1] Pike for Xenial?
I'm adding a task for the cloud archive to consider picking this up as well, but will set it to incomplete. If this really is the "target" you have thought of set it to "new".
[1]: https://wiki.ubuntu.com/OpenStack/CloudArchive#Queens
** Also affects: cloud-archive
Importance: Undecided
Status: New
** Changed in: cloud-archive
Status: New => Incomplete
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1789659
Title:
libvirt-daemon error "virHashSearch:727 : Hash operation not allowed
during iteration"
Status in Ubuntu Cloud Archive:
Incomplete
Status in libvirt package in Ubuntu:
Fix Released
Status in libvirt source package in Bionic:
Fix Released
Bug description:
[Impact]
* Libvirt does an overzealous check on concurrent hash usage which breaks
some automation like for example Terraform. Upstream moved the need to
lock up the stack where applicable and dropped the checks as they were
superfluous.
* Backport the upstream change dropping the extra checks on hash
usage
[Test Case]
* Run the attached script test-lp1789659.sh
(based on ideas of https://bugzilla.redhat.com/show_bug.cgi?id=1581364#c7)
With the aforementioned patch, this error is not produced any more.
[Regression Potential]
* If the assumption that all remaining callers are safe isn't correct
there could be code using invalid hashes accessing guest information.
This needs thorough tests and regression checks as well as an extra
review if it can be applied to the libvirt 4.0 we have in Bionic.
This was done by me and also upstream did the same.
Quote from the patch: "Most important uses seem to be covered. Callers
have now a greater responsability, notably the ability to execute some
operations while iterating were reliably forbidden before are now
accepted."
It is important to see that the internal locking was inherently racy
(boolean) so we are replaceing a broken lock-ckeck with no check as
upper layers were supposed to do it correctly. That is true for what is
in the archive but there could be out-of archive things not doing so
calling directly into the python api bindings for example.
Never the less those are today just as affected by the locking being
broken, so those do not "loose" a lot.
All major exploiters we know of are good - the rest is a regression
risk we are willing to take in favor of fixing all the valid use cases
being affected by the broken lock handling.
[Other Info]
* n/a
----
In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2],
a tool to automate infrastructure deployments, together with
terraform-provider-libvirt[3], a Terraform provider (plugin) that
allow interaction with libvirt.
By default, this setup will issue several libvirt API calls
concurrently in an hard-to-predict order, as demonstrated by logs that
I am attaching.
In the "bad log" example, the following line appears:
error : virHashSearch:727 : Hash operation not allowed during
iteration
According to analysis of a similar problem by Red Hat in an OpenStack
scenario[4], this is has been fixed in upstream libvirt via commit
4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].
I tested the patch and it applies cleanly to the 4.0.0 package
shipping with Bionic and that successfully resolves this issue.
Please evaluate including this patch.
Thanks in advance
[1] http://uyuni-project.org/
[2] https://www.terraform.io/
[3] https://github.com/dmacvicar/terraform-provider-libvirt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
[5] https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt-daemon 4.0.0-1ubuntu8.3
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
Uname: Linux 4.15.0-33-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 29 16:05:10 2018
InstallationDate: Installed on 2014-06-12 (1539 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: libvirt
UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago)
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1789659/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list