[Bug 1789659] Re: libvirt-daemon error "virHashSearch:727 : Hash operation not allowed during iteration"

[ Christian Ehrhardt ]

@Govinda - the remaining issue you mention is unrelated to the one we discussed and fixed here.
For everybody else that wants to follow, Govinda file bug 1794997 about that (Thanks).

@Marc - ok then all our bug tagging is correct, lets see what they say.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1789659

Title:
   libvirt-daemon error "virHashSearch:727 : Hash operation not allowed
  during iteration"

Status in Ubuntu Cloud Archive:
  New
Status in libvirt package in Ubuntu:
  Fix Released
Status in libvirt source package in Bionic:
  Fix Released

Bug description:
  [Impact]

   * Libvirt does an overzealous check on concurrent hash usage which breaks
     some automation like for example Terraform. Upstream moved the need to
     lock up the stack where applicable and dropped the checks as they were
     superfluous.

   * Backport the upstream change dropping the extra checks on hash
  usage

  [Test Case]

   * Run the attached script test-lp1789659.sh
    (based on ideas of https://bugzilla.redhat.com/show_bug.cgi?id=1581364#c7)
   
  With the aforementioned patch, this error is not produced any more.

  [Regression Potential]

   * If the assumption that all remaining callers are safe isn't correct
     there could be code using invalid hashes accessing guest information.
     This needs thorough tests and regression checks as well as an extra
     review if it can be applied to the libvirt 4.0 we have in Bionic.
     This was done by me and also upstream did the same.
     Quote from the patch: "Most important uses seem to be covered. Callers
     have now a greater responsability, notably the ability to execute some
     operations while iterating were reliably forbidden before are now
     accepted."
     It is important to see that the internal locking was inherently racy
     (boolean) so we are replaceing a broken lock-ckeck with no check as
     upper layers were supposed to do it correctly. That is true for what is
     in the archive but there could be out-of archive things not doing so
     calling directly into the python api bindings for example.
     Never the less those are today just as affected by the locking being
     broken, so those do not "loose" a lot.
     All major exploiters we know of are good - the rest is a regression
     risk we are willing to take in favor of fixing all the valid use cases
     being affected by the broken lock handling.

  [Other Info]

   * n/a

  ----

  In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2],
  a tool to automate infrastructure deployments, together with
  terraform-provider-libvirt[3], a Terraform provider (plugin) that
  allow interaction with libvirt.

  By default, this setup will issue several libvirt API calls
  concurrently in an hard-to-predict order, as demonstrated by logs that
  I am attaching.

  In the "bad log" example, the following line appears:

  error : virHashSearch:727 : Hash operation not allowed during
  iteration

  According to analysis of a similar problem by Red Hat in an OpenStack
  scenario[4], this is has been fixed in upstream libvirt via commit
  4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].

  I tested the patch and it applies cleanly to the 4.0.0 package
  shipping with Bionic and that successfully resolves this issue.

  Please evaluate including this patch.

  Thanks in advance

  [1] http://uyuni-project.org/
  [2] https://www.terraform.io/
  [3] https://github.com/dmacvicar/terraform-provider-libvirt
  [4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
  [5] https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch

  ProblemType: Bug
  DistroRelease: Ubuntu 18.04
  Package: libvirt-daemon 4.0.0-1ubuntu8.3
  ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
  Uname: Linux 4.15.0-33-generic x86_64
  NonfreeKernelModules: nvidia_modeset nvidia
  ApportVersion: 2.20.9-0ubuntu7.2
  Architecture: amd64
  CurrentDesktop: ubuntu:GNOME
  Date: Wed Aug 29 16:05:10 2018
  InstallationDate: Installed on 2014-06-12 (1539 days ago)
  InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
  SourcePackage: libvirt
  UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago)

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1789659/+subscriptions