[Bug 1583009] Re: Error starting domain since update

[ChristianEhrhardt]

I still think this is a "new" issue and we should keep the old one fix
released and upen a new bug, but for a minor discussion here an update:

If the socket is not around usually the service isn't running.
Which would make sense for the workaround in comment #14 that essentially restarted things.
But your examples show the socket still opened by the PID, just not available at the path.

I wonder if in your cases some part of the upgrade procedure eliminated
the directories (and recreated them) but leaving the old sockets open.

That might explain why you see no file via "ls" but you see one in lsof.

I tried to take a Trusty and upgrade to UCA-Mitaka as mentioned as one of the triggering cases.
Any virsh command should do, but using the net-edit as reported int he comments to be sure.

Before:
$ service libvirt-bin status
libvirt-bin start/running, process 5305
$ virsh list
 Id    Name                           State
----------------------------------------------------
 2     kvmtest                        running
$ ls -laF /var/run/libvirt/libvirt-sock
srwxrwx--- 1 root libvirtd 0 Aug  7 09:23 /var/run/libvirt/libvirt-sock=
$ lsof /var/run/libvirt/libvirt-sock
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
libvirtd 5305 root   12u  unix 0xffff880152537c00      0t0 13001500 /var/run/libvirt/libvirt-sock

Ok all seems normal before the upgrade, service is running and working.
The socket is around in file-system and owned by the service's PID.

Now upgrading to UCA-Mitaka - the upgrade worked and lifted the versions to current Mitaka
(at the moment 1.3.1-1ubuntu10.11~cloud0).

$ service libvirt-bin status
libvirt-bin start/running, process 7266
$ virsh list
 Id    Name                           State
----------------------------------------------------
 2     kvmtest                        running
$ ls -laF /var/run/libvirt/libvirt-sock
srwxrwx--- 1 root libvirtd 0 Aug  7 09:33 /var/run/libvirt/libvirt-sock=
$ lsof /var/run/libvirt/libvirt-sock
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
libvirtd 7266 root   12u  unix 0xffff88001d404c00      0t0 13031405 /var/run/libvirt/libvirt-sock


Ok, so a "normal" upgrade does not trigger this in general.
We'd need someone who is in the error state to debug what happened to push his system into this state.

@Jeff (and others) - you seem to have the socket still open by the process, but not in the path.
Are there any mounts over that path that might hide them?
Anything on the upgrade output that might indicate a failed restart or something like it?

Issue could be something like:
# ls -l /var/run/libvirt/lib*
srwxrwx--- 1 root libvirtd 0 Aug  7 09:33 /var/run/libvirt/libvirt-sock
srwxrwxrwx 1 root libvirtd 0 Aug  7 09:33 /var/run/libvirt/libvirt-sock-ro
# mkdir /tmp/test
# mount -o bind /tmp/test /var/run/libvirt
# ls -l /var/run/libvirt/lib*
ls: cannot access /var/run/libvirt/lib*: No such file or directory
# virsh list
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

Also the following would lead to such a case:
# rm /var/run/libvirt/libvirt-sock
# virsh list
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
# lsof -p 7266 | grep -- libvirt-sock
libvirtd 7266 root   12u     unix 0xffff88001d404c00      0t0   13031405 /var/run/libvirt/libvirt-sock

So the question is - where would a wild rm or mount come from in your cases.
Setting the Cloud-Archive Task to incomplete until info is provided that allows further debugging.

** Changed in: cloud-archive
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1583009

Title:
  Error starting domain since update

Status in Ubuntu Cloud Archive:
  Incomplete
Status in libvirt package in Ubuntu:
  Fix Released

Bug description:
  Had no problems yesterday using virt-manager to open (or create new
  virtual machines)

  Received updates - unable to now open existing or create:

  Error starting domain: Failed to connect socket to '/var/run/libvirt
  /virtlogd-sock': No such file or directory

  ProblemType: Bug
  DistroRelease: Ubuntu 16.10
  Package: libvirt-bin 1.3.4-1ubuntu1
  ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
  Uname: Linux 4.4.0-22-generic x86_64
  ApportVersion: 2.20.1-0ubuntu4
  Architecture: amd64
  CurrentDesktop: XFCE
  Date: Wed May 18 07:18:47 2016
  InstallationDate: Installed on 2016-01-11 (127 days ago)
  InstallationMedia: Xubuntu 16.04 LTS "Xenial Xerus" - Alpha amd64 (20160111)
  SourcePackage: libvirt
  UpgradeStatus: No upgrade log present (probably fresh install)
  modified.conffile..etc.init.d.virtlogd: [modified]
  modified.conffile..etc.libvirt.libvirtd.conf: [modified]
  modified.conffile..etc.libvirt.libxl.conf: [modified]
  modified.conffile..etc.libvirt.qemu.conf: [inaccessible: [Errno 13] Permission denied: '/etc/libvirt/qemu.conf']
  modified.conffile..etc.libvirt.qemu.networks.default.xml: [inaccessible: [Errno 13] Permission denied: '/etc/libvirt/qemu/networks/default.xml']
  mtime.conffile..etc.init.d.virtlogd: 2016-05-01T04:06:42
  mtime.conffile..etc.libvirt.libvirtd.conf: 2016-05-14T15:01:59
  mtime.conffile..etc.libvirt.libxl.conf: 2016-05-14T15:01:55

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1583009/+subscriptions