[Bug 1267191] Re: openstack-nova-compute service fails with - libvirtError: internal error: CPU feature `avx' specified more than once

Tue Jan 26 05:06:58 UTC 2016

** Summary changed:

-  openstack-nova-compute service fails with - libvirtError: internal error: CPU feature `avx' specified more than once
+ openstack-nova-compute service fails with - libvirtError: internal error: CPU feature `avx' specified more than once

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1267191

Title:
  openstack-nova-compute service fails with - libvirtError: internal
  error: CPU feature `avx' specified more than once

Status in OpenStack Compute (nova):
  Invalid
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Trusty:
  Fix Released

Bug description:
  [Impact]

   * cpu features list which is being sent to libvirt,
   when creating a domain or calling compareCPU, must contain only
   unique entries. Multiple issues arise when we are updating the
   features attribute in LibvirtConfigCPU class (for example during
   migration).

  [Test Case]

  * A two node openstack havana on ubuntu 14.04. Migrating a instance to
  other node.

   - Migrate the instance
   - Observe /var/log/nova/compute.log and /var/log/libvirt.log.
   - /var/log/libvirtd.log: 2014-03-27 18:23:17.141+0000: 2659: error : virCPUDefParseXML:413 : XML error: CPU feature `wdt' specified more than once.

  [Regression Potential]

   * Same fix for 1303536 applied to Icehouse.

   * Fix is running in a big production without any problem.

   * A regression would continue to cause nova not to start (as is happening
     today with this bug) under described conditions.

  [Other Info]

  --------------------------------------------
  Original Description (Clone of a Fedora Bug)
  --------------------------------------------

  Restarting openstack-nova-compute serivce fails with:

    libvirtError: internal error: CPU feature `avx' specified more than
  once

  Version
  ---------

      $ rpm -q openstack-nova libvirt qemu-system-x86
      openstack-nova-2014.1-0.4.b1.fc21.noarch
      libvirt-1.1.3.2-1.fc20.x86_64
      qemu-system-x86-1.6.1-3.fc20.x86_64

  Test env
  ----------

  A two node OpenStack RDO set-up configured manually on two Fedora 20
  VMs:

    - Controller node: Nova, Keystone, Cinder, Glance, Neutron (using Open
      vSwitch plugin and GRE tunneling).

    - Compute node: Nova (nova-compute), Neutron (openvswitch-agent)

  How reproducible: Consistently.

  Steps to Reproduce
  ------------------

      $ systemctl restart openstack-nova-compute

  Observe /var/log/nova/compute.log

  Actual results
  --------------

  $ tail -f /var/log/nova/compute.log
  2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 187, in doit
  2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 147, in proxy_call
  2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     rv = execute(f,*args,**kwargs)
  2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 76, in tworker
  2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     rv = meth(*args,**kwargs)
  2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3622, in baselineCPU
  2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     if ret is None: raise libvirtError ('virConnectBaselineCPU() failed', conn=self)
  2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup libvirtError: internal error: CPU feature `avx' specified more than once
  2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup

  Expected results
  ----------------

  Compute service should start successfully.

  Additional info
  ---------------

  Status of openstack-nova-compute service

  $ systemctl status openstack-nova-compute
  openstack-nova-compute.service - OpenStack Nova Compute Server
     Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled)
     Active: inactive (dead) since Tue 2014-01-08 07:00:07 EST; 13min ago
    Process: 1529 ExecStart=/usr/bin/nova-compute --logfile /var/log/nova/compute.log (code=exited, status=0/SUCCESS)
   Main PID: 1529 (code=exited, status=0/SUCCESS)

  Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib/pyth...in doit
  Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     result = proxy_ca...kwargs)
  Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib/pyth...xy_call
  Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     rv = execute(f,*a...kwargs)
  Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib/pyth...tworker
  Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     rv = meth(*args,**kwargs)
  Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/py...lineCPU
  Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     if ret is None: r...n=self)
  Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup libvirtError: interna...an once
  Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-08 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup

  [reply] [−]
  Private
  Comment 1 Kashyap Chamarthy 2014-01-08 09:21:17 EST

  More contextual trace from compute.log:

  [. . .]
  2014-12-10 22:12:38.789 1429 TRACE nova.virt.libvirt.driver
  2014-12-10 22:12:39.319 1429 ERROR nova.openstack.common.threadgroup [-] internal error: CPU feature `avx' specified more than once
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 117, in wait
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     x.wait()
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 49, in wait
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/openstack/common/service.py", line 448, in run_service
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     service.start()
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/service.py", line 164, in start
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     self.manager.pre_start_hook()
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 822, in pre_start_hook
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     self.update_available_resource(nova.context.get_admin_context())
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4971, in update_available_resource
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     nodenames = set(self.driver.get_available_nodes())
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/driver.py", line 980, in get_available_nodes
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     stats = self.get_host_stats(refresh=refresh)
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4569, in get_host_stats
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     return self.host_state.get_host_stats(refresh=refresh)
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 429, in host_state
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     self._host_state = HostState(self)
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4960, in __init__
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     self.update_status()
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4999, in update_status
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     self.driver.get_instance_capabilities()
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3702, in get_instance_capabilities
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     caps = self.get_host_capabilities()
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2742, in get_host_capabilities
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     libvirt.VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES)
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 187, in doit
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 147, in proxy_call
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     rv = execute(f,*args,**kwargs)
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 76, in tworker
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     rv = meth(*args,**kwargs)
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3622, in baselineCPU
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     if ret is None: raise libvirtError ('virConnectBaselineCPU() failed', conn=self)
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup libvirtError: internal error: CPU feature `avx' specified more than once
  2014-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup
  2014-12-10 22:12:58.904 1467 WARNING nova.virt.libvirt.driver [req-88723bba-e677-4956-9b70-1f7fe66dbff7 None None] Cannot update service status on host: node2-compute,due to an unexpected exception.
  [. . .]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1267191/+subscriptions