[Bug 1749655] [NEW] ceilometer agent managers bails out if a libvirt XML is missing a field

Junien Fridrick 1749655 at bugs.launchpad.net
Thu Feb 15 07:56:20 UTC 2018


Public bug reported:

Hi,

Using python-ceilometer 1:8.1.2-0ubuntu1~cloud0 on Ubuntu 16.04.3 LTS,
we just got the traceback below. The problem is that a single libvirt
XML was missing the "nova:owner" block (I don't know why), and it looks
like this single problem was impacting meter collection for all the
other VMs. Once this VM got removed, the meters were properly populated
(see below for before and after the VM removal).

A single bad XML should not block collection for all the VMs.

Thanks


Traceback :

2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager [-] Unable to discover resources: 'NoneType' object has no attribute 'attrib'
2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager Traceback (most recent call last):
2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 505, in disco
ver
2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager     discovered = discoverer.discover(self, param)
2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/dist-packages/ceilometer/compute/discovery.py", line 133, in d
iscover
2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager     return self.discover_libvirt_polling(manager, param=None)
2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/dist-packages/ceilometer/compute/virt/libvirt/utils.py", line 
90, in decorator
2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager     return function(self, *args, **kwargs)
2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/dist-packages/ceilometer/compute/discovery.py", line 184, in d
iscover_libvirt_polling
2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager     user_id = metadata_xml.find("./owner/user").attrib["uuid"]
2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager AttributeError: 'NoneType' object has no attribute 'attrib'
2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager 
2018-02-15 07:26:43.780 1904096 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.bytes.rate, no resources found this cycle


Before the fix :

$ ceilometer meter-list --query resource=3bfc7f11-a0b9-46af-b7cc-733361fe0efc
+-------------------------------+-------+------+--------------------------------------+------------------------------------------------------------------+----------------------------------+
| Name                          | Type  | Unit | Resource ID                          | User ID                                                          | Project ID                       |
+-------------------------------+-------+------+--------------------------------------+------------------------------------------------------------------+----------------------------------+
| compute.instance.booting.time | gauge | sec  | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| disk.ephemeral.size           | gauge | GB   | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| disk.root.size                | gauge | GB   | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| memory                        | gauge | MB   | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| vcpus                         | gauge | vcpu | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
+-------------------------------+-------+------+--------------------------------------+------------------------------------------------------------------+----------------------------------+


After the fix :

$ ceilometer meter-list --query resource=3bfc7f11-a0b9-46af-b7cc-733361fe0efc
+-------------------------------+------------+---------+--------------------------------------+------------------------------------------------------------------+----------------------------------+
| Name                          | Type       | Unit    | Resource ID                          | User ID                                                          | Project ID                       |
+-------------------------------+------------+---------+--------------------------------------+------------------------------------------------------------------+----------------------------------+
| compute.instance.booting.time | gauge      | sec     | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| cpu                           | cumulative | ns      | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| disk.allocation               | gauge      | B       | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| disk.capacity                 | gauge      | B       | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| disk.ephemeral.size           | gauge      | GB      | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| disk.read.bytes               | cumulative | B       | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| disk.read.requests            | cumulative | request | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| disk.root.size                | gauge      | GB      | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| disk.usage                    | gauge      | B       | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| disk.write.bytes              | cumulative | B       | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| disk.write.requests           | cumulative | request | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| memory                        | gauge      | MB      | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| memory.resident               | gauge      | MB      | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| memory.usage                  | gauge      | MB      | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
| vcpus                         | gauge      | vcpu    | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
+-------------------------------+------------+---------+--------------------------------------+------------------------------------------------------------------+----------------------------------+

** Affects: ceilometer (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceilometer in Ubuntu.
https://bugs.launchpad.net/bugs/1749655

Title:
  ceilometer agent managers bails out if a libvirt XML is missing a
  field

Status in ceilometer package in Ubuntu:
  New

Bug description:
  Hi,

  Using python-ceilometer 1:8.1.2-0ubuntu1~cloud0 on Ubuntu 16.04.3 LTS,
  we just got the traceback below. The problem is that a single libvirt
  XML was missing the "nova:owner" block (I don't know why), and it
  looks like this single problem was impacting meter collection for all
  the other VMs. Once this VM got removed, the meters were properly
  populated (see below for before and after the VM removal).

  A single bad XML should not block collection for all the VMs.

  Thanks


  Traceback :

  2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager [-] Unable to discover resources: 'NoneType' object has no attribute 'attrib'
  2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager Traceback (most recent call last):
  2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 505, in disco
  ver
  2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager     discovered = discoverer.discover(self, param)
  2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/dist-packages/ceilometer/compute/discovery.py", line 133, in d
  iscover
  2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager     return self.discover_libvirt_polling(manager, param=None)
  2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/dist-packages/ceilometer/compute/virt/libvirt/utils.py", line 
  90, in decorator
  2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager     return function(self, *args, **kwargs)
  2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/dist-packages/ceilometer/compute/discovery.py", line 184, in d
  iscover_libvirt_polling
  2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager     user_id = metadata_xml.find("./owner/user").attrib["uuid"]
  2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager AttributeError: 'NoneType' object has no attribute 'attrib'
  2018-02-15 07:26:43.779 1904096 ERROR ceilometer.agent.manager 
  2018-02-15 07:26:43.780 1904096 INFO ceilometer.agent.manager [-] Skip pollster disk.device.write.bytes.rate, no resources found this cycle



  Before the fix :

  $ ceilometer meter-list --query resource=3bfc7f11-a0b9-46af-b7cc-733361fe0efc
  +-------------------------------+-------+------+--------------------------------------+------------------------------------------------------------------+----------------------------------+
  | Name                          | Type  | Unit | Resource ID                          | User ID                                                          | Project ID                       |
  +-------------------------------+-------+------+--------------------------------------+------------------------------------------------------------------+----------------------------------+
  | compute.instance.booting.time | gauge | sec  | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | disk.ephemeral.size           | gauge | GB   | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | disk.root.size                | gauge | GB   | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | memory                        | gauge | MB   | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | vcpus                         | gauge | vcpu | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  +-------------------------------+-------+------+--------------------------------------+------------------------------------------------------------------+----------------------------------+


  After the fix :

  $ ceilometer meter-list --query resource=3bfc7f11-a0b9-46af-b7cc-733361fe0efc
  +-------------------------------+------------+---------+--------------------------------------+------------------------------------------------------------------+----------------------------------+
  | Name                          | Type       | Unit    | Resource ID                          | User ID                                                          | Project ID                       |
  +-------------------------------+------------+---------+--------------------------------------+------------------------------------------------------------------+----------------------------------+
  | compute.instance.booting.time | gauge      | sec     | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | cpu                           | cumulative | ns      | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | disk.allocation               | gauge      | B       | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | disk.capacity                 | gauge      | B       | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | disk.ephemeral.size           | gauge      | GB      | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | disk.read.bytes               | cumulative | B       | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | disk.read.requests            | cumulative | request | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | disk.root.size                | gauge      | GB      | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | disk.usage                    | gauge      | B       | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | disk.write.bytes              | cumulative | B       | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | disk.write.requests           | cumulative | request | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | memory                        | gauge      | MB      | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | memory.resident               | gauge      | MB      | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | memory.usage                  | gauge      | MB      | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  | vcpus                         | gauge      | vcpu    | 3bfc7f11-a0b9-46af-b7cc-733361fe0efc | c25152c3f94599f7dbc7f81ddb01dd8922f68b433e745819509d9745d07b6a2e | b18ffb39e718417bb28e806e6c1e802a |
  +-------------------------------+------------+---------+--------------------------------------+------------------------------------------------------------------+----------------------------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceilometer/+bug/1749655/+subscriptions



More information about the Ubuntu-openstack-bugs mailing list