[Bug 1930446] Re: Wallaby ceilometer.compute.discovery fails to get domain metadata

[James Page]

For reference, I've dropped ceilometer point releases into:

  ppa:james-page/wallaby
  ppa:james-page/xena

for the Wallaby and Xena Ubuntu Cloud Archive pockets - I've tested at
Xena and the point release does the trick with regards to resolving the
namespace issue.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceilometer in Ubuntu.
https://bugs.launchpad.net/bugs/1930446

Title:
  Wallaby ceilometer.compute.discovery fails to get domain metadata

Status in Ceilometer:
  In Progress
Status in Ubuntu Cloud Archive:
  Triaged
Status in Ubuntu Cloud Archive wallaby series:
  Triaged
Status in Ubuntu Cloud Archive xena series:
  Triaged
Status in Ubuntu Cloud Archive yoga series:
  Triaged
Status in ceilometer package in Ubuntu:
  Triaged
Status in ceilometer source package in Hirsute:
  Won't Fix
Status in ceilometer source package in Impish:
  Triaged
Status in ceilometer source package in Jammy:
  Triaged

Bug description:
  Hi!

  Ceilometer compute agent fails to get libvirt domain metadata on an
  Ubuntu 20.04 LTS with the latest updates, kernel 5.4.0-65-generic and
  Openstack Wallaby Nova compute services installed using the official
  Wallaby repo for Ubuntu 20.04. All components have been deployed
  manually.

  Ceilometer agent is configured with instance_discovery_method =
  libvirt_metadata. The agent is unable to fetch the domain metadata,
  and the following error messages appear in
  /var/log/ceilometer/ceilometer-agent-compute.log on agent start and
  periodic polling attempts:

  2021-06-01 16:01:18.297 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid baf06f57-ac5b-4661-928c-7adaeaea0311 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
  2021-06-01 16:01:18.298 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid 208c0d7a-41a3-4fa6-bf72-2f9594ac6b8d metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
  2021-06-01 16:01:18.300 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid d979a527-c1ba-4b29-8e30-322d4d2efcf7 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
  2021-06-01 16:01:18.301 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid a41f21b6-766d-4979-bbe1-84f421b0c3f2 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
  2021-06-01 16:01:18.302 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid fd5ffe32-c6d6-4898-9ba2-2af1ffebd502 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
  2021-06-01 16:01:18.302 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid aff042c9-c311-4944-bc42-09ccd5a90eb7 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
  2021-06-01 16:01:18.303 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid 9510bc46-e4e2-490c-9cbe-c9eb5e349b8d metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
  2021-06-01 16:01:18.304 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid 4d2c2c9b-4eff-460a-a00b-19fdbe33f5d4 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
  2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster cpu_l3_cache, no  resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
  2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.write.bytes, no  resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
  2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.incoming.packets, no  resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
  2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.read.requests, no  resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
  2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.outgoing.packets, no  resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
  2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.outgoing.bytes, no  resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
  2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.incoming.bytes, no  resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
  2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster cpu, no  resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
  2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.write.requests, no  resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
  2021-06-01 16:01:18.307 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.read.bytes, no  resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
  2021-06-01 16:01:18.307 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster memory.usage, no  resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177

  All domains exist and their metadata is readily available using virsh
  or a simple Python script. Nova compute service is fully functional,
  Ceilometer agent is partially functional as it is able to export
  compute.node.cpu.* metrics but nothing related to libvirt domains.

  Installed Ceilometer-related packages:

  ceilometer-agent-compute 2:16.0.0-0ubuntu1~cloud0
  ceilometer-common 2:16.0.0-0ubuntu1~cloud0
  python3-ceilometer 2:16.0.0-0ubuntu1~cloud0

  Installed Nova-related packages:

  nova-common 3:23.0.0-0ubuntu1~cloud0
  nova-compute 3:23.0.0-0ubuntu1~cloud0
  nova-compute-kvm 3:23.0.0-0ubuntu1~cloud0
  nova-compute-libvirt 3:23.0.0-0ubuntu1~cloud0
  python3-nova 3:23.0.0-0ubuntu1~cloud0
  python3-novaclient 2:17.4.0-0ubuntu1~cloud0

  Installed Libvirt-related packages:

  libvirt-clients 6.0.0-0ubuntu8.9
  libvirt-daemon 6.0.0-0ubuntu8.9
  libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.9
  libvirt-daemon-driver-storage-rbd 6.0.0-0ubuntu8.9
  libvirt-daemon-system 6.0.0-0ubuntu8.9
  libvirt-daemon-system-systemd 6.0.0-0ubuntu8.9
  libvirt0:amd64 6.0.0-0ubuntu8.9
  python3-libvirt 6.1.0-1

  Installed Qemu-related packages:

  libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.9
  qemu-block-extra:amd64 1:4.2-3ubuntu6.16
  qemu-kvm 1:4.2-3ubuntu6.16
  qemu-system-common 1:4.2-3ubuntu6.16
  qemu-system-data 1:4.2-3ubuntu6.16
  qemu-system-gui:amd64 1:4.2-3ubuntu6.16
  qemu-system-x86 1:4.2-3ubuntu6.16
  qemu-utils 1:4.2-3ubuntu6.16

  Apparmor is enabled and running the default configuration, no messages
  related to apparmor and libvirt, qemu, nova-compute, ceilometer-agent,
  etc are visible in the logs. I am also attaching the relevant
  Ceilometer agent and Nova configuration files.

  Please let me know if further information is required.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ceilometer/+bug/1930446/+subscriptions