[Bug 1848286] Re: octavia is not reporting metrics like lbaasv2

Mon Dec 2 16:39:05 UTC 2019

** Tags removed: verification-needed verification-needed-disco verification-needed-eoan verification-rocky-needed verification-stein-needed verification-train-needed
** Tags added: verification-done verification-done-disco verification-done-eoan verification-rocky-done verification-stein-done verification-train-done

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1848286

Title:
  octavia is not reporting metrics like lbaasv2

Status in Ceilometer:
  Fix Released
Status in OpenStack Octavia Charm:
  Triaged
Status in Ubuntu Cloud Archive:
  Triaged
Status in Ubuntu Cloud Archive queens series:
  Triaged
Status in Ubuntu Cloud Archive rocky series:
  Fix Committed
Status in Ubuntu Cloud Archive stein series:
  Fix Committed
Status in Ubuntu Cloud Archive train series:
  Fix Committed
Status in Ubuntu Cloud Archive ussuri series:
  Triaged
Status in ceilometer package in Ubuntu:
  Fix Released
Status in ceilometer source package in Bionic:
  Triaged
Status in ceilometer source package in Disco:
  Fix Committed
Status in ceilometer source package in Eoan:
  Fix Committed
Status in ceilometer source package in Focal:
  Fix Released

Bug description:
  [Environment]

  Stein OpenStack
  Ubuntu Bionic

  [Description]

  From my understanding the current Octavia API should be backwards
  compatible with LBaaS v2, therefore, the current LBaaS v2 pollster [1]
  should suffice to gather the following meters:

  meters:

  - network.services.lb.outgoing.bytes
  - network.services.lb.incoming.bytes
  - network.services.lb.pool
  - network.services.lb.listener
  - network.services.lb.member
  - network.services.lb.health_monitor
  - network.services.lb.loadbalancer
  - network.services.lb.total.connections
  - network.services.lb.active.connections

  However, the following warning is noticed when restarted the
  ceilometer services.

  2999:2019-11-07 15:05:41.665 22487 WARNING ceilometer.publisher.gnocchi [-] metric network.services.lb.loadbalancer is not handled by Gnocchi
  3467:2019-11-07 15:06:21.830 3916 WARNING ceilometer.publisher.gnocchi [-] metric network.services.lb.loadbalancer is not handled by Gnocchi

  In fact, by checking at the gnocchi metric list (https://pastebin.canonical.com/p/zyMTFv8vww/) the network.services.lb
  metrics/resources haven't been created.

  The reason is that for a gnocchi resource to exist, ceilometer has to create the resource type explicitly.
  This action is commanded by the ceilometer gnocchi client [2] and it uses the file /usr/lib/python3/dist-packages/ceilometer/publisher/data/gnocchi_resources.yaml
  for the resource definitions, which doesn't includes the lbaasv2 directives.

  [Test case]

  1) Deploy a gnocchi backed ceilometer unit with octavia deployed.
  2) check that lb meters aren't being gathered

  $ gnocchi metric list|grep loadbalancer | wc -l
  0
  3) install the patched version
  4) configure all pollsters
  $  juju config ceilometer enable-all-pollsters=true
  5) sudo systemctl restart ceilometer*
  6) sudo ceilometer-upgrade
  7) Check that lb metrics are being gathered

  ubuntu at niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep loadbalancer | wc -l
  72

  ubuntu at niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep network.services
  | 3947b249-b1ed-489d-9f51-b7549b5b78ce | ceilometer-low | network.services.lb.loadbalancer | loadbalancer | f97838f1-215d-4fcb-bd78-61e1f9859507 |
  | 5527c5b6-d17e-4218-868b-c8d5326575a6 | ceilometer-low | network.services.lb.active.connections | connection | f97838f1-215d-4fcb-bd78-61e1f9859507 |
  | 57d63bcb-48e1-47fb-825c-1109ad7d966d | ceilometer-low | network.services.lb.loadbalancer | loadbalancer | 82d129f3-64a5-40d6-a72b-7ae704046176 |
  | 5d93c751-0645-4b7b-a315-26b0e96dda3d | ceilometer-low | network.services.lb.outgoing.bytes | B | 82d129f3-64a5-40d6-a72b-7ae704046176 |
  | 71ace776-9216-40a4-848c-42e9cc6f93a0 | ceilometer-low | network.services.lb.listener | listener | e7a4dd96-73a1-4aca-946d-8e87ba8f7a37 |
  | 738b1dd0-0930-4bfa-b358-1a27fb63121e | ceilometer-low | network.services.lb.total.connections | connection | f97838f1-215d-4fcb-bd78-61e1f9859507 |
  | 7b4726f6-9490-4d6c-a586-7e12b738b001 | ceilometer-low | network.services.lb.health_monitor | health_monitor | 3ebc3ac0-8e1c-47d9-8789-bb51e6d29eba |
  | 8689e872-c6d0-450f-bb73-d19197c62418 | ceilometer-low | network.services.lb.incoming.bytes | B | 82d129f3-64a5-40d6-a72b-7ae704046176 |
  | ae72c5f3-4d8b-400b-ba86-2aaf068c20fc | ceilometer-low | network.services.lb.incoming.bytes | B | f97838f1-215d-4fcb-bd78-61e1f9859507 |
  | b59ef5ae-9d71-4884-bb7c-66e0d46f5ecb | ceilometer-low | network.services.lb.outgoing.bytes | B | f97838f1-215d-4fcb-bd78-61e1f9859507 |
  | bdf259a1-7471-4c23-ab2b-cac11f522f45 | ceilometer-low | network.services.lb.active.connections | connection | 82d129f3-64a5-40d6-a72b-7ae704046176 |
  | e5d2e433-9482-4e2a-9485-e73d7a13271a | ceilometer-low | network.services.lb.listener | listener | c87e0f88-26bf-496f-a882-71b1c67956b4 |
  | eb634e4e-8e94-4ee4-8fac-4f4db12d56aa | ceilometer-low | network.services.lb.total.connections | connection | 82d129f3-64a5-40d6-a72b-7ae7040

  [Regression Potential]

  There is no regression potential identified on this patch as it uses
  the new metrics upgrade path and it doesn't
  enables any new metrics if the lbaasv2 pollster isn't configured.

  [Proposed solution]

  Enable the resources on /usr/lib/python3/dist packages/ceilometer/publisher/data/gnocchi_resources.yaml
  and modify the gnocchi client provider. I am able to gather the metrics for the octavia load balancers.

  ubuntu at niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep network.services
  | 3947b249-b1ed-489d-9f51-b7549b5b78ce | ceilometer-low | network.services.lb.loadbalancer | loadbalancer | f97838f1-215d-4fcb-bd78-61e1f9859507 |
  | 5527c5b6-d17e-4218-868b-c8d5326575a6 | ceilometer-low | network.services.lb.active.connections | connection | f97838f1-215d-4fcb-bd78-61e1f9859507 |
  | 57d63bcb-48e1-47fb-825c-1109ad7d966d | ceilometer-low | network.services.lb.loadbalancer | loadbalancer | 82d129f3-64a5-40d6-a72b-7ae704046176 |
  | 5d93c751-0645-4b7b-a315-26b0e96dda3d | ceilometer-low | network.services.lb.outgoing.bytes | B | 82d129f3-64a5-40d6-a72b-7ae704046176 |
  | 71ace776-9216-40a4-848c-42e9cc6f93a0 | ceilometer-low | network.services.lb.listener | listener | e7a4dd96-73a1-4aca-946d-8e87ba8f7a37 |
  | 738b1dd0-0930-4bfa-b358-1a27fb63121e | ceilometer-low | network.services.lb.total.connections | connection | f97838f1-215d-4fcb-bd78-61e1f9859507 |
  | 7b4726f6-9490-4d6c-a586-7e12b738b001 | ceilometer-low | network.services.lb.health_monitor | health_monitor | 3ebc3ac0-8e1c-47d9-8789-bb51e6d29eba |
  | 8689e872-c6d0-450f-bb73-d19197c62418 | ceilometer-low | network.services.lb.incoming.bytes | B | 82d129f3-64a5-40d6-a72b-7ae704046176 |
  | ae72c5f3-4d8b-400b-ba86-2aaf068c20fc | ceilometer-low | network.services.lb.incoming.bytes | B | f97838f1-215d-4fcb-bd78-61e1f9859507 |
  | b59ef5ae-9d71-4884-bb7c-66e0d46f5ecb | ceilometer-low | network.services.lb.outgoing.bytes | B | f97838f1-215d-4fcb-bd78-61e1f9859507 |
  | bdf259a1-7471-4c23-ab2b-cac11f522f45 | ceilometer-low | network.services.lb.active.connections | connection | 82d129f3-64a5-40d6-a72b-7ae704046176 |
  | e5d2e433-9482-4e2a-9485-e73d7a13271a | ceilometer-low | network.services.lb.listener | listener | c87e0f88-26bf-496f-a882-71b1c67956b4 |
  | eb634e4e-8e94-4ee4-8fac-4f4db12d56aa | ceilometer-low | network.services.lb.total.connections | connection | 82d129f3-64a5-40d6-a72b-7ae704046176 |
  | ef04f494-884a-4097-ba58-74845f5fbf19 | ceilometer-low | network.services.lb.pool | pool | 6c4a5a4f-0455-42e6-8bf4-ed9c24e2c06c |

  ubuntu at niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi measures show ef04f494-884a-4097-ba58-74845f5fbf19
  +---------------------------+-------------+-------+
  | timestamp | granularity | value |
  +---------------------------+-------------+-------+
  | 2019-11-07T18:30:00+00:00 | 300.0 | 1.0 |
  | 2019-11-07T18:35:00+00:00 | 300.0 | 1.0 |
  | 2019-11-07T18:40:00+00:00 | 300.0 | 1.0 |
  +---------------------------+-------------+-------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/ceilometer/+bug/1848286/+subscriptions