[Bug 1946793] Re: aodh uses deprecated gnocchi api to aggregate metrics and doesn't work properly

[Tristan Zhang]

@James I don't think there are any errors in logs. The deprecated api
works, it just doesn't give us a proper aggregation metric.

You can try both these API with openstack commands as I pasted above.

# openstack metric aggregates
# openstack metric measures aggregation

For only 1 running instance, they give the same value. But for 2
instances, they value are different, and the one from old method is
wrong.

Or you can just call the old and new gnocchi api with a curl command,
you will see the difference.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to aodh in Ubuntu.
https://bugs.launchpad.net/bugs/1946793

Title:
  aodh uses deprecated gnocchi api to aggregate metrics and doesn't work
  properly

Status in Aodh:
  New
Status in aodh package in Ubuntu:
  Incomplete

Bug description:
  On gnocchi API docs, there are 2 API methods to aggregate metrics

  1. /v1/aggregation/metric?

  See: https://gnocchi.osci.io/rest.html#aggregation-across-metrics-
  deprecated

  This one is deprecated

  2. /v1/aggregates?

  See: https://gnocchi.osci.io/rest.html#dynamic-aggregates

  aodh uses the 1st one to aggregate metrics, for example:

  ```
          if isinstance(start, datetime.datetime):
              start = start.isoformat()
          if isinstance(stop, datetime.datetime):
              stop = stop.isoformat()

          params = dict(start=start, stop=stop, aggregation=aggregation,
                        reaggregation=reaggregation, granularity=granularity,
                        needed_overlap=needed_overlap, groupby=groupby,
                        refresh=refresh, resample=resample, fill=fill)
          if query is None:
              for metric in metrics:
                  self._ensure_metric_is_uuid(metric)
              params['metric'] = metrics
              measures = self._get("v1/aggregation/metric",
                                   params=params).json()
  ```

  aodh doesn't work properly in our production environment after
  upgraded to Ussuri.

  When there is only 1 instance, aodh works properly and alarms can be
  triggered when the load on the instance is higher than the threshold.

  However, after the stack is scaled up, and the second instance is
  created. The average cpu usage got from gnocchi by aodh evaluator is
  not correct. The metric measures are negative sometimes.

  I manually pulled metrics with gnocchi command

  The aggregation of metrics is correct with command

  ```
  openstack metric aggregates
  ```

  It uses new API in the backend

  The aggregation of metrics is not correct with command

  ```
  openstack metric measures aggregation
  ```

  It uses the deprecated API which aodh is using.

To manage notifications about this bug go to:
https://bugs.launchpad.net/aodh/+bug/1946793/+subscriptions