[Bug 1799406] Re: [SRU] Alarms fail on Rocky

Steve Langasek steve.langasek at canonical.com
Thu Nov 1 19:42:05 UTC 2018 

Hello Liam, or anyone else affected,

Accepted aodh into cosmic-proposed. The package will build now and be
available at https://launchpad.net/ubuntu/+source/aodh/7.0.0-0ubuntu4 in
a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-cosmic to verification-done-cosmic. If it does not
fix the bug for you, please add a comment stating that, and change the
tag to verification-failed-cosmic. In either case, without details of
your testing we will not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Tags added: verification-needed verification-needed-cosmic

** Also affects: aodh (Ubuntu)
   Importance: Undecided
       Status: New

** Also affects: aodh (Ubuntu Cosmic)
   Importance: Undecided
       Status: New

** Changed in: aodh (Ubuntu Cosmic)
       Status: New => Fix Committed

** Changed in: aodh (Ubuntu)
       Status: New => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to aodh in Ubuntu.
https://bugs.launchpad.net/bugs/1799406

Title:
  [SRU] Alarms fail on Rocky

Status in Aodh:
  In Progress
Status in OpenStack AODH Charm:
  New
Status in oslo.i18n:
  New
Status in aodh package in Ubuntu:
  Fix Committed
Status in aodh source package in Cosmic:
  Fix Committed

Bug description:
  [Impact]
  In a rocky deployment aodh is failing to raise alarms due to what seems like a locale issue. If I purge packages:

  apt-get purge aodh-api aodh-evaluator aodh-expirer aodh-notifier aodh-
  listener python3-aodh libapache2-mod-wsgi-py3 python-apt python-aodh
  python-memcache

  Then disable the AodhCharmRocky class and rerun install so that it
  pulls in the py2 packages then things start working.

  The error from /var/log/aodh/aodh-listener.log:

  http://paste.ubuntu.com/p/FCft2GGnfN/

  ERROR oslo_db.sqlalchemy.exc_filters [-] DB exception wrapped.: AttributeError: 'list' object has no attribute 'lower'
  ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
  ERROR oslo_db.sqlalchemy.exc_filters     context)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 508, in do_execute
  ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 163, in execute
  ERROR oslo_db.sqlalchemy.exc_filters     query = self.mogrify(query, args)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 142, in mogrify
  ERROR oslo_db.sqlalchemy.exc_filters     query = query % self._escape_args(args, conn)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in _escape_args
  ERROR oslo_db.sqlalchemy.exc_filters     return dict((key, conn.literal(val)) for (key, val) in args.items())
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in <genexpr>
  ERROR oslo_db.sqlalchemy.exc_filters     return dict((key, conn.literal(val)) for (key, val) in args.items())
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 819, in literal
  ERROR oslo_db.sqlalchemy.exc_filters     return self.escape(obj, self.encoders)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 806, in escape
  ERROR oslo_db.sqlalchemy.exc_filters     return "'" + self.escape_string(obj) + "'"
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 825, in escape_string
  ERROR oslo_db.sqlalchemy.exc_filters     return converters.escape_string(s)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/pymysql/converters.py", line 73, in _escape_unicode
  ERROR oslo_db.sqlalchemy.exc_filters     return value.translate(_escape_table)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 85, in translate
  ERROR oslo_db.sqlalchemy.exc_filters     self.has_plural_form)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 117, in _translate_msgid
  ERROR oslo_db.sqlalchemy.exc_filters     fallback=True)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3.6/gettext.py", line 525, in translation
  ERROR oslo_db.sqlalchemy.exc_filters     mofiles = find(domain, localedir, languages, all=True)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3.6/gettext.py", line 482, in find
  ERROR oslo_db.sqlalchemy.exc_filters     for nelang in _expand_lang(lang):
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3.6/gettext.py", line 206, in _expand_lang
  ERROR oslo_db.sqlalchemy.exc_filters     loc = locale.normalize(loc)
  ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python3.6/locale.py", line 397, in normalize
  ERROR oslo_db.sqlalchemy.exc_filters     code = localename.lower()
  ERROR oslo_db.sqlalchemy.exc_filters AttributeError: 'list' object has no attribute 'lower'
  ERROR oslo_db.sqlalchemy.exc_filters
  ERROR aodh.evaluator [-] alarm state update failed: oslo_db.exception.DBError: 'list' object has no attribute 'lower'
  ERROR aodh.evaluator Traceback (most recent call last):
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
  ERROR aodh.evaluator     context)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 508, in do_execute
  ERROR aodh.evaluator     cursor.execute(statement, parameters)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 163, in execute
  ERROR aodh.evaluator     query = self.mogrify(query, args)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 142, in mogrify
  ERROR aodh.evaluator     query = query % self._escape_args(args, conn)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in _escape_args
  ERROR aodh.evaluator     return dict((key, conn.literal(val)) for (key, val) in args.items())
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in <genexpr>
  ERROR aodh.evaluator     return dict((key, conn.literal(val)) for (key, val) in args.items())
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 819, in literal
  ERROR aodh.evaluator     return self.escape(obj, self.encoders)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 806, in escape
  ERROR aodh.evaluator     return "'" + self.escape_string(obj) + "'"
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 825, in escape_string
  ERROR aodh.evaluator     return converters.escape_string(s)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/converters.py", line 73, in _escape_unicode
  ERROR aodh.evaluator     return value.translate(_escape_table)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 85, in translate
  ERROR aodh.evaluator     self.has_plural_form)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 117, in _translate_msgid
  ERROR aodh.evaluator     fallback=True)
  ERROR aodh.evaluator   File "/usr/lib/python3.6/gettext.py", line 525, in translation
  ERROR aodh.evaluator     mofiles = find(domain, localedir, languages, all=True)
  ERROR aodh.evaluator   File "/usr/lib/python3.6/gettext.py", line 482, in find
  ERROR aodh.evaluator     for nelang in _expand_lang(lang):
  ERROR aodh.evaluator   File "/usr/lib/python3.6/gettext.py", line 206, in _expand_lang
  ERROR aodh.evaluator     loc = locale.normalize(loc)
  ERROR aodh.evaluator   File "/usr/lib/python3.6/locale.py", line 397, in normalize
  ERROR aodh.evaluator     code = localename.lower()
  ERROR aodh.evaluator AttributeError: 'list' object has no attribute 'lower'
  ERROR aodh.evaluator
  ERROR aodh.evaluator The above exception was the direct cause of the following exception:
  ERROR aodh.evaluator
  ERROR aodh.evaluator Traceback (most recent call last):
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/aodh/evaluator/__init__.py", line 128, in _refresh
  ERROR aodh.evaluator     self._storage_conn.update_alarm(alarm)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/aodh/storage/impl_sqlalchemy.py", line 273, in update_alarm
  ERROR aodh.evaluator     alarm.as_dict())
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3426, in update
  ERROR aodh.evaluator     update_op.exec_()
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/orm/persistence.py", line 1326, in exec_
  ERROR aodh.evaluator     self._do_exec()
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/orm/persistence.py", line 1493, in _do_exec
  ERROR aodh.evaluator     self._execute_stmt(update_stmt)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/orm/persistence.py", line 1333, in _execute_stmt
  ERROR aodh.evaluator     mapper=self.mapper)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/orm/session.py", line 1176, in execute
  ERROR aodh.evaluator     bind, close_with_result=True).execute(clause, params or {})
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 948, in execute
  ERROR aodh.evaluator     return meth(self, multiparams, params)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
  ERROR aodh.evaluator     return connection._execute_clauseelement(self, multiparams, params)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
  ERROR aodh.evaluator     compiled_sql, distilled_params
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
  ERROR aodh.evaluator     context)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
  ERROR aodh.evaluator     util.raise_from_cause(newraise, exc_info)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
  ERROR aodh.evaluator     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 186, in reraise
  ERROR aodh.evaluator     raise value.with_traceback(tb)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
  ERROR aodh.evaluator     context)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 508, in do_execute
  ERROR aodh.evaluator     cursor.execute(statement, parameters)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 163, in execute
  ERROR aodh.evaluator     query = self.mogrify(query, args)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 142, in mogrify
  ERROR aodh.evaluator     query = query % self._escape_args(args, conn)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in _escape_args
  ERROR aodh.evaluator     return dict((key, conn.literal(val)) for (key, val) in args.items())
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in <genexpr>
  ERROR aodh.evaluator     return dict((key, conn.literal(val)) for (key, val) in args.items())
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 819, in literal
  ERROR aodh.evaluator     return self.escape(obj, self.encoders)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 806, in escape
  ERROR aodh.evaluator     return "'" + self.escape_string(obj) + "'"
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 825, in escape_string
  ERROR aodh.evaluator     return converters.escape_string(s)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/pymysql/converters.py", line 73, in _escape_unicode
  ERROR aodh.evaluator     return value.translate(_escape_table)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 85, in translate
  ERROR aodh.evaluator     self.has_plural_form)
  ERROR aodh.evaluator   File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 117, in _translate_msgid
  ERROR aodh.evaluator     fallback=True)
  ERROR aodh.evaluator   File "/usr/lib/python3.6/gettext.py", line 525, in translation
  ERROR aodh.evaluator     mofiles = find(domain, localedir, languages, all=True)
  ERROR aodh.evaluator   File "/usr/lib/python3.6/gettext.py", line 482, in find
  ERROR aodh.evaluator     for nelang in _expand_lang(lang):
  ERROR aodh.evaluator   File "/usr/lib/python3.6/gettext.py", line 206, in _expand_lang
  ERROR aodh.evaluator     loc = locale.normalize(loc)
  ERROR aodh.evaluator   File "/usr/lib/python3.6/locale.py", line 397, in normalize
  ERROR aodh.evaluator     code = localename.lower()
  ERROR aodh.evaluator oslo_db.exception.DBError: 'list' object has no attribute 'lower'
  ERROR aodh.evaluator

  [Test Case]
  1) Deploy openstack with aodh (we're deploying with the development versions of the openstack charms which deploy py3 payloads)
  2) Once deployed/configured create an alarm with:
  openstack alarm create \
    --type event \
    --name instance_off \
    --description 'Instance powered OFF' \
    --event-type "compute.instance.power_off.*" \
    --enable True \
    --query "traits.instance_id=string::INSTANCE_ID" \
    --alarm-action 'log://' \
    --ok-action 'log://' \
    --insufficient-data-action 'log://'
  3) create a server instance
  4) power server instance off to trigger alarm
  5) ensure alarm is successfully raised with no traceback (see /var/log/aodh/aodh-listener.log)

  [Regression Potential]
  The fix is a minimal fix that should result in low regression potential.

To manage notifications about this bug go to:
https://bugs.launchpad.net/aodh/+bug/1799406/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list