[Bug 1773765] Re: There is a possibility that 'running' notification will remain

Mon Oct 26 19:00:57 UTC 2020

Verified successfully for all corresponding proposed package versions:

== focal-proposed ==
root at f1:/usr/lib/python3/dist-packages# apt policy masakari-engine
masakari-engine:
  Installed: 9.0.0-0ubuntu0.20.04.3
  Candidate: 9.0.0-0ubuntu0.20.04.3
  Version table:
 *** 9.0.0-0ubuntu0.20.04.3 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
root at f1:~# cd /usr/lib/python3/dist-packages
root at f1:/usr/lib/python3/dist-packages# python3 -m unittest masakari.tests.unit.engine.test_engine_mgr.EngineManagerUnitTestCase.test_check_expired_notifications
2020-10-26 18:39:39.505 4505 INFO masakari.engine.driver [-] Loading masakari notification driver 'taskflow_driver'
/usr/lib/python3/dist-packages/taskflow/atom.py:31: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  _sequence_types = (list, tuple, collections.Sequence)
/usr/lib/python3/dist-packages/taskflow/atom.py:32: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  _set_types = (set, collections.Set)
/usr/lib/python3/dist-packages/masakari/context.py:92: DeprecationWarning: Using the 'user' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'user_id' argument instead
  super(RequestContext, self).__init__(
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'tenant' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'project_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'domain_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'user_domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'user_domain_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'project_domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'project_domain_id' argument instead
  return wrapped(*args, **kwargs)
2020-10-26 18:39:40.122 4505 ERROR masakari.engine.manager [req-03b9518f-a271-4d8c-aeee-f6c3d3252a09 - - - - -] Periodic task 'check_expired_notifications': Notification a03ac64b-a29e-474f-b94c-77767454b323 is expired.
.
----------------------------------------------------------------------
Ran 1 test in 0.917s

OK

== ussuri-proposed ==
root at b1:~# apt policy masakari-engine
masakari-engine:
  Installed: 9.0.0-0ubuntu0.20.04.3~cloud0
  Candidate: 9.0.0-0ubuntu0.20.04.3~cloud0
  Version table:
 *** 9.0.0-0ubuntu0.20.04.3~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages
        100 /var/lib/dpkg/status
root at b1:~# cd /usr/lib/python3/dist-packages
root at b1:/usr/lib/python3/dist-packages# python3 -m unittest masakari.tests.unit.engine.test_engine_mgr.EngineManagerUnitTestCase.test_check_expired_notifications
2020-10-26 18:51:13.254 3750 INFO masakari.engine.driver [-] Loading masakari notification driver 'taskflow_driver'
/usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/lib/python3/dist-packages/masakari/context.py:107: DeprecationWarning: Using the 'user' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'user_id' argument instead
  global_request_id=kwargs.pop('global_request_id', None))
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'tenant' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'project_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'domain_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'user_domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'user_domain_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'project_domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'project_domain_id' argument instead
  return wrapped(*args, **kwargs)
2020-10-26 18:51:14.906 3750 ERROR masakari.engine.manager [req-8b9c8a87-2d43-43a1-a255-638ee165dd1e - - - - -] Periodic task 'check_expired_notifications': Notification 9ecc2b97-79d0-46cd-b0d8-58f41edf9bd4 is expired.
.
----------------------------------------------------------------------
Ran 1 test in 2.030s

OK

== train-proposed ==
root at b2:~# apt policy masakari-engine
masakari-engine:
  Installed: 8.0.0-0ubuntu1~cloud1
  Candidate: 8.0.0-0ubuntu1~cloud1
  Version table:
 *** 8.0.0-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/train/main amd64 Packages
        100 /var/lib/dpkg/status
root at b2:~# cd /usr/lib/python3/dist-packages
root at b2:/usr/lib/python3/dist-packages# python3 -m unittest masakari.tests.unit.engine.test_engine_mgr.EngineManagerUnitTestCase.test_check_expired_notifications
/usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
2020-10-26 18:52:36.627 3929 INFO masakari.engine.driver [-] Loading masakari notification driver 'taskflow_driver'
2020-10-26 18:52:36.961 3929 INFO root [-] Generating grammar tables from /usr/lib/python3.6/lib2to3/Grammar.txt
2020-10-26 18:52:37.031 3929 INFO root [-] Generating grammar tables from /usr/lib/python3.6/lib2to3/PatternGrammar.txt
/usr/lib/python3/dist-packages/masakari/context.py:107: DeprecationWarning: Using the 'user' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'user_id' argument instead
  global_request_id=kwargs.pop('global_request_id', None))
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'tenant' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'project_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'domain_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'user_domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'user_domain_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'project_domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'project_domain_id' argument instead
  return wrapped(*args, **kwargs)
2020-10-26 18:52:37.395 3929 ERROR masakari.engine.manager [req-c34a9634-e544-4841-afa1-39aa25f026be - - - - -] Periodic task 'check_expired_notifications': Notification 6b52aa9a-7402-4dc4-8e96-89fb6b16a63e is expired.
.
----------------------------------------------------------------------
Ran 1 test in 0.941s

OK

== stein-proposed ==
root at b3:~# apt policy masakari-engine
masakari-engine:
  Installed: 7.0.0-0ubuntu1~cloud1
  Candidate: 7.0.0-0ubuntu1~cloud1
  Version table:
 *** 7.0.0-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/stein/main amd64 Packages
        100 /var/lib/dpkg/status
root at b3:~# cd /usr/lib/python3/dist-packages
root at b3:/usr/lib/python3/dist-packages# python3 -m unittest masakari.tests.unit.engine.test_engine_mgr.EngineManagerUnitTestCase.test_check_expired_notifications
/usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
2020-10-26 18:52:39.210 3904 INFO masakari.engine.driver [-] Loading masakari notification driver 'taskflow_driver'
2020-10-26 18:52:39.410 3904 INFO root [-] Generating grammar tables from /usr/lib/python3.6/lib2to3/Grammar.txt
2020-10-26 18:52:39.515 3904 INFO root [-] Generating grammar tables from /usr/lib/python3.6/lib2to3/PatternGrammar.txt
/usr/lib/python3/dist-packages/masakari/context.py:107: DeprecationWarning: Using the 'user' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'user_id' argument instead
  global_request_id=kwargs.pop('global_request_id', None))
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'tenant' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'project_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'domain_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'user_domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'user_domain_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'project_domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'project_domain_id' argument instead
  return wrapped(*args, **kwargs)
2020-10-26 18:52:39.904 3904 ERROR masakari.engine.manager [req-43c7e834-2586-475a-ab28-9ff667a7fac3 - - - - -] Periodic task 'check_expired_notifications': Notification 596a42dc-fe4b-44ab-8aba-888524b6f21b is expired.
.
----------------------------------------------------------------------
Ran 1 test in 0.825s

OK

** Tags removed: verification-needed verification-needed-focal verification-stein-needed verification-train-needed verification-ussuri-needed
** Tags added: verification-done verification-done-focal verification-stein-done verification-train-done verification-ussuri-done

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1773765

Title:
  There is a possibility that 'running' notification will remain

Status in OpenStack Masakari Charm:
  In Progress
Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive stein series:
  Fix Committed
Status in Ubuntu Cloud Archive train series:
  Fix Committed
Status in Ubuntu Cloud Archive ussuri series:
  Fix Committed
Status in Ubuntu Cloud Archive victoria series:
  Fix Released
Status in masakari:
  Fix Released
Status in masakari stein series:
  Fix Committed
Status in masakari train series:
  Fix Committed
Status in masakari ussuri series:
  Fix Committed
Status in masakari victoria series:
  Fix Released
Status in masakari package in Ubuntu:
  Fix Released
Status in masakari source package in Focal:
  Fix Committed
Status in masakari source package in Groovy:
  Fix Released

Bug description:
  [Impact]
  masakari-engine has two periodic tasks, one for processing 'new' notifications and the other for processing 'error' notifications But it doesn't have a periodic task for processing 'running' notifications.

  Looking at the code of masakari-engine, if the process of it goes down
  immediately after it changes notification status from 'new' to
  'running', then the notification which status is 'running' will remain
  will not be processed by periodic tasks.

  So, should masakari-engine's periodic task process the 'running' notification?
  (Although it need to make such a logic that main process doesn't compete with periodic tasks.)
  Or should the 'running' notification be handled by the operator?

  [Test Case]
  lxc launch ubuntu-daily:groovy g1 (or other corresponding release combination)
  lxc exec g1 /bin/bash
  sudo apt install masakari-engine

  == expect test failure with old code ==
  setup:
  * copy new test code from patch to /usr/lib/python3/dist-packages/masakari/tests/unit/engine/test_engine_mgr.py
  * modify /usr/lib/python3/dist-packages/masakari/tests/unit/engine/test_engine_mgr.py to set EXPIRED_TIME = NOW, and comment out call to test_check_expired_notifications since it doesn't exist without new patch applied.
  test:
  * cd /usr/lib/python3/dist-packages
  * python3 -m unittest masakari.tests.unit.engine.test_engine_mgr.EngineManagerUnitTestCase.test_check_expired_notifications

  == expect test success with patched code ==
  setup: enable corresponding -proposed pocket
  test:
  * cd /usr/lib/python3/dist-packages
  * python3 -m unittest masakari.tests.unit.engine.test_engine_mgr.EngineManagerUnitTestCase.test_check_expired_notifications

  [Regression Potential]
  A regression in this code could occur if either of the time intervals were calculated incorrectly which means a notification could be marked as failed perhaps long before the expiration interval. The defaults can be changed for check_expired_notifications_interval and notifications_expired_interval via masakari-engine config options which helps mitigate that risk. The defaults seem reasonable with a 10 minute periodic check and setting expired notifications to failed status only after 24 hours.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-masakari/+bug/1773765/+subscriptions