[Bug 1773765] Re: There is a possibility that 'running' notification will remain

OpenStack Infra 1773765 at bugs.launchpad.net
Tue Nov 10 00:51:39 UTC 2020


Reviewed:  https://review.opendev.org/758899
Committed: https://git.openstack.org/cgit/openstack/charm-masakari/commit/?id=62bc9c05b9cd438d45067e0efafe6fa75ef5b500
Submitter: Zuul
Branch:    master

commit 62bc9c05b9cd438d45067e0efafe6fa75ef5b500
Author: Billy Olsen <billy.olsen at gmail.com>
Date:   Mon Oct 19 19:48:26 2020 -0700

    Add config options for expired notifications
    
    Adds config options to configure the notification expiration time
    as well as how frequent to check for expired notifications.
    
    Change-Id: I9607f01a800baded37cfc23b6bc9f8e6aecc2d74
    Closes-Bug: #1773765


** Changed in: charm-masakari
       Status: In Progress => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1773765

Title:
  There is a possibility that 'running' notification will remain

Status in OpenStack Masakari Charm:
  Fix Committed
Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive stein series:
  Fix Released
Status in Ubuntu Cloud Archive train series:
  Fix Released
Status in Ubuntu Cloud Archive ussuri series:
  Fix Released
Status in Ubuntu Cloud Archive victoria series:
  Fix Released
Status in masakari:
  Fix Released
Status in masakari stein series:
  Fix Committed
Status in masakari train series:
  Fix Committed
Status in masakari ussuri series:
  Fix Committed
Status in masakari victoria series:
  Fix Released
Status in masakari package in Ubuntu:
  Fix Released
Status in masakari source package in Focal:
  Fix Released
Status in masakari source package in Groovy:
  Fix Released

Bug description:
  [Impact]
  masakari-engine has two periodic tasks, one for processing 'new' notifications and the other for processing 'error' notifications But it doesn't have a periodic task for processing 'running' notifications.

  Looking at the code of masakari-engine, if the process of it goes down
  immediately after it changes notification status from 'new' to
  'running', then the notification which status is 'running' will remain
  will not be processed by periodic tasks.

  So, should masakari-engine's periodic task process the 'running' notification?
  (Although it need to make such a logic that main process doesn't compete with periodic tasks.)
  Or should the 'running' notification be handled by the operator?

  [Test Case]
  lxc launch ubuntu-daily:groovy g1 (or other corresponding release combination)
  lxc exec g1 /bin/bash
  sudo apt install masakari-engine

  == expect test failure with old code ==
  setup:
  * copy new test code from patch to /usr/lib/python3/dist-packages/masakari/tests/unit/engine/test_engine_mgr.py
  * modify /usr/lib/python3/dist-packages/masakari/tests/unit/engine/test_engine_mgr.py to set EXPIRED_TIME = NOW, and comment out call to test_check_expired_notifications since it doesn't exist without new patch applied.
  test:
  * cd /usr/lib/python3/dist-packages
  * python3 -m unittest masakari.tests.unit.engine.test_engine_mgr.EngineManagerUnitTestCase.test_check_expired_notifications

  == expect test success with patched code ==
  setup: enable corresponding -proposed pocket
  test:
  * cd /usr/lib/python3/dist-packages
  * python3 -m unittest masakari.tests.unit.engine.test_engine_mgr.EngineManagerUnitTestCase.test_check_expired_notifications

  [Regression Potential]
  A regression in this code could occur if either of the time intervals were calculated incorrectly which means a notification could be marked as failed perhaps long before the expiration interval. The defaults can be changed for check_expired_notifications_interval and notifications_expired_interval via masakari-engine config options which helps mitigate that risk. The defaults seem reasonable with a 10 minute periodic check and setting expired notifications to failed status only after 24 hours.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-masakari/+bug/1773765/+subscriptions



More information about the Ubuntu-openstack-bugs mailing list