[Bug 1773765] Re: There is a possibility that 'running' notification will remain

[Corey Bryant]

** Changed in: masakari (Ubuntu Focal)
   Importance: Medium => High

** Changed in: masakari (Ubuntu Focal)
       Status: New => Triaged

** Changed in: masakari (Ubuntu Groovy)
   Importance: Medium => High

** Changed in: masakari (Ubuntu Groovy)
       Status: New => Triaged

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1773765

Title:
  There is a possibility that 'running' notification will remain

Status in Ubuntu Cloud Archive:
  In Progress
Status in Ubuntu Cloud Archive stein series:
  In Progress
Status in Ubuntu Cloud Archive train series:
  In Progress
Status in Ubuntu Cloud Archive ussuri series:
  In Progress
Status in Ubuntu Cloud Archive victoria series:
  In Progress
Status in masakari:
  Fix Released
Status in masakari stein series:
  Fix Committed
Status in masakari train series:
  Fix Committed
Status in masakari ussuri series:
  Fix Committed
Status in masakari victoria series:
  Fix Released
Status in masakari package in Ubuntu:
  Triaged
Status in masakari source package in Focal:
  Triaged
Status in masakari source package in Groovy:
  Triaged

Bug description:
  [Impact]
  masakari-engine has two periodic tasks, one for processing 'new' notifications and the other for processing 'error' notifications But it doesn't have a periodic task for processing 'running' notifications.

  Looking at the code of masakari-engine, if the process of it goes down
  immediately after it changes notification status from 'new' to
  'running', then the notification which status is 'running' will remain
  will not be processed by periodic tasks.

  So, should masakari-engine's periodic task process the 'running' notification?
  (Although it need to make such a logic that main process doesn't compete with periodic tasks.)
  Or should the 'running' notification be handled by the operator?

  [Test Case]
  lxc launch ubuntu-daily:groovy g1 (or other corresponding release combination)
  lxc exec g1 /bin/bash
  sudo apt install masakari-engine

  == expect test failure with old code ==
  setup:
  * copy new test code from patch to /usr/lib/python3/dist-packages/masakari/tests/unit/engine/test_engine_mgr.py
  * modify /usr/lib/python3/dist-packages/masakari/tests/unit/engine/test_engine_mgr.py to set EXPIRED_TIME = NOW, and comment out call to test_check_expired_notifications since it doesn't exist without new patch applied.
  test:
  * cd /usr/lib/python3/dist-packages
  * python3 -m unittest masakari.tests.unit.engine.test_engine_mgr.EngineManagerUnitTestCase.test_check_expired_notifications

  == expect test success with patched code ==
  setup: enable corresponding -proposed pocket
  test:
  * cd /usr/lib/python3/dist-packages
  * python3 -m unittest masakari.tests.unit.engine.test_engine_mgr.EngineManagerUnitTestCase.test_check_expired_notifications

  [Regression Potential]
  A regression in this code could occur if either of the time intervals were calculated incorrectly which means a notification could be marked as failed perhaps long before the expiration interval. The defaults can be changed for check_expired_notifications_interval and notifications_expired_interval via masakari-engine config options which helps mitigate that risk. The defaults seem reasonable with a 10 minute periodic check and setting expired notifications to failed status only after 24 hours.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1773765/+subscriptions