[Bug 2086710] Re: watcher's use of apscheduler is incompatible with python 3.12 and eventlet

[James Page]

** Changed in: watcher (Ubuntu Noble)
   Importance: Undecided => High

** Changed in: watcher (Ubuntu Oracular)
   Importance: Undecided => High

** Changed in: watcher (Ubuntu Plucky)
   Importance: Undecided => High

** Changed in: watcher (Ubuntu Noble)
       Status: New => Triaged

** Changed in: watcher (Ubuntu Oracular)
       Status: New => Triaged

** Changed in: watcher (Ubuntu Plucky)
       Status: New => Triaged

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to watcher in Ubuntu.
https://bugs.launchpad.net/bugs/2086710

Title:
  watcher's use of apscheduler is incompatible with python 3.12 and
  eventlet

Status in watcher:
  In Progress
Status in watcher package in Ubuntu:
  Triaged
Status in watcher source package in Noble:
  Triaged
Status in watcher source package in Oracular:
  Triaged
Status in watcher source package in Plucky:
  Triaged

Bug description:
  in the newton release a background job scheduler was added to the
  Decision Engine.

  
  https://github.com/openstack/watcher/commit/06c6c4691b103bf0b3fd3304a1a45fb22aedad50

  to facilitate this the apscheduler lib was introduced as a depency to watcher.
  apscheduler has a lost of capability but does not officially support eventlet.

  since its introduction to watcher it has mostly worked partly by accident.
  over the year as oslo, apscheduler and eventlet have evolved and adapted to newer python
  release watcher has continued to use apscheduler even though that is not technically supported.

  with the move to python 3.12 it became apparent that the background jobs executed on the apscheduler
  BackgroundScheduler instances were accellign shared global state from a non-monkeypatched native thread.

  that results in greenthread sometimes calling into objects that are
  using un monkey patched code.

  for example oslo.db uses time.sleep to yield executions.
  when that oslo.db function is first imported from a non patched thread if its invoked after that in the main thread it will block.

  this can by this expction "RuntimeError: do not call blocking
  functions from the mainloop" here
  https://paste.opendev.org/show/bGPgfURx1cZYOsgmtDyw/

  this has been repdocuded in ci as part of moving the ci jobs to ubutnu
  24.04 and python 3.12

  https://review.opendev.org/c/openstack/watcher/+/932963/comments/f54005d7_b0f831bb

  to address this issue we need to ensure that the background thread
  used to schedule background task is properly monkey patched.

To manage notifications about this bug go to:
https://bugs.launchpad.net/watcher/+bug/2086710/+subscriptions