[Bug 1832105] Re: ceph-mgr dashboard incompatible with cython >= 0.29 (disco)

[Harry Coin]

Here's some help for others facing this:

If the ceph dashboard was working in before upgrading to disco (which 
killed it in a regression), then your hope to get it working via upgrade 
to nautilus (owing to 'fixed-released' advertising in the bug report) 
was to move to ceph v14/nautlius available in ubuntu-eoan.

After 'do-release-upgrade --devel' to eoan / ceph nautilus on every 
system running ceph do:

systemctl status ceph<esc>  and make sure there is only one entry there 
for every osd/mon/mgr/mds.  On my system there were entries there with 
the hostname and with the hostname.domainname as well.

There are a number of other instructions involved in getting nautilus 
running, see them here:

http://docs.ceph.com/docs/nautilus/releases/nautilus/

Also one of the osd's that was not managed by LVM was ignored and not 
started.  I 'replaced it' with itself and it started backfilling normally.

On the systems meant to run the dashboard, this is now necessary:

apt install ceph-mgr-dashboard

    The following will tell you 'the module is already enabled'.

And when you think you're done and ready to log in ... the screen 
accepts your password then does nothing further other than redisplay the 
login screen.  If you put in the wrong password, it tells you. The 
correct password does nothing.  So,  __on every instance of ceph mgr 
even the ones you are not using __ you have to

ceph mgr module disable dashboard

then edit

/usr/share/ceph/mgr/dashboard/services/access_control.py

and change line 186 from self.lastUpdate =
int(time.mktime(time.gmtime()))

to

self.lastUpdate = int(time.time())

Be sure to use spaces and not tabs.

Then

ceph mgr module enable dashboard

ceph dashboard ac-user-set-password admin <new password>

And then, you get to where you were before the update to disco with a 
working dashboard.  Hopefully this saved you a day or two.

I'm no longer able/interested to test whether mimic's dashboard works in 
disco, sorry.   If I'd somehow known an official release would break a 
major user facing function on something as central to operations as ceph 
I would have skipped disco entirely and waited for eoan.


On 7/12/19 8:33 AM, James Page wrote:
> Sorry wrong PPA:
>
> https://launchpad.net/~ci-train-ppa-
> service/+archive/ubuntu/3534/+packages
>
> ** Description changed:
>
> - If Ubuntu is really committed to ceph as I think I've been reading:
> - Notice the ceph dashboard went entirely broken in a major regression of
> - the disco upgrade.  It won't load at all in 13.2.4+dfsg1-0ubuntu2.
> + [Impact]
> + The ceph-mgr daemon is unable to load additional module due to a new check in cython >= 0.29.  This limits the function of the manager.
> +
> +
> + [Test Case]
> + Deploy ceph
> + Check /var/log/ceph/ceph-mgr.`hostname`.log
> + Errors about loading rados module in subprocesses will be seen.
> +
> + [Regression Potential]
> + The fix from upstream actually just works around this issue by overriding the check that cython does; the code works in a subprocess when loaded multiple times.  Regression potential low; cython may produce a longer term fix which means we can drop this patch.
> +
> + [Original Bug Report]
> + If Ubuntu is really committed to ceph as I think I've been reading:  Notice the ceph dashboard went entirely broken in a major regression of the disco upgrade.  It won't load at all in 13.2.4+dfsg1-0ubuntu2.
>    
>    The detail is ceph-mgr (and lots of ceph) relied on a non-feature in
>    cython that went away in cython v29, to do with sub-interpreters.  The
>    ceph folks responded with a hack/workaround to avoid the bug being
>    noticed, and a requirement of the package for an earlier version of
>    cython.  This was done some weeks and months ago.  Actually fixing the
>    problem is a major project the ceph maintainers are struggling to
>    engage, perhaps waiting for later versions of cython to provide a
>    different way forward.
>    
>    However, as of today, on disco ths error message remains:
>    
>    Module 'dashboard' has failed dependency: Interpreter change detected -
>    this module can only be loaded into one interpreter per process.
>    
>    The ceph primary development platform is Debian, on which the workaround
>    has been available for some time.
>    
>    However in our ubuntu case, a major feature of a core packge (web
>    health/monitoring/config interface of  a distributed file system), was
>    allowed to both ship broken and remain so for a long time, even through
>    today.
>    
>    I urge quick attention to the necessary backports.
>    https://github.com/ceph/ceph/pull/25585
>    http://tracker.ceph.com/issues/38788
>    http://tracker.ceph.com/issues/37472
>

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1832105

Title:
  ceph-mgr dashboard incompatible with cython >= 0.29 (disco)

Status in ceph package in Ubuntu:
  Fix Released
Status in ceph source package in Disco:
  In Progress
Status in ceph source package in Eoan:
  Fix Released

Bug description:
  [Impact]
  The ceph-mgr daemon is unable to load additional module due to a new check in cython >= 0.29.  This limits the function of the manager.

  
  [Test Case]
  Deploy ceph
  Check /var/log/ceph/ceph-mgr.`hostname`.log
  Errors about loading rados module in subprocesses will be seen.

  [Regression Potential]
  The fix from upstream actually just works around this issue by overriding the check that cython does; the code works in a subprocess when loaded multiple times.  Regression potential low; cython may produce a longer term fix which means we can drop this patch.

  [Original Bug Report]
  If Ubuntu is really committed to ceph as I think I've been reading:  Notice the ceph dashboard went entirely broken in a major regression of the disco upgrade.  It won't load at all in 13.2.4+dfsg1-0ubuntu2.

  The detail is ceph-mgr (and lots of ceph) relied on a non-feature in
  cython that went away in cython v29, to do with sub-interpreters.  The
  ceph folks responded with a hack/workaround to avoid the bug being
  noticed, and a requirement of the package for an earlier version of
  cython.  This was done some weeks and months ago.  Actually fixing the
  problem is a major project the ceph maintainers are struggling to
  engage, perhaps waiting for later versions of cython to provide a
  different way forward.

  However, as of today, on disco ths error message remains:

  Module 'dashboard' has failed dependency: Interpreter change detected
  - this module can only be loaded into one interpreter per process.

  The ceph primary development platform is Debian, on which the
  workaround has been available for some time.

  However in our ubuntu case, a major feature of a core packge (web
  health/monitoring/config interface of  a distributed file system), was
  allowed to both ship broken and remain so for a long time, even
  through today.

  I urge quick attention to the necessary backports.
  https://github.com/ceph/ceph/pull/25585
  http://tracker.ceph.com/issues/38788
  http://tracker.ceph.com/issues/37472

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1832105/+subscriptions