[Bug 1827690] Re: [19.04][stein] barbican-worker is down: Requested revision 1a0c2cdafb38 overlaps with other requested revisions 39cf2e645cba

[David Ames]

Doing some research.

1) I don't think this is more than one charm unit trying to do migrations at the same time (comment #2)
2) The problem is barbican itself failing per the Traceback
3) My current theory is a failed migration attempt that leaves behind two alembic versions.

Per [1] we should only ever expect 1 alembic version at a time. As a
work around we may need to do some DB surgery per [1]. After a backup of
course.

To find what gets barbican into this state will take a bit more digging
through logs.

[1] https://stackoverflow.com/questions/42424320/how-do-i-fix-alembics-
requested-revision-overlaps-with-other-requested-revisio

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to barbican in Ubuntu.
https://bugs.launchpad.net/bugs/1827690

Title:
  [19.04][stein] barbican-worker is down: Requested revision
  1a0c2cdafb38 overlaps with other requested revisions 39cf2e645cba

Status in OpenStack Barbican Charm:
  In Progress
Status in barbican package in Ubuntu:
  New

Bug description:
  After deploying barbican (stein, 19.04 charms) I got into a situation
  where both workers (two-node HA) are down due to a failed alembic
  migration:

  https://private-fileshare.canonical.com/~dima/charm-dumps/04-05-2019-barbican-0-var-log.tar.gz
  https://private-fileshare.canonical.com/~dima/charm-dumps/04-05-2019-barbican-1-var-log.tar.gz

  https://paste.ubuntu.com/p/J6GvwkXMWc/ - bundle.yaml

  May 03 20:51:11 juju-35b20e-3-lxd-0 systemd[1]: Started OpenStack Barbican Key Management Workers.
  May 03 20:51:39 juju-35b20e-3-lxd-0 systemd[1]: barbican-worker.service: Main process exited, code=exited, status=1/FAILURE
  May 03 20:51:39 juju-35b20e-3-lxd-0 systemd[1]: barbican-worker.service: Failed with result 'exit-code'.

  juju status barbican
  Model      Controller  Cloud/Region  Version  SLA          Timestamp
  openstack  samaas      samaas        2.6-rc1  unsupported  23:27:23Z

  App                 Version  Status   Scale  Charm           Store       Rev  OS      Notes
  barbican            8.0.0    blocked      2  barbican        jujucharms   18  ubuntu
  barbican-vault      1.2.2    active       2  barbican-vault  jujucharms    2  ubuntu
  hacluster-barbican           active       2  hacluster       jujucharms   54  ubuntu

  Unit                     Workload  Agent  Machine  Public address  Ports              Message
  barbican/0               blocked   idle   3/lxd/0  10.232.46.135   9311/tcp,9312/tcp  Services not running that should be: barbican-worker
    barbican-vault/1       active    idle            10.232.46.135                      Unit is ready
    hacluster-barbican/1   active    idle            10.232.46.135                      Unit is ready and clustered
  barbican/1*              blocked   idle   2/lxd/0  10.232.46.130   9311/tcp,9312/tcp  Services not running that should be: barbican-worker
    barbican-vault/0*      active    idle            10.232.46.130                      Unit is ready
    hacluster-barbican/0*  active    idle            10.232.46.130                      Unit is ready and clustered

  https://paste.ubuntu.com/p/BS3fHw287r/
  2019-05-03 21:13:46.582 115638 ERROR barbican alembic.util.exc.CommandError: Requested revision 1a0c2cdafb38 overlaps with other requested revisions 39cf2e645cba

  mysql> use barbican;
  mysql> select * from alembic_version;
  +--------------+
  | version_num  |
  +--------------+
  | 1a0c2cdafb38 |
  | 39cf2e645cba |
  +--------------+
  2 rows in set (0.00 sec)

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-barbican/+bug/1827690/+subscriptions