Juju upgrade-juju error on 1.25.0 in production

[Darryl Weaver]

Hi,

I have seen some very odd behaviour with Juju 1.25.0.
For some reason the Juju state server has gone into an upgrade mode and
reduced Juju's functionality to the bare minimum.

It is odd, because no upgrade was requested and now although Juju state
server reports that Juju is upgrading there does not appear to be any
attempt to upgrade the juju tools version that is currently installed.
Trying to use:
juju upgrade-juju --reset-previous-upgrade
Reports that an upgrade is in progress.

The output from Juju status only shows all units and machines as version
1.25.0 and does not report any newer version that it is attempting to
upgrade to.

Due to all the units spewing log file lines due to Juju state server being
unavailable, we have had to manually stop all Juju daemons on all units and
the juju state server just to stop the log files from filling up and
rotating constantly.  Unfortunately, this large log file output and
rotation means that the original cause of the problem is no longer in the
remaining log files to analyse.

My question is how does juju actually decide that it had received an
upgrade request and is this in mongodb somewhere or the config files on the
state server?
I'd like to check where the upgrade is flagged and preferably recover by
removing the upgrade request so Juju can be started again on the state
server and all units connect to it.

Another alternative is how to make Juju actually upgrade, but I do not know
which version Juju thinks it should upgrade to to manually download the
juju tools and recreate the links to the newer version of the tools
manually.  If that is the way forward where can I check the version Juju is
requesting and is it just a case of relinking the directory in
/var/lib/juju/tools/ or are there other steps required too?

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160517/d772fe25/attachment.html>