Warning: do not use state in API server initialisation

[roger peppe]

We have recently made a change in a feature branch that causes the API
server to read the environment configuration from the mongo state at
startup. This seemed reasonable because the API server layers on top
of the mongo state, so should be able to use it anywhere it likes,
surely... ?

Unfortunately our change triggered a problem [†] where we could not
upgrade from an older system. The problem was a Catch-22 situation:
the environment configuration format has changed recently, so the
API server could not start until the format has been upgraded, but the
upgrader worker requires the API server to be started before it can run.

One way to fix this might be to run upgrades.PerformUpgrades twice,
once for state-specific upgrade steps, and again for the steps that
require an API connection.

After some consideration, I think that is the wrong approach.  For one, it
makes the upgrade logic considerably more complex (you need to coordinate
two things rather than one).  For another, shutting everything out of the
API is an extreme thing to do; currently the only thing that can cause the
API server initialisation to fail is if it has an invalid TLS certificate,
which is something that really *is* required to run the API server.

So I suggest we make it a rule that the apiserver.NewServer should never
read the state before any API requests have been received, and that we
add an appropriate doc comment to that effect.

The down side of this is that there is the possibility of erroneously
making API requests that expect an upgraded version of the state,
but we are careful to avoid this in the existing logic by refusing
logins until the state has been upgraded and by running all state-server
upgrades before API-based upgrades in the upgrades package. We can make
it a little more certain still by ensuring that the API server is not
actually available to state server upgrade steps.

Any thoughts?

  cheers,
      rog.

[†] https://bugs.launchpad.net/juju-core/+bug/1504578