thoughts on priorities

[William Reade]

On Thu, 2013-05-02 at 11:40 +0100, roger peppe wrote:
> Not at all. But I think a cheap (and sufficient and not inelegant) solution
> is just to move the version into another document and have a watcher
> for that, meaning that almost all our logic remains unchanged and
> we have more time for implementing other stuff.

Well, I'm open to discussion... but my instinct argues against it at the
moment. ISTM that the bulk of the logic under threat *is* the dance from
config-watcher to environment to tools-list to version-picking. I'm not
sure how changing the environment config data in order to maintain a
single aspect of that is going to be cheaper than recording machine arch
(which can be done over the API before we start watching the machine
config over the API, so we can guarantee that field'll be filled in
before watching, without demanding a major version change), and wrapping
the existing watcher so that we only deliver the tools we know we need
(series is already known to the machine).

> This is something that every agent will be watching. If every agent
> is watching for exactly the same data (the global version changing)
> then we're doing less work on the API hub and we can potentially
> use less resources by caching the information there.

If having a machine available implies having series/arch available, I
think it's probably smarter to just send the necessary information down.
I don't think that N series/arch-specific watchers is much worse than N
identical environment watchers...

> If the version changes, an agent *will* respond to it, apart from
> in the rare case that the tools are not available for that agent. That's a case
> not a case we need to be too concerned about IMO.

I'm more concerned about every agent responding by making an API
roundtrip to gather loads of data it doesn't need. The thundering herds
of Next()s will be dwarfed by those, won't they?

> I also feel that from an architectural point of view, it makes sense
> to have each agent responsible for choosing what tools it will run.
> I'm not sure the centralised logic is a big help here, and
> may make some things harder in the future - for example
> in a cross-provider juju we may want different agents to fetch
> from different tools repos.

I'm not sure the distributed answer solves it any better or cheaper
either. I'm not sure where you got the idea of cross-provider
environments from, though: the common wisdom is that cross-environment
relations will handle that use case.

> > Yes; but to do that we'd need to know the arches of all the machines in
> > play; and if we had all that information accessible to the API in the
> > first place I'm not sure why we'd want to spend roundtrips on getting
> > redundant data, rather than just sending the right stuff in the first
> > place.
> 
> I'm not sure I understand what you're saying here.

Arch information is valuable for other use cases, even if they're not
top priority. Arch information per machine also helps us to implement a
tools watcher that's (1) accurate and (2) causes a smaller traffic spike
on the server. I'm not sure the benefit of preserving a fragment of the
original logic in-place is overwhelming.

(I just wish we could update all the tools on a machine at the same
time, to avoid watching (and, probably, concurrently downloading) N
identical copies of the tools. Same problem as major-version upgrades,
in a way, just writ small.)

> > Hmm, I think that's still potentially racy. If *agents* cut off their
> > own API access, though... yeah, we can probably make it work. Nice.
> 
> It's only racy if there are still some agents around that are directly
> accessing mongo AFAICS. If by "cutting off their own API access"
> you mean "stop talking to mongo directly", then I'm with you.

Well, I mean "starting up talking to the API only, and explicitly
revoking their own state access by making some API call". But then I'm
not sure how we'd downgrade that safely... compatibility is a tighter
constraint than it seems at first glance, and a "make things insecure"
API is I think somewhat questionable.

Cheers
William