[Maas-devel] Scaling to 72k nodes

[Julian Edwards]

On Thursday 18 Oct 2012 21:58:59 John Arbash Meinel wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 10/18/2012 6:00 PM, Francis J. Lacoste wrote:
> > On 12-10-17 05:02 AM, John Arbash Meinel wrote:
> >> 18 nodegroups, each nodegroup has 4,000 nodes for 72,000 total
> >> nodes. Each node has a ~25kB XML string associated with it.
> > 
> > The 1 cluster controller can host 4k nodes is an assumption we
> > haven't empirically validated.
> > 
> > We should probably either validate it, or try variation on the the
> > number of nodes to clusters ratio (say with 500 per node, 1000,
> > 2000), just to see how the scaling behave differently.

We've based the 4k on references (which I don't have to hand) that indicate 
that broadcast traffic and DHCP server performance becomes problematic at that 
level.  (Having said that, in a case study with a customer whose name I cannot 
mention here had 5k nodes with one DHCP server on a flat network)

So we should treat it as an advised maxmium and recommend that the real 
maximum will depend on your networking usage.

> I'll poke one more time, but being able to say "rebuilding tags from 1
> Region Controller to 20 Cluster Controllers" is network bandwidth
> bound, rather than CPU bound or DB load bound is a pretty good place
> to be.

Is rebuilding all tags at once across all machines a realistic scenario?  I 
would have thought that a more likely one is the introduction of a new rack 
(or racks) containing a few nodes at a time.

Cheers.