[ubuntu-us] Ubuntu US cluster

[Tony Yarusso]

On Thu, May 14, 2009 at 10:20 PM, Nick Ali <nali at ubuntu.com> wrote:
> During UDS Jaunty, Dan and I had talked about creating an Ubuntu US
> cluster of servers that LoCos could use. We all know the various
> problems with trying to use Canonical resources. Assuming we get to
> the point where every LoCo in the US is approved, each LoCo worrying
> about hosting resources is a waste of time, energy, and money.

This sounds like the sort of thing that has to happen at some point -
it's just a question of when we're able to pull it off.  Thanks for
bringing it up.  (btw, unapproved teams often use web resources too -
they just can't be Canonical's.  We'd have to make a decision about
whether those are allowed under this system.)

> This email is just a brain dump, not organized in any way:
>
>  - initially we could start out with a bunch of VPS accounts. A few
> for web servers, a few for databases. Depending on needs, maybe we
> don't even need to separate them out.

Absolutely.  It would be helpful to know traffic / load statistics
from existing sites to know what level / number of such accounts would
be needed to start.

>  - if VPS accounts don't work because of loads, we could look into
> getting our own servers.

My off-the-cuff estimate thinks this is probably a ways down the road,
but I could be horribly wrong.

>  - ubuntu-eu owns a boat load of servers, mostly used machines that they bought

What has their experience been with those machines?  Have they had any
trouble due to their used nature?

>  - noris (which is smurf's company)  pays for the datacenter costs,
> then -eu did fundraising to buy the servers, maybe we could do
> something similar or find a sponsor. Hopefully, just for VPSs we could
> get someone like linode or slicehost to pick up part of the cost if we
> let them add a banner or something.

One I have in mind for the server hardware end is approaching
System76, as they are a US company that has repeatedly shown that they
are interested in the Ubuntu community.

>  - since most LoCos use drupal, we could be set up (hopefully) one big
> drupal install and map the subdomains to use "sites" under it. Give
> each LoCo their own theme. There should also be some sort of access
> control. Can't trust the Florida team with the Georgia team's site :-P

Basic UNIX file permissions will be sufficient for this - Drupal is
designed to work quite well for such situations.

>  - The drupal install should be pretty basic (and/or with a set of
> approved modules), no random "cool" modules just for the hell of it.
> This will make upgrades and maintenance much easier.

There has been some nifty work going on lately by the Ubuntu Drupal
team to set up just such a package of things commonly used for LoCo
sites.  I haven't had time to check it out yet, but it sounds like a
good place to start.

>  - we might want to separate out the resources into 2 parts: official
> supported resources and experimental. Official would be the main LoCo
> websites, with planets, etc. Experimental can be for building and
> testing out new tools that *might* eventually get merged into official
> after some kind of vetting process.

I would propose a third category - things a particular LoCo wants to
use, but are unsupported.  For instance, while we may not want to
maintain an installation of everything under the sun, perhaps a
particular team would like to use a Drupal module that we don't, or as
one reply has mentioned use Django instead.  I would hope that we
could be more liberal about allowing such things than Canonical has
been.

>  - If part of the reason we want to move away from Canonical is their
> slow responsiveness, we need to find some sysadmins who we can trust
> to be around and accessible at least during US times. There are a lot
> of people who are active for 2 or 3 months and then disappear. We
> can't have that.

And preferably a lot of them - tiny bits of work among many is the
only way this will work, since I doubt anyone is willing to give up
their full-time job to do this pro bono...

>  - Does one person from each LoCo get access to the servers? Or a
> small team that is responsible for maintaining it? Everything needs to
> be locked down with ssh keys and strong passwords.

Depends on the team.  The Team Contact should be the authoritative
source for communicating to us who those people will be, but multiple
people should be allowed.  Use the SSH keys from those people's
Launchpad pages.  (I believe there is even a facility for
automatically downloading those given a list of usernames - I think I
bookmarked it on one of these machines...)

>  - Who can make requests? Team contacts or anyone from the LoCos?

By requests do you mean requests for access or requests for action,
like "add $module to our account"?  If you mean access, I would say
that would be appropriate to go through the Team Contacts.

>  - Who gets final say? Say person A wants to do something one way,
> person B want to do it differently. Who gets to make the call?

Again, can you be more specific?  At this level, there would probably
need to be a US Team meeting of some sort to address some matters, and
depending on the issue some things would probably come down to just a
vote while others may be delegated to a small new team of individuals.
 If you mean for individual LoCo sites, that will need to be
determined within the LoCos through their own processes and
communicated upstream through the TCs.

>  - Back to the official vs experimental: experimental can have a copy
> of the official databases, with passwords scrubbed out so LoCos can
> play with it. An obvious example would be creating a new theme. LoCos
> can actually see what it looks like with real data and once its
> working to their liking, it can be deployed to the official servers.

Kind of interesting, but I'm not sure how it would be applied yet.
Why wouldn't teams just be able to use a copy of their own database
only?  Which database(s) were you thinking of making available?

>  - Backups are critical. Maybe all files are checked into LP (after
> scrubbing passwords)? Maybe a cron job to dump the databases nightly
> and email them to a bunch of people?

I like the idea of using version control, both for backups and just
tracking what changes have been made.  I'm not at all thrilled about
trying to e-mail databases regularly - e-mail just isn't designed for
that.  If we ever did e-mail database dumps, the file needs to be
encrypted with GPG.

>  - Some kind of issue tracking application would be nice. Issues that
> sysadmins need to deal with including priorities.

Can we just use Launchpad's bugs facility for this?  My irks with LP
are supposed to be remedied by June or July, so I'm okay with using it
again, and it seems silly to set up yet another system unless we
really have to.

>  - Some kind of collaboration tool would be nice. I think doctormo had
> talked about something similar before. If we are doing a campaign
> where lots of LoCos could be part of, and the wiki isn't a good way of
> tracking, something to help us with organization and workflow.

For collaborating on documents, gobby is pretty cool, and we could run
an instance of the server.  IRC obviously is very useful for
conversation.  I'm not sure exactly what else you had in mind on this
point.

> Dan made an interesting point that I hadn't thought about it. What
> exactly defines a LoCo website? I've been assuming the typical Drupal
> site. Does it have to be that? Who gets to decide?

The LoCo's individual needs define a LoCo web site.  If that happens
to be a Drupal install, great.  If not, we should allow for that too.
As I alluded to before, there should be some set of things that we're
willing to support in terms of dealing with problems and helping
people set up and maintain, but also allow for some level of "here's
your space, you're on your own" kinds of installations for the teams
that want something different.

> Ideas? Thoughts? Too much work? Or should we all just use the -eu resources?
> nick

I like it.  I think it's ultimately less work than either trying to do
everything through Canonical or making every team figure out their own
solution.  I like the idea of a US solution separate from -eu's, as
this allows for dealing with any needs that are different this side of
the pond plus potentially better network connectivity for the users.

In terms of specifics, I would like to put my vote solidly in favor of
using Linode for VPSs to start.  I chose them for my personal use
based on them being by far the most recommended among my contacts in
Ubuntuland, and haven't been disappointed.  They are US-based, have
reasonable pricing and features, and seem pretty on top of Ubuntu
things internally (for instance, Jaunty was available as a
ready-to-deploy image within hours of release).

I would also like to raise the issue of non-US LoCos for
consideration.  Would we consider allowing teams from the region
outside of the US use these resources?  For instance, I help maintain
the Canadian Team's site, which is hosted on Canonical's resources,
and as such has pretty much stagnated since nobody has time to jump
through the endless hoops it takes to get anything done.  I haven't
mentioned it to the rest of that team yet, but if we were willing to
include other teams from the Americas they might be interested.

And yes, I'd be willing to help and have sysadmin and Drupal experience.  :P

-- 
Tony Yarusso
http://tonyyarusso.com/