Changes to LoCo Server Policy
jono at ubuntu.com
Sat Aug 11 01:23:46 BST 2007
Recently we have had some problems with regards to LoCo servers, and the
situation has become untenable. I spent some time this week chatting
with James Troup, also known as elmo, and he leads the Canonical
sysadmin team. He has written up a mail outlining the issues well and
summarising the change in policy. This is important to encourage great
services, but also secure services.
I would like to schedule an IRC meetin in #ubuntu-locoteams on Tuesday
14th August at 2pm UTC to discuss any of the issues covered here.
The email is to follow:
On Monday evening (UK time) it was reported that one of the hosted
community servers that Canonical sponsors had been compromised. After
investigation, it became apparent that 5 of the 8 machines had been
compromised. Since it was reported that they were actively attacking
other machines (and because it's What You Do), the decision was taken to
shut the machines down.
On Tuesday morning we started the procedure of bringing these machines
up in a safe state so that we could recover data from them.
Unfortunately, this took far longer than we would have hoped or liked
due to a combination of having to use remote hands, arbitrary limits
imposed by those remote hands and (relative) lack of bandwidth to copy
data off site.
This process is still ongoing (though only one remain has yet to be
fully recovered - tiber).
How did this happen
a) the servers, especially zambezi were running an incredible
amount of web software (over 15 packages that we recognised)
of all the ones where it's trivial to determine a version, they
were without exception out-of-date and missing security patches.
An attacker could have gotten a shell through almost any of
b) FTP (not sftp, without SSL) was being used to access the
machines, so an attacker (in the right place) could also have
gotten access by sniffing the clear-text passwords.
c) The servers have not been upgraded past breezy due to problems
with the network card and later kernels. This probably allowed
the attacker to gain root.
What happens next
We're obviously working as fast as we can to restore services, however,
we need to make sure they won't immediately be compromised
Our first thought (and as previously mentioned on the loco-contacts
mailing list) was to simply move all these services into the Canonical
data centre, which would solve (b) and (c) above. However, at the time,
we weren't aware of (a).
Unfortunately it's simply not possible for us to maintain that amount of
software in any sane or secure fashion. So we've changed plan
slightly and now plan to do two things:
Loco teams/services can choose to either:
(1) be migrated to the Canonical data centre. This comes with both
restrictions and benefits:
+ Better hardware and bandwidth.
+ Fulltime support from Canonical's sysadmin team including
software maintenance and integration into our existing backup
- root access will not be available.
- Access by per-user SSH key only, limited number of accounts
per loco team / service.
- Can only support certain software (e.g. drupal, wordpress,
planet, moin, ...).
- No ability to run arbitrary CGIs.
(2) or stay on the hosted/outsourced servers.
However, assuming anyone chooses option (2), some things will have to
change with how we handle these servers. Specifically, Canonical will
continue to sponsor the servers but they will have to become entirely
community run, i.e.
+ Community admin team liaise with hosting company for reboots, etc.
+ Community have sole responsibility for all aspects of
administration of servers, including but not limited to day to
day sysadmin tasks, backups, security, upgrades, recovery if
+ Use of servers for loco team services only unless previously
- Both the Community Council and Canonical have oversight on this
 art-web, gallery, drupal, phpmyadmin, wordpress, postnuke, phpbb,
smf, moodle, planet, aspseek, moin, taskfreak, cms made simple,
So, if you have any questions, come to the meeting and discuss them
there where both James and I will be present. :)
Ubuntu Community Manager
www.ubuntu.com / www.jonobacon.org
More information about the loco-contacts