Changes to LoCo Server Policy

Jono Bacon jono at ubuntu.com
Sat Aug 11 01:23:46 BST 2007


Hi all,

Recently we have had some problems with regards to LoCo servers, and the
situation has become untenable. I spent some time this week chatting
with James Troup, also known as elmo, and he leads the Canonical
sysadmin team. He has written up a mail outlining the issues well and
summarising the change in policy. This is important to encourage great
services, but also secure services.

I would like to schedule an IRC meetin in #ubuntu-locoteams on Tuesday
14th August at 2pm UTC to discuss any of the issues covered here.

The email is to follow:

--------------

Hi,


What happened
-------------

On Monday evening (UK time) it was reported that one of the hosted
community servers that Canonical sponsors had been compromised.  After
investigation, it became apparent that 5 of the 8 machines had been
compromised.  Since it was reported that they were actively attacking
other machines (and because it's What You Do), the decision was taken to
shut the machines down.

On Tuesday morning we started the procedure of bringing these machines
up in a safe state so that we could recover data from them.
Unfortunately, this took far longer than we would have hoped or liked
due to a combination of having to use remote hands, arbitrary limits
imposed by those remote hands and (relative) lack of bandwidth to copy
data off site.

This process is still ongoing (though only one remain has yet to be
fully recovered - tiber).

How did this happen
-------------------

Unfortunately:

   a) the servers, especially zambezi were running an incredible
      amount of web software (over 15 packages[1] that we recognised)
and
      of all the ones where it's trivial to determine a version, they
      were without exception out-of-date and missing security patches.
      An attacker could have gotten a shell through almost any of
      these sites.

   b) FTP (not sftp, without SSL) was being used to access the
      machines, so an attacker (in the right place) could also have
      gotten access by sniffing the clear-text passwords.
 
   c) The servers have not been upgraded past breezy due to problems
      with the network card and later kernels.  This probably allowed
      the attacker to gain root.

What happens next
-----------------

We're obviously working as fast as we can to restore services, however,
we need to make sure they won't immediately be compromised
again.

Our first thought (and as previously mentioned on the loco-contacts
mailing list) was to simply move all these services into the Canonical
data centre, which would solve (b) and (c) above.  However, at the time,
we weren't aware of (a).

Unfortunately it's simply not possible for us to maintain that amount of
software in any sane or secure fashion.  So we've changed plan
slightly and now plan to do two things:

Loco teams/services can choose to either:

 (1) be migrated to the Canonical data centre.  This comes with both
     restrictions and benefits:

       + Better hardware and bandwidth.
       + Fulltime support from Canonical's sysadmin team including
         software maintenance and integration into our existing backup
         infrastructure.

       - root access will not be available.
       - Access by per-user SSH key only, limited number of accounts
         per loco team / service.
       - Can only support certain software (e.g. drupal, wordpress,
         planet, moin, ...).
       - No ability to run arbitrary CGIs.

 (2) or stay on the hosted/outsourced servers.

However, assuming anyone chooses option (2), some things will have to
change with how we handle these servers.  Specifically, Canonical will
continue to sponsor the servers but they will have to become entirely
community run, i.e.

   + Community admin team liaise with hosting company for reboots, etc.

   + Community have sole responsibility for all aspects of
     administration of servers, including but not limited to day to
     day sysadmin tasks, backups, security, upgrades, recovery if
     compromised, etc.

   + Use of servers for loco team services only unless previously
agreed.
     - Both the Community Council and Canonical have oversight on this
 
-- 
James

[1] art-web, gallery, drupal, phpmyadmin, wordpress, postnuke, phpbb,
    smf, moodle, planet, aspseek, moin, taskfreak, cms made simple,
    mediawiki, ...


----------------

So, if you have any questions, come to the meeting and discuss them
there where both James and I will be present. :)

	Jono

-- 
Jono Bacon
Ubuntu Community Manager
jono(at)ubuntu(dot)com
www.ubuntu.com / www.jonobacon.org




More information about the loco-contacts mailing list