Server problems affecting some loco teams

[Matthew Nuzum]

Here's an update for everyone who has been affected by the hosting
servers being down.

The servers we use to host the loco teams are not in the Canonical
data center. Instead, they're hosted in a server farm in the US where
we rent space. For the most part, these machines are managed by the
various teams who use them.

Unfortunately, we were made aware yesterday (Monday, Aug 6th) that one
of the machines had been compromised and was being used to try and
hack other machines. We took a moment to assess the risk and evaluate
the situation, then quickly took the machine down. The machine was
known as 'ganges' and was the host to dc.ubuntu-us.org and a few other
sites, most of which are not very active.

We analyzed the other machines we rent in that data center and found a
few to be acting suspiciously. Because of the late hour, we chose to
shut down four other servers so that we can safely inspect them. These
four machines are known as 'hudson', which is used by the doc team and
a couple of the more active loco team websites (Sweden and Finland),
also 'zambezi', which hosted the bulk of the smaller loco teams,
'parana', used by the software freedom day team and 'tiber', used by
motu-revu.

As part of the recovery process, we'd like to migrate the services
running on these machines from the rented servers to machines in the
Canonical Data Centre.  This will give several benefits:

 o Immediate upgrade to a security supported release of Ubuntu
 o Greatly improved hardware
 o Full time sysadmin support from Canonical

It will however come with one restriction which the external servers
don't have and that is that full root access won't be available.
FWIw, The Italian team already migrated to a server in the Canonical
DC a while ago.

Unfortunately the recovery process will take some time.  We've asked
the hosting company to boot the servers up into Live CDs already.
Once they've done that we'll recover the data and content from the
existing machines and re-setup the services on new machines.

We'll keep you updated with progres reports and an ETA as soon as we
have one.

If you have a very recent backup of your site, we can probably get
your site up quicker. Please notify me.

Of greater concern, please consider carefully if you are a user of
'ganges' or one of the suspected compromised servers.  Any sensitive
files (e.g. gpg or ssh keys) or data (e.g. typed passwords, forwarded
SSH keys) stored on or that passed through those servers recently
should be considered compromised and revoked & new ones generated.

You may want to check your machine and any machine you logged into
from the (potentially) compromised servers.  Here is a webpage that
contains more details, a bit on the paranoid side, about what steps
you should consider taking.

<http://www.wiggy.net/debian/developer-securing/>

If you have any concerns or questions, please refer to the above link.
If that does not answer your question, please contact me.

It's too early to be sure, but at the moment we're not expecting any
data loss.

-- 
Matthew Nuzum
newz2000 on freenode