Server hacked?

Wed Jan 2 00:35:23 UTC 2008

Johan, NoOp,

Thanks for your comments.
It seems I can be quite lucky, as the damage seems to be rather
contained to a very limited set of my system. The processes are of the
user www-data. So it seems a web site has been hacked instead. (Count
myself lucky this time)

Evidence:
root at shushan:/proc/29965# ls -l
total 0
[snip]
lrwxrwxrwx 1 www-data www-data 0 2008-01-01 23:07 exe -> /usr/bin/perl
[snip]
root at shushan:/proc/29965# ls fd -l
total 10
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 0 -> socket:[543920]
l-wx------ 1 www-data www-data 64 2008-01-01 23:01 1 -> pipe:[543929]
l-wx------ 1 www-data www-data 64 2008-01-01 23:01 2 ->
/var/log/apache2/error.log
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 3 -> /tmp/.apc.TeL4il
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 4 -> /tmp/.apc.MpL1Yi
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 5 -> /tmp/.apc.2d14Oh
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 6 -> /tmp/.apc.5zT9Eg
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 7 -> /tmp/.apc.Njjgvf
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 8 -> socket:[525267]
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 9 -> socket:[784974]

It seems it limited to the www-data user. Of course the fd/3 file gives
a very good hint that it is indeed apache. The apc comes from PHP. Of
course its odd that the webpage runs perl, since all processes are
supposed to be PHP.
Up to a certain point, this restores my faite (hope) that the system can
be trusted (up to some extend).

Some oddities are these on a site running Joomla are:
===
htdocs/.wonk/motd/USER1.MOTD.old::arcor.de.eu.dal.net 372 _mugo1`dealz
:-     You can always reach this server by typing /server arcor.dal.net
6667
htdocs/.wonk/motd/USER2.MOTD.old::arcor.de.eu.dal.net 372 [X]enzo :-
You can always reach this server by typing /server arcor.dal.net 6667
htdocs/.wonk/src/p_client.c:
ap_snprintf(irccontent,sizeof(irccontent),"6667");
htdocs/.wonk/src/p_inifunc.c:   if (ern != 0) { user(usernum)->port =
6667; } else { user(usernum)->port = atoi(value); }
Binary file htdocs/.wonk/src/p_client.o matches
Binary file htdocs/.wonk/vi matches
htdocs/dimenso:my $porta="6667";
htdocs/dimenso.1:my $porta="6667";
htdocs/scanner.pl:my $porta="6667";
===

It also has wonk.tar.gz from 2007-03-18.

Anyone familiar with this?

I hope to diagnose when the incident occurred an how to protect against
it better in the future.

At least there are some lessons in this:

* Use one-user-per-website only (easier auditing).
* Deploy you firewall with strict rules.
* Do auditing & automated monitoring.
* Keep longer logs if you don't automatically monitor your systems.
* Keep ALL software up to date (something automatic for websites?)

* Its good policy to deny traffic, except if required for system/website
operation.

I'll be moving all stuff to a new box with Xen and more isolation, these
seem good lessons to get started. Still it seems to be quite a lot of
work for doing it right (or at least better) the second time. Any
suggestions from experienced people that might help me?

Thanks,

- Joris

>-----Original Message-----
>From: ubuntu-users-bounces at lists.ubuntu.com 
>[mailto:ubuntu-users-bounces at lists.ubuntu.com] On Behalf Of johanb
>Sent: Tuesday, 1 January 2008 23:26
>To: Ubuntu user technical support,not for general discussions
>Subject: Re: Server hacked?
>
>
>Joris,
>
>There seems to be some kind of rootkit running on your server. 
>It concerns syn_sent , towars numbering devices, which 
>inclines scans for open dcc-servers, to sent packets to. What 
>I would advice you, is to install a package searching for 
>installed rootkits and scan your system with ex. chkrootkit or 
>rkhunter.
>To answer your question : yes according to my humble opinion 
>it concerns some kind of rootkit or trojan. (rather rootkit 
>then trojan) greetz, Johan
>
>Joris Dobbelsteen schreef:
>> Dear,
>>
>> I'm having a Linux server running Ubuntu 6.06 LTS (under vmware) and 
>> it seems to have some quite weird behaviour. For some reason (or 
>> another) the box seems to create OUTGOING connections to an 
>IRC server 
>> from a supposed kernel address. Below is a snapshot of the 
>netstat output...
>>
>> Can anyone provide some detailed information about this?
>> (i.e. confirm my suspicion it is indeed a trojan or something)
>>
>> If so, I'm desiring to do some more diagnosis/forensics on 
>the box to 
>> get to know what may caused this strange behaviour. Can 
>anyone provide 
>> some help on this?
>>
>> The box has PostFix, PowerDNS, Apache2 and SSH exposed to 
>the Internet.
>> Unfortunally its connected to the single LAN segment I have at home.
>> Fortunally I have a strict firewall that doesn't allow IRC out (I 
>> don't use it, so I do not need to allow it).
>>
>> Thanks in advance...
>>
>> - Joris
>>
>>
>> root at shushan:~# netstat -tnp
>> Active Internet connections (w/o servers)
>> Proto Recv-Q Send-Q Local Address           Foreign Address
>> State       PID/Program name
>> tcp        0      1 192.168.10.xx:52020     216.152.66.54:6667
>> SYN_SENT   18687/[kjournald]
>> tcp        0      1 192.168.10.xx:49383     216.152.66.48:6667
>> SYN_SENT   18599/[kjournald]
>> tcp        0      1 192.168.10.xx:32936     216.152.66.46:6667
>> SYN_SENT   11304/[kjournald]
>> tcp        0      1 192.168.10.xx:56901     216.152.67.49:6667
>> SYN_SENT   29965/[kjournald]
>> tcp        0      1 192.168.10.xx:54354     216.152.67.30:6667
>> SYN_SENT   24235/[kjournald]
>> tcp        0      1 192.168.10.xx:60278     216.152.66.47:6667
>> SYN_SENT   15412/[kjournald]
>> [trusted entries removed]
>>
>>
>>   
>
>
>--
>ubuntu-users mailing list
>ubuntu-users at lists.ubuntu.com
>Modify settings or unsubscribe at: 
>https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
>