Anton Chuvakin FTP Server Intrusion Investigation


Published on

Now famous FTP server intrusion investigation, including log analysis, disk forensics as well as lessons learned; all still fun and useful, but circa 2002

Published in: Technology
1 Comment
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Anton Chuvakin FTP Server Intrusion Investigation

  1. 1. FTP Server Attack Case Study <ul><li>Company FTP Server Erased by Hackers: </li></ul><ul><li>Detection, Response, Prevention </li></ul><ul><li>Anton Chuvakin, Ph.D </li></ul><ul><li>Senior Security Analyst </li></ul><ul><li>InfraGard NJ </li></ul><ul><li>May 29, 2002 </li></ul>
  2. 2. FTP Server Attack: Summary <ul><li>What happened? </li></ul><ul><ul><li>Crime scene: network environment and incident discovery </li></ul></ul><ul><li>What was found? </li></ul><ul><ul><li>Case: network traffic analysis and system forensics </li></ul></ul><ul><li>What should have been done? </li></ul><ul><ul><li>Verdict: security prevention measures: what was done and what wasn’t </li></ul></ul>
  3. 3. FTP Attack: Detailed Summary <ul><li>Victim: </li></ul><ul><ul><li>Medium computer hardware manufacturer </li></ul></ul><ul><li>FTP server found with erased disks </li></ul><ul><li>IDS, firewall log analysis </li></ul><ul><li>Hard-drive forensics </li></ul><ul><ul><li>What worked and what didn’t </li></ul></ul><ul><li>Security lessons from the event </li></ul><ul><ul><li>Lessons on prevention, detection and response </li></ul></ul>
  4. 4. FTP Server Attack Case Study <ul><li>Proverbial Monday Morning – FTP Server is not responding </li></ul><ul><li>All remote communication fails </li></ul><ul><li>Upon rebooting doesn’t come back </li></ul><ul><li>Discovered that there is no OS anymore (was Linux RedHat 7.2) </li></ul><ul><li>Incident Response plan is activated! </li></ul>
  5. 5. Network Infrastructure
  6. 6. Network Defences <ul><li>External Firewall </li></ul><ul><ul><li>No DMZ -> External traffic </li></ul></ul><ul><ul><li>No DMZ -> DMZ traffic </li></ul></ul><ul><ul><li>Internet -> DMZ traffic: </li></ul></ul><ul><ul><ul><li>Only port 80 (HTTP) to web server allowed </li></ul></ul></ul><ul><ul><ul><li>Only port 21 (FTP) to FTP server allowed (stateful) </li></ul></ul></ul><ul><ul><ul><li>etc </li></ul></ul></ul><ul><ul><li>IDS 1 and 2 see all DMZ traffic and report through a separate LAN (not shown) to a central database </li></ul></ul>
  7. 7. Network Defences <ul><li>Internal Firewall </li></ul><ul><ul><li>No DMZ -> internal LAN traffic </li></ul></ul><ul><ul><li>No external firewall -> internal LAN traffic </li></ul></ul><ul><ul><li>Proxy installed to handle internal to Internet </li></ul></ul><ul><ul><li>Internal -> DMZ traffic: </li></ul></ul><ul><ul><ul><li>Only Secure Shell (SSH) from select hosts to all DMZ machines for management and file transfers </li></ul></ul></ul><ul><li>Additional defences: </li></ul><ul><ul><li>Host-firewalls on all DMZ hosts </li></ul></ul><ul><ul><li>Integrity checking software </li></ul></ul>
  8. 8. Investigation Plan <ul><li>Decision: full analysis before redeployment </li></ul><ul><li>Main investigation focus: </li></ul><ul><ul><li>What happened? </li></ul></ul><ul><ul><li>Is the network still exposed? </li></ul></ul><ul><ul><li>How to prevent it in the future? </li></ul></ul><ul><li>Secondary focus: </li></ul><ul><ul><li>If it was a hacker attack, track the actions of the attacker to learn more about them and/or recover their tools </li></ul></ul>
  9. 9. Network Log Forensics <ul><li>Source of information : firewall logs, IDS logs, system logs aggregated and presented by a log management software </li></ul><ul><li>What can we learn from them? </li></ul><ul><ul><li>All inbound connections to DMZ servers (source: firewall) </li></ul></ul><ul><ul><li>All outbound attempts from DMZ (firewall) </li></ul></ul><ul><ul><li>Attacks launched (IDS 1, 2) </li></ul></ul><ul><ul><li>Intra-DMZ connection attempts (host logs) </li></ul></ul><ul><ul><li>Protocol and other details (IDS 1) </li></ul></ul>
  10. 10. Network Log Forensics: Timeline <ul><li>Classic scenario (known from the honeynet): </li></ul><ul><ul><li>“ Horizontal” FTP scan (00:10:20-00:10:37) </li></ul></ul><ul><ul><li>FTP server found, more probing (00:12-00:13) </li></ul></ul><ul><ul><li>First connection established (00:13:30) </li></ul></ul><ul><ul><li>Attack launched (00:16:02) </li></ul></ul><ul><ul><li>Outbound connection from FTP attempted (00:17:14) SERVER IS 'OWNED'! </li></ul></ul><ul><ul><li>More probing and trying to get out (00:17-00:30) </li></ul></ul><ul><ul><li>File deployed on server (00:31:59) </li></ul></ul><ul><ul><li>Server erased (00:38) </li></ul></ul>
  11. 11. Network Log Forensics: Results <ul><li>Attack happened from 00:10 to 00:38 </li></ul><ul><li>Attacked penetrated the FTP server using WU-FTPD globbing vulnerability dated Nov 2001 (from IDS signatures) </li></ul><ul><li>Attacker managed to upload a file on the server (most likely rootkit) </li></ul><ul><li>Intruder failed to gain further ground (from IDS, firewall and host logs) </li></ul>
  12. 12. Disk Forensics <ul><li>Focus: </li></ul><ul><ul><li>Get system logs to confirm findings of network forensics (no remote logging) </li></ul></ul><ul><ul><li>Recover attacker's toolkit </li></ul></ul><ul><li>UNIX disk forensics tools </li></ul><ul><ul><li>Strings, grep, awk, file, TCT, TASK, foremost </li></ul></ul>
  13. 13. Disk Forensics: Procedure I <ul><li>Procedure </li></ul><ul><ul><li>Get the disk </li></ul></ul><ul><ul><li>Install in analysis machine </li></ul></ul><ul><ul><li>Mount read-only ( mount -ro ) </li></ul></ul><ul><ul><li>Copy the whole disk into image files ( dd ) </li></ul></ul><ul><li>Looking for log files: strings, grep </li></ul><ul><li>strings /home/hacked-ftp-hdc1 | grep 'Mar 17' </li></ul><ul><li>Found: FTPD logs, messages, inetd logs </li></ul><ul><li>Show attacker’s connections </li></ul>
  14. 14. Disk Forensics: Procedure II <ul><li>Further into the disk... </li></ul><ul><ul><li>Use strings and less to look through the disk </li></ul></ul><ul><ul><li>Search for ATTACK_IP, incoming, rootkit name, etc </li></ul></ul><ul><li>Discovered: </li></ul><ul><ul><li>FTP client logs (with attempts to get out) </li></ul></ul><ul><ul><li>Rootkit installation script! </li></ul></ul><ul><ul><li>List of rootkit files </li></ul></ul>
  15. 15. Disk Forensics: Rootkit <ul><li>Unsophisticated LKM kit (all public stuff?) </li></ul><ul><ul><li>Blocks shell history and syslog audit trail </li></ul></ul><ul><ul><li>Unpacks and installs components </li></ul></ul><ul><ul><li>Deploys hidden backdoor SSH daemon </li></ul></ul><ul><ul><li>Builds and deploys hiding LKM </li></ul></ul><ul><ul><li>Installs some DoS tools and sniffer </li></ul></ul><ul><ul><li>Makes changes to system configuration files </li></ul></ul><ul><li>Comments in Romanian </li></ul><ul><li>Enough to hide from regular system admin </li></ul>
  16. 16. Disk Forensics: Tools I <ul><li>Coroner Toolkit (TCT) </li></ul><ul><ul><li>Erased file recovery (several methods) </li></ul></ul><ul><ul><li>Intrusion timeline creation </li></ul></ul><ul><ul><li>Live and post-mortem system data collection </li></ul></ul><ul><ul><li>Filesystem changes analysis </li></ul></ul><ul><li>TASK </li></ul><ul><ul><li>Improved version of TCT </li></ul></ul><ul><ul><li>Better Windows support (NTFS, etc) </li></ul></ul>
  17. 17. Disk Forensics: Tools II <ul><li>Autopsy </li></ul>
  18. 18. Disk Forensics: Tools III <ul><li>UNIX file removal options </li></ul><ul><ul><li>Files are gone, but the directory entries and content remain </li></ul></ul><ul><ul><li>Files and dir entries and gone, content remains </li></ul></ul><ul><ul><li>Files are gone and some of the content is gone </li></ul></ul><ul><ul><li>Files are gone and content overwritten </li></ul></ul><ul><li>TCT and TASK operate on 1) , TCT can also operate on 2) with certain probability </li></ul><ul><li>Our case: 2) </li></ul>
  19. 19. Disk Forensics: Unerase <ul><li>TCT lazarus tool: the hard way to unerase </li></ul><ul><ul><li>Get all the unallocated space from disk </li></ul></ul><ul><ul><li>Look at chunks of data to determine their type (text, binary) </li></ul></ul><ul><ul><li>Analyses the chunks to further determine type (C code, email, log files, executables, compressed) </li></ul></ul><ul><ul><li>Concatenates consecutive chunks of the same type into one file </li></ul></ul><ul><li>Very slow (hours...) </li></ul><ul><li>Recovery is a game of chance </li></ul><ul><li>Results need further processing </li></ul>
  20. 20. Disk Forensics: Hunt for the Kit I <ul><li>List of files:, adore.tgz, sshutils.tgz, utils.tgz, mount.tar.gz </li></ul><ul><ul><li>Look for all the lazarus results of the compressed type and try to uncompress them </li></ul></ul><ul><ul><li>Look for these filenames within other files </li></ul></ul><ul><ul><li>Look for the files of similar size (from logs) </li></ul></ul><ul><ul><li>Look for files adjacent to found rootkit script </li></ul></ul><ul><ul><li>Look for other archived files (e.g. tar ) </li></ul></ul><ul><ul><li>Look for C source code </li></ul></ul><ul><li>All fails! </li></ul>
  21. 21. Disk Forensics: Hunt for the Kit II <ul><li>Enter foremost (new tool from USAF OSI) </li></ul><ul><ul><li>Signature-based file searching </li></ul></ul><ul><ul><li>Tool uses custom binary strings typically found in files and other parameters </li></ul></ul><ul><ul><li>Had signatures for images and docs </li></ul></ul><ul><ul><li>Support for gzip files added by me </li></ul></ul>
  22. 22. Disk Forensics: Hunt for the Kit III <ul><li>Foremost found two files: </li></ul><ul><ul><li>adore.tgz </li></ul></ul><ul><ul><li>utilz.tgz </li></ul></ul><ul><li>Adore is standard LKM (backdoor, file, process, connection hiding, etc) </li></ul><ul><li>Utils : imp, lil, slc, sense, fsch2 </li></ul><ul><li>After some hostile binary analysis: </li></ul><ul><ul><li>Easy: imp, slc - DoS tools; lil, sense – sniffer; fsch2 – remains a mystery </li></ul></ul>
  23. 23. Lessons: Detection <ul><li>If disk content is missing – a problem is suspected...  </li></ul><ul><li>But! If attacker had decided to keep access? </li></ul><ul><li>IDS with current (!) signature base helps </li></ul><ul><li>But! IDS is only as good as the person looking at the alerts (thus, nobody looking is as good as no IDS ) </li></ul><ul><li>Otherwise, IDS turns into an expensive incident response helper </li></ul>
  24. 24. Lessons: Prevention Done Right <ul><li>Great DMZ architecture and rule sets </li></ul><ul><ul><li>Principle of least privilege utilized </li></ul></ul><ul><ul><li>Block outgoing from DMZ is a must! </li></ul></ul><ul><ul><li>Block DMZ to DMZ needed too </li></ul></ul><ul><li>Up-to-date IDS signatures </li></ul><ul><li>Presence of log management software </li></ul><ul><ul><li>Incident response is very easy since all information is in one place </li></ul></ul>
  25. 25. Lessons: Prevention to Improve <ul><li> still got hacked... </li></ul><ul><li>Patching timely! </li></ul><ul><li>Secure UNIX/Linux? Intrusion prevention? </li></ul><ul><li>Remote logging </li></ul><ul><li>No anonymous uploads for FTP </li></ul><ul><li>Vulnerability assessment </li></ul><ul><li>Real-time log monitoring </li></ul><ul><li>Who controls security? Admins or dedicated stuff? </li></ul>
  26. 26. Lessons: Patching? <ul><li>Security patching: </li></ul><ul><ul><li>Automatic patching? A BIG question. </li></ul></ul><ul><ul><ul><li>Fast automated attacks (Honeynet) – manual patching might not be timely enough </li></ul></ul></ul><ul><ul><li>Patch testing, system backup, mass (and timely!) deployments </li></ul></ul><ul><ul><li>Who drives the patching process? System admin or security department? </li></ul></ul>
  27. 27. Lessons: Response <ul><li>Network log forensics is enough if you have enough audit trail (from firewalls and IDS) and a tool to analyse it </li></ul><ul><ul><li>Centralized audit trail storage very handy! </li></ul></ul><ul><li>Disk forensics is often not reliable – it might or might not produce what you want </li></ul><ul><ul><li>Finding something specific vs finding any evidence </li></ul></ul><ul><li>Detailed analysis is for controlled environment (i.e. honeynets) </li></ul>
  28. 28. Lessons: Attackers I <ul><li>Hackers are after YOU (and its not FUD!): </li></ul><ul><ul><li>For CPU, disk, memory, network connection (rootkit stats) </li></ul></ul><ul><ul><li>Crime of opportunity is MUCH more likely that a targeted attack. Is it thus a bigger threat? </li></ul></ul><ul><ul><ul><li>Massive scanning -> you slip once and your server is owned. </li></ul></ul></ul><ul><ul><li>Early warning about such attacks does not exist (fully automated scan-attack-exploit in one package) </li></ul></ul>
  29. 29. Lessons: Intruder Operations <ul><li>Attacker's purposes (from deployed tools): </li></ul><ul><ul><li>DoS platform </li></ul></ul><ul><ul><li>Scanning for vulnerabilities </li></ul></ul><ul><ul><li>Sniffing </li></ul></ul><ul><ul><li>IRC </li></ul></ul><ul><ul><li>Hoarding hacked boxes for unknown future goals </li></ul></ul><ul><li>Note: only applies to low-level attackers </li></ul><ul><li>Use many hops (origin is unclear) </li></ul>
  30. 30. Conclusions <ul><li>Good network design </li></ul><ul><ul><li>A one time cost and adds a lot to security </li></ul></ul><ul><li>Keeping systems up to date </li></ul><ul><ul><li>Painful but necessary </li></ul></ul><ul><li>IR and forensics </li></ul><ul><ul><li>Network and disk forensics together give complete picture </li></ul></ul><ul><li>Constant vigilance is expensive but necessary </li></ul>
  31. 31. Thanks for Viewing the Presentation <ul><li>Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA </li></ul><ul><li> </li></ul><ul><li>Author of “Security Warrior” (O’Reilly) – </li></ul><ul><li>Book on logs is coming soon! </li></ul><ul><li>See for my papers, books, reviews and other security resources related to logs </li></ul>