Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Anton Chuvakin FTP Server Intrusion Investigation


Published on

Now famous FTP server intrusion investigation, including log analysis, disk forensics as well as lessons learned; all still fun and useful, but circa 2002

Published in: Technology

Anton Chuvakin FTP Server Intrusion Investigation

  1. 1. FTP Server Attack Case Study <ul><li>Company FTP Server Erased by Hackers: </li></ul><ul><li>Detection, Response, Prevention </li></ul><ul><li>Anton Chuvakin, Ph.D </li></ul><ul><li>Senior Security Analyst </li></ul><ul><li>InfraGard NJ </li></ul><ul><li>May 29, 2002 </li></ul>
  2. 2. FTP Server Attack: Summary <ul><li>What happened? </li></ul><ul><ul><li>Crime scene: network environment and incident discovery </li></ul></ul><ul><li>What was found? </li></ul><ul><ul><li>Case: network traffic analysis and system forensics </li></ul></ul><ul><li>What should have been done? </li></ul><ul><ul><li>Verdict: security prevention measures: what was done and what wasn’t </li></ul></ul>
  3. 3. FTP Attack: Detailed Summary <ul><li>Victim: </li></ul><ul><ul><li>Medium computer hardware manufacturer </li></ul></ul><ul><li>FTP server found with erased disks </li></ul><ul><li>IDS, firewall log analysis </li></ul><ul><li>Hard-drive forensics </li></ul><ul><ul><li>What worked and what didn’t </li></ul></ul><ul><li>Security lessons from the event </li></ul><ul><ul><li>Lessons on prevention, detection and response </li></ul></ul>
  4. 4. FTP Server Attack Case Study <ul><li>Proverbial Monday Morning – FTP Server is not responding </li></ul><ul><li>All remote communication fails </li></ul><ul><li>Upon rebooting doesn’t come back </li></ul><ul><li>Discovered that there is no OS anymore (was Linux RedHat 7.2) </li></ul><ul><li>Incident Response plan is activated! </li></ul>
  5. 5. Network Infrastructure
  6. 6. Network Defences <ul><li>External Firewall </li></ul><ul><ul><li>No DMZ -> External traffic </li></ul></ul><ul><ul><li>No DMZ -> DMZ traffic </li></ul></ul><ul><ul><li>Internet -> DMZ traffic: </li></ul></ul><ul><ul><ul><li>Only port 80 (HTTP) to web server allowed </li></ul></ul></ul><ul><ul><ul><li>Only port 21 (FTP) to FTP server allowed (stateful) </li></ul></ul></ul><ul><ul><ul><li>etc </li></ul></ul></ul><ul><ul><li>IDS 1 and 2 see all DMZ traffic and report through a separate LAN (not shown) to a central database </li></ul></ul>
  7. 7. Network Defences <ul><li>Internal Firewall </li></ul><ul><ul><li>No DMZ -> internal LAN traffic </li></ul></ul><ul><ul><li>No external firewall -> internal LAN traffic </li></ul></ul><ul><ul><li>Proxy installed to handle internal to Internet </li></ul></ul><ul><ul><li>Internal -> DMZ traffic: </li></ul></ul><ul><ul><ul><li>Only Secure Shell (SSH) from select hosts to all DMZ machines for management and file transfers </li></ul></ul></ul><ul><li>Additional defences: </li></ul><ul><ul><li>Host-firewalls on all DMZ hosts </li></ul></ul><ul><ul><li>Integrity checking software </li></ul></ul>
  8. 8. Investigation Plan <ul><li>Decision: full analysis before redeployment </li></ul><ul><li>Main investigation focus: </li></ul><ul><ul><li>What happened? </li></ul></ul><ul><ul><li>Is the network still exposed? </li></ul></ul><ul><ul><li>How to prevent it in the future? </li></ul></ul><ul><li>Secondary focus: </li></ul><ul><ul><li>If it was a hacker attack, track the actions of the attacker to learn more about them and/or recover their tools </li></ul></ul>
  9. 9. Network Log Forensics <ul><li>Source of information : firewall logs, IDS logs, system logs aggregated and presented by a log management software </li></ul><ul><li>What can we learn from them? </li></ul><ul><ul><li>All inbound connections to DMZ servers (source: firewall) </li></ul></ul><ul><ul><li>All outbound attempts from DMZ (firewall) </li></ul></ul><ul><ul><li>Attacks launched (IDS 1, 2) </li></ul></ul><ul><ul><li>Intra-DMZ connection attempts (host logs) </li></ul></ul><ul><ul><li>Protocol and other details (IDS 1) </li></ul></ul>
  10. 10. Network Log Forensics: Timeline <ul><li>Classic scenario (known from the honeynet): </li></ul><ul><ul><li>“ Horizontal” FTP scan (00:10:20-00:10:37) </li></ul></ul><ul><ul><li>FTP server found, more probing (00:12-00:13) </li></ul></ul><ul><ul><li>First connection established (00:13:30) </li></ul></ul><ul><ul><li>Attack launched (00:16:02) </li></ul></ul><ul><ul><li>Outbound connection from FTP attempted (00:17:14) SERVER IS 'OWNED'! </li></ul></ul><ul><ul><li>More probing and trying to get out (00:17-00:30) </li></ul></ul><ul><ul><li>File deployed on server (00:31:59) </li></ul></ul><ul><ul><li>Server erased (00:38) </li></ul></ul>
  11. 11. Network Log Forensics: Results <ul><li>Attack happened from 00:10 to 00:38 </li></ul><ul><li>Attacked penetrated the FTP server using WU-FTPD globbing vulnerability dated Nov 2001 (from IDS signatures) </li></ul><ul><li>Attacker managed to upload a file on the server (most likely rootkit) </li></ul><ul><li>Intruder failed to gain further ground (from IDS, firewall and host logs) </li></ul>
  12. 12. Disk Forensics <ul><li>Focus: </li></ul><ul><ul><li>Get system logs to confirm findings of network forensics (no remote logging) </li></ul></ul><ul><ul><li>Recover attacker's toolkit </li></ul></ul><ul><li>UNIX disk forensics tools </li></ul><ul><ul><li>Strings, grep, awk, file, TCT, TASK, foremost </li></ul></ul>
  13. 13. Disk Forensics: Procedure I <ul><li>Procedure </li></ul><ul><ul><li>Get the disk </li></ul></ul><ul><ul><li>Install in analysis machine </li></ul></ul><ul><ul><li>Mount read-only ( mount -ro ) </li></ul></ul><ul><ul><li>Copy the whole disk into image files ( dd ) </li></ul></ul><ul><li>Looking for log files: strings, grep </li></ul><ul><li>strings /home/hacked-ftp-hdc1 | grep 'Mar 17' </li></ul><ul><li>Found: FTPD logs, messages, inetd logs </li></ul><ul><li>Show attacker’s connections </li></ul>
  14. 14. Disk Forensics: Procedure II <ul><li>Further into the disk... </li></ul><ul><ul><li>Use strings and less to look through the disk </li></ul></ul><ul><ul><li>Search for ATTACK_IP, incoming, rootkit name, etc </li></ul></ul><ul><li>Discovered: </li></ul><ul><ul><li>FTP client logs (with attempts to get out) </li></ul></ul><ul><ul><li>Rootkit installation script! </li></ul></ul><ul><ul><li>List of rootkit files </li></ul></ul>
  15. 15. Disk Forensics: Rootkit <ul><li>Unsophisticated LKM kit (all public stuff?) </li></ul><ul><ul><li>Blocks shell history and syslog audit trail </li></ul></ul><ul><ul><li>Unpacks and installs components </li></ul></ul><ul><ul><li>Deploys hidden backdoor SSH daemon </li></ul></ul><ul><ul><li>Builds and deploys hiding LKM </li></ul></ul><ul><ul><li>Installs some DoS tools and sniffer </li></ul></ul><ul><ul><li>Makes changes to system configuration files </li></ul></ul><ul><li>Comments in Romanian </li></ul><ul><li>Enough to hide from regular system admin </li></ul>
  16. 16. Disk Forensics: Tools I <ul><li>Coroner Toolkit (TCT) </li></ul><ul><ul><li>Erased file recovery (several methods) </li></ul></ul><ul><ul><li>Intrusion timeline creation </li></ul></ul><ul><ul><li>Live and post-mortem system data collection </li></ul></ul><ul><ul><li>Filesystem changes analysis </li></ul></ul><ul><li>TASK </li></ul><ul><ul><li>Improved version of TCT </li></ul></ul><ul><ul><li>Better Windows support (NTFS, etc) </li></ul></ul>
  17. 17. Disk Forensics: Tools II <ul><li>Autopsy </li></ul>
  18. 18. Disk Forensics: Tools III <ul><li>UNIX file removal options </li></ul><ul><ul><li>Files are gone, but the directory entries and content remain </li></ul></ul><ul><ul><li>Files and dir entries and gone, content remains </li></ul></ul><ul><ul><li>Files are gone and some of the content is gone </li></ul></ul><ul><ul><li>Files are gone and content overwritten </li></ul></ul><ul><li>TCT and TASK operate on 1) , TCT can also operate on 2) with certain probability </li></ul><ul><li>Our case: 2) </li></ul>
  19. 19. Disk Forensics: Unerase <ul><li>TCT lazarus tool: the hard way to unerase </li></ul><ul><ul><li>Get all the unallocated space from disk </li></ul></ul><ul><ul><li>Look at chunks of data to determine their type (text, binary) </li></ul></ul><ul><ul><li>Analyses the chunks to further determine type (C code, email, log files, executables, compressed) </li></ul></ul><ul><ul><li>Concatenates consecutive chunks of the same type into one file </li></ul></ul><ul><li>Very slow (hours...) </li></ul><ul><li>Recovery is a game of chance </li></ul><ul><li>Results need further processing </li></ul>
  20. 20. Disk Forensics: Hunt for the Kit I <ul><li>List of files:, adore.tgz, sshutils.tgz, utils.tgz, mount.tar.gz </li></ul><ul><ul><li>Look for all the lazarus results of the compressed type and try to uncompress them </li></ul></ul><ul><ul><li>Look for these filenames within other files </li></ul></ul><ul><ul><li>Look for the files of similar size (from logs) </li></ul></ul><ul><ul><li>Look for files adjacent to found rootkit script </li></ul></ul><ul><ul><li>Look for other archived files (e.g. tar ) </li></ul></ul><ul><ul><li>Look for C source code </li></ul></ul><ul><li>All fails! </li></ul>
  21. 21. Disk Forensics: Hunt for the Kit II <ul><li>Enter foremost (new tool from USAF OSI) </li></ul><ul><ul><li>Signature-based file searching </li></ul></ul><ul><ul><li>Tool uses custom binary strings typically found in files and other parameters </li></ul></ul><ul><ul><li>Had signatures for images and docs </li></ul></ul><ul><ul><li>Support for gzip files added by me </li></ul></ul>
  22. 22. Disk Forensics: Hunt for the Kit III <ul><li>Foremost found two files: </li></ul><ul><ul><li>adore.tgz </li></ul></ul><ul><ul><li>utilz.tgz </li></ul></ul><ul><li>Adore is standard LKM (backdoor, file, process, connection hiding, etc) </li></ul><ul><li>Utils : imp, lil, slc, sense, fsch2 </li></ul><ul><li>After some hostile binary analysis: </li></ul><ul><ul><li>Easy: imp, slc - DoS tools; lil, sense – sniffer; fsch2 – remains a mystery </li></ul></ul>
  23. 23. Lessons: Detection <ul><li>If disk content is missing – a problem is suspected...  </li></ul><ul><li>But! If attacker had decided to keep access? </li></ul><ul><li>IDS with current (!) signature base helps </li></ul><ul><li>But! IDS is only as good as the person looking at the alerts (thus, nobody looking is as good as no IDS ) </li></ul><ul><li>Otherwise, IDS turns into an expensive incident response helper </li></ul>
  24. 24. Lessons: Prevention Done Right <ul><li>Great DMZ architecture and rule sets </li></ul><ul><ul><li>Principle of least privilege utilized </li></ul></ul><ul><ul><li>Block outgoing from DMZ is a must! </li></ul></ul><ul><ul><li>Block DMZ to DMZ needed too </li></ul></ul><ul><li>Up-to-date IDS signatures </li></ul><ul><li>Presence of log management software </li></ul><ul><ul><li>Incident response is very easy since all information is in one place </li></ul></ul>
  25. 25. Lessons: Prevention to Improve <ul><li> still got hacked... </li></ul><ul><li>Patching timely! </li></ul><ul><li>Secure UNIX/Linux? Intrusion prevention? </li></ul><ul><li>Remote logging </li></ul><ul><li>No anonymous uploads for FTP </li></ul><ul><li>Vulnerability assessment </li></ul><ul><li>Real-time log monitoring </li></ul><ul><li>Who controls security? Admins or dedicated stuff? </li></ul>
  26. 26. Lessons: Patching? <ul><li>Security patching: </li></ul><ul><ul><li>Automatic patching? A BIG question. </li></ul></ul><ul><ul><ul><li>Fast automated attacks (Honeynet) – manual patching might not be timely enough </li></ul></ul></ul><ul><ul><li>Patch testing, system backup, mass (and timely!) deployments </li></ul></ul><ul><ul><li>Who drives the patching process? System admin or security department? </li></ul></ul>
  27. 27. Lessons: Response <ul><li>Network log forensics is enough if you have enough audit trail (from firewalls and IDS) and a tool to analyse it </li></ul><ul><ul><li>Centralized audit trail storage very handy! </li></ul></ul><ul><li>Disk forensics is often not reliable – it might or might not produce what you want </li></ul><ul><ul><li>Finding something specific vs finding any evidence </li></ul></ul><ul><li>Detailed analysis is for controlled environment (i.e. honeynets) </li></ul>
  28. 28. Lessons: Attackers I <ul><li>Hackers are after YOU (and its not FUD!): </li></ul><ul><ul><li>For CPU, disk, memory, network connection (rootkit stats) </li></ul></ul><ul><ul><li>Crime of opportunity is MUCH more likely that a targeted attack. Is it thus a bigger threat? </li></ul></ul><ul><ul><ul><li>Massive scanning -> you slip once and your server is owned. </li></ul></ul></ul><ul><ul><li>Early warning about such attacks does not exist (fully automated scan-attack-exploit in one package) </li></ul></ul>
  29. 29. Lessons: Intruder Operations <ul><li>Attacker's purposes (from deployed tools): </li></ul><ul><ul><li>DoS platform </li></ul></ul><ul><ul><li>Scanning for vulnerabilities </li></ul></ul><ul><ul><li>Sniffing </li></ul></ul><ul><ul><li>IRC </li></ul></ul><ul><ul><li>Hoarding hacked boxes for unknown future goals </li></ul></ul><ul><li>Note: only applies to low-level attackers </li></ul><ul><li>Use many hops (origin is unclear) </li></ul>
  30. 30. Conclusions <ul><li>Good network design </li></ul><ul><ul><li>A one time cost and adds a lot to security </li></ul></ul><ul><li>Keeping systems up to date </li></ul><ul><ul><li>Painful but necessary </li></ul></ul><ul><li>IR and forensics </li></ul><ul><ul><li>Network and disk forensics together give complete picture </li></ul></ul><ul><li>Constant vigilance is expensive but necessary </li></ul>
  31. 31. Thanks for Viewing the Presentation <ul><li>Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA </li></ul><ul><li> </li></ul><ul><li>Author of “Security Warrior” (O’Reilly) – </li></ul><ul><li>Book on logs is coming soon! </li></ul><ul><li>See for my papers, books, reviews and other security resources related to logs </li></ul>