Anton Chuvakin FTP Server Intrusion Investigation

Slide 1: FTP Server Attack Case Study Company FTP Server Erased by Hackers: Detection, Response, Prevention Anton Chuvakin, Ph.D Senior Security Analyst InfraGard NJ

Slide 2: FTP Server Attack: Summary • What happened? – Crime scene: network environment and incident discovery • What was found? – Case: network traffic analysis and system forensics • What should have been done? – Verdict: security prevention measures: what was done and what wasn’t

Slide 3: FTP Attack: Detailed Summary • Victim: Example.com – Medium computer hardware manufacturer • FTP server found with erased disks • IDS, firewall log analysis • Hard-drive forensics – What worked and what didn’t • Security lessons from the event – Lessons on prevention, detection and response

Slide 4: FTP Server Attack Case Study • Proverbial Monday Morning – FTP Server is not responding • All remote communication fails • Upon rebooting doesn’t come back • Discovered that there is no OS anymore (was Linux RedHat 7.2) • Incident Response plan is activated!

Slide 5: Network Infrastructure

Slide 6: Network Defences External Firewall • No DMZ -> External traffic • No DMZ -> DMZ traffic • Internet -> DMZ traffic: • Only port 80 (HTTP) to web server allowed • Only port 21 (FTP) to FTP server allowed • (stateful) etc • IDS 1 and 2 see all DMZ traffic and report • through a separate LAN (not shown) to a central database

Slide 7: Network Defences Internal Firewall • No DMZ -> internal LAN traffic • No external firewall -> internal LAN traffic • Proxy installed to handle internal to Internet • Internal -> DMZ traffic: • Only Secure Shell (SSH) from select hosts to all • DMZ machines for management and file transfers Additional defences: • Host-firewalls on all DMZ hosts • Integrity checking software •

Slide 8: Investigation Plan • Decision: full analysis before redeployment • Main investigation focus: – What happened? – Is the network still exposed? – How to prevent it in the future? • Secondary focus: – If it was a hacker attack, track the actions of the attacker to learn more about them and/or recover their tools

Slide 9: Network Log Forensics • Source of information: firewall logs, IDS logs, system logs aggregated and presented by a log management software • What can we learn from them? – All inbound connections to DMZ servers (source: firewall) – All outbound attempts from DMZ (firewall) – Attacks launched (IDS 1, 2) – Intra-DMZ connection attempts (host logs) – Protocol and other details (IDS 1)

Slide 10: Network Log Forensics: Timeline • Classic scenario (known from the honeynet): – “Horizontal” FTP scan (00:10:20-00:10:37) – FTP server found, more probing (00:12- 00:13) – First connection established (00:13:30) – Attack launched (00:16:02) – Outbound connection from FTP attempted (00:17:14) SERVER IS 'OWNED'! – More probing and trying to get out (00:17- 00:30) – File deployed on server (00:31:59)

Slide 11: Network Log Forensics: Results Attack happened from 00:10 to 00:38  Attacked penetrated the FTP server using  http://www.cert.org/advisories/CA-2001-33.html WU- FTPD globbing vulnerability dated Nov 2001 (from IDS signatures) Attacker managed to upload a file on the  server (most likely rootkit) Intruder failed to gain further ground (from  IDS, firewall and host logs)

Slide 12: Disk Forensics • Focus: – Get system logs to confirm findings of network forensics (no remote logging) – Recover attacker's toolkit • UNIX disk forensics tools – Strings, grep, awk, file, TCT, TASK, foremost

Slide 13: Disk Forensics: Procedure I • Procedure –Get the disk –Install in analysis machine –Mount read-only (mount -ro) –Copy the whole disk into image files (dd) • Looking for log files: strings, grep strings /home/hacked-ftp-hdc1 | grep 'Mar 17' • • Found: FTPD logs, messages, inetd logs • Show attacker’s connections

Slide 14: Disk Forensics: Procedure II • Further into the disk... – Use strings and less to look through the disk – Search for ATTACK_IP, incoming, rootkit name, etc • Discovered: – FTP client logs (with attempts to get out) – Rootkit installation script! – List of rootkit files

Slide 15: Disk Forensics: Rootkit • Unsophisticated LKM kit (all public stuff?) – Blocks shell history and syslog audit trail – Unpacks and installs components – Deploys hidden backdoor SSH daemon – Builds and deploys hiding LKM – Installs some DoS tools and sniffer – Makes changes to system configuration files • Comments in Romanian • Enough to hide from regular system admin

Slide 16: Disk Forensics: Tools I • Coroner Toolkit (TCT) http://www.porcupine.org/forensics/ – Erased file recovery (several methods) – Intrusion timeline creation – Live and post-mortem system data collection – Filesystem changes analysis • TASK http://www.atstake.com/research/tools/task/ – Improved version of TCT – Better Windows support (NTFS, etc)

Slide 17: Disk Forensics: Tools II • Autopsy http://www.atstake.com/research/tools/autopsy/

Slide 18: Disk Forensics: Tools III • UNIX file removal options 1)Files are gone, but the directory entries and content remain 2)Files and dir entries and gone, content remains 3)Files are gone and some of the content is gone 4)Files are gone and content overwritten • TCT and TASK operate on 1), TCT can also operate on 2) with certain probability • Our case: 2)

Slide 19: Disk Forensics: Unerase TCT lazarus tool: the hard way to unerase • Get all the unallocated space from disk – Look at chunks of data to determine their type – (text, binary) Analyses the chunks to further determine type (C – code, email, log files, executables, compressed) Concatenates consecutive chunks of the same – type into one file Very slow (hours...) • Recovery is a game of chance • Results need further processing •

Slide 20: Disk Forensics: Hunt for the Kit I • List of files: a.sh, adore.tgz, sshutils.tgz, utils.tgz, mount.tar.gz – Look for all the lazarus results of the compressed type and try to uncompress them – Look for these filenames within other files – Look for the files of similar size (from logs) – Look for files adjacent to found rootkit script – Look for other archived files (e.g. tar) – Look for C source code • All fails!

Slide 21: Disk Forensics: Hunt for the Kit II • Enter foremost (new tool from USAF OSI) – Signature-based file searching – Tool uses custom binary strings typically found in files and other parameters – Had signatures for images and docs – Support for gzip files added by me

Slide 22: Disk Forensics: Hunt for the Kit III • Foremost found two files: adore.tgz – utilz.tgz – Adore is standardLKM (backdoor, file, • process, connection hiding, etc) Utils: imp, lil, slc, sense, fsch2 • • After some hostile binary analysis: –Easy: imp, slc - DoS tools; lil, sense – sniffer; fsch2 – remains a mystery

Slide 23: Lessons: Detection If disk content is missing – a problem is • suspected...  But! If attacker had decided to keep • access? IDS with current (!) signature base helps • But! IDS is only as good as the person • looking at the alerts (thus, nobody looking is as good as no IDS) Otherwise, IDS turns into an expensive • incident response helper

Slide 24: Lessons: Prevention Done Right • Great DMZ architecture and rule sets – Principle of least privilege utilized – Block outgoing from DMZ is a must! – Block DMZ to DMZ needed too • Up-to-date IDS signatures • Presence of log management software – Incident response is very easy since all information is in one place

Slide 25: Lessons: Prevention to Improve Example.com still got hacked... • Patching timely! • Secure UNIX/Linux? Intrusion prevention? • Remote logging • No anonymous uploads for FTP • Vulnerability assessment • Real-time log monitoring • Who controls security? Admins or dedicated stuff?

Slide 26: Lessons: Patching? • Security patching: – Automatic patching? A BIG question. • Fast automated attacks (Honeynet) – manual patching might not be timely enough – Patch testing, system backup, mass (and timely!) deployments – Who drives the patching process? System admin or security department?

Slide 27: Lessons: Response • Network log forensics is enough if you have enough audit trail (from firewalls and IDS) and a tool to analyse it – Centralized audit trail storage very handy! • Disk forensics is often not reliable – it might or might not produce what you want – Finding something specific vs finding any evidence • Detailed analysis is for controlled environment (i.e. honeynets)

Slide 28: Lessons: Attackers I • Hackers are after YOU (and its not FUD!): – For CPU, disk, memory, network connection (rootkit stats) – Crime of opportunity is MUCH more likely that a targeted attack. Is it thus a bigger threat? • Massive scanning -> you slip once and your server is owned. – Early warning about such attacks does not exist (fully automated scan-attack-exploit in one package)

Slide 29: Lessons: Intruder Operations • Attacker's purposes (from deployed tools): – DoS platform – Scanning for vulnerabilities – Sniffing – IRC – Hoarding hacked boxes for unknown future goals • Note: only applies to low-level attackers • Use many hops (origin is unclear)

Slide 30: Conclusions • Good network design – A one time cost and adds a lot to security • Keeping systems up to date – Painful but necessary • IR and forensics – Network and disk forensics together give complete picture • Constant vigilance is expensive but necessary

Slide 31: Thanks for Viewing the Presentation Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA http://www.chuvakin.org Author of “Security Warrior” (O’Reilly) – http://www.securitywarrior.org Book on logs is coming soon! See http://www.info-secure.org for my papers, books, reviews and other security resources related to logs