FTP Server Attack Case Study Company FTP Server Erased by Hackers: Detection, Response, Prevention Anton Chuvakin, Ph.D Senior Security Analyst InfraGard NJ May 29, 2002

Company FTP Server Erased by Hackers:

Detection, Response, Prevention

Anton Chuvakin, Ph.D

Senior Security Analyst

InfraGard NJ

May 29, 2002

FTP Server Attack: Summary What happened? Crime scene: network environment and incident discovery What was found? Case: network traffic analysis and system forensics What should have been done? Verdict: security prevention measures: what was done and what wasn’t

What happened?

Crime scene: network environment and incident discovery

What was found?

Case: network traffic analysis and system forensics

What should have been done?

Verdict: security prevention measures: what was done and what wasn’t

FTP Attack: Detailed Summary Victim: Example.com Medium computer hardware manufacturer FTP server found with erased disks IDS, firewall log analysis Hard-drive forensics What worked and what didn’t Security lessons from the event Lessons on prevention, detection and response

Victim: Example.com

Medium computer hardware manufacturer

FTP server found with erased disks

IDS, firewall log analysis

Hard-drive forensics

What worked and what didn’t

Security lessons from the event

Lessons on prevention, detection and response

FTP Server Attack Case Study Proverbial Monday Morning – FTP Server is not responding All remote communication fails Upon rebooting doesn’t come back Discovered that there is no OS anymore (was Linux RedHat 7.2) Incident Response plan is activated!

Proverbial Monday Morning – FTP Server is not responding

All remote communication fails

Upon rebooting doesn’t come back

Discovered that there is no OS anymore (was Linux RedHat 7.2)

Incident Response plan is activated!

Network Infrastructure

Network Defences External Firewall No DMZ -> External traffic No DMZ -> DMZ traffic Internet -> DMZ traffic: Only port 80 (HTTP) to web server allowed Only port 21 (FTP) to FTP server allowed (stateful) etc IDS 1 and 2 see all DMZ traffic and report through a separate LAN (not shown) to a central database

External Firewall

No DMZ -> External traffic

No DMZ -> DMZ traffic

Internet -> DMZ traffic:

Only port 80 (HTTP) to web server allowed

Only port 21 (FTP) to FTP server allowed (stateful)

etc

IDS 1 and 2 see all DMZ traffic and report through a separate LAN (not shown) to a central database

Network Defences Internal Firewall No DMZ -> internal LAN traffic No external firewall -> internal LAN traffic Proxy installed to handle internal to Internet Internal -> DMZ traffic: Only Secure Shell (SSH) from select hosts to all DMZ machines for management and file transfers Additional defences: Host-firewalls on all DMZ hosts Integrity checking software

Internal Firewall

No DMZ -> internal LAN traffic

No external firewall -> internal LAN traffic

Proxy installed to handle internal to Internet

Internal -> DMZ traffic:

Only Secure Shell (SSH) from select hosts to all DMZ machines for management and file transfers

Additional defences:

Host-firewalls on all DMZ hosts

Integrity checking software

Investigation Plan Decision: full analysis before redeployment Main investigation focus: What happened? Is the network still exposed? How to prevent it in the future? Secondary focus: If it was a hacker attack, track the actions of the attacker to learn more about them and/or recover their tools

Decision: full analysis before redeployment

Main investigation focus:

What happened?

Is the network still exposed?

How to prevent it in the future?

Secondary focus:

If it was a hacker attack, track the actions of the attacker to learn more about them and/or recover their tools

Network Log Forensics Source of information : firewall logs, IDS logs, system logs aggregated and presented by a log management software What can we learn from them? All inbound connections to DMZ servers (source: firewall) All outbound attempts from DMZ (firewall) Attacks launched (IDS 1, 2) Intra-DMZ connection attempts (host logs) Protocol and other details (IDS 1)

Source of information : firewall logs, IDS logs, system logs aggregated and presented by a log management software

What can we learn from them?

All inbound connections to DMZ servers (source: firewall)

All outbound attempts from DMZ (firewall)

Attacks launched (IDS 1, 2)

Intra-DMZ connection attempts (host logs)

Protocol and other details (IDS 1)

Network Log Forensics: Timeline Classic scenario (known from the honeynet): “ Horizontal” FTP scan (00:10:20-00:10:37) FTP server found, more probing (00:12-00:13) First connection established (00:13:30) Attack launched (00:16:02) Outbound connection from FTP attempted (00:17:14) SERVER IS 'OWNED'! More probing and trying to get out (00:17-00:30) File deployed on server (00:31:59) Server erased (00:38)

Classic scenario (known from the honeynet):

“ Horizontal” FTP scan (00:10:20-00:10:37)

FTP server found, more probing (00:12-00:13)

First connection established (00:13:30)

Attack launched (00:16:02)

Outbound connection from FTP attempted (00:17:14) SERVER IS 'OWNED'!

More probing and trying to get out (00:17-00:30)

File deployed on server (00:31:59)

Server erased (00:38)

Network Log Forensics: Results Attack happened from 00:10 to 00:38 Attacked penetrated the FTP server using http://www.cert.org/advisories/CA-2001-33.html WU-FTPD globbing vulnerability dated Nov 2001 (from IDS signatures) Attacker managed to upload a file on the server (most likely rootkit) Intruder failed to gain further ground (from IDS, firewall and host logs)

Attack happened from 00:10 to 00:38

Attacked penetrated the FTP server using http://www.cert.org/advisories/CA-2001-33.html WU-FTPD globbing vulnerability dated Nov 2001 (from IDS signatures)

Attacker managed to upload a file on the server (most likely rootkit)

Intruder failed to gain further ground (from IDS, firewall and host logs)

Disk Forensics Focus: Get system logs to confirm findings of network forensics (no remote logging) Recover attacker's toolkit UNIX disk forensics tools Strings, grep, awk, file, TCT, TASK, foremost

Focus:

Get system logs to confirm findings of network forensics (no remote logging)

Recover attacker's toolkit

UNIX disk forensics tools

Strings, grep, awk, file, TCT, TASK, foremost

Disk Forensics: Procedure I Procedure Get the disk Install in analysis machine Mount read-only ( mount -ro ) Copy the whole disk into image files ( dd ) Looking for log files: strings, grep strings /home/hacked-ftp-hdc1 | grep 'Mar 17' Found: FTPD logs, messages, inetd logs Show attacker’s connections

Procedure

Get the disk

Install in analysis machine

Mount read-only ( mount -ro )

Copy the whole disk into image files ( dd )

Looking for log files: strings, grep

strings /home/hacked-ftp-hdc1 | grep 'Mar 17'

Found: FTPD logs, messages, inetd logs

Show attacker’s connections

Disk Forensics: Procedure II Further into the disk... Use strings and less to look through the disk Search for ATTACK_IP, incoming, rootkit name, etc Discovered: FTP client logs (with attempts to get out) Rootkit installation script! List of rootkit files

Further into the disk...

Use strings and less to look through the disk

Search for ATTACK_IP, incoming, rootkit name, etc

Discovered:

FTP client logs (with attempts to get out)

Rootkit installation script!

List of rootkit files

Disk Forensics: Rootkit Unsophisticated LKM kit (all public stuff?) Blocks shell history and syslog audit trail Unpacks and installs components Deploys hidden backdoor SSH daemon Builds and deploys hiding LKM Installs some DoS tools and sniffer Makes changes to system configuration files Comments in Romanian Enough to hide from regular system admin

Unsophisticated LKM kit (all public stuff?)

Blocks shell history and syslog audit trail

Unpacks and installs components

Deploys hidden backdoor SSH daemon

Builds and deploys hiding LKM

Installs some DoS tools and sniffer

Makes changes to system configuration files

Comments in Romanian

Enough to hide from regular system admin

Disk Forensics: Tools I Coroner Toolkit (TCT) http://www.porcupine.org/forensics/ Erased file recovery (several methods) Intrusion timeline creation Live and post-mortem system data collection Filesystem changes analysis TASK http://www.atstake.com/research/tools/task/ Improved version of TCT Better Windows support (NTFS, etc)

Coroner Toolkit (TCT) http://www.porcupine.org/forensics/

Erased file recovery (several methods)

Intrusion timeline creation

Live and post-mortem system data collection

Filesystem changes analysis

TASK http://www.atstake.com/research/tools/task/

Improved version of TCT

Better Windows support (NTFS, etc)

Disk Forensics: Tools II Autopsy http://www.atstake.com/research/tools/autopsy/

Autopsy http://www.atstake.com/research/tools/autopsy/

Disk Forensics: Tools III UNIX file removal options Files are gone, but the directory entries and content remain Files and dir entries and gone, content remains Files are gone and some of the content is gone Files are gone and content overwritten TCT and TASK operate on 1) , TCT can also operate on 2) with certain probability Our case: 2)

UNIX file removal options

Files are gone, but the directory entries and content remain

Files and dir entries and gone, content remains

Files are gone and some of the content is gone

Files are gone and content overwritten

TCT and TASK operate on 1) , TCT can also operate on 2) with certain probability

Our case: 2)

Disk Forensics: Unerase TCT lazarus tool: the hard way to unerase Get all the unallocated space from disk Look at chunks of data to determine their type (text, binary) Analyses the chunks to further determine type (C code, email, log files, executables, compressed) Concatenates consecutive chunks of the same type into one file Very slow (hours...) Recovery is a game of chance Results need further processing

TCT lazarus tool: the hard way to unerase

Get all the unallocated space from disk

Look at chunks of data to determine their type (text, binary)

Analyses the chunks to further determine type (C code, email, log files, executables, compressed)

Concatenates consecutive chunks of the same type into one file

Very slow (hours...)

Recovery is a game of chance

Results need further processing

Disk Forensics: Hunt for the Kit I List of files: a.sh, adore.tgz, sshutils.tgz, utils.tgz, mount.tar.gz Look for all the lazarus results of the compressed type and try to uncompress them Look for these filenames within other files Look for the files of similar size (from logs) Look for files adjacent to found rootkit script Look for other archived files (e.g. tar ) Look for C source code All fails!

List of files: a.sh, adore.tgz, sshutils.tgz, utils.tgz, mount.tar.gz

Look for all the lazarus results of the compressed type and try to uncompress them

Look for these filenames within other files

Look for the files of similar size (from logs)

Look for files adjacent to found rootkit script

Look for other archived files (e.g. tar )

Look for C source code

All fails!

Disk Forensics: Hunt for the Kit II Enter foremost (new tool from USAF OSI) Signature-based file searching Tool uses custom binary strings typically found in files and other parameters Had signatures for images and docs Support for gzip files added by me

Enter foremost (new tool from USAF OSI)

Signature-based file searching

Tool uses custom binary strings typically found in files and other parameters

Had signatures for images and docs

Support for gzip files added by me

Disk Forensics: Hunt for the Kit III Foremost found two files: adore.tgz utilz.tgz Adore is standard LKM (backdoor, file, process, connection hiding, etc) Utils : imp, lil, slc, sense, fsch2 After some hostile binary analysis: Easy: imp, slc - DoS tools; lil, sense – sniffer; fsch2 – remains a mystery

Foremost found two files:

adore.tgz

utilz.tgz

Adore is standard LKM (backdoor, file, process, connection hiding, etc)

Utils : imp, lil, slc, sense, fsch2

After some hostile binary analysis:

Easy: imp, slc - DoS tools; lil, sense – sniffer; fsch2 – remains a mystery

Lessons: Detection If disk content is missing – a problem is suspected...  But! If attacker had decided to keep access? IDS with current (!) signature base helps But! IDS is only as good as the person looking at the alerts (thus, nobody looking is as good as no IDS ) Otherwise, IDS turns into an expensive incident response helper

If disk content is missing – a problem is suspected... 

But! If attacker had decided to keep access?

IDS with current (!) signature base helps

But! IDS is only as good as the person looking at the alerts (thus, nobody looking is as good as no IDS )

Otherwise, IDS turns into an expensive incident response helper

Lessons: Prevention Done Right Great DMZ architecture and rule sets Principle of least privilege utilized Block outgoing from DMZ is a must! Block DMZ to DMZ needed too Up-to-date IDS signatures Presence of log management software Incident response is very easy since all information is in one place

Great DMZ architecture and rule sets

Principle of least privilege utilized

Block outgoing from DMZ is a must!

Block DMZ to DMZ needed too

Up-to-date IDS signatures

Presence of log management software

Incident response is very easy since all information is in one place

Lessons: Prevention to Improve Example.com still got hacked... Patching timely! Secure UNIX/Linux? Intrusion prevention? Remote logging No anonymous uploads for FTP Vulnerability assessment Real-time log monitoring Who controls security? Admins or dedicated stuff?

Example.com still got hacked...

Patching timely!

Secure UNIX/Linux? Intrusion prevention?

Remote logging

No anonymous uploads for FTP

Vulnerability assessment

Real-time log monitoring

Who controls security? Admins or dedicated stuff?

Lessons: Patching? Security patching: Automatic patching? A BIG question. Fast automated attacks (Honeynet) – manual patching might not be timely enough Patch testing, system backup, mass (and timely!) deployments Who drives the patching process? System admin or security department?

Security patching:

Automatic patching? A BIG question.

Fast automated attacks (Honeynet) – manual patching might not be timely enough

Patch testing, system backup, mass (and timely!) deployments

Who drives the patching process? System admin or security department?

Lessons: Response Network log forensics is enough if you have enough audit trail (from firewalls and IDS) and a tool to analyse it Centralized audit trail storage very handy! Disk forensics is often not reliable – it might or might not produce what you want Finding something specific vs finding any evidence Detailed analysis is for controlled environment (i.e. honeynets)

Network log forensics is enough if you have enough audit trail (from firewalls and IDS) and a tool to analyse it

Centralized audit trail storage very handy!

Disk forensics is often not reliable – it might or might not produce what you want

Finding something specific vs finding any evidence

Detailed analysis is for controlled environment (i.e. honeynets)

Lessons: Attackers I Hackers are after YOU (and its not FUD!): For CPU, disk, memory, network connection (rootkit stats) Crime of opportunity is MUCH more likely that a targeted attack. Is it thus a bigger threat? Massive scanning -> you slip once and your server is owned. Early warning about such attacks does not exist (fully automated scan-attack-exploit in one package)

Hackers are after YOU (and its not FUD!):

For CPU, disk, memory, network connection (rootkit stats)

Crime of opportunity is MUCH more likely that a targeted attack. Is it thus a bigger threat?

Massive scanning -> you slip once and your server is owned.

Early warning about such attacks does not exist (fully automated scan-attack-exploit in one package)

Lessons: Intruder Operations Attacker's purposes (from deployed tools): DoS platform Scanning for vulnerabilities Sniffing IRC Hoarding hacked boxes for unknown future goals Note: only applies to low-level attackers Use many hops (origin is unclear)

Attacker's purposes (from deployed tools):

DoS platform

Scanning for vulnerabilities

Sniffing

IRC

Hoarding hacked boxes for unknown future goals

Note: only applies to low-level attackers

Use many hops (origin is unclear)

Conclusions Good network design A one time cost and adds a lot to security Keeping systems up to date Painful but necessary IR and forensics Network and disk forensics together give complete picture Constant vigilance is expensive but necessary

Good network design

A one time cost and adds a lot to security

Keeping systems up to date

Painful but necessary

IR and forensics

Network and disk forensics together give complete picture

Constant vigilance is expensive but necessary

Thanks for Viewing the Presentation Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA http://www.chuvakin.org Author of “Security Warrior” (O’Reilly) – http://www.securitywarrior.org Book on logs is coming soon! See http://www.info-secure.org for my papers, books, reviews and other security resources related to logs

Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA

http://www.chuvakin.org

Author of “Security Warrior” (O’Reilly) – http://www.securitywarrior.org

Book on logs is coming soon!

See http://www.info-secure.org for my papers, books, reviews and other security resources related to logs