The third session from a two day course for potential first responders I ran for a large financial services client.

1. Phil Huggins
February 2004

2.  The 2003 survey identified a combined total loss of
$201,797,340 for 251 organisations
 This was lower than each of the previous 3 three
years
 The average cost of a security incident is at it’s
lowest for some time in 2003 at $47,107 down from
a high of $80,000 dollars in 2002
 Based on organisations responses to the survey. No
idea how those estimates were arrived at or what
industries the loss reporting companies were in.

3.  At a basic level answering these questions is a good
start:
 Who worked on responding to or investigating the
incident?
 How many hours did each of them spend?
 How many people were prevented from working because
of the incident?
 How much productive time did each of them lose?
 How much do you pay each of those people to work for
you?
 How much overhead do you pay for your employees?
 Did you need to purchase new or replacement equipment?

4.  More complicated and difficult questions include:
 What business opportunities did you lose?
 How much revenue did you lose as a direct result of the
incident?
 How much damage was done to your reputation?
 While these shouldn’t be ignored it is probably more
useful to get a good measure of the simple costs for
immediate management purposes

5.  Established by a group of large universities in the
90’s
 First used in 1998 at the University of Washington in
New Zealand
 “The largest security incident in new Zealand
history” where 18 servers at the university were
compromised
 They estimated the average cost per compromised
host of the incident was approximately $1,544

6.  Individual cost per hour
 Wage divided by 52 Weeks = Weekly Cost
 Weekly cost divided by 40 hours = hourly cost
 Benefit rate of 28% added (Average of the
universities involved
 Incidental Expenses
 Hardware Stolen / Damaged
 Phone Bills
 etc

7.  http://project.honeynet.org/challenge/
 The Forensic Challenge is an opportunity for
incident responders to compete using copies of
compromised systems to see who can find the most
useful information
 Using I-CAMP they estimated that the average cost
of the forensic analyses in the challenge was
$2,067.46 +/- $810.12
 Using a consulting rate of $300 an hour they
estimate $22,620 +/- $393

8.  Incident Response is likely to cost
approximately $1,544 a system at university
pay scales
 Forensic Analysis of a compromised system is
likely to cost about $22,620
 These are both manpower intensive, costly
activities
 Need to plug our own figures into these to
get a better picture of the risks we carry

9.  https://cirdb.ceria.purdue.edu/website/
 Web-based system that tracks the on-going
costs of an incident response
 Can compare many incidents against each
other to track changing costs over time
 Can use the online database hosted at Cerias
if you are brave
 Otherwise requires PHP and MySQL on a
server in your network

10.  Call Tree & Escalation list
 Communication methods
 Public Relations and Legal
 Inventory and Contacts
 Procedures and policies
 Acquisition kit

11.  Goal: To clearly document who should be called
during an incident, and how (phone numbers)
 The structure of this tree depends on the size and
structure of the organization
 The escalation tree describes what people should be
called based on the severity of the incident
 The CEO will likely not need to immediately know that a
user got the latest email virus
 For internal incidents, the list may not be strictly
followed, in order to keep it quiet

12.  Response Team
 Public Relations
 Legal Counsel
 Compliance
 Data Protection Officer (Europe / Safe Harbor in US)
 Human Resources
 Firewall Group
 IDS & Monitoring Group
 Remote Access / VPN Group
 Physical Security Group
 Head of IT
 CIO / CSO (or CEO)

13. Executive Team
Public XSP:
Relations ISP, MSP
Response
Team
Detection Security IT Group
Manager Legal Compromised
General Counsel
Physical Forensics
Security Process
Law
Enforcement
Monitoring:
Firewall/IDS

14.  Goal: To clearly document how parties should communicate
 The needed infrastructure should be built ahead of time
 Normal business communication methods could be compromised
during an incident and should not be explicitly trusted until proven
otherwise.
 Out of band communication:
 Dedicated voice mail system
 3rd party email network and dialups
 Fax machines and non-networked printers
 Mobile phones
 Encryption for in-band and secure out-of-band email
 Exchange all keys and required tools ahead of time

15.  A single reporting method will make the response
more efficient
 An awareness program can teach users to call a
“hotline” number when incidents are suspected
(virus, DoS)
 The hotline can be the normal IT Help Desk or a
dedicated incident line
 The hotline operators will report the incident to the
proper responders
 The size of an organization will depend on how
formal this process is

16.  Public Relations may want to prepare press releases
and webpages for incidents in advance
 PR should be ready to ask the right questions if an
incident is reported to them by the media
 Legal Counsel may want to have all documentation
directed towards them so that they are “Attorney
Work Products”.
 This reduces the chances that it is given to the defense in
the discovery process

17.  Goal: To clearly document what systems exist and who is
responsible for each of them.
 Identify other systems that could be involved more easily
 Decrease the amount of time required to get accounts and
passwords by storing contact information
 For internal incidents, contact information may decrease the
number of people that hear about it
 Network maps will help when installing network monitors
and analyzing network traffic

18.  Goal: To script what steps will be taken when an
incident occurs
 Every incident is unique, but procedures give a
baseline to follow
 The level of detail depends on the organization:
 Outsourced response team or in-house
 Small IT group or several hundred members
 Special medical or legal privacy laws or internal policies

19.  Whom to call? (the Call & Escalation Tree)
 How to identify the scope of the incident?
 Which systems can never be shutdown?
 In which cases will an acquisition be performed for
forensics?
 How do I acquire data from a Windows system?
(Acquisition Procedures)
 How do I store the acquired data? (Data Handling
Procedures)
 Am I allowed to increase user monitoring?

20.  Basic
 Call the Head of IT Security
 Head of IT Security calls:
▪ Response Team: 617-621-3500
▪ Public Relations: 617-555-1000
▪ Legal: 617-555-1200
 Data Handling
 All acquired data must be stored in the safe in the Security
Office
 Original data must never be used during an analysis
 Chain of Custody forms must be created for all acquired
data

21.  Solaris Dead Acquisition
1. Insert SCSI drive into suspect system
2. Boot the suspect system from “Response Kit Solaris CD-ROM”
3. Mount the SCSI disk to /mnt:
mount /dev/dsk/c0t6d0s6 /mnt
4. Calculate MD5 checksum of suspect slices (repeat for each slice):
dd if=/dev/rdsk/c0t0d0s4 bs=8k | md5sum
5. dd the suspect slice to a file on the mounted disk (repeat for each
slice):
dd if=/dev/rdsk/c0t0d0s4 of=/mnt/c0t0d0s4.dd
bs=8k
6. Verify the MD5 checksum of the acquired slices:
md5sum /mnt/c0t0d0s4.dd

22.  Goal: Build (or at least identify) hardware and
software to safely and quickly collect data
during an incident.
 A pre-built kit will save time during an
incident.
 Acquisition procedures should be written for
the dedicated hardware

23.  An acquisition computer should have large amounts
of disk space
 Data is sent to the computer from either the
network interface or an IDE or SCSI bus
 UNIX (Linux) is better suited for this system because
it does not try to mount new disks
 This system can also be used for network
monitoring
 Examples:
 Linux rackmount system with a SCSI card
 Mac OS X laptop with Firewire IDE Enclosures

24.  Trusted binaries are needed during an acquisition
(the local ones could be trojaned)
 CDs should exist that can collect the needed data
from a live system and that can boot the hardware
into a trusted kernel with the needed tools
 Bootable Examples:
 @stake Pocket Security Toolkit
 Penguin Sleuth Kit
 Solaris Install CD
 AIX Install CD

25.  Large IDE and SCSI disks that are wiped with
all zeros
 Hand-held imaging devices
 Hub and network cables
 Power strip
 Digital camera
 Required forms