My presentation from Cloud Security World 2015 on effective incident response within cloud environments

1. Effective IR in Cloud
Environments
Andrew Case
Volatility

2. Presentation will be available at:
www.misti.com/download
Download password is available in your Show Guide

3. Slide 3
 Core Volatility developer
 Co-Author “Art of Memory Forensics”
 Lead-investigator on large-scale investigations
 Performed many RE efforts, pentests, and source code
audits
 Previously presented at Black Hat, RSA, Source,
DFRWS, BSides, and others
Who Am I?

4. Slide 4
 (Brief) overview of traditional incident response settings
 Challenges faced when traditional approaches are
applied to cloud environments
 Overcoming the challenges
 Leveraging unique features of the cloud for scalable and
effective incident response
Agenda

5. Slide 5
 Focused on mostly static networks
 IT has full control over system start, stop, reset,
refresh, etc.
 Collection usually performed directly on affected
systems or at least on the same internal network
Traditional IR

6. Slide 6
 Analysts have the ability to gather files, volatile data,
physical memory, and (full) disk images as needed
 Analysts have full control over both the host and guest
virtual machines of servers
 Logs are locally and easily accessible
Traditional IR Cont.

7. Slide 7
 The cloud is not static
 Systems may start and stop automatically in response
to processing load
 Including systems that were or still are compromised
 Volatile data is gone forever…
 IT staff generally has little control over the architecture
and resource allocation
Traditional IR vs Cloud IR – Environment

8. Slide 8
 Collection is done outside of the local environment,
over the Internet
 Collect to storage within the cloud?
 Secure credentials?
 Cost?
 Collect to the local environment?
 Speed?
 Capture in real-time vs in-cloud copies?
 Cost?
Traditional IR vs Cloud IR - Collection

9. Slide 9
 Acquiring traditional data sets is often difficult
 Full disk images are usually impossible
 Full memory captures possible, but chances of a
smeared image greatly increase with high system
activity
 Number of systems that may be comprised can be
enormous
 Live analysis tools trivially lied to by malware
 Particularly on Linux
Traditional IR vs Cloud IR - Collection

10. Slide 10
 Gathering logs faces many of the same issues as disk
and volatile data collection
 In-cloud SIEM may prevent reasonable local download
of logs
 Periodic transfer of logs from cloud to the local network
may leave gaps in real-time view
Traditional IR vs Cloud IR - Collection

11. Slide 11
 You (or your client) generally have little to no control
over the VM host when using the cloud
 This prevents acquisition of data from guests through
the host
 This necessitates the use of software within the guest
to acquire data
Traditional IR vs Cloud IR – VM Control

12. Slide 12
 During an incident is a bad time to move acquisition
tools to system(s)
 Many fully automated deployments don’t enable SSH
 Administrators may have no remote access to the
system
 What then?
 Agents?
 “Backdoor” to enable remote administration?
Traditional IR vs Cloud IR – Acquisition Tools

13. Slide 13
 Incident response needs must be considered at all
stages of development, deployment, and ongoing
operations
 The goal of these efforts is to enable effective and
immediate response as well as ongoing detection of
threats
 Richard Mogull has done great work in this space
related to application and network security
 https://securosis.com/blog
Making Cloud IR Seamless

14. Slide 14
 Applications should be verified that all relevant logging
features are enabled
 In-house applications should be built with detailed
logging built-in and enabled
 This includes every action that you as an
investigator might want to later know about
 Malicious insiders and remote attackers should never be
able to use your own app against you and you not be
able to later track down exactly what they did
Making Cloud IR Seamless – App Dev

15. Slide 15
 As systems are built, automated forensics tools should
be used to base line the system’s “normal” state
 Both on-disk and in-memory artifacts
 Prevent guessing during incidents
 Immediately pinpoint suspicious artifacts
 Systems should be checked to ensure that all relevant
logging is enabled
Making Cloud IR Seamless – Pre-Deployment

16. Slide 16
 Tools required for collection of forensics artifacts need
to be installed with the base system
 How to collect if entirety of disk is not acquirable?
 “Select” files
 What to do with memory?
 Acquisition of artifacts through APIs is vulnerable to
malware interference
 “Live” memory forensics isn’t
 A good compromise when you can’t get a full sample
of RAM
Making Cloud IR Seamless – Enabling Collection

17. Slide 17
 If the system can be automatically spun down, ensure
the logging is remote
 Scalable, remote logging is preferred in most cases
even if the system is stable
 Have automated methods to gather data of interest
 Be proactive about finding threats – don’t wait for
signatures (AV, HIPs, IDS) to fire!
Making Cloud IR Seamless – Post-Deployment

18. Slide 18
 Required in both traditional and cloud environments
 Over 60% of breaches were “discovered” after 3rd party
notification
 Existing technology will only catch skilled adversaries if
they make a mistake
Proactive Incident Response - Motivation

19. Slide 19
 Constantly gather and evaluate system state
 Processes
 Network connections
 AutoRun locations
 … many more data points
 Compare current state to baselines
 Use IOCs, threat intel data, etc. to find known badness
Proactive Incident Response – Howto

20. Slide 20
 Leverage IR-only credentials
 Leverage IR-only instances
 Stop any auto termination of (potentially) affected
hosts
 Use automated scripts to gather as much data as
possible
 Leverage features of the cloud to enhance response
and minimize disruption
Making Cloud IR Seamless – Active Incident

21. Slide 21
 While IR in the cloud has many challenges, it also has
unique features that can be very beneficial
 When used correctly, large-scale, automated detection
and collection becomes possible
Leveraging the Cloud for Better IR

22. Slide 22
 Pre-built instances that have the tools (software) and
storage needed to support IR
 No need to configure and install tools during an
incident
 Removes bottlenecks related to people power as well
as processing power
 Can use credentials separate from the rest of the
environment
IR-Only Instances

23. Slide 23
 Production instances are often under medium to heavy
load
 This pollutes forensics data and makes live analysis
challenging
 Fix:
 Isolate (potentially) affected instances
 Spin up new production instances to replace
compute power
 Benefits:
 More time to gather data in a stable manner
 No adverse effects on customers or performance
Virtual Guest Isolation

24. Slide 24
 Can inspect the state of VM guests without direct
interaction
 Avoids the issue of malware interference or notifying
attackers of forensics activity
 Much simpler to automate and scale
 Collected data can be safely stored on the VM host until
needed
 A huge security boost to private clouds and managed
security from public providers
Virtual Machine Introspection

25. Slide 25
 Snapshots include both volatile memory (RAM) and the
file system (disk)
 The guest cannot detect itself being snapshotted*
 Again - no chance for malware interference or
attacker notification
 Can periodically snapshot and keep for days or weeks
after
 Determine exact time of infection and state changes
since then
Virtual Machine Guest Snapshots

26. Slide 26
 Coming from traditional IR settings, the cloud can be
quite challenging
 Pre-planning is required to effective
 Agreed upon processes to capture and analyze data
 Pre-allocation of resources
 Full-scale exercises to test all points of response
 Automated as much as possible
 Continuous threat hunting
 The cloud also provides unique features that, if
leveraged properly, can make IR much more effective
Summary

27. Contact:
andrew@dfir.org
@attrc
Please Remember To Fill Out Your
Session Evaluation Forms!