1. Hit „em Where it Hurts:
A Live Security Exercise on
Cyber Situational Awareness
Adam Doupé, Manuel Egele, Benjamin Caillat,
Gianluca Stringhini, Gorkem Yakin, Ali Zand,
Ludovico Cavedon, and Giovanni Vigna
University of California, Santa Barbara
ACSAC 2011 – 7/12/11

2. What Are Live Security
Competitions?
• AKA Hacking Competitions
• Useful educational tool for teaching computer
security
• Born as a way to showcase security skills
– DefCon‟s CTF
• Various forms
– Challenge set (DefCon quals, iCTF challenges,
CMU‟s competition, DIMVA competition, RuCTF)
– Capture the flag (DefCon, iCTF 2003-2007, CIPHER)
– Other designs
• Attack-only (e.g., iCTF 2008)
• Defense-only (e.g., Cyber Defense eXercise)

3. Why Live Security Competitions?
• Real-time factor enhances understanding
• Forces teams to:
– Analyze unknown services/binaries
– Defend systems from attack
– Utilize different security skills
– Work as a team
– Create novel tools
Doupé - 7/12/11

4. Key Insight
• Security competitions can be designed to
generate datasets for research
• In the 2010 international Capture The Flag
(iCTF), we structured the competition to
create a Cyber Situational Awareness
dataset
Doupé - 7/12/11

5. Situational Awareness
• By putting perceived events into the context
of the currently executing mission, one can
improve decision making
• Mission
– Series of tasks that an organization wishes to
carry out
• Task
– Discrete step that is carried out using a service
• Service
– Provided to users to accomplish a task
Doupé - 7/12/11

6. Cyber Situational Awareness
• Situational awareness extended to the
cyber domain
• Large organizations constantly under
attack
– Which attacks are important?
– Which assets are important?
• “What if” scenarios
Doupé - 7/12/11

7. Overview
• Live Security Competitions
• Situational Awareness
• Design of the 2010 iCTF
• Cyber Situational Awareness Metrics
• Lessons Learned
• Conclusion
Doupé - 7/12/11

8. The 2010 iCTF: A Cyber SA
Competition
• Introduced the concept of cyber-mission
• “Not all attacks are created equal”
• Participants must be aware of cyber-
missions and cyber-assets
• Attackers must time their attacks to cause
the maximum amount of damage

9. The Setting
• Teams are part of a coalition to bring down the
rogue nation of Litya
• LityaLeaks site used to leak description of Litya‟s
cyber-missions
• Litya‟s network protected by a firewall and an IDS
– If an attack is detected, nation‟s access is shut off
– Nations can bribe network administrator
• Litya has a botnet in each nation, stealing their
money
– If botnet is disabled, nation‟s access shut off
• Money made by solving side challenges.

10. CARGODSTR-TQ-1442

11. COMSAT-WK-1127

12. SEDAFER-GOT-BKT-8217

13. DRIVEBY-DEPLOY-QFK-9751

14. Petri-net Representation of Mission
T8
T13
T2
T12
Failure Analysis
Establish Drive-by Detect Clean-up T10
Attack Failed
T7
T11
Reeval Deliver Attack
T9
T1
Start T4
T5
Blackhat SEO
Search Engine Result Analysis End
T3
Doupé - 7/12/11

15. .
. .
.
Service 1 Service 2 … Service 10

16. .
. The Bank
.
.
Service 1 Service 2 … Service 10 ScoreBot

17. .
. The Bank
.
.
Service 1 Service 2 … Service 10 ScoreBot Botnet C&C
Internal
Network
VPN server
…

18. .
. The Bank
.
.
Service 1 Service 2 … Service 10 ScoreBot Botnet C&C
Internal
Network
Firewall/IDS
Briber
Flag Submission
VPN server
…

19. .
. The Bank
.
.
Service 1 Service 2 … Service 10 ScoreBot Botnet C&C
LityaLeaks
Internal
Network Challenges
Firewall/IDS
Briber
ScoreBoard
Flag Submission
VPN server
…

20. .
. The Bank
.
.
Service 1 Service 2 … Service 10 ScoreBot Botnet C&C
LityaLeaks
Internal
Network Challenges
Firewall/IDS
Briber
ScoreBoard
Flag Submission
VPN server
…

21. Competition Overview
• December 3rd 2010 ~8 hours
• 72 teams
• ~900 participants (largest at the time)
• 7 of 10 services compromised
• 39 teams submitted 872 flags
• 69 of 72 teams solved at least 1 challenge
• 37 GB of traffic
Doupé - 7/12/11

22. Analysis of iCTF Data
• Use the data to validate models and
theories
• We introduce two Situational Awareness
metrics:
– Toxicity
• Capture the amount of damage an attacker has
caused
– Effectiveness
• Capture how effective the attacker was at causing
damage
Doupé - 7/12/11

23. Analysis – CAD - Criticality
• C(s, t): service criticality [0,1]
– Expresses the criticality of service s at time t
– Function can have any shape
• iCTF: 1 when service active, 0 otherwise
Service: MostWanted

24. Analysis – CAD - Attacker
• A(a, s, t): attacker activity [0, 1]
– Represent the attacker‟s activity with respect
to a service
– Can have any shape
• iCTF: 1 when team attacked a service, 0 if no attack
Team: PPP Service: MostWanted

25. Analysis – CAD - Damage
• D(s, t): Damage to the attacker [0, 1]
– Represents the penalty for performing an
attack against service s at time t
– Can have any shape
• iCTF: 1 when service is inactive, 0 when active
Service: MostWanted

26. Analysis – Toxicity
ò
t2
Toxicity(a, s, t1, t2 ) = A(a, s, t)× (C(s, t) - D(s, t)) dt
t1
ì 1 if C(s, t) - D(s, t) > 0
ï
OptimalAttacker(s, t) = í
ï 0
î otherwise

27. Analysis – Effectiveness
ò
t2
MaxToxicity(s, t1, t2 ) = OptimalAttacker(s, t)× (C(s, t) - D(s, t)) dt
t1
Toxicity(a, s, t1, t2 )
Effectiveness(a, s, t1, t2 ) =
MaxToxicity(s, t1, t2 )

28. Analysis – Toxicity of PPP
Team: PPP Service: OvertCovert

29. Analysis – Toxicity and
Effectiveness

30. Overview
• Live Security Competitions
• Situational Awareness
• Design of the 2010 iCTF
• Cyber Situational Awareness Metrics
• Lessons Learned
• Conclusion
Doupé - 7/12/11

31. Lessons Learned
• The Good
– Pre-competition information prepared teams who
took advantage
– Winning team automatically qualified for DefCon
• The Bad
– Structure of the competition was complex and
was understood by a subset of the teams
– Services too hard
• The Ugly
– Intentionally put a root backdoor into bot
– Losing points sucks
Doupé - 7/12/11

32. Conclusions
• Live security exercises great for learning
and security education
• They can be designed to create a
research dataset
• Designed the 2010 iCTF to produce the
first publically available dataset on CSA
• Presented SA metrics: toxicity and
effectiveness

33. Questions?
Data: http://ictf.cs.ucsb.edu/data/ictf2010/
Email: adoupe@cs.ucsb.edu
Twitter: @adamdoupe
Doupé - 7/12/11

34. Service Exploitation