1. Introduction to Adversary
Evaluation Tools
@uccu 2018.6.30
<ajblane0612@gmail.com>
AjMaChInE

2. Outline - Adversary Evaluation Tools
● What
● Why
● How
● CVE-based Adervasry Emulation
● ATT&CK-Based CALDERA
● Metasploit-based Deep Exploit

3. Market: Endpoint Security [0]
2017, Gartner
EndPoint Protection + Endpoint Detection and Respose

4. IThome 2018

5. Why Market Exists
"When taken together, these issues of cost,
time, personnel, training, and design have
created a muddied arena where the meaning
and utility of red teaming can be extensively
debated, making it difficult for organizations to
incorporate red teaming into their internal
security procedures." [4]

6. Defensive security goes hand in
hand with automation

7. modern enterprises largely rely on autonomous
security software able to block and address
threats without needing constant human
supervision. Examples include firewalls, intrusion
detection/prevention systems, network proxies,
host anti-viruses, access control mechanisms, etc.
While system administrators are still needed to
configure, manage, and operate these suites,
automation has taken much of the grunt work out,
making the task of securely defending a network
much easier. Automation here provides significant
advances when assessing network security
posture, measuring software patch levels, security
controls, known threat indicators, and others with
ease." [4]

8. Automation of defensive
tools is not enough

9. These approaches, while successful in identifying
potential vulnerabilities, fail to consider the network’s
response in the face of a real-world cyber adversary.
Attacks are not limited to vulnerability exploits: after
the initial compromise, adversaries commonly
leverage intrinsic security dependencies to
strengthen and expand their footholds on a system.
In these scenarios, select actions that the adversary
executes may be benign by themselves – such as
using internal Windows commands – but when
combined or used maliciously, can cause significant
damage. Adversaries often execute complex attacks
that are missed by simple scanning tools; because
adversaries do not always exploit vulnerabilities, their
actions will often go unnoticed." [4]

10. The False Negative Problem for
blue teams who have no idea what
blue teams miss

11. Idea: Adversary Emulation
- Pentesting and red teams is out
- Automation
- Improve false negative problem, But ...

12. [5]

13. [5]
adversary evaluation tools
1. 在什麼樣的平台 ?
2. 在什麼情境下 ? e.g. post-compromise, exploit hosts
3. 有哪些 Tactics 與 Techniques 可使用 ?
4. 如何找到一個可行的攻擊 ?
5. Reports ?

14. Automated Exploit Execution V.S.
Automated Red Teaming
"More interesting than automated exploit
execution is the general automated red teaming
problem – here, the task is not to execute a single
exploit (interfacing with an operator or not), but
rather to conduct a lengthy, in-depth analysis of
the security of the network via real-world testing,
exposing the intrinsic security dependencies of the
target system" [4]

15. Adversary Emulation Tools
● CALDERA
● Metta
● APT Simulator
● Red Team Automation
● Invoke-Adversary
● Atomic Red Team
● Infection Monkey
● Blue Team Training Toolkit (BT3)
● DumpsterFire AutoTTP
● ...
[6]

16. 1. 在什麼樣的平台 ?
2. 在什麼情境下 ? e.g.: post-compromise, exploit hosts
3. 有哪些 Tactics 與 Techniques 可使用 ?
4. 如何找到一個可行的攻擊 ?
5. Reports ?

17. adversary evaluation tools
1. 在什麼樣的平台 ?
2. 在什麼情境下 ? e.g. post-compromise, exploit hosts
3. 有哪些 Tactics 與 Techniques 可使用 ?
4. 如何找到一個可行的攻擊 ?
5. Reports ?
https://attack.mitre.org/wiki/Main_Page
or Metasploit
or CVE

18. powershell_args.py in
Red Team Automation
[1]
E.G. Let’s Quickly
Map these Generic
Tactics to the
ATT&CK™ matrix
(techniques)

19. 1. 在什麼樣的平台 ?
2. 在什麼情境下 ? e.g. post-compromise, exploit hosts
3. 有哪些 Tactics 與 Techniques 可使用 ?
4. 如何找到一個可行的攻擊 ?
5. Reports
logic and decision engine

20. Finite State Machie(FSM) to
Implement Logic and Decision Engine
● Need to be explicitly told what to do
● Hard to extend to new techniques (software
engineer view)
● Action needs to be coded into FSM
● FSM logic needs to be recomputed
● Inflexible in operation; hard to configure

21. FSM

22. FSM?

23. Reports – E.G. Attack Graph
1. 在什麼樣的平台 ?
2. 在什麼情境下 ? e.g. post-compromise, exploit hosts
3. 有哪些 Tactics 與 Techniques 可使用 ?
4. 如何找到一個可行的攻擊 ?
5. Reports
Host
Exploit Technique

24. Reports – E.G. CALDERD

25. Adversary Emulation Tools
● CALDERA
● Metta
● APT Simulator
● Red Team Automation
● Invoke-Adversary
● Atomic Red Team
● Infection Monkey
● Blue Team Training Toolkit (BT3)
● DumpsterFire AutoTTP
● ...
[6]
實作上不完整

26. Implementation Reference
● CVE-based Adervasry Emulation [7]
● ATT&CK-Based CALDERA [5]
● Metasploit-based Deep Exploit [3]
● ...

27. CVE-based Adervasry Emulation [7]

28. Reducing Action Arsenal [7]
critical action
critical action

29. ATT&CK-Based CALDERA – The
Targeted Platform [5]

30. ATT&CK-Based CALDERA -
Designing an Adversary Decision
Engine - Example Scenario [5]
action

31. ATT&CK-Based CALDERA -
Designing an Adversary Decision
Engine - Example Scenario [5]

32. ATT&CK-Based CALDERA -
Designing an Adversary Decision
Engine
new actions
choose actions (planner.py)
data model
Finite State Machine

33. ATT&CK-Based CALDERA -
Designing an Adversary Decision
Engine – Data Model
operation.py objects.py

34. ATT&CK-Based CALDERA -
Designing an Adversary Decision
Engine – Declaring Actions
operation_steps.py

35. ATT&CK-Based CALDERA -
Designing an Adversary Decision
Engine – Declaring Actions
operation_steps.py

36. ATT&CK-Based CALDERA -
Designing an Adversary Decision
Engine – Heuristic [5]?

37. ATT&CK-Based CALDERA -
Designing an Adversary Decision
Engine – Planning Algorithm

38. Emulation (world.py)
functions in new environments?
In reality – handling uncertainty is a very
hard problem
Your environment

39. Dijkstra Turing Test++
(uncertainty)
[10]
Assumptions:
Known network graph
Known host configurations
Static network
Monotonic actions
(Partially Observable Markov
Decision Process: 部分可觀察馬可
夫決策過程 ) POMDP model
* action 給機率是否成功

40. e.g. Metasploit-based Deep Exploit
Adversary simulation Tools
* Design a model of the system at hand
* Use model-based attack planning to
generate the attacks

41. [3]

42. similar to Caldera's data model similar to Caldera's actions

43. GyoiThon
[3]

44. Reference
● [0] 2017, Putting the P, D, & R back into Endpoint
Protection Detection and Response
● [1] 2018, Introducing Endgame Red Team Automation
● [2] 2018, Metasploit Meets Machine Learning
● [3] 2018, GyoiThon/ Deep Expoloit
● [4] 2016, MITRE, Intelligent, automated red team
emulation
● [5] 2017, blackhat, The MITRE Corporation, CALDERA
Automating Adversary Emulation
● [6] 2018, List of Adversary Emulation Tools
● [7] 2009, Tools for Generating and Analyzing Attack
Graphs

45. Reference
● [8] "Adding New Adversary Techniques",
caldera/docs/add_technique.rst
● [9] Getting CALDERA to do What You Want,
caldera/docs/getting_to_do.rst
● [10] 2015, Simulated Penetration Testing: From "Dijkstra"
to "Turing Test++"