Download to read offline
![19/52
Optimal Stopping Strategy
I The defender can compute the belief
bt , P[Si,t = C | b1, o1, o2, . . . ot]
I Stopping strategy: π(b) : [0, 1] → {S, C}](https://image.slidesharecdn.com/hammarericssonresearchdec2023-231208130205-8fcc76b1/85/Learning-Automated-Intrusion-Response-38-320.jpg)
![19/52
Optimal Threshold Strategy
Theorem
There exists an optimal defender strategy of the form:
π?
(b) = S ⇐⇒ b ≥ α?
α?
∈ [0, 1]
i.e., the stopping set is S = [α?, 1]
b
0 1
belief space B = [0, 1]
S1
α?
1](https://image.slidesharecdn.com/hammarericssonresearchdec2023-231208130205-8fcc76b1/85/Learning-Automated-Intrusion-Response-39-320.jpg)
![21/52
Optimal Multi-Threshold Strategy
Theorem
I Stopping sets are nested Sl−1 ⊆ Sl for l = 2, . . . L.
I If (ot)t≥1 is totally positive of order 2 (TP2), there exists an
optimal defender strategy of the form:
π?
l (b) = S ⇐⇒ b ≥ α?
l , l = 1, . . . , L
where α?
l ∈ [0, 1] is decreasing in l.
b
0 1
belief space B = [0, 1]
S1
S2
.
.
.
SL
α?
1
α?
2
α?
L
. . .](https://image.slidesharecdn.com/hammarericssonresearchdec2023-231208130205-8fcc76b1/85/Learning-Automated-Intrusion-Response-41-320.jpg)
![22/52
Optimal Stopping Game
I Suppose the attacker is dynamic and decides when to start
and abort its intrusion.
Attacker
Defender
t = 1
t = T
τ1,1 τ1,2 τ1,3
τ2,1
t
Stopped
Game episode
Intrusion
I Find the optimal stopping times
maximize
τD,1,...,τD,L
minimize
τA,1,τA,2
E[J]
where J is the defender’s objective.](https://image.slidesharecdn.com/hammarericssonresearchdec2023-231208130205-8fcc76b1/85/Learning-Automated-Intrusion-Response-42-320.jpg)
![25/52
Efficient Computation of Best Responses
Algorithm 1: Threshold Optimization
1 Input: Objective function J, number of thresholds L,
parametric optimizer PO
2 Output: A approximate best response strategy π̂θ
3 Algorithm
4 Θ ← [0, 1]L
5 For each θ ∈ Θ, define πθ(bt) as
6 πθ(bt) ,
(
S if bt ≥ θi
C otherwise
7 Jθ ← Eπθ
[J]
8 π̂θ ← PO(Θ, Jθ)
9 return π̂θ
I Examples of parameteric optimization algorithmns: CEM, BO,
CMA-ES, DE, SPSA, etc.](https://image.slidesharecdn.com/hammarericssonresearchdec2023-231208130205-8fcc76b1/85/Learning-Automated-Intrusion-Response-45-320.jpg)
![28/52
Learning Curves in Simulation and Digital Twin
0
50
100
Novice
Reward per episode
0
50
100
Episode length (steps)
0.0
0.5
1.0
P[intrusion interrupted]
0.0
0.5
1.0
1.5
P[early stopping]
5
10
Duration of intrusion
−50
0
50
100
experienced
0
50
100
150
0.0
0.5
1.0
0.0
0.5
1.0
1.5
0
5
10
0 20 40 60
training time (min)
−50
0
50
100
expert
0 20 40 60
training time (min)
0
50
100
150
0 20 40 60
training time (min)
0.0
0.5
1.0
0 20 40 60
training time (min)
0.0
0.5
1.0
1.5
2.0
0 20 40 60
training time (min)
0
5
10
15
20
πθ,l simulation πθ,l emulation (∆x + ∆y) ≥ 1 baseline Snort IPS upper bound
0 20 40 60 80 100
# training iterations
0
1
2
3
Exploitability
0 20 40 60 80 100
# training iterations
−5.0
−2.5
0.0
2.5
5.0
Defender reward per episode
0 20 40 60 80 100
# training iterations
0
1
2
3
Intrusion length
(π1,l, π2,l) emulation (π1,l, π2,l) simulation Snort IPS ot ≥ 1 upper bound](https://image.slidesharecdn.com/hammarericssonresearchdec2023-231208130205-8fcc76b1/85/Learning-Automated-Intrusion-Response-48-320.jpg)
![28/52
Learning Curves in Simulation and Digital Twin
0
50
100
Novice
Reward per episode
0
50
100
Episode length (steps)
0.0
0.5
1.0
P[intrusion interrupted]
0.0
0.5
1.0
1.5
P[early stopping]
5
10
Duration of intrusion
−50
0
50
100
experienced
0
50
100
150
0.0
0.5
1.0
0.0
0.5
1.0
1.5
0
5
10
0 20 40 60
training time (min)
−50
0
50
100
expert
0 20 40 60
training time (min)
0
50
100
150
0 20 40 60
training time (min)
0.0
0.5
1.0
0 20 40 60
training time (min)
0.0
0.5
1.0
1.5
2.0
0 20 40 60
training time (min)
0
5
10
15
20
πθ,l simulation πθ,l emulation (∆x + ∆y) ≥ 1 baseline Snort IPS upper bound
0 20 40 60 80 100
# training iterations
0
1
2
3
Exploitability
0 20 40 60 80 100
# training iterations
−5.0
−2.5
0.0
2.5
5.0
Defender reward per episode
0 20 40 60 80 100
# training iterations
0
1
2
3
Intrusion length
(π1,l, π2,l) emulation (π1,l, π2,l) simulation Snort IPS ot ≥ 1 upper bound
Stopping is about timing; now we consider timing + action selection](https://image.slidesharecdn.com/hammarericssonresearchdec2023-231208130205-8fcc76b1/85/Learning-Automated-Intrusion-Response-49-320.jpg)
![37/52
The Intrusion Response Problem
maximize
πD∈ΠD
minimize
πA∈ΠA
E(πD,πA) [J]
subject to s
(D)
t+1 ∼ fD · | A
(D)
t , A
(D)
t
∀t
s
(A)
t+1 ∼ fA · | S
(A)
t , At
∀t
ot+1 ∼ Z · | S
(D)
t+1, A
(A)
t ) ∀t
a
(A)
t ∼ πA · | H
(A)
t
, a
(A)
t ∈ AA(st) ∀t
a
(D)
t ∼ πD · | H
(D)
t
, a
(D)
t ∈ AD ∀t
E(πD,πA) denotes the expectation of the random vectors
(St, Ot, At)t∈{1,...,T} when following the strategy profile (πD, πA)
(1) can be formulated as a zero-sum Partially Observed Stochastic
Game with Public Observations (a PO-POSG):
Γ = hN, (Si )i∈N , (Ai )i∈N , (fi )i∈N , u, γ, (b
(i)
1 )i∈N , O, Zi](https://image.slidesharecdn.com/hammarericssonresearchdec2023-231208130205-8fcc76b1/85/Learning-Automated-Intrusion-Response-68-320.jpg)
Uploaded byKim Hammar
Learning Automated Intrusion Response
The document discusses a framework for automated intrusion response using reinforcement learning. It involves creating a digital twin of the target infrastructure, learning defender strategies through simulation, and evaluating strategies. The goal is to develop self-learning systems that can optimize intrusion response over time as attacks evolve.
More Related Content
Similar to Learning Automated Intrusion Response
Learning Automated Intrusion Response
- 1. 0/52 Learning Automated IntrusionResponse Ericsson Research Kim Hammar & Rolf Stadler kimham@kth.se Division of Network and Systems Engineering KTH Royal Institute of Technology December 8, 2023
- 2. 1/52 Use Case: IntrusionResponse I A defender owns an infrastructure I Consists of connected components I Components run network services I Defender defends the infrastructure by monitoring and active defense I Has partial observability I An attacker seeks to intrude on the infrastructure I Has a partial view of the infrastructure I Wants to compromise specific components I Attacks by reconnaissance, exploitation and pivoting Attacker Clients . . . Defender 1 IPS 1 alerts Gateway 7 8 9 10 11 6 5 4 3 2 12 13 14 15 16 17 18 19 21 23 20 22 24 25 26 27 28 29 30 31
- 3. 2/52 Automated Intrusion Response Levelsof security automation No automation. Manual detection. Manual prevention. Lack of tools. 1980s 1990s 2000s-Now Research Operator assistance. Audit logs Manual detection. Manual prevention. Partial automation. Manual configuration. Intrusion detection systems. Intrusion prevention systems. High automation. System automatically updates itself.
- 4. 2/52 Automated Intrusion Response Levelsof security automation No automation. Manual detection. Manual prevention. Lack of tools. 1980s 1990s 2000s-Now Research Operator assistance. Audit logs Manual detection. Manual prevention. Partial automation. Manual configuration. Intrusion detection systems. Intrusion prevention systems. High automation. System automatically updates itself. Can we find effective security strategies through decision-theoretic methods?
- 5. 3/52 Our Framework forAutomated Intrusion Response s1,1 s1,2 s1,3 . . . s1,n s2,1 s2,2 s2,3 . . . s2,n . . . . . . . . . . . . . . . Digital Twin Target Infrastructure Model Creation & System Identification Strategy Mapping π Selective Replication Strategy Implementation π Simulation System Reinforcement Learning & Optimization Strategy evaluation & Model estimation Automation & Self-learning systems
- 6. 3/52 Our Framework forAutomated Intrusion Response s1,1 s1,2 s1,3 . . . s1,n s2,1 s2,2 s2,3 . . . s2,n . . . . . . . . . . . . . . . Digital Twin Target Infrastructure Model Creation & System Identification Strategy Mapping π Selective Replication Strategy Implementation π Simulation System Reinforcement Learning & Optimization Strategy evaluation & Model estimation Automation & Self-learning systems
- 7. 3/52 Our Framework forAutomated Intrusion Response s1,1 s1,2 s1,3 . . . s1,n s2,1 s2,2 s2,3 . . . s2,n . . . . . . . . . . . . . . . Digital Twin Target Infrastructure Model Creation & System Identification Strategy Mapping π Selective Replication Strategy Implementation π Simulation System Reinforcement Learning & Optimization Model estimation Automation & Self-learning systems
- 8. 3/52 Our Framework forAutomated Intrusion Response s1,1 s1,2 s1,3 . . . s1,n s2,1 s2,2 s2,3 . . . s2,n . . . . . . . . . . . . . . . Digital Twin Target Infrastructure Model Creation & System Identification Strategy Mapping π Selective Replication Strategy Implementation π Simulation System Reinforcement Learning & Optimization Model estimation Automation & Self-learning systems
- 9. 3/52 Our Framework forAutomated Intrusion Response s1,1 s1,2 s1,3 . . . s1,n s2,1 s2,2 s2,3 . . . s2,n . . . . . . . . . . . . . . . Digital Twin Target Infrastructure Model Creation & System Identification Strategy Mapping π Selective Replication Strategy Implementation π Simulation System Reinforcement Learning & Optimization Strategy evaluation & Model estimation Automation & Self-learning systems
- 10. 3/52 Our Framework forAutomated Intrusion Response s1,1 s1,2 s1,3 . . . s1,n s2,1 s2,2 s2,3 . . . s2,n . . . . . . . . . . . . . . . Digital Twin Target Infrastructure Model Creation & System Identification Strategy Mapping π Selective Replication Strategy Implementation π Simulation System Reinforcement Learning & Optimization Strategy evaluation Automation & Self-learning systems
- 11. 3/52 Our Framework forAutomated Intrusion Response s1,1 s1,2 s1,3 . . . s1,n s2,1 s2,2 s2,3 . . . s2,n . . . . . . . . . . . . . . . Digital Twin Target Infrastructure Model Creation & System Identification Strategy Mapping π Selective Replication Strategy Implementation π Simulation System Reinforcement Learning & Optimization Strategy evaluation & Model estimation Automation & Self-learning systems
- 12. 3/52 Our Framework forAutomated Intrusion Response s1,1 s1,2 s1,3 . . . s1,n s2,1 s2,2 s2,3 . . . s2,n . . . . . . . . . . . . . . . Digital Twin Target Infrastructure Model Creation & System Identification Strategy Mapping π Selective Replication Strategy Implementation π Simulation System Reinforcement Learning & Optimization Strategy evaluation & Model estimation Automation & Self-learning systems
- 13. 3/52 Creating a DigitalTwin of the Target Infrastructure s1,1 s1,2 s1,3 . . . s1,n s2,1 s2,2 s2,3 . . . s2,n . . . . . . . . . . . . . . . Digital Twin Target Infrastructure Model Creation & System Identification Strategy Mapping π Selective Replication Strategy Implementation π Simulation System Reinforcement Learning & Optimization Strategy evaluation & Model estimation Automation & Self-learning systems
- 14. 3/52 Learning of DefenderStrategies s1,1 s1,2 s1,3 . . . s1,n s2,1 s2,2 s2,3 . . . s2,n . . . . . . . . . . . . . . . Digital Twin Target Infrastructure Model Creation & System Identification Strategy Mapping π Selective Replication Strategy Implementation π Simulation System Reinforcement Learning & Optimization Strategy evaluation & Model estimation Automation & Self-learning systems
- 15. 4/52 Example Infrastructure Configuration I64 nodes I 24 ovs switches I 3 gateways I 6 honeypots I 8 application servers I 4 administration servers I 15 compute servers I 11 vulnerabilities I cve-2010-0426 I cve-2015-3306 I etc. I Management I 1 sdn controller I 1 Kafka server I 1 elastic server r&d zone App servers Honeynet dmz admin zone workflow Gateway idps quarantine zone alerts Defender . . . Attacker Clients 2 1 3 12 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
- 16. 5/52 Emulating Physical Components Containers Physicalserver Operating system Docker engine Our framework I We emulate physical components with Docker containers I Focus on linux-based systems I Our framework provides the orchestration layer
- 17. 6/52 Emulating Network Connectivity Managementnode 1 Emulated IT infrastructure Management node 2 Emulated IT infrastructure Management node n Emulated IT infrastructure VXLAN VXLAN . . . VXLAN IP network I We emulate network connectivity on the same host using network namespaces I Connectivity across physical hosts is achieved using VXLAN tunnels with Docker swarm
- 18. 7/52 Emulating Network Conditions ITraffic shaping using NetEm I Allows to configure: I Delay I Capacity I Packet Loss I Jitter I Queueing delays I etc. User space . . . Application processes Kernel TCP/UDP IP/Ethernet/802.11 OS TCP/IP stack Queueing discipline Device driver queue (FIFO) NIC Netem config: latency, jitter, etc. Sockets
- 19. 8/52 Emulating Clients Clientpopulation . . . Arrival rate λ Departure Service time µ . . . . . . . . . . . . w1 w2 w|W| Workflows (Markov processes) I Homogeneous client population I Clients arrive according to Po(λ) I Client service times Exp(µ) I Service dependencies (St)t=1,2,... ∼ mc
- 20. 9/52 Emulating The Attackerand The Defender I API for automated defender and attacker actions I Attacker actions: I Exploits I Reconnaissance I Pivoting I etc. I Defender actions: I Shut downs I Redirect I Isolate I Recover I Migrate I etc. Markov Decision Process s1,1 s1,2 s1,3 . . . s1,4 s2,1 s2,2 s2,3 . . . s2,4 Digital Twin . . . Virtual network Virtual devices Emulated services Emulated actors IT Infrastructure Configuration & change events System traces Verified security policy Optimized security policy
- 21. 10/52 Software framework Metastore Python libraries Managementapi rest api cli grpc Management api Digital twins I More details about the software framework I Source code: https://github.com/Limmen/csle I Documentation: http://limmen.dev/csle/ I Demo: https://www.youtube.com/watch?v=iE2KPmtIs2A
- 22. 11/52 System Identification s1,1 s1,2s1,3 . . . s1,n s2,1 s2,2 s2,3 . . . s2,n . . . . . . . . . . . . . . . Digital Twin Target Infrastructure Model Creation & System Identification Strategy Mapping π Selective Replication Strategy Implementation π Simulation System Reinforcement Learning & Generalization Strategy evaluation & Model estimation Automation & Self-learning systems
- 23. 12/52 System Model H C ∅ Crashed HealthyCompromised Model complexity Static attacker Small set of responses Dynamic attacker Small set of responses Dynamic attacker Large set of responses I Intrusion response can be modeled in many ways I As a parametric optimization problem I As an optimal stopping problem I As a dynamic program I As a game I etc.
- 24. 13/52 Related Work onLearning Automated Intrusion Response External validity model complexity Goal Georgia et al. 2000. (Next generation intrusion detection: reinforcement learning) Xu et al. 2005. (An RL approach to host-based intrusion detection) Servin et al. 2008. (Multi-agent RL for intrusion detection) Malialis et al. 2013. (Decentralized RL response to DDoS attacks) Zhu et al. 2019. (Adaptive Honeypot engagement) Apruzzese et al. 2020. (Deep RL to evade botnets) Xiao et al. 2021. (RL approach to APT) etc. 2022-2023 Our work 2020-2023
- 25. 14/52 Intrusion Response throughOptimal Stopping I Suppose I The attacker follows a fixed strategy (no adaptation) I We only have one response action, e.g., block the gateway I Formulate intrusion response as optimal stopping Intrusion event time-step t = 1 Intrusion ongoing t t = T Early stopping times Stopping times that affect the intrusion Episode
- 26. 15/52 Intrusion Response fromthe Defender’s Perspective 20 40 60 80 100 120 140 160 180 200 0.5 1 t b 20 40 60 80 100 120 140 160 180 200 50 100 150 200 t Alerts 20 40 60 80 100 120 140 160 180 200 5 10 15 t Logins
- 27. 15/52 Intrusion Response fromthe Defender’s Perspective 20 40 60 80 100 120 140 160 180 200 0.5 1 t b 20 40 60 80 100 120 140 160 180 200 50 100 150 200 t Alerts 20 40 60 80 100 120 140 160 180 200 5 10 15 t Logins
- 28. 15/52 Intrusion Response fromthe Defender’s Perspective 20 40 60 80 100 120 140 160 180 200 0.5 1 t b 20 40 60 80 100 120 140 160 180 200 50 100 150 200 t Alerts 20 40 60 80 100 120 140 160 180 200 5 10 15 t Logins
- 29. 15/52 Intrusion Response fromthe Defender’s Perspective 20 40 60 80 100 120 140 160 180 200 0.5 1 t b 20 40 60 80 100 120 140 160 180 200 50 100 150 200 t Alerts 20 40 60 80 100 120 140 160 180 200 5 10 15 t Logins
- 30. 15/52 Intrusion Response fromthe Defender’s Perspective 20 40 60 80 100 120 140 160 180 200 0.5 1 t b 20 40 60 80 100 120 140 160 180 200 50 100 150 200 t Alerts 20 40 60 80 100 120 140 160 180 200 5 10 15 t Logins
- 31. 15/52 Intrusion Response fromthe Defender’s Perspective 20 40 60 80 100 120 140 160 180 200 0.5 1 t b 20 40 60 80 100 120 140 160 180 200 50 100 150 200 t Alerts 20 40 60 80 100 120 140 160 180 200 5 10 15 t Logins
- 32. 15/52 Intrusion Response fromthe Defender’s Perspective 20 40 60 80 100 120 140 160 180 200 0.5 1 t b 20 40 60 80 100 120 140 160 180 200 50 100 150 200 t Alerts 20 40 60 80 100 120 140 160 180 200 5 10 15 t Logins
- 33. 15/52 Intrusion Response fromthe Defender’s Perspective 20 40 60 80 100 120 140 160 180 200 0.5 1 t b 20 40 60 80 100 120 140 160 180 200 50 100 150 200 t Alerts 20 40 60 80 100 120 140 160 180 200 5 10 15 t Logins
- 34. 15/52 Intrusion Response fromthe Defender’s Perspective 20 40 60 80 100 120 140 160 180 200 0.5 1 t b 20 40 60 80 100 120 140 160 180 200 50 100 150 200 t Alerts 20 40 60 80 100 120 140 160 180 200 5 10 15 t Logins When to take a defensive action?
- 35. 16/52 The Defender’s OptimalStopping Problem (1/3) I Infrastructure is a discrete-time dynamical system (st)T t=1 I Defender observes a noisy observation process (ot)T t=1 I Two options at each time t: (C)ontinue and (S)stop I Find the optimal stopping time τ?: τ? ∈ arg max τ Eτ " τ−1 X t=1 γt−1 RC st st+1 + γτ−1 RS sτ sτ # where RS ss0 & RC ss0 are the stop/continue rewards and τ is τ = inf{t : t > 0, at = S} t ot τ∗
- 36. 17/52 The Defender’s OptimalStopping Problem (2/3) I Objective: stop the attack as soon as possible I Let the state space be S = {H, C, ∅} H C ∅ Stopped Healthy Compromised
- 37. 18/52 The Defender’s OptimalStopping Problem (3/3) I Let the observation process (ot)T t=1 represent ids alerts 0 2000 4000 6000 8000 probability cve-2010-0426 0 2000 4000 6000 8000 cve-2015-3306 0 2000 4000 6000 8000 cve-2015-5602 0 2000 4000 6000 8000 cve-2016-10033 0 2000 4000 6000 8000 cwe-89 0 2000 4000 6000 8000 O probability cve-2017-7494 0 2000 4000 6000 8000 O cve-2014-6271 0 5000 10000 15000 20000 O ftp brute force 0 5000 10000 15000 20000 O ssh brute force 0 5000 10000 15000 20000 O telnet brute force intrusion no intrusion intrusion model normal operation model I Estimate the observation distribution based on M samples from the twin I E.g., compute empirical distribution b Z as estimate of Z I b Z →a.s Z as M → ∞ (Glivenko-Cantelli theorem)
- 38. 19/52 Optimal Stopping Strategy IThe defender can compute the belief bt , P[Si,t = C | b1, o1, o2, . . . ot] I Stopping strategy: π(b) : [0, 1] → {S, C}
- 39. 19/52 Optimal Threshold Strategy Theorem Thereexists an optimal defender strategy of the form: π? (b) = S ⇐⇒ b ≥ α? α? ∈ [0, 1] i.e., the stopping set is S = [α?, 1] b 0 1 belief space B = [0, 1] S1 α? 1
- 40. 20/52 Optimal Multiple Stopping ISuppose the defender can take L ≥ 1 response actions I Find the optimal stopping times τ? L , τ? L−1, . . . , τ? 1 : (τ? l )l=1,...,L ∈ arg max τ1,...,τL Eτ1,...,τL " τL−1 X t=1 γt−1 RC st st+1 + γτL−1 RS sτL sτL + τL−1−1 X t=τL+1 γt−1 RC st st+1 + γτL−1−1 RS sτL−1 sτL−2 + . . . + τ1−1 X t=τ2+1 γt−1 RC st st+1 + γτ1−1 RS sτ1 sτ1 # where τl denotes the stopping time with l stops remaining. t ot τ? L τ? L−1 τ? L−2 . . .
- 41. 21/52 Optimal Multi-Threshold Strategy Theorem IStopping sets are nested Sl−1 ⊆ Sl for l = 2, . . . L. I If (ot)t≥1 is totally positive of order 2 (TP2), there exists an optimal defender strategy of the form: π? l (b) = S ⇐⇒ b ≥ α? l , l = 1, . . . , L where α? l ∈ [0, 1] is decreasing in l. b 0 1 belief space B = [0, 1] S1 S2 . . . SL α? 1 α? 2 α? L . . .
- 42. 22/52 Optimal Stopping Game ISuppose the attacker is dynamic and decides when to start and abort its intrusion. Attacker Defender t = 1 t = T τ1,1 τ1,2 τ1,3 τ2,1 t Stopped Game episode Intrusion I Find the optimal stopping times maximize τD,1,...,τD,L minimize τA,1,τA,2 E[J] where J is the defender’s objective.
- 43. 23/52 Best-Response Multi-Threshold Strategies(1/2) Theorem I The defender’s best response is of the form: π̃D,l (b) = S ⇐⇒ b ≥ α̃l , l = 1, . . . , L I The attacker’s best response is of the form: π̃A,l (b) = C ⇐⇒ π̃D,l (S | b) ≥ β̃H,l , l = 1, . . . , L, s = H π̃A,l (b) = S ⇐⇒ π̃D,l (S | b) ≥ β̃C,l , l = 1, . . . , L, s = C
- 44. 24/52 Best-Response Multi-Threshold Strategies(2/2) b Defender 0 1 S (D) 1,π2,l S (D) 2,π2,l . . . S (D) L,π2,l α̃1 α̃2 α̃L . . . b Attacker 0 1 S (A) C,1,πD,l S (A) C,L,πD,l β̃C,1 β̃C,L β̃H,1 . . . β̃H,L . . . S (A) H,1,πD,l S (A) H,L,πD,l
- 45. 25/52 Efficient Computation ofBest Responses Algorithm 1: Threshold Optimization 1 Input: Objective function J, number of thresholds L, parametric optimizer PO 2 Output: A approximate best response strategy π̂θ 3 Algorithm 4 Θ ← [0, 1]L 5 For each θ ∈ Θ, define πθ(bt) as 6 πθ(bt) , ( S if bt ≥ θi C otherwise 7 Jθ ← Eπθ [J] 8 π̂θ ← PO(Θ, Jθ) 9 return π̂θ I Examples of parameteric optimization algorithmns: CEM, BO, CMA-ES, DE, SPSA, etc.
- 46. 26/52 Threshold-Fictitious Play toApproximate an Equilibrium π̃A ∈ BA(πD) πA πD π̃D ∈ BD(πA) π̃0 A ∈ BA(π0 D) π0 A π0 D π̃0 D ∈ BD(π0 A) . . . π? A ∈ BA(π? D) π? D ∈ BD(π? A) Fictitious play: iterative averaging of best responses. I Learn best response strategies iteratively I Average best responses to approximate the equilibrium
- 47. 27/52 Comparison against State-of-the-artAlgorithms 0 10 20 30 40 50 60 training time (min) 0 50 100 Reward per episode against Novice 0 10 20 30 40 50 60 training time (min) Reward per episode against Experienced 0 10 20 30 40 50 60 training time (min) Reward per episode against Expert PPO ThresholdSPSA Shiryaev’s Algorithm (α = 0.75) HSVI upper bound
- 48. 28/52 Learning Curves inSimulation and Digital Twin 0 50 100 Novice Reward per episode 0 50 100 Episode length (steps) 0.0 0.5 1.0 P[intrusion interrupted] 0.0 0.5 1.0 1.5 P[early stopping] 5 10 Duration of intrusion −50 0 50 100 experienced 0 50 100 150 0.0 0.5 1.0 0.0 0.5 1.0 1.5 0 5 10 0 20 40 60 training time (min) −50 0 50 100 expert 0 20 40 60 training time (min) 0 50 100 150 0 20 40 60 training time (min) 0.0 0.5 1.0 0 20 40 60 training time (min) 0.0 0.5 1.0 1.5 2.0 0 20 40 60 training time (min) 0 5 10 15 20 πθ,l simulation πθ,l emulation (∆x + ∆y) ≥ 1 baseline Snort IPS upper bound 0 20 40 60 80 100 # training iterations 0 1 2 3 Exploitability 0 20 40 60 80 100 # training iterations −5.0 −2.5 0.0 2.5 5.0 Defender reward per episode 0 20 40 60 80 100 # training iterations 0 1 2 3 Intrusion length (π1,l, π2,l) emulation (π1,l, π2,l) simulation Snort IPS ot ≥ 1 upper bound
- 49. 28/52 Learning Curves inSimulation and Digital Twin 0 50 100 Novice Reward per episode 0 50 100 Episode length (steps) 0.0 0.5 1.0 P[intrusion interrupted] 0.0 0.5 1.0 1.5 P[early stopping] 5 10 Duration of intrusion −50 0 50 100 experienced 0 50 100 150 0.0 0.5 1.0 0.0 0.5 1.0 1.5 0 5 10 0 20 40 60 training time (min) −50 0 50 100 expert 0 20 40 60 training time (min) 0 50 100 150 0 20 40 60 training time (min) 0.0 0.5 1.0 0 20 40 60 training time (min) 0.0 0.5 1.0 1.5 2.0 0 20 40 60 training time (min) 0 5 10 15 20 πθ,l simulation πθ,l emulation (∆x + ∆y) ≥ 1 baseline Snort IPS upper bound 0 20 40 60 80 100 # training iterations 0 1 2 3 Exploitability 0 20 40 60 80 100 # training iterations −5.0 −2.5 0.0 2.5 5.0 Defender reward per episode 0 20 40 60 80 100 # training iterations 0 1 2 3 Intrusion length (π1,l, π2,l) emulation (π1,l, π2,l) simulation Snort IPS ot ≥ 1 upper bound Stopping is about timing; now we consider timing + action selection
- 50. 29/52 General Intrusion ResponseGame I Suppose the defender and the attacker can take L actions per node I G = h{gw} ∪ V, Ei: directed tree representing the virtual infrastructure I V: set of virtual nodes I E: set of node dependencies I Z: set of zones r&d zone App servers Honeynet dmz admin zone workflow Gateway idps quarantine zone alerts Defender . . . Attacker Clients 2 1 3 12 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
- 51. 29/52 General Intrusion ResponseGame I Suppose the defender and the attacker can take L actions per node I G = h{gw} ∪ V, Ei: directed tree representing the virtual infrastructure I V: set of virtual nodes I E: set of node dependencies I Z: set of zones r&d zone App servers Honeynet dmz admin zone workflow Gateway idps quarantine zone alerts Defender . . . Attacker Clients 2 1 3 12 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
- 52. 30/52 State Space I Eachi ∈ V has a state vi,t = (v (Z) t,i |{z} D , v (I) t,i , v (R) t,i | {z } A ) I System state st = (vt,i )i∈V ∼ St I Markovian time-homogeneous dynamics: st+1 ∼ f (· | St, At) At = (A (A) t , A (D) t ) are the actions. s1 s2 s3 s4 s5 s4 . . . . . . . . .
- 53. 30/52 State Space I Eachi ∈ V has a state vi,t = (v (Z) t,i |{z} D , v (I) t,i , v (R) t,i | {z } A ) I System state st = (vt,i )i∈V ∼ St I Markovian time-homogeneous dynamics: st+1 ∼ f (· | St, At) At = (A (A) t , A (D) t ) are the actions. s1 s2 s3 s4 s5 s4 . . . . . . . . .
- 54. 30/52 State Space I Eachi ∈ V has a state vi,t = (v (Z) t,i |{z} D , v (I) t,i , v (R) t,i | {z } A ) I System state st = (vt,i )i∈V ∼ St I Markovian time-homogeneous dynamics: st+1 ∼ f (· | St, At) At = (A (A) t , A (D) t ) are the actions. s1 s2 s3 s4 s5 s4 . . . . . . . . .
- 55. 31/52 Workflows I Services areconnected into workflows W = {w1, . . . , w|W|}.
- 56. 31/52 Workflows I Services areconnected into workflows W = {w1, . . . , w|W|}. gw fw idps lb http servers auth server search engine db cache
- 57. 32/52 Workflows I Services areconnected into workflows W = {w1, . . . , w|W|}. I Each w ∈ W is realized as a subtree Gw = h{gw} ∪ Vw, Ewi of G I W = {w1, . . . , w|W|} induces a partitioning V = [ wi ∈W Vwi such that i 6= j =⇒ Vwi ∩ Vwj = ∅ Zone a Zone b Zone c gw 1 2 3 4 5 6 7 A workflow tree
- 58. 32/52 Workflows I Services areconnected into workflows W = {w1, . . . , w|W|}. I Each w ∈ W is realized as a subtree Gw = h{gw} ∪ Vw, Ewi of G I W = {w1, . . . , w|W|} induces a partitioning V = [ wi ∈W Vwi such that i 6= j =⇒ Vwi ∩ Vwj = ∅ Zone a Zone b Zone c gw 1 2 3 4 5 6 7 A workflow tree
- 59. 33/52 Observations I idpss inspectnetwork traffic and generate alert vectors: ot , ot,1, . . . , ot,|V| ∈ N |V| 0 ot,i is the number of alerts related to node i ∈ V at time-step t. I ot = (ot,1, . . . , ot,|V|) is a realization of the random vector Ot with joint distribution Z idps idps idps idps alerts Defender . . . Attacker Clients 2 1 3 12 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
- 60. 33/52 Observations I idpss inspectnetwork traffic and generate alert vectors: ot , ot,1, . . . , ot,|V| ∈ N |V| 0 ot,i is the number of alerts related to node i ∈ V at time-step t. I ot = (ot,1, . . . , ot,|V|) is a realization of the random vector Ot with joint distribution Z idps idps idps idps alerts Defender . . . Attacker Clients 2 1 3 12 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
- 61. 34/52 probability ZO1 ZO2 ZO3 ZO4 ZO5 ZO6 ZO7 ZO8 probability ZO9 ZO10 ZO11 ZO12 ZO13 ZO14 ZO15 ZO16 probability ZO17 ZO18 ZO19 ZO20 ZO21 ZO22 ZO23 ZO24 probability ZO25 ZO26 ZO27 ZO28 ZO29 ZO30 ZO31 ZO32 probability ZO33 ZO34 ZO35 ZO36 ZO37 ZO38 ZO39 ZO40 probability ZO41 ZO42 ZO43 ZO44 ZO45 ZO46 ZO47 ZO48 probability ZO49 ZO50 ZO51 ZO52 ZO53 ZO54 ZO55 ZO56 250 500 750 O probability ZO57 250500 750 O ZO58 250 500 750 O ZO59 250 500 750 O ZO60 250 500 750 O ZO61 250 500 750 O ZO62 250 500 750 O ZO63 250 500 750 O ZO64 Distributions of # alerts weighted by priority ZOi (Oi | S (D) i , A (A) i ) per node i ∈ V no intrusion intrusion
- 62. 35/52 Defender I Defender action: a (D) t∈ {0, 1, 2, 3, 4}|V| I 0 means do nothing. 1 − 4 correspond to defensive actions (see fig) I A defender strategy is a function πD ∈ ΠD : HD → ∆(AD), where h (D) t = (s (D) 1 , a (D) 1 , o1, . . . , a (D) t−1, s (D) t , ot) ∈ HD I Objective: (i) maintain workflows; and (ii) stop a possible intrusion: J , T X t=1 γt−1 η |W| X i=1 uW(wi , st) | {z } workflows utility − (1 − η) |V| X j=1 cI(st,j, at,j) | {z } intrusion and defense costs ! dmz rd zone admin zone Old path New path Honeypot App node Defender Revoke certificates Blacklist IP 1) Node migration 2) Flow migration and blocking 3) Shut down node 4) Access control
- 63. 35/52 Defender I Defender action: a (D) t∈ {0, 1, 2, 3, 4}|V| I 0 means do nothing. 1 − 4 correspond to defensive actions (see fig) I A defender strategy is a function πD ∈ ΠD : HD → ∆(AD), where h (D) t = (s (D) 1 , a (D) 1 , o1, . . . , a (D) t−1, s (D) t , ot) ∈ HD I Objective: (i) maintain workflows; and (ii) stop a possible intrusion: J , T X t=1 γt−1 η |W| X i=1 uW(wi , st) | {z } workflows utility − (1 − η) |V| X j=1 cI(st,j, at,j) | {z } intrusion and defense costs ! dmz rd zone admin zone Old path New path Honeypot App node Defender Revoke certificates Blacklist IP 1) Node migration 2) Flow migration and blocking 3) Shut down node 4) Access control
- 64. 35/52 Defender I Defender action: a (D) t∈ {0, 1, 2, 3, 4}|V| I 0 means do nothing. 1 − 4 correspond to defensive actions (see fig) I A defender strategy is a function πD ∈ ΠD : HD → ∆(AD), where h (D) t = (s (D) 1 , a (D) 1 , o1, . . . , a (D) t−1, s (D) t , ot) ∈ HD I Objective: (i) maintain workflows; and (ii) stop a possible intrusion: J , T X t=1 γt−1 η |W| X i=1 uW(wi , st) | {z } workflows utility − (1 − η) |V| X j=1 cI(st,j, at,j) | {z } intrusion and defense costs ! dmz rd zone admin zone Old path New path Honeypot App node Defender Revoke certificates Blacklist IP 1) Node migration 2) Flow migration and blocking 3) Shut down node 4) Access control
- 65. 36/52 Attacker I Attacker action:a (A) t ∈ {0, 1, 2, 3}|V| I 0 means do nothing. 1 − 3 correspond to attacks (see fig) I An attacker strategy is a function πA ∈ ΠA : HA → ∆(AA), where HA is the space of all possible attacker histories h (A) t = (s (A) 1 , a (A) 1 , o1, . . . , a (A) t−1, s (A) t , ot) ∈ HA I Objective: (i) disrupt workflows; and (ii) compromise nodes: − J . . . Attacker login attempts configure Automated system Server 2) Brute-force 1) Reconnaissance 3) Code execution Attacker Server TCP SYN TCP SYN ACK port open Attacker Service Server malicious request inject code execution
- 66. 36/52 Attacker I Attacker action:a (A) t ∈ {0, 1, 2, 3}|V| I 0 means do nothing. 1 − 3 correspond to attacks (see fig) I An attacker strategy is a function πA ∈ ΠA : HA → ∆(AA), where HA is the space of all possible attacker histories h (A) t = (s (A) 1 , a (A) 1 , o1, . . . , a (A) t−1, s (A) t , ot) ∈ HA I Objective: (i) disrupt workflows; and (ii) compromise nodes: − J . . . Attacker login attempts configure Automated system Server 2) Brute-force 1) Reconnaissance 3) Code execution Attacker Server TCP SYN TCP SYN ACK port open Attacker Service Server malicious request inject code execution
- 67. 36/52 Attacker I Attacker action:a (A) t ∈ {0, 1, 2, 3}|V| I 0 means do nothing. 1 − 3 correspond to attacks (see fig) I An attacker strategy is a function πA ∈ ΠA : HA → ∆(AA), where HA is the space of all possible attacker histories h (A) t = (s (A) 1 , a (A) 1 , o1, . . . , a (A) t−1, s (A) t , ot) ∈ HA I Objective: (i) disrupt workflows; and (ii) compromise nodes: − J . . . Attacker login attempts configure Automated system Server 2) Brute-force 1) Reconnaissance 3) Code execution Attacker Server TCP SYN TCP SYN ACK port open Attacker Service Server malicious request inject code execution
- 68. 37/52 The Intrusion ResponseProblem maximize πD∈ΠD minimize πA∈ΠA E(πD,πA) [J] subject to s (D) t+1 ∼ fD · | A (D) t , A (D) t ∀t s (A) t+1 ∼ fA · | S (A) t , At ∀t ot+1 ∼ Z · | S (D) t+1, A (A) t ) ∀t a (A) t ∼ πA · | H (A) t , a (A) t ∈ AA(st) ∀t a (D) t ∼ πD · | H (D) t , a (D) t ∈ AD ∀t E(πD,πA) denotes the expectation of the random vectors (St, Ot, At)t∈{1,...,T} when following the strategy profile (πD, πA) (1) can be formulated as a zero-sum Partially Observed Stochastic Game with Public Observations (a PO-POSG): Γ = hN, (Si )i∈N , (Ai )i∈N , (fi )i∈N , u, γ, (b (i) 1 )i∈N , O, Zi
- 69. 38/52 Existence of aSolution Theorem Given the po-posg Γ, the following holds: (A) Γ has a mixed Nash equilibrium and a value function V ? : BD × BA → R. (B) For each strategy pair (πA, πD) ∈ ΠA × ΠD, the best response sets BD(πA) and BA(πD) are non-empty.
- 70. 39/52 The Curse ofDimensionality I While Γ has a value, computing it is intractable. The state, action, and observation spaces of the game grow exponentially with |V|. 1 2 3 4 5 104 105 2 105 |S| |O| |Ai | |V| Growth of |S|, |O|, and |Ai | in function of the number of nodes |V|
- 71. 39/52 The Curse ofDimensionality I While (1) has a solution (i.e the game Γ has a value (Thm 1)), computing it is intractable since the state, action, and observation spaces of the game grow exponentially with |V|. 1 2 3 4 5 104 105 2 105 |S| |O| |Ai | |V| Growth of |S|, |O|, and |Ai | in function of the number of nodes |V| We tackle the scability challenge with decomposition
- 72. 40/52 Intuitively.. rd zone App serversHoneynet dmz admin zone workflow Gateway idps quarantine zone alerts Defender . . . Attacker Clients 2 1 3 12 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 65 The optimal action here... Does not directly depend on the state or action of a node down here
- 73. 40/52 Intuitively.. rd zone App serversHoneynet dmz admin zone workflow Gateway idps quarantine zone alerts Defender . . . Attacker Clients 2 1 3 12 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 65 The optimal action here... But they are not completely independent either. How can we exploit this structure? Does not directly depend on the state or action of a node down here
- 74. 41/52 Our Approach: SystemDecomposition To avoid explicitly enumerating the very large state, observation, and action spaces of Γ, we exploit three structural properties. 1. Additive structure across workflows. I The game decomposes into additive subgames on the workflow-level 2. Optimal substructure within a workflow. I The subgame for each workflow decomposes into subgames on the node-level with optimal substructure 3. Threshold properties of local defender strategies. I Optimal node-level strategies exhibit threshold structures
- 75. 42/52 Additive Structure AcrossWorkflows (Intuition) “=” I If there is no path between i and j in G, then i and j are independent in the following sense: I Compromising i has no affect on the state of j. I Compromising i does not make it harder or easier to compromise j. I Compromising i does not affect the service provided by j. I Defending i does not affect the state of j. I Defending i does not affect the service provided by j.
- 76. 43/52 Additive Structure AcrossWorkflows Definition (Transition independence) A set of nodes Q are transition independent iff the transition probabilities factorize as f (St+1 | St, At) = Y i∈Q f (St+1,i | St,i , At,i ) Definition (Utility independence) A set of nodes Q are utility independent iff there exists functions u1, . . . , u|Q| such that the utility function u decomposes as u(St, At) = f (u1(St,1, At,1), . . . , u1(St,|Q|, At,Q)) and ui ≤ u0 i ⇐⇒ f (u1, . . . , ui , . . . , u|Q|) ≤ f (u1, . . . , u0 i , . . . , u|Q|)
- 77. 44/52 Additive Structure AcrossWorkflows Theorem (Node independencies) (A) All nodes V in the game Γ are transition independent. (B) If there is no path between i and j in the topology graph G, then i and j are utility independent. Corollary (Additive structure across workflows) Γ decomposes into |W| additive subproblems that can be solved independently and in parallel. π (w1) k π (w2) k π (w|W|) k ot,w1 ot,w2 ot,w|W| . . . ⊕ a (k) w1 a (k) w2 a (k) w|W| a (k) t
- 78. 45/52 Optimal Substructure Withina Workflow I Nodes in the same workflow are utility dependent. I =⇒ Adding locally-optimal strategies does not yield an optimal workflow strategy. I However, the locally-optimal strategies satisfy the optimal substructure property. I =⇒ there exists an algorithm for constructing an optimal workflow strategy from locally-optimal strategies for each node. Zone 1 Zone 2 1 2 3 gw w1 w2 V = {1, 2, 3}, E = {(1, 2)}, W = {w1, w2}, Z = {1, 2} IT infrastructure Utility dependencies St, At Ut S (D) t,1 A (D) t,1 Ut,1 S (A) t,1 S (D) t,2 A (D) t,2 Ut,2 S (A) t,2 S (D) t,3 A (D) t,3 Ut,3 S (A) t,3
- 79. 45/52 Optimal Substructure Withina Workflow I Nodes in the same workflow are utility dependent. I =⇒ Adding locally-optimal strategies does not yield an optimal workflow strategy. I However, the locally-optimal strategies satisfy the optimal substructure property. I =⇒ there exists an algorithm for constructing an optimal workflow strategy from locally-optimal strategies for each node. Zone 1 Zone 2 1 2 3 gw w1 w2 V = {1, 2, 3}, E = {(1, 2)}, W = {w1, w2}, Z = {1, 2} IT infrastructure Utility dependencies St, At Ut S (D) t,1 A (D) t,1 Ut,1 S (A) t,1 S (D) t,2 A (D) t,2 Ut,2 S (A) t,2 S (D) t,3 A (D) t,3 Ut,3 S (A) t,3
- 80. 46/52 Scalable Learning throughDecomposition 1 2 3 4 5 6 7 8 9 10 2 4 6 8 10 linear measured # parallel processes n |V| = 10 Speedup S n Speedup of best response computation for the decomposed game; Tn denotes the completion time with n processes; the speedup is calculated as Sn = T1 Tn ; the error bars indicate standard deviations from 3 measurements.
- 81. 47/52 Threshold Properties ofLocal Defender Strategies. Belief space B (j) D Switching curve Υ Continuation set C Stopping set S (1, 0, 0) j healthy (0, 1, 0) j discovered (0, 0, 1) j compromised I A node can be in three attack states s (A) t : Healthy, Discovered, Compromised. I The defender has a belief state b (D) t
- 82. 48/52 Proof Sketch (ThresholdProperties) I Let L(e1, b̂) denote the line segment that starts at the belief state e1 = (1, 0, 0) and ends at b̂, where b̂ is in the sub-simplex that joins e2 and e3. I All beliefs on L(e1, b̂) are totally ordered according to the Monotone Likelihood Ratio (MLR) order. =⇒ a threshold belief state αb̂ ∈ L(e1, b̂) exists where the optimal strategy switches from C to S. I Since the entire belief space can be covered by the union of lines L(e1, b̂), the threshold belief states αb̂1 , αb̂2 , . . . yield a switching curve Υ. Belief space B (j) D (the 2-dimensional unit simplex) sub-simplex B (j) D,e1 joining e2 and e3 b̂5 b̂4 b̂3 b̂2 b̂1 b̂6 b̂7 b̂8 b̂9 L(e1, b̂5) Switching curve Υ Threshold belief state αb̂9 e1 (1, 0, 0) e2 (0, 1, 0) e3 (0, 0, 1)
- 83. 49/52 Decompositional Fictitious Play(DFSP) π̃A ∈ BA(πD) πA πD π̃D ∈ BD(πA) π̃0 A ∈ BA(π0 D) π0 A π0 D π̃0 D ∈ BD(π0 A) . . . π? A ∈ BA(π? D) π? D ∈ BD(π? A) Fictitious play: iterative averaging of best responses. I Learn best response strategies iteratively through the parallel solving of subgames in the decomposition I Average best responses to approximate the equilibrium
- 84. 50/52 Learning Equilibrium Strategies 020 40 60 80 100 running time (h) 0 5 b δ = 0.4 Approximate exploitability b δ 0 20 40 60 80 100 running time (h) 0.0 0.5 1.0 Defender utility per episode dfsp simulation dfsp digital twin upper bound oi,t 0 random defense Learning curves obtained during training of dfsp to find optimal (equilibrium) strategies in the intrusion response game; red and blue curves relate to dfsp; black, orange and green curves relate to baselines.
- 85. 51/52 Comparison with NFSP 010 20 30 40 50 60 70 80 running time (h) 0.0 2.5 5.0 7.5 Approximate exploitability dfsp nfsp Learning curves obtained during training of dfsp and nfsp to find optimal (equilibrium) strategies in the intrusion response game; the red curve relate to dfsp and the purple curve relate to nfsp; all curves show simulation results.
- 86. 52/52 Conclusions I We developa framework to automatically learn security strategies. I We apply the framework to an intrusion response use case. I We derive properties of optimal security strategies. I We evaluate strategies on a digital twin. I Questions → demonstration s1,1 s1,2 s1,3 . . . s1,n s2,1 s2,2 s2,3 . . . s2,n . . . . . . . . . . . . . . . Emulation Target System Model Creation System Identification Strategy Mapping π Selective Replication Strategy Implementation π Simulation Learning