Computer networks are undergoing a phenomenal growth, driven by the rapidly increasing number of nodes constituting the networks. At the same time, the number of security threats on Internet and intranet networks is constantly increasing, and the testing and experimentation of cyber defense solutions require the availability of separate, test environments that best reflect the complexity of a real system. Such environments support the deployment and monitoring of complex mission-driven network scenarios, and cyber security training activities, thus enabling enterprises to study cyber defense strategies and allowing security researchers to evaluate their algorithms at scale. The main objective is delivering to researchers and practitioners an overview of the technological means and the practical steps to setup a private cloud platform based on OpenNebula for the creation and management of virtual environments that support cyber-security activities of training and testing, as well as an overview of its possible applications in the cyber security domain. In particular: 1. We describe our infrastructure based on OpenNebula 2. We overview our application, sitting on top of OpenNebula, as well as the technological tools involved in the management of its lifecycle (e.g., Ansible)
. 3. We show how the platform can support various examples of security research activities
 [References] Building an emulation environment for cyber security analyses of complex networked systems, Tanasache, Florin Dragos and Sorella, Mara and Bonomi, Silvia and Rapone, Raniero and Meacci, Davide, ICDCN '19, ACM, 2019

1. Building Virtual Environments for Security Analyses
of Complex Networked Systems
Mara Sorella, Ph.D.
Research center on Cyber Intelligence and Information Security (CIS)
Department of Computer, Control and Management Engineering
Sapienza University of Rome

2. Starting from the past decade, cyber attacks have become increasingly sophisticated,
stealthy, targeted and multi-faceted, featuring zero-day exploits and highly creative
interdisciplinary attack methods.
Introduction

3. Starting from the past decade, cyber attacks have become increasingly sophisticated,
stealthy, targeted and multi-faceted, featuring zero-day exploits and highly creative
interdisciplinary attack methods.
A common strategy is trying to play the role of the attacker and stress the network that is
aimed to protect. Another key aspect is personnel training.
Introduction

4. Starting from the past decade, cyber attacks have become increasingly sophisticated,
stealthy, targeted and multi-faceted, featuring zero-day exploits and highly creative
interdisciplinary attack methods.
A common strategy is trying to play the role of the attacker and stress the network that is
aimed to protect. Another key aspect is personnel training.
Need to have a separate, dedicated environment that should be able to:
▪ represent realistic scenarios that fit the security testing objectives
▪ support the definition of new scenarios and cyber threats in a cost and time-effective
manner
Introduction

5. Starting from the past decade, cyber attacks have become increasingly sophisticated,
stealthy, targeted and multi-faceted, featuring zero-day exploits and highly creative
interdisciplinary attack methods.
A common strategy is trying to play the role of the attacker and stress the network that is
aimed to protect. Another key aspect is personnel training.
Need to have a separate, dedicated environment that should be able to:
▪ represent realistic scenarios that fit the security testing objectives
▪ support the definition of new scenarios and cyber threats in a cost and time-effective
manner
Introduction
This is typically achieved by instrumenting virtual environments, referred as cyber ranges

6. Starting from the past decade, cyber attacks have become increasingly sophisticated,
stealthy, targeted and multi-faceted, featuring zero-day exploits and highly creative
interdisciplinary attack methods.
A common strategy is trying to play the role of the attacker and stress the network that is
aimed to protect. Another key aspect is personnel training.
Need to have a separate, dedicated environment that should be able to:
▪ represent realistic scenarios that fit the security testing objectives
▪ support the definition of new scenarios and cyber threats in a cost and time-effective
manner
Introduction
This is typically achieved by instrumenting virtual environments, referred as cyber ranges

7. Our Project: Motivation

8. Our Project: Motivation
▪ Research focus: threat modeling, network hardening algorithms
▪ Goal: test and evaluate our research products in realistic scenarios

9. Our Project: Motivation
▪ Research focus: threat modeling, network hardening algorithms
▪ Goal: test and evaluate our research products in realistic scenarios
▪ Issues
very few existing datasets available
limited information available
typically small scale networks (<10 nodes)

10. Our Project: Motivation
▪ Research focus: threat modeling, network hardening algorithms
▪ Goal: test and evaluate our research products in realistic scenarios
▪ Solution
A combination of techniques of network and security assessment, and
cloud technologies to enable the deployment of fully virtualized instances of computer
networks with high degree of affinity to actual reference scenarios
▪ Issues
very few existing datasets available
limited information available
typically small scale networks (<10 nodes)

11. Solution overview

12. Solution overview

13. Solution overview
Testbed
Specification

14. Solution overview
Testbed
Specification

15. Solution overview
Testbed
Specification

16. Solution overview
Testbed
Specification

17. Solution overview
Testbed
Specification

18. Solution overview
Testbed
Specification

19. Solution overview
Testbed
Specification

20. Virtual Environment Infrastructure
Design choices

21. Major open source solutions: OpenNebula vs OpenStack
Private cloud management, Infrastructure as a Service platforms
Virtual Environment Infrastructure: IaaS

22. Major open source solutions: OpenNebula vs OpenStack
Private cloud management, Infrastructure as a Service platforms
vendor
stacks
Virtual Environment Infrastructure: IaaS

23. Major open source solutions: OpenNebula vs OpenStack
Private cloud management, Infrastructure as a Service platforms
- Complex, multitiered, vendor-driven
- Many subprojects, each with different maturity levels
vendor
stacks
Virtual Environment Infrastructure: IaaS

24. Major open source solutions: OpenNebula vs OpenStack
Private cloud management, Infrastructure as a Service platforms
- Complex, multitiered, vendor-driven
- Many subprojects, each with different maturity levels
- Ease of setup and use
- free, yet production ready
vendor
stacks
Virtual Environment Infrastructure: IaaS

25. Storage Layer
Maintaining VM OS Images (“templates”) repository: distributed/replicated filesystem

26. • Replicated mode: exact copies of the data are maintained on the bricks
• Fosters data locality at VM instantiation time
Storage Layer
Maintaining VM OS Images (“templates”) repository: distributed/replicated filesystem

27. • Replicated mode: exact copies of the data are maintained on the bricks
• Fosters data locality at VM instantiation time
/Images — GlusterFS mount point, OS images
/System — instantiated machines disks
/Files & Kernels — plain text files such as scripts
OpenNebula
Datastores
Storage Layer
Maintaining VM OS Images (“templates”) repository: distributed/replicated filesystem

28. Inter- and intra- LAN comms, across different physical nodes
Virtual switches: OpenVirtualSwitch, Linux Ethernet Bridge
• Keeps a MAC database:
tap0 — eth0
Network Layer
Inter/intra Virtual LAN communications across physical nodes
OVS
Software implementation of
a virtual multilayer network
switch

29. Inter- and intra- LAN comms, across different physical nodesNetwork Layer
OpenVirtualSwitch: software implementation of a virtual multilayer network switch
also enables efficient data
collection at the bridge level
SPAN (Switched Port Analyzer)

30. Virtual Infrastructure: Overview

31. Virtual Infrastructure: Overview

32. server 1 server 2 … server n
Virtual Infrastructure: Overview

33. server 1 server 2 … server n
oned
(master)
Virtual Infrastructure: Overview

34. server 1 server 2 … server n
opennebula-kvm opennebula-kvmoned
(master)
Virtual Infrastructure: Overview

35. server 1 server 2 … server n
opennebula-kvm opennebula-kvmoned
(master)
Virtual Infrastructure: Overview

36. switch (backbone)
server 1 server 2 … server n
opennebula-kvm opennebula-kvmoned
(master)
Virtual Infrastructure: Overview

37. switch (backbone)
server 1 server 2 … server n
br1
br2
br3
opennebula-kvm opennebula-kvmoned
(master)
Virtual Infrastructure: Overview

38. switch (backbone)
server 1 server 2 … server n
br1
br2
br3
switch (service)
opennebula-kvm opennebula-kvmoned
(master)
Virtual Infrastructure: Overview

39. firewall
switch (backbone)
server 1 server 2 … server n
br1
br2
br3
switch (service)
opennebula-kvm opennebula-kvmoned
(master)
Virtual Infrastructure: Overview

40. firewall
switch (backbone)
server 1 server 2 … server n
br1
br2
br3
switch (service)
opennebula-kvm opennebula-kvmoned
(master)
Virtual Infrastructure: Overview

41. firewall
switch (backbone)
server 1 server 2 … server n
br1
br2
br3
switch (service)
opennebula-kvm opennebula-kvmoned
(master)
Virtual Infrastructure: Overview

42. firewall
switch (backbone)
server 1 server 2 … server n
br1
br2
br3
switch (service)
EMULATION ENVIRONMENT INFRASTRUCTURE
opennebula-kvm opennebula-kvmoned
(master)
Virtual Infrastructure: Overview

43. firewall
switch (backbone)
server 1 server 2 … server n
br1
br2
br3
switch (service)
VIRTUAL TESTBED EMULATION ENVIRONMENT INFRASTRUCTURE
opennebula-kvm opennebula-kvmoned
(master)
Virtual Infrastructure: Overview

44. Testbed Design and Deployment

45. - Cyber range Laboratory
- Deploys a testbed starting from a YAML file (“infrastructure as a code”)
Automatic Testbed Deployment: Cylab

46. - Cyber range Laboratory
- Deploys a testbed starting from a YAML file (“infrastructure as a code”)
Automatic Testbed Deployment: Cylab
No opennebula provider

47. - Cyber range Laboratory
- Deploys a testbed starting from a YAML file (“infrastructure as a code”)
Automatic Testbed Deployment: Cylab
No opennebula provider

48. 1. VLANs
A text-only configuration file (YAML representation)
A Testbed “spec”
A text-only configuration file (YAML representation)
Testbed
Specification

49. 1. VLANs
2. VMs
A text-only configuration file (YAML representation)
A Testbed “spec”
A text-only configuration file (YAML representation)
Testbed
Specification

50. 1. VLANs
2. VMs
A text-only configuration file (YAML representation)
A Testbed “spec”
A text-only configuration file (YAML representation)
+custom init script support
(CONTEXT / START_SCRIPT)
Testbed
Specification

51. 1. VLANs
2. VMs
3. Virtual
Routers
A text-only configuration file (YAML representation)
A Testbed “spec”
A text-only configuration file (YAML representation)
+custom init script support
(CONTEXT / START_SCRIPT)
Testbed
Specification

52. 1. VLANs
2. VMs
3. Virtual
Routers
4. Firewalls
A text-only configuration file (YAML representation)
A Testbed “spec”
A text-only configuration file (YAML representation)
+custom init script support
(CONTEXT / START_SCRIPT)
Testbed
Specification

53. Cylab:Architecture overview

54. Cylab:Architecture overview

55. Cylab:Architecture overview

56. Cylab:Architecture overview
service
installation

57. Applications

58. The infrastructure can support various activitiesApplications: Overview

59. 1. Cyber-range deployment for security training and testing
• cyber security scenario awareness
• incident management (detection, investigation, response)
The infrastructure can support various activitiesApplications: Overview

60. [ICDCN ‘19] Tanasache, Sorella, Bonomi, Rapone, Meacci. Building an emulation environment for cyber security analyses of complex networked systems
1. Cyber-range deployment for security training and testing
• cyber security scenario awareness
• incident management (detection, investigation, response)
2. Dataset generation
The infrastructure can support various activities
case study [ICDCN ‘19]
Applications: Overview

61. [ICDCN ‘19] Tanasache, Sorella, Bonomi, Rapone, Meacci. Building an emulation environment for cyber security analyses of complex networked systems
1. Cyber-range deployment for security training and testing
• cyber security scenario awareness
• incident management (detection, investigation, response)
2. Dataset generation
3. Threat modeling & risk management
The infrastructure can support various activities
case study [ICDCN ‘19]
Applications: Overview

62. [ICDCN ‘19] Tanasache, Sorella, Bonomi, Rapone, Meacci. Building an emulation environment for cyber security analyses of complex networked systems
1. Cyber-range deployment for security training and testing
• cyber security scenario awareness
• incident management (detection, investigation, response)
2. Dataset generation
3. Threat modeling & risk management
• dynamic attack graph generation
The infrastructure can support various activities
case study [ICDCN ‘19]
Applications: Overview

63. [ICDCN ‘19] Tanasache, Sorella, Bonomi, Rapone, Meacci. Building an emulation environment for cyber security analyses of complex networked systems
1. Cyber-range deployment for security training and testing
• cyber security scenario awareness
• incident management (detection, investigation, response)
2. Dataset generation
3. Threat modeling & risk management
• dynamic attack graph generation
The infrastructure can support various activities
case study [ICDCN ‘19]
Applications: Overview

64. [ICDCN ‘19] Tanasache, Sorella, Bonomi, Rapone, Meacci. Building an emulation environment for cyber security analyses of complex networked systems
1. Cyber-range deployment for security training and testing
• cyber security scenario awareness
• incident management (detection, investigation, response)
2. Dataset generation
3. Threat modeling & risk management
• dynamic attack graph generation
• network hardening
• automatic attack path instantiation
The infrastructure can support various activities
case study [ICDCN ‘19]
Applications: Overview

65. [ICDCN ‘19] Tanasache, Sorella, Bonomi, Rapone, Meacci. Building an emulation environment for cyber security analyses of complex networked systems
1. Cyber-range deployment for security training and testing
• cyber security scenario awareness
• incident management (detection, investigation, response)
2. Dataset generation
3. Threat modeling & risk management
• dynamic attack graph generation
• network hardening
• automatic attack path instantiation
The infrastructure can support various activities
case study [ICDCN ‘19]
Applications: Overview

66. Applications
Dataset Generation

67. Software agents deployed on the hosts, capturing
different behavioral patterns
Dataset Generation: benign traffic agents
Protocols
▪ HTTP/HTTPS
▪ SSH
▪ SMB
▪ SFTP

68. Software agents deployed on the hosts, capturing
different behavioral patterns
Dataset Generation: benign traffic agents
Protocols
▪ HTTP/HTTPS
▪ SSH
▪ SMB
▪ SFTP

69. Malicious activities performed in the testbed, covering a diverse set of attack
scenarios.
Web attack - Drupal
Ransomware Attack (WannaCry)
We collected a publicly released dataset containing complete network traces, enriched with labeled
features
Dataset Generation: cyber attacks

70. LAN1 LAN2
br1 br1
LAN3
br2 br2
LAN1
Data collection: network traffic

71. LAN1 LAN2
br1 br1
LAN3
br2 br2
LAN1
Data collection: network traffic

72. LAN1 LAN2
br1 br1
LAN3
br2 br2
LAN1
Data collection: network traffic
For each network to be monitored, OVS port mirroring (SPAN) allows to mirror the traffic from all VM
network interfaces toward a specific output port (1 x br x node)

73. LAN1 LAN2
br1 br1
LAN3
br2 br2
LAN1
Data collection: network traffic
For each network to be monitored, OVS port mirroring (SPAN) allows to mirror the traffic from all VM
network interfaces toward a specific output port (1 x br x node)

74. Information to be gathered from the virtual testbed include:
• routing tables
• system logs
• firewall rules
• ACLs from network devices
• installed applications (+CVE)
• running services
• open ports
This info is using an out-of-band “management” interface for each machine
Data collection: metadata

75. Toward a flexible and fully automated testbed
▪ Service + host behavior on-demand installation
Ansible server + Catalog server
Ongoing work

76. Toward a flexible and fully automated testbed
▪ Service + host behavior on-demand installation
Ansible server + Catalog server
▪ Terraform Integration (opennebula provider)
Ongoing work

77. Toward a flexible and fully automated testbed
▪ Service + host behavior on-demand installation
Ansible server + Catalog server
▪ Terraform Integration (opennebula provider)
Ongoing work
fork
fork

78. Toward a flexible and fully automated testbed
▪ Service + host behavior on-demand installation
Ansible server + Catalog server
▪ Terraform Integration (opennebula provider)
Ongoing work
fork
fork
oneuser
oneacl
onehost
onecluster
API support still lacking:
…