Cyber-infrastructure and meta-cloud testbeds, such as GENI, CloudLab, and Chameleon, are shared facilities that can be configured to provide a diverse and yet controllable environment for testing network protocols and distributed applications. Combined with emulation capabilities, these testbeds provide automated tools for allocating resources, instantiating applications, and collecting measurements. To facilitate reproducibility, they provide support for re-creating the execution environment between experiment runs. A major issue, however, with reproducibility on these systems is the lack of accurate control of time, especially when the experiment faces resource oversubscription. Virtual time management has been proposed for scheduling time dilated virtual machines to increase time fidelity. We hereby propose a unified resource and time scheme on cyber-infrastructure and meta-cloud testbeds to enable large-scale, high-capacity, high-fidelity, reproducible distributed emulation.

1. VIRTUAL TIME MACHINE FOR
LARGE-SCALE REPRODUCIBLE
DISTRIBUTED EMULATION
Jason Liu
Florida International University
GEFI Workshop, Oct 26-27, 2017, Rio de Janeiro, Brazil

2. ABOUT REPRODUCIBILITY
• Simulation observes strict event timestamp ordering
time
event eventevent event event
eventevent event
E2E1 E7E3 E8E5E4 E6

3. time
event eventevent event event
eventevent event
timeLP1
LP2 E1 E8E4 E6
E2 E7E5
E3
ABOUT REPRODUCIBILITY
• Simulation observes strict event timestamp ordering

4. ABOUT REPRODUCIBILITY
• Simulation observes strict event timestamp ordering
• Pseudo-randomness (same seed, same sequence)
• Explicit tie-breaking rules for simultaneous events
• Totally repeatable
GTNeTS
SSFNet

5. ABOUT REPRODUCIBILITY
• Emulation and physical testbeds provide “best effort” timing fidelity
• Inherent nondeterministic behavior
• Interference from competing applications and background services
• Interference from cross traffic
• Assuming reproducible if with “same“ hardware/software provisioning
• Nondeterminism amplified by resource contention

6. H3
H2H5
H6
H1
H4
R1
R2
R3
R4
R5 R6
R7 R8
R9 R10
R11
R12
R13
R14
R15
R16
6
OpenVSwitch Linux namespace
Case in Point: Mininet
Bob Lantz, Brandon Heller, and Nick McKeown. 2010. A network in a laptop: rapid prototyping for
software-defined networks. In Proceedings of the 9th ACM Workshop on Hot Topics in Networks. 19:1–19:6.

7. H3
H2H5
H6
H1
H4
R1
R2
R3
R4
R5 R6
R7 R8
R9 R10
R11
R12
R13
R14
R15
R16
7
OpenVSwitch Linux namespace
Maxinet is a Distributed Mininet
P. Wette, M. Draxler, A. Schwabe, F. Wallaschek, M. Zahraee, and H. Karl, “Maxinet: Distributed emulation of
software-defined networks,” in Proceedings of the 2014 IFIP Networking Conference, 2014, pp. 1–9.

8. MININET/MAXINET
• Virtualization:
• Linux containers (LXC, namespace)
• OVS, veth, GRE tunnels
• CPU limit: cgroup
• Bandwidth and latency: netem, tc
• Problems in experiment capacity and fidelity
• Mininet has processing and memory limitations
• For small experiments only (both in network size and activities)
• Fidelity severely deteriorated by resource underprovisioning
• Maxinet allows large experiments, but limited by cross-machine latency and throughput
• No timing fidelity ☛ no reproducibility

9. STUDY SHREW DDOS ATTACK ON MININET
0
0.2
0.4
0.6
0.8
1
sequential/normal
sequential/dos
distributed/normal
distributed/dos
NormalizedThroughput
RTT=40ms
RTT=120ms
RTT=200ms
RTT=280ms
RTT=360ms
9
Victim
Attacker
Common
Bottleneck
Link

10. RESOURCE UNDERPROVISIONING: VIRTUAL TIME
• Time dilation [1]:
• Slow down clock time in VMs w.r.t. real time
• Time dilation factor (TDF) = real time / virtual time
• Equivalent to be an update in system resources
• Observable time:
• gettimeofday(): return t = t0 + (r - r0) * TDF
• sleep(t): hrtimer_nanosleep(t * TDF)
• Implementation through kernel modification [2]
• Change Linux task_struct: add TDF, t0, r0, …
• Add/replace system calls, such as clone(), gettimeofday(), sleep()
Real Time
VirtualTime
r0 r = now()
t0
t
[1] D. Gupta, K. Yocum, M. McNett, A. C. Snoeren, A. Vahdat, and G. M. Voelker, “To infinity and beyond:
time warped network emulation,” SOSP 2005.
[2] J. Lamps, D. M. Nicol, and M. Caesar, “Timekeeper: a lightweight virtual time system for linux,”
SIGSIM-PADS 2014.

11. PRIME Emulation Infrastructure
PRIME Network Simulator
Applications
TCP/IP
Stack
Virtual NICs
VM1
Applications
TCP/IP
Stack
Virtual NICs
VM2
TCP EVALUATION USING TIME DILATION
• SVEET: Scalable Virtualized Evaluation Environment for TCP
• M. A. Erazo, Y. Li, and J. Liu, “SVEET! A scalable virtualized evaluation environment
for TCP,” TRIDENTCOM 2009.
• Static time dilation for all VMs and real-time simulator
11

12. WSC'10 12JASON LIU, FLORIDA INTERNATIONAL UNIVERSITY

13. Throughput under random loss over 100 Mb/s bottleneck link
13
Throughput under random loss over 10 Mb/s bottleneck link

14. TIMING FIDELITY: TRADITIONAL EMULATION
Real Time
VirtualTime
VM1
VM2
VM3
VM4
• No time dilation: virtual time = real time
• Timing depends on CPU scheduling
• CPU contention deteriorates timing fidelity

15. TIMING FIDELITY: FIXED TIME DILATION
Real Time
VirtualTime
VM1
VM2
VM3
VM4
• Upgraded resources: more throughput, less latency
• TDF derived from bottleneck constraint
• Timing still depends on CPU scheduling

16. TIMING FIDELITY: VIRTUAL TIME SCHEDULING
Real Time
VirtualTime
• Dynamic time dilation for each VM:
• TDF derived from resource (e.g., processing speed)
• TDF from degree of multiplexing (e.g., #VMs)
• TDF from utilization (e.g., packet send rate)
• CPU scheduling: schedule VMs according to virtual time
• Ability to start and stop any VM
• Virtual-time aware netem, tc, GRE tunnel
• Better VM control ☛ improved timing fidelity
• Improved timing fidelity ☛ better reproducibility
VM1
VM2
VM3
VM4

17. VIRTUAL TIME MACHINE
• Local controller
• Coordinate virtual time by communicating/schedulingVMs and SIMs
• Monitor resource utilization, manage network traffic between VMs
• Global controller
• VM migration and model repartitioning
• Manage overall virtual time advancement
• Manage network traffic between machines
17