A real-time network simulation infrastructure based on OpenVPN


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

A real-time network simulation infrastructure based on OpenVPN

  1. 1. ARTICLE IN PRESS The Journal of Systems and Software xxx (2008) xxx–xxx Contents lists available at ScienceDirect The Journal of Systems and Software journal homepage: www.elsevier.com/locate/jssA real-time network simulation infrastructure based on OpenVPN qJason Liu *, Yue Li, Nathanael Van Vorst, Scott Mann, Keith HellmanSchool of Computing and Information Sciences, Florida International University, Miami, 11200 SW 8th Street, ECS 261B, FL 33199, USADepartment of Mathematical and Computer Sciences, Colorado School of Mines, Golden, CO 80401, USAa r t i c l e i n f o a b s t r a c tArticle history: We present an open and flexible software infrastructure that embeds physical hosts in a simulated net-Received 6 February 2008 work. In real-time network simulation, where real-world implementations of distributed applicationsReceived in revised form 4 August 2008 and network services can run together with the network simulator that operates in real-time, real net-Accepted 4 August 2008 work packets are injected into the simulation system and subject to the simulated network conditionsAvailable online xxxx computed as a result of both real and virtual traffic traversing the network and competing for network resources. Our real-time simulation infrastructure has been implemented based on Open Virtual PrivateKeywords: Network (OpenVPN), modified and customized to bridges traffic between the physical hosts and the sim-Network simulationNetwork emulation ulated network. We identify the performance advantages and limitations of our approach via a set ofReal-time simulation experiments. We also present two interesting application scenarios to show the capabilities of theVirtual private network real-time simulation infrastructure. Published by Elsevier Inc.1. Introduction Peterson et al., 2002), through analytical models (Liu et al., 2004), to simulation and emulation (Breslau et al., 2000; Vahdat Over the years, we have seen a major shift in the global network et al., 2002; Guruprasad et al., 2005). In addition, advanced paralleltraffic in response to emerging ‘‘killer apps” of the Internet. For network simulation tools, such as SSFNet (Cowie et al., 1999),example, subject to interpretation, one recent study shows that GTNetS (Riley, 2003), and ROSSNet (Yaun et al., 2003), aim atpeer-to-peer (P2P) applications currently contribute over 60% of extending the simulator’s capability to execute extremely largethe total Internet traffic (CacheLogic, 2005). During the same network models and has helped improve our understanding ofperiod, we have also seen a growth of other applications, such as the global-scale network phenomena.voice-over-IP (VoIP), Internet teleconferencing, and multimedia As the scale of the network we are able to simulate increases,data streaming, all having the potential to become the next however, one cannot underestimate the importance of simulationdominating traffic over the global network. The landscape of com- validation, which seems particularly elusive for large-scale net-puter networks changes rapidly as a result of the interplay of many work models. There have been efforts (e.g., Fall, 1999; Simmondssocial and technological factors of highly unpredictable nature. et al., 2000; Song et al., 2000) in augmenting the network simula-This poses a considerable challenge to researchers designing the tion with emulation capabilities, which we call real-time networknext-generation high-performance networking and software infra- simulation (Liu, 2008). With the emulation extension, a networkstructures to satisfy the growing demand of these distributed simulator operating in real-time can run together with real-worldapplications. The success of advancing critical network technolo- distributed applications and network services. These applicationsgies, to a large extent, depends on the available tools that can effec- and services generate real packets, which are injected into the sim-tively prototype, analyze, and evaluate new designs and new ideas. ulation system. Packet delays and losses are calculated as a func-Traditionally, networking research has relied on a variety of tools, tion of the simulated network condition. To a certain extent,ranging from physical testbeds (Barford and Landweber, 2003; real-time network simulation not only alleviates the burden of developing separate models for applications in simulation, but also increases the confidence level of network simulation as real sys- q This article is an extended version of early work published at the 2007 INFOCOM tems are included in the network model.MiniSymposium (Liu et al., 2007). Part of this work was done when Dr. Jason Liu Large-scale real-time network simulation requires simulationwas at the Colorado School of Mines. be able to characterize the behavior of a network—potentially with * Corresponding author. Tel./fax: +1 305 348 1625. millions of network entities and with realistic traffic load. In addi- E-mail addresses: liux@cis.fiu.edu (J. Liu), yueli@cis.fiu.edu (Y. Li), vanvorst@ieee.org (N.V. Vorst), sunix13@yahoo.com (S. Mann), khellman@mines.edu tion, the simulation must be scalable and able to keep up with the(K. Hellman). wall-clock time. To this end, parallel discrete-event simulation URL: http://www.cis.fiu.edu/~liux/ (J. Liu). techniques can be applied to increase the simulation event0164-1212/$ - see front matter Published by Elsevier Inc.doi:10.1016/j.jss.2008.08.015 Please cite this article in press as: Liu, J. et al., A real-time network simulation infrastructure based on OpenVPN, J. Syst. Software (2008), doi:10.1016/j.jss.2008.08.015
  2. 2. ARTICLE IN PRESS2 J. Liu et al. / The Journal of Systems and Software xxx (2008) xxx–xxxprocessing capabilities and to reduce the possibility of missed rithms can significantly improve the timeliness of the interac-deadlines (Simmonds and Unger, 2003). The multi-scale modeling tive network simulator.techniques have also been investigated that can selectively include Dynamic configuration and continuous network operation. Thecoarse-grain traffic models to effectively reduce the time complex- virtual network testbed is expected to accommodate multipleity of a large-scale network simulation (Nicol and Yan, 2004; Zhou experiments running either simultaneously or in succession withet al., 2004; Liu, 2006). potentially different network configurations without disruption. Scalability also implies high performance of the emulation Scalable interaction and effortless integration with real applica-interface for real applications to interact with the simulator. This tions. We need to provide a real-time simulation infrastructureis an area where we have not seen significant progress being made. for the network simulator run on supercomputers to dynami-Consider a scenario, shown in Fig. 1, where collaborative and geo- cally interface with a large number of real applications.graphically distributed research teams conduct a remote scientificexperiment over the network. The experiment involves physical This article focuses on the last issue. In particular, we propose aexperimental instruments, supercomputing facilities, and large- simulation infrastructure using a Virtual Private Network (VPN)scale data storage centers. A high-performance software infrastruc- server farm as the simulation gateway for distributed applicationsture is in place to seamlessly provide coordination between the to dynamically connect to the real-time network simulator. Realexperimental facilities and the distributed research teams. From distributed applications run on computers configured as VPN cli-the perspective of network design, our goal is to use a large-scale ents, which automatically forward network packets generated byvirtual network testbed (presumably run at a supercomputing cen- the applications and targeted the virtual network to the VPN serv-ter) to connect these experimental sites in order to evaluate both ers. A user-level daemon process run at each simulation gatewaysoftware and networking support for this large-scale distributed manages emulation connections to the simulator and redirectsscience application. For networking research, the experimental the traffic from the VPN servers to the corresponding simulationdata—including, for example, intermediate results that need to be processes via TCP connections. At the simulation site, the networktransfered and stored, data streams for remote visualization, and packets are translated into simulation events and then injectedcontrol flows for remote experiment steering—provides a realistic into the simulator. On the reverse path, packets arriving at a virtualtraffic load, invaluable for testing new network protocols and node in the virtual network are translated into real network pack-network services. In this scenario, multiple entry points to the net- ets and sent to the simulation gateway via TCP. The user-level dae-work simulator must be provided to facilitate real-time interaction mon process receives the packets and then forwards the packets toinitiated from different geographical locations and with different the VPN server, which sends the packets to the client machine thattraffic demand. corresponds to the virtual node in simulation and runs real This article presents a real-time simulation infrastructure that applications.facilitates interaction between the real applications and the net- Using VPN as the simulation gateway between real applicationswork simulation. This effort is part of the PRIME (Parallel Real-time and the simulator brings several benefits:Immersive network Modeling Environment) project, designed toprovide a self-sustained large-scale virtual network environment (1) The simulation gateway is placed outside the firewall of thefor researchers of distributed systems and networks (Liu, 2008). supercomputing center where we run the large-scale net-PRIME is derived from an interactive network simulator called work simulation. The gateway is necessary because directRINSE (Liljenstam et al., 2005) and aims at addressing several emulation connections to the simulator are typically blockedimportant issues, including: by the firewall. The gateway provides a natural rendezvous point between the real distributed applications and the Simulation scalability and real-time performance guarantee. We simulator. envision that using parallel simulation and multi-scale modeling (2) On a client machine, each VPN connection is implemented techniques and combining with novel real-time scheduling algo- through a network tunnel device assigned with a proper IP Fig. 1. A PRIME application. Please cite this article in press as: Liu, J. et al., A real-time network simulation infrastructure based on OpenVPN, J. Syst. Software (2008), doi:10.1016/j.jss.2008.08.015
  3. 3. ARTICLE IN PRESS J. Liu et al. / The Journal of Systems and Software xxx (2008) xxx–xxx 3 address. We use an automatically generated VPN configura- lated network is ad hoc and potentially transient. More important, tion file to customize the IP routing table to allow network aside from conducting traffic generated by the emulated applica- traffic targeting the virtual IP address space to be forwarded tions, our network simulator can simulate additional traffic in via the VPN connection. Since in most cases the tunnel the background providing a more realistic and yet controllable net- device is treated as a regular network interface, this process work condition. is therefore transparent to the application we intend to Bradford et al. gave a survey of techniques for importing and emulate. exporting real network packets, such as using raw sockets and (3) The VPN server is capable of handling a large number of VPN the pcap library, that allow real-time interactions with the net- connections. Allowing dynamic behavior is important work simulator (Bradford et al., 2001). In MaSSF (Liu and Chien, because we expect the virtual network testbed to run for a 2006), applications can also run unmodified together with the sim- long period of time. Individual emulation connections may ulator. This is accomplished by intercepting communication-re- comparatively have a shorter life span. lated and time-sensitive systems calls, either through the use of (4) Several VPN servers can be used to balance the traffic going linker wrapper functions or with the replacement of dynamic link- in and coming out of the real-time simulator. A client ing libraries. To enable interaction with the emulation system, machine emulating a real application can choose to connect these techniques invariably require the applications to receive to a VPN server according to its geographical location and/or some special treatment, either at compile/link time, which is prob- the quality-of-service demand. One can also adopt a load lematic if one does not have access to the source code, or at run balancing strategy to select a simulation gateway in a VPN time, which tends to be difficult to implement and maintain. Fur- server farm. thermore, these techniques do not address issues, such as scalabil- (5) VPN provides support for enterprise-scale configurations, ity and load balancing, which we regard as important for a large- such as fine-grain access controls, authentication, and scale emulation system. encryption. It also provides fault-tolerance measures, such OpenVPN is an open-source VPN solution implemented based as fail-over. Data compression is also used for packet trans- on IP tunneling, i.e., using TUN/TAP devices to encapsulate IP pack- mission through the VPN connection for better performance. ets inside UDP (Yonan, 2008). Previously we used IP tunneling to We inherit all these features automatically. connect a wireless ad hoc network simulator with the real imple- (6) VPN packages are in general highly portable. For example, mentation of routing protocols to validate wireless models (Liu OpenVPN, the implementation we use for the simulation et al., 2005). NCTUns is a network simulator that uses IP tunneling gateway, is open source and freely available on Windows, to connect with real-life network protocols on the same physical Mac OS X, and most Unix platforms (Yonan, 2008). host (Wang et al., 2007). VINI uses OpenVPN to connect with the end users for network experimentation on PlanetLab (Bavier The remainder of this article is organized as follows. In Section et al., 2006).2, we describe previous work related to our approach. Section 3 Our approach extends the previous effort in the RINSE simulator,presents our real-time simulation infrastructure in detail. We which also allows real-time interactions with real applications pre-implemented a prototype of this system to study the feasibility sumably running on geographically distributed client machinesof such an approach. To demonstrate the performance of this sim- (Liljenstam et al., 2005). RINSE prescribes a blueprint for a large-ulation infrastructure and assess its limit, we conducted prelimin- scale emulation system. In particular, the design of RINSE is the firstary experiments. We report the results in Section 4. We also used we know to designate a data server to connect the client applica-this real-time simulation infrastructure to interact with existing tions with the real-time simulator. By using a database, the servernetwork routing protocols and a real network device designed for can perform authentication and event bookkeeping for client appli-content filtering and cyber-security applications. Section 5 pre- cations accessing the simulated network. The real-time simulationsents our case studies. We conclude this article with a discussion infrastructure we describe in this article presents a viable approachof future work in Section 6. to large-scale network emulation, which not only supports an effi- cient and flexible pathway to the real-time simulator, but also pro- vides a scalable solution for managing client applications.2. Related work Advanced emulation techniques can now provide full-scale 3. The real-time simulation infrastructureemulation capabilities for large-scale networks (Vahdat et al.,2002; Zheng and Ni, 2003; White et al., 2002. For example, Model- The real-time simulation infrastructure consists of three majorNet (Vahdat et al., 2002) multiplexes real network applications at components: the I/O threads collocated with the simulator, thethe peripheral of the system called ‘‘edge nodes”. Multiple applica- simulation gateways, and the client machines running real applica-tions run unmodified on each edge node and, by setting up explicit tions. An example of the real-time simulation infrastructure isroutes, send traffic through the emulation core that runs on paral- illustrated in Fig. 2.lel computers. Network operations are emulated using a time-dri-ven approach to calculate packet delays and losses. At its core, (1) The I/O threads are collocated with the real-time networkModelNet uses an IP firewall (ipfw) rule to intercept incoming simulator run on parallel computers (most likely in a super-packets and a kernel module is then used to simulate packet for- computing center behind a firewall). The I/O threads arewarding in the network. The ModelNet approach is appropriate used by the real-time simulator to import real networkfor a closed emulation system, where edge nodes are configured packets and export simulated packets. The threads are con-statically as an integral part of the system. A careful setup of the nected via TCP connections to the simulation gateways, con-physical testbed with a tight coupling between the edge nodes figured to tunnel network packets to and from theand the core can achieve a very high throughput allowing the emu- applications running on client machines.lation to scale up to tens of thousands of network entities. In com- (2) Each simulation gateway runs a modified OpenVPN serverparison, our real-time simulation infrastructure is open and and a daemon process called ssfgwd, which is responsibleflexible, and allows dynamic connections to the virtual network for shunting IP packets between the virtual network andtestbed—the association between real applications and the simu- the client applications. Please cite this article in press as: Liu, J. et al., A real-time network simulation infrastructure based on OpenVPN, J. Syst. Software (2008), doi:10.1016/j.jss.2008.08.015
  4. 4. ARTICLE IN PRESS4 J. Liu et al. / The Journal of Systems and Software xxx (2008) xxx–xxx virtual network simulation gateway VPN Server VPN Client VPN Client ssfgwd I/O threads VPN Client VPN Server parallel machines VPN Client ssfgwd VPN Client TCP connections simulation gateway Firewall Fig. 2. The real-time simulation infrastructure. (3) The client machines each run one or several OpenVPN cli- session forwards the event to the designated I/O agent in the same ents configured to connect to a designated simulation gate- simulation process (on the same processor), which exports the way. There is a one-to-one mapping between a client packet to the writer thread by translating the simulated packet machine and an emulated virtual node inside the simulation. to a real IP packet. The write thread sends the IP packet to the sim- OpenVPN set up the IP addresses for the client machine ulation gateway via TCP. On the reverse path, when the reader using the same IP addresses of the corresponding virtual thread receives an IP packet, it translates the packet to a simulation node. OpenVPN also set up the IP routing table at the client event and presents it to the I/O agent inside the simulator. The I/O machine automatically for the client applications to forward agent then forwards it to the emulation protocol session on the traffic to the virtual network through the VPN connection. corresponding virtual host, which pushes the packet down the simulated protocol stack. This procedure is illustrated in Fig. 3. In the remainder of this section, we describe the details of thesethree components, as well as a mechanism to emulate routers with 3.2. The simulation gatewaymultiple network interfaces. The simulation gateway is a critical component of the real-time3.1. The real-time network simulator and the I/O threads simulation infrastructure, residing between the network simulator and the client applications run on distributed machines. We set up PRIME extends the SSFNet network simulator for real-time an OpenVPN server at each simulation gateway managing incom-large-scale network simulations. SSFNet was developed based on ing connections from the client machines. There can be multiplea simulation kernel that implements the Scalable Simulation simulation gateways for balancing the traffic load or for differenti-Framework (SSF) for high-performance parallel and distributeddiscrete-event simulation (SSF Research Network, 2008). The sim-ulator has been augmented to allow real-time interactions with virtual hostexternal network applications. simulation gateway A virtual network is partitioned among parallel processors to be TCP UDP VPN Serversimulated in parallel. Each processor participating in the parallel Emulation IP Sessionnetwork simulation spawns two threads in addition to the simula-tion process assigned to this processor for event processing. An I/O MAC MAC ssfgwdagent in the simulation process is responsible for both exporting writerand importing packets. A writer thread takes the packets exported PHY PHY threadfrom the simulation process and forwards them to a simulation I/Ogateway, which then delivers the packets to the corresponding cli- virtual host Agentent machine running real applications. A reader thread accepts TCP UDP readerpackets from the client applications through a simulation gateway thread Emulationand inserts the packets into the simulator. IP Session ssfgwd The I/O agent and the I/O threads use established SSF functionsdesigned specifically to support emulation, including both export- MAC MAC VPN Servering simulation events and importing real network packets (see Simulation PHY PHY Process simulation gatewayLiljenstam et al., 2005 for more details). Each emulated virtualnode in the simulation contains an emulation protocol session thatintercepts packets at the IP layer. Upon receiving a simulated pack-et deemed to be sent out to the client application, the emulation Fig. 3. Connecting the simulator and the simulation gateway. Please cite this article in press as: Liu, J. et al., A real-time network simulation infrastructure based on OpenVPN, J. Syst. Software (2008), doi:10.1016/j.jss.2008.08.015
  5. 5. ARTICLE IN PRESS J. Liu et al. / The Journal of Systems and Software xxx (2008) xxx–xxx 5ated services (for example, each simulate gateway could offer a dif- throughput is anticipated. Alternatively, the clients may dynami-ferent quality of service guarantee for its connected clients). At cally choose among a set of simulation gateways at the time ofeach gateway, traffic to and from the client applications are han- its connection. For example, the IP Virtual Server (IPVS) can bedled by OpenVPN. We defer the details of the OpenVPN setup to used to implement a rather sophisticated load balancing schemethe next section. at a front-end server, which subsequently redirect services to a In addition to running the OpenVPN server that manages the cluster of servers at the back-end (IP virtual server, 2008). In otherOpenVPN clients, we create another process at the simulation gate- situations, the clients may choose to connect to a gateway that canway to handle traffic to and from the real-time network simulator. satisfy a particular quality-of-service demand. This allows us toThe ssfgwd daemon is a process responsible for shunting IP pack- differentiate client traffic to and from the real-time simulator.ets between the simulator’s I/O threads and the OpenVPN server. It The quality-of-service demand could be as simple as finding a sim-maintains a separate TCP connection with each of the I/O threads ulation gateway that is geographically close to the client, or moreon the parallel machines running the simulation. Packets to be in- complicated as to find a simulation gateway that can sustain trafficserted into the real-time simulator are sent from ssfgwd via a ded- with certain throughput and latency requirements. In our imple-icated TCP connection to the reader thread associated with the mentation, we adopted the OpenVPN’s simple load-balancingsimulation process on the parallel machines. Packets exported scheme in which clients can choose randomly to connect to a ser-from the real-time simulator are sent from the writer thread via ver from a set of simulation gateways at the time of connection. Inanother TCP connection to ssfgwd. In either case, it is important any case, the I/O threads at the simulator must make separate TCPto have the TCP connections initiated from the simulator, since connections to all simulation gateways. We implemented mecha-the simulator is expected to run at the back-end of a supercom- nisms to allow a client each time to connect to a different gateway;puter, which is normally situated behind a firewall and/or cannot traffic from the simulator will be routed to the client machinebe determined a priori. It is quite possible that incoming TCP con- through the gateway with which the client is currently engaged.nections to the supercomputer are blocked by the firewall rules. The type of the TCP connections to a simulation gateway is dis- 3.3. The VPN connectionstinguished by the data subsequently sent from the I/O threads viathe TCP connections. A one-byte identifier is used to label a con- OpenVPN was chosen to allow applications to dynamically con-nection from either a reader thread or a writer thread. In addition, nect to the simulation gateway and to emulate network interfacesthe IP addresses of all virtual hosts to be emulated by the corre- on the client machines. We chose OpenVPN in our implementation,sponding simulation process are subsequently sent by the reader since it is publicly available and runs on most operating systems.thread. The ssfgwd process records this list of IP addresses and OpenVPN uses the TUN/TAP interface provided by the system.creates a mapping from the IP addresses to the corresponding As shown in Fig. 4, each OpenVPN instance creates a virtual net-TCP connection to the reader thread that sends these addresses. La- work device (named tun0 in this case). The virtual network deviceter, ssfgwd uses this mapping to forward traffic from the client has two end-points: one as a regular network interface with an as-machines to the correct simulation processes. signed IP address and the other with a file interface through which In order to support emulation of hosts with multiple network one can apply read and write operations. We assign the virtual net-interfaces (for reasons which we discuss in Section 3.4), we modi- work interface with an IP address that matches the IP address offied the VPN server to spawn the ssfgwd process directly and com- the emulated network interface in simulation and add an entrymunicate with it using the standard Unix pipes. An IP packet to the kernel forwarding table to direct all traffic to the virtual IPreceived by the VPN server from one of its clients is preceded with address space to be sent through the virtual network device. In thisthe client’s IP address before it is sent to ssfgwd via the pipe. The way, the client machine will assume the proper identity as applica-IP address is used by ssfgwd to deliver the packet to the corre- tions running on this machine can transparently forward traffic tosponding simulation process. Recall that a mapping from the IP ad- the simulated network. IP packets sent to the virtual networkdress to the simulation process that contains the client’s virtual interface are read by OpenVPN through the file interface. Subse-host is established immediately after the TCP connection is made quently, OpenVPN applies proper data compression and encryptionby the reader thread. Once the TCP connection is located, the pack- to the packets before sending them via UDP to the OpenVPN serveret is sent via the TCP connection to the designated reader thread at running at the simulation gateway. In the opposite direction, UDPthe simulator. The packet is subsequently inserted into the proto- packets received from the remote OpenVPN server are decryptedcol stack of the virtual node that carries the same IP address of the and decompressed before the packets are written out the virtualclient machine that initiates this packet. In such a way, the packet network interface. Thus the packets arrive at the application as ifappears as if it were generated directly by the virtual host. Simi- they come directly from the virtual network interface.larly, when an IP packet emanating from a virtual host is sent to To set up the real-time simulation infrastructure, one runs athe simulation gateway through the TCP connection established modified OpenVPN server at the simulation gateway using anby the writer thread, the packet is preceded with an IP address OpenVPN server configuration file automatically generated fromidentifying the virtual host (more precisely, a network interface the virtual network specification. Each OpenVPN client connectson the virtual host) that exports the packet. The ssfgwd process to a chosen simulation gateway using a separate client configura-sends both the IP address and the packet via the pipe to the Open- tion file, also generated automatically beforehand. The procedureVPN server, which forwards the packet to the corresponding Open- for creating these configuration files is illustrated in Fig. 5, whereVPN client with the IP address. In such a way, the packet arrives at the original components for setting up parallel network simulationthe client machine as if it receives the packet directly from a phys- execution are shown on the left and the shaded components to theical network. right are added to support the real-time simulation infrastructure. Our scheme supports the use of multiple simulation gateways Same with SSFNet, one specifies the virtual network in PRIMEto alleviate the traffic load placed otherwise on a single gateway using a simple configuration script written in the Domain Model-forming a bottleneck. Such load balancing decisions can be made ing Language (DML). A DML network description includes theeither statically or dynamically. All virtual nodes in the virtual net- topology of the network (i.e., sub-networks, hosts, routers, andwork can be partitioned and assigned to different simulation gate- links connecting network interfaces in the hosts and routers), theways at the configuration time. The entire emulation traffic is network protocols running at each host or router, as well as thetherefore divided among the simulation gateways and a better traffic traversing the network (see SSF Research Network, 2008 Please cite this article in press as: Liu, J. et al., A real-time network simulation infrastructure based on OpenVPN, J. Syst. Software (2008), doi:10.1016/j.jss.2008.08.015
  6. 6. ARTICLE IN PRESS6 J. Liu et al. / The Journal of Systems and Software xxx (2008) xxx–xxx Client Machine Application (e.g., iperf) OpenVPN Client Simulation Gateway user kernel OpenVPN eth0 Server tun0 ssfgwd Client Machine Application (e.g., iperf) OpenVPN eth0 Client user kernel PRIME Simulator tun0 eth0 Fig. 4. OpenVPN uses virtual network interfaces for tunneling application traffic. DML Network Description dmlenv VPN Server VPN ConÞguration Server DML Auxiliary SSFNet Information vpnscript dmlpart VPN Client VPN ConÞguration Client DML Partitioning Information Fig. 5. OpenVPN configurations generated automatically from DML network specification.for details). The DML network description is then processed by a script first creates a certificate authority used by the OpenVPNutility program called dmlenv, which automatically assigns IP ad- servers to validate the encrypted incoming client connections.dresses to all network interfaces defined in the virtual network. The script then generates a pair of public and private keys for eachdmlenv generates another DML file which contains the result of OpenVPN server or client. Finally, for convenience, vpnscriptthe IP address assignment, as well as other auxiliary information puts all files needed to start an OpenVPN server or client in a com-related to setting up the network simulation, including the map- pressed archive file to be distributed separately. For each simula-ping of communication channels between simulation entities and tion gateway, the archive includes the private key of thethe alignment of hosts and routers to form simulation processes. OpenVPN server and the public keys of all OpenVPN clients man-The auxiliary information will also be given to dmlpart, another aged by this simulation gateway (each corresponding to an emu-utility program that partitions the virtual network and generates lated network interface). Also included in the archive is ayet another DML file describing the assignment of the simulation mapping from public keys to IP addresses. The mapping will beprocesses to parallel processors. All these DML files will be needed used by the OpenVPN server to assign IP addresses to incoming cli-to start the SSFNet network simulator. ent connections, which are distinguished by the clients’ public We provide another utility program called vpnscript to auto- keys. For each OpenVPN client, the archive includes the privatematically generate the configuration files needed for setting up the key of the client and the public keys of all designated simulationOpenVPN servers and clients. We use a server configuration file to gateways. Another script is included in these archives to automateset up each simulation gateway and a client configuration file for the start of the OpenVPN servers and clients.each emulated network interface. Routers and hosts in the virtual We also added a plug-in module to the OpenVPN server to up-network with multiple network interfaces require multiple Open- date the network simulator of the connection status of the clientVPN connections, which we discuss in the next section. The vpn- machines. When a client establishes a VPN connection, the plug- Please cite this article in press as: Liu, J. et al., A real-time network simulation infrastructure based on OpenVPN, J. Syst. Software (2008), doi:10.1016/j.jss.2008.08.015
  7. 7. ARTICLE IN PRESS J. Liu et al. / The Journal of Systems and Software xxx (2008) xxx–xxx 7 Fig. 6. Emulation of a router with multiple network interfaces.in code will be run to send a notification packet to the emulation rived at the network interface, the writer thread sendsprotocol session on the corresponding virtual node. Upon receiving the address ahead of the packet to the simulation gate-this notification, the virtual node is then allowed to export packets way. The OpenVPN server is modified to use this leading IP addressto the client machine. Similarly, when a client machine disconnects to differentiate traffic to the client machines,1 in this case, forward-from the server, a notification is sent from the plug-in to the sim- ing the ICMP packet to the client machine labeled as so that simulated packets to the virtual host will be dropped through the corresponding VPN connection. The client machineat the simulator side (rather than being carried all the way to the therefore receives the ICMP packet from the logical network inter-simulation gateway and then getting dropped). face, exactly as it happens in the simulated network. Sup- pose the forwarding table of the client machine is set up so that the3.4. IP forwarding packet will be sent out immediately through the logical network interface The OpenVPN server, upon receiving this pack- Each VPN connection at the client machine corresponds to an et, will attach an IP address to the packet before sendingemulated network interface inside the simulation. For hosts with it onward to the ssfgwd process and subsequently to the simula-multiple network interfaces, multiple VPN connections are needed. tor. The reader thread at the simulator uses this leading IP addressIt would not be necessary to modify the VPN server if these virtual to dispatch the packet to the corresponding virtual host. The ICMPnodes were simply end hosts. In this case, packets arrived at the packet is then sent down the protocol stack of router B, out fromsimulation gateway from a client machine could be sent to the virtual network interface, continuing its journey on thesimulation process and the corresponding virtual host according virtual network.their source IP addresses. Similarly, packets arrived at the simula-tion gateway from the simulator could be distinguished by their 4. Experimentsdestination IP addresses and sent to the client machines via thecorresponding VPN connections. 4.1. Throughput and Latency This scheme, however, does not work if one wants to emulate arouter (e.g., to test a real routing protocol implementation). Con- We conducted several experiments to evaluate the throughputsider an example shown in Fig. 6, where we emulate the router B and latency of our real-time simulation infrastructure. As men-in a virtual network of three nodes. Since router B has two network tioned in Section 3, the simulation infrastructure consists of threeinterfaces with IP addresses and, we run two components on the server side: VPN server, ssfgwd, and I/Oseparate OpenVPN clients on the client machine. Suppose that threads. Each packet imported into (or exported from) the real-node A pings node C. An IP packet with a source address of time simulator goes through these three components in sequence., a destination address of, and a payload of an To evaluate the throughput and the latency induced by each com-ICMP ECHO-REQUEST message reaches the network interface ponent, we added the components one at a time in the experi- and is then exported to the simulation gateway as ex- ments and analyzed the associated overhead.pected. The ssfgwd process cannot determine how to forward We first measured the overhead of OpenVPN by running the ori-the packet to the client machine by inspecting the packet alone. ginal OpenVPN server with the IP forwarding option enabled. Thus, To overcome this problem, each packet sent between the net- client machines connected to the same OpenVPN server can simplywork simulator and the simulation gateway is preceded with anIP address indicating the network interface with which the packetis associated. For example, when exporting the ICMP packet ar- 1 Note that the changes only apply to the OpenVPN server. Please cite this article in press as: Liu, J. et al., A real-time network simulation infrastructure based on OpenVPN, J. Syst. Software (2008), doi:10.1016/j.jss.2008.08.015
  8. 8. ARTICLE IN PRESS8 J. Liu et al. / The Journal of Systems and Software xxx (2008) xxx–xxxcommunicate with each other via the UDP tunnels set up by Open- Table 1VPN; the server functions simply as a traffic stop between the cli- Latency in milliseconds when various components of the real-time simulation infrastructure are includedents. To expose the overhead induced by ssfgwd, we added aLOOPBACK option in its implementation. Packets generated by Minimum Average Maximum Standardone client and forwarded to the OpenVPN server and then to deviationssfgwd are directly sent back to the OpenVPN server and finally Through OpenVPN server 0.201 0.269 0.399 0.021delivered to the other client. In a similar fashion, we added a Through OpenVPN server 0.209 0.294 0.449 0.023 and ssfgwdLOOPBACK option to the real-time simulator. Packets sent from Through all components 0.366 0.442 0.576 0.029one client via the OpenVPN server, ssfgwd, the reader thread,and finally reached the real-time simulator are immediately ex-ported to the writer thread, meandering through ssfgwd and theOpenVPN server to the other client. We used ping to measure the latency of the real-time simula- We set up the experiment on a Xen virtual machine environ- tion infrastructure; the results are shown in Table 1. The tablement Barham et al., 2003. The host machine is a DELL OPTIPLEX shows the round-trip times between the two clients as traffic goes745 workstation with Intel Core 2 Duo 2.4 GHz processors and through the OpenVPN server, the OpenVPN server and ssfgwd,3 GB memory. Xen is a high-performance virtual machine monitor; and then all three components, respectively. As expected the la-we use Xen to minimize the overhead caused by transmitting net- tency increases when more components are involved in the path.work packets between physical hosts, which would depend on the Overall, it sums to a total of 0.442 ms on average if the packetsphysical interconnect. In Xen terminology, each domain is a virtual are forwarded through the entire simulation infrastructure.machine running on the host machine. Domain 0 is the first do-main created when the host is powered on and has privileges to ac- 4.2. Accuracy evaluationcess physical machine resources. It also provides an interface tomanage other user domains (called Domain U’s). We use another set of experiments to assess the impact on the Both the host OS and guest OS ran Debian Linux 4.0 distribu- emulation accuracy as the real-time simulation infrastructuretions with kernel version We put both the simulation inevitably introduces overheads when network traffic is sent be-gateway and the network simulator in domain 0 and created two tween the applications running on the client machines and theuser domains each containing a client, where we used iperf Iperf real-time simulator. The overhead manifested as packet delays– the TCP/UDP bandwidth measurement tool, 2008 to generate and losses is not part of the simulated network and could becomeUDP traffic at a certain rate and ran ping to measure the round- a major contributor to emulation errors.trip delays. We varied the traffic load from 50 Mb/s to 170 Mb/s. In our particular experiment setup, the machines running theFig. 7 shows the throughput achieved between the clients. OpenVPN clients and the simulation gateway were both AMD Ath- If the traffic only goes through the OpenVPN server, the lon64 machines with a 2.2 GHz processor and 2 GB memory. Thethroughput can reach as much as 145 Mb/s. A major part of the machines were connected via a gigabit switch. We chose two loca-overhead is from data encryption and compression, in addition to tions to run the network simulator. One is a 23-node Linux clusterthe overhead incurred by IP tunneling over UDP. If we allow the located on campus (each node equipped with a Pentium IV 3.2 GHztraffic to go through ssfgwd, the maximum throughput drops CPU and 512 KB memory). The campus network is a 100 Mb/sdown to 125 Mb/s, mainly due to the overhead from context Ethernet. The other location is the IBM terascale supercomputerswitching and memory copying. Once we include the real-time named DataStar (consisting of 272 8-way p655+ and 7 32-waysimulator, which means that all three components of the simula- p690 compute nodes) at the San Diego Supercomputer Center attion infrastructure are fully engaged in packet forwarding, the UCSD (2008).throughput drops further down to 90 Mb/s. Although this speed The simulated network is shaped like a dumbbell as shown inis sufficient for a common 100 Mb/s Ethernet network setup, it is Fig. 8, which has been commonly used to evaluate TCP congestionobvious that OpenVPN is not a proper choice to emulate faster net- control algorithms. There are N nodes aligned on either side of thework connections. OpenVPN is better suited for managing remote dumbbell connected by two routers. A server node on one side ofemulation connections that have lower throughput demands. the dumbbell is sending traffic via TCP to the corresponding client node on the other side of the dumbbell. The delay of the link con- necting a router with one of its adjacent client or server nodes is 150 set to be 100 ms and its bandwidth is 100 Kb/s. The link connecting OpenVPN Server OpenVPN Server and ssfgwd the two routers in the middle of the network has a delay of 300 ms 140 All Components and a bandwidth of (100 Â N) Kb/s. In our experiments, we emu- 130 lated the router R1, which is located on the server side of the dumb- bell network. On the client machine, we simultaneously ran (N + 1) Throughput (Mb/s) 120 OpenVPN clients, each corresponding to a network interface of rou- 110 ter R1. We also enabled IP forwarding on the client machine and set 100 the kernel routing table accordingly so that the Linux system 90 would forward packets to and from the virtual tunnel devices (cre- ated by OpenVPN). We use this dumbbell network specifically to 80 expose the overhead of the real-time simulation infrastructure. 70 By changing the number of TCP client/server pairs (N) we can vary the amount of network traffic traversing the real-time simulation 60 infrastructure. 50 Before the TCP transfers began, we first ran ping separately 60 80 100 120 140 160 from the machine running the simulator and the machine running Offered Load (Mb/s) the VPN clients to the simulation gateway to measure the round-Fig. 7. Throughput achieved when various components of the real-time simulation trip time of the segments between the simulator and the simulationinfrastructure are included. gateway (represented as x in Table 2) and between the simulation Please cite this article in press as: Liu, J. et al., A real-time network simulation infrastructure based on OpenVPN, J. Syst. Software (2008), doi:10.1016/j.jss.2008.08.015
  9. 9. ARTICLE IN PRESS J. Liu et al. / The Journal of Systems and Software xxx (2008) xxx–xxx 9 Fig. 8. Emulation of a dumbbell-shaped network.Table 2 100 %Round-trip times measured in milliseconds Local Linux cluster SDSC supercomputer 80 %x 0.210 ± 0.064 46.674 ± 3.383y 0.042 ± 0.007 0.045 ± 0.007 Data Transferredz 200.794 ± 0.889 247.197 ± 2.827 0.39% 23.59% 60 %d 0.529 0.464 40 %gateway and the client machine (represented as y). In the meantime, the client machine, on behalf of router R1, started pinging 20 %one of the simulated server nodes (say, S1, with the result being rep-resented as z). Let z0 be the simulated round-trip time between rou- Sim, N=1,2,4,8,16,32,64 N=128 (10 runs)ter R1 and S1 when R1 is not emulated (200.013 ms in this case). We 0%calculate the emulation error to be = |z À z0 |/z0 . Also, the difference 0 5 10 15 20 25 30 35 40in the round-trip times d = z À z0 À x À y reflects the delay overhead Time (in seconds)in addition to the imposed network delays of the two segments be-tween the client machine and the gateway and between the gate- Fig. 9. Emulation of TCP data transfers on local Linux cluster.way and the simulator. This delay overhead includes, for example,the time it takes to import and export simulation events and to transfer. The maximum height of staircase shows that there weretransport packets between ssfgwd and the OpenVPN server inside at most 64 segments in flight at the peak transfer, which can bethe simulation gateway. The baseline measurement reported in translated to an aggregate throughput of 32 Mb/s for all 64 TCPTable 2 was collected during a relative network quiescence. The re- connections. As we increased N to 128, the performance diverged.sults shown are averages of 10 separate runs. In both case, the delay We show the amount of data transferred during the first 40 sec-overhead accounts for only about 1/2 ms on average, which coin- onds or so for each of the 10 runs on the right side of Fig. 9. Ascides with the previous experiment. the traffic demand went beyond the capacity of the real-time sim- Next we set up all server nodes in the dumbbell network to each ulation infrastructure (and the 100 Mb/s campus network), signif-send a file of 300 KB simultaneously over TCP to their client coun- icant and unpredictable packet losses occurred causing the TCPterparts.2 We varied the number of client/server pairs, thus the congestion control to reduce the send rates accordingly—a phe-amount of network traffic traversing the real-time simulation nomenon that would not have occurred in the simulated network.infrastructure, during the experiment. Figs. 9 and 10 show the The emulation result in these cases deviates from the expectedaggregate data transferred over all TCP connections as we ran the network behavior.network simulator on the local Linux cluster and at the SDSC Fig. 10 shows the result of the network simulator running onsupercomputing center, respectively. Scaled up to 64 TCP connec- the SDSC supercomputer. Even with a single TCP connection, thetions, the emulation on the local cluster produced results almost overall throughput decreased markedly to 88% of the expected va-indistinguishable from the expected behavior. The telltale staircase lue, which is mainly due to the significant round-trip time betweenshape of the curve on the left of the figure is typical of the TCP data the simulator and the client machine (with an average of about 47 ms). More TCP connections further exacerbate this problem as 2 A small jitter was added to avoid the artificial synchrony among the multiple TCP additional traffic further lengthens the delays experienced by thetransfers. packets crossing the real-time simulation infrastructure. Different Please cite this article in press as: Liu, J. et al., A real-time network simulation infrastructure based on OpenVPN, J. Syst. Software (2008), doi:10.1016/j.jss.2008.08.015
  10. 10. ARTICLE IN PRESS10 J. Liu et al. / The Journal of Systems and Software xxx (2008) xxx–xxx 100 % real-time simulator to proportionally slow down with respect to real-time. That is, we fix the simulation time advancement to only a constant fraction of the real time; we make the simulation time 80% to advance by f 6 1 s for each wall-clock second. In relative terms, this essentially increases the packet processing capability of the Data Transferred simulation gateways and the client machines, as well as the capac- 60% ity of the physical network connections—by a factor of 1/f. The abil- ity to throttle simulation clock plays an important role in one of our case studies, described in the next section. 40% Sim 5. Case studies N=1 20% N=2 N=4 In this section we present two case studies involving real net- N=8 N=16 work applications using our real-time simulation infrastructure. N=32 0% 0 5 10 15 20 25 5.1. Content routing and distribution Time (in seconds) Fig. 10. Emulation of TCP data transfers on SDSC supercomputer. For the first case study we use the real-time simulation infra- structure to test a network device, named Content Aware Policy Engine (CAPE), developed at Northrop Grumman. CAPE allows one to examine, search, log, and modify the content of the packetsfrom the previous scenario, the latency—rather than packet loss— that pass through the device, which can be configured using a sim-plays a major role in admitting the emulation errors. As shown ple language. CAPE has both hardware and software implementa-in Fig. 11, the throughput reduced to 57% for 32 TCP connections, tions. For this case study we used a software version that runsplacing the outcome of the emulation poignantly different from on a Linux box. The software, written in python, uses the Berkleythe expected behavior. packet filter to snatch packets from the logical interfaces and then We should emphasize here that the exact results from these applies rules to the packets, possibly modifying them as specifiedexperiments are unimportant since they merely reflect a particular in the configuration language before forwarding the resultingsetup of our emulation system. The experiments, however, show packets onward. CAPE uses IP chains to drop the original packetsthat the losses and delays experienced by the packets as they travel to prevent the host OS from processing them unintentionally.through the real-time simulation infrastructure can have a signifi- In this study we use CAPE for content routing and distribution.cant impact on the emulation accuracy. On the one hand, to avoid We set up a simulation network that consists of two sub-networksthe damaging effect from network latency depicted in Fig. 10, the with a total of 60 routers and 1016 end hosts. This network is asimulator should be placed close to both the simulation gateway scaled-down version of the baseline network model from the DAR-and the client machines. This is especially important if we are to PA NMS program that has been used for large-scale simulationsupport applications that are sensitive to network latencies (such studies. Each sub-network (shown in Fig. 12) has a server clusteras TCP). On the other hand, with a tight coupling provided between (in net 1) that acts as the traffic source. Also, in net 2 and net 3,the simulator, the gateway, and the client machines, the latency there are a total of 12 local area network clouds, each with 42 cli-overhead is insignificant. More important, our experiments con- ent hosts acting as the traffic sink. To populate the network withfirm that the throughput of the real-time simulation infrastructurecan scale up close to the network’s 100 Mb/s physical capacity. We also need to emphasize that, although it is for certain thatsufficient bandwidth must be provided to allow the system to sus- net 0 net 1 0:1tain further emulation traffic in demanding scenarios, in cases 1:0where no reference to the wall-clock time is required (such as to the other 0:0 campusthe first case study in the following section), we can allow the 0:2 1:1 CAPE 4 5 Accuracy of Throughput (95% Confidence) 90% server 85% 2:0 2:1 3:0 3:1 80% 75% 2:2 2:3 3:3 3:2 70% 2:4 2:5 65% net 3 60% 2:6 55% LAN with 42 hosts 50% net 2 1 2 4 8 16 32 Number of TCP Connections Fig. 11. Accuracy of TCP throughput using SDSC supercomputer. Fig. 12. The DARPA NMS campus network instrumented with CAPE. Please cite this article in press as: Liu, J. et al., A real-time network simulation infrastructure based on OpenVPN, J. Syst. Software (2008), doi:10.1016/j.jss.2008.08.015
  11. 11. ARTICLE IN PRESS J. Liu et al. / The Journal of Systems and Software xxx (2008) xxx–xxx 11 5% We ran the experiments on the same AMD machines with giga- sim bit connections shown in the previous section. We varied the Avg Pkt Loss Rate (95% Confidence) emu/forward emu/replicate number of clients N from 5 to 160, resulting an aggregate traffic 4.5% of 4–128 Mb/s. We found the python-based software-based CAPE device was unable to process packets at a rate higher than 4 K packets per second without significantly dropping them. Therefore, 4% for experiments with more than 40 clients, the speed of the real- time simulator was throttled down to accommodate the slow pro- cessing of the CAPE device. During the experiment, we also found 3.5% the device was unable to handle more than 160 client connections due to a bug in the software-based CAPE device. Fig. 13 shows that, 3% as expected, the packet loss rate (averaged over 10 runs) increases with the increase of clients when content distribution is disabled. The results from simulation and emulation are close. When repli- 2.5% cation is activated in CAPE, however, the loss rate remains stable, 5 10 20 40 80 160 implying that the packet losses occur primarily at the network seg- Number of Clients (N) ment between the CAPE device and the streaming video server. Fig. 13. Using CAPE content distribution to reduce packet loss. 5.2. Intra-domain routingsufficient background traffic, at each LAN, we designated 320 fluid In our second case study, we use the real-time simulation infra-TCP flows, 2 packet TCP flows, and 2 packet UDP flows, each down- structure to investigate network routing. Specifically, we testloading data from a randomly selected server from both campuses XORP, which is an open-source software router platform for re-(see Liu, 2006 for more detail on the fluid TCP model). The applica- search and development (Handley et al., 2003). XORP supportstion we focus on is streaming video, as a sequence of 1 KB UDP dat- implementations of common Internet routing protocols, includingagrams sent at a constant rate. We randomly chose N clients in net BGP4+, OSPFv2, OSPFv3, RIPv2, RIPng, and several multicast rout-2 at one campus network to each request a 100 KB/s video stream ing protocols. We modified XORP to support split routing and for-from a server randomly chosen from the other campus network. warding: routing protocols are run by XORP on the client machinesWe placed the CAPE device between the routers labeled 0:0 and and the resulting routes are used to construct forwarding tables4 (in Fig. 12) so that it could intercept the streaming traffic. We used by the corresponding routers in simulation to make packetmeasured the overall average packet loss rate of all clients as an forwarding decisions (Li et al., 2008).indication of the receiving video quality. Here we study intro-domain routing using OSPFv2. We used a We ran three tests. As the baseline, we took out the CAPE device test scenario representing the Abilene backbone network, whichand simply ran the simulation. In the second test, we emulated the consists of eleven routers at major cities in the Unite State (seeCAPE device but programmed the device to forward all the traffic Fig. 14). The test case is modified from the one developed originallywithout inspecting the content. In the third test we configured by Bavier et al. for evaluating VINI (Bavier et al., 2006), a virtualthe CAPE device to allow only one outstanding request from the network testbed for supporting experimentation on the PlanetLab.clients to reach the server and the rest were cached inside the CAPE We used fluid models to represent traffic on the backbone net-device. When the server started the video stream back to the de- work (Liu, 2006). The number of TCP flows between the routersvice, CAPE replicated the video stream to all other clients. In this was set to be proportional to the link utilization shown on the Indi-way, the network path between the CAPE device and the server ana University Abilene Network Operations Center Weathermapwas only burdened with one video stream and better network per- (Abilene backbone network, 2008). Since the fluid traffic is calcu-formance was expected. lated inside the real-time network simulator, the fluid background Seattle Chicago New York Sunnyvale Kansas City Washington, DC Denver Indianapolis Los Angeles Atlanta Houston Fig. 14. Abilene backbone routers running OSPFv2. Please cite this article in press as: Liu, J. et al., A real-time network simulation infrastructure based on OpenVPN, J. Syst. Software (2008), doi:10.1016/j.jss.2008.08.015