Embedded Virtual Machines: Architecture Description,                        Technical Report ESE-2009-01                  ...
Figure 1: (a) A wireless sensor, actuator and controller network. (b) Algorithm assignment to a setof controllers, each ma...
functional invariants, such as a control law and para-functional invariants such as timeliness con-straints, fault toleran...
4. Fault diagnostics, repair and recovery are manual and template-driven for amajority of networked control systems. In th...
the option of static or dynamic binding depending on the capabilities and timing requirements forthe given control problem...
Table 1: Control EVM PID ( p KP KI KD addr_in addr_out - )                                Proportional-Integral-Derivative...
Table 2: Common EVM:                                                                                             Dene a wo...
3    EVM ArchitectureWe now describe the node-specic architecture which implements the mechanisms for the virtualmachine o...
Motor            Overload             State      Focus of EVM work                 Apps                               Cont...
(a) Virtual Tasks Description Table                            (b) nanoRK TCB extension  Global ID                        ...
3.4 Virtual Component ManagerThe fundamental dierence between native nano-RK and the VCM is that the scope of nano-RKsacti...
2. Redistribution of VTs when adding/removing nodes3. Planned VT migration triggered by the user4. Unplanned VT migration ...
problems with native support for task migration and on-line task assignment. Kairos [22] allowsa programmer to describe co...
Upcoming SlideShare
Loading in...5

Tech reportese01 09


Published on

Published in: Design, Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Tech reportese01 09

  1. 1. Embedded Virtual Machines: Architecture Description, Technical Report ESE-2009-01 Miroslav Pajic and Rahul Mangharam September, 2009 Abstract Embedded wireless networks have largely focused on open-loop sensing and monitoring. To address actuation in closed-loop wireless control systems there is a strong need to re-think the communication architectures and protocols for reliability, coordination and control. As the links, nodes and topology of wireless systems are inherently unreliable, such time-critical and safety-critical applications require programming abstractions where the tasks are assigned to the sensors, actuators and controllers as a single component rather than statically mapping a set of tasks to a specic physical node at design time. To this end, we introduce the Embed- ded Virtual Machine (EVM), a powerful and exible programming abstraction where virtual components and their properties are maintained across node boundaries. In the context of pro- cess and discrete control, an EVM is the distributed runtime system that dynamically selects primary-backup sets of controllers to guarantee QoS given spatial and temporal constraints of the underlying wireless network. The EVM architecture denes explicit mechanisms for control, data and fault communication within the virtual component. EVM-based algorithms introduce new capabilities such as predictable outcomes and provably minimal graceful degra- dation during sensor/actuator failure, adaptation to mode changes and runtime optimization of resource consumption. Through the design of a natural gas process plant hardware-in- loop simulation we aim to demonstrate the preliminary capabilities of EVM-based wireless networks.1 IntroductionAutomation control systems form the basis for signicant pieces of our nations critical infrastruc-ture. Time-critical and safety-critical automation systems are at the heart of essential infrastruc-tures such as oil reneries, automated factories, logistics and power generation systems. Discreteand process control represent an important domain for real-time embedded systems with over atrillion dollars in installed systems and $90 billion in projected revenues for 2008 [1]. In order to meet the reliability requirements, automation systems are traditionally severelyconstrained along three dimensions, namely, operating resources, scalability of interconnectedsystems and exibility to mode changes. Oil reneries, for example, are built to operate withoutinterruption for over 25 years and can never be shutdown for preventive maintenance or upgrades.They are built with rigid ranges of operating throughput and require a signicant re-haul to adaptto changing market conditions. This rigidity has resulted in proprietary systems with limited scopefor re-appropriation of resources during faults and retooling to match design changes on-demand.For example, automotive assembly lines lose an average of $22,000 per minute of downtime [2]during system faults. This has created a culture where the operating engineer is forced to patch afaulty unit in an ad hoc manner which often necessitates masking certain sensor inputs to let theoperation proceed. This process of unsystematic alteration to the system exacerbates the problemand makes the assembly line dicult and expensive to operate, maintain and modify. Embedded Wireless Sensor-Actuator-Controller (WSAC) networks are emerging as a practicalmeans to monitor and operate automation systems with lower setup/maintenance costs. While 1
  2. 2. Figure 1: (a) A wireless sensor, actuator and controller network. (b) Algorithm assignment to a setof controllers, each mapped to the respective nodes. (c) Three Virtual Components, each composed ofseveral network elementsthe physical benets of wireless, in terms of cable replacement, are apparent, automation manu-facturers and plant owners have increasing interest in the logical benets. With multi-hop WSAC networks, it is possible to build modular systems which can be swappedout for o-line maintenance during faults. Modular systems can be dynamically assigned tobe primary or backup on the basis of available resources or availability of the desired calibra-tion. Modularity allows for incremental expansion of the plant and is a major consideration inemerging economies. WSAC networks allow for runtime conguration where resources can bere-appropriated on-demand, for example when throughput targets change due to lower price elec-tricity during o-peak hours or due to seasonal changes in end-to-end demand. While WSAC networks facilitate both planned and unplanned mode changes, runtime pro-grammable WSAC networks allow for exible item-by-item process customization. For example,a high demand for fuel-ecient Toyota Prius will require major retooling of a traditional wiredfactory that is designed for the Toyota Camry chassis. With re-programmable WSAC, the assem-bly line stations can adapt to a schedule where every 3 Camrys are interleaved with 2 Prius withsynchronized changes in operation modes and assembly line operations.1.1 Embedded Virtual MachinesThe current generation of embedded wireless systems has largely focused on open-loop sensing andmonitoring applications. To address actuation in closed-loop wireless control systems there is astrong need to re-think the communication architectures and protocols for reliability, coordinationand control. As the links, nodes and topology of wireless systems are inherently unreliable, suchtime-critical and safety-critical applications require programming abstractions where the tasks areassigned to the sensors, actuators and controllers as a single component rather than staticallymapping a set of tasks to a specic physical node at design time (see Fig. 1). Such wirelesscontroller grids are composed of many wireless nodes, each of which share a common sense ofthe control application but without regard to physical node boundaries. Our approach, as shownin Fig. 2(a), is to decouple the functionality (i.e. tasks) from the inherently unreliable physicalsubstrate (i.e. nodes) and allow tasks to migrate/adapt to changes in the underlying topology. To this end, we introduce the Embedded Virtual Machine (EVM), a powerful and exibleprogramming abstraction where virtual components and their properties are maintained acrossnode boundaries. EVMs dier from classical virtual machines (VM). In the enterprise or on PCs,one (powerful) physical machine may be partitioned to host multiple virtual machines for higherresource utilization. On the other hand, in the embedded domain, an EVM is composed acrossmultiple physical nodes with a goal to maintain correct and high-delity operation even underchanges in the physical composition of the network. The goal of the EVM is to maintain a set of 2
  3. 3. functional invariants, such as a control law and para-functional invariants such as timeliness con-straints, fault tolerance and safety standards across a set of controllers given the spatio-temporalchanges in the physical network. By incorporating EVMs in existing and future wireless automa-tion systems, our aim is to realize: 1. Predictable outcomes in the presence of controller failure. During node or link faults, EVMalgorithms determine if and when tasks should be reassigned and provide the mechanisms fortimely state migration. 2. Provably minimal QoS degradation without violating safety. In the case of (unplanned)topology changes of the wireless control network, potential safety violations are routine occurrencesand hence the EVM must reorganize resources and task assignments to suit the current resourceavailability (i.e. link bandwidth, available processing capacity, memory usage, sensor inputs, etc.). 3. Composable and recongurable runtime system through synthesis In the EVM approach,a collection of sensors, actuators and controllers make a Virtual Component as shown in Fig. 1.A Virtual Component is a composition of interconnected communicating physical componentsdened by object transfer relationships. At runtime, nodes determine the task-set and operatingpoints of dierent controllers in the Virtual Component. This machine-to-machine coordinationrequires task-set generation, task migration and remote algorithm activation which are executedvia synthesis at runtime. 4. Runtime Resource Re-appropriation and Optimization for dynamic changes in service.For planned system changes such as a factory shift, increase in output or retooling for a dierentchassis, nodes are required to be re-scheduled in a timely and work conserving manner.1.2 Challenges with Wireless ControlWhile there has been considerable research in the general area of wireless sensor networks, a ma-jority of the work has been on open-loop and non-real time monitoring application. As we extendthe existing programming paradigm to closed-loop control applications with tight timeliness andsafety requirements, we identify four primary challenges with the design, analysis and deploymentof extending such networks: 1. Programming motes in the event-triggered paradigm is tedious for controlnetworks. It is hard to provide any analytical bounds on the response time and stability oftasks in an event-driven regime [3, 4]. Real-time tasks are time-triggered while sensor inputs areevent-triggered. It is easier to incorporate sporadic tasks in a time-triggered regime than viceversa. 2. Programming of sensor networks is at the physical node-level where the tasks arebound to the node at compile-time. It is non-trivial to decompose a large control problem intodening components and applications for each mote. 3. Design of systems with exible topologies is hardTaskphysical node-level program- Runtime with Managementming as the set of tasks (or responsibility) is associated with the physical node. Thus, a change • Task_Migratein the link capacity, node energy level or connectivity in the current topology will render the ap- – Task instructionsplication useless. It is necessary to associate – Stack mapping of tasks to nodes and incorporate a logicalmechanisms to transfer responsibilities during physical and environmental changes in the network – Data(as in Fig. 2(a)). – Associate libraries – Control/Schedule/Resource meta data VT1 VT2 ... VTn Task_Migrate(T2) Node Node 1 2 N1 N2 N3 ... Nm Current Task-set: T1, T2, T3 Current Task-set: T’1, T’2, …, T’7 (a) Decoupled virtual tasks and physical (b) Task migration for real-time operation (instructions, stack, nodes with runtime task mapping data timing/fault tolerance meta-data) on one physical node to another. 29 Figure 2: Programmatic control with decoupled virtual tasks and physical controller nodes 3
  4. 4. 4. Fault diagnostics, repair and recovery are manual and template-driven for amajority of networked control systems. In the case of WSAC networks, it is not plausible toexhaustively capture all possible faults at design time and thus provisions must be made forruntime diagnostics and recovery.1.3 Overview of the EVM ApproachWhile wireless system engineers optimize the physical, link and network layers to provide an ex-pected packet error rate, this does not translate accurately to the stability of the control problem atthe application layer. For example, planned and unplanned changes in the network topology withnode/link failures are currently not easily captured or speciable in the metrics and requirementsfor the control engineer. For a given plant connected to its set of controllers via wireless links it isnecessary that the controller process the sensor inputs and perform actuation within a boundedsampling interval. While one approach is to design specialized wireless control algorithms thatare robust to specied range packet errors [5, 6], it is non-trivial to design the same for frequenttopological changes. Furthermore, it is dicult to extend the current network infrastructure toadd/remove nodes and to redistribute control algorithms to suit environmental changes such asbattery drain for battery-operated nodes, increased production during o-peak electricity pricing,seasonal production throughput targets and operation mode changes. The EVM approach is to allow the control engineer to utilize the same wired-network controlalgorithms on the wireless network without knowledge of the underlying network protocols, node-specic operating systems or hardware platforms. The virtual machine executing on each nodewithin the virtual component instruments the VC to adapt and recongure to changes whileensuring the control algorithm is within its stability constraints. This approach is complementaryto the body of network control algorithms [5, 6] as it provides a logical abstraction of the underlyingphysical node topology.2 EVM Design FlowOur focus is on the design and implementation of wireless controllers and in providing suchcontrollers with runtime mechanisms for robust operation in the face of spatio-temporal topo-logical changes. Our focus is exclusively on robust controller design and not on sensors or ac-tuators, as the latter are largely physical devices with node-bound functionality. In order toenable control engineers to design wireless control systems in a manner that is both largely plat-form/protocol/hardware architecture independent and extensible to dierent domains of controlsystems in process, discrete, aviation, medical, etc. we present a three-layered design process. Wenow describe the design ow from control problem formulation, automatic translation of controlmodels to platform-independent EVM interpreter-based code and nally to platform-dependentbinaries (see Fig. 2). These binaries are assigned to nodes within a VC using assignment andscheduling algorithms presented in [7]. The binaries are executed as virtual tasks within thenode-specic architecture described in Section 3. At design time, control systems are usually designed using software tools, such as Mat-lab/Simulink, that incorporate both modeling and simulating capabilities. Therefore, in orderto automatize whole design ow, the EVM is able to automatically generate functional modelsfrom the Simulink control system description to dene the processes by which input sampleddata is manipulated into output data for feedback and actuation. These functional models arerepresented by generated code and meta data that are platform and node independent systemdescription, thus providing the system designer to exclusively focus on the control problem design. From the functional description in the platform-independent and domain specic language,the EVM design ow automatically extracts additional para-functional properties, like timing,inter task dependencies, etc from the Simulink model. All these properties, along with functionaldescription are used to dene a platform optimized binary for each virtual task (VT). VTs have 4
  5. 5. the option of static or dynamic binding depending on the capabilities and timing requirements forthe given control problem specication.2.1 Control Problem SynthesisA functional description of a VT is automaticallyextracted from the Simulink design, using the Design Control System Descriptionfact that each block can be represented as a com- Time (Matlab/Simulink, C…)position of other Simulink blocks. Thus, eachmodel can be presented as a hierarchical com-position of basic Simulink blocks.This organiza- Platform Domain-specific EVM Descriptiontion of Simulink models allows for natural extrac- Independent (Common-EVM and Control-EVM)tion of a functional description using predenedwords from the EVM programming language dic- Platformtionary (described in the next sub-section). Simi- Dependent Bin. Virtual Component Manager/RTOS (nano-RK)larly, when a new block in Simulink is dened as Formata composition of previously dened blocks, forthe EVM functional description, a new word is Figure 3: EVM Design owdescribed using previously dened words, until alevel is reached where all words belong to theEVM dictionary. Therefore, a VT description is obtained using a parser that processes theSimulink .mdl le as a .txt le by searching for a new block denitions along with the inter-connections between blocks. Since each block in Simulink design has to have an unique name, astraightforward functional extraction would result in number of EVM words equal to the numberof used (sub)blocks. In order to avoid this, before each new word is dened the parser checkswhether a word with same structure already exist. This check can contain several hierarchicallevels and thus it can increase parsers execution time. An example of creation of a VTs func-tional description from a Simulink design model in Fig. 4(a) is presented in Fig. 4(b). As theplatform-independent language is stack-based, the notation is in reverse.2.2 Platform Independent Domain Specic LanguageIn order to generate functional description of the designed system, the EVM programming languageis based on FORTH, a structured, stack-based, extensible programming language[8]. On the otherhand, since the goal of EVM design is to allow exibility and designing utilities independentof chosen programming language, the intermediate programming language is not constrained toFORTH. This allows users to have a degree of freedom to replace Simulink with C, .NET, VB orother descriptions for control system implementation. The use of the EVM intermediate programming language enables domain-specic con-structs, where basic programming libraries are related to the type of application that is beingdeveloped. For example, for use in embedded wireless networks for industrial control, we devel-oped two predened libraries, Common EVM ( Table 2), based on the standard FORTH library[8], and Control EVM that contains functionality widely used for developing control applications Fcn1 1 f(u) out 1 PID_controller 1 R1 2 Sum 2 in out 2 in 2 1 Sum Saturation out 2 : VTout1 SWAP f SUM ( x x – y ) ; in 1 Fcn : VTout2 NEGATE R1 SUM kp1 ki1 kd1 0 0 PID SATURATE ( x – x ) ; R2 f(u) : VTout4 NEGATE R2 SUM kp ki kd 0 0 PID SATURATE ( x – x ) ; in out 4 3 Sum 3 out 4 : VT_ID ROT DUP ROT VTout1 DUP VTout4 2SWAP SWAP VTout1 DUP Sum 1 PID_controller in 3 Saturation 1 VTout4 2SWAP ( in1 in2 in3 – out1 out2 out3 out4 ) ; 3 out 3(a) Example of a control algorithm (VT) modeled in (b) Generated codeSimulink Figure 4: Generation of EVM functional description from Simulink model 5
  6. 6. Table 1: Control EVM PID ( p KP KI KD addr_in addr_out - ) Proportional-Integral-Derivative controller calculating p outputs in one invocation, result is updated directly in memory, since previous values are needed PI ( p KP KI addr_in addr_out - ) Proportional-Integral controller TF ( p m n addr_alpha addr_beta addr_in add_out - ) Transfer function described with mth (nth ) order polynomial in nominator (denominator) with coecients stored in α and β LTI ( p m n addr_A addr_B addr_C a_in a_x a_out - ) LTI system, x[k + 1] = Ax[k] + Bu[k], y[k] = Cx[k], where p, m, n are sizes of vectors n, y and x(a list of all APIs can be seen in Table 2 and Table 1). In addition the extensibility of EVMallows to dene the Automotive EVM, Aviation EVM or Medical EVM libraries that will containfunctionalities specic to each of these application elds. Using these libraries code generatorcreates a system description from a predened component, thus creating a task description le foreach of the Virtual Tasks (VTs). Timing parameters (period and worst-case execution time) are also extracted from the model.Since we consider only discrete controllers as potential VTs, Simulink design rules force the designerto dene a sampling rate for each discrete block, allowing us to extract period as least commondivider for all task timing. These values are also extracted from the model (.mdl) le and addedto the VTs description le, along with the functional description. List of the Control EVM words is presented in Table 1. First three words specied in the tableare used for the Singe-Input-Single-Output (SISO) systems. Although all these systems can bedescribed using the last word (LTI - describing Linear Time Invariant system), the fact that theyare widely exploited recommended its use. 6
  7. 7. Table 2: Common EVM: Dene a wordArithmetic operations (on 16bit):+ ( n1 n2 - n3 ) Adds n1 and n2, result returned on the stack- ( n1 n2 - n3 ) Subtract n2 from n1, result returned on the stack1+ ( n1 - n2) Add 1 to n1, return the result on the stack1- ( n1 - n2) Subtract 1 from n1, return the result on the stack* ( n1 n2 - n3 ) Multiply (signed) n1 and n2, result returned on the stack/ ( n1 n2 - n3 ) Divide n1 with n2, quotient returned on the stack/MOD ( n1 n2 - n3 n4 ) Divide n1 with n2, quotient n3 and reminder n4 returned on the stackMOD ( n1 n2 - n3 ) Divide n1 with n2, reminder returned on the stackL ( x1 u - x2 ) Logical left shift of u places on x1, result returned on the stackL ( x1 u - x2 ) Logical right shift of u places on x1, result returned on the stackA ( x1 u - x2 ) Arithmetical left shift of u places on x1, result returned on the stackA ( x1 u - x2 ) Arithmetical right shift of u places on x1, result returned on the stackABS ( n - |n|) Replace the top stack item with absolute valueMIN ( x1 x2 - x3 ) Lower of the two values is returned on the stackMAX ( x1 x2 - x3 ) Higher of the two values is returned on the stackLogical operations (on 16bit):INV ( x1 - x3 ) Invert x1, result returned on the stackAND ( x1 x2 - x3 ) ands x1 and x2, result returned on the stackXOR ( x1 x2 - x3 ) xors x1 and x2, result returned on the stackOR ( x1 x2 - x3 ) ors x1 and x2, result returned on the stackNOR ( x1 x2 - x3 ) nors x1 and x2, result returned on the stackNAND ( x1 x2 - x3 ) nands x1 and x2, result returned on the stackComparison and testing:0 ( n - ag )0 ( n - ag )0 ( n - ag )0= ( n - ag )= ( n1 n2 - ag ) ( n1 n2 - ag ) ( n1 n2 - ag ) ( n1 n2 - ag )Memory manipulation:@ ( addr - x) fetch! ( x addr - ) writemalloc ( size - addr ) Allocation of 128B memory blockmfree ( addr - ) Frees 128B memory blockSensor/Actuator handling:RDSensG ( sensID - n1 ) Read sensor value from the sensor described by the global sensor IDRDSensL ( sensID - n1 ) Read sensor value from the sensor described by the local sensor IDWRActG ( value actID - ) Write value to the actuator described by the global actuator IDWRActL ( value actID - ) Write value to the actuator described by the local actuator IDNetworking:PktSendG ( addr n nodeID - ) Send n-byte message stored at address addr to node described by global ID 7
  8. 8. 3 EVM ArchitectureWe now describe the node-specic architecture which implements the mechanisms for the virtualmachine on each node. The Common-EVM and Control-EVM description for each set of controllersare scoped within Virtual Tasks that are mapped at runtime by the the task Assignment Algorithmdescribed in [7]. This description is interpreted by the interpreter running on each node. The EVMruntime system is built on top of the nano-RK real-time operating system[9] as a supertask, allowingnode-specic tasks to execute natively and virtual tasks (VTs), i.e. those that are dynamicallycoupled with a node, to run within the EVM. The EVM services are shown in Fig. 6 and the EVMblock-level reference architecture is presented in Fig. 5. This allows EVM to maintain node specicfunctionalities and be extensible to runtime task evocation of existing or new virtual components. The interface between nano-RK and all VTs is realized using the Virtual Component Manager(VCM) supertask. The VCM maintains local resource reservations (CPU, network slots, memory,etc.) within nano-RK, the local state of the VTs and global mapping of VTs within the virtualcomponent. The VCM is responsible for memory and network management for all VTs-to-physicalnodes and presents a mapping between local and remote ports which is transparent to all local VTs.The VCM includes a FORTH-like interpreter for generic and domain-specic runtime operationsand a Fault/Failure Manager (FFM) for runtime fault-tolerant operation.3.1 nano-RK Real-Time OSTo address the need for timing precision, priority scheduling and ne-grained resource managementthe nano-RK resource kernel [9] has been previously developed with timeliness as a rst-classconcern. nano-RK is a fully preemptive RTOS with multi-hop networking support that runs ona variety of sensor network platforms (8-bit Atmel-AVR, 16-bit TI-MSP430, Crossbow motes,FireFly). All networking is conducted over the RT-Link [10] real-time link protocol that is nativeto nano-RK. It supports xed-priority preemptive scheduling for ensuring that task deadlines aremet, along with support for and enforcement of CPU and network bandwidth reservations. Taskscan specify their resource demands and the operating system provides timely, guaranteed andcontrolled access to CPU cycles and network packets in resource constrained embedded sensorenvironments. It also supports the concept of virtual energy reservations that allows the OS toenforce energy budgets associated with a sensing task by controlling resource accesses. nano-RKprovides various medium access control and networking protocols including a low-power-listenCSMA protocol called B-MAC, an implicit tree routing protocol and RT-Link. For networked control systems, it is essential that the underlying sensor operating system ex-pose precision timing, scheduled tasks and synchronized networking so that the trade-os betweenenergy-consumption (node lifetime), reliability and responsiveness are speciable and enforceableboth at design-time and runtime. Support for the above services is required for low-duty cycle andenergy-constrained sensor networks too because the computation and communication are packedinto a short duration so all nodes maximize their common sleep time. nano-RK has been design as fully static OS, congured at the design time. Therefore to allow Interp. FFM Task Task native tasks VT 1 virtual tasks ... VCM Task 1 ... Task n VT m nanoRKFigure 5: EVM architecture with Virtual Component Manager running as a supertask along side nativenano-RK tasks 8
  9. 9. Motor Overload State Focus of EVM work Apps Control Detection Migration Task Task Task Reserves Reserves Reserves Adaptive Virtual Machine Real-Time Scheduler Network Runtime System Management Kernel Task Peripheral Parametric Control Management Drivers Task Partitioning RT-Link RX Buffer Scheduleability Analysis Reservations TX Buffer Software Attestation Algorithm Activation Protocol Adaptation Hardware Time Sync RX Power Control Policy Negotiation Data Migration Microcontroller 802.15.4 Radio Online Fault Diagnosis Figure 6: nano-RK RTOS architecture with EVM extensionsparametric and programmatic run-time changes in the code nano-RK was redesigned and extendedwith several new features (see Fig. 6):• Runtime Parametric Control: Support for dynamically changing the sampling rates, runtimetask and peripheral activation/deactivation and runtime modication to the task utilization hasbeen added. These facilities are exposed and executed via the Common-EVM programmer inter-face.• Runtime Programmatic Control: As a part of EVM design dynamic task migration has beenimplemented. This requires runtime schedulability analysis, capability checks to migrate a subsetof the task data, instructions, required libraries and task control block. Based on the Assignment[], tasks may be activated or migrated between primary and backup nodes. Such facilities aretriggered by the primary-backup policy implemented on top of the assignment algorithm.• Dynamic Memory Management: Both Best-t and First-t memory allocation methods aresupported. In addition a Garbage Collector (GC) has been designed to reclaim all memory seg-ments owned by tasks that had been terminated. The GC is scheduled only when its executiondoes not inuence execution of other tasks. To allow for dynamic memory (re)allocation dur-ing code execution, a memory management mechanism was implemented with functions void ∗nrk _malloc(uint16_t) and void nrk _f ree(void ∗).3.2 Virtual Component InterpreterThe Virtual Component Interpreter provides the EVM programmer an interface to dene and exe-cute all VTs. Every VT is dened as a word within the VCM library. When a new VT descriptionis received over the network, the VCM calls the interpreter which denes a new word using thedescription le of a task and existing virtual component libraries. When a VT is activated, eachexecution of VT is implemented as a scheduled activation of the interpreter with the VTs word asan input. To allow preemptivity of the tasks, each call of the interpreter uses a VT-specic stackand dedicated memory segments. In addition, since each VT is capable of dynamically allocatingmemory during its execution in cases when there is no space in already dedicated memory, theEVMs memory manager allocates a new block of xed size (currently 128B). Therefore, the inter-preter is designed to use logical addresses in form (block_index, local_address_inside_block).In order to be able to use the nodes physical memory, the VCM supports address translation andgarbage collection for the interpreter. Each node has a local copy of standard Common-EVM dictionary and Control-EVM dictio-naries. If a new word needs to be included in the existing library, upon reception of a messagethat contains the word, VCM initiates execution of the interpreter. The interpreter rst checksthe global word identier and revision number in order to discard obsolete versions. If a newerversion is at hand, the interpreter (re)denes the word in the dictionary. Since all VTs executions 9
  10. 10. (a) Virtual Tasks Description Table (b) nanoRK TCB extension Global ID Local ID Processing Units needed CPU Reserve Deployment constraints Type (Fixed/Flexible) Memory requirements Stack size Number of nodes (n) 1st Block pointer Node 1 ... ... nth Block pointer Node n Number of Tx slots Pointer to slot Memory requirements Stack size index array Number of Blocks Number of Rx slots Pointer to slot Network Dependencies Number of Tx slots index array Number of Rx slots EVM description le Table 3: Local and Global task descriptionare performed using the interpreter, there is no need for any additional kind of provider-subscriberrelation as described in [11]. In addition, potential failure modes that may result from inconsistentversions on each node are avoided using this simple approach. In the case when a when a nodereceives an update for a word used to dene a task, it checks the revision number in the denitionle of the task in the case of a newer revision, after updating denition of the word, the interpreterinforms the VCM to update the VTs description le. Each local update, also updates the revisionnumber of the VT word, therefore keeping VT fully updated without requiring consensus acrossthe VC.3.3 Virtual TasksEach VT is described using Virtual Tasks De-scription Table, comprised of a global and local INTERPRETERdescription of a VT (see Table 2(a)). Copies ofthe Descriptor Table are stored on all members User Definedof the VC assigned within the VT. While this re- - Common EVMquirement for consistency currently results in an - Control EVMissue of scalability, the current project with Hon-eywell Process Control (OneWireless) requireshigh-speed control networks with less than 20nodes and is hence within the practical limits Standardized - Message formatof the current approach. Each task species itsDeployment constraints which describe specic ...distribution requirements of a task as to whethera task is Fixed, meaning that it has to run ona specic subset of physical nodes, or Flexible, Figure 7: VCM/Interpreter relationi.e. a task can be deployed on any physical nodein the VC. If a task is declared as Fixed, the number of the suitable nodes is specied along withthe node IDs. Each VTs global description has information about memory requirements, stacksize and number of xed size memory blocks (e.g.128B) that are used. In addition to the abovemeta data, Network requirements in terms of number of active RT-Link transmit and receive slotsare specied at design time. The above descriptors are specied as within the VCMs Task ControlBlock (TCB) for each task. The EVM TCB is an extension to the native nano-RK TCB[9]. Upon VT denition on a node, VCM creates Task Control Block (TCB) for the task. Besideparts of native TCB (more details can be found in [6]) VCM initializes the TCB extension presentedin Table 2(b) which describes local resources used by the VT. 10
  11. 11. 3.4 Virtual Component ManagerThe fundamental dierence between native nano-RK and the VCM is that the scope of nano-RKsactivities is local, node-specic and dened completely at design time, while the scope of theVCM is the Virtual Component that may possibly span multiple physical nodes. The current setof functionalities supported is:1. Virtual Task handling:1.1 VC state is maintained, keeping every node with a consistent mapping of VTs. The VCMin each controller node within the VC periodically broadcasts its information about VT mappingin order to keep consistency between all members of the VC. Currently a centralized consensusprotocol is used and a distributed consensus protocol is needed to scale operations.1.2 Task migration that can be triggered as a result of a fault/failure procedure or by request ofeither the VT or VCM. As a part of a task migration, tasks VTDT is sent along with all memoryblocks used by the task. If the VT is already dened on a node (checked by exchange of hashvalues), only task parameters are exchanged. In addition, due to physical limitations (network,dierences between controller/actuator nodes, etc) before migrating VT to a particular node, net-work and CPU schedulability analysis are performed for nodes that are potential candidates. Ifanalysis show that no node can execute the task correctly, an error message is returned.1.4 VT activation: After a VT is dened, the VCM performs local CPU and network schedulabil-ity analysis prior to task activation. This is done in order to ensure that task will not adverselyaect correct execution of previously dened VTs.1.5 Control of tasks executed on other nodes: For all VTs in Backup mode, the VCM shadows theexecution of the VT in Primary mode. If a departure from the desired operation is observed (i.e.low battery level, decreased received packet signal strength), Backup nodes may be assigned toPrimary mode based on policy.2. Network Management:2.1 Transparent radio interface: Using the message header which contains information about mes-sage type, the VCM determines which task should be informed about message arrival. Messagescontaining tasks and their parameter denitions are rst processed by a VCM, before the VCMactivates the Interpreter (for remote ports). All local task-specic messages are transferred totheir respective local ports.2.2 Logical-to-physical address mapping: All communication between VTs is via the VCM. Sincea VT does not have information on which nodes other VTs are deployed, the VCM performslogical-to-physical address mapping. In cases when both tasks are on the same node, the VCMdirectly passes a message to a receiving task buer.4 ImplementationTo evaluate the EVMs performance in a real setting with multiple coordinated controller opera-tions, we used a factory simulation module (FischerTechnik model factory). The factory consistsof 22 sensors and actuators that are to be controlled in a coordinated and timely manner. A blockof wood is passed through a conveyor, pushed by a rammer on to a turn table and operated uponby up to three milling/cutting/pneumatic machines. The factory module was initially controlledby wired programmable logic controllers (PLCs). We converted it to use wireless control withFireFly nodes controlling all sensors and actuators via a set of electical relays. FireFly [12] is alow-cost, low-power, platform based on the Atmel ATmega1281 8-bit micro-controller with 8KBof RAM and 128KB of ROM along with a Chipcon CC2420 IEEE 802.15.4 standard-compliantradio transceiver. FireFly nodes support tight global hardware-based time synchronization forreal-time TDMA-based communication with the RT-Link protocol [10]. The EVM also workson TI MSP-EXP430F5xxx and MSP430FG4xxx architectures for more ecient execution of theFORTH-like stack-based language on a Von Neumann architecture. We have demonstrated:1. On-line capacity expansion when a node joins the VC 11
  12. 12. 2. Redistribution of VTs when adding/removing nodes3. Planned VT migration triggered by the user4. Unplanned VT migration as a result of node failure or communication link failure5. Multiple coordinated work-ows In a second experiment setup we considered multiple concurrent workows, where each work-ow uses a subset of the actuators. Each new block type triggers the assignment of the associatedVT and migrates from one controller to the next as the block physically moves through the factory.Therefore, with every new type of block a reconguration of the VC is performed. We have testedthe setup with a batch of 9 input blocks consisting of 3 dierent types. This is an example of thelogical benets of the EVM as it enables a more agile form of manufacturing. Details on bothexperiments with videos can be seen in [13].5 Related workThere have been several variants of virtual machines, such as Maté [4], Syclla [14] and SwissQM[15],and exible operating systems, such as SOS[11], Contiki[16], Mantis[17], Pixie[18] and LiteOS[19],for wireless sensor networks. The primary dierences that set EVM apart from prior work is thatit is centered on real-time operation of controllers and actuators. Within the design of EVMsoperating system, link protocol, programming abstractions and operation, timeliness is a rst-classcitizen and all operations are synchronized. The EVM does not have a single node-perspectiveof mapping operations to one virtualized processor on a particular node but rather maintainscoordinated operation across a set of controllers within a virtual component. Furthermore, EVMhas been designed to operate in an autonomous manner and to be extensible at runtime. One of the rst virtual machines (VM) for sensor networks was Maté. Maté implementsa simple, communication-centric VM built on top of the TinyOS [3]. It is designed as a highlevel interface where code is written using limited instruction set, dened at design-time, andexecuted with a FORTH-like interpreter. EVM utilizes a similar FORTH-like interpreter but isextensible at runtime and allows for fully preemptive tasks. Syclla is a more conventional systemVM that allows code mobility by providing a virtualized processor abstraction on each node.It uses a standard approach where one physical machine exposes interfaces to a single logicalmachine. On the other hand, EVM uses several physical nodes and allows user to consider thevirtual component as a single logical entity. SOS operating systems for sensor nodes [11] wasdesigned to allow exible operation of nodes in a sensor network. As TinyOS, it has an event-drive execution of all components, with a dierence that components can be installed/modiedduring runtime. To allow this, SOS uses dynamically-loaded modules while kernel implementsmessaging, dynamic memory, and module loading and unloading. EVM allows for dynamic taskevocation and runtime function loading but in addition has mechanism for interaction with othernodes, which SOS lacks. In addition, SOS utilizes a signicant portion of the TinyOS codeand all tasks are run-to-completion. EVM is built on the nano-RK sensor RTOS and hence alltasks are scheduled by xed priority scheduling and are fully preemptable. Finally, the VirtualNode Layer [20] provides a programing abstraction for a virtual infrastructure where each virtualnode is identied with a particular region and it is emulated by one of the physical nodes in itsregion. In last few years several dierent systems for macro-programming in WSN have beendeveloped[21, 22, 23]. Welsh et al. [21] have dened a set of abstractions representing localcommunication between nodes in order to expose control over resource consumption along withproviding feedback on its performance. EVM along with data sharing incorporates performancetracking of algorithm execution and automatically readjust available resource in order to maximizesystem robustness. An extension of these ideas is used to develop the Regiment [23], a high-levellanguage based on the functional reactive programming. The Regiment uses two types of objects,Signals used to represent sensor states and Regions, collection of signals, where membership inRegion can vary in time. In addition the authors dene operators for creating new regions alongwith three basic operations (applying function to a signal and aggregating values in the region).EVM is not a generic macroprogramming system as it focuses on closed-loop control and actuation 12
  13. 13. problems with native support for task migration and on-line task assignment. Kairos [22] allowsa programmer to describe code execution for each of the nodes in a network using a centralizedapproach where details about code generation, remote data access and management along withnode interactions are hidden from the programmer. Kairos uses three constructs: remote dataaccess, iterating through 1-hop neighbors and addressing arbitrary nodes. Basic of Kairos isa data sharing algorithm which assumes eventual consistency between shared memories on thenodes. Although EVM provides consensus between shared memories store on dierent nodes witha shared memory access for all nodes in the networks its list of features is not only limited to that.References [1] Frost and Sullivan, North American Sensor Markets, Technical Report A-761-32, 2004. [2] Nielsen Research, Downtime Costs Auto Industry, March 2006. [3] J. Hill and et al. System architecture directions for network sensors. ASPLOS, 2000. [4] P. Levis and D. Culler. Mate: A tiny virtual machine for sensor networks. ACM ASPLOS-X, 2002. [5] J. P. Hespanha, P. Naghshtabrizi, and Y. Xu. A survey of recent results in networked control systems. Proc. of the IEEE, 2007. [6] W. Zhang, M.S. Branicky, and S.M. Phillips. Stability of networked control systems. IEEE Control Systems Magazine, 2001. [7] M. Pajic and R. Mangharam. The Assignemnt Problem. http://mlab.seas.upenn.edu/evm, 2009. [8] L. Brodie. Starting FORTH. 2nd Ed., Prentice-Hall. ISBN 0-13-843079-9, 1987. [9] nano-rk sensor rtos. http://nanork.org.[10] A. Rowe, R. Mangharam, and R. Rajkumar. RT-Link: A Time-Synchronized Link Protocol for Energy-Constrained Multi-hop Wireless Networks. IEEE SECON, 2006.[11] C.C. Han and et.al. SOS : A Dynamic Operating System for Sensor Nodes. ACM Mobisys, 2005.[12] R. Mangharam and et.al. FireFly: A Cross-layer Platform for Real-time Embedded Wireless Net- works. Real-Time System Journal, 2007.[13] EVM website - http://mlab.seas.upenn.edu/evm, 2009.[14] P. Marbell and L. Iftode. Scylla: A smart virtual machine for mobile embedded systems. In WMCSA, 2000.[15] R Muller, G Alonso, and D Kossmann. A virtual machine for sensor networks. ACM EuroSys, 2007.[16] A. Dunkels, B. Gronvall, and T. Voigt. Contiki - A Lightweight Operating System for Tiny Networked Sensors. IEEE LCN, 2004.[17] Shah Bhatti and et. al. MANTIS OS: an embedded multithreaded operating system for wireless platforms. Mob. Netw. Appl., 2005.[18] K. Lorincz and et. al. Resource aware programming in the Pixie OS. ACM SenSys, 2008.[19] Q. Caoand and et. al. The LiteOS: Towards Unix-Like Abstractions for Wireless Sensor Networks. IEEE IPSN, 2008.[20] M. Brown, S. Gilbert, N. Lynch, C. Newport, T. Nolte, and M. Spindel. The Virtual Node Layer: A Programming Abstraction for Wireless Sensor Networks. WWSNA, 2007.[21] M. Welsh and G. Mainland. Programming Sensor Networks using Abstract Regions. NSDI, 2004.[22] R. Gummadi et al. Macro-programming Wireless Sensor Networks Using Kairos. In Dist. Comp. in Sensor Sys., 2005.[23] R. Newton et al. The Regiment Macroprogramming System. ACM IPSN, 2007. 13