JPD Developing Distributed Multimedia Applications

1. 1.0 Abstract Distributed multimedia applications consist of a mixture of elements scattered at different locations on a network. Each element on the network has different transmission needs and each link of the network has different transmission characteristics. Just as an orchestra conductor must match the characteristics of instruments to the performance skills of the musicians - then orchestrate the piece according to their location in the orchestra and the acoustic properties of the hall. So too, a multimedia orchestration service must take A/V elements, match them to the A/V servers with appropriate responsiveness, and distribute the elements to appropriate locations on the network. To do this, we have created an orchestration service that integrates and centralizes the orchestration task thereby relieving the individual elements from being aware of how they are being composited to form a combined application and hopefully also leading to globally optimal and balanced networks. Keywords: orchestration, synchronization, compound documents, distributed software architectures CircusTalk: An Orchestration Service for Distributed MultiMedia Yechezkal-Shimon Gutfreund, Jose Diaz-Gonzalez, Russell Sasnett, Vincent Phuah Distributed Multi-Media Applications Project GTE Laboratories, Inc. 40 Sylvan Road Waltham, MA 02254 USA sgutfreund@gte.com, jdiaz@gte.com, rsasnett@gte.com, vphuah@gte.com 2.0 Problem Overview Distributed multimedia applications consist of a mixture of elements distributed over a network. For example, in figure 1, we show a collaborative media-space [3] where two scientists are conducting a joint transcontinental experiment. One scientist has a high-resolution SEM microscope, the other provides the NMR scanner. Both are producing real-time video which they are also processing in real-time. The results of the image processing is being used to drive a real-time simulation which is providing a paral- lel representation of the results. They will be sharing the video, but having separate simulators and renderers so that they can view different aspects of the simulation. In order to build this system, processing tasks (e.g. image processing) will have to be matched to appropriate compute servers, input/output data flows will have to be characterized, and appropriate network connections established. However, this binding cannot be static. Loading changes on the compute servers and changes in traffic flow patterns on the underlying ATM network must be constantly monitored. In response to load changes, alternative virtual connections or alternative compute servers may have to be re-scheduled to maintain the QoS guarantees. From this specific example, we can create a general statement of the multimedia orchestration problem. Distributed multimedia applications consist of a set of elements. Elements can act as either Video Camera Scientist #1 Scientist #2 Image Processor NMR Scanner Particle Injector Simulator Volume Renderer Object Tracker IsoSurface Extractor SEM Microcope Renderer Video Camera Image Processor Simulator Renderer Figure 1: A Transcontinental MultiMedia Collaboration

2. sources, sinks, or transformational elements. An element can be either continuous media (e.g. video camera, VCR, or audio/video file), or a computational element (e.g. an image extraction routine, scientific visualization simulation, or compression algorithm). These elements are then “wired” together to form compound multimedia applications. Each element is a generic anonymous service, and each connection is a virtual path. Therefore, there can be multiple redundant copies of elements at different points on the network. Furthermore, elements are not aware of who they are “connected” to. Instead, an independent orchestration service ex- ists that handles the binding, translation, and synchronization functions needed to coordinate these independent elements. It is the orchestration server that monitors network loads, host loads, and other factors to decide dynamically which of the generic elements to bind and how they are to be coordinated. This frees the elements from having to be “network aware” and allows us to implement uniform coordination and scheduling algorithms between the elements. 3.0 Scope of Problem Within the general multimedia orchestration problem, there are many specific orchestration issues that have to be addressed. We see four aspects of orchestration that have to be provided by an orchestration server. In order of ascending complexity they are: binding, trading, translating, and coordinating. 3.1 Binding Binding refers to the formation of static connections between distributed elements. Elements can post fixed attributes lists with the orchestration server that describe the desired attributes of their sources and sinks. These attributes can describe QoS characteristics such as throughput and end-to-end latency, as well as type (e.g. a compression service) or format (e.g. Px64) of the stream. To perform binding the orchestration server uses a matching algorithm to determine if source and sink match and if a link should be established. Examples of binding mechanisms can be found in OSF’s DCE [22], OMG’s CORBA [16], Microsoft’s OLE [21], and TE- NET’s channel reservation protocol [1]. 3.2 Trading For some compound multimedia applications static binding will be sufficient. However, for other applications a more dynamic inter- connection mechanism is required. Traders are autonomous processes that exist inside the orchestration server. Information on host loading, network connectivity, line error rates, and network traffic load is continuously being posted to the orchestration server. A trader uses this information together with the requests from sink and source elements to make connections. If a match cannot be made, the trader can enter into negotiation with the requesting elements to get them to modify their requests. Changes in connectivity, load, or line errors, can awaken a trader to renegotiate the terms of connection between elements. For example, additional load could cause a request for addition money to pay for same grade of connection. Examples of traders can be found in ODP [13], ANSA [14], OSF’s DME [17], and TINA [24]. 3.3 Translating Since we are dealing with autonomous elements that are unaware of their input or output connections, the orchestration server must sometimes perform translation services between elements. The orchestrator can either do this itself, or it can bind a filter package between the source and sink sides of a connection. For example, a filter can compress, decompress, or make format changes on a stream. Another aspect of translation involves the interactive controls (widgets) that come with the off-the-shelf application elements. Frequently, one would like to bind the buttons, sliders, and knobs from different packages into one master control switch. Al- ternatively, one might like to have one unified user interface that transparently drives all the component multimedia elements. For example, figure 1 depicts a collaborative scientific experiment. It is quite likely that the scientists will want to use off-the-shelf programs for the simulator, image processors, renderers, etc. Each of the packages might come with it own peculiar GUI. We would like to provide one unified GUI (call it a remote control panel) and then use the orchestrator to map widget actions on the remote control panel to multiple corresponding actions on the underlying multimedia elements. Another aspect of translation is the ability to replace or upgrade elements dynamically. Thus, by means of the translation service in the orchestrator we can replace the underlying elements, keep the current interface, and only rewrite the mapping and translation rules in the orchestrator. Alternatively, we could replace our remote control panel. Instead of an interactive GUI we could record GUI actions and play them back, or replace the entire GUI with speech or audio events. For example, one could insert a filter into an audio stream to locate speech events and use these to drive a VCR. 3.4 Coordinating Finally, there is a need for the orchestration server to provide synchronization services. However, hard synchronization of multimedia elements will not be achievable until all components of the operating system (X-windows, file system, network driver, OS ker- nel, etc.) are capable of providing real-time guarantees of service. Until then we see the orchestration server providing a weak form of synchronization we call coordination. That is, events such as start of audio, start of video, or the next frame of a rendered graph- ic can be posted to the orchestration server. In response to these events, coordination agents running in the server can test for com- binations of events and signal multimedia elements. Since the orchestration server is independent of the multimedia elements, we can add and replace coordination controls without modifying the underlying elements that are being controlled. Thus we could replace the button event that signals “step to next frame” on a movie viewer with an event from the video stream (e.g. gesturing with one’s hand). 4.0 Related Work Currently, these orchestration problems are being tackled as separate issues. Synchronization and rate reservation schemes have been explored at Lancaster [4,6] and at Berkeley’s TENET [1]. Ar- chitectures for expressing synchronization have be suggested by the ISIS group [15], the NESTOR project [2], and SUN [20]. Trad- er concepts are being looked into as part of the Berkcom project Y

3. [19], ANSA [14], OSF’s DME [17], and TINA [24]. Coordinating and creating compound documents is also an issue in general (i.e. non-multimedia) computing. Work such as HP’s NewWave [12], OSF’s GAIA [18], MicroSoft’s Cairo [8], and OMG’s CORBA [16] present models for the integration and composition of distributed multimedia applications. Given the current popularity of multimedia computing, one should expect to see these standards expanded to include multimedia data types. However, none of these schemes deal with all the orchestration issues (mentioned in section 3.0) in a single unified approach. We believe that there is enough interaction between these types of orchestration issues that one unified high-level mechanism and ab- straction for orchestration will be more useful than a piecemeal approach or one that separates each of these orchestration functions into a separate API layer. 5.0 Approach It is our belief that the basket of orchestration issues (section 3.0) should not be handled in a piecemeal approach. Just as an orchestra needs a single conductor, so do the “performers” in a distributed multimedia ensemble. In an orchestra, there is a single centralized decision maker who makes the scheduling, layout,1 and arrangement decisions. Global issues of tonal balance, matching layout to the hall’s acoustics, and balancing volume across the different pieces of the orchestra is not done on a piece-by-piece ba- sis. Rather, there is a single centralized point of control where these decisions are carried out. Nor is each orchestration issue handled separately or sequentially. Instead, due to the interaction between orchestration issues, they are all balanced within one sophisticated rule making system. In the Circus multimedia environment we have tried to employ the same principles in the construction of our orchestration server. Thus our approach is characterized by: 1. The conductor has some, but not unlimited, freedom to change the layout of the orchestra. • A single logical orchestration server.2 • A software content addressable memory for storage of network events and connection information. • A single language and metaphor for dealing with the entire basket of orchestration issues (i.e. not a piece-meal approach of separate system calls).3 • A general-purpose interpretive language for writing orchestration scripts. • Powerful pattern-directed search primitives for locating patterns of information in the content addressable store. In figure 2 we present the architecture of our system: The Circus Orchestrator is implemented as a blackboard system. It has a large software implemented content addressable memory called TupleSpace.4 Information from applications is posted as tuples to TupleSpace. Applications do not have to be specially written to be orchestrated. Each application is wrapped with a special Circus library. In this manner Circus allows for the incorporation of commercial off the shelf (COTS) applications. Wrapping is ac- complished by replacing the dynamically bound Xt library with a special library of our own. In this library we have special routines to: trap X-events, trap widget callbacks, modify applications variables, and change X resources. The library executes scripts written 2. Our implementation is based on a single physical orchestrator. However, the architecture is flexible enough to allow the implementation to be done via an ensemble of orchestration servers. 3. In [7] Gelernter and Carriero note that languages such as Algol left all interaction issues (communication to users, objects in the file system, inter- program communication, and inter-processes coordination) out of the language specification. The result has been an ad-hoc piecemeal approach to performing these coordination tasks. They call for the creation of an Anti- Algol to present a unified approach for dealing with coordination issues. The CircusTalk language can be viewed as our attempt to provide such a facility for orchestrating distributed multimedia ensembles. 4. We have borrowed the idea of tuple spaces from the Linda language [5] and enhanced it. Application Orchestrator Tuple Space Script List Figure 2: Architecture of Circus Wrapper Application Wrapper Script Editor Persistent Object Store

4. in a special purpose interpretive language.5 This scheme allows us to write scripts that can peer into applications, notice widget events, X-events, and application events, and post them to the Or- chestrator. The Orchestrator itself has two components: a TupleSpace and a ScriptList. TupleSpace is a large software-based content addressable memory. The ScriptList is a sequential list of rules. When tuples are posted to the TupleSpace, the ScriptList is scanned, and all applicable rules are executed. The result of execution of a rule may result in sending zero or more messages to the wrapped applications. Scripts consist of two parts: (1) a head in the form of a pattern- template, (2) a body in the form of an action-block of CircusTalk commands. When a tuple is posted to the Orchestrator, scripts whose head/pattern match the form of one of the tuples will have their body executed. Patterns can specify specific tuple values, ranges of values, apply arbitrarily complex test operations, or look for patterns of values across tuples. This pattern matching capacity is crucial to discovering and locating simultaneous events occur- ring in the network (as reflected in TupleSpace). The body of the script consists of CircusTalk commands. CircusT- alk is a general purpose language built on top of Mathematica [25]. It contains commands for testing, iteration, functional & logic programming and object oriented programming. It is a very expres- sive language capable of representing a wide variety of complex orchestration services. Clients can either use the standard orchestration scripts or post their own. Scripts can be dynamically posted by the elements, uploaded from the persistent object store, or interactively edited via the Script Editor, even while the Orchestrator is running. We believe this framework provides us with a very flexible platform to explore aspects of orchestration without having to commit to any particular fixed approach. 6.0 An Introductory Example One of the common usages for the Orchestrator is to translate an X-widget action (e.g. button press) on one client, into an X-widget action on a different client. By writing the appropriate script in the Orchestrator one can transform one type of action (e.g. button press) into a different kind of action (e.g. slider motion). In the following example we have two X-clients each with a slider gauge. When the slider of one of the clients goes up, we want the slider of the other client to move down (and vice-versa). The code to implement this is as follows: void slideOut(widget, client, call) Widget widget; caddr_t client; XmScaleCallbackStruct *call; { char tuple[100]; sprintf(tuple, “{”slider”, %d}”, call->value); 5. The details of the Application Wrapper will be described in a forthcom- ing paper. postTuple(tuple); } This is the slider callback procedure. It is called each time the slider bar is moved. The Xt call to establish this callback is: XtAd- dCallback(scaleOut,XmNvalueChangedCallback, slideOut, NULL). Each time this callback is entered it cre- ates a string form of a tuple: {“slider”, value}. The tuple has two values. The first is the string: “slider”. The second is the integer value of the sliders position (0-100). The tuple is then posted to the Orchestrator. {char script[100] = “[{“slider”,val_}] := (ReturnTuple[{100-val}]; RemoveTuple[])” addScript(script, slideIn, NULL); } This piece of code adds a script to the Orchestrator’s ScriptList. It only needs to be run once at start-up. The first argument of the addScript function is the string form of an Orchestrator script. The second argument is the address of a function that will receive the results of this script. The script itself consist of two parts, a head: [{“slider”, val_}] and an action: (ReturnTuple[{100-val}]; Re- moveTuple[]). The head of the script gives a pattern-template for the tuple to match. When new tuples are posted to the Orchestrator, the Or- chestrator will scan its script list to see if pattern-template in the head of any scripts match a tuple. If so, it will then execute the action of the script. In this case, the pattern-template says that the tuple should have a string of the form “slider” as the first argument. The second argument is the wildcard val_ which gets bound to whatever the second value of the tuple is. The action of the script consists of two CircusTalk statements. The first statement reflects the value w.r.t. 100 and returns this in the form of a tuple to scripts callback. In our example the slideIn function that was specified in the addScript call will receive this tuple. The second CircusTalk statement removes the tuple from the Or- chestrator’s TupleSpace so that no other scripts will be able to match it. void slideIn(tupleString, dispatch) char *tupleString; int dispatch; { int value; sscanf(tupleString, “{%d}”, &value); XmScaleSetValue(scaleIn,value); } This routine receives values from the script that was added to the Orchestrator. Tuples arrive encoded in string form. They are scanned and the value is used to set the value of the slider widget. attach(“NONE”); detach(“NONE”); Finally, we need a mechanism to establish and tear-down the connection to the Orchestrator. This is done by the attach and detach routines.

5. 7.0 CircusTalk CircusTalk is an interpretive language built on top of Mathematica. Mathematica is a general-purpose language in which one can program in a variety of paradigms: procedural, functional, logic, transformational, or object-oriented. It has extensive facilities for pattern matching and for rule-based transformations which were key to the implementation of many CircusTalk features. CircusTalk is typically used to create scripts. A script has two parts: a head (which consists of a pattern-template) and a body (which consists of an action-block). The following is in an example of a script: [{“slider”,val_}] :=(ReturnTuple[{100-val}] RemoveTuple[]) The head of this script is: [{“slider”,val_}], and the body is: (ReturnTuple[{100-val}]; RemoveTuple[]). 7.1 Pattern-Templates Pattern-Templates all have the following form: [{pattern1, pattern2, ..., patternN}] The Orchestrator uses the pattern-template to determine if a script should execute. If each of the values in a tuple match the corresponding patterns in the pattern-template (with no extras) then that pattern-template is said to match. Patterns can have a variety of forms. The simplest is a literal (e.g. 10, or “widget30”). Literal patterns must exactly match the corresponding value in the tuple. A second kind of pattern is a variable pattern. For example the variable pattern x_ will match any value and assign the value to the variable x. One can restrict the binding of variable patterns by qualifying it with a type. The pattern x_Integer will only match a value of type Integer while the pattern x_String will only match a string. One can also use the simple pattern _ to universally match any value, and not bind the value to any variable. Tests can also be applied in a pattern. The pattern x_ ? OddQ will only match an odd value while the pattern x_ ? VectorQ will only match a vector. Conditions can also be applied to patterns. The pattern x_ /; x < 34 will only match a value which is less than 34. The pattern x_ /; 10< x <= 100 looks for a value in the range (10, 100]. Conditions can also be applied to the entire tuple. For example the pattern-template [{a_Integer, b_Integer} /; Mod[a, b] == 0] will only match tuples with two values which are modulo each other. One can also use compound statements for conditions. In this way one can turn conditions into small programs. These programs can be used to perform more complex tests, or can be used look for simultaneous events (tuples) in tuple space. For further information on patterns one should read section 2.3 of the Mathematica Book [25]. 7.2 Action-Blocks An action block has the following form: (statement1; statement2; ...; statementN) Action-Blocks are used by the Orchestrator in Attach and Detach actions as well as for the body of a script. In Attach and Detach commands the action block executes unconditionally. While in a script the action-block only executes if the pattern-template in the head of the script matched a tuple. When a pattern-template matches a tuple the Orchestrator will perform the following actions: 1. Bind values in the tuple to the matching variable names indi- cated in the pattern-template. 2. Assign the variable TUPLE to the value of the entire tuple. 3. Assign the variable CLIENT to the host-address of the machine that posted this script (which will receive the results of any ReturnTuple[] statements). 4. Assign the variable CALLBACK the address of the callback routine of the CLIENT. 5. Assign the variable DISPATCH the value of the dispatch argument that was provided in the addScript call. 6. Execute the statements of the action-block in sequential order (unless there are conditionals or iterative blocks within the action-block). The Attach and Detach action-blocks operate in the same manner, however the only variable that gets bound is the CLIENT variable. There is an alternative form of the action-block: Block[{var1, var2, ..., varN}, statement1; statement2; ...; statementN] In this form one can have local variables whose scope is limited to the scope of the action-block. In this example the local variables are var1, var2, varN. One can give default values to the local variables. For example one can state var1=Pi to give var1 the initial value of Pi. 7.2.1 CircusTalk statements ReturnTuple[] is the command for sending a tuple back to the client program that originally posted this script. For example, the statement ReturnTuple[{“test”, 5}] in an action block will return the tuple {“test”, 5} to the client program at its callback function address. The callback function will receive a string form of this tuple. An alternate form of the ReturnTuple[] statement is ReturnTupl- e[tuple, client, callback, dispatch]. In this form one can send the tuple to any client or callback that one wishes. That is, one is not restricted to sending the tuple only to the client that posted the script. This feature can be used to implement multi-casts of events from the Orchestrator. TupleSpace itself is maintained as a list of tuples kept in the variable TS. For example, if the value of TS is { {“string1”, 4}, {“wid-

6. get1”, 50} } then TupleSpace has two tuples in it. Individual tuples can be addressed in scripts. In our case, TS[[1]] is the tuple {“string1”, 4}. One can also access individual values in the Tuple. For example TS[[2,2]] is 50. CircusTalk gives additional mechanisms to access tuples. The FindTuple[pattern-template] can be used to return the index of the tuple that matches the pattern-template (or a list of indexes if more than one matches). This index can then be used to access the tuple (TS[[index]]) or it can be used by RemoveTuple[index] to remove the tuple from TupleSpace. RemoveTuple[TUPLE] is the mechanism for removing the tuple that matched the head of this script. Tuples can also be added and replaced in TupleSpace. The Ap- pend[TS, tuple] will add a tuple to TupleSpace and the ReplaceTu- ple[tuple] will replace the current tuple, TUPLE. Much of the power of the CircusTalk language comes from the fact that it is built upon the Mathematica programming language. Mathematica provides extensive facilities and functions for functional programming, object-oriented programming, logic programming, pattern recognition, and procedural programming. We cannot detail all the features of Mathematica in this short document, but we can give a few short examples. For a more complete treatment one should read the Mathematica Book [25], especially section 2, or one of the numerous (and well written) books that give an introduction to Mathematica. One of the functions that frequently gets used in CircusTalk is the If[] command. If[expression,TRUE, FALSE] evaluates the expression and if true executes the statements in the TRUE clause other- wise it will execute the statements in the FALSE clause. The Map statement can be used to apply an operation over every item in a list. For example, Map[ Log, list] will take the Logarithm of every item in the list. The statement test1 = 2+4; is an example of a Set[] command which sets the variable test1 to the value 2+4 (i.e. 6). 8.0 Internals & Performance 8.1 Trace of Execution When a client program wishes to communicate with the Orchestra- tor the following steps occur: 1. The client program makes a call to the API (e.g. PostTuple(), or AddScript()). 2. The API sends a message via ToolTalk [23] to the Orchestra- tor. 3. The message is received by the “C” program Orchestrator.c. This program runs as an independent process on the host indi- cated by the Orchestrator environment variable. 4. This program, in turn, sends a message (via Unix sockets) to Mathematica. 5. A Mathematica package (called Orchestrator.m) receives the message via the MathLink protocol. 6. If there is a result of the operation, the Orchestrator.m package will send it back to the Mathematica.c program which in turn will send a message back to the API via ToolTalk. Tooltalk is capable of translating data types between machine, starting the Orchestrator process if it is not already running, guar- antee of message delivery across the network, and queueing of messages that arrive until they need to be read. 8.2 Performance Given all the communications and the use of the non-compiled (in- terpreted) CircusTalk language one might be concerned about performance issues. However, we have found that performance on a SparcStation1 is quite adequate for our needs. That is, we can process approximately 60 messages per second between SparcSta- tion1s.6 Performance actually improves when we distribute the different components to separate machines, since both ToolTalk and Mathematica are CPU bound and having them run on separate machines gives us more CPU cycles. Currently we are not attempting to do any hard synchronization with the Orchestrator. However, we are now upgrading to a network of SparcStation10s connected to an 8-port 155Mb/s Fore systems ATM switch. On such a platform we would like to attempt some more “real-time-like” synchronization. The SparcStation10 will probably boost performance of the current system to 240 Mes- sages/Sec. However, at a later time, we may wish to re-visit our decision to have a single orchestration server and to base the orchestrator on top of Mathematica. 9.0 Experiences Working with the Orchestrator has been tremendous fun. We have been able to play with working ensembles of distributed multimedia applications. Using the Orchestrator we can “play” with the control panel widgets of remote programs, create pseudo control panels, make an ensemble of programs “play” accompaniment to a lead program (i.e. a distributed Karoke system), and intercept mail messages. Bugs in CircusTalk scripts have also been interesting. One bug involved forgetting to remove a tuple from TupleSpace after it had been processed. This resulted in the application running in something akin to “player piano” mode. We strongly recommend to other designers of orchestration services to use interpretive scripts to explore orchestration issues. We were often able to rewire the control structure of an ensemble of programs in less than ten minutes, due to the fact that CircusTalk is an interpretive language. We also feel that having a central server is a big advantage when conducting exploratory reseach into orchestration issues. If we had to distribute the coordination code into the various applications (such as one would do with CORBA [16] or OLE [21]) then there would be multiple places to modify code to change the coordination characteristics of our test programs. With the Orchestrator we can carry out and see all the ramifications of making orchestration changes - since all this control information is centrally located. 10.0 Future Work Our work with the Orchestrator has really only begun. For example, in the distributed Karoke system we use a control panel on one 6. This figure is based on using ONC-RPC as the IPC mechanism. Our ear- ly experience with ToolTalk taught us that ToolTalk can only deliver 20 Messages/sec. Therefore, we do not plan to use ToolTalk in future versions.

7. system to drive a suite of programs on other machines which are supposed to play in harmony with the “lead” system. We do this by posting the GUI events from the “lead” control panel to the Or- chestrator, which then relays these events (via scripts) to the systems keeping harmony. This corresponds to the translation level service we described in section 3.3. Obviously this needs to be ex- tended to take into account the trading (section 3.2) and coordination (section 3.4) issues. What we would like to do is have network-based applications (such as nfs, ftp, etc.) place bandwidth request information to the Orchestrator. In addition traffic monitoring programs would keep track of instaneous network traffic such as that from X-windows programs. We would also have load monitoring programs that would keep track of CPU loading at the hosts. All this information would be posted to the Orchestrator at regular intervals. The Orchestrator would use this information to provide a super- scheduling service for the distributed Karoke system. It would di- rect the audio, video, and GUI data from the “lead” system to the distributed “harmony” systems. The harmonizing systems would merge this data with their own to produce composite audio, video, and graphics.7 This would be a “super-scheduler” in that most plans for schedul- ers for ATM circuit set-up do not incorporate such detailed information about the continous streams that they control. In our case, we would be providing the circuit scheduling algorithm with detailed information about the streams and, in addition, information about host loading - which we believe is also important in the delivery of smooth continous A/V information on the user’s workstation. That is, the problem does not end at the network interface but most also incorporate information about the actual ability of the workstation to deliver the A/V stream to the user. 11.0 Conclusions It is our belief that coordination, control, translation, and orchestration issues are going to become major topics in the creation of distributed multimedia applications. Each element of a multimedia application will have different features and performance. In order to bind together these elements one will have to carefully choose among the potential network multimedia elements and qualify their usage by their type, performance, and the QoS characteristics of the transport links that connects them. We see the issues of binding, coordination, translation, and orchestrator as being inherently linked issues. In the Circus Orchestrator we provide a a single architecture and platform where one can create orchestration services that can deal with these linkages. In addition, the Orchestrator mechanism allows one to have one central repository where all information about the network, processors, available applications, and translation needs can be centrally stored (i.e. in TupleSpace). The scripting facility allows all the decisions about coordination and orchestration to be performed lo- cally on the data. Not only can this lead to better performance and more timely re-assessment of binding decisions, but it also frees the individual multimedia network elements from having to be aware of how they are bound together in an application. 7. Audio and video data would not pass through TupleSpace, but through independent channels whose QoS characterisics are dynamically controlled by the Orchestrator. The Orchestrator provides a very simple metaphor and model for handling coordination and orchestration needs. It is basically a large content addressable memory (TupleSpace) coupled to a rule based system (the ScriptList). Although the basic model is simple, it is also very powerful. This power stems largely from the general purpose language (CircusTalk) and its extensive libraries for doing pattern-based programming, object-oriented programming, and procedural programming. 12.0 References 0. 1. Anindo Bannerjea and Bruce A. Mah. The Real-Time Channel Administration Protocol. In Network and Operating System Support for Digital Audio and Video, Second Interna- tional Workshop, Heidelberg Germany, November 1991. Springer-Verlag 1992. 209--216. 2. Gerold Blakowski, Jens Hubel, Ulirike Langrehr, and Max Muhlhauser. Tool Support for the Synchronization and Presentation of Distributed MultiMedia. In Computer Com- munications, Vol. 15, No. 10 (Dec. 1992). 611--618. 3. Sara A. Bly, Steve R. Harrison, and Susan Irwin. Media Spaces: Bringing People Together in a Video, Audio, and Computing Environment. In Communications of the ACM. Vol. 36, No. 1 (Jan. 1993). 28--47. 4. Andrew Campbell, Geoff Coulson, Fancisco Garcia, and David Hutchison. A Continuous Media Transport and Or- chestration Service. In Computer Communications Review. Vol. 22, No. 4 (Oct 1992). SIGCOMM’92 Conference Pro- ceedings. 99--110. 5. Nicholas Carriero and David Gelernter. How to Write Par- allel Programs: A First Course. MIT Press, Cambridge MA. 1990. 6. Geoff Coulson, Francisco Garcia, David Hutchison, and Doug Shepard. Protocol Support for Distributed MultiMedia Applications. In Network and Operating System Support for Digital Audio and Video, Second International Workshop, Heidelberg Germany, November 1991. Springer-Verlag 1992. 45--56. 7. David Gelernter and Nicholas Carriero. Coordination Lan- guages and their Significance. In Communications of the ACM. Vol. 35, No. 2 (Feb. 1992). 97--107. 8. Ed Foster. Microsoft’s Cairo takes us right into the heart of darkness. In InfoWorld, Vol. 14, No. 10 (Mar. 9, 1992). page 58. 9. Yechezkal-Shimon Gutfreund, Jose Diaz-Gonzalez, Rus- sell Sasnett, and Vincent Phuah. Dynamicity Issues in Broadband Network Computing. In Network and Operating System Support for Digital Audio and Video, Second Interna- tional Workshop, Heidelberg Germany, November 1991. Springer-Verlag 1992. 209--216. 10. Yechezkal-Shimon Gutfreund. ThinkerToy: An Environ- ment for Decision Support. Ph.D. Thesis. Computer and In- formation Sciences Department, COINS Technical Report 88-72. 1988.

8. 11. Yechezkal-Shimon Gutfreund. ManiplIcons in ThinkerToy. Proceedings of OOPSLA’87 Object Oriented Programming Systems, Languages, and Applications. Norman Meyrowitz editor. October 4-8 1987. Kissimee, Florida. SIGPLAN Notic- es, Vol. 22, No. 12, (December 1987) 307--317. 12. Hewlett-Packard. HP NewWave General Information Manu- al. Hewlett-Packard. 1990. 13. ISO, ODP. Working Document on Topic 9.1 - ODP Trader. In ISO/IEC JTC1/SC21/WG7 and SGVII.Q16 (ODP) N645, N737. 27 November, 1992. 14. David S. Marshak. ANSA: A Model for Distributed Com- puting. In Network Monitor: Guide to Distributed Comput- ing. Vol. 6, No. 11 (Nov. 1991) 15. Keith Marzullo, Mark Wood, Robert Cooper, Kenneth Birman. Tools for Distributed Application Management. In IEEE Computer, Vol. 24, No. 8 (Aug. 1991). 42--51. 16. OMG. The Common Object Request Broker: Architecture and Specification. Object Management Group, Framingham, MA. 1991. 17. Open Software Foundation. OSF Distributed Management Environment (DME) Architecture. Open Software Founda- tion, Cambridge, MA. 1992. 18. OSF. GUI-based Application Interoperability Architecture (GAIA). In GAIA Workshop Proceedings. August 18-19. Open Software Foundation, Cambridge, MA. 1992. 19. R. Popescu-Zeletin, V. Tschammer, and M. Tschichholz. ‘Y’ Distributed Application Platform. In Computer Commu- nications, Vol. 14, No. 6 (July 1991). 366--374. 20. J. Duane Northcutt and Eugene M. Kuerner. System Sup- port for Time-Critical Applications. In Network and Operat- ing System Support for Digital Audio and Video, Second International Workshop, Heidelberg, Germany, November 1991. Springer-Verlag 1992. 242--254. 21. Bruce D. Schatzman. Next-generation OLE: OLE 2.0 is fast- er and simpler for users,but it’s trickier for developers. Byte Vol. 18, No. 3 (March 1993). 209-211. 22. John Shirley. Guide to Writing DCE Applications. O’Reilly & Associates, Inc. Sebastopol, CA. 1992. 23. SunSoft. ToolTalk 1.0 Programmer’s Guide. SunSoft, Moun- tain View, CA. 1991. 24. TINA. Tina Consortium Meeting Notes. For meeting notes contact Harvey Rubin at Bellcore 1-908-758-2223 or ti- na93@rs.sip.stet.it. 1992. 25. Stephen Wolfram. Mathematica: A System for Doing Math- ematics by Computer. 2nd Edition. Addison-Wesley, Read- ing, MA. 1991.

13. 7.0 CircusTalk CircusTalk is an interpretive language built on top of Mathematica. Mathematica is a general-purpose language in which one can program in a variety of paradigms: procedural, functional, logic, transformational, or object-oriented. It has extensive facilities for pattern matching and for rule-based transformations which were key to the implementation of many CircusTalk features. CircusTalk is typically used to create scripts. A script has two parts: a head (which consists of a pattern-template) and a body (which consists of an action-block). The following is in an example of a script: [{“slider”,val_}] :=(ReturnTuple[{100-val}] RemoveTuple[]) The head of this script is: [{“slider”,val_}], and the body is: (ReturnTuple[{100-val}]; RemoveTuple[]). 7.1 Pattern-Templates Pattern-Templates all have the following form: [{pattern1, pattern2, ..., patternN}] The Orchestrator uses the pattern-template to determine if a script should execute. If each of the values in a tuple match the corresponding patterns in the pattern-template (with no extras) then that pattern-template is said to match. Patterns can have a variety of forms. The simplest is a literal (e.g. 10, or “widget30”). Literal patterns must exactly match the corresponding value in the tuple. A second kind of pattern is a variable pattern. For example the variable pattern x_ will match any value and assign the value to the variable x. One can restrict the binding of variable patterns by qualifying it with a type. The pattern x_Integer will only match a value of type Integer while the pattern x_String will only match a string. One can also use the simple pattern _ to universally match any value, and not bind the value to any variable. Tests can also be applied in a pattern. The pattern x_ ? OddQ will only match an odd value while the pattern x_ ? VectorQ will only match a vector. Conditions can also be applied to patterns. The pattern x_ /; x < 34 will only match a value which is less than 34. The pattern x_ /; 10< x <= 100 looks for a value in the range (10, 100]. Conditions can also be applied to the entire tuple. For example the pattern-template [{a_Integer, b_Integer} /; Mod[a, b] == 0] will only match tuples with two values which are modulo each other. One can also use compound statements for conditions. In this way one can turn conditions into small programs. These programs can be used to perform more complex tests, or can be used look for simultaneous events (tuples) in tuple space. For further information on patterns one should read section 2.3 of the Mathematica Book [25]. 7.2 Action-Blocks An action block has the following form: (statement1; statement2; ...; statementN) Action-Blocks are used by the Orchestrator in Attach and Detach actions as well as for the body of a script. In Attach and Detach commands the action block executes unconditionally. While in a script the action-block only executes if the pattern-template in the head of the script matched a tuple. When a pattern-template matches a tuple the Orchestrator will perform the following actions: 1. Bind values in the tuple to the matching variable names indi- cated in the pattern-template. 2. Assign the variable TUPLE to the value of the entire tuple. 3. Assign the variable CLIENT to the host-address of the machine that posted this script (which will receive the results of any ReturnTuple[] statements). 4. Assign the variable CALLBACK the address of the callback routine of the CLIENT. 5. Assign the variable DISPATCH the value of the dispatch argument that was provided in the addScript call. 6. Execute the statements of the action-block in sequential order (unless there are conditionals or iterative blocks within the action-block). The Attach and Detach action-blocks operate in the same manner, however the only variable that gets bound is the CLIENT variable. There is an alternative form of the action-block: Block[{var1, var2, ..., varN}, statement1; statement2; ...; statementN] In this form one can have local variables whose scope is limited to the scope of the action-block. In this example the local variables are var1, var2, varN. One can give default values to the local variables. For example one can state var1=Pi to give var1 the initial value of Pi. 7.2.1 CircusTalk statements ReturnTuple[] is the command for sending a tuple back to the client program that originally posted this script. For example, the statement ReturnTuple[{“test”, 5}] in an action block will return the tuple {“test”, 5} to the client program at its callback function address. The callback function will receive a string form of this tuple. An alternate form of the ReturnTuple[] statement is ReturnTupl- e[tuple, client, callback, dispatch]. In this form one can send the tuple to any client or callback that one wishes. That is, one is not restricted to sending the tuple only to the client that posted the script. This feature can be used to implement multi-casts of events from the Orchestrator. TupleSpace itself is maintained as a list of tuples kept in the variable TS. For example, if the value of TS is { {“string1”, 4}, {“wid-

14. get1”, 50} } then TupleSpace has two tuples in it. Individual tuples can be addressed in scripts. In our case, TS[[1]] is the tuple {“string1”, 4}. One can also access individual values in the Tuple. For example TS[[2,2]] is 50. CircusTalk gives additional mechanisms to access tuples. The FindTuple[pattern-template] can be used to return the index of the tuple that matches the pattern-template (or a list of indexes if more than one matches). This index can then be used to access the tuple (TS[[index]]) or it can be used by RemoveTuple[index] to remove the tuple from TupleSpace. RemoveTuple[TUPLE] is the mechanism for removing the tuple that matched the head of this script. Tuples can also be added and replaced in TupleSpace. The Ap- pend[TS, tuple] will add a tuple to TupleSpace and the ReplaceTu- ple[tuple] will replace the current tuple, TUPLE. Much of the power of the CircusTalk language comes from the fact that it is built upon the Mathematica programming language. Mathematica provides extensive facilities and functions for functional programming, object-oriented programming, logic programming, pattern recognition, and procedural programming. We cannot detail all the features of Mathematica in this short document, but we can give a few short examples. For a more complete treatment one should read the Mathematica Book [25], especially section 2, or one of the numerous (and well written) books that give an introduction to Mathematica. One of the functions that frequently gets used in CircusTalk is the If[] command. If[expression,TRUE, FALSE] evaluates the expression and if true executes the statements in the TRUE clause other- wise it will execute the statements in the FALSE clause. The Map statement can be used to apply an operation over every item in a list. For example, Map[ Log, list] will take the Logarithm of every item in the list. The statement test1 = 2+4; is an example of a Set[] command which sets the variable test1 to the value 2+4 (i.e. 6). 8.0 Internals & Performance 8.1 Trace of Execution When a client program wishes to communicate with the Orchestra- tor the following steps occur: 1. The client program makes a call to the API (e.g. PostTuple(), or AddScript()). 2. The API sends a message via ToolTalk [23] to the Orchestra- tor. 3. The message is received by the “C” program Orchestrator.c. This program runs as an independent process on the host indi- cated by the Orchestrator environment variable. 4. This program, in turn, sends a message (via Unix sockets) to Mathematica. 5. A Mathematica package (called Orchestrator.m) receives the message via the MathLink protocol. 6. If there is a result of the operation, the Orchestrator.m package will send it back to the Mathematica.c program which in turn will send a message back to the API via ToolTalk. Tooltalk is capable of translating data types between machine, starting the Orchestrator process if it is not already running, guar- antee of message delivery across the network, and queueing of messages that arrive until they need to be read. 8.2 Performance Given all the communications and the use of the non-compiled (in- terpreted) CircusTalk language one might be concerned about performance issues. However, we have found that performance on a SparcStation1 is quite adequate for our needs. That is, we can process approximately 60 messages per second between SparcSta- tion1s.6 Performance actually improves when we distribute the different components to separate machines, since both ToolTalk and Mathematica are CPU bound and having them run on separate machines gives us more CPU cycles. Currently we are not attempting to do any hard synchronization with the Orchestrator. However, we are now upgrading to a network of SparcStation10s connected to an 8-port 155Mb/s Fore systems ATM switch. On such a platform we would like to attempt some more “real-time-like” synchronization. The SparcStation10 will probably boost performance of the current system to 240 Mes- sages/Sec. However, at a later time, we may wish to re-visit our decision to have a single orchestration server and to base the orchestrator on top of Mathematica. 9.0 Experiences Working with the Orchestrator has been tremendous fun. We have been able to play with working ensembles of distributed multimedia applications. Using the Orchestrator we can “play” with the control panel widgets of remote programs, create pseudo control panels, make an ensemble of programs “play” accompaniment to a lead program (i.e. a distributed Karoke system), and intercept mail messages. Bugs in CircusTalk scripts have also been interesting. One bug involved forgetting to remove a tuple from TupleSpace after it had been processed. This resulted in the application running in something akin to “player piano” mode. We strongly recommend to other designers of orchestration services to use interpretive scripts to explore orchestration issues. We were often able to rewire the control structure of an ensemble of programs in less than ten minutes, due to the fact that CircusTalk is an interpretive language. We also feel that having a central server is a big advantage when conducting exploratory reseach into orchestration issues. If we had to distribute the coordination code into the various applications (such as one would do with CORBA [16] or OLE [21]) then there would be multiple places to modify code to change the coordination characteristics of our test programs. With the Orchestrator we can carry out and see all the ramifications of making orchestration changes - since all this control information is centrally located. 10.0 Future Work Our work with the Orchestrator has really only begun. For example, in the distributed Karoke system we use a control panel on one 6. This figure is based on using ONC-RPC as the IPC mechanism. Our ear- ly experience with ToolTalk taught us that ToolTalk can only deliver 20 Messages/sec. Therefore, we do not plan to use ToolTalk in future versions.

15. system to drive a suite of programs on other machines which are supposed to play in harmony with the “lead” system. We do this by posting the GUI events from the “lead” control panel to the Or- chestrator, which then relays these events (via scripts) to the systems keeping harmony. This corresponds to the translation level service we described in section 3.3. Obviously this needs to be ex- tended to take into account the trading (section 3.2) and coordination (section 3.4) issues. What we would like to do is have network-based applications (such as nfs, ftp, etc.) place bandwidth request information to the Orchestrator. In addition traffic monitoring programs would keep track of instaneous network traffic such as that from X-windows programs. We would also have load monitoring programs that would keep track of CPU loading at the hosts. All this information would be posted to the Orchestrator at regular intervals. The Orchestrator would use this information to provide a super- scheduling service for the distributed Karoke system. It would di- rect the audio, video, and GUI data from the “lead” system to the distributed “harmony” systems. The harmonizing systems would merge this data with their own to produce composite audio, video, and graphics.7 This would be a “super-scheduler” in that most plans for schedul- ers for ATM circuit set-up do not incorporate such detailed information about the continous streams that they control. In our case, we would be providing the circuit scheduling algorithm with detailed information about the streams and, in addition, information about host loading - which we believe is also important in the delivery of smooth continous A/V information on the user’s workstation. That is, the problem does not end at the network interface but most also incorporate information about the actual ability of the workstation to deliver the A/V stream to the user. 11.0 Conclusions It is our belief that coordination, control, translation, and orchestration issues are going to become major topics in the creation of distributed multimedia applications. Each element of a multimedia application will have different features and performance. In order to bind together these elements one will have to carefully choose among the potential network multimedia elements and qualify their usage by their type, performance, and the QoS characteristics of the transport links that connects them. We see the issues of binding, coordination, translation, and orchestrator as being inherently linked issues. In the Circus Orchestrator we provide a a single architecture and platform where one can create orchestration services that can deal with these linkages. In addition, the Orchestrator mechanism allows one to have one central repository where all information about the network, processors, available applications, and translation needs can be centrally stored (i.e. in TupleSpace). The scripting facility allows all the decisions about coordination and orchestration to be performed lo- cally on the data. Not only can this lead to better performance and more timely re-assessment of binding decisions, but it also frees the individual multimedia network elements from having to be aware of how they are bound together in an application. 7. Audio and video data would not pass through TupleSpace, but through independent channels whose QoS characterisics are dynamically controlled by the Orchestrator. The Orchestrator provides a very simple metaphor and model for handling coordination and orchestration needs. It is basically a large content addressable memory (TupleSpace) coupled to a rule based system (the ScriptList). Although the basic model is simple, it is also very powerful. This power stems largely from the general purpose language (CircusTalk) and its extensive libraries for doing pattern-based programming, object-oriented programming, and procedural programming. 12.0 References 0. 1. Anindo Bannerjea and Bruce A. Mah. The Real-Time Channel Administration Protocol. In Network and Operating System Support for Digital Audio and Video, Second Interna- tional Workshop, Heidelberg Germany, November 1991. Springer-Verlag 1992. 209--216. 2. Gerold Blakowski, Jens Hubel, Ulirike Langrehr, and Max Muhlhauser. Tool Support for the Synchronization and Presentation of Distributed MultiMedia. In Computer Com- munications, Vol. 15, No. 10 (Dec. 1992). 611--618. 3. Sara A. Bly, Steve R. Harrison, and Susan Irwin. Media Spaces: Bringing People Together in a Video, Audio, and Computing Environment. In Communications of the ACM. Vol. 36, No. 1 (Jan. 1993). 28--47. 4. Andrew Campbell, Geoff Coulson, Fancisco Garcia, and David Hutchison. A Continuous Media Transport and Or- chestration Service. In Computer Communications Review. Vol. 22, No. 4 (Oct 1992). SIGCOMM’92 Conference Pro- ceedings. 99--110. 5. Nicholas Carriero and David Gelernter. How to Write Par- allel Programs: A First Course. MIT Press, Cambridge MA. 1990. 6. Geoff Coulson, Francisco Garcia, David Hutchison, and Doug Shepard. Protocol Support for Distributed MultiMedia Applications. In Network and Operating System Support for Digital Audio and Video, Second International Workshop, Heidelberg Germany, November 1991. Springer-Verlag 1992. 45--56. 7. David Gelernter and Nicholas Carriero. Coordination Lan- guages and their Significance. In Communications of the ACM. Vol. 35, No. 2 (Feb. 1992). 97--107. 8. Ed Foster. Microsoft’s Cairo takes us right into the heart of darkness. In InfoWorld, Vol. 14, No. 10 (Mar. 9, 1992). page 58. 9. Yechezkal-Shimon Gutfreund, Jose Diaz-Gonzalez, Rus- sell Sasnett, and Vincent Phuah. Dynamicity Issues in Broadband Network Computing. In Network and Operating System Support for Digital Audio and Video, Second Interna- tional Workshop, Heidelberg Germany, November 1991. Springer-Verlag 1992. 209--216. 10. Yechezkal-Shimon Gutfreund. ThinkerToy: An Environ- ment for Decision Support. Ph.D. Thesis. Computer and In- formation Sciences Department, COINS Technical Report 88-72. 1988.

16. 11. Yechezkal-Shimon Gutfreund. ManiplIcons in ThinkerToy. Proceedings of OOPSLA’87 Object Oriented Programming Systems, Languages, and Applications. Norman Meyrowitz editor. October 4-8 1987. Kissimee, Florida. SIGPLAN Notic- es, Vol. 22, No. 12, (December 1987) 307--317. 12. Hewlett-Packard. HP NewWave General Information Manu- al. Hewlett-Packard. 1990. 13. ISO, ODP. Working Document on Topic 9.1 - ODP Trader. In ISO/IEC JTC1/SC21/WG7 and SGVII.Q16 (ODP) N645, N737. 27 November, 1992. 14. David S. Marshak. ANSA: A Model for Distributed Com- puting. In Network Monitor: Guide to Distributed Comput- ing. Vol. 6, No. 11 (Nov. 1991) 15. Keith Marzullo, Mark Wood, Robert Cooper, Kenneth Birman. Tools for Distributed Application Management. In IEEE Computer, Vol. 24, No. 8 (Aug. 1991). 42--51. 16. OMG. The Common Object Request Broker: Architecture and Specification. Object Management Group, Framingham, MA. 1991. 17. Open Software Foundation. OSF Distributed Management Environment (DME) Architecture. Open Software Founda- tion, Cambridge, MA. 1992. 18. OSF. GUI-based Application Interoperability Architecture (GAIA). In GAIA Workshop Proceedings. August 18-19. Open Software Foundation, Cambridge, MA. 1992. 19. R. Popescu-Zeletin, V. Tschammer, and M. Tschichholz. ‘Y’ Distributed Application Platform. In Computer Commu- nications, Vol. 14, No. 6 (July 1991). 366--374. 20. J. Duane Northcutt and Eugene M. Kuerner. System Sup- port for Time-Critical Applications. In Network and Operat- ing System Support for Digital Audio and Video, Second International Workshop, Heidelberg, Germany, November 1991. Springer-Verlag 1992. 242--254. 21. Bruce D. Schatzman. Next-generation OLE: OLE 2.0 is fast- er and simpler for users,but it’s trickier for developers. Byte Vol. 18, No. 3 (March 1993). 209-211. 22. John Shirley. Guide to Writing DCE Applications. O’Reilly & Associates, Inc. Sebastopol, CA. 1992. 23. SunSoft. ToolTalk 1.0 Programmer’s Guide. SunSoft, Moun- tain View, CA. 1991. 24. TINA. Tina Consortium Meeting Notes. For meeting notes contact Harvey Rubin at Bellcore 1-908-758-2223 or ti- na93@rs.sip.stet.it. 1992. 25. Stephen Wolfram. Mathematica: A System for Doing Math- ematics by Computer. 2nd Edition. Addison-Wesley, Read- ing, MA. 1991.

JPD Developing Distributed Multimedia Applications

Recommended

Recommended

More Related Content

Similar to JPD Developing Distributed Multimedia Applications

Similar to JPD Developing Distributed Multimedia Applications (20)

Recently uploaded

Recently uploaded (20)

JPD Developing Distributed Multimedia Applications