03 synchronized distance learning

1,951 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,951
On SlideShare
0
From Embeds
0
Number of Embeds
970
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

03 synchronized distance learning

  1. 1. Synchronized Distance Learning Timothy K. Shih
  2. 2. Outline • White Board and Chat Room – Designing Shared Whiteboard • Multimedia Synchronization – Microsoft ASF – SMIL • Synchronization Models – Petri Net – Interval Temporal Logic (optional)
  3. 3. White Board and Chat Room • Chat Room – Almost Everywhere • White Board – On-Line Annotation with Floor Control
  4. 4. Text Acknowledgement Timothy K. Shih and Yi-Chun Liao, MINE Lab, Tamkang University, Taiwan • Text and simple graphic drawing • Annotation message broadcasting via multiple uni-casting • NetMeeting 3.0, COM (Component Object Model), ActiveX control • Acknowledgement from Remote Stations is included • Floor Control: Control Modes, Channel, and Group On-Line Annotation
  5. 5. Request Button Request to SpeakErrorAcknowledgement Question Button Speaking Return Control Interface and Demo
  6. 6. Demo Annotation with Feedbacks
  7. 7. • 30 Client connections • Heterogeneous Network Demo Video
  8. 8. • The control of ―Who to Speak‖ in a communication environment • Concepts: – Channel: communication modal (text, audio, video, etc.) – Mode: communication style of a channel (Free Access vs. Equal Right) – Group: the IP domain of a channel • Example: 4 Channels, 3 Groups, 10 Persons, 2 Modes Video Conferencing (ER) and Chat Room Tool (FA) Chat Room Tool (FA) Annotation Tool (ER) Floor Control
  9. 9. Chat RoomAnnotation Virtual PanoramaShared Notebook Video Audio Video-on-Demand Multi-modal and Multi- channel Communication
  10. 10. • Enhance the clearness of video presentation • Add bitmapped animation to the video Augmented Video Conferencing Tool
  11. 11. Augmented Video Presentation Screen Capture Video Control Video + Audio MPEG-2
  12. 12. Using Direct Show
  13. 13. Implementation • Configuration • Five Control Modes – Normal – Resized Screen – Resized Video – Full Video – Full Screen
  14. 14. • Audio and Video Devices and Codec • Screen and CCD Camera Resolutions Device and Layout Configuration
  15. 15. Video Control Modes
  16. 16. Normal Mode
  17. 17. Resized Screen Mode
  18. 18. Resized Video Mode
  19. 19. Full Video Mode
  20. 20. Full Screen Mode
  21. 21. • Problems to Solve – Distance Precision – Space Discontinuity – Video Precision – Eye Contact – Object Tracking • The instructor • The whiteboard – Interaction Protocol • Floor Control – Performance Challenges
  22. 22. Demo Demo
  23. 23. Multimedia Synchronization • Why Multimedia Synchronization • Synchronization Methods – Microsoft ASF – SMIL • Synchronization Models (Theories) – Petri Net – Interval Temporal Logic
  24. 24. Video Music Audio Slide Message Annotation ASF Records Timothy K. Shih, Y. R. Liu and Sheng-En Yeh, MINE Lab, Tamkang University, Taiwan • Automatic Recording • Use Commercial Codec and the Advanced Streaming Format (ASF) from Microsoft • Video and Slide Synchronization • Text and Drawing Annotation Synchronization • Slide Selection • Presentation Editing • Automatic Loading Multimedia Synchronization Using ASF
  25. 25. • Event markers are embedded in a ASF file • Arbitrary activation of markers • Do not record the entire stream as one video clip • Video streaming depends on Codec Video Stream Text User Interrupt User Interrupt User Interrupt Slide Slide Slide  MS Office XP – Presentation Broadcast provides video and slide synchronization (PowerPoint and IE) Event-based Synchronization
  26. 26. Choose Device Choose Codec Presentation Recording Tool
  27. 27. Select Output Slide Selection Presentation Playback Control Functions Presentation Tool
  28. 28. • Hard to record the entire lecture without making a mistake • Different presentation needs different level of details • Lecture summarization and editing • Atomic presentation object – a PowerPoint Slide (with its video and annotation events) Text Atomic Object Adaptive Presentation
  29. 29. Three Levels of Details Automatic Loading Presentation Editing Multi-Level Presentation
  30. 30. Demo Video Demonstration
  31. 31. Synchronized Multimedia Integration Language (SMIL) • Recommended by W3C • XML-based language • Designed for Internet use • Easy to write similar to HTML • Integrates several media formats • Plain text documents with a ―.smi‖ or ―.smil‖ suffix Part of slides about SMIL are summarized from: http://service.real.com/help/library/guides/production/realpgd.htm
  32. 32. History • Dec 1995 - SMIL designing started • Nov 1997 - SMIL 1.0 Recommendation • Mar 1998 - First SMIL implementation • Aug 1999 - First public draft of SMIL 2.0 • Aug 2001 - SMIL 2.0 Recommendation
  33. 33. Support for SMIL in Players / Browsers 2.0 • X-Smiles 0.40: SMIL 2.0 Basic Profile 2.0 • Oratrix GRiNS player: SMIL 2.0 Language 2.0 • IE 5: many SMIL 2.0 modules 1.0 • QuickTime Player 4.1 1.0 • RealPlayer 8 1.0 • Compaq HPAS 1.0 • Productivity Works Lp Player 1.0 • SOJA from Helio 1.0 • S2M2 by NIST 1.0 • Schmunzel by SunTREC Salzburg
  34. 34. Use Cases for SMIL • Slideshows • Advertisements • Internet TV • Education • Corporate communications • Product information • User’s Guides • Net meetings • etc.
  35. 35. General Rules • Documents start with <smil> and end with </smil> • <head> section, which is optional; and a <body> section which is required • Tags and Attributes in lowercase only • All elements must have closing tags or / • Attribute values must be enclosed in double- quotes • HTML-style comments are allowed. – <!-- comments go here -->
  36. 36. Typical SMIL Elements <head> - Head element <meta> - Meta data, such as author, copyright... <layout> - Layout of the presentation <region> - Region describing positioning <body> - Body element <seq> - Sequential time container <par> - Parallel time container <img>, <audio>, <video> - Media to be played <switch> - Selects between elements <a>, <area> - Linking
  37. 37. A SMIL Example <smil> <head> <layout> <region id=”reg1” top=”0” left=”0” width=”300” height=”200” /> <region id=”reg2” left=”300” width=”300” height=”200” /> </layout> </head> <body> <seq> <img id=”id1” src=”intro.jpg” region=”reg1” dur=”4s” /> <audio id=”id2” src=”music.wav” dur=”2s” /> <par> <video id=”id3” src=”movie5s.mpg” region=”reg1” /> <img id=”id4” src=”intro.jpg” begin=”3s” region=”reg2” /> </par> </seq> </body> </smil>
  38. 38. <head> element • Information not related to the temporal presentation of the material • May contain the following children – <meta> – <switch> – <layout>
  39. 39. <meta> element • Just like the <meta> element in HTML • An element used to specify information about the document • Common uses include: – <meta name="author" content="Jim Dabrowski"/> – <meta name="title" content="Jim’s SMIL Presentation"/> • Other uses: copyright, abstract, etc.
  40. 40. <layout> element • Probably the most important element • Specify the layout of the document, as well as the layout of the multimedia elements in the document • If not specified, how document looks will depend on the player • Two children allowed here – <root-layout> – <region>
  41. 41. <root-layout> element • An empty element • Defines the size of the main window • You cannot specify media elements in here • There can only be one <root-layout>
  42. 42. <root-layout> attributes • background-color – Can use named or hexadecimal values. • height, width – All specified in pixels • title
  43. 43. <region> element • Areas within <root-layout> in which media elements will appear • May define as many as you want • Very similar to frames in HTML • Are allowed to overlap — controlled by z- order • Any regions that lay outside the <root- layout> will be cut off
  44. 44. Required attributes • id – A unique text name for the region • top, left, width, height – Pixels or percentages or mixed
  45. 45. Example • <head> <root-layout width="480" height="300"/> <region id=“someid" top="60" left="120" width="240" height="180"/> </head> Screen (60, 120)  240  180
  46. 46. <body> element • Information related to the temporal and linking behavior of the document • Children include – Synchronization elements • <par>, <seq> – Media elements • <animation>, <audio>, <img>, <text>, <textstream>, <video>, <ref>
  47. 47. Synchronization elements • <par> element — children of a <par> element occur in parallel • <seq>element — children of a <seq> element appear in sequential order • Valid children of <seq> and <par> elements include – <seq> & <par> – Any of the media elements
  48. 48. Combining <seq> and <par> tags <seq> clip 1 <par> clip 2 clip 3 </par> clip 4 </seq> <par> clip 1 <seq> clip 2 clip 3 </seq> clip 4 </par> 1 2 3 4 3 1 2 4
  49. 49. Media elements • The various pieces of multimedia that go into the SMIL document • Continuous media – <animation>, <audio>, <textstream>, <video> – Have an intrinsic duration • Discrete media – <text>, <img> – Have no intrinsic duration • The actual tag name doesn’t matter • Type attribute is used to determine media type • Could use <ref> tag
  50. 50. Media element attributes • All media elements have certain attributes in common • You use these attributes to specify how the children of <seq> and <par> elements will appear in the presentation
  51. 51. Specifying media and location • src – Specifies the source of the media element – Identical to the src attribute of the <img> tag in html • region – The area of the root-layout in which the media element is to be displayed – e.g., <video src="hal.mov" region="video_region" /> – If you don’t specify a region, it’s implementation dependent
  52. 52. Setting begin and end time • begin & end attributes • Specifies the explicit beginning and ending time of the element – e.g., <video src="hal.mov" begin="10.5s" end="45.5s" /> • Relative to parent element regardless of own length • May use with <par> and <seq> elements to affect entire group
  53. 53. Setting internal begin & end times • clip-begin & clip-end attributes • Specifies the internal beginning and ending time of a clip – E.g., <video src="hal.mov" clip-begin="2.5s" clip- end="15.8s" /> • May combine with begin and end attributes
  54. 54. Setting a duration • dur attribute • Explicit duration of the media element • Time value, or ―indefinite‖ • Should not use dur with end attributes
  55. 55. Filling up time • fill attribute – Specifies what happens to the clip when it is done – Freeze or remove • repeat – Specifies how many times the clip should repeat itself – Any integer is valid – Can add repeat attribute to the <par> and <seq> elements
  56. 56. Synchronizing elements • By default, <par> elements end when all elements finish • Can be modified with the end attribute to force an end time for the <par> element • Or you can use the endsync attribute of the <par> element to synchronize the ending of its clips • Three values for the endsync attribute – first, last, id(element id)
  57. 57. Linking • SMIL provides two ways to link to other documents • <a> tag – Identical to the <a> tag in html – Specify an href attribute with a URL – Can surround and turn an entire media element into a link • <anchor> tag – Defines a ―hot spot‖ inside a media element – Similar to HTML image maps
  58. 58. <anchor> tag • Specified as a child of a media element – <video src="video.rm" region="video_region"> <anchor href="url here" coords="20, 40, 80, 120"> </video> • Where the coords attribute specifies the location within the media element that is the hot spot • May use pixels or percentages
  59. 59. Control Structure • SMIL provides a <switch> tag • Can use it to allow player to choose one of several test-attributes • Format <switch> <choice1 test-attribute="value1"/> <choice2 test-attribute="value2"/> … </switch> • Various test-attributes have been defined
  60. 60. Using <switch> for language • You can use the <switch> tag to allow player to choose the appropriate language • <par> <video src="video/seattle.rm" /> <switch> <audio src="french/seattle.rm" system-language="fr"/> <audio src="german/seattle.rm" system-language="de"/> <audio src="spanish/seattle.rm" system-language="es"/> <audio src="english/seattle.rm"/> </switch> </par>
  61. 61. Using <switch> for system bitrates • <switch> <par system-bitrate="75000"> <!-- dual ISDN or faster --> <audio src="audio/newsong1.rm"/> <video src="video/newsong1.rm"/> </par> <par system-bitrate="4700"> <!-- single ISDN --> <audio src="audio/newsong2.rm"/> <video src="video/newsong2.rm"/> </par> <par system-bitrate="20000"> <!-- 28.8 modem --> <audio src="audio/newsong3.rm"/> <video src="video/newsong3.rm"/> </par> </switch>
  62. 62. For information • W3C’s Synchronized Multimedia Group – http://www.w3.org/AudioVideo • Just SMIL – http://smw.internet.com/smil
  63. 63. Petri Net • What are Petri Nets – A tool for system study • System Modeling – Structure – Dynamic Behaviors • System Evaluation Source: Feng-Cheng Yang These slides are modified based on Feng-Cheng Yang’s presentation
  64. 64. Using Petri Net • Modeling – Representation for a System – Often in Mathematical Terms – Manipulated to reveal information • Example of Discrete System Analysis – Bank System • Teller Components – Idle state – Busy state • Customer Components – Idle state (waiting in the line) – Busy state (being served)
  65. 65. Multimedia Presentation SMIL <par> clip 1 <seq> clip 2 clip 3 </seq> clip 4 </par> 3 1 2 4 Petri Net (simplified) 3 1 2 4
  66. 66. Features of a System • Components – Subsystems – Separated and Interacting • States – Depend on the past history – Change over time
  67. 67. System Behaviors • Concurrency (Parallelism) – Activities of different components occur simultaneously – Difficult to model • Sequencing – Serial activities occur one by one between components – Easy to model
  68. 68. History • Basic concept was created in 1962 by Carl Adam Petri in his doctoral thesis (W. Germany) – Communication between asynchronous components of a computer system • Early theory, notation, and representation are developed by the Information System Theory Project (Applied Data Research, US) – Conduct by A.W. Holt (1968)
  69. 69. History (cont.) • Considerable research and publication in 1970 – Directed by Prof. Jack B. Dennis at M.I.T. • Conference on Petri Nets and Related Methods in 1975 at M.I.T. • Workshop on Petri Nets held in Paris in 1977 • In 1979 an advanced course on General Net Theory is held in Hamburg • A Special Interest Group on Petri nets was formed in Germany
  70. 70. Petri Net Structure (Mathematical Model) • Four parts – P: a set of places – T: a set of transitions – I: an input function • A mapping from a transition tj to a collection of places I(tj ) • The places are input places of tj – O: an output function • A mapping from a transition tj to a collection of places O(tj ) • The places are output places of tj Pi tj Pi tj
  71. 71. Definition 1 • A Petri net structure is a four-tuple: C = (P,T,I,O) • P = { p1, p2,…, pn }, n ≥ 0 • T= { t1, t2,…, tm }, m≥0 • P∩T = ∅ • I:T→P∞, a mapping from transitions to bags of places • O:T → P∞, a mapping from transitions to bags of places
  72. 72. Petri Net Representation • A place pi is an input place of a transition tj if pi ∈ I(tj) • A place pi is an output place of a transition tj if pi ∈ O(tj) • A bag is a generalization of sets which allows multiple occurrences of an elements in a bag. – E.g.: O(tj) = { p1, p1, p1, p2, p3, p4, p4, p5 } Pi tj Pi tj
  73. 73. Example 1 P2 P4P1 P5 P3 t1 t2 t3 t4
  74. 74. Example 1 • C = ( P,T,I,O ) • P = { p1 , p2 , p3 , p4 , p5 }, n = 5 • T = { t1 , t2 , t3, t4 }, m = 4 • I(t1 ) = { p1 } O(t1 ) = { p2 , p3 , p5 } • I(t2 ) = { p2 , p3 , p5 } O(t2 ) = { p5 } • I(t3 ) = { p3 } O(t3 ) = { p4 } • I(t4 ) = { p4 } O(t4 ) = { p2 , p3 } P2 P4P1 P5 P3 t1 t2 t3 t4
  75. 75. Multiplicity • #(a, A) : The number of occurrences of element a in bag A. • The multiplicity of an input place pi for a transition tj is #( pi, I(tj) ) • A transition tj is an input of a place pi, if pi is an output of tj • A transition tj is an output of a place pi, if pi is an input of tj Pi tj Pi tj # = 3
  76. 76. Definition 2 • Input and output functions are extended as follows: – I:T→P∞, O:T → P∞ , such that • #( tj, I(pi) ) = #( pi, O(tj) ) and • #( pi, I(tj) ) = #( tj, O(pi) ) Pi tj Pi tj
  77. 77. Example 2 P2 P4P1 P5 P3 t1 t2 t3 t4 P6 t5
  78. 78. Example 2 • C = ( P,T,I,O ) • P = { p1 , p2 , p3 , p4 , p5 , p6 }, n = 6 • T = { t1 , t2 , t3, t4, t5 }, m = 5 • I(t1 ) = { p1 } O(t1 ) = { p2 , p3 } • I(t2 ) = { p3 } O(t2 ) = { p3 , p5 , p5 } • I(t3 ) = { p2, p3 } O(t3 ) = { p2, p4 } • I(t4 ) = { p4 , p5 , p5 , p5 } O(t4 ) = { p4 } • I(t5 ) = { p2 } O(t5 ) = { p6 } P2 P4P1 P5 P3 t1 t2 t3 t4 P6 t5
  79. 79. Example 3 P2 P4 P1 P5 P3 t1 t2 t3 t4 P6 t5 P7 P8 P9 t6
  80. 80. Example 3 • C = ( P,T,I,O ) • P = { p1 , p2 , p3 , p4 , p5 , p6 , p7 , p8 , p9 }, n = 9 • T = { t1 , t2 , t3, t4, t5 , t6 }, m = 6 • I(t1 ) = { p1 } O(t1 ) = { p2 , p3 } • I(t2 ) = { p8 } O(t2 ) = { p1 , p7 } • I(t3 ) = { p2 , p5 } O(t3 ) = { p6 } • I(t4 ) = { p3 } O(t4 ) = { p4 } • I(t5 ) = { p6 , p7 } O(t5 ) = { p9 } • I(t6 ) = { p4 , p9 } O(t6 ) = { p5 , p8 } P2 P4 P1 P5 P3 t1 t2 t3 t4 P6 t5 P7 P8 P9 t6
  81. 81. Petri Net Graphs • Bipartite directed multigraph • Vertices: – Circles (places) and Bars (transitions) • Links: – Directed arcs (represent the relations between transitions and places) • Represented by functions I and O • Multiple arcs directed from a place to a transition is allowed
  82. 82. Definition 3 • A Petri net graph G is a bipartite directed multigraph, G = ( V, A ), where V = { v1, v2, …, vs } is a set of vertices and A = {a1, a2, …, ar} is a bag of directed arcs, ai = (vj , vk ), with vj, vk ∈ V • The set V can be partitioned into two disjoint sets P and T such that V = P ∪ T, P ∩ T = ∅ , and for each directed arc, ai ∈ A, if ai = (vj , vk ), then either vj ∈ P and vk ∈ T or vj ∈ T and vk ∈ P
  83. 83. Definition 4 • Define V = P ∪ T. • Define A as a bag of directed arcs such that for all pi ∈ P and tj ∈ T , – #( (pi , tj ), A ) = #( pi , I(tj) ) and – #( (tj , pi ), A ) = #( pi , O(tj) ) • G = ( V, A ) is a Petri net graph which is equivalent to the Petri net structure C = ( P, T, I, O )
  84. 84. The Dual of a Petri Net • The dual of a Petri net C = ( P, T, I, O ) is the Petri net C’ = (T, P, I, O ) which results from interchanging places and transitions
  85. 85. The Dual of Example 1 P2 P4P1 P3 t1 t5 t3 t4 t2 P2 P4P1 P5 P3 t1 t2 t3 t4
  86. 86. The Inverse Petri Net • The inverse Petri net for a Petri net C = ( P, T, I, O ) is defined by interchanging the input and output functions, -C = ( P, T, O , I ).
  87. 87. The Inverse of the Dual of Example 1 P2 P4P1 P3 t1 t5 t3 t4 t2 P2 P4P1 P3 t1 t5 t3 t4 t2
  88. 88. Petri Net Markings • A marking μ is an assignment of tokens to the places of a Petri net • The number and position of tokens may change during the execution of a Petri net • The tokens are used to define the execution of a Petri net
  89. 89. Definition 5 • A marking μ of a Petri net C = ( P,T,I,O ) is a function from the set of places P to the nonnegative integers N; μ : P → N • The marking can also be defined as an n-vector, μ = [ μ1 μ2 …μn ] • The number of tokens in place pi is μi, i = 1, …, n – μ(pi ) = μi
  90. 90. Marked Petri Nets • M = ( C, μ ) is a Petri net structure and a marking μ – M = (P,T,I,O, μ ) • Tokens are represented by small dots in the circles which represent the places of a Petri net • The number of tokens assigned to a place is unbounded • The number of possible markings of a Petri net is infinitive
  91. 91. A Marked Petri Net of Example 1 • The marking μ = [ 1 2 0 0 1 ] t1 t2 t3 t4 P1 P5 P2 P4 P3
  92. 92. Another Marked Petri Net of Example 1 • The marking μ = [ 0 2 0 5 1 ] t1 t2 t3 t4 P5 P2 P4P1 P3
  93. 93. Execution Rules for Petri Nets • Tokens reside in places and control the execution of the transitions of the net • A Petri net executes by firing transition • A transition fires by removing tokens from its input places and creating new tokens which are distributed to its output places – The tokens are not moving from one place to the next place – They are consumed by the transition and regenerated from the transition to the next place • A transition may fire if it is enabled
  94. 94. Execution Rules for Petri Nets (Conti.) • A transition is enabled if each of its input places has at least as many tokens in it as arcs from the place to the transition • The tokens in the input places which enable a transition are its enabling tokens • A transition fires by removing all of its enabling tokens from its input places and then depositing into each of its output places one token for each arc from the transition to the place
  95. 95. Definition 6 • A transition tj ∈ T in a marked Petri net C = ( P,T,I,O ) with marking μ is enabled if for all places pi , μ(pi) ≥ #( pi, I(tj) ) tj 1 token ≥ one arc 3 tokens ≥ three arcs 2 tokens ≥ two arcs
  96. 96. Definition 7 • A transition tj in a marked Petri net with marking μ may fire whenever it is enabled • Firing an enabled transition tj results in a new marking μ’ defined by • μ’(pi ) = μ (pi ) - #(pi , I(tj) ) + #(pi , O(tj) ) tj Pi tj Pi Remove 2 tokens (for 2 arcs) Remove 3 tokens (for 3 arcs) Add 1 token (for 1 arc) Add 1 token (for 1 arc)
  97. 97. Firing Examples • Example a: – I(t3 ) = { p2} O(t3 ) = {p7, p13 } – t3 is enabled whenever there is at least one token in place p2 • Transition t3 fires by removing one token from p2 and depositing one token in place p7 and one token in place p13 t3 t3 P2 P7 P13 P7 P13 P7P2
  98. 98. Firing Examples (Cont.) • Example b: – I(t2 ) = {p21, p23} O( t2 ) = {p23, p25, p25 } – t2 is enabled whenever there is at least one token in place p21 and at least one token in place p23 – Transition t2 fires by removing one token from p21 and one token from p23 and then deposits one token in place p23 and two tokens in place p25 t2 t2 P21 P23 P25 P23 P25 P21
  99. 99. Firing Examples (Cont.) • t1 , t3 , and t4 are enabled t1 t2 t3 t4 P4 P1 P3 P5P2
  100. 100. • t4 is fired t1 t2 t3 t4 P4 P1 P3 P5P2
  101. 101. • t1 is fired t1 t2 t3 t4 P4 P1 P3 P5P2
  102. 102. • t3 is fired t1 t2 t3 t4 P4 P1 P3 P5P2
  103. 103. Summary • Petri Net Definitions • Petri Net Examples • For multimedia presentations – Transitions are used for synchronization – Places can represent multimedia resources • Related to Interval Temporal Logic 3 1 2 4
  104. 104. Multimedia Presentation Based on Petri Net • Defining a operation model of multimedia presentation by features of Petri Net • Constructing an algorithm based on Timed Petri Net for multimedia presentations
  105. 105. Background • Petri net is a graph and a mathematical tool • Multimedia presentations includes timing control and user interaction • Describe multimedia objects by Timed Petri Net • Create new definitions based on Petri Net for interactions • Implement a tool for multimedia presentation designs
  106. 106. An operation example of Petri net Place A rc Tokens Transition A rc A rc Place Place Tokens Place Place Place A rc A rc A rc Transition Token Token Token (a) A initial marking before transition firing (b) A new marking after transition firing
  107. 107. Interpretations of places and transitions Tansitions A ttributesInput Places O utput Places Input Places Transition A ttributes O utput Places D ata Processing Inform ation R equest A vailable Service R esource Perm ission A pproach Precondition Event Postcondition M ultim dia Tim ing Playing
  108. 108. Two special cases in Petri net (a) A sink transition of Petri Net (End of Presentation) (b) A source transition of Petri Net (Begin of Presentation)
  109. 109. Self-loop and Weights 1 1 1 2 1 (a) A self_loop of Petri Net (b) Weights of synchronous arcs of Petri Net
  110. 110. New Components for Multimedia Objects : Place and token is for playing a m ultim edia resource. : U ser transition is for accepting a m essage and causing the activation of connected transitions. : U se r arc is fo r connec ting fr om a user transition to a transition. : Sy nc hro nou s arc is for con ne cting fr om a p la ce to a transition or from a transition to a place. : Transition is for synchronous control. New New
  111. 111. Example of Multimedia Petri net im age text video sound m usic Syn. arc Syn. arc Syn. arc Syn. arc Syn. arc Syn. arc Syn. arc Syn. arc Syn. arc Syn. arc U ser arc U ser arc navigation m essage navigation m essage user transition transition transition transition transition
  112. 112. Definitions of place, transition and user transition in the Multimedia Petri net O bject attributes File form at D uration File nam e Execution tool File attributes Place node Transition U ser Transition A ttributes M essages Tim ing O bject relations Interrupt Pause Stop C hange
  113. 113. Algorithm: Compute_Node_Time • Construct a Transition_Graph reachable from one of the starting • Navigation_Messages; • Initialize the node pointed by the Navigation_Message to • Node_Time = 0; • Sort the Transition_Graph to a Transition_List by Topological sort; • For each node N in the Transition_List, except the first, • Begin • For each incoming edge, compute the Estimated_Time = • Node_Time + Edge_Duration; • Set Node_Time of node N to the maximum of all Estimated_Time • of incoming edges; • End
  114. 114. Algorithm: Run_Petri_Net – Begin – For each node N in the Transition_List Do Begin – Collect places pointed by Sync_Arcs from node N in Sync_Set(N); – Sort the Transition_List according to Node_Time in a non- – decreasing order; – End – CoBegin – Process1: – For each node N in the Transition_List Do Begin – Sequentially play resources in Sync_Set(N) at Node_Time(N) – concurrently; – End – Process2: – LoopBegin – If Process1 ends, Then Process2 ends; – If there is a Navigation_Message of a User_Transition Then – Exits with the next starting Navigation_Message; – If there is a Selection, Assignment, or Condition Then – Perform the action accordingly; – LoopEnd – CoEnd – End – Algorithm: Run_Presentation – Begin – LoopBegin – If there is a starting Navigation_Message Then – Compute_Node_Time; – Run_Petri_Net; – Else – End Presentation – LoopEnd – End
  115. 115. A initial graph of Multimedia Petri Net Text V ideo W ave Sound Bm p M idi
  116. 116. Transition graph for simplified Multimedia Petri Net Sound Bm p Text V ideo U ser m essage W ave U ser m essage M idi
  117. 117. An example for the Compute-Node-Time algorithm 70 35 20 15 15 20 40 30 20 a b c d e
  118. 118. Topological sort of the transition nodes a c d e b 20 70 152030 35 15 40 20
  119. 119. A Multimedia Presentation Example sound text musicanimation video image a b c d
  120. 120. Presentation Schedule of the Example A nim ation Sound Text M usic Im age V ideo Tim e Transition "a" Transition "b" Transition "c" Transition "d" U ser transition
  121. 121. Synchronous Sets for the Order of Transitions Firing a b c d b c d C ase1 : W ithout firing of user transition : {anim ation,sound,text} : {m usic,im age,text} : {m usic,video} : End C ase2 : W ith firing of user transition : {m usic,im age,video} : End
  122. 122. Layered Multimedia Presentation Subnet Subnet Subnet Transition Transition U ser Transition
  123. 123. Implementation Example
  124. 124. Implementation Example (cont’)
  125. 125. Implementation Example (cont’)
  126. 126. Summary • Provide a model for the multimedia presentation design by extending concepts of Petri Net • Could be extended for layered multimedia presentations • Petri Net is used as an example of synchronization control
  127. 127. Collaborative Learning with Petri Net • Different students learn different contents and in different speed • Team members have to learn skill and gathered together to solve a mission start C B A Dispatch Discuss Grade
  128. 128. The Same Structure but Different Skill Learner A Cooperative Learning Learner B Learner C L2L1 L3 L4 L1,L3,L4 L1,L2,L3,L4 L1,L2,L3,L4
  129. 129. Capability & Assessment Learner A Learner B Learner C L1,L3,L4 L1,L2,L3,L4 Exam Give different color tokens condition Learner D L1,L2,L3,L4 L1,L2,L3,L4 Learner E L1,L3,L4 C.L.
  130. 130. Cooperative Learning Environment Enter Communication Vote Suspend ExitExam
  131. 131. Interval Temporal Logic • Temporal Intervals • Fast Interval Relation Composition • Solving the Composition Conflicts • Interval Relation Distances • Qualitative Spatial Knowledge • The Applications
  132. 132. • James F. Allen, "Maintaining Knowledge about Temporal Intervals," Communications of the ACM, Vol. 26, No. 11, 1983. • Relations between two time intervals • Relation composition • Can be used in many research areas Temporal Intervals
  133. 133. Allen’s 13 Temporal Interval Relations
  134. 134. Composition of the 13 Interval Relations
  135. 135. From Point Relations to Interval Relations
  136. 136. Revised 18 Temporal Interval Relations
  137. 137. Fast Interval Relation Composition • Relation composition involves set operations • The amount of relation sets is finite • Table13
  138. 138. The Relation Composition Algorithm Algorithm : RelComp Input : rs1 29RelSet, rs2 29RelSet Output : rs 29RelSet Preconditions : true Postconditions : true Steps : 1. rs = r1 rs1 , r2 rs2 (r1 , r2) rs1 rs2 Table13 (r1 , r2)
  139. 139. Computing All Possible Relations Algorithm : ComputeTable29 Input : Table13 Output : Table29 Preconditions : true Postconditions : relation composition is closed under I Steps: 1. Construct a set of 13 atomic sets from the 13 relations, assuming that this set is called I, which is an index set for table look up. 2. Let Table29( i, j ) = Table13( i, j ), i I, j I 3. Table29( i, j ), i I , j I , do 3.1: if k = Table29( i, j ) I then 3.1.1 : I = I Table29( i, j ) 3.1.2 : m I ,do 3.1.2.1 Table29( k, m ) = Relcomp( k, m ) 3.1.2.2 Table29( m, k ) = Relcomp(m, k )
  140. 140. The 29 Possible Relation Sets IDs Relation Sets IDs Relation Sets ------------------------------------------------------------------------------------------ 1 { < } 16 { o, d ,s } 2 { > } 17 { oi, di, si } 3 { d } 18 { <, o, m } 4 { di } 19 { >, oi, mi } 5 { o } 20 { f, fi, e }* 6 { oi } 21 { s, si, e }* 7 { m } 22 { <, o, m, d, s } 8 { mi } 23 { >, oi, mi, di, si } 9 { s } 24 { <, o, m, di, fi } 10 { si } 25 { > , oi, mi, d, f } 11 { f } 26 { o, oi, d, di, s, si, f, fi, e }* 12 { fi } 27 { <, m, d, di, o, oi, f, fi, s, si, e } 13 { e }* 28 { > , mi, di , d, oi, o, fi, f , si, s, e } 14 { o, di, fi } 29 { < , > ,m, mi, di , d, oi, o, fi, f , si, s, e }* 15 { oi, d, f } rs rs-1 rs rs-1 ------------------------------------------- 1 2 18 19 3 4 20 20 5 6 21 21 7 8 22 23 9 10 24 25 11 12 26 26 13 13 27 28 14 15 29 29 16 17 The Inverse Relations
  141. 141. Table29 o | 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 ----------------------------------------------------------------------------------------------------------------------------- -------------- 01 |01 29 22 01 01 02 01 22 01 01 22 01 01 01 22 22 22 01 29 22 01 22 29 01 29 22 22 29 29 02 |29 02 25 02 25 02 25 02 25 02 02 02 02 25 25 25 02 29 02 02 25 29 02 29 25 25 29 25 29 03 |01 02 03 29 22 25 01 02 03 25 03 22 03 29 25 22 29 22 25 22 25 22 29 29 25 29 29 29 29 04 |24 23 26 04 14 17 14 17 14 04 17 04 04 14 26 26 17 24 23 17 14 27 23 24 28 26 27 28 29 05 |01 23 16 24 18 26 01 17 05 14 16 18 05 24 26 22 27 18 28 22 14 22 29 24 28 27 27 29 29 06 |24 02 15 23 26 19 14 02 15 19 06 17 06 28 25 26 23 27 19 17 25 27 23 29 25 28 29 28 29 07 |01 23 16 01 01 16 01 20 07 07 16 01 07 01 16 22 22 01 28 22 07 22 29 01 28 22 22 29 29 08 |24 02 15 02 15 02 21 02 15 02 08 08 08 25 25 15 02 27 02 08 25 27 02 29 25 25 29 25 29 09 |01 02 03 24 18 15 01 08 09 21 03 18 09 24 15 22 27 18 25 22 21 22 29 24 25 27 27 29 29 10 |24 02 15 04 14 06 14 08 21 10 06 04 10 14 15 26 17 24 19 17 21 27 23 24 25 26 27 28 29 11 |01 02 03 23 16 19 07 02 03 19 11 20 11 28 25 16 23 22 19 20 25 22 23 29 25 28 29 28 29 12 |01 23 16 04 05 17 07 17 05 04 20 12 12 14 26 16 17 18 23 20 14 22 23 24 28 26 27 28 29 13 |01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 14 |24 23 26 24 24 26 24 17 14 14 26 24 14 24 26 27 27 24 28 27 14 27 29 24 28 27 27 29 29 15 |24 02 15 29 27 25 24 02 15 25 15 27 15 29 25 27 29 27 25 27 25 27 29 29 25 29 29 29 29 16 |01 23 16 29 22 28 01 23 16 28 16 22 16 29 28 22 29 22 28 22 28 22 29 29 28 29 29 29 29 17 |24 23 26 23 26 23 14 23 26 23 17 17 17 28 28 26 23 27 23 17 28 27 23 29 28 28 29 28 29 18 |01 29 22 24 18 27 01 27 18 24 22 18 18 24 27 22 27 18 29 22 24 22 29 24 29 27 27 29 29 19 |29 02 25 23 28 19 28 02 25 19 19 23 19 28 25 28 23 29 19 23 25 29 23 29 25 28 29 28 29 20 |01 23 16 23 16 23 07 23 16 23 20 20 20 28 28 16 23 22 23 20 28 22 23 29 28 28 29 28 29 21 |24 02 15 24 24 15 24 08 21 21 15 24 21 24 15 27 27 24 25 27 21 27 29 24 25 27 27 29 29 22 |01 29 22 29 22 29 01 29 22 29 22 22 22 29 29 22 29 22 29 22 29 22 29 29 29 29 29 29 29 23 |29 23 28 23 28 23 28 23 28 23 23 23 23 28 28 28 23 29 23 23 28 29 23 29 28 28 29 28 29 24 |24 29 27 24 24 27 24 27 24 24 27 24 24 24 27 27 27 24 29 27 24 27 29 24 29 27 27 29 29 25 |29 02 25 29 29 25 29 02 25 25 25 29 25 29 25 29 29 29 25 29 25 29 29 29 25 29 29 29 29 26 |24 23 26 29 27 28 24 23 26 28 26 27 26 29 28 27 29 27 28 27 28 27 29 29 28 29 29 29 29 27 |24 29 27 29 27 29 24 29 27 29 27 27 27 29 29 27 29 27 29 27 29 27 29 29 29 29 29 29 29 28 |29 23 28 29 29 28 29 23 28 28 28 29 28 29 28 29 29 29 28 29 28 29 29 29 28 29 29 29 29 29 |29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29 29
  142. 142. Solving the Composition Conflicts • Unknown derivation If X < Y and Y > Z, then X ? Z • Multiple derivation If X < Y and Y d Z, then X {<, o, m, d, s} Z • Conflict derivation If X < Y, Y < Z, and X > Z are declared by the user, then there exists a conflict
  143. 143. The Domain of Relations • An user edge denotes a relation between a pair of objects defined by the user. The relation may be reasonable or non-reasonable. • A derived edge holds a non-empty set of reasonable relations derived by our algorithm. The relation of the two objects connected by the derived edge can be any reasonable relation in the set. • complete relation domain (a complete graph): contains user edges and derived edges, with possible cycles and possible conflicts. • reasonable relation domain (a graph): contains user edges and derived edges, with possible cycles but no conflict. • reduced relation domain (a graph): contains only user edges, with possible cycles and possible conflicts. • restricted relation domain (a tree): contains only user edges, without cycle.
  144. 144. Relation among Domains
  145. 145. Computes the Reasonable RD from the Reduced RD Algorithm : ComputeRD1 Input : G = ( GV,GE ) Output: Kn = ( KnV, KnE ) Preconditions : true Postconditions : GV = KnV GE KnE Steps : 1 : G = EliminateConflicts (G) 2 : Kn = G pl = 2 3 : repeat until | KnE | = | KnV | * ( | KnV | -1 ) / 2 3.1 : for each e = ( a, b ) e KnE a KnV b KnV such that there is a path of user edges from a to b , with path length = pl 3.2 : suppose (( n1, n2 ), ( n2, n3 ),…, ( nk-1, nk ) ) is a path with a = n1 b = nk k = pl + 1 3.3 : set e.rs = Table29 (( a, nk-1 ).rs, (nk-1, b ).rs ) 3.4 : KnE = KnE { e } 3.5 : pl = pl + 1
  146. 146. Eliminate Conflicts Algorithm : EliminateConflicts Input : G = ( GV, GE ) Output : G = ( G V, G E) Preconditions : G contains only user edges G = G Postconditions : G = G , but the reasonable sets of edges in G may be changed. Steps : 1. for each P = ((n1, n2), (n2, n3), , (nk-1, nk) ) in G with n1 = nk k >3 1.1 : for each i, 1 i k-2 1.1.1 : set (ni, ni+2).rs = Table29 ((ni, ni+1).rs, (ni+1, ni+2).rs ) 1.2 : rs = Table29 ((nk, nk-2).rs, (nk-2, nk-1).rs ) 1.3 : if (nk, nk-1).r rs then 1.3.1 : ask user to choose a r rs 1.3.2 : set (nk, nk-1).r = r
  147. 147. An Example
  148. 148. User Edges and Derived Edges User edges: ( A, B ) = { < } = [1] ( B, C ) = { m } = [7] ( C, D ) = { d } = [3] ( C, E ) = { s } = [9] ( F, D ) = { < } = [1] 2. Path Length = 3 (A, E) = (A, B ) o (B, C) o (C, E) = (A, C ) o (C, E) = [1] o [9] = [1] = { < } (A, D) = (A, B ) o (B, C) o (C, D) = (A, C ) o (C, D) = [1] o [3] = [22] = { <, o, m, d, s } (B, F) = (B, C ) o (C, D) o (D, F) = (B, D ) o (D, F) = [16] o [1]-1 = [23] = { >, oi, mi, di, si } (E, F) = (E, C ) o (C, D) o (D, F) = (E, D ) o (D, F) =[14] -1 o [2] = [15] o [2] = [2] = { > } 3. Path Length = 4 (A, F) = (A, B ) o (B, C) o (C, D ) o (D, F) = ( (A, B ) o (B, C) ) o ( (C, D ) o (D, F) ) = (A, C ) o (C, F) = [1] o [2] = [29] = { <, >, d, di, o, oi, m, mi, f, fi, s, si, e } Derived edges: 1. Path Length = 2 (A, C) = (A, B ) o (B, C) = [1] o [7] = [1] = { < } (B, D) = (B, C ) o (C, D) = [7] o [3] = [16] = {o, d, s } (C, F) = (C, D ) o (D, F) = [3] o [1]-1 = [3] o [2] = { > } (D, E) = (D, C ) o (C, E) = [4] o [9] = [14] = {o, di, fi } (B, E) = (B, C ) o (C, E) = [7] o [9] = [7] = { m }
  149. 149. Adds User Edges to the Reasonable RD Algorithm : AddUERD Input : l = (a, b ), Kn = ( KnV, KnE ) Output : Kn+1 = ( Kn+1V, Kn+1E ) Preconditions : l KnE a KnV b KnV Postconditions : | Kn+1V | = | KnV | + 1 | Kn+1E | = | KnE | + n Steps : 1: Kn+1E = KnE { l } 2: for each e = ( c, b ) c a c KnV 2.1: e.rs = d KnV, ( c, d ) KnE, (d, b ) KnE ( Table29(( c, d ).rs, ( d, b ).rs )) 2.2: Kn+1E = Kn+1E {e} 3: Kn+1V = KnV { b }
  150. 150. Adding Edges Adding an user edge to Example 4.1: Add ( G, E ) = { f } = [11] Derived edges : Derive ( E, G ) = ( G, E )-1 = [11]-1 = [12] = { fi } ( inverse ) Derive ( A, G ) = ( A, E ) o ( E, G ) = [1] o [12] = [1] = { < } ( association ) Derive ( B, G ) = ( B, E ) o ( E, G ) = [7] o [12] = [1] = { < } ( association ) Derive ( C, G ) = ( C, E ) o ( E, G ) = [9] o [12] = [18] = {<, o, m } Derive ( D, G ) = ( D, E ) o ( E, G ) = [14] o [12] = [24] = { <, o, m, di, fi } ( association ) Derive ( F, G ) = ( F, E ) o ( E, G ) = ( E, F )-1 o ( E, G ) = [1] o [12] = [1] = { < } ( inverse, association )
  151. 151. Interval Relation Distances • Use relation distances to estimate the similarity of object positions • There are three types of relation distances – The Point Relation Distance – The Interval Relation Distance – The Extended Point-Interval Relation Distance
  152. 152. The Point Relation Distance
  153. 153. The Interval Relation Distance
  154. 154. The Extended Point-Interval Relation Distance
  155. 155. The Distance Graph < ols m loe o s d fi e f di si oi m i ole los > oo
  156. 156. Qualitative Spatial Knowledge • Relation Embedding – two points on a line – two points on a plan – two line segments on a line – two line segments on a plan – two 2-D objects on a plan
  157. 157. Projection of Two Line Segments on a 2-D Plan
  158. 158. Projection of Two 2-D Objects on a 2-D Plan
  159. 159. Generalized Spatial Relation f 1 = 29RelSet o 29RelSet 29RelSet f 2 = 29RelSet o 29RelSet 29RelSet o 29RelSet 29RelSet 29RelSet f 3 = 29RelSet o 29RelSet 29RelSet o 29RelSet 29RelSet o 29RelSet 29RelSet 29RelSet 29RelSet where 29RelSet 29RelSet { {<} {<}, {<} {>}, … , { = } { = } } 29RelSet 29RelSet 29RelSet { {<} {<} {<}, {<} {<} {>},…, {=} {=} {=}} i1 j1, i2 j2 P (29RelSet 29RelSet ) f 2 (i1 j1, i2 j2 ) = f 1 (i1 , i2 ) f 1 ( j1 , j2 ) i1 j1 k1 , i2 j2 k2 P (29RelSet 29RelSet 29RelSet ) f 3 (i1 j1 k1, i2 j2 k2) = f 1 (i1 , i2 ) f 1 ( j1 , j2 ) f 1 (k1 , k2 ) where A B = { a b | a A, b B } A B C = { a b c | a A, b B, c C }
  160. 160. The Applications • Multimedia Presentation Generation • Shape-Based Image Retrieval (not discussed)
  161. 161. Multimedia Presentation Generation
  162. 162. The Presentation System
  163. 163. The Presentation System
  164. 164. The Presentation System
  165. 165. The Presentation System
  166. 166. The Presentation System
  167. 167. The Presentation System
  168. 168. Notes • Synchronization of Multimedia Objects • Computation Efficiency • Extension toward N-Dimension • Multimedia Presentation Scheduling
  169. 169. Summary • White Board and Chat Room • Multimedia Synchronization • SMIL • Petri Net • Interval Temporal Logic

×