Successfully reported this slideshow.
Your SlideShare is downloading. ×

Structured Interactive Scores

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 167 Ad
Advertisement

More Related Content

Similar to Structured Interactive Scores (20)

Advertisement

Recently uploaded (20)

Structured Interactive Scores

  1. 1. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra “If I cannot draw it, I do not understand it” –Anonymous. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 1
  2. 2. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structured interactive scores: From a structural description of a multimedia scenario to a real-time capable implementation with formal semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. Ph.D defense Supervised by Myriam Desainte-Catherine and Camilo Rueda September 25th 2012 Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 2
  3. 3. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Some domains shaped by the arose of technology Tape Music Electroacoustic Music Installations Interactive Multimedia Experimental Music Non-linear Multimedia Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 3
  4. 4. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 4
  5. 5. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Contemporary Dance ./videos/theatre.mov Minchoul synchronizes human gestes, video and sound in real-time in cleaning (2009). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 5
  6. 6. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive multimedia installations ./videos/installation.mov Torres uses the participation of the audience to co-author the meaning of the installation in pollen (2008). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 6
  7. 7. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Meta-instruments for physically disabled people ./videos/baopao.mov The Bao-Pao1 is a meta-instrument. As an example, this meta-instrument can control the start and end of notes. 1 http://www.bao-pao.com/index.php Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 7
  8. 8. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 8
  9. 9. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Problems with most existing tools 1. No formal semantics [TDCR12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 9
  10. 10. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Problems with most existing tools 1. No formal semantics [TDCR12] 2. Time models are unrelated [DCAA12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 10
  11. 11. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Cue-Lists model As an example, consider theater cue-list manager Qlab 2 . 2 http://figure53.com/qlab/ Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 11
  12. 12. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timeline model As an example, consider sequencer Pro Tools 3 . 3 http://www.avid.com/US/resources/digi-orientation Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 12
  13. 13. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time models are unrelated: Cue Lists Vs. Timeline Vs. It is argued in [DCAA12] that having both time models temporally related will bring new possibilities for the creation of interactive multimedia. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 13
  14. 14. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Problems with most existing tools 1. No formal semantics [TDCR12] 2. Time models are unrelated [DCAA12] 3. Schedulers are not appropriate for real-time [TDCC12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 14
  15. 15. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Scheduler problem: aggravated under high cpu load • Most tools do not allow the users to interact with them without letting them experience noticeable delays • Karplus-Strong is a well-known digital signal processing algorithm to simulate the sound of metallic strings • A Karplus-Strong arpeggio implementation in Pure Data4 . ./sounds/barry-loaded.mov • A Karplus-Strong arpeggio implementation using Faust ([OFL04]) and interactive scores [TDCC12]. ./sounds/faust-loaded.mov Results from [TDCC12]. 4 Colin Barry’s implementation found at www.loomer.co.uk Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 15
  16. 16. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Problems with most existing tools 1. No formal semantics [TDCR12] 2. Time models are unrelated [DCAA12] 3. Schedulers are not appropriate for real-time [TDCC12] 4. No hierarchy (e.g., movements, parts, measures, motives and notes) [Vic04] 5. No unified model for conditional branching and temporal relations [Vic03, Ran09, All09, Vic11] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 16
  17. 17. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 17
  18. 18. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solution: Interactive Scores Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d overlaps starts starts ends ends ends time A formalism for interactive multimedia developed at LaBRI: [BDC01, DCB03, ADC05, AADCR06, AADC07, ADCLA08, AADC08, Ran09, All09, OR09, BAM+ 09, Sar08, ABM+ 10], [TDCB10, TDC10, Tor10], [MADC11], [ADCT11, TDCC12, TDCR12, Tor12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 18
  19. 19. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 19
  20. 20. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra History of interactive scores • Boxes: A software to model a hierarchy and temporal constraints [BDC01] • A model of interactive scores [ADC05] • Hierarchical Time Stream Petri nets (hts petri nets) semantics for interactive scores [AADC07] • An extension of interactive scores with temporal reductions [ADCLA08] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 20
  21. 21. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra History of interactive scores • First non-deterministic timed concurrent constraint (ntcc) model of interactive scores [AADCR06] • A model change the hierarchy of the temporal objects during execution [OR09] • A model to represent conditional branching and temporal relations separately [Ran09, All09] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 21
  22. 22. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 22
  23. 23. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Existing tools for interactive scores • Boxes [BDC01] • Iscore developed for OpenMusic [AADC08] • Acousmouscribe • Stand-alone i-score (currently maintained by LaBRI) • Prototypes using Ntccrt, a real-time interpreter for ntcc [TDCB10, TDC10, TDCC12] • Virage [BAM+ 09, ABM+ 10, MADC11] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 23
  24. 24. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Virage: An implementation of interactive scores ./videos/virage.mov Software Virage developed under Virage project [BAM+ 09]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 24
  25. 25. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 25
  26. 26. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Related formalisms to interactive scores • Globally Asynchronous, Locally Synchronous (gals) [TGL07] • Hierarchical time stream Petri nets (hts petri nets) [SSSW95] • Temporal constraints: qualitative [All83], quantitative [Gen98] and combination of both [Mei96] • Concurrent constraint programming (ccp) calculi [Sar92, SJG94, Sar08, OV08]. • Non-deterministic timed concurrent constraint programming (ntcc) [NPV02] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 26
  27. 27. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Non-deterministic Timed Concurrent Constraint Programming (ntcc) • Process calculus to model reactive systems with discrete time, non-determinism, partial information and asynchrony [NPV02, Val02] • Includes a temporal logic to verify properties [NPV02] • A logic formula can be translated into a process and the strongest post-condition5 of a process into a B¨uchi automaton6 [Val05] 5 Sequences that the process can output under any environment. 6 A B¨uchi automaton recognizes sequences of infinite length. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 27
  28. 28. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Non-deterministic Timed Concurrent Constraint Programming (ntcc) • Has been used for the verification and simulation of other interactive multimedia systems [RV02, RV05, RV04] • Has been used for interactive scores [AADCR06, Sar08, OR09] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 28
  29. 29. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: Interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalism to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 29
  30. 30. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Related tools to interactive scores • Score following (e.g., Antescofo [Con08, ECGJ11]) Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 30
  31. 31. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra An example of score following ./videos/antescofo.mov Example of the score following system Antescofo. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 31
  32. 32. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Related tools to interactive scores • Score Following (e.g., Antescofo [Con08, ECGJ11]) • Asynchronous Dataflow Languages (e.g., Max/msp [PAZ98] and Pure Data[Puc96]) • Synchronous Dataflow Languages (e.g., CSound, Faust [OFL04], Esterel, Signal [GLGB87] and Lustre) • Machine Improvisation (e.g., OMax [ABC+ 06] and Continuator [Pac02]) • Computer Assisted Composition (e.g., OpenMusic’s Maquettes [BAA11]) Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 32
  33. 33. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. We saw some examples of multimedia interaction 2. There are problems with most existing tools 3. Our solution is interactive scores (is) 4. is is a mature formalism 5. There are several tools for is 6. Score following and Maquettes are related to is 7. gals, htspn and ntcc are related to is Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 33
  34. 34. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 34
  35. 35. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Hierarchical model of interactive scores 1. Structural definition [TDCR12] 2. Abstract semantics [TDCR12] 3. Operational semantics [TDCR12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 35
  36. 36. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition: Temporal objects Green (g) Red (r) Sound (u) time Temporal objects are aligned on a timeline, thus they have nominal start time and nominal duration. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 36
  37. 37. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition: Hierarchy of temporal objects Green (g) Red (r) Sound (u) Lights (l) Scenario (c) A temporal object can contain other objects to constraint the execution start and end times of its children. The hierarchical model has been previously discussed in [BDC01, All09]. We presented a new formalization in [TDCR12]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 37
  38. 38. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition: Interactive objects Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d Interactive objects (gray circles) allow to control the start or end of other temporal objects. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 38
  39. 39. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition: Temporal relations Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d overlaps starts starts ends ends ends time Temporal relations constraint the set of possible execution times and durations of the temporal objects. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 39
  40. 40. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d ∆r ∆g ∆u ∆l ∆c {0} {0} {0} {0} {0} [0, ∞) [0, ∞) [0, ∞) Durations and temporal relations can be represented by point-to-point temporal relations labeled by integer intervals [Mei96]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 40
  41. 41. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Hierarchical model of interactive scores 1. Structural definition 2. Abstract semantics 3. Operational semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 41
  42. 42. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timed Event Structures (tes) • Langerak’s timed event structures is a mathematical model to represent systems with non-determinism, real-time and concurrency [BKL98]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 42
  43. 43. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timed Event Structures (tes) a b c d e Figure : Circles are events. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 43
  44. 44. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timed Event Structures (tes) a b c d e [0,7] [0,5] [4,4] Figure : Circles are events. Full-line arrows represent event delays. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 44
  45. 45. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timed Event Structures (tes) a b c d e [0,7] [0,5] [4,4] Figure : Circles are events. Full-line arrows represent event delays. Pointed-line arrows represent conflicts; when they have no direction, the conflict is symmetrical. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 45
  46. 46. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Temporal objects sc su sb sl sr sg sa sd ec eu er eg el STRUCTURAL DEFINITION tes semantics for interactive scores [TDCR12]. Colored circles represent events of static temporal objects. Gray circles represent events of interactive objects. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 46
  47. 47. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Duration of temporal objects ∆r ∆g ∆u ∆c ∆l sc su sb sl sr sg sa sd ec eu er eg el STRUCTURAL DEFINITION Durations of the temporal objects are represent by event delays. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 47
  48. 48. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Temporal relations 0 0 0 0 (0,∞ )(0,∞ ) (0,∞ ) sc su sb sl sr sg 0 sa sd ec eu er eg STRUCTURAL DEFINITION Red arrows are event delays of the form ei [0,∞) → ej , derived from the hierarchy. Black arrows are event delays derived from the explicit temporal relations. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 48
  49. 49. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Interactive scores 0 0 0 0 (0,∞ )(0,∞ ) (0,∞ ) ∆r ∆g ∆u ∆c ∆l sc su sb sl sr sg 0 sa sd ec eu er eg STRUCTURAL DEFINITION tes semantics is obtained by combining the events, the event delays from the duration of temporal objects, and the event delays from the temporal relations. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 49
  50. 50. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Hierarchical model of interactive scores 1. Structural definition 2. Abstract semantics 3. Operational semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 50
  51. 51. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Operational semantics: Intuition sc su sb sl sr sg sa sd ec eu er eg 0 0 0 0 (0 ,∞ )(0,∞) (0 , ∞ ) ∆r ∆g ∆u ∆c ∆l sc su sb sl sr sg 0 sa sd ec eu er eg CONSTRAINT STORESTRUCTURAL DEFINITION x Agent that represents the behavior of point x • Ntcc semantics for interactive scores were introduced in [AADCR06] and later formalized in [TDCR12]. • Operational semantics respects the event structures semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 51
  52. 52. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Hierarchical model of interactive scores 1. The structural definition is composed by temporal objects and point-to-point temporal relations 2. Abstract semantics are defined in timed event structures 3. Operational semantics are defined in ntcc Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 52
  53. 53. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 53
  54. 54. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model of interactive scores 1. Structural definition [TDCB10, TDC10, ADCT11] 2. Abstract semantics 3. Operational semantics [TDCB10, TDC10, ADCT11] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 54
  55. 55. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model: Structural definition Verse 1 (v) Chorus (ch) Verse 3 (x) Verse 2 (w) Verse 4 (y) Chorus (c) ∆v ∆w∆ch ∆c ∆ = {0} ∆ = {0} ∆ = {0} ∆ = [2, 5] ∆ = [1, 4] ϕ > 0 ∨ ψ ≤ 3 φ > 0 ∧ ψ ≤ 5 d ∆x ∆y ∆a ∆b V = {φ, ψ, ϕ} main (m) ∆ = {0} ϕ > 0 ∨ ψ ≤ 3 V = {φ, ψ, ϕ} Timed conditional relation duration condition Variables Interactive object Choice The model was introduced in [TDCB10, TDC10, ADCT11]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 55
  56. 56. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model of interactive scores 1. Structural definition 2. Abstract semantics 3. Operational semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 56
  57. 57. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: “Visible”-action events sa sb ea ecscsv ev d Structural Definition Colored circles are the “visible”-action events. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 57
  58. 58. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: “Invisible”-action events sa sb eb ea ecscsv ev d s a s b Structural Definition Black circles are the “invisible”-action events. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 58
  59. 59. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Temporal relations 0 0 sa sb eb ea ecscsv ev d s a s b Structural Definition Temporal relations are translated to event delays. Event delays are represented as arrows. For simplicity, the arrows have no labels. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 59
  60. 60. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Conflicts 0 0 sa sb eb ea ecscsv ev d s a s b Structural Definition Dashed lines represent symmetrical conflicts: It means that two events are mutually exclusive. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 60
  61. 61. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: More conflicts 0 0 sa sb eb ea ecscsv ev d s a s b Structural Definition Dashed-line arrows represent asymmetrical conflicts: It means that one event must happen before another or it will not happen. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 61
  62. 62. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Why visible and invisible events? • Because it is a simple way to represent conditional branching in a single bi-dimensional plane. A b s t r a c t i o n Tim e (Im plication) Values Paralelism (and) Alternative (or) Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 62
  63. 63. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra David Janin’s dimensions A b s t r a c t i o n Tim e (Im plication) Values Paralelism (and) Alternative (or) There are several dimensions in multimedia interaction7 . 7 http://www.labri.fr/perso/janin/ Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 63
  64. 64. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model of interactive scores 1. Structural definition 2. Abstract semantics 3. Operational semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 64
  65. 65. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model: Operational semantics b a c when finish unless finish ∆b = 3 ∆c = 2 d ∆d = 1 ∆ = 0 ∆ = 1 ∆ = 1 ∆ = 1 ∆ = 1 ∆ = 0 V = {finish} Operational semantics for the conditional branching extension with loops were introduced in [TDCB10, TDC10, ADCT11]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 65
  66. 66. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model of interactive scores 1. Structural definition is given for scores with and without loops 2. Abstract semantics are defined in timed event structures for scores without loops 3. Operational semantics are defined in ntcc, support loops, but are not related to the abstract semantics, as it was the case in the hierarchical model Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 66
  67. 67. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 67
  68. 68. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive scores with signal processing 1. Structural definition [TDCC12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 68
  69. 69. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores with signal processing Karplus (k1) Karplus (k2) Karplus (k3) Output An extension in which ntcc and the synchronous programming language with formal semantics Faust [OFL04] interact. Introduced in [TDCC12] and it is inspired from [All09]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 69
  70. 70. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores: Temporal objects Karplus (k1) Karplus (k2) Karplus (k3) ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] Output Temporal objects may contain a Faust audio process, before they used to control external audio processes. Input8 and Output streams are also temporal objects. 8 In this score there is no audio input. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 70
  71. 71. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores: Temporal relations Karplus (k1) Karplus (k2) Karplus (k3) 100smp [2s, 4s] [0s, 0s] [0s, 0s] Output In addition to the usual temporal relations, dashed-line arrows represent high-precision temporal relations. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 71
  72. 72. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores: Dataflow relations Karplus (k1) Karplus (k2) Karplus (k3) Output White-headed arrows represent dataflow relations: To describe how the sound is transferred from one object to another. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 72
  73. 73. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores: Interactive objects Karplus (k1) Karplus (k2) Karplus (k3)a b Output Interactive objects are represented (and behave) as usual. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 73
  74. 74. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Extension of Interactive Scores Karplus (k1) Karplus (k2) Karplus (k3)a b [100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Temporal object Temporal relation High-precision temporal relation Dataflow relation Interactive object Interactive score Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 74
  75. 75. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 75
  76. 76. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation of Interactive Scores 1. Ntccrt [TAAR09] 2. Results on the conditional branching model [TDC10] 3. Results on the signal processing extension [TDCC12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 76
  77. 77. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation: Interpreter for ntcc OpenMusic interface Ntccrt compiler Pure Data external Max/Msp external Common Lisp interface C++ interface Stand-alone program User Programmer Ntccrt9 is a simulation tool for ntcc introduced in [TAAR09]. Experience on the usage of Ntccrt is described in [ORS+ 11]. 9 http://sourceforge.net/projects/ntccrt/ Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 77
  78. 78. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation: Example using Pure Data Structural Definition “Bang” objects are controlled by the Ntccrt plugin [TDC10]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 78
  79. 79. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation 1. Ntccrt 2. Results on the conditional branching model 3. Results on the signal processing extension Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 79
  80. 80. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Results on the conditional branching extension 0
 5
 10
 15
 20
 25
 30
 35
 40
 45
 50
 0
 200
 400
 600
 800
 1000
 1200
 1400
 1600
 1800
 Average
dura*on
of
a
*me
unit
(ms)
 Number
of
points
in
the
Interac*ve
Score
 Stress test to compute the average duration of a discrete time unit of the simulation of a conditional branching score using Ntccrt [TDC10]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 80
  81. 81. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation 1. Ntccrt 2. Results on the conditional branching model 3. Results on the signal processing extension Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 81
  82. 82. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Results on the signal processing extension Average relative jitter: The average time difference between the expected starting time of each string, with respect to the first string of an arpeggio, and the time obtained during execution. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 82
  83. 83. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Results on the signal processing extension Implementation 85% cpu load 3% cpu load Using object block˜ 1 in Pd 7991 ms 1 ms Using object z˜ 1 in Pd 9231 ms 2 ms Using Faust object in Pd 0.5 ms 0.5 ms Table : Average relative jitter of each string of an arpeggio with respect to the beep. All implementations use the Karplus-Strong algorithm to simulate the sound of the strings [TDCC12]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 83
  84. 84. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Results on the signal processing extension In some cases –such as programs with feedback loops– our interactive scores implementation using Faust outperforms Pure Data under high-cpu load. Pure data’s performance is unacceptable, in such cases, for real-time interaction. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 84
  85. 85. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation 1. Our models are simulated using Ntccrt, a real-time capable interpreter for ntcc 2. Results on the conditional branching model show that we can execute up to 400 temporal objects 3. Results on the signal processing extension show that interactive scores outperforms Pure Data under high-cpu load Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 85
  86. 86. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 86
  87. 87. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification 1. Motivation 2. Constraint linear-time temporal logic (cltl) [NPV02] 3. Model checking algorithm Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 87
  88. 88. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification of interactive scores • Existing semantics for interactive scores were given in hts petri nets. • Existing verification tools for time Petri nets, such as Rom´eo [GLMR05] and Tapaal [BJS09], cannot be used to verify hts petri nets [SSSW95] because hts petri nets cannot be translated to time Petri nets [BD99]. • Existing verification tools for linear-time logic, such as : Uppaal [BDL+ 01], Kronos [BDM+ 98] and Spin [Hol97], cannot be used directly for ntcc. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 88
  89. 89. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification of interactive scores • Results on bisimilarity for ccp [Ari10, ABPV12] are quite promising, but not yet applicable to ntcc. • Strongest post-condition of a process can be translated into a B¨uchi automaton [Val05], but using classic Schimpf’s algorithm [SMS09] is intractable because the encoding heavily relies on the complement of such automata [Saf88]. • We proposed a bounded-time finite-state-automata-based model checker for ntcc, ntccmc10 . 10 http://sourceforge.net/projects/ntccmc/ Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 89
  90. 90. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification 1. Motivation 2. Constraint linear-time temporal logic (cltl) 3. Model checking algorithm Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 90
  91. 91. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra cltl properties Examples of constraint linear-time logic (cltl) [NPV02] formulae are • “always the constraint pitch = 60 can be deduced from the output store”, namely (pitch = 60); and • “eventually, both object a and object b are launched at the same time”, namely ♦(launcha ˙∧launchb). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 91
  92. 92. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification 1. Motivation 2. Constraint linear-time temporal logic (cltl) 3. Model checking algorithm Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 92
  93. 93. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Is the language empty? Bounded-time automata-based model checking algorithm for ntcc A. intersection P → Pt F → Ft Ft Pt Pt → A Pt → A Ft → Pt Qt A A BF P t A B B Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Adaptation of classic model checking algorithm for linear-time logic [SMS09] to ntcc, based on the encoding of ntcc to B¨uchi automata given by Valencia et al. [Val05]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 93
  94. 94. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc Bounded-time automata-based model checking algorithm for ntcc ntcc Process CLTL formulaF P t Number of discrete time units Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 94
  95. 95. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Does a ntcc process P satisfy a CLTL formula F for a bounded time t ? Bounded-time automata-based model checking algorithm for ntcc P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → FtF P t Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 95
  96. 96. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Bounded-time automata-based model checking algorithm for ntcc P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process F P t Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 96
  97. 97. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Bounded-time automata-based model checking algorithm for ntcc P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Bounded-time process to deterministic automaton (A) Pt → A Pt → A Pt → A Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process A BF P t Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 97
  98. 98. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Bounded-time automata-based model checking algorithm for ntcc P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Bounded-time process to deterministic automaton (A) Pt → A Pt → A Pt → A Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process A A A. complement A BF P t B A Automaton Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 98
  99. 99. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Bounded-time automata-based model checking algorithm for ntcc A. intersection P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Bounded-time process to deterministic automaton (A) Pt → A Pt → A Pt → A Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process A A A. complement A BF P t A B B A Automaton Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 99
  100. 100. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Is the language empty? Bounded-time automata-based model checking algorithm for ntcc A. intersection P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Bounded-time process to deterministic automaton (A) Pt → A Pt → A Pt → A Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process A A A. complement A BF P t A B B A Automaton Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 100
  101. 101. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification 1. Existing tools cannot be used directly to verify interactive scores and we want to prove properties of interactive scores automatically 2. In particular, constraint linear-time temporal logic (cltl) properties 3. We presented our bounded-time automata-based model checking algorithm for ntcc, ntccMC, for cltl properties Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 101
  102. 102. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 102
  103. 103. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 103
  104. 104. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Summary • A Formal model for multimedia interaction that combines • timeline model • cue-list model [TDCR12] • An extension for conditional branching [TDCB10, TDC10, ADCT11] • An extension for audio processing [Tor10, TDCC12] • A simulation tool [TAAR09, ORS+ 11] • A verification tool [Tor10] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 104
  105. 105. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Contributions • 2 book chapters in “Constraint Programming in Music” [ADCT11, ORS+ 11] • 1 journal article in “Mathematics and Music” [TDCR12] • 2 papers in “Sound and Music Computing” conference [TDC10, TDCC12] • 1 extended abstract in “International Conference on Logic Programming” [Tor10] • 1 paper in “Journ´ees d’Informatique Musicale” [TDCB10] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 105
  106. 106. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 106
  107. 107. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • Time models are unrelated • In interactive scores, time models are temporally related Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d overlaps starts starts ends ends ends time Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 107
  108. 108. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • Schedulers are not appropriate for real-time • Using interactive scores and Faust, it is possible to have real-time interaction even under high cpu load Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 108
  109. 109. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • No hierarchy • Interactive scores provide a hierarchy of temporal objects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 109
  110. 110. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • No formal semantics • We defined abstract and operational semantics for interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 110
  111. 111. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • No unified model for conditional branching and temporal relations • We provide a unified model for conditional branching and temporal relations in which all branches have the same set of possible durations Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 111
  112. 112. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 112
  113. 113. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Comparison w.r.t previous models of interactive scores • Model to change the hierarchy during performance [OR09] • Previous model developed at LaBRI [All09] • Our models [TDC10, TDCB10, TDCR12, TDCC12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 113
  114. 114. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Model to change the hierarchy during performance • Provides an elegant solution to change the structure of the hierarchy during performance • Has not been implemented Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 114
  115. 115. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Previous model developed at LaBRI • Supports different mechanisms of temporal reductions • Average duration of a discrete time unit outperforms our simulations • Durations are intervals are limited to {0}, (0, ∞] and [0, ∞] • Conditional branching and temporal relations are presented separately • Processes associated are meant to be handled outside the formalism; in the implementation as external programs Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 115
  116. 116. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Our models • Abstract semantics to describe the execution traces • Arbitrary integer intervals in the operational semantics • Unified model for conditional branching and temporal relations • Automatic verification • Does not allow temporal reductions • Simulation tool is a prototype; it is not user friendly • Average duration of time units is slower than in the implementation of previous model Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 116
  117. 117. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 117
  118. 118. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools Karplus (k1) Karplus (k2) Karplus (k3)a b[100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) • Qlab and Ableton Live do not allow to model delays of 100 samples • Protools does not allow interaction In all the programs mentioned above, it is very hard to synchronize processes whose durations are integer intervals such as duration ∈ [2, 4]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 118
  119. 119. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 119
  120. 120. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Future work directions • Event structures semantics for scores with loops • Model interactive theater performances • Model interactive museum exhibitions • Verify scores in which ntcc and Faust interact • Model music pieces combining the timeline and cue-list model Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 120
  121. 121. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Music pieces combining both time models Baltazar et al. used i-score to compose miniature # 1 (2012). I-score controls the smoke, fan, sound and lights. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 121
  122. 122. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Music pieces combining both time models ./videos/pascal.mp4 Baltazar et al. needed to have objects with flexible duration because the behavior of the smoke is unpredictable. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 122
  123. 123. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 123
  124. 124. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Ongoing projects • Project Interactivity in the Writing of Time and Interactions (inedit) (2012-2015) founded by the french national research agency (anr). • Project Robust theories for Emerging Applications in Concurrency Theory: Processes and Logic Used in emergent Systems(react+) (2011-2013) founded by the colombian national research agency (Colciencias). • Project ossia (2012-2015) founded by the anr. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 124
  125. 125. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Ongoing projects • Project Interactivity in the Writing of Time and Interactions (inedit) (2012-2015) founded by the french national research agency (anr). • Project Robust theories for Emerging Applications in Concurrency Theory: Processes and Logic Used in emergent Systems(react+) (2011-2013) founded by the colombian national research agency (Colciencias). • Project ossia (2012-2015) founded by the anr. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 124
  126. 126. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Ongoing projects • Project Interactivity in the Writing of Time and Interactions (inedit) (2012-2015) founded by the french national research agency (anr). • Project Robust theories for Emerging Applications in Concurrency Theory: Processes and Logic Used in emergent Systems(react+) (2011-2013) founded by the colombian national research agency (Colciencias). • Project ossia (2012-2015) founded by the anr. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 124
  127. 127. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra M E R C I. Avez-vous des questions? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 125
  128. 128. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions 7. Extra Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  129. 129. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Antoine Allombert, G´erard Assayag, and Myriam Desainte-Catherine. A system of interactive scores based on petri nets. In Proc. of SMC ’07, Athens, Greece, 2007. Antoine Allombert, G´erard Assayag, and Myriam Desainte-Catherine. Iscore: a system for writing interaction. In Proc. of 3rd International Conference on Digital Interactive Media in Entertainment and Arts (DIMEA) ’08, pages 360–367, New York, NY, USA, 2008. ACM. Antoine Allombert, G´erard Assayag, M. Desainte-Catherine, and Camilo Rueda. Concurrent constraint models for interactive scores. In Proc. of Sound and Music Computing (SMC) ’06, Marseille, France, 2006. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  130. 130. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra G´erard Assayag, Georges Bloch, Marc Chemillier, Arshia Cont, and Shlomo Dubnov. Omax brothers: a dynamic topology of agents for improvization learning. In Proc. of the 1st ACM workshop on Audio and music computing multimedia (AMCMM) ’06, pages 125–132, New York, NY, USA, 2006. ACM. Antoine Allombert, Pascal Baltazar, Rapha¨el Marczak, Myriam Desainte-Catherine, and Laurent Garnier. Designing an interactive intermedia sequencer from users requirements and theoretical background. In Proc. of International Computer Music Conference (ICMC) ’10, 2010. A. Aristizabal, F. Bonchi, L. Pino, and F. Valencia. Partition refinement for bisimilarity in ccp. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  131. 131. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of the 27th ACM Symposium On Applied Computing (SAC 2012), 2012. Antoine Allombert and Myriam Desainte-Catherine. Interactive scores: A model for specifying temporal relations between interactive and static events. In Journal of New Music Research, 2005. Antoine Allombert, Myriam Desainte-Catherine, J. Larralde, and G´erard Assayag. A system of interactive scores based on qualitative and quantitative temporal constraints. In Proc. of 4th International Conference on Digital Arts (Artech) ’08, Porto, Portugal, 2008. The Artech International Association. Antoine Allombert, Myriam Desainte-Catherine, and Mauricio Toro. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  132. 132. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Modeling temporal constrains for a system of interactive score. In G´erard Assayag and Charlotte Truchet, editors, Constraint Programming in Music, chapter 1, pages 1–23. Wiley, 2011. James F. Allen. Maintaining knowledge about temporal intervals. Communication of ACM, 26, 1983. Antoine Allombert. Aspects temporels d’un syst`eme de partitions num`eriques interactives pour la composition et l’interpr´etation. PhD thesis, Universit´e de Bordeaux, November 2009. Andr´es Aristiz´abal. Bisimilarity in concurrent constraint programming. In 26th International Conference on Logic Programming (ICLP) 2010, 2010. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  133. 133. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Jean Bresson, Carlos Agon, and G´erard Assayag. Openmusic: visual programming environment for music composition, analysis and research. In Proceedings of the 19th ACM international conference on Multimedia, MM ’11, pages 743–746, New York, NY, USA, 2011. ACM. P. Baltazar, A. Allombert, R. Marczak, J.M. Couturier, M. Roy, A. S`edes, and M. Desainte-Catherine. Virage : Une reflexion pluridisciplinaire autour du temps dans la creation numerique. In in Proc. of JIM, 2009. Marc Boyer and Michel Diaz. Non equivalence between time petri nets and time stream petri nets. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  134. 134. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of the The 8th International Workshop on Petri Nets and Performance Models, number 98 in 1, Washington, DC, USA, 1999. IEEE Computer Society. Anthony Beuriv´e and Myriam Desainte-Catherine. Representing musical hierarchies with constraints. In 7th International Conference on Principles and Practice of Constraint Programming, Musical Constraints Workshop, Paphos, 2001. Gerd Behrmann, Alexandre David, Kim G. Larsen, Oliver M¨oller, Paul Pettersson, and Wang Yi. Uppaal - present and future. In Proc. of 40th IEEE Conference on Decision and Control. IEEE Computer Society Press, 2001. Marius Bozga, Conrado Daws, Oded Maler, Alfredo Olivero, Stavros Tripakis, and Sergio Yovine. Kronos: A model-checking tool for real-time systems. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  135. 135. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Anders Ravn and Hans Rischel, editors, Formal Techniques in Real-Time and Fault-Tolerant Systems, volume 1486 of Lecture Notes in Computer Science, pages 298–302. Springer Berlin / Heidelberg, 1998. Joakim Byg, Kenneth Yrke Jørgensen, and Jiˇr´ı Srba. Tapaal: Editor, simulator and verifier of timed-arc petri nets. In Proceedings of the 7th International Symposium on Automated Technology for Verification and Analysis (ATVA) ’09, pages 84–89, Berlin, Heidelberg, 2009. Springer-Verlag. Christel Baier, Joost-Pieter Katoen, and Diego Latella. Metric semantics for true concurrent real time. In Proc. of International Conference on Automata, Languages and Programming (ICALP) ’98, Berlin, Germany, 1998. Springer. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  136. 136. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Arshia Cont. Antescofo: Anticipatory synchronization and control of interactive parameters in computer music. In Proc. of ICMC ’08, 2008. Myriam Desainte-Catherine, Antoine Allombert, and G´erard Assayag. Towards a hybrid temporal paradigm for musical composition and performance: The case of musical interpretation. Computer Music Journal, To appear in fall, 2012. M. Desainte-Catherine and N. Brousse. Towards a specification of musical interactive pieces. In Proc. of the XIX Colloquium on Musical Informatics (CIM), Firenze, Italy, 2003. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  137. 137. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Jos´e Echeveste, Arshia Cont, Jean-Louis Giavitto, and Florent Jacquemard. Formalisation des relations temporelles entre une partition et une performance musicale dans un contexte d’accompagnement automatique : Accompagnement musical automatique. In Journal Europ´een des Syst`emes Automatis´es, 2011. Rosella Gennari. Temporal resoning and constraint programming - a survey. CWI Quaterly, 11:3–163, 1998. Thierry Gautier, Paul Le Guernic, and L¨oic Besnard. Signal: A declarative language for synchronous programming of real-time systems. In Proc. of FPCA ’87, 1987. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  138. 138. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Guillaume Gardey, Didier Lime, Morgan Magnin, and Olivier (H.) Roux. Rom´eo: A tool for analyzing time Petri nets. In 17th International Conference on Computer Aided Verification (CAV’05), volume 3576 of Lecture Notes in Computer Science, pages 418–423, Edinburgh, Scotland, UK, 2005. Springer. Gerard J. Holzmann. The model checker spin. Software Engineering, 23(5):279–295, 1997. Rapha¨el Marczak, Antoine Allombert, and Myriam Desainte-Catherine. Real-time temporal control of musical processes. In Proc. of the International Conferences on Advances in Multimedia (MMEDIA) ’11, 2011. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  139. 139. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Itay Meiri. Combining qualitative and quantitative constraints in temporal reasoning. Artificial Inteligence, 87(1-2):343–385, 1996. M. Nielsen, C. Palamidessi, and F. Valencia. Temporal concurrent constraint programming: Denotation, logic and applications. Nordic Journal of Computing, 1(9):145–188, 2002. Yann Orlarey, Dominique Fober, and Stephane Letz. Syntactical and semantical aspects of faust. Soft Comput., 8(9):623–632, 2004. Carlos Olarte and Camilo Rueda. A Declarative Language for Dynamic Multimedia Interaction Systems. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  140. 140. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of Mathematics and Computation in Music, volume 38, Berlin, Germany, july 2009. Springer. Carlos Olarte, Camilo Rueda, Gerardo Sarria, Mauricio Toro, and Frank Valencia. Concurrent constraints models of music interaction. In G´erard Assayag and Charlotte Truchet, editors, Constraint Programming in Music, chapter 6, pages 133–153. Wiley, Hoboken, NJ, USA., 2011. Carlos Olarte and Frank D. Valencia. The expressivity of universal timed ccp: undecidability of monoadic fltl and closure operators for security. In PPDP ’08: Proceedings of the 10th international ACM SIGPLAN conference on Principles and practice of declarative programming, pages 8–19. ACM, 2008. Fran¸cois Pachet. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  141. 141. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Playing with virtual musicians: the continuator in practice. IEEE Multimedia, 9:77–82, 2002. Miller Puckette, Theodore Apel, and David Zicarelli. Real-time audio analysis tools for pd and max/msp. In Proc. of ICMC ’98, Ann Arbor, USA, 1998. M. Puckette. Pure data. In Proceedings of the International Computer Music Conference. San Francisco 1996, 1996. Nirina Ranaivoson. R´eflexion sur la mise en place des structures logiques dans un logiciel de partitions musicales interactives. Master’s thesis, Universit´e de Bordeaux, 2009. Camilo Rueda and Frank Valencia. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  142. 142. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Proving musical properties using a temporal concurrent constraint calculus. In Proc. of the 28th International Computer Music Conference (ICMC) ’02, 2002. C. Rueda and F. Valencia. On validity in modelization of musical problems by ccp. Soft Computing, 8(9):641–648, 2004. Camilo Rueda and Frank Valencia. A temporal concurrent constraint calculus as an audio processing framework. In SMC ’05, 2005. S. Safra. On the complexity of omega-automata. In Proceedings of the 29th Annual Symposium on Foundations of Computer Science (SFCS) ’88, pages Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  143. 143. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra 319–327, Washington, DC, USA, 1988. IEEE Computer Society. Vijay A. Saraswat. Concurrent Constraint Programming. MIT Press, Cambridge, MA, 1992. Gerardo Sarria. Formal Models of Timed Musical Processes. PhD thesis, Universidad del Valle, Colombia, 2008. Vijay A. Saraswat, Radha Jagadeesan, and Vineet Gupta. Foundations of timed concurrent constraint programming. In Proceedings of the Ninth Annual IEEE Symposium on Logic in Computer Science, pages 71–80. IEEE Computer Press, 1994. Alexander Schimpf, Stephan Merz, and Jan-Georg Smaus. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  144. 144. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Construction of b¨uchi automata for ltl model checking verified in isabelle/hol. In Proceedings of the 22nd International Conference on Theorem Proving in Higher Order Logics, TPHOLs ’09, pages 424–439, Berlin, Heidelberg, 2009. Springer-Verlag. Patrick S´enac, Pierre de Saqui-Sannes, and Roberto Willrich. Hierarchical time stream petri net: A model for hypermedia systems. In Proc. of the 16th International Conference on Application and Theory of Petri Nets, pages 451–470, London, UK, 1995. Springer-Verlag. Mauricio Toro, Carlos Ag´on, G´erard Assayag, and Camilo Rueda. Ntccrt: A concurrent constraint framework for real-time interaction. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  145. 145. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of ICMC ’09, Montreal, Canada, 2009. Mauricio Toro and Myriam Desainte-Catherine. Concurrent constraint conditional branching interactive scores. In Proc. of SMC ’10, Barcelona, Spain, 2010. Mauricio Toro, Myriam Desainte-Catherine, and P. Baltazar. A model for interactive scores with temporal constraints and conditional branching. In Proc. of Journ´ees d’Informatique Musical (JIM) ’10, May 2010. Mauricio Toro, Myriam Desainte-Catherine, and Julien Castet. An extension of interactive scores for multimedia scenarios with temporal relations for micro and macro controls. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  146. 146. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of Sound and Music Computing (SMC) ’12, Copenhagen, Denmark, July 2012. Mauricio Toro, Myriam Desainte-Catherine, and Camilo Rueda. Formal semantics for interactive music scores: A framework to design, specify properties and execute interactive scenarios. Journal of Mathematics and Music, To be published., 2012. Paul Teehan, Mark R. Greenstreet, and Guy G. Lemieux. A survey and taxonomy of gals design styles. IEEE Design Test of Computers, 24(5):418–428, 2007. Mauricio Toro. Structured musical interactive scores (short). In Proc. of the doctoral consortium in ICLP ’10, Edinburgh, Scotland, UK, 2010. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  147. 147. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Mauricio Toro. Structured Interactive Scores: From a simple structural definition to a real-time capable implementation. PhD thesis, Universit´e de Bordeaux, Bordeaux, September 2012. Frank D. Valencia. Temporal Concurrent Constraint Programming. PhD thesis, University of Aarhus, 2002. Frank D. Valencia. Decidability of infinite-state timed ccp processes and first-order ltl. Journal of Theoretical Computer Science - Expressiveness in concurrency, 330(3):557–607, 2005. Lindsay Vickery. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  148. 148. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Non-linear structures for real-time interactive musical works. In Proceedings of the Australasian Computer Music Conference (ACMC) ’03, 2003. Lindsay Vickery. Interactive control of higher order musical structures. In Proc. of ACMC ’04, Victoria University, New Zealand, July 2004. ACMA. Lindsay Vickery. The possibilities of novel format structures through computer controlled live performance. In Proc. of ACMC ’11, 2011. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 127
  149. 149. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra More Future Work Score with conditions Event Structures semantics Block Diagram semantics Abstract semantics Structural Definition Ntcc model Ntcc and Faust interconnected Model checker Temporal Property Operational semantics Pure Data Program Score with audio proc. Score with interval durations Score with arbitrary durations Event Structures semantics Event Structures semantics Stand Alone Programs Programs Music Properties Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 127
  150. 150. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Limitation of time conditional branching b e Choose either d or e after aa c d ∆c = {3} ∆e = {2} ∆d = {1} ∆b = [1, 2] = = = = = = = Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 128
  151. 151. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conditional branching semantics in detail Red (r) ∆r ∆r sr er ∆r s r e r Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 129
  152. 152. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conditional branching semantics in detail φ ∆ ∆ ∆ ∆ ∆ ∆a ∆b ∆a ∆b ∆a Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 130
  153. 153. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conditional branching semantics in detail Green (g) Red (r) Lights (l) a ∆l ∆r ∆g ∆a = {0} ∆r ∆r ∆g ∆g sr sr er er sg sg eg eg a sl el ∆l ∆l Video (e) ee e e e l s l φ ≥ 10 ∆ = {0} Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 131
  154. 154. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Equivalence between interactive score constraints and its event traces Theorem The temporal constraints of a score are equivalent to the temporal constraints of the event structures semantics. Proof. This holds by replacing points in the temporal constraints of the score by its corresponding events. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 132
  155. 155. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Correctness of the operational semantics with respect to the event structures semantics Theorem The possible time units in which a point process can be launched in the operational semantics are included in the possible time units in which an event can appear in an event trace in the tes semantics [TDCR12]. Proof. By induction on the structure of the timed event structures (tes) semantics of a score. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 133
  156. 156. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Correctness of the operational semantics with respect to the event structures semantics Theorem Let s be a score, ε∗ = des(normal(es(s))) = E, l∗ , R its event structures semantics, tc(ε∗ ) its temporal constraints, P = ScoreS,I,Pr,R the ntcc process that represents the score, [[P]] the denotation of the ntcc process, and Ti is the set of indexes j such that [[P]]j+1 entails launchi . It holds for all sequences in [[P]], n = card(E), that T1 × T2... × Tn ⊆ Solutions(tc(ε∗ )). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 134
  157. 157. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Correctness of the operational semantics with respect to the event structures semantics ε∗ Values of Ti for process Scoreε∗ tc(ε∗ ) ((i)) Ti ⊆ [1, n∞] ti ∈ N (i) ∆ → (j) Ti = {1} ∧ Tj ∈ Ti + min(∆) ti ∈ N ∧ tj ∈ ti + ((i)) ∆ → (j) Ti ⊆ [1, n∞] ∧ Tj ∈ Ti + min(∆) ti ∈ N ∧ tj ∈ ti + ((i)) ∆ → ((j)) Ti ⊆ [1, n∞] ∧ Tj ∈ Ti + ∆ ti ∈ N ∧ tj ∈ ti + (i) ∆ → ((j)) Ti = {1} ∧ Tj ∈ Ti + ∆ ti ∈ N ∧ tj ∈ ti + Table : Temporal constraints to launch the events in the ntcc model Vs. temporal constraints of the event structures semantics. We denote an interactive object as ((i)) and a static one as (i). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 135
  158. 158. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Extension of Interactive Scores Karplus (k1) Karplus (k2) Karplus (k3)a b [100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Temporal object Temporal relation High-precision temporal relation Dataflow relation Interactive object Interactive score Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 136
  159. 159. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra The novelty of our extension: Faust and ntcc interconnection 1. A model based on the Non-deterministic Timed Concurrent Constraint (ntcc) calculus for concurrency, user interactions and temporal relations, and 2. a model based on the Faust programming language for sound processing and micro controls. buton Label mouse down mouse up 0 1 user ntcc Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 137
  160. 160. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Extension of Interactive Scores Karplus (k1) Karplus (k2) Karplus (k3)a b [100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Temporal object Temporal relation High-precision temporal relation Dataflow relation Interactive object Interactive score Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 138
  161. 161. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Example: An arpeggio with three strings Karplus (k1) Karplus (k2) Karplus (k3)a b[100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 139
  162. 162. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Example: An arpeggio with three strings a b [2s, 4s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] Figure : The constraint graph Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 140
  163. 163. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Example: An arpeggio with three strings Karplus (k1) Karplus (k2) Karplus (k3) @100 threeCords(f) output sk1 ek1 sk1 ek2 sk2 Figure : Faust’s Block diagram Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 141
  164. 164. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Example: An arpeggio with three strings Figure : Implementation Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 142
  165. 165. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Application: Three user controlled arpeggios ∆ = 10 ∆ = 10 ∆ = 10 Figure : The double-headed arrow represents an inequality (≤) and a white-headed arrow represents an equality relation (=). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 143
  166. 166. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Application: An arpeggio without “clicks” a b [100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] Anti Click Three Karplus (f)Karplus (k1) Karplus' AC[9.5s, 9.5s] [0.5s, 0.5s] Karplus (k2) Karplus' AC [0.5s, 0.5s] ∆k2 = [5s, 10s] [4.5s, 9.5s] Karplus (k3) Karplus' AC [0.5s, 0.5s] ∆k3 = [4s, 4s] [3.5s, 3.5s] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 144
  167. 167. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Application: Sound source perception Karplus (k1) L Output (o1) [0s, 0s] ∆k1 = [10s, 10s] ∆o1 = [10s, 10s] R Output (o2) ∆o2 = [10s, 10s] Karplus (k1) L Output (o1) ∆k1 = [10s, 10s] ∆o1 = [10s, 10s] R Output (o2) ∆o2 = [10s, 10s] 500µs Time Time Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 145

×