An extension of interactive scores for multimedia scenarios with temporal micro and macro controls

1. Introduction Extension Applications Current Work
An extension of interactive scores for
multimedia scenarios with temporal micro
and macro controls
Mauricio TORO∗
– LaBRI, Universit´e de Bordeaux.
R´eunion avec Yann Orlarey
∗joint work with Myriam Desainte-Catherine and Julien Castet
December 5th 2011

2. Introduction Extension Applications Current Work
Problems with existing tools
• Time models are unrelated
• No hierarchy
• Time scales are unrelated

3. Introduction Extension Applications Current Work
Problems with Max and Pd
• Max and Pd
• They are no compositional tools
• The scheduler is not good

4. Introduction Extension Applications Current Work
Interactive Scores
• Interactive scores is a formalism for the design of
scenarios.
• We study interactive scores limited to
• hierarchical relations represented as a directed tree and
• point-to-point temporal relations without disjunction nor
inequality (=) [2] 1 and quantitative temporal relations.
1
which are equivalent to Allen’s relations without disjunction [1]

5. Introduction Extension Applications Current Work
Interactive Scores
• Interactive scores is a formalism for the design of
scenarios.
• We study interactive scores limited to
• hierarchical relations represented as a directed tree and
• point-to-point temporal relations without disjunction nor
inequality (=) [2] 1 and quantitative temporal relations.
1
which are equivalent to Allen’s relations without disjunction [1]

6. Introduction Extension Applications Current Work
Solution
• Extend interactive scores with micro controls and signal
processing
• Ntcc for control and user events, and Faust for micro
controls and signal processing.
buton
Label
mouse
down
mouse
up
0
1
user
ntcc

7. Introduction Extension Applications Current Work
Extension to Interactive Scores
• Temporal objects
• Interactive objects
• Temporal relations
• Micro temporal relations [n, n]
• Dataflow relations

8. Introduction Extension Applications Current Work
Example of Dataflow relations
Acquisition (y)
Delay (x)
Filter (z)
Two diffusions (u)
Sound (v)
Output (o)
Dataflow
Figure: Dataflow Vs. Time view of a score. Tick arrows represent
the flow of data over time.

9. Introduction Extension Applications Current Work
Example: An arpeggio with three strings
Karplus (k1)
Karplus (k2)
Karplus (k3)a
b[100smp, 100smp]
[2s, 4s]
[0s, 0s]
[0s, 0s]
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
ThreeStrings(f)
Figure: The Score

10. Introduction Extension Applications Current Work
Example: An arpeggio with three strings
a
b
[2s, 4s]
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
Figure: The constraint graph

11. Introduction Extension Applications Current Work
Example: An arpeggio with three strings
Karplus (k1)
Karplus (k2)
Karplus (k3)
@100
threeCords(f)
output
sk1
ek1
sk1
ek2
sk2
Figure: Faust’s Block diagram

12. Introduction Extension Applications Current Work
Example: An arpeggio with three strings
Figure: Implementation

13. Introduction Extension Applications Current Work
Three user controled arpeggios
∆ = 10 ∆ = 10 ∆ = 10
Figure: The double-headed arrow represents an inequality (≤) and
a white-headed arrow represents an equality relation (=).

14. Introduction Extension Applications Current Work
The anti “click” score
a
b
[100smp, 100smp]
[2s, 4s]
[0s, 0s]
[0s, 0s]
∆k1 = [10s, 10s] Anti Click Three Karplus (f)Karplus (k1)
Karplus' AC[9.5s, 9.5s] [0.5s, 0.5s]
Karplus (k2)
Karplus' AC [0.5s, 0.5s]
∆k2 = [5s, 10s]
[4.5s, 9.5s]
Karplus (k3)
Karplus' AC [0.5s, 0.5s]
∆k3 = [4s, 4s]
[3.5s, 3.5s]

15. Introduction Extension Applications Current Work
A delay of 500µs
Karplus (k1)
L Output (o1)
[0s, 0s]
∆k1 = [10s, 10s]
∆o1 = [10s, 10s]
R Output (o2) ∆o2 = [10s, 10s]
Karplus (k1)
L Output (o1)
∆k1 = [10s, 10s]
∆o1 = [10s, 10s]
R Output (o2) ∆o2 = [10s, 10s]
[500µs, 500µs]
Time
Simultaneity
Time
Data Flow

16. Introduction Extension Applications Current Work
Example of Dataflow relations
Acquisition (y)
Delay (x)
Filter (z)
Two diffusions (u)
Sound (v)
Output (o)
Dataflow
Figure: Dataflow Vs. Time view of a score. Tick arrows represent
the flow of data over time.

17. Introduction Extension Applications Current Work
Translation to Faust (1)

18. Introduction Extension Applications Current Work
Translation to Faust (2)

19. Introduction Extension Applications Current Work
Translation to Faust (3)

20. Introduction Extension Applications Current Work
From Score to Faust –Mauricio
∆
∆
δ
δ
@( , δ) @( , δ)
s1
s2
s1 e1
e2
e1
control
s
e
Figure: Control signals for start and end of a temporal object.

21. Introduction Extension Applications Current Work
From Score to Faust –Mauricio
input
output
delay
0
0
input
output
delay
checkbox
∆
∆
∆
Figure: The audio output is delayed.

22. Introduction Extension Applications Current Work
From Score to Faust –Mauricio
input
output
10 smp
input
ouput
checkbox
∆
∆@( , δ)
Figure: The audio output starts after the input.

23. Introduction Extension Applications Current Work
Example of Dataflow relations
Acquisition (y)
Delay (x)
Filter (z)
Two diffusions (u)
Sound (v)
Output (o)
Dataflow
Figure: Dataflow Vs. Time view of a score. Tick arrows represent
the flow of data over time.

24. Introduction Extension Applications Current Work
Temporal object calculus –Myriam
input
delayed
+ gain
+ gain + gain
d
d
Figure: Thin arrows are macrotemporal relations, dashed arrows
are micro temporal, tick arrows are dataflow. Horizontal axis is the

25. Introduction Extension Applications Current Work
Temporal object calculus –Myriam
Let u be x(0); x(t + φ), x be a delay, f be a Faust filter.
We have an expresion o(uzxy; v).
o(uzxy; v)
o(uzys,e
s+δ,e+δ; v)
o(uf (ys,e
s+δ,e+δ); v)
o(uf (ys,e
s+δ,e+δ); v)
o(f (ys,e
s+δ,e+δ); f (ys+φ,e+φ
s+δ+φ,e+δ+φ); v)
of (ys,e
s+δ,e+δ); of (ys+φ,e+φ
s+δ+φ,e+δ+φ); ov

26. Introduction Extension Applications Current Work
M E R C I !!!

27. Introduction Extension Applications Current Work
J. F. Allen.
Maintaining knowledge about temporal intervals.
Communication of ACM, 26, 1983.
R. Gennari.
Temporal resoning and constraint programming - a survey.
CWI Quaterly, 11:3–163, 1998.