Slides from the ACM Multimedia 2014 (Full Paper) Paper: Benjamin Rainer, Christian Timmerer, Self-Organized Inter-Destination Multimedia Synchronization For Adaptive Media Streaming, In Proceedings of the 22st ACM International Conference on Multimedia (ACM, ed.), ACM, New York, NY, USA, pp. 10, 2014.

1. SELF-ORGANIZED
INTER-DESTINATION MULTIMEDIA
SYNCHRONIZATION
FOR ADAPTIVE MEDIA STREAMING
Benjamin Rainer, Christian Timmerer
Alpen-Adria-Universität Klagenfurt,
Institute of Information Technology

2. TYPES OF SYNCHRONIZATION
§ Intra-Stream Synchronization
§ Avoid jitter between the presentation of two consecutive media units
§ Inter-Stream Synchronization
§ E.g., Audio + Video + Subtitles
Network Mul$media
Playback
Video
Time
(t)
Audio
Time
(t)
Receiver
Network
40
ms
Time
(t)
Receiver
Video Playback
Slide 2 of 19

3. MOTIVATION
§ Why do we need Inter-Destination Multimedia
Synchronization?
§ Friends are watching a soccer match and they communicate via text messaging/
phone/…
User
1
User
2
Goal!
Did
you
see
the
goal?
Which
goal?
Thanks
for
the
spoiler!
Slide 3 of 19

4. STATE OF THE ART
§ Most IDMS solutions
§ Extend sender/receiver reports of RTCP [4]
§ Assume multicast [4, 18]
§ Demand a centralized instance [4, 24]
• Master-/Slave- scheme
• Synchronization Master
§ Push-based multimedia streaming [4, 24, 9, 18]
§ Overcoming the asynchronism
§ By pausing and skipping media content
§ Increasing or decreasing the playback rate [17]
• Adaptive Media Playout (AMP)
Slide 4 of 19

5. OUR APPROACH
Content Provider
Application Layer
Peer-to-Peer Overlay
Geographically
Distributed
Clients
MPD Provides MPDs
MPD
enriched with
Session Information
MPD Server
Content Servers
§ Research Objectives
§ Adapt MPEG-DASH for IDMS
§ Agree on a reference in a distributed and self-organized manner
§ Carry out the actual synchronization more smartly than using plain AMP
Slide 5 of 19

6. SESSION MANAGEMENT
§ Inter-Destination Multimedia Synchronization Session Object
(ISO)
§ Time bounded entity, contains a set of peers, uniquely identifiable
§ (IP, port) and the type of the Network Address Translator (NAT)
• NAT covered in the paper
§ Allows a (unique) numbering of peers
§ ISO is identified by session key
§ Provided by 3rd party application or the user
§ Integrated into the MPD of MPEG-DASH
§ Server imports the corresponding ISO when requested
§ E.g., a peer requests the MPD with a session key
Slide 6 of 19

7. SYNCHRONIZATION
§ Two phase synchronization using non reliable communication (UDP)
§ Coarse synchronization
§ Fine synchronization
§ Coarse synchronization
§ Overlay creation
§ Educated guess where to start downloading
§ Fine synchronization
§ Distributed algorithm – Merge and Forward
§ Periodical sends constants sized message to neighbors
§ Negotiate on reference playback timestamp
§ Overcome the identified asynchronism
§ Dynamic Adaptive Media Playout
Slide 7 of 19

8. COARSE SYNCHRONIZATION
§ Overlay creation
§ Ask peers in the ISO for their current Playback Timestamp
(PTS) + NTP TS
• If the response is received, the peer is added to the list of
known peers
§ May lead to a not fully connected network
• E.g., due to packet loss
§ Strategies for selecting the segment to start with
§ Maximum PTS from peers
§ Minimum PTS from peers
§ Weighted average PTS from peers
Slide 8 of 19

9. MERGE AND FORWARD - EXAMPLE
1
2
3
Initial state:
{BF, LID, HID, Cnt, ATS}
1: {{1}, 1, 1, 1, P1}
2: {{2}, 2, 2, 1, P2}
3: {{3}, 3, 3, 1, P3}
1
2
3
τ =1
1: {{1,2}, 1, 2, 2, (P1+P2)/2}
2.1: {{1,2}, 1, 2, 2, (P1+P2)/2}
3: {{2,3}, 2, 3, 2, (P2+P3)/2}
τ =1 1
τ = 2
BF
…
Bloom
filter
Cnt
…
Cumula$ve
Count
H/LID
…
Lowest/Highest
Peer
ID
ATS
…
(Weighted)
Average
TS
Slide 9 of 19
1
2
3
1: {{1,2}, 1, 2, 2, (P1+P2)/2}
2: {{1,2,3}, 1, 3, 3, (P3+2*(P1+P2/)/2)/3}
3: {{2,3}, 2, 3, 2, (P2+P3)/2}
2
3
1: {{1,2,3}, 1, 3, 3, (P1+P2+P3)/3}
2: {{1,2,3}, 1, 3, 3, (P1+P2+P3)/3}
3: {{1,2,3}, 1, 3, 3, (P1+P2+P3)/3}

10. M&F - EVALUATION
§ Simulation environment OMNeT++ with INET
framework
§ Random networks (Erdős-Rényi)
§ 40, 60, and 80 peers
§ Probabilities for creating connections between peers: 0.1
to 0.9 (uniformly distributed)
§ Period of 250ms
§ RTT of 300ms between peers
§ Compared to Aggregate
§ Periodically sends list of known playback timestamps and
NTP timestamps [10]
§ Peers extend this list and/or update their PTS and NTP TS
Slide 10 of 19

11. M&F - EVALUATION
§ Y-axis denotes the average traffic generated per peer in kbit
§ X-axis denotes the connectivity of the overlay network
Slide 11 of 19

12. M&F - EVALUATION
§ Y-axis denotes the time required for the synchronization process
§ X-axis denotes the connectivity of the overlay network
Slide 12 of 19

13. DYNAMIC ADAPTIVE MEDIA PLAYOUT
§ Increase/decrease playback rate
§ Pausing decreases QoE [7]
§ Overcome asynchronism by increasing or decreasing the playback
rate
§ Select those content sections which mask the playback rate variation
§ Content features for measuring the distortion caused by AMP
§ Audio: spectral energy of audio frames
§ Video: motion intensity between consecutive video frames
§ Metrics for the distortion
§ Difference between the impaired and the unimpaired case for both
modalities
Slide 13 of 19

14. DYNAMIC ADAPTIVE MEDIA PLAYOUT
§ Combined metric:
f (X) =|| d(X) ||2, d(X)= (dv (X), da (X))T
§ Constrained optimization problem for finding appropriate content
sections:
x1...
x2...
x3...
ξ...
L...
B...
bc...
br...
T...
tmax...
X ∈ R3
(1)argmin
X
f (X)
sign(ξ ) −1) ⋅ sign(ξ ) = ξ
(2)x2 ⋅ (x3
(3)L ≤ B − x2 ⋅ x3 + x2 ⋅
bc
br
(4)x1 ≤ T
(5)x2 ≤ tmax
start time
duration
playback rate
asynchronism
lower buffer threshold
current buffer fill state
content bit-rate
bandwidth
maximum start time
maximum duration
Slide 14 of 19

15. DYNAMIC AMP - EVALUATION
§ Subjective Quality Assessment using Crowdsourcing
§ Microworkers
§ 15 minutes
§ $0.25 as reward
§ 80 participants
§ Sequences
§ Babylon A.D. for training
• {1, 0.5, 2} times the nominal playback rate
§ Big Buck Bunny for the main evaluation
• {0.5, 0.6, 0.8, 1,1.2, 1.4, 1.6, 1.8, 2} times the nominal playback rate
• Selected content sections
Slide 15 of 19

16. DYNAMIC AMP - EVALUATION
§ Y-axis denotes the Mean Opinion Score, x-axis denotes the average f(X) and playback
rate (μ)
§ High linear correlation between distortion metric and QoE
• μ > 1:
• μ < 1:
ρ = 0.975
ρ = −0.995
Slide 16 of 19

17. Slide 17 of 19
DEMO
§ Demo video available on YouTube
§ https://www.youtube.com/watch?v=2V9rO5SbI7A
§ Source Code available at: https://github.com/grishnagkh/mf

18. CONCLUSION
§ Introduced IDMS to Adaptive Streaming
§ MPEG-DASH
§ Distributed Control Scheme that scales with the number of peers
§ Can be combined with any streaming protocol
§ Not coupled with the session management or the overlay creation
§ Dynamic AMP for carrying out the actual synchronization
§ (General) Optimization problem that aims on finding appropriate content
sections
§ Demo video available on YouTube
§ https://www.youtube.com/watch?v=2V9rO5SbI7A
§ Search for: MergeAndForward
§ Source Code available at: https://github.com/grishnagkh/mf
Slide 18 of 19

19. THANK YOU!
Q & A

20. REFERENCES
[4] F. Boronat Segu, J. Guerri Cebollada, and J. Lloret Mauri. An RTP/RTCP based
approach for multimedia group and inter-stream synchronization. MTAP, pp. 40:285-319,
2008.
[9] C. Hesselman, D. Abbadessa, W. Van Der Beek, D. Gorgen, K. Shepherd, S. Smit, M.
Gulbahar, I. Vaishnavi, J. Zoric, D. Lowet, R. De Groote, J. O'Connell, and O. Friedrich.
Sharing enriched multimedia experiences across heterogeneous network infrastructures.
IEEE Comm. Mag., pp. 48(6):54-65, 2010.
[10] T. Hossfeld, M. Seufert, M. Hirth, T. Zinner, P. Tran-Gia, and R. Schatz. Quantification
of YouTube QoE via Crowdsourcing. In IEEE ISM, pages 494{499, 2011.
[17] M. Montagud and F. Boronat. On the Use of Adaptive Media Playout for Inter-
Destination Synchronization. IEEE Communications Letters, pp. 15(8):863-865, 2011.
[18] M. Montagud, F. Boronat, and H. Stokking. Design and Simulation of a Distributed
Control Scheme for Inter-destination Media Synchronization. In IEEE 27th AINA, pp.
937-944, March 2013.
[24] H. Stokking, M. Van Deventer, O. Niamut, F. Walraven, and R. Mekuria. IPTV inter-destination
synchronization: A network-based approach. In 14th ICIN, pp. 1-6, 2010.
Slide 20 of 19