Paper presented by George Ghinea at MMSYS'18, 9th ACM on Multimedia Systems Conference (Amsterdam, Netherlands - June 12 - 15, 2018) Authors: Estêvão Bissoli Saleme, Celso A. Saibel Santos, and George Ghinea DOI: https://doi.org/10.1145/3204949.3204965

1. Improving Response Time Interval in
Networked Event-based Mulsemedia
Systems
SALEME1, E. B., SANTOS1, C. A. S., and GHINEA2, G.
1Federal University of Espírito Santo, Brazil
2Brunel University London, UK

2. 2ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
Outline
▪ Introduction and motivation
▪ Our research
▪ Issues in networked event-based mulsemedia systems
▪ Experimental environment
▪ Test setup, mock applications, PlaySEM SER
▪ Procedure
▪ Results
▪ Conclusion

3. 3ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
Intro and Motivation
▪ Multimedia -> requirements stems from the continuous,
temporal nature of audio and video
▪ Mulsemedia -> integrate these classic media types with non-
traditional ones such as olfactory, haptic, and gustatory
▪ New valences: lingering effects, distribution, temporality,
production and rendering
▪ Acceptable delays can vary depending on the context of the
interaction (primary media, type of interaction, feedback, etc.)
▪ Nonetheless, some types of effects such as vibration and
lighting require prompt responses in contrast to lingering
effects.

4. 4ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
Our research
▪ Main goal -> improve temporal aspects in networked event-
based mulsemedia systems through the:
▪ Addition of lightweight communication protocols to reduce
network overhead
▪ Pre-processing sensory effects metadata (SEM) to give swift
responses to event-based applications
▪ Event-based applications: activated by events that occur in the
virtual world, such as an explosion in a game, or as a response
to a stimulus from the real world captured through sensors by
the virtual world. They MUST BE responsive
▪ Ex.: Games, VR and interactive applications

5. 5ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
Issues in networked event-based
mulsemedia systems
▪ SALEME et al. (2017) highlighted four major aspects needed to
be dealt with to improve networked event-based mulsemedia
systems:
i. repetitive actions of sending metadata scripts in real time
ii. high number of exchanged messages
iii. use of a default wireless network
iv. use of suitable programming languages for processing
sensory effect metadata described in MPEG-V
SALEME, E. B., Celestrini, J. R., and Santos, C. A. S. 2017. Time Evaluation for the Integration of a Gestural Interactive Application with a Distributed Mulsemedia Platform. In Proceedings
of the 8th ACM on Multimedia Systems Conference (MMSys'17)

6. 6ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
PlaySEM SER -> decoupled software to process SEM
and convert it into commands for different
hardware allowing timeline and event-based apps to
support mulsemedia.
Experimental environment
▪ What did we do?
▪ FIRST: we changed the
architecture of the PlaySEM
SER to support lightweight
protocols such as CoAP,
MQTT, and WebSocket

7. 7ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
Past version of PlaySEM SER
▪ SECOND: we changed the
communication strategy of
the PlaySEM SER
▪ How was it?
▪ GIA integrated with the
PlaySEM SER
▪ In each event, a SEM was
transmitted from the GIA to
the PlaySEM SER
▪ This process ran again and
again doing repetitive
tasks for the same event

8. 8ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
Current version of PlaySEM SER
▪ How did we do to improve it? Two steps communication
strategy:
1) Loading process (SEM sent when handshaking) 2) Optimized real-time flow (whenever a gesture occurs)

9. 9ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
Setup and Procedure
Computer A
GIA (4 mocks)
Router R
CoAP, MQTT, WebSocket,
and UPnP messages
For each protocol:
i) 8 MPEG-V SEM transmitted (loading process)
ii) 400 interactions (50 for each action)
iii) Network communication captured through Wireshark
CoAP time = CON (client) on endpoint SetPlayEvent – ACK message (server)
MQTT time = Publish Message on topic SetPlayEvent – Publish Complete
UPnP time = POST on service SetPlayEvent – HTTP/1.1 200 OK
WebSocket time = Websocket text with JSON SetPlayEvent – [ACK]
SETUP
GIA: Intel Core i7-6700HQ, 16GB
RAM, 254GB SSD, running on
Windows 10 64 bits
PlaySEM SER: Intel Core i7-3537U,
4GB RAM, 128 GB SSD, running on
Windows 8.1 64 bits
Router: Wireless Dual Band AC750
Archer C20 (wireless turned off)
Computer B
PlaySEM SER

10. 10ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
Results
1. CoaP is the fastest followed by
MQTT, UPnP, and WebSocket
2. SE mean -> little variability
between sample means
3. SD higher for faster protocols
(sensitive to variations)
1 2
3

11. 11ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
Results
CoAP-> fastest, more
variability
WebSocket -> slowest,
less dispersion

12. 12ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
Protocols timing and selection
▪ Unlike timeline applications, event-based ones are not deterministic.
An event might happen unexpectedly
▪ Synchronism must be tight, and response to an interaction should be
given swiftly. Target: cut down time wasting when integrating an
event-based application to the PlaySEM SER via network to avoid
undesirable delays
▪ CoAP and MQTT tend to deliver a response quickly (1.4ms and 3.2ms).
UPnP ~ 8ms. WebSocket hit 51ms on average. Despite being the
slowest, the latter is handy for web apps
▪ Developers may devise their own communication protocol to enhance
even more the response time. But we are looking for interoperable
and reusable solutions

13. 13ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
Communication strategy optimization
▪ Packaging (t2 ), Transmission (t3 ) and Metadata processing (t4 ) tasks
eliminated in real-time (loading process created to pre-process SEM). The
gain over the original strategy was between 19ms and 46ms
▪ t5 decreased for CoAP and MQTT, held steady for UPnP, and increased
significantly for WebSocket. Tc , Tm , and Tu were far superior the original Tu
strategy whereas Tw is considered equivalent in terms of superior threshold
1
2

14. 14ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
Conclusion
▪ Pre-processing MPEG-V SEM for real-time communication in
networked event-based mulsemedia systems cut down time wasting
significantly
▪ The use of lightweight communication protocols such as CoAP and
MQTT boost response time up to four times compared to the native
way of the PlaySEM’s prior strategy. Worst case for CoAP, MQTT, and
UPnP outperforms the best case of the original strategy.
▪ This scenario is useful for any event-based mulsemedia application
such as games, VR/AR software, and interactive applications that
describes its sensory effects in MPEG-V whereby the renderer is
decoupled from the application
▪ Future works: find the influence of computational delays on the user’s
experience and add support to other connectivity protocols to
transform the PlaySEM SER into a framework

15. 15ACM Multimedia Systems 2018 - Improving Response Time Interval in Networked Event-based Mulsemedia Systems
Thanks for your attention!
Questions?
George Ghinea
george.ghinea at brunel.ac.uk
Estêvão Bissoli Saleme, Celso A. S. Santos, and Gheorghita Ghinea. 2018. Improving response
time interval in networked event-based mulsemedia systems. In MMSys’18: 9th ACM
Multimedia Systems Conference, June 12–15 2018, Amsterdam, Netherlands. ACM, New York,
NY, USA, 9 pages. https://doi.org/10.1145/3204949.3204965