Paper presented by Frederic Andres at The 10th International Conference on Management of Emergent Digital EcoSystems (MEDES'18). (Tokyo, Japan - September 25-28, 2018) Authors: Estêvão Bissoli Saleme, Celso A. Saibel Santos, Ricardo de Almeida Falbo, George Ghinea, and Frederic Andres DOI: https://doi.org/10.1145/3281375.3281378

1. Towards a Reference Ontology on
Mulsemedia Systems
SALEME1, E. B., SANTOS1, C. A. S., FALBO1, R. A., GHINEA2, G., ANDRES 3, F.
1Federal University of Espírito Santo, Brazil
2Brunel University London, UK
3National Institute of Informatics, Japan
The 10th International Conference on Management of
Emergent Digital EcoSystems (MEDES'18)

2. 2MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
Outline
▪ Introduction and motivation
▪ Our Research
▪ Brief Background on Mulsemedia
▪ MulseOnto
▪ Scope and Requirements
▪ Mulsemedia System sub-ontology
▪ Medium sub-ontology
▪ Sensory Effects sub-ontology
▪ Evaluation and results
▪ Conclusion and Future Works

3. 3MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
Intro and Motivation
▪ Multimedia -> requirements stems from the continuous,
temporal nature of audio and video
▪ Mulsemedia -> integrate these classic media types with non-
traditional ones such as olfactory, haptic, and gustatory
▪ Heterogeneous digital ecosystem
▪ Lack of formal and explicit representation of what
mulsemedia is
▪ Potential issues:
▪ Misconception -> lame (weak) solutions
▪ Overlook standardization, reuse, compatibility, and so on

4. 4MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
Our research
▪ Capture the common conceptualization of Mulsemedia systems
and what it entails to
▪ Reduce ambiguity and inaccuracy by creating an ontology
▪ Make the best possible description of the domain in reality and
then improve communication, learning and problem-solving
▪ What did we do?
▪ Developed a domain reference ontology (≠ operational ontology) -
> common vocabulary
▪ Big picture of mulsemedia (entities, interactions, kinds of
sensory effects, physical realizations)
▪ Evaluation: assessment by humans and a data-driven
approach

5. 5MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
Brief Background on Mulsemedia
▪ Common concepts:
▪ Mulsemedia system -> a multimedia system (constituted of
programs) + at least one non-traditional media (embed in
mulsemedia programs or descriptions)
▪ Media types and creation -> text, graphics, sound, and
motion created by humans, acquired through sensors, or
synthesized
▪ Sensory Effects -> contain light, vibration, spraying, wind,
scent, taste, temperature descriptions
▪ Mulsemedia Presentation -> it coordinates the integration
of outputs emanating from the media and sensory-effect
specific devices and considers resource limitations, user
preferences, and other presentation features

6. 6MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
MulseOnto - Scope
▪ What mulsemedia systems and their small parts are
▪ Which entities they interact with
▪ Where the sensory effects come from
▪ What physical realizations different kinds of media do in this
context
▪ The current MulseOnto does not intend to answer:
▪ QoE related aspects, how stimuli are perceived, how media are loaded,
composed and presented, timing aspects management, set up of systems
and devices, neither attribute details (computer media extensions,
encoding methods for computer medium, kinds of flavors or scent, the
intensity of effects, temporality constraints, colors, etc)

7. 7MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
MulseOnto - Requirements
▪ Functional requirements specified as CQs

8. 8MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
MulseOnto - Requirements
▪ Non-functional requirements:
▪ SEON is organized in layers:
▪ Top level: Unifed Foundational Ontology (UFO), a foundational ontology
that is based on a number of theories from Formal Ontology, Philosophical
Logics, Philosophy of Language, Linguistics and Cognitive Psychology
▪ Core level: on software (Software Ontology - SwO) and software process
(Software Process Ontology - SPO)

9. 9MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
MulseOnto – sub-ontologies

10. 10MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
MulseOnto – Mulsemedia sub-
ontology
1
2
3

11. 11MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
MulseOnto – Medium sub-ontology

12. 12MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
MulseOnto – Sensory Effects sub-
ontology

13. 13MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
Evaluation and Results
▪ First: expert judgments to analyse whether concepts and
relations defined in the ontology are able to answer their
competency questions
▪ Are there elements besides the point? If so, they are not part
of this current ontology
▪ Second: data-driven approach -> mulsemedia system instance
taking into account concepts and relations
▪ Validated whether MulseOnto is capable of representing a
real-world case through an elastic hypothetical
mulsemedia scenario

14. 14MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
Hypothetical mulsemedia scenario
▪ It involves all of the concepts presented in the ontology in
terms of instances
A recently created science museum in Tokyo will provide a multi-sensory experience to its visitors. It
is elaborating an astonishing exhibition on exploring the earth through places like volcanoes,
glaciers, earthquakes, aurora borealis, waterfalls, and locations with exotic food. The goal is that
visitors not only see videos and pictures of those places, but also feel immersed in each environment
they are experiencing. To do so, the museum will be using a mulsemedia platform called PlaySEM,
which has a video player and a renderer that support sensory effects. The video-clips are annotated
with sensory effect descriptions in MPEG-V (with the help of SEVino), that include light, vibration,
spraying, kinesthetic, tactile, wind, scent, taste, fog, and temperature effects. Visual media will be
displayed on screens, projectors, head-mounted displays. Some visual media objects will also be read
from a behavior presentation specification and be presented in holographic devices. LED strips and
lightbulbs will come up with lighting effects. Eventually, a live event from a real place can be directly
transmitted to the museum. The behavior presentation specification will cater to mulsemedia
program with audio media sources. To provide auditory experiences, the museum will arrange box
speakers, headphones, earbuds, and rumble devices. To deliver haptic experiences, there will be a
motion chair, local air conditioners, spray machines, wind turbines and vibration vests. Olfactory and
gustatory effects will be conveyed in two ways. Scent diffusers and electric smell devices to feel the
smell of the object media presented to the visitors. A digital lollipop and a beverage electronic device
will be set for taste.

15. 15MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
Evaluation and Results
Excerpt of Competency Questions Verification
Excerpt of MulseOnto Instantiation

16. 16MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
Conclusion
▪ Mulsemedia is not easy to understand at a glance -> complex digital
ecosystem and different views
▪ We established a common conceptualization of mulsemedia systems,
described a common vocabulary to discuss mulsemedia solutions
▪ We captured and adapted the ontology to reflect concepts sometimes
neglected when reading a plain English text by
▪ Evaluating the reference ontology against the competency
questions
▪ Instantiating the concepts to see if a hypothetical mulsemedia
system adheres to it
▪ MulseOnto -> initial step to discuss the mulsemedia ecosystem that
involves capturing, distributing, rendering and perceiving sensory
effects

17. 17MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
Future Works
▪ Besides our elastic hypothetical mulsemedia system, further
scenarios from real-world environments must be subject of
study to strengthen the accuracy of our ontology
▪ Create a mulsemedia ontology network to cope with the
process of capturing, distributing, rendering and perceiving
sensory effects in detail
▪ Materialization of this conceptual model to an operational
ontology to exchange sensory effects metadata
▪ Model to integrate different mulsemedia standards enabling
interoperability between different solutions by mapping them
to this reference ontology

18. 18MEDES 2018 - Towards a Reference Ontology on Mulsemedia Systems
Thanks for your attention!
Questions?
Frederic Andres
andres at nii.ac.jp
Estêvão B. Saleme, Celso A. S. Santos, Ricardo A. Falbo, Gheorghita Ghinea, and Frederic
Andres. 2018. Towards a Reference Ontology on Mulsemedia Systems. In Proceedings of
International Conference on Management of Emergent Digital EcoSystems, Tokyo, Japan,
September, 2018 (MEDES’18), 8 pages. https://doi.org/10.1145/3281375.3281378