PlaySEM: a Platform for Rendering MulSeMedia Compatible with MPEG-V
•
1 like•482 views
Paper presentation by Celso A. S. Santos in the XXI Brazilian Symposium on Multimedia and the Web (October 27-30, 2015, Manaus) Authors: Estêvão B. Saleme and Celso A. Saibel Santos DOI: http://dx.doi.org/10.1145/2820426.2820450 PlaySEM demo video is available at: http://youtu.be/aka4jBZcoKk Another application using SER: http://youtu.be/a5QPsB2zKpk
![“Traditional” multimedia (video, audio, text, ...)
MulSeMedia - stimulate other human senses:
◦ Mechanoreceptors, Thermoreceptors, Chemoreceptors
Subjective experiments -> QoE improvement compared
to Media without sensory effects (WALTL et. al, 2009)
WALTL, M.; TIMMERER, C.; HELLWAGNER, H. A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects.
In: First International Workshop on Quality of Multimedia Experience (QoMEX 2009). [s.n.], 2009.
Introduction](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
Viewers also liked
Similar to PlaySEM: a Platform for Rendering MulSeMedia Compatible with MPEG-V
Similar to PlaySEM: a Platform for Rendering MulSeMedia Compatible with MPEG-V (20)
Recently uploaded
Recently uploaded (20)
PlaySEM: a Platform for Rendering MulSeMedia Compatible with MPEG-V
- 1. PlaySEM: a Platform for Rendering MulSeMedia Compatible with MPEG-V Estêvão SALEME & Celso. A. S. SANTOS Manaus, AM, Brazil – 28/10/2015 Networks and Multimedia Research Lab – LPRM Federal University of Espírito Santo – UFES
- 2. “Traditional” multimedia (video, audio, text, ...) MulSeMedia - stimulate other human senses: ◦ Mechanoreceptors, Thermoreceptors, Chemoreceptors Subjective experiments -> QoE improvement compared to Media without sensory effects (WALTL et. al, 2009) WALTL, M.; TIMMERER, C.; HELLWAGNER, H. A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects. In: First International Workshop on Quality of Multimedia Experience (QoMEX 2009). [s.n.], 2009. Introduction
- 3. How it works? WALTL, M.; TIMMERER, C.; HELLWAGNER, H. A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects. In: First International Workshop on Quality of Multimedia Experience (QoMEX 2009). [s.n.], 2009. Sensory Effects (WALTL et. al, 2009)
- 4. Problem • Are there any integration or interoperability issues with other applications? • The development of interoperable solutions (MPEG-V) is required. Heterogeneous solution • Is it not feasible to reutilize the sensory effects processor? • There is a need to reutilize solutions in different ways, such as Media Players, TV Digital, games, etc. Strong coupling
- 5. New approach for simulating and rendering sensory effects, independently from Media Players, communicating through UPnP that is compatible with MPEG-V It provides: ◦ SE Video Player ◦ MPEG Metadata library ◦ Simple platform for simulating and rendering sensory effects (SE Video Player + SE Renderer) ◦ Open-source, fuel for future researches on the sensory effects area Objective of our work
- 6. Use embedded sensory effect renderers (guide virtual x real world): i. SEMP (WALTL et al., 2013) ii. SESim (WALTL et al., 2013) iii. Sensorama (CHO, 2010) iv. Sensible Media Simulator (KIM; JOO, 2014) WALTL, M. et al. An end-to-end tool chain for sensory experience based on mpeg-v. Sig. Proc.: Image Comm., v. 28, n. 2, p. 136–150, 2013. KIM, S.-K.; JOO, Y. S. Sensible media simulation in an automobile application and human responses to sensory effects. ETRI Journal, ETRI, v. 35, n. 6, p. 1001–1010, 2014. ISSN 1225-6463. CHO, H.-Y. Event-Based control of 4D effects using MPEG RoSE. 2010. Sensory Effects Media Players
- 7. SEMP WALTL, M. et al. An end-to-end tool chain for sensory experience based on mpeg-v. Sig. Proc.: Image Comm., v. 28, n. 2, p. 136–150, 2013. (WALTL et. al, 2013)
- 8. SESim WALTL, M. et al. An end-to-end tool chain for sensory experience based on mpeg-v. Sig. Proc.: Image Comm., v. 28, n. 2, p. 136–150, 2013. (WALTL et. al, 2013)
- 9. Sensorama CHO, H.-Y. Event-Based control of 4D effects using MPEG RoSE. 2010. (CHO, 2010)
- 10. Sensible Media Simulation KIM, S.-K.; JOO, Y. S. Sensible media simulation in an automobile application and human responses to sensory effects. ETRI Journal, ETRI, v. 35, n. 6, p. 1001–1010, 2014. ISSN 1225-6463. (KIM; JOO, 2014)
- 11. Sensory Effect Renderer is decoupled from the presentation interface ◦ It can be used with another Media Player or application It is not restricted to timeline based applications Multi-platform (coded in Java) What is different about PlaySEM?
- 13. SE Video Player is responsible for playing video and reading MPEG-V SEM. Main functionalities include: ◦ open a video and related SEM ◦ control media state (play, pause and stop) ◦ position the video at a specific time in the timeline ◦ switch to full screen mode ◦ locate SER devices ◦ obtain information regarding device capabilities and configure a variety of application parameters SE Video Player - functionalities
- 14. SE Video Player - interface
- 15. SE Renderer is responsible for transforming SEM into commands and for driving devices Central component and can be reutilized in other applications. Its services include: SE Renderer - services
- 16. Instance of the PlaySEM
- 17. Real environment
- 18. No approaches were found in literature that decouples sensory effects renderer from media players, so: Would it cause any relevant delays or synchronization loss during the playback of sensory effects at the user environment? Open issue
- 19. Data traffic over a local network does not introduce significant delays H1 SE Renderer architecture has little impact on synchronization loss H2 Hypotheses
- 20. Latency was measured by capturing network packages http Stats parameter (SE Video Player): Execution time = T3 (T4 for SetSem) - T1 Data from 5 test rounds was collected and then the average time was calculated ID Description T1 Before command execution T2 After command execution T3 SE Renderer processing confirmation T4 SEM processing confirmation H1 - Network latency
- 21. Commands execution time vs. Latency variation
- 22. Time readings came from two clocks: C1, from SE Video Player, and C2, from SE Renderer The VLCJ clock has a 300ms precision. Another clock with a better precision was created Times were recorded on 5 test rounds Synchronization loss = time elapsed between C2 and C1 H2 - Synchronization loss
- 23. Time difference between SE Video Player and SER clocks after pause command
- 24. The PlaySEM platform is based on parts 2, 3, 6 and 7 of the MPEG-V standard Flexibility for replacing “components”, taking advantage of its architecture Multi-platform support Network Latency: ◦ Basic control functions (Play, Pause, Stop): average time < 16ms ◦ Average latency of all services < 25ms (except SetSem) ◦ Delay exclusion would bring no significant performance gains Conclusion
- 25. Synchronization loss: ◦ There were periods when the video playback was ahead or behind the rendering of sensory effects, although never beyond 34 milliseconds The synchronization loss can be considered negligible, although it does not account for the time that the hardware was actually working Effect type Acceptable time Haptic 1s behind video Air flow 5s ahead of or 3s behind video (YUAN et. al, 2015) Conclusion YUAN, Z. et al. Perceived synchronization of mulsemedia services. Multimedia, IEEE Transactions on, PP, n. 99, p. 1–1, 2015. ISSN 1520-9210.
- 26. THANK YOU! Celso Santos and Estêvão Saleme The end
Editor's Notes
- No modo tradicional da multimedia visao e audicao, seja lendo um livro, escutando musica, vendo um filme ou visualizando imagens Porque não estimular outros sentidos?
- Imersão como uma experiência realista, vista como condição necessária para criação de "presença“. Imersão como um estado psicológico caracterizado por perceber-se envolvido, incluído e interagindo com um ambiente que fornece um fluxo contínuo de estímulos e experiências. Presença refere-se a experimentar o ambiente gerado por computador, em vez da localidade física real (WITMER; SINGER, 1998).
- Não pretendo processar mídia pois quero que outros tipos de aplicações reusem o renderizador Dispositivo que não pode interoperar com outros produtos que o cliente espera interoperar perde valor* Capacidade de diferentes produtos interoperar entre eles permite aos consumidores uma escolha MPEG-V = interoperabilidade mundo virtual (ex. jogos, simuladores e aplicações) e mundo real (ex. sensores e atuadores) Acoplamento dificulta a adoção de múltiplos efeitos em outras áreas como medicina, jogos e educação, que geralmente são aplicativos baseados em eventos Preço do amBX é atualmente 800 libras esterlinas (kit premium)
- Renderizar significa entregar alguma coisa a alguém, como por exemplo, um serviço Porque é relevante? Permite que o renderizador de efeitos sensoriais seja reaproveitado para diferentes tipos de aplicação, não somente baseadas em linha de tempo Exemplo: 1) Peças teatrais, onde o ator proporciona efeitos sensoriais para os expectadores; 2) Jogos de vídeo games, podem se comunicar com o renderizador para promover efeitos sensoriais; 3) Aplicações na área de educação para ensinar o frutas e vegetais para crianças com associação de aroma.
- A abordagem do SE Renderer desacoplado do SE Video Player para o processamento do SEM, o torna capaz de ser utilizado em qualquer Media Player
- AL shows messages network Average Latency, (standard deviation) indicates the time variation on AL in order to project how higher it could grow, from a statistical standpoint. VC indicates the Variation Coecient, as a percentage, projected in AMI, Average Minimum Latency, on the lower bound and AMA, Average Maximum Latency, on the upper bound. APT indicates the Average Processing Time on SER, deducting latency time from execution time. AET indicates the Average Execution Time for each of the service provided by SER.