SlideShare a Scribd company logo
Quality assessment of immersive media:
Recent activities within VQEG
5 October 2021
Jesús Gutiérrez
Grupo de Tratamiento de Imágenes (GTI)
Universidad Politécnica de Madrid (UPM)
jesus.gutierrez@upm.es
https://www.gti.ssr.upm.es
www.vqeg.org
MPEG AG 5 Workshop on Quality of Immersive
Media: Assessment and Metrics
2
Outline
• Presentation of VQEG and IMG.
• Subjective Evaluation of Visual Quality and Simulator Sickness of
Short 360° Videos: ITU-T Rec. P.919.
• Ongoing and future work:
• Evaluation of QoE with long 360° Videos.
• Evaluation of immersive and interactive communication systems.
3
VQEG
Video Quality Experts Group
• VQEG provides a forum, via email lists and face-to-face meetings for video quality
assessment experts to exchange information and work together on common goals.
• Main objectives:
• To formulate test plans that clearly and specifically define the procedures for performing
subjective assessment tests and objective models validations.
• To produce open-source databases of multimedia material and test results, as well as
software tools.
• To conduct subjective studies of multimedia and immersive technologies and provide a
place for collaborative model developments.
• VQEG is open to all interested from industry, academia, government organizations
and Standard-Developing Organizations (SDOs). There are no fees involved, no
membership applications and no invitations are needed.
• Subscription to the main VQEG email list (ituvidq@vqeg.org) constitutes membership.
• Check the website for more details: http://www.vqeg.org/
4
IMG
Immerisve Media Group
• Mission: Quality assessment of immersive media.
• Technologies:
• 360-degree content / Virtual Reality.
• Augmented / Mixed reality.
• 3D, FVV, multiview technologies, light field.
• Goals: Baseline quality assessment of immersive systems.
• Generate and publish open-source datasets of immersive media content.
• Develop subjective test methods and provide QoE guidelines.
• Validation and development of objective metrics.
• Chairs: Jesús Gutiérrez (UPM), Pablo Pérez (Nokia Bell Labs), Zhenzhong Chen (Wuhan University).
• Website: https://www.its.bldrdoc.gov/vqeg/projects/immersive-media-group.aspx
5
IMG
Immerisve Media Group
• Some works and presentations:
• “Project Vertigo: monitoring sickness and discomfort in high-motion 360 video”, Pablo Perez (Nokia Bell-Labs)
• “Subjective evaluation of 360 degree video quality during head movement”, Glenn Van Wallendael (Ghent University)
• “A dataset of head and eye movements for 360 degree images”, Jesús Gutiérrez (University of Nantes)
• “QoE of Omnidirectional (360°) Videos”, Ashutosh Singla (Audiovisual Technology Group, TU Ilmenau)
• “A framework for adaptive delivery of omnidirectional video”, Christian Timmerer (University of Klagenfurt)
• “Spherical SSIM for Objective Omnidirectional Video Quality Assessment”, Zhenzhong Chen (Wuhan University)
• “Video Multimethod Assessment Fusion (VMAF) on 360VR”, Narciso García (Universidad Politécnica de Madrid).
• “Quality of Experience for a Virtual Reality simulator“, Kjell Brunnström (RISE Acreo AB)
• “Evaluation of immersion and presence in VR”, Francesca De Simone (CWI)
• “Measuring User Quality of Experience in Social VR Systems”, Francesca de Simone (CWI).
• “Perceptual Characterization of 3D Graphical Contents based on Visual Attention patterns”, Mona Abid (Univ. of Nantes)
• “Comparison of Subjective Methods for Quality Assessment”, Yana Nehmé (LIRIS/INSA Lyon)
• “3 datasets for quality assessment in the context of Free Viewpoint Video”, Suiyi Ling (University of Nantes)
• “Quality Assessment of FTV”, Patrick Le Callet (Université de Nantes).
• “Considerations on FTV quality assessment”, Narciso García (Universidad Politécnica de Madrid)
• “Characterization and selection of light field content for perceptual assessment”, Jesús Gutiérrez (University of Nantes)
• “No reference quality evaluation of light field content based on structural representation of the epipolar plane image”, Ali Ak (U. Nantes)
• … Many more in the public ftp (check the meetings): https://www.its.bldrdoc.gov/vqeg/meetings-home.aspx
360° video
VR
FVV
LF
AR
6
Outline
• Presentation of VQEG and IMG.
• Subjective Evaluation of Visual Quality and Simulator Sickness
of Short 360° Videos: ITU-T Rec. P.919.
• Ongoing and future work:
• Evaluation of QoE with long 360° Videos.
• Evaluation of immersive and interactive communication systems.
7
ITU-T Rec. P.919
Subjective evaluation of short 360º videos
• Global target of the test plan:
• Design and execute a cross-lab test where we can assess and validate subjective
evaluation methodologies for 360° video.
• Contribute to the standardization of such methodologies under ITU-T P.919 (Q13/12).
• Generate a dataset of subjectively assessed content for future research.
• What was covered:
• Assessment of video quality and simulator sickness…
• … of short sequences (<=30s), in the spirit of ITU-R BT.500, ITU-T P.910, etc.
• More details in our recently published paper:
• J. Gutiérrez, P. Pérez, M. Orduna, A. Singla, C. Cortés, P. Mazumdar, I. Viola, K. Brunnström, F. Battisti, N. Cieplińska, D. Juszka,
L. Janowski, M. Leszczuk, A. Adeyemi-Ejeye, Y. Hu, Z. Chen, G. Van Wallendael, P. Lambert, C. Díaz, J. Hedlund, O. Hamsis, S.
Fremerey, F. Hofmeyer, A. Raake, P. César, M. Carli, N. García, "Subjective evaluation of visual quality and simulator sickness
of short 360° videos: ITU-T Rec. P.919", IEEE Transactions on Multimedia, Jul. 2021 (Early access).
8
ITU-T Rec. P.919
Subjective evaluation of short 360º videos
• Audiovisual Quality
• Test methodology
• ACR vs DCR
• Sequence duration
• 10s vs 20s
• 20s vs 30s
• 10s vs 30s
• Test setup
• HTC Vive vs Samsung Gear VR vs HTC Vive Pro
and Tethered vs Untethered
• Scoring app vs “Saying out lout”
• Audio vs no audio
• Simulator sickness
• When/how to assess
simulator sickness
• Short vs. long
questionnaires
• One single question
• Reduced SSQ
9
ITU-T Rec. P.919
Subjective evaluation of short 360º videos
• Distribution of test conditions and participant labs
10
ITU-T Rec. P.919
Subjective evaluation of short 360º videos
• Test set: Source sequences
• Wide range of content
characteristics.
• SI/TI computed over
equirectangular (ER), cube-map
(CM) and spherical (SP), to account
for possible projection distortions
[De Simone, 2019].
11
ITU-T Rec. P.919
Subjective evaluation of short 360º videos
• Test set: Encoded sequences
• 8 HRCs (including Hidden Reference):
• Kvazaar encoder, HEVC, fixed QP.
• 4 homogeneous QPs: 15 (HR), 22, 32, 42.
• 4 non-homogeneous QPs (tiles).
#Tiles Transition ROI QPs
8x5** Smooth 90O 42 37 32 22 22 32 37 42
6x3 Smooth 120O 42 32 22 22 32 42
8x5** Abrupt 180O 42 42 22 22 22 22 42 42
6x3 Abrupt 120O 37 37 22 22 37 37
12
ITU-T Rec. P.919
Subjective evaluation of short 360º videos
• Session structure and duration:
• Each subject evaluates the same test clip under two different conditions:
• Use within-subject statistics for better statistical power.
• Each condition is tested in one active period:
• After each active session: rest period of 15 min.
• Red arrows: score simulator sickness.
• App Miro360: Developed, tested, and validated subjective assessment SW:
• For desktop (HTC and Oculus) and mobile (GearVR and Daydream).
Intro Test Condition A Rest (15 min) Test Condition B Wrap-up
Instructions in written form Visual acuity & color tests Training session Doubts/Questions
C. Cortés, P. Pérez, and N. García,
“Unity3D-based app for 360VR
subjective quality assessment with
customizable questionnaires”, IEEE
ICCE 2019.
https://git.gti.ssr.upm.es/pub/Miro360
13
ITU-T Rec. P.919
Subjective evaluation of short 360º videos
• Simulator Sickness:
• In each measurement, subjects scored both:
• First, a single question (1-5 scale): always the same (A or B), randomly assigned
• Second, SSQ [Kennedy et al., 1993]
No problem No perceptible effect, natural feeling
Light effects Slight discomfort, but no sickness
Uncomfortable Moderate discomfort, but tolerable for a while
Unpleasant
Strong discomfort or sickness, but can continue the
test
Unbearable
Strong discomfort or sickness, and want to stop
test
A. Are you feeling any sickness or discomfort now?
(Vertigo scale) [Pérez et al., 2018]
Very dizzy
Dizzy
Slightly dizzy
Not dizzy
Absolutely not dizzy
B. How is your level of dizziness or
nausea? (Short-SSQ) [Tran et al., 2017]
14
ITU-T Rec. P.919
Subjective evaluation of short 360º videos
• Observers
• At least 28 participants per lab.
• Visual screening: Visual acuity and color vision.
• Subjects were sitting in a swivel chair.
15
ITU-T Rec. P.919
Subjective evaluation of short 360º videos
• Results of audiovisual quality: Wuhan, ACR, videos of 10s (blue) and 20s (orange).
16
ITU-T Rec. P.919
Subjective evaluation of short 360º videos
• Results of audiovisual quality: All laboratories (all tested conditions), VSenseLuther.
17
ITU-T Rec. P.919
Subjective evaluation of short 360º videos
• Results of SSQ:
• Exploration results:
Can we use a subset of the SSQ?
PLCC among different SSQ
18
ITU-T Rec. P.919
Subjective evaluation of short 360º videos
• Outcomes included in the recommendation ITU-T P.919:
• Methodology: Both ACR and DCR can be used for assessment of coding quality.
• Duration of videos: subjective tests of coding degradations with 360° videos can
be done with sequences of 10 seconds.
• HMD: any commercial HMD (tethered or untethered) can be used in visual
quality tests (providing enough resolution and refresh rate)
• Audio: it is possible to use test stimuli either with or without audio to evaluate
visual quality (spatial audio was not considered).
• Method to collect ratings: Voting interfaces and verbal voting.
• Minimum number of observers: 28 participants.
• SSQ: both Vertigo scale and VRSQ [Kim et al., 2018] can be alternatives to SSQ.
• Dataset will be available soon in https://www.its.bldrdoc.gov/vqeg/video-datasets-and-organizations.aspx
19
Outline
• Presentation of VQEG and IMG.
• Subjective Evaluation of Visual Quality and Simulator Sickness of
Short 360° Videos: ITU-T Rec. P.919.
• Ongoing and future work:
• Evaluation of QoE with long 360° Videos.
• Evaluation of immersive and interactive communication systems.
20
Evaluation of QoE with long 360° Videos
Ongoing and future work
• After studying the evaluation of short videos à Long 360° videos.
• Definition: a short (2-10 minutes) full content item.
• Requires content immersive evaluation [Pinson et al., 2014].
• Evaluation:
• Single-Stimulus Continuous Quality Evaluation [ITU-T P.910] à In-sequence
• Single-Stimulus Discrete Quality Evaluation (SSDQE) [Gutiérrez et al., 2012] à In-sequence
• ACR à Post-sequence
• Outcomes:
• SSDQE provides similar results to what ACR would provide.
• SSCQE à to be analyzed.
• ACR is significantly affected by in-sequence evaluation.
• Quality evaluation does not affect sense of presence.
HRC HR HRC HR HRC HR
HR HR
20s 5s
VOTE VOTE VOTE
continuous voting
SSCQE
SSDQE
Control
Post-
Seq
Eval
SSDQE / SSCQE (?) / ACR are valid
for long sequences
M. Orduna, P. Pérez, J. Gutiérrez, N. García,
“Methodology to Assess Quality, Presence,
Empathy, Attitude, and Attention in Social
VR: International Experiences Use Case”,
arXiv:2103.02550
21
Outline
• Presentation of VQEG and IMG.
• Subjective Evaluation of Visual Quality and Simulator Sickness of
Short 360° Videos: ITU-T Rec. P.919.
• Ongoing and future work:
• Evaluation of QoE with long 360° Videos.
• Evaluation of immersive and interactive communication systems.
22
Evaluation of immersive and interactive communication systems
Ongoing and future work
• How can we test bi-directional immersive communication systems?
• Evaluate effect of technical factors (e.g., latency / bitrate / etc.) in QoE (e.g.,
immersion, interactivity, etc.)
• Compare with other systems / experiments.
• Related recommendations:
• ITU-T P.920 - Interactive test methods for audiovisual communications.
• Some tasks proposed to evaluate effect of technical factors of video-
conference services (05/2000):
• E.g.: one of the subjects shows and describes a plastic building block and the
other one is required to reproduce it.
• ITU-T P.13xx series: telemeeting assessment.
• ITU-T P.QXM - QoE Assessment of eXtended Reality (XR) Meeting.
• Best practices for QoE assessment of tele-meetings with XR elements.
23
Evaluation of immersive and interactive communication systems
Ongoing and future work
• Test plan:
• Run a cross-lab experiment using any available collaborative immersive
technology:
• “The same” experiment in completely different setups
• Gather a set of immersive communication systems:
• Real-time 360° / Free-viewpoint video telepresence.
• Social VR with pointcloud transmission / with avatars.
• XR collaboration.
• …
• Create an experiment that covers all basic functionalities:
• Conversation between people.
• Discussion about objects in the immersive space.
• Interaction with (local / remote / virtual) objects in the immersive space.
24
Evaluation of immersive and interactive communication systems
Ongoing and future work
• Identified technologies:
FVV Live (GTI-UPM)
Narciso García
narciso@gti.ssr.upm.es
The Owl (Nokia Bell Labs)
Pablo Pérez
pablo.perez@nokia-bell-labs.com
VRTogether (CWI)
Pablo César
p.s.cesar@cwi.nl
25
Evaluation of immersive and interactive communication systems
Ongoing and future work
• Interested groups:
• UPM (Spain)
• Nokia Bell-Labs (Spain)
• CWI (The Netherlands)
• TU Ilmenau (Germany)
• Ghent University (Belgium)
• RISE (Sweden)
• Wuhan University (China)
• University of Surrey (United Kingdom)
• AGH University of Science and Technology (Poland)
• …
• You?
26
Acknowledgment and references
• Pablo Pérez
• Marta Orduna
• Ashutosh Singla
• Carlos Cortés
• Pramit Mazumdar
• Irene Viola
• Kjell Brunnström
• Federica Battisti
• Natalia Cieplinska
• Dawid Juszka
• Lucjan Janowski
• Mikolaj Igor Leszczuk
• Anthony Adeyemi-Ejeye
• Yaosi Hu
• Zhengzhong Chen
• Glenn Van Wallendael
• Peter Lambert
• Cesar Diaz
• John Hedlund
• Omar Hamsis
• Stephan Fremerey
• Frank Hofmeyer
• Alexander Raake
• Pablo Cesar
• Marco Carli
• Narciso García
• Philip Corriveau
• Patrick Le Callet
• …
• F. De Simone, J. Gutiérrez, P. Le Callet, "Complexity measurement and characterization of 360-degree content", Human Vision and Electronic
Imaging Conference, Jan. 2019.
• J. Gutiérrez, P. Pérez, M. Orduna, A. Singla, C. Cortés, P. Mazumdar, I. Viola, K. Brunnström, F. Battisti, N. Cieplińska, D. Juszka, L.
Janowski, M. Leszczuk, A. Adeyemi-Ejeye, Y. Hu, Z. Chen, G. Van Wallendael, P. Lambert, C. Díaz, J. Hedlund, O. Hamsis, S. Fremerey, F.
Hofmeyer, A. Raake, P. César, M. Carli, N. García, "Subjective evaluation of visual quality and simulator sickness of short 360° videos: ITU-T
Rec. P.919", IEEE Transactions on Multimedia, Jul. 2021 (Early access).
• R. S. Kennedy, N. E. Lane, K. S. Berbaum, and M. G. Lilienthal, “Simulator sickness questionnaire: An enhanced method for quantifying
simulator sickness”, Int. J. Aviation Psychol., vol. 3, no. 3, pp. 203–220, Nov. 1993.
• P. Pérez, N. Oyaga, J. J. Ruiz, and A. Villegas, “Towards systematic analysis of cybersickness in high motion omnidirectional video”, Int. Conf.
Qual. Multimedia Experience, 2018.
• H. T. T. Tran, N. P. Ngoc, C. T. Pham, Y. J. Jung, and T. C. Thang, “A subjective study on QoE of 360 video for VR communication”, IEEE Int.
Workshop Multimedia Signal Process., 2017.
• H. K. Kim, J. Park, Y. Choi, and M. Choe, “Virtual reality sickness questionnaire (VRSQ):Motion sickness measurement index in a VR
environment”, Appl. Ergonom., vol. 69, pp. 66–73, May 2018.
• M. H. Pinson, M. Sullivan, and A. Catellier, “A New Method for Immersive Audiovisual Subjective Testing”, Int. Work. Video Process. Qual.
Metrics Consum. Electron., Jan. 2014.
• J. Gutierrez, P. Perez, F. Jaureguizar, J. Cabrera, and N. Garcia, “Validation of a novel approach to subjective quality evaluation of conventional
and 3D broadcasted video services”, International Workshop on Quality of Multimedia Experience, Jul. 2012.
• M. Orduna, P. Pérez, J. Gutiérrez, N. García, “Methodology to Assess Quality, Presence, Empathy, Attitude, and Attention in Social VR:
International Experiences Use Case”, arXiv:2103.02550.
• P. Carballeira, C. Carmona, C. Díaz, D. Berjón, D. Corregidor, J. Cabrera, F. Morán, C. Doblado, S. Arnaldo, M.M. Martín, N. García, "FVV
Live: A real-time free-viewpoint video system with consumer electronics hardware”, IEEE Transactions on Multimedia, 2021.
• D. Berjón, P. Carballeira, J. Cabrera, C. Carmona, D. Corregidor, C. Díaz, F. Morán, N. García, “FVV Live: real-time, low-cost free viewpoint
video”, IEEE Int. Conf. on Multimedia and Expo Workshops, Jul. 2020,
• R. Kachach, M. Orduna, J. Rodríguez, P. Pérez, A. Villegas, J. Cabrera, N. Garcia, “Immersive Telepresence in Remote Education”, International
Workshop on Immersive Mixed and Virtual Environment Systems, Sep. 2021.
• R. Kachach, D. González-Morín, F. Pereira, P. Pérez, I. Benito, J. Ruiz, E. González, A, Villegas, "A Multi-Peer, Low Cost Immersive
Communication System for Pandemic Times”, IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, Mar. 2021.
• S. Rossi, I. Viola, J. Jansen, S. Subramanyam, L. Toni, and P. Cesar, “Influence of Narrative Elements on User Behaviour in Photorealistic Social
VR”, International Workshop on Immersive Mixed and Virtual Environment Systems, Sep. 2021.
• R. Mekuria, K. Blom, and P. Cesar Design, “Implementation and Evaluation of a Point Cloud Codec for Tele-Immersive Video”, IEEE
Transactions on Circuits and Systems for Video Technology, 27(4), pp. 828-842, Apr. 2017.
Quality assessment of immersive media:
Recent activities within VQEG
5 October 2021
Jesús Gutiérrez
Grupo de Tratamiento de Imágenes (GTI)
Universidad Politécnica de Madrid (UPM)
jesus.gutierrez@upm.es
https://www.gti.ssr.upm.es
www.vqeg.org
MPEG AG 5 Workshop on Quality of Immersive
Media: Assessment and Metrics

More Related Content

What's hot

Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR ...
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR ...Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR ...
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR ...
Förderverein Technische Fakultät
 
Continual/Lifelong Learning with Deep Architectures, Vincenzo Lomonaco
Continual/Lifelong Learning with Deep Architectures, Vincenzo LomonacoContinual/Lifelong Learning with Deep Architectures, Vincenzo Lomonaco
Continual/Lifelong Learning with Deep Architectures, Vincenzo Lomonaco
Data Science Milan
 
Overview of Selected Current MPEG Activities
Overview of Selected Current MPEG ActivitiesOverview of Selected Current MPEG Activities
Overview of Selected Current MPEG Activities
Alpen-Adria-Universität
 
Translational R&D and Innovation Fund (TIF) - MoE
Translational R&D and Innovation Fund (TIF) - MoETranslational R&D and Innovation Fund (TIF) - MoE
Translational R&D and Innovation Fund (TIF) - MoE
joserojasnus
 

What's hot (20)

2018 FiTCE congress
2018 FiTCE congress2018 FiTCE congress
2018 FiTCE congress
 
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR ...
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR ...Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR ...
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR ...
 
Hard-Negatives Selection Strategy for Cross-Modal Retrieval
Hard-Negatives Selection Strategy for Cross-Modal RetrievalHard-Negatives Selection Strategy for Cross-Modal Retrieval
Hard-Negatives Selection Strategy for Cross-Modal Retrieval
 
Recent advances in quality of experience in multimedia communication
Recent advances in quality of experience in multimedia communicationRecent advances in quality of experience in multimedia communication
Recent advances in quality of experience in multimedia communication
 
Tactile Internet with Human-in-the-Loop
Tactile Internet with Human-in-the-LoopTactile Internet with Human-in-the-Loop
Tactile Internet with Human-in-the-Loop
 
ICIP2016 image compression grand challenge
ICIP2016 image compression grand challengeICIP2016 image compression grand challenge
ICIP2016 image compression grand challenge
 
Continual/Lifelong Learning with Deep Architectures, Vincenzo Lomonaco
Continual/Lifelong Learning with Deep Architectures, Vincenzo LomonacoContinual/Lifelong Learning with Deep Architectures, Vincenzo Lomonaco
Continual/Lifelong Learning with Deep Architectures, Vincenzo Lomonaco
 
New Media Services from a Mobile Chipset Vendor and Standardization Perspective
New Media Services from a Mobile Chipset Vendor and Standardization PerspectiveNew Media Services from a Mobile Chipset Vendor and Standardization Perspective
New Media Services from a Mobile Chipset Vendor and Standardization Perspective
 
Overview of Selected Current MPEG Activities
Overview of Selected Current MPEG ActivitiesOverview of Selected Current MPEG Activities
Overview of Selected Current MPEG Activities
 
Mtech Second progresspresentation ON VIDEO SUMMARIZATION
Mtech Second progresspresentation ON VIDEO SUMMARIZATIONMtech Second progresspresentation ON VIDEO SUMMARIZATION
Mtech Second progresspresentation ON VIDEO SUMMARIZATION
 
Translational R&D and Innovation Fund (TIF) - MoE
Translational R&D and Innovation Fund (TIF) - MoETranslational R&D and Innovation Fund (TIF) - MoE
Translational R&D and Innovation Fund (TIF) - MoE
 
Continual/Lifelong Learning with Deep Architectures
Continual/Lifelong Learning with Deep ArchitecturesContinual/Lifelong Learning with Deep Architectures
Continual/Lifelong Learning with Deep Architectures
 
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
 
A Framework for Adaptive Delivery of Omnidirectional Video
A Framework for Adaptive Delivery of Omnidirectional VideoA Framework for Adaptive Delivery of Omnidirectional Video
A Framework for Adaptive Delivery of Omnidirectional Video
 
Quality of Experience in emerging visual communications
Quality of Experience in emerging visual communicationsQuality of Experience in emerging visual communications
Quality of Experience in emerging visual communications
 
Perceptually Lossless Compression with Error Concealment for Periscope and So...
Perceptually Lossless Compression with Error Concealment for Periscope and So...Perceptually Lossless Compression with Error Concealment for Periscope and So...
Perceptually Lossless Compression with Error Concealment for Periscope and So...
 
PERCEPTUALLY LOSSLESS COMPRESSION WITH ERROR CONCEALMENT FOR PERISCOPE AND SO...
PERCEPTUALLY LOSSLESS COMPRESSION WITH ERROR CONCEALMENT FOR PERISCOPE AND SO...PERCEPTUALLY LOSSLESS COMPRESSION WITH ERROR CONCEALMENT FOR PERISCOPE AND SO...
PERCEPTUALLY LOSSLESS COMPRESSION WITH ERROR CONCEALMENT FOR PERISCOPE AND SO...
 
Macroblock Level Rate Control for Low Delay H.264/AVC based Video Communication
Macroblock Level Rate Control for Low Delay H.264/AVC based Video CommunicationMacroblock Level Rate Control for Low Delay H.264/AVC based Video Communication
Macroblock Level Rate Control for Low Delay H.264/AVC based Video Communication
 
HPC on Cloud for SMEs. The case of bolt tightening.
HPC on Cloud for SMEs. The case of bolt tightening.HPC on Cloud for SMEs. The case of bolt tightening.
HPC on Cloud for SMEs. The case of bolt tightening.
 
Fasti c pix vfinal
Fasti c pix vfinalFasti c pix vfinal
Fasti c pix vfinal
 

Similar to Quality assessment of immersive media: Recent activities within VQEG

ROLE OF DIGITAL IMAGING IN PATHOLOGY.pptx
ROLE OF DIGITAL IMAGING IN PATHOLOGY.pptxROLE OF DIGITAL IMAGING IN PATHOLOGY.pptx
ROLE OF DIGITAL IMAGING IN PATHOLOGY.pptx
aditisikarwar2
 
Ultra-High-Definition Quality of Experience with MPEG-DASH
Ultra-High-Definition Quality of Experience with MPEG-DASHUltra-High-Definition Quality of Experience with MPEG-DASH
Ultra-High-Definition Quality of Experience with MPEG-DASH
Bitmovin Inc
 

Similar to Quality assessment of immersive media: Recent activities within VQEG (20)

NMSL_2017summer
NMSL_2017summerNMSL_2017summer
NMSL_2017summer
 
A Ensemble Learning-based No Reference QoE Model for User Generated Contents
A Ensemble Learning-based No Reference QoE Model for User Generated ContentsA Ensemble Learning-based No Reference QoE Model for User Generated Contents
A Ensemble Learning-based No Reference QoE Model for User Generated Contents
 
Research Proposal Presentation Pitch
Research Proposal Presentation PitchResearch Proposal Presentation Pitch
Research Proposal Presentation Pitch
 
Selected Aspects of the New Recommendation on Subjective Methods of Assessing...
Selected Aspects of the New Recommendation on Subjective Methods of Assessing...Selected Aspects of the New Recommendation on Subjective Methods of Assessing...
Selected Aspects of the New Recommendation on Subjective Methods of Assessing...
 
Presentation of the InVID verification technologies at IPTC 2018
Presentation of the InVID verification technologies at IPTC 2018Presentation of the InVID verification technologies at IPTC 2018
Presentation of the InVID verification technologies at IPTC 2018
 
Research and activity report
Research and activity reportResearch and activity report
Research and activity report
 
Presentación Tesis 08022016
Presentación Tesis 08022016Presentación Tesis 08022016
Presentación Tesis 08022016
 
ROLE OF DIGITAL IMAGING IN PATHOLOGY.pptx
ROLE OF DIGITAL IMAGING IN PATHOLOGY.pptxROLE OF DIGITAL IMAGING IN PATHOLOGY.pptx
ROLE OF DIGITAL IMAGING IN PATHOLOGY.pptx
 
Ultra-High-Definition Quality of Experience with MPEG-DASH
Ultra-High-Definition Quality of Experience with MPEG-DASHUltra-High-Definition Quality of Experience with MPEG-DASH
Ultra-High-Definition Quality of Experience with MPEG-DASH
 
Video production pedagogy
Video production pedagogyVideo production pedagogy
Video production pedagogy
 
Keyframe-based Video Summarization Designer
Keyframe-based Video Summarization DesignerKeyframe-based Video Summarization Designer
Keyframe-based Video Summarization Designer
 
QoE Research
QoE ResearchQoE Research
QoE Research
 
A Cross-Layer Framework for Multi-user360-Degree Video Streaming over Cellula...
A Cross-Layer Framework for Multi-user360-Degree Video Streaming over Cellula...A Cross-Layer Framework for Multi-user360-Degree Video Streaming over Cellula...
A Cross-Layer Framework for Multi-user360-Degree Video Streaming over Cellula...
 
Visual Search for Musical Performances and Endoscopic Videos
Visual Search for Musical Performances and Endoscopic VideosVisual Search for Musical Performances and Endoscopic Videos
Visual Search for Musical Performances and Endoscopic Videos
 
Event Understanding in Endoscopic Surgery Videos
Event Understanding in Endoscopic Surgery VideosEvent Understanding in Endoscopic Surgery Videos
Event Understanding in Endoscopic Surgery Videos
 
Fast object re detection and localization in video for spatio-temporal fragme...
Fast object re detection and localization in video for spatio-temporal fragme...Fast object re detection and localization in video for spatio-temporal fragme...
Fast object re detection and localization in video for spatio-temporal fragme...
 
Immersive Video Delivery: From Omnidirectional Video to Holography
Immersive Video Delivery: From Omnidirectional Video to HolographyImmersive Video Delivery: From Omnidirectional Video to Holography
Immersive Video Delivery: From Omnidirectional Video to Holography
 
Interactive Video Search: Where is the User in the Age of Deep Learning?
Interactive Video Search: Where is the User in the Age of Deep Learning?Interactive Video Search: Where is the User in the Age of Deep Learning?
Interactive Video Search: Where is the User in the Age of Deep Learning?
 
Near-Duplicate Video Retrieval by Aggregating Intermediate CNN Layers
Near-Duplicate Video Retrieval by Aggregating Intermediate CNN LayersNear-Duplicate Video Retrieval by Aggregating Intermediate CNN Layers
Near-Duplicate Video Retrieval by Aggregating Intermediate CNN Layers
 
Fast object re-detection and localization in video for spatio-temporal fragme...
Fast object re-detection and localization in video for spatio-temporal fragme...Fast object re-detection and localization in video for spatio-temporal fragme...
Fast object re-detection and localization in video for spatio-temporal fragme...
 

More from Alpen-Adria-Universität

Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Alpen-Adria-Universität
 
Content-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive StreamingContent-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive Streaming
Alpen-Adria-Universität
 
Evaluation of Quality of Experience of ABR Schemes in Gaming Stream
Evaluation of Quality of Experience of ABR Schemes in Gaming StreamEvaluation of Quality of Experience of ABR Schemes in Gaming Stream
Evaluation of Quality of Experience of ABR Schemes in Gaming Stream
Alpen-Adria-Universität
 
Policy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-Driven Dynamic HTTP Adaptive Streaming Player EnvironmentPolicy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Alpen-Adria-Universität
 
Energy Consumption in Video Streaming: Components, Measurements, and Strategies
Energy Consumption in Video Streaming: Components, Measurements, and StrategiesEnergy Consumption in Video Streaming: Components, Measurements, and Strategies
Energy Consumption in Video Streaming: Components, Measurements, and Strategies
Alpen-Adria-Universität
 
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
Alpen-Adria-Universität
 
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming ApplicationsSARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications
Alpen-Adria-Universität
 

More from Alpen-Adria-Universität (20)

VEED: Video Encoding Energy and CO2 Emissions Dataset for AWS EC2 instances
VEED: Video Encoding Energy and CO2 Emissions Dataset for AWS EC2 instancesVEED: Video Encoding Energy and CO2 Emissions Dataset for AWS EC2 instances
VEED: Video Encoding Energy and CO2 Emissions Dataset for AWS EC2 instances
 
GREEM: An Open-Source Energy Measurement Tool for Video Processing
GREEM: An Open-Source Energy Measurement Tool for Video ProcessingGREEM: An Open-Source Energy Measurement Tool for Video Processing
GREEM: An Open-Source Energy Measurement Tool for Video Processing
 
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
 
VEEP: Video Encoding Energy and CO₂ Emission Prediction
VEEP: Video Encoding Energy and CO₂ Emission PredictionVEEP: Video Encoding Energy and CO₂ Emission Prediction
VEEP: Video Encoding Energy and CO₂ Emission Prediction
 
Content-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive StreamingContent-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive Streaming
 
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Video...
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Video...Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Video...
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Video...
 
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Vid...
Empowerment of Atypical Viewers  via Low-Effort Personalized Modeling  of Vid...Empowerment of Atypical Viewers  via Low-Effort Personalized Modeling  of Vid...
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Vid...
 
Optimizing Video Streaming for Sustainability and Quality: The Role of Prese...
Optimizing Video Streaming  for Sustainability and Quality: The Role of Prese...Optimizing Video Streaming  for Sustainability and Quality: The Role of Prese...
Optimizing Video Streaming for Sustainability and Quality: The Role of Prese...
 
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
 
Machine Learning Based Resource Utilization Prediction in the Computing Conti...
Machine Learning Based Resource Utilization Prediction in the Computing Conti...Machine Learning Based Resource Utilization Prediction in the Computing Conti...
Machine Learning Based Resource Utilization Prediction in the Computing Conti...
 
Evaluation of Quality of Experience of ABR Schemes in Gaming Stream
Evaluation of Quality of Experience of ABR Schemes in Gaming StreamEvaluation of Quality of Experience of ABR Schemes in Gaming Stream
Evaluation of Quality of Experience of ABR Schemes in Gaming Stream
 
Network-Assisted Delivery of Adaptive Video Streaming Services through CDN, S...
Network-Assisted Delivery of Adaptive Video Streaming Services through CDN, S...Network-Assisted Delivery of Adaptive Video Streaming Services through CDN, S...
Network-Assisted Delivery of Adaptive Video Streaming Services through CDN, S...
 
Multi-access Edge Computing for Adaptive Video Streaming
Multi-access Edge Computing for Adaptive Video StreamingMulti-access Edge Computing for Adaptive Video Streaming
Multi-access Edge Computing for Adaptive Video Streaming
 
Policy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-Driven Dynamic HTTP Adaptive Streaming Player EnvironmentPolicy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-Driven Dynamic HTTP Adaptive Streaming Player Environment
 
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...
 
Energy Consumption in Video Streaming: Components, Measurements, and Strategies
Energy Consumption in Video Streaming: Components, Measurements, and StrategiesEnergy Consumption in Video Streaming: Components, Measurements, and Strategies
Energy Consumption in Video Streaming: Components, Measurements, and Strategies
 
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
 
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningVideo Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
 
Optimizing QoE and Latency of Live Video Streaming Using Edge Computing a...
Optimizing  QoE and Latency of  Live Video Streaming Using  Edge Computing  a...Optimizing  QoE and Latency of  Live Video Streaming Using  Edge Computing  a...
Optimizing QoE and Latency of Live Video Streaming Using Edge Computing a...
 
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming ApplicationsSARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications
 

Recently uploaded

Recently uploaded (20)

SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
Motion for AI: Creating Empathy in Technology
Motion for AI: Creating Empathy in TechnologyMotion for AI: Creating Empathy in Technology
Motion for AI: Creating Empathy in Technology
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
 
Enterprise Security Monitoring, And Log Management.
Enterprise Security Monitoring, And Log Management.Enterprise Security Monitoring, And Log Management.
Enterprise Security Monitoring, And Log Management.
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAK
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 

Quality assessment of immersive media: Recent activities within VQEG

  • 1. Quality assessment of immersive media: Recent activities within VQEG 5 October 2021 Jesús Gutiérrez Grupo de Tratamiento de Imágenes (GTI) Universidad Politécnica de Madrid (UPM) jesus.gutierrez@upm.es https://www.gti.ssr.upm.es www.vqeg.org MPEG AG 5 Workshop on Quality of Immersive Media: Assessment and Metrics
  • 2. 2 Outline • Presentation of VQEG and IMG. • Subjective Evaluation of Visual Quality and Simulator Sickness of Short 360° Videos: ITU-T Rec. P.919. • Ongoing and future work: • Evaluation of QoE with long 360° Videos. • Evaluation of immersive and interactive communication systems.
  • 3. 3 VQEG Video Quality Experts Group • VQEG provides a forum, via email lists and face-to-face meetings for video quality assessment experts to exchange information and work together on common goals. • Main objectives: • To formulate test plans that clearly and specifically define the procedures for performing subjective assessment tests and objective models validations. • To produce open-source databases of multimedia material and test results, as well as software tools. • To conduct subjective studies of multimedia and immersive technologies and provide a place for collaborative model developments. • VQEG is open to all interested from industry, academia, government organizations and Standard-Developing Organizations (SDOs). There are no fees involved, no membership applications and no invitations are needed. • Subscription to the main VQEG email list (ituvidq@vqeg.org) constitutes membership. • Check the website for more details: http://www.vqeg.org/
  • 4. 4 IMG Immerisve Media Group • Mission: Quality assessment of immersive media. • Technologies: • 360-degree content / Virtual Reality. • Augmented / Mixed reality. • 3D, FVV, multiview technologies, light field. • Goals: Baseline quality assessment of immersive systems. • Generate and publish open-source datasets of immersive media content. • Develop subjective test methods and provide QoE guidelines. • Validation and development of objective metrics. • Chairs: Jesús Gutiérrez (UPM), Pablo Pérez (Nokia Bell Labs), Zhenzhong Chen (Wuhan University). • Website: https://www.its.bldrdoc.gov/vqeg/projects/immersive-media-group.aspx
  • 5. 5 IMG Immerisve Media Group • Some works and presentations: • “Project Vertigo: monitoring sickness and discomfort in high-motion 360 video”, Pablo Perez (Nokia Bell-Labs) • “Subjective evaluation of 360 degree video quality during head movement”, Glenn Van Wallendael (Ghent University) • “A dataset of head and eye movements for 360 degree images”, Jesús Gutiérrez (University of Nantes) • “QoE of Omnidirectional (360°) Videos”, Ashutosh Singla (Audiovisual Technology Group, TU Ilmenau) • “A framework for adaptive delivery of omnidirectional video”, Christian Timmerer (University of Klagenfurt) • “Spherical SSIM for Objective Omnidirectional Video Quality Assessment”, Zhenzhong Chen (Wuhan University) • “Video Multimethod Assessment Fusion (VMAF) on 360VR”, Narciso García (Universidad Politécnica de Madrid). • “Quality of Experience for a Virtual Reality simulator“, Kjell Brunnström (RISE Acreo AB) • “Evaluation of immersion and presence in VR”, Francesca De Simone (CWI) • “Measuring User Quality of Experience in Social VR Systems”, Francesca de Simone (CWI). • “Perceptual Characterization of 3D Graphical Contents based on Visual Attention patterns”, Mona Abid (Univ. of Nantes) • “Comparison of Subjective Methods for Quality Assessment”, Yana Nehmé (LIRIS/INSA Lyon) • “3 datasets for quality assessment in the context of Free Viewpoint Video”, Suiyi Ling (University of Nantes) • “Quality Assessment of FTV”, Patrick Le Callet (Université de Nantes). • “Considerations on FTV quality assessment”, Narciso García (Universidad Politécnica de Madrid) • “Characterization and selection of light field content for perceptual assessment”, Jesús Gutiérrez (University of Nantes) • “No reference quality evaluation of light field content based on structural representation of the epipolar plane image”, Ali Ak (U. Nantes) • … Many more in the public ftp (check the meetings): https://www.its.bldrdoc.gov/vqeg/meetings-home.aspx 360° video VR FVV LF AR
  • 6. 6 Outline • Presentation of VQEG and IMG. • Subjective Evaluation of Visual Quality and Simulator Sickness of Short 360° Videos: ITU-T Rec. P.919. • Ongoing and future work: • Evaluation of QoE with long 360° Videos. • Evaluation of immersive and interactive communication systems.
  • 7. 7 ITU-T Rec. P.919 Subjective evaluation of short 360º videos • Global target of the test plan: • Design and execute a cross-lab test where we can assess and validate subjective evaluation methodologies for 360° video. • Contribute to the standardization of such methodologies under ITU-T P.919 (Q13/12). • Generate a dataset of subjectively assessed content for future research. • What was covered: • Assessment of video quality and simulator sickness… • … of short sequences (<=30s), in the spirit of ITU-R BT.500, ITU-T P.910, etc. • More details in our recently published paper: • J. Gutiérrez, P. Pérez, M. Orduna, A. Singla, C. Cortés, P. Mazumdar, I. Viola, K. Brunnström, F. Battisti, N. Cieplińska, D. Juszka, L. Janowski, M. Leszczuk, A. Adeyemi-Ejeye, Y. Hu, Z. Chen, G. Van Wallendael, P. Lambert, C. Díaz, J. Hedlund, O. Hamsis, S. Fremerey, F. Hofmeyer, A. Raake, P. César, M. Carli, N. García, "Subjective evaluation of visual quality and simulator sickness of short 360° videos: ITU-T Rec. P.919", IEEE Transactions on Multimedia, Jul. 2021 (Early access).
  • 8. 8 ITU-T Rec. P.919 Subjective evaluation of short 360º videos • Audiovisual Quality • Test methodology • ACR vs DCR • Sequence duration • 10s vs 20s • 20s vs 30s • 10s vs 30s • Test setup • HTC Vive vs Samsung Gear VR vs HTC Vive Pro and Tethered vs Untethered • Scoring app vs “Saying out lout” • Audio vs no audio • Simulator sickness • When/how to assess simulator sickness • Short vs. long questionnaires • One single question • Reduced SSQ
  • 9. 9 ITU-T Rec. P.919 Subjective evaluation of short 360º videos • Distribution of test conditions and participant labs
  • 10. 10 ITU-T Rec. P.919 Subjective evaluation of short 360º videos • Test set: Source sequences • Wide range of content characteristics. • SI/TI computed over equirectangular (ER), cube-map (CM) and spherical (SP), to account for possible projection distortions [De Simone, 2019].
  • 11. 11 ITU-T Rec. P.919 Subjective evaluation of short 360º videos • Test set: Encoded sequences • 8 HRCs (including Hidden Reference): • Kvazaar encoder, HEVC, fixed QP. • 4 homogeneous QPs: 15 (HR), 22, 32, 42. • 4 non-homogeneous QPs (tiles). #Tiles Transition ROI QPs 8x5** Smooth 90O 42 37 32 22 22 32 37 42 6x3 Smooth 120O 42 32 22 22 32 42 8x5** Abrupt 180O 42 42 22 22 22 22 42 42 6x3 Abrupt 120O 37 37 22 22 37 37
  • 12. 12 ITU-T Rec. P.919 Subjective evaluation of short 360º videos • Session structure and duration: • Each subject evaluates the same test clip under two different conditions: • Use within-subject statistics for better statistical power. • Each condition is tested in one active period: • After each active session: rest period of 15 min. • Red arrows: score simulator sickness. • App Miro360: Developed, tested, and validated subjective assessment SW: • For desktop (HTC and Oculus) and mobile (GearVR and Daydream). Intro Test Condition A Rest (15 min) Test Condition B Wrap-up Instructions in written form Visual acuity & color tests Training session Doubts/Questions C. Cortés, P. Pérez, and N. García, “Unity3D-based app for 360VR subjective quality assessment with customizable questionnaires”, IEEE ICCE 2019. https://git.gti.ssr.upm.es/pub/Miro360
  • 13. 13 ITU-T Rec. P.919 Subjective evaluation of short 360º videos • Simulator Sickness: • In each measurement, subjects scored both: • First, a single question (1-5 scale): always the same (A or B), randomly assigned • Second, SSQ [Kennedy et al., 1993] No problem No perceptible effect, natural feeling Light effects Slight discomfort, but no sickness Uncomfortable Moderate discomfort, but tolerable for a while Unpleasant Strong discomfort or sickness, but can continue the test Unbearable Strong discomfort or sickness, and want to stop test A. Are you feeling any sickness or discomfort now? (Vertigo scale) [Pérez et al., 2018] Very dizzy Dizzy Slightly dizzy Not dizzy Absolutely not dizzy B. How is your level of dizziness or nausea? (Short-SSQ) [Tran et al., 2017]
  • 14. 14 ITU-T Rec. P.919 Subjective evaluation of short 360º videos • Observers • At least 28 participants per lab. • Visual screening: Visual acuity and color vision. • Subjects were sitting in a swivel chair.
  • 15. 15 ITU-T Rec. P.919 Subjective evaluation of short 360º videos • Results of audiovisual quality: Wuhan, ACR, videos of 10s (blue) and 20s (orange).
  • 16. 16 ITU-T Rec. P.919 Subjective evaluation of short 360º videos • Results of audiovisual quality: All laboratories (all tested conditions), VSenseLuther.
  • 17. 17 ITU-T Rec. P.919 Subjective evaluation of short 360º videos • Results of SSQ: • Exploration results: Can we use a subset of the SSQ? PLCC among different SSQ
  • 18. 18 ITU-T Rec. P.919 Subjective evaluation of short 360º videos • Outcomes included in the recommendation ITU-T P.919: • Methodology: Both ACR and DCR can be used for assessment of coding quality. • Duration of videos: subjective tests of coding degradations with 360° videos can be done with sequences of 10 seconds. • HMD: any commercial HMD (tethered or untethered) can be used in visual quality tests (providing enough resolution and refresh rate) • Audio: it is possible to use test stimuli either with or without audio to evaluate visual quality (spatial audio was not considered). • Method to collect ratings: Voting interfaces and verbal voting. • Minimum number of observers: 28 participants. • SSQ: both Vertigo scale and VRSQ [Kim et al., 2018] can be alternatives to SSQ. • Dataset will be available soon in https://www.its.bldrdoc.gov/vqeg/video-datasets-and-organizations.aspx
  • 19. 19 Outline • Presentation of VQEG and IMG. • Subjective Evaluation of Visual Quality and Simulator Sickness of Short 360° Videos: ITU-T Rec. P.919. • Ongoing and future work: • Evaluation of QoE with long 360° Videos. • Evaluation of immersive and interactive communication systems.
  • 20. 20 Evaluation of QoE with long 360° Videos Ongoing and future work • After studying the evaluation of short videos à Long 360° videos. • Definition: a short (2-10 minutes) full content item. • Requires content immersive evaluation [Pinson et al., 2014]. • Evaluation: • Single-Stimulus Continuous Quality Evaluation [ITU-T P.910] à In-sequence • Single-Stimulus Discrete Quality Evaluation (SSDQE) [Gutiérrez et al., 2012] à In-sequence • ACR à Post-sequence • Outcomes: • SSDQE provides similar results to what ACR would provide. • SSCQE à to be analyzed. • ACR is significantly affected by in-sequence evaluation. • Quality evaluation does not affect sense of presence. HRC HR HRC HR HRC HR HR HR 20s 5s VOTE VOTE VOTE continuous voting SSCQE SSDQE Control Post- Seq Eval SSDQE / SSCQE (?) / ACR are valid for long sequences M. Orduna, P. Pérez, J. Gutiérrez, N. García, “Methodology to Assess Quality, Presence, Empathy, Attitude, and Attention in Social VR: International Experiences Use Case”, arXiv:2103.02550
  • 21. 21 Outline • Presentation of VQEG and IMG. • Subjective Evaluation of Visual Quality and Simulator Sickness of Short 360° Videos: ITU-T Rec. P.919. • Ongoing and future work: • Evaluation of QoE with long 360° Videos. • Evaluation of immersive and interactive communication systems.
  • 22. 22 Evaluation of immersive and interactive communication systems Ongoing and future work • How can we test bi-directional immersive communication systems? • Evaluate effect of technical factors (e.g., latency / bitrate / etc.) in QoE (e.g., immersion, interactivity, etc.) • Compare with other systems / experiments. • Related recommendations: • ITU-T P.920 - Interactive test methods for audiovisual communications. • Some tasks proposed to evaluate effect of technical factors of video- conference services (05/2000): • E.g.: one of the subjects shows and describes a plastic building block and the other one is required to reproduce it. • ITU-T P.13xx series: telemeeting assessment. • ITU-T P.QXM - QoE Assessment of eXtended Reality (XR) Meeting. • Best practices for QoE assessment of tele-meetings with XR elements.
  • 23. 23 Evaluation of immersive and interactive communication systems Ongoing and future work • Test plan: • Run a cross-lab experiment using any available collaborative immersive technology: • “The same” experiment in completely different setups • Gather a set of immersive communication systems: • Real-time 360° / Free-viewpoint video telepresence. • Social VR with pointcloud transmission / with avatars. • XR collaboration. • … • Create an experiment that covers all basic functionalities: • Conversation between people. • Discussion about objects in the immersive space. • Interaction with (local / remote / virtual) objects in the immersive space.
  • 24. 24 Evaluation of immersive and interactive communication systems Ongoing and future work • Identified technologies: FVV Live (GTI-UPM) Narciso García narciso@gti.ssr.upm.es The Owl (Nokia Bell Labs) Pablo Pérez pablo.perez@nokia-bell-labs.com VRTogether (CWI) Pablo César p.s.cesar@cwi.nl
  • 25. 25 Evaluation of immersive and interactive communication systems Ongoing and future work • Interested groups: • UPM (Spain) • Nokia Bell-Labs (Spain) • CWI (The Netherlands) • TU Ilmenau (Germany) • Ghent University (Belgium) • RISE (Sweden) • Wuhan University (China) • University of Surrey (United Kingdom) • AGH University of Science and Technology (Poland) • … • You?
  • 26. 26 Acknowledgment and references • Pablo Pérez • Marta Orduna • Ashutosh Singla • Carlos Cortés • Pramit Mazumdar • Irene Viola • Kjell Brunnström • Federica Battisti • Natalia Cieplinska • Dawid Juszka • Lucjan Janowski • Mikolaj Igor Leszczuk • Anthony Adeyemi-Ejeye • Yaosi Hu • Zhengzhong Chen • Glenn Van Wallendael • Peter Lambert • Cesar Diaz • John Hedlund • Omar Hamsis • Stephan Fremerey • Frank Hofmeyer • Alexander Raake • Pablo Cesar • Marco Carli • Narciso García • Philip Corriveau • Patrick Le Callet • … • F. De Simone, J. Gutiérrez, P. Le Callet, "Complexity measurement and characterization of 360-degree content", Human Vision and Electronic Imaging Conference, Jan. 2019. • J. Gutiérrez, P. Pérez, M. Orduna, A. Singla, C. Cortés, P. Mazumdar, I. Viola, K. Brunnström, F. Battisti, N. Cieplińska, D. Juszka, L. Janowski, M. Leszczuk, A. Adeyemi-Ejeye, Y. Hu, Z. Chen, G. Van Wallendael, P. Lambert, C. Díaz, J. Hedlund, O. Hamsis, S. Fremerey, F. Hofmeyer, A. Raake, P. César, M. Carli, N. García, "Subjective evaluation of visual quality and simulator sickness of short 360° videos: ITU-T Rec. P.919", IEEE Transactions on Multimedia, Jul. 2021 (Early access). • R. S. Kennedy, N. E. Lane, K. S. Berbaum, and M. G. Lilienthal, “Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness”, Int. J. Aviation Psychol., vol. 3, no. 3, pp. 203–220, Nov. 1993. • P. Pérez, N. Oyaga, J. J. Ruiz, and A. Villegas, “Towards systematic analysis of cybersickness in high motion omnidirectional video”, Int. Conf. Qual. Multimedia Experience, 2018. • H. T. T. Tran, N. P. Ngoc, C. T. Pham, Y. J. Jung, and T. C. Thang, “A subjective study on QoE of 360 video for VR communication”, IEEE Int. Workshop Multimedia Signal Process., 2017. • H. K. Kim, J. Park, Y. Choi, and M. Choe, “Virtual reality sickness questionnaire (VRSQ):Motion sickness measurement index in a VR environment”, Appl. Ergonom., vol. 69, pp. 66–73, May 2018. • M. H. Pinson, M. Sullivan, and A. Catellier, “A New Method for Immersive Audiovisual Subjective Testing”, Int. Work. Video Process. Qual. Metrics Consum. Electron., Jan. 2014. • J. Gutierrez, P. Perez, F. Jaureguizar, J. Cabrera, and N. Garcia, “Validation of a novel approach to subjective quality evaluation of conventional and 3D broadcasted video services”, International Workshop on Quality of Multimedia Experience, Jul. 2012. • M. Orduna, P. Pérez, J. Gutiérrez, N. García, “Methodology to Assess Quality, Presence, Empathy, Attitude, and Attention in Social VR: International Experiences Use Case”, arXiv:2103.02550. • P. Carballeira, C. Carmona, C. Díaz, D. Berjón, D. Corregidor, J. Cabrera, F. Morán, C. Doblado, S. Arnaldo, M.M. Martín, N. García, "FVV Live: A real-time free-viewpoint video system with consumer electronics hardware”, IEEE Transactions on Multimedia, 2021. • D. Berjón, P. Carballeira, J. Cabrera, C. Carmona, D. Corregidor, C. Díaz, F. Morán, N. García, “FVV Live: real-time, low-cost free viewpoint video”, IEEE Int. Conf. on Multimedia and Expo Workshops, Jul. 2020, • R. Kachach, M. Orduna, J. Rodríguez, P. Pérez, A. Villegas, J. Cabrera, N. Garcia, “Immersive Telepresence in Remote Education”, International Workshop on Immersive Mixed and Virtual Environment Systems, Sep. 2021. • R. Kachach, D. González-Morín, F. Pereira, P. Pérez, I. Benito, J. Ruiz, E. González, A, Villegas, "A Multi-Peer, Low Cost Immersive Communication System for Pandemic Times”, IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, Mar. 2021. • S. Rossi, I. Viola, J. Jansen, S. Subramanyam, L. Toni, and P. Cesar, “Influence of Narrative Elements on User Behaviour in Photorealistic Social VR”, International Workshop on Immersive Mixed and Virtual Environment Systems, Sep. 2021. • R. Mekuria, K. Blom, and P. Cesar Design, “Implementation and Evaluation of a Point Cloud Codec for Tele-Immersive Video”, IEEE Transactions on Circuits and Systems for Video Technology, 27(4), pp. 828-842, Apr. 2017.
  • 27. Quality assessment of immersive media: Recent activities within VQEG 5 October 2021 Jesús Gutiérrez Grupo de Tratamiento de Imágenes (GTI) Universidad Politécnica de Madrid (UPM) jesus.gutierrez@upm.es https://www.gti.ssr.upm.es www.vqeg.org MPEG AG 5 Workshop on Quality of Immersive Media: Assessment and Metrics