Quality of Experience: Measuring Quality from the End-User Perspective

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

QUALITY OF EXPERIENCE
Measuring Quality from the End-user Perspective
TEWI Kolloquium 20 Nov 2019
Dr. Raimund Schatz, AIT/...

About Me Raimund Schatz
Senior Scientist @ AIT
Austrian Institute of Technology
Recently joined ATHENA @ AAU
Dr. (Informat...

Agenda
§ Welcome & Introduction
§ What is QoE?
§ Origins & History
§ QoE Definition(s)
§ How to measure QoE?
§ Overview: O...

PART 1: WHAT IS QOE?
4AIT | 2019

Let‘s Warm Up a Bit …
How would you define „Quality“? How is the term being used?
What is Quality?

Quality – Is in Fact an Elephant!
The blind men and the elephant: Poem by John Godfrey Saxe
And so these men of Indostan
D...

What is Quality, anyway?
§ QD1: Quality as Qualitas
§ Essential nature, inherent characteristics, characteristic attribute...

QoS
Technology-centric:
throughput, delay, packet loss, etc.
QoE
User-centric:
what really matters to the
end-user: respon...

QoE: Some Definition Attempts
§ QoE as a reloaded buzzword:
QoE has been defined as an extension of the traditional QoS in...

User Expectations
regarding system
performance
Smartphone vs.
Tablet vs. TV
Influence of
network speed,
latencies, etc.
Ap...

“Quality of Experience (QoE) is the degree of delight or
annoyance of the user of an application or service. It results
fr...

“Quality of Experience (QoE) is the degree of delight or annoyance of a
person whose experiencing involves an application,...

§ Fundamental relationships and data on quality perception
§ QoE as f(System, User state, Content, Context)
§ WQL Hypothes...

PART 2: HOW TO „MEASURE“ QOE?
14AIT | 2019

Common Quality Issues for Networked Multimedia
§ Web Browsing:
§ Long waiting time until anything happens
§ Slow page rend...

General Question: Can we directly „measure“
experienced Quality (QoE)?
§ Answer: NO, not yet!
§ Why?
§ Elusive concept
§ N...

How to assess or estimate QoE?
A) Subjective QoE Tests
§ Based on end-user involvement
§ Subjective measures: e.g. user op...

User
App
Net
Application
Log
Analysis
Traffic
Analysis
Subjective
Testing
Method
User Quality
Perception Data
Insights
QoE...

Overview: QoE Assessment Approaches
QoE
Measurement
Subjective
Controlled
Experiments
(Lab)
Crowdsourcing
Field / Real
Ser...

Subjective QoE Assessment
Key Question: How to assess QoE at maximum validity?
Answer: Subjective QoE Testing with Human P...

The Process
21
Reporting
Results Analysis
Execution
Pilot & Refinement
Setup
Planning

PLANNING & SETUP
22

Rule #1: Know your Purpose!
§ Every study is done for a purpose …
à know your purpose and clearly define the problem that ...

Test Methodology & Design
§ Variables
§ Which ones to manipulate, control, observe or ignore?
à Avoid unintended/unnoticed...

Recommended Reading
§ http://www.doesinc.com/knowledge.htm
§ http://www.statsoft.com/textbook/experimental-design/
§ ITU-T...

Example: Subjective Image Quality Testing
§ Given: Source Image, System that impairs image (compression,
transmission erro...

A typical lab assessment involves …
•15 to 30 participants
•1-2 hours per participant
•Informed consent / GDPR (signed)
•I...

Test Content
§ Has considerable impact on quality perception
§ Content choice depends on study goals, e.g.
§ Typical conte...

Which User Data/Correlates with QoE can we collect?
§ Subjective Opinion / Assessment
§ Quantative: Ratings
§ Qualitative:...

How to Obtain User Ratings: Scaling Methods
Scaling
Direct
Sg Stimulus
ACR
Dbl Stimulus
DCR
Continuous
SAMVIQ &
MUSHRA
Ind...

Key Measure: MOS
§ Mean Opinion Score
§ Widely used in many fields:
§ Politics/Elections
§ Marketing/Advertisement
§ Food ...

Direct Scaling: ACR (Absolute Category Rating)
§ Discrete
§ Single stimulus
§ Multiple dimensions addressable
§ Usually 5-...

Direct Scaling: DCR (Degradation Category Rating)
§ Discrete
§ Direct Comparison à Relative
§ Reference vs. processed samp...

Scaling: Continuous
§ Continuous / Sg or Dbl Stimulus
§ For assessing transient quality
artifacts in longer (media)
sample...

Exercise: Which Direct Scaling to Use?
§ Assessment Tasks
1) Impact of (constant) noise on QoE (image)
2) Impact infrequen...

Scales: How to Map Opinions to Numbers?
§ Not all features that an entity has can be described by numbers
§ Example: a per...

Consequence: Another essential Rule …
§ ANTICIPATE, ANTICIPATE, ANTICIPATE!
§ Beyond clearly defining the problem …
§ … pr...

Some Scale Designs (ACR)
38

Some Scale Designs (ACR) ctd.
§ Ideal for Crowdsourcing (Gardlo, Egger & Hossfeld 2015)
39
Gardlo, B., Egger, S., & Hoßfel...

Setup: Test Environment
§ Measurements have to be valid, objective & reliable
§ Subjective testing methodologies
§ High re...

Planning: Time & Test Conditions
§ Subject‘s Time & Energy = scarce resource
§ Max. 90 min of net testing time, requires a...

Number of Subjects - The dreaded „N“!
§ Huge Trade-off: Time & Money vs. Statistical Power / Reliability
§ Higher N à smal...

Analysis by Brunnström & Barkowsky (2018)
§ Traditional recommendations too optimistic, particularly when you have to corr...

Sample Test Plan
44

PILOT & REFINEMENT & EXECUTION
45

46
§ Subjective tests are like a live performance
§ When the audience arrives and the show starts, everything has to be in...

47
§ Treat your subjects well – for them participation should be a positive
experience in itself!
§ Monitor your data, you...

48
§ By TU–Ilmenau, AVT Group
§ https://github.com/Telecommunication-Telemedia-Assessment/avrateNG
Rating Tools/Systems#1:...

Setting up a new user study can be tedious…
TheFragebogen.de
A software framework for user studies made simple.
Used in se...

ANALYSIS & REPORTING
50
Conditions
Items
Subjects

!"" #"" $"" %"" !&""
!
!'(
#
#'(
)
)'(
$
$'(
(
*+, -./01,2./03456
7*8
!
!
89:
;89:
<=>
MOS Data Analysis and Reporting
• M...

!"" #"" $"" %"" !&""
!
!'(
#
#'(
)
)'(
$
$'(
(
*+, -./01,2./03456
7*8
!
!
89:
;89:
<=>
MOS Data Analysis and Reporting, ct...

!"" #"" $"" %"" !&""
!
!'(
#
#'(
)
)'(
$
$'(
(
*+, -./01,2./03456
7*8
!
!
89:
;89:
<=>
MOS Data Analysis and Reporting, ct...

!"" #"" $"" %"" !&""
!
!'(
#
#'(
)
)'(
$
$'(
(
*+, -./01,2./03456
7*8
!
!
89:
;89:
<=>
MOS Data Analysis and Reporting
• M...

Remember: ALWAYS be Cautious with Summary
Statistics!
F. J. Anscombe: Graphs in
Statistical Analysis. In:
American Statist...

More Extreme: The Datasaurus Dozen
https://www.autodeskresearch.com/publications/samestats

Reporting (as recommended by ITU-T P.910)
57

Further Reporting Examples …
58
Schatz et al. 2018

Further Reporting Examples (ctd)
59
Schatz et al. 2018

How to navigate the seven seas of
statistical analysis?
60
Ratings,etc.
QoS, Encoder settings, etc.

61
How to navigate the seven seas of statistical analysis? (ctd)
Source: J. Wobbrock http://depts.washington.edu/acelab/pr...

62
§ http://www.panelcheck.com/
Tools from Food Sciences can make your life a bit easier …
Hint: check out the work (paper...

63
§ Think of Reproducible Science!
1. Publish the Dataset
§ Ideally: Content + Rating Data
2. When sharing: use standardi...

64
§ Emerging technologies: XR, VR video,
point-cloud, mulsemedia, …
§ New targets à more work = good J
à Many more DOF à ...

Thank you for your attention!
Any questions?

QoE-related Books
§ Quality Engineering (Möller, 2011)
§ Very good introductory book on quality engineering
§ Focus on aud...

Related Philosophy: Pirsig (1974)
§ Zen and the Art of Motorcycle Maintenance
§ possibly the most widely read philosophy b...

Further Reading, ctd.
§ European Network on Quality of Experience in Multimedia Systems and
Services (QUALINET), “Definiti...

Further Reading: Crowdsourcing
§ T. Hoßfeld, C. Keimel, M. Hirth, B. Gardlo, J. Habigt, K. Diepold, and P.
Tran-Gia, “Best...

Further Reading, ctd.
Balachandran, Athula, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, and
Hui Zhang. 2012....

Further Reading, ctd.
Akamai. June 2006: Retail Web Site Performance: Consumer Reaction to a Poor Online
Shopping Experien...

Further Reading, ctd.
Strohmeier, Dominik. „Open profiling of quality: a mixed methods research approach
for audiovisual q...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1052471 views
Storytelling For The Web: Integrate... by Chiara Aliotta 967761 views
Artificial Intelligence, Data and C... by OECD Directorate ... 900577 views
How to Leverage AI to Boost Employe... by SocialHRCamp 380598 views
2024 State of Marketing Report – by... by Marius Sescu 215763 views
Everything You Need To Know About C... by Expeed Software 233752 views

View on Slideshare

1 of 72 Ad

View on Slideshare

1 of 72 Ad