Survey on the State-Of-The-Art Methods for Objective Video Quality Assessment in Recognition Tasks

Akademia G´orniczo-Hutnicza
im. Stanislawa Staszica w Krakowie
AGH University of Science
and Technology
Video Quality Assessment in
Recognition Tasks
Kamil Kawa1 Mikolaj Leszczuk1 Atanas Boev2
1AGH University of Science and Technology, PL-30059 Krak´ow, Poland
vq@kt.agh.edu.pl
http://vq.kt.agh.edu.pl
2Huawei Technologies Duesseldorf GmbH, 40549 Duesseldorf, Germany
atanas.boev@huawei.com
Kamil Kawa, Mikolaj Leszczuk, Atanas Boev Video Quality Assessment in Recognition Tasks 1 / 23

www.agh.edu.pl
Outline
1 Problem Introduction

www.agh.edu.pl
Outline
2 Assessments Environment
General Viewing Conditions for Subjective Assessments in Laboratory
Environment
General Viewing Conditions for Subjective Assessments in Home
Environment

www.agh.edu.pl
Outline
Environment
Environment
3 Methods
Single Choice Method
Multiple Choice Method
Timed Task Method
Scenes

www.agh.edu.pl
Outline
Environment
Environment
3 Methods
Timed Task Method
Scenes
4 Conclusion

www.agh.edu.pl Problem Introduction
Problem Introduction
Nowadays, many metrics for overall Quality of Experience (QoE),
successfully used in video processing systems for video quality
evaluation
Both:
Full-Reference ones, like Peak Signal–to–Noise Ratio – PSNR or
Structural Similarity – SSIM
Non-Reference ones, like video quality indicators
However – not appropriate for recognition tasks analytic (Target
Recognition Video, TRV)

“Target” – object on video that tester needs to identify, e.g.:
Face
Object,
Number
TRV – video used as tool checking ability to recognise specific targets
of interests in video stream
TRV applicable in various services such as:
Surveillance
Licence place identification
Human identification
Telemedicine
According to, one can divide the target into three categories:
1 Human identification (including facial recognition)
2 Object identification
3 Alphanumeric identification

Given use of TRV, qualitative tests:
Not focusing on the subject’s satisfaction with video sequence quality
Measure how subject using TRV to accomplish certain tasks
Example purposes of this:
Video surveillance – recognition of vehicle license plate numbers
Telemedicine/remote diagnostics – correct diagnosis
Fire safety – ﬁre detection
Rear backup cameras – parking the car
Games – spotting and correctly reacting to virtual enemy
Video newscasts and reports editing – video summarization

Traditional approach to video quality assessment mostly focusing on
Quality of Service (QoS) techniques
However – obsolete method now
To prepare more accurate assessment of video quality, one to take into
account perception of user
Based on limitation of QoS for video applications, QoE describing
performance of whole, end-to-end video delivery system from user’s
point of view
Several essential factors aﬀecting perceived video QoE:
Quality degradation during the content production phase
Artefact introduced by lossy compression
Network transmission errors
Application and display device-speciﬁc parameters
End-user’s preferences and perception model

Moreover, methods and metrics, to fulﬁl following expectations:
In-service applicable
Non-reference quality assessment
High performance for diverse video content
Coverage of all mentioned factors contributing to overall QoE
Mapping between measured parameters (QoS, artefacts level) and QoE

Many parameters impacting ability to achieve recognition task, but
selecting ﬁve of them as most important ones:
1 Usage time frame – specifying if one in need to analyse video in
real-time or to be stored and analysed later
2 Discrimination level – specifying ﬁne level of detail sought from video
3 Target size – specifying whether predicted region of interest in video
occupies relatively small or large percentage of video
4 Lightning level – specifying anticipated lighting level of scene
5 Level of motion – specifying anticipated motion level in video scene

www.agh.edu.pl Assessments Environment
Assessments Environment
General viewing condition for subjective assessments to be met.
Conditions divided into:
Home environment
Laboratory environment

www.agh.edu.pl Assessments Environment General Viewing Conditions for Subjective Assessments in Laboratory Environment
General Viewing Conditions for Assessments in Laboratory Environment
The assessors’ viewing conditions should be arranged as follows:
Table: Viewing condition for subjective assessments in laboratory environment.
Ratio of luminance of inactive screen to peak luminance: <=0.02
Ratio of the luminance of the screen, when displaying only black
level in a completely dark room, to that corresponding to peak white:
≈ 0.01
Maximum observation angle relative to the normal 30
Ratio of luminance of background behind picture monitor
to peak luminance of picture
≈ 0.15
Chromaticity of background D65
Other room illumination low

www.agh.edu.pl Assessments Environment General Viewing Conditions for Subjective Assessments in Home Environment
General Viewing Conditions for Assessments in Home Environment
Viewing distance and screen size to be selected in order to satisfy PVD
(Preferred Viewing Distance).
Table: Viewing condition for subjective assessments in home environment.
Inactive screen vs. peak luminance <=0.02
Maximum relative vs. normal observation angle 30◦
Screen size for a 4/3 format ratio Screen size to satisfy PVD rules
Screen size for a 16/9 format ratio Screen size to satisfy PVD rules
Monitor processing without digital processing
Peak luminance 200 cd/m2
Environmental illuminance on the screen 200lux

PVD in function of screen size shown in Table
Information in table and function recommending PVD related screen
size that should be used
Table: Information on PVD and related screen sizes.
Screen diagonal
(in)
Screen height
(H)
PVD
4/3 ratio 16/9 ratio (m) (H)
12 15 0.18 9
15 18 0.23 8
20 24 0.30 7
29 36 0.45 6
60 73 0.91 5
>100 >120 >1.53 3-5

Figure: PVD for moving images.

www.agh.edu.pl Methods
Methods
ITU-T P.912 Recommendation introducing lot of useful methods for
recognition tasks
This Recommendation deﬁning subjective assessment methods for
evaluating quality of one-way video used for target recognition tasks

www.agh.edu.pl Methods Single Choice Method
Method to be used when single,
unambiguous answer to
identiﬁcation question
Technique utilisable for
alphanumeric character
recognition scenarios
Experimenter asking tester which
letter(s), or number(s) appearing
in speciﬁc area of video
Answer to be evaluated only as
binary one:
Correct
Incorrect
According to that, Yes or No
test – also acceptable

www.agh.edu.pl Methods Single Choice Method
For example, one asking viewer
if certain object present in scene
In such a method – essential to
ensure availability of easy to
understand answers
Care also to be taken to avoid
terminology diﬀerences
Use of “unsure” answer allowed,
but not recommended
Reason for that: over-usage of
this option by testers, leading to
contamination of experiment
result

www.agh.edu.pl Methods Multiple Choice Method
This method especially
appropriate for all discrimination
class levels (introduced in ITU-T
P.912) and target categories
For this method, experimenter
showing:
Video (this slide)
List of possible answers (next
slide)
After presenting video, viewers
to choose label being closest to
what recognised on clip

www.agh.edu.pl Methods Multiple Choice Method
Use of ﬁxed multiple choices
eliminating any possible
misunderstanding possibly
arising from open questions
Due to that, more accurate
measurements possible
Number of choices oﬀered to
tester depending on number of
alternative scenes presented
As in previous method, one to
take special care when ”unsure”
is one of listed choices

www.agh.edu.pl Methods Timed Task Method
Timed Task Method
Viewer to be asked to watch for particular action or object that viewer
about to recognise in video clip
When tester perceiving target occurrence, pushing button
In timed task, experimenter to determine whether time falling within
acceptable time-frame for decision making
This time-frames applicable for example in video scenarios:
“A tester is responding to a violent situation and needs to identify
whether crowd members have real weapons.”
“A person is chasing a vehicle and needs to read the license plate.”

www.agh.edu.pl Methods Scenes
Scenes
TRV used to perform a
recognition task
Scenes to contain targets
consistent with the application
under study
Measurement of test mostly
focused on subject’s ability to
identify objects and actions
Problem: viewer possibly
memorising scene content and
using other visual clues to
remember identity of target
Therefore, set of scenes
containing multiple versions to
replace particular scene

Scenes
It is best way to control
differences between versions
Example scenario:
“A person walks across the
field of view carrying objects.”
Set of videos to consist of
multiple shots using different
object and different people
Number of scenarios in set to be
large enough to reduce likelihood
of scene memorisation
Of course, experts to determine
content of each scene

Scenes
Diﬀerence to be, for example,
object carried by person on video
Experts to identify critical tasks
or parameters of scenes
One to base set of
multiple-choice answer and
experiment design on these
parameters
One to create scene in way that,
object of interest appearing in
video at resolution realistically
expected in practice

www.agh.edu.pl Conclusion
Conclusion
More general review of possible methods that could be used to video
quality assessment presented in paper
Based on limitation of QoS for video applications, QoE describing
performance of whole end-to-end video delivery system from user’s
point of view
Key point for every test session: general viewing conditions for
subjective assessments in laboratory and home environment
Various methods applicable to recognition task deﬁned in paper
Each method containing list with brief description of used methods in
scientiﬁc experiments

Survey on the State-Of-The-Art Methods for Objective Video Quality Assessment in Recognition Tasks

Recommended

Recommended

More Related Content

Similar to Survey on the State-Of-The-Art Methods for Objective Video Quality Assessment in Recognition Tasks

Similar to Survey on the State-Of-The-Art Methods for Objective Video Quality Assessment in Recognition Tasks (20)

More from Mikolaj Leszczuk

More from Mikolaj Leszczuk (20)

Recently uploaded

Recently uploaded (20)

Survey on the State-Of-The-Art Methods for Objective Video Quality Assessment in Recognition Tasks