It was once thought that high QoS (Quality of Service) performance solves recurrent problems of low-quality multimedia services. Since then, solutions have been proposed to ensure a high level of QoE (Quality of Experience). In this document, the authors attempt to outline his understanding of an accurate meaning of quality of multimedia services. Starting from QoS and passing through generalised QoE, the authors focus on aspects of subjective and objective quality modelling and optimisation of visual performance for TRV (Target Recognition Video) applications (such as video surveillance), outlining the path of ITU-T standardisation in this area. The authors revised the ITU-T Recommendation P.912 to reflect improved subjective test techniques developed since this Recommendation was approved. The authors also attempt to predict at least some existing errors of reasoning, which are likely to become evident for the industry in the next decade.

1. Selected Aspects of the New Recommendation on
Subjective Methods of Assessing Video Quality in Recognition Tasks
Opole, 18.10.2022
Mikołaj Leszczuk, Lucjan Janowski
Akademia Górniczo-Hutnicza im. Stanisława Staszica w Krakowie
AGH University of Science and Technology

2. Presentation Outline
• Introduction
• Target Recognition Video, TRV
• Who is doing it?
• Standardization of methods for subjective evaluation of TRV
• Objective video quality assessment methods for TRV

3. Introduction

4. Introduction (1/2)
• Some years ago: proposals suggesting quality measured not only at on
Quality of Service (QoS) level but on user level - Quality of Experience
(QoE)
• Special frameworks of integrated assessment of quality of video
sequences
• Frameworks including solutions that attempt to model overall quality,
operating at:
• Intersection of QoS and QoE, or
• Only in QoE

5. Introduction (2/2)
• QoE depending on number of contextual parameters
• Only full understanding, possible with limitations of area of application,
makes possible to measure and optimize quality
• High numbers of contextual parameters mean research question still
open
• Effect:
• General approach OK for entertainment
• Not OK for Target Recognition Video, TRV

6. Target Recognition Video (TRV)

7. TRV Specifics
Why do we need special quality approach for TRV?

8. 240p,
24 BPP,
low bitrate:
• Perceived quality:
fair
• Recognition rate:
bad
1080p HD,
24 BPP,
high bitrate:
• Perceived quality:
excellent
• Recognition rate:
excellent
1080p HD,
1 BPP,
low bitrate:
• Perceived quality:
bad
• Recognition rate:
excellent

9. TRV Specifics
In many visual applications, the quality of the motion picture is not as
important as the ability of the visual system to perform specific tasks
for which it is created, given the processed video sequences

10. Typical TRV Examples
Video
surveillance
Telemedicine/
remote
diagnostics
Fire safety
Backup
cameras
Computer
games

11. TRV Tests (1/2)
• Different TRV quality
understanding, but similar
problems
• Verification? Quality tests
needed
• Purpose: (for task) relate
TRV quality with desired:
• Detection probability, or
• Recognition accuracy

12. TRV Tests (2/2)
• TRV qualitative tests:
• Not focusing on subject’s satisfaction with quality
• Focusing on measuring TRV to accomplish tasks
• Purposes of this may include:
• Video surveillance -> licence plate recognition
• Telemedicine -> correct diagnosis
• Fire safety -> fire detection
• Backup cameras -> parking car
• Games -> spotting and reacting to virtual enemy

13. TRV Quality Assessment Issues
• Method of selecting source signal (TRV sequences)
• Testing methods and general manner of conducting experiment
• Selecting group of subjects in experiment
• Instructing and training subjects before experiment
• Conditions in which test will be carried out
• Statistical analysis and presentation of results

14. Who Is Doing It?

15. Video Quality Experts Group
• Vision: “To advance the field of video quality assessment...”
• International experts: industry, academia, government organizations,
standard-developing organizations (ITU, …)
• Main activities today:
• Improving subjective testing methods
• Joint model development
• Validation of quality models
• Submissions to ITU Recommendations
• More information: http://www.vqeg.org/

16. Standardization of Methods for
Subjective Evaluation of TRV

17. Recommendation
ITU-T P.912
I n t e r n a t i o n a l T e l e c o m m u n i c a t i o n U n i o n
ITU-T P.916
TELECOMMUNICATION
STANDARDIZATION SECTOR
OF ITU
(03/2016)
SERIES P: TERMINALS AND SUBJECTIVE AND
OBJECTIVE ASSESSMENT METHODS
Audiovisual quality in multimedia services
Information and guidelines for assessing and
minimizing visual discomfort and visual fatigue
from 3D video
Recommendation ITU-T P.916
• Methods for TRV
Evaluation
• Addressing questions
formulated in previous
section
• Organizes terminology
related to subjective TRV
testing
• Introducing appropriate
definitions for methods of
testing (psychophysical
experiments)

18. Objective Video Quality
Assessment Methods for TRV

19. Objective Assessment Methods for TRV
• It is possible to develop the new concept of an objective models for
evaluating video quality for TRV
• For this purpose, third-party systems are used:
• Values of various image indicators
• Results from automatic recognition engines
• Modelling using Neural Networks (NN) turns out to be promising, e.g.:
• Facial recognition: 0.869
• Automatic License-Plate Recognition: 0.777
• Object detection: 0.730
• Modelling with the use of NN is facilitated by the possibility of
automated creation of training sets

20. Thank you for attention!
https://qoe.agh.edu.pl
qoe@agh.edu.pl