Highlihgts of the results and outcomes of ITU-T Study Group 12 meeting in January 2017 : Performance, Quality of Service and Quality of Experience Author : Kwame BAAH-ACHEAMFUOR, Study Group 12 Chairman

1. ITU-T Study Group 12:
Performance, Quality of Service and
Quality of Experience
Highlights of January 2017 meeting
Kwame BAAH-ACHEAMFUOR
Study Group 12 Chairman
Geneva, 10-19 January 2017

2. 2

3. WP1/12: Terminals and multimedia
subjective assessment
Highlights of January 2017 meeting
Lars Birger Nielsen
WP1/12 Chairman
Gunilla Berndtsson
WP1/12 Vice-Chairman

4. P.381 “Technical requirements and test methods
for the universal wired headset or headphone
interface of digital mobile terminals”
Produced by Q.3 (Speech transmission characteristics
of communication terminals for fixed circuit-switched,
mobile and packet-switched (IP) networks)
Requirements for electroacoustic test and
corresponding test methods – incl. EN 50332-1 and 2
Specification of pin assignments of a socket connector
with three/four contact points
For characterisation of headset/headphones as well
as the interface of digital wireless terminals
P.381: one microphone
Revision for support of super-wideband and fullband
4
Revised Recommendations Consented

5. P.11x0 “In vehicle Recommendations”
Produced by Q.4 (Hands-free communication in
vehicles)
Revised Recommendation P.1100 (NB), P.1110
(WB) and P.1140 (eCall)
Measurement of Delay was aligned with 3GPP specifications
Adaptation of a new background noise equalization method
Alignment of descriptions and editorial updates
New Recommendation P.1120 (SWB + FB)
5
Recommendations Consented

6. P.501 “Test signals for use in telephonometry”
Produced by Q.6 (Analysis methods using complex
measurement signals including their application for
speech enhancement techniques and hands-free
telephony)
Texts of amendments have been moved into the
P.501 document
All speech samples and other samples will become
available for download once the approval process of
P.501 has been concluded.
6
Revised Recommendations Consented

7. New Recommendations Consented
P.1310 “(Spatial audio meetings quality
evaluation”
Produced by Q.10 (Conferencing and
telemeeting assessment)
Provide guidance on appropriate subjective
testing methods for evaluating of spatial audio
telemeeting systems – w/wo video
Methods for spatial audio on how to
Collect subjective quality ratings
Assess accuracy of the spatial alignment of
audiovisual systems
Obtain task performance indicators for the
intelligibility of concurrent speakers
Measure communication effectiveness using task
performance
Obtain task performance indicators for the cognitive
load experienced
7
Method Annex A Annex B Annex C Annex D Annex E
Rendering
Scenario
Audio-only
or audio component of
an audiovisual system
    
Audiovisual  
Test Focus Overall System     
Individual Components    
Target
Variables
Overall Quality    (3
 (3
Media
Quality
Non-spatial
quality

Spatial
quality
 
Communication quality   (2
(2
(2
Measurement
Sensitivity
Sufficient (test added
value of spatial audio)
    
High (differentiate
variants of spatial
audio)
1)  
Test
Paradigm
Listening-Only (Non-
interactive) Test
   
Conversation Test   
Measurement
Paradigm
Subjective Ratings  
Performance Measures   
Notes:
1)
Depending on the actual implementation of the method details, e.g. training phase, it is possible to
achieve high measurement sensitivity.
2)
These methods (indirectly) measure individual aspects of communication quality.
3)
These methods are not measuring overall quality, but they may be combined with subjective ratings
of overall quality.

8. New work items
Two new work items has been started
New work item on “Transmission characteristics for In
Car Communication” or P.ICC
New work item on “Subjective methodology for
Speech and Audio multi-channel Test” or P.SAT
8

9. Rapporteurs & SG12 Meetings
9
Q.4/12 “Objective methods for speech and audio
evaluation in vehicles”
Date: August 2nd
Place: Geneva, Switzerland
Topic: To progress on P.ICC
Q.5/12 “Telephonometric methodologies for handset
and headset terminals”
Date: May 29th and 30th
Place: Bern, Switzerland
Topic: To progress on P.TBN and P.Loudness
SG12 meeting
Date: September 19-28 (2017)
Place: Geneva, Switzerland

10. WP1/12: Terminals and multimedia
subjective assessment
Highlights of January 2017 meeting
Lars Birger Nielsen
WP1/12 Chairman
Gunilla Berndtsson
WP1/12 Vice-Chairman
END of WP1 PRESENTATION

11. WP2/12: Objective models and tools
for multimedia quality
Highlights of January 2017 meeting
Vincent Barriac
WP2/12 Chairman
Ludovic Malfait
WP2/12 Vice-Chairman

12. List of Questions under study
Q9/12
Perceptual-based objective methods for voice, audio and
visual quality measurements in telecommunication services
Q14/12
Development of parametric models and tools for multimedia
quality assessment of packet-based video services
Q15/12
Parametric and E-model-based planning, prediction and
monitoring of conversational speech quality
Q16/12 Framework for diagnostic functions
Q19/12 Objective and subjective methods for evaluating perceptual
audiovisual quality in multimedia services
12

13. Q.9: summary of discussion (1/2)
13
P.863
A modified version of P.863 that minimizes the under-
prediction in case of extended micropauses was introduced.
This version is available for testing to all interested parties in
ITU-T and commercial POLQA customers.
The results of a P.800 ACR subjective test focusing on the
evaluation of EVS and Opus coding was introduced and
discussed. Contributions providing additional experimental
results would be greatly appreciated to verify the performance
of P.863 and other objective measures on Opus and other
proprietary codecs.

14. Q.9: summary of discussion (2/2)
14
P.AMD and P.SAMD
Next SG12 meeting is targeted as the submission date for
candidate models for both sets A (5 dimensions) and B (7
dimensions).
For the new work item P.SAMD the existing Requirement
Specification is extended to cover both P.AMD and P.SAMD in
a single document.
P.SPELQ
The latest results are looking promising for narrowband
databases. Up to now, there are no published training results
for super-wideband databases.
A stable version of the P.SPELQ model is expected to be
submitted to TSB before the next SG12 meeting so the
evaluation can start

15. Q.9: outputs
15
No liaison statement
One document agreed:
Supplement for P series addressing P.863 and P.863.1:
“Application of P.863 and P.863.1 for speech processed by
blind bandwidth extension approaches”, available as TD163
rev 1.
One interim meeting is requested
Proposed time frame is June / Beginning of July
Main topics for discussion: P.AMD/P.SAMD and probably
P.SPELQ.
Exact date and location will be discussed and agreed via the
Q9/12 reflector. Discussion topics will be updated according
to the submitted contributions.

16. Q.14: summary of discussion (1/2)
16
P.1203: Parametric bitstream-based quality
assessment of progressive download and adaptive
audiovisual streaming services over reliable transport
amendment with the final performance figures for all four
modes, concluding the P.NATS phase 1 work
New work item for a new recommendation being the
extension of the P.1201 Recommendations
This new work item focuses on UDP-based video for new
codecs and higher video resolutions.

17. Q.14: summary of discussion (2/2)
17
New study item AVHD-AS/P.NATS phase 2 (joint
project with VQEG)
extension of the scope of P.1203 with new video codecs and
resolutions (including UHD), inclusion of a larger variety of
model types (new: pixel-based)
11 proponents have responded positively to the provisional
call for participation.
44 subjective test databases to be created
device types and viewing distance discussed and agreed
distribution of short vs. long sequence databases set to 70/30, training
vs validation databases to 50/50 for short sequence databases and
25/75 for long sequence databases.

18. Q.14: outputs
18
One document agreed
Amendment to P.1203 (available as TD139)
No liaison statements
Two Rapporteur Meetings are requested
Q.13, 14 and 17/12 (“Q.44”) plus VQEG AVHD (to discuss
AVHD / P.NATS Phase 2): End of March with the most likely
dates 22-24 March 2017, hosted by Deutsche Telekom / T-
Labs
Q.14/12 / AVHD-AS meeting on AVHD/P.NATS Phase 2 at
VQEG meeting to be held 8-12 May 2017 in Los Gatos,
California, USA, hosted by Netflix

19. Q.15: summary of discussion
19
Wideband extension to G.107.1
The decision has been taken to work on a basic SWB
version of the E-model. Will be re-used by Q.13 for
G.1070
P.CQO
The data received on the validation of the underlying
subjective procedure (P.CQS) was considered very useful,
but should be augmented by results from other test labs,
involving different test set-ups, participant groups, and
potentially languages to substantiate the subjective
method.

20. Q.16: summary of discussion
20
P.TCA: Technical Cause Analysis
It was proposed to draft a Recommendation that
provides P.TCA degradation classes, their description
and exemplary audio samples. This Recommendation
would provide the industry with a reference for common
terminology.
E.FINAD: Framework for Intelligent Network
Analytics and Diagnostics
A new work item is proposed to recommend a
framework for identifying anomalies and diagnosing root
cause from data collected from production networks
such as logs, statistics, alerts, etc…

21. Q.19: summary of discussion
21
J op-tr (Methods for Optimizing Bitrates and
Transmission Resolution by Considering Display
Characteristics and Available Bandwidth)
New progress based on experimental results and statistical
analyses provided in C 67
showed that equivalent perceptual video quality could be provided with
reduced bandwidth by considering the display characteristics (size and
resolutions).
J.noref
Potential interaction with Q.14 (P.NATS phase 2) in the near
future
test data of P.NATS-Phase 2 could be re-used for the validation of J.noref
possibility to any model originating from Q.19 to be submitted as
candidate models to P.NATS phase 2 competition and work item
distinction between scopes: P.NATS phase 2 work item is focused on TCP
based video, while J.noref work item originates from broadcast type of
video and includes UDP based video and packet loss.

22. IRG AVQA
22
Decision to continue
Co-chairman on behalf of SG12 unchanged: Jens
Berger (Q.9 rapporteur)
One liaison statement sent to SG9 (available as TD
160)

23. WP2/12: Objective models and tools
for multimedia quality
Highlights of January 2017 meeting
Vincent Barriac
WP2/12 Chairman
Ludovic Malfait
WP2/12 Vice-Chairman
END OF WP2 PRESENTATION

24. WP3/12: Multimedia QoS and QoE
Highlights of January 2017 meeting
Paul Coverdale
WP3/12 Chairman
Al Morton
Tiago Sousa Prado
WP3/12 Vice-Chairmen

25. New Recommendations Consented
E.847 (G.PoiCong) “QoS norms for
TDM Interconnection between
Telecom Networks”
This Recommendation analyses and
identifies QoS parameters for TDM
interconnection between telecom
networks, which are needed to facilitate
effective interconnections with
reasonable traffic handling capacities.
25

26. New Recommendations Consented
Y.1545.1 (Y.FMIPQoS) “Framework
for Monitoring the QoS of IP network
services”
This Recommendation is a diagnostic
reference for IP network quality of
service monitoring, and primarily as a
guide to assist regulators monitor the
quality of service of internet that is
provided by service providers (although
subscribers and network service
providers may also derive benefit). 26

27. New Recommendations Consented
E.811 (E.QMME) “Quality
Measurement in Major Events”
This Recommendation addresses the
quality assessment of mobile broadband
and voice services provided during
major events, such as FIFA World Cups
and Olympic and Paralympic Games, by
creating a useful international reference
to be considered by operators and
regulators when preparing to host such
events. 27

28. Amended Recommendations
Consented
E.802 “New Annex A on guidelines
on selection of representative
samples”
provides guidance on the selection of
representative samples in the
measurement of QoS parameters
28

29. Liaisons
To SG11 and TSAG regarding the
work item Y.FMIPQoS (now
Y.1545.1)
We have consented Y.1545.1
The WTSA-16 guidance indicates that
SG 12 has clear responsibility for
various aspects of QoS, including
measurement methodologies
29

30. Liaisons
To 3GPP on KPIs for mobile services
requests information and appropriate
references related with Voice and Data
QoS KPI thresholds on mobile networks.
30

31. Liaisons
To SG16, VQEG, MPEG, 3GPP SA4
Informs about a new study item on
quality of experience (QoE) for virtual
reality (VR) services (G.QoE-VR)
31

32. Liaisons
To SG2 on study of website traffic
characterization
Informs that we have removed G.MFWT
from our work items
32

33. Liaisons
To SG5 on proposed collaboration on
accuracy and availability
requirements regarding soft error
requirements
Recommends to recognize the evolution
to VNF-based telecom architectures in
K.soft_des
33

34. New work items
G.ViLTE
End-to-end QoS for Video Telephony
over 4G mobile networks
E.RQUAL
Strategies to Establish Quality
Measurement Frameworks
E.NetPerfRank
Statistical Framework for QoE Centric
Benchmarking Scoring and Ranking
34

35. New work items
E.RQST
Voice and Data QoS KPI thresholds on
mobile networks
G.QoE-VR
QoE for Virtual Reality
35

36. Interim meetings
To be co-located with Q14/12
Question 13/12
One meeting requested
Tentatively March 2017 in Berlin
Question 17/12
One meeting requested
Tentatively March 2017 in Berlin
36

37. WP3/12: Multimedia QoS and QoE
Highlights of January 2017 meeting
Paul Coverdale
WP3/12 Chairman
Al Morton
Tiago Sousa Prado
WP3/12 Vice-Chairmen
END OF WP3 PRESENTATION

38. Next Study Group 12 meeting
19-28 September 2017, Geneva,
Switzerland
SG12 Website: www.itu.int/go/tsg12
38

39. 39