The document discusses quality of experience (QoE) evaluation for next generation networks starting from ITU recommendations. It describes the background of telecom carriers moving to new IP/MPLS networks due to the obsolescence of TDM technology. Metrics and tools like FreeProbe are presented for validating QoE by analyzing video and audio quality over IP networks, along with test results from an IPTV service testing model that show the impact of packet loss and delay on subjective quality ratings.

1. Eng. Luis Enrique Conde del Oso.
ETECSA. Cuba
luis.conde@etecsa.cu
Telephone: 537 2666825 Mobile 53 52884472
“Evaluation of the QoE in NGN starting from UIT recommendations.”
“ITU-T Standardization and other key ITU Activities”
Havana, Cuba from 8 to 9 February 2011.

2. • Background.
Current scenario
of Telecom
Carriers
• Obsolescence of TDM Technology.
• Requirements for new services and increase of the current services.
- Fixed telephone service growth.
- Mobile service growth
- Wide development of data services.
- Introduction of novel services such as IPTV.
Proposal of Development
• Development of Services on NGN networks with NAM (GIE).
• Optimization of National Transport Network IP/MPLS.
• Designs and implementation of Local and Regional Metro
Networks.
• Introduction of the objective quality validation of services in real
time (and its extension to quality profile of telecommunications
network).
Beginning of the scheduled development
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 1

3. • The Metro Ethernet like support of the
Broadband.
Source: Telecom Italia – VPDT ETECSA.
Standards defined in the T-MPLS:
G.8110.1 Network Architecture of T-MPLS
G.8113 Requirements OAM
G.8114 OAM
G.8112 UNI/NNI
G.8131 Lineal Protection
G.8132 Rings Protection
G.8121 Equipment T-MPLS
G.8151 EMF
G.8152 Infomodel
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso.
Ring
Ring
Ring
To the Service POP
To the Service POP
To the Service POP
2

4. • Pseudo rings of the Provincial Metro Ethernet of Havana. Network simulations in TILAB.
Source: VPDT. ETECSA.
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 3

5. BRAS PE
IP/MPLS
AR BRAS PEAR
METRO
REGIONAL
Service PoPService PoP
METRO
PROVINCIAL ADMADM
ADMADM
ADMADM
Eth/SDH
Source: VPDT. ETECSA.
E1
STMn
FE
GE
Platform
of IPTV
VoD
VoD
Punto 1
Validación
Punto 2
Validación
Punto 3
Validación
(Signal encrypted)
(Signal no
encrypted)
The Operators of
Telecommunications
generally use unique
network which
include the return
channel, and they are
IP broadband network
with solutions FTTH,
FTTB and FTTC and
last mile
Technologies ADSL,
ADSL2 +, VDSL,
VDSL2 + and GPON.
In these network
protocols like PIM,
IGMP and BGP should
be activated.
• Different levels of a NGN Network.
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 4

6. • Quality Profile on networks, according to the following ITU-T Recommendations G.1050, and ITU-T
Y.1541.
Source: UIT-T.
Source Spirent.
DSL Forum's WT-126
•SD. 1 pack./loss every
30 minutes.
(MLR=0.004 pack/seg)
•HD 1pack./loss. every
4 hours
(MLR=0.0005 pack/seg)
Source: DSL Forum's WT-126
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 5

7. Source Asistencia Técnica de Telecom Italia.
• Quality Profile of ToIP, according to the ITU-T Recommendations .
ITU-T G.114 Delay ≤ 300 mseg.
(Bidirectional).
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 6

8. • Obtaining of the information for the validation of the quality of the video signal.
A video signal to be
transmitted over a IP
network should be
compressed. The codec
MPEG defines a
sequence of images I, B
and P, which a GOP
conforms. In the header
of these packets they are
fields that possess
outstanding information's
to be able to make the
validation of the quality
of the video signal
Source: Shenick.
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 7

9. 8 Bytes
• Main fields of the header in Transport Stream.
Source: UIT-T
They group in 3 levels of priority:
Priority 1. Correct decoding.
Priority 2. Quality Tx of the IP
Package.
Priority 3. It Influences in specific
applications.
Fecha: 4/11/2010 Vicepresidencia Desarrollo y Tecnología
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 8

10. • QoE and QoS.
SLA SLA SLA
Human Vision Constraints.
Perceived Quality (QoE), at service level end to end.
STB influence
Quality
dependence of
the Service
Network Quality (QoS), transport packet functionSource: VPDT. ETECSA.
Data Metrics
“Same value of PSNR”
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 9

11. Metrics
(VQM)
Subjective
Metrics
Implicit comparison (DSCQS).
Explicit comparison (DSIS)
Absolute valuations (SSCQE)
Valuations with Absolute Category (ACR).Objective
Metrics
Ref. Complete.
Ref. Reduced.
Without Reference.
Picture Metrics
Analysis of the Information
visual contained.
VHS Metric.
(Aspects of the Human vision)
Data Metrics
Analysis of Videos
Decoded.
MSE
PSNR
(Evaluation of Packages)
Flows of Video
Metrics
T- Factor (Deep Inspection of the flow of the Video (TS
and UDP). The impact of the disturbances of the Network
depends on the Codec, the losses and its propagation
(space and time).
MDI (Slight Inspection of the Video flow (TS and UDP).
Independent of the Codec IP Network.).
Correlation
Source: VPDT. ETECSA.
• State of the art of validation Metrics for video quality.
BT.500.11 (Images of TV).
P.910 (Aplic. of Multimedia), Video Quality.
P.911 (Aplic. of Multimedia), Audiovisual Quality.
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 10

12. FreeProbe with T- Factor Metric.
• Development of metrics in FreeProbe Tools. (Software open source).
Source: Telecom Italia - VPDT. ETECSA.
FreeProbe with MDI Metric.
Freeprobe for Codec H.264 AVC.
FreeprobeJPER (ToIP Service)
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 11

13. • Windows (GUI) of FreeProbe (T-Factor Metric ) .
SA: Service Availability.
SD: Service degradation.
SI: Service Impairment.
It is based on the
propagation of errors
inside the GOP, among
images I, P and B during
the transport of the same
one in the IP network,
taking in consideration the
behavior of the error
accountants SAE, SDE and
SIE, in the period of
analysis of the quality,
having in bill determined
preset thresholds and the
priority defined in the
obtained parameters of the
head of the packages.
Fuente Telecom Italia - VPDT. ETECSA.
Delay
Jitter
Packet
Loss
Dependent of the codification
Metric with capacity of identification of
errors at level of MPEG-2 caused by
losses of packages UDP/RTP.
Metric of macro indicators SA,SD y SI
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 12

14. Here the impact of the transcodificación of the signal was evaluated due to not having the contents the
same code format as well as the employment of a Hybrid solution in Havana City and its repercussion
starting from the behavior of the error accountants SAE, SDE and SIE in the period of evaluation of the
quality and its correlation with the subjective observation of the sign.
This test allowed the
evaluation of the
potentialities and the
weaknesses of the
ETECSA access network
to support a service of
BTV, MPEG-2, and it
consisted on the design
of a Hybrid Network that
simulates the solution
used by Orange in its
service of IPTV, which
allows to increase the
covering of a service of
BTV in areas where you
cannot offer an
appropriate band width
for this service. It
facilitates the
determination of Eligible
and Not Eligible Users
• Test with the FreeProbe (T-factor), simulation of the solution of Orange.
Source. VPDT. ETECSA.
200 Watts
Elegible Users
≤ 500 m @ 4 Mbps
Not Eligible Users
2 Km. @ 512 Kbps
SA: Service Availability.
SD: Service Degradation.
SI: Service Impairment.
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 13

15. It is based on the analysis of the
behavior of the buffer,
“Underflow” or “Overflow” of the
same one, that which can cause
the freezing of the image or the
appearance of defects in the
same product of excessive
losses of packages, having
bigger repercussion if these lost
packages contain images of the
type I or P.
Fuente Telecom Italia
DF: MLR
(RFC- 4445)
Propuesta
It doesn't depend on the codification
DF = [VB(max) – VB(min)] / MR
MLR = (Prospective packages-
Received Packages) / interval of
time in secondsDelay
Jitter
Packet
Loss
A Buffer Virtual minimum 1Mb, requires
which adds a retard of 266.67 mseg
(1Mb / 3.75 Mbps)
Metric Media Delivery Index
DF= Delay Factor
MLR= Media Loss Rate
DF:MLR
• Windows (GUI) of FreeProbe (Metric MDI) .
Source Telecom Italia - VPDT. ETECSA.
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 14

16. Source VPDT. ETECSA.
The introduction of the
metric MDI in the
Freeprobe and the
necessity of making
experimental tests to
define the necessary
objective-subjective
correlation and to
determine the respective
thresholds in each
disturbance type, made
necessary the design of a
laboratory testing model.
• IPTV Service Testing Model applied at ETECSA test.
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 15

17. • Services implemented in IPTV Platform in Testing Model.
Source: VPDT. ETECSA.
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 16

18. Video Server
NetDisturb
Source VPDT. ETECSA.
Source: Telecom Italia – VPDT ETECSA.
• Training preparation of the subjective observer.
Delay
Jitter
Packet Loss
UIT-T J.144R y UIT-R BT.1683.
Muy Molesto Presencia elevada de errores que perjudican el audio y el video
(1)
(2)
(3)
(5)
(4)
MOS DMOS
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 17

19. Source: Telecom Italia - VPDT. ETECSA.
• Test results done in the testing model (Channel SD to 4 Mbps)
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 18

20. Source: Telecom Italia - VPDT. ETECSA.
Results of the Simulations Tests.
Uniform packet loss: Superior losses to 0.001% are inadequate
for a service of IPTV and they have bigger impact in the audio
one that in the video.
Packet Loss in Burst: Bigger affectation in the video that in the
audio one, however they are less annoying for the user than the
uniform losses.
Delay and the Jitter: Affectations are not observed in the range
between 0 and 50 mseg, what demonstrates the
recommendations of the WT-126 of the DSL Forum´s.
• Test results done in the testing model (Channel SD to 4 Mbps)
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 19

21. Serv
POP 2
DWDM
Serv
POP 4
Serv
POP 1
Serv
POP 3
Pseudo ring
Ethernet
Pseudo ring
Ethernet
Pseudo ring
Ethernet
CUB
Source: Telecom Italia - VPDT. ETECSA.
FreeProbe
FreeProbe Server
Measurement point
Data collect
FreeProbe
IPTV
Head End
FreeProbe
•Centralization of QoE Validation with FreeProbe to substitute solutions
with Shadow Router. (Cisco, Tektronix, JDSU, Huawei).
Metric MDI
Metric MDI
Metric T-Factor
SNMP
SNMP
SNMP
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 20

22. VLAN
QoS
Source: VPDT. ETECSA.
VLAN 10
VLAN 20
VLAN 30
• Results of the Tests of the Triple Service Play with FreeProbe (MDI).
Results of the tests of a
Triple Play service with
the use of the FreeProbe
with metric MDI in the
testing model, with the
services BTV (1.5 Mbps
MPEG-2), Access to
Internet and ToIP
(G.729A) on a Network
with QoS and without
implemented QoS
(Network BPL vs Intranet
of ETECSA).
QoS
BPL
Intranet
Nacional
de
ETECSA
Red
Eléctrica
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 21

23. Video Server
Source: VPDT. ETECSA.
• Methodology for the measurement of the capacity of memory of temporary
storage (Buffer) of the STB AmiNET110.
1.5 seg
4 Mbps and codification MPG2
Methodology for the
measurement of the capacity of
memory of temporary storage
(Buffer) of the STB AmiNET110.
Capacity of Memory = 6 Mb.
(Without Packets Loss)
HUB
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 22

24.  Conclusions.
The design of the IP network of broadband in ETECSA with the implementation of protocols
like the PIM, IGMP and BGP among others constitutes the fundamental pillar of the
development of the Objective Network of ETECSA, allowing the expansion of the NGN and the
possibility of offering novel services like the IPTV.
The valuable theoretical and investigative experiences developed for the appropriate
correlation of the QoS and the QoE with the employment of the probe FreeProbe, with metric
T-factor and MDI, they have allowed ETECSA to have a tool personalized for the validation of
the quality of the service of IPTV, which was extended, with the addition of new metric in the
Freeprobe that include the validation of the service of ToIP (at the moment in test).
The Platform of Services of IPTV developed in ETECSA, jointly with the laboratory testing
model that incorporates a IP network emulator was an important aspects that minimized the
costs of the investments of the investigative stage were and they allowed to obtain important
theoretical and practical results.
The proposal of use of the FreeProbe (MDI) to validate the Profile of Quality of the IP Network
of ETECSA, it transforms it into an important alternative in front of the expensive acquisition of
the Shadow Router dedicated to this only function.
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 23

25.  Recommendations.
The study of the open source code of the VideoLAN is an important recommendation that
will facilitate the creation of new developments of applications of IPTV.
To use the probe FreeProbe with Metric T-factor and Metric H.264 for the tests of validation
of systems of TDT and for a future service of IPTV about the mobile service 3G.
To extend the employment of the FreeProbe with Metric MDI to the IP networks of ETECSA
for the evaluation of their Profile of Quality and the detection of the network element that
originates the disturbance.
To validate the employment of the FreeProbeJPER for the evaluation of the channels of
ToIP in the NGN network.
The design of the testing model that incorporates a generator of disturbances is an
important tool that will allow the appropriate Objective-Subjective correlation in the FreeProbe
with new metric with capacities of evaluating the quality of services in real time as ToIP.
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 24

26. In line with the world…
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso. 25

27. • Original video VS Original video with 1% Packet Loss Rate.
• Original Video. •Video with 1% Packet Loss Rate.
Source VPDT. ETECSA.
ITU-T Standardization and other key ITU Activities
Eng. Luís Enrique Conde del Oso.