Institut Mines-Télécom
Subjective quality and HTTP adaptive
streaming: a review of psychophysical studies
Francesca De Sim...
Institut Mines-Télécom
Content
Basic concepts
• Quality of Service (QoS) vs Quality of Experience (QoE)
• Subjective vs ob...
Institut Mines-Télécom
Basic Concepts
3
Institut Mines-Télécom
QoS vs QoE
Example of Quality of Service (QoS) metrics
• Available bandwidth
• Round Trip Time (RTT...
Institut Mines-Télécom
QoS vs QoE
Example of Quality of Service (QoS) metrics
• Available bandwidth
• Round Trip Time (RTT...
Institut Mines-Télécom
Subjective vs objective assessment
Subjective quality assessment
In controlled environment
• standa...
Institut Mines-Télécom
Subjective vs objective assessment
Objective quality assessment
• Full-Reference (FR)
• Reduced-Ref...
Institut Mines-Télécom
QoE and DASH
HTTP server TCP transport layer DASH client
128 kbps
500kbps
1Mbps
• Terminology
0 2 4...
Institut Mines-Télécom
QoE and DASH
No packet losses
• Reliable transport layer
Stalling events
• Playout interruptions
du...
Institut Mines-Télécom
QoE and HAS
No packet losses
• Reliable transport layer
Stalling events
• Playout interruptions
due...
Institut Mines-Télécom
Goal of this talk
11
Identify key questions
concerning subjective video
quality perception and DASH...
Institut Mines-Télécom
Subjective QoE and DASH:
what do we know
12
Institut Mines-Télécom
Perception of stalling events
13
Stallings should be avoided at all times
[Dobrian2011, Mok2011, Fl...
Institut Mines-Télécom
Perception of stalling events
14
Stallings should be avoided at all times
[Dobrian2011, Mok2011, Fl...
Institut Mines-Télécom
Perception of stalling events
Conclusions
• Threshold to detect stalling: 80ms [Pastrana-Vidal2004]...
Institut Mines-Télécom
Perception of quality switches
16
Assumptions
• Each representation corresponds to a
different bitr...
Institut Mines-Télécom
Perception of quality switches
17
“What is the impact of switching amplitude and
frequency?”
Institut Mines-Télécom
Perception of quality switches
18
“What is the impact of switching amplitude and
frequency?”
Questi...
Institut Mines-Télécom
Perception of quality switches
19
– Studies on layer encoded videos and SVC [Zink 2003, Ni 2011]
– ...
Institut Mines-Télécom
Perception of quality switches
20
Conclusions
• Smooth variations preferred to abrupt variations
• ...
Institut Mines-Télécom
Perception of quality switches
21
But…
• Conclusion might not be generalizable to switches between
...
Institut Mines-Télécom
Perception of quality switches
22
Question 2: “Do users prefer shorter but more
frequent variation ...
Institut Mines-Télécom
Perception of quality switches
23
• [Zink 2003, Ni 2011] layer
encoded videos and SVC
• [Moorthy 20...
Institut Mines-Télécom24
Conclusions
• Less frequent quality variations are preferred to more frequent
variations
• Freque...
Institut Mines-Télécom25
Conclusions
• Less frequent quality variations are preferred to more
frequent variations
• Freque...
Institut Mines-Télécom
Perception of quality switches
26
Question 3: “is it better to switch quality levels or to
stay at ...
Institut Mines-Télécom
Perception of quality switches
27
Question 3: “is it better to switch quality levels or to
stay at ...
Institut Mines-Télécom
Perception of quality switches
28
Conclusions
• Constant quality preferred to varying quality
• Sho...
Institut Mines-Télécom
Perception of quality switches
29
But…
• Preferable to switch to a higher rate, if the duration of ...
Institut Mines-Télécom
Perception of quality switches
30
Temporal trend
• The end quality of the video makes a definite im...
Institut Mines-Télécom
Perception of quality switches
When the bitrate has to be lowered (increased), for
instance due to ...
Institut Mines-Télécom
Perception of quality switches
32
When the bitrate has to be lowered (increased), for
instance due ...
Institut Mines-Télécom
Perception of quality switches
33
Conclusions
• Spatial quality more important
than frame rate at l...
Institut Mines-Télécom
Perception of quality switches
34
Methodological findings:
• [RehmanQomex2013] Subjects are resiste...
Institut Mines-Télécom
Tradeoffs: initial delay vs stalling
35
Stallings can be avoided by employing larger
client buffers...
Institut Mines-Télécom
Tradeoffs: initial delay vs stalling
36
Stallings can be avoided by employing larger
client buffers...
Institut Mines-Télécom
Tradeoffs: initial delay vs stalling
Conclusions
• Initial startup delay does not significantly inf...
Institut Mines-Télécom
Conclusions
38
Institut Mines-Télécom
Conclusions
Agreed upon conclusions:
• Frequency of stalling is the main factor affecting QoE
• The...
Institut Mines-Télécom
Challenges
40
Why is it difficult to design a
subjective test for HAS?
• Temporal variation of qual...
Institut Mines-Télécom
Thank You for your
Attention!
francesca.de-simone@telecom-paristech.fr
frederic.dufaux@telecom-pari...
Institut Mines-Télécom
Further reading and resources…
1. Qualinet White Paper on Definitions of Quality of Experience (201...
Upcoming SlideShare
Loading in …5
×

ConférenSquad #3 : Subjective Quality and HTTP Adaptive Streaming: a Review of Psychophysical Studies (Frédéric Dufaux, Telecom Paris Tech)

3,505 views

Published on

Conférence OVFSquad du 15/05/14 :
QoS et QoE : double nécessité ou double peine ?
http://ovfsquad.fr

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
3,505
On SlideShare
0
From Embeds
0
Number of Embeds
1,268
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

ConférenSquad #3 : Subjective Quality and HTTP Adaptive Streaming: a Review of Psychophysical Studies (Frédéric Dufaux, Telecom Paris Tech)

  1. 1. Institut Mines-Télécom Subjective quality and HTTP adaptive streaming: a review of psychophysical studies Francesca De Simone, Frédéric Dufaux Institut Mines-Télécom; Télécom ParisTech; CNRS LTCI
  2. 2. Institut Mines-Télécom Content Basic concepts • Quality of Service (QoS) vs Quality of Experience (QoE) • Subjective vs objective assessment • QoE and DASH Subjective QoE and DASH: what do we know • Perception of stalling events • Perception of quality switches • Tradeoffs Conclusions 2
  3. 3. Institut Mines-Télécom Basic Concepts 3
  4. 4. Institut Mines-Télécom QoS vs QoE Example of Quality of Service (QoS) metrics • Available bandwidth • Round Trip Time (RTT) • Packet loss rate (PLR) • Etc.. Quality of Experience (QoE)? 4
  5. 5. Institut Mines-Télécom QoS vs QoE Example of Quality of Service (QoS) metrics • Available bandwidth • Round Trip Time (RTT) • Packet loss rate (PLR) • Etc.. Quality of Experience (QoE)? [Qualinet WhitePaper 2012] 5 multimedia experiencemultimedia experience MULTIMEDIA SERVICE Visual signal Price Novelty Interface CONTEXT etc… Audio signal etc…USER Human Visual System Expectations Emotions Auditory System Attention etc…
  6. 6. Institut Mines-Télécom Subjective vs objective assessment Subjective quality assessment In controlled environment • standard methodologies for video designed for analog television [ITU Rec BT.500-11] • adapted for multimedia applications [ITU Rec. P.910] In uncontrolled environment • crowdsourcing • field experiments • users’ feedback from real products 6
  7. 7. Institut Mines-Télécom Subjective vs objective assessment Objective quality assessment • Full-Reference (FR) • Reduced-Reference (RR) • No-Reference (NR) 7 Input/Reference signal Output/Processed signalImage/ video processing FR METRIC Input/Reference signal Output/Processed signalImage/ video processing NR METRIC Input/Reference signal Output/Processed signalImage/ video processing Feature extraction RR METRIC Example of standardized solutions • FR video quality metric for HDTV for digital cable television [ITU-T Rec. J.341] • NR model for audiovisual quality assessment in IPTV [ITU-T Rec. P.1201.2]
  8. 8. Institut Mines-Télécom QoE and DASH HTTP server TCP transport layer DASH client 128 kbps 500kbps 1Mbps • Terminology 0 2 4 6 8 10 12 14 time (s) Realization Realization Realization Chunk Adaptation Set 128 kbps 500kbps 1Mbps
  9. 9. Institut Mines-Télécom QoE and DASH No packet losses • Reliable transport layer Stalling events • Playout interruptions due to video player buffer underflow Quality switches • Quality variation due to switch between representations 9
  10. 10. Institut Mines-Télécom QoE and HAS No packet losses • Reliable transport layer Stalling events • Playout interruptions due to video player buffer underflow Quality switches • Quality variation due to switch between representations 10 Not addressed in traditional video QoE research!
  11. 11. Institut Mines-Télécom Goal of this talk 11 Identify key questions concerning subjective video quality perception and DASH Review existing answers • Perception of video stalling • Perception of video quality switches • Tradeoffs
  12. 12. Institut Mines-Télécom Subjective QoE and DASH: what do we know 12
  13. 13. Institut Mines-Télécom Perception of stalling events 13 Stallings should be avoided at all times [Dobrian2011, Mok2011, Floris2012, Atzori2013, Hobfeld2013] but if they occur… “what is the impact of stalling frequency and duration?”
  14. 14. Institut Mines-Télécom Perception of stalling events 14 Stallings should be avoided at all times [Dobrian2011, Mok2011, Floris2012, Atzori2013, Hobfeld2013] but if they occur… “what is the impact of stalling frequency and duration?” • Studies before DASH [Pastrana-Vidal2004, Staelens2010] • Studies on DASH [Moorthy2011, vanKester2011]
  15. 15. Institut Mines-Télécom Perception of stalling events Conclusions • Threshold to detect stalling: 80ms [Pastrana-Vidal2004] • Subjects prefer a single stalling of longer duration compared to multiple short stallings [Moorthy2011] • Subjects prefer regular stallings over irregular ones [Moorthy2011] • Stallings up to 400ms can still be tolerated in the case of few stalling events and long video sequences [vanKester2011] [Staelens2010] 15
  16. 16. Institut Mines-Télécom Perception of quality switches 16 Assumptions • Each representation corresponds to a different bitrate • Representations may differ in terms of spatial resolution, temporal resolution and encoding quantization settings • Three kinds of switches are possible • Encoding switch (ESW) • Spatial switch (SSW) • Temporal switch (TSW) 128 kbps 500kbps 1Mbps
  17. 17. Institut Mines-Télécom Perception of quality switches 17 “What is the impact of switching amplitude and frequency?”
  18. 18. Institut Mines-Télécom Perception of quality switches 18 “What is the impact of switching amplitude and frequency?” Question 1: “shall the bitrate be reduced (increased) once with a big quality drop or gradually?” time Layer or Rate time Layer or Rate
  19. 19. Institut Mines-Télécom Perception of quality switches 19 – Studies on layer encoded videos and SVC [Zink 2003, Ni 2011] – Studies on HAS [Moorthy 2012, Mok 2012, Yitong 2013] “What is the impact of switching amplitude and frequency?” Question 1: “shall the bitrate be reduced (increased) once with a big quality drop or gradually?” time Layer or Rate time Layer or Rate
  20. 20. Institut Mines-Télécom Perception of quality switches 20 Conclusions • Smooth variations preferred to abrupt variations • Amplitude of the variation must be kept as small as possible • Examples of amplitude thresholds to deliver generally acceptable quality [Ni 2011] • Quality differences should not exceed 4 QPs for ESW, one third of the original frame rate for TSW, half the original size for SSW
  21. 21. Institut Mines-Télécom Perception of quality switches 21 But… • Conclusion might not be generalizable to switches between levels exhibiting small quality separation [Moorthy 2012] Conclusions • Smooth variations preferred to abrupt variations • Amplitude of the variation must be kept as small as possible • Examples of amplitude thresholds to deliver generally acceptable quality [Ni 2011] • Quality differences should not exceed 4 QPs for ESW, one third of the original frame rate for TSW, half the original size for SSW
  22. 22. Institut Mines-Télécom Perception of quality switches 22 Question 2: “Do users prefer shorter but more frequent variation or longer but less frequent variations?”
  23. 23. Institut Mines-Télécom Perception of quality switches 23 • [Zink 2003, Ni 2011] layer encoded videos and SVC • [Moorthy 2012, Robinson 2012, Yitong 2013] Question 2: “Do users prefer shorter but more frequent variation or longer but less frequent variations?” time Layer or Rate time Layer or Rate
  24. 24. Institut Mines-Télécom24 Conclusions • Less frequent quality variations are preferred to more frequent variations • Frequency of variations should be kept as small as possible • For SNR or spatial resolution switches, low frequency can relieve the annoyance of quality switch [Ni 2011] Perception of quality switches
  25. 25. Institut Mines-Télécom25 Conclusions • Less frequent quality variations are preferred to more frequent variations • Frequency of variations should be kept as small as possible • For SNR or spatial resolution switches, low frequency can relieve the annoyance of quality switch [Ni 2011] But… • More frequent switches are preferred over fewer switches, if the subject is able to view the highest quality video for longer duration [Moorthy 2012] • For temporal resolution switches, the frequency does not seem to have significant influence on QoE [Ni 2011] Perception of quality switches
  26. 26. Institut Mines-Télécom Perception of quality switches 26 Question 3: “is it better to switch quality levels or to stay at a lower quality? i.e. to switch or not to switch?”
  27. 27. Institut Mines-Télécom Perception of quality switches 27 Question 3: “is it better to switch quality levels or to stay at a lower quality? i.e. to switch or not to switch?” • [Zink 2003, Ni 2011] layer encoded videos and SVC • [Moorthy 2012, Robinson 2012, Mok 2012, Yitong 2013, Villa 2013, Rehman 2013, Tavakoli 2014] time Layer or Rate time Layer or Rate
  28. 28. Institut Mines-Télécom Perception of quality switches 28 Conclusions • Constant quality preferred to varying quality • Short-term spikes degrade QoE [Yitong 2013] • Constant (lower) quality better than decreasing quality (from higher to lower) [Zink 2003] • Constant or nearly constant quality is preferable to frequently varying quality (even if mean quality is lower) [Robinson 2012] [Ni 2011] • Providing as high bitrate as possible does not necessarily lead to the highest QoE [Mok 2012]
  29. 29. Institut Mines-Télécom Perception of quality switches 29 But… • Preferable to switch to a higher rate, if the duration of the higher rate is at least half the duration of the lower rates [Moorthy 2012] • If constant quality is too low, any adaptation is preferred [Tavakoli 2014] [Ni 2011] • Maintaining a reasonable quality for longer duration results in a small bias towards better subjective experience [Rehman 2013]
  30. 30. Institut Mines-Télécom Perception of quality switches 30 Temporal trend • The end quality of the video makes a definite impact on the perceived quality: all is well that ends well [Zink 2003][Moorthy 2012] • [TavakoliSPIE2014] For increasing quality, overall results did not show any statistically significantly difference between different strategies. For the decreasing scenarios, there was a statistically significantly preference to gradual change with 10 seconds chunk compared to the other scenarios. Content dependency • [NiMMsys2011] [RobinsonBellTJ2012] Effect of switch of spatial and temporal resolution varies on different content types, even for the same amplitude • difficult to spot quality oscillations when frequent scene changes in the scene. • Quality change is more noticeable in steady, pan, tracking shots and when there are number of hard edges. • More complex texture details , more strongly affected by the loss of spatial resolution • The lack of relative movement between objects in the scene may limit the visible effect of frame dropping.
  31. 31. Institut Mines-Télécom Perception of quality switches When the bitrate has to be lowered (increased), for instance due to restricted (better) network conditions, … Question 1: “is it better to reduce (increase) encoded bitrate, frame rate, or resolution?” 31
  32. 32. Institut Mines-Télécom Perception of quality switches 32 When the bitrate has to be lowered (increased), for instance due to restricted bandwidth (better network conditions), … Question 1: “is it better to reduce (increase) the encoded bitrate, the video frame rate, or the video resolution?” H.264/SVC encoded versions at same bitrate 1/16 of the original resolution upscaledStronger quantization • Many subjective studies on optimal combinations (SNR, spatial resolution, temporal resolution) for scalable video coding [Lee 2012]
  33. 33. Institut Mines-Télécom Perception of quality switches 33 Conclusions • Spatial quality more important than frame rate at low bit rates and viceversa at high bit rates • Low vs high bit rate threshold higher for content with faster motion • Among quality switch of SNR levels, spatial resolutions or temporal resolutions, SNR switch is the most recommended as less noticeable [Ni 2011]
  34. 34. Institut Mines-Télécom Perception of quality switches 34 Methodological findings: • [RehmanQomex2013] Subjects are resistent in updating their opinions: when there is a small quality variation between consecutive scenes, subjects tend to keep their opinions or change only slightly. • [RehmanQomex2013] [ChenIP2014] Subjects use asymmetric strategies in updating their opinions. A significant quality degradation between consecutive scenes results in a large penalty, as compared to the reward obtained by a similar quality improvement between consecutive scenes. Behavioral response of viewers to quality variation is more sensitive in low quality region than in high quality region. • [ChenIP2014] The QoE of a video at a particular moment depends on the viewing experience before the moment: the current video quality can affect the perceived quality in the next 15 second • [ChenIP2014] A viewer suffering a previous unpleasant viewing experience tends to penalize the perceived quality in the future • [VillaNIK2013] Significant differences in terms of self-reported pleasure, arousal, and the degree of delight between the two usage scenarios (user alone vs group of users). All significantly higher in the first usage scenario.
  35. 35. Institut Mines-Télécom Tradeoffs: initial delay vs stalling 35 Stallings can be avoided by employing larger client buffers but this will affect the initial startup delay of the video, so… Question 2: “what is the impact of initial loading?” Question 3: “what is the impact of combined effect (stalling + initial loading)?”
  36. 36. Institut Mines-Télécom Tradeoffs: initial delay vs stalling 36 Stallings can be avoided by employing larger client buffers but this will affect the initial startup delay of the video, so… Question 2: “what is the impact of initial loading?” Question 3: “what is the impact of combined effect (stalling + initial loading)?” [Hobfeld2013, HoBfeldQomex2012, Staelens2010]
  37. 37. Institut Mines-Télécom Tradeoffs: initial delay vs stalling Conclusions • Initial startup delay does not significantly influence quality perception [Hoßfeld 2013] • End-users are willing to tolerate larger startup delays, if this results in less video stallings [Staelens 2010] • Different amounts of startup delay are tolerated, depending on the specific type of application [HoBfeld 2012] 37
  38. 38. Institut Mines-Télécom Conclusions 38
  39. 39. Institut Mines-Télécom Conclusions Agreed upon conclusions: • Frequency of stalling is the main factor affecting QoE • The amplitude of the quality switch is the most dominant factor for the perception of quality switch (more than the frequency) Issue: • How generalizable are the findings of these studies? 39
  40. 40. Institut Mines-Télécom Challenges 40 Why is it difficult to design a subjective test for HAS? • Temporal variation of quality ─ Longer sequences may be needed: which test method to use? • Variety of terminals in the real scenario ─ Different network conditions ─ Different users’ expectations
  41. 41. Institut Mines-Télécom Thank You for your Attention! francesca.de-simone@telecom-paristech.fr frederic.dufaux@telecom-paristech.fr Thanks to: Marie-Neige Garcia and Alexander Raake (T-Labs TU Berlin), Nicolas Staelens (Ghent University), Samira Tavakoli (Universidad Politecnica de Madrid), Sebastian Egger (ftw), Kjell Brunnstrom (Acreo) 41
  42. 42. Institut Mines-Télécom Further reading and resources… 1. Qualinet White Paper on Definitions of Quality of Experience (2012), European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003), Patrick Le Callet, Sebastian Möller and Andrew Perkis, eds., Lausanne, Switzerland, Version 1.1, June 3, 2012. 2. Zink et al., Subjective impression of variations in layer encoded videos, ACM 2003 3. R.R. Pastrana-Vidal, J.C. Gicquel, C. Colomes, and H. Cherifi, “Sporadic frame dropping impact on quality perception,” in Proceedings of SPIE, 2004, vol. 5292, p. 182 4. Q. Huynh-Thu and M. Ghanbari, “Temporal Aspect of Perceived Quality in Mobile Video Broadcasting”, IEEE Transactions on Broadcasting, vol.54, no.3, pp.641-651, Sept. 2008 5. J.-S. Lee, F. De Simone, N. Ramzan, E. Izquierdo and T. Ebrahimi, Quality assessment of multidimensional video scalability, in IEEE Communications Magazine, vol. 50, num. 4, p. 38-46, 2012 6. J.-S. Lee, F. De Simone and T. Ebrahimi, Subjective quality evaluation via paired comparison: application to scalable video coding, in IEEE Transactions on Multimedia, vol. 13, num. 5, p. 882-893, 2011 7. Venkataraman et al., Effects of Internet path selection on video QoE, MMSys 2011 8. Chen et al., Quantifying QoS requirements of network services: a cheat-proof framework, MMSys 2011 9. R. Mok, E. Chan, and R. Chang. Measuring the quality of experience of HTTP video streaming. In Proc. IEEE/IFIP 10. IM (pre-conf.), 2011 11. R. Mok, E. Chan, X. Luo, and R. Chang., Inferring the QoE of HTTP Video Streaming from User-Viewing Activities, ACM W- MUST’11, 2011 12. Zhou Wang; Bovik, A.C., "Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures," Signal Processing Magazine, IEEE , vol.26, no.1, pp.98-117, Jan. 2009 13. Kusching at el., An evaluation of TCP-based rate-control algorithms for adaptive internet streaming of H.264/SVC, MMSys 2010 14. Lederer at el., Dynamic adaptive streaming over HTTP dataset, MMSys 2012 15. K. Seshadrinathan and A. C. Bovik, Motion Tuned Spatio-temporal Quality Assessment of Natural Videos, vol. 19, no. 2, pp. 335-350, IEEE Transactions on Image Processing, Feb. 2010 16. NTIA VQM http://www.its.bldrdoc.gov/resources/video-quality-research/software.aspx 17. Yim et al., Evaluation of temporal variation of video quality in packet loss network, Signal Processing Image Communication, 2011 18. Singh, K.D.; Hadjadj-Aoul, Y.; Rubino, G.; , "Quality of experience estimation for adaptive HTTP/TCP video streaming using H.264/AVC," Consumer Communications and Networking Conference (CCNC), 2012 IEEE , vol., no., pp.127-131, 14-17 Jan. 2012 42

×