Successfully reported this slideshow.

PSI_aboveFold_parvez_2016_07.pptx

2

Share

Upcoming SlideShare
Speed_Perception_Phase1
Speed_Perception_Phase1
Loading in …3
×
1 of 24
1 of 24

PSI_aboveFold_parvez_2016_07.pptx

2

Share

Download to read offline

Slide deck that explains the rationale behind perceptual speed index, and the collaborative work that happened over last year.

Slide deck that explains the rationale behind perceptual speed index, and the collaborative work that happened over last year.

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

PSI_aboveFold_parvez_2016_07.pptx

  1. 1. Parvez Ahammad Principal Engineer / Head of DS&ML Team Instart Logic Inc. July 2016 Perceptual Speed Index (PSI) for Measuring above-fold visual Quality of Experience of webpages
  2. 2. Acknowledgments • WPO-Foundation/visualmetrics · GitHub • Pat Meenan (Google & WebPageTest) • DS&ML team @ Instart Logic – Clark Gao – Matthew Mok • Karan Kumar (Instart Logic)
  3. 3. Why we (@ InstartLogic) care about metrics for end-user QoE ● Towards a data-driven ‘Control System’ for improving web performance ● Accurate measurement of end-user experience is key!
  4. 4. Page Load Idle Response to Input Animation & Scrolling Reference: How Users Perceive the Speed of The Web (2015): Paul Irish / Google Four phases of interaction: end-user’s perception
  5. 5. The space of end-to-end perf. metrics • PA’s taxonomy of web performance measurements A. Milestone timings (of various kinds) B. Byte size counts (relative to various milestones) C. Request counts D. Metrics on above-fold visual QoE E. ...
  6. 6. Quick Refresher on Speed Index ● Popular public metric on above-fold visual QoE ● Patrick Meenan (WPT / Google Inc.) ● Aggregate function on quickness of visual completion above-fold ● Goes beyond milestone timer based approaches
  7. 7. Measurement of visual progress in Speed Index ● Frame-by-frame VC progress is computed from pixel-histogram comparisons ● Pixel-wise similarity (mean histogram difference / MHD) doesn’t capture visual perception ! ● Perception of Shape / Color / Object similarity
  8. 8. Pixel-wise similarity doesn’t capture shape similarity Black / White :: 50 / 50 MHD (Mean Histogram Difference) = 0
  9. 9. Pixel-wise similarity doesn’t capture color similarity WWW Edward Adelson / MIT
  10. 10. Pixel-wise similarity doesn’t capture object similarity 1 2 3 4 5 6 7 Antonio Torralba / MIT
  11. 11. Proposal for a perceptually oriented visual QoE metric ● Update: Frame-by-frame VC progress computation using SSIM Original, MHD = 0; SSIM = 1 MHD = 144, SSIM = 0.988 MHD = 144, SSIM = 0.913 MHD = 144, SSIM = 0.840 MHD = 144, SSIM = 0.694 MHD = 142, SSIM = 0.662 Wang / Bovik / Simoncelli - 2004
  12. 12. pSI (perceptual Speed Index) ● Frame-by-frame VC progress computation using SSIM perceptual Via SSIM
  13. 13. Experiment #1 ● Key question: Does the MHD  SSIM change capture “real” website behavior? ● Experiments with IR-100 websites (April 2015) ● Presented to Pat Meenan in May 2015 ● Let’s look at a few illustrative examples
  14. 14. Tracking Perceptual Change: SI vs. pSI – RalphLauren (2015) Green: % visual completion via SSIM (PSI) Red: % visual completion via MHD (SI)
  15. 15. Tracking Perceptual Change: SI vs. pSI – ChicosFas (2015) Green: % visual completion via SSIM (PSI) Red: % visual completion via MHD (SI)
  16. 16. Tracking Perceptual Change: SI vs. pSI – Etsy (2015) Green: % visual completion via SSIM (PSI) Red: % visual completion via MHD (SI)
  17. 17. Tracking Perceptual Change: SI vs. pSI – Express (2015) Green: % visual completion via SSIM (PSI) Red: % visual completion via MHD (SI)
  18. 18. Recap: Experiment #1 ● Does the “MHD  SSIM change” capture “real” website behavior? ● Experiments with IR-100 websites (April 2015) ● Presented to Pat Meenan in May 2015 ● Preliminary conclusions ● SSIM based visual progress measurements match human perception more closely than MHD ● Key observation: PSI appears higher when visual jitter exists ● Pop-up ads / large lay-out changes / etc. ● SSIM / MHD swap doesn’t affect websites without visual jitter ● ChicosFas / Etsy examples
  19. 19. Experiment #2 ● How do SI/PSI compare on a larger sample of Alexa top-K websites? ● Experiments with Alexa top mobile websites (March/April 2016) ● Dataset of 500+ mobile above-fold videos ● Dataset provided by Pat Meenan ● All analysis complete (May 2016) ● Results sent to Pat ● What did we find out?
  20. 20. #1: SI and PSI are highly (linearly) correlated (ρ = 0.91)
  21. 21. #2: Median PSI (5020) is slightly higher than median SI (4857)
  22. 22. Experiment #2 ● How do SI/PSI compare on a larger sample of Alexa top-K websites? ● Experiments with Alexa top mobile websites (March/April 2016) ● Dataset of 500+ mobile above-fold videos ● Dataset provided by Pat Meenan ● All analysis complete (May 2016) ● Results sent to Pat ● What did we find out? ● SI and PSI are linearly correlated ● Key observation: PSI is much higher than SI for webpages with visual jitter / layout thrashing behavior
  23. 23. Summary (1) ● Metrics on above-fold visual QoE of web application delivery must take human end-user perception into account. ● Speed Index ● Primarily focused on progress of above-fold loading ● Does not account for layout stability ● Perceptual Speed Index ● A perceptually oriented metric to measure above-fold visual QoE ● Designed to account for visual jitter (layout stability) ● Complementary to SI ● PSI implementation complete ● Open sourced  GitHub: WPO-Foundation/visualmetrics
  24. 24. Summary (2) ● For more details and explanations: ● http://www.parvez-ahammad.org/blog/perceptual-speed-index- psi-for-measuring-above-fold-visual-performance-of-webpages

Editor's Notes

  • Add SI/pSI values on top
  • ×