Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Parvez Ahammad
Principal Engineer / Head of DS&ML Team
Instart Logic Inc.
July 2016
Perceptual Speed Index (PSI)
for Measu...
Acknowledgments
• WPO-Foundation/visualmetrics · GitHub
• Pat Meenan (Google & WebPageTest)
• DS&ML team @ Instart Logic
–...
Why we (@ InstartLogic) care about metrics for end-user QoE
● Towards a data-driven ‘Control System’ for improving web per...
Page
Load
Idle
Response
to Input
Animation
&
Scrolling
Reference: How Users Perceive the Speed of The Web (2015): Paul Iri...
The space of end-to-end perf. metrics
• PA’s taxonomy of web performance measurements
A. Milestone timings (of various kin...
Quick Refresher on Speed Index
● Popular public metric on above-fold visual QoE
● Patrick Meenan (WPT / Google Inc.)
● Agg...
Measurement of visual progress in Speed Index
● Frame-by-frame VC progress is computed from pixel-histogram comparisons
● ...
Pixel-wise similarity doesn’t capture shape similarity
Black / White :: 50 / 50
MHD (Mean Histogram Difference) = 0
Pixel-wise similarity doesn’t capture color similarity
WWW
Edward Adelson / MIT
Pixel-wise similarity doesn’t capture object similarity
1 2
3
4
5
6
7
Antonio Torralba / MIT
Proposal for a perceptually oriented visual QoE metric
● Update: Frame-by-frame VC progress computation using SSIM
Origina...
pSI (perceptual Speed Index)
● Frame-by-frame VC progress computation using SSIM
perceptual Via SSIM
Experiment #1
● Key question: Does the MHD  SSIM change capture “real” website
behavior?
● Experiments with IR-100 websit...
Tracking Perceptual Change: SI vs. pSI – RalphLauren (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual comple...
Tracking Perceptual Change: SI vs. pSI – ChicosFas (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completi...
Tracking Perceptual Change: SI vs. pSI – Etsy (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion vi...
Tracking Perceptual Change: SI vs. pSI – Express (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion...
Recap: Experiment #1
● Does the “MHD  SSIM change” capture “real” website behavior?
● Experiments with IR-100 websites (A...
Experiment #2
● How do SI/PSI compare on a larger sample of Alexa top-K websites?
● Experiments with Alexa top mobile webs...
#1: SI and PSI are highly (linearly) correlated (ρ = 0.91)
#2: Median PSI (5020) is slightly higher than median SI (4857)
Experiment #2
● How do SI/PSI compare on a larger sample of Alexa top-K websites?
● Experiments with Alexa top mobile webs...
Summary (1)
● Metrics on above-fold visual QoE of web application delivery must
take human end-user perception into accoun...
Summary (2)
● For more details and explanations:
● http://www.parvez-ahammad.org/blog/perceptual-speed-index-
psi-for-meas...
Upcoming SlideShare
Loading in …5
×

PSI_aboveFold_parvez_2016_07.pptx

1,163 views

Published on

Slide deck that explains the rationale behind perceptual speed index, and the collaborative work that happened over last year.

Published in: Internet
  • Be the first to comment

PSI_aboveFold_parvez_2016_07.pptx

  1. 1. Parvez Ahammad Principal Engineer / Head of DS&ML Team Instart Logic Inc. July 2016 Perceptual Speed Index (PSI) for Measuring above-fold visual Quality of Experience of webpages
  2. 2. Acknowledgments • WPO-Foundation/visualmetrics · GitHub • Pat Meenan (Google & WebPageTest) • DS&ML team @ Instart Logic – Clark Gao – Matthew Mok • Karan Kumar (Instart Logic)
  3. 3. Why we (@ InstartLogic) care about metrics for end-user QoE ● Towards a data-driven ‘Control System’ for improving web performance ● Accurate measurement of end-user experience is key!
  4. 4. Page Load Idle Response to Input Animation & Scrolling Reference: How Users Perceive the Speed of The Web (2015): Paul Irish / Google Four phases of interaction: end-user’s perception
  5. 5. The space of end-to-end perf. metrics • PA’s taxonomy of web performance measurements A. Milestone timings (of various kinds) B. Byte size counts (relative to various milestones) C. Request counts D. Metrics on above-fold visual QoE E. ...
  6. 6. Quick Refresher on Speed Index ● Popular public metric on above-fold visual QoE ● Patrick Meenan (WPT / Google Inc.) ● Aggregate function on quickness of visual completion above-fold ● Goes beyond milestone timer based approaches
  7. 7. Measurement of visual progress in Speed Index ● Frame-by-frame VC progress is computed from pixel-histogram comparisons ● Pixel-wise similarity (mean histogram difference / MHD) doesn’t capture visual perception ! ● Perception of Shape / Color / Object similarity
  8. 8. Pixel-wise similarity doesn’t capture shape similarity Black / White :: 50 / 50 MHD (Mean Histogram Difference) = 0
  9. 9. Pixel-wise similarity doesn’t capture color similarity WWW Edward Adelson / MIT
  10. 10. Pixel-wise similarity doesn’t capture object similarity 1 2 3 4 5 6 7 Antonio Torralba / MIT
  11. 11. Proposal for a perceptually oriented visual QoE metric ● Update: Frame-by-frame VC progress computation using SSIM Original, MHD = 0; SSIM = 1 MHD = 144, SSIM = 0.988 MHD = 144, SSIM = 0.913 MHD = 144, SSIM = 0.840 MHD = 144, SSIM = 0.694 MHD = 142, SSIM = 0.662 Wang / Bovik / Simoncelli - 2004
  12. 12. pSI (perceptual Speed Index) ● Frame-by-frame VC progress computation using SSIM perceptual Via SSIM
  13. 13. Experiment #1 ● Key question: Does the MHD  SSIM change capture “real” website behavior? ● Experiments with IR-100 websites (April 2015) ● Presented to Pat Meenan in May 2015 ● Let’s look at a few illustrative examples
  14. 14. Tracking Perceptual Change: SI vs. pSI – RalphLauren (2015) Green: % visual completion via SSIM (PSI) Red: % visual completion via MHD (SI)
  15. 15. Tracking Perceptual Change: SI vs. pSI – ChicosFas (2015) Green: % visual completion via SSIM (PSI) Red: % visual completion via MHD (SI)
  16. 16. Tracking Perceptual Change: SI vs. pSI – Etsy (2015) Green: % visual completion via SSIM (PSI) Red: % visual completion via MHD (SI)
  17. 17. Tracking Perceptual Change: SI vs. pSI – Express (2015) Green: % visual completion via SSIM (PSI) Red: % visual completion via MHD (SI)
  18. 18. Recap: Experiment #1 ● Does the “MHD  SSIM change” capture “real” website behavior? ● Experiments with IR-100 websites (April 2015) ● Presented to Pat Meenan in May 2015 ● Preliminary conclusions ● SSIM based visual progress measurements match human perception more closely than MHD ● Key observation: PSI appears higher when visual jitter exists ● Pop-up ads / large lay-out changes / etc. ● SSIM / MHD swap doesn’t affect websites without visual jitter ● ChicosFas / Etsy examples
  19. 19. Experiment #2 ● How do SI/PSI compare on a larger sample of Alexa top-K websites? ● Experiments with Alexa top mobile websites (March/April 2016) ● Dataset of 500+ mobile above-fold videos ● Dataset provided by Pat Meenan ● All analysis complete (May 2016) ● Results sent to Pat ● What did we find out?
  20. 20. #1: SI and PSI are highly (linearly) correlated (ρ = 0.91)
  21. 21. #2: Median PSI (5020) is slightly higher than median SI (4857)
  22. 22. Experiment #2 ● How do SI/PSI compare on a larger sample of Alexa top-K websites? ● Experiments with Alexa top mobile websites (March/April 2016) ● Dataset of 500+ mobile above-fold videos ● Dataset provided by Pat Meenan ● All analysis complete (May 2016) ● Results sent to Pat ● What did we find out? ● SI and PSI are linearly correlated ● Key observation: PSI is much higher than SI for webpages with visual jitter / layout thrashing behavior
  23. 23. Summary (1) ● Metrics on above-fold visual QoE of web application delivery must take human end-user perception into account. ● Speed Index ● Primarily focused on progress of above-fold loading ● Does not account for layout stability ● Perceptual Speed Index ● A perceptually oriented metric to measure above-fold visual QoE ● Designed to account for visual jitter (layout stability) ● Complementary to SI ● PSI implementation complete ● Open sourced  GitHub: WPO-Foundation/visualmetrics
  24. 24. Summary (2) ● For more details and explanations: ● http://www.parvez-ahammad.org/blog/perceptual-speed-index- psi-for-measuring-above-fold-visual-performance-of-webpages

×