SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
1.
Parvez Ahammad
Principal Engineer / Head of DS&ML Team
Instart Logic Inc.
July 2016
Perceptual Speed Index (PSI)
for Measuring above-fold visual
Quality of Experience of webpages
2.
Acknowledgments
• WPO-Foundation/visualmetrics · GitHub
• Pat Meenan (Google & WebPageTest)
• DS&ML team @ Instart Logic
– Clark Gao
– Matthew Mok
• Karan Kumar (Instart Logic)
3.
Why we (@ InstartLogic) care about metrics for end-user QoE
● Towards a data-driven ‘Control System’ for improving web performance
● Accurate measurement of end-user experience is key!
4.
Page
Load
Idle
Response
to Input
Animation
&
Scrolling
Reference: How Users Perceive the Speed of The Web (2015): Paul Irish / Google
Four phases of interaction: end-user’s perception
5.
The space of end-to-end perf. metrics
• PA’s taxonomy of web performance measurements
A. Milestone timings (of various kinds)
B. Byte size counts (relative to various milestones)
C. Request counts
D. Metrics on above-fold visual QoE
E. ...
6.
Quick Refresher on Speed Index
● Popular public metric on above-fold visual QoE
● Patrick Meenan (WPT / Google Inc.)
● Aggregate function on quickness of visual completion above-fold
● Goes beyond milestone timer based approaches
7.
Measurement of visual progress in Speed Index
● Frame-by-frame VC progress is computed from pixel-histogram comparisons
● Pixel-wise similarity (mean histogram difference / MHD) doesn’t capture visual
perception !
● Perception of Shape / Color / Object similarity
8.
Pixel-wise similarity doesn’t capture shape similarity
Black / White :: 50 / 50
MHD (Mean Histogram Difference) = 0
9.
Pixel-wise similarity doesn’t capture color similarity
WWW
Edward Adelson / MIT
10.
Pixel-wise similarity doesn’t capture object similarity
1 2
3
4
5
6
7
Antonio Torralba / MIT
12.
pSI (perceptual Speed Index)
● Frame-by-frame VC progress computation using SSIM
perceptual Via SSIM
13.
Experiment #1
● Key question: Does the MHD SSIM change capture “real” website
behavior?
● Experiments with IR-100 websites (April 2015)
● Presented to Pat Meenan in May 2015
● Let’s look at a few illustrative examples
14.
Tracking Perceptual Change: SI vs. pSI – RalphLauren (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion via MHD (SI)
15.
Tracking Perceptual Change: SI vs. pSI – ChicosFas (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion via MHD (SI)
16.
Tracking Perceptual Change: SI vs. pSI – Etsy (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion via MHD (SI)
17.
Tracking Perceptual Change: SI vs. pSI – Express (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion via MHD (SI)
18.
Recap: Experiment #1
● Does the “MHD SSIM change” capture “real” website behavior?
● Experiments with IR-100 websites (April 2015)
● Presented to Pat Meenan in May 2015
● Preliminary conclusions
● SSIM based visual progress measurements match human
perception more closely than MHD
● Key observation: PSI appears higher when visual jitter exists
● Pop-up ads / large lay-out changes / etc.
● SSIM / MHD swap doesn’t affect websites without visual jitter
● ChicosFas / Etsy examples
19.
Experiment #2
● How do SI/PSI compare on a larger sample of Alexa top-K websites?
● Experiments with Alexa top mobile websites (March/April 2016)
● Dataset of 500+ mobile above-fold videos
● Dataset provided by Pat Meenan
● All analysis complete (May 2016)
● Results sent to Pat
● What did we find out?
20.
#1: SI and PSI are highly (linearly) correlated (ρ = 0.91)
21.
#2: Median PSI (5020) is slightly higher than median SI (4857)
22.
Experiment #2
● How do SI/PSI compare on a larger sample of Alexa top-K websites?
● Experiments with Alexa top mobile websites (March/April 2016)
● Dataset of 500+ mobile above-fold videos
● Dataset provided by Pat Meenan
● All analysis complete (May 2016)
● Results sent to Pat
● What did we find out?
● SI and PSI are linearly correlated
● Key observation: PSI is much higher than SI for webpages with
visual jitter / layout thrashing behavior
23.
Summary (1)
● Metrics on above-fold visual QoE of web application delivery must
take human end-user perception into account.
● Speed Index
● Primarily focused on progress of above-fold loading
● Does not account for layout stability
● Perceptual Speed Index
● A perceptually oriented metric to measure above-fold visual QoE
● Designed to account for visual jitter (layout stability)
● Complementary to SI
● PSI implementation complete
● Open sourced GitHub: WPO-Foundation/visualmetrics
24.
Summary (2)
● For more details and explanations:
● http://www.parvez-ahammad.org/blog/perceptual-speed-index-
psi-for-measuring-above-fold-visual-performance-of-webpages