Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
Stephen Thair, profile picture
Uploaded byStephen Thair
PPTX, PDF2,699 views

Measuring web performance. Velocity EU 2011

The document discusses various methods and metrics for measuring web performance, highlighting both subjective and objective approaches. Key raw metrics include Time to First Byte, render start time, and DOMContentLoaded, while it emphasizes the importance of context and the role of different tools like browser extensions and proxy timing. It also contrasts synthetic and real-user monitoring, advocating for a hybrid approach to gain comprehensive insights into website performance.

Technology◦
Downloaded 118 times
MEASURING WEB PERFORMANCE Steve Thair Seriti Consulting @TheOpsMgr
Every measurement of web performance you will ever make will be wrong (C) SERITI CONSULTING, 2011 08/11/2011 2
(C) SERITI CONSULTING, 2011 08/11/2011 3
“The human perception of duration is both subjective and variable” http://en.wikipedia.org/wiki/Time_perception
“PERCEPTION IS VARIABLE…” Go read Stoyan’s talk! http://velocityconf.com/velocity2010/public/schedule/detail/13019 (C) SERITI CONSULTING, 2011 08/11/2011 5
Web Performance Subjective Objective (C) SERITI CONSULTING, 2011 08/11/2011 6
Case Studies Subjective Focus Groups “Qualitative Interviews techniques” Video Analysis Surveys (C) SERITI CONSULTING, 2011 08/11/2011 7
Javascript Navigation timing Objective Browser Extensions “Quantitative Custom Browsers techniques” Proxy timings Web Server mods Network sniffing (C) SERITI CONSULTING, 2011 08/11/2011 8
“I keep six honest serving-men (They taught me all I knew); Their names are What and Why and When And How and Where and Who.” Rudyard Kipling, The Elephant’s Tale (C) SERITI CONSULTING, 2011 08/11/2011 9
WHAT LEVEL DO YOU MEASURE? Journey Page Object
CHOOSE YOUR METRIC! https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html (C) SERITI CONSULTING, 2011 08/11/2011 11
4 Key “Raw” Metrics • Time to First Byte (TTFB) • Render Start Time • DOMContentLoaded • Page (onLoad) Load Time (PLT) (C) SERITI CONSULTING, 2011 08/11/2011 12
What about “Above the Fold” time? • How long to “render of the static stuff in the viewable area of the page”? Limitations of AFT – Only applicable to lab setting – Does not reflect user perceived latency based on functionality http://assets.en.oreilly.com/1/event/62/Above%20the%20Fold%20Time_%20Measuring%20Web%20Page%20Performance %20Visually%20Presentation.pdf (C) SERITI CONSULTING, 2011 08/11/2011 13
(C) SERITI CONSULTING, 2011 08/11/2011 14
WHAT OTHER METRICS? Apdex Statistical Metrics Counts/Histograms Raw Metrics
Apdex (t) = (Satisfied Count + Tolerated Count / 2) / Total Samples • A number between 0 and 1 that represents “user satisfaction” • For technical reasons the “Tolerated” threshold is set to four times the “Satisfied” Threshold so if your “Satisfied” threshold (t) was 4 seconds then: • 0 to 4 seconds = Satisfied 4 to 16 seconds = Tolerated over 16 seconds = Frustrated. http://apdex.org/ (C) SERITI CONSULTING, 2011 08/11/2011 16
PERFORMANCE IS MULTI-DIMENSIONAL Multiple Metrics For Multiple URLS From Different Locations Using Different Tools Across the Lifecycle Over Time (C) SERITI CONSULTING, 2011 08/11/2011 17
The importance of CONTEXT (C) SERITI CONSULTING, 2011 08/11/2011 18
Location Bandwidth Wired, WiFi, 3G Latency Operating Cached objects System Addons & Antivirus Extensions Browser Device Time of Day Context Resolution (C) SERITI CONSULTING, 2011 08/11/2011 19
(C) SERITI CONSULTING, 2011 08/11/2011 20
Who? When? User Experience Design (UX) Developers Prod Develop Testers Ops SDLC WebOps “The Boss” Build QA (CI) (C) SERITI CONSULTING, 2011 08/11/2011 21
WHERE – DEPENDS ON THE HOW & WHY… Web Browser Proxy Server Internet Synthetic versus Real-User “Real User” Firewall / Synthetic Agent Load-Balancer Web Server (Reverse) Proxy Server SPAN port or Network tap WiFi or 3G Smartphone Signal/Noise Ratio increases…. Network “Sniffer” User/Browser metrics Server-based metrics (C) SERITI CONSULTING, 2011 08/11/2011 22
The Synthetic Versus Real-User Debate (C) SERITI CONSULTING, 2011 08/11/2011 23
“…it's a question of when, “Because you’re skipping the “last mile” not if active monitoring of websites between the server and the user’s for availability and performance will browser, you’re not seeing how your be obsolete.” site actually performs in the real world” - Pat Meenan - Josh Bixby “You can have my active monitoring when you pry it from my cold, dead hands…” - Steve Thair http://blog.patrickmeenan.com/2011/05/demise-of-active-website-monitoring.html http://www.webperformancetoday.com/2011/07/05/web-performance-measurement-island-is-sinking/ http://www.seriticonsulting.com/blog/2011/5/21/you-can-have-my-active-monitoring-when-you-pry-it-from-my-co.html (C) SERITI CONSULTING, 2011 08/11/2011 24
Observational Study Versus Experiment (C) SERITI CONSULTING, 2011 08/11/2011 25
Experiment versus Observational Study • Both typically have the goal of detecting a relationship between the explanatory and response variables. Experiment • create differences in the explanatory variable and examine any resulting changes in the response variable (cause-and-effect conclusion) Observational Study • observe differences in the explanatory variable and notice any related differences in the response variable (association between variables) http://www.math.utah.edu/~joseph/Chapter_09.pdf (C) SERITI CONSULTING, 2011 08/11/2011 26
Observational Study = Real-User • “Watching” what happens in a given population sample • We can only observe… and try to infer what is actually happening • Many “confounding variables” • High signal to noise • Correlation (C) SERITI CONSULTING, 2011 08/11/2011 27
Location Bandwidth Wired, Latency WiFi, 3G Cached Operating objects System Addons & Antivirus Extensions Browser Device Time of Day Context Resolution (C) SERITI CONSULTING, 2011 08/11/2011 28
Observational Study = Real-User Experiment = Synthetic • “Watching” what happens in a • We “design” our experiment given population sample • We chose when, where, what, • We can only observe… and try to how etc infer what is actually happening • We control the variables (as • Many “confounding variables” much as possible) • High signal to noise • Lower signal to noise • Correlation • Causation* * OK, real “root cause” analysis will probably take a lot more investigation, I admit… but you get closer! (C) SERITI CONSULTING, 2011 08/11/2011 29
So which one is better? Neither. Complementary not Competing “…Ultimately I'd love to see a hybrid model where synthetic tests are triggered based on something detected in the data (slowdown, drop in volume, etc) to validate the issue or collect more data. - Pat Meenan (C) SERITI CONSULTING, 2011 08/11/2011 30
API Call to Synthetic Real-User Monitoring Controlled Test and Use RUM as “Reality Check” detect a change in a compare to baseline. page’s performance From Observation… By controlling the variables To Experiment… (C) SERITI CONSULTING, 2011 08/11/2011 31
Javascript Back to the “How”… Navigation timing Objective Browser Extensions “Quantitative Custom Browsers techniques” Proxy timings Web Server mods Network sniffing (C) SERITI CONSULTING, 2011 08/11/2011 32
7 WAYS OF MEASURING WEBPERF 1. JavaScript timing e.g. Souder’s Episodes or Yahoo! Boomerang* 2. Navigation-Timing e.g GA SiteSpeed 3. Browser Extension e.g. HTTPwatch 4. Custom browser e.g. 3pmobile.com or (headless) PhantomJS.org 5. Proxy timing e.g. Charles proxy 6. Web Server Mod e.g. APM solutions 7. Network sniffing e.g. Atomic Labs Pion (C) SERITI CONSULTING, 2011 08/11/2011 33
COMPARING METHODS… Measurement Method Navigation- Browser Custom Proxy Web Server Network Metric JavaScript Timing API Extension Browser Debugger Mod sniffing Charles APM Example Product WebTuna SiteSpeed HTTPWatch 3PMobile Pion Proxy Modules "Blocked/Wait" No No Yes Yes Yes No No DNS No Yes Yes Yes Yes No No Connect No Yes Yes Yes Yes No Yes Time to First Byte Partially Yes Yes Yes Yes Yes Yes "Render Start" No No Yes Yes No No No DOMReady Partially Yes Yes Yes No No No "Page/HTTP Partially Yes Yes Yes Yes No Partially Complete" OnLoad Event Yes Yes Yes Yes No No No JS Execution Time Partially No Yes Yes No No No Page-Level Yes Yes Yes Yes Partially Partially Partially Object Level No No Yes Yes Yes Yes Yes Good for RUM? Yes Yes Partially No No Partially Yes Good for Mobile? Partially Partially Partially Partially Partially Partially Partially Affects Measurement Yes No Yes Yes Yes Yes No (C) SERITI CONSULTING, 2011 08/11/2011 34
JAVASCRIPT TIMING – HOW IT WORKS unLoad Event var start = new Stick it in a Cookie Load the next page Date().getTime() PLT = onLoad Event Send a beacon var end = new beacon.gif?time=plt end - start Date().getTime() https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html (C) SERITI CONSULTING, 2011 08/11/2011 35
PROS & CONS OF JAVASCRIPT TIMING Metric JavaScript • Pro’s Example Product WebTuna • Simple "Blocked/Wait" No • Episodes/Boomerang provide custom timing for DNS No developer instrumentation Connect No Time to First Byte Partially • Cons "Render Start" No DOMReady Partially • Relies on Javascript and Cookies "Page/HTTP Partially Complete" • Only accurate for 2 nd page in journey OnLoad Event Yes JS Execution Time Partially • Can only really get a “page load metric” and a Page-Level Object Level Yes No partial TTFB metric Good for RUM? Good for Mobile? Yes Partially • “Observer effect” (and Javascript can break!) Affects Measurement Yes (C) SERITI CONSULTING, 2011 08/11/2011 36
NAVIGATION-TIMING – HOW IT WORKS onLoad Event var plt = now - Send a beacon var end = new performance.timing. beacon.gif?time=plt Date().getTime() navigationStart; (C) SERITI CONSULTING, 2011 08/11/2011 37
NAVIGATION TIMING METRICS https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html (C) SERITI CONSULTING, 2011 08/11/2011 38
PROS & CONS OF NAVIGATION-TIMING Metric Navigation- • Pro’s Timing API Example Product SiteSpeed • Even simpler! "Blocked/Wait" No • Lots more metrics DNS Yes Connect Yes • More accurate Time to First Byte Yes "Render Start" No • Cons DOMReady Yes "Page/HTTP • Need browser support for API Yes Complete" OnLoad Event Yes • IE9+ / Chrome 6+ / Firefox 7+ JS Execution Time No Page-Level Yes • Relies on Javascript (for querying API & beacon) Object Level No Good for RUM? Yes • “Observer effect” Good for Mobile? Partially Affects Measurement No • Page-level only (C) SERITI CONSULTING, 2011 08/11/2011 39
A BIT MORE ABOUT GA SITESPEED… • Just add one line for basic, free, real-user monitoring! _gaq.push(['_setAccount', 'UA-12345-1']); _gaq.push(['_trackPageview']); _gaq.push(['_trackPageLoadTime']); • Sampling appears to vary (a lot!) • 10% of page visits by design but reported 2% to 100% • Falls back to Google Toolbar if available (but NOT javascript timing) • Will probably make you think perf is better than it really is… (C) SERITI CONSULTING, 2011 08/11/2011 40
(C) SERITI CONSULTING, 2011 08/11/2011 41
(C) SERITI CONSULTING, 2011 08/11/2011 42
(C) SERITI CONSULTING, 2011 08/11/2011 43
BROWSER EXTENSION – HOW IT WORKS That subscribes to Write a browser Get your users to a whole lot of API extension… install it… event listeners… Send the timing back to collector E.g. showslow.com https://developer.mozilla.org/en/XPCOM_Interface_Reference (C) SERITI CONSULTING, 2011 08/11/2011 44
PROS & CONS OF BROWSER EXTENSIONS Metric Browser • Pros Extension • Very complete metrics Example Product HTTPWatch "Blocked/Wait" Yes • Object and Page level DNS Connect Yes Yes • No javascript (in the page at least)!!! Time to First Byte Yes • Great for continuous integration perf testing "Render Start" Yes DOMReady Yes • Cons "Page/HTTP Complete" Yes • Getting users to install it… OnLoad Event Yes JS Execution Time Yes • Not natively cross-browser Page-Level Yes Object Level Yes • Some browsers don’t support extensions Good for RUM? Good for Mobile? Partially Partially • Especially mobile browsers! Affects Measurement Yes • “Observer effect” (C) SERITI CONSULTING, 2011 08/11/2011 45
CUSTOM BROWSER – HOW IT WORKS Add custom Take some open Like WebKit or the instrumentation for source browser code Android Browser performance measurement Send the timing back Get users to to collector install it… E.g. 3pmobile.com (C) SERITI CONSULTING, 2011 08/11/2011 46
PROS & CONS OF CUSTOM BROWSER Metric Custom • Pros Browser Example Product 3PMobile • Great when you can’t use extensions / javascript / cookies "Blocked/Wait" Yes ie. For mobile performance e.g. 3Pmobile.com DNS Yes Connect Yes • Great for automation e.g. http://www.PhantomJS.org/ Time to First Byte Yes "Render Start" Yes • Good metrics (depending on OS API availability) DOMReady Yes "Page/HTTP • Cons Yes Complete" OnLoad Event Yes • Requires installation JS Execution Time Yes Page-Level Yes • Maintaining fidelity to “real browser” measurements Object Level Yes Good for RUM? No • “Observer Effect” (due to instrumentation code) Good for Mobile? Partially Affects Measurement Yes (C) SERITI CONSULTING, 2011 08/11/2011 47
PROXY DEBUGGER – HOW IT WORKS Change browser to use debugging Proxy Debugging proxy Export data to log e.g. Charles or records each request Fiddler (C) SERITI CONSULTING, 2011 08/11/2011 48
PROS & CONS OF PROXY DEBUGGER Metric Proxy • Pros Debugger Example Product Fiddler • One simple change to browser config Proxy "Blocked/Wait" Yes • No Javascript / Cookies DNS Yes Connect Yes • Can offer bandwidth throttling Time to First Byte Yes "Render Start" No • Cons DOMReady "Page/HTTP No • Proxies significantly impact HTTP traffic Yes Complete" • http://insidehttp.blogspot.com/2005/06/using-fiddler-for- OnLoad Event No JS Execution Time No performance.html Page-Level Partially Object Level Yes • No access to browser events Good for RUM? No • Concept of a “page” be problematic… Good for Mobile? Partially Affects Measurement Yes (C) SERITI CONSULTING, 2011 08/11/2011 49
6 Keep-Alive connections per SERVER Versus 8 Keep-Alive connections TOTAL per PROXY (Firefox 7.0.1) (C) SERITI CONSULTING, 2011 08/11/2011 50
WEB SERVER MOD – HOW IT WORKS Write a webserver Start a timer on Stop Timer on Mod or ISAPI filter Request Response Send the timing back to collector E.g. AppDynamics http://www.apachetutor.org/dev/request (C) SERITI CONSULTING, 2011 08/11/2011 51
PROS & CONS OF WEB SERVER MOD Metric Web Server • Pros Mod APM • Great for Application Performance Management (APM) Example Product Modules • Can be used in a “hybrid mode” with Javascript timing "Blocked/Wait" No DNS No • Measuring your “back-end” performance Connect No Time to First Byte Yes • Can be easy to deploy* "Render Start" No DOMReady No • Cons "Page/HTTP Complete" No • Limited metrics, ignores network RTT and only sees origin OnLoad Event No requests JS Execution Time No • “Observer Effect” (~5% server perf hit with APM?) Page-Level Partially Object Level Yes • Concept of a “page” be problematic… Good for RUM? Partially Good for Mobile? Partially • Can be a pain to deploy* Affects Measurement Yes (C) SERITI CONSULTING, 2011 08/11/2011 52
NETWORK SNIFFING – HOW IT WORKS Create a SPAN Promiscuous Assemble TCP/IP port or network mode packet packets into tap sniffing HTTP Requests Record the timing Assemble HTTP data in a Requests into database “pages” (C) SERITI CONSULTING, 2011 08/11/2011 53
PROS & CONS OF NETWORK SNIFFING Metric Network • Pros sniffing • No “observer effect” (totally “passive”) Example Product Pion "Blocked/Wait" No • Very common “appliance-based” RUM solution DNS No Connect Yes • Can be used in a “hybrid mode” with Javascript timing Time to First Byte "Render Start" Yes No • Can be easy to deploy* DOMReady No • Cons "Page/HTTP Partially Complete" • Limited metrics and only sees origin requests OnLoad Event No JS Execution Time No • Not “cloud friendly” at present Page-Level Partially Object Level Yes • Concept of a “page” be problematic… Good for RUM? Yes Good for Mobile? Partially • Can be a pain to deploy* Affects Measurement No (C) SERITI CONSULTING, 2011 08/11/2011 54
SUMMARY • Performance is subjective (but we try to make it objective) • Performance is Multi-dimensional • Context is critical • “Observational Studies AND Experiments” • Real User Monitoring AND Synthetic Monitoring • 7 different measurement techniques each with Pros & Cons (C) SERITI CONSULTING, 2011 08/11/2011 55
@LDNWEBPERF USER GROUP! • Join our London Web Performance Meetup • http://www.meetup.com/London-Web-Performance-Group/ • Next Wednesday 16 th Nov - 7pm – London (Bank) • WPO case study from www.thetimes.co.uk! • Follow us on Twitter @LDNWebPerf • #LDNWebPerf & #WebPerf (C) SERITI CONSULTING, 2011 08/11/2011 56
QUESTIONS? http://mobro.co/TheOpsMgr (C) SERITI CONSULTING, 2011 08/11/2011 57