SlideShare a Scribd company logo
1 of 47
Download to read offline
Real User Monitoring
@Walmarthttp://www.destructoid.com
Presentation Schedule
• Cliff - 10-20m on RUM
• Aaron – 10-20m on Monitoring in real-time
• Balaji – 10-20m on Correlating RUM and Business Analytics
• Final Q&A
Cliff Crocker
- Performance
- Reliability
- Platform & Site Analytics
Twitter: @cliffcrocker
shhhh…..
• We are not the
fastest retail site
on the internet
today.
IE 8 – Web Page Test Comp
Index – Item Page Performance
11s – Fully Loaded time
4
What gets measured, gets done…
• Before we start to optimize – see where our pain is
and prepare to measure for success.
• You never go on a diet without first stepping on the
scale…
5
Source: Flickr
Synthetic Monitoring
Pros
– Technology is great
– Real Browsers (IE, Chrome,
FF)
– Built in Alerting
– Charting/Trending
– Waterfall analysis
– Screen shots & headers
– Object level detail
Cons
– False Positives
– Impossible/impractical to
measure everything
– Fixed number of browser/OS
combinations
– Simulated bandwidth
constraints at best
– Too few data points for
statistical relevance
6
~2.2% of Global Internet
usage - Thanksgiving this year
7
• If only those users
could tell us about their
site experience…
8
Source: Flickr
Enter boomerang.js - https://github.com/yahoo/boomerang
boomerang.js with NavTiming (thanks Buddy & Phil)
• Doc Complete, Page Response, Page Processing, TTFB, DNS, Connect & more
• Cookie data (for parent domain)
• Location (geolookup on IP)
• Referrer
• User Agent
• Anything else you want to stuff into the beacon
+
9
Source: Flickr
10
Start with a url group (i.e. ‘Item Page’)
Find offenders that
have the most
impact
Identify usage patterns for browsers
(Chrome and Safari users more
active on weekends)
12
13
What we found scared us…
Home Page Performance – Jan 2012
14
Set some goals and SLAs
• Focus on ‘Page Processing Time’ First
– see Golden Rule – 80% of time spent here (more like 90% for Walmart)
• Look at backend & network response time SLAs later
• Use 95th
Percentile
• Set Achievable SLAs
• Revisit Monthly
• Celebrate Wins!
15
Case Study: Item Page
• Problem:
– Page takes ~24s for slowest 5% of users
• Too many elements
• Slow third party modules
• Several other pagespeed ‘no-nos’
• Goal: Meet SLA for February
– 20s (95th
percentile)
• Approach:
– Scrum team dedicated to perf optimization for 1 sprint
– Team pools resources and ideas - focuses on biggest bang
16
Success!
8s off the 95th
%tile
17
Success!
2.7s under February SLA
18
RUM in Near Real Time
BEACON
SERVER
BEACON
SERVERCLIENTCLIENT INTERNETINTERNET
MAGIC SMOKE
How boomerang.js Works
1. client downloads JavaScript payload
2. client sends RUM payload to beacon
3. beacon server responds with HTTP 204
WHO CARES!WHO CARES!
20
Aaron Kulick
- Performance
- Founder: SF & SV Web Performance Group
http://www.sfwebperf.org
Twitter: @GoFastWeb
WHO AM I?
Configuring boomerang.js is EASY
Getting Actionable Data Is HARD
22
Where’s the data?
The initial incarnation of RUM @ WMT leveraged Akamai at the edge.
• data reliability issues
• data availability issues
• data “freshness” issues
It works… for certain values of work
Source: Flickr
Source: Flickr
23
HUBBLEHUBBLE
Enter Hubble
JETTYJETTY
DURABLE QUEUEDURABLE QUEUE
APACHE FLUME NGAPACHE FLUME NG
HDFSHDFS UDPUDP
HADOOPHADOOP
24
REAL TIMEREAL TIME
The Bridge Between Then and Now
• Etsy/StatsD - https://github.com/etsy/statsd
– Timers
– Counters
– Simple Aggregation
• Min, Max, Mean, Median, 95th
Percentile
• Whisper, Carbon & Graphite - https://launchpad.net/graphite
– Scalable Realtime Graphing
– RRD-like Storage Requirements + Federated Option
– Supports Irregular Updates
The Now Bits
HUBBLEHUBBLE
UDPUDP
StatsDStatsD
AGGREGATION
Graphite
+
Whisper
Graphite
+
Whisper
UDP/TCPUDP/TCP
STORAGE
+
DISPLAY
• Fixed some calculation problems in Etsy/StatsD
• Added median calculation for timers
• Incorporated multiple flush interval patches
Pretty Pictures
RAW DATA FORMAT
uswmt.all.t_page.upper_95(9289.0),1329259510,1329260710,10|15904.0,9184.0,9125.0,12736.0,11735.0,16776.0,8484.0,
10839.0,14620.0,7579.0,8871.0,8240.0,12390.0,5211.0,10301.0,24784.0,9410.0,16554.0,9609.0,11871.0,12751.0,9797.0,
11003.0,15962.0,7953.0,7707.0,4181.0,11616.0,11746.0,12814.0,10566.0,24782.0,18303.0,20904.0,7718.0,8531.0,7312.0,
9614.0,8749.0,11671.0,5989.0,9832.0,10592.0,11611.0,16946.0,18858.0,14360.0,15927.0,10470.0,10140.0,11307.0,9739.0,
9772.0,9875.0,13641.0,11626.0,14758.0,6529.0,11727.0,10194.0,8003.0,10639.0,7297.0,9891.0,10312.0,12497.0,11557.0,
11406.0,12456.0,12939.0,11029.0,10813.0,11737.0,10618.0,14128.0,16879.0,15865.0,6255.0,14605.0,8861.0,27425.0,
10948.0,19666.0,7185.0,13266.0,13156.0,15111.0,13110.0,15151.0,8666.0,16775.0,10110.0,10387.0,17274.0,22183.0,
8937.0,13168.0,12267.0,11891.0,9635.0,10446.0,8129.0,9550.0,9229.0,8375.0,8657.0,11119.0,6799.0,9094.0,21952.0,
14989.0,16828.0,9001.0,13444.0,10332.0,13609.0,9266.0,13349.0,11546.0,9289.0
uswmt.all.t_page.median(1844.0),1329259510,1329260710,10|4165.0,2333.5,2073.5,2584.0,2547.0,2627.5,2401.0,1575.0,
2170.0,1169.0,1970.0,1838.0,2083.0,5211.0,2496.0,3242.5,1541.0,1437.5,1928.0,1971.0,1776.0,3108.0,2010.5,2044.0,2325.5,
2640.0,1733.0,3924.0,2629.0,1867.5,1782.0,2370.5,2921.0,4783.0,2260.0,1340.0,3256.0,2297.0,2565.0,1874.0,2000.0,2483.5,
2705.5,2432.0,1809.0,2826.0,2204.0,2695.0,1045.0,1615.5,2250.0,2387.0,1562.5,1998.0,2512.0,2139.0,1482.0,2138.5,2100.0,
2583.0,2652.0,3277.0,2549.0,1755.0,2196.5,2766.0,2989.5,3638.0,3034.0,3615.5,2650.5,5207.5,3023.0,1941.0,1918.5,1768.0,
3048.0,1522.5,2710.0,1392.0,2402.0,2005.0,3246.0,1383.0,1880.0,2398.0,1833.0,2579.0,2052.0,2622.0,2089.0,1102.0,1296.0,
3339.0,2132.5,2831.0,3466.0,2131.0,2026.0,2754.0,3228.5,1000.0,2075.0,2011.5,2428.0,4019.5,2788.0,1665.0,1968.0,2695.5,
2873.0,1752.0,2314.5,1766.0,2971.0,3091.5,2205.5,3033.0,2476.0,1844.0
27
The Work Tomorrow…
The Good
•Metric Throughput
•Commodity Storage
•Commodity Infrastructure
The Bad
•More Metric Throughput
•Calculation Complexity
•Web Sockets (pretty)
•Metric Fan-out
28
WATCH THIS SPACE
29
Is Page Performance a Factor of
Site Conversion? And how big is
it?
February, 2012
v
s
31
Walmart.com - Fun Facts
• Reach
–Millions of Shoppers/week.
–Billions of page requests/year - Spikes up to 1500%
–Billions of internal product search volumes/year
• Scale
–Millions of active product SKUs + Market Place
–Millions of pages indexed in search engines
• Complexity
–1/4th
of page contents served by partners, affiliates and Marketplace
–Multiple departments, 10+ checkout paths
Page Performance & Site Conversion – Feb 2012
32
So, how do you monitor?...
Page Performance & Site Conversion – Feb 2012
33
Few Industry Benchmarks…
• Factoid 1: Large eCommerce site extensively A/B
tested page performance and published a study
showing 100 millisecond delay = 1% drop in revenue
• Factoid 2: Search Engines A/B tested performance
and found that a 500 millisecond delay caused a 20%
drop in traffic.
• Factoid 3: In an experiment across multiple retailers,
a 1 second delay caused a 7% decline in conversion
Page Performance & Site Conversion – Feb 2012
34
So, how big is it for Walmart.com?
Page Performance & Site Conversion – Feb 2012
35
Agenda
• Phase 1 – Baseline Measurement - Impact of Site
Performance on Conversion, Bounce rates &
Revenue
• Phase 2 - Targets for Page Performance
• Phase 3 – Optimization Results
• Key Highlights & Takeaways
Page Performance & Site Conversion – Feb 2012
36
Agenda
• Phase 1 – Baseline Measurement - Impact of Site
Performance on Conversion, Bounce rates &
Revenue
• Phase 2 – Targets for Page Performance
• Phase 3 – Optimization Results
• Key Highlights & Takeaways
Page Performance & Site Conversion – Feb 2012
37
Impact of site performance on overall site conversion rate….
Baseline – 1 in 2 site visits had response time > 4 seconds
* Sharp decline in conversion rate as average site load time increases from 1 to 4 seconds
* Overall average site load time is lower for the converted population (3.22 Seconds) than the non-
converted population (6.03 Seconds)
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
38
@ Page level….
Page load time is lower for Buyers compared to Non-Buyers
* The Page load time is highest for certain pages - 6.38 secs when there was a conversion and
8.06 where there was no conversion.
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
39
@ Department level….
Department load time is lower for Buyers compared to Non-Buyers
* Key Categories has 2-3 seconds difference b/w buyer Vs non-buyer
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
40
What about bounce?
Page Bounce Rate Vs Response Time
* Key pages have high bounce rates which correlates with high T_Page as well
* Significant difference ( up to 9secs) in T_Page between bounced and non-bounced for landing
pages.
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
41
Bounce rates @ department level….
Department Bounce Rate Vs Response Time
* High T_Page for key pages (up to 19.82s) and key department making Bounce rate significantly
higher
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
42
Agenda
• Phase 1 – Baseline Measurement - Impact of Site
Performance on Conversion, Bounce rates &
Revenue
• Phase 2 - Targets for Page Performance
• Phase 3 – Optimization Results
• Key Highlights & Takeaways
Page Performance & Site Conversion – Feb 2012
43
Phase 2 – Targets for Page Performance……
Conversion & Bounce Rate Impacts Drives Prioritization
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
44
Agenda
• Phase 1 – Baseline Measurement - Impact of Site
Performance on Conversion, Bounce rates &
Revenue
• Phase 2 - Targets for Page Performance
• Phase 3 – Optimization Results
• Key Highlights & Takeaways
Page Performance & Site Conversion – Feb 2012
45
Phase 3 – Success Story….
First Win….and yes conversion had positive improvements…
Page Performance & Site Conversion – Feb 2012
46
• Page speed matters for site conversion!
• Monitor real user performance in a “Big Data” way!!
• Every 1s improvement = Up to 2% increase in CVR
• 100ms improvement = Up to 1% incremental revenue
• SEO benefits for entry pages and reduce bounces
• Test & Learn - Target segments and run A/B Tests focused on
improving page performance
Key Highlights
Page Performance & Site Conversion – Feb 2012
We’re Hiring for Everything!!!!
47
Source: Flickr
http://www.walmartlabs.com/open-positions/
https://walmartstores.com/careers/apply/?ba=eCom
@cliffcrocker or @GoFastWeb

More Related Content

Similar to Walmart Web Performance Circa 2013

Lean Six Sigma Green Belt Certification 1
Lean Six Sigma Green Belt Certification 1Lean Six Sigma Green Belt Certification 1
Lean Six Sigma Green Belt Certification 1Fred Zuercher
 
Metrics, Metrics Everywhere (but where the heck do you start?)
Metrics, Metrics Everywhere (but where the heck do you start?)Metrics, Metrics Everywhere (but where the heck do you start?)
Metrics, Metrics Everywhere (but where the heck do you start?)SOASTA
 
Metrics, Metrics Everywhere (but where the heck do you start?)
Metrics, Metrics Everywhere (but where the heck do you start?)Metrics, Metrics Everywhere (but where the heck do you start?)
Metrics, Metrics Everywhere (but where the heck do you start?)SOASTA
 
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)Tammy Everts
 
improving the performance of Rails web Applications
improving the performance of Rails web Applicationsimproving the performance of Rails web Applications
improving the performance of Rails web ApplicationsJohn McCaffrey
 
Windy cityrails performance_tuning
Windy cityrails performance_tuningWindy cityrails performance_tuning
Windy cityrails performance_tuningJohn McCaffrey
 
Ruby on Rails Performance Tuning. Make it faster, make it better (WindyCityRa...
Ruby on Rails Performance Tuning. Make it faster, make it better (WindyCityRa...Ruby on Rails Performance Tuning. Make it faster, make it better (WindyCityRa...
Ruby on Rails Performance Tuning. Make it faster, make it better (WindyCityRa...John McCaffrey
 
Enterprise mobilelogix overview
Enterprise mobilelogix overviewEnterprise mobilelogix overview
Enterprise mobilelogix overviewMobileLogix
 
Case study migration from cm13 to cm14 - Oracle Primavera P6 Collaborate 14
Case study migration from cm13 to cm14 - Oracle Primavera P6 Collaborate 14Case study migration from cm13 to cm14 - Oracle Primavera P6 Collaborate 14
Case study migration from cm13 to cm14 - Oracle Primavera P6 Collaborate 14p6academy
 
Ahead of the Curve: How 23andMe Improved UX with Performance Edge
Ahead of the Curve: How 23andMe Improved UX with Performance EdgeAhead of the Curve: How 23andMe Improved UX with Performance Edge
Ahead of the Curve: How 23andMe Improved UX with Performance EdgeOptimizely
 
H:\Documents\Life After Upgrading To R12
H:\Documents\Life After Upgrading To  R12H:\Documents\Life After Upgrading To  R12
H:\Documents\Life After Upgrading To R12InSync Conference
 
British American Tobacco GP Upgrade
British American Tobacco GP UpgradeBritish American Tobacco GP Upgrade
British American Tobacco GP UpgradeCedar Consulting
 
Demystifying Website Performance and Its Impact on Revenue
Demystifying Website Performance and Its Impact on RevenueDemystifying Website Performance and Its Impact on Revenue
Demystifying Website Performance and Its Impact on Revenuesection_io
 
A Designer's Guide to Web Performance
A Designer's Guide to Web PerformanceA Designer's Guide to Web Performance
A Designer's Guide to Web PerformanceKevin Mandeville
 
Managing Productivity With Unifier
Managing Productivity With UnifierManaging Productivity With Unifier
Managing Productivity With Unifierp6academy
 
A (Fairly) Complete Guide to Performance Budgets [SmashingConf SF 2023]
A (Fairly) Complete Guide to Performance Budgets [SmashingConf SF 2023]A (Fairly) Complete Guide to Performance Budgets [SmashingConf SF 2023]
A (Fairly) Complete Guide to Performance Budgets [SmashingConf SF 2023]Tammy Everts
 
Edge 2014: A Modern Approach to Performance Monitoring
Edge 2014: A Modern Approach to Performance MonitoringEdge 2014: A Modern Approach to Performance Monitoring
Edge 2014: A Modern Approach to Performance MonitoringAkamai Technologies
 
From PHP to React - case study
From PHP to React - case studyFrom PHP to React - case study
From PHP to React - case studySparkbit
 

Similar to Walmart Web Performance Circa 2013 (20)

Lean Six Sigma Green Belt Certification 1
Lean Six Sigma Green Belt Certification 1Lean Six Sigma Green Belt Certification 1
Lean Six Sigma Green Belt Certification 1
 
Metrics, Metrics Everywhere (but where the heck do you start?)
Metrics, Metrics Everywhere (but where the heck do you start?)Metrics, Metrics Everywhere (but where the heck do you start?)
Metrics, Metrics Everywhere (but where the heck do you start?)
 
Metrics, Metrics Everywhere (but where the heck do you start?)
Metrics, Metrics Everywhere (but where the heck do you start?)Metrics, Metrics Everywhere (but where the heck do you start?)
Metrics, Metrics Everywhere (but where the heck do you start?)
 
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
 
improving the performance of Rails web Applications
improving the performance of Rails web Applicationsimproving the performance of Rails web Applications
improving the performance of Rails web Applications
 
Windy cityrails performance_tuning
Windy cityrails performance_tuningWindy cityrails performance_tuning
Windy cityrails performance_tuning
 
Ruby on Rails Performance Tuning. Make it faster, make it better (WindyCityRa...
Ruby on Rails Performance Tuning. Make it faster, make it better (WindyCityRa...Ruby on Rails Performance Tuning. Make it faster, make it better (WindyCityRa...
Ruby on Rails Performance Tuning. Make it faster, make it better (WindyCityRa...
 
Enterprise mobilelogix overview
Enterprise mobilelogix overviewEnterprise mobilelogix overview
Enterprise mobilelogix overview
 
Case study migration from cm13 to cm14 - Oracle Primavera P6 Collaborate 14
Case study migration from cm13 to cm14 - Oracle Primavera P6 Collaborate 14Case study migration from cm13 to cm14 - Oracle Primavera P6 Collaborate 14
Case study migration from cm13 to cm14 - Oracle Primavera P6 Collaborate 14
 
Ahead of the Curve: How 23andMe Improved UX with Performance Edge
Ahead of the Curve: How 23andMe Improved UX with Performance EdgeAhead of the Curve: How 23andMe Improved UX with Performance Edge
Ahead of the Curve: How 23andMe Improved UX with Performance Edge
 
H:\Documents\Life After Upgrading To R12
H:\Documents\Life After Upgrading To  R12H:\Documents\Life After Upgrading To  R12
H:\Documents\Life After Upgrading To R12
 
British American Tobacco GP Upgrade
British American Tobacco GP UpgradeBritish American Tobacco GP Upgrade
British American Tobacco GP Upgrade
 
Demystifying Website Performance and Its Impact on Revenue
Demystifying Website Performance and Its Impact on RevenueDemystifying Website Performance and Its Impact on Revenue
Demystifying Website Performance and Its Impact on Revenue
 
Web performance e-book
Web performance e-bookWeb performance e-book
Web performance e-book
 
Designers Guide to Web Performance Yotta 2013
Designers Guide to Web Performance Yotta 2013Designers Guide to Web Performance Yotta 2013
Designers Guide to Web Performance Yotta 2013
 
A Designer's Guide to Web Performance
A Designer's Guide to Web PerformanceA Designer's Guide to Web Performance
A Designer's Guide to Web Performance
 
Managing Productivity With Unifier
Managing Productivity With UnifierManaging Productivity With Unifier
Managing Productivity With Unifier
 
A (Fairly) Complete Guide to Performance Budgets [SmashingConf SF 2023]
A (Fairly) Complete Guide to Performance Budgets [SmashingConf SF 2023]A (Fairly) Complete Guide to Performance Budgets [SmashingConf SF 2023]
A (Fairly) Complete Guide to Performance Budgets [SmashingConf SF 2023]
 
Edge 2014: A Modern Approach to Performance Monitoring
Edge 2014: A Modern Approach to Performance MonitoringEdge 2014: A Modern Approach to Performance Monitoring
Edge 2014: A Modern Approach to Performance Monitoring
 
From PHP to React - case study
From PHP to React - case studyFrom PHP to React - case study
From PHP to React - case study
 

More from Cliff Crocker

Measuring What Matters - Fluent Conf 2018
Measuring What Matters - Fluent Conf 2018Measuring What Matters - Fluent Conf 2018
Measuring What Matters - Fluent Conf 2018Cliff Crocker
 
Measuring what matters
Measuring what mattersMeasuring what matters
Measuring what mattersCliff Crocker
 
Velocity NYC: Metrics, metrics everywhere (but where the heck do you start?)
Velocity NYC: Metrics, metrics everywhere (but where the heck do you start?)Velocity NYC: Metrics, metrics everywhere (but where the heck do you start?)
Velocity NYC: Metrics, metrics everywhere (but where the heck do you start?)Cliff Crocker
 
Synthetic and RUM - Best of bo
Synthetic and RUM - Best of boSynthetic and RUM - Best of bo
Synthetic and RUM - Best of boCliff Crocker
 
A Modern Approach to Performance Monitoring
A Modern Approach to Performance MonitoringA Modern Approach to Performance Monitoring
A Modern Approach to Performance MonitoringCliff Crocker
 
Velocity NY - How to Measure Revenue in Milliseconds
Velocity NY - How to Measure Revenue in MillisecondsVelocity NY - How to Measure Revenue in Milliseconds
Velocity NY - How to Measure Revenue in MillisecondsCliff Crocker
 
Velocity NY 2014 - The Natives are Getting Restless
Velocity NY 2014 - The Natives are Getting RestlessVelocity NY 2014 - The Natives are Getting Restless
Velocity NY 2014 - The Natives are Getting RestlessCliff Crocker
 
Managing the unmanageable - Third Party RUM
Managing the unmanageable - Third Party RUMManaging the unmanageable - Third Party RUM
Managing the unmanageable - Third Party RUMCliff Crocker
 
Rum first london web perf meetup
Rum first   london web perf meetupRum first   london web perf meetup
Rum first london web perf meetupCliff Crocker
 
"Rum First" NYC Webperf Meetup
"Rum First" NYC Webperf Meetup"Rum First" NYC Webperf Meetup
"Rum First" NYC Webperf MeetupCliff Crocker
 

More from Cliff Crocker (10)

Measuring What Matters - Fluent Conf 2018
Measuring What Matters - Fluent Conf 2018Measuring What Matters - Fluent Conf 2018
Measuring What Matters - Fluent Conf 2018
 
Measuring what matters
Measuring what mattersMeasuring what matters
Measuring what matters
 
Velocity NYC: Metrics, metrics everywhere (but where the heck do you start?)
Velocity NYC: Metrics, metrics everywhere (but where the heck do you start?)Velocity NYC: Metrics, metrics everywhere (but where the heck do you start?)
Velocity NYC: Metrics, metrics everywhere (but where the heck do you start?)
 
Synthetic and RUM - Best of bo
Synthetic and RUM - Best of boSynthetic and RUM - Best of bo
Synthetic and RUM - Best of bo
 
A Modern Approach to Performance Monitoring
A Modern Approach to Performance MonitoringA Modern Approach to Performance Monitoring
A Modern Approach to Performance Monitoring
 
Velocity NY - How to Measure Revenue in Milliseconds
Velocity NY - How to Measure Revenue in MillisecondsVelocity NY - How to Measure Revenue in Milliseconds
Velocity NY - How to Measure Revenue in Milliseconds
 
Velocity NY 2014 - The Natives are Getting Restless
Velocity NY 2014 - The Natives are Getting RestlessVelocity NY 2014 - The Natives are Getting Restless
Velocity NY 2014 - The Natives are Getting Restless
 
Managing the unmanageable - Third Party RUM
Managing the unmanageable - Third Party RUMManaging the unmanageable - Third Party RUM
Managing the unmanageable - Third Party RUM
 
Rum first london web perf meetup
Rum first   london web perf meetupRum first   london web perf meetup
Rum first london web perf meetup
 
"Rum First" NYC Webperf Meetup
"Rum First" NYC Webperf Meetup"Rum First" NYC Webperf Meetup
"Rum First" NYC Webperf Meetup
 

Recently uploaded

2024_hackersuli_mobil_ios_android ______
2024_hackersuli_mobil_ios_android ______2024_hackersuli_mobil_ios_android ______
2024_hackersuli_mobil_ios_android ______hackersuli
 
draft-harrison-sidrops-manifest-number-01, presented at IETF 119
draft-harrison-sidrops-manifest-number-01, presented at IETF 119draft-harrison-sidrops-manifest-number-01, presented at IETF 119
draft-harrison-sidrops-manifest-number-01, presented at IETF 119APNIC
 
Basic Security.pptx is a awsome PPT on your mobiel
Basic Security.pptx is a awsome PPT on your mobielBasic Security.pptx is a awsome PPT on your mobiel
Basic Security.pptx is a awsome PPT on your mobielpratamakiki860
 
Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119
Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119
Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119APNIC
 
Summary IGF 2013 Bali - English (tata kelola internet / internet governance)
Summary  IGF 2013 Bali - English (tata kelola internet / internet governance)Summary  IGF 2013 Bali - English (tata kelola internet / internet governance)
Summary IGF 2013 Bali - English (tata kelola internet / internet governance)ICT Watch - Indonesia
 
Tari Eason Warriors Come Out To Play T Shirts
Tari Eason Warriors Come Out To Play T ShirtsTari Eason Warriors Come Out To Play T Shirts
Tari Eason Warriors Come Out To Play T Shirtsrahman018755
 
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...vmzoxnx5
 
Summary ID-IGF 2016 National Dialogue - English (tata kelola internet / int...
Summary  ID-IGF 2016 National Dialogue  - English (tata kelola internet / int...Summary  ID-IGF 2016 National Dialogue  - English (tata kelola internet / int...
Summary ID-IGF 2016 National Dialogue - English (tata kelola internet / int...ICT Watch - Indonesia
 
Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...
Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...
Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...Andreas Sfakianakis
 
Is DNS ready for IPv6, presented by Geoff Huston at IETF 119
Is DNS ready for IPv6, presented by Geoff Huston at IETF 119Is DNS ready for IPv6, presented by Geoff Huston at IETF 119
Is DNS ready for IPv6, presented by Geoff Huston at IETF 119APNIC
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119APNIC
 
Power of Social Media for E-commerce.pdf
Power of Social Media for E-commerce.pdfPower of Social Media for E-commerce.pdf
Power of Social Media for E-commerce.pdfrajats19920
 
IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119
IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119
IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119APNIC
 

Recently uploaded (13)

2024_hackersuli_mobil_ios_android ______
2024_hackersuli_mobil_ios_android ______2024_hackersuli_mobil_ios_android ______
2024_hackersuli_mobil_ios_android ______
 
draft-harrison-sidrops-manifest-number-01, presented at IETF 119
draft-harrison-sidrops-manifest-number-01, presented at IETF 119draft-harrison-sidrops-manifest-number-01, presented at IETF 119
draft-harrison-sidrops-manifest-number-01, presented at IETF 119
 
Basic Security.pptx is a awsome PPT on your mobiel
Basic Security.pptx is a awsome PPT on your mobielBasic Security.pptx is a awsome PPT on your mobiel
Basic Security.pptx is a awsome PPT on your mobiel
 
Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119
Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119
Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119
 
Summary IGF 2013 Bali - English (tata kelola internet / internet governance)
Summary  IGF 2013 Bali - English (tata kelola internet / internet governance)Summary  IGF 2013 Bali - English (tata kelola internet / internet governance)
Summary IGF 2013 Bali - English (tata kelola internet / internet governance)
 
Tari Eason Warriors Come Out To Play T Shirts
Tari Eason Warriors Come Out To Play T ShirtsTari Eason Warriors Come Out To Play T Shirts
Tari Eason Warriors Come Out To Play T Shirts
 
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
 
Summary ID-IGF 2016 National Dialogue - English (tata kelola internet / int...
Summary  ID-IGF 2016 National Dialogue  - English (tata kelola internet / int...Summary  ID-IGF 2016 National Dialogue  - English (tata kelola internet / int...
Summary ID-IGF 2016 National Dialogue - English (tata kelola internet / int...
 
Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...
Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...
Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...
 
Is DNS ready for IPv6, presented by Geoff Huston at IETF 119
Is DNS ready for IPv6, presented by Geoff Huston at IETF 119Is DNS ready for IPv6, presented by Geoff Huston at IETF 119
Is DNS ready for IPv6, presented by Geoff Huston at IETF 119
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119
 
Power of Social Media for E-commerce.pdf
Power of Social Media for E-commerce.pdfPower of Social Media for E-commerce.pdf
Power of Social Media for E-commerce.pdf
 
IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119
IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119
IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119
 

Walmart Web Performance Circa 2013

  • 2. Presentation Schedule • Cliff - 10-20m on RUM • Aaron – 10-20m on Monitoring in real-time • Balaji – 10-20m on Correlating RUM and Business Analytics • Final Q&A
  • 3. Cliff Crocker - Performance - Reliability - Platform & Site Analytics Twitter: @cliffcrocker
  • 4. shhhh….. • We are not the fastest retail site on the internet today. IE 8 – Web Page Test Comp Index – Item Page Performance 11s – Fully Loaded time 4
  • 5. What gets measured, gets done… • Before we start to optimize – see where our pain is and prepare to measure for success. • You never go on a diet without first stepping on the scale… 5 Source: Flickr
  • 6. Synthetic Monitoring Pros – Technology is great – Real Browsers (IE, Chrome, FF) – Built in Alerting – Charting/Trending – Waterfall analysis – Screen shots & headers – Object level detail Cons – False Positives – Impossible/impractical to measure everything – Fixed number of browser/OS combinations – Simulated bandwidth constraints at best – Too few data points for statistical relevance 6
  • 7. ~2.2% of Global Internet usage - Thanksgiving this year 7
  • 8. • If only those users could tell us about their site experience… 8 Source: Flickr
  • 9. Enter boomerang.js - https://github.com/yahoo/boomerang boomerang.js with NavTiming (thanks Buddy & Phil) • Doc Complete, Page Response, Page Processing, TTFB, DNS, Connect & more • Cookie data (for parent domain) • Location (geolookup on IP) • Referrer • User Agent • Anything else you want to stuff into the beacon + 9 Source: Flickr
  • 10. 10
  • 11. Start with a url group (i.e. ‘Item Page’) Find offenders that have the most impact
  • 12. Identify usage patterns for browsers (Chrome and Safari users more active on weekends) 12
  • 13. 13
  • 14. What we found scared us… Home Page Performance – Jan 2012 14
  • 15. Set some goals and SLAs • Focus on ‘Page Processing Time’ First – see Golden Rule – 80% of time spent here (more like 90% for Walmart) • Look at backend & network response time SLAs later • Use 95th Percentile • Set Achievable SLAs • Revisit Monthly • Celebrate Wins! 15
  • 16. Case Study: Item Page • Problem: – Page takes ~24s for slowest 5% of users • Too many elements • Slow third party modules • Several other pagespeed ‘no-nos’ • Goal: Meet SLA for February – 20s (95th percentile) • Approach: – Scrum team dedicated to perf optimization for 1 sprint – Team pools resources and ideas - focuses on biggest bang 16
  • 17. Success! 8s off the 95th %tile 17
  • 19. RUM in Near Real Time
  • 20. BEACON SERVER BEACON SERVERCLIENTCLIENT INTERNETINTERNET MAGIC SMOKE How boomerang.js Works 1. client downloads JavaScript payload 2. client sends RUM payload to beacon 3. beacon server responds with HTTP 204 WHO CARES!WHO CARES! 20
  • 21. Aaron Kulick - Performance - Founder: SF & SV Web Performance Group http://www.sfwebperf.org Twitter: @GoFastWeb WHO AM I?
  • 22. Configuring boomerang.js is EASY Getting Actionable Data Is HARD 22
  • 23. Where’s the data? The initial incarnation of RUM @ WMT leveraged Akamai at the edge. • data reliability issues • data availability issues • data “freshness” issues It works… for certain values of work Source: Flickr Source: Flickr 23
  • 24. HUBBLEHUBBLE Enter Hubble JETTYJETTY DURABLE QUEUEDURABLE QUEUE APACHE FLUME NGAPACHE FLUME NG HDFSHDFS UDPUDP HADOOPHADOOP 24 REAL TIMEREAL TIME
  • 25. The Bridge Between Then and Now • Etsy/StatsD - https://github.com/etsy/statsd – Timers – Counters – Simple Aggregation • Min, Max, Mean, Median, 95th Percentile • Whisper, Carbon & Graphite - https://launchpad.net/graphite – Scalable Realtime Graphing – RRD-like Storage Requirements + Federated Option – Supports Irregular Updates
  • 26. The Now Bits HUBBLEHUBBLE UDPUDP StatsDStatsD AGGREGATION Graphite + Whisper Graphite + Whisper UDP/TCPUDP/TCP STORAGE + DISPLAY • Fixed some calculation problems in Etsy/StatsD • Added median calculation for timers • Incorporated multiple flush interval patches
  • 27. Pretty Pictures RAW DATA FORMAT uswmt.all.t_page.upper_95(9289.0),1329259510,1329260710,10|15904.0,9184.0,9125.0,12736.0,11735.0,16776.0,8484.0, 10839.0,14620.0,7579.0,8871.0,8240.0,12390.0,5211.0,10301.0,24784.0,9410.0,16554.0,9609.0,11871.0,12751.0,9797.0, 11003.0,15962.0,7953.0,7707.0,4181.0,11616.0,11746.0,12814.0,10566.0,24782.0,18303.0,20904.0,7718.0,8531.0,7312.0, 9614.0,8749.0,11671.0,5989.0,9832.0,10592.0,11611.0,16946.0,18858.0,14360.0,15927.0,10470.0,10140.0,11307.0,9739.0, 9772.0,9875.0,13641.0,11626.0,14758.0,6529.0,11727.0,10194.0,8003.0,10639.0,7297.0,9891.0,10312.0,12497.0,11557.0, 11406.0,12456.0,12939.0,11029.0,10813.0,11737.0,10618.0,14128.0,16879.0,15865.0,6255.0,14605.0,8861.0,27425.0, 10948.0,19666.0,7185.0,13266.0,13156.0,15111.0,13110.0,15151.0,8666.0,16775.0,10110.0,10387.0,17274.0,22183.0, 8937.0,13168.0,12267.0,11891.0,9635.0,10446.0,8129.0,9550.0,9229.0,8375.0,8657.0,11119.0,6799.0,9094.0,21952.0, 14989.0,16828.0,9001.0,13444.0,10332.0,13609.0,9266.0,13349.0,11546.0,9289.0 uswmt.all.t_page.median(1844.0),1329259510,1329260710,10|4165.0,2333.5,2073.5,2584.0,2547.0,2627.5,2401.0,1575.0, 2170.0,1169.0,1970.0,1838.0,2083.0,5211.0,2496.0,3242.5,1541.0,1437.5,1928.0,1971.0,1776.0,3108.0,2010.5,2044.0,2325.5, 2640.0,1733.0,3924.0,2629.0,1867.5,1782.0,2370.5,2921.0,4783.0,2260.0,1340.0,3256.0,2297.0,2565.0,1874.0,2000.0,2483.5, 2705.5,2432.0,1809.0,2826.0,2204.0,2695.0,1045.0,1615.5,2250.0,2387.0,1562.5,1998.0,2512.0,2139.0,1482.0,2138.5,2100.0, 2583.0,2652.0,3277.0,2549.0,1755.0,2196.5,2766.0,2989.5,3638.0,3034.0,3615.5,2650.5,5207.5,3023.0,1941.0,1918.5,1768.0, 3048.0,1522.5,2710.0,1392.0,2402.0,2005.0,3246.0,1383.0,1880.0,2398.0,1833.0,2579.0,2052.0,2622.0,2089.0,1102.0,1296.0, 3339.0,2132.5,2831.0,3466.0,2131.0,2026.0,2754.0,3228.5,1000.0,2075.0,2011.5,2428.0,4019.5,2788.0,1665.0,1968.0,2695.5, 2873.0,1752.0,2314.5,1766.0,2971.0,3091.5,2205.5,3033.0,2476.0,1844.0 27
  • 28. The Work Tomorrow… The Good •Metric Throughput •Commodity Storage •Commodity Infrastructure The Bad •More Metric Throughput •Calculation Complexity •Web Sockets (pretty) •Metric Fan-out 28
  • 30. Is Page Performance a Factor of Site Conversion? And how big is it? February, 2012 v s
  • 31. 31 Walmart.com - Fun Facts • Reach –Millions of Shoppers/week. –Billions of page requests/year - Spikes up to 1500% –Billions of internal product search volumes/year • Scale –Millions of active product SKUs + Market Place –Millions of pages indexed in search engines • Complexity –1/4th of page contents served by partners, affiliates and Marketplace –Multiple departments, 10+ checkout paths Page Performance & Site Conversion – Feb 2012
  • 32. 32 So, how do you monitor?... Page Performance & Site Conversion – Feb 2012
  • 33. 33 Few Industry Benchmarks… • Factoid 1: Large eCommerce site extensively A/B tested page performance and published a study showing 100 millisecond delay = 1% drop in revenue • Factoid 2: Search Engines A/B tested performance and found that a 500 millisecond delay caused a 20% drop in traffic. • Factoid 3: In an experiment across multiple retailers, a 1 second delay caused a 7% decline in conversion Page Performance & Site Conversion – Feb 2012
  • 34. 34 So, how big is it for Walmart.com? Page Performance & Site Conversion – Feb 2012
  • 35. 35 Agenda • Phase 1 – Baseline Measurement - Impact of Site Performance on Conversion, Bounce rates & Revenue • Phase 2 - Targets for Page Performance • Phase 3 – Optimization Results • Key Highlights & Takeaways Page Performance & Site Conversion – Feb 2012
  • 36. 36 Agenda • Phase 1 – Baseline Measurement - Impact of Site Performance on Conversion, Bounce rates & Revenue • Phase 2 – Targets for Page Performance • Phase 3 – Optimization Results • Key Highlights & Takeaways Page Performance & Site Conversion – Feb 2012
  • 37. 37 Impact of site performance on overall site conversion rate…. Baseline – 1 in 2 site visits had response time > 4 seconds * Sharp decline in conversion rate as average site load time increases from 1 to 4 seconds * Overall average site load time is lower for the converted population (3.22 Seconds) than the non- converted population (6.03 Seconds) Note: Load Time here is the time taken from head of the page to page ready (T_Page) Page Performance & Site Conversion – Feb 2012
  • 38. 38 @ Page level…. Page load time is lower for Buyers compared to Non-Buyers * The Page load time is highest for certain pages - 6.38 secs when there was a conversion and 8.06 where there was no conversion. Note: Load Time here is the time taken from head of the page to page ready (T_Page) Page Performance & Site Conversion – Feb 2012
  • 39. 39 @ Department level…. Department load time is lower for Buyers compared to Non-Buyers * Key Categories has 2-3 seconds difference b/w buyer Vs non-buyer Note: Load Time here is the time taken from head of the page to page ready (T_Page) Page Performance & Site Conversion – Feb 2012
  • 40. 40 What about bounce? Page Bounce Rate Vs Response Time * Key pages have high bounce rates which correlates with high T_Page as well * Significant difference ( up to 9secs) in T_Page between bounced and non-bounced for landing pages. Note: Load Time here is the time taken from head of the page to page ready (T_Page) Page Performance & Site Conversion – Feb 2012
  • 41. 41 Bounce rates @ department level…. Department Bounce Rate Vs Response Time * High T_Page for key pages (up to 19.82s) and key department making Bounce rate significantly higher Note: Load Time here is the time taken from head of the page to page ready (T_Page) Page Performance & Site Conversion – Feb 2012
  • 42. 42 Agenda • Phase 1 – Baseline Measurement - Impact of Site Performance on Conversion, Bounce rates & Revenue • Phase 2 - Targets for Page Performance • Phase 3 – Optimization Results • Key Highlights & Takeaways Page Performance & Site Conversion – Feb 2012
  • 43. 43 Phase 2 – Targets for Page Performance…… Conversion & Bounce Rate Impacts Drives Prioritization Note: Load Time here is the time taken from head of the page to page ready (T_Page) Page Performance & Site Conversion – Feb 2012
  • 44. 44 Agenda • Phase 1 – Baseline Measurement - Impact of Site Performance on Conversion, Bounce rates & Revenue • Phase 2 - Targets for Page Performance • Phase 3 – Optimization Results • Key Highlights & Takeaways Page Performance & Site Conversion – Feb 2012
  • 45. 45 Phase 3 – Success Story…. First Win….and yes conversion had positive improvements… Page Performance & Site Conversion – Feb 2012
  • 46. 46 • Page speed matters for site conversion! • Monitor real user performance in a “Big Data” way!! • Every 1s improvement = Up to 2% increase in CVR • 100ms improvement = Up to 1% incremental revenue • SEO benefits for entry pages and reduce bounces • Test & Learn - Target segments and run A/B Tests focused on improving page performance Key Highlights Page Performance & Site Conversion – Feb 2012
  • 47. We’re Hiring for Everything!!!! 47 Source: Flickr http://www.walmartlabs.com/open-positions/ https://walmartstores.com/careers/apply/?ba=eCom @cliffcrocker or @GoFastWeb