Web Performance Culture and Tools
at Etsy Mike Brittain Dir. of Engineering, Infrastructure Etsy Boston Web Performance Meetup Aug 24, 2011

Overview Etsy and Engineering Make
Performance Matter Tools and Process

http://qacreate.etsy.com

Monthly Weather Report $38 MM
in sales 1.9 MM items sold http://etsy.me/weather-report-june-2011

Monthly Weather Report 990 MM
page views http://etsy.me/weather-report-june-2011

Engineering Technologies Linux, Apache, MySQL,
PHP, Memcache Solr, Squid, Hadoop, Amazon S3, EC2, EMR Ganglia, Cacti, Nagios, Graphite, Splunk, and some of our own...

Engineering Teams 90 Engineers 3-6
engineers + product developer and/or designer

Engineering Continuous Deployment ~40 releases
per day inc. app code and config changes

1. Write code 2. Code
review 3. Automated tests

1. Write code 2. Code
review 3. Automated tests 4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod

1. Write code 2. Code
review 3. Automated tests 4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod 5. Monitor! 6. Monitor! 7. Monitor!

Engineering Data-Driven Development 45,000+ metrics
50+ dashboards

Make Performance Matter Have a
story

Business Impact Measure “Our own
years of testing have conclusively shown that when speed of a feature or product improves, usage, quite simply, goes up.” ~Google Abundance of research from Google, Bing, AOL, Amazon, Shopzilla, etc. http://googleblog.blogspot.com/2009/12/this-week-in-search-121809.html

Business Impact Measure Bounce rate
Search conversion

Business Impact Measure Purchase funnel
Ad impressions, page views and tracking discrepancies Social, engagement

Operations Site Stability Measure perf
for discrete pieces of your infrastructure Contention for shared resources like database, memcache, solr, or even web server processes

Morale
Happy Engineers

Getting to work Where to
Start? Focus your efforts where it makes sense for your business

Getting to work Where to
Optimize? Tiers of service SLAs

Getting to work Focus Beware
of CTS: Constant Tweaking Syndrome

Getting to work Focus Friends
don’t let friends tweak without graphing

Make Performance Matter Have your
story

Tools and
Process at
Etsy

Process Etsy’s Perf Team Standardize
patterns Create tools and reports Coordinate efforts

Process Server-side Performance There is
nothing worse than a blank page or a spinning thing-a-ma-jigger

Process Server-side Performance 95th Percentile
> 800 ms is a P2 bug

Process
Server-side
Performance

Tools Logging Perf-related data belongs
in your server logs apache_note()

Tools Logging $timer_start = microtime(true);
... $timer_diff = microtime(true) - $timer_start; register_shutdown_function() apache_note('php_microsec', $timer_diff);

Tools Logging LogFormat "%{True-Client-IP}i %l
%u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %{etsy_user_id}n %{php_bytes}n %{php_microsec}n %D" combined

Tools Logging LogFormat "%{True-Client-IP}i %l
%u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %{etsy_user_id}n %{php_bytes}n %{php_microsec}n %D" combined

Tools Logging LogFormat "%{True-Client-IP}i %l
%u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %{etsy_user_id}n %{php_bytes}n %{php_microsec}n %D" combined

Tools Logging web0060 66.249.71.110 -
- [24/Aug/2011:04:16:52 +0000] "GET /listing/12189259/tropical-etched- pair-of-lampwork-glass HTTP/1.1" 200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http:// www.google.com/bot.html)" - 13399576 505780 554876

Tools Analyze grep ... access.log
| awk ... grep "/listing/" access.log | awk '{sum=sum+$(NF-1)} END {print sum/NR}' grep "/listing/" access.log | awk 'BEGIN {max=0} {if ($(NF-1)>max) max=$(NF-1)} END {print max}'

Tools
Analyze

Tools Analyze 1. Capture perf
data in logs page gen., Boomerang, Gomez 2. Aggregate Splunk, Logster, StatsD 3. Record in Graphite http://graphite.wikidot.com

Tools
Analyze

Tools Logster 1. Read new
log entries every minute 2. Parse and aggregate useful numbers 3. Send to Graphite http://github.com/etsy

web0001 [04:28:54 2011] [error] [client
10.101.x.x] Oh noooooo! web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo! web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh! web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllpp web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a g web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo! web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a g web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!! web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling. web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling. web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling. web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo! web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling. web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo

Fatals Errors Warnings

:)

Tools StatsD Collects metrics from
your app code and sticks them in Graphite StatsD::increment("logins.success"); StatsD::timing("gearman.time", $msec);

Tools StatsD 90th pct average
lower

:) :)

Tools
Perf Dashboard

Tools I/O Profiler Lightweight, inline
profiling Start and end times wrapped around service calls databases, memcache, apc, etc.

Tools
I/O Profiler

Tools
I/O Profiler

Tools Client-side Testing Gomez (API),
Boomerang, WebPagetest

Tools ShowSlow Tracks front-end best
practices

Tools ShowSlow Internal instance Automated
testing End-point for I/O Profiler Trending on individual rules http://showslow.com

Other thoughts YSlow & Page
Speed Mind your development process 3rd-party content and ga.js “Use a CDN”

Other thoughts Device-Specific Design The
mobile web is very much about designing for performance Screen size, pixel density, connection speed, wi-fi vs. cellular, browser cache size, local storage, connections per host, metered pricing, etc.

Make Performance Matter Have your
story Use tools and process to focus your efforts

We are Hiring http://etsy.com/careers Software
Engineering positions available in a number of teams, including Analytics, Operations, Web Performance, Payments, Core Platform, Front-End, Internal Apps, Search, Security, and more...

Thank You Mike Brittain mike@etsy.com
@mikebrittain CodeAsCraft.etsy.com github.com/etsy graphite.wikidot.com showslow.com