Web Performance Culture and Tools at Etsy

Web PerformanceCulture and Toolsat EtsyMike BrittainDir. of Engineering, InfrastructureEtsyBoston Web Performance MeetupAug 24, 2011

OverviewEtsy and EngineeringMake PerformanceMatterTools and Process

http://qacreate.etsy.com

MonthlyWeather Report$38 MM in sales1.9 MM items soldhttp://etsy.me/weather-report-june-2011

MonthlyWeather Report990 MM page viewshttp://etsy.me/weather-report-june-2011

EngineeringTechnologiesLinux, Apache, MySQL,PHP, MemcacheSolr, Squid, Hadoop, Amazon S3, EC2, EMRGanglia, Cacti, Nagios, Graphite, Splunk, andsome of our own...

EngineeringTeams90 Engineers3-6 engineers+ product developer and/or designer

EngineeringContinuousDeployment~40 releases per day inc. app code and conﬁg changes

1. Write code2. Code review3. Automated tests

1. Write code2. Code review3. Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod

1. Write code2. Code review3. Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod5. Monitor!6. Monitor!7. Monitor!

EngineeringData-DrivenDevelopment45,000+ metrics50+ dashboards

MakePerformanceMatterHave a story

Business ImpactMeasure“Our own years of testing have conclusively shown that when speed of a feature or product improves, usage, quite simply, goes up.” ~GoogleAbundance of research from Google, Bing,AOL, Amazon, Shopzilla, etc.http://googleblog.blogspot.com/2009/12/this-week-in-search-121809.html

Business ImpactMeasureBounce rateSearch conversion

Business ImpactMeasurePurchase funnelAd impressions, page views andtracking discrepanciesSocial, engagement

OperationsSite StabilityMeasure perf for discrete piecesof your infrastructureContention for shared resources likedatabase, memcache, solr, or even webserver processes

MoraleHappy Engineers

Getting to workWhere toStart?Focus your eﬀorts where it makessense for your business

Getting to workWhere toOptimize?Tiers of serviceSLAs

Getting to workFocusBeware of CTS:Constant Tweaking Syndrome

Getting to workFocusFriends don’t let friendstweak without graphing

MakePerformanceMatterHave your story

Tools andProcess atEtsy

ProcessEtsy’sPerf TeamStandardize patternsCreate tools and reportsCoordinate eﬀorts

ProcessServer-sidePerformanceThere is nothing worsethan a blank page or a spinningthing-a-ma-jigger

ProcessServer-sidePerformance95th Percentile> 800 ms is a P2 bug

ProcessServer-sidePerformance

ToolsLoggingPerf-related data belongs inyour server logsapache_note()

ToolsLogging$timer_start = microtime(true); ...$timer_diff = microtime(true) - $timer_start;register_shutdown_function()apache_note(php_microsec, $timer_diff);

ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined

ToolsLoggingweb0060 66.249.71.110 - - [24/Aug/2011:04:16:52+0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - 13399576 505780554876

ToolsAnalyzegrep ... access.log | awk ...grep "/listing/" access.log | awk {sum=sum+$(NF-1)} END {print sum/NR}grep "/listing/" access.log | awk BEGIN {max=0} {if ($(NF-1)>max) max=$(NF-1)}END {print max}

ToolsAnalyze

ToolsAnalyze1. Capture perf data in logs page gen., Boomerang, Gomez2. Aggregate Splunk, Logster, StatsD3. Record in Graphite http://graphite.wikidot.com

ToolsAnalyze

ToolsLogster1. Read new log entries every minute2. Parse and aggregate useful numbers3. Send to Graphite http://github.com/etsy

web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] Youve been eaten by a gweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] Youve been eaten by a gweb0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo

Fatals Errors Warnings

ToolsStatsDCollects metrics from your appcode and sticks them in GraphiteStatsD::increment("logins.success");StatsD::timing("gearman.time", $msec);

ToolsStatsD 90th pct average lower

:) :)

ToolsPerf Dashboard

ToolsI/O ProﬁlerLightweight, inline proﬁlingStart and end times wrappedaround service calls databases, memcache, apc, etc.

ToolsI/O Proﬁler

ToolsClient-sideTestingGomez (API), Boomerang,WebPagetest

ToolsShowSlowTracks front-end best practices

ToolsShowSlowInternal instanceAutomated testingEnd-point for I/O ProﬁlerTrending on individual rules http://showslow.com

Other thoughtsYSlow &Page SpeedMind your development process3rd-party content and ga.js“Use a CDN”

Other thoughtsDevice-SpeciﬁcDesignThe mobile web is very muchabout designing for performanceScreen size, pixel density, connection speed,wi-ﬁ vs. cellular, browser cache size, localstorage, connections per host, meteredpricing, etc.

MakePerformanceMatterHave your storyUse tools and processto focus your eﬀorts

We areHiringhttp://etsy.com/careersSoftware Engineering positions available ina number of teams, including Analytics,Operations, Web Performance, Payments,Core Platform, Front-End, Internal Apps,Search, Security, and more...

ThankYouMike Brittainmike@etsy.com@mikebrittainCodeAsCraft.etsy.comgithub.com/etsygraphite.wikidot.comshowslow.com