Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Web PerformanceCulture and Toolsat EtsyMike BrittainDir. of Engineering, InfrastructureEtsyBoston Web Performance MeetupAu...
OverviewEtsy and EngineeringMake PerformanceMatterTools and Process
http://qacreate.etsy.com
MonthlyWeather Report$38 MM in sales1.9 MM items soldhttp://etsy.me/weather-report-june-2011
MonthlyWeather Report990 MM page viewshttp://etsy.me/weather-report-june-2011
EngineeringTechnologiesLinux, Apache, MySQL,PHP, MemcacheSolr, Squid, Hadoop, Amazon S3, EC2, EMRGanglia, Cacti, Nagios, G...
EngineeringTeams90 Engineers3-6 engineers+ product developer and/or designer
EngineeringContinuousDeployment~40 releases per day inc. app code and config changes
1. Write code2. Code review3. Automated tests
1. Write code2. Code review3. Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod
1. Write code2. Code review3. Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod5. Monitor!6. Monitor!7. Monitor!
EngineeringData-DrivenDevelopment45,000+ metrics50+ dashboards
MakePerformanceMatterHave a story
Business ImpactMeasure“Our own years of testing have conclusively shown that when speed of a feature or product improves, ...
Business ImpactMeasureBounce rateSearch conversion
Business ImpactMeasurePurchase funnelAd impressions, page views andtracking discrepanciesSocial, engagement
OperationsSite StabilityMeasure perf for discrete piecesof your infrastructureContention for shared resources likedatabase...
MoraleHappy Engineers
Getting to workWhere toStart?Focus your efforts where it makessense for your business
Getting to workWhere toOptimize?Tiers of serviceSLAs
Getting to workFocusBeware of CTS:Constant Tweaking Syndrome
Getting to workFocusFriends don’t let friendstweak without graphing
MakePerformanceMatterHave your story
Tools andProcess atEtsy
ProcessEtsy’sPerf TeamStandardize patternsCreate tools and reportsCoordinate efforts
ProcessServer-sidePerformanceThere is nothing worsethan a blank page or a spinningthing-a-ma-jigger
ProcessServer-sidePerformance95th Percentile> 800 ms is a P2 bug
ProcessServer-sidePerformance
ToolsLoggingPerf-related data belongs inyour server logsapache_note()
ToolsLogging$timer_start = microtime(true); ...$timer_diff = microtime(true) - $timer_start;register_shutdown_function()ap...
ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n...
ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n...
ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n...
ToolsLoggingweb0060 66.249.71.110 - - [24/Aug/2011:04:16:52+0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-...
ToolsAnalyzegrep ... access.log | awk ...grep "/listing/" access.log | awk {sum=sum+$(NF-1)} END {print sum/NR}grep "/list...
ToolsAnalyze
ToolsAnalyze1. Capture perf data in logs page gen., Boomerang, Gomez2. Aggregate Splunk, Logster, StatsD3. Record in Gra...
ToolsAnalyze
ToolsLogster1. Read new log entries every minute2. Parse and aggregate useful numbers3. Send to Graphite http...
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [warning] [client 10.10...
Fatals Errors Warnings
:)
ToolsStatsDCollects metrics from your appcode and sticks them in GraphiteStatsD::increment("logins.success");StatsD::timin...
ToolsStatsD 90th pct average lower
:) :)
ToolsPerf Dashboard
ToolsI/O ProfilerLightweight, inline profilingStart and end times wrappedaround service calls databases, memcache, apc, etc.
ToolsI/O Profiler
ToolsI/O Profiler
ToolsClient-sideTestingGomez (API), Boomerang,WebPagetest
ToolsShowSlowTracks front-end best practices
ToolsShowSlowInternal instanceAutomated testingEnd-point for I/O ProfilerTrending on individual rules http://show...
Other thoughtsYSlow &Page SpeedMind your development process3rd-party content and ga.js“Use a CDN”
Other thoughtsDevice-SpecificDesignThe mobile web is very muchabout designing for performanceScreen size, pixel density, co...
MakePerformanceMatterHave your storyUse tools and processto focus your efforts
We areHiringhttp://etsy.com/careersSoftware Engineering positions available ina number of teams, including Analytics,Opera...
ThankYouMike Brittainmike@etsy.com@mikebrittainCodeAsCraft.etsy.comgithub.com/etsygraphite.wikidot.comshowslow.com
Web Performance Culture and Tools at Etsy
Upcoming SlideShare
Loading in …5
×

Web Performance Culture and Tools at Etsy

20,215 views

Published on

This talk covers some of the tools that Etsy uses for measuring performance, how to instill a culture of performance, how Etsy tracks performance wins and regressions, and where to get started if you don't have a formalized performance team in your company.

Originally presented at the Boston Web Performance Meetup on Aug 24, 2011.

Published in: Technology
no profile picture user

  • Login to see the comments

Show More

Web Performance Culture and Tools at Etsy

  1. Web PerformanceCulture and Toolsat EtsyMike BrittainDir. of Engineering, InfrastructureEtsyBoston Web Performance MeetupAug 24, 2011
  2. OverviewEtsy and EngineeringMake PerformanceMatterTools and Process
  3. http://qacreate.etsy.com
  4. MonthlyWeather Report$38 MM in sales1.9 MM items soldhttp://etsy.me/weather-report-june-2011
  5. MonthlyWeather Report990 MM page viewshttp://etsy.me/weather-report-june-2011
  6. EngineeringTechnologiesLinux, Apache, MySQL,PHP, MemcacheSolr, Squid, Hadoop, Amazon S3, EC2, EMRGanglia, Cacti, Nagios, Graphite, Splunk, andsome of our own...
  7. EngineeringTeams90 Engineers3-6 engineers+ product developer and/or designer
  8. EngineeringContinuousDeployment~40 releases per day inc. app code and config changes
  9. 1. Write code2. Code review3. Automated tests
  10. 1. Write code2. Code review3. Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod
  11. 1. Write code2. Code review3. Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod5. Monitor!6. Monitor!7. Monitor!
  12. EngineeringData-DrivenDevelopment45,000+ metrics50+ dashboards
  13. MakePerformanceMatterHave a story
  14. Business ImpactMeasure“Our own years of testing have conclusively shown that when speed of a feature or product improves, usage, quite simply, goes up.” ~GoogleAbundance of research from Google, Bing,AOL, Amazon, Shopzilla, etc.http://googleblog.blogspot.com/2009/12/this-week-in-search-121809.html
  15. Business ImpactMeasureBounce rateSearch conversion
  16. Business ImpactMeasurePurchase funnelAd impressions, page views andtracking discrepanciesSocial, engagement
  17. OperationsSite StabilityMeasure perf for discrete piecesof your infrastructureContention for shared resources likedatabase, memcache, solr, or even webserver processes
  18. MoraleHappy Engineers
  19. Getting to workWhere toStart?Focus your efforts where it makessense for your business
  20. Getting to workWhere toOptimize?Tiers of serviceSLAs
  21. Getting to workFocusBeware of CTS:Constant Tweaking Syndrome
  22. Getting to workFocusFriends don’t let friendstweak without graphing
  23. MakePerformanceMatterHave your story
  24. Tools andProcess atEtsy
  25. ProcessEtsy’sPerf TeamStandardize patternsCreate tools and reportsCoordinate efforts
  26. ProcessServer-sidePerformanceThere is nothing worsethan a blank page or a spinningthing-a-ma-jigger
  27. ProcessServer-sidePerformance95th Percentile> 800 ms is a P2 bug
  28. ProcessServer-sidePerformance
  29. ToolsLoggingPerf-related data belongs inyour server logsapache_note()
  30. ToolsLogging$timer_start = microtime(true); ...$timer_diff = microtime(true) - $timer_start;register_shutdown_function()apache_note(php_microsec, $timer_diff);
  31. ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined
  32. ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined
  33. ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined
  34. ToolsLoggingweb0060 66.249.71.110 - - [24/Aug/2011:04:16:52+0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - 13399576 505780554876
  35. ToolsAnalyzegrep ... access.log | awk ...grep "/listing/" access.log | awk {sum=sum+$(NF-1)} END {print sum/NR}grep "/listing/" access.log | awk BEGIN {max=0} {if ($(NF-1)>max) max=$(NF-1)}END {print max}
  36. ToolsAnalyze
  37. ToolsAnalyze1. Capture perf data in logs page gen., Boomerang, Gomez2. Aggregate Splunk, Logster, StatsD3. Record in Graphite http://graphite.wikidot.com
  38. ToolsAnalyze
  39. ToolsLogster1. Read new log entries every minute2. Parse and aggregate useful numbers3. Send to Graphite http://github.com/etsy
  40. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] Youve been eaten by a gweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] Youve been eaten by a gweb0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
  41. Fatals Errors Warnings
  42. :)
  43. ToolsStatsDCollects metrics from your appcode and sticks them in GraphiteStatsD::increment("logins.success");StatsD::timing("gearman.time", $msec);
  44. ToolsStatsD 90th pct average lower
  45. :) :)
  46. ToolsPerf Dashboard
  47. ToolsI/O ProfilerLightweight, inline profilingStart and end times wrappedaround service calls databases, memcache, apc, etc.
  48. ToolsI/O Profiler
  49. ToolsI/O Profiler
  50. ToolsClient-sideTestingGomez (API), Boomerang,WebPagetest
  51. ToolsShowSlowTracks front-end best practices
  52. ToolsShowSlowInternal instanceAutomated testingEnd-point for I/O ProfilerTrending on individual rules http://showslow.com
  53. Other thoughtsYSlow &Page SpeedMind your development process3rd-party content and ga.js“Use a CDN”
  54. Other thoughtsDevice-SpecificDesignThe mobile web is very muchabout designing for performanceScreen size, pixel density, connection speed,wi-fi vs. cellular, browser cache size, localstorage, connections per host, meteredpricing, etc.
  55. MakePerformanceMatterHave your storyUse tools and processto focus your efforts
  56. We areHiringhttp://etsy.com/careersSoftware Engineering positions available ina number of teams, including Analytics,Operations, Web Performance, Payments,Core Platform, Front-End, Internal Apps,Search, Security, and more...
  57. ThankYouMike Brittainmike@etsy.com@mikebrittainCodeAsCraft.etsy.comgithub.com/etsygraphite.wikidot.comshowslow.com

×