Web Performance Culture and Tools at Etsy

  • 8,903 views
Uploaded on

This talk covers some of the tools that Etsy uses for measuring performance, how to instill a culture of performance, how Etsy tracks performance wins and regressions, and where to get started if you …

This talk covers some of the tools that Etsy uses for measuring performance, how to instill a culture of performance, how Etsy tracks performance wins and regressions, and where to get started if you don't have a formalized performance team in your company.

Originally presented at the Boston Web Performance Meetup on Aug 24, 2011.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
8,903
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
114
Comments
0
Likes
20

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Web PerformanceCulture and Toolsat EtsyMike BrittainDir. of Engineering, InfrastructureEtsyBoston Web Performance MeetupAug 24, 2011
  • 2. OverviewEtsy and EngineeringMake PerformanceMatterTools and Process
  • 3. http://qacreate.etsy.com
  • 4. MonthlyWeather Report$38 MM in sales1.9 MM items soldhttp://etsy.me/weather-report-june-2011
  • 5. MonthlyWeather Report990 MM page viewshttp://etsy.me/weather-report-june-2011
  • 6. EngineeringTechnologiesLinux, Apache, MySQL,PHP, MemcacheSolr, Squid, Hadoop, Amazon S3, EC2, EMRGanglia, Cacti, Nagios, Graphite, Splunk, andsome of our own...
  • 7. EngineeringTeams90 Engineers3-6 engineers+ product developer and/or designer
  • 8. EngineeringContinuousDeployment~40 releases per day inc. app code and config changes
  • 9. 1. Write code2. Code review3. Automated tests
  • 10. 1. Write code2. Code review3. Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod
  • 11. 1. Write code2. Code review3. Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod5. Monitor!6. Monitor!7. Monitor!
  • 12. EngineeringData-DrivenDevelopment45,000+ metrics50+ dashboards
  • 13. MakePerformanceMatterHave a story
  • 14. Business ImpactMeasure“Our own years of testing have conclusively shown that when speed of a feature or product improves, usage, quite simply, goes up.” ~GoogleAbundance of research from Google, Bing,AOL, Amazon, Shopzilla, etc.http://googleblog.blogspot.com/2009/12/this-week-in-search-121809.html
  • 15. Business ImpactMeasureBounce rateSearch conversion
  • 16. Business ImpactMeasurePurchase funnelAd impressions, page views andtracking discrepanciesSocial, engagement
  • 17. OperationsSite StabilityMeasure perf for discrete piecesof your infrastructureContention for shared resources likedatabase, memcache, solr, or even webserver processes
  • 18. MoraleHappy Engineers
  • 19. Getting to workWhere toStart?Focus your efforts where it makessense for your business
  • 20. Getting to workWhere toOptimize?Tiers of serviceSLAs
  • 21. Getting to workFocusBeware of CTS:Constant Tweaking Syndrome
  • 22. Getting to workFocusFriends don’t let friendstweak without graphing
  • 23. MakePerformanceMatterHave your story
  • 24. Tools andProcess atEtsy
  • 25. ProcessEtsy’sPerf TeamStandardize patternsCreate tools and reportsCoordinate efforts
  • 26. ProcessServer-sidePerformanceThere is nothing worsethan a blank page or a spinningthing-a-ma-jigger
  • 27. ProcessServer-sidePerformance95th Percentile> 800 ms is a P2 bug
  • 28. ProcessServer-sidePerformance
  • 29. ToolsLoggingPerf-related data belongs inyour server logsapache_note()
  • 30. ToolsLogging$timer_start = microtime(true); ...$timer_diff = microtime(true) - $timer_start;register_shutdown_function()apache_note(php_microsec, $timer_diff);
  • 31. ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined
  • 32. ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined
  • 33. ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined
  • 34. ToolsLoggingweb0060 66.249.71.110 - - [24/Aug/2011:04:16:52+0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - 13399576 505780554876
  • 35. ToolsAnalyzegrep ... access.log | awk ...grep "/listing/" access.log | awk {sum=sum+$(NF-1)} END {print sum/NR}grep "/listing/" access.log | awk BEGIN {max=0} {if ($(NF-1)>max) max=$(NF-1)}END {print max}
  • 36. ToolsAnalyze
  • 37. ToolsAnalyze1. Capture perf data in logs page gen., Boomerang, Gomez2. Aggregate Splunk, Logster, StatsD3. Record in Graphite http://graphite.wikidot.com
  • 38. ToolsAnalyze
  • 39. ToolsLogster1. Read new log entries every minute2. Parse and aggregate useful numbers3. Send to Graphite http://github.com/etsy
  • 40. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] Youve been eaten by a gweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] Youve been eaten by a gweb0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
  • 41. Fatals Errors Warnings
  • 42. :)
  • 43. ToolsStatsDCollects metrics from your appcode and sticks them in GraphiteStatsD::increment("logins.success");StatsD::timing("gearman.time", $msec);
  • 44. ToolsStatsD 90th pct average lower
  • 45. :) :)
  • 46. ToolsPerf Dashboard
  • 47. ToolsI/O ProfilerLightweight, inline profilingStart and end times wrappedaround service calls databases, memcache, apc, etc.
  • 48. ToolsI/O Profiler
  • 49. ToolsI/O Profiler
  • 50. ToolsClient-sideTestingGomez (API), Boomerang,WebPagetest
  • 51. ToolsShowSlowTracks front-end best practices
  • 52. ToolsShowSlowInternal instanceAutomated testingEnd-point for I/O ProfilerTrending on individual rules http://showslow.com
  • 53. Other thoughtsYSlow &Page SpeedMind your development process3rd-party content and ga.js“Use a CDN”
  • 54. Other thoughtsDevice-SpecificDesignThe mobile web is very muchabout designing for performanceScreen size, pixel density, connection speed,wi-fi vs. cellular, browser cache size, localstorage, connections per host, meteredpricing, etc.
  • 55. MakePerformanceMatterHave your storyUse tools and processto focus your efforts
  • 56. We areHiringhttp://etsy.com/careersSoftware Engineering positions available ina number of teams, including Analytics,Operations, Web Performance, Payments,Core Platform, Front-End, Internal Apps,Search, Security, and more...
  • 57. ThankYouMike Brittainmike@etsy.com@mikebrittainCodeAsCraft.etsy.comgithub.com/etsygraphite.wikidot.comshowslow.com