Web Performance Culture and Tools at Etsy
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Web Performance Culture and Tools at Etsy

  • 9,427 views
Uploaded on

This talk covers some of the tools that Etsy uses for measuring performance, how to instill a culture of performance, how Etsy tracks performance wins and regressions, and where to get started if......

This talk covers some of the tools that Etsy uses for measuring performance, how to instill a culture of performance, how Etsy tracks performance wins and regressions, and where to get started if you don't have a formalized performance team in your company.

Originally presented at the Boston Web Performance Meetup on Aug 24, 2011.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
9,427
On Slideshare
7,848
From Embeds
1,579
Number of Embeds
12

Actions

Shares
Downloads
113
Comments
0
Likes
20

Embeds 1,579

http://www.mikebrittain.com 1,517
http://www.linkedin.com 29
http://socialwebkurs.wikispaces.com 12
http://twitter.com 7
http://us-w1.rockmelt.com 5
http://coderwall.com 3
http://translate.googleusercontent.com 1
http://webcache.googleusercontent.com 1
http://www.mikebrittain.com. 1
http://www.slideshare.net 1
http://www.onlydoo.com 1
https://twitter.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Web PerformanceCulture and Toolsat EtsyMike BrittainDir. of Engineering, InfrastructureEtsyBoston Web Performance MeetupAug 24, 2011
  • 2. OverviewEtsy and EngineeringMake PerformanceMatterTools and Process
  • 3. http://qacreate.etsy.com
  • 4. MonthlyWeather Report$38 MM in sales1.9 MM items soldhttp://etsy.me/weather-report-june-2011
  • 5. MonthlyWeather Report990 MM page viewshttp://etsy.me/weather-report-june-2011
  • 6. EngineeringTechnologiesLinux, Apache, MySQL,PHP, MemcacheSolr, Squid, Hadoop, Amazon S3, EC2, EMRGanglia, Cacti, Nagios, Graphite, Splunk, andsome of our own...
  • 7. EngineeringTeams90 Engineers3-6 engineers+ product developer and/or designer
  • 8. EngineeringContinuousDeployment~40 releases per day inc. app code and config changes
  • 9. 1. Write code2. Code review3. Automated tests
  • 10. 1. Write code2. Code review3. Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod
  • 11. 1. Write code2. Code review3. Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod5. Monitor!6. Monitor!7. Monitor!
  • 12. EngineeringData-DrivenDevelopment45,000+ metrics50+ dashboards
  • 13. MakePerformanceMatterHave a story
  • 14. Business ImpactMeasure“Our own years of testing have conclusively shown that when speed of a feature or product improves, usage, quite simply, goes up.” ~GoogleAbundance of research from Google, Bing,AOL, Amazon, Shopzilla, etc.http://googleblog.blogspot.com/2009/12/this-week-in-search-121809.html
  • 15. Business ImpactMeasureBounce rateSearch conversion
  • 16. Business ImpactMeasurePurchase funnelAd impressions, page views andtracking discrepanciesSocial, engagement
  • 17. OperationsSite StabilityMeasure perf for discrete piecesof your infrastructureContention for shared resources likedatabase, memcache, solr, or even webserver processes
  • 18. MoraleHappy Engineers
  • 19. Getting to workWhere toStart?Focus your efforts where it makessense for your business
  • 20. Getting to workWhere toOptimize?Tiers of serviceSLAs
  • 21. Getting to workFocusBeware of CTS:Constant Tweaking Syndrome
  • 22. Getting to workFocusFriends don’t let friendstweak without graphing
  • 23. MakePerformanceMatterHave your story
  • 24. Tools andProcess atEtsy
  • 25. ProcessEtsy’sPerf TeamStandardize patternsCreate tools and reportsCoordinate efforts
  • 26. ProcessServer-sidePerformanceThere is nothing worsethan a blank page or a spinningthing-a-ma-jigger
  • 27. ProcessServer-sidePerformance95th Percentile> 800 ms is a P2 bug
  • 28. ProcessServer-sidePerformance
  • 29. ToolsLoggingPerf-related data belongs inyour server logsapache_note()
  • 30. ToolsLogging$timer_start = microtime(true); ...$timer_diff = microtime(true) - $timer_start;register_shutdown_function()apache_note(php_microsec, $timer_diff);
  • 31. ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined
  • 32. ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined
  • 33. ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined
  • 34. ToolsLoggingweb0060 66.249.71.110 - - [24/Aug/2011:04:16:52+0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - 13399576 505780554876
  • 35. ToolsAnalyzegrep ... access.log | awk ...grep "/listing/" access.log | awk {sum=sum+$(NF-1)} END {print sum/NR}grep "/listing/" access.log | awk BEGIN {max=0} {if ($(NF-1)>max) max=$(NF-1)}END {print max}
  • 36. ToolsAnalyze
  • 37. ToolsAnalyze1. Capture perf data in logs page gen., Boomerang, Gomez2. Aggregate Splunk, Logster, StatsD3. Record in Graphite http://graphite.wikidot.com
  • 38. ToolsAnalyze
  • 39. ToolsLogster1. Read new log entries every minute2. Parse and aggregate useful numbers3. Send to Graphite http://github.com/etsy
  • 40. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] Youve been eaten by a gweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] Youve been eaten by a gweb0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
  • 41. Fatals Errors Warnings
  • 42. :)
  • 43. ToolsStatsDCollects metrics from your appcode and sticks them in GraphiteStatsD::increment("logins.success");StatsD::timing("gearman.time", $msec);
  • 44. ToolsStatsD 90th pct average lower
  • 45. :) :)
  • 46. ToolsPerf Dashboard
  • 47. ToolsI/O ProfilerLightweight, inline profilingStart and end times wrappedaround service calls databases, memcache, apc, etc.
  • 48. ToolsI/O Profiler
  • 49. ToolsI/O Profiler
  • 50. ToolsClient-sideTestingGomez (API), Boomerang,WebPagetest
  • 51. ToolsShowSlowTracks front-end best practices
  • 52. ToolsShowSlowInternal instanceAutomated testingEnd-point for I/O ProfilerTrending on individual rules http://showslow.com
  • 53. Other thoughtsYSlow &Page SpeedMind your development process3rd-party content and ga.js“Use a CDN”
  • 54. Other thoughtsDevice-SpecificDesignThe mobile web is very muchabout designing for performanceScreen size, pixel density, connection speed,wi-fi vs. cellular, browser cache size, localstorage, connections per host, meteredpricing, etc.
  • 55. MakePerformanceMatterHave your storyUse tools and processto focus your efforts
  • 56. We areHiringhttp://etsy.com/careersSoftware Engineering positions available ina number of teams, including Analytics,Operations, Web Performance, Payments,Core Platform, Front-End, Internal Apps,Search, Security, and more...
  • 57. ThankYouMike Brittainmike@etsy.com@mikebrittainCodeAsCraft.etsy.comgithub.com/etsygraphite.wikidot.comshowslow.com