Your SlideShare is downloading. ×
0
Metrics-DrivenEngineeringMike BrittainENGINEERING DIRECTOR, ETSY@mikebrittain
PROCESSANDTOOLSSupportingacultureofContinuousDeployment
Howmanynewvisitors?Howmanylistingscreated?Howmanyregistrations?HowdopeopleuseEtsy?Howmanymessagessent?Howmanypurchases?How...
Searchindexing?Howfastarepagesgenerating?Asynctaskscurrentlyinqueue?Howistheapplicationbehaving?DeveloperAPIauthandratelim...
Replicationslavelag?Memcachehits/misses?Availableconnections?AretheserversandnetworkOK?Databasequeriespersecond?Totaloutgo...
BusinessMetrics
ApplicationMetrics
SystemMetrics
SystemMetrics
VisibilityEVERYWHERE
Metricshelpyouidentifygoals
Metricshelpyouidentifygoals...butalsotellyouwhenyou’vebrokensomething.
AlwaysBeShippingcredit: ibailemon (flickr)
1st dayPutyourselfonthewebsite.
2nd dayCompletetax,insurance,andbenefitsforms.credit: ktpupp (flickr)
DevSandbox Trunk/master ProductionYou!Test
7e9a814 -> 63a2bb3Deploy to Production
50+Deploys/day200+Committers15Productteams8Infrastructureteams50+Deploys/daycredit: misswired (flickr)
credit: digidave (flickr)
PeerReviewCodereviews,Architecturereviews,Operabilityreviews
AutomatedTestsStaticanalysis,Unittests,Integrationtests,Functionaltests
May2013$102.9Millioningoodsold1.37Billionpageviewshttps://www.etsy.com/blog/news/2013/etsy-statistics-may-2013-weather-rep...
Failureisnotanoption
Failureisnotanoptioninevitable
Failureisnotanoptionanddetectable!inevitable
Access
Soundslikealotofwork,who’sgoingtobuildallofthis?Q:
Well,theOpsteammanagesthenetwork,rackstheservers,installedthemonitoringtools,wearsthepagers,blah,blah,blah...A:Soundslikea...
Engineersbuildtheapplication
OPSLoggingGraphingTrendingAlertingENG
Metricsarepartofeveryfeature(andsoareconfigflags)
MakeitDEADSIMPLE
Ganglia (application,servers,network)Logster* (application,servers)Cacti (network,SNMP)FITB* (network)*github.com/etsySimp...
Ganglia
Cluster-orientedHugecommunitycontributedrecipesCustommetrics(gmetad)Ganglia
Graphite
Single-instanceCreatenewmetricson-the-flyCustomizeviaURLsanddisplayfunctionshttp://www.aosabook.org/en/graphite.htmlGraphite
LogFormats
Time, remote address, http method, request uri, referrer, user-agent,response size, response code, execution time, memory ...
LogFormat %l %t "%r" %>s %b "%{Referer}i""%{User-Agent}i" %{custom_field}n ...apache_note(“custom_field”, $whatever);
LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b"%{Referer}i" "%{User-Agent}i"%{display_mode}n %{user_id}n %{php_bytes}n...
LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b"%{Referer}i" "%{User-Agent}i"%{display_mode}n %{user_id}n %{php_bytes}n...
Logger::error("User login failed. Reason:$msg for $email_addr", “login”);Methodnamedenoteslog“level”—error,fatal,warning,n...
Logger::error("User login failed. Reason:$msg for $email_addr", “login”);web0054 [Fri Mar 04 16:27:48 2011] [error] [login...
web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] Invalid charset conveweb0102 [Fri Mar 04 16:27:48 2011] [w...
web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] Invalid charset conveweb0102 [Fri Mar 04 16:27:48 2011] [w...
web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] Invalid charset conveweb0102 [Fri Mar 04 16:27:48 2011] [w...
github.com/etsy/logsterRunbycron(e.g.1mintervals)Keeps acursoronyourlogfileParseandaggregatevalueshoweveryouwantOutputtoGan...
web0054 [Fri Mar 04 16:27:48 2011] [error] [login][mk04gw1p71] User login failed. Reason: wrongpassword was submitted for ...
if (fields[log_level] == “fatal”):self.fatals += 1elif (fields[log_level] == “error”):self.errors += 1elif (fields[log_lev...
MetricObject("fatals",(self.fatals / self.duration), "per sec")MetricObject("errors",(self.errors / self.duration), "per s...
github.com/etsy/logsterFATALS ERRORS WARNINGSLogster
StatsD
github.com/etsy/statsdStatsDNetworkdaemon(node.js)AcceptsdataoverUDPFlushestoGraphiteevery10secOne-lineofcode
StatsD::increment("logins.success");
StatsD::increment("logins.success");Logins
StatsD::timing("profile.time", $msec);
StatsD::timing("profile.time", $msec);90thpctaveragelower
Adhocname value timestamp
echo "events.deploy.site 1 `date +%s`" | nc graphite.etsycorp.com 2003
VerticalLineTechnology!target=drawAsInfinite(events.deploy.site)
User Logins
PHP Warnings
PHP Fatal Errors
250,000+metricsatEtsySystems,Applications,Business
github.com/etsy/dashboardDashboards
<a href="http://graphite.etsycorp.com/render?from=-1hours&width=800&height=600&title=File+or+Script+Not+Found&yMin=0&targe...
$g = new Graphite($time);$g->setTitle(File Not Found);$g->addMetric(webs.errorLog.notExist, #00cc00);echo $g->getDashboard...
But,yousaid...“250,000+metricsatEtsy”Systems,Applications,Business
http://graphite/render?from=-1hours&width=600&height=200&target=webs.errorLog.warning&rawData=1
http://graphite/render?from=-1hours&width=600&height=200&target=webs.errorLog.warning&rawData=1webs.errorLog.warning,13184...
Holt-WintersConfidenceBandslowerupper
Holt-WintersAberration
Businessmetrics+ Confidencebands_____________Alertablemetrics
Metrics!Metrics+EventsMetrics+AlertsMetrics+Metrics
High-level,real-timevisibility
Detectproblemsearly,andresolvethemquickly.
MakethemaccessibleMakethemrequiredfeaturesMakethemdeadsimple
Merci!These slides will be available atmikebrittain.com/talkscodeascraft.etsy.comgithub.com/etsySay“Hello!”mike@etsy.com@m...
Metrics Driven Engineering
Metrics Driven Engineering
Metrics Driven Engineering
Upcoming SlideShare
Loading in...5
×

Metrics Driven Engineering

28,700

Published on

Presented at USI 2013 in Paris, France.

In this talk I discuss how Etsy's engineering team collects and uses real-time metrics to add confidence to our Continuous Deployment culture.

Like what you've read? We're frequently hiring for a variety of engineering roles at Etsy. If you're interested, drop me a line or send me your resume: mike@etsy.com.

http://www.etsy.com/careers

Published in: Technology
0 Comments
25 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
28,700
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
79
Comments
0
Likes
25
Embeds 0
No embeds

No notes for slide

Transcript of "Metrics Driven Engineering"

  1. 1. Metrics-DrivenEngineeringMike BrittainENGINEERING DIRECTOR, ETSY@mikebrittain
  2. 2. PROCESSANDTOOLSSupportingacultureofContinuousDeployment
  3. 3. Howmanynewvisitors?Howmanylistingscreated?Howmanyregistrations?HowdopeopleuseEtsy?Howmanymessagessent?Howmanypurchases?Howmanynewshops?
  4. 4. Searchindexing?Howfastarepagesgenerating?Asynctaskscurrentlyinqueue?Howistheapplicationbehaving?DeveloperAPIauthandratelimiting?Imagesresizedandstored?Errorandwarningrates?
  5. 5. Replicationslavelag?Memcachehits/misses?Availableconnections?AretheserversandnetworkOK?Databasequeriespersecond?Totaloutgoingbandwidth?CPU,Memory,I/O?
  6. 6. BusinessMetrics
  7. 7. ApplicationMetrics
  8. 8. SystemMetrics
  9. 9. SystemMetrics
  10. 10. VisibilityEVERYWHERE
  11. 11. Metricshelpyouidentifygoals
  12. 12. Metricshelpyouidentifygoals...butalsotellyouwhenyou’vebrokensomething.
  13. 13. AlwaysBeShippingcredit: ibailemon (flickr)
  14. 14. 1st dayPutyourselfonthewebsite.
  15. 15. 2nd dayCompletetax,insurance,andbenefitsforms.credit: ktpupp (flickr)
  16. 16. DevSandbox Trunk/master ProductionYou!Test
  17. 17. 7e9a814 -> 63a2bb3Deploy to Production
  18. 18. 50+Deploys/day200+Committers15Productteams8Infrastructureteams50+Deploys/daycredit: misswired (flickr)
  19. 19. credit: digidave (flickr)
  20. 20. PeerReviewCodereviews,Architecturereviews,Operabilityreviews
  21. 21. AutomatedTestsStaticanalysis,Unittests,Integrationtests,Functionaltests
  22. 22. May2013$102.9Millioningoodsold1.37Billionpageviewshttps://www.etsy.com/blog/news/2013/etsy-statistics-may-2013-weather-report/
  23. 23. Failureisnotanoption
  24. 24. Failureisnotanoptioninevitable
  25. 25. Failureisnotanoptionanddetectable!inevitable
  26. 26. Access
  27. 27. Soundslikealotofwork,who’sgoingtobuildallofthis?Q:
  28. 28. Well,theOpsteammanagesthenetwork,rackstheservers,installedthemonitoringtools,wearsthepagers,blah,blah,blah...A:Soundslikealotofwork,who’sgoingtobuildallofthis?Q:
  29. 29. Engineersbuildtheapplication
  30. 30. OPSLoggingGraphingTrendingAlertingENG
  31. 31. Metricsarepartofeveryfeature(andsoareconfigflags)
  32. 32. MakeitDEADSIMPLE
  33. 33. Ganglia (application,servers,network)Logster* (application,servers)Cacti (network,SNMP)FITB* (network)*github.com/etsySimple,open-sourcetoolsGraphite (application)Statsd* (application)Logformats (application,servers)Nagios (alerting)
  34. 34. Ganglia
  35. 35. Cluster-orientedHugecommunitycontributedrecipesCustommetrics(gmetad)Ganglia
  36. 36. Graphite
  37. 37. Single-instanceCreatenewmetricson-the-flyCustomizeviaURLsanddisplayfunctionshttp://www.aosabook.org/en/graphite.htmlGraphite
  38. 38. LogFormats
  39. 39. Time, remote address, http method, request uri, referrer, user-agent,response size, response code, execution time, memory consumed,plus custom fields...• Signed-in/out (user_id vs. “-”)• display mode (“desktop” vs. “mobile”)• i10n/i18n (“en-US”)• etc.AccessLogs
  40. 40. LogFormat %l %t "%r" %>s %b "%{Referer}i""%{User-Agent}i" %{custom_field}n ...apache_note(“custom_field”, $whatever);
  41. 41. LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b"%{Referer}i" "%{User-Agent}i"%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1"200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876
  42. 42. LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b"%{Referer}i" "%{User-Agent}i"%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1"200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876
  43. 43. Logger::error("User login failed. Reason:$msg for $email_addr", “login”);Methodnamedenoteslog“level”—error,fatal,warning,notice,debug.A“namespace”parameterisprovidedsowecanaggregatelogentrieswithsimilarconcerns.
  44. 44. Logger::error("User login failed. Reason:$msg for $email_addr", “login”);web0054 [Fri Mar 04 16:27:48 2011] [error] [login][mk04gw1p71] User login failed. Reason: wrongpassword was submitted for mike@etsy.comUniquerequestIDServernameDateandtime LevelNamespace
  45. 45. web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] Invalid charset conveweb0102 [Fri Mar 04 16:27:48 2011] [warning] [login] [47dd608551] User login failed.web0012 [Fri Mar 04 16:27:48 2011] [warning] [login] [mk04gw1p71] User login failed.web0081 [Fri Mar 04 16:27:48 2011] [error] [register] [39e08e6692] Duplicate user IDweb0100 [Fri Mar 04 16:27:49 2011] [fatal] [register] [f9c2b23702] Invalid charset coweb0003 [Fri Mar 04 16:27:49 2011] [error] [register] [39e08e6692] Duplicate user IDweb0050 [Fri Mar 04 16:27:49 2011] [error] [register] [2e468a9bb6] Duplicate user IDweb0054 [Fri Mar 04 16:27:49 2011] [warning] [login] [mk04gw1p71] User login failed.web0200 [Fri Mar 04 16:27:49 2011] [error] [login] [f9c2b23702] User login failed. Reweb0064 [Fri Mar 04 16:27:49 2011] [error] [login] [47dd608551] Duplicate user ID encweb0012 [Fri Mar 04 16:27:49 2011] [warning] [login] [32976da59c] User login failed.web0041 [Fri Mar 04 16:27:49 2011] [fatal] [login] [mk04gw1p71] Invalid charset conveweb0012 [Fri Mar 04 16:27:49 2011] [error] [login] [2f297b40a5] User login failed. Reweb0025 [Fri Mar 04 16:27:49 2011] [warning] [register] [32976da59c] User login faileweb0088 [Fri Mar 04 16:27:49 2011] [warning] [register] [2e468a9bb6] User login faileweb0050 [Fri Mar 04 16:27:50 2011] [warning] [register] [39e08e6692] User login faileweb0035 [Fri Mar 04 16:27:50 2011] [warning] [login] [2f297b40a5] User login failed.web0072 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [2f297b40a5] User login failedweb0050 [Fri Mar 04 16:27:50 2011] [error] [login] [2e468a9bb6] User login failed. Reweb0054 [Fri Mar 04 16:27:50 2011] [warning] [login] [mk04gw1p71] User login failed.web0200 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [f9c2b23702] User login failedweb0064 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [47dd608551] Invalid charset cweb0012 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.web0041 [Fri Mar 04 16:27:50 2011] [fatal] [login] [mk04gw1p71] Invalid charset conveweb0012 [Fri Mar 04 16:27:50 2011] [error] [register] [2f297b40a5] Duplicate user IDweb0025 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.web0088 [Fri Mar 04 16:27:50 2011] [warning] [login] [2e468a9bb6] User login failed.web0050 [Fri Mar 04 16:27:51 2011] [warning] [login] [39e08e6692] User login failed.web0035 [Fri Mar 04 16:27:51 2011] [warning] [login] [2f297b40a5] User login failed.web0072 [Fri Mar 04 16:27:51 2011] [error] [login] [2f297b40a5] User login failed. Re
  46. 46. web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] Invalid charset conveweb0102 [Fri Mar 04 16:27:48 2011] [warning] [login] [47dd608551] User login failed.web0012 [Fri Mar 04 16:27:48 2011] [warning] [login] [mk04gw1p71] User login failed.web0081 [Fri Mar 04 16:27:48 2011] [error] [register] [39e08e6692] Duplicate user IDweb0100 [Fri Mar 04 16:27:49 2011] [fatal] [register] [f9c2b23702] Invalid charset coweb0003 [Fri Mar 04 16:27:49 2011] [error] [register] [39e08e6692] Duplicate user IDweb0050 [Fri Mar 04 16:27:49 2011] [error] [register] [2e468a9bb6] Duplicate user IDweb0054 [Fri Mar 04 16:27:49 2011] [warning] [login] [mk04gw1p71] User login failed.web0200 [Fri Mar 04 16:27:49 2011] [error] [login] [f9c2b23702] User login failed. Reweb0064 [Fri Mar 04 16:27:49 2011] [error] [login] [47dd608551] Duplicate user ID encweb0012 [Fri Mar 04 16:27:49 2011] [warning] [login] [32976da59c] User login failed.web0041 [Fri Mar 04 16:27:49 2011] [fatal] [login] [mk04gw1p71] Invalid charset conveweb0012 [Fri Mar 04 16:27:49 2011] [error] [login] [2f297b40a5] User login failed. Reweb0025 [Fri Mar 04 16:27:49 2011] [warning] [register] [32976da59c] User login faileweb0088 [Fri Mar 04 16:27:49 2011] [warning] [register] [2e468a9bb6] User login faileweb0050 [Fri Mar 04 16:27:50 2011] [warning] [register] [39e08e6692] User login faileweb0035 [Fri Mar 04 16:27:50 2011] [warning] [login] [2f297b40a5] User login failed.web0072 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [2f297b40a5] User login failedweb0050 [Fri Mar 04 16:27:50 2011] [error] [login] [2e468a9bb6] User login failed. Reweb0054 [Fri Mar 04 16:27:50 2011] [warning] [login] [mk04gw1p71] User login failed.web0200 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [f9c2b23702] User login failedweb0064 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [47dd608551] Invalid charset cweb0012 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.web0041 [Fri Mar 04 16:27:50 2011] [fatal] [login] [mk04gw1p71] Invalid charset conveweb0012 [Fri Mar 04 16:27:50 2011] [error] [register] [2f297b40a5] Duplicate user IDweb0025 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.web0088 [Fri Mar 04 16:27:50 2011] [warning] [login] [2e468a9bb6] User login failed.web0050 [Fri Mar 04 16:27:51 2011] [warning] [login] [39e08e6692] User login failed.web0035 [Fri Mar 04 16:27:51 2011] [warning] [login] [2f297b40a5] User login failed.web0072 [Fri Mar 04 16:27:51 2011] [error] [login] [2f297b40a5] User login failed. Re
  47. 47. web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] Invalid charset conveweb0102 [Fri Mar 04 16:27:48 2011] [warning] [login] [47dd608551] User login failed.web0012 [Fri Mar 04 16:27:48 2011] [warning] [login] [mk04gw1p71] User login failed.web0081 [Fri Mar 04 16:27:48 2011] [error] [register] [39e08e6692] Duplicate user IDweb0100 [Fri Mar 04 16:27:49 2011] [fatal] [register] [f9c2b23702] Invalid charset coweb0003 [Fri Mar 04 16:27:49 2011] [error] [register] [39e08e6692] Duplicate user IDweb0050 [Fri Mar 04 16:27:49 2011] [error] [register] [2e468a9bb6] Duplicate user IDweb0054 [Fri Mar 04 16:27:49 2011] [warning] [login] [mk04gw1p71] User login failed.web0200 [Fri Mar 04 16:27:49 2011] [error] [login] [f9c2b23702] User login failed. Reweb0064 [Fri Mar 04 16:27:49 2011] [error] [login] [47dd608551] Duplicate user ID encweb0012 [Fri Mar 04 16:27:49 2011] [warning] [login] [32976da59c] User login failed.web0041 [Fri Mar 04 16:27:49 2011] [fatal] [login] [mk04gw1p71] Invalid charset conveweb0012 [Fri Mar 04 16:27:49 2011] [error] [login] [2f297b40a5] User login failed. Reweb0025 [Fri Mar 04 16:27:49 2011] [warning] [register] [32976da59c] User login faileweb0088 [Fri Mar 04 16:27:49 2011] [warning] [register] [2e468a9bb6] User login faileweb0050 [Fri Mar 04 16:27:50 2011] [warning] [register] [39e08e6692] User login faileweb0035 [Fri Mar 04 16:27:50 2011] [warning] [login] [2f297b40a5] User login failed.web0072 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [2f297b40a5] User login failedweb0050 [Fri Mar 04 16:27:50 2011] [error] [login] [2e468a9bb6] User login failed. Reweb0054 [Fri Mar 04 16:27:50 2011] [warning] [login] [mk04gw1p71] User login failed.web0200 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [f9c2b23702] User login failedweb0064 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [47dd608551] Invalid charset cweb0012 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.web0041 [Fri Mar 04 16:27:50 2011] [fatal] [login] [mk04gw1p71] Invalid charset conveweb0012 [Fri Mar 04 16:27:50 2011] [error] [register] [2f297b40a5] Duplicate user IDweb0025 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.web0088 [Fri Mar 04 16:27:50 2011] [warning] [login] [2e468a9bb6] User login failed.web0050 [Fri Mar 04 16:27:51 2011] [warning] [login] [39e08e6692] User login failed.web0035 [Fri Mar 04 16:27:51 2011] [warning] [login] [2f297b40a5] User login failed.web0072 [Fri Mar 04 16:27:51 2011] [error] [login] [2f297b40a5] User login failed. ReFATALS ERRORS WARNINGSLogster
  48. 48. github.com/etsy/logsterRunbycron(e.g.1mintervals)Keeps acursoronyourlogfileParseandaggregatevalueshoweveryouwantOutputtoGanglia,Graphite,AmazonCloudWatchSimpleparsersLogster
  49. 49. web0054 [Fri Mar 04 16:27:48 2011] [error] [login][mk04gw1p71] User login failed. Reason: wrongpassword was submitted for mike@etsy.com^.+ [.+] [(?P<log_level>.+)]1.Patternmatchonfieldsofinterest
  50. 50. if (fields[log_level] == “fatal”):self.fatals += 1elif (fields[log_level] == “error”):self.errors += 1elif (fields[log_level] == “warning”):self.warnings += 1...2.Aggregatevalues (sum,average,percentile,etc.)
  51. 51. MetricObject("fatals",(self.fatals / self.duration), "per sec")MetricObject("errors",(self.errors / self.duration), "per sec")MetricObject("warning",(self.warnings / self.duration), "per sec")3.Sendthevaluesas“metricobjects”tothecollectors
  52. 52. github.com/etsy/logsterFATALS ERRORS WARNINGSLogster
  53. 53. StatsD
  54. 54. github.com/etsy/statsdStatsDNetworkdaemon(node.js)AcceptsdataoverUDPFlushestoGraphiteevery10secOne-lineofcode
  55. 55. StatsD::increment("logins.success");
  56. 56. StatsD::increment("logins.success");Logins
  57. 57. StatsD::timing("profile.time", $msec);
  58. 58. StatsD::timing("profile.time", $msec);90thpctaveragelower
  59. 59. Adhocname value timestamp
  60. 60. echo "events.deploy.site 1 `date +%s`" | nc graphite.etsycorp.com 2003
  61. 61. VerticalLineTechnology!target=drawAsInfinite(events.deploy.site)
  62. 62. User Logins
  63. 63. PHP Warnings
  64. 64. PHP Fatal Errors
  65. 65. 250,000+metricsatEtsySystems,Applications,Business
  66. 66. github.com/etsy/dashboardDashboards
  67. 67. <a href="http://graphite.etsycorp.com/render?from=-1hours&width=800&height=600&title=File+or+Script+Not+Found&yMin=0&target=webs.errorLog.notExist&target=drawAsInfinite%28deploys.config.production%29&target=drawAsInfinite%28deploys.web.production%29&target=drawAsInfinite%28deploys.search.production%29&target=drawAsInfinite%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,%23ff0000,%23006633,%23cc6600"><img src="http://graphite.etsycorp.com/render?from=-1hours&width=280&height=220&title=File+or+Script+Not+Found&hideLegend=1&yMin=0&target=webs.errorLog.notExist&target=drawAsInfinite%28deploys.config.production%29&target=drawAsInfinite%28deploys.web.production%29&target=drawAsInfinite%28deploys.search.production%29&target=drawAsInfinite%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,%23ff0000,%23006633,%23cc6600"></a>KindofHard :-/github.com/etsy/dashboard
  68. 68. $g = new Graphite($time);$g->setTitle(File Not Found);$g->addMetric(webs.errorLog.notExist, #00cc00);echo $g->getDashboardHTML(280, 220);SuperEasy!github.com/etsy/dashboard
  69. 69. But,yousaid...“250,000+metricsatEtsy”Systems,Applications,Business
  70. 70. http://graphite/render?from=-1hours&width=600&height=200&target=webs.errorLog.warning&rawData=1
  71. 71. http://graphite/render?from=-1hours&width=600&height=200&target=webs.errorLog.warning&rawData=1webs.errorLog.warning,1318444930,1318448530,60|5.0,1.0,3.0,1.0,0.0,9.0,0.0,1.0,3.0,2.0,1.0,6.0,2.0,6.0,3.0,6.0,4.0,4.0,2.0,1.0,1.0,8.0,2.0,3.0,6.0,3.0,5.0,3.0,0.0,4.0,6.0,2.0,0.0,2.0,0.0,4.0,0.0,3.0,1.0,3.0,4.0,2.0,10.0,3.0,0.0,6.0,0.0,4.0,2.0,5.0,18.0,1.0,1.0,2.0,1.0,8.0,5.0,1.0,1.0,None
  72. 72. Holt-WintersConfidenceBandslowerupper
  73. 73. Holt-WintersAberration
  74. 74. Businessmetrics+ Confidencebands_____________Alertablemetrics
  75. 75. Metrics!Metrics+EventsMetrics+AlertsMetrics+Metrics
  76. 76. High-level,real-timevisibility
  77. 77. Detectproblemsearly,andresolvethemquickly.
  78. 78. MakethemaccessibleMakethemrequiredfeaturesMakethemdeadsimple
  79. 79. Merci!These slides will be available atmikebrittain.com/talkscodeascraft.etsy.comgithub.com/etsySay“Hello!”mike@etsy.com@mikebrittainMetrics-DrivenEngineering
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×