Metrics Driven Engineering

Metrics-DrivenEngineeringMike BrittainENGINEERING DIRECTOR, ETSY@mikebrittain

PROCESSANDTOOLSSupportingacultureofContinuousDeployment

Howmanynewvisitors?Howmanylistingscreated?Howmanyregistrations?HowdopeopleuseEtsy?Howmanymessagessent?Howmanypurchases?Howmanynewshops?

Searchindexing?Howfastarepagesgenerating?Asynctaskscurrentlyinqueue?Howistheapplicationbehaving?DeveloperAPIauthandratelimiting?Imagesresizedandstored?Errorandwarningrates?

Replicationslavelag?Memcachehits/misses?Availableconnections?AretheserversandnetworkOK?Databasequeriespersecond?Totaloutgoingbandwidth?CPU,Memory,I/O?

BusinessMetrics

ApplicationMetrics

SystemMetrics

VisibilityEVERYWHERE

Metricshelpyouidentifygoals

Metricshelpyouidentifygoals...butalsotellyouwhenyou’vebrokensomething.

AlwaysBeShippingcredit: ibailemon (ﬂickr)

1st dayPutyourselfonthewebsite.

2nd dayCompletetax,insurance,andbeneﬁtsforms.credit: ktpupp (ﬂickr)

DevSandbox Trunk/master ProductionYou!Test

7e9a814 -> 63a2bb3Deploy to Production

50+Deploys/day200+Committers15Productteams8Infrastructureteams50+Deploys/daycredit: misswired (ﬂickr)

credit: digidave (ﬂickr)

PeerReviewCodereviews,Architecturereviews,Operabilityreviews

AutomatedTestsStaticanalysis,Unittests,Integrationtests,Functionaltests

May2013$102.9Millioningoodsold1.37Billionpageviewshttps://www.etsy.com/blog/news/2013/etsy-statistics-may-2013-weather-report/

Failureisnotanoption

Failureisnotanoptioninevitable

Failureisnotanoptionanddetectable!inevitable

Access

Soundslikealotofwork,who’sgoingtobuildallofthis?Q:

Well,theOpsteammanagesthenetwork,rackstheservers,installedthemonitoringtools,wearsthepagers,blah,blah,blah...A:Soundslikealotofwork,who’sgoingtobuildallofthis?Q:

Engineersbuildtheapplication

OPSLoggingGraphingTrendingAlertingENG

Metricsarepartofeveryfeature(andsoareconﬁgﬂags)

MakeitDEADSIMPLE

Ganglia (application,servers,network)Logster* (application,servers)Cacti (network,SNMP)FITB* (network)*github.com/etsySimple,open-sourcetoolsGraphite (application)Statsd* (application)Logformats (application,servers)Nagios (alerting)

Ganglia

Cluster-orientedHugecommunitycontributedrecipesCustommetrics(gmetad)Ganglia

Graphite

Single-instanceCreatenewmetricson-the-ﬂyCustomizeviaURLsanddisplayfunctionshttp://www.aosabook.org/en/graphite.htmlGraphite

LogFormats

Time, remote address, http method, request uri, referrer, user-agent,response size, response code, execution time, memory consumed,plus custom ﬁelds...• Signed-in/out (user_id vs. “-”)• display mode (“desktop” vs. “mobile”)• i10n/i18n (“en-US”)• etc.AccessLogs

LogFormat %l %t "%r" %>s %b "%{Referer}i""%{User-Agent}i" %{custom_field}n ...apache_note(“custom_ﬁeld”, $whatever);

LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b"%{Referer}i" "%{User-Agent}i"%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1"200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876

Logger::error("User login failed. Reason:$msg for $email_addr", “login”);Methodnamedenoteslog“level”—error,fatal,warning,notice,debug.A“namespace”parameterisprovidedsowecanaggregatelogentrieswithsimilarconcerns.

Logger::error("User login failed. Reason:$msg for $email_addr", “login”);web0054 [Fri Mar 04 16:27:48 2011] [error] [login][mk04gw1p71] User login failed. Reason: wrongpassword was submitted for mike@etsy.comUniquerequestIDServernameDateandtime LevelNamespace

web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] Invalid charset conveweb0102 [Fri Mar 04 16:27:48 2011] [warning] [login] [47dd608551] User login failed.web0012 [Fri Mar 04 16:27:48 2011] [warning] [login] [mk04gw1p71] User login failed.web0081 [Fri Mar 04 16:27:48 2011] [error] [register] [39e08e6692] Duplicate user IDweb0100 [Fri Mar 04 16:27:49 2011] [fatal] [register] [f9c2b23702] Invalid charset coweb0003 [Fri Mar 04 16:27:49 2011] [error] [register] [39e08e6692] Duplicate user IDweb0050 [Fri Mar 04 16:27:49 2011] [error] [register] [2e468a9bb6] Duplicate user IDweb0054 [Fri Mar 04 16:27:49 2011] [warning] [login] [mk04gw1p71] User login failed.web0200 [Fri Mar 04 16:27:49 2011] [error] [login] [f9c2b23702] User login failed. Reweb0064 [Fri Mar 04 16:27:49 2011] [error] [login] [47dd608551] Duplicate user ID encweb0012 [Fri Mar 04 16:27:49 2011] [warning] [login] [32976da59c] User login failed.web0041 [Fri Mar 04 16:27:49 2011] [fatal] [login] [mk04gw1p71] Invalid charset conveweb0012 [Fri Mar 04 16:27:49 2011] [error] [login] [2f297b40a5] User login failed. Reweb0025 [Fri Mar 04 16:27:49 2011] [warning] [register] [32976da59c] User login faileweb0088 [Fri Mar 04 16:27:49 2011] [warning] [register] [2e468a9bb6] User login faileweb0050 [Fri Mar 04 16:27:50 2011] [warning] [register] [39e08e6692] User login faileweb0035 [Fri Mar 04 16:27:50 2011] [warning] [login] [2f297b40a5] User login failed.web0072 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [2f297b40a5] User login failedweb0050 [Fri Mar 04 16:27:50 2011] [error] [login] [2e468a9bb6] User login failed. Reweb0054 [Fri Mar 04 16:27:50 2011] [warning] [login] [mk04gw1p71] User login failed.web0200 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [f9c2b23702] User login failedweb0064 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [47dd608551] Invalid charset cweb0012 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.web0041 [Fri Mar 04 16:27:50 2011] [fatal] [login] [mk04gw1p71] Invalid charset conveweb0012 [Fri Mar 04 16:27:50 2011] [error] [register] [2f297b40a5] Duplicate user IDweb0025 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.web0088 [Fri Mar 04 16:27:50 2011] [warning] [login] [2e468a9bb6] User login failed.web0050 [Fri Mar 04 16:27:51 2011] [warning] [login] [39e08e6692] User login failed.web0035 [Fri Mar 04 16:27:51 2011] [warning] [login] [2f297b40a5] User login failed.web0072 [Fri Mar 04 16:27:51 2011] [error] [login] [2f297b40a5] User login failed. Re

web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] Invalid charset conveweb0102 [Fri Mar 04 16:27:48 2011] [warning] [login] [47dd608551] User login failed.web0012 [Fri Mar 04 16:27:48 2011] [warning] [login] [mk04gw1p71] User login failed.web0081 [Fri Mar 04 16:27:48 2011] [error] [register] [39e08e6692] Duplicate user IDweb0100 [Fri Mar 04 16:27:49 2011] [fatal] [register] [f9c2b23702] Invalid charset coweb0003 [Fri Mar 04 16:27:49 2011] [error] [register] [39e08e6692] Duplicate user IDweb0050 [Fri Mar 04 16:27:49 2011] [error] [register] [2e468a9bb6] Duplicate user IDweb0054 [Fri Mar 04 16:27:49 2011] [warning] [login] [mk04gw1p71] User login failed.web0200 [Fri Mar 04 16:27:49 2011] [error] [login] [f9c2b23702] User login failed. Reweb0064 [Fri Mar 04 16:27:49 2011] [error] [login] [47dd608551] Duplicate user ID encweb0012 [Fri Mar 04 16:27:49 2011] [warning] [login] [32976da59c] User login failed.web0041 [Fri Mar 04 16:27:49 2011] [fatal] [login] [mk04gw1p71] Invalid charset conveweb0012 [Fri Mar 04 16:27:49 2011] [error] [login] [2f297b40a5] User login failed. Reweb0025 [Fri Mar 04 16:27:49 2011] [warning] [register] [32976da59c] User login faileweb0088 [Fri Mar 04 16:27:49 2011] [warning] [register] [2e468a9bb6] User login faileweb0050 [Fri Mar 04 16:27:50 2011] [warning] [register] [39e08e6692] User login faileweb0035 [Fri Mar 04 16:27:50 2011] [warning] [login] [2f297b40a5] User login failed.web0072 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [2f297b40a5] User login failedweb0050 [Fri Mar 04 16:27:50 2011] [error] [login] [2e468a9bb6] User login failed. Reweb0054 [Fri Mar 04 16:27:50 2011] [warning] [login] [mk04gw1p71] User login failed.web0200 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [f9c2b23702] User login failedweb0064 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [47dd608551] Invalid charset cweb0012 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.web0041 [Fri Mar 04 16:27:50 2011] [fatal] [login] [mk04gw1p71] Invalid charset conveweb0012 [Fri Mar 04 16:27:50 2011] [error] [register] [2f297b40a5] Duplicate user IDweb0025 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.web0088 [Fri Mar 04 16:27:50 2011] [warning] [login] [2e468a9bb6] User login failed.web0050 [Fri Mar 04 16:27:51 2011] [warning] [login] [39e08e6692] User login failed.web0035 [Fri Mar 04 16:27:51 2011] [warning] [login] [2f297b40a5] User login failed.web0072 [Fri Mar 04 16:27:51 2011] [error] [login] [2f297b40a5] User login failed. ReFATALS ERRORS WARNINGSLogster

github.com/etsy/logsterRunbycron(e.g.1mintervals)Keeps acursoronyourlogﬁleParseandaggregatevalueshoweveryouwantOutputtoGanglia,Graphite,AmazonCloudWatchSimpleparsersLogster

web0054 [Fri Mar 04 16:27:48 2011] [error] [login][mk04gw1p71] User login failed. Reason: wrongpassword was submitted for mike@etsy.com^.+ [.+] [(?P<log_level>.+)]1.Patternmatchonﬁeldsofinterest

if (fields[log_level] == “fatal”):self.fatals += 1elif (fields[log_level] == “error”):self.errors += 1elif (fields[log_level] == “warning”):self.warnings += 1...2.Aggregatevalues (sum,average,percentile,etc.)

MetricObject("fatals",(self.fatals / self.duration), "per sec")MetricObject("errors",(self.errors / self.duration), "per sec")MetricObject("warning",(self.warnings / self.duration), "per sec")3.Sendthevaluesas“metricobjects”tothecollectors

github.com/etsy/logsterFATALS ERRORS WARNINGSLogster

StatsD

github.com/etsy/statsdStatsDNetworkdaemon(node.js)AcceptsdataoverUDPFlushestoGraphiteevery10secOne-lineofcode

StatsD::increment("logins.success");

StatsD::increment("logins.success");Logins

StatsD::timing("profile.time", $msec);

StatsD::timing("profile.time", $msec);90thpctaveragelower

Adhocname value timestamp

echo "events.deploy.site 1 `date +%s`" | nc graphite.etsycorp.com 2003

VerticalLineTechnology!target=drawAsInfinite(events.deploy.site)

User Logins

PHP Warnings

PHP Fatal Errors

250,000+metricsatEtsySystems,Applications,Business

github.com/etsy/dashboardDashboards

<a href="http://graphite.etsycorp.com/render?from=-1hours&width=800&height=600&title=File+or+Script+Not+Found&yMin=0&target=webs.errorLog.notExist&target=drawAsInfinite%28deploys.config.production%29&target=drawAsInfinite%28deploys.web.production%29&target=drawAsInfinite%28deploys.search.production%29&target=drawAsInfinite%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,%23ff0000,%23006633,%23cc6600"><img src="http://graphite.etsycorp.com/render?from=-1hours&width=280&height=220&title=File+or+Script+Not+Found&hideLegend=1&yMin=0&target=webs.errorLog.notExist&target=drawAsInfinite%28deploys.config.production%29&target=drawAsInfinite%28deploys.web.production%29&target=drawAsInfinite%28deploys.search.production%29&target=drawAsInfinite%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,%23ff0000,%23006633,%23cc6600"></a>KindofHard :-/github.com/etsy/dashboard

$g = new Graphite($time);$g->setTitle(File Not Found);$g->addMetric(webs.errorLog.notExist, #00cc00);echo $g->getDashboardHTML(280, 220);SuperEasy!github.com/etsy/dashboard

But,yousaid...“250,000+metricsatEtsy”Systems,Applications,Business

http://graphite/render?from=-1hours&width=600&height=200&target=webs.errorLog.warning&rawData=1

http://graphite/render?from=-1hours&width=600&height=200&target=webs.errorLog.warning&rawData=1webs.errorLog.warning,1318444930,1318448530,60|5.0,1.0,3.0,1.0,0.0,9.0,0.0,1.0,3.0,2.0,1.0,6.0,2.0,6.0,3.0,6.0,4.0,4.0,2.0,1.0,1.0,8.0,2.0,3.0,6.0,3.0,5.0,3.0,0.0,4.0,6.0,2.0,0.0,2.0,0.0,4.0,0.0,3.0,1.0,3.0,4.0,2.0,10.0,3.0,0.0,6.0,0.0,4.0,2.0,5.0,18.0,1.0,1.0,2.0,1.0,8.0,5.0,1.0,1.0,None

Holt-WintersConﬁdenceBandslowerupper

Holt-WintersAberration

Businessmetrics+ Conﬁdencebands_____________Alertablemetrics

Metrics!Metrics+EventsMetrics+AlertsMetrics+Metrics

High-level,real-timevisibility

Detectproblemsearly,andresolvethemquickly.

MakethemaccessibleMakethemrequiredfeaturesMakethemdeadsimple

Merci!These slides will be available atmikebrittain.com/talkscodeascraft.etsy.comgithub.com/etsySay“Hello!”mike@etsy.com@mikebrittainMetrics-DrivenEngineering