Metrics-Driven Engineering

Metrics-Driven EngineeringMike Brittain @ mikebrittainDirector of engineering, Infrastructure October 13, 2011

Tools and Process at Etsy

How many new visits? How many listings created? How many registrations?How do people use Etsy? How many convos sent? How many purchases? How many new shops?

Search indexing? How fast are pages generating? Async tasks currently in queue?What is the application doing? Developer API auth and rate limiting? Images resized and stored? Error and warning rates?

Replication slave lag? Memcache hits/misses? Available connections?Are the servers in good shape ? Database queries per second? Total outgoing bandwidth? CPU, Memory, I/O?

Business Metrics

Application Metrics

System Metrics

Visibility EVERYWHERE

Constant Change

$314 Million GMS 2010 $180 Million GMS 2009 $87 Million GMS 2008 $26 Million GMS 2007credit: pentarux (ﬂickr)

25 Million Unique Visitors 1 Billion page views per monthcredit: pentarux (ﬂickr)

Engineering team grew 500% over 18 monthscredit: martin_heigan (ﬂickr)

Less talk, more do.

Always Be Shippingcredit: ibailemon (ﬂickr)

Always Be Shipping (even if it’s your ﬁrst day)credit: ibailemon (ﬂickr)

90+ Engineers 40+ Deploys / daycredit: misswired (ﬂickr)

credit: digidave (ﬂickr)

Code Reviews

Automated Tests

$cfg = array( checkout => array(enabled => on), homepage => array(enabled => on), profiles => array(enabled => on), new_search => array(enabled => off),); Conﬁg FlagsEnable and disable features quickly

$cfg = array( checkout => array(enabled => on), homepage => array(enabled => on), profiles => array(enabled => on), new_search => array(enabled => off),); Conﬁg FlagsEnable and disable features quicklyPlus “admin-only,” percentage ramp-up, A/B testing,whitelists, blacklists, etc...

Failure is not an option

inevitable!Failure is not an option

inevitable!Failure is not an option a learning opportunity!

inevitable!Failure is not an option a learning opportunity! DETECTABLE!

Access

Detect problems quickly

CONFIDENCE

A: Well, the Ops team manages the network, racks the servers, installed the monitoring tools, wears the pagers, blah, blah, blah...

Engineers build the application

Logging GraphingOPS ENG Trending Alerting

“Engineers are too busy writing features to build metrics.”

Metrics are part of every feature ...and so are conﬁg ﬂags

Dead Simple

Simple, open source tools

Cacti (network, SNMP)Ganglia (machines)Graphite (application)Splunk (log analysis, nightly reports)Nagios (alerting) Logging Logster StatsD

Ganglia

GangliaCluster-orientedHuge community contributed recipesCustom metrics (gmetad)

Graphite

Graphite Single-instance Create new metrics on-the-ﬂy Customize via URLs and display functions

Logging

It’s 2:48 PM.Do you know where your logs are?

Logger::log_error("User login failed.Reason: $msg for $username", “login”);

web0054 [Fri Mar 04 16:27:48 2011][error] [login] [mk04gw1p71] User login failed. Reason: wrong password for ...

LogFormat "%h %l %u %t "%r" %>s %b" common

LogFormat %{True-Client-IP}i %l %t "%r " %>s %b "%{Referer}i" "%{User-Agent}i" %{etsy_shop_id}n %{etsy_uaid}n %V %{etsy_ab_selections}n %{etsy_request_uuid}n %{etsy_api_consumer_key}n %{etsy_api_method_name}n %{php_memory_usage_bytes}n %{php_time_microsec}n %D" combined

apache_note()

LogFormat %{True-Client-IP}i %l %t "%r " %>s %b "%{Referer}i" "%{User-Agent}i" %{etsy_shop_id}n %{etsy_uaid}n %V %{etsy_ab_selections}n %{etsy_request_uuid}n %{etsy_api_consumer_key}n %{etsy_api_method_name}n %{php_memory_usage_bytes}n %{php_time_microsec}n %D" combined

grep "/listing/" access.log | awk {sum=sum+$(NF-2)} END {print sum/NR}

web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppppp!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] Youve been eaten by a grue.web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0003 [04:28:54 2011] [error] [client 10.101.x.x] Youve been eaten by a grue.web0066 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!

LogsterFatals Errors Warnings

LogsterRun by cronKeeps a cursor on your log ﬁleAggregate lines anyway you wantOutput to Ganglia or GraphiteSimple parsers github.com/etsy

web0054 [Fri Mar 04 16:27:48 2011][error] [login] [mk04gw1p71] User login failed. Reason: wrong password for ...

^.+ [.+] [(?P<log_level>.+)]

if (fields[log_level] == “fatal”): self.fatals += 1elif (fields[log_level] == “error”): self.errors += 1elif (fields[log_level] == “warning”): self.warnings += 1...

MetricObject("fatals", (self.fatals / self.duration), "per sec")MetricObject("errors", (self.errors / self.duration), "per sec")MetricObject("warning", (self.warnings / self.duration), "per sec")

Fatals Errors Warnings

StatsD

StatsD Network daemon (node.js) Accepts data over UDP Flushes to Graphite every 10 sec One-line of codegithub.com/etsy

StatsD::increment("logins.success");

StatsD::increment("logins.success"); logins

StatsD::timing("gearman.time", $msec);

StatsD::timing("gearman.time", $msec); 90th pct average lower

Ad hocname value timestamp

echo "events.deploy.site 1 `date +%s`" | nc graphite.etsycorp.com 2003

Vertical Line Technology!target=drawAsInfinite(events.deploy.site)

We could stare at graphs all day...

http://graphite/render? from=-1hours&width=600&height=200&target=webs.errorLog.warning&rawData=1

http://graphite/render? from=-1hours&width=600&height=200 &target=webs.errorLog.warning&rawData=1webs.errorLog.warning,1318444930,1318448530,60|5.0,1.0,3.0,1.0,0.0,9.0,0.0,1.0,3.0,2.0,1.0,6.0,2.0,6.0,3.0,6.0,4.0,4.0,2.0,1.0,1.0,8.0,2.0,3.0,6.0,3.0,5.0,3.0,0.0,4.0,6.0,2.0,0.0,2.0,0.0,4.0,0.0,3.0,1.0,3.0,4.0,2.0,10.0,3.0,0.0,6.0,0.0,4.0,2.0,5.0,18.0,1.0,1.0,2.0,1.0,8.0,5.0,1.0,1.0,None

Holt-Winters Conﬁdence Bandsupper lower

Holt-Winters Aberration

Business metrics + Conﬁdence bands_____________ Alertable metrics

40,000+ metrics at Etsy Systems, Applications, Business

Dashboards

Kind of Hard :-/<a href="http://graphite.etsycorp.com/render?from=-1hours&width=800&height=600&title=File+or+Script+Not+Found&yMin=0&target=webs.errorLog.notExist&target=drawAsInfinite%28deploys.config.production%29&target=drawAsInfinite%28deploys.web.production%29&target=drawAsInfinite%28deploys.search.production%29&target=drawAsInfinite%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,%23ff0000,%23006633,%23cc6600"> <img src="http://graphite.etsycorp.com/render?from=-1hours&width=280&height=220&title=File+or+Script+Not+Found&hideLegend=1&yMin=0&target=webs.errorLog.notExist&target=drawAsInfinite%28deploys.config.production%29&target=drawAsInfinite%28deploys.web.production%29&target=drawAsInfinite%28deploys.search.production%29&target=drawAsInfinite%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,%23ff0000,%23006633,%23cc6600"></a>

Super Easy!$g = new Graphite($time);$g->setTitle(File Not Found);$g->addMetric(webs.errorLog.notExist, #00cc00);echo $g->getDashboardHTML(280, 220);

Metrics!

Metrics!Metrics + Events

Metrics!Metrics + EventsMetrics + Alerts

Metrics!Metrics + EventsMetrics + AlertsMetrics + Metrics

High-level, real-time visibility

Detect problems quickly

CONFIDENCE

Make them required features

Make them dead simple

Make them accessible

Make them!

Homeworkcodeascraft.etsy.comgithub.com/etsy Get in touch mike @ etsy . comWe’re always looking for people @ mikebrittainwho are interested in this kind ofstuﬀ...Thank Youetsy.com/careers

http://benjaminwootton.co.uk	3185
http://java.dzone.com	1043
http://blog.livedoor.jp	420
http://storify.com	102
http://agile.dzone.com	90
http://www.web2expo.com	56
http://ruby.dzone.com	47
http://tatsukii.tumblr.com	47
https://twitter.com	37
http://a0.twimg.com	33
http://server.dzone.com	29
http://architects.dzone.com	22
http://python.dzone.com	17
http://playtherapy.co.uk	12
http://paper.li	11
http://www.linkedin.com	10
http://webcache.googleusercontent.com	8
http://safe.tumblr.com	8
http://do-nothing.tumblr.com	3
http://devopsfriday.com	3
http://translate.googleusercontent.com	3
http://coderwall.com	3
http://rascasse.com	3
http://lanyrd.com	3
http://mrsy.tumblr.com	2
http://twitter.com	2
http://us-w1.rockmelt.com	1
http://rss.qoli.de	1
http://pfmusique.tumblr.com	1
http://www.dzone.com	1
http://rritw.com	1
http://www.twylah.com	1
http://ranksit.com	1
http://nosqlfriday.com	1

Metrics-Driven Engineering

by Mike Brittain

Accessibility

Categories

Upload Details

Usage Rights

34 Embeds 5,207

Statistics