Winning the metrics battle

Winning the metrics battle (ﬁnally)

Winning the metrics battle (ﬁnally) Simon Hildrew Nick Satterly Infrastructure Developer Monitoring Engineer The Guardian The Guardian

The metrics battleﬁeld

Total metrics 180,000 50,0001,400 2,800

http://www.ﬂickr.com/photos/ghostsigns/6676069121 5 minutes every 15 seconds http://www.ﬂickr.com/photos/millynet/134071210

developer dashboards

Physical screens Screensaver hacks201510 5 0

devhack

business dashboards

metrics + dashboards = culture change

http://www.ﬂickr.com/photos/chrisjames_taylor/5454315456

our approach Side project ➡ PrioritiseIncremental upgrade ➡ Understand the real problemUse off the shelf tool ➡ Question the tools Pragmatic solution ➡ Be ambitious Done in a year ➡ Keep learning

Prioritise

drowning in workhttp://www.ﬂickr.com/photos/iampeas/246738971

a dedicated monitoring and metrics engineer

Understand the real problem

Urgent issue -current tool end of life

The story so far...

metrics were not helping us solve production outages

ballooning number of applications

but... difﬁcult to instrument applications

T.T. Detect +T.T. Fix = T.T. Diagnose + T.T. Resolve

inaccessible tools http://www.ﬂickr.com/photos/kdashy/2678539087

inconsistent datahttp://www.ﬂickr.com/photos/sybrenstuvel/2468506922

hypothesising & arguing easier than measuring http://www.ﬂickr.com/photos/nouqraz/200049988

The ‘right’ thing• measure everything• measure frequently• measure each data point once• input and output must be open

Question the tools

Brute force?http://www.ﬂickr.com/photos/epublicist/3546059144

The safe option?http://www.ﬂickr.com/photos/alicebartlett/2361209195

Unintuitive?http://www.ﬂickr.com/photos/merlijnhoek/2841785343

Imposing a ﬂawed model?http://www.ﬂickr.com/photos/evansville/8953838/

Too difﬁcult / no progress?http://www.ﬂickr.com/photos/ginja_andy/4165849136/

Nagios• the “IBM” of monitoring tools• compromise over quantity and frequency of checks• < insert your criticism of nagios here >

Zabbix• metric collection tightly coupled to monitoring tool• confusing UI with poor visualisation• needed brute force to make limited API work

The ‘right’ thing• measure everything• measure frequently• measure each data point once• input and output must be open

don’t compromise

Be ambitious

http://www.ﬂickr.com/photos/mugley/2961131550 Throw work away

Draw your dream

http://www.ﬂickr.com/photos/sk8geek/7358702704 Get as far as you can

screens users db? alerting? Etsy dashboard message queue graphite SNMP? syslog? FITB ganglia api?network hosts applications

Develop missing pieces http://www.ﬂickr.com/photos/kalexanderson/5969012589

screens users mongodb alerta elastic search Etsy dashboard message queue syslog SNMP graphite ganglia alerts alerts alerts FITB ganglia ganglia-apinetwork hosts applications

Guardian Managementhttps://github.com/guardian/guardian-management

Ganglia APIhttps://github.com/guardian/ganglia-api

rescale image??? Alertahttps://github.com/guardian/alerta

Current stack• Ganglia • Guardian management https://github.com/guardian/guardian-management• FITB • Guardian ganglia-api https://github.com/guardian/ganglia-api• Graphite • Guardian alerta• Etsy dashboards https://github.com/guardian/alerta

Keep learning

we are not there yet

Watch the cultural changes

detecting

diagnosis

performance testing

conﬁrmation

#monitoringsucks

➡ Prioritise➡ Understand the real problem➡ Question the tools➡ Be ambitious➡ Keep learning

tools can change culture

Thank you http://github.com/guardian http://gu.com/p/3ap5f Simon Hildrew Nick Satterly @sihil @nicksatterlysimon.hildrew@guardian.co.uk nick.satterly@guardian.co.uk

http://www.guardian.co.uk	3755
http://www.guprod.gnl	4
http://cms.guprod.gnl	1

Winning the metrics battle

by sihil on Oct 09, 2012

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 3,760

Statistics

Winning the metrics battle Presentation Transcript