SlideShare a Scribd company logo
From #MonitoringSucks toFrom #MonitoringSucks to
#MonitoringLove#MonitoringLove
Open Source Monitoring in 2016Open Source Monitoring in 2016
@KrisBuytaert
FlossUK 2016, London , UK
Kris BuytaertKris Buytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source
Consultant @inuits.euConsultant @inuits.eu
● Everything is an effing DNS ProblemEverything is an effing DNS Problem
● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore
● Organising ConferencesOrganising Conferences
● Evangelizing devopsEvangelizing devops
An opinionated talk about the Open SourceAn opinionated talk about the Open Source
Monitoring tooling landscapeMonitoring tooling landscape
In which I hope to learn from YOUIn which I hope to learn from YOU
#devops=~C(L)AMS#devops=~C(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● Monitoring and MeasurementMonitoring and Measurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim
Monitoring is usually anMonitoring is usually an
aftertoughtaftertought
ENOBUDGET, ENOTIMEENOBUDGET, ENOTIME
An 2008 OLS PaperAn 2008 OLS Paper
● We have bloated Java toolsWe have bloated Java tools
● Some open Core stuffSome open Core stuff
● DYI folks want traditional NagiosDYI folks want traditional Nagios
● DBA RequiredDBA Required
#monitoringsucks#monitoringsucks
● John Vincent (@lusis), june 2011John Vincent (@lusis), june 2011
● A sub #devops movementA sub #devops movement
● https://github.com/monitoringsucks/https://github.com/monitoringsucks/
Why #monitoringsucksWhy #monitoringsucks
● Manual config (gui)Manual config (gui)
● Not in sync with realityNot in sync with reality
● Hosts onlyHosts only
● Services sometimesServices sometimes
● Application neverApplication never
● Chaos or out of sync with realityChaos or out of sync with reality
● Alert FatigueAlert Fatigue
#monitoringlove#monitoringlove
•
•
Ulf Mansson #devopsdays Rome 2011Ulf Mansson #devopsdays Rome 2011
•
A new era of toolingA new era of tooling
•
#monitoringlove hacksessions @inuits#monitoringlove hacksessions @inuits
•
#monitorama#monitorama
What we wantWhat we want
● Small , well suited componentsSmall , well suited components
•
CollectCollect
•
Transport / MangleTransport / Mangle
•
StoreStore
•
AnalyseAnalyse
•
Act / AlertAct / Alert
•
VisualizeVisualize
#monitoringlove#monitoringlove
But the love was about :But the love was about :
SensuSensu
● Awesome for non staticAwesome for non static
environmentsenvironments
● Scaling a clustered RabbitMQ ?Scaling a clustered RabbitMQ ?
● This is Europe, U no do cloudThis is Europe, U no do cloud
Automation ofAutomation of
#monitoring#monitoring
brought backbrought back
thethe #love#love
AutomationAutomation
Monitoring aMonitoring a serviceservice
vsvs
Monitoring aMonitoring a ServiceService
definition of done:definition of done:
monitored and in productionmonitored and in production
A software project is not doneA software project is not done
until your last end user is deaduntil your last end user is dead
Culture,Culture,
Automation,Automation,
Measurement :Measurement :
measure all the thingsmeasure all the things
SharingSharing
CollectD all the metrics,CollectD all the metrics,
at high intervalsat high intervals
Oldschool graphiteOldschool graphite
Graphite++Graphite++
● APIAPI
● DashboardsDashboards
•
GrafanaGrafana
•
GdashGdash
● Engines :Engines :
•
InfluxDBInfluxDB
•
CyaniteCyanite
Draw as InfiniteDraw as Infinite
● Time To DeployTime To Deploy
● DeployDeploy
FrequencyFrequency
● LifecycleLifecycle
frequencyfrequency
● Map to otherMap to other
metricsmetrics
Graph-ExplorerGraph-Explorer
● (Vimeo)(Vimeo)
● Metrics 2.0Metrics 2.0
● Add EventsAdd Events
GrafanaGrafana
Graphs to KnowledgeGraphs to Knowledge
SkylineSkyline
•
OculusOculus
•
Creating Information out of this dataCreating Information out of this data
•
Big dataBig data
•
Machine LearningMachine Learning
AggregationAggregation
● Alert on streamsAlert on streams
● Alert on aggregated metricsAlert on aggregated metrics
RiemannRiemann
● I still don't get it ?I still don't get it ?
● Distributed TopDistributed Top
● Do you like Clojure ?Do you like Clojure ?
● Riemann Health plugin ?Riemann Health plugin ?
● s/riemann-health/collectd/g;s/riemann-health/collectd/g;
● Output to graphiteOutput to graphite
PrometheusPrometheus
● Started 2012Started 2012
● SoundCloudSoundCloud
● Metrics BasedMetrics Based
● ScrapesScrapes
EndpointsEndpoints
•
ExistingExisting
endpoints forendpoints for
limited toolslimited tools
● GraphiteGraphite
ExporterExporter
● Push GatewayPush Gateway
● Great AlertingGreat Alerting
● Might needMight need
some codingsome coding
But I have log files..But I have log files..
Logs and MetricsLogs and Metrics
● Graylog2Graylog2
● ELSA (Enterprise Log Search andELSA (Enterprise Log Search and
Archive)Archive)
● ELK StackELK Stack
● Collect fromCollect from
anywhereanywhere
● FilterFilter
● Send anywhereSend anywhere
Infitnite Diskspace ?Infitnite Diskspace ?
● Logstash outputLogstash output
•
Statsd => GraphiteStatsd => Graphite
•
Keep patterns around,Keep patterns around,
•
Selectively purge dataSelectively purge data
•
APMAPM
But what about my apps ?But what about my apps ?
Half the world cheers about SAASHalf the world cheers about SAAS
tools :(tools :(
PacketbeatPacketbeat
● Traffic FlowTraffic Flow
through networkthrough network
● TransactionsTransactions
causing errroscausing errros
● SQL per HTTPSQL per HTTP
● API call usageAPI call usage
Old PacketBeatOld PacketBeat
Beats ?Beats ?
● Elasti.coElasti.co
● Collect, Parse and ShipCollect, Parse and Ship
● Q: Is all the data you care aboutQ: Is all the data you care about
suitable for Elastic Search ?suitable for Elastic Search ?
● What about Long Term Storage ?What about Long Term Storage ?
● Do you even want to build alertingDo you even want to build alerting
from this ?from this ?
Checking for FailureChecking for Failure
● IcingaIcinga
•
Automated config generationAutomated config generation
● SensuSensu
•
CloudstyleCloudstyle
● PrometheusPrometheus
•
Metric basedMetric based
Waking you up at nightWaking you up at night
● FlapjackFlapjack
flapjack.ioflapjack.io
monitoring notification routing +monitoring notification routing +
event processing systemevent processing system
● OpenDutyOpenDuty
github.com/szechuen/OpenDutygithub.com/szechuen/OpenDuty
Duty managementDuty management
AggregatingAggregating
● ThrukThruk
● GrafanaGrafana
● DashingDashing
Our Current StackOur Current Stack
I love where Monitoring is headingI love where Monitoring is heading
We have much less false positives these daysWe have much less false positives these days
ContactContact
Kris Buytaert kris.buytaert@inuits.euKris Buytaert kris.buytaert@inuits.eu
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.eu/http://www.inuits.eu/
Find Inuits inFind Inuits in
Brasschaat,Ghent,Brasschaat,Ghent,
Rotterdam,Prague,Rotterdam,Prague,
Kiev,BrnoKiev,Brno

More Related Content

What's hot

devops is a reorg
devops is a reorgdevops is a reorg
devops is a reorg
Kris Buytaert
 
The influence of "Distributed platforms" on #devops
The influence of "Distributed platforms" on #devopsThe influence of "Distributed platforms" on #devops
The influence of "Distributed platforms" on #devops
Kris Buytaert
 
No, we can't do continuous delivery
No, we can't do continuous deliveryNo, we can't do continuous delivery
No, we can't do continuous delivery
Kris Buytaert
 
Another 7 tools for your #devops stack
Another 7 tools for your #devops stackAnother 7 tools for your #devops stack
Another 7 tools for your #devops stack
Kris Buytaert
 
Open Source Monitoring in 2015
Open Source Monitoring in 2015Open Source Monitoring in 2015
Open Source Monitoring in 2015
Kris Buytaert
 
Is there a future for devops ?
Is there a future for devops ?Is there a future for devops ?
Is there a future for devops ?
Kris Buytaert
 
Moby is killing your devops efforts
Moby is killing your devops effortsMoby is killing your devops efforts
Moby is killing your devops efforts
Kris Buytaert
 
From devoops to devops
From devoops to devopsFrom devoops to devops
From devoops to devops
Kris Buytaert
 
Continous Delivery of your Infrastructure
Continous Delivery of your InfrastructureContinous Delivery of your Infrastructure
Continous Delivery of your Infrastructure
Kris Buytaert
 
Devops is dead, Long Live Devops
Devops is dead, Long Live DevopsDevops is dead, Long Live Devops
Devops is dead, Long Live Devops
Kris Buytaert
 
Adopting Devops , Stories from the trenches
Adopting Devops , Stories from the trenchesAdopting Devops , Stories from the trenches
Adopting Devops , Stories from the trenches
Kris Buytaert
 
Devops, The future is here, it's just not evenly distributed
Devops, The future is here, it's just not evenly distributedDevops, The future is here, it's just not evenly distributed
Devops, The future is here, it's just not evenly distributed
Kris Buytaert
 
Continuous Infrastructure First
Continuous Infrastructure FirstContinuous Infrastructure First
Continuous Infrastructure First
Kris Buytaert
 
The Return of the Dull Stack Engineer
The Return of the Dull Stack EngineerThe Return of the Dull Stack Engineer
The Return of the Dull Stack Engineer
Kris Buytaert
 
Docker is killing your #devops Efforts
Docker is killing your #devops EffortsDocker is killing your #devops Efforts
Docker is killing your #devops Efforts
Kris Buytaert
 
Repositories as Code
Repositories as CodeRepositories as Code
Repositories as Code
Kris Buytaert
 
Devops is a Security Requirement
Devops is a Security RequirementDevops is a Security Requirement
Devops is a Security Requirement
Kris Buytaert
 
Pipeline as Code
Pipeline as CodePipeline as Code
Pipeline as Code
Kris Buytaert
 
Devops 101 QA with Kris Buytaert
Devops 101 QA  with Kris BuytaertDevops 101 QA  with Kris Buytaert
Devops 101 QA with Kris Buytaert
Kris Buytaert
 
Automating MySQL operations with Puppet
Automating MySQL operations with PuppetAutomating MySQL operations with Puppet
Automating MySQL operations with Puppet
Kris Buytaert
 

What's hot (20)

devops is a reorg
devops is a reorgdevops is a reorg
devops is a reorg
 
The influence of "Distributed platforms" on #devops
The influence of "Distributed platforms" on #devopsThe influence of "Distributed platforms" on #devops
The influence of "Distributed platforms" on #devops
 
No, we can't do continuous delivery
No, we can't do continuous deliveryNo, we can't do continuous delivery
No, we can't do continuous delivery
 
Another 7 tools for your #devops stack
Another 7 tools for your #devops stackAnother 7 tools for your #devops stack
Another 7 tools for your #devops stack
 
Open Source Monitoring in 2015
Open Source Monitoring in 2015Open Source Monitoring in 2015
Open Source Monitoring in 2015
 
Is there a future for devops ?
Is there a future for devops ?Is there a future for devops ?
Is there a future for devops ?
 
Moby is killing your devops efforts
Moby is killing your devops effortsMoby is killing your devops efforts
Moby is killing your devops efforts
 
From devoops to devops
From devoops to devopsFrom devoops to devops
From devoops to devops
 
Continous Delivery of your Infrastructure
Continous Delivery of your InfrastructureContinous Delivery of your Infrastructure
Continous Delivery of your Infrastructure
 
Devops is dead, Long Live Devops
Devops is dead, Long Live DevopsDevops is dead, Long Live Devops
Devops is dead, Long Live Devops
 
Adopting Devops , Stories from the trenches
Adopting Devops , Stories from the trenchesAdopting Devops , Stories from the trenches
Adopting Devops , Stories from the trenches
 
Devops, The future is here, it's just not evenly distributed
Devops, The future is here, it's just not evenly distributedDevops, The future is here, it's just not evenly distributed
Devops, The future is here, it's just not evenly distributed
 
Continuous Infrastructure First
Continuous Infrastructure FirstContinuous Infrastructure First
Continuous Infrastructure First
 
The Return of the Dull Stack Engineer
The Return of the Dull Stack EngineerThe Return of the Dull Stack Engineer
The Return of the Dull Stack Engineer
 
Docker is killing your #devops Efforts
Docker is killing your #devops EffortsDocker is killing your #devops Efforts
Docker is killing your #devops Efforts
 
Repositories as Code
Repositories as CodeRepositories as Code
Repositories as Code
 
Devops is a Security Requirement
Devops is a Security RequirementDevops is a Security Requirement
Devops is a Security Requirement
 
Pipeline as Code
Pipeline as CodePipeline as Code
Pipeline as Code
 
Devops 101 QA with Kris Buytaert
Devops 101 QA  with Kris BuytaertDevops 101 QA  with Kris Buytaert
Devops 101 QA with Kris Buytaert
 
Automating MySQL operations with Puppet
Automating MySQL operations with PuppetAutomating MySQL operations with Puppet
Automating MySQL operations with Puppet
 

Similar to From MonitoringSucks to Monitoring Love , 2016 Edition

Open Source Monitoring in 2019
Open Source Monitoring in 2019 Open Source Monitoring in 2019
Open Source Monitoring in 2019
Kris Buytaert
 
Monitoring in an Infrastructure as Code Age
Monitoring in an Infrastructure as Code AgeMonitoring in an Infrastructure as Code Age
Monitoring in an Infrastructure as Code Age
Puppet
 
OSDC 2016 - Another 7 Tools for your #devops Stack by Kris Buytaert
OSDC 2016 - Another 7 Tools for your #devops Stack by Kris BuytaertOSDC 2016 - Another 7 Tools for your #devops Stack by Kris Buytaert
OSDC 2016 - Another 7 Tools for your #devops Stack by Kris Buytaert
NETWAYS
 
OSMC 2014: From monitoringsucks to monitoringlove (and back) | Kris Buytaert
OSMC 2014: From monitoringsucks to monitoringlove (and back) | Kris BuytaertOSMC 2014: From monitoringsucks to monitoringlove (and back) | Kris Buytaert
OSMC 2014: From monitoringsucks to monitoringlove (and back) | Kris Buytaert
NETWAYS
 
OSMC 2014 | From monitoringsucks to monitoringlove, and back by Kris Buytaert
OSMC 2014 | From monitoringsucks to monitoringlove, and back by Kris BuytaertOSMC 2014 | From monitoringsucks to monitoringlove, and back by Kris Buytaert
OSMC 2014 | From monitoringsucks to monitoringlove, and back by Kris Buytaert
NETWAYS
 
From Config Management Sucks to #cfgmgmtlove
From Config Management Sucks to #cfgmgmtlove From Config Management Sucks to #cfgmgmtlove
From Config Management Sucks to #cfgmgmtlove
Kris Buytaert
 
Run stuff, Deploy Stuff, Jax London 2017 Edition
Run stuff, Deploy Stuff, Jax London 2017 EditionRun stuff, Deploy Stuff, Jax London 2017 Edition
Run stuff, Deploy Stuff, Jax London 2017 Edition
Kris Buytaert
 
When traditional configuration management is to slow for your needs
When traditional configuration management is to slow for your needsWhen traditional configuration management is to slow for your needs
When traditional configuration management is to slow for your needs
Kris Buytaert
 
Run stuff, Deploy Stuff
Run stuff, Deploy StuffRun stuff, Deploy Stuff
Run stuff, Deploy Stuff
Kris Buytaert
 
Continuous Infrastructure First
Continuous Infrastructure FirstContinuous Infrastructure First
Continuous Infrastructure First
Kris Buytaert
 
Pipeline as code for your infrastructure as Code
Pipeline as code for your infrastructure as CodePipeline as code for your infrastructure as Code
Pipeline as code for your infrastructure as Code
Kris Buytaert
 
Can we fix dev-oops ?
Can we fix dev-oops ?Can we fix dev-oops ?
Can we fix dev-oops ?
Kris Buytaert
 
Monitoring with ElasticSearch
Monitoring with ElasticSearch Monitoring with ElasticSearch
Monitoring with ElasticSearch
Kris Buytaert
 
OSDC 2015: Kris Buytaert | From ConfigManagementSucks to ConfigManagementLove
OSDC 2015: Kris Buytaert | From ConfigManagementSucks to ConfigManagementLoveOSDC 2015: Kris Buytaert | From ConfigManagementSucks to ConfigManagementLove
OSDC 2015: Kris Buytaert | From ConfigManagementSucks to ConfigManagementLove
NETWAYS
 
Its not about the tooling
Its not about the toolingIts not about the tooling
Its not about the tooling
Bram Vogelaar
 
Horacio Gonzalez - Monitoring OVH - Codemotion Amsterdam 2019
Horacio Gonzalez - Monitoring OVH - Codemotion Amsterdam 2019Horacio Gonzalez - Monitoring OVH - Codemotion Amsterdam 2019
Horacio Gonzalez - Monitoring OVH - Codemotion Amsterdam 2019
Codemotion
 
Icinga Camp Amsterdam - Infrastructure as Code
Icinga Camp Amsterdam - Infrastructure as CodeIcinga Camp Amsterdam - Infrastructure as Code
Icinga Camp Amsterdam - Infrastructure as Code
Icinga
 
Continuous Delivery of (y)our infrastructure.
Continuous Delivery of (y)our infrastructure.Continuous Delivery of (y)our infrastructure.
Continuous Delivery of (y)our infrastructure.
Kris Buytaert
 
Pinto+Stratopan+Love
Pinto+Stratopan+LovePinto+Stratopan+Love
Pinto+Stratopan+Love
Jeffrey Ryan Thalhammer
 
Continuous Infrastructure First Ignite Edition
Continuous Infrastructure First  Ignite EditionContinuous Infrastructure First  Ignite Edition
Continuous Infrastructure First Ignite Edition
Kris Buytaert
 

Similar to From MonitoringSucks to Monitoring Love , 2016 Edition (20)

Open Source Monitoring in 2019
Open Source Monitoring in 2019 Open Source Monitoring in 2019
Open Source Monitoring in 2019
 
Monitoring in an Infrastructure as Code Age
Monitoring in an Infrastructure as Code AgeMonitoring in an Infrastructure as Code Age
Monitoring in an Infrastructure as Code Age
 
OSDC 2016 - Another 7 Tools for your #devops Stack by Kris Buytaert
OSDC 2016 - Another 7 Tools for your #devops Stack by Kris BuytaertOSDC 2016 - Another 7 Tools for your #devops Stack by Kris Buytaert
OSDC 2016 - Another 7 Tools for your #devops Stack by Kris Buytaert
 
OSMC 2014: From monitoringsucks to monitoringlove (and back) | Kris Buytaert
OSMC 2014: From monitoringsucks to monitoringlove (and back) | Kris BuytaertOSMC 2014: From monitoringsucks to monitoringlove (and back) | Kris Buytaert
OSMC 2014: From monitoringsucks to monitoringlove (and back) | Kris Buytaert
 
OSMC 2014 | From monitoringsucks to monitoringlove, and back by Kris Buytaert
OSMC 2014 | From monitoringsucks to monitoringlove, and back by Kris BuytaertOSMC 2014 | From monitoringsucks to monitoringlove, and back by Kris Buytaert
OSMC 2014 | From monitoringsucks to monitoringlove, and back by Kris Buytaert
 
From Config Management Sucks to #cfgmgmtlove
From Config Management Sucks to #cfgmgmtlove From Config Management Sucks to #cfgmgmtlove
From Config Management Sucks to #cfgmgmtlove
 
Run stuff, Deploy Stuff, Jax London 2017 Edition
Run stuff, Deploy Stuff, Jax London 2017 EditionRun stuff, Deploy Stuff, Jax London 2017 Edition
Run stuff, Deploy Stuff, Jax London 2017 Edition
 
When traditional configuration management is to slow for your needs
When traditional configuration management is to slow for your needsWhen traditional configuration management is to slow for your needs
When traditional configuration management is to slow for your needs
 
Run stuff, Deploy Stuff
Run stuff, Deploy StuffRun stuff, Deploy Stuff
Run stuff, Deploy Stuff
 
Continuous Infrastructure First
Continuous Infrastructure FirstContinuous Infrastructure First
Continuous Infrastructure First
 
Pipeline as code for your infrastructure as Code
Pipeline as code for your infrastructure as CodePipeline as code for your infrastructure as Code
Pipeline as code for your infrastructure as Code
 
Can we fix dev-oops ?
Can we fix dev-oops ?Can we fix dev-oops ?
Can we fix dev-oops ?
 
Monitoring with ElasticSearch
Monitoring with ElasticSearch Monitoring with ElasticSearch
Monitoring with ElasticSearch
 
OSDC 2015: Kris Buytaert | From ConfigManagementSucks to ConfigManagementLove
OSDC 2015: Kris Buytaert | From ConfigManagementSucks to ConfigManagementLoveOSDC 2015: Kris Buytaert | From ConfigManagementSucks to ConfigManagementLove
OSDC 2015: Kris Buytaert | From ConfigManagementSucks to ConfigManagementLove
 
Its not about the tooling
Its not about the toolingIts not about the tooling
Its not about the tooling
 
Horacio Gonzalez - Monitoring OVH - Codemotion Amsterdam 2019
Horacio Gonzalez - Monitoring OVH - Codemotion Amsterdam 2019Horacio Gonzalez - Monitoring OVH - Codemotion Amsterdam 2019
Horacio Gonzalez - Monitoring OVH - Codemotion Amsterdam 2019
 
Icinga Camp Amsterdam - Infrastructure as Code
Icinga Camp Amsterdam - Infrastructure as CodeIcinga Camp Amsterdam - Infrastructure as Code
Icinga Camp Amsterdam - Infrastructure as Code
 
Continuous Delivery of (y)our infrastructure.
Continuous Delivery of (y)our infrastructure.Continuous Delivery of (y)our infrastructure.
Continuous Delivery of (y)our infrastructure.
 
Pinto+Stratopan+Love
Pinto+Stratopan+LovePinto+Stratopan+Love
Pinto+Stratopan+Love
 
Continuous Infrastructure First Ignite Edition
Continuous Infrastructure First  Ignite EditionContinuous Infrastructure First  Ignite Edition
Continuous Infrastructure First Ignite Edition
 

More from Kris Buytaert

Years of (not) learning , from devops to devoops
Years of (not) learning , from devops to devoopsYears of (not) learning , from devops to devoops
Years of (not) learning , from devops to devoops
Kris Buytaert
 
Observability will not fix your Broken Monitoring ,Ignite
Observability will not fix your Broken Monitoring ,IgniteObservability will not fix your Broken Monitoring ,Ignite
Observability will not fix your Broken Monitoring ,Ignite
Kris Buytaert
 
Infrastructure as Code Patterns
Infrastructure as Code PatternsInfrastructure as Code Patterns
Infrastructure as Code Patterns
Kris Buytaert
 
From devoops to devops 13 years of (not) learning
From devoops to devops 13 years of (not) learningFrom devoops to devops 13 years of (not) learning
From devoops to devops 13 years of (not) learning
Kris Buytaert
 
Pipeline all the Dashboards as Code
Pipeline all the Dashboards as CodePipeline all the Dashboards as Code
Pipeline all the Dashboards as Code
Kris Buytaert
 
Help , My Datacenter is on fire
Help , My Datacenter is on fireHelp , My Datacenter is on fire
Help , My Datacenter is on fire
Kris Buytaert
 
GitOps , done Right
GitOps , done RightGitOps , done Right
GitOps , done Right
Kris Buytaert
 
Devops is Dead, Long live Devops
Devops is Dead, Long live DevopsDevops is Dead, Long live Devops
Devops is Dead, Long live Devops
Kris Buytaert
 
10 years of #devopsdays, but what have we really learned ?
10 years of #devopsdays, but what have we really learned ? 10 years of #devopsdays, but what have we really learned ?
10 years of #devopsdays, but what have we really learned ?
Kris Buytaert
 
Is there a Future for devops ?
Is there a Future for devops   ? Is there a Future for devops   ?
Is there a Future for devops ?
Kris Buytaert
 
10 Years of #devopsdays weirdness
10 Years of #devopsdays weirdness10 Years of #devopsdays weirdness
10 Years of #devopsdays weirdness
Kris Buytaert
 
ADDO 2019: Looking back at over 10 years of Devops
ADDO 2019:    Looking back at over 10 years of DevopsADDO 2019:    Looking back at over 10 years of Devops
ADDO 2019: Looking back at over 10 years of Devops
Kris Buytaert
 
Migrating to Puppet 5
Migrating to Puppet 5Migrating to Puppet 5
Migrating to Puppet 5
Kris Buytaert
 
Deploying your SaaS stack OnPrem
Deploying your SaaS stack OnPremDeploying your SaaS stack OnPrem
Deploying your SaaS stack OnPrem
Kris Buytaert
 
Looking back at 5 years of #cfgmgmtcamp
Looking back at 5 years of #cfgmgmtcampLooking back at 5 years of #cfgmgmtcamp
Looking back at 5 years of #cfgmgmtcamp
Kris Buytaert
 
Looking back at 7.5 years of Devopsdays , DOd PDX
Looking back at 7.5 years of Devopsdays , DOd PDXLooking back at 7.5 years of Devopsdays , DOd PDX
Looking back at 7.5 years of Devopsdays , DOd PDX
Kris Buytaert
 
Devopsdays Amsterdam 2017 Keynote, looking back at 5 years of AMS
Devopsdays Amsterdam 2017 Keynote, looking back at 5 years of AMSDevopsdays Amsterdam 2017 Keynote, looking back at 5 years of AMS
Devopsdays Amsterdam 2017 Keynote, looking back at 5 years of AMS
Kris Buytaert
 

More from Kris Buytaert (17)

Years of (not) learning , from devops to devoops
Years of (not) learning , from devops to devoopsYears of (not) learning , from devops to devoops
Years of (not) learning , from devops to devoops
 
Observability will not fix your Broken Monitoring ,Ignite
Observability will not fix your Broken Monitoring ,IgniteObservability will not fix your Broken Monitoring ,Ignite
Observability will not fix your Broken Monitoring ,Ignite
 
Infrastructure as Code Patterns
Infrastructure as Code PatternsInfrastructure as Code Patterns
Infrastructure as Code Patterns
 
From devoops to devops 13 years of (not) learning
From devoops to devops 13 years of (not) learningFrom devoops to devops 13 years of (not) learning
From devoops to devops 13 years of (not) learning
 
Pipeline all the Dashboards as Code
Pipeline all the Dashboards as CodePipeline all the Dashboards as Code
Pipeline all the Dashboards as Code
 
Help , My Datacenter is on fire
Help , My Datacenter is on fireHelp , My Datacenter is on fire
Help , My Datacenter is on fire
 
GitOps , done Right
GitOps , done RightGitOps , done Right
GitOps , done Right
 
Devops is Dead, Long live Devops
Devops is Dead, Long live DevopsDevops is Dead, Long live Devops
Devops is Dead, Long live Devops
 
10 years of #devopsdays, but what have we really learned ?
10 years of #devopsdays, but what have we really learned ? 10 years of #devopsdays, but what have we really learned ?
10 years of #devopsdays, but what have we really learned ?
 
Is there a Future for devops ?
Is there a Future for devops   ? Is there a Future for devops   ?
Is there a Future for devops ?
 
10 Years of #devopsdays weirdness
10 Years of #devopsdays weirdness10 Years of #devopsdays weirdness
10 Years of #devopsdays weirdness
 
ADDO 2019: Looking back at over 10 years of Devops
ADDO 2019:    Looking back at over 10 years of DevopsADDO 2019:    Looking back at over 10 years of Devops
ADDO 2019: Looking back at over 10 years of Devops
 
Migrating to Puppet 5
Migrating to Puppet 5Migrating to Puppet 5
Migrating to Puppet 5
 
Deploying your SaaS stack OnPrem
Deploying your SaaS stack OnPremDeploying your SaaS stack OnPrem
Deploying your SaaS stack OnPrem
 
Looking back at 5 years of #cfgmgmtcamp
Looking back at 5 years of #cfgmgmtcampLooking back at 5 years of #cfgmgmtcamp
Looking back at 5 years of #cfgmgmtcamp
 
Looking back at 7.5 years of Devopsdays , DOd PDX
Looking back at 7.5 years of Devopsdays , DOd PDXLooking back at 7.5 years of Devopsdays , DOd PDX
Looking back at 7.5 years of Devopsdays , DOd PDX
 
Devopsdays Amsterdam 2017 Keynote, looking back at 5 years of AMS
Devopsdays Amsterdam 2017 Keynote, looking back at 5 years of AMSDevopsdays Amsterdam 2017 Keynote, looking back at 5 years of AMS
Devopsdays Amsterdam 2017 Keynote, looking back at 5 years of AMS
 

Recently uploaded

Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
Wouter Lemaire
 
CAKE: Sharing Slices of Confidential Data on Blockchain
CAKE: Sharing Slices of Confidential Data on BlockchainCAKE: Sharing Slices of Confidential Data on Blockchain
CAKE: Sharing Slices of Confidential Data on Blockchain
Claudio Di Ciccio
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
SitimaJohn
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
David Brossard
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
Daiki Mogmet Ito
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
tolgahangng
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Edge AI and Vision Alliance
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
Claudio Di Ciccio
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 

Recently uploaded (20)

Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
 
CAKE: Sharing Slices of Confidential Data on Blockchain
CAKE: Sharing Slices of Confidential Data on BlockchainCAKE: Sharing Slices of Confidential Data on Blockchain
CAKE: Sharing Slices of Confidential Data on Blockchain
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 

From MonitoringSucks to Monitoring Love , 2016 Edition

  • 1. From #MonitoringSucks toFrom #MonitoringSucks to #MonitoringLove#MonitoringLove Open Source Monitoring in 2016Open Source Monitoring in 2016 @KrisBuytaert FlossUK 2016, London , UK
  • 2. Kris BuytaertKris Buytaert ● I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @inuits.euConsultant @inuits.eu ● Everything is an effing DNS ProblemEverything is an effing DNS Problem ● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore ● Organising ConferencesOrganising Conferences ● Evangelizing devopsEvangelizing devops
  • 3. An opinionated talk about the Open SourceAn opinionated talk about the Open Source Monitoring tooling landscapeMonitoring tooling landscape In which I hope to learn from YOUIn which I hope to learn from YOU
  • 4. #devops=~C(L)AMS#devops=~C(L)AMS ● CultureCulture ● (Lean)(Lean) ● AutomationAutomation ● Monitoring and MeasurementMonitoring and Measurement ● SharingSharing Damon Edwards and John WillisDamon Edwards and John Willis Gene KimGene Kim
  • 5. Monitoring is usually anMonitoring is usually an aftertoughtaftertought ENOBUDGET, ENOTIMEENOBUDGET, ENOTIME
  • 6. An 2008 OLS PaperAn 2008 OLS Paper ● We have bloated Java toolsWe have bloated Java tools ● Some open Core stuffSome open Core stuff ● DYI folks want traditional NagiosDYI folks want traditional Nagios ● DBA RequiredDBA Required
  • 7. #monitoringsucks#monitoringsucks ● John Vincent (@lusis), june 2011John Vincent (@lusis), june 2011 ● A sub #devops movementA sub #devops movement ● https://github.com/monitoringsucks/https://github.com/monitoringsucks/
  • 8. Why #monitoringsucksWhy #monitoringsucks ● Manual config (gui)Manual config (gui) ● Not in sync with realityNot in sync with reality ● Hosts onlyHosts only ● Services sometimesServices sometimes ● Application neverApplication never ● Chaos or out of sync with realityChaos or out of sync with reality ● Alert FatigueAlert Fatigue
  • 9. #monitoringlove#monitoringlove • • Ulf Mansson #devopsdays Rome 2011Ulf Mansson #devopsdays Rome 2011 • A new era of toolingA new era of tooling • #monitoringlove hacksessions @inuits#monitoringlove hacksessions @inuits • #monitorama#monitorama
  • 10. What we wantWhat we want ● Small , well suited componentsSmall , well suited components • CollectCollect • Transport / MangleTransport / Mangle • StoreStore • AnalyseAnalyse • Act / AlertAct / Alert • VisualizeVisualize
  • 11.
  • 12.
  • 13.
  • 14. #monitoringlove#monitoringlove But the love was about :But the love was about :
  • 15. SensuSensu ● Awesome for non staticAwesome for non static environmentsenvironments ● Scaling a clustered RabbitMQ ?Scaling a clustered RabbitMQ ? ● This is Europe, U no do cloudThis is Europe, U no do cloud
  • 16. Automation ofAutomation of #monitoring#monitoring brought backbrought back thethe #love#love
  • 18. Monitoring aMonitoring a serviceservice vsvs Monitoring aMonitoring a ServiceService
  • 19. definition of done:definition of done: monitored and in productionmonitored and in production
  • 20. A software project is not doneA software project is not done until your last end user is deaduntil your last end user is dead
  • 21. Culture,Culture, Automation,Automation, Measurement :Measurement : measure all the thingsmeasure all the things SharingSharing
  • 22. CollectD all the metrics,CollectD all the metrics, at high intervalsat high intervals
  • 25. Draw as InfiniteDraw as Infinite ● Time To DeployTime To Deploy ● DeployDeploy FrequencyFrequency ● LifecycleLifecycle frequencyfrequency ● Map to otherMap to other metricsmetrics
  • 26. Graph-ExplorerGraph-Explorer ● (Vimeo)(Vimeo) ● Metrics 2.0Metrics 2.0 ● Add EventsAdd Events
  • 28. Graphs to KnowledgeGraphs to Knowledge SkylineSkyline • OculusOculus • Creating Information out of this dataCreating Information out of this data • Big dataBig data • Machine LearningMachine Learning
  • 29. AggregationAggregation ● Alert on streamsAlert on streams ● Alert on aggregated metricsAlert on aggregated metrics
  • 30.
  • 31.
  • 32. RiemannRiemann ● I still don't get it ?I still don't get it ? ● Distributed TopDistributed Top ● Do you like Clojure ?Do you like Clojure ? ● Riemann Health plugin ?Riemann Health plugin ? ● s/riemann-health/collectd/g;s/riemann-health/collectd/g; ● Output to graphiteOutput to graphite
  • 33. PrometheusPrometheus ● Started 2012Started 2012 ● SoundCloudSoundCloud ● Metrics BasedMetrics Based ● ScrapesScrapes EndpointsEndpoints • ExistingExisting endpoints forendpoints for limited toolslimited tools ● GraphiteGraphite ExporterExporter ● Push GatewayPush Gateway ● Great AlertingGreat Alerting ● Might needMight need some codingsome coding
  • 34. But I have log files..But I have log files..
  • 35.
  • 36.
  • 37.
  • 38. Logs and MetricsLogs and Metrics ● Graylog2Graylog2 ● ELSA (Enterprise Log Search andELSA (Enterprise Log Search and Archive)Archive) ● ELK StackELK Stack
  • 39. ● Collect fromCollect from anywhereanywhere ● FilterFilter ● Send anywhereSend anywhere
  • 40.
  • 41. Infitnite Diskspace ?Infitnite Diskspace ? ● Logstash outputLogstash output • Statsd => GraphiteStatsd => Graphite • Keep patterns around,Keep patterns around, • Selectively purge dataSelectively purge data •
  • 42. APMAPM But what about my apps ?But what about my apps ? Half the world cheers about SAASHalf the world cheers about SAAS tools :(tools :(
  • 43. PacketbeatPacketbeat ● Traffic FlowTraffic Flow through networkthrough network ● TransactionsTransactions causing errroscausing errros ● SQL per HTTPSQL per HTTP ● API call usageAPI call usage
  • 45. Beats ?Beats ? ● Elasti.coElasti.co ● Collect, Parse and ShipCollect, Parse and Ship ● Q: Is all the data you care aboutQ: Is all the data you care about suitable for Elastic Search ?suitable for Elastic Search ? ● What about Long Term Storage ?What about Long Term Storage ? ● Do you even want to build alertingDo you even want to build alerting from this ?from this ?
  • 46. Checking for FailureChecking for Failure ● IcingaIcinga • Automated config generationAutomated config generation ● SensuSensu • CloudstyleCloudstyle ● PrometheusPrometheus • Metric basedMetric based
  • 47. Waking you up at nightWaking you up at night ● FlapjackFlapjack flapjack.ioflapjack.io monitoring notification routing +monitoring notification routing + event processing systemevent processing system ● OpenDutyOpenDuty github.com/szechuen/OpenDutygithub.com/szechuen/OpenDuty Duty managementDuty management
  • 49. Our Current StackOur Current Stack
  • 50. I love where Monitoring is headingI love where Monitoring is heading We have much less false positives these daysWe have much less false positives these days
  • 51. ContactContact Kris Buytaert kris.buytaert@inuits.euKris Buytaert kris.buytaert@inuits.eu Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.eu/http://www.inuits.eu/ Find Inuits inFind Inuits in Brasschaat,Ghent,Brasschaat,Ghent, Rotterdam,Prague,Rotterdam,Prague, Kiev,BrnoKiev,Brno