From #monitoringsucks to #monitoringlove , the State of Open Source monitoring in #devops in 2019

1. From #MonitoringSucks toFrom #MonitoringSucks to
#MonitoringLove#MonitoringLove
Open Source Monitoring in 2018-2019Open Source Monitoring in 2018-2019
@KrisBuytaert
Devops Meetup, Brno

2. Kris BuytaertKris Buytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● Chief Twitter Ofcer and Open SourceChief Twitter Ofcer and Open Source
Consultant @inuits.euConsultant @inuits.eu
● Everything is an efng DNS ProblemEverything is an efng DNS Problem
● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore
● Organising ConferencesOrganising Conferences
● Evangelizing devopsEvangelizing devops

3. An opinionated talk about the Open SourceAn opinionated talk about the Open Source
Monitoring tooling landscapeMonitoring tooling landscape
In which I hope to learn from YOUIn which I hope to learn from YOU

4. #devops=~C(L)AMS#devops=~C(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● Monitoring and MeasurementMonitoring and Measurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim

5. Monitoring is usually anMonitoring is usually an
aftertoughtaftertought
ENOBUDGET, ENOTIMEENOBUDGET, ENOTIME

6. AnAn 20082008 OLS PaperOLS Paper
● We have bloated Java toolsWe have bloated Java tools
● Some open Core stufSome open Core stuf
● DYI folks want traditional NagiosDYI folks want traditional Nagios
● DBA RequiredDBA Required

7. #monitoringsucks#monitoringsucks
● John Vincent (@lusis), june 2011John Vincent (@lusis), june 2011
● A sub #devops movementA sub #devops movement
● https://github.com/monitoringsucks/https://github.com/monitoringsucks/

8. Why #monitoringsucksWhy #monitoringsucks
● Manual confg (gui)Manual confg (gui)
● Not in sync with realityNot in sync with reality
● Hosts onlyHosts only
● Services sometimesServices sometimes
● Application neverApplication never
● Chaos or out of sync with realityChaos or out of sync with reality
● Alert FatigueAlert Fatigue

9. #monitoringlove#monitoringlove
•
•
Ulf Mansson #devopsdays Rome 2011Ulf Mansson #devopsdays Rome 2011
•
A new era of toolingA new era of tooling
• #monitoringlove hacksessions @inuits#monitoringlove hacksessions @inuits
•
#monitorama#monitorama

10. What we wantWhat we want
● Small , well suited componentsSmall , well suited components
•
CollectCollect
•
Transport / MangleTransport / Mangle
•
StoreStore
•
AnalyseAnalyse
•
Act / AlertAct / Alert
•
VisualizeVisualize

13. The love was : SensuThe love was : Sensu
● Awesome for non staticAwesome for non static
environmentsenvironments
● Scaling a clustered RabbitMQ ?Scaling a clustered RabbitMQ ?
● Looking more and more likeLooking more and more like
PrometheusPrometheus
● This is Europe, U no do cloudThis is Europe, U no do cloud

14. Automation ofAutomation of
#monitoring#monitoring
brought backbrought back
thethe #love#love

15. There is no such thingThere is no such thing
as “Service” discoveryas “Service” discovery

16. Monitoring aMonitoring a serviceservice
vsvs
Monitoring aMonitoring a ServiceService

17. AutomationAutomation

18. defnition of done:defnition of done:
monitored and in productionmonitored and in production

19. A software project is not doneA software project is not done
until your last end user is deaduntil your last end user is dead

20. Culture,Culture,
Automation,Automation,
Measurement :Measurement :
measure all the thingsmeasure all the things
SharingSharing

21. Collection :Collection :
● Collectd,Collectd,
● DiamondDiamond
● From applicationFrom application
● Custom ExportersCustom Exporters
● LogsLogs
● LogsLogs

22. NetDataNetData
● GranularityGranularity
● Debug vsDebug vs
constant prod ?constant prod ?
●

23. Transport / Ship / Mangle:Transport / Ship / Mangle:
● Collectd / Diamond / TelegrafCollectd / Diamond / Telegraf
● (r)syslog, Beats, logstash(r)syslog, Beats, logstash
● Q , Nats, ActiveMQ, RabbitMQQ , Nats, ActiveMQ, RabbitMQ

24. ● Collect fromCollect from
anywhereanywhere
● FilterFilter
● Send anywhereSend anywhere

25. Store :Store :
● TSDB : Time Series DBTSDB : Time Series DB
● Optimized DB for Time SeriesOptimized DB for Time Series
● Graphite/ Infux / OpenTSDB / ....Graphite/ Infux / OpenTSDB / ....
● ElasticElastic
● Long Term vs Short Term StorageLong Term vs Short Term Storage

26. Oldschool graphiteOldschool graphite

27. PrometheusPrometheus
● Started 2012Started 2012
● SoundCloudSoundCloud
● Metrics BasedMetrics Based
● ScrapesScrapes
EndpointsEndpoints
•
ExistingExisting
endpoints forendpoints for
limited toolslimited tools
● GraphiteGraphite
ExporterExporter
● Push GatewayPush Gateway
● Great AlertingGreat Alerting

28. PrometheusPrometheus
● Mostly for Short TermMostly for Short Term
● Still Ship longterm metrics to otherStill Ship longterm metrics to other
TSDBTSDB
● Nginx gw’s all over the placeNginx gw’s all over the place
•
(ssl fun)(ssl fun)

29. Infnite Diskspace ?Infnite Diskspace ?
● Logstash outputLogstash output
•
Statsd => GraphiteStatsd => Graphite
•
Keep patterns around,Keep patterns around,
•
Selectively purge dataSelectively purge data
● Prometheus for Short TermPrometheus for Short Term
•
Graphite for Long termGraphite for Long term

30. Log AlternativesLog Alternatives
● Graylog2Graylog2
● ELSA (Enterprise Log Search andELSA (Enterprise Log Search and
Archive)Archive)
● ELK StackELK Stack
● FluentdFluentd

31. Prometheus ?Prometheus ?
● Only For Containers ?Only For Containers ?
● Also for other setups !Also for other setups !
● Is this sufcient ?Is this sufcient ?

32. Act / Alert:Act / Alert:

33. Checking for FailureChecking for Failure
● IcingaIcinga
•
Automated confg generationAutomated confg generation
● SensuSensu
•
CloudstyleCloudstyle
● PrometheusPrometheus
•
AlertManagerAlertManager

34. Waking you up at nightWaking you up at night
● FlapjackFlapjack
fapjack.iofapjack.io
monitoring notifcation routing +monitoring notifcation routing +
event processing systemevent processing system
● OpenDutyOpenDuty
github.com/szechuen/OpenDutygithub.com/szechuen/OpenDuty
Duty managementDuty management

35. Waking you up at nightWaking you up at night
● AnagAnag
● Custom written stufCustom written stuf

36. Analyse:Analyse:

37. Basic SearchBasic Search

38. Graphs to KnowledgeGraphs to Knowledge
SkylineSkyline
•
OculusOculus
•
Creating Information out of this dataCreating Information out of this data
•
Big dataBig data
•
Machine LearningMachine Learning
•
Hastic.ioHastic.io

39. Hastic.ioHastic.io
● Open Source Pattern DetectionOpen Source Pattern Detection
● Label patterns → Wait for learning toLabel patterns → Wait for learning to
complete → Get detectionscomplete → Get detections
● Hastic Server + Grafana AppHastic Server + Grafana App

40. LogIslandsLogIslands
● Complex event processing &Complex event processing &
patterns mining at scalepatterns mining at scale
● Kafka, Nif, Spark, HadoopKafka, Nif, Spark, Hadoop

41. Visualize:Visualize:

42. KibanaKibana

43. GrafanaGrafana

44. ChallengeChallenge
● *ana as code*ana as code
● Template your ...Template your ...
● e.g grafonnet-libe.g grafonnet-lib
•
A jsonnet lib to generate GrafanaA jsonnet lib to generate Grafana
dashboards ...dashboards ...

45. AggregatingAggregating
● ThrukThruk
● GrafanaGrafana
● DashingDashing

46. Lack of change ?Lack of change ?
● Limited # new toolsLimited # new tools
● Feature Complete ?Feature Complete ?

47. Is prometheus the newIs prometheus the new
Docker ?Docker ?

48. APMAPM
Application Performance MonitoringApplication Performance Monitoring
But what about my apps ?But what about my apps ?
● agent required that ties to codeagent required that ties to code
● Code modifcationsCode modifcations

49. Old PacketBeatOld PacketBeat

50. Open Source “APM”Open Source “APM”
● Scouter
● Jaeger
● Kamon
● Zipkin
● Beats ...
● Performance Co Pilot
● Kamon
● Pinpoint
● Micrometer
● StageMonitor
● SkyWalking
● Kieker
=> Huge focus on the Java Ecosystem , little
options for PHP/ Python / Ruby shops.

51. OpenAPM.ioOpenAPM.io

52. OpenTracing 101OpenTracing 101
● The problem : It was not reasonable to ask all OSS services and all OSSThe problem : It was not reasonable to ask all OSS services and all OSS
packages and all application-specifc code to use a single tracingpackages and all application-specifc code to use a single tracing
vendor => Open Ttracingvendor => Open Ttracing
● Distributed Tracing StandardDistributed Tracing Standard
● CNCFCNCF
● Dapper inside GoogleDapper inside Google
● ““OpenTracing is not a download or a program. Distributed tracingOpenTracing is not a download or a program. Distributed tracing
requires that software developers add instrumentation to the code ofrequires that software developers add instrumentation to the code of
an application, or to the frameworks used in the application”an application, or to the frameworks used in the application”

53. Complexity is the EnemyComplexity is the Enemy
of Reliabilityof Reliability

54. I love where Monitoring is headingI love where Monitoring is heading
““Wait , was I oncall last week ?”Wait , was I oncall last week ?”
True words said by one of our oncall engineersTrue words said by one of our oncall engineers

55. OpservabilityOpservability

56. ContactContact
Kris Buytaert kris.buytaert@inuits.euKris Buytaert kris.buytaert@inuits.eu
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.eu/http://www.inuits.eu/
Find Inuits inFind Inuits in
Brasschaat,Ghent,Brasschaat,Ghent,
Rotterdam,Prague,Rotterdam,Prague,
Kiev,BrnoKiev,Brno