Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Open Source Monitoring in 2019

929 views

Published on

From #monitoringsucks to #monitoringlove , the State of Open Source monitoring in #devops in 2019

Published in: Technology
  • Be the first to comment

Open Source Monitoring in 2019

  1. 1. From #MonitoringSucks toFrom #MonitoringSucks to #MonitoringLove#MonitoringLove Open Source Monitoring in 2018-2019Open Source Monitoring in 2018-2019 @KrisBuytaert Devops Meetup, Brno
  2. 2. Kris BuytaertKris Buytaert ● I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Twitter Ofcer and Open SourceChief Twitter Ofcer and Open Source Consultant @inuits.euConsultant @inuits.eu ● Everything is an efng DNS ProblemEverything is an efng DNS Problem ● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore ● Organising ConferencesOrganising Conferences ● Evangelizing devopsEvangelizing devops
  3. 3. An opinionated talk about the Open SourceAn opinionated talk about the Open Source Monitoring tooling landscapeMonitoring tooling landscape In which I hope to learn from YOUIn which I hope to learn from YOU
  4. 4. #devops=~C(L)AMS#devops=~C(L)AMS ● CultureCulture ● (Lean)(Lean) ● AutomationAutomation ● Monitoring and MeasurementMonitoring and Measurement ● SharingSharing Damon Edwards and John WillisDamon Edwards and John Willis Gene KimGene Kim
  5. 5. Monitoring is usually anMonitoring is usually an aftertoughtaftertought ENOBUDGET, ENOTIMEENOBUDGET, ENOTIME
  6. 6. AnAn 20082008 OLS PaperOLS Paper ● We have bloated Java toolsWe have bloated Java tools ● Some open Core stufSome open Core stuf ● DYI folks want traditional NagiosDYI folks want traditional Nagios ● DBA RequiredDBA Required
  7. 7. #monitoringsucks#monitoringsucks ● John Vincent (@lusis), june 2011John Vincent (@lusis), june 2011 ● A sub #devops movementA sub #devops movement ● https://github.com/monitoringsucks/https://github.com/monitoringsucks/
  8. 8. Why #monitoringsucksWhy #monitoringsucks ● Manual confg (gui)Manual confg (gui) ● Not in sync with realityNot in sync with reality ● Hosts onlyHosts only ● Services sometimesServices sometimes ● Application neverApplication never ● Chaos or out of sync with realityChaos or out of sync with reality ● Alert FatigueAlert Fatigue
  9. 9. #monitoringlove#monitoringlove • • Ulf Mansson #devopsdays Rome 2011Ulf Mansson #devopsdays Rome 2011 • A new era of toolingA new era of tooling • #monitoringlove hacksessions @inuits#monitoringlove hacksessions @inuits • #monitorama#monitorama
  10. 10. What we wantWhat we want ● Small , well suited componentsSmall , well suited components • CollectCollect • Transport / MangleTransport / Mangle • StoreStore • AnalyseAnalyse • Act / AlertAct / Alert • VisualizeVisualize
  11. 11. The love was : SensuThe love was : Sensu ● Awesome for non staticAwesome for non static environmentsenvironments ● Scaling a clustered RabbitMQ ?Scaling a clustered RabbitMQ ? ● Looking more and more likeLooking more and more like PrometheusPrometheus ● This is Europe, U no do cloudThis is Europe, U no do cloud
  12. 12. Automation ofAutomation of #monitoring#monitoring brought backbrought back thethe #love#love
  13. 13. There is no such thingThere is no such thing as “Service” discoveryas “Service” discovery
  14. 14. Monitoring aMonitoring a serviceservice vsvs Monitoring aMonitoring a ServiceService
  15. 15. AutomationAutomation
  16. 16. defnition of done:defnition of done: monitored and in productionmonitored and in production
  17. 17. A software project is not doneA software project is not done until your last end user is deaduntil your last end user is dead
  18. 18. Culture,Culture, Automation,Automation, Measurement :Measurement : measure all the thingsmeasure all the things SharingSharing
  19. 19. Collection :Collection : ● Collectd,Collectd, ● DiamondDiamond ● From applicationFrom application ● Custom ExportersCustom Exporters ● LogsLogs ● LogsLogs
  20. 20. NetDataNetData ● GranularityGranularity ● Debug vsDebug vs constant prod ?constant prod ? ●
  21. 21. Transport / Ship / Mangle:Transport / Ship / Mangle: ● Collectd / Diamond / TelegrafCollectd / Diamond / Telegraf ● (r)syslog, Beats, logstash(r)syslog, Beats, logstash ● Q , Nats, ActiveMQ, RabbitMQQ , Nats, ActiveMQ, RabbitMQ
  22. 22. ● Collect fromCollect from anywhereanywhere ● FilterFilter ● Send anywhereSend anywhere
  23. 23. Store :Store : ● TSDB : Time Series DBTSDB : Time Series DB ● Optimized DB for Time SeriesOptimized DB for Time Series ● Graphite/ Infux / OpenTSDB / ....Graphite/ Infux / OpenTSDB / .... ● ElasticElastic ● Long Term vs Short Term StorageLong Term vs Short Term Storage
  24. 24. Oldschool graphiteOldschool graphite
  25. 25. PrometheusPrometheus ● Started 2012Started 2012 ● SoundCloudSoundCloud ● Metrics BasedMetrics Based ● ScrapesScrapes EndpointsEndpoints • ExistingExisting endpoints forendpoints for limited toolslimited tools ● GraphiteGraphite ExporterExporter ● Push GatewayPush Gateway ● Great AlertingGreat Alerting
  26. 26. PrometheusPrometheus ● Mostly for Short TermMostly for Short Term ● Still Ship longterm metrics to otherStill Ship longterm metrics to other TSDBTSDB ● Nginx gw’s all over the placeNginx gw’s all over the place • (ssl fun)(ssl fun)
  27. 27. Infnite Diskspace ?Infnite Diskspace ? ● Logstash outputLogstash output • Statsd => GraphiteStatsd => Graphite • Keep patterns around,Keep patterns around, • Selectively purge dataSelectively purge data ● Prometheus for Short TermPrometheus for Short Term • Graphite for Long termGraphite for Long term
  28. 28. Log AlternativesLog Alternatives ● Graylog2Graylog2 ● ELSA (Enterprise Log Search andELSA (Enterprise Log Search and Archive)Archive) ● ELK StackELK Stack ● FluentdFluentd
  29. 29. Prometheus ?Prometheus ? ● Only For Containers ?Only For Containers ? ● Also for other setups !Also for other setups ! ● Is this sufcient ?Is this sufcient ?
  30. 30. Act / Alert:Act / Alert:
  31. 31. Checking for FailureChecking for Failure ● IcingaIcinga • Automated confg generationAutomated confg generation ● SensuSensu • CloudstyleCloudstyle ● PrometheusPrometheus • AlertManagerAlertManager
  32. 32. Waking you up at nightWaking you up at night ● FlapjackFlapjack fapjack.iofapjack.io monitoring notifcation routing +monitoring notifcation routing + event processing systemevent processing system ● OpenDutyOpenDuty github.com/szechuen/OpenDutygithub.com/szechuen/OpenDuty Duty managementDuty management
  33. 33. Waking you up at nightWaking you up at night ● AnagAnag ● Custom written stufCustom written stuf
  34. 34. Analyse:Analyse:
  35. 35. Basic SearchBasic Search
  36. 36. Graphs to KnowledgeGraphs to Knowledge SkylineSkyline • OculusOculus • Creating Information out of this dataCreating Information out of this data • Big dataBig data • Machine LearningMachine Learning • Hastic.ioHastic.io
  37. 37. Hastic.ioHastic.io ● Open Source Pattern DetectionOpen Source Pattern Detection ● Label patterns → Wait for learning toLabel patterns → Wait for learning to complete → Get detectionscomplete → Get detections ● Hastic Server + Grafana AppHastic Server + Grafana App
  38. 38. LogIslandsLogIslands ● Complex event processing &Complex event processing & patterns mining at scalepatterns mining at scale ● Kafka, Nif, Spark, HadoopKafka, Nif, Spark, Hadoop
  39. 39. Visualize:Visualize:
  40. 40. KibanaKibana
  41. 41. GrafanaGrafana
  42. 42. ChallengeChallenge ● *ana as code*ana as code ● Template your ...Template your ... ● e.g grafonnet-libe.g grafonnet-lib • A jsonnet lib to generate GrafanaA jsonnet lib to generate Grafana dashboards ...dashboards ...
  43. 43. AggregatingAggregating ● ThrukThruk ● GrafanaGrafana ● DashingDashing
  44. 44. Lack of change ?Lack of change ? ● Limited # new toolsLimited # new tools ● Feature Complete ?Feature Complete ?
  45. 45. Is prometheus the newIs prometheus the new Docker ?Docker ?
  46. 46. APMAPM Application Performance MonitoringApplication Performance Monitoring But what about my apps ?But what about my apps ? ● agent required that ties to codeagent required that ties to code ● Code modifcationsCode modifcations
  47. 47. Old PacketBeatOld PacketBeat
  48. 48. Open Source “APM”Open Source “APM” ● Scouter ● Jaeger ● Kamon ● Zipkin ● Beats ... ● Performance Co Pilot ● Kamon ● Pinpoint ● Micrometer ● StageMonitor ● SkyWalking ● Kieker => Huge focus on the Java Ecosystem , little options for PHP/ Python / Ruby shops.
  49. 49. OpenAPM.ioOpenAPM.io
  50. 50. OpenTracing 101OpenTracing 101 ● The problem : It was not reasonable to ask all OSS services and all OSSThe problem : It was not reasonable to ask all OSS services and all OSS packages and all application-specifc code to use a single tracingpackages and all application-specifc code to use a single tracing vendor => Open Ttracingvendor => Open Ttracing ● Distributed Tracing StandardDistributed Tracing Standard ● CNCFCNCF ● Dapper inside GoogleDapper inside Google ● ““OpenTracing is not a download or a program. Distributed tracingOpenTracing is not a download or a program. Distributed tracing requires that software developers add instrumentation to the code ofrequires that software developers add instrumentation to the code of an application, or to the frameworks used in the application”an application, or to the frameworks used in the application”
  51. 51. Complexity is the EnemyComplexity is the Enemy of Reliabilityof Reliability
  52. 52. I love where Monitoring is headingI love where Monitoring is heading ““Wait , was I oncall last week ?”Wait , was I oncall last week ?” True words said by one of our oncall engineersTrue words said by one of our oncall engineers
  53. 53. OpservabilityOpservability
  54. 54. ContactContact Kris Buytaert kris.buytaert@inuits.euKris Buytaert kris.buytaert@inuits.eu Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.eu/http://www.inuits.eu/ Find Inuits inFind Inuits in Brasschaat,Ghent,Brasschaat,Ghent, Rotterdam,Prague,Rotterdam,Prague, Kiev,BrnoKiev,Brno

×