Monitoring with ElasticSearch

Kris Buytaert
Kris BuytaertDevops, Linux and Open Source Expert at Inuits
Monitoring with the ELKMonitoring with the ELK
StackStack
Kris Buytaert
@krisbuytaert
KrisKris BuytaertBuytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source
Consultant @inuits.euConsultant @inuits.eu
● Everything is an effing DNS ProblemEverything is an effing DNS Problem
● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore
● Organising too many confs , #devopsdays,Organising too many confs , #devopsdays,
#loadays, #cfgmgmtcamp ...#loadays, #cfgmgmtcamp ...
● Evangelizing devopsEvangelizing devops
#devops=~C(L)AMS#devops=~C(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● Monitoring and MeasurementMonitoring and Measurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim
#monitoringsucks#monitoringsucks
● John Vincent (@lusis), june 2011John Vincent (@lusis), june 2011
● A sub #devops movementA sub #devops movement
● https://github.com/monitoringsucks/https://github.com/monitoringsucks/
#monitoringlove#monitoringlove
•
•
Ulf Mansson #devopsdays Rome 2011Ulf Mansson #devopsdays Rome 2011
•
A new era of toolingA new era of tooling
•
#monitoringlove hacksessions @inuits#monitoringlove hacksessions @inuits
•
#monitorama#monitorama
What we wantWhat we want
● Small , well suited componentsSmall , well suited components
•
CollectCollect
•
Transport / MangleTransport / Mangle
•
StoreStore
•
AnalyseAnalyse
•
Act / AlertAct / Alert
•
VisualizeVisualize
•
Monitoring with ElasticSearch
What do YOU do withWhat do YOU do with
your logfiles ?your logfiles ?
Logs & CollectionLogs & Collection
● Syslog,Syslog,
● RsyslogRsyslog
● Syslog-ngSyslog-ng
● Log4jLog4j
● Graylog2Graylog2
● ELSA (Enterprise LogELSA (Enterprise Log
Search and Archive)Search and Archive)
● ELK StackELK Stack
LogstashLogstash
● Not your average centralized logging toolNot your average centralized logging tool
● Elasticsearch backedElasticsearch backed
● ShipperShipper
● IndexerIndexer
● WebWeb
● Collect fromCollect from
anywhereanywhere
● FilterFilter
● Send anywhereSend anywhere
● QueingQueing
Collect logsCollect logs
● rsyslog -> rsyslog inputrsyslog -> rsyslog input
● dedicated shippers (logstash-shipper,dedicated shippers (logstash-shipper,
lumberjack)lumberjack)
•
Direct or via queueDirect or via queue
● Write logs -> json format -> redis -> json inputWrite logs -> json format -> redis -> json input
● * -> your favourite logstash input* -> your favourite logstash input
●
FiltersFilters
● GrokGrok
● MutateMutate mutate {mutate {
# Lowercase some values that are always in uppercase# Lowercase some values that are always in uppercase
lowercase => [ "EventType", "FileName", "Hostname", "Severity" ]lowercase => [ "EventType", "FileName", "Hostname", "Severity" ]
}}
● dropdrop
● TagsTags
GrokGrok
grok {grok {
match => [match => [
"message", "%{IPORHOST:host} - %{USER:remote_user} [%"message", "%{IPORHOST:host} - %{USER:remote_user} [%
{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{ISO8601_TIMEZONE}] %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{ISO8601_TIMEZONE}] %
{QS:request} %{INT:status} %{INT:bytes_sent} %{QS:http_referer} %{QS:request} %{INT:status} %{INT:bytes_sent} %{QS:http_referer} %
{QS:http_user_agent}"{QS:http_user_agent}"
]]
add_field => ["grok_type", "nginx-access"]add_field => ["grok_type", "nginx-access"]
add_tag => ["grokked"]add_tag => ["grokked"]
}}
}}
OutputsOutputs
● Mostly ElasticSearchMostly ElasticSearch
● Plenty of outputsPlenty of outputs
•
EmailEmail
•
NagiosNagios
•
RiemannRiemann
•
StatsdStatsd
•
GraphiteGraphite
Monitoring with ElasticSearch
Monitoring with ElasticSearch
Monitoring with ElasticSearch
Same tool used by devs to debugSame tool used by devs to debug
as by ops to debugas by ops to debug
Long Term MetricsLong Term Metrics
● Disk space is cheapDisk space is cheap
● But some people don't care about those logsBut some people don't care about those logs
after X weeks / months / yearsafter X weeks / months / years
● Send statistics via statsd to graphite ,Send statistics via statsd to graphite ,
● Keep graphite data for long term storage,Keep graphite data for long term storage,
● Purge elasticsearch contentPurge elasticsearch content
(curator is incomplete, working on patches)(curator is incomplete, working on patches)
GraphiteGraphite
● Graphing at ScaleGraphing at Scale
● Graphing at EaseGraphing at Ease
● Any metric is a graphAny metric is a graph
● echo "somestring $somevalue $timestamp" |echo "somestring $somevalue $timestamp" |
nc <%= graphitehost %> 2003nc <%= graphitehost %> 2003
Monitoring with ElasticSearch
Friends of GraphiteFriends of Graphite
● CCollection :ollection :
●
Statsd,Statsd,
●
Collectd + CarbonpluginCollectd + Carbonplugin
●
JmxtransJmxtrans
●
LogsterLogster
● DashboardsDashboards
●
TattleTattle
●
GdashGdash
●
GrafanaGrafana
Alerting on Events/LogsAlerting on Events/Logs
● Logstash -> icingaLogstash -> icinga
output {output {
if [message] =~ /(error|ERROR|CRITICAL)/ and [logsource] =~ /edc-if [message] =~ /(error|ERROR|CRITICAL)/ and [logsource] =~ /edc-
app/ and [program] =~ /^edc-/ {app/ and [program] =~ /^edc-/ {
nagios_nsca {nagios_nsca {
host => "10.0.64.28"host => "10.0.64.28"
nagios_status => "1"nagios_status => "1"
nagios_host => "%{logsource}"nagios_host => "%{logsource}"
nagios_service => "Log check - %{program}"nagios_service => "Log check - %{program}"
}}
}}
Buy this guy
a beer
Remember, it's notRemember, it's not
about the toolsabout the tools
ContactContact
Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.be/http://www.inuits.be/
InuitsInuits
Duboistraat 50Duboistraat 50
2060 Antwerpen2060 Antwerpen
BelgiumBelgium
891.514.231891.514.231
+32 475 961221+32 475 961221
1 of 27

Recommended

From Config Management Sucks to #cfgmgmtlove by
From Config Management Sucks to #cfgmgmtlove From Config Management Sucks to #cfgmgmtlove
From Config Management Sucks to #cfgmgmtlove Kris Buytaert
2K views42 slides
Attack monitoring using ElasticSearch Logstash and Kibana by
Attack monitoring using ElasticSearch Logstash and KibanaAttack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaPrajal Kulkarni
67.6K views59 slides
Mobile Analytics mit Elasticsearch und Kibana by
Mobile Analytics mit Elasticsearch und KibanaMobile Analytics mit Elasticsearch und Kibana
Mobile Analytics mit Elasticsearch und Kibanainovex GmbH
2.5K views53 slides
Frontera распределенный робот для обхода веба в больших объемах / Александр С... by
Frontera распределенный робот для обхода веба в больших объемах / Александр С...Frontera распределенный робот для обхода веба в больших объемах / Александр С...
Frontera распределенный робот для обхода веба в больших объемах / Александр С...Ontico
882 views159 slides
Large Scale Log collection using LogStash & mongoDB by
Large Scale Log collection using LogStash & mongoDB Large Scale Log collection using LogStash & mongoDB
Large Scale Log collection using LogStash & mongoDB Gaurav Bhardwaj
4.9K views22 slides
Open Source Monitoring Tools by
Open Source Monitoring ToolsOpen Source Monitoring Tools
Open Source Monitoring Toolsm_richardson
29.4K views59 slides

More Related Content

What's hot

Bringing code to the data: from MySQL to RocksDB for high volume searches by
Bringing code to the data: from MySQL to RocksDB for high volume searchesBringing code to the data: from MySQL to RocksDB for high volume searches
Bringing code to the data: from MySQL to RocksDB for high volume searchesIvan Kruglov
787 views41 slides
GraalVMの紹介とTruffleでPHPぽい言語を実装したら爆速だった話 by
GraalVMの紹介とTruffleでPHPぽい言語を実装したら爆速だった話GraalVMの紹介とTruffleでPHPぽい言語を実装したら爆速だった話
GraalVMの紹介とTruffleでPHPぽい言語を実装したら爆速だった話なおき きしだ
3.4K views35 slides
Pharo Update by
Pharo Update Pharo Update
Pharo Update ESUG
950 views82 slides
Sphinx at Craigslist in 2012 by
Sphinx at Craigslist in 2012Sphinx at Craigslist in 2012
Sphinx at Craigslist in 2012Jeremy Zawodny
5.3K views31 slides
Logstash family introduction by
Logstash family introductionLogstash family introduction
Logstash family introductionOwen Wu
1.3K views38 slides
Real time fulltext search with sphinx by
Real time fulltext search with sphinxReal time fulltext search with sphinx
Real time fulltext search with sphinxAdrian Nuta
12.6K views35 slides

What's hot(20)

Bringing code to the data: from MySQL to RocksDB for high volume searches by Ivan Kruglov
Bringing code to the data: from MySQL to RocksDB for high volume searchesBringing code to the data: from MySQL to RocksDB for high volume searches
Bringing code to the data: from MySQL to RocksDB for high volume searches
Ivan Kruglov787 views
GraalVMの紹介とTruffleでPHPぽい言語を実装したら爆速だった話 by なおき きしだ
GraalVMの紹介とTruffleでPHPぽい言語を実装したら爆速だった話GraalVMの紹介とTruffleでPHPぽい言語を実装したら爆速だった話
GraalVMの紹介とTruffleでPHPぽい言語を実装したら爆速だった話
なおき きしだ3.4K views
Pharo Update by ESUG
Pharo Update Pharo Update
Pharo Update
ESUG950 views
Sphinx at Craigslist in 2012 by Jeremy Zawodny
Sphinx at Craigslist in 2012Sphinx at Craigslist in 2012
Sphinx at Craigslist in 2012
Jeremy Zawodny5.3K views
Logstash family introduction by Owen Wu
Logstash family introductionLogstash family introduction
Logstash family introduction
Owen Wu1.3K views
Real time fulltext search with sphinx by Adrian Nuta
Real time fulltext search with sphinxReal time fulltext search with sphinx
Real time fulltext search with sphinx
Adrian Nuta12.6K views
«Scrapy internals» Александр Сибиряков, Scrapinghub by it-people
«Scrapy internals» Александр Сибиряков, Scrapinghub«Scrapy internals» Александр Сибиряков, Scrapinghub
«Scrapy internals» Александр Сибиряков, Scrapinghub
it-people446 views
Realtime Search Infrastructure at Craigslist (OpenWest 2014) by Jeremy Zawodny
Realtime Search Infrastructure at Craigslist (OpenWest 2014)Realtime Search Infrastructure at Craigslist (OpenWest 2014)
Realtime Search Infrastructure at Craigslist (OpenWest 2014)
Jeremy Zawodny14.5K views
Reactive database access with Slick3 by takezoe
Reactive database access with Slick3Reactive database access with Slick3
Reactive database access with Slick3
takezoe5.2K views
Ceph BlueStore - новый тип хранилища в Ceph / Максим Воронцов, (Redsys) by Ontico
Ceph BlueStore - новый тип хранилища в Ceph / Максим Воронцов, (Redsys)Ceph BlueStore - новый тип хранилища в Ceph / Максим Воронцов, (Redsys)
Ceph BlueStore - новый тип хранилища в Ceph / Максим Воронцов, (Redsys)
Ontico2.4K views
MySQL And Search At Craigslist by Jeremy Zawodny
MySQL And Search At CraigslistMySQL And Search At Craigslist
MySQL And Search At Craigslist
Jeremy Zawodny14.8K views
Data Processing and Ruby in the World by SATOSHI TAGOMORI
Data Processing and Ruby in the WorldData Processing and Ruby in the World
Data Processing and Ruby in the World
SATOSHI TAGOMORI6.5K views
みんなのNode.js by ogom_
みんなのNode.jsみんなのNode.js
みんなのNode.js
ogom_940 views
Tracing Microservices with Zipkin by takezoe
Tracing Microservices with ZipkinTracing Microservices with Zipkin
Tracing Microservices with Zipkin
takezoe13.8K views
Batch import of large RDF datasets into Semantic MediaWiki by Samuel Lampa
Batch import of large RDF datasets into Semantic MediaWikiBatch import of large RDF datasets into Semantic MediaWiki
Batch import of large RDF datasets into Semantic MediaWiki
Samuel Lampa1.1K views
Ruby in office time reboot by Kentaro Goto
Ruby in office time rebootRuby in office time reboot
Ruby in office time reboot
Kentaro Goto1.7K views
使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化-曾書庭 by 台灣資料科學年會
使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化-曾書庭使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化-曾書庭
使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化-曾書庭
Pycon 2012 What Python can learn from Java by jbellis
Pycon 2012 What Python can learn from JavaPycon 2012 What Python can learn from Java
Pycon 2012 What Python can learn from Java
jbellis5.3K views
Sharding - patterns & antipatterns, Константин Осипов, Алексей Рыбак by Ontico
Sharding -  patterns & antipatterns, Константин Осипов, Алексей РыбакSharding -  patterns & antipatterns, Константин Осипов, Алексей Рыбак
Sharding - patterns & antipatterns, Константин Осипов, Алексей Рыбак
Ontico4.1K views

Viewers also liked

Wiesbaden Magazin Ausgabe Juni 2011 by
Wiesbaden Magazin Ausgabe Juni 2011Wiesbaden Magazin Ausgabe Juni 2011
Wiesbaden Magazin Ausgabe Juni 2011Landeshauptstadt Wiesbaden
5.9K views28 slides
Agenturportrait brandpolice 2011 small by
Agenturportrait brandpolice 2011 smallAgenturportrait brandpolice 2011 small
Agenturportrait brandpolice 2011 smallBrandpolice GmbH
2.5K views30 slides
Nare Blast Presentation by
Nare Blast PresentationNare Blast Presentation
Nare Blast Presentationikayadev
528 views28 slides
Jason Stanley, Secure-24 - Own IT Through Proactive IT Monitoring by
Jason Stanley, Secure-24 - Own IT Through Proactive IT MonitoringJason Stanley, Secure-24 - Own IT Through Proactive IT Monitoring
Jason Stanley, Secure-24 - Own IT Through Proactive IT MonitoringZenoss
1.2K views17 slides
Intro to Zenoss by Andrew Kirch by
Intro to Zenoss by Andrew KirchIntro to Zenoss by Andrew Kirch
Intro to Zenoss by Andrew Kirchbuildacloud
1.8K views13 slides
Open Source Monitoring in 2014, from #monitoringssucks to #monitoringlove and... by
Open Source Monitoring in 2014, from #monitoringssucks to #monitoringlove and...Open Source Monitoring in 2014, from #monitoringssucks to #monitoringlove and...
Open Source Monitoring in 2014, from #monitoringssucks to #monitoringlove and...Kris Buytaert
5.5K views64 slides

Viewers also liked(20)

Agenturportrait brandpolice 2011 small by Brandpolice GmbH
Agenturportrait brandpolice 2011 smallAgenturportrait brandpolice 2011 small
Agenturportrait brandpolice 2011 small
Brandpolice GmbH2.5K views
Nare Blast Presentation by ikayadev
Nare Blast PresentationNare Blast Presentation
Nare Blast Presentation
ikayadev528 views
Jason Stanley, Secure-24 - Own IT Through Proactive IT Monitoring by Zenoss
Jason Stanley, Secure-24 - Own IT Through Proactive IT MonitoringJason Stanley, Secure-24 - Own IT Through Proactive IT Monitoring
Jason Stanley, Secure-24 - Own IT Through Proactive IT Monitoring
Zenoss1.2K views
Intro to Zenoss by Andrew Kirch by buildacloud
Intro to Zenoss by Andrew KirchIntro to Zenoss by Andrew Kirch
Intro to Zenoss by Andrew Kirch
buildacloud1.8K views
Open Source Monitoring in 2014, from #monitoringssucks to #monitoringlove and... by Kris Buytaert
Open Source Monitoring in 2014, from #monitoringssucks to #monitoringlove and...Open Source Monitoring in 2014, from #monitoringssucks to #monitoringlove and...
Open Source Monitoring in 2014, from #monitoringssucks to #monitoringlove and...
Kris Buytaert5.5K views
IT Infrastructure Monitoring Strategies in Healthcare by CA Technologies
IT Infrastructure Monitoring Strategies in HealthcareIT Infrastructure Monitoring Strategies in Healthcare
IT Infrastructure Monitoring Strategies in Healthcare
CA Technologies3.2K views
Rootconf by akbarabi
RootconfRootconf
Rootconf
akbarabi567 views
Mesoscon 2015 by Skand Gupta
Mesoscon 2015Mesoscon 2015
Mesoscon 2015
Skand Gupta1.8K views
The devops approach to monitoring, Open Source and Infrastructure as Code Style by Julien Pivotto
The devops approach to monitoring, Open Source and Infrastructure as Code StyleThe devops approach to monitoring, Open Source and Infrastructure as Code Style
The devops approach to monitoring, Open Source and Infrastructure as Code Style
Julien Pivotto14.6K views
Blending ITIL, Agile, DevOps and LeanUX at Auto Trader UK by Andrew Humphrey
Blending ITIL, Agile, DevOps and LeanUX at Auto Trader UKBlending ITIL, Agile, DevOps and LeanUX at Auto Trader UK
Blending ITIL, Agile, DevOps and LeanUX at Auto Trader UK
Andrew Humphrey1.8K views
Building Product from ground up using Open Source Technologies by Amit Goel
Building Product from ground up using Open Source TechnologiesBuilding Product from ground up using Open Source Technologies
Building Product from ground up using Open Source Technologies
Amit Goel778 views
TechWiseTV Workshop: APIC-EM by Robb Boyd
TechWiseTV Workshop: APIC-EMTechWiseTV Workshop: APIC-EM
TechWiseTV Workshop: APIC-EM
Robb Boyd15.5K views
Data science team, a practice to setup by Omid Mogharian
Data science team, a practice to setupData science team, a practice to setup
Data science team, a practice to setup
Omid Mogharian359 views
Send that (damn) elevator down ! by Ekta Grover
Send that (damn) elevator down !Send that (damn) elevator down !
Send that (damn) elevator down !
Ekta Grover1.5K views
(R)Evolutionize APM - APM in Continuous Delivery and DevOps by Martin Etmajer
(R)Evolutionize APM - APM in Continuous Delivery and DevOps(R)Evolutionize APM - APM in Continuous Delivery and DevOps
(R)Evolutionize APM - APM in Continuous Delivery and DevOps
Martin Etmajer1.3K views
Experiences in ELK with D3.js for Large Log Analysis and Visualization by Surasak Sanguanpong
Experiences in ELK with D3.js  for Large Log Analysis  and VisualizationExperiences in ELK with D3.js  for Large Log Analysis  and Visualization
Experiences in ELK with D3.js for Large Log Analysis and Visualization
Monitoring in the DevOps Era by Mike Kavis
Monitoring in the DevOps EraMonitoring in the DevOps Era
Monitoring in the DevOps Era
Mike Kavis10.7K views
Centralized Logging System Using ELK Stack by Rohit Sharma
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK Stack
Rohit Sharma3.3K views

Similar to Monitoring with ElasticSearch

Open Source Monitoring in 2019 by
Open Source Monitoring in 2019 Open Source Monitoring in 2019
Open Source Monitoring in 2019 Kris Buytaert
1.5K views56 slides
When traditional configuration management is to slow for your needs by
When traditional configuration management is to slow for your needsWhen traditional configuration management is to slow for your needs
When traditional configuration management is to slow for your needsKris Buytaert
1.4K views32 slides
Monitoring Drupal In an Infrastructure as Code Age by
Monitoring Drupal In an Infrastructure as Code AgeMonitoring Drupal In an Infrastructure as Code Age
Monitoring Drupal In an Infrastructure as Code AgeKris Buytaert
10.6K views45 slides
The Return of the Dull Stack Engineer by
The Return of the Dull Stack EngineerThe Return of the Dull Stack Engineer
The Return of the Dull Stack EngineerKris Buytaert
2.4K views51 slides
Automating MySQL operations with Puppet by
Automating MySQL operations with PuppetAutomating MySQL operations with Puppet
Automating MySQL operations with PuppetKris Buytaert
1.8K views51 slides
From MonitoringSucks to Monitoring Love , 2016 Edition by
From MonitoringSucks to Monitoring Love , 2016 EditionFrom MonitoringSucks to Monitoring Love , 2016 Edition
From MonitoringSucks to Monitoring Love , 2016 EditionKris Buytaert
29.4K views51 slides

Similar to Monitoring with ElasticSearch (20)

Open Source Monitoring in 2019 by Kris Buytaert
Open Source Monitoring in 2019 Open Source Monitoring in 2019
Open Source Monitoring in 2019
Kris Buytaert1.5K views
When traditional configuration management is to slow for your needs by Kris Buytaert
When traditional configuration management is to slow for your needsWhen traditional configuration management is to slow for your needs
When traditional configuration management is to slow for your needs
Kris Buytaert1.4K views
Monitoring Drupal In an Infrastructure as Code Age by Kris Buytaert
Monitoring Drupal In an Infrastructure as Code AgeMonitoring Drupal In an Infrastructure as Code Age
Monitoring Drupal In an Infrastructure as Code Age
Kris Buytaert10.6K views
The Return of the Dull Stack Engineer by Kris Buytaert
The Return of the Dull Stack EngineerThe Return of the Dull Stack Engineer
The Return of the Dull Stack Engineer
Kris Buytaert2.4K views
Automating MySQL operations with Puppet by Kris Buytaert
Automating MySQL operations with PuppetAutomating MySQL operations with Puppet
Automating MySQL operations with Puppet
Kris Buytaert1.8K views
From MonitoringSucks to Monitoring Love , 2016 Edition by Kris Buytaert
From MonitoringSucks to Monitoring Love , 2016 EditionFrom MonitoringSucks to Monitoring Love , 2016 Edition
From MonitoringSucks to Monitoring Love , 2016 Edition
Kris Buytaert29.4K views
London devops logging by Tomas Doran
London devops loggingLondon devops logging
London devops logging
Tomas Doran17.3K views
Icinga Camp Amsterdam - Infrastructure as Code by Icinga
Icinga Camp Amsterdam - Infrastructure as CodeIcinga Camp Amsterdam - Infrastructure as Code
Icinga Camp Amsterdam - Infrastructure as Code
Icinga3.3K views
On the Importance of Infrastructure as Code by Kris Buytaert
On the Importance of Infrastructure as CodeOn the Importance of Infrastructure as Code
On the Importance of Infrastructure as Code
Kris Buytaert1.8K views
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ... by Hernan Costante
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Hernan Costante1.9K views
Open Source Monitoring in 2015 by Kris Buytaert
Open Source Monitoring in 2015Open Source Monitoring in 2015
Open Source Monitoring in 2015
Kris Buytaert2.3K views
Dev secops opsec, devsec, devops ? by Kris Buytaert
Dev secops opsec, devsec, devops ?Dev secops opsec, devsec, devops ?
Dev secops opsec, devsec, devops ?
Kris Buytaert17.4K views
OSDC 2012 | Devops and Open Source by Kris Buytaert by NETWAYS
OSDC 2012 | Devops and Open Source by Kris BuytaertOSDC 2012 | Devops and Open Source by Kris Buytaert
OSDC 2012 | Devops and Open Source by Kris Buytaert
NETWAYS23 views
OSDC 2012 | Devops and Open Source by Kris Buyaert by NETWAYS
OSDC 2012 | Devops and Open Source by Kris BuyaertOSDC 2012 | Devops and Open Source by Kris Buyaert
OSDC 2012 | Devops and Open Source by Kris Buyaert
NETWAYS18 views
Continous Delivery of your Infrastructure by Kris Buytaert
Continous Delivery of your InfrastructureContinous Delivery of your Infrastructure
Continous Delivery of your Infrastructure
Kris Buytaert857 views
Groovy there's a docker in my application pipeline by Kris Buytaert
Groovy there's a docker in my application pipelineGroovy there's a docker in my application pipeline
Groovy there's a docker in my application pipeline
Kris Buytaert1.3K views
OSMC 2017 | Groovy There is a Docker in my Dashing Pipeline by Kris Buytaert by NETWAYS
OSMC 2017 | Groovy There is a Docker in my Dashing Pipeline by Kris Buytaert OSMC 2017 | Groovy There is a Docker in my Dashing Pipeline by Kris Buytaert
OSMC 2017 | Groovy There is a Docker in my Dashing Pipeline by Kris Buytaert
NETWAYS152 views
Pipeline as code for your infrastructure as Code by Kris Buytaert
Pipeline as code for your infrastructure as CodePipeline as code for your infrastructure as Code
Pipeline as code for your infrastructure as Code
Kris Buytaert1.9K views
Monitoring in an Infrastructure as Code Age by Puppet
Monitoring in an Infrastructure as Code AgeMonitoring in an Infrastructure as Code Age
Monitoring in an Infrastructure as Code Age
Puppet7.2K views

More from Kris Buytaert

Years of (not) learning , from devops to devoops by
Years of (not) learning , from devops to devoopsYears of (not) learning , from devops to devoops
Years of (not) learning , from devops to devoopsKris Buytaert
65 views44 slides
Observability will not fix your Broken Monitoring ,Ignite by
Observability will not fix your Broken Monitoring ,IgniteObservability will not fix your Broken Monitoring ,Ignite
Observability will not fix your Broken Monitoring ,IgniteKris Buytaert
167 views20 slides
Infrastructure as Code Patterns by
Infrastructure as Code PatternsInfrastructure as Code Patterns
Infrastructure as Code PatternsKris Buytaert
117 views53 slides
From devoops to devops 13 years of (not) learning by
From devoops to devops 13 years of (not) learningFrom devoops to devops 13 years of (not) learning
From devoops to devops 13 years of (not) learningKris Buytaert
185 views40 slides
Pipeline all the Dashboards as Code by
Pipeline all the Dashboards as CodePipeline all the Dashboards as Code
Pipeline all the Dashboards as CodeKris Buytaert
644 views20 slides
Help , My Datacenter is on fire by
Help , My Datacenter is on fireHelp , My Datacenter is on fire
Help , My Datacenter is on fireKris Buytaert
542 views38 slides

More from Kris Buytaert(20)

Years of (not) learning , from devops to devoops by Kris Buytaert
Years of (not) learning , from devops to devoopsYears of (not) learning , from devops to devoops
Years of (not) learning , from devops to devoops
Kris Buytaert65 views
Observability will not fix your Broken Monitoring ,Ignite by Kris Buytaert
Observability will not fix your Broken Monitoring ,IgniteObservability will not fix your Broken Monitoring ,Ignite
Observability will not fix your Broken Monitoring ,Ignite
Kris Buytaert167 views
Infrastructure as Code Patterns by Kris Buytaert
Infrastructure as Code PatternsInfrastructure as Code Patterns
Infrastructure as Code Patterns
Kris Buytaert117 views
From devoops to devops 13 years of (not) learning by Kris Buytaert
From devoops to devops 13 years of (not) learningFrom devoops to devops 13 years of (not) learning
From devoops to devops 13 years of (not) learning
Kris Buytaert185 views
Pipeline all the Dashboards as Code by Kris Buytaert
Pipeline all the Dashboards as CodePipeline all the Dashboards as Code
Pipeline all the Dashboards as Code
Kris Buytaert644 views
Help , My Datacenter is on fire by Kris Buytaert
Help , My Datacenter is on fireHelp , My Datacenter is on fire
Help , My Datacenter is on fire
Kris Buytaert542 views
Devops is Dead, Long live Devops by Kris Buytaert
Devops is Dead, Long live DevopsDevops is Dead, Long live Devops
Devops is Dead, Long live Devops
Kris Buytaert289 views
10 years of #devopsdays, but what have we really learned ? by Kris Buytaert
10 years of #devopsdays, but what have we really learned ? 10 years of #devopsdays, but what have we really learned ?
10 years of #devopsdays, but what have we really learned ?
Kris Buytaert594 views
Continuous Infrastructure First by Kris Buytaert
Continuous Infrastructure FirstContinuous Infrastructure First
Continuous Infrastructure First
Kris Buytaert568 views
Is there a Future for devops ? by Kris Buytaert
Is there a Future for devops   ? Is there a Future for devops   ?
Is there a Future for devops ?
Kris Buytaert478 views
10 Years of #devopsdays weirdness by Kris Buytaert
10 Years of #devopsdays weirdness10 Years of #devopsdays weirdness
10 Years of #devopsdays weirdness
Kris Buytaert400 views
ADDO 2019: Looking back at over 10 years of Devops by Kris Buytaert
ADDO 2019:    Looking back at over 10 years of DevopsADDO 2019:    Looking back at over 10 years of Devops
ADDO 2019: Looking back at over 10 years of Devops
Kris Buytaert578 views
Continuous Infrastructure First Ignite Edition by Kris Buytaert
Continuous Infrastructure First  Ignite EditionContinuous Infrastructure First  Ignite Edition
Continuous Infrastructure First Ignite Edition
Kris Buytaert476 views
Continuous Infrastructure First by Kris Buytaert
Continuous Infrastructure FirstContinuous Infrastructure First
Continuous Infrastructure First
Kris Buytaert521 views
Devops is a Security Requirement by Kris Buytaert
Devops is a Security RequirementDevops is a Security Requirement
Devops is a Security Requirement
Kris Buytaert699 views
Is there a future for devops ? by Kris Buytaert
Is there a future for devops ?Is there a future for devops ?
Is there a future for devops ?
Kris Buytaert3.5K views

Recently uploaded

Spesifikasi Lengkap ASUS Vivobook Go 14 by
Spesifikasi Lengkap ASUS Vivobook Go 14Spesifikasi Lengkap ASUS Vivobook Go 14
Spesifikasi Lengkap ASUS Vivobook Go 14Dot Semarang
35 views1 slide
Empathic Computing: Delivering the Potential of the Metaverse by
Empathic Computing: Delivering  the Potential of the MetaverseEmpathic Computing: Delivering  the Potential of the Metaverse
Empathic Computing: Delivering the Potential of the MetaverseMark Billinghurst
470 views80 slides
Kyo - Functional Scala 2023.pdf by
Kyo - Functional Scala 2023.pdfKyo - Functional Scala 2023.pdf
Kyo - Functional Scala 2023.pdfFlavio W. Brasil
165 views92 slides
Business Analyst Series 2023 - Week 3 Session 5 by
Business Analyst Series 2023 -  Week 3 Session 5Business Analyst Series 2023 -  Week 3 Session 5
Business Analyst Series 2023 - Week 3 Session 5DianaGray10
209 views20 slides
Transcript: The Details of Description Techniques tips and tangents on altern... by
Transcript: The Details of Description Techniques tips and tangents on altern...Transcript: The Details of Description Techniques tips and tangents on altern...
Transcript: The Details of Description Techniques tips and tangents on altern...BookNet Canada
130 views15 slides
Perth MeetUp November 2023 by
Perth MeetUp November 2023 Perth MeetUp November 2023
Perth MeetUp November 2023 Michael Price
15 views44 slides

Recently uploaded(20)

Spesifikasi Lengkap ASUS Vivobook Go 14 by Dot Semarang
Spesifikasi Lengkap ASUS Vivobook Go 14Spesifikasi Lengkap ASUS Vivobook Go 14
Spesifikasi Lengkap ASUS Vivobook Go 14
Dot Semarang35 views
Empathic Computing: Delivering the Potential of the Metaverse by Mark Billinghurst
Empathic Computing: Delivering  the Potential of the MetaverseEmpathic Computing: Delivering  the Potential of the Metaverse
Empathic Computing: Delivering the Potential of the Metaverse
Mark Billinghurst470 views
Business Analyst Series 2023 - Week 3 Session 5 by DianaGray10
Business Analyst Series 2023 -  Week 3 Session 5Business Analyst Series 2023 -  Week 3 Session 5
Business Analyst Series 2023 - Week 3 Session 5
DianaGray10209 views
Transcript: The Details of Description Techniques tips and tangents on altern... by BookNet Canada
Transcript: The Details of Description Techniques tips and tangents on altern...Transcript: The Details of Description Techniques tips and tangents on altern...
Transcript: The Details of Description Techniques tips and tangents on altern...
BookNet Canada130 views
Perth MeetUp November 2023 by Michael Price
Perth MeetUp November 2023 Perth MeetUp November 2023
Perth MeetUp November 2023
Michael Price15 views
Attacking IoT Devices from a Web Perspective - Linux Day by Simone Onofri
Attacking IoT Devices from a Web Perspective - Linux Day Attacking IoT Devices from a Web Perspective - Linux Day
Attacking IoT Devices from a Web Perspective - Linux Day
Simone Onofri15 views
Lilypad @ Labweek, Istanbul, 2023.pdf by Ally339821
Lilypad @ Labweek, Istanbul, 2023.pdfLilypad @ Labweek, Istanbul, 2023.pdf
Lilypad @ Labweek, Istanbul, 2023.pdf
Ally3398219 views
Case Study Copenhagen Energy and Business Central.pdf by Aitana
Case Study Copenhagen Energy and Business Central.pdfCase Study Copenhagen Energy and Business Central.pdf
Case Study Copenhagen Energy and Business Central.pdf
Aitana12 views
DALI Basics Course 2023 by Ivory Egg
DALI Basics Course  2023DALI Basics Course  2023
DALI Basics Course 2023
Ivory Egg14 views
STPI OctaNE CoE Brochure.pdf by madhurjyapb
STPI OctaNE CoE Brochure.pdfSTPI OctaNE CoE Brochure.pdf
STPI OctaNE CoE Brochure.pdf
madhurjyapb12 views
AMAZON PRODUCT RESEARCH.pdf by JerikkLaureta
AMAZON PRODUCT RESEARCH.pdfAMAZON PRODUCT RESEARCH.pdf
AMAZON PRODUCT RESEARCH.pdf
JerikkLaureta15 views
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N... by James Anderson
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
James Anderson33 views
Igniting Next Level Productivity with AI-Infused Data Integration Workflows by Safe Software
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software225 views
Piloting & Scaling Successfully With Microsoft Viva by Richard Harbridge
Piloting & Scaling Successfully With Microsoft VivaPiloting & Scaling Successfully With Microsoft Viva
Piloting & Scaling Successfully With Microsoft Viva

Monitoring with ElasticSearch

  • 1. Monitoring with the ELKMonitoring with the ELK StackStack Kris Buytaert @krisbuytaert
  • 2. KrisKris BuytaertBuytaert ● I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @inuits.euConsultant @inuits.eu ● Everything is an effing DNS ProblemEverything is an effing DNS Problem ● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore ● Organising too many confs , #devopsdays,Organising too many confs , #devopsdays, #loadays, #cfgmgmtcamp ...#loadays, #cfgmgmtcamp ... ● Evangelizing devopsEvangelizing devops
  • 3. #devops=~C(L)AMS#devops=~C(L)AMS ● CultureCulture ● (Lean)(Lean) ● AutomationAutomation ● Monitoring and MeasurementMonitoring and Measurement ● SharingSharing Damon Edwards and John WillisDamon Edwards and John Willis Gene KimGene Kim
  • 4. #monitoringsucks#monitoringsucks ● John Vincent (@lusis), june 2011John Vincent (@lusis), june 2011 ● A sub #devops movementA sub #devops movement ● https://github.com/monitoringsucks/https://github.com/monitoringsucks/
  • 5. #monitoringlove#monitoringlove • • Ulf Mansson #devopsdays Rome 2011Ulf Mansson #devopsdays Rome 2011 • A new era of toolingA new era of tooling • #monitoringlove hacksessions @inuits#monitoringlove hacksessions @inuits • #monitorama#monitorama
  • 6. What we wantWhat we want ● Small , well suited componentsSmall , well suited components • CollectCollect • Transport / MangleTransport / Mangle • StoreStore • AnalyseAnalyse • Act / AlertAct / Alert • VisualizeVisualize •
  • 8. What do YOU do withWhat do YOU do with your logfiles ?your logfiles ?
  • 9. Logs & CollectionLogs & Collection ● Syslog,Syslog, ● RsyslogRsyslog ● Syslog-ngSyslog-ng ● Log4jLog4j ● Graylog2Graylog2 ● ELSA (Enterprise LogELSA (Enterprise Log Search and Archive)Search and Archive) ● ELK StackELK Stack
  • 10. LogstashLogstash ● Not your average centralized logging toolNot your average centralized logging tool ● Elasticsearch backedElasticsearch backed ● ShipperShipper ● IndexerIndexer ● WebWeb
  • 11. ● Collect fromCollect from anywhereanywhere ● FilterFilter ● Send anywhereSend anywhere ● QueingQueing
  • 12. Collect logsCollect logs ● rsyslog -> rsyslog inputrsyslog -> rsyslog input ● dedicated shippers (logstash-shipper,dedicated shippers (logstash-shipper, lumberjack)lumberjack) • Direct or via queueDirect or via queue ● Write logs -> json format -> redis -> json inputWrite logs -> json format -> redis -> json input ● * -> your favourite logstash input* -> your favourite logstash input ●
  • 13. FiltersFilters ● GrokGrok ● MutateMutate mutate {mutate { # Lowercase some values that are always in uppercase# Lowercase some values that are always in uppercase lowercase => [ "EventType", "FileName", "Hostname", "Severity" ]lowercase => [ "EventType", "FileName", "Hostname", "Severity" ] }} ● dropdrop ● TagsTags
  • 14. GrokGrok grok {grok { match => [match => [ "message", "%{IPORHOST:host} - %{USER:remote_user} [%"message", "%{IPORHOST:host} - %{USER:remote_user} [% {MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{ISO8601_TIMEZONE}] %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{ISO8601_TIMEZONE}] % {QS:request} %{INT:status} %{INT:bytes_sent} %{QS:http_referer} %{QS:request} %{INT:status} %{INT:bytes_sent} %{QS:http_referer} % {QS:http_user_agent}"{QS:http_user_agent}" ]] add_field => ["grok_type", "nginx-access"]add_field => ["grok_type", "nginx-access"] add_tag => ["grokked"]add_tag => ["grokked"] }} }}
  • 15. OutputsOutputs ● Mostly ElasticSearchMostly ElasticSearch ● Plenty of outputsPlenty of outputs • EmailEmail • NagiosNagios • RiemannRiemann • StatsdStatsd • GraphiteGraphite
  • 19. Same tool used by devs to debugSame tool used by devs to debug as by ops to debugas by ops to debug
  • 20. Long Term MetricsLong Term Metrics ● Disk space is cheapDisk space is cheap ● But some people don't care about those logsBut some people don't care about those logs after X weeks / months / yearsafter X weeks / months / years ● Send statistics via statsd to graphite ,Send statistics via statsd to graphite , ● Keep graphite data for long term storage,Keep graphite data for long term storage, ● Purge elasticsearch contentPurge elasticsearch content (curator is incomplete, working on patches)(curator is incomplete, working on patches)
  • 21. GraphiteGraphite ● Graphing at ScaleGraphing at Scale ● Graphing at EaseGraphing at Ease ● Any metric is a graphAny metric is a graph ● echo "somestring $somevalue $timestamp" |echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003nc <%= graphitehost %> 2003
  • 23. Friends of GraphiteFriends of Graphite ● CCollection :ollection : ● Statsd,Statsd, ● Collectd + CarbonpluginCollectd + Carbonplugin ● JmxtransJmxtrans ● LogsterLogster ● DashboardsDashboards ● TattleTattle ● GdashGdash ● GrafanaGrafana
  • 24. Alerting on Events/LogsAlerting on Events/Logs ● Logstash -> icingaLogstash -> icinga output {output { if [message] =~ /(error|ERROR|CRITICAL)/ and [logsource] =~ /edc-if [message] =~ /(error|ERROR|CRITICAL)/ and [logsource] =~ /edc- app/ and [program] =~ /^edc-/ {app/ and [program] =~ /^edc-/ { nagios_nsca {nagios_nsca { host => "10.0.64.28"host => "10.0.64.28" nagios_status => "1"nagios_status => "1" nagios_host => "%{logsource}"nagios_host => "%{logsource}" nagios_service => "Log check - %{program}"nagios_service => "Log check - %{program}" }} }}
  • 26. Remember, it's notRemember, it's not about the toolsabout the tools
  • 27. ContactContact Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.be/http://www.inuits.be/ InuitsInuits Duboistraat 50Duboistraat 50 2060 Antwerpen2060 Antwerpen BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221