SlideShare a Scribd company logo
1 of 25
ELK – From zero to (coding class) hero
Josip Kovaček
Elk => Elastic Stack
Agenda
 Logs
 Elastic Stack
 Setup
 Monitoring
 Alerting
Logs
[ 2017-10-21 12:03:55.478, 8.8.8.8, Pero, /search, q=Changecon
ticket ]
Logs
Logs
Logs
Elastic Stack
Elastic Stack
Plugins…Lots of plugins
Elastic Stack
Beats
 Lightweight data shippers
 Filebeat, Metricbeat, Packetbeat, Winlogbeat, Heartbeat
 60+ community beats
Logstash
 Input/output from 50+ destinations
 Extracts and transforms data
 30+ plugins for data manipulation
Elasticsearch
 Real time search and analytics engine
 Restful JSON API
 Scalable, reliable
 Index and archives
Kibana
 Queries Elasticsearch for data
 Predefined set of visualization types
 Dashboards, plugins
Setup
 Running each component manually
 Installing as a service
 Docker, Vagrant image
 Configure input/output and component specific parameters
 Configure “index pattern”
Setup
logstash.yml configuration
Setup
*beatname.template.json
curl -XPUT "http://localhost:9200/_template/filebeat?pretty" -
d@filebeat.template.json
Monitoring
 Application/integration endpoints heartbeat > HeartBeat
 System performance counters > MetricBeat
 Logs > FileBeat
 DB/performance counters > logstash-input-jdbc
 Spring Boot (Actuator) > HttpBeat
Alerting
 X-Pack
 ElastAlert – free alternative
 Alert via email, JIRA, Slack, Telegram, execute command...
 Rules:
Alerting
Conclusion
 Benefits/drawbacks
 Learn from logs
 Getting bigger picture, detecting hard to find patterns
 Predicting upcoming service disruption
 Automated alerts, buying time when service fails
Q/A
 https://demo.elastic.co
 https://hub.docker.com/r/sebp/elk/
 https://github.com/Yelp/elastalert
 https://github.com/dzharii/awesome-elasticsearch
Thank you!
Conference and Media Partners

More Related Content

What's hot

Our challenge for Bulkload reliability improvement
Our challenge for Bulkload reliability  improvementOur challenge for Bulkload reliability  improvement
Our challenge for Bulkload reliability improvement
Satoshi Akama
 

What's hot (20)

OpenStack Ceilometer
OpenStack CeilometerOpenStack Ceilometer
OpenStack Ceilometer
 
ELK Stack
ELK StackELK Stack
ELK Stack
 
Our challenge for Bulkload reliability improvement
Our challenge for Bulkload reliability  improvementOur challenge for Bulkload reliability  improvement
Our challenge for Bulkload reliability improvement
 
Introduction openstack horizon
Introduction openstack horizonIntroduction openstack horizon
Introduction openstack horizon
 
Analyzing MySQL Logs with ClickHouse, by Peter Zaitsev
Analyzing MySQL Logs with ClickHouse, by Peter ZaitsevAnalyzing MySQL Logs with ClickHouse, by Peter Zaitsev
Analyzing MySQL Logs with ClickHouse, by Peter Zaitsev
 
Adventures in Observability: How in-house ClickHouse deployment enabled Inst...
 Adventures in Observability: How in-house ClickHouse deployment enabled Inst... Adventures in Observability: How in-house ClickHouse deployment enabled Inst...
Adventures in Observability: How in-house ClickHouse deployment enabled Inst...
 
Experiences in ELK with D3.js for Large Log Analysis and Visualization
Experiences in ELK with D3.js  for Large Log Analysis  and VisualizationExperiences in ELK with D3.js  for Large Log Analysis  and Visualization
Experiences in ELK with D3.js for Large Log Analysis and Visualization
 
OpenStack Log Mining
OpenStack Log MiningOpenStack Log Mining
OpenStack Log Mining
 
From Ceilometer to Telemetry: not so alarming!
From Ceilometer to Telemetry: not so alarming!From Ceilometer to Telemetry: not so alarming!
From Ceilometer to Telemetry: not so alarming!
 
Reactive programming on Android
Reactive programming on AndroidReactive programming on Android
Reactive programming on Android
 
LowlaDB intro March 2015
LowlaDB intro March 2015LowlaDB intro March 2015
LowlaDB intro March 2015
 
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander ZaitsevMigration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
 
Clickhouse at Cloudflare. By Marek Vavrusa
Clickhouse at Cloudflare. By Marek VavrusaClickhouse at Cloudflare. By Marek Vavrusa
Clickhouse at Cloudflare. By Marek Vavrusa
 
nuclio Overview October 2017
nuclio Overview October 2017nuclio Overview October 2017
nuclio Overview October 2017
 
Supercharge your Analytics with ClickHouse, v.2. By Vadim Tkachenko
Supercharge your Analytics with ClickHouse, v.2. By Vadim TkachenkoSupercharge your Analytics with ClickHouse, v.2. By Vadim Tkachenko
Supercharge your Analytics with ClickHouse, v.2. By Vadim Tkachenko
 
Flink Forward SF 2017: Chinmay Soman - Real Time Analytics in the real World ...
Flink Forward SF 2017: Chinmay Soman - Real Time Analytics in the real World ...Flink Forward SF 2017: Chinmay Soman - Real Time Analytics in the real World ...
Flink Forward SF 2017: Chinmay Soman - Real Time Analytics in the real World ...
 
Cost-Based Optimizer in Apache Spark 2.2
Cost-Based Optimizer in Apache Spark 2.2 Cost-Based Optimizer in Apache Spark 2.2
Cost-Based Optimizer in Apache Spark 2.2
 
Kafka Summit NYC 2017 - Easy, Scalable, Fault-tolerant Stream Processing with...
Kafka Summit NYC 2017 - Easy, Scalable, Fault-tolerant Stream Processing with...Kafka Summit NYC 2017 - Easy, Scalable, Fault-tolerant Stream Processing with...
Kafka Summit NYC 2017 - Easy, Scalable, Fault-tolerant Stream Processing with...
 
Apache Stratos (incubating) Hangout IV - Stratos Controller and CLI Internals
Apache Stratos (incubating) Hangout IV - Stratos Controller and CLI InternalsApache Stratos (incubating) Hangout IV - Stratos Controller and CLI Internals
Apache Stratos (incubating) Hangout IV - Stratos Controller and CLI Internals
 
Making Nested Columns as First Citizen in Apache Spark SQL
Making Nested Columns as First Citizen in Apache Spark SQLMaking Nested Columns as First Citizen in Apache Spark SQL
Making Nested Columns as First Citizen in Apache Spark SQL
 

Similar to ELK - from zero to coding class hero

Visualizing HPCC Systems Log Data Using ELK
Visualizing HPCC Systems Log Data Using ELKVisualizing HPCC Systems Log Data Using ELK
Visualizing HPCC Systems Log Data Using ELK
HPCC Systems
 

Similar to ELK - from zero to coding class hero (20)

Elk for applications on k8s
Elk for applications on k8sElk for applications on k8s
Elk for applications on k8s
 
Elastic stack Presentation
Elastic stack PresentationElastic stack Presentation
Elastic stack Presentation
 
Visualizing HPCC Systems Log Data Using ELK
Visualizing HPCC Systems Log Data Using ELKVisualizing HPCC Systems Log Data Using ELK
Visualizing HPCC Systems Log Data Using ELK
 
MySQL Audit using Percona audit plugin and ELK
MySQL Audit using Percona audit plugin and ELKMySQL Audit using Percona audit plugin and ELK
MySQL Audit using Percona audit plugin and ELK
 
Introduction to ELK
Introduction to ELKIntroduction to ELK
Introduction to ELK
 
Présentation ELK/SIEM et démo Wazuh
Présentation ELK/SIEM et démo WazuhPrésentation ELK/SIEM et démo Wazuh
Présentation ELK/SIEM et démo Wazuh
 
Workshop splunk 6.5-saint-louis-mo
Workshop splunk 6.5-saint-louis-moWorkshop splunk 6.5-saint-louis-mo
Workshop splunk 6.5-saint-louis-mo
 
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container EraLogging for Production Systems in The Container Era
Logging for Production Systems in The Container Era
 
Serhii Matynenko "How to Deal with Logs, Migrating from Monolith Architecture...
Serhii Matynenko "How to Deal with Logs, Migrating from Monolith Architecture...Serhii Matynenko "How to Deal with Logs, Migrating from Monolith Architecture...
Serhii Matynenko "How to Deal with Logs, Migrating from Monolith Architecture...
 
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
 
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...
 
Apache Eagle Architecture Evolvement
Apache Eagle Architecture EvolvementApache Eagle Architecture Evolvement
Apache Eagle Architecture Evolvement
 
Alfresco monitoring with Nagios and ELK stack
Alfresco monitoring with Nagios and ELK stackAlfresco monitoring with Nagios and ELK stack
Alfresco monitoring with Nagios and ELK stack
 
Play framework : A Walkthrough
Play framework : A WalkthroughPlay framework : A Walkthrough
Play framework : A Walkthrough
 
ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)
 
Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays N...
Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays N...Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays N...
Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays N...
 
WebCamp 2016: DevOps. Ярослав Погребняк: Gobetween - новый лоад балансер для ...
WebCamp 2016: DevOps. Ярослав Погребняк: Gobetween - новый лоад балансер для ...WebCamp 2016: DevOps. Ярослав Погребняк: Gobetween - новый лоад балансер для ...
WebCamp 2016: DevOps. Ярослав Погребняк: Gobetween - новый лоад балансер для ...
 
elk_stack_alexander_szalonnas
elk_stack_alexander_szalonnaselk_stack_alexander_szalonnas
elk_stack_alexander_szalonnas
 
Search and analyze data in real time
Search and analyze data in real timeSearch and analyze data in real time
Search and analyze data in real time
 
2015 03-16-elk at-bsides
2015 03-16-elk at-bsides2015 03-16-elk at-bsides
2015 03-16-elk at-bsides
 

Recently uploaded

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
Muhammad Subhan
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
FIDO Alliance
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
FIDO Alliance
 

Recently uploaded (20)

Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
 
Vector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxVector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptx
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps Productivity
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 

ELK - from zero to coding class hero