SlideShare a Scribd company logo
1 of 37
Download to read offline
Logs aggregation
and analysis
Agenda
● Problems with logs
● How do we do it at Divante - ELK Stack
o ElasticSearch
o Logstash
o Kibana
o Architecture
o Additional tools
● Summary
● Questions
What problems with logs
may we encounter?
Problems with logs
No consistent log format
http://blog.tersmitten.nl/how-to-colorize-your-log-files-with-ccze.html
Problems with logs
• cat
• grep
• awk
• sed
• tail
• regular expressions
Hampered log analysis, increased response time.
Log search, analysis – old school
Problems with logs
Heavy server load, low application performance
• Synchronous, blocking writing
• IO operations burdening the server
• Limited amount of inodes in the file system
• Relatively slow write speed
• Logs cleanup
http://wiki.processmaker.com/index.php/Advanced_Performance_Monitor_Dashboards
Problems with logs
Complex architecture - read and write problem
• Read/write on servers after the load balancer
- NFS application?
https://www.digitalocean.com/community/tutorials/5-common-server-setups-for-your-web-application
Problems with logs
Continuous monitoring
• Continuous monitoring of all application parameters is not easy
• Technical knowledge is required
How do we do it at Divante?
ELK Stack
ELK Stack - what is it?
ELK Stack is a set of tools:
It provides a centralized log management in distributed, high-availability systems.
E as in ElasticSearch
ElasticSearch
● + NoSQL database
● + Full-text search
● + REST Api (json)
● + Based on Apache Lucene
● + Replication, snapshots
● + Official PHP and JavaScript libraries available
● - No transactions
● Requirements: Java
E as in ElasticSearch
Installation and setup
● Java Oracle installation
o sudo apt-get install python-software-properties
o sudo add-apt-repository -y ppa:webupd8team/java
o sudo apt-get update
o sudo apt-get -y install oracle-java8-installer
● ElasticSearch installation
o wget and unzip of the latest distribution available at
https://www.elastic.co/downloads
o Setup in the elasticsearch.yml file
● Running ElasticSearch:
o ./bin/elasticsearch -d
L as in Logstash
● Aggregation of logs from multiple sources
● Normalization (parsing, filtering) of the collected data
● Sending the normalized data to various sources
● Requirements: Java
INPUT
FILTER
PARSE
OUTPUT
L as in Logstash
Input: (41)
● elasticsearch, imap
● rabbitmq, redis
● sqlite, syslog
● tcp, twitter
● varnishlog, websocket
Output: (55)
● elasticsearch, mongodb,
● redis, email, file, csv
● hipchat, http
● jira, redmine, rabbitmq
● tcp, websocket, zabbix
More at: http://logstash.net/docs/1.4.2/
INPUT
FILTER
PARSE
OUTPUT
L as in Logstash
Filters: (50)
● checksum
● csv, date
● elasticsearch
● fingerprint
● geoip, grep, grok
● json, json_encode
● ruby, split
● translate
● urldecode
● useragent, xml
More at: http://logstash.net/docs/1.4.2/
INPUT
FILTER
PARSE
OUTPUT
L as in Logstash
GROK filter
● Parsing and analysis of any text
● Grok is the best way to process even the most unstructured data
● Over 120 samples and possibility to create new ones
[ERROR] - 2015/13/03-11:01:31 - 192.168.0.1 - Some error message
[%{GREEDYDATA:message}] - %{DATESTAMP} - %{IP:client} - %{GREEDYDATA:message}
L as in Logstash
Installation and setup
● Java Oracle and Logstash installation
o wget and unzip of the latest distribution available at
https://www.elastic.co/downloads
● Setup:
input {
file {
type => "syslog"
path => ["/var/log/auth.log", "/var/log/syslog"]
}}
● Running Logstash:
o ./bin/logstash
K as in Kibana
● Data visualization in the form of a web app
● Data search, filters and analysis
● Intuitive interface, not only for the programmers
● Instant sharing and embedding of multiple dashboards
● Easy dashboard adjustment - JSON
● Export of the results
K as in Kibana
https://www.elastic.co/blog/kibana-4-literally
K as in Kibana
Bar chart:
https://www.elastic.co/blog/kibana-4-beta-2-get-now
K as in Kibana
Pie chart:
https://www.elastic.co/blog/kibana-4-for-investigating-pacs-super-pacs-and-your-neighbors
K as in Kibana
Histogram:
http://blog.qbox.io/kibana-4-and-elasticsearch-v-1-4-4-and-1-3-9
K as in Kibana
Data table:
https://www.elastic.co/blog/kibana-4-for-investigating-pacs-super-pacs-and-your-neighbors
K as in Kibana
Geolocation:
https://www.elastic.co/blog/kibana-4-literally
K as in Kibana
Relation chart:
http://demo.packetbeat.com/#/dashboard/elasticsearch/Packetbeat%2520Statistics
K as in Kibana
Installation and setup
● Kibana installation:
o wget and unzip of the latest distribution available at
https://www.elastic.co/downloads
● Setup:
o Kibana is set up on the default ElasticSearch port - 9300
● Running Kibana:
o ./bin/kibana
● Start in a browser:
o http://YOURDOMAIN.com:5601
ELK Stack
ARCHITECTURE
ELK - Architecture
Basic:
What is the potential problem?
ElasticSearch REST
API
ELK - Architecture
Performance + scalability
Setup with Redis
Providers
Indexer
ELK - Architecture
Performance + scalability
Setup with the use of a queuing mechanism
Indexer
Providers
Additional
TOOLS
Tools
ElasticSearch Plugin: Head
ElasticSearch monitoring and management panel
Installation:
● elasticsearch/bin/plugin -install mobz/elasticsearch-head
● http://localhost:9200/_plugin/head/
Tools
ElasticSearch Plugin: Morfologik
Plugin enabling the use of Polish characters
in ElasticSearch queries
Installation:
● cd elasticsearch
● bin/plugin -install com.github.chytreg/elasticsearch-analysis-
morfologik/2.3.1
Tools
Marvel
Application for the continuous monitoring of ElasticSearch
http://blog.qbox.io/elasticsearch-marvel-released
ELK Stack
Summary
● Consistent log structure
● Centralized log system
● Dashboard supporting the search and analysis processes
● Continuous monitoring
● Reduced response time
● Happy customer receives new charts :)
VS
http://blog.rootshell.be/tag/unix/
http://imgbuddy.com/kibana-demo.asp
Thank you!
Bartosz Picho
sales@divante.co
divante.co/blog

More Related Content

What's hot

Elk devops
Elk devopsElk devops
Elk devopsIdeato
 
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...ForgeRock
 
The ELK Stack - Get to Know Logs
The ELK Stack - Get to Know LogsThe ELK Stack - Get to Know Logs
The ELK Stack - Get to Know LogsGlobalLogic Ukraine
 
Experiences in ELK with D3.js for Large Log Analysis and Visualization
Experiences in ELK with D3.js  for Large Log Analysis  and VisualizationExperiences in ELK with D3.js  for Large Log Analysis  and Visualization
Experiences in ELK with D3.js for Large Log Analysis and VisualizationSurasak Sanguanpong
 
ELK, a real case study
ELK,  a real case studyELK,  a real case study
ELK, a real case studyPaolo Tonin
 
Logstash + Elasticsearch + Kibana Presentation on Startit Tech Meetup
Logstash + Elasticsearch + Kibana Presentation on Startit Tech MeetupLogstash + Elasticsearch + Kibana Presentation on Startit Tech Meetup
Logstash + Elasticsearch + Kibana Presentation on Startit Tech MeetupStartit
 
Log management with ELK
Log management with ELKLog management with ELK
Log management with ELKGeert Pante
 
Introduction to ELK
Introduction to ELKIntroduction to ELK
Introduction to ELKYuHsuan Chen
 
How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.Renzo Tomà
 
'Scalable Logging and Analytics with LogStash'
'Scalable Logging and Analytics with LogStash''Scalable Logging and Analytics with LogStash'
'Scalable Logging and Analytics with LogStash'Cloud Elements
 
Open Source Logging and Monitoring Tools
Open Source Logging and Monitoring ToolsOpen Source Logging and Monitoring Tools
Open Source Logging and Monitoring ToolsPhase2
 
Toronto High Scalability meetup - Scaling ELK
Toronto High Scalability meetup - Scaling ELKToronto High Scalability meetup - Scaling ELK
Toronto High Scalability meetup - Scaling ELKAndrew Trossman
 
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaLogging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaAmazee Labs
 
Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackRohit Sharma
 

What's hot (20)

ELK Stack
ELK StackELK Stack
ELK Stack
 
Elk devops
Elk devopsElk devops
Elk devops
 
Introducing ELK
Introducing ELKIntroducing ELK
Introducing ELK
 
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
 
The ELK Stack - Get to Know Logs
The ELK Stack - Get to Know LogsThe ELK Stack - Get to Know Logs
The ELK Stack - Get to Know Logs
 
Experiences in ELK with D3.js for Large Log Analysis and Visualization
Experiences in ELK with D3.js  for Large Log Analysis  and VisualizationExperiences in ELK with D3.js  for Large Log Analysis  and Visualization
Experiences in ELK with D3.js for Large Log Analysis and Visualization
 
ELK, a real case study
ELK,  a real case studyELK,  a real case study
ELK, a real case study
 
Logstash + Elasticsearch + Kibana Presentation on Startit Tech Meetup
Logstash + Elasticsearch + Kibana Presentation on Startit Tech MeetupLogstash + Elasticsearch + Kibana Presentation on Startit Tech Meetup
Logstash + Elasticsearch + Kibana Presentation on Startit Tech Meetup
 
Log management with ELK
Log management with ELKLog management with ELK
Log management with ELK
 
Introduction to ELK
Introduction to ELKIntroduction to ELK
Introduction to ELK
 
elk_stack_alexander_szalonnas
elk_stack_alexander_szalonnaselk_stack_alexander_szalonnas
elk_stack_alexander_szalonnas
 
Elk
Elk Elk
Elk
 
How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.
 
'Scalable Logging and Analytics with LogStash'
'Scalable Logging and Analytics with LogStash''Scalable Logging and Analytics with LogStash'
'Scalable Logging and Analytics with LogStash'
 
Open Source Logging and Monitoring Tools
Open Source Logging and Monitoring ToolsOpen Source Logging and Monitoring Tools
Open Source Logging and Monitoring Tools
 
Elk scilifelab
Elk scilifelabElk scilifelab
Elk scilifelab
 
Introduction to ELK
Introduction to ELKIntroduction to ELK
Introduction to ELK
 
Toronto High Scalability meetup - Scaling ELK
Toronto High Scalability meetup - Scaling ELKToronto High Scalability meetup - Scaling ELK
Toronto High Scalability meetup - Scaling ELK
 
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaLogging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & Kibana
 
Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK Stack
 

Viewers also liked

ELK - Stack - Munich .net UG
ELK - Stack - Munich .net UGELK - Stack - Munich .net UG
ELK - Stack - Munich .net UGSteve Behrendt
 
OIS Architecture Review
OIS Architecture ReviewOIS Architecture Review
OIS Architecture ReviewForgeRock
 
ELK at LinkedIn - Kafka, scaling, lessons learned
ELK at LinkedIn - Kafka, scaling, lessons learnedELK at LinkedIn - Kafka, scaling, lessons learned
ELK at LinkedIn - Kafka, scaling, lessons learnedTin Le
 
Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014
Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014
Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014clairvoyantllc
 
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et KibanaJournée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et KibanaPublicis Sapient Engineering
 
Open source log analytics
Open source log analyticsOpen source log analytics
Open source log analyticsVinod Nayal
 
Monitor your Atlassian stack like the NSA
Monitor your Atlassian stack like the NSAMonitor your Atlassian stack like the NSA
Monitor your Atlassian stack like the NSAACA IT-Solutions
 
Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...
Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...
Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...Amazon Web Services
 

Viewers also liked (10)

ELK - Stack - Munich .net UG
ELK - Stack - Munich .net UGELK - Stack - Munich .net UG
ELK - Stack - Munich .net UG
 
OIS Architecture Review
OIS Architecture ReviewOIS Architecture Review
OIS Architecture Review
 
ELK at LinkedIn - Kafka, scaling, lessons learned
ELK at LinkedIn - Kafka, scaling, lessons learnedELK at LinkedIn - Kafka, scaling, lessons learned
ELK at LinkedIn - Kafka, scaling, lessons learned
 
Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014
Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014
Log analysis using Logstash,ElasticSearch and Kibana - Desert Code Camp 2014
 
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et KibanaJournée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
 
Open source log analytics
Open source log analyticsOpen source log analytics
Open source log analytics
 
Monitor your Atlassian stack like the NSA
Monitor your Atlassian stack like the NSAMonitor your Atlassian stack like the NSA
Monitor your Atlassian stack like the NSA
 
On Centralizing Logs
On Centralizing LogsOn Centralizing Logs
On Centralizing Logs
 
Elk stack
Elk stackElk stack
Elk stack
 
Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...
Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...
Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...
 

Similar to Logs aggregation and analysis

Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsExperiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsCeph Community
 
Guider: An Integrated Runtime Performance Analyzer on AGL
Guider: An Integrated Runtime Performance Analyzer on AGLGuider: An Integrated Runtime Performance Analyzer on AGL
Guider: An Integrated Runtime Performance Analyzer on AGLPeace Lee
 
OSMC 2010 | Monitoring mit Icinga by Icinga Team
OSMC 2010 | Monitoring mit Icinga by Icinga TeamOSMC 2010 | Monitoring mit Icinga by Icinga Team
OSMC 2010 | Monitoring mit Icinga by Icinga TeamNETWAYS
 
Elk ruminating on logs
Elk ruminating on logsElk ruminating on logs
Elk ruminating on logsMathew Beane
 
Chef for OpenStack - OpenStack Fall 2012 Summit
Chef for OpenStack  - OpenStack Fall 2012 SummitChef for OpenStack  - OpenStack Fall 2012 Summit
Chef for OpenStack - OpenStack Fall 2012 SummitMatt Ray
 
Architectures, Frameworks and Infrastructure
Architectures, Frameworks and InfrastructureArchitectures, Frameworks and Infrastructure
Architectures, Frameworks and Infrastructureharendra_pathak
 
Creating PostgreSQL-as-a-Service at Scale
Creating PostgreSQL-as-a-Service at ScaleCreating PostgreSQL-as-a-Service at Scale
Creating PostgreSQL-as-a-Service at ScaleSean Chittenden
 
Openstack India May Meetup
Openstack India May MeetupOpenstack India May Meetup
Openstack India May MeetupDeepak Garg
 
Scaling ELK Stack - DevOpsDays Singapore
Scaling ELK Stack - DevOpsDays SingaporeScaling ELK Stack - DevOpsDays Singapore
Scaling ELK Stack - DevOpsDays SingaporeAngad Singh
 
Technology behind-real-time-log-analytics
Technology behind-real-time-log-analytics Technology behind-real-time-log-analytics
Technology behind-real-time-log-analytics Data Science Thailand
 
COUG_AAbate_Oracle_Database_12c_New_Features
COUG_AAbate_Oracle_Database_12c_New_FeaturesCOUG_AAbate_Oracle_Database_12c_New_Features
COUG_AAbate_Oracle_Database_12c_New_FeaturesAlfredo Abate
 
Icinga 2011 at Chemnitzer Linuxtage
Icinga 2011 at Chemnitzer LinuxtageIcinga 2011 at Chemnitzer Linuxtage
Icinga 2011 at Chemnitzer LinuxtageIcinga
 
ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)Mathew Beane
 
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kuberneteshbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on KubernetesHBaseCon
 
(BDT404) Large-Scale ETL Data Flows w/AWS Data Pipeline & Dataduct
(BDT404) Large-Scale ETL Data Flows w/AWS Data Pipeline & Dataduct(BDT404) Large-Scale ETL Data Flows w/AWS Data Pipeline & Dataduct
(BDT404) Large-Scale ETL Data Flows w/AWS Data Pipeline & DataductAmazon Web Services
 
SharePoint Saturday San Antonio: SharePoint 2010 Performance
SharePoint Saturday San Antonio: SharePoint 2010 PerformanceSharePoint Saturday San Antonio: SharePoint 2010 Performance
SharePoint Saturday San Antonio: SharePoint 2010 PerformanceBrian Culver
 
Australian OpenStack User Group August 2012: Chef for OpenStack
Australian OpenStack User Group August 2012: Chef for OpenStackAustralian OpenStack User Group August 2012: Chef for OpenStack
Australian OpenStack User Group August 2012: Chef for OpenStackMatt Ray
 
PLSSUG - Troubleshoot SQL Server performance problems like a Microsoft Engineer
PLSSUG - Troubleshoot SQL Server performance problems like a Microsoft EngineerPLSSUG - Troubleshoot SQL Server performance problems like a Microsoft Engineer
PLSSUG - Troubleshoot SQL Server performance problems like a Microsoft EngineerMarek Maśko
 

Similar to Logs aggregation and analysis (20)

Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsExperiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah Watkins
 
Guider: An Integrated Runtime Performance Analyzer on AGL
Guider: An Integrated Runtime Performance Analyzer on AGLGuider: An Integrated Runtime Performance Analyzer on AGL
Guider: An Integrated Runtime Performance Analyzer on AGL
 
Logstash
LogstashLogstash
Logstash
 
OSMC 2010 | Monitoring mit Icinga by Icinga Team
OSMC 2010 | Monitoring mit Icinga by Icinga TeamOSMC 2010 | Monitoring mit Icinga by Icinga Team
OSMC 2010 | Monitoring mit Icinga by Icinga Team
 
Elk ruminating on logs
Elk ruminating on logsElk ruminating on logs
Elk ruminating on logs
 
Chef for OpenStack - OpenStack Fall 2012 Summit
Chef for OpenStack  - OpenStack Fall 2012 SummitChef for OpenStack  - OpenStack Fall 2012 Summit
Chef for OpenStack - OpenStack Fall 2012 Summit
 
Chef for OpenStack- Fall 2012.pdf
Chef for OpenStack- Fall 2012.pdfChef for OpenStack- Fall 2012.pdf
Chef for OpenStack- Fall 2012.pdf
 
Architectures, Frameworks and Infrastructure
Architectures, Frameworks and InfrastructureArchitectures, Frameworks and Infrastructure
Architectures, Frameworks and Infrastructure
 
Creating PostgreSQL-as-a-Service at Scale
Creating PostgreSQL-as-a-Service at ScaleCreating PostgreSQL-as-a-Service at Scale
Creating PostgreSQL-as-a-Service at Scale
 
Openstack India May Meetup
Openstack India May MeetupOpenstack India May Meetup
Openstack India May Meetup
 
Scaling ELK Stack - DevOpsDays Singapore
Scaling ELK Stack - DevOpsDays SingaporeScaling ELK Stack - DevOpsDays Singapore
Scaling ELK Stack - DevOpsDays Singapore
 
Technology behind-real-time-log-analytics
Technology behind-real-time-log-analytics Technology behind-real-time-log-analytics
Technology behind-real-time-log-analytics
 
COUG_AAbate_Oracle_Database_12c_New_Features
COUG_AAbate_Oracle_Database_12c_New_FeaturesCOUG_AAbate_Oracle_Database_12c_New_Features
COUG_AAbate_Oracle_Database_12c_New_Features
 
Icinga 2011 at Chemnitzer Linuxtage
Icinga 2011 at Chemnitzer LinuxtageIcinga 2011 at Chemnitzer Linuxtage
Icinga 2011 at Chemnitzer Linuxtage
 
ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)
 
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kuberneteshbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
 
(BDT404) Large-Scale ETL Data Flows w/AWS Data Pipeline & Dataduct
(BDT404) Large-Scale ETL Data Flows w/AWS Data Pipeline & Dataduct(BDT404) Large-Scale ETL Data Flows w/AWS Data Pipeline & Dataduct
(BDT404) Large-Scale ETL Data Flows w/AWS Data Pipeline & Dataduct
 
SharePoint Saturday San Antonio: SharePoint 2010 Performance
SharePoint Saturday San Antonio: SharePoint 2010 PerformanceSharePoint Saturday San Antonio: SharePoint 2010 Performance
SharePoint Saturday San Antonio: SharePoint 2010 Performance
 
Australian OpenStack User Group August 2012: Chef for OpenStack
Australian OpenStack User Group August 2012: Chef for OpenStackAustralian OpenStack User Group August 2012: Chef for OpenStack
Australian OpenStack User Group August 2012: Chef for OpenStack
 
PLSSUG - Troubleshoot SQL Server performance problems like a Microsoft Engineer
PLSSUG - Troubleshoot SQL Server performance problems like a Microsoft EngineerPLSSUG - Troubleshoot SQL Server performance problems like a Microsoft Engineer
PLSSUG - Troubleshoot SQL Server performance problems like a Microsoft Engineer
 

More from Divante

The eCommerce Platforms in the Global Setup
The eCommerce Platforms in the Global Setup	The eCommerce Platforms in the Global Setup
The eCommerce Platforms in the Global Setup Divante
 
eCommerce Trends 2020
eCommerce Trends 2020eCommerce Trends 2020
eCommerce Trends 2020Divante
 
Async & Bulk REST API new possibilities of communication between systems
Async & Bulk REST API new possibilities of communication  between systemsAsync & Bulk REST API new possibilities of communication  between systems
Async & Bulk REST API new possibilities of communication between systemsDivante
 
Magento Functional Testing Framework a way to seriously write automated tests...
Magento Functional Testing Framework a way to seriously write automated tests...Magento Functional Testing Framework a way to seriously write automated tests...
Magento Functional Testing Framework a way to seriously write automated tests...Divante
 
Die Top 10 Progressive Web Apps in der Modernbranche
Die Top 10 Progressive Web Apps in der ModernbrancheDie Top 10 Progressive Web Apps in der Modernbranche
Die Top 10 Progressive Web Apps in der ModernbrancheDivante
 
progressive web apps - pwa as a game changer for e-commerce - meet magento i...
 progressive web apps - pwa as a game changer for e-commerce - meet magento i... progressive web apps - pwa as a game changer for e-commerce - meet magento i...
progressive web apps - pwa as a game changer for e-commerce - meet magento i...Divante
 
Customer churn - how to stop it?
Customer churn - how to stop it?Customer churn - how to stop it?
Customer churn - how to stop it?Divante
 
eCommerce trends 2019 by Divante.co
eCommerce trends 2019 by Divante.coeCommerce trends 2019 by Divante.co
eCommerce trends 2019 by Divante.coDivante
 
How to create a Vue Storefront theme
How to create a Vue Storefront themeHow to create a Vue Storefront theme
How to create a Vue Storefront themeDivante
 
Game changer for e-commerce - Vue Storefront - open source pwa
Game changer for e-commerce - Vue Storefront - open source pwa Game changer for e-commerce - Vue Storefront - open source pwa
Game changer for e-commerce - Vue Storefront - open source pwa Divante
 
Vue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speech
Vue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speechVue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speech
Vue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speechDivante
 
How to successfully onboard end-clients to a B2B Platform - Magento Imagine ...
How to successfully onboard  end-clients to a B2B Platform - Magento Imagine ...How to successfully onboard  end-clients to a B2B Platform - Magento Imagine ...
How to successfully onboard end-clients to a B2B Platform - Magento Imagine ...Divante
 
eCommerce trends from 2017 to 2018 by Divante.co
eCommerce trends from 2017 to 2018 by Divante.coeCommerce trends from 2017 to 2018 by Divante.co
eCommerce trends from 2017 to 2018 by Divante.coDivante
 
Designing for PWA (Progressive Web Apps)
Designing for PWA (Progressive Web Apps)Designing for PWA (Progressive Web Apps)
Designing for PWA (Progressive Web Apps)Divante
 
Why is crud a bad idea - focus on real scenarios
Why is crud a bad idea - focus on real scenariosWhy is crud a bad idea - focus on real scenarios
Why is crud a bad idea - focus on real scenariosDivante
 
vue-storefront - PWA eCommerce for Magento2 MM17NYC presentation
vue-storefront - PWA eCommerce for Magento2 MM17NYC presentationvue-storefront - PWA eCommerce for Magento2 MM17NYC presentation
vue-storefront - PWA eCommerce for Magento2 MM17NYC presentationDivante
 
Pimcore Overview - Pimcore5
Pimcore Overview - Pimcore5Pimcore Overview - Pimcore5
Pimcore Overview - Pimcore5Divante
 
Pimcore E-Commerce Framework - Pimcore5
Pimcore E-Commerce Framework - Pimcore5Pimcore E-Commerce Framework - Pimcore5
Pimcore E-Commerce Framework - Pimcore5Divante
 
The biggest stores on Magento
The biggest stores on MagentoThe biggest stores on Magento
The biggest stores on MagentoDivante
 
B2B Commerce - how to become successful
B2B Commerce - how to become successfulB2B Commerce - how to become successful
B2B Commerce - how to become successfulDivante
 

More from Divante (20)

The eCommerce Platforms in the Global Setup
The eCommerce Platforms in the Global Setup	The eCommerce Platforms in the Global Setup
The eCommerce Platforms in the Global Setup
 
eCommerce Trends 2020
eCommerce Trends 2020eCommerce Trends 2020
eCommerce Trends 2020
 
Async & Bulk REST API new possibilities of communication between systems
Async & Bulk REST API new possibilities of communication  between systemsAsync & Bulk REST API new possibilities of communication  between systems
Async & Bulk REST API new possibilities of communication between systems
 
Magento Functional Testing Framework a way to seriously write automated tests...
Magento Functional Testing Framework a way to seriously write automated tests...Magento Functional Testing Framework a way to seriously write automated tests...
Magento Functional Testing Framework a way to seriously write automated tests...
 
Die Top 10 Progressive Web Apps in der Modernbranche
Die Top 10 Progressive Web Apps in der ModernbrancheDie Top 10 Progressive Web Apps in der Modernbranche
Die Top 10 Progressive Web Apps in der Modernbranche
 
progressive web apps - pwa as a game changer for e-commerce - meet magento i...
 progressive web apps - pwa as a game changer for e-commerce - meet magento i... progressive web apps - pwa as a game changer for e-commerce - meet magento i...
progressive web apps - pwa as a game changer for e-commerce - meet magento i...
 
Customer churn - how to stop it?
Customer churn - how to stop it?Customer churn - how to stop it?
Customer churn - how to stop it?
 
eCommerce trends 2019 by Divante.co
eCommerce trends 2019 by Divante.coeCommerce trends 2019 by Divante.co
eCommerce trends 2019 by Divante.co
 
How to create a Vue Storefront theme
How to create a Vue Storefront themeHow to create a Vue Storefront theme
How to create a Vue Storefront theme
 
Game changer for e-commerce - Vue Storefront - open source pwa
Game changer for e-commerce - Vue Storefront - open source pwa Game changer for e-commerce - Vue Storefront - open source pwa
Game changer for e-commerce - Vue Storefront - open source pwa
 
Vue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speech
Vue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speechVue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speech
Vue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speech
 
How to successfully onboard end-clients to a B2B Platform - Magento Imagine ...
How to successfully onboard  end-clients to a B2B Platform - Magento Imagine ...How to successfully onboard  end-clients to a B2B Platform - Magento Imagine ...
How to successfully onboard end-clients to a B2B Platform - Magento Imagine ...
 
eCommerce trends from 2017 to 2018 by Divante.co
eCommerce trends from 2017 to 2018 by Divante.coeCommerce trends from 2017 to 2018 by Divante.co
eCommerce trends from 2017 to 2018 by Divante.co
 
Designing for PWA (Progressive Web Apps)
Designing for PWA (Progressive Web Apps)Designing for PWA (Progressive Web Apps)
Designing for PWA (Progressive Web Apps)
 
Why is crud a bad idea - focus on real scenarios
Why is crud a bad idea - focus on real scenariosWhy is crud a bad idea - focus on real scenarios
Why is crud a bad idea - focus on real scenarios
 
vue-storefront - PWA eCommerce for Magento2 MM17NYC presentation
vue-storefront - PWA eCommerce for Magento2 MM17NYC presentationvue-storefront - PWA eCommerce for Magento2 MM17NYC presentation
vue-storefront - PWA eCommerce for Magento2 MM17NYC presentation
 
Pimcore Overview - Pimcore5
Pimcore Overview - Pimcore5Pimcore Overview - Pimcore5
Pimcore Overview - Pimcore5
 
Pimcore E-Commerce Framework - Pimcore5
Pimcore E-Commerce Framework - Pimcore5Pimcore E-Commerce Framework - Pimcore5
Pimcore E-Commerce Framework - Pimcore5
 
The biggest stores on Magento
The biggest stores on MagentoThe biggest stores on Magento
The biggest stores on Magento
 
B2B Commerce - how to become successful
B2B Commerce - how to become successfulB2B Commerce - how to become successful
B2B Commerce - how to become successful
 

Recently uploaded

eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolsosttopstonverter
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slidesvaideheekore1
 
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jGraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jNeo4j
 
Osi security architecture in network.pptx
Osi security architecture in network.pptxOsi security architecture in network.pptx
Osi security architecture in network.pptxVinzoCenzo
 
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...OnePlan Solutions
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingShane Coughlan
 
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shardsChristopher Curtin
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLionel Briand
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxSasikiranMarri
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...OnePlan Solutions
 
Effectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorEffectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorTier1 app
 
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...kalichargn70th171
 
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptxThe Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptxRTS corp
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?Alexandre Beguel
 
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonLeveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonApplitools
 
Amazon Bedrock in Action - presentation of the Bedrock's capabilities
Amazon Bedrock in Action - presentation of the Bedrock's capabilitiesAmazon Bedrock in Action - presentation of the Bedrock's capabilities
Amazon Bedrock in Action - presentation of the Bedrock's capabilitiesKrzysztofKkol1
 
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfPros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfkalichargn70th171
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecturerahul_net
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldRoberto Pérez Alcolea
 

Recently uploaded (20)

eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration tools
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slides
 
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jGraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
 
Osi security architecture in network.pptx
Osi security architecture in network.pptxOsi security architecture in network.pptx
Osi security architecture in network.pptx
 
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
 
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and Repair
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
 
Effectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorEffectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryError
 
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
 
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptxThe Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?
 
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonLeveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
 
Amazon Bedrock in Action - presentation of the Bedrock's capabilities
Amazon Bedrock in Action - presentation of the Bedrock's capabilitiesAmazon Bedrock in Action - presentation of the Bedrock's capabilities
Amazon Bedrock in Action - presentation of the Bedrock's capabilities
 
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfPros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecture
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository world
 

Logs aggregation and analysis

  • 2. Agenda ● Problems with logs ● How do we do it at Divante - ELK Stack o ElasticSearch o Logstash o Kibana o Architecture o Additional tools ● Summary ● Questions
  • 3. What problems with logs may we encounter?
  • 4. Problems with logs No consistent log format http://blog.tersmitten.nl/how-to-colorize-your-log-files-with-ccze.html
  • 5. Problems with logs • cat • grep • awk • sed • tail • regular expressions Hampered log analysis, increased response time. Log search, analysis – old school
  • 6. Problems with logs Heavy server load, low application performance • Synchronous, blocking writing • IO operations burdening the server • Limited amount of inodes in the file system • Relatively slow write speed • Logs cleanup http://wiki.processmaker.com/index.php/Advanced_Performance_Monitor_Dashboards
  • 7. Problems with logs Complex architecture - read and write problem • Read/write on servers after the load balancer - NFS application? https://www.digitalocean.com/community/tutorials/5-common-server-setups-for-your-web-application
  • 8. Problems with logs Continuous monitoring • Continuous monitoring of all application parameters is not easy • Technical knowledge is required
  • 9. How do we do it at Divante? ELK Stack
  • 10. ELK Stack - what is it? ELK Stack is a set of tools: It provides a centralized log management in distributed, high-availability systems.
  • 11. E as in ElasticSearch ElasticSearch ● + NoSQL database ● + Full-text search ● + REST Api (json) ● + Based on Apache Lucene ● + Replication, snapshots ● + Official PHP and JavaScript libraries available ● - No transactions ● Requirements: Java
  • 12. E as in ElasticSearch Installation and setup ● Java Oracle installation o sudo apt-get install python-software-properties o sudo add-apt-repository -y ppa:webupd8team/java o sudo apt-get update o sudo apt-get -y install oracle-java8-installer ● ElasticSearch installation o wget and unzip of the latest distribution available at https://www.elastic.co/downloads o Setup in the elasticsearch.yml file ● Running ElasticSearch: o ./bin/elasticsearch -d
  • 13. L as in Logstash ● Aggregation of logs from multiple sources ● Normalization (parsing, filtering) of the collected data ● Sending the normalized data to various sources ● Requirements: Java INPUT FILTER PARSE OUTPUT
  • 14. L as in Logstash Input: (41) ● elasticsearch, imap ● rabbitmq, redis ● sqlite, syslog ● tcp, twitter ● varnishlog, websocket Output: (55) ● elasticsearch, mongodb, ● redis, email, file, csv ● hipchat, http ● jira, redmine, rabbitmq ● tcp, websocket, zabbix More at: http://logstash.net/docs/1.4.2/ INPUT FILTER PARSE OUTPUT
  • 15. L as in Logstash Filters: (50) ● checksum ● csv, date ● elasticsearch ● fingerprint ● geoip, grep, grok ● json, json_encode ● ruby, split ● translate ● urldecode ● useragent, xml More at: http://logstash.net/docs/1.4.2/ INPUT FILTER PARSE OUTPUT
  • 16. L as in Logstash GROK filter ● Parsing and analysis of any text ● Grok is the best way to process even the most unstructured data ● Over 120 samples and possibility to create new ones [ERROR] - 2015/13/03-11:01:31 - 192.168.0.1 - Some error message [%{GREEDYDATA:message}] - %{DATESTAMP} - %{IP:client} - %{GREEDYDATA:message}
  • 17. L as in Logstash Installation and setup ● Java Oracle and Logstash installation o wget and unzip of the latest distribution available at https://www.elastic.co/downloads ● Setup: input { file { type => "syslog" path => ["/var/log/auth.log", "/var/log/syslog"] }} ● Running Logstash: o ./bin/logstash
  • 18. K as in Kibana ● Data visualization in the form of a web app ● Data search, filters and analysis ● Intuitive interface, not only for the programmers ● Instant sharing and embedding of multiple dashboards ● Easy dashboard adjustment - JSON ● Export of the results
  • 19. K as in Kibana https://www.elastic.co/blog/kibana-4-literally
  • 20. K as in Kibana Bar chart: https://www.elastic.co/blog/kibana-4-beta-2-get-now
  • 21. K as in Kibana Pie chart: https://www.elastic.co/blog/kibana-4-for-investigating-pacs-super-pacs-and-your-neighbors
  • 22. K as in Kibana Histogram: http://blog.qbox.io/kibana-4-and-elasticsearch-v-1-4-4-and-1-3-9
  • 23. K as in Kibana Data table: https://www.elastic.co/blog/kibana-4-for-investigating-pacs-super-pacs-and-your-neighbors
  • 24. K as in Kibana Geolocation: https://www.elastic.co/blog/kibana-4-literally
  • 25. K as in Kibana Relation chart: http://demo.packetbeat.com/#/dashboard/elasticsearch/Packetbeat%2520Statistics
  • 26. K as in Kibana Installation and setup ● Kibana installation: o wget and unzip of the latest distribution available at https://www.elastic.co/downloads ● Setup: o Kibana is set up on the default ElasticSearch port - 9300 ● Running Kibana: o ./bin/kibana ● Start in a browser: o http://YOURDOMAIN.com:5601
  • 28. ELK - Architecture Basic: What is the potential problem? ElasticSearch REST API
  • 29. ELK - Architecture Performance + scalability Setup with Redis Providers Indexer
  • 30. ELK - Architecture Performance + scalability Setup with the use of a queuing mechanism Indexer Providers
  • 32. Tools ElasticSearch Plugin: Head ElasticSearch monitoring and management panel Installation: ● elasticsearch/bin/plugin -install mobz/elasticsearch-head ● http://localhost:9200/_plugin/head/
  • 33. Tools ElasticSearch Plugin: Morfologik Plugin enabling the use of Polish characters in ElasticSearch queries Installation: ● cd elasticsearch ● bin/plugin -install com.github.chytreg/elasticsearch-analysis- morfologik/2.3.1
  • 34. Tools Marvel Application for the continuous monitoring of ElasticSearch http://blog.qbox.io/elasticsearch-marvel-released
  • 35. ELK Stack Summary ● Consistent log structure ● Centralized log system ● Dashboard supporting the search and analysis processes ● Continuous monitoring ● Reduced response time ● Happy customer receives new charts :)