The document discusses the challenges of managing large amounts of data across multiple systems and the limitations of traditional logging solutions. It presents the ELK stack (Elasticsearch, Logstash, Kibana) as an open-source alternative that facilitates data collection, storage, and visualization. The document highlights the benefits of each component within the stack, including ease of installation, scalability, and real-time analytics capabilities.

1. ELK STACK
Master the data
BY EL MAHDI BENZEKRI

2. PLAN
 Ourproblem
 Old solutions limits
 Other solutions
 ELK Stack overview
 Logstash/Elasticsearch/Kibana
 Demo

3. Who are we?

4. Our problem
Lot of
users
Lots of
data
Lots of
systems
Integrated
collection
of logs

5. Old solutions limits
• Tail & grep impossible with multiple systems
• DBs dont scale and its difficult to extract statistics
• Syslog :
• Only log history
• Not flexible : syslog client to syslog server

7. Other solutions
- Biggest tool
- 537 apps
- High cost and
complexity
- Up to
~10000$
SPLUNK
- Easy setup
- Establish
baseline and
notify
- Sensitive data
- Data
transmition
overhead
- Complex
pricing strategy
SUMO LOGIC (SAAS)

8. ELK Stack
- Open source
- Easy install
- Mature
components
- Complexity of
three different
products
- Diffrent
machines in
production

9. Logstash
Over 200 plugins and all pluggable and mixable
Unify and democratize data from different sources and
into different destinations
Data collection engine

10. Logstash loves data

11. Logstash popular plugins
- Date
- mutate
- Grok
- Multiline
FILTER
- File
- Stdin
- syslog
INPUT
- Elasticsearch
- File
- Mongodb
- Email
- stdout
OUTPUT

12. Elasticsearch
• Build on top of Apache Lucene(java)
• Developer-Friendly, RESTful API
• High Availability
• Massively Distributed
• Real-Time data and Advanced Analytics(1s from indexing to
searching)

13. Elasticsearch architecture

14. Elasticsearch REST API

15. Kibana
Real time dashboards
Designed to interact with elasticsearch data
Full JS and HTML5 analytics and visualization platform

16. Search query syntax
• status:active
• title:(quick OR brown)
• title:(quick brown)
• author:"John Smith«
• book.*:(quick brown)
• _missing_:title
• _exists_:title
• qu?ck bro*
• name:/joh?n(ath[oa]n)/
• Fuzziness :
• quikc~ brwn~ foks~
• quikc~1
• Proximity seach:
• "fox quick"~5
• Ranges
• date:[2012-01-01 TO 2012-
12-31]
• count:{10 TO *] = count:>10
• Boosting
• quick^2 fox
• Boolean operators
• quick brown +fox –news
• Avoid
• ((quick AND fox) OR
(brown AND fox) OR
fox) AND NOT news
• *ing

17. Demo architecture
TOMCAT Logs
Accounts.json
REST API
File input plugin

18. 01110100 01101000 01100001
01101110 01101011 00100000
01111001 01101111 01110101
:D