Logstash is an open source tool for collecting, parsing, and storing logs and other event data. It can input data from multiple sources, parse and transform the data, and output it to multiple destinations such as Elasticsearch. Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene. It allows storing, searching, and analyzing large volumes of data quickly and in near real-time. Together, Logstash can collect and parse log files, enriching the data, and outputting it to Elasticsearch for storage, search, and visualization, making log event data searchable and analyzable.

1. Logstash & Elasticsearch
Tatooine

2. Agenda
• Introduction
• What is logstash
• logstash in action
• What is Elasticsearch
• Elasticsearch in action

3. Introduction
• Issue : find out TPS based on ELB ’s log files
• What is the goals
• make life is easier
• No repeat to write program again and agains
• Make log event searchable
• Able to analyse log event

5. • Free & open source
• Event processing log file
• Support multiple sources and destinations
• logstash can manipulate data
• pipeline = input + (filter) + output
• codec is change data representation
• Flexible configuration
• Not limit only process log event

6. • Middle guy that sit between sources and destinations that
• manage event and logs
• collect data
• parse data
• enrich data
• store data (search & visualise)

7. logstash = inputs + filters + outputs

8. Elastic search
• NoSQL: Document Oriented
- Insert, delete, update, retrieve, analytic and search
• Built on top of apache lucene
- lucene is most popular java based full text search index implement
• Distributed text search engine
- Inverted Index
- Cluster

9. Ancient Search Engine

10. Why
Free & open source
Easy to scale (distributed)
Everything is one JSON call (Restful API)
Unleash power of Lucene under the hood
•
•
•
•
• Excellent query DSL
• Support for advance search features
(full text search)
Document oriented
Schema free
•
•
• Active community

11. What does it added to lucene?
• RESTFUL Service
- JSON API over HTTP
• High Available & Performance
- node form cluster
- distributed data using shard
- replicas request load, fault tolerance
• Long terms persistency
- write through persistent storage system

12. Document Oriented
name address hired_date department
Ball Paris 22/06/2015 Business
JenJa Tokyo 18/01/2016 Accounting
Kook London 1/04/2017 Marketing
{
…..
“hit”: [
{
“_index”: “general”
“_type”: “employee”,
“name”: “Ball”,
“address”: “Paris”,
“hired_date”: “22/06/2015”,
“department”: “business”
},
{
“_index”: “general”
“_type”: “employee”,
“name”: “JenJa”,
“address”: “Tokyo”,
“hired_date”: “18/01/2016”,
“department”: “Accounting”
},
…..
]
}
Table: employee
Database: general

13. Elastic search
Elasticsearch Relational MySQL
Index Database
Type Table
Document Row
Field Column

14. Who use elasticsearch ?

15. Conclusion
• Logstash used to load,parse and structured data
to elasticsearch
• Elasticsearch used to find number of TPS for each
API

16. Q & A
Thanks you