This provide details about ELK Stack and the usage of ELK in log analysis. Which gives analytics on log data and helps in predictive analysis.

2. Agenda:
Presented by: Asish KumarPresented by: Asish Kumar

3. What is Elasticsearch?
 Elasticsearch is a search engine.
 It is based on NoSQL Database and Framework build on top of Apache
Lucene.
 Elasticsearch is an open source distributed, REST full search and analytics
engine capable of solving a growing number of use cases.
 Elasticsearch is a highly scalable open-source full-text search and analytics
engine. It allows you to store, search, and analyze big volumes of data quickly
and in near real time. It is generally used as the underlying engine/technology
that powers applications that have complex search features and requirements.
 It use indexes to search the stored data, which makes it faster.
Presented by: Asish Kumar

4. Where to implement?
 You run an online web store where you allow your customers to search for products that you sell. In
this case, you can use Elasticsearch to store your entire product catalog and inventory and provide
search and autocomplete suggestions for them.
 You want to collect log or transaction data and you want to analyze and mine this data to look for
trends, statistics, summarizations, or anomalies. In this case, you can use Logstash (part of the
Elasticsearch/Logstash/Kibana stack) to collect, aggregate, and parse your data, and then have
Logstash feed this data into Elasticsearch. Once the data is in Elasticsearch, you can run searches and
aggregations to mine any information that is of interest to you.
 You run a price alerting platform which allows price-savvy customers to specify a rule like "I am
interested in buying a specific electronic gadget and I want to be notified if the price of gadget falls
below $X from any vendor within the next month". In this case you can scrape vendor prices, push
them into Elasticsearch and use its reverse-search (Percolator) capability to match price movements
against customer queries and eventually push the alerts out to the customer once matches are found.
 You have analytics/business-intelligence needs and want to quickly investigate, analyze,
visualize, and ask ad-hoc questions on a lot of data (think millions or billions of records).
In this case, you can use Elasticsearch to store your data and then use Kibana (part of the
Elasticsearch/Logstash/Kibana stack) to build custom dashboards that can visualize
aspects of your data that are important to you
Presented by: Asish Kumar

5. Architecture of Elasticsearch?
Presented by: Asish Kumar

6. Advantages of Multi-Cluster Elasticsearch
 Better Reliability: Issues in a single cluster will only affect a small proportion
of your customers.
 Better Application Performance: In a multi-cluster environment, you can
more effectively allocate resources for indexing, searching, and cluster state,
because each cluster is smaller.
 Easier Upgrades: Not only are you upgrading smaller clusters, but you can also
roll out the upgrade cluster by cluster, reducing the risk of a “Grand Slam”
failure. In the worst case, it is easier to completely replace the cluster, because
no cluster is so big that this becomes prohibitively expensive.
 Higher Overall Uptime: Even if you do have downtime, it is very unlikely to
take out every cluster.
Presented by: Asish Kumar

7. Need for Log Analysis.
Presented by: Asish Kumar

8. Presented by: Asish Kumar
Logstash Architecture.

9. Featurs of Logstash
 Data Pipeline tool
 Centralize the data processing
 Collect, analysis large verity of structured /Unstructured data.
 Provide plugin to connect with various types of input source.
 Provide features to turn data into meaningful information.
Presented by: Asish Kumar

10. Need for Log Analysis
Presented by: Asish Kumar

11. Problem with Log Analysis Each application writs log in its own format,
depending on the technology like log4net
,IIS,TomCat,Apache
• ‘’
• [ DD/MM/YYYY, MM/DD/YYYY, UTC and GMT]
• Different app server different Log, User has to login to
• the environment to access Log.
• Domain & Technical Expertise required to understand log
Presented by: Asish Kumar

12. What is
 Its Data visualization tool
 Provide real-time analysis, Summarization and
charting.
 Provide user friendly interface.
 Permits saving of dashboard & managing multiple
dashboard.
Presented by: Asish Kumar

13. Presented by: Asish Kumar

14. Presented by: Asish Kumar

15. Presented by: Asish Kumar