Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
ELK - Stack
A perfect match for your Log Management
Steve Behrendt
@derSteve
dersteve.com
2
The problem
 Distributed systems
 Service-oriented Architectures
 Microservices
 Multi-language systems
 Multi-techno...
Traditional Architecture
4
Browser
IIS
Store App
SQL Server
Product
Schema
Invoice
Schema
Billing
Schema
Customer
Schema
P...
Azure DocumentDB
MySQL
Azure SQL Server
Microservices
5
Browser
IIS
Apache / Tomcat
Azure
Product
UI
Service
Customer
UI
S...
Azure DocumentDB
MySQL
Azure SQL Server
Be the logs with you
6
Browser
IIS
Apache / Tomcat
Azure
Product
UI
Service
Custom...
The challenges
 Different log formats
 Each log has its expert
 Different log locations (machines/servers)
 Different ...
One solution: ELK
8
ELK
 E - Elasticsearch
 L - Logstash
 K – Kibana
 …
 Lucene
 Shield
 Marvel
9
The ELK architecture
10
Logs Logstash Elasticsearch Kibana
IIS
Syslog
EntLib
Broker Indexer
Search
Storage
Visualize
Visua...
Logstash
 Collecting,
 Filtering,
 Normalizing,
 Sending logs to a central location
 Understands the logs
11
Logstash Pipeline
12
•Log File
Input
•grok
•date
•geoip
•useragent
Filter
•Elasticsearch
•ConsoleOutput
Grok debugger
http://grokdebug.herokuapp.com/
13
Forwarder - lumberjack
 Separate service to forward messages to a
remote endpoint, e.g. logstash instance or
elasticsearc...
Logstash DEMO
15
Elasticsearch
 Based on lucene for indexing and searching - but
lucene is just a library and very complex
 Provides (sim...
Elasticsearch - Scaling
 Supports vertical (bigger hardware) and
horizontal scaling (more hardware)
 Horizontal scaling ...
The empty cluster
18
Node: Is a running instance of elasticsearch
Cluster: A cluster consists of one or more nodes with th...
Index and shards
19
Shard: low-level worker holding a slice of data. A single
instance of lucene.
Index: logical namespace...
Replicas / Failover
20
Primary and replica shards: Primary and associated replica
shard store the same documents.
Newly in...
Horizontal scaling
21
3 shards spread across 3 from 2 nodes.
Each shard is full fledged search engine. Scaling by
increasi...
Cluster Discovery
 Discovering nodes inside a cluster and electing a
master node
 Zen discovery
22
Types, Documents, Fields
23
Relational Database
Elasticsearch
Databases
Indices
Tables
Types
Rows
Documents
Columns
Fields
Storing documents
PUT /customer/employee/1
{
"first_name" : "John",
"last_name" : "Smith",
"age" : 25,
"about" : "I love t...
Retrieving document
GET /customer/employee/1
Search lite
GET /customer/employee/search
25
Query DSL
GET /customer/employee/_search
{
"query" : {
"match" : {
"last_name" : "Smith"
}
}
}
26
Elasticsearch DEMO
27
Kibana
 Data Visualization + Data Discovery
28
Kibana DEMO
29
What’s missing?
 Security
 Alerting
30
Alternatives?
31
32
Go and grok some logs
Upcoming SlideShare
Loading in …5
×

ELK - Stack - Munich .net UG

1,436 views

Published on

ELK Stack (Elasticsearch, Logstash, Kibana) as a Log-Management solution for the Microsoft developer presented at the .net Usergroup in Munich in June 2015.

Published in: Technology
  • Writing a good research paper isn't easy and it's the fruit of hard work. For help you can check writing expert. Check out, please ⇒ www.HelpWriting.net ⇐ I think they are the best
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I think you need a perfect and 100% unique academic essays papers have a look once this site i hope you will get valuable papers, ⇒ www.WritePaper.info ⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

ELK - Stack - Munich .net UG

  1. 1. ELK - Stack A perfect match for your Log Management
  2. 2. Steve Behrendt @derSteve dersteve.com 2
  3. 3. The problem  Distributed systems  Service-oriented Architectures  Microservices  Multi-language systems  Multi-technology stack  Multiple Datastores (SQL, noSQL, File stores) 3
  4. 4. Traditional Architecture 4 Browser IIS Store App SQL Server Product Schema Invoice Schema Billing Schema Customer Schema Product Module Customer Module
  5. 5. Azure DocumentDB MySQL Azure SQL Server Microservices 5 Browser IIS Apache / Tomcat Azure Product UI Service Customer UI Service Invoicing Service CMS Service Customer Schema SQL Server Product Schema Content Schema Billing Schema
  6. 6. Azure DocumentDB MySQL Azure SQL Server Be the logs with you 6 Browser IIS Apache / Tomcat Azure Product UI Service Customer UI Service Invoicing Service CMS Service Customer Schema SQL Server Product Schema Content Schema Billing Schema
  7. 7. The challenges  Different log formats  Each log has its expert  Different log locations (machines/servers)  Different date formats  Internet of things – decentralised log creation and storing  Searching files by keyword is hard  Combination of different messages  Setting the log context 7
  8. 8. One solution: ELK 8
  9. 9. ELK  E - Elasticsearch  L - Logstash  K – Kibana  …  Lucene  Shield  Marvel 9
  10. 10. The ELK architecture 10 Logs Logstash Elasticsearch Kibana IIS Syslog EntLib Broker Indexer Search Storage Visualize Visualize Visualize Visualize
  11. 11. Logstash  Collecting,  Filtering,  Normalizing,  Sending logs to a central location  Understands the logs 11
  12. 12. Logstash Pipeline 12 •Log File Input •grok •date •geoip •useragent Filter •Elasticsearch •ConsoleOutput
  13. 13. Grok debugger http://grokdebug.herokuapp.com/ 13
  14. 14. Forwarder - lumberjack  Separate service to forward messages to a remote endpoint, e.g. logstash instance or elasticsearch 14
  15. 15. Logstash DEMO 15
  16. 16. Elasticsearch  Based on lucene for indexing and searching - but lucene is just a library and very complex  Provides (simple) Restful - API abstraction on top of lucene  Stores documents in json format 16
  17. 17. Elasticsearch - Scaling  Supports vertical (bigger hardware) and horizontal scaling (more hardware)  Horizontal scaling is hard, but Elasticsearch is distributed by nature 17
  18. 18. The empty cluster 18 Node: Is a running instance of elasticsearch Cluster: A cluster consists of one or more nodes with the same cluster name that are working together to share their data and workload
  19. 19. Index and shards 19 Shard: low-level worker holding a slice of data. A single instance of lucene. Index: logical namespace that points to one or more physical shards
  20. 20. Replicas / Failover 20 Primary and replica shards: Primary and associated replica shard store the same documents. Newly indexed document first stored on a primary shard, then copied in parallel to the associated replica shard(s).
  21. 21. Horizontal scaling 21 3 shards spread across 3 from 2 nodes. Each shard is full fledged search engine. Scaling by increasing number of replica shards.
  22. 22. Cluster Discovery  Discovering nodes inside a cluster and electing a master node  Zen discovery 22
  23. 23. Types, Documents, Fields 23 Relational Database Elasticsearch Databases Indices Tables Types Rows Documents Columns Fields
  24. 24. Storing documents PUT /customer/employee/1 { "first_name" : "John", "last_name" : "Smith", "age" : 25, "about" : "I love to go rock climbing", "interests": [ "sports", "music" ] } 24
  25. 25. Retrieving document GET /customer/employee/1 Search lite GET /customer/employee/search 25
  26. 26. Query DSL GET /customer/employee/_search { "query" : { "match" : { "last_name" : "Smith" } } } 26
  27. 27. Elasticsearch DEMO 27
  28. 28. Kibana  Data Visualization + Data Discovery 28
  29. 29. Kibana DEMO 29
  30. 30. What’s missing?  Security  Alerting 30
  31. 31. Alternatives? 31
  32. 32. 32 Go and grok some logs

×