Recommended
More Related Content
What's hot
What's hot (20)
Similar to ELK - Stack - Munich .net UG
Similar to ELK - Stack - Munich .net UG (20)
Recently uploaded
Recently uploaded (20)
ELK - Stack - Munich .net UG
- 1. ELK - Stack A perfect match for your Log Management
- 3. The problem Distributed systems Service-oriented Architectures Microservices Multi-language systems Multi-technology stack Multiple Datastores (SQL, noSQL, File stores) 3
- 4. Traditional Architecture 4 Browser IIS Store App SQL Server Product Schema Invoice Schema Billing Schema Customer Schema Product Module Customer Module
- 5. Azure DocumentDB MySQL Azure SQL Server Microservices 5 Browser IIS Apache / Tomcat Azure Product UI Service Customer UI Service Invoicing Service CMS Service Customer Schema SQL Server Product Schema Content Schema Billing Schema
- 6. Azure DocumentDB MySQL Azure SQL Server Be the logs with you 6 Browser IIS Apache / Tomcat Azure Product UI Service Customer UI Service Invoicing Service CMS Service Customer Schema SQL Server Product Schema Content Schema Billing Schema
- 7. The challenges Different log formats Each log has its expert Different log locations (machines/servers) Different date formats Internet of things – decentralised log creation and storing Searching files by keyword is hard Combination of different messages Setting the log context 7
- 9. ELK E - Elasticsearch L - Logstash K – Kibana … Lucene Shield Marvel 9
- 10. The ELK architecture 10 Logs Logstash Elasticsearch Kibana IIS Syslog EntLib Broker Indexer Search Storage Visualize Visualize Visualize Visualize
- 11. Logstash Collecting, Filtering, Normalizing, Sending logs to a central location Understands the logs 11
- 14. Forwarder - lumberjack Separate service to forward messages to a remote endpoint, e.g. logstash instance or elasticsearch 14
- 15. Logstash DEMO 15
- 16. Elasticsearch Based on lucene for indexing and searching - but lucene is just a library and very complex Provides (simple) Restful - API abstraction on top of lucene Stores documents in json format 16
- 17. Elasticsearch - Scaling Supports vertical (bigger hardware) and horizontal scaling (more hardware) Horizontal scaling is hard, but Elasticsearch is distributed by nature 17
- 18. The empty cluster 18 Node: Is a running instance of elasticsearch Cluster: A cluster consists of one or more nodes with the same cluster name that are working together to share their data and workload
- 19. Index and shards 19 Shard: low-level worker holding a slice of data. A single instance of lucene. Index: logical namespace that points to one or more physical shards
- 20. Replicas / Failover 20 Primary and replica shards: Primary and associated replica shard store the same documents. Newly indexed document first stored on a primary shard, then copied in parallel to the associated replica shard(s).
- 21. Horizontal scaling 21 3 shards spread across 3 from 2 nodes. Each shard is full fledged search engine. Scaling by increasing number of replica shards.
- 22. Cluster Discovery Discovering nodes inside a cluster and electing a master node Zen discovery 22
- 23. Types, Documents, Fields 23 Relational Database Elasticsearch Databases Indices Tables Types Rows Documents Columns Fields
- 24. Storing documents PUT /customer/employee/1 { "first_name" : "John", "last_name" : "Smith", "age" : 25, "about" : "I love to go rock climbing", "interests": [ "sports", "music" ] } 24
- 25. Retrieving document GET /customer/employee/1 Search lite GET /customer/employee/search 25
- 26. Query DSL GET /customer/employee/_search { "query" : { "match" : { "last_name" : "Smith" } } } 26
- 28. Kibana Data Visualization + Data Discovery 28
- 29. Kibana DEMO 29
- 30. What’s missing? Security Alerting 30
- 31. Alternatives? 31
- 32. 32 Go and grok some logs
Editor's Notes
- ELK solves the challenge of bringing all of this together, combine it and make it accessible to the user
- Lucene – Search engine Elasticsearch is built on top of Marvel – Monitoring Elasticsearch cluster Shield – Secure and encrypt your data
- Lucene – Search engine Elasticsearch is built on top of Marvel – Monitoring Elasticsearch cluster Shield – Secure and encrypt your data
- One node in the cluster is elected to be the master node, which is in charge of managing cluster-wide changes like creating or deleting an index, or adding or removing a node from the cluster.
- A shard is a low-level worker unit that holds just a slice of all the data in the index. In Inside a Shard, we explain in detail how a shard works, but for now it is enough to know that a shard is a single instance of Lucene, and is a complete search engine in its own right. Shards are how Elasticsearch distributes data around your cluster. Think of shards as containers for data. Documents are stored in shards, and shards are allocated to nodes in your cluster. A shard can be either a primary shard or a replica shard. To add data to Elasticsearch, we need an index—a place to store related data. In reality, an index is just a logical namespace that points to one or more physical shards.
- 3 nodes share work instead of 2 nodes.
- By default, every field in a document is indexed (has an inverted index) and thus is searchable. A field without an inverted index is not searchable. We discuss inverted indexes in more detail in Inverted Index. An Elasticsearch cluster can contain multiple indices (databases), which in turn contain multiple types(tables). These types hold multiple documents (rows), and each document has multiple fields (columns).
- Shield – Secure and encrypt your data
- Shield – Secure and encrypt your data Watcher – Alerting
- Splunk – similar to ELK AppDynamics – Instrumentation New Relic