Elastic Stack is a suite of open source tools for log analytics and data processing including Beats, Logstash, Elasticsearch, Kibana, Curator, and hosted cloud solutions. Beats are lightweight data shippers that collect data from endpoints and send to Logstash or Elasticsearch. Logstash is used for data collection, transformation, and transport to Elasticsearch for storage and search. Kibana provides data visualization and dashboards. Curator manages Elasticsearch indices. The Elastic Stack can be self-hosted or used via cloud offerings.

1. Elastic Stack
A deeper dive into the stack

2. Elastic Stack
● Beats – Files, Metrics, Audit, Packets
● Logstash – Inputs, Groks, Plugins, Ruby code
● Elasticsearch – clusters, routing, security
● Kibana – dashboards, searches, visualizations
● Curator – index management
● Cloud – hosted solutions

3. Elastic Stack
Cluster design
● Resiliency
● Speed
● Capacity
● Cost

4. Elastic Stack
Cluster example

5. Beats
The Lightweight Shipper
● Agent on each endpoint
● Sends to logstash or direct to elasticsearch
● Data shipper – not just logs
● Build your own beat with libbeat
● Community written beats

6. Beats
Filebeat
● Designed for logfiles
● Modules for Apache, NGINX, System, MySQL, and
more
● Multiline pattern matching
● Adjust volume based on logstash feedback
● Input file glob patterns and harvesters
– Log, stdin, redis, udp, docker, tcp, syslog
● At least once delivery*

7. Beats
WinLogBeat
● Designed for Windows style event logs
● Multiple fields exported
– Docker and Kubernetes metadata
– Error events
– Beat and host fields
● Processor for events before shipment
– Drop events
– Drop or rename fields
– add metadata and dissect strings

8. Beats
Metricbeat
● System monitoring – CPU, Memory, I/O, etc.
● Hosted Docker and Kubernetes containers
● Service modules – Metricset
– Apache, MongoDB, Prometheus, MySQL
– Custom built modules in Go
● No aggregation at collection
● Multiple metrics per event and can include strings

9. Beats
Packetbeat
● Decoders for common protocols
– Http, ICMP, MySQL, Redis, MongoDB, etc
● Dedicated server or on the application server
● Correlates requests and responses into
transactions sent to Logstash/Elasticsearch
● Records interesting fields based on protocol
● Flow statistics including packet and byte counts

10. Beats
Auditbeat and Heartbeat
● Audit beat
– Linux audit framework shipper
– Similar data as Auditd – user/process
– Spools audit data to disk for resiliency
● Heartbeat
– Uptime monitoring of protocols
– Supports TLS, authentication, and proxies
– Dynamic inventory management

11. Logstash
Collect, Enrich, Transport
● Inputs, filters, outputs configured in pipelines
● Community extensible
● Multiple input sources
● Powerful and extensible filters
● Multiple output destinations

12. Logstash
Inputs
● Log/data using beats, log4j, syslog, TCP/UDP
● Metrics/data over TCP/UDP
● Http web hooks, requests, and end point polling
● Datastores with JDBC
● Datastreams with kafka, RabbitMQ, or Amazon
SQS
● Sensor data or custom data

13. Logstash
Filters to transform
● Grok
– Parse and structure arbitrary text
– 120 default patterns (date, IP, word, URIpath)
– %{SYNTAX:semantic} %{IP:client} %
{DATE:timestamp}
– Saved as strings by default
● kv, mutate, geoip, csv, fingeprint
● Ruby, Json, XML, and more plugins and codecs

14. Logstash
Output plugins
● More than just elasticsearch
● Monitoring – nagios, zabbix, AWS cloudwatch
● Database – influxDB, MongoDB, openTSDB
● Notification – pagerduty, email, XMPP, Amazon
SNS, kafka
● Logging – greylog, loggly, syslog, timber.io
● Pipe, file, stdout, syslog, tcp/udp
● Custom ruby code output plugin

15. Logstash
demo - example
● https://grokdebug.herokuapp.com
● https://github.com/agolo/logstash-test-runner
● Debug with stdout {codec: ruby } or json

16. Kibana
Visualizations
● Query and discovery
● Graphs and dashboards
● Metrics
● Reporting
● Open source dashboards

17. Kibana
demo - example
● Cloud hosted sample flight data
● Even more examples:
https://github.com/elastic/examples

18. Other products
● Curator
– Index management
– Reprocessing and routing
– Delete, close, and allocation
● APM (Application Performance Monitoring)
– Language specific agents (node, python, ruby, go, java)
– APM Server to collect performance metrics
● ElasticCloud and Cloud Enterprise
– Hosted and centralized management