'Scalable Logging and Analytics with LogStash'

9,330 views

Published on

Rich Viet, Principal Engineer at Cloud Elements presents 'Scalable Logging and Analytics with LogStash' at All Things API meetup in Denver, CO.

Learn more about scalable logging and analytics using LogStash. This will be an overview of logstash components, including getting started, indexing, storing and getting information from logs.

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching).

Published in: Technology

'Scalable Logging and Analytics with LogStash'

  1. 1. Log Scaling and Analytics With Logstash Richard Viet Principal Engineer Cloud Elements
  2. 2. Problems  Logging to a database or filesystem  Logging has placed a load on the database and filesystem.  Multiple log formats  No easy way to search logs  No easy method to gather statistics
  3. 3. Logstash  Open source, Apache licence  Written in JRuby. Runs on jvm.  Plugins easily written in Ruby.  Part of the Elasticsearch family.  www.logstash.net
  4. 4. Logstash  Scalable: Elasticsearch for indexing, search and retrieval  Process multiple log formats  Receive logs from multiple sources  Output logs to multiple destinations  Kibana provides web interface for search and analytics  Easily extended with plugins written in Ruby
  5. 5. Logstash Architecture Shipper Broker Indexer Search Storage Shipper Shipper Web Interface
  6. 6. Logstash Pipeline  Input → filters → output  Separate threads  Filters are applied in order of config file  Outputs processed in order of config file
  7. 7. Logstash Plugins  Input – read input stream – File input – Log4j – Redis – Syslog  Codecs – decoding log messages – Json – Multiline
  8. 8. Logstash Plugins  Filters – processing messages – Csv – define fields in a csv – Date – define date field formats – Mutate – change date type – Xml – extract xml – Grok – parses arbitrary text
  9. 9. Logstash Plugins  Output – Elasticsearch – Elasticsearch_http – Mongodb – Email – Nagios
  10. 10. Indexer  Send message to Elasticsearch for indexing  An index is created for each day  Each index split into 5 shards by default  Original message is stored  Each field indexed
  11. 11. Elasticsearch  Apache Lucene search engine  An elasticsearch index is made up of multiple shards  Each shard is a lucene index  Primary shard and at least one replica  Shards are moved between servers when servers are added or removed
  12. 12. Elasticsearch Configuration  Self discovery – Multicast • Simplest if all nodes on same network – Unicast • Provide a list of servers – Combination
  13. 13. Elasticsearch  Adding more nodes improves indexing and search time.  Primary node is indexed first then replicas  Number of shards determined when index is created.  Number of replicas is configurable
  14. 14. Elasticsearch  Adding more nodes improves indexing and search time.  Primary node is indexed first then replicas  Number of shards determined when index is created.  Number of replicas is configurable
  15. 15. Kibana  Browser based analytics for time-stamped data  Included in the logstash jar  Connect to the logstash server port 9292  Sends multiple requests to avoid overloading the server.
  16. 16. Log4j to Logstash App Logstash Redis Elasticsearch Cluster App App Logstash
  17. 17. Logstash Log4j Server  Configure logstash as a Log4j server input { log4j { mode => "server" port => 9501 } }
  18. 18. Send to a broker  Configure broker output { stdout {} redis { host => "redis1" data_type => "list" key => "logstash" } }
  19. 19. Indexing input { redis { host => “redis” data_type => “list” key => “logstash” }} output { elasticsearch { cluster => “logstash” host => "elasticsearch" port => "9200" }}
  20. 20. Scaling Broker Indexer Search Storage Shipper Web Interface Broker Indexer
  21. 21. Sending to Broker output { stdout {} redis { host => ["redis1", “redis2”] data_type => "list" shuffle_hosts => true key => "logstash" } }
  22. 22. Indexing input { redis { host => “redis1” data_type => “list” key => “logstash” redis { host => “redis2” data_type => “list” key => “logstash” } } output { ...
  23. 23. Quick Start  Logstash, elasticsearch and kibana configured to run from the logstash jar  Download and untar  bin/logstash agent -f config.file  bin/logstash web

×