Downloaded 27 times
![How logstash works?
• logstash process events, not (only)
loglines!
• “The logstash agent is a processing
pipeline with 3 stages:
– inputs -> filters -> outputs.”
– separate threads
• “Inputs generate events, filters modify
them, outputs ship them elsewhere.”
• -- [the life of an event in logstash]
](https://image.slidesharecdn.com/logstashfamilyintroduction-140710043836-phpapp02/75/Logstash-family-introduction-15-2048.jpg)
Uploaded byOwen Wu
Logstash family introduction
This document provides an introduction to the Logstash log processing tool. It begins by defining what a log is, then describes the theoretical and actual life cycle of a log. It notes common problems with log management. The document introduces Logstash as an open source log processing solution and describes its architecture and components, including inputs, filters, outputs, and how events flow through the system. It provides examples of using Logstash plugins and processing logs.
More Related Content
Similar to Logstash family introduction
Logstash family introduction
- 1.
- 2. What is a log • Oxford Dic4onary – a thick piece of wood that is cut from or has fallen from a tree – (also logbook) an official record of events during a par4cular period of 4me, especially a journey on a ship • 4me + data
- 3. In theory, life cycle of log Record Transmit Analyze Store Delete
- 4. In design, life cycle of log Record Transmit Store Delete
- 5. In fact, life cycle of log Record Delete
- 6. Problems • Logging to a database or filesysytem • Logging has placed a load on the database and filesystem • Mul4ple log formats • No easy way to search logs • No easy method to gather sta4s4cs
- 7. Find the logs of 16 computers 6 months ago ?
- 8. Why use Logstash? • A lot choices! • But we want a free & high-integrality & easy to use solution • splunk (finding your faults, just like mom) • facebookarchive/scribe (2682 ★) • Graylog2(Server+WUI 1683 ★) • fluentd (2038 ★) • logstash (2689 ★)
- 9. logstash and other things hRps://www.youtube.com/watch?v=RuUFnog29M4
- 10. Logstash • Open Source, Apache Licence • WriRen in JRuby, Runs on JVM • Plugins easily wriRen in Ruby • Process mul4ple format ( input, output ) • Logstash Family! ( Elas4cSearch , Kibana)
- 11.
- 12. Elas4cSearch • A response to the claim : “Search is hard” • Powerful indexing & search tool • search & index data available Rescully as JSON over HTTP
- 13.
- 14.
- 15. How logstash works? • logstash process events, not (only) loglines! • “The logstash agent is a processing pipeline with 3 stages: – inputs -> filters -> outputs.” – separate threads • “Inputs generate events, filters modify them, outputs ship them elsewhere.” • -- [the life of an event in logstash]
- 16. In my thinking, Event Life Cycle Input filter output
- 17. In fact, Event Life Cycle event (Input -‐> output) event -‐-‐-‐-‐-‐-‐-‐-‐-‐ input filter output
- 18. Logstash is a wooden tube Input Input Input filter output codec filter filter output output
- 19. Logstash plugins Workflow • inputs – How events get into LogStash. • codecs – convert an incoming format into an internal representa4on • filters – processing ac4ons on events : modify events or drop events • outputs – How output events from LogStash
- 20.
- 21. What is an event!? • A @4mestamp (ISO 8601 4mestamp) • A messsage field ( data ) • A @version • host ( the host of sender) • type( syslog, irc, etc)
- 22. Exersice: Hello Word! java -‐jar logstash-‐1.1.12-‐flatjar.jar agent -‐f hello.conf java -‐jar logstash.jar agent -‐f hello.conf
- 23. Input • tcp • udp • unix • file • syslog • redis • logstash-‐fowarder(former Lumberjack)
- 24. Codecs • plain • json • rubydebug • mul4line
- 25. Outputs • mongodb • elas4cSearch • email • file • jira
- 26. Exercise: Mul4ple input & output
- 27. logstash-‐forwader • ♫ I'm a lumberjack and I'm ok! I sleep when idle, then I ship logs all day! I parse your logs, I eat the JVM agent for lunch! ♫ • WriRen in Go • lumberjack is reserved for protocol • Resource Usage Concerns • Need an SSL CA to verify the server
- 28. lumberjack • Encryp4on & Authen4ca4on (TLS) • Compression ( reduce bandwidth) • Sequence & ack behavior like TCP • Low latency • Reliable Aplica4on-‐Level message transport
- 29.
- 30. Filters • date • grok • drop • geoIP • mutate • mul4line
- 31.
- 32.
- 33. powerful grok • Parse arbitrary text and structure it. • The syntax for a grok paRern is – %{SYNTAX:SEMANTIC} • 55.3.244.1 GET /index.html 15824 – %{IP:client} – %{WORD:method} – %{URIPATHPARAM:request} – %{NUMBER:bytes} • hRps://github.com/elas4csearch/logstash/blob/ v1.4.2/paRerns/grok-‐paRerns
- 34.
- 35.
- 36. mutate • Muta4ons on fields. – rename – remove – replace – join – split – upper – lower
- 37. mul4line • Codecs & filter
- 38. Reference • hRps://www.digitalocean.com/community/ tutorials/how-‐to-‐use-‐logstash-‐and-‐kibana-‐to-‐ centralize-‐and-‐visualize-‐logs-‐on-‐ubuntu-‐14-‐04 • hRp://www.vmdoh.com/blog/centralizing-‐ logs-‐lumberjack-‐logstash-‐and-‐elas4csearch • hRp://jpmens.net/2012/08/09/i-‐grok-‐how-‐to-‐ mutate-‐a-‐file-‐with-‐logstash/ • hRp://gleenders.blogspot.tw/2014/02/ logstash-‐glassfish.html