Your SlideShare is downloading. ×
Logstash family introduction
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Logstash family introduction

243

Published on

Logstash Introduction

Logstash Introduction

Published in: Engineering, Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
243
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Logstash  Family  Introduc4on Owen
  • 2. What  is  a  log •  Oxford  Dic4onary   – a  thick  piece  of  wood  that  is  cut  from  or  has  fallen   from  a  tree   – (also  logbook)  an  official  record  of  events  during  a   par4cular  period  of  4me,  especially  a  journey  on  a   ship     •  4me  +  data        
  • 3. In  theory,  life  cycle  of  log Record Transmit Analyze Store Delete
  • 4. In  design,  life  cycle  of  log Record Transmit Store Delete
  • 5. In  fact,  life  cycle  of  log Record Delete
  • 6. Problems •  Logging  to  a  database  or  filesysytem   •  Logging  has  placed  a  load  on  the  database  and   filesystem   •  Mul4ple  log    formats   •  No  easy  way  to  search  logs   •  No  easy  method  to  gather  sta4s4cs  
  • 7. Find  the  logs  of  16  computers  6   months  ago  ?
  • 8. Why  use  Logstash? •  A lot choices!  •  But we want a free & high-integrality & easy to use solution •  splunk (finding your faults, just like mom) •  facebookarchive/scribe (2682 ★) •  Graylog2(Server+WUI 1683 ★) •  fluentd (2038 ★) •  logstash (2689 ★)
  • 9. logstash  and  other  things hRps://www.youtube.com/watch?v=RuUFnog29M4
  • 10. Logstash •  Open  Source,  Apache  Licence   •  WriRen  in  JRuby,  Runs  on  JVM   •  Plugins  easily  wriRen  in  Ruby   •  Process  mul4ple  format  (  input,  output  )   •  Logstash  Family!  (  Elas4cSearch  ,  Kibana)  
  • 11. LogStash  Family  architecture
  • 12. Elas4cSearch •  A  response  to  the  claim  :  “Search  is  hard”   •  Powerful  indexing  &  search  tool   •  search  &  index  data  available  Rescully  as   JSON  over  HTTP  
  • 13. Kibana
  • 14. All-­‐in-­‐one!
  • 15. How logstash works? •  logstash process events, not (only) loglines! •  “The logstash agent is a processing pipeline with 3 stages: – inputs -> filters -> outputs.” – separate threads •  “Inputs generate events, filters modify them, outputs ship them elsewhere.” •  -- [the life of an event in logstash] 
  • 16. In  my  thinking,  Event  Life  Cycle Input filter     output
  • 17. In  fact,  Event  Life  Cycle event  (Input  -­‐>  output) event   -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐   input   filter     output
  • 18. Logstash  is  a  wooden  tube Input Input Input filter output codec filter filter output output
  • 19. Logstash  plugins  Workflow   •  inputs   –  How  events  get  into  LogStash.   •  codecs     –   convert  an  incoming  format  into  an  internal     representa4on   •  filters   –   processing  ac4ons  on  events  :  modify  events  or  drop   events   •  outputs   –  How  output  events  from  LogStash
  • 20. Logstash  plugins
  • 21. What  is  an  event!? •  A  @4mestamp  (ISO  8601  4mestamp)   •  A  messsage  field  (  data  )   •  A  @version   •  host  (  the  host  of  sender)   •  type(  syslog,  irc,  etc)  
  • 22. Exersice:  Hello  Word! java  -­‐jar  logstash-­‐1.1.12-­‐flatjar.jar  agent  -­‐f   hello.conf java  -­‐jar  logstash.jar  agent  -­‐f  hello.conf
  • 23. Input   •  tcp   •  udp   •  unix   •  file   •  syslog   •  redis   •  logstash-­‐fowarder(former  Lumberjack)
  • 24. Codecs •  plain   •  json   •  rubydebug   •  mul4line
  • 25. Outputs   •  mongodb   •  elas4cSearch   •  email   •  file   •  jira  
  • 26. Exercise:  Mul4ple  input  &  output
  • 27. logstash-­‐forwader •  ♫  I'm  a  lumberjack  and  I'm  ok!  I  sleep  when   idle,  then  I  ship  logs  all  day!  I  parse  your  logs,  I   eat  the  JVM  agent  for  lunch!  ♫   •  WriRen  in  Go   •  lumberjack  is  reserved  for  protocol   •  Resource  Usage  Concerns   •  Need  an  SSL  CA  to  verify  the  server  
  • 28. lumberjack •  Encryp4on  &  Authen4ca4on  (TLS)   •  Compression  (  reduce  bandwidth)   •  Sequence  &  ack  behavior  like  TCP   •  Low  latency   •  Reliable  Aplica4on-­‐Level  message  transport
  • 29. Forwarder  Sample
  • 30. Filters   •  date   •  grok   •  drop   •  geoIP   •  mutate     •  mul4line  
  • 31. Exercise:  Parse  Data
  • 32. filter  config  
  • 33. powerful  grok •  Parse  arbitrary  text  and  structure  it.   •  The  syntax  for  a  grok  paRern  is       –  %{SYNTAX:SEMANTIC}   •  55.3.244.1        GET          /index.html          15824   –  %{IP:client}     –  %{WORD:method}     –  %{URIPATHPARAM:request}     –  %{NUMBER:bytes}   •  hRps://github.com/elas4csearch/logstash/blob/ v1.4.2/paRerns/grok-­‐paRerns  
  • 34. grok  sample  
  • 35. drop
  • 36. mutate   •  Muta4ons  on   fields.     –   rename   –   remove   – replace   – join   – split   – upper   – lower
  • 37. mul4line •  Codecs  &  filter
  • 38. Reference •  hRps://www.digitalocean.com/community/ tutorials/how-­‐to-­‐use-­‐logstash-­‐and-­‐kibana-­‐to-­‐ centralize-­‐and-­‐visualize-­‐logs-­‐on-­‐ubuntu-­‐14-­‐04   •  hRp://www.vmdoh.com/blog/centralizing-­‐ logs-­‐lumberjack-­‐logstash-­‐and-­‐elas4csearch   •  hRp://jpmens.net/2012/08/09/i-­‐grok-­‐how-­‐to-­‐ mutate-­‐a-­‐file-­‐with-­‐logstash/   •  hRp://gleenders.blogspot.tw/2014/02/ logstash-­‐glassfish.html  

×