Logstash	
  Family	
  Introduc4on
Owen
What	
  is	
  a	
  log
•  Oxford	
  Dic4onary	
  
– a	
  thick	
  piece	
  of	
  wood	
  that	
  is	
  cut	
  from	
  or	
...
In	
  theory,	
  life	
  cycle	
  of	
  log
Record
Transmit
Analyze
Store
Delete
In	
  design,	
  life	
  cycle	
  of	
  log
Record
Transmit
Store
Delete
In	
  fact,	
  life	
  cycle	
  of	
  log
Record
Delete
Problems
•  Logging	
  to	
  a	
  database	
  or	
  filesysytem	
  
•  Logging	
  has	
  placed	
  a	
  load	
  on	
  the	
...
Find	
  the	
  logs	
  of	
  16	
  computers	
  6	
  
months	
  ago	
  ?
Why	
  use	
  Logstash?
•  A lot choices!	

•  But we want a free & high-integrality &
easy to use solution
•  splunk (fi...
logstash	
  and	
  other	
  things
hRps://www.youtube.com/watch?v=RuUFnog29M4
Logstash
•  Open	
  Source,	
  Apache	
  Licence	
  
•  WriRen	
  in	
  JRuby,	
  Runs	
  on	
  JVM	
  
•  Plugins	
  easi...
LogStash	
  Family	
  architecture
Elas4cSearch
•  A	
  response	
  to	
  the	
  claim	
  :	
  “Search	
  is	
  hard”	
  
•  Powerful	
  indexing	
  &	
  sea...
Kibana
All-­‐in-­‐one!
How logstash works?
•  logstash process events, not (only)
loglines!
•  “The logstash agent is a processing
pipeline with ...
In	
  my	
  thinking,	
  Event	
  Life	
  Cycle
Input
filter	
  	
  
output
In	
  fact,	
  Event	
  Life	
  Cycle
event	
  (Input	
  -­‐>	
  output)
event	
  
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐	
  
input	
...
Logstash	
  is	
  a	
  wooden	
  tube
Input
Input
Input
filter
output
codec
filter
 filter
output
output
Logstash	
  plugins	
  Workflow 	
  
•  inputs	
  
–  How	
  events	
  get	
  into	
  LogStash.	
  
•  codecs	
  	
  
–  	
...
Logstash	
  plugins
What	
  is	
  an	
  event!?
•  A	
  @4mestamp	
  (ISO	
  8601	
  4mestamp)	
  
•  A	
  messsage	
  field	
  (	
  data	
  )	...
Exersice:	
  Hello	
  Word!
java	
  -­‐jar	
  logstash-­‐1.1.12-­‐flatjar.jar	
  agent	
  -­‐f	
  
hello.conf
java	
  -­‐ja...
Input	
  
•  tcp	
  
•  udp	
  
•  unix	
  
•  file	
  
•  syslog	
  
•  redis	
  
•  logstash-­‐fowarder(former	
  Lumberj...
Codecs
•  plain	
  
•  json	
  
•  rubydebug	
  
•  mul4line
Outputs	
  
•  mongodb	
  
•  elas4cSearch	
  
•  email	
  
•  file	
  
•  jira	
  
Exercise:	
  Mul4ple	
  input	
  &	
  output
logstash-­‐forwader
•  ♫	
  I'm	
  a	
  lumberjack	
  and	
  I'm	
  ok!	
  I	
  sleep	
  when	
  
idle,	
  then	
  I	
  sh...
lumberjack
•  Encryp4on	
  &	
  Authen4ca4on	
  (TLS)	
  
•  Compression	
  (	
  reduce	
  bandwidth)	
  
•  Sequence	
  &...
Forwarder	
  Sample
Filters	
  
•  date	
  
•  grok	
  
•  drop	
  
•  geoIP	
  
•  mutate 	
  	
  
•  mul4line	
  
Exercise:	
  Parse	
  Data
filter	
  config	
  
powerful	
  grok
•  Parse	
  arbitrary	
  text	
  and	
  structure	
  it.	
  
•  The	
  syntax	
  for	
  a	
  grok	
  paRe...
grok	
  sample 	
  
drop
mutate	
  
•  Muta4ons	
  on	
  
fields.	
  	
  
– 	
  rename	
  
– 	
  remove	
  
– replace	
  
– join	
  
– split	
  
– u...
mul4line
•  Codecs	
  &	
  filter
Reference
•  hRps://www.digitalocean.com/community/
tutorials/how-­‐to-­‐use-­‐logstash-­‐and-­‐kibana-­‐to-­‐
centralize-...
Upcoming SlideShare
Loading in …5
×

Logstash family introduction

503
-1

Published on

Logstash Introduction

Published in: Engineering, Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
503
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
19
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Logstash family introduction

  1. 1. Logstash  Family  Introduc4on Owen
  2. 2. What  is  a  log •  Oxford  Dic4onary   – a  thick  piece  of  wood  that  is  cut  from  or  has  fallen   from  a  tree   – (also  logbook)  an  official  record  of  events  during  a   par4cular  period  of  4me,  especially  a  journey  on  a   ship     •  4me  +  data        
  3. 3. In  theory,  life  cycle  of  log Record Transmit Analyze Store Delete
  4. 4. In  design,  life  cycle  of  log Record Transmit Store Delete
  5. 5. In  fact,  life  cycle  of  log Record Delete
  6. 6. Problems •  Logging  to  a  database  or  filesysytem   •  Logging  has  placed  a  load  on  the  database  and   filesystem   •  Mul4ple  log    formats   •  No  easy  way  to  search  logs   •  No  easy  method  to  gather  sta4s4cs  
  7. 7. Find  the  logs  of  16  computers  6   months  ago  ?
  8. 8. Why  use  Logstash? •  A lot choices!  •  But we want a free & high-integrality & easy to use solution •  splunk (finding your faults, just like mom) •  facebookarchive/scribe (2682 ★) •  Graylog2(Server+WUI 1683 ★) •  fluentd (2038 ★) •  logstash (2689 ★)
  9. 9. logstash  and  other  things hRps://www.youtube.com/watch?v=RuUFnog29M4
  10. 10. Logstash •  Open  Source,  Apache  Licence   •  WriRen  in  JRuby,  Runs  on  JVM   •  Plugins  easily  wriRen  in  Ruby   •  Process  mul4ple  format  (  input,  output  )   •  Logstash  Family!  (  Elas4cSearch  ,  Kibana)  
  11. 11. LogStash  Family  architecture
  12. 12. Elas4cSearch •  A  response  to  the  claim  :  “Search  is  hard”   •  Powerful  indexing  &  search  tool   •  search  &  index  data  available  Rescully  as   JSON  over  HTTP  
  13. 13. Kibana
  14. 14. All-­‐in-­‐one!
  15. 15. How logstash works? •  logstash process events, not (only) loglines! •  “The logstash agent is a processing pipeline with 3 stages: – inputs -> filters -> outputs.” – separate threads •  “Inputs generate events, filters modify them, outputs ship them elsewhere.” •  -- [the life of an event in logstash] 
  16. 16. In  my  thinking,  Event  Life  Cycle Input filter     output
  17. 17. In  fact,  Event  Life  Cycle event  (Input  -­‐>  output) event   -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐   input   filter     output
  18. 18. Logstash  is  a  wooden  tube Input Input Input filter output codec filter filter output output
  19. 19. Logstash  plugins  Workflow   •  inputs   –  How  events  get  into  LogStash.   •  codecs     –   convert  an  incoming  format  into  an  internal     representa4on   •  filters   –   processing  ac4ons  on  events  :  modify  events  or  drop   events   •  outputs   –  How  output  events  from  LogStash
  20. 20. Logstash  plugins
  21. 21. What  is  an  event!? •  A  @4mestamp  (ISO  8601  4mestamp)   •  A  messsage  field  (  data  )   •  A  @version   •  host  (  the  host  of  sender)   •  type(  syslog,  irc,  etc)  
  22. 22. Exersice:  Hello  Word! java  -­‐jar  logstash-­‐1.1.12-­‐flatjar.jar  agent  -­‐f   hello.conf java  -­‐jar  logstash.jar  agent  -­‐f  hello.conf
  23. 23. Input   •  tcp   •  udp   •  unix   •  file   •  syslog   •  redis   •  logstash-­‐fowarder(former  Lumberjack)
  24. 24. Codecs •  plain   •  json   •  rubydebug   •  mul4line
  25. 25. Outputs   •  mongodb   •  elas4cSearch   •  email   •  file   •  jira  
  26. 26. Exercise:  Mul4ple  input  &  output
  27. 27. logstash-­‐forwader •  ♫  I'm  a  lumberjack  and  I'm  ok!  I  sleep  when   idle,  then  I  ship  logs  all  day!  I  parse  your  logs,  I   eat  the  JVM  agent  for  lunch!  ♫   •  WriRen  in  Go   •  lumberjack  is  reserved  for  protocol   •  Resource  Usage  Concerns   •  Need  an  SSL  CA  to  verify  the  server  
  28. 28. lumberjack •  Encryp4on  &  Authen4ca4on  (TLS)   •  Compression  (  reduce  bandwidth)   •  Sequence  &  ack  behavior  like  TCP   •  Low  latency   •  Reliable  Aplica4on-­‐Level  message  transport
  29. 29. Forwarder  Sample
  30. 30. Filters   •  date   •  grok   •  drop   •  geoIP   •  mutate     •  mul4line  
  31. 31. Exercise:  Parse  Data
  32. 32. filter  config  
  33. 33. powerful  grok •  Parse  arbitrary  text  and  structure  it.   •  The  syntax  for  a  grok  paRern  is       –  %{SYNTAX:SEMANTIC}   •  55.3.244.1        GET          /index.html          15824   –  %{IP:client}     –  %{WORD:method}     –  %{URIPATHPARAM:request}     –  %{NUMBER:bytes}   •  hRps://github.com/elas4csearch/logstash/blob/ v1.4.2/paRerns/grok-­‐paRerns  
  34. 34. grok  sample  
  35. 35. drop
  36. 36. mutate   •  Muta4ons  on   fields.     –   rename   –   remove   – replace   – join   – split   – upper   – lower
  37. 37. mul4line •  Codecs  &  filter
  38. 38. Reference •  hRps://www.digitalocean.com/community/ tutorials/how-­‐to-­‐use-­‐logstash-­‐and-­‐kibana-­‐to-­‐ centralize-­‐and-­‐visualize-­‐logs-­‐on-­‐ubuntu-­‐14-­‐04   •  hRp://www.vmdoh.com/blog/centralizing-­‐ logs-­‐lumberjack-­‐logstash-­‐and-­‐elas4csearch   •  hRp://jpmens.net/2012/08/09/i-­‐grok-­‐how-­‐to-­‐ mutate-­‐a-­‐file-­‐with-­‐logstash/   •  hRp://gleenders.blogspot.tw/2014/02/ logstash-­‐glassfish.html  
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×