Logonomics

1,081 views

Published on

Published in: Technology, Business
1 Comment
2 Likes
Statistics
Notes
No Downloads
Views
Total views
1,081
On SlideShare
0
From Embeds
0
Number of Embeds
25
Actions
Shares
0
Downloads
10
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide

Logonomics

  1. 1. LOGONOMICS:  The  Hidden  Side  of  Blackboard  Logs     by  steve  feldman   @PerfForensics  
  2. 2. Logging  Doesn’t  Suck  
  3. 3. It’s  Like  Fishing  in  the  Night…  
  4. 4. So  Why  Don’t  We  Talk  About  Logs   More  OJen?  
  5. 5. At  least  20%  of  all  people  in  this  room   don’t  know  where  to  find  their  logs.  
  6. 6. At  least  50%  of  all  people  in  this  room   don’t  look  at  their  logs.  
  7. 7. At  least  60%  of  all  people  in  this  room   don’t  visualize  their  log  data.  
  8. 8. At  least  75%  of  all  people  in  this  room   don’t  correlate  data  between  logs.  
  9. 9. At  least  90%  of  all  people  in  this  room   don’t  standardize  the  management  of   logs  to  a  centralized  service.  
  10. 10. At  least  95%  of  all  people  in  this  room   don’t  alert  IT  staff  based  on  a  specific   log  event.  
  11. 11. If  a  System  Doesn’t  Output  to  a  Log  Do   We  Assume  Nobody  is  Using  it?  
  12. 12. If  a  System  ConZnuously  Spews  Data   to  a  Log  Do  We  Ignore  it?  
  13. 13. What  We  Can  Do  With  Our  Log  Data   LOGONOMICS:  The  Hidden  Side  of     Blackboard  Logs    
  14. 14. Trending  and  Intelligence     Service  Levels     Threats  and  VulnerabiliZes     Responsiveness    Reliability    
  15. 15. Primer  Data  Points  Everyone  Should   Know   Unique  Requests   Time  Series  of  Requests   ConcentraZon  of  Request  Types   Origin  of  Requests   Quick  Averages   Cascading  Issues  Across  Logs  
  16. 16. Combining  Other  Data  with  Log  Data   CorrelaZon   Root  Cause   InterpretaZon   CompleZon  of  Message   Full  Picture   Sequence  and  Timelines  
  17. 17. Types  of  Data  We  Can  Get  LOGONOMICS:  The  Hidden  Side  of  Blackboard  Logs    
  18. 18. Business  AnalyZcs:  AdopZon  and   Growth  
  19. 19. System  Health  
  20. 20. Capacity  Planning  
  21. 21. Security  and  Threat  Analysis  
  22. 22. Quality  and  Experience:  MeeZng  SLAs  
  23. 23. Replay  and  Benchmarking  
  24. 24. Insight  into  the  BbLogs  LOGONOMICS:  The  Hidden  Side  of  Blackboard  Logs    
  25. 25. Four  Horseman  of  Logs  
  26. 26. Bablefield  of  Other  Logs   •  AuthenZcaZon   •  Plugins  Directory   •  NauZlus  for  events   •  Monitoring  (System  Logs)   – Syslogs  and  Rsyslogs  (/var/messages)   – Windows  Event  Logs  
  27. 27. Is  there  a  Most  Important  Log?  
  28. 28. Access  Log   Log  Formafng  Mabers   Log  Levels     (INFO,  WARN,  ERROR)   mod_log_forensic   Use  %k,  %T  and  %D   Decompose  the  URI   Log  Formafng  Mabers  
  29. 29. Is  there  a  2nd  Most  Important  Log?  
  30. 30. Tomcat  and  Java  Logs   Stack  Traces   Startup  OpZons   GC  Events   GC  Pauses  and  Status  
  31. 31. Tools  We  Should  Consider  LOGONOMICS:  The  Hidden  Side  of  Blackboard  Logs    
  32. 32. It’s  All  About  the  Right  Fishing  Rod  
  33. 33. CAT! GREP! TAIL! SED!AWK! SORT!
  34. 34. GROK!
  35. 35. SomeZmes  a  Net  is  Beber  to  Cast  
  36. 36. Log  CentralizaZon  
  37. 37. Please  Take  All  My  Logs     Format  Lots  of  Log  Data     Send  it  Down  the  River  
  38. 38. •  amqp   •  exec   •  file   •  gelf   •  redis   •  stdin   •  stomp   •  syslog   •  tcp   •  twiber   •  xmpp   •  zeromq   •  amqp   •  elasZcsearch   •  elasZcsearch_ river   •  file   •  ganglia   •  gelf   •  graphite   •  internal   •  loggly   •  mongodb   •  nagios   •  date   •  dns   •  gelfify   •  grep   •  grok   •  grokdisco very   •  json   •  mulZline   •  mutate   •  split   •  null   •  redis   •  statsd   •  stdout   •  stomp   •  tcp   •  websocket   •  xmpp   •  zabbix   •  zeromq   Inputs   Filters   Outputs  
  39. 39. Configure  Apache  for  JSON  log   •  hbp://cookbook.logstash.net/recipes/apache-­‐ json-­‐logs/  
  40. 40. Configure  Tomcat  for  MulZ-­‐Line  Filter  
  41. 41. Setup  Bb  to  feed  logstash  
  42. 42. What  We  Use  Logstash   Log  AggregaZon   Non-­‐FuncZonal   Requirements   Event  NoZficaZon   IntegraZon  with   Zabbix   Kibana  Front-­‐End   Redis  Inputs  &  Outputs   Indexing  
  43. 43. Simple  Challenge  to  All   •  Setup  Logstash  architecture  (All  Single  Node)   •  Start  shipping  basic  log  files   – Apache  2.X  access  log  or  IIS  web  server  log   – Tomcat  Catalina  log  file   •  Output  results  to  statsD  (Etsy  Project)   – Simple  Use  Case:  IncremenZng  HTTP  codes  (200,   300,  400)   •  Visualize  statsD  data  with  Graphite  
  44. 44. Bonus  Challenge  to  All   •  Take  the  Vagrant  VM  and  integrate  Logstash   shipper  with  configuraZon  files.   •  Add  Postgres  support  (Development  Only)   •  Basic  syslog  funcZonality  for  CentOs   •  Custom  Log  Interface  for  a  B2  
  45. 45. Let’s  Add-­‐on  to  the  IniZaZve   developer.blackboard.com    

×