Logonomics
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Steve, where can the Vagrant VM be downloaded from?
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
1,073
On Slideshare
1,042
From Embeds
31
Number of Embeds
1

Actions

Shares
Downloads
7
Comments
1
Likes
2

Embeds 31

https://twitter.com 31

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. LOGONOMICS:  The  Hidden  Side  of  Blackboard  Logs     by  steve  feldman   @PerfForensics  
  • 2. Logging  Doesn’t  Suck  
  • 3. It’s  Like  Fishing  in  the  Night…  
  • 4. So  Why  Don’t  We  Talk  About  Logs   More  OJen?  
  • 5. At  least  20%  of  all  people  in  this  room   don’t  know  where  to  find  their  logs.  
  • 6. At  least  50%  of  all  people  in  this  room   don’t  look  at  their  logs.  
  • 7. At  least  60%  of  all  people  in  this  room   don’t  visualize  their  log  data.  
  • 8. At  least  75%  of  all  people  in  this  room   don’t  correlate  data  between  logs.  
  • 9. At  least  90%  of  all  people  in  this  room   don’t  standardize  the  management  of   logs  to  a  centralized  service.  
  • 10. At  least  95%  of  all  people  in  this  room   don’t  alert  IT  staff  based  on  a  specific   log  event.  
  • 11. If  a  System  Doesn’t  Output  to  a  Log  Do   We  Assume  Nobody  is  Using  it?  
  • 12. If  a  System  ConZnuously  Spews  Data   to  a  Log  Do  We  Ignore  it?  
  • 13. What  We  Can  Do  With  Our  Log  Data   LOGONOMICS:  The  Hidden  Side  of     Blackboard  Logs    
  • 14. Trending  and  Intelligence     Service  Levels     Threats  and  VulnerabiliZes     Responsiveness    Reliability    
  • 15. Primer  Data  Points  Everyone  Should   Know   Unique  Requests   Time  Series  of  Requests   ConcentraZon  of  Request  Types   Origin  of  Requests   Quick  Averages   Cascading  Issues  Across  Logs  
  • 16. Combining  Other  Data  with  Log  Data   CorrelaZon   Root  Cause   InterpretaZon   CompleZon  of  Message   Full  Picture   Sequence  and  Timelines  
  • 17. Types  of  Data  We  Can  Get  LOGONOMICS:  The  Hidden  Side  of  Blackboard  Logs    
  • 18. Business  AnalyZcs:  AdopZon  and   Growth  
  • 19. System  Health  
  • 20. Capacity  Planning  
  • 21. Security  and  Threat  Analysis  
  • 22. Quality  and  Experience:  MeeZng  SLAs  
  • 23. Replay  and  Benchmarking  
  • 24. Insight  into  the  BbLogs  LOGONOMICS:  The  Hidden  Side  of  Blackboard  Logs    
  • 25. Four  Horseman  of  Logs  
  • 26. Bablefield  of  Other  Logs   •  AuthenZcaZon   •  Plugins  Directory   •  NauZlus  for  events   •  Monitoring  (System  Logs)   – Syslogs  and  Rsyslogs  (/var/messages)   – Windows  Event  Logs  
  • 27. Is  there  a  Most  Important  Log?  
  • 28. Access  Log   Log  Formafng  Mabers   Log  Levels     (INFO,  WARN,  ERROR)   mod_log_forensic   Use  %k,  %T  and  %D   Decompose  the  URI   Log  Formafng  Mabers  
  • 29. Is  there  a  2nd  Most  Important  Log?  
  • 30. Tomcat  and  Java  Logs   Stack  Traces   Startup  OpZons   GC  Events   GC  Pauses  and  Status  
  • 31. Tools  We  Should  Consider  LOGONOMICS:  The  Hidden  Side  of  Blackboard  Logs    
  • 32. It’s  All  About  the  Right  Fishing  Rod  
  • 33. CAT! GREP! TAIL! SED!AWK! SORT!
  • 34. GROK!
  • 35. SomeZmes  a  Net  is  Beber  to  Cast  
  • 36. Log  CentralizaZon  
  • 37. Please  Take  All  My  Logs     Format  Lots  of  Log  Data     Send  it  Down  the  River  
  • 38. •  amqp   •  exec   •  file   •  gelf   •  redis   •  stdin   •  stomp   •  syslog   •  tcp   •  twiber   •  xmpp   •  zeromq   •  amqp   •  elasZcsearch   •  elasZcsearch_ river   •  file   •  ganglia   •  gelf   •  graphite   •  internal   •  loggly   •  mongodb   •  nagios   •  date   •  dns   •  gelfify   •  grep   •  grok   •  grokdisco very   •  json   •  mulZline   •  mutate   •  split   •  null   •  redis   •  statsd   •  stdout   •  stomp   •  tcp   •  websocket   •  xmpp   •  zabbix   •  zeromq   Inputs   Filters   Outputs  
  • 39. Configure  Apache  for  JSON  log   •  hbp://cookbook.logstash.net/recipes/apache-­‐ json-­‐logs/  
  • 40. Configure  Tomcat  for  MulZ-­‐Line  Filter  
  • 41. Setup  Bb  to  feed  logstash  
  • 42. What  We  Use  Logstash   Log  AggregaZon   Non-­‐FuncZonal   Requirements   Event  NoZficaZon   IntegraZon  with   Zabbix   Kibana  Front-­‐End   Redis  Inputs  &  Outputs   Indexing  
  • 43. Simple  Challenge  to  All   •  Setup  Logstash  architecture  (All  Single  Node)   •  Start  shipping  basic  log  files   – Apache  2.X  access  log  or  IIS  web  server  log   – Tomcat  Catalina  log  file   •  Output  results  to  statsD  (Etsy  Project)   – Simple  Use  Case:  IncremenZng  HTTP  codes  (200,   300,  400)   •  Visualize  statsD  data  with  Graphite  
  • 44. Bonus  Challenge  to  All   •  Take  the  Vagrant  VM  and  integrate  Logstash   shipper  with  configuraZon  files.   •  Add  Postgres  support  (Development  Only)   •  Basic  syslog  funcZonality  for  CentOs   •  Custom  Log  Interface  for  a  B2  
  • 45. Let’s  Add-­‐on  to  the  IniZaZve   developer.blackboard.com