LOGONOMICS:	
  The	
  Hidden	
  Side	
  of	
  Blackboard	
  Logs	
  	
  
by	
  steve	
  feldman	
  
@PerfForensics	
  
Logging	
  Doesn’t	
  Suck	
  
It’s	
  Like	
  Fishing	
  in	
  the	
  Night…	
  
So	
  Why	
  Don’t	
  We	
  Talk	
  About	
  Logs	
  
More	
  OJen?	
  
At	
  least	
  20%	
  of	
  all	
  people	
  in	
  this	
  room	
  
don’t	
  know	
  where	
  to	
  find	
  their	
  logs.	...
At	
  least	
  50%	
  of	
  all	
  people	
  in	
  this	
  room	
  
don’t	
  look	
  at	
  their	
  logs.	
  
At	
  least	
  60%	
  of	
  all	
  people	
  in	
  this	
  room	
  
don’t	
  visualize	
  their	
  log	
  data.	
  
At	
  least	
  75%	
  of	
  all	
  people	
  in	
  this	
  room	
  
don’t	
  correlate	
  data	
  between	
  logs.	
  
At	
  least	
  90%	
  of	
  all	
  people	
  in	
  this	
  room	
  
don’t	
  standardize	
  the	
  management	
  of	
  
lo...
At	
  least	
  95%	
  of	
  all	
  people	
  in	
  this	
  room	
  
don’t	
  alert	
  IT	
  staff	
  based	
  on	
  a	
  sp...
If	
  a	
  System	
  Doesn’t	
  Output	
  to	
  a	
  Log	
  Do	
  
We	
  Assume	
  Nobody	
  is	
  Using	
  it?	
  
If	
  a	
  System	
  ConZnuously	
  Spews	
  Data	
  
to	
  a	
  Log	
  Do	
  We	
  Ignore	
  it?	
  
What	
  We	
  Can	
  Do	
  With	
  Our	
  Log	
  Data	
  
LOGONOMICS:	
  The	
  Hidden	
  Side	
  of	
  	
  
Blackboard	
 ...
Trending	
  and	
  Intelligence	
  
	
  
Service	
  Levels	
  
	
  
Threats	
  and	
  VulnerabiliZes	
  
	
  
Responsivene...
Primer	
  Data	
  Points	
  Everyone	
  Should	
  
Know	
  
Unique	
  Requests	
  
Time	
  Series	
  of	
  Requests	
  
Co...
Combining	
  Other	
  Data	
  with	
  Log	
  Data	
  
CorrelaZon	
  
Root	
  Cause	
  
InterpretaZon	
  
CompleZon	
  of	
...
Types	
  of	
  Data	
  We	
  Can	
  Get	
  LOGONOMICS:	
  The	
  Hidden	
  Side	
  of	
  Blackboard	
  Logs	
  	
  
Business	
  AnalyZcs:	
  AdopZon	
  and	
  
Growth	
  
System	
  Health	
  
Capacity	
  Planning	
  
Security	
  and	
  Threat	
  Analysis	
  
Quality	
  and	
  Experience:	
  MeeZng	
  SLAs	
  
Replay	
  and	
  Benchmarking	
  
Insight	
  into	
  the	
  BbLogs	
  LOGONOMICS:	
  The	
  Hidden	
  Side	
  of	
  Blackboard	
  Logs	
  	
  
Four	
  Horseman	
  of	
  Logs	
  
Bablefield	
  of	
  Other	
  Logs	
  
•  AuthenZcaZon	
  
•  Plugins	
  Directory	
  
•  NauZlus	
  for	
  events	
  
•  Mo...
Is	
  there	
  a	
  Most	
  Important	
  Log?	
  
Access	
  Log	
  
Log	
  Formafng	
  Mabers	
  
Log	
  Levels	
  	
  
(INFO,	
  WARN,	
  ERROR)	
  
mod_log_forensic	
  
U...
Is	
  there	
  a	
  2nd	
  Most	
  Important	
  Log?	
  
Tomcat	
  and	
  Java	
  Logs	
  
Stack	
  Traces	
  
Startup	
  OpZons	
  
GC	
  Events	
  
GC	
  Pauses	
  and	
  Status...
Tools	
  We	
  Should	
  Consider	
  LOGONOMICS:	
  The	
  Hidden	
  Side	
  of	
  Blackboard	
  Logs	
  	
  
It’s	
  All	
  About	
  the	
  Right	
  Fishing	
  Rod	
  
CAT!
GREP!
TAIL!
SED!AWK!
SORT!
GROK!
SomeZmes	
  a	
  Net	
  is	
  Beber	
  to	
  Cast	
  
Log	
  CentralizaZon	
  
Please	
  Take	
  All	
  My	
  Logs	
  
	
  
Format	
  Lots	
  of	
  Log	
  Data	
  
	
  
Send	
  it	
  Down	
  the	
  Riv...
•  amqp	
  
•  exec	
  
•  file	
  
•  gelf	
  
•  redis	
  
•  stdin	
  
•  stomp	
  
•  syslog	
  
•  tcp	
  
•  twiber	
...
Configure	
  Apache	
  for	
  JSON	
  log	
  
•  hbp://cookbook.logstash.net/recipes/apache-­‐
json-­‐logs/	
  
Configure	
  Tomcat	
  for	
  MulZ-­‐Line	
  Filter	
  
Setup	
  Bb	
  to	
  feed	
  logstash	
  
What	
  We	
  Use	
  Logstash	
  
Log	
  AggregaZon	
  
Non-­‐FuncZonal	
  
Requirements	
  
Event	
  NoZficaZon	
  
Integr...
Simple	
  Challenge	
  to	
  All	
  
•  Setup	
  Logstash	
  architecture	
  (All	
  Single	
  Node)	
  
•  Start	
  shipp...
Bonus	
  Challenge	
  to	
  All	
  
•  Take	
  the	
  Vagrant	
  VM	
  and	
  integrate	
  Logstash	
  
shipper	
  with	
 ...
Let’s	
  Add-­‐on	
  to	
  the	
  IniZaZve	
  
developer.blackboard.com	
  	
  
Logonomics
Logonomics
Logonomics
Logonomics
Logonomics
Logonomics
Logonomics
Upcoming SlideShare
Loading in...5
×

Logonomics

843

Published on

Published in: Technology, Business
1 Comment
2 Likes
Statistics
Notes
No Downloads
Views
Total Views
843
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide

Logonomics

  1. 1. LOGONOMICS:  The  Hidden  Side  of  Blackboard  Logs     by  steve  feldman   @PerfForensics  
  2. 2. Logging  Doesn’t  Suck  
  3. 3. It’s  Like  Fishing  in  the  Night…  
  4. 4. So  Why  Don’t  We  Talk  About  Logs   More  OJen?  
  5. 5. At  least  20%  of  all  people  in  this  room   don’t  know  where  to  find  their  logs.  
  6. 6. At  least  50%  of  all  people  in  this  room   don’t  look  at  their  logs.  
  7. 7. At  least  60%  of  all  people  in  this  room   don’t  visualize  their  log  data.  
  8. 8. At  least  75%  of  all  people  in  this  room   don’t  correlate  data  between  logs.  
  9. 9. At  least  90%  of  all  people  in  this  room   don’t  standardize  the  management  of   logs  to  a  centralized  service.  
  10. 10. At  least  95%  of  all  people  in  this  room   don’t  alert  IT  staff  based  on  a  specific   log  event.  
  11. 11. If  a  System  Doesn’t  Output  to  a  Log  Do   We  Assume  Nobody  is  Using  it?  
  12. 12. If  a  System  ConZnuously  Spews  Data   to  a  Log  Do  We  Ignore  it?  
  13. 13. What  We  Can  Do  With  Our  Log  Data   LOGONOMICS:  The  Hidden  Side  of     Blackboard  Logs    
  14. 14. Trending  and  Intelligence     Service  Levels     Threats  and  VulnerabiliZes     Responsiveness    Reliability    
  15. 15. Primer  Data  Points  Everyone  Should   Know   Unique  Requests   Time  Series  of  Requests   ConcentraZon  of  Request  Types   Origin  of  Requests   Quick  Averages   Cascading  Issues  Across  Logs  
  16. 16. Combining  Other  Data  with  Log  Data   CorrelaZon   Root  Cause   InterpretaZon   CompleZon  of  Message   Full  Picture   Sequence  and  Timelines  
  17. 17. Types  of  Data  We  Can  Get  LOGONOMICS:  The  Hidden  Side  of  Blackboard  Logs    
  18. 18. Business  AnalyZcs:  AdopZon  and   Growth  
  19. 19. System  Health  
  20. 20. Capacity  Planning  
  21. 21. Security  and  Threat  Analysis  
  22. 22. Quality  and  Experience:  MeeZng  SLAs  
  23. 23. Replay  and  Benchmarking  
  24. 24. Insight  into  the  BbLogs  LOGONOMICS:  The  Hidden  Side  of  Blackboard  Logs    
  25. 25. Four  Horseman  of  Logs  
  26. 26. Bablefield  of  Other  Logs   •  AuthenZcaZon   •  Plugins  Directory   •  NauZlus  for  events   •  Monitoring  (System  Logs)   – Syslogs  and  Rsyslogs  (/var/messages)   – Windows  Event  Logs  
  27. 27. Is  there  a  Most  Important  Log?  
  28. 28. Access  Log   Log  Formafng  Mabers   Log  Levels     (INFO,  WARN,  ERROR)   mod_log_forensic   Use  %k,  %T  and  %D   Decompose  the  URI   Log  Formafng  Mabers  
  29. 29. Is  there  a  2nd  Most  Important  Log?  
  30. 30. Tomcat  and  Java  Logs   Stack  Traces   Startup  OpZons   GC  Events   GC  Pauses  and  Status  
  31. 31. Tools  We  Should  Consider  LOGONOMICS:  The  Hidden  Side  of  Blackboard  Logs    
  32. 32. It’s  All  About  the  Right  Fishing  Rod  
  33. 33. CAT! GREP! TAIL! SED!AWK! SORT!
  34. 34. GROK!
  35. 35. SomeZmes  a  Net  is  Beber  to  Cast  
  36. 36. Log  CentralizaZon  
  37. 37. Please  Take  All  My  Logs     Format  Lots  of  Log  Data     Send  it  Down  the  River  
  38. 38. •  amqp   •  exec   •  file   •  gelf   •  redis   •  stdin   •  stomp   •  syslog   •  tcp   •  twiber   •  xmpp   •  zeromq   •  amqp   •  elasZcsearch   •  elasZcsearch_ river   •  file   •  ganglia   •  gelf   •  graphite   •  internal   •  loggly   •  mongodb   •  nagios   •  date   •  dns   •  gelfify   •  grep   •  grok   •  grokdisco very   •  json   •  mulZline   •  mutate   •  split   •  null   •  redis   •  statsd   •  stdout   •  stomp   •  tcp   •  websocket   •  xmpp   •  zabbix   •  zeromq   Inputs   Filters   Outputs  
  39. 39. Configure  Apache  for  JSON  log   •  hbp://cookbook.logstash.net/recipes/apache-­‐ json-­‐logs/  
  40. 40. Configure  Tomcat  for  MulZ-­‐Line  Filter  
  41. 41. Setup  Bb  to  feed  logstash  
  42. 42. What  We  Use  Logstash   Log  AggregaZon   Non-­‐FuncZonal   Requirements   Event  NoZficaZon   IntegraZon  with   Zabbix   Kibana  Front-­‐End   Redis  Inputs  &  Outputs   Indexing  
  43. 43. Simple  Challenge  to  All   •  Setup  Logstash  architecture  (All  Single  Node)   •  Start  shipping  basic  log  files   – Apache  2.X  access  log  or  IIS  web  server  log   – Tomcat  Catalina  log  file   •  Output  results  to  statsD  (Etsy  Project)   – Simple  Use  Case:  IncremenZng  HTTP  codes  (200,   300,  400)   •  Visualize  statsD  data  with  Graphite  
  44. 44. Bonus  Challenge  to  All   •  Take  the  Vagrant  VM  and  integrate  Logstash   shipper  with  configuraZon  files.   •  Add  Postgres  support  (Development  Only)   •  Basic  syslog  funcZonality  for  CentOs   •  Custom  Log  Interface  for  a  B2  
  45. 45. Let’s  Add-­‐on  to  the  IniZaZve   developer.blackboard.com    
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×