Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Application Logging Good Bad Ugly ... Beautiful?


Published on

Application Logging Good Bad Ugly ... Beautiful? The presentation for application developers on how to log - and how not to!

Published in: Technology, Business

Application Logging Good Bad Ugly ... Beautiful?

  1. 1. Application Logging “Worst Practices” Dr Anton Chuvakin, GCIA, GCIH, GCFA Chief Logging Evangelist LogLogic, Inc
  2. 2. Outline <ul><li>Why logging? Why NOT logging? </li></ul><ul><li>Developers and logging </li></ul><ul><li>Bad log – good log? </li></ul><ul><li>Conclusions and action items </li></ul>
  3. 3. Log Data Overview <ul><li>Audit logs </li></ul><ul><li>Transaction logs </li></ul><ul><li>Intrusion logs </li></ul><ul><li>Connection logs </li></ul><ul><li>System performance records </li></ul><ul><li>User activity logs </li></ul><ul><li>Various alerts and other messages </li></ul><ul><li>Firewalls/intrusion prevention </li></ul><ul><li>Routers/switches </li></ul><ul><li>Intrusion detection </li></ul><ul><li>Servers, desktops, mainframes </li></ul><ul><li>Business applications </li></ul><ul><li>Databases </li></ul><ul><li>Anti-virus </li></ul><ul><li>VPNs </li></ul>What logs? From Where?
  4. 4. “ Standard” Messages 10/09/200317:42:57,,48352,,909,,,accept,tcp,,,,909,,,0,,4,3,,' 9Oct2003 17:42:57,accept,labcpngfp3,inbound,eth2c0,0,VPN-1 & FireWall-1,product=VPN-1 & FireWall-1[db_tag={0DE0E532-EEA0-11D7-BDFC-927F5D1DECEC};mgmt= labcpngfp3;date=1064415722;policy_name= Standard],labdragon,48352,,909, tcp,,',eth2c0,inbound Oct 9 16:29:49 [] Oct 09 2003 16:44:50: %PIX-6-302013: Built outbound TCP connection 2245701 for outside: ( to inside: ( PIX 2003-10-20|15:25:52|dragonapp-nids|TCP-SCAN|||0|0|X|------S-|0|total=484,min=1,max=1024,up=246,down=237,flags=------S-,Oct20-15:25:34,Oct20-15:25:52| Oct 20 15:35:08 labsnort snort: [1:1421:2] SNMP AgentX/tcp request [Classification: Attempted Information Leak] [Priority: 2]: {TCP} -> SENSORDATAID=&quot;138715&quot; SENSORNAME=&quot;; ALERTID=&quot;QPQVIOAJKBNC6OONK6FTNLLESZ&quot; LOCALTIMEZONEOFFSET=&quot;14400&quot; ALERTNAME=&quot;pcAnywhere_Probe“ ALERTDATETIME=&quot;2003-10-20 19:35:21.0&quot; SRCADDRESSNAME=&quot;; SOURCEPORT=&quot;42444&quot; INTRUDERPORT=&quot;42444&quot; DESTADDRESSNAME=&quot;; VICTIMPORT=&quot;5631&quot; ALERTCOUNT=&quot;1&quot; ALERTPRIORITY=&quot;3&quot; PRODUCTID=&quot;3&quot; PROTOCOLID=&quot;6&quot; REASON=&quot;RSTsent&quot;
  5. 5. What Commonly “Gets Logged”? <ul><li>System or software startup, shutdown, restart, and abnormal termination (crash) </li></ul><ul><li>Various thresholds being exceeded or reaching dangerous levels such as disk space full, memory exhausted, or processor load too high </li></ul><ul><li>Hardware health messages that the system can troubleshoot or at least detect and log </li></ul><ul><li>Access to resources and authentication decisions </li></ul><ul><li>Network connections , failed and successful </li></ul><ul><li>User access privilege changes such as the su command—both failed and successful </li></ul><ul><li>User credentials and access right changes , such as account updates, creation, and deletion—both failed and successful </li></ul><ul><li>System configuration changes and software updates—both failed and successful </li></ul>
  6. 6. Why Logs? What Are They Good For? <ul><li>Security teams </li></ul><ul><ul><li>Monitor, detect, investigate, track, analyze </li></ul></ul><ul><li>Auditors </li></ul><ul><ul><li>Well, audit  </li></ul></ul><ul><li>The rest of IT </li></ul><ul><ul><li>Troubleshoot, measure, verify, monitor </li></ul></ul><ul><li>Finally, developers </li></ul><ul><ul><li>“ Hmmmm… maybe for debugging… I dunno  ” </li></ul></ul>
  7. 7. Why NOT Log? Why Logs Are Baaaaaaad!  <ul><li>“ Who needs logs? I sure don’t!” - “ Debugger is better! Printf is fine too…” </li></ul><ul><li>Isn’t it what system (network) infrastructure does? </li></ul><ul><li>It slows down the systems, uses up disk and memory </li></ul><ul><li>“ It was not in the spec ” </li></ul><ul><li>We don’t know what to log – we will log something (and then hate it) </li></ul><ul><li>We don’t know how to do it – we will do it somehow (and then hate it more…) </li></ul>
  8. 8. Two Log Realms – And A Chasm In Between! <ul><li>“ Audit” logs vs “ debug” logs </li></ul>
  9. 9. In Detail: Audit vs Debug Logs Not known Known in advance Scope Might come handy (but debugger is better …) Slow down systems, clutter disk, unclear purpose, hard to update – yuck!  Developer view Useful for hours/days Useful for years Time scope Faults, failures, errors Attacks, activities, faults Content Sometimes on Always on Condition System operator, developer Security, audit Primary “consumer” Debug Logs Audit Logs
  10. 10. Result? <ul><li>%PIX|ASA-3-713185 Error: Username too long - connection aborted </li></ul><ul><li>%PIX|ASA-5-501101 User transitioning priv level </li></ul><ul><li>ERROR: transport error 202: send failed: Success </li></ul><ul><li>sles10sp1oes oesaudit: type=CWD msg=audit(09/27/07 22:09:45.683:318) :  cwd=/home/user1 </li></ul>
  11. 11. More results? <ul><li>userenv[error] 1030 RCI-CORPwsupx No description available </li></ul><ul><li>Aug 11 09:11:19 xx null pif ? exit! 0 </li></ul><ul><li>Apr 23 23:03:08 support last message repeated 3 times </li></ul><ul><li>Apr 23 23:04:23 support last message repeated 5 times </li></ul><ul><li>Apr 23 23:05:38 support last message repeated 5 times </li></ul>
  12. 12. Categorizing Log Badness - I <ul><li>Not logging what is essential </li></ul><ul><ul><li>E.g. not recording authentication failures </li></ul></ul><ul><li>Lacking severity/priority/rank </li></ul><ul><ul><li>E.g. no matter how limited </li></ul></ul><ul><li>Missing essential details </li></ul><ul><ul><li>E.g. user access denied – what user? </li></ul></ul><ul><li>Missing other details / context </li></ul><ul><ul><li>E.g. error 124 – what is it, exactly? </li></ul></ul>
  13. 13. Categorizing Log Badness - II <ul><li>Format bad for humans </li></ul><ul><ul><li>E.g. XML with binary blurbs </li></ul></ul><ul><li>Format bad for systems </li></ul><ul><ul><li>E.g. English  </li></ul></ul><ul><li>Not machine-correlatable </li></ul><ul><ul><li>E.g. related messages with no unifying ID </li></ul></ul><ul><li>All messages within an application use different syntax </li></ul><ul><ul><li>E.g. “login failed’ and ‘logon success=no’ </li></ul></ul><ul><li>Logging pieces of source code or confidential data </li></ul><ul><li>Log message subjective/interpretative </li></ul>
  14. 14. But This … This Here Takes The Cake … <ul><li>Logging username AND passwords to “debug” authentication (niiiice!  ) </li></ul><ul><li>Logging numeric error codes – and not having documentation ANYWHERE (please read my mind!  ) </li></ul><ul><li>Logging chunks of source code to syslog (care to see a 67kB syslog message?  ) </li></ul>
  15. 15. Alien (logging) vs Predator (performance) <ul><li>A curiously persistent myth … </li></ul><ul><li>A myth? Try this on your Oracle database! </li></ul><ul><li>Logging FOR performance!? </li></ul><ul><li>Yes, logging hurts performance in some cases, BUT … </li></ul><ul><ul><li>Laws tell you that you MUST log (this trumps it) </li></ul></ul><ul><ul><li>Logging helps troubleshoot performance issues </li></ul></ul><ul><ul><li>Finally, disk is cheap! </li></ul></ul>
  16. 16. On The Other Hand, Best Logs … <ul><li>… tell you exactly what happened – when, where and how </li></ul><ul><li>… are suitable for manual, semi-automated and automated analysis </li></ul><ul><li>… are suitable for remote collection and analysis </li></ul><ul><li>… can be analyzed without having the application that produced them at hand </li></ul><ul><li>… don’t slow the system down </li></ul><ul><li>… can be proven reliable (if used as evidence) </li></ul>
  17. 17. So, How to Change That?! <ul><li>Logging resources for developers: </li></ul><ul><li>Central logging APIs/libraries/modules </li></ul><ul><ul><li>Log4j and its brethren (log4net, log4cxx, log4php, etc) </li></ul></ul><ul><ul><li>XDAS and OpenXDAS APIs </li></ul></ul><ul><ul><li>Windows Event Log API </li></ul></ul><ul><ul><li>Unix syslog() </li></ul></ul><ul><li>So? </li></ul><ul><li>A bit of how , not much of what </li></ul>
  18. 18. <ul><li>Some Answers – No More Questions for A Change …. </li></ul>
  19. 19. Brief “Treatise” On What To Log - Events <ul><li>AAA (Authentication, Authorization, Access) </li></ul><ul><ul><li>Authentication/authorization decisions </li></ul></ul><ul><ul><li>System access, data access </li></ul></ul><ul><li>Change </li></ul><ul><ul><li>System/application changes (especially privilege changes) </li></ul></ul><ul><ul><li>Data change (creation and destruction are changes too) </li></ul></ul><ul><li>“ Badness” / Threats </li></ul><ul><ul><li>Invalid input </li></ul></ul><ul><li>Resource Issues </li></ul><ul><ul><li>Resource exhausted, capacity exceeded, etc </li></ul></ul><ul><ul><li>Limit reached </li></ul></ul><ul><li>Mixed Availability Issues </li></ul><ul><ul><li>Startups and shutdowns </li></ul></ul><ul><ul><li>Faults and errors </li></ul></ul><ul><ul><li>Backups success / failure </li></ul></ul>
  20. 20. Brief “Treatise” On What To Log – Core Details <ul><li>Timestamp + TZ ( when ) </li></ul><ul><li>System, application or component ( where ) </li></ul><ul><li>User ( who ) </li></ul><ul><li>Action ( what ) </li></ul><ul><li>Status ( result ) </li></ul><ul><li>Priority (severity, importance, rank, level, etc) </li></ul><ul><li>Reason ( can I dream for a second?  ) </li></ul>
  21. 21. Brief “Treatise” On What To Log – Context <ul><li>Source IP (DNS name, other name, etc) </li></ul><ul><li>Logging system (process, application, component, sub-component) </li></ul><ul><li>Affected system (process, application, component, sub-component) </li></ul>
  22. 22. Brief “Treatise” On How To Log <ul><li>Use globally standard APIs, libraries, modules </li></ul><ul><li>User language-standard APIs, libraries, modules </li></ul><ul><li>Use application-standard APIs, libraries, modules </li></ul><ul><li>Darn it, just log something  </li></ul><ul><li>P.S. Gross oversimplification alert! </li></ul><ul><li>In reality, different log call for different mechanisms: don’t syslog what you need to file dump, etc. </li></ul>
  23. 23. Who Will Do This? Inside Your Organization <ul><li>Organization security team </li></ul><ul><ul><li>Create a logging standard for internally-developed software (considering the regulations, security needs, etc) </li></ul></ul><ul><ul><li>Involve IT and business owners in the process </li></ul></ul><ul><li>IT / development managers </li></ul><ul><ul><li>Enforce logging standard as a “MUST-have” feature </li></ul></ul><ul><li>Software architects </li></ul><ul><ul><li>“ Get” the value of logging </li></ul></ul><ul><ul><li>Understand audit vs debug logging </li></ul></ul><ul><li>Software developers </li></ul><ul><ul><li>Follow the standard, use libraries and APIs, add logging features </li></ul></ul>
  24. 24. Who Will Do This? Outside Your Organization <ul><li>Commercial software vendors </li></ul><ul><ul><li>Start developing (and then adopting) log standards (CEE) </li></ul></ul><ul><li>Open source community </li></ul><ul><ul><li>Create logging libraries </li></ul></ul><ul><ul><li>Popularize standard logging routines </li></ul></ul><ul><li>Log analysis vendors </li></ul><ul><ul><li>Encourage the log standardization </li></ul></ul><ul><ul><li>Use their [universally painful!] log analysis experience to drive all of the above!  </li></ul></ul>
  25. 25. Mini-conclusion <ul><li>YOU MUST LOG! </li></ul><ul><li>The important of logging will ONLY GROW. </li></ul><ul><li>Software architects and developer need to “get” logging – there is NO other way ( infrastructure logging won’t cut it) </li></ul><ul><li>Security team will need to guide them! </li></ul><ul><li>Logging standards (global and organization-wide) are a MUST – and they will happen </li></ul><ul><ul><li>Pending a global standard - use your own, but standard across your infrastructure </li></ul></ul>
  26. 26. Thank You for Attending! <ul><li>Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA </li></ul><ul><li>Chief Logging Evangelist </li></ul><ul><li>LogLogic, Inc </li></ul><ul><li> </li></ul><ul><li>See for my papers, books, reviews </li></ul><ul><li>and other security and logging resources; </li></ul><ul><li>check my blog at </li></ul>