Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The Ultimate Logging Architecture You know you WANT it! Michele Leroux Bustamante michelebusta@solliance.net @micheleb...
The Hello World Of Logging 1992
Hello World!
Hello World!
Logging Today 2014
Web Browsers Mobile Apps Client Apps
Why do we log? • Troubleshooting visibility • Security audits, review, early detection • Post incident forensics • Tra...
What to log? EXAMPLE: Application Events Windows Logs IIS Logs Trace Output EXAMPLE: Login Attempts Unauthorized/ ...
Make Logging EASY
Implement a Log Helper ILogger Logger Trace Debug() Trace Information() Trace Warning() Trace Error() Throw() ...
Failure is NOT an option.
Event Logging
Just Do It • Whatever is built in • Whatever you know best • Just do it
Encapsulate the Mechanism ILogger Logger ELMAH / SLAB Azure Diagnostics log4j / log4net ElasticSearch
Audit Logging
Logs and Compliance • Contain no user credentials • No PII, PHI or identifiable user data • Retention period (1 year is...
Implement an Audit Helper Logger.Current.TraceInformation(); Logger.Current.Throw(ex); ILogger Logger Trace xxx() T...
Benefits of noSQL • Log details tend to evolve – Schema-less storage is best – Re-indexing may be necessary • Co-locat...
Audit Log Use Cases • Every login attempt (success or failure) • Excessive login attempts and lockouts • Blocking/black...
Audit Log Fields • Date/time of event • Machine name/instance • Process ID • User ID (possibly encrypted) / Session ID...
History and Activity Logging
History Logs • Changes made to key tables • Describes – Who changed the record? – From which application? – Which fie...
Implement a History Log Helper HistoryLogger.Current.Write(); IHistoryLogger HistoryLogger History Logs DocumentDB C...
Wrap History in the DAL History Logs OrdersDal UsersDal ContentDal Relational DB Orders Claims Users Content
Wrap History in the DAL History Logs OrdersDal UsersDal ContentDal Relational DB Orders Claims Users Content
What happened with my order? History Logs OrdersDal UsersDal ContentDal Relational DB Orders Claims Users Content
Activity Logs • Not specific to code execution and troubleshooting, diagnostics • Specific to the application, user act...
Implement an Activity Log Helper ActivityLogger.Current.UserDownload(); ActivityLogger.Current.ReportRequest(); Activit...
What happened with my order? History Logs OrdersDal Relational DB Orders Activity Logs
Automate Logging Where Possible • View controllers • API controllers • Authorization hooks • Outbound calls • Data Ac...
To Queue Or NOT To Queue
Client and Server Logging Client Apps Mobile API Client API Log API Client API Log API Loggers Web Browsers Mobile ...
What can I queue? Loggers ETW DocDB Event Logs Audit Logs Activity Logs History Logs
ETW Goal Loggers ETW History Publisher Activity Publisher Audit Publisher ALERTS Stream Analytics Events Publ...
Queued Logging • Considerations – Timestamps matter – Correlation across nodes matters (to a point) – Guaranteed exact...
Troubleshooting Is Important!
Problem Statement • We need immediate access to what the HECK is going on when there is a problem • Sometimes I use (in...
Elasticsearch Architecture Logger AuditLogger HistoryLogger ActivityLogger Elasticsearch
Kibana Visualization
LogStash LogStash Elasticsearch Identity Server Web Server / IIS / Event Logs CPU / Memory Perf Counters Blob CSVs ...
Archives, Aggregation and Analytics
ARCHIVE Elastic Search Audit Logs Activity Logs History Logs HDInsight PoweShell Spin up, analyze, spin down Inges...
What you’re looking for is… • Manageable implementation • Ability to “evolve” log content • Reduce IO / socket overhead...
References • Conference resources: – http://michelebusta.com • Contact me: – michelebusta@solliance.net – @michelebus...
Upcoming SlideShare
Loading in …5
×
Technology
9,588 views
Dec. 06, 2014

The Ultimate Logging Architecture - You KNOW you want it!

Logging is one of those things that everyone complains about, but doesn't dedicate time to. Of course, the first rule of logging is "do it". Without that, you have no visibility into system activities when investigations are required. But, the end goal is much, much more than this. Almost all applications require security audit logs for compliance; application logs for visibility across all cloud properties; and application tracing for tracking usage patterns and business intelligence. The latter is that magic sauce that helps businesses learn about their customer or in some cases the data is FOR the customer. Without a strategy this can get very messy, fast. In this session Michele will discuss design patterns for a sound logging and audit strategy; considerations for security and compliance; the benefits of a noSQL approach; and more.

no profile picture user

  • Login to see the comments

Show More

The Ultimate Logging Architecture - You KNOW you want it!

  1. 1. The Ultimate Logging Architecture You know you WANT it! Michele Leroux Bustamante michelebusta@solliance.net @michelebusta http://solliance.net http://michelebusta.com
  2. 2. The Hello World Of Logging 1992
  3. 3. Hello World!
  4. 4. Hello World!
  5. 5. Logging Today 2014
  6. 6. Web Browsers Mobile Apps Client Apps
  7. 7. Why do we log? • Troubleshooting visibility • Security audits, review, early detection • Post incident forensics • Track change history • Insights into user activity • Reporting and analysis
  8. 8. What to log? EXAMPLE: Application Events Windows Logs IIS Logs Trace Output EXAMPLE: Login Attempts Unauthorized/ Authorized Access Password Resets EXAMPLE: Session Trace Purchase Flow Report Generation Feature Access EXAMPLE: Change history for any critical system records Live Streaming / Analytics Event Logs Audit Logs Activity Logs History Logs
  9. 9. Make Logging EASY
  10. 10. Implement a Log Helper ILogger Logger Trace Debug() Trace Information() Trace Warning() Trace Error() Throw() Logger.Current.TraceInformation(); Logger.Current.Throw(ex);
  11. 11. Failure is NOT an option.
  12. 12. Event Logging
  13. 13. Just Do It • Whatever is built in • Whatever you know best • Just do it
  14. 14. Encapsulate the Mechanism ILogger Logger ELMAH / SLAB Azure Diagnostics log4j / log4net ElasticSearch
  15. 15. Audit Logging
  16. 16. Logs and Compliance • Contain no user credentials • No PII, PHI or identifiable user data • Retention period (1 year is good baseline) • A structured archival process • Alert if log reaches capacity • Authorized access • Protections from modifications (write-only)
  17. 17. Implement an Audit Helper Logger.Current.TraceInformation(); Logger.Current.Throw(ex); ILogger Logger Trace xxx() Throw() AuditLogger.Current.Write(); AuditLogger.Current.Throw(ex); Write() Throw() IAuditLogger AuditLogger Azure Blobs Event Logs Audit Logs DocumentDB
  18. 18. Benefits of noSQL • Log details tend to evolve – Schema-less storage is best – Re-indexing may be necessary • Co-location with mainline databases – Adds complexity and overhead (potentially) – Does not allow a separate “evolution” team around telemetry and analysis
  19. 19. Audit Log Use Cases • Every login attempt (success or failure) • Excessive login attempts and lockouts • Blocking/blacklisting users, IP addresses, access ports • Every logout • Every modification to user table, including permissions • All configuration changes • Attempts to access restricted resources, APIs from unexpected paths • All access to PII / PHI in an individually identifiable way
  20. 20. Audit Log Fields • Date/time of event • Machine name/instance • Process ID • User ID (possibly encrypted) / Session ID • Type of event • Success or failure of the event (if applicable) • Seriousness of the event violation (if applicable) • Message (free form) • Stack Trace (if applicable)
  21. 21. History and Activity Logging
  22. 22. History Logs • Changes made to key tables • Describes – Who changed the record? – From which application? – Which fields changed? • Need the ability to surface this to applications – Sometimes to users – Always to operations to solve problems
  23. 23. Implement a History Log Helper HistoryLogger.Current.Write(); IHistoryLogger HistoryLogger History Logs DocumentDB Claims Users Orders Claims Claims …
  24. 24. Wrap History in the DAL History Logs OrdersDal UsersDal ContentDal Relational DB Orders Claims Users Content
  25. 25. Wrap History in the DAL History Logs OrdersDal UsersDal ContentDal Relational DB Orders Claims Users Content
  26. 26. What happened with my order? History Logs OrdersDal UsersDal ContentDal Relational DB Orders Claims Users Content
  27. 27. Activity Logs • Not specific to code execution and troubleshooting, diagnostics • Specific to the application, user activity • COULD be informative to users as well – History of recent activity in the site – Reports they requested, downloads, other… • Provides insights to the business regarding user activity, trends and patterns – Non-critical analysis
  28. 28. Implement an Activity Log Helper ActivityLogger.Current.UserDownload(); ActivityLogger.Current.ReportRequest(); ActivityLogger.Current.PurchaseOrder(); IActivityLogger ActivityLogger Activity Logs DocumentDB
  29. 29. What happened with my order? History Logs OrdersDal Relational DB Orders Activity Logs
  30. 30. Automate Logging Where Possible • View controllers • API controllers • Authorization hooks • Outbound calls • Data Access layers
  31. 31. To Queue Or NOT To Queue
  32. 32. Client and Server Logging Client Apps Mobile API Client API Log API Client API Log API Loggers Web Browsers Mobile Apps Event Logs Audit Logs Activity Logs History Logs
  33. 33. What can I queue? Loggers ETW DocDB Event Logs Audit Logs Activity Logs History Logs
  34. 34. ETW Goal Loggers ETW History Publisher Activity Publisher Audit Publisher ALERTS Stream Analytics Events Publisher Event Logs Audit Logs Activity Logs History Logs
  35. 35. Queued Logging • Considerations – Timestamps matter – Correlation across nodes matters (to a point) – Guaranteed exactly one in order doesn’t exist – Async is good (mostly) • That said – Priority matters (hot, warm, default) – Simplicity matters – Throughput matters
  36. 36. Troubleshooting Is Important!
  37. 37. Problem Statement • We need immediate access to what the HECK is going on when there is a problem • Sometimes I use (in order): – Google Analytics – Event Logs (Azure Website) – Table Storage queries (STRIKE THAT, USELESS) – Blob storage CSVs (good enough, not realtime)
  38. 38. Elasticsearch Architecture Logger AuditLogger HistoryLogger ActivityLogger Elasticsearch
  39. 39. Kibana Visualization
  40. 40. LogStash LogStash Elasticsearch Identity Server Web Server / IIS / Event Logs CPU / Memory Perf Counters Blob CSVs …
  41. 41. Archives, Aggregation and Analytics
  42. 42. ARCHIVE Elastic Search Audit Logs Activity Logs History Logs HDInsight PoweShell Spin up, analyze, spin down Ingest Blob Storage Event Logs OR, just…
  43. 43. What you’re looking for is… • Manageable implementation • Ability to “evolve” log content • Reduce IO / socket overhead (monitor this) • Prioritization • Real-time analytics, troubleshooting • Accessibility for UI lookups (history, activity) • Archival and mass analysis
  44. 44. References • Conference resources: – http://michelebusta.com • Contact me: – michelebusta@solliance.net – @michelebusta • Founder, CIO of Solliance – http://solliance.net

×