Developing Highly Instrumented Applications with Minimal Effort


Published on

Presentation from Silicon Valley Code Camp 2013. Related code on github:


Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Many products promise to provide insights and value from whatever data you throw at them. There’s no doubt they can provide something, but you can’t escape the law of GIGO.
  • As a result, it’s on you as a developer, product manager and business to care about the data you are creating. This presentation will demonstrate some best practices on how to go about this, regardless of your platform.
  • Now for some definitions. There is a lot of stuff generated by the system – let’s classify the stuff broadly and narrow on what I am going to be talking about.
  • Quick review of patterns we’ll use. Interception is the secret sauce here. We want to be able to get in front of every call to every object or component in our system. That lets us see what data is being passed around, who is doing it, and how long it took.
  • Indent the bullets
  • Developing Highly Instrumented Applications with Minimal Effort

    1. 1. Developing highly instrumented applications with minimal effort Tim Hobson Principal Engineer, Chief Caffeine Officer - Intuit
    2. 2. About Me 2
    3. 3. Agenda Concepts • Quality Data • Application Instrumentation • Application Logging Theory • Patterns • Supporting Frameworks • Supporting Components Practice • .Net Example • Java Example • Node.js Example GOAL: You should be able to apply this to your application today 3
    4. 4. Concepts
    5. 5. If You Do Nothing… OR 5
    6. 6. There’s No Free Lunch Garbage In, Garbage Out Insight! 6
    7. 7. Quality In, Insight Out Most apps start with only the framework or app server logging (or nothing!) None of the above is interesting to the business or the developer. You can’t get if you don’t give – there are many ways to give, and many classes of data to provide. 7
    8. 8. Classes of System Output System Instrumentation • JMX/WMI/SNMP monitoring • Apache/IIS/nginx access logs App Logging App Instrumentation • • • • • • • • Intentional Business Transactions Overtly triggers alerts Source of business metrics • Aids in troubleshooting failures, bugs Cross-cutting (free) App Activity Passively triggers alerts Source of performance data • Source of usage data 8
    9. 9. Theory and Best Practices
    10. 10. Best Practices (© splunk>) Create human readable events Clearly timestamp events Use key-value pairs Be aware of multi-value fields Log unique identifiers 10
    11. 11. Best Practices (© Tim) Global timestamps (UTC – 2013-08-21 22:43:31,990) Context setting (who/what/where/when/how) Categories/taxonomy (what tier, what component) Timing (time everything!) Security (never log sensitive data: password=***) Consistency in naming – (action=purchase; sale=oct13; productId=123123) – (action=buy; promo=oct13; sku=123123) 11
    12. 12. Example Output Context 2013-08-21 22:55:36,504; LogLevel=INFO; sid=q3prv41kt511vzojytnx1d42; rid=6500583; userLogin=(null); ipAddress=; thread=249; category=Web.Controllers.BaseWebController; msg=RequestInfo; server=ws001prod; url=; method=GET; languages=en-US,en;q=0.8; referrer=; userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36; userId=123456; controller=Account; action=Logon; Metrics 2013-08-21 22:55:36,519; LogLevel=INFO; sid=q3prv41kt511vzojytnx1d42; rid=6500583; userLogin=(null); ipAddress=; thread=249; category=TraceInterceptor; timeTracing=Service.Users.GetUser, time=3; 12
    13. 13. Patterns Pipeline – Filter – Interceptor Dependency injection – Proxy Pointcuts/aspects 13
    14. 14. HTTP Pipelines var app = express(); app.use(func1); app.use(func2); app.use(func3); app.use(func4); app.use(func5); app.use(func6); app.use(func7); app.use(func8); app.listen(8080); Java (from Oracle Docs) ASP.Net (from MSFT Docs) 14 express.js
    15. 15. Dependency Injection • Enforces interface-based programming • Forbids circular references • Lends itself to testability • Flexibility in changing implementations • For our purposes: cross-cutting capabilities 15
    16. 16. Intercepting Calls With Dynamic Proxies Interceptor Dependency A Implementation DI Container 16 Interceptor Interceptor Proxy Interceptor Interceptor Interceptor Caller Interceptor Proxy Interceptor Interceptor Proxy Dependency B
    17. 17. Supporting Frameworks The Front Door The Inner Sanctum Annotations & Attributes • ASP.Net Modules/ActionFi lters • Java Servlet Filters/Spring Interceptors • Node.js interceptors • Unity Dynamic Proxies • Spring @AspectJ Pointcuts • JavaScript Mixins • @Instrumentable • [Instrumentable] 17
    18. 18. Supporting Components .Net • Log4Net • NLog Java node.js • LogBack • SLF4J • Log4J 18 • SenchaLabs Connect • Winston
    19. 19. Intervention!
    20. 20. Sample App Intervention 3 Platforms 3 Sample Apps 20 3 Interventions
    21. 21. Goals Leverage an HTTP pipeline for context Leverage aspects for interception Apply best practices Minimize impact on existing code 21
    22. 22. MVC Music Store Intervention (Demo) Missing dependency injection/interception framework: Unity Missing logging library: Log4Net, Buche 1. 2. 3. 4. 5. 6. 7. 8. Add DI framework (Unity) Configure aspects (Interception) Configure controller factory, container locator Set up logger (Log4Net) Configure log pattern and targets (Log4Net.xml) Wire up logging interceptor Wire up LogActionFilter and BaseLoggingController Deploy! 22
    23. 23. Spring Pet Clinic Intervention (Demo) 1. Set up logger a. Configure log pattern (to support context data) b. Configure targets (rotating file) 2. Create and @Instrumentable attribute a. Capture calling context b. Start/stop timing 3. Create a. Capture request metadata b. Set MDC c. Start/stop timing 4. Configure aspect bean 1. Configure interceptor bean 1. Annotate the methods we care about 23
    24. 24. NodeCellar Intervention (Demo) Missing Interception Framework: Scarlet Missing Logging Library: Winston 1. 2. 3. 4. 5. 6. Set up Scarlet Configure log pattern Configure method interception Create logging interceptor Create LoggingFilter Deploy! 24
    25. 25. Closing Thoughts
    26. 26. Got Log? Now Get Intimate With Your App Ops Dashboards Business Dashboards Quality Assurance Pro-Active Service Degradation Alerting SLA Tracking Security Alerting 2 6 Customer Support Performance Metrics Pre-Release Sanity Testing
    27. 27. Key Takeaways • It is YOUR responsibility as a developer to provide useful operational and business data. • It is not hard, and most of it is for free once you have the patterns in place. • The same patterns and practices can be applied to practically any platform, and any type of application or service. • When you provide consistent and predictable data others can build on your greatness 27
    28. 28. Get the Code @hoserdude Spring Pet Clinic Intervention: MVC Music Store Intervention: NodeCellar Intervention: 28
    29. 29. Intuit Speakers @ Silicon Code Camp 2013: SATURDAY 9:45 a.m. - Ramakrishna Kollipara – “Complete Automation of Performance Testing” 1:45 p.m. - Joe Wells - “QBO: Journey From legacy Java app to a Client-side HTML5 app” 3:30 p.m. - Naga Addagadde & Sangeeta Narang – “Intuit APIs for Financial Transaction Aggregation” 5:00 p.m. Ted Drake –“Hitting the Accessibility High Notes with ARIA” SUNDAY 9:15 a.m. - Eugene Krivopaltsev –“Building Native Mobile Apps with Custom Views” 1:15 p.m. - Tim Hobson – “Developing Highly Instrumental Applications with Minimal Effort” For more information about joining our organization visit our booth or connect with our onsite recruiter: You don't want to miss out on a chance to win this cool headset. Stop by our booth to enter!
    30. 30. THANK YOU Want to talk more? I’ll be at the Intuit booth today 3-5 PM. 30