In the past, applications where created as monolithic entities running on a single server. If this is the past for you, too, you will have experienced the downside of modern distributed and cloud applications, as debugging, troubleshooting, and monitoring is not easily accomplished with traditional approaches. Within this session, we will explore different possibilities for collecting and analyzing the needed information to solve issues on modern distributed application and discuss the advantages and disadvantages of each approach like debugger, log files, performance counter and third party solutions. The focus of this session will be on Developer and DevOps need, as increased release cycles and third party dependency more and more result in the need for troubleshooting also on production system, rather than in an isolated test environment. This session requires a solid understanding of distributed applications and knowledge of SOA, but most principles also apply to and can be beneficial for more traditional application design approaches. The used code examples are in .NET but the shown principles generally apply to other languages, too, and shown software is often available for a variety of environments.

1. Debugging,
Troubleshooting
Distributed Web
& Cloud Applications
Foothill College, Oct. 5th, 2013

2. Theo Jungeblut
• Engineering manager & lead by day
at AppDynamics in San Francisco
• Coder & software craftsman by night
• Architects decoupled solutions
tailored to business needs & crafts
maintainable code to last
• Worked in healthcare and factory
automation, building mission critical
applications, framework & platforms
• Degree in Software Engineering
and Network Communications
• Enjoys cycling, running and eating
theo@designitright.net
www.designitright.net

3. Warning
• Contains also proprietary
software
• This areas are pointed out
• Please walk away now,
if that is not acceptable
for you, it’s okay

4. Overview
• Inquiry about the Status Quo
• Some Facts -> Dev Process & Ops Reality
• Monolithic Applications
• Debugging & Logging
• Distributed Applications
• Side Effects you need to troubleshot
• Performance Counter
• Microsoft IntelliTrace
• AppDynamics
• Summary
• Q & A

5. We need YOU!

6. How agile are You?

7. The Impact of Process
* from Wikipedia: http://en.wikipedia.org/wiki/File:Agile-vs-iterative-flow.jpg

8. How many releases
do your company
have a year?

9. Typical Releases a Year*
58%
17%
21%
3%
2+ months
monthly
weekly
daily
http://www.appdynamics.com/blog/2011/12/14/storm-clouds-in-2012-summary-of-
appdynamics-apm-customer-survey/

10. Development + Operations = DevOps
* from Wikipedia: http://en.wikipedia.org/wiki/File:Agile-vs-iterative-flow.jpg

11. Difference between Dev & Ops
Dev
• Getting / being agile: need for frequent updates
with new feature and bug fixes
• Achieves stability trough running latest
Ops
• Highest goal: stable execution flawless execution,
by knowing the environment and software
• Achieves stability through careful rollouts

12. “The Past/Present”*
Web
Server DB
User
* or maybe still your present?

13. “The Past/Present” - Architecture
Data Access Layer
Business Logic Layer
Presentation Layer
DB

14. How are you
troubleshooting?

15. Traditional Debugging

16. Traditional Debugging
Advantages
• Visibility for executed code path
Disadvantages
• Performance impact
• Requires software installation
• Often not possible in production
• Typical does not find easily distributes issues
• Only real time results

17. Low Impact Debugger &
Crash Dump Debugging
http://www.windbg.info

18. Automatic Crash Dump Analysis
http://blogs.msdn.com/b/debugdiag/archive/2013/10/03/debugdiag-2-0-is-now-rtw.aspx

19. Low Impact Debugger &
Crash Dump Debugging
Advantages
• Low overhead
• Dump collection can be done in production
• Allows for offline analysis of dump files
• Can be setup to automatically collect dump files for crashes
• Allows very deep analysis and debugging
Disadvantages
• Automatic analysis limited
• Manual analysis has steep learning curve
• Offline review requires to have precise dll version available

20. The Book
http://www.amazon.com/Advanced-NET-Debugging-Mario-Hewardt/dp/0321578899

21. Log Files
Advantages
• Output adjustable without restart
• Accepted in Production
Disadvantages
• Manual, time intense creation of log statements
• Manual, time intense process of result review
• Limited scope & visibility
• Potentially high performance impact based on IO

22. “The Present/Future”- Architecture
Microsoft MSDN: http://msdn.microsoft.com/es-es/architecture/gg189193

23. “The Present/Future”
ASP.NET
MVC 3/4
(IIS)
User
WAS Tomcat
Azure
Web Role
SQL Server MySQL
Active
Directory
Azure
Worker Role
WAS
3rd Party
Web Services
Java
Application
MongoDB
3rd Party
Web Services
WCF HTTP

24. The (Netflix) Distributed Reality *
* http://techblog.netflix.com/2012/06/netflix-operations-part-i-going.html

25. What has all this
to do with
Debugging
and
Troubleshooting?

26. Who does really know
how their application
is performing?

27. What impacts
Application
Performance?

28. Concurrency
http://en.wikipedia.org/wiki/File:Smithposter.jpg
• Deadlocks
• Raise Conditions
• Unexpected complexity
• …

29. Data Volume
http://en.wikipedia.org/wiki/File:Smithposter.jpg
• constantly increasing
• Volume hides details
• Handling volume
impacts resources
• …

30. Resources
http://blog.laptopmag.com/amd-cries-foul-but-tests-show-battery-life-inferior-to-intels/battery_morpheus
• are limited
• Danger of Starvation
• …

31. Who is
Performance Testing
every release?

32. http://www.slideshare.net/appdynamics/what-can-devops-learn-from-formula-1

33. Dev + Ops = Success
• DevOps has to be a combined approach
• Mostly no visibility in real performance
& production state

34. IntelliTrace™
• “Flight Recorder” for your Process
• Records configurable details
• Allows offline analysis
• Microsoft specific Tool in Visual Studio
• Java Flight Recorder within the Oracle world
IntelliTrace http://msdn.microsoft.com/en-us/library/hh398365.aspx
Java FlightRecorder: http://gotocon.com/dl/goto-aar-
2013/slides/RickardBckman_JavaFlightRecorderNextGenerationDiagnosticsAndProfiling.pdf

35. Summary IntelliTrace
Advantages
• Allows great visibility into single process/AppPool
• Allows to play back test or production issue
Disadvantages
• Limited to Single Process / AppPool Visibility
• Should be only temporally enable in production
• Requires configuration effort for balance between
details & performance

36. Performance Counter / Perfmon
http://msdn.microsoft.com/en-us/library/aa645516(v=vs.71).aspx

37. Performance Counter
• Collect machine wide and process
specific data
• Allows on- and offline analysis
• Microsoft specific tool build into
Windows
http://www.codeproject.com/Articles/8590/An-Introduction-To-Performance-Counters

38. Summary Performance Counter
Advantages
• Huge OOTB support in Windows/IIS/etc.
• Low overhead
• Extensible
Disadvantages
• Collecting application specific data requires
source code changes

39. AppDynamics
• Application Performance Management
• Optimized for Production
• Monitoring instead of investigating
• Providing the “Right-Data”
Let’s take a look
http://www.appdynamics.com/products-editions.php

40. Summary AppDynamics
Advantages
• Works in production on high performance sides
• Monitors, Notify, Troubleshoot
• Allows instance handling in cloud environments
• Free lite version
• Collects required application details
Disadvantages
• Full version not free
• Does not include every application detail

41. Build your own Opinion (for Free)
• Lite Version
free but
limited
• 30 days free
Trial of Pro &
Azure Version
http://www.appdynamics.com/products-free-download.php

42. Summary
• Understand Dev Impact to Ops Execution
• Understand the business impact of
production issues
• Monolithic is simpler to Troubleshoot
• Distributed is more complex
• The right tools can make a Big Difference

43. Downloads,
Feedback & Comments:
Q & A
Graphic by Nathan Sawaya
courtesy of brickartist.com
theo@designitright.net
www.designitright.net
www.speakerrate.com/theoj
www. slideshare.net/theojungeblut

44. Blog, Rating, Slides
http://www.DesignItRight.net
www.speakerrate.com/theoj
www.slideshare.net/theojungeblut

45. Time to say Thank You!
The Organizers
Foothill College (team)
The volunteers (how about you?)
The
Sponsors
Pictures
from
http://blog.siliconvalley-codecamp.com

46. … thanks for you attention!
And visit and support the
www.siliconvalley-codecamp.com
Please fill out the
feedback, and…
www.speakerrate.com/theoj