The document discusses root cause analysis in IT problem resolution. It defines root cause analysis and explains its importance in improving IT processes and meeting organizational goals. It describes the typical problem resolution process and challenges, including lack of a structured process and "blame game" between teams. The document reviews different problem detection methods and common root cause analysis approaches, and proposes ways to improve processes through mapping of services and infrastructure, implementing a structured collaborative process, and leveraging tools that provide historical context and topology-based correlation.

1. The Elusive Root Cause Of IT Problems
And How To Easily Identify It
Noam Biran
Director of Product Management

2. Introduction
Mr. Biran
• Director of Product Management at Neebula
• 20 years experience in systems management & BSM
• Innovation Product Management at BMC
• Co-founder of Appilog (now HP uCMDB & DDMA)
About Neebula
Neebula provides the first and only automatic service-centric IT management
solution allowing IT organizations to improve the service provided to the business
by shifting from managing disparate technology silos to managing the services
running in the data center. Leveraging unique technology that automatically maps
business services to the underlying infrastructure, Neebula enables the IT team to
increase availability of the main services they manage and reduce the time to
repair of problems.

3. Agenda
• Introduction
• Root cause analysis defined
• The problem resolution process
• Problem detection
• Root cause analysis methods
• Improving root cause analysis processes

4. Root Cause Analysis Definition
ITIL V3
An Activity that identifies the Root Cause of
an Incident or Problem.
Root Cause Analysis typically concentrates on
IT Infrastructure failures.
Wikipedia
Root Cause Analysis is any structured
approach to identify the factors that resulted
in the harmful consequences of one or more
past events

5. The importance of Root Cause Analysis
• Root Cause Analysis has a high impact on
– IT processes
• The efficiency of the overall incident/problem
management process
• Good RCA discipline requires well established
configuration management
– Organizational goals
• Meeting internal and external SLAs
• Financial (budget & revenue) implications
• Brand / customer loyalty

6. Root Cause Analysis Nowadays

7. The Critical Role of Root Cause Analysis
• Improper (or lack of) identification of the real
root cause may yield:
– Repeating problems
– Increased downtime
– Waste of human
resources on
“fixing” the wrong
issues
– Risk to the business

8. The Life of The Operator
We expect the operator
– To handle 1000’s of cryptic events
– Understand impact on 100’s of services
– Understand the correlation to
customers service complaints
– Understand what changed
– Orchestrate the resolution
And make these decisions within minutes to
reduce MTTR
Are we giving our operators the tools to
succeed?

9. Problem Resolution Process

10. Problem Resolution Process
• Events coming in to the NOC
• NOC performs some investigation
• Root cause analysis is shared between NOC
& 2nd/3rd level support (admins)
• Low level diagnostics & problem resolution
is done by 2nd/3rd level support (admins)

11. Involved Parties & Tools
• Tools
– Monitoring tools
– Configuration management tools
• People
– Users
– NOC
– Admins – specialized teams focused on specific
area, e.g. system, database, network
– Application support / developers

12. The Common Process – Blame Game
• No structured process
• Lack of overall cross-domain view
• Each team has its own terminology and view
• Each team is working on its own

13. Problem Detection

14. Potential Problem Symptoms
• Lack of certain functionality
– A certain transaction does not work
• Performance degradation
– Fund transfer response time is above 2 sec.
• Availability issue
– Application doesn’t work
• None
– Unnoticeable failure due to high availability
configuration

15. Problem Detection
• Good problem detection methods are key for a
structured root cause analysis process
• Problem detection tools should provide sufficient
data to the root cause analysis process
• There are various distinct methods each with its
pros and cons
• There is no single superior detection method

16. Detection – Users
• What it does
– Compensates for unknown / unreported
problems
• What it doesn’t
– Supposedly accurate – actually might point in
the wrong direction
– Usually takes place
too late for a quick fix
& impact to business

17. Detection – Infrastructure Monitoring
• What it does
– Monitor each technical element
comprising the service
– Great way to identify
specific availability failures
• What it doesn’t
– Hard to correlate with real user experience
– Too many false positives
– Lots of events on symptoms rather on actual problem

18. Detection – End User Experience
• What it does
– Measure overall response time of user transactions
– Synthetic or real user transactions
– The ultimate problem detection method
• What it doesn’t
– No real breakdown to assist
in pinpointing the problem
or even the domain

19. Detection – Transaction Breakdown
• What it does
– Discovery of each transaction’s path
within the data center
– Highlight potential performance
problems within the transaction
execution
• What it doesn’t
– No correlation to infrastructure
monitoring
– Cannot cover the entire data center
– domain specific

20. Detection – Domain Specific Tools
• What it does
– Drill down in a specific application
– Great analysis & diagnostics within an application
• What it doesn’t
– No data center wide view
– Lack of insight into the
connections between
applications

21. Detection - Synergy

22. Root Cause Analysis Methods

23. Potential Root Cause Types
• Configuration change
• Version upgrade
• Hardware fault
• Software bug
• Capacity problem
• Resource collision

24. Common Ways for Root Cause Analysis
• War room scenario
• The log file approach
• APM tools
• Transaction management
• Manual event correlation / analysis

25. War Room Scenario
• Getting everyone in the same room
• Each has its own data and terminology
• Blame game
• Takes a lot of time

26. The Log File Approach
• An admin sits and analyzes log files and
other historical data from various sources
• A domain specific approach
• Certain degree of structured process
• Might identify problems that
are not the root cause
(distractions)

27. APM Tools
• An admin sits and analyzes log files and
other historical data from various sources
• A domain specific approach
• Certain degree of structured process
• Might identify problems that
are not the root cause
(distractions)

28. Transaction Management
• A great tool to point to the probable area
where the root cause resides
• Limited to specific domains
• Inability to correlate with infrastructure
metrics / failures

29. Manual Event Correlation / Analysis
• Requires cross-domain expertise
• Requires understanding of dependencies
between components
• Time consuming
• Lack of insight into other
non-event data

30. Improving Root Cause Analysis
Processes

31. Making The Best From Existing Tools
• Choose problem detection methods that
assist in the root cause analysis process
• Turn the root cause analysis into a
structured process
– Internal team processes
– Inter-team processes
• Common language & visibility between
teams

32. New Methods: Mapping
• Mapping of Business service & applications
and the supporting infrastructure
• Ties symptoms (user) to problems
(technology)
• Introduces a common language between
teams
• Enables a high level cross-domain view

33. New Methods: Structured Process
• Define a structured process for problem
investigation and root cause analysis
• Define how collaboration should occur
during root cause analysis between teams

34. New Methods: Tools
• Use tools that provide a historical
dimension for problem investigation
• Use tools that enable the correlation of
problems to configuration changes
• Use topology based correlation instead of
rule based (or manual based) correlation