How Allscripts Streamlined Root Cause Analysis - AppSphere16

How Allscripts
streamlines root cause
analysis –
The gift of time
Jeff Brady
Allscripts

Copyright © 2016 Allscripts Healthcare Solutions, Inc. 2
Allscripts
• Brand New Allscripts Tower
opening in 2017 in Raleigh NC
• Allscripts serves more than 180,000 physician users,
solutions in 2700 hospitals and 13,000 extended care
organizations users on 48 different solutions
• Supporting hybrid on-prem and SaaS systems

About Jeff
• “Program of Programs” manager working within the
Common Components and Solutions (CCnS)
• Focus is the timely certification and implementation of an
industry wide regulatory initiative know as Meaningful use.
• Coordination of the activities for 28 inter related
development areas that requiring approx. 1000 team
members to execute.
• Married, 1 Grandson, 5 Children, and an old Dog

“To improve the interconnectivity of patient data while remaining in
compliance with Regulatory initiatives our systems are becoming
increasingly interconnected. The ability to assess the health and
performance of these system arrays is critical and needs to evolve at a
pace faster than the products.”
Problem Statement

Requirements from the business/challenges for IT
• Continuous performance improvement
– Better, Faster, Cheaper
• SME focus on reactive actions
– Expertise to root cause
– Junior engineers may not have the full view of the application
– Just working in their one area of the array
• Minimized exposure when incidents occur;
Extended time to root cause
– Forced separation of development and hosting due to PHI
– SCOM (indication of a deeper issue)
– Difficulty in identification of problem source
– Log file analysis
– Issue triage and ownership
– War rooms

Growth is necessary, but it can be expensive
System Expansion
(organic or net new opportunity)
– New regulatory initiatives
• Growth with limited additional income
– Data analytics
• Data warehousing of ever expanding data elements
• Timely retrieval of data and report execution
(key to client satisfaction)
• Expansion of reporting base
– System expansion
• State by state rollouts
• Additional market opportunities
• Feature expansions
• Onboarding of additional customers
– Shifts in market needs
• Transition of core business model

Delivering the Business’
Goals

Relieve the pressure on SMEs
Transition subject matter experts
(SME) into proactive behaviors
– Streamline root cause and detection
– Elevate (top line) junior engineers
Augmentation of existing tools
– Symbiotic relationship between tools such as
SCOM and AppDynamics
– Performance lab capability enhancements
Hosting
SCOM Enhancements
SCOM Trouble Shooting
Architectural Validation
Production insights / Details
Development
QA Feedback
Performance Feedback
Upgrade Comparison
Development Insights
Architectural Validation
Dev Ops Communications
Improved inter-group
communications
– High fidelity targeting of the source of the issue or
performance bottleneck
• Exact SQL Query that is limiting database
performance
• Precise .NET Method and Class
Enhance current engineering
processes
– Architectural validation
– Production details easily conveyed to
development team
Improved system performance

Smart Growth
• Economic scaling evaluations
– Extensibility of performance lab into value optimization
– Ability to numerically validate various design options performance
• System configuration
• Software design options
– Determination of best fit hardware scaling versus software scaling
– Augmentation of existing performance lab tooling makes this possible
BEFORE production deployment.
B1 B2
B3 B4
A1
A2
C1
C2
B1 B2
B3 B4
A1
A2
C1
C2
Capacity Doubling Via Hardware
B1 B2
B3 B4
A1
A2
C1
C2
Baseline Deployment
B1 B2 A1C1
B3 B4 A2C2
8x Capacity Via Config & Software

Action plan
• Identify stakeholders and understand their expectations
• Start small with a well understood solution and solve
problems
• Form a knowledge sharing community or working group of
dedicated and naturally curious engineers
• Align the deployment of AppDynamics from development
through production on your pilot
• Augment your existing tools, the relationship is symbiotic

-- “ had an issue with “Change Password” and they were hunting it for 2 weeks, so they
applied AppD to it. They found it was not in their code but in an unmanaged library. AppD to provided visibility
into the Method and Class hierarchy where the failure was occurring put them on it in a single run. Issue
solved, it was related to a security update from Microsoft.…”
--This was root caused within 1 hour
-- “downloads chase request files from our payers was one of our
longest running processes. After drilling down into the
Tx.ChaseRequests we were surprised to see that DNS resolution was
consuming over 90% of our execution time.”
--”Our SQL.Open calls were often the most "expensive" calls
performed by the message handlers. Often these calls took 300ms and up
to 800ms. “--
-- Analysis and development are in-process this sprint.
Value – AppDynamics “Win Wire”
-- “We are working to change our traffic models to reflect a
production like mix as well as enhancing test scripts” -- “This should
go a long way to addressing the gaps noted below.”

Value – “Smart Growth” Example
Sales opportunity – “Double throughput required”
– Applied AppDynamics to load test lab
• Augmentation of existing tool set to more accurately identify improvement
areas
• Extended performance lab into value optimization
– 2 Optimization runs
• Utilized AppDynamics database monitoring capability to determine most expensive SQL calls
• Utilized AppDynamics ability to compare different releases to numerically assess impact
– Est. of 4 hours of engineering development / test
• Resulted in the ability to meet the sales request w/o the need for an additional system
• Systems of this complexity can easily exceed $1MM in total cost
– This approach completely changed the business case for this opportunity
“Save Result” “Check EMPI” Engineering effort
Baseline 450ms 130ms 1 hour issue ID
Round 1 250ms 17ms 2hrs to corrective action
Round 2 150ms N/A Next Morning~ 2 hrs to Corrective action
Impact 200% Increase 665% Increase
Second Round

Recap
• Started with a small well understood solution to stabilize our
AppDynamics understanding
• To understand impacts to inter-team communication we expanded
to a more complex product and folded AppDynamics into the
product development progression, development through
production.
• To better understand the full range of AppDynamics capabilities we
expanded to a larger product base, mostly at the development
level.

2016 #1 Top Ambulatory EHR Vendor in user
satisfaction poll from Black Book Rankings
Company benefit
• Simple Identification of performance
limitations
– By the most junior of engineers, freeing SME to higher level
tasks
– Preproduction deployment optimization
• Direct and measurable performance impacts
– Client satisfaction increase where response time
matters
• Feedback from Production systems into
Development
• Time to Root Cause, Time to Corrective actions
• System performance
– Ability to rapidly measure overall performance of design options
– Optimize solution SW and HW to maximize value to the company
“AppDynamics is the only tool that we have that
looks at the specific WCF calls to identify
Current and Future issues, in the Web Code”

Please give us your feedback—Session B5619
• Complete the online survey you'll receive via
email later today or via text at:
Text this number: 878787
Text this word: APPSPHERE
• Every time you submit a session survey, your
name will be entered in a random drawing.
We're giving away Amazon Echos
to 5 lucky winners!
• Thank you for your input
APPDYNAMICS CONFIDENTIAL AND PROPRIETARY 17
Win!

How Allscripts Streamlined Root Cause Analysis - AppSphere16

More Related Content

What's hot

Viewers also liked

Similar to How Allscripts Streamlined Root Cause Analysis - AppSphere16

More from AppDynamics

Recently uploaded

How Allscripts Streamlined Root Cause Analysis - AppSphere16