This document discusses how a major hospital company has used Splunk across several areas to address key challenges and drive success. It summarizes how Splunk was used to: 1) Identify user errors, not device issues, that were impacting adoption of a mobile vitals monitoring program involving 5000 wireless devices. 2) Provide a unified, real-time view of connected medical devices and clinical applications that improved monitoring, troubleshooting, and compliance. 3) Rapidly develop an alternative to the vendor-provided but ineffective data analysis tool for their health information exchange initiative.

1. Copyright © 2012 Splunk, Inc.
Major Hospital Company

2. About
Founded in 1968
On Fortune’s “Most Admired
Healthcare Companies” list
Ranked on “The World’s Most Ethical
Companies” list
Computerworld “Top 100 best places
to work”
2

3. IT&S 2014
3

4. Connected Devices in Hospitals
Improve workflow and allows for efficiency in
hospitals
Used every day to document patient care
activities
Requires accurate data transfer to comply
with Meaningful Use provisions of HITECH Act
Connected devices lead to better healthcare
outcomes thru better quality management
“The FDA assures devices are accurate – we assure accuracy is maintained
through to the patient record”
4

5. Rolling Out Mobile Vital Signs Monitoring Technology
5000 wireless devices across 165
hospitals
65,000 patients monitored
17,000 nurses using devices
90% Vitals Signs taken electronically
The plan: Vitals automatically posted
in electronic medical record
“The size and implications of this project were larger
than anything IT&S had ever attempted.”
5

6. MDI Key Challenges
6
Troubleshooting to
address nurse
complaints about
devices
Changing nurse
behavior to ensure
effective use of
devices
Compliance with
meaningful use
requirements of
healthcare act

7. Why Splunk? (MDI)
Need unified view of 5000 devices
across 56 servers
Fill gap between existing monitoring
systems and out of box application
monitoring
Create consolidated view of
application
We Needed a Platform
7

8. Solving the Issue of Broken Devices
Widespread complaints of devices failing
Variation in complaints across facilities/servers
Minimal onsite support
Nurses will not troubleshoot devices—devices work or they sit
Nurses can be averse to and threatened by new technology
Splunk rapidly identified it was user error, not a device issue
“Splunk rapidly identified it was user error, not a device issue.”
8

9. Splunk ad-hoc Search Showed
• Only 30% of vitals were being taken via electronic method
• Devices were unusable due to inadequate charge
• Wireless coverage was inconsistent
• Devices responded slowly to loss of signal
“A combination of end user behaviors and technical issues were
responsible for poor adoption of the technology, threatening a
$7 million investment.”
9

10. Vitals Tracking-The Data
Device Logs
EHR Reports
AD Lookups
Application DB
Logs alone lack the context to provide meaning to the data
“Logs alone lack the context to provide meaning to the data.”
10

11. Driving Success with Splunk (MDI)
• Avoided $3 million incremental Bonus – Using the same data
spend on new batteries
• Reduces time to resolve and
correct issues by viewing all
dashboard data at once
• Enabled change in nurse behavior
with real-time clinician dashboards
• Cross-enterprise view of application
performance
• System problem alerting
• Automated service and support
records
“We were able to very quickly determine that there was nothing wrong with
the batteries at all, the problem was simply that they weren’t being charged.”
11

12. Splunk inflationary epoch!
Expanding Splunk to become a major application management resource
Expand to other teams (Security, Networking & Infrastructure)
Splunk is becoming the cornerstone for management of clinical
applications and a trusted single pane of glass for the enterprise
“Splunk is our enterprise management platform for clinical
informatics.”
12

13. Health Information Exchange (HIE)
HIE is one of the most important aspects of Meaningful Use Stage II
At the point we began work on its own HIE, the failure rate of Regional
Health Information Organizations (RHIOs – state, regional, and
community-based HIE initiatives) was around 30%.
After over a year in planning, vendor selection, and build/development;
the finished product was ready for testing
Testing did not go well…
13

14. HIE Key Challenges
(WE) being the largest healthcare organization in the US, once again
managed to overwhelm all aspects of the vendors solution
– 6 Million transactional messages per day
– Some messages can be in excess of 40,000 lines of text
– So many transactions and events that logs were being overwritten every 6-8
hours
If something happened to crash the system after 5PM, by the time the
testing team got back to the office the next morning; the cause of the
system crash had likely been lost forever
14

15. Enter Splunk (HIE)
It took almost a week to comb through 45 servers, all different
application subsystems, for logs and to classify those logs
Due to the “unique” (unstable) circumstance of the application,
deploying universal forwarders to those systems was strictly forbidden
– Two servers were commandeered (slightly illegitimately) from a past initiative
– Splunk was installed on commandeered servers , actively monitoring and parsing
out “junk data” and PHI for over 300 logs via UNC
– This was a substantial leap forward, logs were no longer lost!
15

16. 16

17. Why Splunk? (HIE)
Splunk released the DBX application
– Directly connected to the vast number of databases contained within this HIE
solution
– We now have a truly complete whole system view in real-time
This holistic view of HIE data empowered our team to see problems in
real-time and begin doing something about it!
– Used correlated WMI and database data to compare 4 key system health metrics
 Incoming Rate
 Completed Rate
 FIFO Queue
 Stuck Messages
17

18. HIE Data Analysis Tool (DAT)
Plagued with integration issues there was an immediate need to get
access to a tool that allowed users to:
– Research failed message transactions
– Investigate slow end-user response times
If a clinician can access data from the source system and not access the
same data in this tool, adoption will be nonexistent
– Without clinician adoption of the HIE solution, tens of millions of dollars will be
wasted
18

19. HIE DAT Key Challenges
HIE solution has its own tool built in, making it the default choice for the
task.
– There is a major flaw with it though, it doesn’t work.
 No timeline for a fix
Without a data analysis tool for constant monitoring of system response
times and failed message transactions the reliability of the entire
solution was in jeopardy
– In depth auditing would be unattainable
– Data integrity could not be assured
– Meaningful Use attestation would be inconceivable
19

20. Enter Splunk (HIE DAT)
In a world where vendors can’t… Splunk can and does, quickly!
– Splunk DB Connect app leveraged to pass real-time database queries to various
databases
– Uses Vendor supplied Queries
 This data was made searchable via a series of dashboards, letting the individual
doing the research drill down to individual message ID’s with 0 impact to the
production databases
Development time for this solution, planning to production was 6 hours
with a 30 days historical backload
– Comparable solution from internal DBA group?
 3-6 Months
20

21. Driving Success with Splunk (HIE)
Splunk was paramount to the success of HIE
– Splunk empowered rapid root-cause determination
– Facilitated application health metric monitoring
 Alerting of trends that were actionable
– Splunk empowered rapid development
 Reduction in DAT development from a minimum of 3 months to 6 hours
– 8,000% Reduction in effort
– $23,700 cost avoidance
The failure of this application could have led to millions of dollars in lost
reimbursements and millions more lost in man hours and development
work
21

22. Payment Card Industry (PCI) Compliance
The Payment Card Industry Data Security Standard (PCI DSS) is a set of
requirements designed to ensure that ALL companies that process,
store or transmit credit card information maintain a secure
environment.
– Penalty
 $5,000 to $100,000 per month for PCI compliance violations
22

23. PCI Compliance Key Challenges
RSA enVision solution is expensive
– Roughly $200K/year in support costs
RSA enVision solution requires hard coded inputs
– Will not automatically adjust reporting as new methods are added
 Requires Vendor PS to make supported modifications to add additional inputs
Initial setup took 6 months development
23

24. Enter Splunk PCI Compliance
Clone of RSA tool created in 2 weeks, with the following enhancements:
– Completely automated
– Real-time data
– Designed to be dynamic
– Added UI access
– Drillable data model that takes end-user context to intelligently filter data
24

25. Driving Success with Splunk (PCI)
Annual cost avoidance of ~$200K
Byproducts:
– Real-time IVR dashboards tracking revenue
– Real-time transaction failure notification
– Real-time potential fraud notifications
25

26. 26

27. Executive Dashboard
Team of 3 using C# building custom solution for over a year
– Significant strides
– Incomplete
Created complete POC solution in Splunk with 24 hours effort
27

28. 28

29. Doctor Access Login Time Monitoring
Problem: Doc’s login in times to internal systems via Citrix were taking
to long and there was no way for IT to monitor and measure the time it
takes (Doctor Experience is CIO level issue)
Solution: Splunk now measures the exact experience that Doc’s are
seeing on their screen
– Citrix XenDesktop, Citrix Broker, Juniper, Netscaler, SSO
29

30. Splunk Growth Plans
Licensing:
– Increase License by 430% in the next 2 weeks
– Increase License by an additional 256% by Jan 2015
Hardware:
– Purchase dedicated storage solution for Splunk in excess of 3M IOPS by 1st
quarter 2015 (1PB)
– Purchase additional 4 double spec’d indexers by 1st quarter 2015
30

31. Enter SkyNet
Coming Q2 2015
31

32. Thank You!