2. 2
Agenda
About me and The Vancouver Clinic
How we started
Splunk deployment
Splunk use cases at The Vancouver Clinic
Best practices
3. 3
Vancouver Clinic Overview
Serving the Southwest Washington
since 1936
Locally owned and governed
Comprehensive and high quality of
patient care
4. 4
My Background and Role
IT team supports all clinic’s IT operations needs
– Infrastructure monitoring and sizing
– Root cause analysis
System analyst at The Vancouver Clinic
– Integration of medical systems
– Improvement of business processes
5. 5
How We Got Started
Needed real-time solution for event logging and proactive
monitoring across the entire IT infrastructure
– Predicting failures and understanding performance of the systems
– Before Splunk, slow and manual process of collecting event data from
multiple client machines
– Centralized logging for PCI compliance
Started with Splunk two years ago for medical records privacy
monitoring
– Pioneered using Splunk for patient privacy monitoring in PNW
– Huge interest from other clinics and hospitals in this use case
6. 6
Splunk at The Vancouver Clinic Today
6
Splunk data types: Firewall logs,
DNS lookups, application logs,
Windows events and performance
logs, MS SQL logs, Infrastructure
syslog, SAN metrics, etc.
IT operations team is the main
user of Splunk
Active users #: 15
Splunk Apps deployed: Windows
Infrastructure App, DB Connect,
Splunk on Splunk, Palo Alto,
Citrix, Symantec
2 search heads
2 indexers
>1500 forwarders
7. 7
Planning for Expansion
• Estimating capacity
growth and proactive
expansion plans
• Disk latency and IOPS
monitoring
– Identifying causation
• Disk Group Usage
balancing
9. 9
Capacity Planning and Database Optimization
• Proactive capacity planning and
estimating database growth
• Trending changes in load times
• Identifying anomalous load timings
• Visibility into how long SQL queries
are taking
• Correlation of High CPU usage to
poorly written SQL queries
10. 10
Servers and Applications
• Monitoring VMWare Clients
and Hosts
– CPU
– Memory
– Disk Usage/Performance
• Exchange Performance
• EMR
– BLOB storage
• Citrix PVS
12. 12
Patient Privacy Monitoring
• Splunk helps us comply with
patient privacy laws
• Highlights anomalous patient
record access
– Employee accessing medical records
without authorization
– Prior to Splunk lack of visibility
• Other “turn-key” tools we
evaluated were expensive and less
flexible
– Still required huge time investment
• Interest from other regional
hospitals and clinics
13. Splunk’s Value for The Vancouver Clinic
• Splunk easier to use and more cost effective
• Splunk is flexible and we can modify reports
Saved over 50K
• We are able to catch problems proactively before they happen
• Increased confidence and satisfaction toward our IT teamProactive Monitoring
• Our data is centralized
• Less need for hunting in various locations for log data
• Ability to see trends/patterns in our logs
Faster Support Response
14. 14
Lessons Learned
Value of Splunk community
– Users on answers.splunk.com are very helpful
Make your custom logs more Splunk friendly
– Easier to index key/value pairs
Trust your Splunk data
– Hard to break out of old habits of going to the source.
– Much easier to correlate disparate data within Splunk.
Re-evaluate your Splunk data every once in a while
15. 15
What’s Next
Extending Splunk deployment for proactive monitoring
– Building more alerts and dashboards
Creating executive dashboard and reports
Look into the SDK’s and REST API
Editor's Notes
Please insert a better picture if you find it and please review
Feel free to edit the red text
Could you specify which applications logs?
Example – AV issue that caused Server disk latency to go high
Example – Helping report writers identify poorly written queries that are affecting machine performance
Example – Nurse reaction to proactively being called by IT
If you have any of the AHA moments, could you add? Could be other team members
I added this one from our discusison or other
Example –
Feel free to edit XYZ
Mention the nurse reaction when you fixed here problem.