Justin Hardeman is a Unix administrator at Availity LLC, a company that processes over 2 billion healthcare transactions annually. He has over 5 years of experience using Splunk for monitoring Availity's large, multi-datacenter infrastructure consisting of 500+ virtual machines. Splunk has allowed Availity to move from a reactive to proactive approach by providing real-time visibility into issues, transactions, and workflows across their environment.

1. Justin Hardeman
Unix Administrator
Availity LLC

2. WHO IS AVAILITY?
Availity delivers revenue cycle and
related business solutions for health
care professionals who want to build
healthy, thriving organizations

3. AVAILITY KNOWS THE MARKET
MORE THAN 2 BILLION TRANSACTIONS PROCESSED ANNUALLY
connectivity
with every
health plan
nationwide
575,000+
active
providers
2,700+
hospitals
675+ vendor
partners
225 million+
consumers

4. 4
ABOUT ME
• Availity Employee since 2005
• Unix/Linux Administrator –
RHCSA, 3+ Years
• Resident Splunk Admin – 5+
Years
• Splunk “Revolution Award”
Winner - 2012
• Production operations –
monitoring, incident response,
24/7 support all using Splunk

5. 5
BEFORE SPLUNK
• Narrow focus – Application has degraded, no view into
the complete environment
• Reactive approach to problems
• Lack of visibility into root-cause of failures / issues
• Long resolution times – always needed the availability of
the right person at the right time
“Before I wasn't doing fun stuff like I am doing now with Splunk”

6. 6
ENVIRONMENT OVERVIEW
• 100 GB+ license
• Multi Datacenter
• 500+ Virtual Machines
• Splunk 6.2 in Production
• Splunk Applications – Splunk for Google maps,
Splunk for Initiate, Splunk for Unix, Splunk for
Squid, etc.
“It’s really not hard to get information in Splunk—and the value that
results is well worth the effort”

7. 7
FINDING FAULTS
PROBLEM:
Homegrown batch healthcare claim transactions
processing system using SQL query for queues without any
monitoring requiring hours spent on tracking bugs/issues
• Splunkgivesusreal-timevisibilityintoissuesastheyarise
• Real-timealertinghelpsusmovefromreactivetoproactive
• ReportsandDashboardsshowrealtimetransactionsandB2B
workflows

8. 8
USABILITY CHANGES
PROBLEM:
Due to certain browser vulnerabilities, our site no
longer supports certain browsers. How successful
were the announcements and alerts we provided?
• Identifiedthetrafficoftheusersbeforeandafterupgrade
• Builtdashboardsdisplayinguserinfo,whatbrowsersthey
used,andhowmanyupgradedbasedonourmarketing.
• Abletoshare/displaythistomultipleshareholdersallthe
waytotheSeniorLeadershiplevel.

9. 9
REAL-TIME ALERTS
• Splunk helps us quickly identify problems
• Real-time alerts over a wide spectrum – Transaction,
Portal Logins, CPU memory, NFS write times, average
response times, etc.
• All real-time alerting in our own app
• An important component of troubleshooting
“The ability to search, report, and alert all from the same interface has
been one of the greatest values.”

10. 10
OPERATIONS CENTER

11. 11
OPERATIONS VIEW

12. 12
APPLICATIONS VIEW

13. 13
SPLUNK ADOPTION IS STILL VIRAL
• Role-based Dashboard views for different groups:
– CTO’s dashboards in Splunk show KPIs
– Sys Admin Group
– Developers–secured role-based access restricted to
certain indexes—speeding app troubleshooting time
– Power Users: Operations Center
“Splunk is THE tool that will allow you to begin your data
democracy.”

14. 14
FUTURE PLANS
• Increase usage of Splunk for Compliance
• Increase Splunk datacenter footprint
• Gain efficiencies by being more proactive in problem
resolution with Splunk
• Splunk for a private cloud?
• Splunk Stream
“It’s really not hard to get information in Splunk—and the
value that results is well worth the effort”

15. 15
THANK YOU
Justin Hardeman
Unix Administrator
Availity LLC