Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Copyright © 2013 Splunk Inc.

Cardinal Health
Patrick Farrell
Sr. Software Engineer
My Background and Role
Patrick Farrell, Sr. Software Engineer
– Resident Splunk Administrator and Champion
– Started using...
Company Overview
•
•
•
•
•

Founded in 1971
Over 30,000 employees
Headquarters in Dublin, Ohio
Ranked #19 on the Fortune 5...
Before Splunk
Manual search on 30+ servers
using Unix command-line
programs (Awk, Grep, Tail)
Operational support and
deve...
Splunk at Cardinal Health
Data sources
– Application Logs
– Access Logs, System Out,
System Err, GC, and other
custom appl...
Splunk Use Cases
“Splunk is our Swiss Army Knife”

Improving Root
Cause Analysis

Gathering Customer
Usage Statistics

Inc...
Return on Investment
“One of the most important benefits of
using Splunk from an application development
standpoint is ill...
Increased Efficiency
100+ developers on a single application,
there can be lines of erroneous code
– 1.2 million severe er...
Improved Systems Uptime and Performance
Writing Splunk friendly code
– Inventory Manager

Splunk’s search processing langu...
Inventory Manager Operational Dashboard
Improving Customer Satisfaction
•

Splunk alerts us when customers
see the contact help desk
message on our site
– Reach o...
Reducing Root Cause Analysis Time
Searching logs across many
application servers can take
hours. Remember, time is
money!
...
Reducing Root Cause Analysis Times
Normal Execution Scenario
Reducing Root Cause Analysis Time
Load Reduction Scenario
Reducing Root Cause Analysis Times
Abnormal Execution Scenario
Results with Splunk
Reduced
Down time

The most important
benefit to our large ecommerce application
is reduced down time....
Best Practice Recommendations
Splunk is an amazing platform as long as you are prepared for it!
Create a roadmap that outl...
Best Practice Recommendations
Generate a unique identifier for each transaction and write it to the log
as part of each ev...
Future Plans
Expanding use of Splunk to our Medical eCommerce Platform
Creation of additional operational and business das...
Thank You
SplunkLive! Customer Presentation - Cardinal Health
SplunkLive! Customer Presentation - Cardinal Health
SplunkLive! Customer Presentation - Cardinal Health
SplunkLive! Customer Presentation - Cardinal Health
SplunkLive! Customer Presentation - Cardinal Health
SplunkLive! Customer Presentation - Cardinal Health
Upcoming SlideShare
Loading in …5
×

SplunkLive! Customer Presentation - Cardinal Health

3,312 views

Published on

At SplunkLive! Columbus, November 2013

Published in: Technology, Business
  • Be the first to comment

SplunkLive! Customer Presentation - Cardinal Health

  1. 1. Copyright © 2013 Splunk Inc. Cardinal Health Patrick Farrell Sr. Software Engineer
  2. 2. My Background and Role Patrick Farrell, Sr. Software Engineer – Resident Splunk Administrator and Champion – Started using Splunk two years ago as a developer for our eCommerce platform – Responsible for Splunk administration, maintenance, custom application development, and dashboards – Splunk Community of Practice owner at Cardinal Health
  3. 3. Company Overview • • • • • Founded in 1971 Over 30,000 employees Headquarters in Dublin, Ohio Ranked #19 on the Fortune 500 Cardinal Health helps pharmacies, hospitals, ambulatory surgery centers and physician offices focus on patient care while reducing costs, enhancing efficiency and improving quality
  4. 4. Before Splunk Manual search on 30+ servers using Unix command-line programs (Awk, Grep, Tail) Operational support and development groups spent hours on root cause analysis and problem resolution No insight into customer usage of our applications No ability to be proactive with customer support
  5. 5. Splunk at Cardinal Health Data sources – Application Logs – Access Logs, System Out, System Err, GC, and other custom application logs – 25 individual source types – 250+ individual sources Indexer, Search Head, Deployment Server, and License Master 60 GB Per Day Splunk used in pre-production and production environments More than thirty individuals actively using Splunk on a regular basis Forwarder Forwarder 30+ Forwarders (5 Server Classes)
  6. 6. Splunk Use Cases “Splunk is our Swiss Army Knife” Improving Root Cause Analysis Gathering Customer Usage Statistics Increasing Efficiency Proactive Customer Support
  7. 7. Return on Investment “One of the most important benefits of using Splunk from an application development standpoint is illustrated by how it has helped us clean up our logging code.”
  8. 8. Increased Efficiency 100+ developers on a single application, there can be lines of erroneous code – 1.2 million severe error messages / hour Splunk is used to analyze application logs during performance/endurance testing The punct command is your friend Key benefit: Splunk helps us clean up our code – Capacity savings (storage, license) – Improved efficiency (speed) – Reduced spam
  9. 9. Improved Systems Uptime and Performance Writing Splunk friendly code – Inventory Manager Splunk’s search processing language allowed us to easily perform analysis once considered impossible from the Unix prompt. Analytics for: – – – – – Most active accounts Most invoked operations SQL Database contention Longest running operations Exceptions encountered
  10. 10. Inventory Manager Operational Dashboard
  11. 11. Improving Customer Satisfaction • Splunk alerts us when customers see the contact help desk message on our site – Reach out to customer immediately • Immediate support = happier customers = more revenue • Gathering customer usage data to identify which functionality should be enhanced or retired
  12. 12. Reducing Root Cause Analysis Time Searching logs across many application servers can take hours. Remember, time is money! Now an alert or search helps us identify most issues in seconds!
  13. 13. Reducing Root Cause Analysis Times Normal Execution Scenario
  14. 14. Reducing Root Cause Analysis Time Load Reduction Scenario
  15. 15. Reducing Root Cause Analysis Times Abnormal Execution Scenario
  16. 16. Results with Splunk Reduced Down time The most important benefit to our large ecommerce application is reduced down time. Every minute of down time results in a significant loss of revenue. Improved Customer Satisfaction Increase Efficiencies We were able to reduce our daily indexing volume by 3 GB by identifying and eliminating defects that produced in excess of 1.2 million severe events per hour. Reduced MTTR Application Enhancements We can determine the focus of future enhancements by monitoring how our customers are using the site. Likewise, we can also identify unused functionality. Thank you, punct! 22 Searching and Reporting Ability to drill down to specific areas and find issues in seconds instead of hours.
  17. 17. Best Practice Recommendations Splunk is an amazing platform as long as you are prepared for it! Create a roadmap that outlines how you intend to use Splunk and where you would like to take the product within your organization. Plan your environment and account for future growth (users, searches, license volume, hardware capacity, storage, etc.). 23
  18. 18. Best Practice Recommendations Generate a unique identifier for each transaction and write it to the log as part of each event so that you may easily identify all related events. Take advantage of automatic field extraction using key-value pairs or use a logging format such as JSON that can provide automatic field extraction. Capture execution time in log events for an added dimension 24
  19. 19. Future Plans Expanding use of Splunk to our Medical eCommerce Platform Creation of additional operational and business dashboards Evaluate the possibility of using Splunk in DEV and QA 25
  20. 20. Thank You

×