SplunkLive! Detroit April 2013 - Domino's Pizza


Published on

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • The content marked in red and orange needs editing or additional detail.
  • SplunkLive! Detroit April 2013 - Domino's Pizza

    1. 1. Copyright © 2012 Splunk Inc.Domino’s PizzaRussell Turner & Seth Porta
    2. 2. Domino’s Pizza OverviewWorld leader in pizza deliveryMore than 10,000 corporate andfranchised stores in US andinternational marketsOnline ordering, apps forAndroid, Kindle and iPhoneFounded in 19602012 sales: $7.4 billion 2
    3. 3. Our Background and RoleRussell Turner, Manager of Site Reliability EngineeringSeth Porta, Site Reliability EngineerOur Team is responsible for ensuring our online customers have the bestexperience possible– Maintain ecommerce uptime– Middleware, infrastructure, servers, global and local load balancing– Architecting and deployments of new business initiatives– Closely tied into development workflow 3
    4. 4. How We Started?POC’ed Splunk for the first time in 2009 (within the Infrastructure team)Needed a solution to analyze and aggregate logging data from our OS (Linux and Solaris) andmiddleware in a timely mannerInfoSec team used HP ArcSight for log aggregation, but Splunk offered the following advantages:– Faster and easier searches in Splunk– Real-time insights– Better reporting with Apache access logs– Much faster alerting in Splunk– Cost and scalability– Ease of deployment 4
    5. 5. Splunk Usage at Domino’sRange of uses from monitoring to business insights including:– What we are selling, orders per minute, coupon usage, etc.– Online ordering trends, efficiency of marketing promotions– Splunk provides us answers 24-48h prior to analysis from our data warehousing tools– Significant reduction in troubleshooting time– Streamlined developer insight into debugging development code– Overall Order to Cash system health monitoring 5
    6. 6. Splunk at Domino’s TodaySplunk deployed across two data Two indexers with Distributed Searchcenters (live and failover) MI Datacenter KY DatacenterFour different productionenvironmentsTeams using Splunk: SiteReliability team, InfoSec and webdevelopers 360+ forwarders25-40GB data indexed per dayDozen unique users per monthSplunk Apps: DeploymentMonitor, Google Maps 6
    7. 7. Results with Splunk Proactive Operational Reduced MTTR Cost Savings Alerting and Intelligence Baselining Saved $300,000 vs. Real-time alerting = alternate APM tools Tracking: businessIssue resolution proactivity relevantfrom 2-3 hours to Engineering resources information, trends,less than 5 minutes. Historical baselining freed up for other promotion has been huge needs. success, customer behavior 7
    8. 8. 8
    9. 9. Using Splunk For Data Correlation?Domino’s Splunk Environment: Logs from 800 Virtual and Middleware, Apache Web Proprietary Database Server Logs Application logs physical System Logs (over 20 types) servers, Linux/Sola logs ris Before Splunk Enter Splunk Gathering logs manually “Million times easier with Splunk” Sifting through aggregated Java messages from Proactive alarms alert us to dips in our sales middleware (Grep) Baselining and trending Reactive 9
    10. 10. AHA! Moments with SplunkRussell: Seth:IT team started taking Splunk “When asked to show response times ofhome and working on their own data stores, I was able to providetime with Splunk. answers within 30s (just pipe one search“Splunk is much bigger than into another) and got a list of storesmonitoring tool. We are sitting on instantly. In the past we had to work ona gold mine of data!” that for weeks.” 10
    11. 11. Splunk For Operational Analysis of Payment ProcessingMeasuring response time forvarious order channelsInstant analysis of cash vs.credit card orderingperformanceTroubleshooting cardprocessor issues 11
    12. 12. Splunk for GEO Sales Tracking• Splunk RESTful APIs integrate with Domino’s GEO sales tracking applications (Java based)• Sales monitoring by regions• We have been able to identify ISP outages in certain regions 12
    13. 13. Splunk for Domino’s MarketingBefore Splunk• Someone at midnight pulling data and Splunk dashboard to track 50% off crunching numbers daily online coupon promotionResults• Automated information• Report submitted to our leadership team, including the CIO and CEO• Monitoring promotion success in real-time 13
    14. 14. Best Practice RecommendationsBuild a full blown POC to demonstrate Splunk’s valueFind real use cases to demonstrate Splunk’s effectivenessPlan your Splunk deployment (distributed environment); understandwhere config files liveSplunk documentation is helpful – use it!Leverage the huge online communityTake scoping seriously 14
    15. 15. Splunk at Domino’s: FutureCreate real time dashboards for any departments to view OLOhealth, not just reports mailed to the LTUse Splunk for more key performance analysesExpand Splunk Apps deployment: Linux and Unix monitoring, VMwareApp, F5 integrationOptimize middleware application logs for Splunk consumptionStart to leverage Splunk to monitor Corporate applications built on ourstack (Liferay) and Store health 15
    16. 16. SummarySplunk empowers us to better utilize our technology to gain acompetitive edgeHelps to ensure exceptional customer satisfactionEnables us to be agile and make marketing decisions based oncurrent promotion successSplunk helps us not just save cost but boosts morale as well 16
    17. 17. Thank You! 17