This can serve as an agenda slide…I’ll walk you through how Splunk helped us initially to simply find the data we needed to do our jobThen become proactiveTo deliver a better customer experienceand now Splunk has helped us gain internal credibility (and additional work) by delivering dashboards that showcase the data driving our business.
Other vendors, LogLogic, SenSage and Snareserver offer a lot of canned reports and forms, but to truly understand our environment—and get up and running quickly, Splunk was the best answer.
• Offering widest range of benchmark futures and options products available on any exchange, covering all major asset classes• Interest rates, equities, FX, commodities, and alternative investments such as weather and real estate• Joint venture owning 90% of Dow Jones Indexes• Our customers include brokerage firms, banks, hedge funds, pension funds• We monitor network infrastructure and the artifacts our apps generate 2
About Bob BeardUsing Splunk for the past 6 yearsDirector, Network EngineeringDesign and implement monitoring solutions for applications andnetworking for the ExchangeTeam responsible for Monitoring for fault tolerance andperformanceServed in various management and engineering roles for 20+ years 3
Before and After Splunk Problem • No solid log collection platform • Multiple Monitoring solutions • All visualizations or Analytics required custom programming Results • Search functionality allows for quick and easy isolation • Single log monitoring infrastructure for all IT and Executive staff 4
Our Splunk Architecture• Our Splunk • 3 Data centers • 56 Indexers • 2 Search heads for ad hoc searches • 2 Search heads for Real time searches • 1 Search head for saved searches and alerts • 2500 Forwarders+• 1 TB per day
Real-time Analytics• Moving from reacting to proactive—avoiding downtime before it happens• Our apps teams don’t log in a standard way• Troubleshooting across lots of apps and log types very time consuming• Research took too long and was often incomplete
Real-time Dashboards Across Multiple Departments• Each team sees specific statistics/ • NOC dashboards/ reports/ searches in • Operations Center real time • Customer Service• Role-based access limits access to • Various specific indexes/ data Development Teams• People have direct access to the data they need• 300+ folks using Splunk Even senior management can log in and get value
Real-time Reactions• Tried months and several homegrown solutions to surface real time insight—with Splunk it was working in 1 week• Threshold-based alerting supports proactive customer engagement • Thresholds based on exchange activity • Could also indicate application problem• Lookup function makes it easy to correlate various alerts
Real-time Improvements• Match engine dashboards show Key Performance Indicators• Developers can see how changes they make to match engines affect performance in real time in a parallel environment
Getting “De-Used” to your Database• Anyone can search in Splunk• No need to learn a query language• Splunk encourages exploration which helps lead to other discoveries• Knowing sourcetypes just makes parsing through the data easier “The speed of finding answers in Splunk is amazing. I’m fascinated by how quickly it returns results from across our entire data set.”
AHA!Took me two days from nothing to a workingenvironmentThe ability to correlate the log typesThe ability to keep improving parsing over timeBeing able to pull reports for upper level management inminutes vs. taking hours to produce a single monthlyreport. 15
Deployment GotchasHave to restart the indexersSeparate search heads for real timeGood index planning for delegation of access 16
Looking ForwardIncrease number of search heads to 8Indexer replicationConnection poolingUpgrade to Splunk 5Search heads pooling 17