Did you use Splunk for your data center move at all? (John Bauer): It’s currently in progress. Yes, basically – oh so, yes, I’ll tell you about the whole top down view. We’re basically planning on having Splunk be the main aggregation point for all these divergent systems that kind of compete in the same space. So we use (Scum), we use SolarWinds, we use vCOPS for VMware, we use Nagios internally in some places. So what I’ve done with like SolarWinds and vCOPS is now I trap alerts to Splunk so, even though I don’t have like the fully enriched highly finely grained metric data, I at least have, when those other external systems notice events I get traps about them so I can – right now I’m not using that data but it’s basically – Splunk will be the central like reporting and like a stash-board spot to correlate all those things – all these events coming in from all these different systems. And actually, I have a white-board screen that has a lot more points but I could basically put together like a screen shot for you.
SplunkLive! Minneapolis April 2013 - Digital River
Digital RiverFounded in 1994Global leader in e-commerceCloud commerce, marketing and paymentsolutions1400 global employeesBuilds and manages online businesses formore than 40,000 software publishers,consumer technology manufacturers,distributors, online retailers and affiliatesClients include Microsoft, EA, Adobe,Autodesk, Nuance, Logictech, nVidia,Trend Micro, Kaspersky, Capcom, Square Enix2
About MeComputer science background – 15+ years IT and misc developmentApplication Security Analyst at Digital River for 2 years– Static and dynamic analysis of our applications– Manage our web application firewalls and security policies– Help push discovered defects through the development process– Security researchSystems Administrator at Digital River for 1.5 years- Monitor and assist with troubleshooting issuesHave become one of the two Splunk owners in the organizationPhotographer, inventor, security, 3D Printer and Unix enthusiast. Using Linuxsince ’93, Slackware 1.0. Irony, Digital River acquired ftp.cdrom.com withSimtel3
Splunk for Real-time Monitoring of AttacksWeb Application Firewall (WAF)– System can only store 300,000 events (3 days worth of events)– Needed at least a years retention of data– Filtering capability very limitedSplunk natural fit for logging these security events– Splunk tracked security events across our data centers– Not only logging security events for historical purpose but also real-timedashboards– Work-flow action to understand exact request attacker used and how webapplication responded4
Correlated SearchesCurrently correlating searches based on IP AddressInvestigate IDS events and WAF events from same IP address ormultiple attacks based on destination addressesJust scratching the surface6Most security tools have limited search capabilities. Splunk makes any search youcan imagine possible.
Replacing Databases for Real-time MonitoringCurrently use many databases for logging– Biggest use is for tracking global site page hits– Millions of rows per day across 20+ databases with 70+ million rows eachPulled page hit data into Splunk with Splunk DB Connect– Showed Developer, Engineering and Ops Teams– Suggested to log application directly to Splunk– RESULT: real-time views of all traffic across all data centers8Onboarding new data into Splunk was easy with Splunk DB Connect
Global View of Malicious Attacks• Global view of maliciousattacks and where they arecoming from9• Heat map and alerts allow fordrill down to events forinvestigation
Help visualize the Business withReal-Time Sales DataBy generating events on the final check out page from the e-commercesystem, we can see sales as they happen across our globalinfrastructureReal-time trending items – what are people placing in their cart andpurchasing10
Real-Time monitoring of response times• Big push for better operational monitoring of external processingpipelines• External entity had problem with monitoring health of their environment• Creative solutions like watching data over the wire and generating eventsto Splunk with our Web Application Firewalls.12
Splunk for Central ReportingSplunk to be main aggregation point for divergent data1414SolarWindsSCOMvCOPS forVmwareSynergeticInteroperabilitySecurity EventsLegacyMonitoringLogs…everything else
App by Security Architect Matt Kirby, LogCompactorhttp://splunk-base.splunk.com/apps/69654/logcompactor15
Future• Database audit logs going into Splunk for reporting and trends• Use Cisco app for telecom/networking planning• Monitoring site deployments, 200+ per day- Visualizing operational change, performance and incidents in one view• Dashboards for executives/business data• Automatic validation of malicious attacks- Kick of a security scan or other analysis to confirm or reject malicious attempts16With Splunk it’s easy to bring together many disparate data sources.