Managing Security with Splunk Enterprise

4. 4 My Role Member of the CIO office Responsible for all privacy in IT security for University of Illinois-Urbana campus Community of 45,000 students and 13,000 employees Monitor breaches and implement response Work to centralize data from numerous stakeholders

5. 5 Where we were…

6. 6 Where we were… Built our own Syslog environment Searching logs was a task in and of itself Needed system would layer on top of back-end logging environment Enter Splunk>

7. 7 Enter Splunk> Primarily security applications – Incident response – Network baselines Secondary uses – Operations – Analytics

8. 8 Use Case: Account Compromise External/Splunk notification ID look-up and research Flagging malicious activity Detected 1500% abuse

9. 9 Use Case: Copyright Infringement Notification • Investigate notifications received from music and movie industries • Use Splunk to validate illegal activity and automate response • Free up human capital to focus on incident response and security Screenshot here

10. 10 Use Case: Phishing Campaigns • All mail relays and data integrated in Splunk • Mail admins and security team easily correlate problems – Correlation prior to Splunk took overnight – Now takes a few seconds with pre- indexing in Splunk • Next steps to control phishing attacks Screenshot here

11. 11 Crime Stoppers! Interfaced with local police department and campus police Assisted in locating suspect of off-campus crime 20 minutes to locate suspect – 19 minutes to find log owner (logging source outside of Splunk) – 1 minute to find suspect after log was integrated into Splunk

12. 12 What’s Next Continue to add alerting and log sources to correlate across Automation with infrastructure (network defends itself) Expand use outside of IT – Business side of the house – Research community

13. 13 Top Takeaways Provides that missing insight Managed correctly = life becomes easier Distribute the power

14. Thank You

Editor's Notes

“Wild West” May not be large compared to corporate environments, but we are a city (police, housing, farms, streets & buildings, labs, etc.) open in nature You had to keep in mind that not only are we running enterprise-level ERP systems and exchange environments and various systems like that, but I also have a pocket of researchers on this network that are doing very interesting things and very cutting-edge research. I think we have a top-ranked computer science program in the world. So they're always taking our network and our research network to the extreme. How do I differentiate between what they're doing in research versus some sort of assault from a country? It's hard to determine at times. So we had all of these logging environments (that are not all centrally controlled) and we were trying to figure out a single scalable solution where because I don't control, because central IT doesn't control the IT resources, we needed a way for these systems to be able to communicate back to a central environment and then we could do analysis on them.
Systems, name the OS we have, probably have some that don’t exist anywhere else Pockets of logging already happening – enter syslog environment to collect high priority logs (allow access to log owners & security) Terrible front end, hard to search, long wait times for results (24, 28, 72, 1 week),– example of email/lync client upgrade – targeted email 15 minutes to put search together, 3 days to get results Hopefully you didn’t have a typo in your search and assume you didn’t have to correlate results against another log source Needed to do something better – time to find a good front-end Evaluated open source and commercial products, Splunk floats to top – becomes option as part of I2 net+ offering
The Splunk purchase was driven through security. Sell it to the organization operational uses and research value Tip every meeting you attend say Splunk can do that and within a few weeks everyone will say that for you Ops side = web analytics, wireless usage, target communications lytics and some research. System performance, outage research and alerting (ex. Shib logons 70k concurrent connection to less than 7k in 15 minutes, somethings going on That was sort of a secondary . . . those were some of the partners we formed inside of the building to move forward. But most of my experience and most of my conversations come out of using the basic tool as an incident response and sort of network-based lining (?) tool from a security standpoint.
We realize an account is compromised either from an external notification or from using Splunk and be able to very quickly look up the individual's ID and get a sense for what their network activity looks like. We can very quickly correlate to user before the compromise and try to isolate where the compromise was coming from and what was obviously done with the account after it was not owned by the owner. Lately we take a various set of our log sources . . . any log source where an individual has to be present to log in to. Ex. For some of our Web authentication, our VPN connections, a couple of these other places; we actually are taking that logging information, correlating it with their geo location and we have a series of rules set up that say "If an individual is connecting to a university resource and they're in three countries simultaneously or within a certain time frame. Or if the person appears to be traveling greater than 600 miles per hour" and a couple of other parameters, we basically assume the account is compromised and then Splunk fires off an alert into our picketing queue and then we follow our procedure to scramble those credentials and follow up with the end user.
Our office deals with passing along copyright notifications from individuals who have illegally downloaded copywritten material. Notifications received from the music and movie industry. We use Splunk now to help automate that response. So when those notifications come in, it pulls all the logging sources. It validates whether or not the individual did actually download that material and then kicks off another process which allows us to automate notification and resolve those issues. We're almost there. One more separate way that we've almost basically taken the work of an individual away by using Splunk. It just allowed us to reallocate that person to actually do incident response and security work.
51B firewalls (6TB), 33B (4.6TB) DNS, 24B (4.3TB) internal email, 6B mail relays (0.7TB) We've been able to use Splunk because all of our mail relays and all of that information is in it. Our mail environment is architecturally segregated across around 100 systems to run it and trying to correlate those logs to the past has been a pain. Now that they're all in Splunk, our e-mail administrators and our security team can easily correlate problems, whether it be from phishing or just delivery issues within . . . I'm looking at the stat here. I'm trying to see. They've got a new time frame. Again, it takes overnight. Previously, it used to take overnight to run with the pre-indexing done in Splunk now in a few seconds. Mail administrator is starting to automate some of that as well. It just gets easier and easier. Historically we were only able to minimize the impact that we feel and that with a couple other tools, we're able to spend less time cleaning it up. Recently, we were a victim of a pretty good spear phishing attack. it was after identity theft around payroll data. Luckily we were able to stop it, but we are now using Splunk as our alerting mechanism. So we know the IPs that this was coming from and we know what the pattern looks like. So we essentially have queries sitting there that look over our logs and say "If anything like this hits again, it's an immediate notification if you're on queue.” We haven't seen any repeat offenders, but again, that was something we couldn't subdue very well before using Splunk.
Insight – Security, IT operations, research efforts – If you have the data it likely can answer the question, if you don’t know the question to ask sometimes splunk will even guide you down that path Life easier – manage correctly means devote person to tool develop and use. reduce time looking things up, but as you mature use you start to automate reporting and eventually maybe even actions Power – get the tool in as many people’s hands in the organization, show them how they can use it. Maybe they stop bugging you with less work, maybe they gain insight and move the organization forward (side note helps with funding..give example of enterprise reporting tool and not having someone work it.

Managing Security with Splunk Enterprise

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (7)

Similar to Managing Security with Splunk Enterprise

Similar to Managing Security with Splunk Enterprise (20)

More from Splunk

More from Splunk (20)

Recently uploaded

Recently uploaded (20)

Managing Security with Splunk Enterprise

Editor's Notes