Recommended
Recommended
More Related Content
Similar to Duke at SplunkLive! Charlotte
Similar to Duke at SplunkLive! Charlotte (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
Duke at SplunkLive! Charlotte
- 1. Splunk@Duke Presented by: Jeremy Hopkins, Sr. IT Analyst
- 2. Agenda Introductions Early Operational Usage and Reactive Analysis Moving from Reactive to Proactive
- 3. • 15,000 Students • 3,300 Faculty • 36,000 Staff • Total of 68,000+ Active Users • University and Medical Center • Worldwide Presence
- 4. Splunk @ Duke University 200+ Indices & Sourcetypes • Syslog • OS (Win & *nix) • Web • Network • IPS/IDS/Firewall • Shibboleth • Mail • LDAP • VPN • Many More Departments & Uses • IT Security Office • System Admin • Messaging • Network • Database • Emergency Notification Tracking • Departmental IT Groups • Many uses by many groups
- 5. Forwarders set to "autoLB = true" Distributed Search Indexing Tier • 4 x Indexer • 5 TB local storage each Collection Tier • Various systems with forwarders • Syslog aggregator w/ Universal Forwarder for syslog events Search Tier • 2 x Search Head • Search Head Pooling enabled • Looking forward to Clustering LS Shared License • Department quotas Department Instance • Multiple "all-in- one" instance for different department needs Splunk @ Duke University
- 6. But I barely know you… “Who has authenticated the most in the last hour?” --Your Manager These questions become distractions, because we know it isn’t just one question… Where are the users logging in from? Where is SPAM coming from? Where is legit mail coming from? Log analysis from a shell prompt using the ancient sysadmin combo of: grep | awk | sort | uniq grep sasl_username logfile | awk '{print $9}' | sort | uniq -c | sort -n | tail -5 7 sasl_username=user1@oit.duke.edu 7 sasl_username=user2@oit.duke.edu 8 sasl_username=user3@oit.duke.edu 10 sasl_username=user4@oit.duke.edu 58 sasl_username=user5@oit.duke.edu
- 7. Dashboards to the Rescue Top 10 SMTP Logins using previous search example
- 8. Take NULL locations and puts a custom label on them Creates a Location label such as “Charlotte, NC” instead of a column for city and state. Top SMTP Logins Finds SMTP logins and builds a table with username, login count, and location eventtype=smtpauth | iplocation client_ip | eval Location= if(CountryCode == "US",if(City=="",if(Region=="","Unknown Location, "+Country,Region),if(Region=="",City+", ?? "+Country,City+", "+Region)),if(City=="",Country,City+", "+Country)) | eval Location= if(isnotnull(Location), Location, if(cidrmatch(”10.0.0.0/8",client_ip), "Duke - Private”, client_ip ) ) | stats values(Location) count(netid) by netid | rename … | table netid Count Location | sort - Count
- 9. Event types – SMTP Auth Converted to a Splunk eventtype index=mail host=mail* process=postfix/smtpd sasl_username=* – Duke created event types for various login events from various sources: VPN – eventtype=vpnlogin index=network sourcetype=vpn vpn_user=* vpn_inner_ipv4=* vpn_source_ip=* Shibboleth (single sign on) - eventtype=shiblogin index=idms_shib sourcetype=idp-process (shib_success=“[password]” OR SSO=“true”)
- 10. Play nicely with others In theory, we can join the various login types together to find all login events – eventtype=dukelogin is defined with the following search eventtype=vpnlogin OR eventtype=smtpauth OR eventtype=shiblogin Event types allow others to use your logs without knowing the specifics of your application aka: Collaboration
- 12. Fun with Maps Source of the mail we accepted as “not spam”
- 13. Guess Where our Spam Originates…
- 14. Was this really a DDOS? | inputlookup ddos_20150123.csv | geoip src_ip
- 15. Neat, but how did it help? • Plotting mail sources on maps provided a visual for management which allow implementation of geographically isolated mail flow and acceptance rules • Dashboards and eventtypes allowed the ITSO to quickly see account abuse and provided the groundwork for collaboration with other teams
- 16. And then this happened… Phishing to Fraud • Phish targeting 600+ faculty/staff • Typical phish emails (nothing overly sophisticated) – Pay Raise, Login Verification, etc. – Cloned Login Page • Compromised accounts used to access HR/Payroll sites • After successful login, bank routing numbers for direct deposit changed • Reports of monthly salaries not received Source: The News & Observer
- 17. Example of Phishing Email Received Clicking here leads to URL on next slide A pay rise… interesting.
- 18. Link from Phish in Previous Slide Believe it or not, Duke does not own nl- tour.ru
- 19. Gone phishin’ Example of Dashboard to Record Email into Phish Tracking Lookup Table Search Duration Actual Subject of Email How we identify a particular campaign Your DUKE Pay Increase Pay Increase - 20141006 Adds to PhishList lookup table
- 20. User Lookup in the PhishList | inputlookup PhishList.csv | search To=dukeuser@duke.edu
- 21. Phishing DashboardAggregate of PhishList Data with Search Panel
- 22. Once the Bleeding Stopped… • How do we utilize Splunk to become more proactive? 1. Look for “non-Duke” IPs with multiple user logins to HR/Payroll system 2. Non-Duke IPs with multiple user logins regardless of destination 3. Query the number of cities an account has logged in from in past 24 hrs 4. Query the number of countries an account has logged in from in past 24 hrs
- 23. • Multiple Iterations of Direct Deposit Phish • Team moves forward from authentication only investigations • Chum Accounts to help locate attackers early This type of visibility into logs just did not exist for the ITSO prior to Splunk The Fight Continues… It becomes organizational, and not just the responsibility of security • App developments • MFA
- 26. Example Using Radio Buttons <fieldset> <input type="text" token="search_netid"> <label>NetID</label> <suffix/> </input> <input type=”radio” token=”mail_button”> <label>Mail Logins</label> <default>YES</default> <choice value=”index=mail”>YES</choice> <choice value=”index=NONESUCH”>NO</choice> </input> <input type="time"> <default>Last 7 days</default> <label>Time</label> </input> <fieldset> ~ <searchString>$mail_button$ host=”mail-gw-*” user=$search_netid$</searchString>
- 27. Example of Combining Details Using Case | eval combined_event_type=case( isnotnull( vpn_login ), “VPN Login”, isnotnull( sasl_username ), “Email”, match( event_action, “lock” ), “Account Lock”, match( event_action, “unlock” ), “Account Unlock”, isnotnull( clientip ) AND isnotnull( user ), “WEB HIT”, 1=1, “-” )
- 28. Benefits of Splunk to Duke • Splunk allowed Duke to begin leveraging multiple data sources almost immediately with very little ramp up time • Detailed information on recipients of phishing messages is now available to Security in minutes instead of hours • Splunk has allowed Duke to more than double the number of compromised accounts we detect and lock each month • Splunk provided the ability to create a custom SIEM like solution tailored to Duke’s needs
- 29. • Use Splunk to bridge the gap between teams and log knowledge • Use event types, macros, and saved searches to make your long crazy searches usable by others • Use the eval command to create custom algorithms • Continue to educate your users about Phishing Key Takeaways
- 30. Thanks for Listening! Email: Jeremy.Hopkins@duke.edu Socials: hopkeno Lets Go Duke!
Editor's Notes
- Good Morning! I’m Jeremy Hopkins and I’ve been asked to speak to you about how Duke University has leveraged Splunk in our environment.
- While Duke’s use of Splunk is much more intensive than what we’ll discuss in this presentation, here’s the agenda for this morning. We’ll go through the Intros and “Abouts”, then move on to Duke’s early operational use of Splunk for reactive analysis. And finally give you a glimpse of where we are today as we’ve moved, and continue to move from reactive to proactive.
- For you Architect and Infrastructure folk, here’s a look at our deployment. Starting at the bottom we have about 3000 forwarders sending to 4 core indexers. At the top we have 2 pooled search heads. We split the license between the University and the Medical Center with DU using about 200GB/day of our 250GB portion. Additionally, we have about 16 departmental instances.
- Prior to splunk, we answered questions with long command lines and/or scripts of one flavor or another. While this works, it can be limiting, and not easily digestible by management.
- Example of our Top 10 Logins table using custom built “Location” field that contains GeoIP Data and Duke specific labels
- This search is really application independent because the heavy lifting is done in the event type. What’s unique about this search is the eval to put a combined and/or custom location label in the table instead of columns for City, State, Country, etc. The first eval says: If the Country Code is US then the Location should show as City, Region (state), otherwise the Location should be City and Country. The second eval looks for a null Location value and assesses it for our private IP space. If the IP matches our private space (10.0.0.0/8) then we set the location to a custom label.
- Example of eventtypes. We noticed that we were doing the same types of searches over and over again. These searches produced results that were a type of event or action, such as a login, but weren’t usually specific. Ie: show me all logins in application X. So we created event types these, Mail, VP, and another for our Single Sign On system (Shibboleth).
- We then created a rollup eventtype of “dukelogin” which is simply a search of the various login eventtypes: vpn, smtp, and shib
- It was about this point where this xkcd comic really was how we felt. Not so much the elaborate fantasy scenarios, though some in mgmt did come up with some pretty radical scenarios they wanted us to tackle, but more so that we could swing in, splunk a few things, and swing out into the sunset as a hero.
- So, we had some fun with maps! What this map tells us is that most of the mail we accept and consider legit originates from the US.
- This is the inverse of our previous example. It shows the locations where we block mail or drop it as Spam. The combination of this map and the previous map were used to sell to management that we needed to treat inbound mail originating from different places differently. As a result, we implemented a geoip based mail system that enforces much stricter connection rates and delivery limits from non-United States based IP addresses.
- Mid-January we underwent a DDOS attack. I don’t have my load balancer connection data readily available to Splunk, but I was able to grab a snapshot of the connection table from the load balancer. Think of this similar to a netstat. I took this data, and parsed out the connecting IP addresses that had idle/abandoned connections to the site being attacked and fed that into the Google Maps app in Splunk. The result is a nice map that shows where these connections originated from. This is a good example of a way you can leverage splunk to present raw data visually.
- What had we learned thus far?
- http://www.newsobserver.com/2014/01/07/3512350/email-scam-nets-thieves-a-months.html (Jan) Original phish on 11/19/13 First direct deposit issues reported after December pay period (early Jan) Add’l attacks occurred between Jan-Mar and then again resurfaced in July http://www.newsobserver.com/2014/07/22/4021696_duke-warns-workers-of-email-scam.html?rh=1 (July)
- Screen shot of Phishing Dashboard. Email Subject is the subject of the email you are search for. Phish Name is an arbitrary label that we will use to identify the phishing message. The Add to PhishList button exports the results of the search to a Splunk Lookup Table for ingestion by our Security Office.
- Example of searching the PhishList by To address. The columns you see here unpopulated “affiliation” and “netid” are filled for actual users based on other lookup tables that we keep about our users.
- The Phishing Dashboard pulls in statistics about the unique phishing messages and also allows for a quick user search to see which messages a specific user has received. The bottom of the dashboard compares data from another lookup table which contains a list of direct deposit changes. It looks for users that have had their Direct Deposit info changed after receiving a phishing message. This Direct Deposit Search was the first visualization of the data, but a very rudimentary way of detecting these events. The Messaging folks then punted to the IT Security Office.
- At this stage we begin asking questions about what we can do w/ this new visibility of Shib and web logs. We then developed some saved searches and reports to monitor these things and called it a high risk report. That report included answers to the questions here, but also incorporated the information we had been collecting about who was receiving phish messages and who has had their direct deposit changed.
- During Q1, we saw multiple iterations of DD phish, which helped us tweak our Splunk use One of the 1st changes, utilize Web Hits, not relying specifically on authentication logins just b/c someone logged into the site, doesn’t mean they went to the specific page for DD changes Additionally, web logs have referrer info – now we can see IP info of attackers who’ve cloned our SSO page plus we can look for our users visiting the cloned page -- lock accounts Lastly, we begin using Chum accts to look track attacker activity App developers made changes to help thwart fraud and MFA became a much higher priority across campus
- User Investigation form in use today.