SplunkSummit 2015 - Security Ninjitsu
•
2 likes•844 views
SplunkSummit 2015 - Sydney, Canberra, Melbourne
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to SplunkSummit 2015 - Security Ninjitsu
Similar to SplunkSummit 2015 - Security Ninjitsu (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
SplunkSummit 2015 - Security Ninjitsu
- 1. Copyright © 2015 Splunk Inc. Original talk by David Veuve Senior SE, Security SME, Splunk Security Ninjitsu Andrew Phillips Senior SE, Splunk
- 2. Disclaimer 2 During the course of this presentaKon, we may make forward looking statements regarding future events or the expected performance of the company. We cauKon you that such statements reflect our current expectaKons and esKmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements, please review our filings with the SEC. The forward-‐ looking statements made in the this presentaKon are being made as of the Kme and date of its live presentaKon. If reviewed aTer its live presentaKon, this presentaKon may not contain current or accurate informaKon. We do not assume any obligaKon to update any forward looking statements we may make. In addiKon, any informaKon about our roadmap outlines our general product direcKon and is subject to change at any Kme without noKce. It is for informaKonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligaKon either to develop the features or funcKonality described or to include any such feature or funcKonality in a future release.
- 3. 3
- 4. 4 Check the Non-‐PresentaKon Version and the Security Ninjitsu App 3200 Words 1800 Words
- 5. Personal introducKon 5 David Veuve – Senior Sales Engineer for Major Accounts in Northern California Security SME, Former customer, author of Search AcKvity app dveuve@splunk.com
- 6. Who Are You? 1. Someone technical who cares about security 2. All Splunk skill levels 3. No Enterprise Security required 6
- 7. Who is this session for? 1. Someone technical who cares about security 2. All Splunk skill levels 3. No Enterprise Security required 7
- 8. Agenda Four types of security correlaKon rules you probably want 1. CorrelaKon across many sourcetypes and events 2. Privileged user monitoring 3. Conquering alert faKgue 4. Threat Intel hits All driven by customer requirements / requests 8
- 9. What Experience Are You About to Have? 9 | eval state=If(SplunkExperience<Ninja, "InformaKon Overload", "Neato") | eval state=mvappend(state, "Excitement??") Don’t fear – the Security Ninjitsu app is available on SplunkBase. Feedback welcome!
- 10. Security CorrelaKon In Splunk
- 11. Mainframe Data VMware Plakorm for Machine Data Splunk Solu0ons > Easy to Adopt Exchange PCI Security RelaKonal Databases Mobile Forwarders Syslog / TCP / Other Sensors & Control Systems Across Data Sources, Use Cases & Consump0on Models Wire Data 11 Mobile Intel MINT CIM
- 12. Mainframe Data VMware Plakorm for Machine Data Splunk Solu0ons > Easy to Adopt Exchange PCI Security RelaKonal Databases Mobile Forwarders Syslog / TCP / Other Sensors & Control Systems Across Data Sources, Use Cases & Consump0on Models Wire Data 12 Mobile Intel MINT CIM
- 13. Mainframe Data VMware Plakorm for Machine Data Splunk Solu0ons > Easy to Adopt Exchange PCI Security RelaKonal Databases Mobile Forwarders Syslog / TCP / Other Sensors & Control Systems Across Data Sources, Use Cases & Consump0on Models Wire Data 13 Mobile Intel MINT CIM
- 14. ● Easy in Enterprise Security ● In ES or Core Splunk, any search can: – Send an email – Trigger ServiceNow / etc – Run a script – Add FW Blocks, Increase Logging, etc. ● CorrelaKon in Splunk is just searching Splunk CorrelaKon Rules
- 15. 15
- 16. 16 Security-‐relevant data models from Common InformaKon Model Common Informa0on Model Standard Language
- 18. Comparison 18 Without Common InformaKon Model (Sourcetype=WinSecurity EventID=…) OR (sourcetype=linux_secure password OR key) OR sourcetype=… | eval user=coalesce(Windows_Account, user, Webstore_Admin_User…) With Common InformaKon Model tag=authenKcaKon
- 19. • AcceleraKon facilitates beser and broader analysis • Splunk has a few ways of acceleraKng content: • Report AcceleraKon • Data Model AcceleraKon • Summary Indexing • TSCollect • Pre-‐Processing of logs • Go View Last Year’s talk: Security Ninjitsu (conf.splunk.com, 2014 Sessions) How To Accelerate 19
- 20. Search Example 20 Raw Search 71 Seconds With Data Model AcceleraKon 9.8 Seconds
- 21. CorrelaKon Across MulKple Sourcetypes
- 22. CorrelaKon Across MulKple Sourcetypes • CorrelaKon is easy in Splunk. • Easy: – Across many auth log types – Across auth logs and event logs – Complex scenarios • Now, some techniques! 22
- 23. Technique 1 – Common InformaKon Model 23 tag=authenKcaKon | chart count over src by acKon | where success>0 AND failure>10 If you leverage Splunk’s Common InformaKon Model you can write one search across many products. The above search could cover twenty different products, all with matching field extracKons Most searches in this session will be based on the common informaKon model Try with the ES Sandbox!
- 24. Techniques – Common InformaKon Model 24 tag=authenKcaKon | chart count over src by acKon | where success>0 AND failure>10 Many sourcetypes with one search!
- 25. Technique 2 – Flexible Stats 25 Example: | stats count(eval(acKon="success")) as successes count(eval(acKon="failure")) as failures by user • Almost anything from eval works in stats eval
- 26. Technique 2 – Flexible Stats 26 Great Techniques: • If statements (use null for non-‐valid results) • values(eval(if(acKon="success",user,null))) as "Successful Users" • vs.. values(eval(acKon="success")) as "# of Successful Users" • Searchmatch and match for flexible matching • AND OR NOT • If(searchmatch("sudo") AND user!="service" AND (host="emailserver" OR host="webserver")…)
- 27. Techniques 3 – Expand Base Search 27 Joins are computaKonally expensive, and limited Subsearches are beser, but not by a lot – Super sparse (rare) search as subsearch – good! Both limited to 60 seconds and 10k results Best to expand your base search
- 28. Technique 3 – Expand Base Search 28 Bad: tag=malware …… | join host [search tag=proxy ……. ] Good: tag=malware OR tag=proxy | stats count(eval(tag="malware")) as malware count(eval(tag="proxy")) as proxy by host AccounKng for Host SubtleKes: | eval mydest=if(tag="malware", dest, src) | stats … by mydest
- 29. Technique 3 – Expand Base Search 29 Incorrect (10k results!) – Join Version Maybe Incorrect (400 seconds, 10k malware hits) – Subsearch Version Beser (72 seconds) – Expanded Base Search Best (14 seconds) – tstats Search
- 30. Technique 4 – The other stats 30 SomeKmes you need more flexibility TransacKon is powerful, but expensive Consider: – streamstats – ordered processing – eventstats – addiKve (non-‐destrucKve) stats processing – geostats – be world aware
- 31. Techniques – Breaking Subsearch Limits 31 Common Usage: [search index=malware | table host] index=proxy Interpreted as: (host=vicKm1 OR host=vicKm2) index=proxy Easy specificity creates huge performance improvements (Did you know you can do | eval myhost=[search tag=malware | return dest]) Subsearches limited to 10,000 results and 60 seconds by default You can also return a literally interpreted search string: [search tag=malware | stats values(dest) as search | eval search=“(dest=“ . mvjoin(search, “ OR dest=“) . “)”] • Can’t break 60 second limit without limits.conf change
- 32. Techniques – Higher Confidence 32 Trigger your components and register to a summary index – Hey, ES does that already! Example: Find sources or desKnaKons of brute force, vicKms of IDS hits, or malware events (clean or not) and determine if those hosts have new uncategorized web proxy acKvity We’ll look at that later
- 33. Core Use Case 33 New Process Launch and uncategorized proxy acKvity within 15 minutes of anK-‐virus alert (successful or failed) High Probability C&C AcKvity Advanced use case, simple search
- 34. Core Use Case 34 [search tag=malware earliest=-‐20m@m latest=-‐15m@m | table dest | rename dest as src ] earliest=-‐20m@m (sourcetype=sysmon OR sourcetype=carbon_black evensype=process_launch) OR (sourcetype=proxy category=uncategorized) | stats count(eval(sourcetype="proxy")) as proxy_events count(eval(sourcetype="carbon_black" OR sourcetype="sysmon")) as endpoint_events by src | where proxy_events > 0 AND endpoint_events > 0 First, find our infected hosts.
- 35. Core Use Case 35 [search tag=malware earliest=-‐20m@m latest=-‐15m@m | table dest | rename dest as src ] earliest=-‐20m@m (sourcetype=sysmon OR sourcetype=carbon_black evensype=process_launch) OR (sourcetype=proxy category=uncategorized) | stats count(eval(sourcetype="proxy")) as proxy_events count(eval(sourcetype="carbon_black" OR sourcetype="sysmon")) as endpoint_events by src | where proxy_events > 0 AND endpoint_events > 0 Pull endpoint + proxy data for those hosts
- 36. Core Use Case 36 [search tag=malware earliest=-‐20m@m latest=-‐15m@m | table dest | rename dest as src ] earliest=-‐20m@m (sourcetype=sysmon OR sourcetype=carbon_black evensype=process_launch) OR (sourcetype=proxy category=uncategorized) | stats count(eval(sourcetype="proxy")) as proxy_events count(eval(sourcetype="carbon_black" OR sourcetype="sysmon")) as endpoint_events by src | where proxy_events > 0 AND endpoint_events > 0 See how many proxy and endpoint events per host
- 37. Core Use Case 37 [search tag=malware earliest=-‐20m@m latest=-‐15m@m | table dest | rename dest as src ] earliest=-‐20m@m (sourcetype=sysmon OR sourcetype=carbon_black evensype=process_launch) OR (sourcetype=proxy category=uncategorized) | stats count(eval(sourcetype="proxy")) as proxy_events count(eval(sourcetype="carbon_black" OR sourcetype="sysmon")) as endpoint_events by src | where proxy_events > 0 AND endpoint_events > 0 Filter to just hosts that have the known bad events
- 38. Core Use Case 38 [search tag=malware earliest=-‐20m@m latest=-‐15m@m | table dest | rename dest as src ] earliest=-‐20m@m (sourcetype=sysmon OR sourcetype=carbon_black evensype=process_launch) OR (sourcetype=proxy category=uncategorized) | stats count(eval(sourcetype="proxy")) as proxy_events count(eval(sourcetype="carbon_black" OR sourcetype="sysmon")) as endpoint_events by src | where proxy_events > 0 AND endpoint_events > 0 Four Lines, but not hard
- 39. Scalability Improvements 39 Raw Search: 21 seconds Tstats: 2.76 seconds
- 40. About Endpoint Logs 40 Curious about Endpoint Monitoring? Check out the epic talk from Splunk Rockstar James Brodsky: Splunking The Endpoint hJp://conf.splunk.com/session/2015/recordings/2015-‐ splunk-‐119.mp4
- 42. Privileged User Monitoring 1. Start by detecKng something bad 2. Focus on highly visible or highly privileged users Our use case: Alert for users who log into way more systems than normal 42
- 43. How to Build StaKsKcal Analysis in Splunk 43 Understand Your Use Cases Begin by pulling your data – Establish the base dataset tag=authenKcaKon | bucket _Kme span=1d | stats count by user, host, _Kme – Pull trend per host | stats avg(count) as avg first(count) as recent by user, host – Pull overall trends | eventstats dc(host) as NumServers by user Apply your business logic
- 44. Techniques in Analysis 44 Understand Normal versus Now: | eval isRecent=if(_Kme>relaKve_Kme(now(),"-‐1d"), "yes", "no") Report on Causes for Analysis | eval Cause=if(NumServersHistorically*3 < NumServersRecently, "SubstanKal increase in the number of servers logged on to","") | where Cause!=""
- 45. AcceleraKon Analysis 45 Raw Searching can be slow over big datasets tag=authenKcaKon earliest=-‐30d@d| bucket _Kme span=1d | stats count by user, host, _Kme Accelerated searching is fast! | tstats count from datamodel=AuthenKcaKon where earliest=-‐30d@d groupby AuthenKcaKon.dest AuthenKcaKon.user _Kme span=1d | rename AuthenKcaKon.dest as dest AuthenKcaKon.user as user tag=authenKcaKon earliest=-‐30d@d| bucket _Kme span=1d | stats count by user, host, _Kme | eval isRecent=if(_Kme>relaKve_Kme( now(),"-‐1d"), "yes", "no") | stats avg(eval(if(isRecent="no",count,n ull))) as avg first(count) as recent by user, host | eventstats count(eval(if(avg>0,"yes",null))) as NumServersHistorically dc(eval(if(recent>0,"yes",null))) as NumServersRecently by user | eval Cause=if(isnull(avg) AND NumServersHistorically>0, "This is the first logon to this server", "") | eval Cause=if(NumServersHistorically* 3 < NumServersRecently, mvappend(Cause,"SubstanKal increase in the number of servers logged on to"), Cause) | where Cause!=""
- 46. • AcceleraKon facilitates beser and broader analysis • Splunk has a few ways of acceleraKng content: • Report AcceleraKon • Data Model AcceleraKon via Pivot • Summary Indexing • TSCollect • Pre-‐Processing of logs • Go View Last Year’s talk: Security Ninjitsu (conf.splunk.com, 2014 Sessions) How To Accelerate 46
- 47. Analysis – Part Two 47 You know high risk, high exposure users – Sys Admins – ExecuKves – Contractors – First 3 months of employment, last 3 months of employment Sources: – AD Group Membership – AD Title – HRIS Employment Status
- 48. Analysis – Part Two -‐ Example 48 | inputlookup LDAPSearch | eval risk = 1 | eval risk = case(NumWhoReportIn>100, risk+10, risk) | eval risk = case(like(Groups, "%OU=Groups,OU=IT Security,%"), risk + 10, risk) | fields risk sAMAccountName | outputlookup RiskPerUser IdenKty Data IniKalize Risk Business Logic New Lookup
- 49. Analysis – Pu€ng it Together 49 [… insert your Privileged User AcKvity Search …] | stats count by user | lookup RiskPerUser sAMAccountName as user | eval AggRisk = risk * count | eval DescripKveRisk = case(AggRisk > 100, "very high", AggRisk>30, "medium", AggRisk>5, "low", 1=1, "very low") Summarize Per User Add Org-‐wide Risk Create a new Lookup Describe Risk
- 50. Analysis – Pu€ng it Together 50 Oh Yeah. Jack Bauer has gone rogue.
- 51. AcKon (ES Specific) 51 In ES, pass the following overrides in your search: – severity – risk_score – risk_object – risk_object_type Beser yet, use the built-‐in ES IdenKty Framework!
- 53. Conquering Alert FaKgue • Typical Ker one analyst: one event per 10-‐15 min. – Only 50 events per shiT. • You will always have more alert data than you have staff • Many great techniques for managing this • Let’s dig into my favorite five 53
- 54. (1/5) Analysis Technique – Risk-‐Based 54 • Great for general purpose events • Increase the risk associated with an enKty (user, system, signature, etc.) • Focus acKvity on high risk enKKes • Out of the box with ES (index=risk) • Consider building your own by chaining | collect
- 55. (2/5) Analysis Technique -‐ StaKsKcal 55 Understand Your Environment Begin by pulling your data – Establish the base dataset | bucket _Kme span=1d | stats sum(param1) as sum count(param1) as count by host, _Kme – Pull trend per host | stats avg(sum) as avg stdev(sum) as stdev first(sum) as recent by host – Pull overall trends | eventstats avg(avg) as overallavg ….. Apply your business logic
- 56. (2/5) Analysis Technique – StaKsKcal – Part Two 56 Example Where Clause | where (avg_earliest > relaKve_Kme(now(), "-‐1d")) OR (earliest > relaKve_Kme(now(), "-‐1d") OR priority>3 )) ….. Most of the hosts infected in last day High Priority host infected in the last day
- 57. (3/5) Analysis Technique – Combine MulKple Vectors 57 With mulKple correlaKon searches, do a meta analysis on events. – ES: index=notable – Alert Manager: | rest "/services/alerts/fired_alerts" – TickeKng system: API or DBConnect Search for hosts with mulKple alerts to create a high confidence high severity event.
- 58. (3/5) Analysis Technique – Combine MulKple Vectors 58 Example: index=notable | stats dc(search_name) as NumRules by dest More Powerful Example: (index=notable AnKvirus OR ids) OR (tag=proxy category=uncategorized) [… use Stats Eval example for correlaKon …] In ES >= 3.2, search index=risk for correlaKons w/o notables
- 59. (4/5) Analysis Technique – Increase Logging 59 Increase logging on suspect hosts With ES, use Splunk Stream. Also use your ETDR soluKon. Leverage panblock, expect scripts to add to increased logging groups Write new correlaKon rules based on that increased logging – Higher confidence, higher severity
- 60. (5/5) Analysis Techniques – Machine Learning 60 With Machine Learning, you can build extremely powerful models and techniques for finding outliers programmaKcally. Look at Splunk UBA – this is what they do. – Ask your SE! Look at the new ML App! – Ask your SE! (Watch him look bewildered!)
- 61. Threat Feeds
- 62. Threat Feeds • You know enough to build a threat intel engine • Don’t 62
- 63. Great Threat Feed Tools 63 ES is officially supported with nine types of threat intel Without ES, look at SA-‐Splice on Splunkbase – not supported, but works for many customers. Please, please don’t build it yourself! IPs Domains User Names Process Names Hashs CerKficate Hashes CerKficate Common Names Email Addresses File Names
- 64. But that’s not all for Threat Intel 64 Lots of things you can do with Threat Intel – Turning Indica0ons of Compromise into Tangible Protec0on hsp://conf.splunk.com/session/2015/recordings/2015-‐splunk-‐94.mp4 – Managed Threat Intelligence in Splunk ES Splunk’s Brian Luger (ES Developer) – hsp://conf.splunk.com/session/2015/recordings/2015-‐splunk-‐148b.mp4 Generate it yourself (go ask your SE and tell them Andrew Phillips sent you)
- 65. Demo the Security Ninjitsu App
- 66. Wrap Up
- 67. How to Be Successful 67 Install the app on a non-‐producKon instance! – Example Searches for Every Use Case/Technique – One enKrely new use case Check out security sessions on hsp://conf.splunk.com Post to hsp://answers.splunk.com with tag "correlaKonsearch"! Talk to the person next to you! Hunt down your nearest Splunk Security SME SE!
- 68. Give Me Feedback! 68 Rate it in the app $50 Amazon GiT Card will be randomly given to those who also submit feedback here: hsp://www.davidveuve.com/go/ conf2015 Download the app, play around with it, and give me feedback. hsp://www.davidveuve.com/go/conf2015 Another $50 Amazon GiT Card will be randomly given!
- 69. THANK YOU
- 70. QuesKons hsp://www.davidveuve.com/go/conf2015
- 71. THANK YOU