Data-Driven Threat Intelligence: Useful Methods and Measurements for Handling Indicators
•Download as PPTX, PDF•
7 likes•4,417 views
This session will consist of a technological exploration of commercial and open-source threat intelligence feeds that are commonly offered as a way to improve the capabilities of incident response teams. While not all Threat Intelligence can be represented as "indicator feeds", this space has enough market attention that it deserves a proper scientific, evidence-based investigation so that practitioners and decision makers can maximize the results they are able to get for the data they have available. We will present a data-driven analysis of a cross-section of threat intelligence feeds (both open-source and commercial) to measure their statistical bias, overlap, and representability of the unknown population of breaches worldwide, in addition to some tidbits as indicator age and uniqueness across feeds. All the statistical code written and research data used (from the publicly available feeds) will be made available in the spirit of reproducible research. The tool itself (tiq-test) will be able to be used by attendees to perform the same type of tests on their own data. We will also provide an additional open-source tool (combine) for attendees to extract, normalize and export data from threat intelligence feeds to use in their internal projects and systems. It will be pre-configured with a good mix of current publicly available network feeds and easily extensible for private or commercial feeds.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Data-Driven Threat Intelligence: Useful Methods and Measurements for Handling Indicators
Similar to Data-Driven Threat Intelligence: Useful Methods and Measurements for Handling Indicators (20)
Recently uploaded
Recently uploaded (20)
Data-Driven Threat Intelligence: Useful Methods and Measurements for Handling Indicators
- 1. Data-Driven Threat Intelligence: Useful Methods and Measurements for Handling Indicators (#ddti) Alex Pinto Chief Data Scientist Niddel / MLSec Project @alexcpsec @MLSecProject Alexandre Sieira CTO Niddel @AlexandreSieira @NiddelCorp
- 2. MLSec Project / Niddel • MLSec Project – research-focused branch of Niddel for open- source tools and community building • Niddel builds Magnet, the Applied Threat Intelligence Platform focused on detecting breaches and malware activity • Looking for trial prospects and research collaboration • More info at: • niddel.com • mlsecproject.org
- 3. • Cyber War… Threat Intel – What is it good for? • Combine and TIQ-test • Using TIQ-test • Dataset description • Tests and more tests • Use case: Feed Comparison • Future research direction Agenda
- 4. What is TI good for (1) Attribution
- 5. What is TI good for anyway? TY to @bfist for his work on http://sony.attributed.to
- 6. What is TI good for (2) – Cyber Maps!! TY to @hrbrmstr for his work on https://github.com/hrbrmstr/pewpew
- 7. What is TI good for anyway? • (3) How about actual defense? • Strategic and tactical: planning • Technical indicators: DFIR and monitoring
- 8. Affirming the Consequent Fallacy 1. If A, then B. 2. B. 3. Therefore, A. 1. Evil malware talks to 8.8.8.8. 2. I see traffic to 8.8.8.8. 3. ZOMG, APT!!!
- 9. Combine and TIQ-Test • Combine (https://github.com/mlsecproject/combine) • Gathers TI data (ip/host) from Internet and local files • Normalizes the data and enriches it (AS / Geo / pDNS) • Can export to CSV, “tiq-test format” and CRITs • Coming Soon™: CybOX / STIX / SILK /ArcSight CEF • TIQ-Test (https://github.com/mlsecproject/tiq-test) • Runs statistical summaries and tests on TI feeds • Generates charts based on the tests and summaries • Written in R (because you should learn a stat language)
- 10. Using TIQ-TEST • Available tests and statistics: • NOVELTY – How often do they update themselves? • AGING – How long does an indicator sit on a feed? • POPULATION – How does this population distribution compare to another one? • OVERLAP – How do they compare to what you got? • UNIQUENESS – How many indicators are found in only one feed?
- 13. Using TIQ-TEST – Feeds Selected • Dataset was separated into “inbound” and “outbound” TY to @kafeine and John Bambenek for access to their feeds
- 14. Using TIQ-TEST – Data Prep • Extract the “raw” information from indicator feeds • Both IP addresses and hostnames were extracted
- 15. Using TIQ-TEST – Data Prep • Convert the hostname data to IP addresses: • Active IP addresses for the respective date (“A” query) • Passive DNS from Farsight Security (DNSDB) • For each IP record (including the ones from hostnames): • Add asnumber and asname (from MaxMind ASN DB) • Add country (from MaxMind GeoLite DB) • Add rhost (again from DNSDB) – most popular “PTR”
- 16. Using TIQ-TEST – Data Prep Done
- 17. Novelty Test Measuring added and dropped indicators
- 18. Novelty Test - Inbound
- 19. Aging Test Is anyone cleaning this mess up eventually?
- 20. INBOUND
- 21. OUTBOUND
- 22. Population Test • Let us use the ASN and GeoIP databases that we used to enrich our data as a reference of the “true” population. • But, but, human beings are unpredictable! We will never be able to forecast this!
- 24. Is your sampling poll as random as you think?
- 25. Can we get a better look? • Statistical inference-based comparison models (hypothesis testing) • Exact binomial tests (when we have the “true” pop) • Chi-squared proportion tests (similar to independence tests)
- 27. Overlap Test More data can be better, but make sure it is not the same data
- 28. Overlap Test - Inbound
- 29. Overlap Test - Outbound
- 30. Uniqueness Test
- 31. Uniqueness Test • “Domain-based indicators are unique to one list between 96.16% and 97.37%” • “IP-based indicators are unique to one list between 82.46% and 95.24% of the time”
- 33. Intermission
- 34. • “You Data Scientists and your algorithms, how quaint.” • “Why aren’t you doing some useful research like nation-state attribution?” OPTION 1: Cool Story, Bro!
- 35. OPTION 2: How can I use this awesomeness on my data?
- 36. • How about using TIQ-TEST to evaluate a private intel feed? • Trying stuff before you buy is usually a good idea. Just sayin’ • Let’s compare a new feed, “private1”, against our combined outbound indicators Use Case: Comparing Private Feeds
- 37. Population Test
- 38. Population Test
- 39. Population Test
- 40. Aging Test • I guess most DGAs rotate every 24 hours, right? • Rotation means the private data is still “fresh”, from research or DGA generation procedures Mostly DGA Related Churn
- 43. A++ WOULD THREAT INTEL AGAIN (or would I?)
- 44. I hate quoting myself, but…
- 45. Take Aways • Analyze your data. Extract more value from it! • If you ABSOLUTELY HAVE TO buy Threat Intelligence or data, evaluate it first. • Try the sample data, replicate the experiments: • https://github.com/mlsecproject/tiq-test-Summer2015 • http://rpubs.com/alexcpsec/tiq-test-Summer2015 • Share data with us. I’ll make sure it gets proper exercise!
- 46. Future Research • Updating this presentation for Black Hat USA • Analyzing Threat Intelligence Sharing behavior • We need anonymous indicator sharing counts: • “C1 shared X indicators on day Y” • “C2 marked x indicators as FPs / or down voted them”
- 48. Thanks! • Q&A? • Feedback! ”The measure of intelligence is the ability to change." - Albert Einstein Alex Pinto @alexcpsec @MLSecProject @NiddelCorp Alexandre Sieira @AlexandreSieira @MLSecProject @NiddelCorp
Editor's Notes
- During this presentation I’ll be making a quick introduction on some important concepts and our views on what threat intelligence is and the many useful things that can and cannot be done with it. Then, Mr. Pinto will present a couple of open source tools made available at the MLSec Project GitHub repository and an analysis of the metrics we gathered from a set of publicly available and private threat intelligence feeds.
- Based on what you read on the media and one some online marketing material, attribution is one of the sexiest parts about threat intelligence. If you understand your adversaries, their intent and capabilities, there is a lot you can do within your risk management and even business decisions to better prepare yourself. However, if you want to be realistic you’ll have to admit that as in the case demonstrated here, attribution is really hard to do. This reflects both on the cost of of getting good CTI data with attribution. Also it is not reasonable to expect that any significant proportion of attacks out there will be attributed at all, given the sheer amount of smaller threat actors out there.
- This is beautifully illustrated by the Sony breach controversy. So data science to the rescue! The good humored sony.attributed.to website creates plausible attribution reports for the Sony hack. It is based on data sampled from actual attacks data in the VERIS and DBIR databases consistent with how frequently threat actors, locations and methods are actually used.
- One other really common application of threat intelligence is building a threat map. After all, how else will upper management know that your team really has what it takes to prevent pandas, bears and even maybe capybaras from infiltrating your networks? Who knows what those samba-dancing Brazilian hackers are up to, after all? Fear not, now there’s an open source threat map that you can use called Pew pew. And you don’t even need to stand up your own honeypots. Pew pew displays plausible attack patterns that are sampled randomly from public data made available by Arbor Networks. So it is exactly as useful as the real thing.
- But seriously, what about using the threat intelligence data to actually defend organizations? The high level data is awesome to help with the high level decisions, of course. However, how do you go about using the technical indicators? It’s straightforward enough to pull a list of IP addresses, domain names and URLs into SIEMs or IDS rules. As Gavin Reid mentioned in his talk yesterday, this can be a great way to reduce the time it takes to detect a novel threat and bypass the detection, rule writing and policy update cycle of your sensors. However, this has to be done with great care.
- Threat intelligence feeds are mostly providing indicators of things that malware does: for example IP addresses, domain names and URLs they communicate with. Knowing this can be invaluable for all sorts of investigation and forensics activities. But we really believe the holy grail is detection. A breach detection example the fallacy: some malware observed talking to a destination does NOT mean that all communication towards that destination is indicative of malware. Also you need evaluate the quality and the applicability of the indicators you are consuming (and perhaps paying for) to decide which combination of sources is optimal for your organization. Now Mr. Pinto will tell you a bit about two open source projects we develop to help you perform such an evaluation.
- TESTING THE DATA Population is tricky: - Could mean the entire world (all IPv4 space) - Should ideally mean your world (use your data)
- 2014 was not a leap year 150k outbound 300k inbound 450k X 365 -> 164,250,000 / 165 MILLION
- 28 feeds
- - For the hostname / domains feeds: - We extracted the "active" IP addresses for those hostnames on the dates they were reported (using pDNS from Farsight Security) - Passive DNS query of active "A" responses on the reported day (from 00:00 to 23:59) - For this experiment, we got rid of "non-public IPs" (localhost, RFC1918) - Then for each IP record (including the ones got above) enrich it with: - asnumber and asname (from MaxMind ASN DB) - country (from MaxMind GeoLite DB) - rhost - the more popular reverse DNS entry ("PTR") from passive DNS on that date
- - Then for each IP record (including the ones got above) enrich it with: - asnumber and asname (from MaxMind ASN DB) - country (from MaxMind GeoLite DB) - rhost - the more popular reverse DNS entry ("PTR") from passive DNS on that date - we are not playing around with this on this talk, data is too sparse
- NOVELTY: Always request a trial of the data feed (15/30 days) Measure addition and churn
- OVERLAP
- OVERLAP
- OVERLAP
- POPULATION: We will use the ASN and GEO databases as our population - They should cover all the existing IPv4 space, give or take a few anomalies With this, we can simulate a draw from this population for people that are going to be attacking us because human beings are unpredictable and will never be able to forecast where the attacks are coming from right? :troll: ----- COUNTRY / ASN (graphs of country proportions on different feeds)
- Inbound – Turkey WINS! US is highly above the population average, and CN is slightly below
- ---- HYPOTHESIS TESTING OF PROPORTIONS (explain exact binom test vs. Chi-squared)
- ---- HYPOTHESIS TESTING OF PROPORTIONS (explain exact binom test vs. Chi-squared)
- -> Explain the diferences - These differences describe a higher probability of specific actors targeting you from specific locations (GEO/ASN) in relation to a completely random actor. - Was one of the 1st features I used for MLSec
- Get 100 fish in a pond Tag all Get 100 more fish – how many were tagged? 5?-> 20x more fish
- Intermission