How to Normalize Threat Intelligence Data from Multiple Sources - Tech Talk Tuesday


Published on

Ever feel like you spend more time converting security information from one format to another, than actually connecting the dots hidden within it? The Collective Intelligence Framework (CIF) is a data processor for pulling in and normalizing out all these threat intel sources into a single combined dataset. Watch it on-demand #TTTSec

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

How to Normalize Threat Intelligence Data from Multiple Sources - Tech Talk Tuesday

  1. 1. How to Normalize ThreatIntelligence Data fromMultiple Sources#TTTsec @AlienVaultYour HostConrad ConstantineCommunity Manager,AlienVault@cpconstantineTodd LeethamCyber Threat Intelligence Lead, EMC@rudehimself
  2. 2. Covered In This Talk• Getting the Collective IntelligenceFramework installed, collecting intelligencefrom external sources and generating acustom feed to use with your securitycontrols.• Making sense of the Threat IntelligenceResources out there as part of yoursecurity monitoring program.
  3. 3. What You’ll Need to Build andUse CIF• A Linux system, (a Debian-based distro ispreferred) with plenty of resources allocated –4GB and 20GB of storage recommended forexperimentation, 16GB and 500GB recommendedfor production.• Experience installing Linux software from source.• Basic DBA skills with the PostGreSQL Database.• Admin experience with the BIND DNS resolver• Admin experience with the Apache Webserver• Know how to locate and install Perl Modules.• Familiarity with essential internet topologymechanisms (BGP AS’s, registrars, etc)
  4. 4. Collective Intelligence Framework:Redux• Just as a SIEM allows the consumption of log data,normalizing to allow queries, transforms and correlations to berun against them, CIF does the same for Threat Intel Data.• IP addresses, Domains, URI substrings – threat intel comes inmany formats and we don’t have time to spend our daysconverting datasets by hand – automate once, use it forever.• Store Data from multiple sources, combine, process andproduce customized output in formats suitable forconsumption by the security controls you have in placealready.• Query the intelligence data via programming API or human-readable web interface.• Customize output for different audiences, maintain accessthrough a key-based API system, share tokenized, sanitizedintelligence amongst multiple organizations without disclosingsensitive information in the process.
  5. 5. CIF Architecture
  6. 6. Threat Intelligence For MereMortals• Security Controls (for the most part) detect technicalthreats – they can’t determine intent.• Malicious activity can be indistinguishable fromlegitimate, to a software control.• Nothing identifies a False Positive like a second (orthird, or fourth) opinion.• Attackers have agility that defenders do not –keeping them on the move and unable to launch anattack from the same place twice raises their costsof ‘doing business’• Information about where they are launching attacksfrom, what tools they are using – any piece ofinformation that can make the difference betweenresponding to an Alert, and responding to a Threat.
  7. 7. Putting Threat Intel to Work• Security Controls generate hundreds of alerts per day(on a slow day).• Threat Intelligence allows you to prioritize responseefforts around caused by external parties known to beconducting malicious activities.• Threat Intel allows you to group individual alerts togetherinto a larger picture of coordinated activity against yourassets, and enable you to strike at the roots of an attackcampaign instead of chasing each compromiseindividually.• 50 compromised machines? Or one Command AndControl system to identify and block communications to?
  8. 8. The Threat IntelligenceMarketplace• Public internet threat intelligence began with Anti-Spam Blacklists.• Now covers a multitude of open repositories ofhost/network reputation, malware and exploitsignatures and other more specialized information.• Several Public and Private organizations maintainprivate (or commercial subscription) feeds of ThreatIntelligence, ranging from IP Reputation tospecialized research about the individuals carryingout attacks.• Many emerging standards for defining andexchanging threat information – and securitycontrols often have only limited support forconsuming this information.
  9. 9. Building your first CIF Server• You either:– Want to start incorporating some public ThreatData into your security controls• Or– You’re currently consuming several threatdata feeds and want a better way to combine,aggregate and query them, and process themwith your security controls and analysis tools
  10. 10. Polling Question#TTTsec @AlienVault
  11. 11. Prerequisites andEnvironment• A working BIND installation on the CIF server, configured touse trusted public DNS servers for upstream forwarding:• A working PostgreSQL installation on the CIF server,configured for user/pass based auth• An Apache Webserver installation, with Mod_Perl loaded.• A fairly extensive collection of Perl modules
  12. 12. CIF Server Installation• Download the CIF archive, extract it, run the configure scripts.• Build and ‘make install’• Run ‘make initdb’ this will fail if postgresql was not configured.• Create a new service account – ‘cif’, generate the base CIFconfiguration file for it. ~/.cif• Configure Apache to load the CIF http API perl modules via mod_perl.• Install the Cron entries for CIF to update its threat sources periodically• CIF installs to /opt/cif by default.
  13. 13. Creating API Keys• Access to the CIF datastore is done via client apps using an API key.• You’ll need to generate an access key for each client that will haveaccess to the CIF datastore.• The initial key creation is going to look something like this:$ cif_apikeys -u -a -g everyone -G everyoneuserid key description guid default_guid access writerevoked expires 249cd5fd-04e3-46ad-bf0f-c02030cc864a 8c864306-d21a-37b1-8705-746a786719bf true all 2012-08-01 11:50:15.969724+00• You’re going to need this API key to configure a CIF client
  14. 14. Installing a Client• The Client is contained in the ‘libcif’ source package – install the perldependencies and configure && make && make install, as usual.• This contains the ‘cif’ binary used for commandline interaction withthe CIF server.• Configuration is just the URI for the CIF server API, and the client’sAPI key (generated previous)
  15. 15. Threat Intel Sources• The default threat intel sources are defined in individual configs in{installdir}/etc/• They are updated periodically with the {installdir}/bin/cif_crontoolexecutable.• They define a source of information, and some basic transforms tobegin the normalization process.• Sources are defined with global access rights, confidence levels,that control how their information is used within CIF client queries.detection = dailyfeed = = reputation.alienvault.comguid = everyoneconfidence = 65severity = mediumrestriction = need-to-knowalternativeid = ""alternativeid_restriction = publicmirror = /tmp
  16. 16. CIF and AlienVault Open Threat Exchange.• CIF comes with a few public Threat Intelsources by default.• CleanMX, Zeustracker, MalwareDomainList..• ….and AlienVault Open Threat Exchange.• The same IP reputation and Threat Data weuse in the AlienVault product.• With CIF you can consume it..• ..With AlienVault OSSIM you can contributeto it automatically and help take the fight tothe Threat Actors.
  17. 17. Querying Feeds• Commandline client allows querying thenormalized feed data by confidence level, typeof activity seen, network location, domain, etc• Query if a URI exists in the Threat Feeds:$ cif -q‘• Query for all information about hosts on a given network:$ cif -q• Has anyone seen this file before? Try a SHA-1 Hashquery:$ cif -q a5135ec6f2322cc12f3d9daa38dfb358• Some simple Web Interfaces created for the HTTP API,or query from your own tools if they are capable ofmaking API queries.
  18. 18. Consuming Feeds• CIF comes with a selection of output feed plugins, available via thecommandline tool , using the –p (plugin) argument, using the perlIODEF module or the HTTP API.• Some included formats: snort rules csv json bindzone html table ascii table bro (network monitor) pcap filter iptables
  19. 19. Putting it to Work• Define feeds that query information according to yourconditions Type of Threats observed Confidence Levels Network Locations, etc etc• Export in a format consumable by your security controls.• Automatically block connections, or just raise priority onalerts that show up in aggregate threat data.• Create your own data source from your own SecurityAnalysis work, create limited views on the informationand share with Security Partners.
  20. 20. Taking it from Here• Get a basic system up• Start Experimenting with the CIF query tools• Generate a feed to automatically pass on toone of your security controls or analysis tools.• SIEM WatchLists are excellent things topopulate with Threat Intel, to alert andprioritize on.• Start responding to attacks made by people,not signatures triggered by systems.
  21. 21. • Collective Intelligence Framework (CIF)Website– Server Installation Instructions’t forget to check the dependencies page for your Linux Distro!)– Client Installation Instructions– API Documentation• AlienVault Open Threat Exchange(OTX)
  22. 22. • AlienVault OSSIM, free open-source SIEM• Free 30-day trial of AlienVault USM• AlienVault Labs blog• AlienVault Apps & ThreatReports by: @AlienVaultHELPFUL TOOLS & RESOURCES
  23. 23. Questions?#TTTsec @AlienVault
  24. 24. Thank You.#TTTsec @AlienVaultYour HostConrad ConstantineCommunity Manager,AlienVault@cpconstantineTo learn more about AlienVault please