Ever feel like you spend more time converting security information from one format to another, than actually connecting the dots hidden within it? The Collective Intelligence Framework (CIF) is a data processor for pulling in and normalizing out all these threat intel sources into a single combined dataset. Watch it on-demand http://ow.ly/li8Lf #TTTSec

1. How to Normalize Threat
Intelligence Data from
Multiple Sources
#TTTsec @AlienVault
Your Host
Conrad Constantine
Community Manager,
AlienVault
@cpconstantineTodd Leetham
Cyber Threat Intelligence Lead, EMC
@rudehimself

2. Covered In This Talk
• Getting the Collective Intelligence
Framework installed, collecting intelligence
from external sources and generating a
custom feed to use with your security
controls.
• Making sense of the Threat Intelligence
Resources out there as part of your
security monitoring program.

3. What You’ll Need to Build and
Use CIF
• A Linux system, (a Debian-based distro is
preferred) with plenty of resources allocated –
4GB and 20GB of storage recommended for
experimentation, 16GB and 500GB recommended
for production.
• Experience installing Linux software from source.
• Basic DBA skills with the PostGreSQL Database.
• Admin experience with the BIND DNS resolver
• Admin experience with the Apache Webserver
• Know how to locate and install Perl Modules.
• Familiarity with essential internet topology
mechanisms (BGP AS’s, registrars, etc)

4. Collective Intelligence Framework:
Redux
• Just as a SIEM allows the consumption of log data,
normalizing to allow queries, transforms and correlations to be
run against them, CIF does the same for Threat Intel Data.
• IP addresses, Domains, URI substrings – threat intel comes in
many formats and we don’t have time to spend our days
converting datasets by hand – automate once, use it forever.
• Store Data from multiple sources, combine, process and
produce customized output in formats suitable for
consumption by the security controls you have in place
already.
• Query the intelligence data via programming API or human-
readable web interface.
• Customize output for different audiences, maintain access
through a key-based API system, share tokenized, sanitized
intelligence amongst multiple organizations without disclosing
sensitive information in the process.
http://code.google.com/p/collective-intelligence-framework/

5. CIF Architecture

6. Threat Intelligence For Mere
Mortals• Security Controls (for the most part) detect technical
threats – they can’t determine intent.
• Malicious activity can be indistinguishable from
legitimate, to a software control.
• Nothing identifies a False Positive like a second (or
third, or fourth) opinion.
• Attackers have agility that defenders do not –
keeping them on the move and unable to launch an
attack from the same place twice raises their costs
of ‘doing business’
• Information about where they are launching attacks
from, what tools they are using – any piece of
information that can make the difference between
responding to an Alert, and responding to a Threat.

7. Putting Threat Intel to Work
• Security Controls generate hundreds of alerts per day
(on a slow day).
• Threat Intelligence allows you to prioritize response
efforts around caused by external parties known to be
conducting malicious activities.
• Threat Intel allows you to group individual alerts together
into a larger picture of coordinated activity against your
assets, and enable you to strike at the roots of an attack
campaign instead of chasing each compromise
individually.
• 50 compromised machines? Or one Command And
Control system to identify and block communications to?

8. The Threat Intelligence
Marketplace
• Public internet threat intelligence began with Anti-
Spam Blacklists.
• Now covers a multitude of open repositories of
host/network reputation, malware and exploit
signatures and other more specialized information.
• Several Public and Private organizations maintain
private (or commercial subscription) feeds of Threat
Intelligence, ranging from IP Reputation to
specialized research about the individuals carrying
out attacks.
• Many emerging standards for defining and
exchanging threat information – and security
controls often have only limited support for
consuming this information.

9. Building your first CIF Server
• You either:
– Want to start incorporating some public Threat
Data into your security controls
• Or
– You’re currently consuming several threat
data feeds and want a better way to combine,
aggregate and query them, and process them
with your security controls and analysis tools

10. Polling Question
#TTTsec @AlienVault

11. Prerequisites and
Environment
• A working BIND installation on the CIF server, configured to
use trusted public DNS servers for upstream forwarding:
https://code.google.com/p/collective-intelligence-framework/wiki/BindSetup_v1
• A working PostgreSQL installation on the CIF server,
configured for user/pass based auth
https://code.google.com/p/collective-intelligence-framework/wiki/PostgresInstall_v1
• An Apache Webserver installation, with Mod_Perl loaded.
• A fairly extensive collection of Perl modules
http://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_DebianSqueeze_v1
https://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_DebianWheezy_v1
https://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_Ubuntu12_v1
https://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_CentOS6_v1
https://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_CentOS5_v1

12. CIF Server Installation
• Download the CIF archive, extract it, run the configure scripts.
• Build and ‘make install’
• Run ‘make initdb’ this will fail if postgresql was not configured.
• Create a new service account – ‘cif’, generate the base CIF
configuration file for it. ~/.cif
• Configure Apache to load the CIF http API perl modules via mod_perl.
• Install the Cron entries for CIF to update its threat sources periodically
• CIF installs to /opt/cif by default.
https://code.google.com/p/collective-intelligence-
framework/wiki/ServerInstall_v1

13. Creating API Keys
• Access to the CIF datastore is done via client apps using an API key.
• You’ll need to generate an access key for each client that will have
access to the CIF datastore.
• The initial key creation is going to look something like this:
$ cif_apikeys -u myuser@mydomain.com -a -g everyone -G everyone
userid key description guid default_guid access write
revoked expires created
myuser@mydomain.com 249cd5fd-04e3-46ad-bf0f-c02030cc864a 8c864306-d21a-37b1-
8705-746a786719bf true all 2012-08-01 11:50:15.969724+00
• You’re going to need this API key to configure a CIF client

14. Installing a Client
• The Client is contained in the ‘libcif’ source package – install the perl
dependencies and configure && make && make install, as usual.
• This contains the ‘cif’ binary used for commandline interaction with
the CIF server.
• Configuration is just the URI for the CIF server API, and the client’s
API key (generated previous)
https://code.google.com/p/collective-intelligence-framework/wiki/ClientInstall_v1

15. Threat Intel Sources
• The default threat intel sources are defined in individual configs in
{installdir}/etc/
• They are updated periodically with the {installdir}/bin/cif_crontool
executable.
• They define a source of information, and some basic transforms to
begin the normalization process.
• Sources are defined with global access rights, confidence levels,
that control how their information is used within CIF client queries.
detection = daily
feed = http://reputation.alienvault.com/reputation.data
source = 'reputation.alienvault.com'
guid = everyone
confidence = 65
severity = medium
restriction = 'need-to-know'
alternativeid = "http://reputation.alienvault.com/reputation.generic"
alternativeid_restriction = 'public'
mirror = /tmp

16. CIF and AlienVault Open Threat Exchange.
• CIF comes with a few public Threat Intel
sources by default.
• CleanMX, Zeustracker, MalwareDomainList..
• ….and AlienVault Open Threat Exchange.
• The same IP reputation and Threat Data we
use in the AlienVault product.
• With CIF you can consume it..
• ..With AlienVault OSSIM you can contribute
to it automatically and help take the fight to
the Threat Actors.

17. Querying Feeds
• Commandline client allows querying the
normalized feed data by confidence level, type
of activity seen, network location, domain, etc
• Query if a URI exists in the Threat Feeds:
$ cif -q 'http://www.yahoo.com/example.html‘
• Query for all information about hosts on a given network:
$ cif -q 130.201.0.0/16
• Has anyone seen this file before? Try a SHA-1 Hash
query:
$ cif -q a5135ec6f2322cc12f3d9daa38dfb358
• Some simple Web Interfaces created for the HTTP API,
or query from your own tools if they are capable of
making API queries.

18. Consuming Feeds
• CIF comes with a selection of output feed plugins, available via the
commandline tool , using the –p (plugin) argument, using the perl
IODEF module or the HTTP API.
https://github.com/collectiveintel/iodef-pb-simple-perl/tree/master/lib/Iodef/Pb/Format
• Some included formats:
 snort rules
 csv
 json
 bindzone
 html table
 ascii table
 bro (network monitor)
 pcap filter
 iptables

19. Putting it to Work
• Define feeds that query information according to your
conditions
 Type of Threats observed
 Confidence Levels
 Network Locations, etc etc
• Export in a format consumable by your security controls.
• Automatically block connections, or just raise priority on
alerts that show up in aggregate threat data.
• Create your own data source from your own Security
Analysis work, create limited views on the information
and share with Security Partners.

20. Taking it from Here
• Get a basic system up
• Start Experimenting with the CIF query tools
• Generate a feed to automatically pass on to
one of your security controls or analysis tools.
• SIEM WatchLists are excellent things to
populate with Threat Intel, to alert and
prioritize on.
• Start responding to attacks made by people,
not signatures triggered by systems.

21. • Collective Intelligence Framework (CIF)
Websitehttps://code.google.com/p/collective-intelligence-framework/
– Server Installation Instructionshttps://code.google.com/p/collective-intelligence-
framework/wiki/ServerInstall_v1
(Don’t forget to check the dependencies page for your Linux Distro!)
– Client Installation Instructionshttps://code.google.com/p/collective-intelligence-
framework/wiki/ClientInstall_v1
– API Documentationhttps://code.google.com/p/collective-intelligence-
framework/wiki/API_v1
• AlienVault Open Threat Exchange
(OTX)http://www.alienvault.com/alienvault-labs/open-threat-exchange
REFERENCE

22. • AlienVault OSSIM, free open-source SIEM
http://communities.alienvault.com
• Free 30-day trial of AlienVault USM
http://www.alienvault.com/free-trial
• AlienVault Labs blog
http://labs.alienvault.com/labs/
• AlienVault Apps & Threat
Reportshttp://www.alienvault.com/resource-
center/open-minds-exchange
Sponsored by: AlienVault
www.alienvault.com#TTTsec @AlienVault
HELPFUL TOOLS & RESOURCES

23. Questions?
#TTTsec @AlienVault

24. Thank You.
#TTTsec @AlienVault
Your Host
Conrad Constantine
Community Manager,
AlienVault
@cpconstantine
To learn more about AlienVault please visit:
www.alienvault.com