How to Build the Collective Intelligence Framework - Hacker Hotshots 11/27/2013


Published on

This presentation is how to build and use the Collective Intelligence Framework to start to collect threat intelligence from open-sources and use that to protect your network. This presentation was given for Hacker Hotshots on 11/27/2013.

Published in: Technology
1 Comment
1 Like
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

How to Build the Collective Intelligence Framework - Hacker Hotshots 11/27/2013

  1. 1. Hacker Hotshots – 11/27/2013 How to Build the Collective Intelligence Framework And Start to Protect Your Network John Bambenek Chief Forensic Examiner, Bambenek Consulting
  2. 2. Problem  Lots of people product lots of data, blacklists, indicators of badness out there.  They all have their own formats and means of distribution.  How to take multiple datasets, normalize them and take action?  “Does anyone know anything about X?”
  3. 3. Solution: Collective Intelligence Framework  Developed by REN-ISAC  framework/  Does not generate data, simply takes sources normalizes it and then outputs by given types  Not really a data sharing tool  Up to user to assess confidence in the data  Limited in the types of data it can handle
  4. 4. Data Types URLs  Domains  IPs  MD5s   Certainly more to threat intel than this, but it’s a start
  5. 5. CIF Architecture
  6. 6. CIF Architecture  By default, everything lives in /opt/cif  Configs in /opt/cif/etc/*.cfg (CIF processes all files ending in .cfg)     cif_smrt – queries the feeds cif_feed – generates feeds by assessment cif – command-line client tool cif_crontool – used for querying all feeds automatically
  7. 7. Requirements to Install  For a “real” instance, you would need some disk (250 GB – 500 GB) and RAM (16 GB)  Disk is driven by how long you want to keep old data  Memory is only needing while parsing data  CIF can be placed in a virtual infrastructure easily  Can install it on most everything, Debian/Ubuntu easiest mostly because the instructions are available and clear  Ubuntu 12 probably best, 13 has some undocumented changes that need to be made  Some kernel tweaking is needed
  8. 8. CIF Queries  Generally an analyst investigating will use queries to see what is in the database.  cif –q <IP ADDRESS|DOMAIN NAME|MD5>  Will include search records in the response (unless suppressed)  Exact matching only (can’t search for part of a URL… yet)
  9. 9. CIF Queries  CIF also ships a browser plugin which is a little easier for analysis  Use cif_apikeys –l to get your key, find your amazon IP and configure it now  Can query specific items or feeds
  10. 10. CIF feeds  cif –q feed/assessment –p output type [-z 0]  (-z 0 will prevent truncating URLs)  cif –q infrastructure/scan  Try with a lower confidence level  cif –q url/phishing –c 45  Not all output plugins work for all feeds  Full list at: intelligence-framework/wiki/API_FeedTypes_v1
  11. 11. CIF output types bindzone Bind zone configuration bro bro (network monitor) csv comma separated value html Html-ized table iptables iptables drop rules json json pcapfilter pcap filter (i.e. tcpdump) snort snort alert rules table ascii table (default)
  12. 12. CIF Output  There are dozens of sources (many don’t have configs in CIF), but you can integrate them all into CIF and/or a feed.  What to do with this now?      Snort Rules Feed to web proxy to block/alert Send to border device to blacklist IPs Set up a sinkhole You can also put your own data into CIF for later research
  13. 13. Questions?  Thanks for tuning in!  See more courses and hacker hotshot sessions at