Intersect
Upcoming SlideShare
Loading in...5
×
 

Intersect

on

  • 776 views

 

Statistics

Views

Total Views
776
Views on SlideShare
776
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Shameless plugs before we get started: Don’t forget to check out Hack Fortress, and the finals are at 6:30 in ?. Great, one of a kind hacking/blow stuff up competition that we work hard on.Please go see the last talk of Shmoocon, “All Your Data are Belong to Me” or something like that. It’s about iPhone apps stealing your stuff. But more importantly, it’s Matt’s wife and she’ll be mad at us if we don’t tell you that.
  • What we really wanted to call this presentationNow first a disclaimer…
  • If you went to the opening remarks, you heard Bruce say something like “don’t believe everything everybody says”. I first saw Bruce speak at Defcon 13 a few years back, and I really wanted to find a video clip of him yelling “bow to my firewall”, but that man can make stuff disappear from the Internet like nobody else. George Lucas should have him track down all the copies of Star Wars Christmas Special.But I’ll settle for stealing a slide from him…
  • Hopefully more people will start presentations with a disclaimer like this, rather than just jumping on stage and prosthelytizing. I don’t mean that we’re lyingQuestion everythingJust because we’re on stage, doesn’t mean we know more than you (we’re just better looking)
  • Through that internal R&D program, Matt and I work in a lab dealing with (a term you might hear thrown around a lot), the “Advanced Persistent Threat”That lab has a bunch of people both developing tools and looking at various COTS and FOSS tools, looking for attempted attacks on our networkThrough that lab we came up with the idea for our project and were able to develop itWe based our work on some lessons we “learned”
  • We’ll get into these more in depth later, but they’re the basic assumptions of our research
  • AVWill detect known bad, but “good” may not be trueHigh false negative rateHome GrownEvery new tool needs to ingest files and report resultsDon’t want them in a position where they can affect availability
  • Unique and proprietaryLots of vendors claim to have “top to bottom” solutionEven if they doThey’re expensiveHard to configureNot always the “best of breed” at each levelVendors are catering to the big pictureProbably won’t address your specific threats/needs
  • “One of the AV bypasses we used was a simple wrapper that forced the internal emulators of the AV engines to timeout when scanning. We did this by wrapping the malicious code in an exe that called GetTickCount and waited about 2 minutes, at which point the AV engines had timed out and said the file was clean. Then we decoded the original malware, dropped it to disk and executed it.” - MR
  • Not generally remote, network basedSo your firewalls, IDS/IPS are uselessIf your users are compromised and initiate the connection, you need a different type of scanning/blockingThe most effective way is to get the malware, analyze what it’s doing, make “signatures” from that, and look for those “signatures”
  • Might not want to block a file with these indicators, but they might make you want to analyze it a little deeperThis can work both waysWhat do you know about your adversary? Look for that in new files…Did you get some malware? Look for new indicators
  • This stuff won’t show up in VirusTotal or AV signaturesEven if you submit it to them, it might be a while before it gets incorporated into a signature
  • If something looks a little suspicious to a few different scanners, escalate it to an analystResearchers can develop tools faster than COTSAre your costly tools actually performing?Can you replace them with something cheaper?Will that new tool really do a better job?Time to finally make the COTS developers step up their games
  • Already existing on your network, just leverage them
  • Or just pipe it through Curl
  • Gloss over this. Examples on next slide.
  • Pass toFotios
  • Calculates MD5 and content-type (used as routing key)
  • Skip over producers part. This is explained with example producer. We said we wanted a simple interface, this is as simple as it gets. If you can POST or GET, you can use this
  • So we said to ourselves, “let’s not only learn a whole new paradigm for programming, but let’s do it in a language neither of us has used before”
  • Only time you need to contact INTERSECT directly is to fetch files for analysis
  • Can be simply reused (like if you’re calling `system command`)
  • Bank of America, Barclays, Credit Suisse, Goldman Sachs, JPMorgan ChaseCisco, Microsoft, Novell, Red Hat, VMware
  • Written in ErlangRecently acquired by SpringSource, owned by VMware
  • Create consumers based on intelligence gained through this iterative process or other means
  • Let’s look back at those 3 needs we mentioned earlier
  • If something looks a little suspicious to a few different scanners, escalate it to an analystResearchers can develop tools faster than COTSAre your costly tools actually performing?Can you replace them with something cheaper?Will that new tool really do a better job?Time to finally make the COTS developers step up their games

Intersect Intersect Presentation Transcript

  • INTERSECT: CombiningCommercial/FOSS Tools with Custom Code to Root Out Malware Fotios Lindiakos Matt Pawloski © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • INTERSECT: How to Combine All of the Stuff You Spent Too MuchMoney on With the Cool Free StuffYour Boss Won’t Let You Install to Actually Do Something Useful © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • © 2011 The MITRE Corporation.Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • About Us Fotios Matt• Graduated 2007 from RIT • Graduated 2005 from RIT with a BS in CS with a BS in IT• Attending GMU for a MS in • Graduated 2010 from ISA Capitol College with MS in• Started as an intern at IA MITRE and has been full • Worked at Symantec, KCG time for 3.5 years • Been at MITRE 3 years © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • About MITRE• “Not-for-profit organization chartered to work in the public interest”• “MITRE is a unique organization that assists the United States government with scientific research and analysis, development and acquisition, and/or systems engineering and integration”• “MITRE also has its own independent research and development program that explores new technologies and new uses of technologies to solve our sponsors problems in the near-term and in the future” Sources: http://www.mitre.org/about/ffrdcs.html http://www.mitre.org/about/index.html © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • An Axiom• Client side attacks are the most prevalent attack vector – Users receiving a malicious email attachment – Users receiving a malicious link in an email• We need agile file examination!• Good tools exist, but can be hard to deploy/use• “Real-time” is nice to have, but not practical © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • STATE OF THE INDUSTRY © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Products Pros Cons • Quick indicator • Not effective against targeted attacksAntivirus • Cheap/free • Can block in • Same “signature problem” as real-time antivirusIDS/IPS • Doesn’t examine full files • False positives can cause an outage • Can be very • Getting files can be difficult Home effective for your specific • Lots of reinvention of the wheel • Can be unstable Grown organization if used properly © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Most COTS Products• Difficult to interface with• No great “top to bottom” solution• Expensive• Not agile enough to meet quickly adapting threats• Vendors don’t meet your specific needs• This doesn’t mean they are worthless! © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • If DEFCON has taught us anything…• “Race to Zero” – Signature based scanning is trivial to bypass – Examples • Repacking • Causing AV engines to timeout by wrapping malware with some trivial code – Doesn’t have to attack AV or modify malware • Unhook AV• Targeted defenses are needed for targeted attacks © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • CURRENT THREATS © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Client Side Attacks• We have everything we need in the file – Static analysis • Initial file is usually just a dropper – Behavioral analysis • File will beacon out, download more malware, and commence C&C• Even a small success rate is still a success © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • 0-day• Everybody is sick of talking about this• Detection sometimes possible through – Content detonation – Targeted profiling • ssdeep • HBGary FingerPrint © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Targeted Attacks• How can you expect any non-specialized tools to find something that was made specifically for you? – Targeted attacks need targeted defenses• Only targeted at a select number of users• Phishing email so well crafted your users will definitely click on it © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • What We Need• Need to Gather Intelligence – Accumulating Data • You already have the files • Different tools can tell you different things – Correlating Data• Need to Protect – Detect targeted attacks – Need to react faster than traditional solutions – Different tools may offer overlapping protection• Need to Measure Efficacy – Are your tools actually doing a good job? – Easily evaluate new technologies © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • WHAT WE PROPOSE © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Not a Replacement!• You still need these products• We want to augment and integrate them• Use conventional technology in unconventional ways © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Our Requirements• Simple Interfaces – Front end (user experience) – Back end (developer experience)• Scalable• Resilient• Fast• Awesome (with a catchy name) © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Our Solution…• We call it INTERSECT © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • What Is It?• Middleware that works!• Just a framework – Ties together all the pieces (more on them later) – Gives the users a “single pane of glass” – Handles all of the mundane stuff to let the developers focus on their parts – Helps consolidate results • Can be used to perform correlation and alerting © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Producers• Any services/devices that see full files and can upload• Examples – Web proxy – Email server – File server (SMB, FTP, etc) – Full file extractor on live network stream © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Producer Example#!/usr/bin/rubyrequire ‘rest_client’RestClient.post url, { :upload => { :upload => File.new(filename) }, :transfer => { :param1 => value1, :param2 => value2 }} © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Consumers• Scanners that examine or possibly modify files submitted by producers – Start from scratch • Code it right into your own tools – Leverage existing tools • Write a wrapper for a COTS product you already have• Return results to be correlated © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Examples of Consumers COTS FOSS/Custom• AV • AV – Individual (ClamAV, AVG, etc) – Individual – Aggregate (VirusTotal) (Symantec, McAfee, etc) • Content Detonation – Aggregate (MetaScan) – Honeynet Project • Yara• Content Detonation – Public signatures – FireEye – Create your own! • Archive Extraction• File Profiling – Zip/Tar/ISO – HBGary FingerPrint – DD Image. Forensics anybody? • Covert Data Channels – Find indicators and quickly weaponize them © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Consumer Exampledef subscribe() AMQP.start(:host => HOST ) do amq = MQ.new q = amq.queue(QUEUE_NAME) ex = amq.topic(MESSAGES_EX) q.bind(ex, :key => "image.#") q.subscribe(:ack => true) do |hdr,body| yield hdr,body hdr.ack end endend © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Returning Resultsdef publish(hash) AMQP.start(:host => HOST ) do amq = MQ.new ex = amq.fanout(RESULTS_EX) ex.publish( hash[:body], :headers => hash[:headers] ) endend © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • INTERSECT INTERNALS © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Brains• Keep the trains running…• Accepts file submissions• Submit files to bus• Collect results created by consumers• Correlate results from consumers © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Front End• HTTP Interface• Producers can upload files – By POSTing files• Analysts can view results – Through the Ruby on Rails app• Consumers can download files to analyze – Via a simple static file serving © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Ruby on Rails App• Allows for rapid web development – Abstracts away everything for you• WEBrick – Standard, lightweight development web server – Holding up to pretty much whatever we throw at it• EventMachine – Extremely high scalability, performance and stability for the most demanding production environments – An API that eliminates the complexities of high- performance threaded network programming, allowing engineers to concentrate on their application logic © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Back End• MySQL – Holds all metadata for files and results• File store – All files are stored and renamed to match their MD5 hash to prevent duplication © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Service Bus• The unsung hero• Disclaimer – We know very little about ESB software. We know just enough to say we don’t like them.• Provides basic routing of messages between producers, INTERSECT, and consumers• Allows us to decouple everything – Just connect to the bus © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Tying It Together• Consumers bind to the bus – Simple wrapper script / class to communicate with bus • Use this method to quickly repurpose already existing services and capabilities • Use this method to integrate proprietary solutions with limited interfaces – Integrate directly into your consumer from the start © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Advanced Message Queuing Protocol (AMQP)• Awesome protocol – Lightweight• Developed by some financial companies to facilitate “common business messaging”• Protocol developed by some major technical companies © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • RabbitMQ• Easy to setup – Servers run on Windows, *nix, OSX, OpenVMS?!• Libraries for most languages – Ruby, Java, Perl, Python, .NET, PHP, C, Erlang, Lisp, H askell• Simple to configure and manage – Allows you to spend more time developing – Powerful features • Access control • vhosts • Load balancing / Redundancy © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Exchanges Fanout Topic• Shotgun approach • More precise – All services get all messages – Declare what messages you – Queues fill up want based on routing key – Consumers have to decide – We use filetype • Based on libmagic for now – Reduces load on consumers © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Messages• We keep them as small as possible• Files are not contained in the messages – Some consumers simply operate on metadata• We provide a URL to get the file from INTERSECT © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Message Packet Capture © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Result Packet Capture © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Consumer Workflow © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Load Balancing / Redundancy• RabbitMQ – Multiple instances can be stood up in a cluster• Consumer – Multiple instances can be bound to the same queue – Messages will be delivered to an available instance of a service © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • ADVANTAGES TO OUR APPROACH © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • ‚Single Pane of Glass‛• Consolidate results from disparate services – Correlate those results to find something novel• Search through results and transfers by any amount of metadata• Evaluate efficacy of different services – Did some detect maliciousness and others not? © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Agile• Easy to add new consumers• Resubmit files to new/updated consumers• Provide research projects with relevant test data © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Asynchronous• Queuing allows consumers to take as much time as they need to process files• A failure of one consumer has no effect across the system – Less stable research projects can process real data to better prove their methods © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Leverage Resources• Fully utilize your COTS tools – It was expensive, get your money’s worth – Many of them expect a manual workflow and go underutilized• Throw more hardware at it – Run multiple copies of services © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • WHAT WE NEED © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • What We Need• Need to Gather Intelligence• Need to Protect• Need to Measure Efficacy © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Gather Intelligence• You can never have too much intelligence• Once you have all of the information in one place – Act on it – Analyze it © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Protect• How we’re using it right now• Producers – Web proxy – Network taps – Mail server• Consumers – Lots of file scanners © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Protect• Workflow – File comes in from the network • Somebody is downloading or was sent a file • Scanners do triage • If the file is suspicious – Alert an analyst – They can decide what to do based on your corporate policy • We can accumulate data on files – Retroactively scan files when new tips/signatures come out – Start to tie different files to the same attackers © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Statistics• Began ingesting files from multiple enterprise producers• Since 14-Dec – ~175,000 unique files – Averaging ~4500 unique files daily – Max ~7800 files in a day – No backlog! © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Measure Efficacy• With all of that data – You can see how your COTS tools compare to your research projects – You can see how your research projects are progressing • Run scans with one version • After you make some changes, run new scans and compare © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Other Possible Use Cases• Egress Filtering – Write scanners that look for SSN, Credit Card Numbers, “Dirty Words”…• File Transfer – Make users put files into the system if they want to bring it into the corporation – Don’t allow them to download it directly • They need to put it into your system and then download it from there after if gets scanned © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • FUTURE IMPROVEMENTS © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Correlation Engine Integration• We are working on utilizing Splunk – Other SIEMs would work too• Provide better UI, alerting, searching, etc.• Don’t reinvent the wheel © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Better Filetype Checking• This is a difficult problem to get perfectly correct• Maybe we can develop services that do this for us… > file --mime-type INTERSECT.* INTERSECT.doc: application/msword INTERSECT.docm: application/zip INTERSECT.docx: application/zip © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Archive ‚Explosion‛• Simplest form, unzip a file and resubmit the children – The “threat” of the archive would be an aggregate of the threat of the children• But what is an “archive”? – Office 2007+ file format • PowerPoint stores each slide as a different file, along with each image individually – Disk images • Write a forensic service that can parse through and pull out all files • Resubmit those files to keep track of all files in a disk image © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.
  • Contact Us• Emails – flindiakos@mitre.org – mpawloski@mitre.org © 2011 The MITRE Corporation. Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.