SlideShare a Scribd company logo
1 of 25
Download to read offline
John Munro / jmunro@endgame.com
              Jason Trost / jtrost@endgame.com

FlonCon 2013 | January 7–10 | Albuquerque, New Mexico
Introductions
 • John Munro (jmunro@endgame.com)
   – Network Security Researcher and Data Scientist


 • Jason Trost (jtrost@endgame.com)
   – Senior Software Engineer
   – Specializes in Hadoop/Storm/BigData
Agenda

 •   The Problem
 •   Our Approach
 •   DGA Domain Classifier
 •   String Statistics as Features
 •   Malicious Domain Classifier
 •   Demo
 •   Real-time Streaming Platform
The Problem
               txmxbo.info                              youtube.com

      yahoo.com
                                Ct0u2xj5dbe4.wvw—game465.com

  p4.httzd5e2ufizo.3bawhfuec45dca65.401724.s1.v4.ipv6-exp.l.google.com


           abulqe.com                       za6.limfoklubs.com

  ns3.ohio.gov                                      bibz01.apple.com
                              docs.joomla.org
 Wmk41035u3751s0bgv4n91b0b7h74v.ipcheker.com
The Problem
               txmxbo.info                              youtube.com

      yahoo.com
                                Ct0u2xj5dbe4.wvw—game465.com

  p4.httzd5e2ufizo.3bawhfuec45dca65.401724.s1.v4.ipv6-exp.l.google.com


           abulqe.com                       za6.limfoklubs.com

  ns3.ohio.gov                                      bibz01.apple.com
                              docs.joomla.org
 Wmk41035u3751s0bgv4n91b0b7h74v.ipcheker.com
The Problem

• Massive Volumes
  – Some of our partners
    deal with TBs per day
    of DNS PCAPs
• Incredible Rates
  – One partner sees
    13k requests/sec
  – Another closer to
    100k/sec
Our Approach: Machine Learning!

 • Real-time streaming classification
   – In parallel across multiple servers
 • Markov Models
   – Random Domain Generation Traffic
   – Normal Benign Traffic
 • Random Forests
   – Benign vs Malicious
 • Periodically retrained
   – In order to maintain accuracy
Data Sources
 • Benign Domains
   – Millions of popular, real domains
      • Correlated with the Alexa top 10k domains
 • Malicious Domains
   – 800k domains gathered from an internal malware
     sandbox
   – Public blacklist domains from Conficker and
     Murofet Botnets
Markov Models
Markovian DGA Classifier
 • Domain Generation Algorithm (DGA)
 • Popular Domain Model
   – Trained: 258,039 domains from Day 1 of our Benign set
   – Tested: 331,359 domains from Day 2 of our
     Benign set
   – Accuracy: 99.40 % with 1,458 Unknown
 • Randomly Generated Domain Model
   – Trained: 90,884 domains from Conficker Botnet
   – Tested: 295,306 domains from Murofet Botnet
   – Accuracy: 99.34 % with 1,923 Unknown
String Statistics as Features
Feature Usefulness
Random Forests Algorithm




                  FPO
             VIDEO TO COME
Random Forests
 • Pros:
    –   Very high accuracy
    –   Scalable across many nodes
    –   Built-in protection from over fitting
    –   Can handle very large data sets with many features
    –   Robust with respect to goodness of features
    –   Practical for real world use
    –   Does not assume a distribution
    –   Only two parameters to tune
    –   Memory efficient
 • Cons:
    – Not the quickest classifier, but plenty fast in practice
Malicious Domain Classifier


 • Performance measured by 10 – fold Cross
   Validation
 • Training Set            Cross Validation
   – 200k Benign                         0.06

                                         0.05

   – 200k Malicious
                      Out of Bag Error



                                         0.04

                                         0.03

                                         0.02

                                         0.01

                                           0
                                                10   20   30     40     50      60       70   80   90   100
                                                                      Number of Trees

                                                               K=3      K=5          K = 10
Results
                               Bad Precision                                                                   Good Precision
            0.984                                                                           0.9795
            0.983                                                                            0.979
            0.982                                                                           0.9785
            0.981                                                                            0.978
                                                                                            0.9775




                                                                                Precision
Precision




             0.98
                                                                                             0.977
            0.979
                                                                                            0.9765
            0.978                                                                            0.976
            0.977                                                                           0.9755
            0.976                                                                            0.975
            0.975                                                                           0.9745
                     10   20   30   40      50    60       70   80   90   100                        10   20    30   40      50    60       70    80   90     100
                                         Number of Trees                                                                  Number of Trees
                               Bad Accuracy                                                                    Good Accuracy
            0.9805                                                                          0.9805
              0.98                                                                            0.98
            0.9795                                                                          0.9795
             0.979                                                                           0.979
Accuracy




                                                                                Accuracy

            0.9785                                                                          0.9785
             0.978                                                                           0.978
            0.9775                                                                          0.9775
             0.977                                                                           0.977
            0.9765                                                                          0.9765
             0.976                                                                           0.976
                     10   20   30   40      50    60       70   80   90   100                        10   20    30   40      50    60       70    80   90     100
                                         Number of Trees                                                                  Number of Trees
                                                                                                                                  K=3            K=5        K = 10
Results
                                                                          Model Size
                                              400
                                              350
                                              300
                            Model Size (MB)   250
                                              200                                                                K=3
                                              150                                                                K=5
                                              100                                                                K=10
                                               50
                                                0
                                                    10   20     30   40      50    60       70   80   90   100
                                                                          Number of Trees



                                                              Classification Throughput
                             60,000

                             50,000
      Classifications/sec




                             40,000                                                                              K=3

                             30,000                                                                              K=5
                                                                                                                 K=10
                             20,000

                             10,000

                                                0
                                                    10   20     30   40      50    60       70   80   90   100
                                                                          Number of Trees
Results
Demo
Realtime Streaming Platform

 • Velocity is a platform for processing, analyzing,
   and visualizing large-scale event data in real-
   time
 • It was designed to be horizontally scalable and
   is built using Twitter’s Storm

                • It was built primarily for internal
                  use with DNS events, IDS alerts,
                  and netflow data, but it is in the
                  process of being commercialized
Velocity Pipeline
Conclusion
 • Malicious domain classification
 • DGA domain identification using Markov
   Models
 • Summary Statistics based on domain string
   work well
 • Random Forests are very successful at
   classifying domains as Benign or Malicious
 • Real-time, distributed implementation
Future Work

 • Include more features: TTL, frequency seen,
   etc.
 • Correlation of bad domains based on ASN,
   Country, Organization, etc.
 • Identify subnets that are infected based on
   high traffic to bad domains
 • Identify Content Delivery Networks
 • Self Organizing Maps and other visualizations
Questions
Contact Information
 • John Munro
 • Email: jmunro@endgame.com

 •   Jason Trost
 •   Email: jtrost@endgame.com
 •   Twitter: @jason_trost
 •   Blog: www.covert.io

More Related Content

More from Jason Trost

BSidesNYC 2016 - An Adversarial View of SaaS Malware Sandboxes
BSidesNYC 2016 - An Adversarial View of SaaS Malware SandboxesBSidesNYC 2016 - An Adversarial View of SaaS Malware Sandboxes
BSidesNYC 2016 - An Adversarial View of SaaS Malware SandboxesJason Trost
 
Distributed Sensor Data Contextualization for Threat Intelligence Analysis
Distributed Sensor Data Contextualization for Threat Intelligence AnalysisDistributed Sensor Data Contextualization for Threat Intelligence Analysis
Distributed Sensor Data Contextualization for Threat Intelligence AnalysisJason Trost
 
An Adversarial View of SaaS Malware Sandboxes
An Adversarial View of SaaS Malware SandboxesAn Adversarial View of SaaS Malware Sandboxes
An Adversarial View of SaaS Malware SandboxesJason Trost
 
Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...
Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...
Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...Jason Trost
 
Lessons Learned from Building and Running MHN, the World's Largest Crowdsourc...
Lessons Learned from Building and Running MHN, the World's Largest Crowdsourc...Lessons Learned from Building and Running MHN, the World's Largest Crowdsourc...
Lessons Learned from Building and Running MHN, the World's Largest Crowdsourc...Jason Trost
 
Modern Honey Network at Bay Area Open Source Security Hackers
Modern Honey Network at Bay Area Open Source Security HackersModern Honey Network at Bay Area Open Source Security Hackers
Modern Honey Network at Bay Area Open Source Security HackersJason Trost
 
Modern Honey Network (MHN)
Modern Honey Network (MHN)Modern Honey Network (MHN)
Modern Honey Network (MHN)Jason Trost
 
BinaryPig - Scalable Malware Analytics in Hadoop
BinaryPig - Scalable Malware Analytics in HadoopBinaryPig - Scalable Malware Analytics in Hadoop
BinaryPig - Scalable Malware Analytics in HadoopJason Trost
 
Accumulo Nutch/GORA, Storm, and Pig
Accumulo Nutch/GORA, Storm, and PigAccumulo Nutch/GORA, Storm, and Pig
Accumulo Nutch/GORA, Storm, and PigJason Trost
 

More from Jason Trost (9)

BSidesNYC 2016 - An Adversarial View of SaaS Malware Sandboxes
BSidesNYC 2016 - An Adversarial View of SaaS Malware SandboxesBSidesNYC 2016 - An Adversarial View of SaaS Malware Sandboxes
BSidesNYC 2016 - An Adversarial View of SaaS Malware Sandboxes
 
Distributed Sensor Data Contextualization for Threat Intelligence Analysis
Distributed Sensor Data Contextualization for Threat Intelligence AnalysisDistributed Sensor Data Contextualization for Threat Intelligence Analysis
Distributed Sensor Data Contextualization for Threat Intelligence Analysis
 
An Adversarial View of SaaS Malware Sandboxes
An Adversarial View of SaaS Malware SandboxesAn Adversarial View of SaaS Malware Sandboxes
An Adversarial View of SaaS Malware Sandboxes
 
Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...
Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...
Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...
 
Lessons Learned from Building and Running MHN, the World's Largest Crowdsourc...
Lessons Learned from Building and Running MHN, the World's Largest Crowdsourc...Lessons Learned from Building and Running MHN, the World's Largest Crowdsourc...
Lessons Learned from Building and Running MHN, the World's Largest Crowdsourc...
 
Modern Honey Network at Bay Area Open Source Security Hackers
Modern Honey Network at Bay Area Open Source Security HackersModern Honey Network at Bay Area Open Source Security Hackers
Modern Honey Network at Bay Area Open Source Security Hackers
 
Modern Honey Network (MHN)
Modern Honey Network (MHN)Modern Honey Network (MHN)
Modern Honey Network (MHN)
 
BinaryPig - Scalable Malware Analytics in Hadoop
BinaryPig - Scalable Malware Analytics in HadoopBinaryPig - Scalable Malware Analytics in Hadoop
BinaryPig - Scalable Malware Analytics in Hadoop
 
Accumulo Nutch/GORA, Storm, and Pig
Accumulo Nutch/GORA, Storm, and PigAccumulo Nutch/GORA, Storm, and Pig
Accumulo Nutch/GORA, Storm, and Pig
 

Recently uploaded

SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTxtailishbaloch
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3DianaGray10
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxNeo4j
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationKnoldus Inc.
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024Brian Pichman
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTopCSSGallery
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarThousandEyes
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingFrancesco Corti
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdfThe Good Food Institute
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 

Recently uploaded (20)

SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development Companies
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 

Clairvoyant Squirrel: Large Scale Malicious Domain Classification

  • 1. John Munro / jmunro@endgame.com Jason Trost / jtrost@endgame.com FlonCon 2013 | January 7–10 | Albuquerque, New Mexico
  • 2. Introductions • John Munro (jmunro@endgame.com) – Network Security Researcher and Data Scientist • Jason Trost (jtrost@endgame.com) – Senior Software Engineer – Specializes in Hadoop/Storm/BigData
  • 3. Agenda • The Problem • Our Approach • DGA Domain Classifier • String Statistics as Features • Malicious Domain Classifier • Demo • Real-time Streaming Platform
  • 4. The Problem txmxbo.info youtube.com yahoo.com Ct0u2xj5dbe4.wvw—game465.com p4.httzd5e2ufizo.3bawhfuec45dca65.401724.s1.v4.ipv6-exp.l.google.com abulqe.com za6.limfoklubs.com ns3.ohio.gov bibz01.apple.com docs.joomla.org Wmk41035u3751s0bgv4n91b0b7h74v.ipcheker.com
  • 5. The Problem txmxbo.info youtube.com yahoo.com Ct0u2xj5dbe4.wvw—game465.com p4.httzd5e2ufizo.3bawhfuec45dca65.401724.s1.v4.ipv6-exp.l.google.com abulqe.com za6.limfoklubs.com ns3.ohio.gov bibz01.apple.com docs.joomla.org Wmk41035u3751s0bgv4n91b0b7h74v.ipcheker.com
  • 6. The Problem • Massive Volumes – Some of our partners deal with TBs per day of DNS PCAPs • Incredible Rates – One partner sees 13k requests/sec – Another closer to 100k/sec
  • 7. Our Approach: Machine Learning! • Real-time streaming classification – In parallel across multiple servers • Markov Models – Random Domain Generation Traffic – Normal Benign Traffic • Random Forests – Benign vs Malicious • Periodically retrained – In order to maintain accuracy
  • 8. Data Sources • Benign Domains – Millions of popular, real domains • Correlated with the Alexa top 10k domains • Malicious Domains – 800k domains gathered from an internal malware sandbox – Public blacklist domains from Conficker and Murofet Botnets
  • 10. Markovian DGA Classifier • Domain Generation Algorithm (DGA) • Popular Domain Model – Trained: 258,039 domains from Day 1 of our Benign set – Tested: 331,359 domains from Day 2 of our Benign set – Accuracy: 99.40 % with 1,458 Unknown • Randomly Generated Domain Model – Trained: 90,884 domains from Conficker Botnet – Tested: 295,306 domains from Murofet Botnet – Accuracy: 99.34 % with 1,923 Unknown
  • 13. Random Forests Algorithm FPO VIDEO TO COME
  • 14. Random Forests • Pros: – Very high accuracy – Scalable across many nodes – Built-in protection from over fitting – Can handle very large data sets with many features – Robust with respect to goodness of features – Practical for real world use – Does not assume a distribution – Only two parameters to tune – Memory efficient • Cons: – Not the quickest classifier, but plenty fast in practice
  • 15. Malicious Domain Classifier • Performance measured by 10 – fold Cross Validation • Training Set Cross Validation – 200k Benign 0.06 0.05 – 200k Malicious Out of Bag Error 0.04 0.03 0.02 0.01 0 10 20 30 40 50 60 70 80 90 100 Number of Trees K=3 K=5 K = 10
  • 16. Results Bad Precision Good Precision 0.984 0.9795 0.983 0.979 0.982 0.9785 0.981 0.978 0.9775 Precision Precision 0.98 0.977 0.979 0.9765 0.978 0.976 0.977 0.9755 0.976 0.975 0.975 0.9745 10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100 Number of Trees Number of Trees Bad Accuracy Good Accuracy 0.9805 0.9805 0.98 0.98 0.9795 0.9795 0.979 0.979 Accuracy Accuracy 0.9785 0.9785 0.978 0.978 0.9775 0.9775 0.977 0.977 0.9765 0.9765 0.976 0.976 10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100 Number of Trees Number of Trees K=3 K=5 K = 10
  • 17. Results Model Size 400 350 300 Model Size (MB) 250 200 K=3 150 K=5 100 K=10 50 0 10 20 30 40 50 60 70 80 90 100 Number of Trees Classification Throughput 60,000 50,000 Classifications/sec 40,000 K=3 30,000 K=5 K=10 20,000 10,000 0 10 20 30 40 50 60 70 80 90 100 Number of Trees
  • 19. Demo
  • 20. Realtime Streaming Platform • Velocity is a platform for processing, analyzing, and visualizing large-scale event data in real- time • It was designed to be horizontally scalable and is built using Twitter’s Storm • It was built primarily for internal use with DNS events, IDS alerts, and netflow data, but it is in the process of being commercialized
  • 22. Conclusion • Malicious domain classification • DGA domain identification using Markov Models • Summary Statistics based on domain string work well • Random Forests are very successful at classifying domains as Benign or Malicious • Real-time, distributed implementation
  • 23. Future Work • Include more features: TTL, frequency seen, etc. • Correlation of bad domains based on ASN, Country, Organization, etc. • Identify subnets that are infected based on high traffic to bad domains • Identify Content Delivery Networks • Self Organizing Maps and other visualizations
  • 25. Contact Information • John Munro • Email: jmunro@endgame.com • Jason Trost • Email: jtrost@endgame.com • Twitter: @jason_trost • Blog: www.covert.io