The document discusses the process of malware identification using machine learning, outlining the necessity for automated malware analysis due to the increasing complexity of malware threats. It details the stages of malware detection by antivirus companies, which include in-house analysis, training machine learning models on both benign and malicious samples, and real-time classification on customer machines. Various machine learning algorithms such as logistic regression, k-nearest neighbors, decision trees, and random forests are identified as tools for improving classification efficacy.

1. Malware identification using
Machine Learning
By: Japneet Singh

2. Agenda
• Malware identification/detection process
• Need for automated malware analysis and detection
• Machine Learning
• Identifying malware using Supervised Learning
• Demo
• References

3. What’s a malware?

4. How AV companies identify malware?
Stage 1 - In-House
• Manual/Automated analysis of digital artifacts
• Static analysis
• Dynamic analysis
• Signature generation and packaging
Stage 2 - On customer machines
• Classifying digital artifacts by matching against signatures
• Taking actions

5. What’s in a signature
• Specific indicators like file path, name, size, network connections etc.
• Generic indicators like unique patterns in malware files
• Blacklisted file hashes
• Behavior indicators like sequences of malicious actions

6. Need for automated malware identification
• Malware detection has turned into big data problem
• Most of the new malware is almost identical to
existing malware

7. Machine Learning

8. Supervised learning
• Suitable for Classification problems and Regression problems

9. Classification vs Regression

10. Classification process at high level

11. Classification algorithms
• Logistic regression
• k-nearest neighbors
• Decision trees
• Random forests

12. Visualizing detection boundary: 2 features
Image source: Malware Data Science by Joshua Saxe and Hillary Sanders

13. Visualizing detection boundary: 3 features
Image source: Malware Data Science by Joshua Saxe and Hillary Sanders

14. Example datasets
Image source: Malware Data Science by Joshua Saxe and Hillary Sanders

15. Logistic Regression
Image source: Malware Data Science by Joshua Saxe and Hillary Sanders

16. k-nearest neighbors
Image source: Malware Data Science by Joshua Saxe and Hillary Sanders

17. k-nearest neighbors
Image source: Malware Data Science by Joshua Saxe and Hillary Sanders

18. Decision Tree
Image source: Malware Data Science by Joshua Saxe and Hillary Sanders

19. Decision Tree
Image source: Malware Data Science by Joshua Saxe and Hillary Sanders

20. Random forest

21. Random forest
Image source: Malware Data Science by Joshua Saxe and Hillary Sanders

22. How AV companies identify Malware: Part 2
Stage 1 - In-House
• Collect malware and benign samples
• Divide samples into Train and Test sets
• Train the ML model using Train set
• Extract features from malware and benign files
• Run ML algorithm to train a model from extracted features
• Test ML model efficacy using Test set
• If efficacy does not meet expectations, Iterate
• If efficacy meets expectations, bundle the model with signatures
Stage 2 - On customer machines
• Extract features from files
• Use ML algorithm and trained ML model to classify the file

23. Demo
• Using an open source malware classifier “Ember” to train and classify

24. References
• https://www.av-test.org/en/statistics/malware/
• Ember source: https://github.com/endgameinc/ember
•
https://www.amazon.com/Malware-Data-Science-Detection-
Attribution/dp/1593278594