Downloaded 21 times
Uploaded byDatabricks
AI on Spark for Malware Analysis and Anomalous Threat Detection
The document presents an overview of how Avast employs AI and big data for malware analysis and threat detection, highlighting their structured streaming approach for fast threat identification. It details the machine learning pipeline used for malware classification and the advantages of structured streaming in handling large datasets for real-time anomaly detection. The session concludes with insights on collaboration between science and engineering teams and emphasizes the integration of machine learning tools for effective threat analysis.
More Related Content
Similar to AI on Spark for Malware Analysis and Anomalous Threat Detection
AI on Spark for Malware Analysis and Anomalous Threat Detection
- 1. WIFI SSID:Spark+AISummit |Password: UnifiedDataAnalytics
- 2. Jakub Sanojca &Joāo Da Silva, Avast Researcher Data Engineer
- 3. Jakub Sanojca &Joāo Da Silva, Avast Researcher Data Engineer AI on Spark for Malware Analysis and Anomalous Threat Detection
- 4. Demonstrate how Avastleverages AI and big data to burn malware. Goal
- 5. Demonstrate how Avastleverages AI and big data to burn malware. Goal
- 6. Agenda • What Avastdoes • Malware research • Structured Streaming • AI anomaly detection • Demo
- 7.
- 8. Thank you • BigData Systems • AI team - especially Yura, Olga and Dmitry • Threat researchers and analysts
- 9. Avast is dedicatedto creating a world that provides safety and privacy for all, no matter who you are, where you are, or how you connect.
- 10. Global reach 10#UnifiedDataAnalytics #SparkAISummit Portfolioof security, privacy and utility applications
- 11. World’s Largest DetectionNetwork 300 M+ new files monthly 10,000 + globally distributed servers 200B+ URLs
- 12. 12#UnifiedDataAnalytics #SparkAISummit Training theAvast Machine Learning Engine Purpose-built approach that takes < 12 hours to add new features, train, and deploy into production
- 13. Malware classification 13#UnifiedDataAnalytics #SparkAISummit Data ●>500 handcrafted features from binary files from our experts Task ● Classification to clean/malware/pup files Two step ML Pipeline: ● Cluster data with custom k-means ● Classification inside the cluster is done by Random Forest
- 14. Infrastructure: Underlying datalake - Burger 14#UnifiedDataAnalytics #SparkAISummit
- 15. 15#UnifiedDataAnalytics #SparkAISummit15 Data Features ClusteringTraining Validation Production Clustering Training Validation 3h 4.5h 24 h 24 h 24 h 6 h ● ~700TB of binary files ● patented tailor-made solution Architecture: Malware classification
- 16. Custom application Spark •optimised & performant • takes months to develop • not that easy to change • slower • easy to experiment with • very fast development
- 17.
- 18. 1. Identify -threat researcher 2. Block - operator 3. Analyze and automate - data / AI researcher + engineers 3 step threat approach
- 19. 1. Identify -threat researcher 2. Block - operator 3. Analyze and automate - data / AI researcher + engineers 3 step threat approach
- 20. 1. Identify -threat researcher 2. Block - operator 3. Analyze and automate - data / AI researcher + engineers 3 step threat approach
- 21. • Thousands ofdetection time series • Where should operator focus? Time series of detections
- 22. • Thousands ofdetection time series • Where should operator focus? Time series of detections
- 23. Short response timeis necessary
- 24. Short response timeis necessary
- 25. First idea -custom streaming app • Python because of ML models
- 26. First idea -custom streaming app • Python because of ML models • Big part of code about already solved problems
- 27. First idea -custom streaming app • Python because of ML models • Big part of code about already solved problems • POC written by researchers
- 28. First idea -custom streaming app • Python because of ML models • Big part of code about already solved problems • POC written by researchers • Gets job done, but not easy to maintain or experiment
- 29. Adopted solution: Spark StructuredStreaming 29#UnifiedDataAnalytics #SparkAISummit
- 30.
- 31. Advantages of Structured Streaming forfast threat detection #UnifiedDataAnalytics #SparkAISummit
- 32. Advantages of StructuredStreaming • Unified processing engine 32#UnifiedDataAnalytics #SparkAISummit
- 33. Advantages of StructuredStreaming • Unified processing engine • End to end AI with multiple sinks 33#UnifiedDataAnalytics #SparkAISummit
- 34. Advantages of StructuredStreaming • Unified processing engine • End to end AI with multiple sinks • Window aggregations and Watermarking out of the box 34#UnifiedDataAnalytics #SparkAISummit
- 35. Advantages of StructuredStreaming • Unified processing engine • End to end AI with multiple sinks • Window aggregations and Watermarking out of the box • Resilient streams 35#UnifiedDataAnalytics #SparkAISummit
- 36.
- 37. Structured Streaming Adoption •Unbounded table 37#UnifiedDataAnalytics #SparkAISummit
- 38. Structured Streaming Adoption •Unbounded table • Triggers 38#UnifiedDataAnalytics #SparkAISummit
- 39. Structured Streaming Adoption •Unbounded table • Triggers 39#UnifiedDataAnalytics #SparkAISummit >>> writer = sdf.writeStream.trigger(processingTime='5 seconds')
- 40. Structured Streaming Adoption •Unbounded table • Triggers 40#UnifiedDataAnalytics #SparkAISummit >>> writer = sdf.writeStream.trigger(processingTime='5 seconds') >>> writer = sdf.writeStream.trigger(once=True)
- 41. Structured Streaming Adoption •Unbounded table • Triggers 41#UnifiedDataAnalytics #SparkAISummit >>> writer = sdf.writeStream.trigger(processingTime='5 seconds') >>> writer = sdf.writeStream.trigger(once=True) >>> writer = sdf.writeStream.trigger(continuous='5 seconds')
- 42. Structured Streaming Adoption •Unbounded table • Triggers • Micro Batch Processing vs Continuous processing 42#UnifiedDataAnalytics #SparkAISummit
- 43. Structured Streaming Adoption •Unbounded table • Triggers • Micro Batch Processing vs Continuous processing – org.apache.spark.sql.execution.streaming.MicroBatchExecution 43#UnifiedDataAnalytics #SparkAISummit
- 44. Structured Streaming Adoption •Unbounded table • Triggers • Micro Batch Processing vs Continuous processing – org.apache.spark.sql.execution.streaming.MicroBatchExecution – org.apache.spark.sql.execution.streaming.ContinuousExecution (experimental) 44#UnifiedDataAnalytics #SparkAISummit
- 45. Structured Streaming Adoption •Unbounded table • Triggers • Micro Batch Processing vs Continuous processing 45#UnifiedDataAnalytics #SparkAISummit
- 46.
- 47.
- 48.
- 49.
- 50. #UnifiedDataAnalytics #SparkAISummit AI drivenanomaly detection on time series
- 51. How to quicklyidentify campaigns of malware and potentially unwanted programs. 51#UnifiedDataAnalytics #SparkAISummit AI driven anomaly detection on time series
- 52. How to quicklyidentify campaigns of malware and potentially unwanted programs: • Traditional approaches - find outliers 52#UnifiedDataAnalytics #SparkAISummit AI driven anomaly detection on time series
- 53. How to quicklyidentify campaigns of malware and potentially unwanted programs. • Traditional approaches - find outliers • Machine learning - predict and compare – Neural networks - LSTMs vs CNNs 53#UnifiedDataAnalytics #SparkAISummit AI driven anomaly detection on time series
- 54. How to quicklyidentify campaigns of malware and potentially unwanted programs. • Traditional approaches - find outliers • Machine learning - predict and compare – Neural networks - LSTMs vs CNNs – Other - auto-regressive models etc. 54#UnifiedDataAnalytics #SparkAISummit AI driven anomaly detection on time series
- 55. • Sequential 55#UnifiedDataAnalytics #SparkAISummit Threatanomaly detection: training
- 56. • Sequential • Parallel!mapPartitions / pandas_udf 56#UnifiedDataAnalytics #SparkAISummit Threat anomaly detection: training
- 57. • Sequential • Parallel! •Distributed - TensorflowOnSpark 57#UnifiedDataAnalytics #SparkAISummit Threat anomaly detection: training
- 58. • pandas_udf forparallel predictions • super easy to test on already stored data as batch job 58#UnifiedDataAnalytics #SparkAISummit Threat anomaly detection: stream serving
- 59. Demo + CodeWalkthrough 59#UnifiedDataAnalytics #SparkAISummit
- 60. Challenges 60#UnifiedDataAnalytics #SparkAISummit • Multiplepotential incompatibility surfaces • Unexpected behavior / Unknowns • Silent failures
- 61. Takeaways • Easier collaborationbetween Science and Engineering teams • An excellent toolbox to do anomaly detection in near real time • Easy ML/AI/DL integration • Parallelism 61#UnifiedDataAnalytics #SparkAISummit
- 62. Questions? Jakub Sanojca &Joāo Da Silva, Avast Researcher Data Engineer