Uploaded byDataWorks Summit
Large scale anomaly detection in cyber-security
The document discusses using big data analytics and machine learning techniques for cybersecurity applications like anomaly detection. It describes Thales' CENTAI lab and its work developing batch and real-time anomaly detection algorithms using techniques like copula-based density estimation. The algorithms were initially developed as Pig UDFs and then ported to Spark to enable both batch and streaming analysis on large logs. The goal is to improve detection of targeted and non-targeted cyber attacks.
More Related Content
More from DataWorks Summit
Large scale anomaly detection in cyber-security
- 1. www.thalesgroup.com OPEN Large scale anomaly detectionin cyber-security YVES MABIALA
- 2. 2 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. Agenda ▌The CENTAI lab ▌Challengesin cyber-attacks detection ▌How to improve detection capabilities ? ▌Batch anomaly detection in Pig ▌From batch to real-time ▌Conclusion
- 3. 3 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. CENTAI ecosystem Customers Concepts TRL 1-4 Prototypes TRL4-6 Products / Solutions Applications TRL 7-9 University of Bordeaux LABRI UPMC – LIP6 (Paris) LIRIS (Lyon) TRT & Innovation Hubs Algorithms co-dev and transfer Proof of Concept / Proof of Technos TRL = Technical Readiness Level Start-up & SMEs Thales Business Lines Big Data Big Analytics Visual Analytics CENTAI
- 4. 4 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. Challenges in cyber-attacksdetection Two kinds of attacks Non targeted - General public, Not very complex, Using common attack patterns Targeted - Using very customized attack patterns - Made through large periods of time APT !
- 5. 5 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. How to improvedetection capabilities ? Traditional approaches Based on expert rules (detection or correlation) Often not sized for handling massive data Need innovative tools to Detect abnormal behaviors without relying on known patterns Perform analysis on large periods of time
- 6. 6 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. What about thedata ? Data collected at multiple levels Applicative/ Network / System Data are Big DATA Massive and heterogeneous (hundred of TBs of logs) Highly dynamic (x10GB/s for network flows/streams)
- 7. 7 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. Batch analysis ofapplicative/network logs Context Audit of logs Data Network logs (proxy and firewalls) Hundred billions of events Non labeled Objective Detection of abnormal events in these logs (unsupervised anomaly detection)
- 8. 8 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. Anomaly detection challenges Open-sourcealgorithm libraries will not help much No unsupervised anomaly detection implemented in most big data analytics frameworks Data is massive and highly dimensional (x hundreds variables) Linear complexity mandatory Very law false positives rate must be ensured Outputs must be as “understandable” as possible
- 9. 9 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. Copula anomaly detection Developmentof a set of tools in PIG Probabilistic anomaly detection algorithm - Based on the notion of copula 1 1 2 1 3Event Variable 1 Variable 2 Variable n Density 1 Density n Density 2 Joint Density Anomaly / Normal event Density estimation Copula Thresholding
- 10. 10 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. A few wordson copula Express a multivariate cumulative distribution function as a function of its univariate marginal distributions Π 𝑢1, 𝑢2 = ℙ 𝑈1 ≤ 𝑢1, 𝑈2 ≤ 𝑢2 = C(F 𝑢1 , G 𝑢2 ) Many types of copulas Parametric : archimedean (e.g. : C 𝑢, 𝑣 = 𝑢𝑣) , gaussian Non parametric : empirical Development of a new copula : Indetermination criteria 𝐶 𝑢1, 𝑢2 = 𝑢2 𝐹−1(𝑢1) 𝐴 + 𝑢1 𝐺−1(𝑢2) 𝐵 - 𝐹−1(𝑢1)𝐺−1(𝑢2) 𝐴𝐵
- 11. 11 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. Implementation in Pig Implementationas Pig algebraic UDFs (fully incremental) Marginal estimation using KDE Joint distribution estimation using Copula Threshold estimation with quantile regression/extreme value Integration of the UDFs in two scripts Learning Scoring
- 12.
- 13.
- 14. 14 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. From pure batchanalysis to batch and stream Very satisfied of this first implementation Quite easy to create customized UDF Very good performance But Need to add some real-time processing capabilities - While avoiding the burden of coding the algorithm twice How to combine both capabilities ?
- 15. 15 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. 5 – Querylayer3 – Serving layer 4 – Speed layer 2 – Batch layer 1 - Collect Lambda architecture
- 16. 16 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. Lambda architecture 5 –Query layer3 – Serving layer 4 – Speed layer 2 – Batch layer1 - Collect
- 17. 17 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. From Pig toSpark Implementation as Spark Accumulator Very easy to transfer the code from Pig UDFs Pros An order of magnitude faster Easier to share information between functions Same implementation for batch and real-time Cons Lost data typing and variable names
- 18.
- 19. 19 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. Conclusion Big Data analyticsbecomes essential in cyber-security Cyber-attacks detection Investigation and Forensics Situation awareness Spark greatly helps to leverage the power of big data analytics for batch and real-time applications What is next ? Provide decision aid and support tools
- 20. 20 OPEN Thisdocumentmaynotbereproduced,modified,adapted,published,translated,inanyway,inwholeorin partordisclosedtoathirdpartywithoutthepriorwrittenconsentofThales-©Thales2015Allrightsreserved. Thank you foryour attention ▌Any questions ? Yves MABIALA yves.mabiala@thalesgroup.com