Download to read offline
Uploaded byCSIRO
Getting better at detecting anomalies by using ensembles
The document discusses improving anomaly detection through ensembles and Item Response Theory (IRT), highlighting its applications in areas like fraud detection and social sciences. It explains the process of fitting IRT models to unsupervised anomaly detection methods, yielding a score representing the anomaly levels of observations. The paper presents results indicating the effectiveness of IRT-based ensembles for detecting anomalies in real datasets.
More Related Content
Similar to Getting better at detecting anomalies by using ensembles
Getting better at detecting anomalies by using ensembles
- 1. Getting better atdetecting anomalies by using ensembles Sevvandi Kandanaarachchi ANZIAM 2022 February 7 2022
- 2. Suppose • After yearsof scanning the skies we find intelligent life on a planet far, far, away . . . • This is an anomaly. A very interesting anomaly . . . . • We want to know about this anomaly. • What kind of life forms? • What kind of technology? • Do they use Maths and Stats?
- 3. Why anomalies? • Theytell a different story • Fraudulent credit card transactions amongst billions of legitimate transactions • Computer network intrusions • Astronomical anomalies – solar flares • Training a model on certain fraud/intrusions/cyber attacks is not optimal, because there are new types of fraud/attacks, always! • You want to be alerted when weird things happen. • Anomaly detection is used in these applications. 3
- 4. Overview • Anomaly DetectionEnsembles • Item Response Theory • Mapping IRT to anomaly detection ensembles • Fitting the IRT model • Some Results
- 5. What is ananomaly detection ensemble? Dataset Unsupervised AD methods • The AD methods are heterogenous methods • Ensembles – use existing methods to come up with better anomaly detection/scores AD ensemble Ensemble Score
- 6. What is ItemResponse Theory (IRT)? • A set of models used in psychometrics/social sciences • In education • Find the difficulty and discrimination of test questions • Ability of participants • In social sciences - unobservable characteristics and observed outcomes • Verbal or mathematical ability • Racial prejudice or stress proneness • Political inclinations • Intrinsic “quality” that cannot be measured directly • Latent trait modelling This Photo by Unknown Author is licensed under CC BY-SA
- 7. IRT in education •𝑁 Students (participants) answer 𝑛 questions (test item) • Your input to the IRT model is a matrix of responses/accuracies 𝑌𝑁×𝑛 • Fit the IRT model • You get as your output • Student ability (latent trait continuum) • Test item discrimination • Test item difficulty 7 Q 1 Q 2 Q 3 Q 4 Stu 1 0.95 0.87 0.67 0.84 Stu 2 0.57 0.49 0.78 0.77 Stu n 0.75 0.86 0.57 0.45
- 8. IRT in psychometrics(e.g. racial prejudice) • The matrix 𝑌𝑁×𝑛 contains the original responses • row = person, column = question • Rosenberg's Self-Esteem Scale - I feel I am a person of worth (Strongly Agree/ ... ) • No correct or wrong answer • Latent trait – the self esteem of the participants Participant 1 Participant N Item 1 Item n Matrix of original responses 𝑌𝑁×𝑛 IRT Model
- 9. IRT ensemble foranomaly detection • The matrix 𝑌𝑁×𝑛 contains the anomaly scores – these are not accuracy measures because we are doing unsupervised anomaly detection. • Row = observation, column = AD method • 𝑌𝑁×𝑛 has the output of the AD methods → Fit the IRT model • Latent trait – the anomalousness of the observations – the ensemble score • High values → high anomalousness, low values → low anomalousness Observation 1 Observation N AD method 1 AD method n Matrix of anomaly responses/scores 𝑌𝑁×𝑛 IRT Model
- 10. Fitting the IRTmodel • Maximising the expectation • 𝐸 = 𝑁 𝑗(ln 𝛼𝑗 + ln |𝛾𝑗|) − 1/ 2 𝑖 𝑗 𝛼𝑗 2 𝛽𝑗 + 𝛾𝑗𝑧𝑖𝑗 −
- 11. And the ensemblescores . . . • We get the latent trait parameters • 𝜃𝑖 = 𝑗 𝛼𝑗 2 (𝛽𝑗+𝛾𝑗𝑧𝑖𝑗) 𝑗 𝛼𝑗 2 • 𝜃𝑖 - ensemble score for the 𝑖𝑡ℎ observation • 𝛼𝑗 - discrimination parameter of 𝑗𝑡ℎ AD method • 𝛾𝑗 - scaling parameter for the 𝑗𝑡ℎ AD method • 𝛽𝑗 - difficulty parameter for the • 𝑧𝑖𝑗 - anomaly score of the 𝑗𝑡ℎ AD method on the 𝑖𝑡ℎ observation
- 12. Example 1 • ADmethods (DDoutlier, h2o, e1071) • KNN_AGG • LOF • COF • INFLO • KDEOS • LDF • LDOF • Autoencoders – Deep learning • OCSVM – One class Support Vector Machine • Isolation Forest – Tree based method Nearest neighbourhood-based methods Density/distance based Dataset Unsupervised AD methods AD ensemble Ensemble Score
- 13. Unsupervised AD methodsoutput
- 14.
- 15.
- 16. Example with iterations Data in𝑅6 - first 2 dimensions shown, others normally distributed Evaluation metric – Area under ROC Iteration 5 Iteration 10
- 17. Summary • Anomaly detectionin many applications • IRT – used in social sciences/psychometrics • Use IRT for AD ensembles • Tests on real datasets - good results • More details on the paper - https://arxiv.org/abs/2106.06243- @Information Sciences • R package – outlierensembles – on CRAN - extends R package EstCRM for IRT
- 18. 18 On ArXiv -https://arxiv.org/abs/2106.06243