SlideShare a Scribd company logo
1 of 26
Stony Brook University 
Shebuti & Leman 
Shebuti Rayana Leman Akoglu
Tax evasion Credit card fraud 
& Many More… 
Network intrusion 
Healthcare fraud 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
2
 Problem: Given a sequence of graphs, 
Q1. Event detection: find time points at which 
graph changes significantly 
Q2. Characterization: find (top k) nodes / edges / 
regions that change the most 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
3
 Main framework 
 Compute graph similarity/distance scores 
… … … 
time 
 Find unusual occurrences in time series 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
4
 Flow of Ensemble Approach 
 Event Detection in Dynamic Graphs 
Ensemble Algorithms 
Eigen Behavior based Event Detection (EBED) 
Probabilistic Approach (PTSAD) 
SPIRIT 
Consensus Method 
Rank based 
Score based 
Results 
Dataset 1: Challenge Network flow Data 
Dataset 2: New York Times News Corpus 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
5
Event Detection 
Consensus Rank Merging 
•Rank based 
•Inverse Rank 
•Kemeny Young 
•Score Based 
•Unification 
(avg, max) 
•Mixture Model (avg, 
max) 
• Final Ensemble 
(Inverse Rank) 
Characterization 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 6
 Numerous algorithms for event detection 
 Hard to decide which one will work well 
for a specific data set 
 Our Goal: design an ensemble approach which 
might not give best result 
but “better” than most base algorithms 
 Challenges: 
 Different scores/scales 
 Different merging approaches 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
7
 Extract “typical behavior” (eigen-behavior) of 
nodes/edges 
 eigen-behavior ≡ principal eigen-vector 
 Compare eigen-behavior over time 
 Score the time ticks depending on 
amount of change in behavior 
from previous time tick. 
 Mark the ones with high score as 
anomalous. 
T 
N 
Feature: Degree 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
8
Nodes 
T 
Features 
(egonet) 
Time 
T 
N 
Feature: 
degree 
WW 
past pattern 
right 
singular 
vector 
N 
 
 
 
eigen-behavior at t eigen-behaviors 
change-score 
metric: Z = 1- uTr 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
9
 Individual nodes/edges time series with 
distributions 
 Poisson 
 Zero-inflated Poisson 
 Hurdle Process 
▪ Hurdle Component: Bernoulli & Markov Chain 
▪ Count Component: Zero-truncated Poisson 
 Model Selection: 
 AIC, log likelihood, Vuong’s test and log gain 
 Find single-sided p-value as the probability of 
observing a count as extreme as v [P(X ≥ v)] 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
10
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 11
 Streaming Pattern dIscoveRy in multIple Time-series 
(SPIRIT) [Papadimitriou et al. 2005] 
 Discovers trends – whenever trend changes it 
introduce new hidden variable & remove when not 
needed 
 Detects anomalous points in trends 
 Nodes weights change in each step 
 At a change point the node which has highest weight 
is most anomalous 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
12
Event Detection 
Characterization 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 13
RankList2 
ScoreList2 
Consensus 
RankList1 
ScoreList1 
Rank based Score based 
•Inverse Rank 
•Kemeny Young 
[J. Kemeny 1959] 
RankList3 
ScoreList3 
•Unification [Zimek et al. 2011] 
-avg & max 
•Mixture Model [Jing et al. 2006] 
-avg & max 
Final Ensemble: inverse rank 
FinalRankList 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 14
 We were given a “Cyber Challenge Network” 
from NGAS R&T Space Park 
 Simulated cyber network traffic 
 10 days activities 
 125 hosts 
 To-from information with timestamps 
 Find “suspicious” events and the entities 
associated with the corresponding events in 
Challenge Network. 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
15
Eigen-behaviors 
Probabilistic Approach 
SPIRIT 
Z-score 
1 – norm. 
(sum 
p-value) 
projection 
Time tick 
Shebuti & Leman 16 
Event Detection & Characterization in Dynamic Graphs 
Feature: 
Degree
Eigen-behaviors 
at Time tick 376 
Probabilistic Approach 
SPIRIT 
relative 
activity 
change 
projection 
weight 
nodes 
Shebuti & Leman 17 
Event Detection & Characterization in Dynamic Graphs 
normal. 
|log(p)|
Average Precision Table (Feature: Degree) 
Algorithm Sample rate (10 min) 
Base 
Algorithms 
EBED 0.8333 
PTSAD 0.5722 
SPIRIT 0.7292 
Consensus 
Rank 
Merging 
Algorithms 
Inverse Rank (1/R) 1.0000 
Kemeny Young 0.8095 
Unification (avg) 0.8056 
Unification (max) 0.7255 
Mixture model (avg) 0.1684 
Mixture model (max) 0.1684 
Final Ensemble (1/R) 0.8667 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 18
Average Precision Table for Node anomalies 
Feature: Degree [Sample rate 10 min] 
Algorithm Event at 376 Event at 1126 
Base 
Algorithms 
EBED 1.0000 1.0000 
PTSAD 1.0000 0.2500 
SPIRIT 0.3026 0.0213 
Consensus 
Rank 
Merging 
Algorithms 
Inverse Rank (1/R) 1.0000 0.5000 
Kemeny Young 1.0000 0.2000 
Unification (avg) 1.0000 1.0000 
Unification (max) 0.8333 1.0000 
Mixture model (avg) 1.0000 1.0000 
Mixture model (max) 1.0000 1.0000 
Final Ensemble (1/R) 1.0000 1.0000 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 19
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 20
 ~8 years (Jan 2000- July 2007) of published 
articles of New York Times 
 Graph links: Co-mention of named entities 
(people, places, organization) 
 Sample rate: 1 week 
 No ground truth 
 Big Events detected: 
 January, 2001 – George W. Bush elected US president 
 September 11, 2001 – Terrorist attack in WTC 
 February 1, 2003 – Space Shuttle Columbia Disaster 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
21
Feature: 
Weighted 
Degree 
Eigen-behaviors 
Columbia disaster 
Probabilistic Approach 
SPIRIT 
2001 election 
Z Score 
1 – norm. 
(sum 
p-value) 
projection 
9/11 WTC attack 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 22
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 23
 Heterogeneous detectors 
 different scores 
 different effectiveness (depending on dataset) 
 Ensemble for event detection on dynamic graphs 
 Multiple consensus (merging) approaches 
 two-phase consensus finding 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
24
 Near-future: Robust consensus by automatically 
selecting effective base algorithms 
 Challenge: no ground truth 
 Near-future: real-time detection 
 Event detection under diverse data sources 
(e.g., news media, social media, the Web, …) 
 Challenges: different entity types, 
different time granularity, 
entity resolution 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
25
srayana@cs.stonybrook.edu 
http://www.cs.stonybrook.edu/~datalab/ 
Judge a man by his questions rather than his answers. 
-Voltaire 
Event Detection 
Characterization 
Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 
26

More Related Content

Similar to Event Detection and Characterization in Dynamic Graphs

Talk 2011-buet-perception-event
Talk 2011-buet-perception-eventTalk 2011-buet-perception-event
Talk 2011-buet-perception-eventMahfuzul Haque
 
How we use functional programming to find the bad guys @ Build Stuff LT and U...
How we use functional programming to find the bad guys @ Build Stuff LT and U...How we use functional programming to find the bad guys @ Build Stuff LT and U...
How we use functional programming to find the bad guys @ Build Stuff LT and U...Richard Minerich
 
Comparative study of various approaches for transaction Fraud Detection using...
Comparative study of various approaches for transaction Fraud Detection using...Comparative study of various approaches for transaction Fraud Detection using...
Comparative study of various approaches for transaction Fraud Detection using...Pratibha Singh
 
Multisensor data fusion in object tracking applications
Multisensor data fusion in object tracking applicationsMultisensor data fusion in object tracking applications
Multisensor data fusion in object tracking applicationsSayed Abulhasan Quadri
 
Visual diagnostics for more effective machine learning
Visual diagnostics for more effective machine learningVisual diagnostics for more effective machine learning
Visual diagnostics for more effective machine learningBenjamin Bengfort
 
Quantiative Risk Analysis for the Aerospace Industry
Quantiative Risk Analysis for the Aerospace IndustryQuantiative Risk Analysis for the Aerospace Industry
Quantiative Risk Analysis for the Aerospace IndustryIntaver Insititute
 
Processing Patterns for Predictive Business
Processing Patterns for Predictive BusinessProcessing Patterns for Predictive Business
Processing Patterns for Predictive BusinessTim Bass
 
Musings of kaggler
Musings of kagglerMusings of kaggler
Musings of kagglerKai Xin Thia
 
Less is More: Building Selective Anomaly Ensembles with Application to Event...
Less is More: Building Selective Anomaly Ensembles  with Application to Event...Less is More: Building Selective Anomaly Ensembles  with Application to Event...
Less is More: Building Selective Anomaly Ensembles with Application to Event...Shebuti Rayana
 
Processing Patterns for PredictiveBusiness
Processing Patterns for PredictiveBusinessProcessing Patterns for PredictiveBusiness
Processing Patterns for PredictiveBusinessTim Bass
 
Machine Learning Summary for Caltech2
Machine Learning Summary for Caltech2Machine Learning Summary for Caltech2
Machine Learning Summary for Caltech2Lukas Mandrake
 
Machine Learning ICS 273A
Machine Learning ICS 273AMachine Learning ICS 273A
Machine Learning ICS 273Abutest
 
Intro to Deep Reinforcement Learning
Intro to Deep Reinforcement LearningIntro to Deep Reinforcement Learning
Intro to Deep Reinforcement LearningKhaled Saleh
 
Wang midterm-defence
Wang midterm-defenceWang midterm-defence
Wang midterm-defenceZhipeng Wang
 
Data Philly Meetup - Big (Geo) Data
Data Philly Meetup - Big (Geo) DataData Philly Meetup - Big (Geo) Data
Data Philly Meetup - Big (Geo) DataAzavea
 
Graph Signal Processing for Machine Learning A Review and New Perspectives - ...
Graph Signal Processing for Machine Learning A Review and New Perspectives - ...Graph Signal Processing for Machine Learning A Review and New Perspectives - ...
Graph Signal Processing for Machine Learning A Review and New Perspectives - ...lauratoni4
 
[ICLR2021 (spotlight)] Benefit of deep learning with non-convex noisy gradien...
[ICLR2021 (spotlight)] Benefit of deep learning with non-convex noisy gradien...[ICLR2021 (spotlight)] Benefit of deep learning with non-convex noisy gradien...
[ICLR2021 (spotlight)] Benefit of deep learning with non-convex noisy gradien...Taiji Suzuki
 
Anomaly Detection for Real-World Systems
Anomaly Detection for Real-World SystemsAnomaly Detection for Real-World Systems
Anomaly Detection for Real-World SystemsManojit Nandi
 
Hidden Markov Models for Abnormal Event Processing in Transportation Data Str...
Hidden Markov Models for Abnormal Event Processing in Transportation Data Str...Hidden Markov Models for Abnormal Event Processing in Transportation Data Str...
Hidden Markov Models for Abnormal Event Processing in Transportation Data Str...John Lau
 
Bridging the Gap: Machine Learning for Ubiquitous Computing -- ML and Ubicomp...
Bridging the Gap: Machine Learning for Ubiquitous Computing -- ML and Ubicomp...Bridging the Gap: Machine Learning for Ubiquitous Computing -- ML and Ubicomp...
Bridging the Gap: Machine Learning for Ubiquitous Computing -- ML and Ubicomp...Thomas Ploetz
 

Similar to Event Detection and Characterization in Dynamic Graphs (20)

Talk 2011-buet-perception-event
Talk 2011-buet-perception-eventTalk 2011-buet-perception-event
Talk 2011-buet-perception-event
 
How we use functional programming to find the bad guys @ Build Stuff LT and U...
How we use functional programming to find the bad guys @ Build Stuff LT and U...How we use functional programming to find the bad guys @ Build Stuff LT and U...
How we use functional programming to find the bad guys @ Build Stuff LT and U...
 
Comparative study of various approaches for transaction Fraud Detection using...
Comparative study of various approaches for transaction Fraud Detection using...Comparative study of various approaches for transaction Fraud Detection using...
Comparative study of various approaches for transaction Fraud Detection using...
 
Multisensor data fusion in object tracking applications
Multisensor data fusion in object tracking applicationsMultisensor data fusion in object tracking applications
Multisensor data fusion in object tracking applications
 
Visual diagnostics for more effective machine learning
Visual diagnostics for more effective machine learningVisual diagnostics for more effective machine learning
Visual diagnostics for more effective machine learning
 
Quantiative Risk Analysis for the Aerospace Industry
Quantiative Risk Analysis for the Aerospace IndustryQuantiative Risk Analysis for the Aerospace Industry
Quantiative Risk Analysis for the Aerospace Industry
 
Processing Patterns for Predictive Business
Processing Patterns for Predictive BusinessProcessing Patterns for Predictive Business
Processing Patterns for Predictive Business
 
Musings of kaggler
Musings of kagglerMusings of kaggler
Musings of kaggler
 
Less is More: Building Selective Anomaly Ensembles with Application to Event...
Less is More: Building Selective Anomaly Ensembles  with Application to Event...Less is More: Building Selective Anomaly Ensembles  with Application to Event...
Less is More: Building Selective Anomaly Ensembles with Application to Event...
 
Processing Patterns for PredictiveBusiness
Processing Patterns for PredictiveBusinessProcessing Patterns for PredictiveBusiness
Processing Patterns for PredictiveBusiness
 
Machine Learning Summary for Caltech2
Machine Learning Summary for Caltech2Machine Learning Summary for Caltech2
Machine Learning Summary for Caltech2
 
Machine Learning ICS 273A
Machine Learning ICS 273AMachine Learning ICS 273A
Machine Learning ICS 273A
 
Intro to Deep Reinforcement Learning
Intro to Deep Reinforcement LearningIntro to Deep Reinforcement Learning
Intro to Deep Reinforcement Learning
 
Wang midterm-defence
Wang midterm-defenceWang midterm-defence
Wang midterm-defence
 
Data Philly Meetup - Big (Geo) Data
Data Philly Meetup - Big (Geo) DataData Philly Meetup - Big (Geo) Data
Data Philly Meetup - Big (Geo) Data
 
Graph Signal Processing for Machine Learning A Review and New Perspectives - ...
Graph Signal Processing for Machine Learning A Review and New Perspectives - ...Graph Signal Processing for Machine Learning A Review and New Perspectives - ...
Graph Signal Processing for Machine Learning A Review and New Perspectives - ...
 
[ICLR2021 (spotlight)] Benefit of deep learning with non-convex noisy gradien...
[ICLR2021 (spotlight)] Benefit of deep learning with non-convex noisy gradien...[ICLR2021 (spotlight)] Benefit of deep learning with non-convex noisy gradien...
[ICLR2021 (spotlight)] Benefit of deep learning with non-convex noisy gradien...
 
Anomaly Detection for Real-World Systems
Anomaly Detection for Real-World SystemsAnomaly Detection for Real-World Systems
Anomaly Detection for Real-World Systems
 
Hidden Markov Models for Abnormal Event Processing in Transportation Data Str...
Hidden Markov Models for Abnormal Event Processing in Transportation Data Str...Hidden Markov Models for Abnormal Event Processing in Transportation Data Str...
Hidden Markov Models for Abnormal Event Processing in Transportation Data Str...
 
Bridging the Gap: Machine Learning for Ubiquitous Computing -- ML and Ubicomp...
Bridging the Gap: Machine Learning for Ubiquitous Computing -- ML and Ubicomp...Bridging the Gap: Machine Learning for Ubiquitous Computing -- ML and Ubicomp...
Bridging the Gap: Machine Learning for Ubiquitous Computing -- ML and Ubicomp...
 

Recently uploaded

Virtual memory management in Operating System
Virtual memory management in Operating SystemVirtual memory management in Operating System
Virtual memory management in Operating SystemRashmi Bhat
 
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...Erbil Polytechnic University
 
Internet of things -Arshdeep Bahga .pptx
Internet of things -Arshdeep Bahga .pptxInternet of things -Arshdeep Bahga .pptx
Internet of things -Arshdeep Bahga .pptxVelmuruganTECE
 
Configuration of IoT devices - Systems managament
Configuration of IoT devices - Systems managamentConfiguration of IoT devices - Systems managament
Configuration of IoT devices - Systems managamentBharaniDharan195623
 
Class 1 | NFPA 72 | Overview Fire Alarm System
Class 1 | NFPA 72 | Overview Fire Alarm SystemClass 1 | NFPA 72 | Overview Fire Alarm System
Class 1 | NFPA 72 | Overview Fire Alarm Systemirfanmechengr
 
multiple access in wireless communication
multiple access in wireless communicationmultiple access in wireless communication
multiple access in wireless communicationpanditadesh123
 
11. Properties of Liquid Fuels in Energy Engineering.pdf
11. Properties of Liquid Fuels in Energy Engineering.pdf11. Properties of Liquid Fuels in Energy Engineering.pdf
11. Properties of Liquid Fuels in Energy Engineering.pdfHafizMudaserAhmad
 
US Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of ActionUS Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of ActionMebane Rash
 
Sachpazis Costas: Geotechnical Engineering: A student's Perspective Introduction
Sachpazis Costas: Geotechnical Engineering: A student's Perspective IntroductionSachpazis Costas: Geotechnical Engineering: A student's Perspective Introduction
Sachpazis Costas: Geotechnical Engineering: A student's Perspective IntroductionDr.Costas Sachpazis
 
Energy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptxEnergy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptxsiddharthjain2303
 
Work Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvWork Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvLewisJB
 
Software and Systems Engineering Standards: Verification and Validation of Sy...
Software and Systems Engineering Standards: Verification and Validation of Sy...Software and Systems Engineering Standards: Verification and Validation of Sy...
Software and Systems Engineering Standards: Verification and Validation of Sy...VICTOR MAESTRE RAMIREZ
 
complete construction, environmental and economics information of biomass com...
complete construction, environmental and economics information of biomass com...complete construction, environmental and economics information of biomass com...
complete construction, environmental and economics information of biomass com...asadnawaz62
 
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTIONTHE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTIONjhunlian
 
Crushers to screens in aggregate production
Crushers to screens in aggregate productionCrushers to screens in aggregate production
Crushers to screens in aggregate productionChinnuNinan
 
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort serviceGurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort servicejennyeacort
 
welding defects observed during the welding
welding defects observed during the weldingwelding defects observed during the welding
welding defects observed during the weldingMuhammadUzairLiaqat
 
System Simulation and Modelling with types and Event Scheduling
System Simulation and Modelling with types and Event SchedulingSystem Simulation and Modelling with types and Event Scheduling
System Simulation and Modelling with types and Event SchedulingBootNeck1
 
Past, Present and Future of Generative AI
Past, Present and Future of Generative AIPast, Present and Future of Generative AI
Past, Present and Future of Generative AIabhishek36461
 

Recently uploaded (20)

Virtual memory management in Operating System
Virtual memory management in Operating SystemVirtual memory management in Operating System
Virtual memory management in Operating System
 
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
 
Internet of things -Arshdeep Bahga .pptx
Internet of things -Arshdeep Bahga .pptxInternet of things -Arshdeep Bahga .pptx
Internet of things -Arshdeep Bahga .pptx
 
Configuration of IoT devices - Systems managament
Configuration of IoT devices - Systems managamentConfiguration of IoT devices - Systems managament
Configuration of IoT devices - Systems managament
 
Class 1 | NFPA 72 | Overview Fire Alarm System
Class 1 | NFPA 72 | Overview Fire Alarm SystemClass 1 | NFPA 72 | Overview Fire Alarm System
Class 1 | NFPA 72 | Overview Fire Alarm System
 
multiple access in wireless communication
multiple access in wireless communicationmultiple access in wireless communication
multiple access in wireless communication
 
11. Properties of Liquid Fuels in Energy Engineering.pdf
11. Properties of Liquid Fuels in Energy Engineering.pdf11. Properties of Liquid Fuels in Energy Engineering.pdf
11. Properties of Liquid Fuels in Energy Engineering.pdf
 
US Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of ActionUS Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of Action
 
Sachpazis Costas: Geotechnical Engineering: A student's Perspective Introduction
Sachpazis Costas: Geotechnical Engineering: A student's Perspective IntroductionSachpazis Costas: Geotechnical Engineering: A student's Perspective Introduction
Sachpazis Costas: Geotechnical Engineering: A student's Perspective Introduction
 
Energy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptxEnergy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptx
 
Work Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvWork Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvv
 
Software and Systems Engineering Standards: Verification and Validation of Sy...
Software and Systems Engineering Standards: Verification and Validation of Sy...Software and Systems Engineering Standards: Verification and Validation of Sy...
Software and Systems Engineering Standards: Verification and Validation of Sy...
 
young call girls in Green Park🔝 9953056974 🔝 escort Service
young call girls in Green Park🔝 9953056974 🔝 escort Serviceyoung call girls in Green Park🔝 9953056974 🔝 escort Service
young call girls in Green Park🔝 9953056974 🔝 escort Service
 
complete construction, environmental and economics information of biomass com...
complete construction, environmental and economics information of biomass com...complete construction, environmental and economics information of biomass com...
complete construction, environmental and economics information of biomass com...
 
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTIONTHE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
 
Crushers to screens in aggregate production
Crushers to screens in aggregate productionCrushers to screens in aggregate production
Crushers to screens in aggregate production
 
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort serviceGurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
 
welding defects observed during the welding
welding defects observed during the weldingwelding defects observed during the welding
welding defects observed during the welding
 
System Simulation and Modelling with types and Event Scheduling
System Simulation and Modelling with types and Event SchedulingSystem Simulation and Modelling with types and Event Scheduling
System Simulation and Modelling with types and Event Scheduling
 
Past, Present and Future of Generative AI
Past, Present and Future of Generative AIPast, Present and Future of Generative AI
Past, Present and Future of Generative AI
 

Event Detection and Characterization in Dynamic Graphs

  • 1. Stony Brook University Shebuti & Leman Shebuti Rayana Leman Akoglu
  • 2. Tax evasion Credit card fraud & Many More… Network intrusion Healthcare fraud Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 2
  • 3.  Problem: Given a sequence of graphs, Q1. Event detection: find time points at which graph changes significantly Q2. Characterization: find (top k) nodes / edges / regions that change the most Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 3
  • 4.  Main framework  Compute graph similarity/distance scores … … … time  Find unusual occurrences in time series Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 4
  • 5.  Flow of Ensemble Approach  Event Detection in Dynamic Graphs Ensemble Algorithms Eigen Behavior based Event Detection (EBED) Probabilistic Approach (PTSAD) SPIRIT Consensus Method Rank based Score based Results Dataset 1: Challenge Network flow Data Dataset 2: New York Times News Corpus Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 5
  • 6. Event Detection Consensus Rank Merging •Rank based •Inverse Rank •Kemeny Young •Score Based •Unification (avg, max) •Mixture Model (avg, max) • Final Ensemble (Inverse Rank) Characterization Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 6
  • 7.  Numerous algorithms for event detection  Hard to decide which one will work well for a specific data set  Our Goal: design an ensemble approach which might not give best result but “better” than most base algorithms  Challenges:  Different scores/scales  Different merging approaches Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 7
  • 8.  Extract “typical behavior” (eigen-behavior) of nodes/edges  eigen-behavior ≡ principal eigen-vector  Compare eigen-behavior over time  Score the time ticks depending on amount of change in behavior from previous time tick.  Mark the ones with high score as anomalous. T N Feature: Degree Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 8
  • 9. Nodes T Features (egonet) Time T N Feature: degree WW past pattern right singular vector N    eigen-behavior at t eigen-behaviors change-score metric: Z = 1- uTr Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 9
  • 10.  Individual nodes/edges time series with distributions  Poisson  Zero-inflated Poisson  Hurdle Process ▪ Hurdle Component: Bernoulli & Markov Chain ▪ Count Component: Zero-truncated Poisson  Model Selection:  AIC, log likelihood, Vuong’s test and log gain  Find single-sided p-value as the probability of observing a count as extreme as v [P(X ≥ v)] Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 10
  • 11. Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 11
  • 12.  Streaming Pattern dIscoveRy in multIple Time-series (SPIRIT) [Papadimitriou et al. 2005]  Discovers trends – whenever trend changes it introduce new hidden variable & remove when not needed  Detects anomalous points in trends  Nodes weights change in each step  At a change point the node which has highest weight is most anomalous Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 12
  • 13. Event Detection Characterization Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 13
  • 14. RankList2 ScoreList2 Consensus RankList1 ScoreList1 Rank based Score based •Inverse Rank •Kemeny Young [J. Kemeny 1959] RankList3 ScoreList3 •Unification [Zimek et al. 2011] -avg & max •Mixture Model [Jing et al. 2006] -avg & max Final Ensemble: inverse rank FinalRankList Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 14
  • 15.  We were given a “Cyber Challenge Network” from NGAS R&T Space Park  Simulated cyber network traffic  10 days activities  125 hosts  To-from information with timestamps  Find “suspicious” events and the entities associated with the corresponding events in Challenge Network. Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 15
  • 16. Eigen-behaviors Probabilistic Approach SPIRIT Z-score 1 – norm. (sum p-value) projection Time tick Shebuti & Leman 16 Event Detection & Characterization in Dynamic Graphs Feature: Degree
  • 17. Eigen-behaviors at Time tick 376 Probabilistic Approach SPIRIT relative activity change projection weight nodes Shebuti & Leman 17 Event Detection & Characterization in Dynamic Graphs normal. |log(p)|
  • 18. Average Precision Table (Feature: Degree) Algorithm Sample rate (10 min) Base Algorithms EBED 0.8333 PTSAD 0.5722 SPIRIT 0.7292 Consensus Rank Merging Algorithms Inverse Rank (1/R) 1.0000 Kemeny Young 0.8095 Unification (avg) 0.8056 Unification (max) 0.7255 Mixture model (avg) 0.1684 Mixture model (max) 0.1684 Final Ensemble (1/R) 0.8667 Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 18
  • 19. Average Precision Table for Node anomalies Feature: Degree [Sample rate 10 min] Algorithm Event at 376 Event at 1126 Base Algorithms EBED 1.0000 1.0000 PTSAD 1.0000 0.2500 SPIRIT 0.3026 0.0213 Consensus Rank Merging Algorithms Inverse Rank (1/R) 1.0000 0.5000 Kemeny Young 1.0000 0.2000 Unification (avg) 1.0000 1.0000 Unification (max) 0.8333 1.0000 Mixture model (avg) 1.0000 1.0000 Mixture model (max) 1.0000 1.0000 Final Ensemble (1/R) 1.0000 1.0000 Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 19
  • 20. Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 20
  • 21.  ~8 years (Jan 2000- July 2007) of published articles of New York Times  Graph links: Co-mention of named entities (people, places, organization)  Sample rate: 1 week  No ground truth  Big Events detected:  January, 2001 – George W. Bush elected US president  September 11, 2001 – Terrorist attack in WTC  February 1, 2003 – Space Shuttle Columbia Disaster Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 21
  • 22. Feature: Weighted Degree Eigen-behaviors Columbia disaster Probabilistic Approach SPIRIT 2001 election Z Score 1 – norm. (sum p-value) projection 9/11 WTC attack Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 22
  • 23. Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 23
  • 24.  Heterogeneous detectors  different scores  different effectiveness (depending on dataset)  Ensemble for event detection on dynamic graphs  Multiple consensus (merging) approaches  two-phase consensus finding Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 24
  • 25.  Near-future: Robust consensus by automatically selecting effective base algorithms  Challenge: no ground truth  Near-future: real-time detection  Event detection under diverse data sources (e.g., news media, social media, the Web, …)  Challenges: different entity types, different time granularity, entity resolution Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 25
  • 26. srayana@cs.stonybrook.edu http://www.cs.stonybrook.edu/~datalab/ Judge a man by his questions rather than his answers. -Voltaire Event Detection Characterization Shebuti & Leman Event Detection & Characterization in Dynamic Graphs 26

Editor's Notes

  1. My work focuses on discovering patterns and detecting anomalies in real-world data, using graph analytics techniques, and developing effective and efficient tools to do so .