SlideShare a Scribd company logo
Shebuti Rayana
DATA Lab
http://www.cs.stonybrook.edu/~datalab
Shebuti Rayana
Stony Brook University
Shebuti Rayana 2
Network intrusion
Healthcare fraud
Credit card fraudTax evasion
Event Detection & Characterization in Dynamic Graphs
& Many More…
Shebuti Rayana 3
 Problem: Given a sequence of graphs,
Q1. Event detection: find time points at which
graph changes significantly
Q2. Attribution: find (top k) nodes / edges /
regions that change the most
Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 4
 Main framework
 Compute graph similarity/distance scores
 Find unusual occurrences in time series
 *Note: attribution is a desired property
… ……
time
Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 5
 Flow of Ensemble Approach
 Event Detection in Dynamic Graphs
Ensemble Components
Eigen Behavior based Event Detection (EBED)
Probabilistic Approach (PTSAD)
SPIRIT
Consensus Method
Rank based
Score based
Results
Dataset 1: Challenge Network flow Data
Dataset 2: New York Times News Corpus
Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 6Event Detection & Characterization in Dynamic Graphs
Event Detection
Consensus Rank Merging
•Rank based
•Inverse Rank
•Kemeny Young
•Score Based
•Unification
(avg, max)
•Mixture Model (avg,
max)
• Final Ensemble
(Inverse Rank)
Characterization
Shebuti Rayana 7
 Numerous algorithms for event detection
 Hard to decide which one will work well for a
specific data set
 Our Goal: design an ensemble approach which
might not give us the best result but “better”
than most of the base algorithms
Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 8
 Extract “typical behavior” (eigen-behavior) of
nodes/edges
 eigen-behavior ≡ principal eigen-vector
 Compare eigen-behavior over time
 Score the time ticks depending on
amount of change in behavior
from previous time tick.
 Mark the ones with high score as
anomalous.
T
N
Feature: Degree
Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 9
Nodes
T
Features
(egonet)
Time
T
N
Feature:
degree
W
W
past pattern
eigen-behavior at t eigen-behaviors
N



right
singular
vector
change-score
metric: Z = 1- uTr
Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 10
 individual nodes/edges time series with distributions
 Poisson
 Zero-inflated Poisson
 Hurdle Process
▪ Hurdle Component: Bernoulli & Markov Chain
▪ Count Component: Zero-truncated Poisson
 Model Selection:
 AIC, log likelihood, Vuong’s test and log gain
 Find single-sided p-value as the probability of
observing a count as extreme as v [P(X ≥ v)]
Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 11Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 12
 We use the algorithm Streaming Pattern
Discovery in Multiple Time Series by
Papadimitriou et al. [2005]
 Discovers trends – whenever trend changes it
introduce new hidden variable & remove when not
needed
 Detects anomalous points in trends
 Nodes weights change in each step
 At a change point the node which has highest weight
is most anomalous
Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 13Event Detection & Characterization in Dynamic Graphs
 Base Algorithms reports: Rank list & Score list
 Consensus Rank Merging Approaches
Rank based Score based
•Inverse Rank
•Kemeny Young
•Unification [Zimek et al. 2011]
-avg & max
•Mixture Model [Jing et al. 2006]
-avg & max
Ensemble of Ensemble: inverse rank
Consensus
Shebuti Rayana 18
 We were given a “Cyber Challenge Network”
from NGAS R&T Space Park
 Simulated cyber network traffic
 10 days activities
 125 hosts
 To-from information with timestamps
 We have to find “suspicious” events and the
entities associated with the corresponding
events in this Challenge Network.
Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 19
Eigen-behaviors
Probabilistic Approach
SPIRIT
Z-score
1 – norm.
(sum
p-value)
projection
Event Detection & Characterization in Dynamic Graphs
Time tick
Feature:
Degree
Shebuti Rayana 20
Eigen-behaviors
Probabilistic Approach
SPIRIT
relative
activity
change
projection
weight
Event Detection & Characterization in Dynamic Graphs
at Time tick 376
nodes
normal.
|log(p)|
Shebuti Rayana 21
 The nodes with IP
 10.50.10.14
 10.51.16.1
 10.51.16.129
are showing subtle change in behavior compared to
other nodes.
Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 22Event Detection & Characterization in Dynamic Graphs
Algorithm Sample rate (10 min)
Base
Algorithms
EBED 0.8333
PTSAD 0.5722
SPIRIT 0.7292
Consensus
Rank
Merging
Algorithms
1/R 1.0000
Kemeny Young 0.8095
Unification (avg) 0.8056
Unification (max) 0.7255
Mixture model (avg) 0.1684
Mixture model (max) 0.1684
Ensemble of Ensemble (1/R) 0.8667
Average Precision Table (Feature: Degree)
Shebuti Rayana 23Event Detection & Characterization in Dynamic Graphs
Algorithm Event at 376 Event at 1126
Base
Algorithms
EBED 1.0000 1.0000
PTSAD 1.0000 0.2500
SPIRIT 0.3026 0.0213
Consensus
Rank
Merging
Algorithms
1/R 1.0000 0.5000
Kemeny Young 1.0000 0.2000
Unification (avg) 1.0000 1.0000
Unification (max) 0.8333 1.0000
Mixture model (avg) 1.0000 1.0000
Mixture model (max) 1.0000 1.0000
Ensemble of Ensemble (1/R) 1.0000 1.0000
Average Precision Table for Node anomalies
Feature: Degree [Sample rate 10 min]
Shebuti Rayana 24Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 25
 Around 8 years (Jan 2000- July 2007) of
published articles of New York Times
 Graph links: Co-mention of named entities (e.g.
people, places, organization etc.)
 Sample rate: 1 week
 No ground truth
 Big Events detected:
 January, 2001 – George W. Bush elected president of
United States
 September 11, 2001 – Terrorist attack in Word Trade
Center
 February 1, 2003 – Space Shuttle Columbia Disaster
Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 26Event Detection & Characterization in Dynamic Graphs
Feature:
Weighted
Degree
Eigen-behaviors
Probabilistic Approach
SPIRIT
1 – norm.
(sum
p-value)
projection
2001 election
Columbia disaster
9/11 WTC attack
ZScore
Shebuti Rayana 27Event Detection & Characterization in Dynamic Graphs
Shebuti Rayana 29
Judge a man by his questions rather than his answers.
-Voltaire
Event Detection & Characterization in Dynamic Graphs
Event Detection
Attribution
srayana@cs.stonybrook.edu
http://www.cs.stonybrook.edu/~datalab/

More Related Content

Similar to Event Detection and Characterization in Dynamic Graphs

Bayesian risk assessment of autonomous vehicles
Bayesian risk assessment of autonomous vehiclesBayesian risk assessment of autonomous vehicles
Bayesian risk assessment of autonomous vehicles
Institute for Transport Studies (ITS)
 
Project
ProjectProject
Project
Sujith C.P
 
Process Mining Reloaded: Event Structures as a Unified Representation of Proc...
Process Mining Reloaded: Event Structures as a Unified Representation of Proc...Process Mining Reloaded: Event Structures as a Unified Representation of Proc...
Process Mining Reloaded: Event Structures as a Unified Representation of Proc...
Marlon Dumas
 
Anomaly Detection in Sequences of Short Text Using Iterative Language Models
Anomaly Detection in Sequences of Short Text Using Iterative Language ModelsAnomaly Detection in Sequences of Short Text Using Iterative Language Models
Anomaly Detection in Sequences of Short Text Using Iterative Language Models
Cynthia Freeman
 
icpr_2012
icpr_2012icpr_2012
icpr_2012
Andrew Gilbert
 
How we use functional programming to find the bad guys @ Build Stuff LT and U...
How we use functional programming to find the bad guys @ Build Stuff LT and U...How we use functional programming to find the bad guys @ Build Stuff LT and U...
How we use functional programming to find the bad guys @ Build Stuff LT and U...
Richard Minerich
 
Processing Patterns for Predictive Business
Processing Patterns for Predictive BusinessProcessing Patterns for Predictive Business
Processing Patterns for Predictive Business
Tim Bass
 
ETALIS_RuleML_2011_Retractions
ETALIS_RuleML_2011_RetractionsETALIS_RuleML_2011_Retractions
ETALIS_RuleML_2011_Retractions
Darko Anicic
 
Advances in Bayesian Learning
Advances in Bayesian LearningAdvances in Bayesian Learning
Advances in Bayesian Learning
butest
 
Object Tracking with Instance Matching and Online Learning
Object Tracking with Instance Matching and Online LearningObject Tracking with Instance Matching and Online Learning
Object Tracking with Instance Matching and Online Learning
Jui-Hsin (Larry) Lai
 
CBM Fault Detection by Carl Byington
CBM Fault Detection by Carl ByingtonCBM Fault Detection by Carl Byington
CBM Fault Detection by Carl Byington
Carl Byington
 
Presentation iswc
Presentation iswcPresentation iswc
Presentation iswc
SydGillani
 
Kakfa summit london 2019 - the art of the event-streaming app
Kakfa summit london 2019 - the art of the event-streaming appKakfa summit london 2019 - the art of the event-streaming app
Kakfa summit london 2019 - the art of the event-streaming app
Neil Avery
 
The Art of The Event Streaming Application: Streams, Stream Processors and Sc...
The Art of The Event Streaming Application: Streams, Stream Processors and Sc...The Art of The Event Streaming Application: Streams, Stream Processors and Sc...
The Art of The Event Streaming Application: Streams, Stream Processors and Sc...
confluent
 
Processing Patterns for PredictiveBusiness
Processing Patterns for PredictiveBusinessProcessing Patterns for PredictiveBusiness
Processing Patterns for PredictiveBusiness
Tim Bass
 
Improving Accuracy and Explainability in Event-Case Correlation via Rule Mining
Improving Accuracy and Explainability in Event-Case Correlation via Rule MiningImproving Accuracy and Explainability in Event-Case Correlation via Rule Mining
Improving Accuracy and Explainability in Event-Case Correlation via Rule Mining
Dina Bayomie
 
CEP: Event-Decision Architecture for PredictiveBusiness, July 2006
CEP: Event-Decision Architecture for PredictiveBusiness, July 2006CEP: Event-Decision Architecture for PredictiveBusiness, July 2006
CEP: Event-Decision Architecture for PredictiveBusiness, July 2006
Tim Bass
 
Stream Processing Overview
Stream Processing OverviewStream Processing Overview
Stream Processing Overview
Maycon Viana Bordin
 
Web Service Antipatterns Detection Using Genetic Programming
Web Service Antipatterns Detection Using Genetic ProgrammingWeb Service Antipatterns Detection Using Genetic Programming
Web Service Antipatterns Detection Using Genetic Programming
Ali Ouni
 
Recsys 2016: Modeling Contextual Information in Session-Aware Recommender Sys...
Recsys 2016: Modeling Contextual Information in Session-Aware Recommender Sys...Recsys 2016: Modeling Contextual Information in Session-Aware Recommender Sys...
Recsys 2016: Modeling Contextual Information in Session-Aware Recommender Sys...
Bartlomiej Twardowski
 

Similar to Event Detection and Characterization in Dynamic Graphs (20)

Bayesian risk assessment of autonomous vehicles
Bayesian risk assessment of autonomous vehiclesBayesian risk assessment of autonomous vehicles
Bayesian risk assessment of autonomous vehicles
 
Project
ProjectProject
Project
 
Process Mining Reloaded: Event Structures as a Unified Representation of Proc...
Process Mining Reloaded: Event Structures as a Unified Representation of Proc...Process Mining Reloaded: Event Structures as a Unified Representation of Proc...
Process Mining Reloaded: Event Structures as a Unified Representation of Proc...
 
Anomaly Detection in Sequences of Short Text Using Iterative Language Models
Anomaly Detection in Sequences of Short Text Using Iterative Language ModelsAnomaly Detection in Sequences of Short Text Using Iterative Language Models
Anomaly Detection in Sequences of Short Text Using Iterative Language Models
 
icpr_2012
icpr_2012icpr_2012
icpr_2012
 
How we use functional programming to find the bad guys @ Build Stuff LT and U...
How we use functional programming to find the bad guys @ Build Stuff LT and U...How we use functional programming to find the bad guys @ Build Stuff LT and U...
How we use functional programming to find the bad guys @ Build Stuff LT and U...
 
Processing Patterns for Predictive Business
Processing Patterns for Predictive BusinessProcessing Patterns for Predictive Business
Processing Patterns for Predictive Business
 
ETALIS_RuleML_2011_Retractions
ETALIS_RuleML_2011_RetractionsETALIS_RuleML_2011_Retractions
ETALIS_RuleML_2011_Retractions
 
Advances in Bayesian Learning
Advances in Bayesian LearningAdvances in Bayesian Learning
Advances in Bayesian Learning
 
Object Tracking with Instance Matching and Online Learning
Object Tracking with Instance Matching and Online LearningObject Tracking with Instance Matching and Online Learning
Object Tracking with Instance Matching and Online Learning
 
CBM Fault Detection by Carl Byington
CBM Fault Detection by Carl ByingtonCBM Fault Detection by Carl Byington
CBM Fault Detection by Carl Byington
 
Presentation iswc
Presentation iswcPresentation iswc
Presentation iswc
 
Kakfa summit london 2019 - the art of the event-streaming app
Kakfa summit london 2019 - the art of the event-streaming appKakfa summit london 2019 - the art of the event-streaming app
Kakfa summit london 2019 - the art of the event-streaming app
 
The Art of The Event Streaming Application: Streams, Stream Processors and Sc...
The Art of The Event Streaming Application: Streams, Stream Processors and Sc...The Art of The Event Streaming Application: Streams, Stream Processors and Sc...
The Art of The Event Streaming Application: Streams, Stream Processors and Sc...
 
Processing Patterns for PredictiveBusiness
Processing Patterns for PredictiveBusinessProcessing Patterns for PredictiveBusiness
Processing Patterns for PredictiveBusiness
 
Improving Accuracy and Explainability in Event-Case Correlation via Rule Mining
Improving Accuracy and Explainability in Event-Case Correlation via Rule MiningImproving Accuracy and Explainability in Event-Case Correlation via Rule Mining
Improving Accuracy and Explainability in Event-Case Correlation via Rule Mining
 
CEP: Event-Decision Architecture for PredictiveBusiness, July 2006
CEP: Event-Decision Architecture for PredictiveBusiness, July 2006CEP: Event-Decision Architecture for PredictiveBusiness, July 2006
CEP: Event-Decision Architecture for PredictiveBusiness, July 2006
 
Stream Processing Overview
Stream Processing OverviewStream Processing Overview
Stream Processing Overview
 
Web Service Antipatterns Detection Using Genetic Programming
Web Service Antipatterns Detection Using Genetic ProgrammingWeb Service Antipatterns Detection Using Genetic Programming
Web Service Antipatterns Detection Using Genetic Programming
 
Recsys 2016: Modeling Contextual Information in Session-Aware Recommender Sys...
Recsys 2016: Modeling Contextual Information in Session-Aware Recommender Sys...Recsys 2016: Modeling Contextual Information in Session-Aware Recommender Sys...
Recsys 2016: Modeling Contextual Information in Session-Aware Recommender Sys...
 

Recently uploaded

VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
satpalsheravatmumbai
 
High Profile Girls Call Mumbai 9910780858 Provide Best And Top Girl Service A...
High Profile Girls Call Mumbai 9910780858 Provide Best And Top Girl Service A...High Profile Girls Call Mumbai 9910780858 Provide Best And Top Girl Service A...
High Profile Girls Call Mumbai 9910780858 Provide Best And Top Girl Service A...
shardda patel
 
Aws MLOps Interview Questions with answers
Aws MLOps Interview Questions  with answersAws MLOps Interview Questions  with answers
Aws MLOps Interview Questions with answers
Sathiakumar Chandr
 
CMO MRM_May 2024 WITH BREAKDOWN AND IMPROVEMENTDATA.pdf
CMO MRM_May 2024 WITH BREAKDOWN AND IMPROVEMENTDATA.pdfCMO MRM_May 2024 WITH BREAKDOWN AND IMPROVEMENTDATA.pdf
CMO MRM_May 2024 WITH BREAKDOWN AND IMPROVEMENTDATA.pdf
IndranilDasgupta19
 
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeliveryBDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
erynsouthern
 
Dataguard Switchover Best Practices using DGMGRL (Dataguard Broker Command Line)
Dataguard Switchover Best Practices using DGMGRL (Dataguard Broker Command Line)Dataguard Switchover Best Practices using DGMGRL (Dataguard Broker Command Line)
Dataguard Switchover Best Practices using DGMGRL (Dataguard Broker Command Line)
Alireza Kamrani
 
The Rise of Python in Finance,Automating Trading Strategies: _.pdf
The Rise of Python in Finance,Automating Trading Strategies: _.pdfThe Rise of Python in Finance,Automating Trading Strategies: _.pdf
The Rise of Python in Finance,Automating Trading Strategies: _.pdf
Riya Sen
 
Celonis Busniess Analyst Virtual Internship.pptx
Celonis Busniess Analyst Virtual Internship.pptxCelonis Busniess Analyst Virtual Internship.pptx
Celonis Busniess Analyst Virtual Internship.pptx
AnujaGaikwad28
 
VVIP Girls Call Noida 9873940964 Provide Best And Top Girl Service And No1 in...
VVIP Girls Call Noida 9873940964 Provide Best And Top Girl Service And No1 in...VVIP Girls Call Noida 9873940964 Provide Best And Top Girl Service And No1 in...
VVIP Girls Call Noida 9873940964 Provide Best And Top Girl Service And No1 in...
Ak47
 
Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...
Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...
Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...
revolutionary575
 
Cyber Insurance Mathematical Model & Pricing 2
Cyber Insurance Mathematical Model & Pricing 2Cyber Insurance Mathematical Model & Pricing 2
Cyber Insurance Mathematical Model & Pricing 2
BaraDaniel1
 
Communication-Skills-An-Essential-Toolkit.pptx
Communication-Skills-An-Essential-Toolkit.pptxCommunication-Skills-An-Essential-Toolkit.pptx
Communication-Skills-An-Essential-Toolkit.pptx
sanketdhavale23di
 
Towards an Analysis-Ready, Cloud-Optimised service for FAIR fusion data
Towards an Analysis-Ready, Cloud-Optimised service for FAIR fusion dataTowards an Analysis-Ready, Cloud-Optimised service for FAIR fusion data
Towards an Analysis-Ready, Cloud-Optimised service for FAIR fusion data
Samuel Jackson
 
Celebrity Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
Celebrity Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...Celebrity Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
Celebrity Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
arti singh$A17
 
Semantic Web and organizational data .pptx
Semantic Web and organizational data .pptxSemantic Web and organizational data .pptx
Semantic Web and organizational data .pptx
Kanchana Weerasinghe
 
History and Application of LLM Leveraging Big Data
History and Application of LLM Leveraging Big DataHistory and Application of LLM Leveraging Big Data
History and Application of LLM Leveraging Big Data
Jongwook Woo
 
VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...
VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...
VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...
sukaniyasunnu
 
Water bodies of India - Shubham 8 b.pptx
Water bodies of India - Shubham 8 b.pptxWater bodies of India - Shubham 8 b.pptx
Water bodies of India - Shubham 8 b.pptx
manshinain08
 
Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
sheetal singh$A17
 
Premium Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service A...
Premium Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service A...Premium Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service A...
Premium Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service A...
sheetal singh$A17
 

Recently uploaded (20)

VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
 
High Profile Girls Call Mumbai 9910780858 Provide Best And Top Girl Service A...
High Profile Girls Call Mumbai 9910780858 Provide Best And Top Girl Service A...High Profile Girls Call Mumbai 9910780858 Provide Best And Top Girl Service A...
High Profile Girls Call Mumbai 9910780858 Provide Best And Top Girl Service A...
 
Aws MLOps Interview Questions with answers
Aws MLOps Interview Questions  with answersAws MLOps Interview Questions  with answers
Aws MLOps Interview Questions with answers
 
CMO MRM_May 2024 WITH BREAKDOWN AND IMPROVEMENTDATA.pdf
CMO MRM_May 2024 WITH BREAKDOWN AND IMPROVEMENTDATA.pdfCMO MRM_May 2024 WITH BREAKDOWN AND IMPROVEMENTDATA.pdf
CMO MRM_May 2024 WITH BREAKDOWN AND IMPROVEMENTDATA.pdf
 
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeliveryBDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
 
Dataguard Switchover Best Practices using DGMGRL (Dataguard Broker Command Line)
Dataguard Switchover Best Practices using DGMGRL (Dataguard Broker Command Line)Dataguard Switchover Best Practices using DGMGRL (Dataguard Broker Command Line)
Dataguard Switchover Best Practices using DGMGRL (Dataguard Broker Command Line)
 
The Rise of Python in Finance,Automating Trading Strategies: _.pdf
The Rise of Python in Finance,Automating Trading Strategies: _.pdfThe Rise of Python in Finance,Automating Trading Strategies: _.pdf
The Rise of Python in Finance,Automating Trading Strategies: _.pdf
 
Celonis Busniess Analyst Virtual Internship.pptx
Celonis Busniess Analyst Virtual Internship.pptxCelonis Busniess Analyst Virtual Internship.pptx
Celonis Busniess Analyst Virtual Internship.pptx
 
VVIP Girls Call Noida 9873940964 Provide Best And Top Girl Service And No1 in...
VVIP Girls Call Noida 9873940964 Provide Best And Top Girl Service And No1 in...VVIP Girls Call Noida 9873940964 Provide Best And Top Girl Service And No1 in...
VVIP Girls Call Noida 9873940964 Provide Best And Top Girl Service And No1 in...
 
Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...
Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...
Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...
 
Cyber Insurance Mathematical Model & Pricing 2
Cyber Insurance Mathematical Model & Pricing 2Cyber Insurance Mathematical Model & Pricing 2
Cyber Insurance Mathematical Model & Pricing 2
 
Communication-Skills-An-Essential-Toolkit.pptx
Communication-Skills-An-Essential-Toolkit.pptxCommunication-Skills-An-Essential-Toolkit.pptx
Communication-Skills-An-Essential-Toolkit.pptx
 
Towards an Analysis-Ready, Cloud-Optimised service for FAIR fusion data
Towards an Analysis-Ready, Cloud-Optimised service for FAIR fusion dataTowards an Analysis-Ready, Cloud-Optimised service for FAIR fusion data
Towards an Analysis-Ready, Cloud-Optimised service for FAIR fusion data
 
Celebrity Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
Celebrity Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...Celebrity Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
Celebrity Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
 
Semantic Web and organizational data .pptx
Semantic Web and organizational data .pptxSemantic Web and organizational data .pptx
Semantic Web and organizational data .pptx
 
History and Application of LLM Leveraging Big Data
History and Application of LLM Leveraging Big DataHistory and Application of LLM Leveraging Big Data
History and Application of LLM Leveraging Big Data
 
VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...
VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...
VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...
 
Water bodies of India - Shubham 8 b.pptx
Water bodies of India - Shubham 8 b.pptxWater bodies of India - Shubham 8 b.pptx
Water bodies of India - Shubham 8 b.pptx
 
Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
 
Premium Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service A...
Premium Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service A...Premium Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service A...
Premium Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service A...
 

Event Detection and Characterization in Dynamic Graphs

  • 2. Shebuti Rayana 2 Network intrusion Healthcare fraud Credit card fraudTax evasion Event Detection & Characterization in Dynamic Graphs & Many More…
  • 3. Shebuti Rayana 3  Problem: Given a sequence of graphs, Q1. Event detection: find time points at which graph changes significantly Q2. Attribution: find (top k) nodes / edges / regions that change the most Event Detection & Characterization in Dynamic Graphs
  • 4. Shebuti Rayana 4  Main framework  Compute graph similarity/distance scores  Find unusual occurrences in time series  *Note: attribution is a desired property … …… time Event Detection & Characterization in Dynamic Graphs
  • 5. Shebuti Rayana 5  Flow of Ensemble Approach  Event Detection in Dynamic Graphs Ensemble Components Eigen Behavior based Event Detection (EBED) Probabilistic Approach (PTSAD) SPIRIT Consensus Method Rank based Score based Results Dataset 1: Challenge Network flow Data Dataset 2: New York Times News Corpus Event Detection & Characterization in Dynamic Graphs
  • 6. Shebuti Rayana 6Event Detection & Characterization in Dynamic Graphs Event Detection Consensus Rank Merging •Rank based •Inverse Rank •Kemeny Young •Score Based •Unification (avg, max) •Mixture Model (avg, max) • Final Ensemble (Inverse Rank) Characterization
  • 7. Shebuti Rayana 7  Numerous algorithms for event detection  Hard to decide which one will work well for a specific data set  Our Goal: design an ensemble approach which might not give us the best result but “better” than most of the base algorithms Event Detection & Characterization in Dynamic Graphs
  • 8. Shebuti Rayana 8  Extract “typical behavior” (eigen-behavior) of nodes/edges  eigen-behavior ≡ principal eigen-vector  Compare eigen-behavior over time  Score the time ticks depending on amount of change in behavior from previous time tick.  Mark the ones with high score as anomalous. T N Feature: Degree Event Detection & Characterization in Dynamic Graphs
  • 9. Shebuti Rayana 9 Nodes T Features (egonet) Time T N Feature: degree W W past pattern eigen-behavior at t eigen-behaviors N    right singular vector change-score metric: Z = 1- uTr Event Detection & Characterization in Dynamic Graphs
  • 10. Shebuti Rayana 10  individual nodes/edges time series with distributions  Poisson  Zero-inflated Poisson  Hurdle Process ▪ Hurdle Component: Bernoulli & Markov Chain ▪ Count Component: Zero-truncated Poisson  Model Selection:  AIC, log likelihood, Vuong’s test and log gain  Find single-sided p-value as the probability of observing a count as extreme as v [P(X ≥ v)] Event Detection & Characterization in Dynamic Graphs
  • 11. Shebuti Rayana 11Event Detection & Characterization in Dynamic Graphs
  • 12. Shebuti Rayana 12  We use the algorithm Streaming Pattern Discovery in Multiple Time Series by Papadimitriou et al. [2005]  Discovers trends – whenever trend changes it introduce new hidden variable & remove when not needed  Detects anomalous points in trends  Nodes weights change in each step  At a change point the node which has highest weight is most anomalous Event Detection & Characterization in Dynamic Graphs
  • 13. Shebuti Rayana 13Event Detection & Characterization in Dynamic Graphs  Base Algorithms reports: Rank list & Score list  Consensus Rank Merging Approaches Rank based Score based •Inverse Rank •Kemeny Young •Unification [Zimek et al. 2011] -avg & max •Mixture Model [Jing et al. 2006] -avg & max Ensemble of Ensemble: inverse rank Consensus
  • 14. Shebuti Rayana 18  We were given a “Cyber Challenge Network” from NGAS R&T Space Park  Simulated cyber network traffic  10 days activities  125 hosts  To-from information with timestamps  We have to find “suspicious” events and the entities associated with the corresponding events in this Challenge Network. Event Detection & Characterization in Dynamic Graphs
  • 15. Shebuti Rayana 19 Eigen-behaviors Probabilistic Approach SPIRIT Z-score 1 – norm. (sum p-value) projection Event Detection & Characterization in Dynamic Graphs Time tick Feature: Degree
  • 16. Shebuti Rayana 20 Eigen-behaviors Probabilistic Approach SPIRIT relative activity change projection weight Event Detection & Characterization in Dynamic Graphs at Time tick 376 nodes normal. |log(p)|
  • 17. Shebuti Rayana 21  The nodes with IP  10.50.10.14  10.51.16.1  10.51.16.129 are showing subtle change in behavior compared to other nodes. Event Detection & Characterization in Dynamic Graphs
  • 18. Shebuti Rayana 22Event Detection & Characterization in Dynamic Graphs Algorithm Sample rate (10 min) Base Algorithms EBED 0.8333 PTSAD 0.5722 SPIRIT 0.7292 Consensus Rank Merging Algorithms 1/R 1.0000 Kemeny Young 0.8095 Unification (avg) 0.8056 Unification (max) 0.7255 Mixture model (avg) 0.1684 Mixture model (max) 0.1684 Ensemble of Ensemble (1/R) 0.8667 Average Precision Table (Feature: Degree)
  • 19. Shebuti Rayana 23Event Detection & Characterization in Dynamic Graphs Algorithm Event at 376 Event at 1126 Base Algorithms EBED 1.0000 1.0000 PTSAD 1.0000 0.2500 SPIRIT 0.3026 0.0213 Consensus Rank Merging Algorithms 1/R 1.0000 0.5000 Kemeny Young 1.0000 0.2000 Unification (avg) 1.0000 1.0000 Unification (max) 0.8333 1.0000 Mixture model (avg) 1.0000 1.0000 Mixture model (max) 1.0000 1.0000 Ensemble of Ensemble (1/R) 1.0000 1.0000 Average Precision Table for Node anomalies Feature: Degree [Sample rate 10 min]
  • 20. Shebuti Rayana 24Event Detection & Characterization in Dynamic Graphs
  • 21. Shebuti Rayana 25  Around 8 years (Jan 2000- July 2007) of published articles of New York Times  Graph links: Co-mention of named entities (e.g. people, places, organization etc.)  Sample rate: 1 week  No ground truth  Big Events detected:  January, 2001 – George W. Bush elected president of United States  September 11, 2001 – Terrorist attack in Word Trade Center  February 1, 2003 – Space Shuttle Columbia Disaster Event Detection & Characterization in Dynamic Graphs
  • 22. Shebuti Rayana 26Event Detection & Characterization in Dynamic Graphs Feature: Weighted Degree Eigen-behaviors Probabilistic Approach SPIRIT 1 – norm. (sum p-value) projection 2001 election Columbia disaster 9/11 WTC attack ZScore
  • 23. Shebuti Rayana 27Event Detection & Characterization in Dynamic Graphs
  • 24. Shebuti Rayana 29 Judge a man by his questions rather than his answers. -Voltaire Event Detection & Characterization in Dynamic Graphs Event Detection Attribution srayana@cs.stonybrook.edu http://www.cs.stonybrook.edu/~datalab/

Editor's Notes

  1. My work focuses on discovering patterns and detecting anomalies in real-world data, using graph analytics techniques, and developing effective and efficient tools to do so .