Using Graphs for Data Analysis
•
2 likes•2,762 views
This document discusses using graphs for data analysis and provides examples of different types of graph algorithms and queries that can be performed on graph data. Some key points: - Graphs can be used to represent relational datasets and enable different types of analysis than is possible on traditional relational models. - Common graph algorithms discussed include centrality measures, community detection, pattern matching queries, and shortest path algorithms. - Examples applications highlighted are fraud detection using financial transaction graphs and topic modeling on text data represented as a graph.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Using Graphs for Data Analysis
Similar to Using Graphs for Data Analysis (20)
More from オラクルエンジニア通信
More from オラクルエンジニア通信 (20)
Recently uploaded
Recently uploaded (20)
Using Graphs for Data Analysis
- 3. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Data Analysis, Big Data, Cloud Big Data Data Analytics Cloud • Big data makes data analytics effective • Data analytics make big data valuable • Cloud enables Big Data with large data capacity • Big Data creates market for Cloud • Cloud enables Data Analytics with sufficient computation power • Data Analytics creates market for Cloud
- 4. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • Relational model is widely used for general data management • However, the model is cumbersome for answering certain questions. Your Data As a Graph Account ID Owner ID Creation Date 1111 200 2010-3-10 2222 100 2011-2-13 3333 400 2015-9-16 4444 300 2012-5-25 5555 100 … SRC DEST Type Amount 1111 3333 Wire $20,000 5555 4444 Wire $30,000 4444 2222 Recurring $10,000 3333 5555 Wire $20,000 …. Account Table Transfer Table Owner ID Name 100 Alice 200 Bob 300 Charlie 400 Dave … Customer Table How can Bob and Charlie be related? Is there any money flow between them? Is there any money flow that cycles back to the originating owner?
- 5. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • Represent the dataset as a graph – Entities become vertices – Relationships become edges • Answering becomes easier with graph representation Your Data As a Graph 100 Alice 200 Bob 300 Charlie 400 Dave 1111 2222 3333 4444 5555 $20,000 $20,000 $30,000 $10,000 How can Bob and Charlie be related? Is there any money flow between them? Is there any money flow that cycles back to the originating owner?
- 6. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • Graph representation enables even more powerful data analysis. – Directly examine fine-grained relationships between entities – Indirect (multi-hop) relations are also considered Your Data As a Graph Cluster the accounts into groups from their wire-transfer records Identify important accounts whose suspension would disrupt money flow significantly
- 7. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Need to support both kinds, as well as combinations of the two Graph Workloads Pagerank Modularity Clustering Coefficient Shortest Path Connected Components Conductance Centrality Coloring Spanning Tree Computational Graph Analytics Compute certain values on nodes and edges While (repeatedly) traversing or iterating on the graph In certain procedural ways Graph Query (Pattern Matching) Given a description of a pattern spouse friend friend Find every sub-graph that matches it 7 Images from IMDB.com
- 8. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • Concept in a line – “Find me all groups of entities that are connected like this” • Example Use case – Fraud detection in financial transactions • Create a graph from customers, financial accounts and their transactions • Want to find transaction patterns that look suspicious, for instance, like this … Pattern Matching Graph Query (total amount 1) > τ (total amount 2) > τ Account #1 Account #2 Customer #1 External Entity :has :wires :wires |(total amount 1) - (total amount 2)| < δ
- 9. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • Concept in a line – “Find me all entities that are connected to this vertex .. ” – “… the connection can be indefinitely away but only though certain edges“ • Example Use case – Continued Fraud detection in financial transactions – But now … Pattern Matching Graph Query -- Reachability Customer #1 :has (total amount 1) > τ (total amount 2) > τ Account #1 Account #2 External Entity :wires :wires |(total amount 1) - (total amount 2)| < δ Customer #2 :has #A #B…… Address #341 Email @my.com:addr :addr :email:email
- 10. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • Concept in a line – “Find me the most important vertex (entity) in this data set” • Centrality is a fancy name from graph theory è mathematically, a measure of relative importance of vertices in a graph • Question is ... on what account a vertex is considered as important? • There are indeed many different centralities defined in the graph theory – Betweenness Centrality – Closeness Centrality – Eigenvector Centrality – Pagerank – HITS – … Graph Algorithm -- Centrality Eigenvector centrality (images from Wikipedia)
- 11. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • Rough Idea – Vertices are considered important, if it is located “in between” other vertices • Example Application – Power Grid Network • Given a network of switches, relays, wires, generators, feeders • What are the nodes that will have most negative impact if goes down Betweenness Centrality Note: This algorithm is very expensive in computation. PGX provides a fast implementation by automatically applying complicated implementation technique through DSL compiler optimization Device Connector ON ON ON ON ON ONON ON ON ON ON ON ON ON ONOFF OFF source deviceElectric Network ON ON source device2 ON
- 12. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • Concept in a line “What are other entities that are close to this one?” • Shortest path – Using edge weights as ‘distance’ metric – User can define different metrics … – Traditional algorithms to find the paths and distances: Dijkstra, Bellman-Ford, etc … • Some considerations – What if there are multiple paths? – What about high degree vertices in between? Graph Algorithm -- Closeness A B C DVs. Vs.A B C D
- 13. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • Personalized pagerank – A variant of Pagerank algorithm – When repeating random walk (with restart) from the given starting vertices – Compute probability of visiting each vertex in the graph • Example Application – Finding hidden members of a cartel from flight records – Given a list of known Cartel members – Can we find people who shared same flight with some of these members often? Personalized Pagerank Starting vertices • Vertices that are ‘close’ would be visited more often naturally • Shared edges also would make the vertex visited more often …1 2 3 4 5 31 1 12 10 21 23 ……
- 14. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Similar examples – Product Recommendation • Example – I bought this printer. Can the system automatically recommend the ink cartridge for it? • Approach – Create a graph of customer-product (from a real data set) – Check closeness between products + Mandatory Co-Item For the imputed data entity: Item-5 Optional Co-Items Other items For a real data entity in the original data set lrg white cttn span cami:White:Large lrg black cttn span cami:Black:Large med white cttn span cami:White:Medium
- 15. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • Concept in a line “Can we cluster the data entities into groups from their closeness?” • (Weakly) Connected component – A classic algorithm to identify disjoint groups of entities – This approach works surprisingly well often! • Especially when combined with edge- masking • But what about non-disconnected graphs? Components and Communities
- 16. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • General community detection – Many new algorithms proposed by graph theory – Label propagation, Info Map, Relax Map, … • Example use case: – Students from CS/Math department of a university – Can we classify students by the classes that they take? Community Detection Algorithms Classification Result – For students of ‘Math/CS department’ - Courses taken by community A - Courses taken by community B
- 17. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • Representing the data-set as a graph – Vertices è Posts and tags – Edges è Tag specified in the posts • These relationships (edges) gives very strong signal for clustering – Groups of postings are naturally formed by commonly shared tags – Noises from generic terms (e.g. Linux) are taken care of by graph partitioning algorithm Example – Topic Modeling networ king linux ip- tables kernel debian sed awk apt Text- Process packag ing
- 18. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Comparison to Classic Machine Learning • Topic Modeling – Dataset: on-line postings in unix.stackexchange.com (CY 2016) – Identify groups of topics in these postings (trend analysis) Topic (tags) Bash, shell-script, scripting, mmv Text-processing, awk, sed, grep, perl Centos, rhel, yum, rpm, repository, rpmbuild, redhat-satellite, drupal Networking, ip, routing, dhcp, tcp, router, iproute, isc-dhcp, pcap Ssh, openssh, sshd, ssh-tunneling, key-authentication, ssh-config – Traditional ML approach (LDA) Topic (tags) Bash, shell-script, shell, scripting Linux, ssh, grep, linux-kernel, files, kernel, regular-expression Networking, network-interface, dns, ip, raspberry-pi, raspbian, routing Centos, python, yum, rpm, mysql, php, postgresql, software-installation, repository Permissions, sudo, users, root, sort, aix, chmod, group, executable, acl – Results from graph analysis • More intuitive results • Less susceptible to hyper-parameter selection
- 19. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • Graph analysis can augment Machine Learning – Typical machine learning techniques train models based on directly observed features – Graph analysis can provide additional strong signals by analyzing relationships – .. Which make ML model more accurate Graph Analysis and Machine Learning Feature1 Feature 2 Feature 3 D1 D2 D3 Predictive Model Raw Data Graph View Feature 4 Feature 5 Feature 6 Feature 7
- 20. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Summary • Graph analysis – a methodology in data analytics – Inspects fined-grained relationships between data entities – Can be combined with other approaches, e.g. ML • Graph pattern matching query • Computational graph analysis