Jason Riedy, profile picture
Uploaded byJason Riedy
1,358 views

STING: Spatio-Temporal Interaction Networks and Graphs for Intel Platforms

This document summarizes work on STING (Spatio-Temporal Interaction Networks and Graphs), a tool for analyzing massive graphs on Intel platforms. STING can maintain metrics and a graph under large data changes and is optimized for performance on Intel architectures. The document outlines STING's architecture and capabilities, including its extensible data structure and ability to support dynamic graph algorithms like incremental community detection and PageRank. It also reviews accomplishments like maintaining over 100 million graph updates per second.

Embed presentation

Downloaded 18 times
STING: Spatio-Temporal Interaction Networks and Graphs for Intel Platforms David Bader, Jason Riedy, David Ediger, Rob McColl, Timothy G. Mattson∗, ... Georgia Institute of Technology 17 January 2014
(insert current prefix here)scale data analysis Health care Finding outbreaks, population epidemiology Social networks Advertising, searching, grouping Intelligence Decisions at scale, regulating algorithms Systems biology Understanding interactions, drug design Power grid Disruptions, conservation Simulation Discrete events, cracking meshes Data analysis runs throughout. Many problems can be phrased through graphs. Any many are changing and dynamic. Bader, Riedy— STING 17 Jan. 2014 2 / 47
GT’s part of Intel’s Non-Numeric Computing Program Goal Supporting analysis of massive graphs under rapid change across the spectrum of Intel-based platforms. STING on Intel-based platforms Maintain a graph and metrics under large data changes Performance, documentation, distribution Available at http://www.cc.gatech.edu/stinger/ Bader, Riedy— STING 17 Jan. 2014 3 / 47
Outline Motivation: Graph Algorithms for Analysis STING Review of Accomplishments Selected Details on Recent Work Dynamic Community Updating Incremental PageRank Related Projects (Future Plans) Artifacts of the Intel Non-Numeric Computing Program Bader, Riedy— STING 17 Jan. 2014 4 / 47
Why Graphs? Smaller than raw data. Taught (roughly) to all CS students... Semantic attributions can capture essential relationships. Traversals can be faster than filtering DB joins. Provide clear phrasing for queries about relationships. Often next step after dense and sparse linear algebra. Note: Extracting data into a graph is a separate topic, see Intel’s GraphBuilder. Bader, Riedy— STING 17 Jan. 2014 5 / 47
Graphs: A Fundamental Abstraction Structure for “unstructured” data Traditional uses: Route planning on fixed routes Logistic planning between sources, routes, destinations Increasing uses: Computer security: Identify anomalies (e.g. spam, viruses, hacks) as they occur, insider threats, control access, localize malware Data / intelligence integration: Find smaller, relevant subsets of massive, “unstructured” data piles Recommender systems (industry): Given activities, automatically find other interesting data. Bader, Riedy— STING 17 Jan. 2014 6 / 47
On to STING... Motivation: Graph Algorithms for Analysis STING Review of Accomplishments Selected Details on Recent Work Related Projects (Future Plans) Artifacts of the Intel Non-Numeric Computing Program Bader, Riedy— STING 17 Jan. 2014 7 / 47
STING’s focus Source data predictionaction summary Control VizSimulation / query STING manages queries against changing graph data. Visualization and control often are application specific. Ideal: Maintain many persistent graph analysis kernels. One current graph snapshot, kernels keep smaller histories. Also (a harder goal), coordinate the kernels’ cooperation. Bader, Riedy— STING 17 Jan. 2014 8 / 47
STING: High-level architecture Slide credit: Rob McColl and David Ediger OpenMP + sufficiently POSIX-ish Multiple processes for resilience Bader, Riedy— STING 17 Jan. 2014 9 / 47
STING Extensible Representation: Core data structure source vertex edge type adj. vtx weight mod. time first time edge type index vertex index Initial considerations [Bader, et al.] Be useful for the entire “large graph” community Portable semantics and high-level optimizations across multiple platforms & frameworks (XMT C, MTGL, etc.) Permit good performance: No single structure is optimal for all. Assume globally addressable memory access and atomic operations Support multiple, parallel readers and a single writer process Large graph ⇒ rare conflicts, may ignore synchronization Bader, Riedy— STING 17 Jan. 2014 10 / 47
STING in use Who is using it? Internally, well, us. Some details later. Outside institutions in multiple research projects Center for Adaptive Supercomputing Software – Multithreaded Architectures (CASS-MT) directed by PNNL PRODIGAL team for DARPA ADAMS (Anomaly Detection at Multiple Scales) including GTRI, SAIC, Oregon State U., U. Mass., and CMU Several industrial companies in the area of data analytics and security Well-attended tutorials given at PPoPP, in MD, and locally Bader, Riedy— STING 17 Jan. 2014 11 / 47
STING: Where do you get it? www.cc.gatech.edu/stinger/ Gateway to code, development, documentation, presentations... (working on usage and development screencasts) Remember: Still academic code, but maturing. Bader, Riedy— STING 17 Jan. 2014 12 / 47
Motivation: Graph Algorithms for Analysis STING Review of Accomplishments Selected Details on Recent Work Related Projects (Future Plans) Artifacts of the Intel Non-Numeric Computing Program Bader, Riedy— STING 17 Jan. 2014 13 / 47
Selected Program Accomplishments I In the beginning: Initial streaming algorithms Maintaining clustering coefficients[1] (counting triangles) Up to 130 000 graph updates per second on X5570 (Nehalem-EP, 2.93GHz) 2000× speed-up over static recomputation Connected components[2] Up to 88 700 graph updates per second on X5570, 233× speed-up Community detection[3, 4, 5] First parallel algorithm for agglomerative community detection, 3.3B edges clustered in 505s on a 4-socket, 10-core Westmere-EX. Win 10th DIMACS Implementation Challenge’s Mix Challenge Enabled moving computation from special server + batch to laptops and servers. Bader, Riedy— STING 17 Jan. 2014 14 / 47
Selected Program Accomplishments II Still maintained, used by external groups like Cray, people finally threatening performance. Expanded streaming algorithms Improved connected components[6] Does not fall back on re-computation for deletions Up to 137× speed-up over static re-computation Community maintenance[7] and monitoring∗ Up to 100 million updates per second, speed-ups from 4× to 3 700× Demonstrated at Research@Intel and GraphLab Workshop 2013 PageRank∗ Reduce edge traversals from 30% – 90%. Betweenness centrality∗ [8, 9] Up to 148× speed-up Optimization and software structure of STING[10] Bader, Riedy— STING 17 Jan. 2014 15 / 47
Selected Program Accomplishments III Up to 14× performance improvement on Intel-based platforms[11], best paper award at HPEC12 Research@Intel 2013 demo collaborative with Intel GraphBuilder: extract and transform data from a Hadoop store, transfer to STING for analysis Something different: 10th DIMACS Implementation Challenge on Graph Partitioning and Graph Clustering [12] Very useful workshop, 13-14 February 2012 at Georgia Tech Intel sponsorship: Thank you! Graph archive being used in other projects (although not always cited appropriately) PASQUAL [13] First de novo genome assembler that is both scalable and reliable (doesn’t crash). http://www.cc.gatech.edu/pasqual/ (not STING-based) Bader, Riedy— STING 17 Jan. 2014 16 / 47
Motivation: Graph Algorithms for Analysis STING Review of Accomplishments Selected Details on Recent Work Dynamic Community Updating Incremental PageRank Related Projects (Future Plans) Artifacts of the Intel Non-Numeric Computing Program Bader, Riedy— STING 17 Jan. 2014 17 / 47
Community Detection What do we mean? Partition a graph’s vertices into disjoint communities. Locally optimize some metric, e.g. modularity, conductance, ... Try to capture that vertices are more similar within one community than between communities. Jason’s network via LinkedIn Labs Bader, Riedy— STING 17 Jan. 2014 18 / 47
Parallel agglomerative method A B C D E F G Use a matching to avoid a serializing queue. Compute a heavy weight, large matching. Simple greedy algorithm. Maximal matching, within 2× max weight. Merge all matched communities in parallel. Highly scalable, 5.8 × 106 – 7 × 106 edges/sec, 8× – 16× speed-up on 80-thread E7-8870 Agnostic to weighting, matching... Works with many weighting and termination criteria. Bader, Riedy— STING 17 Jan. 2014 19 / 47
Parallel agglomerative method A B C D E F G C D G Use a matching to avoid a serializing queue. Compute a heavy weight, large matching. Simple greedy algorithm. Maximal matching, within 2× max weight. Merge all matched communities in parallel. Highly scalable, 5.8 × 106 – 7 × 106 edges/sec, 8× – 16× speed-up on 80-thread E7-8870 Agnostic to weighting, matching... Works with many weighting and termination criteria. Bader, Riedy— STING 17 Jan. 2014 19 / 47
Parallel agglomerative method A B C D E F G C D G E B C Use a matching to avoid a serializing queue. Compute a heavy weight, large matching. Simple greedy algorithm. Maximal matching, within 2× max weight. Merge all matched communities in parallel. Highly scalable, 5.8 × 106 – 7 × 106 edges/sec, 8× – 16× speed-up on 80-thread E7-8870 Agnostic to weighting, matching... Works with many weighting and termination criteria. Bader, Riedy— STING 17 Jan. 2014 19 / 47
Community Monitoring Algorithm Given a batch of edge insertions and removals: 1. Collect the set of possibly moved vertices ∆V. Add the endpoints∗ of every edge insertion and removal to ∆V. (STING support routine, scatter/gather and atomic CAS.) 2. Extract each v ∈ ∆V from its community into a singleton. This is the complex piece, see Riedy & Bader, MTAAP13 [7]. 3. Re-start agglomeration from the expanded community graph. ∗: More recent experiments (by Anita Zakrzewska) add larger neighborhoods. Little effect on modularity, large effect on performance. Bader, Riedy— STING 17 Jan. 2014 20 / 47
Platform: Intel® E7-4820-based server Tolerates some latency by hyperthreading: “Westmere-EX:” 2 threads / core, 8 cores / socket, four sockets. Fast cores (2.0 GHz), fast memory (1 066 MHz). Not so many outstanding memory requests (60/socket), but large caches (18 MiB L3 per socket). Good system support Transparent hugepages reduces TLB costs. Fast, user-level locking. (HLE would be better...) OpenMP, although I didn’t tune it... Four processors (64 threads), 1 TiB memory gcc 4.7.2, Linux kernel 3.9 pre-release Acknowledgment: Donated by Oracle. Image: Intel® press kit Bader, Riedy— STING 17 Jan. 2014 21 / 47
Platform: Cray/YarcData XMT2 Tolerates latency by massive multithreading: Hardware: 128 threads per processor Context switch on every cycle (500 MHz) Many outstanding memory requests (180/proc) “No” caches... Flexibly supports dynamic load balancing Globally hashed address space, no data cache Support for fine-grained, word-level synchronization Full/empty bit on with every memory word 64 processor XMT2 at CSCS, the Swiss National Supercomputer Centre. 500 MHz processors, 8192 threads, 2 TiB of shared memory Image: cray.com Bader, Riedy— STING 17 Jan. 2014 22 / 47
Test Data Combination of real and artificial Start with (small) real-world graphs from 10th DIMACS Challenge collection: Name |V| |E| |C| |EC| caidaRouterLevel 192 244 1 218 132 18 343 30 776 coPapersDBLP 540 486 30 866 466 1 401 205 856 eu-2005 862 664 16 138 468 55 624 194 971 Generate artificial edge actions: Compute an initial clustering. Generate random edge insertions between and removals within communities. 15/16: Insertions. Select communities ∝ volume, vertices ∝ 1/degree. 1/16: Removals. Uniform sampling (w/o replacement) of existing edges, then inserted edges. Bader, Riedy— STING 17 Jan. 2014 23 / 47
Peak updates per second There is speed-up per # of threads, etc. on large batches, large graphs. Not the interesting story. Graph Platform #T Batch Upd/Sec Speed-up∗ Latency(s) caidaRouterLevel IA32-64 56 1e5 1.2e+7 4.0 8.3e-3 XMT2 56 1e6 2.5e+6 4.3 4.0e-1 coPapersDBLP IA32-64 20 1e6 2.9e+6 11 3.5e-1 XMT2 48 3e5 2.2e+6 21 1.4e-1 eu-2005 IA32-64 40 1e5 4.8e+6 330 2.1e-2 XMT2 64 1e6 2.1e+6 43 4.9e-1 Large batches, many threads ⇒ > 106 updates per second, but high latency ∗: Speed up over static recomputation. Quality doesn’t decrease significantly v. static recomputation. Bader, Riedy— STING 17 Jan. 2014 24 / 47
Least latency between updates Graph Platform #T Batch Upd./Sec Speed-up∗ Latency (s) caidaRouterLevel IA32-64 2 1 4.6e+2 430 2.2e-3 XMT2 4 30 5.2e+3 230 5.8e-3 coPapersDBLP IA32-64 4 30 7.0e+3 1900 4.3e-3 XMT2 40 1 1.4e+2 380 7.1e-3 eu-2005 IA32-64 12 1 4.2e+2 3500 2.4e-3 XMT2 20 10 1.5e+3 3200 6.5e-3 Small batches, fewer threads ⇒ fast response, but < 104 updates per second ∗: Speed up over static recomputation. Quality doesn’t decrease significantly v. static recomputation. Bader, Riedy— STING 17 Jan. 2014 25 / 47
Incremental PageRank PageRank: Well-understood method for ranking vertices based on random walks (related to minimizing conductance). Equivalent problem, solve (I − αATD−1)x = (1 − α)v given initial weights v. Goal: Use for seed set expansion, sparse v. State-of-the-art for updating x when the graph represented by A changes? Re-start iteration with the previous x. Can do significantly better for low-latency needs. Compute the change ∆x instead of the entire new x. Bader, Riedy— STING 17 Jan. 2014 26 / 47
Incremental PageRank: Iteration Iterative solver Step k → k + 1: ∆x(k+1) = α(A + ∆A)T (D + ∆D)−1 ∆x(k) + α (A + ∆A)T (D + ∆D)−1 − AT D−1 x Additive part: Non-zero only at changes. Operator: Pushes changes outward. Bader, Riedy— STING 17 Jan. 2014 27 / 47
Incremental PageRank: Limiting Expansion Iterative solver Step k → k + 1: ∆ˆx(k+1) = α(A + ∆A)T (D + ∆D)−1 ∆ˆx (k) |ex + α∆ˆx|held α (A + ∆A)T (D + ∆D)−1 − AT D−1 ˆx Additive part: Non-zero only at changes. Operator: Pushes sufficiently large changes outward. Bader, Riedy— STING 17 Jan. 2014 28 / 47
Incremental PageRank: Test Cases Initial, high-level implementation via sparse matrices in Julia. Test graphs from the 10th DIMACS Implementation Challenge. Add uniformly random edges... worst case. Up to 100k in different batch sizes. One sequence of edge actions per graph shared across experiments. Conv. & hold base threshold: 10−12 Graph |V| |E| Avg. Deg. Size (MiB) caidaRouterLevel 192 244 609 066 3.17 5.38 coPapersCiteseer 434 102 1 603 6720 36.94 124.01 coPapersDBLP 540 486 15 245 729 28.21 118.38 great-britain.osm 7 733 822 8 156 517 1.05 91.73 PGPgiantcompo 10 680 24 316 2.28 0.23 power 4 941 6 594 1.33 0.07 Bader, Riedy— STING 17 Jan. 2014 29 / 47
Incremental PageRank: Edge Traversal Savings Percent of edge traversals relative to re-started iteration: 1000 100 10 q q q q q q q q q qq q q q q q q q q qq q q q q q q q q qq q q q q q q q q q qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq q qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq q qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq 0% 20% 40% 60% 80% 0% 20% 40% 60% 80% 0% 20% 40% 60% 80% 0% 20% 40% 60% 80% 0% 20% 40% 60% 80% 0% 20% 40% 60% 80% caidaRouterLevelcoPapersCiteseercoPapersDBLPgreat_britain.osmPGPgiantcompopower 0 2500 5000 7500 10000 0 2500 5000 7500 10000 0 2500 5000 7500 10000 "Time" Fractionofedgesv.restartedPR Graph name q caidaRouterLevel coPapersCiteseer coPapersDBLP great_britain.osm PGPgiantcompo power Held vertex threshold / 10**−12 q q q q10000.0 1000.0 100.0 10.0 Bader, Riedy— STING 17 Jan. 2014 30 / 47
Incremental PageRank: Quality Kendall τ (count of inversions) for ranking result: 1000 100 10 q q q q q q q q q qq q q q q q q q q qq q q q q q q q q qq q q q q q q q q q qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq 0.9995 0.9996 0.9997 0.9998 0.9999 1.0000 0.9995 0.9996 0.9997 0.9998 0.9999 1.0000 0.9995 0.9996 0.9997 0.9998 0.9999 1.0000 0.9995 0.9996 0.9997 0.9998 0.9999 1.0000 0.9995 0.9996 0.9997 0.9998 0.9999 1.0000 0.9995 0.9996 0.9997 0.9998 0.9999 1.0000 caidaRouterLevelcoPapersCiteseercoPapersDBLPgreat_britain.osmPGPgiantcompopower 0 2500 5000 7500 10000 0 2500 5000 7500 10000 0 2500 5000 7500 10000 "Time" Kendalltau Graph name q caidaRouterLevel coPapersCiteseer coPapersDBLP great_britain.osm PGPgiantcompo power Held vertex threshold / 10**−12 q q q q10000.0 1000.0 100.0 10.0 Larger is more similar, max. value one. All values here in (0.995, 1]. Bader, Riedy— STING 17 Jan. 2014 31 / 47
Incremental PageRank: Quality for Seed-Set Use Fagin, et al.’s extension Kendall τ for the top 100 list: 1000 100 10 q q q q q q q q q qq q q q q q q q q qq q q q q q q q q qq q q q q q q q q q qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq0.0000 0.0005 0.0010 0.0015 0.0000 0.0005 0.0010 0.0015 0.0000 0.0005 0.0010 0.0015 0.0000 0.0005 0.0010 0.0015 0.0000 0.0005 0.0010 0.0015 0.0000 0.0005 0.0010 0.0015 caidaRouterLevelcoPapersCiteseercoPapersDBLPgreat_britain.osmPGPgiantcompopower 0 2500 5000 7500 10000 0 2500 5000 7500 10000 0 2500 5000 7500 10000 "Time" K_0(top100) Graph name q caidaRouterLevel coPapersCiteseer coPapersDBLP great_britain.osm PGPgiantcompo power Held vertex threshold / 10**−12 q q q q10000.0 1000.0 100.0 10.0 Smaller is more similar, min. value one. All values here in [0, 0.0018). Bader, Riedy— STING 17 Jan. 2014 32 / 47
Motivation: Graph Algorithms for Analysis STING Review of Accomplishments Selected Details on Recent Work Related Projects (Future Plans) Artifacts of the Intel Non-Numeric Computing Program Bader, Riedy— STING 17 Jan. 2014 33 / 47
GRATEFUL: Graph Analysis Tackling Power Efficiency, Uncertainty, and Locality David A. Bader (PI), Jason Riedy (co-PI) Under DARPA PERFECT (Power Efficiency Revolution for Embedded Computing Technologies) #HR0011-13-2-0001 Goal: Improve ops/watt by factor of 50×–75×! Providing mission-critical analysis of semantic data where needed: as close to the data and data use as possible using the least power necessary to enable rapid decisions with reliable analysis. GRATEFUL is developing new algorithms providing new, fundamental capabilities while coping with real-world uncertainties under run-time power and environmental constraints. STING: Static ⇒ dynamic saves power, lowers latency. Bader, Riedy— STING 17 Jan. 2014 34 / 47
The XScala Project: A Community Repository for Model-driven Design and Tuning of Data- Intensive Applications for Extreme-scale Accelerator-based Systems David A. Bader (PI, GT), Viktor Prasanna (PI, USC), Richard Vuduc (co-PI, GT), Jason Riedy (co-PI, GT) NSF ACI-1339745, an SI2-SSI project Goal: Accelerate science! Map irregular data problems onto “accelerators” and make them usable Science accelerators: multi-threaded, multi-core; GPGPU; cloud; etc. Develop models and analysis kernels together. Not just graphs: also strings, machine learning, ... Not just GPUs: also parallelizing sequential roadblocks, mapping onto FPGAs, finding opportunities for platforms, ... Bader, Riedy— STING 17 Jan. 2014 35 / 47
Graph Algorithm Primitives Community standards effort underway: Timothy Mattson, Jeremy Kepner, John Gilbert, Aydin Buluç, ...) State of the art for “Graphs in the language of Linear algebra” is mature enough to support a standard set of BLAS. Benefits of this standard include: Separation of concerns spares graph algorithm experts the pain of producing optimized primitives (the vendors will produce these ... just as with traditional BLAS) Prevents mass scale “reinvention of the wheel” with each graph research group creating their own primitives. Bader, Riedy— STING 17 Jan. 2014 36 / 47
Higher-Level (aka Fuzzier) Plans More integration into entire analysis tool chains. Continue partnering with ETL researchers (e.g. GraphBuilder), visualization researchers, area experts. Keep STING focused on the graph mapping and algorithms portion. Learn how/what to forget. Cannot keep all data. Make writing high-performance STING kernels easier: Looking into Julia (julialang.org) for dynamically compiled kernels, although large memory footprint. Keeping an eye on other projects (many related through GRATEFUL, e.g. SEJITS in UCB’s ASPIRE) PGAS for distributed memory platforms. Wants a finer-grained interconnect with higher injection rates than Ethernet or IB... And a PGAS mmap... (aka system-level work) Bader, Riedy— STING 17 Jan. 2014 37 / 47
Motivation: Graph Algorithms for Analysis STING Review of Accomplishments Selected Details on Recent Work Related Projects (Future Plans) Artifacts of the Intel Non-Numeric Computing Program Bader, Riedy— STING 17 Jan. 2014 38 / 47
Software & Data Where & What STING/STINGER http://www.cc.gatech.edu/stinger/; framework for dynamic graph analysis. community-el http://www.cc.gatech.edu/~jriedy/ community-detection/ or http://gitorious.org/community-el; separate static community detection code, simple input, no external dependencies. PASQUAL http://www.cc.gatech.edu/pasqual/; shared-memory scalable and dependable de novo assembler DIMACS Challenge http://www.cc.gatech.edu/dimacs10/; data, workshop papers and presentations All software under a modified BSD license. Bader, Riedy— STING 17 Jan. 2014 39 / 47
Publications I Note: Not including presentations, posters, or publications that simply used mirasol... [1] D. Ediger, K. Jiang, E. J. Riedy, and D. A. Bader, “Massive streaming data analytics: A case study with clustering coefficients,” in Proc. Workshop on Multithreaded Architectures and Applications (MTAAP), Atlanta, Georgia, Apr. 2010. [2] D. Ediger, E. J. Riedy, D. A. Bader, and H. Meyerhenke, “Tracking structure of streaming social networks,” in Proc. Workshop on Multithreaded Architectures and Applications (MTAAP), Anchorage, Alaska, May 2011. Bader, Riedy— STING 17 Jan. 2014 40 / 47
Publications II [3] E. J. Riedy, H. Meyerhenke, D. Ediger, and D. A. Bader, “Parallel community detection for massive graphs,” in Proceedings of the 9th International Conference on Parallel Processing and Applied Mathematics, Torun, Poland, Sep. 2011. [4] ——, “Parallel community detection for massive graphs,” in 10th DIMACS Implementation Challenge Workshop - Graph Partitioning and Graph Clustering. Atlanta, Georgia: (workshop paper), Feb. 2012, won first place in the Mix Challenge and Mix Pareto Challenge. [5] E. J. Riedy, D. A. Bader, and H. Meyerhenke, “Scalable multi-threaded community detection in social networks,” in Workshop on Multithreaded Architectures and Applications (MTAAP), Shanghai, China, May 2012. Bader, Riedy— STING 17 Jan. 2014 41 / 47
Publications III [6] R. McColl, O. Green, and D. Bader, “A new parallel algorithm for connected components in dynamic graphs,” in The 20th Annual IEEE International Conference on High Performance Computing (HiPC), Hyderabad, India, Dec. 2013. [7] E. J. Riedy and D. A. Bader, “Scalable multi-threaded community detection in social networks,” in 7th Workshop on Multithreaded Architectures and Applications (MTAAP), Boston, MA, May 2013. [8] O. Green, R. McColl, , and D. Bader, “A fast algorithm for streaming betweenness centrality,” in 4th ASE/IEEE International Conference on Social Computing (SocialCom), Amsterdam, The Netherlands, Sep. 2012. Bader, Riedy— STING 17 Jan. 2014 42 / 47
Publications IV [9] O. Green and D. A. Bader, “A fast algorithm for streaming betweenness centrality,” in 5th ASE/IEEE International Conference on Social Computing (SocialCom), Washington, DC, Sep. 2013. [10] J. Riedy, H. Meyerhenke, D. A. Bader, D. Ediger, and T. G. Mattson, “Analysis of streaming social networks and graphs on multicore architectures,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Japan, Mar. 2012. [Online]. Available: http://www.slideshare.net/jasonriedy/ icassp-2012-analysis-of-streaming-social-networks-and-graph Bader, Riedy— STING 17 Jan. 2014 43 / 47
Publications V [11] D. Ediger, R. McColl, J. Riedy, and D. A. Bader, “STINGER: High performance data structure for streaming graphs,” in The IEEE High Performance Extreme Computing Conference (HPEC), Waltham, MA, Sep. 2012, best paper award. [12] D. A. Bader, H. Meyerhenke, P. Sanders, and D. Wagner, Eds., Graph Partitioning and Graph Clustering. 10th DIMACS Implementation Challenge Workshop, ser. Contemporary Mathematics. Georgia Institute of Technology, Atlanta, GA: American Mathematical Society and Center for Discrete Mathematics and Theoretical Computer Science, 2013, no. 588. Bader, Riedy— STING 17 Jan. 2014 44 / 47
Publications VI [13] X. Liu, P. Pande, H. Meyerhenke, and D. A. Bader, “PASQUAL: Parallel techniques for next generation genome sequence assembly,” IEEE Transactions on Parallel & Distributed Systems, vol. 24, no. 5, pp. 977–986, 2013. [14] J. Fairbanks, D. Ediger, R. McColl, D. Bader, and E. Gilbert, “A statistical framework for analyzing streaming graphs,” in IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), Aug. 2013. Bader, Riedy— STING 17 Jan. 2014 45 / 47
People! GT is an educational institution Related graduate students: David Ediger: graduated, Ph.D., at GTRI Rob McColl: nearing graduation, shared with GTRI Oded Green: nearing graduation James Fairbanks Anita Zakrzewska Xing Liu: PASQUAL Pushkar Pande: PASQUAL, graduated, MS Related post-doc: Henning Meyerhenke: Juniorprof. Dr. at Karlsruhe Inst. Tech. Bader, Riedy— STING 17 Jan. 2014 46 / 47
Acknowledgment of support Bader, Riedy— STING 17 Jan. 2014 47 / 47

More Related Content

PDF
Perspective on HPC-enabled AI
byinside-BigData.com
 
PPTX
Optalysys Optical Processing for HPC
byinside-BigData.com
 
PPT
CI image processing
byMeenakshi Sood
 
PPT
Smart Geo. Guido Satta (Maggio 2015)
byCRS4 Research Center in Sardinia
 
PPTX
IoT 2014 Value Creation Workshop: SDIL
byTill Riedel
 
PDF
Is Your Software Development Process Green?
byFörderverein Technische Fakultät
 
PPTX
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
byVijay Srinivas Agneeswaran, Ph.D
 
PDF
On-the-fly Collaboration for Legacy Business Process Systems in An Open Servi...
byFörderverein Technische Fakultät
 
Perspective on HPC-enabled AI
byinside-BigData.com
 
Optalysys Optical Processing for HPC
byinside-BigData.com
 
CI image processing
byMeenakshi Sood
 
Smart Geo. Guido Satta (Maggio 2015)
byCRS4 Research Center in Sardinia
 
IoT 2014 Value Creation Workshop: SDIL
byTill Riedel
 
Is Your Software Development Process Green?
byFörderverein Technische Fakultät
 
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
byVijay Srinivas Agneeswaran, Ph.D
 
On-the-fly Collaboration for Legacy Business Process Systems in An Open Servi...
byFörderverein Technische Fakultät
 

Viewers also liked

PPTX
Introduction to STINGER
byrobertmccoll
 
PDF
Graph Analyses with Python and NetworkX
byBenjamin Bengfort
 
PDF
Community Detection in Social Media
bySymeon Papadopoulos
 
PDF
Community Detection with Networkx
byErika Fille Legara
 
PDF
STINGER: Multi-threaded Graph Streaming
byJason Riedy
 
PDF
Graph Analysis Trends and Opportunities -- CMG Performance and Capacity 2014
byJason Riedy
 
PDF
Scalable and Efficient Algorithms for Analysis of Massive, Streaming Graphs
byJason Riedy
 
PDF
Network Challenge: Error and Sensitivity Analysis
byJason Riedy
 
PDF
Graph Analysis Beyond Linear Algebra
byJason Riedy
 
PPTX
Hadoop World 2011: Hadoop and Graph Data Management: Challenges and Opportuni...
byCloudera, Inc.
 
PDF
Graph Exploitation Seminar, 2011
byJason Riedy
 
Introduction to STINGER
byrobertmccoll
 
Graph Analyses with Python and NetworkX
byBenjamin Bengfort
 
Community Detection in Social Media
bySymeon Papadopoulos
 
Community Detection with Networkx
byErika Fille Legara
 
STINGER: Multi-threaded Graph Streaming
byJason Riedy
 
Graph Analysis Trends and Opportunities -- CMG Performance and Capacity 2014
byJason Riedy
 
Scalable and Efficient Algorithms for Analysis of Massive, Streaming Graphs
byJason Riedy
 
Network Challenge: Error and Sensitivity Analysis
byJason Riedy
 
Graph Analysis Beyond Linear Algebra
byJason Riedy
 
Hadoop World 2011: Hadoop and Graph Data Management: Challenges and Opportuni...
byCloudera, Inc.
 
Graph Exploitation Seminar, 2011
byJason Riedy
 

Similar to STING: Spatio-Temporal Interaction Networks and Graphs for Intel Platforms

PDF
STING: Spatio-Temporal Interaction Networks and Graphs for Intel Platforms
byJason Riedy
 
PDF
STING: Spatio-Temporal Interaction Networks and Graphs for Intel Platforms
byJason Riedy
 
PPTX
Massive-Scale Analytics Applied to Real-World Problems
byinside-BigData.com
 
PPT
Benchmarking graph databases on the problem of community detection
bySymeon Papadopoulos
 
PPT
Benchmarking graph databases on the problem of community detection
bySotiris Beis
 
PDF
Graph Analysis: New Algorithm Models, New Architectures
byJason Riedy
 
PPTX
Large-Scale Graph Computation on Just a PC: Aapo Kyrola Ph.D. thesis defense
byAapo Kyrölä
 
PDF
PowerGraph
byIgor Shevchenko
 
PDF
Introducing Stig: A New Open Source, Non-relational, Distributed Graph Databa...
byDATAVERSITY
 
ODP
SC10 project slides
byJason Riedy
 
PDF
Big Graph : Tools, Techniques, Issues, Challenges and Future Directions
bycsandit
 
PDF
BIG GRAPH: TOOLS, TECHNIQUES, ISSUES, CHALLENGES AND FUTURE DIRECTIONS
bycscpconf
 
PDF
High-Performance Analysis of Streaming Graphs
byJason Riedy
 
PDF
ICASSP 2012: Analysis of Streaming Social Networks and Graphs on Multicore Ar...
byJason Riedy
 
PDF
Ling liu part 01：big graph processing
byjins0618
 
PDF
Ling liu part 02：big graph processing
byjins0618
 
PDF
STING: Spatio-Temporal Interaction Networks and Graphs for Intel Platforms
byJason Riedy
 
PDF
SIAM Annual Meeting 2012: Streaming Graph Analytics for Massive Graphs
byJason Riedy
 
PDF
ScaleGraph - A High-Performance Library for Billion-Scale Graph Analytics
byToyotaro Suzumura
 
PDF
Graphs as Streams: Rethinking Graph Processing in the Streaming Era
byVasia Kalavri
 
STING: Spatio-Temporal Interaction Networks and Graphs for Intel Platforms
byJason Riedy
 
STING: Spatio-Temporal Interaction Networks and Graphs for Intel Platforms
byJason Riedy
 
Massive-Scale Analytics Applied to Real-World Problems
byinside-BigData.com
 
Benchmarking graph databases on the problem of community detection
bySymeon Papadopoulos
 
Benchmarking graph databases on the problem of community detection
bySotiris Beis
 
Graph Analysis: New Algorithm Models, New Architectures
byJason Riedy
 
Large-Scale Graph Computation on Just a PC: Aapo Kyrola Ph.D. thesis defense
byAapo Kyrölä
 
PowerGraph
byIgor Shevchenko
 
Introducing Stig: A New Open Source, Non-relational, Distributed Graph Databa...
byDATAVERSITY
 
SC10 project slides
byJason Riedy
 
Big Graph : Tools, Techniques, Issues, Challenges and Future Directions
bycsandit
 
BIG GRAPH: TOOLS, TECHNIQUES, ISSUES, CHALLENGES AND FUTURE DIRECTIONS
bycscpconf
 
High-Performance Analysis of Streaming Graphs
byJason Riedy
 
ICASSP 2012: Analysis of Streaming Social Networks and Graphs on Multicore Ar...
byJason Riedy
 
Ling liu part 01：big graph processing
byjins0618
 
Ling liu part 02：big graph processing
byjins0618
 
STING: Spatio-Temporal Interaction Networks and Graphs for Intel Platforms
byJason Riedy
 
SIAM Annual Meeting 2012: Streaming Graph Analytics for Massive Graphs
byJason Riedy
 
ScaleGraph - A High-Performance Library for Billion-Scale Graph Analytics
byToyotaro Suzumura
 
Graphs as Streams: Rethinking Graph Processing in the Streaming Era
byVasia Kalavri
 

More from Jason Riedy

PDF
Novel Architectures for Applications in Data Science and Beyond
byJason Riedy
 
PDF
CRNCH 2018 Summit: Rogues Gallery Update
byJason Riedy
 
PDF
Reproducible Linear Algebra from Application to Architecture
byJason Riedy
 
PDF
Augmented Arithmetic Operations Proposed for IEEE-754 2018
byJason Riedy
 
PDF
MTAAP12: Scalable Community Detection
byJason Riedy
 
PDF
Lucata at the HPEC GraphBLAS BoF
byJason Riedy
 
PDF
Lucata at the HPEC GraphBLAS BoF
byJason Riedy
 
PDF
A New Algorithm Model for Massive-Scale Streaming Graph Analysis
byJason Riedy
 
PDF
ICIAM 2019: A New Algorithm Model for Massive-Scale Streaming Graph Analysis
byJason Riedy
 
PDF
CRNCH Rogues Gallery: A Community Core for Novel Computing Platforms
byJason Riedy
 
PDF
PEARC19: Wrangling Rogues: A Case Study on Managing Experimental Post-Moore A...
byJason Riedy
 
PDF
CRNCH Rogues Gallery: A Community Core for Novel Computing Platforms
byJason Riedy
 
PDF
Updating PageRank for Streaming Graphs
byJason Riedy
 
PDF
GraphBLAS and Emus
byJason Riedy
 
PDF
High-Performance Analysis of Streaming Graphs
byJason Riedy
 
PDF
ICIAM 2019: Reproducible Linear Algebra from Application to Architecture
byJason Riedy
 
PDF
LAGraph 2021-10-13
byJason Riedy
 
PDF
Graph analysis and novel architectures
byJason Riedy
 
PDF
Characterization of Emu Chick with Microbenchmarks
byJason Riedy
 
Novel Architectures for Applications in Data Science and Beyond
byJason Riedy
 
CRNCH 2018 Summit: Rogues Gallery Update
byJason Riedy
 
Reproducible Linear Algebra from Application to Architecture
byJason Riedy
 
Augmented Arithmetic Operations Proposed for IEEE-754 2018
byJason Riedy
 
MTAAP12: Scalable Community Detection
byJason Riedy
 
Lucata at the HPEC GraphBLAS BoF
byJason Riedy
 
Lucata at the HPEC GraphBLAS BoF
byJason Riedy
 
A New Algorithm Model for Massive-Scale Streaming Graph Analysis
byJason Riedy
 
ICIAM 2019: A New Algorithm Model for Massive-Scale Streaming Graph Analysis
byJason Riedy
 
CRNCH Rogues Gallery: A Community Core for Novel Computing Platforms
byJason Riedy
 
PEARC19: Wrangling Rogues: A Case Study on Managing Experimental Post-Moore A...
byJason Riedy
 
CRNCH Rogues Gallery: A Community Core for Novel Computing Platforms
byJason Riedy
 
Updating PageRank for Streaming Graphs
byJason Riedy
 
GraphBLAS and Emus
byJason Riedy
 
High-Performance Analysis of Streaming Graphs
byJason Riedy
 
ICIAM 2019: Reproducible Linear Algebra from Application to Architecture
byJason Riedy
 
LAGraph 2021-10-13
byJason Riedy
 
Graph analysis and novel architectures
byJason Riedy
 
Characterization of Emu Chick with Microbenchmarks
byJason Riedy
 

Recently uploaded

PDF
PeopleSoft Lift and Shift to Oracle Cloud Infrastructure.pdf
byAstuteBusiness
 
PDF
How Does AI Improve Location-Based Mobile App Development for Businesses.pdf
byNet-Craft.com
 
PDF
Ch2 -ENG.pdf COMPUTER ARCHITECTURE L2 INFO
byrogase8895
 
PDF
Code, Coins and Collaboration: How Open Source Shapes Modern Finance.
byDavid Okononfua
 
PPTX
Struggling with Pentaho Limitations How Helical Insight Solves Them.pptx
byVarsha Nayak
 
PPTX
Application Security – Static Application Security Testing (SAST)
byLalit Jagotra
 
PDF
IT Estate Modernization: Transform Your Infrastructure for the Future .pdf
byAstuteBusiness
 
PPTX
Reimagining Service with AI Voice Agents | Webinar
byCEPTES Software
 
PDF
API_SECURITY CONSULTANCY SERVICES IN USA
bydavidjohansen1597
 
PDF
Data structure using C :UNIT-I Introduction to Data structures and Stacks BCA...
byKuvempu University
 
PDF
Database Management Systems(DBMS):UNIT-I Introduction to Database(DBMS) BCA S...
byKuvempu University
 
PDF
Operating System (OS) :UNIT-I Introduction to Operating System BCA SEP SEM-II...
byKuvempu University
 
PPTX
NSF Converter Software to Convert NSF to PST, EML, MSG
byNSF Converter Software
 
PPTX
GDS Integration Solution | GDS Integration Service
byyugababu033
 
PPTX
#15 All About Anypoint MQ - Calicut MuleSoft Meetup Group
bycalicutmulesoftmeetu
 
PPTX
application security presentation 2 by harman
byharmanideapad
 
PPTX
AI Clinic Management Software for Pulmonology Clinics Bringing Clarity, Contr...
byEasy Clinic
 
PDF
SecureChain AI (SCAI) Token – Smart Contract Security Audit Report by EtherAu...
byEtherAuthority
 
PDF
Cybersecurity Alert- What Organisations Must Watch Out For This Christmas Fes...
byCybernetic Global Intelligence
 
PDF
Digitizing Banquet Management_ Why It Matters for Modern Hotels.pdf
byHotelogix
 
PeopleSoft Lift and Shift to Oracle Cloud Infrastructure.pdf
byAstuteBusiness
 
How Does AI Improve Location-Based Mobile App Development for Businesses.pdf
byNet-Craft.com
 
Ch2 -ENG.pdf COMPUTER ARCHITECTURE L2 INFO
byrogase8895
 
Code, Coins and Collaboration: How Open Source Shapes Modern Finance.
byDavid Okononfua
 
Struggling with Pentaho Limitations How Helical Insight Solves Them.pptx
byVarsha Nayak
 
Application Security – Static Application Security Testing (SAST)
byLalit Jagotra
 
IT Estate Modernization: Transform Your Infrastructure for the Future .pdf
byAstuteBusiness
 
Reimagining Service with AI Voice Agents | Webinar
byCEPTES Software
 
API_SECURITY CONSULTANCY SERVICES IN USA
bydavidjohansen1597
 
Data structure using C :UNIT-I Introduction to Data structures and Stacks BCA...
byKuvempu University
 
Database Management Systems(DBMS):UNIT-I Introduction to Database(DBMS) BCA S...
byKuvempu University
 
Operating System (OS) :UNIT-I Introduction to Operating System BCA SEP SEM-II...
byKuvempu University
 
NSF Converter Software to Convert NSF to PST, EML, MSG
byNSF Converter Software
 
GDS Integration Solution | GDS Integration Service
byyugababu033
 
#15 All About Anypoint MQ - Calicut MuleSoft Meetup Group
bycalicutmulesoftmeetu
 
application security presentation 2 by harman
byharmanideapad
 
AI Clinic Management Software for Pulmonology Clinics Bringing Clarity, Contr...
byEasy Clinic
 
SecureChain AI (SCAI) Token – Smart Contract Security Audit Report by EtherAu...
byEtherAuthority
 
Cybersecurity Alert- What Organisations Must Watch Out For This Christmas Fes...
byCybernetic Global Intelligence
 
Digitizing Banquet Management_ Why It Matters for Modern Hotels.pdf
byHotelogix
 

STING: Spatio-Temporal Interaction Networks and Graphs for Intel Platforms

  • 1.
    STING: Spatio-Temporal Interaction Networksand Graphs for Intel Platforms David Bader, Jason Riedy, David Ediger, Rob McColl, Timothy G. Mattson∗, ... Georgia Institute of Technology 17 January 2014
  • 2.
    (insert current prefixhere)scale data analysis Health care Finding outbreaks, population epidemiology Social networks Advertising, searching, grouping Intelligence Decisions at scale, regulating algorithms Systems biology Understanding interactions, drug design Power grid Disruptions, conservation Simulation Discrete events, cracking meshes Data analysis runs throughout. Many problems can be phrased through graphs. Any many are changing and dynamic. Bader, Riedy— STING 17 Jan. 2014 2 / 47
  • 3.
    GT’s part ofIntel’s Non-Numeric Computing Program Goal Supporting analysis of massive graphs under rapid change across the spectrum of Intel-based platforms. STING on Intel-based platforms Maintain a graph and metrics under large data changes Performance, documentation, distribution Available at http://www.cc.gatech.edu/stinger/ Bader, Riedy— STING 17 Jan. 2014 3 / 47
  • 4.
    Outline Motivation: Graph Algorithmsfor Analysis STING Review of Accomplishments Selected Details on Recent Work Dynamic Community Updating Incremental PageRank Related Projects (Future Plans) Artifacts of the Intel Non-Numeric Computing Program Bader, Riedy— STING 17 Jan. 2014 4 / 47
  • 5.
    Why Graphs? Smaller thanraw data. Taught (roughly) to all CS students... Semantic attributions can capture essential relationships. Traversals can be faster than filtering DB joins. Provide clear phrasing for queries about relationships. Often next step after dense and sparse linear algebra. Note: Extracting data into a graph is a separate topic, see Intel’s GraphBuilder. Bader, Riedy— STING 17 Jan. 2014 5 / 47
  • 6.
    Graphs: A FundamentalAbstraction Structure for “unstructured” data Traditional uses: Route planning on fixed routes Logistic planning between sources, routes, destinations Increasing uses: Computer security: Identify anomalies (e.g. spam, viruses, hacks) as they occur, insider threats, control access, localize malware Data / intelligence integration: Find smaller, relevant subsets of massive, “unstructured” data piles Recommender systems (industry): Given activities, automatically find other interesting data. Bader, Riedy— STING 17 Jan. 2014 6 / 47
  • 7.
    On to STING... Motivation:Graph Algorithms for Analysis STING Review of Accomplishments Selected Details on Recent Work Related Projects (Future Plans) Artifacts of the Intel Non-Numeric Computing Program Bader, Riedy— STING 17 Jan. 2014 7 / 47
  • 8.
    STING’s focus Source data predictionaction summary Control VizSimulation/ query STING manages queries against changing graph data. Visualization and control often are application specific. Ideal: Maintain many persistent graph analysis kernels. One current graph snapshot, kernels keep smaller histories. Also (a harder goal), coordinate the kernels’ cooperation. Bader, Riedy— STING 17 Jan. 2014 8 / 47
  • 9.
    STING: High-level architecture Slidecredit: Rob McColl and David Ediger OpenMP + sufficiently POSIX-ish Multiple processes for resilience Bader, Riedy— STING 17 Jan. 2014 9 / 47
  • 10.
    STING Extensible Representation:Core data structure source vertex edge type adj. vtx weight mod. time first time edge type index vertex index Initial considerations [Bader, et al.] Be useful for the entire “large graph” community Portable semantics and high-level optimizations across multiple platforms & frameworks (XMT C, MTGL, etc.) Permit good performance: No single structure is optimal for all. Assume globally addressable memory access and atomic operations Support multiple, parallel readers and a single writer process Large graph ⇒ rare conflicts, may ignore synchronization Bader, Riedy— STING 17 Jan. 2014 10 / 47
  • 11.
    STING in use Whois using it? Internally, well, us. Some details later. Outside institutions in multiple research projects Center for Adaptive Supercomputing Software – Multithreaded Architectures (CASS-MT) directed by PNNL PRODIGAL team for DARPA ADAMS (Anomaly Detection at Multiple Scales) including GTRI, SAIC, Oregon State U., U. Mass., and CMU Several industrial companies in the area of data analytics and security Well-attended tutorials given at PPoPP, in MD, and locally Bader, Riedy— STING 17 Jan. 2014 11 / 47
  • 12.
    STING: Where doyou get it? www.cc.gatech.edu/stinger/ Gateway to code, development, documentation, presentations... (working on usage and development screencasts) Remember: Still academic code, but maturing. Bader, Riedy— STING 17 Jan. 2014 12 / 47
  • 13.
    Motivation: Graph Algorithmsfor Analysis STING Review of Accomplishments Selected Details on Recent Work Related Projects (Future Plans) Artifacts of the Intel Non-Numeric Computing Program Bader, Riedy— STING 17 Jan. 2014 13 / 47
  • 14.
    Selected Program AccomplishmentsI In the beginning: Initial streaming algorithms Maintaining clustering coefficients[1] (counting triangles) Up to 130 000 graph updates per second on X5570 (Nehalem-EP, 2.93GHz) 2000× speed-up over static recomputation Connected components[2] Up to 88 700 graph updates per second on X5570, 233× speed-up Community detection[3, 4, 5] First parallel algorithm for agglomerative community detection, 3.3B edges clustered in 505s on a 4-socket, 10-core Westmere-EX. Win 10th DIMACS Implementation Challenge’s Mix Challenge Enabled moving computation from special server + batch to laptops and servers. Bader, Riedy— STING 17 Jan. 2014 14 / 47
  • 15.
    Selected Program AccomplishmentsII Still maintained, used by external groups like Cray, people finally threatening performance. Expanded streaming algorithms Improved connected components[6] Does not fall back on re-computation for deletions Up to 137× speed-up over static re-computation Community maintenance[7] and monitoring∗ Up to 100 million updates per second, speed-ups from 4× to 3 700× Demonstrated at Research@Intel and GraphLab Workshop 2013 PageRank∗ Reduce edge traversals from 30% – 90%. Betweenness centrality∗ [8, 9] Up to 148× speed-up Optimization and software structure of STING[10] Bader, Riedy— STING 17 Jan. 2014 15 / 47
  • 16.
    Selected Program AccomplishmentsIII Up to 14× performance improvement on Intel-based platforms[11], best paper award at HPEC12 Research@Intel 2013 demo collaborative with Intel GraphBuilder: extract and transform data from a Hadoop store, transfer to STING for analysis Something different: 10th DIMACS Implementation Challenge on Graph Partitioning and Graph Clustering [12] Very useful workshop, 13-14 February 2012 at Georgia Tech Intel sponsorship: Thank you! Graph archive being used in other projects (although not always cited appropriately) PASQUAL [13] First de novo genome assembler that is both scalable and reliable (doesn’t crash). http://www.cc.gatech.edu/pasqual/ (not STING-based) Bader, Riedy— STING 17 Jan. 2014 16 / 47
  • 17.
    Motivation: Graph Algorithmsfor Analysis STING Review of Accomplishments Selected Details on Recent Work Dynamic Community Updating Incremental PageRank Related Projects (Future Plans) Artifacts of the Intel Non-Numeric Computing Program Bader, Riedy— STING 17 Jan. 2014 17 / 47
  • 18.
    Community Detection What dowe mean? Partition a graph’s vertices into disjoint communities. Locally optimize some metric, e.g. modularity, conductance, ... Try to capture that vertices are more similar within one community than between communities. Jason’s network via LinkedIn Labs Bader, Riedy— STING 17 Jan. 2014 18 / 47
  • 19.
    Parallel agglomerative method A B C D E F G Usea matching to avoid a serializing queue. Compute a heavy weight, large matching. Simple greedy algorithm. Maximal matching, within 2× max weight. Merge all matched communities in parallel. Highly scalable, 5.8 × 106 – 7 × 106 edges/sec, 8× – 16× speed-up on 80-thread E7-8870 Agnostic to weighting, matching... Works with many weighting and termination criteria. Bader, Riedy— STING 17 Jan. 2014 19 / 47
  • 20.
    Parallel agglomerative method A B C D E F G C D G Usea matching to avoid a serializing queue. Compute a heavy weight, large matching. Simple greedy algorithm. Maximal matching, within 2× max weight. Merge all matched communities in parallel. Highly scalable, 5.8 × 106 – 7 × 106 edges/sec, 8× – 16× speed-up on 80-thread E7-8870 Agnostic to weighting, matching... Works with many weighting and termination criteria. Bader, Riedy— STING 17 Jan. 2014 19 / 47
  • 21.
    Parallel agglomerative method A B C D E F G C D G E B C Usea matching to avoid a serializing queue. Compute a heavy weight, large matching. Simple greedy algorithm. Maximal matching, within 2× max weight. Merge all matched communities in parallel. Highly scalable, 5.8 × 106 – 7 × 106 edges/sec, 8× – 16× speed-up on 80-thread E7-8870 Agnostic to weighting, matching... Works with many weighting and termination criteria. Bader, Riedy— STING 17 Jan. 2014 19 / 47
  • 22.
    Community Monitoring Algorithm Givena batch of edge insertions and removals: 1. Collect the set of possibly moved vertices ∆V. Add the endpoints∗ of every edge insertion and removal to ∆V. (STING support routine, scatter/gather and atomic CAS.) 2. Extract each v ∈ ∆V from its community into a singleton. This is the complex piece, see Riedy & Bader, MTAAP13 [7]. 3. Re-start agglomeration from the expanded community graph. ∗: More recent experiments (by Anita Zakrzewska) add larger neighborhoods. Little effect on modularity, large effect on performance. Bader, Riedy— STING 17 Jan. 2014 20 / 47
  • 23.
    Platform: Intel® E7-4820-basedserver Tolerates some latency by hyperthreading: “Westmere-EX:” 2 threads / core, 8 cores / socket, four sockets. Fast cores (2.0 GHz), fast memory (1 066 MHz). Not so many outstanding memory requests (60/socket), but large caches (18 MiB L3 per socket). Good system support Transparent hugepages reduces TLB costs. Fast, user-level locking. (HLE would be better...) OpenMP, although I didn’t tune it... Four processors (64 threads), 1 TiB memory gcc 4.7.2, Linux kernel 3.9 pre-release Acknowledgment: Donated by Oracle. Image: Intel® press kit Bader, Riedy— STING 17 Jan. 2014 21 / 47
  • 24.
    Platform: Cray/YarcData XMT2 Tolerateslatency by massive multithreading: Hardware: 128 threads per processor Context switch on every cycle (500 MHz) Many outstanding memory requests (180/proc) “No” caches... Flexibly supports dynamic load balancing Globally hashed address space, no data cache Support for fine-grained, word-level synchronization Full/empty bit on with every memory word 64 processor XMT2 at CSCS, the Swiss National Supercomputer Centre. 500 MHz processors, 8192 threads, 2 TiB of shared memory Image: cray.com Bader, Riedy— STING 17 Jan. 2014 22 / 47
  • 25.
    Test Data Combination ofreal and artificial Start with (small) real-world graphs from 10th DIMACS Challenge collection: Name |V| |E| |C| |EC| caidaRouterLevel 192 244 1 218 132 18 343 30 776 coPapersDBLP 540 486 30 866 466 1 401 205 856 eu-2005 862 664 16 138 468 55 624 194 971 Generate artificial edge actions: Compute an initial clustering. Generate random edge insertions between and removals within communities. 15/16: Insertions. Select communities ∝ volume, vertices ∝ 1/degree. 1/16: Removals. Uniform sampling (w/o replacement) of existing edges, then inserted edges. Bader, Riedy— STING 17 Jan. 2014 23 / 47
  • 26.
    Peak updates persecond There is speed-up per # of threads, etc. on large batches, large graphs. Not the interesting story. Graph Platform #T Batch Upd/Sec Speed-up∗ Latency(s) caidaRouterLevel IA32-64 56 1e5 1.2e+7 4.0 8.3e-3 XMT2 56 1e6 2.5e+6 4.3 4.0e-1 coPapersDBLP IA32-64 20 1e6 2.9e+6 11 3.5e-1 XMT2 48 3e5 2.2e+6 21 1.4e-1 eu-2005 IA32-64 40 1e5 4.8e+6 330 2.1e-2 XMT2 64 1e6 2.1e+6 43 4.9e-1 Large batches, many threads ⇒ > 106 updates per second, but high latency ∗: Speed up over static recomputation. Quality doesn’t decrease significantly v. static recomputation. Bader, Riedy— STING 17 Jan. 2014 24 / 47
  • 27.
    Least latency betweenupdates Graph Platform #T Batch Upd./Sec Speed-up∗ Latency (s) caidaRouterLevel IA32-64 2 1 4.6e+2 430 2.2e-3 XMT2 4 30 5.2e+3 230 5.8e-3 coPapersDBLP IA32-64 4 30 7.0e+3 1900 4.3e-3 XMT2 40 1 1.4e+2 380 7.1e-3 eu-2005 IA32-64 12 1 4.2e+2 3500 2.4e-3 XMT2 20 10 1.5e+3 3200 6.5e-3 Small batches, fewer threads ⇒ fast response, but < 104 updates per second ∗: Speed up over static recomputation. Quality doesn’t decrease significantly v. static recomputation. Bader, Riedy— STING 17 Jan. 2014 25 / 47
  • 28.
    Incremental PageRank PageRank: Well-understoodmethod for ranking vertices based on random walks (related to minimizing conductance). Equivalent problem, solve (I − αATD−1)x = (1 − α)v given initial weights v. Goal: Use for seed set expansion, sparse v. State-of-the-art for updating x when the graph represented by A changes? Re-start iteration with the previous x. Can do significantly better for low-latency needs. Compute the change ∆x instead of the entire new x. Bader, Riedy— STING 17 Jan. 2014 26 / 47
  • 29.
    Incremental PageRank: Iteration Iterativesolver Step k → k + 1: ∆x(k+1) = α(A + ∆A)T (D + ∆D)−1 ∆x(k) + α (A + ∆A)T (D + ∆D)−1 − AT D−1 x Additive part: Non-zero only at changes. Operator: Pushes changes outward. Bader, Riedy— STING 17 Jan. 2014 27 / 47
  • 30.
    Incremental PageRank: LimitingExpansion Iterative solver Step k → k + 1: ∆ˆx(k+1) = α(A + ∆A)T (D + ∆D)−1 ∆ˆx (k) |ex + α∆ˆx|held α (A + ∆A)T (D + ∆D)−1 − AT D−1 ˆx Additive part: Non-zero only at changes. Operator: Pushes sufficiently large changes outward. Bader, Riedy— STING 17 Jan. 2014 28 / 47
  • 31.
    Incremental PageRank: TestCases Initial, high-level implementation via sparse matrices in Julia. Test graphs from the 10th DIMACS Implementation Challenge. Add uniformly random edges... worst case. Up to 100k in different batch sizes. One sequence of edge actions per graph shared across experiments. Conv. & hold base threshold: 10−12 Graph |V| |E| Avg. Deg. Size (MiB) caidaRouterLevel 192 244 609 066 3.17 5.38 coPapersCiteseer 434 102 1 603 6720 36.94 124.01 coPapersDBLP 540 486 15 245 729 28.21 118.38 great-britain.osm 7 733 822 8 156 517 1.05 91.73 PGPgiantcompo 10 680 24 316 2.28 0.23 power 4 941 6 594 1.33 0.07 Bader, Riedy— STING 17 Jan. 2014 29 / 47
  • 32.
    Incremental PageRank: EdgeTraversal Savings Percent of edge traversals relative to re-started iteration: 1000 100 10 q q q q q q q q q qq q q q q q q q q qq q q q q q q q q qq q q q q q q q q q qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq q qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq q qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq 0% 20% 40% 60% 80% 0% 20% 40% 60% 80% 0% 20% 40% 60% 80% 0% 20% 40% 60% 80% 0% 20% 40% 60% 80% 0% 20% 40% 60% 80% caidaRouterLevelcoPapersCiteseercoPapersDBLPgreat_britain.osmPGPgiantcompopower 0 2500 5000 7500 10000 0 2500 5000 7500 10000 0 2500 5000 7500 10000 "Time" Fractionofedgesv.restartedPR Graph name q caidaRouterLevel coPapersCiteseer coPapersDBLP great_britain.osm PGPgiantcompo power Held vertex threshold / 10**−12 q q q q10000.0 1000.0 100.0 10.0 Bader, Riedy— STING 17 Jan. 2014 30 / 47
  • 33.
    Incremental PageRank: Quality Kendallτ (count of inversions) for ranking result: 1000 100 10 q q q q q q q q q qq q q q q q q q q qq q q q q q q q q qq q q q q q q q q q qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq 0.9995 0.9996 0.9997 0.9998 0.9999 1.0000 0.9995 0.9996 0.9997 0.9998 0.9999 1.0000 0.9995 0.9996 0.9997 0.9998 0.9999 1.0000 0.9995 0.9996 0.9997 0.9998 0.9999 1.0000 0.9995 0.9996 0.9997 0.9998 0.9999 1.0000 0.9995 0.9996 0.9997 0.9998 0.9999 1.0000 caidaRouterLevelcoPapersCiteseercoPapersDBLPgreat_britain.osmPGPgiantcompopower 0 2500 5000 7500 10000 0 2500 5000 7500 10000 0 2500 5000 7500 10000 "Time" Kendalltau Graph name q caidaRouterLevel coPapersCiteseer coPapersDBLP great_britain.osm PGPgiantcompo power Held vertex threshold / 10**−12 q q q q10000.0 1000.0 100.0 10.0 Larger is more similar, max. value one. All values here in (0.995, 1]. Bader, Riedy— STING 17 Jan. 2014 31 / 47
  • 34.
    Incremental PageRank: Qualityfor Seed-Set Use Fagin, et al.’s extension Kendall τ for the top 100 list: 1000 100 10 q q q q q q q q q qq q q q q q q q q qq q q q q q q q q qq q q q q q q q q q qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq0.0000 0.0005 0.0010 0.0015 0.0000 0.0005 0.0010 0.0015 0.0000 0.0005 0.0010 0.0015 0.0000 0.0005 0.0010 0.0015 0.0000 0.0005 0.0010 0.0015 0.0000 0.0005 0.0010 0.0015 caidaRouterLevelcoPapersCiteseercoPapersDBLPgreat_britain.osmPGPgiantcompopower 0 2500 5000 7500 10000 0 2500 5000 7500 10000 0 2500 5000 7500 10000 "Time" K_0(top100) Graph name q caidaRouterLevel coPapersCiteseer coPapersDBLP great_britain.osm PGPgiantcompo power Held vertex threshold / 10**−12 q q q q10000.0 1000.0 100.0 10.0 Smaller is more similar, min. value one. All values here in [0, 0.0018). Bader, Riedy— STING 17 Jan. 2014 32 / 47
  • 35.
    Motivation: Graph Algorithmsfor Analysis STING Review of Accomplishments Selected Details on Recent Work Related Projects (Future Plans) Artifacts of the Intel Non-Numeric Computing Program Bader, Riedy— STING 17 Jan. 2014 33 / 47
  • 36.
    GRATEFUL: Graph AnalysisTackling Power Efficiency, Uncertainty, and Locality David A. Bader (PI), Jason Riedy (co-PI) Under DARPA PERFECT (Power Efficiency Revolution for Embedded Computing Technologies) #HR0011-13-2-0001 Goal: Improve ops/watt by factor of 50×–75×! Providing mission-critical analysis of semantic data where needed: as close to the data and data use as possible using the least power necessary to enable rapid decisions with reliable analysis. GRATEFUL is developing new algorithms providing new, fundamental capabilities while coping with real-world uncertainties under run-time power and environmental constraints. STING: Static ⇒ dynamic saves power, lowers latency. Bader, Riedy— STING 17 Jan. 2014 34 / 47
  • 37.
    The XScala Project:A Community Repository for Model-driven Design and Tuning of Data- Intensive Applications for Extreme-scale Accelerator-based Systems David A. Bader (PI, GT), Viktor Prasanna (PI, USC), Richard Vuduc (co-PI, GT), Jason Riedy (co-PI, GT) NSF ACI-1339745, an SI2-SSI project Goal: Accelerate science! Map irregular data problems onto “accelerators” and make them usable Science accelerators: multi-threaded, multi-core; GPGPU; cloud; etc. Develop models and analysis kernels together. Not just graphs: also strings, machine learning, ... Not just GPUs: also parallelizing sequential roadblocks, mapping onto FPGAs, finding opportunities for platforms, ... Bader, Riedy— STING 17 Jan. 2014 35 / 47
  • 38.
    Graph Algorithm Primitives Communitystandards effort underway: Timothy Mattson, Jeremy Kepner, John Gilbert, Aydin Buluç, ...) State of the art for “Graphs in the language of Linear algebra” is mature enough to support a standard set of BLAS. Benefits of this standard include: Separation of concerns spares graph algorithm experts the pain of producing optimized primitives (the vendors will produce these ... just as with traditional BLAS) Prevents mass scale “reinvention of the wheel” with each graph research group creating their own primitives. Bader, Riedy— STING 17 Jan. 2014 36 / 47
  • 39.
    Higher-Level (aka Fuzzier)Plans More integration into entire analysis tool chains. Continue partnering with ETL researchers (e.g. GraphBuilder), visualization researchers, area experts. Keep STING focused on the graph mapping and algorithms portion. Learn how/what to forget. Cannot keep all data. Make writing high-performance STING kernels easier: Looking into Julia (julialang.org) for dynamically compiled kernels, although large memory footprint. Keeping an eye on other projects (many related through GRATEFUL, e.g. SEJITS in UCB’s ASPIRE) PGAS for distributed memory platforms. Wants a finer-grained interconnect with higher injection rates than Ethernet or IB... And a PGAS mmap... (aka system-level work) Bader, Riedy— STING 17 Jan. 2014 37 / 47
  • 40.
    Motivation: Graph Algorithmsfor Analysis STING Review of Accomplishments Selected Details on Recent Work Related Projects (Future Plans) Artifacts of the Intel Non-Numeric Computing Program Bader, Riedy— STING 17 Jan. 2014 38 / 47
  • 41.
    Software & Data Where& What STING/STINGER http://www.cc.gatech.edu/stinger/; framework for dynamic graph analysis. community-el http://www.cc.gatech.edu/~jriedy/ community-detection/ or http://gitorious.org/community-el; separate static community detection code, simple input, no external dependencies. PASQUAL http://www.cc.gatech.edu/pasqual/; shared-memory scalable and dependable de novo assembler DIMACS Challenge http://www.cc.gatech.edu/dimacs10/; data, workshop papers and presentations All software under a modified BSD license. Bader, Riedy— STING 17 Jan. 2014 39 / 47
  • 42.
    Publications I Note: Notincluding presentations, posters, or publications that simply used mirasol... [1] D. Ediger, K. Jiang, E. J. Riedy, and D. A. Bader, “Massive streaming data analytics: A case study with clustering coefficients,” in Proc. Workshop on Multithreaded Architectures and Applications (MTAAP), Atlanta, Georgia, Apr. 2010. [2] D. Ediger, E. J. Riedy, D. A. Bader, and H. Meyerhenke, “Tracking structure of streaming social networks,” in Proc. Workshop on Multithreaded Architectures and Applications (MTAAP), Anchorage, Alaska, May 2011. Bader, Riedy— STING 17 Jan. 2014 40 / 47
  • 43.
    Publications II [3] E.J. Riedy, H. Meyerhenke, D. Ediger, and D. A. Bader, “Parallel community detection for massive graphs,” in Proceedings of the 9th International Conference on Parallel Processing and Applied Mathematics, Torun, Poland, Sep. 2011. [4] ——, “Parallel community detection for massive graphs,” in 10th DIMACS Implementation Challenge Workshop - Graph Partitioning and Graph Clustering. Atlanta, Georgia: (workshop paper), Feb. 2012, won first place in the Mix Challenge and Mix Pareto Challenge. [5] E. J. Riedy, D. A. Bader, and H. Meyerhenke, “Scalable multi-threaded community detection in social networks,” in Workshop on Multithreaded Architectures and Applications (MTAAP), Shanghai, China, May 2012. Bader, Riedy— STING 17 Jan. 2014 41 / 47
  • 44.
    Publications III [6] R.McColl, O. Green, and D. Bader, “A new parallel algorithm for connected components in dynamic graphs,” in The 20th Annual IEEE International Conference on High Performance Computing (HiPC), Hyderabad, India, Dec. 2013. [7] E. J. Riedy and D. A. Bader, “Scalable multi-threaded community detection in social networks,” in 7th Workshop on Multithreaded Architectures and Applications (MTAAP), Boston, MA, May 2013. [8] O. Green, R. McColl, , and D. Bader, “A fast algorithm for streaming betweenness centrality,” in 4th ASE/IEEE International Conference on Social Computing (SocialCom), Amsterdam, The Netherlands, Sep. 2012. Bader, Riedy— STING 17 Jan. 2014 42 / 47
  • 45.
    Publications IV [9] O.Green and D. A. Bader, “A fast algorithm for streaming betweenness centrality,” in 5th ASE/IEEE International Conference on Social Computing (SocialCom), Washington, DC, Sep. 2013. [10] J. Riedy, H. Meyerhenke, D. A. Bader, D. Ediger, and T. G. Mattson, “Analysis of streaming social networks and graphs on multicore architectures,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Japan, Mar. 2012. [Online]. Available: http://www.slideshare.net/jasonriedy/ icassp-2012-analysis-of-streaming-social-networks-and-graph Bader, Riedy— STING 17 Jan. 2014 43 / 47
  • 46.
    Publications V [11] D.Ediger, R. McColl, J. Riedy, and D. A. Bader, “STINGER: High performance data structure for streaming graphs,” in The IEEE High Performance Extreme Computing Conference (HPEC), Waltham, MA, Sep. 2012, best paper award. [12] D. A. Bader, H. Meyerhenke, P. Sanders, and D. Wagner, Eds., Graph Partitioning and Graph Clustering. 10th DIMACS Implementation Challenge Workshop, ser. Contemporary Mathematics. Georgia Institute of Technology, Atlanta, GA: American Mathematical Society and Center for Discrete Mathematics and Theoretical Computer Science, 2013, no. 588. Bader, Riedy— STING 17 Jan. 2014 44 / 47
  • 47.
    Publications VI [13] X.Liu, P. Pande, H. Meyerhenke, and D. A. Bader, “PASQUAL: Parallel techniques for next generation genome sequence assembly,” IEEE Transactions on Parallel & Distributed Systems, vol. 24, no. 5, pp. 977–986, 2013. [14] J. Fairbanks, D. Ediger, R. McColl, D. Bader, and E. Gilbert, “A statistical framework for analyzing streaming graphs,” in IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), Aug. 2013. Bader, Riedy— STING 17 Jan. 2014 45 / 47
  • 48.
    People! GT is aneducational institution Related graduate students: David Ediger: graduated, Ph.D., at GTRI Rob McColl: nearing graduation, shared with GTRI Oded Green: nearing graduation James Fairbanks Anita Zakrzewska Xing Liu: PASQUAL Pushkar Pande: PASQUAL, graduated, MS Related post-doc: Henning Meyerhenke: Juniorprof. Dr. at Karlsruhe Inst. Tech. Bader, Riedy— STING 17 Jan. 2014 46 / 47
  • 49.
    Acknowledgment of support Bader,Riedy— STING 17 Jan. 2014 47 / 47