Deep stream dynamic graph analytics with grapharis - Massimo Perini
•
1 like•383 views
World's toughest and most interesting analysis tasks lie at the intersection of graph data (inter-dependencies in data) and deep learning (inter-dependencies in the model). Classical graph embedding techniques have for years occupied research groups seeking how complex graphs can be encoded into a low-dimensional latent space. Recently, deep learning has dominated the space of embeddings generation due to its ability to automatically generate embeddings given any static graph. Grapharis is a project that revitalizes the concept of graph embeddings, yet it does so in a real setting were graphs are not static but keep changing over time (think of user interactions in social networks). More specifically, we explored how a system like Flink can be used to simplify both the process of training a graph embedding model incrementally but also make complex inferences and predictions in real time using graph structured data streams. To our knowledge, Grapharis is the first complete data pipeline using Flink and Tensorflow for real-time deep graph learning. This talk will cover how we can train, store and generate embeddings continuously and accurately as data evolves over time without the need to re-train the underlying model.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Deep stream dynamic graph analytics with grapharis - Massimo Perini
Similar to Deep stream dynamic graph analytics with grapharis - Massimo Perini (20)
More from Flink Forward
More from Flink Forward (20)
Recently uploaded
Recently uploaded (20)
Deep stream dynamic graph analytics with grapharis - Massimo Perini
- 1. Dynamic Graph Embedding on Event Streams Massimo Perini Advisors: P. Carbone, V. Kalavri, G.Ramponi
- 2. Machine learning on graphs ⬡ Incorporate graph structure information into a machine learning model ⬡ Capture the graph topology and other graph relevant graph information 2
- 3. GraphSAGE ⬡ Each node embedding is represented by the aggregation of its features and those of its neighborhood. ⬡ Learns how to aggregate features from a node neighborhood. 3
- 4. Graph representation - GraphSAGE 4
- 5. 5 GraphSAGE D Aggregate A B C E F G Aggregate Aggregate D Aggregate A B A C E Aggregate B F G Aggregate D Aggregate A B Layer 1 Layer 2 h0 -> h1 h1 -> h2
- 6. Problems ⬡ Inductive neural networks are trained using static graphs ⬡ Available systems perform inference using a batch approach ⬡ We will focus on a supervised vertex classification task 6
- 7. 7 Pipeline Labelled graph stream Unlabelled graph stream Train model Inference embeddings
- 9. Dynamic dataset ⬡ Typical training from static datasets: random sampling ⬡ Dynamic dataset: trains more old samples ⬡ Idea: learn more from some samples than from others 9
- 10. Dynamic dataset ⬡ We can measure how unexpected is a vertex ∙ How much the model can’t predict it ⬡ Vertices are selected using this value ⬡ New vertices get maximum priority 10
- 11. Sampling probabilities and vertex classes 11
- 14. Graph updates ⬡ Recomputes only the sections of the dependency trees affected by changes. ⬡ The recomputed dependency hierarchy contains the newly added, changed or removed node. 14
- 16. Let’s scale!
- 17. Problems with batch approaches ⬡ The straggler task determines the runtime ⬡ New mutations are not processed while computation is ongoing ⬡ No low-latency real-time processing 17
- 18. Asynchronous distributed version ⬡ Each vertex stores the vectors of the neighboring vertices up to the maximum depth ⬡ Node communicates the computed features to neighbors. ⬡ Neighbors replace the vectors and perform the aggregation again 18
- 19. State ⬡ Stores in the managed per-vertex keyed state: ∙ Vertex adjacency list (used to send messages to adjacent vertices) ∙ Vertex and neighbours vectors ∙ Neural networks layers (trained parameters) 19
- 20. Message passing - addition 20 E D A F E C B A CB EDF D F E A CB EDF D A A C E A A D h0 h1 h0 h1 h1 A CB EDF h1 h1 B A C B E h1
- 21. Iterations ⬡ Asynchronous iteration allows message-passing between different vertices of a keyed stream 21 A CB EDF E A D hE 1 hE 0 hE 1 hE 0 hE 1hE 1 Keyed state Window Operator
- 22. Windowing policy ⬡ A neural network layer is a matrix: each vertex needs to run matrix multiplications ⬡ Bottleneck if a vertex receives too many messages in a short amount of time ⬡ We use per-vertex session windows with a maximum window length 22
- 23. Neural network execution ⬡ Neural networks require fast linear algebra operations ⬡ Implemented using Breeze: ∙ Uses a wrapper for low-level BLAS ∙ Used in Flink ML and Spark MLLib too 23
- 24. Execution plan 24 Graph Source Flat Map Forward Window Hash Filter Filter ForwardForward Iteration Sink Iteration Source Hash Map Data sink Forward Forward Stream of edges A key represents a vertex of the graph Opposite filter conditions: the operator filters new vertex embeddings and neighbours messages Vertex prediction Message passing between vertices Forward Preprocessing
- 25. Dynamic model ⬡ Once a new model is deployed, the system needs to run again the inference on the graph ⬡ Can be implemented without an explicit barrier 25
- 26. Dynamic model 26 A CB EDF A CB EDF A CB EDF h1 New model The new model is broadcasted (broadcasted state paradigm) Once each vertex receives the model, emits the new vector at “depth 0” Once a vertex receives the “depth 0” vectors from the neighbours, it aggregates them. It then sends the “depth 1” result to neighbours h0
- 27. Execution plan 27 Graph Source Flat Map Forward Window Hash Filter Filter ForwardForward Iteration Sink Iteration Source Hash Map Data sink Forward Forward Forward Model Source Forward Map Co-process-Broadcast-Keyed BroadcastHash Computes graph updates Defines per-vertex windows of incoming messages Message passing between vertices
- 28. Solving Iteration Deadlocks 28 Flat Map Forward Window Hash Filter Filter ForwardForward Iteration Sink Iteration Source Hash Map Data sink Forward Forward Forward Model Source Forward Map Co-process-Broadcast-Keyed BroadcastHash Window ⬡ Problem: ∙ Backpressure on feedback-edge ∙ All channels in loop block indefinitely ⬡ Solution: ∙ New PrioritizedUnionInputGate* ∙ Max priority on feedback input ∙ Min priority on regular input ∙ *http://tiny.cc/fu1xdz (hotfix by senorcarbone for Flink 1.7)
- 29. Experiments
- 30. Datasets ⬡ Pubmed Diabetes (dynamic vertex addition, publication network) ⬡ Reddit (dynamic edge addition, social network) 30 citation paper Same user comment on both posts post topics Communities (“subreddits”)
- 31. Baselines ⬡ Offline: Train over the full graph with multiple epochs every 10 steps ⬡ No-Rehearsal: Train over the new vertices only. ⬡ Random: Train over a uniform random sample selected on the full graph 31
- 32. F1 Macro - Reddit 32 F1 Macro - Pubmed F1 score
- 33. Training time 33
- 34. Scalability - Throughput 34 Pubmed dataset Reddit dataset
- 37. Thank you