Data Summer Conf 2018, “Analysing Billion Node Graphs (ENG)” — Giorgi Jvaridze, Senior Software Engineer at Zalando
•
0 likes•355 views
Many important data science problems can be approached using graphs. The data that can be represented as graphs are everywhere: Social networks, Economic networks, Biomedical networks, Network of Neurons, the Internet itself. Some of those graphs can become very large. There are many challenges that we have to deal with when we want to process, analyse and visualize large graphs. In his presentation Giorgi Jvaridze will talk about experiences he had working with multi-billion node graphs.
Recommended
Recommended
More Related Content
Similar to Data Summer Conf 2018, “Analysing Billion Node Graphs (ENG)” — Giorgi Jvaridze, Senior Software Engineer at Zalando
Similar to Data Summer Conf 2018, “Analysing Billion Node Graphs (ENG)” — Giorgi Jvaridze, Senior Software Engineer at Zalando (20)
More from Provectus
More from Provectus (20)
Recently uploaded
Recently uploaded (20)
Data Summer Conf 2018, “Analysing Billion Node Graphs (ENG)” — Giorgi Jvaridze, Senior Software Engineer at Zalando
- 1. ANALYSING BILLION NODE GRAPHS DATA SUMMER CONF 2018 ODESSA, UKRAINE GIORGI JVARIDZE SENIOR SOFTWARE ENGINEER - ZALANDO 21-07-2018
- 2. 2 ABOUT ME - Giorgi Jvaridze https://www.linkedin.com/in/jvaridze/ - Senior Software Engineer @ Zalando - Previously at Tripadvisor - Personalization - Dublin Data Science meetup group - Data Science Tbilisi meetup group - DataFest Tbilisi - Annual conference about Data Science, Data Journalism and related topics.
- 3. 3 - Intros - Graphs - Cross Device Graph - Deterministic Approach - Probabilistic Approach - Challenges - Conclusion - Q&A AGENDA
- 5. OUR VISION: CONNECTING PEOPLE AND FASHION
- 6. 6 ZALANDO AT A GLANCE ~ 4.5billion EUR revenue 2017 > 200 million visits per month > 15,000 employees in Europe > 75% of visits via mobile devices > 23 million active customers > 300,000 product choices ~ 2,000 brands 17 countries as at Jun 2018
- 7. 7 WE BRING FASHION TO PEOPLE IN 17 COUNTRIES 2008-2009 2010 2012-2013 2011 2018
- 10. 10 GRAPHS / NETWORKS - Vertices or Nodes - Edges
- 11. 11 GRAPHS / NETWORKS - Vertices or Nodes - Edges - Weighted vs Unweighted
- 12. 12 GRAPHS / NETWORKS - Vertices or Nodes - Edges - Weighted vs Unweighted - Directed vs Undirected
- 13. 13 GRAPHS / NETWORKS - Vertices or Nodes - Edges - Weighted vs Unweighted - Directed vs Undirected - Connected Components
- 14. 14 GRAPHS / NETWORKS - Vertices or Nodes - Edges - Weighted vs Unweighted - Directed vs Undirected - Connected Components - Vertex/Edge attributes g h c d e a b f C1 C2 C3
- 15. 15 GRAPHS / NETWORKS - Vertices or Nodes - Edges - Weighted vs Unweighted - Directed vs Undirected - Connected Components - Vertex/Edge attributes - Many more properties and algorithms
- 16. 16 - Social Networks - The Internet - Biomedical Networks - Network of Neurons - Many Data Science Applications GRAPHS / NETWORKS
- 17. 17 DATA SCIENCE AT ZALANDO
- 19. 19 USE CASES
- 20. 20 USE CASES
- 21. 21 USE CASES
- 23. 23 some_id other_id another_id f1 c1 - f1 c2 - f1 c2 u1 f2 - u1 f3 c3 - f4 c4 u2 f5 c5 u2 f6 - u2 c1 f1 c2 f2u1 f3 c3 f4 f5 f6 c4 c5 u2 GRAPH CREATION
- 24. 24 some_id other_id another_id f1 c1 - f1 c2 - f1 c2 u1 f2 - u1 f3 c3 - f4 c4 u2 f5 c5 u2 f6 - u2 c1 f1 c2 f2u1 f3 c3 f4 f5 f6 c4 c5 u2 GRAPH CREATION
- 25. 25 some_id other_id another_id f1 c1 - f1 c2 - f1 c2 u1 f2 - u1 f3 c3 - f4 c4 u2 f5 c5 u2 f6 - u2 c1 f1 c2 f2u1 f3 c3 f4 f5 f6 c4 c5 u2 GRAPH CREATION
- 26. 26 some_id other_id another_id f1 c1 - f1 c2 - f1 c2 u1 f2 - u1 f3 c3 - f4 c4 u2 f5 c5 u2 f6 - u2 c1 f1 c2 f2u1 f3 c3 f4 f5 f6 c4 c5 u2 GRAPH CREATION
- 27. 27 some_id other_id another_id f1 c1 - f1 c2 - f1 c2 u1 f2 - u1 f3 c3 - f4 c4 u2 f5 c5 u2 f6 - u2 c1 f1 c2 f2u1 f3 c3 f4 f5 f6 c4 c5 u2 GRAPH CREATION
- 28. 28 some_id other_id another_id f1 c1 - f1 c2 - f1 c2 u1 f2 - u1 f3 c3 - f4 c4 u2 f5 c5 u2 f6 - u2 c1 f1 c2 f2u1 f3 c3 f4 f5 f6 c4 c5 u2 CONNECTED COMPONENTS
- 29. 29 ARCHITECTURE
- 30. 30 TECHNOLOGIES AWS S3 AWS EMR AWS Data Pipeline
- 31. 31 APACHE SPARK
- 32. 32 SPARK GRAPHFRAMES - Distributed graph processing framework built on top of DataFrames - GraphX is based on RDDs - Represented by GraphFrame object - GraphFrame object takes two DataFrames: Vertices and Edges - Vertices must have id column - Edges must have src and dst column
- 33. 33 SPARK GRAPHFRAMES - Distributed graph processing framework built on top of DataFrames - GraphX is based on RDDs - Represented by GraphFrame object - GraphFrame object takes two DataFrames: Vertices and Edges - Vertices must have id column - Edges must have src and dst column
- 34. 34 CONNECTED COMPONENTS - NAIVE IMPLEMENTATION (GRAPHX) 1. Assign each vertex unique component ID 2. Iterate until convergence: - For each vertex V, update: Component ID of V ← Smallest component ID in neighbourhood of V
- 35. 35 CONNECTED COMPONENTS - NAIVE IMPLEMENTATION (GRAPHX) 1. Assign each vertex unique component ID 2. Iterate until convergence: - For each vertex V, update: Component ID of V ← Smallest component ID in neighbourhood of V Pro: Easy to implement Con: Slow convergence on large-diameter graphs
- 36. 36 SMALL-/LARGE-STAR ALGORITHM (GRAPHFRAMES) 1. Assign each vertex unique component ID 2. Iterate until convergence: - Small Star: For each vertex, connect all smaller neighbors and self to the min neighbor. - Large Star: For each vertex, connect all strictly larger neighbors to the min neighbor including self Kiveris et al. "Connected Components in MapReduce and Beyond."
- 37. 37 SMALL-/LARGE-STAR ALGORITHM (GRAPHFRAMES) Kiveris et al. "Connected Components in MapReduce and Beyond."
- 38. 38 SMALL-/LARGE-STAR ALGORITHM (GRAPHFRAMES) Kiveris et al. "Connected Components in MapReduce and Beyond."
- 39. 39 some_id other_id another_id f1 c1 - f1 c2 - f1 c2 u1 f2 - u1 f3 c3 - f4 c4 u2 f5 c5 u2 f6 - u2 c1 f1 c2 f2u1 f3 c3 f4 f5 f6 c4 c5 u2 CONNECTED COMPONENTS
- 41. 41
- 42. 42 c1 f1 c2 f2u1 f3 c3 f4 f5 f6 c4 c5 u2 Vertex 1 Features Vertex 2 Features Label c1 f1 true c4 f4 true ... ... ... Training Data CLASSIFICATION PROBLEM
- 43. 43 CLASSIFICATION PROBLEM c1 f1 c2 f2u1 f3 c3 f4 f5 f6 c4 c5 u2 Vertex 1 Features Vertex 2 Features Label c1 f1 true c4 f4 true c1 c4 false c1 c5 false ... ... ... Training Data
- 44. 44 ● ID Metadata - first seen, last seen, number of sessions etc.. ● Session Information - mobile/desktop, device, browser, geoip, activity patterns etc.. ● Product Information - category browse, product similarity ● Fashion Insights ● Features of Neighbours - immediate or within connected component… ● Graph Features - graph centrality, community detection scores etc.. ● Graph Embeddings FEATURE ENGINEERING
- 45. 45 Graph Embeddings DeepWalk and Node2vec ● How do we capture structure in relational data? https://thegradient.pub/structure-learning/ ● Fraud Detection Using Deep Learning on Graph Embeddings and Topology Metrics https://www.experoinc.com/post/fraud-detection-using-deep-learning-on-graph-e mbeddings-and-topology-metrics ● Graph Convolutional Networks - https://tkipf.github.io/graph-convolutional-networks/
- 46. 46 WORD2VEC
- 47. 47 NODE2VEC - TEXT GENERATION c1 f1 c2 f2u1 f3 c3 f4 f5 f6 c4 c5 u2 c1 f1 c2 u1 f1 u1 c2 u1 f2 f3 c3 f3 c3 c4 f4 u2 f5 c5 f5 ...
- 48. 48 Challenges Visualization and Data Quality
- 49. 49 Top largest connected components
- 50. 50 Why is the component #187 so huge? Top largest connected components
- 51. 51 Betweenness Centrality in Florentine Families’ Marriage Network
- 52. 52 Betweenness Centrality in Florentine Families’ Marriage Network
- 53. 53 Betweenness Centrality in Florentine Families’ Marriage Network But there are ~4 Million nodes. That’s ~8,000,000,000,000 pairs of IDs
- 54. 54 “A Fast and Highly Quality Multilevel Scheme for Partitioning Irregular Graphs”. George Karypis and Vipin Kumar. SIAM Journal on Scientific Computing, Vol. 20, No. 1, pp. 359—392, 1999. GRAPH PARTITIONING - METIS
- 55. 55 GRAPH PARTITIONING - METIS - Partition Graph multiple times - See statistics on how many times each node was on the border between partitions - Investigate the most fragile nodes - Potentially filter them out - Iterate
- 56. 56 OTHER IDEAS - Community Detection methods - Spectral Clustering - Exploratory Analysis - Visualization - Others?
- 57. 57 OTHER IDEAS
- 58. 58 VISUALIZATION - Too big - Most of the tools won’t work - Slow Graph Layout - Hairball Problem
- 59. 59 VISUALIZATION - Too big - Most of the tools won’t work - Slow Graph Layout - Hairball Problem https://github.com/anvaka/ngraph - Tools for working with Graphs (mostly for JS) - Library for visualizing large graphs on WebGL - Offline Graph Layout (C++ / OpenMP)
- 60. 60 GRAPH SUMMARIZATION Graph Summarization Methods and Applications: A Survey by Yike Liu, Tara Safavi, Abhilash Dighe, Danai Koutra https://arxiv.org/abs/1612.04883
- 61. 61 Questions?
- 62. This presentation and its contents are strictly confidential. It may not, in whole or in part, be reproduced, redistributed, published or passed on to any other person by the recipient. The information in this presentation has not been independently verified. No representation or warranty, express or implied, is made as to the accuracy or completeness of the presentation and the information contained herein and no reliance should be placed on such information. No responsibility is accepted for any liability for any loss howsoever arising, directly or indirectly, from this presentation or its contents. DISCLAIMER 62