The document proposes a 3-phase algorithm to compute the metric backbone of a weighted graph to improve the performance of graph algorithms and queries. Phase 1 finds 1st-order semi-metric edges by only examining triangles. Phase 2 identifies metric edges in 2-hop paths. Phase 3 runs BFS to label remaining edges. The algorithm removes up to 90% of semi-metric edges and scales to billion-edge graphs. Real-world graphs exhibit significant semi-metricity, and the backbone provides up to 6x speedups for graph queries and analytics.

1. THE SHORTEST PATH IS NOT
ALWAYS A STRAIGHT LINE
leveraging semi-metricity in large-scale graph analysis
Vasiliki Kalavri (kalavri@kth.se) KTH Royal Institute of Technology
Tiago Simas (tiago.simas@telefonica.com)Telefonica Research
Dionysios Logothetis (dionysios@fb.com) Facebook

2. 2
Alice42 likes
Weighted graphs capture
relationship strength
distance
similarity
social proximity
rating
preference
influential nodes
optimal propagation paths
communities
recommendations
BobMax
3 likes

3. 3
Sparsification techniques reduce the
graph size and still give exact or good
approximate results
G G’
f(G) ~ f(G’)

4. THE METRIC BACKBONE
Reduces the graph size while
maintaining relevant structure
The minimum subgraph of a weighted graph, that
preserves the shortest paths of the original graph
4
B
E
DA
C
2
3
10
4
2
1
B
E
DA
C
2
3
2
1

5. WHAT CAN WE USE IT FOR?
• Exact computations
• any algorithm that depends on the shortest paths
• reachability, connectivity
• betweenness centrality, closeness centrality
• Approximation
• PageRank, random walks
• eigenvector centrality
• community detection, clustering
5

6. WHAT CAN WE USE IT FOR?
• Exact computations
• any algorithm that depends on the shortest paths
• reachability, connectivity
• betweenness centrality, closeness centrality
• Approximation
• PageRank, random walks
• eigenvector centrality
• community detection, clustering
5
Improves community detection
modularity and recommender
systems accuracy

7. IMPACT ON LARGE-SCALE SYSTEMS
• Graph Databases
• fewer edges => smaller path search space
• Batch Graph Processing
• CPU and memory requirements depend on #messages
• #messages proportional to #edges
• fewer edges => improved analysis performance
• Graph Compression
• fewer edges => storage reduction
6

8. BACKGROUND

9. SEMI-METRICITY
In a weighted graph, an edge is semi-metric, if there
exists a shorter indirect path between its endpoints
8
B
E
DA
C
2
3
10
4
2
1

10. SEMI-METRICITY
In a weighted graph, an edge is semi-metric, if there
exists a shorter indirect path between its endpoints
9
B
E
DA
C
2
3
10
4
2
1
CE is 1st-order
semi-metric:
C-D-E is a shorter
2-hop path

11. SEMI-METRICITY
In a weighted graph, an edge is semi-metric, if there
exists a shorter indirect path between its endpoints
10
B
E
DA
C
2
3
10
4
2
1
AD is 2nd-order
semi-metric:
A-B-C-D is a shorter
3-hop path
CE is 1st-order
semi-metric:
C-D-E is a shorter
2-hop path

12. SEMI-METRICITY
In a weighted graph, an edge is semi-metric, if there
exists a shorter indirect path between its endpoints
11
B
E
DA
C
2
3
10
4
2
1
CE is 1st-order
semi-metric:
C-D-E is a shorter
2-hop path
AD is 2nd-order
semi-metric:
A-B-C-D is a shorter
3-hop path
AB, BC, CD, DE
are metric

13. BACKBONE ALGORITHM

14. BACKBONE CALCULATION
• Calculating the backbone:
• find all semi-metric edges: 1 BFS per edge?
• compute APSP and store O(N2) paths
13

15. BACKBONE CALCULATION
• Calculating the backbone:
• find all semi-metric edges: 1 BFS per edge?
• compute APSP and store O(N2) paths
Can we calculate or
approximate the backbone
without solving APSP?
13

16. ORDER OF SEMI-METRICITY
14

17. ORDER OF SEMI-METRICITY
14
Most semi-metric edges are
1st-order semi-metric

18. A 3-PHASE BACKBONE ALGORITHM
15
Find 1st-order semi-metric
edges: only look at triangles
1.

19. A 3-PHASE BACKBONE ALGORITHM
15
Find 1st-order semi-metric
edges: only look at triangles
1. Scalable & practical
for large graphs

20. EXAMPLE
16
B
E
DA
C
2
3
10
4
2
1

21. EXAMPLE
17
B
E
DA
C
2
3
10
4
2
1
Phase 1

22. EXAMPLE
18
B
E
DA
C
2
3
10
2
1
Phase 1

23. A 3-PHASE BACKBONE ALGORITHM
19
Find 1st-order semi-metric
edges: only look at triangles
1. Scalable & practical
for large graphs

24. A 3-PHASE BACKBONE ALGORITHM
19
Find 1st-order semi-metric
edges: only look at triangles
1.
Identify metric edges in
2-hop paths
2.
Scalable & practical
for large graphs

25. A 3-PHASE BACKBONE ALGORITHM
19
Find 1st-order semi-metric
edges: only look at triangles
1.
Identify metric edges in
2-hop paths
2.
Scalable & practical
for large graphs
Most semi-metric edges
have been removed

26. EXAMPLE
20
B
E
DA
C
2
3
10
2
1
Phase 2

27. EXAMPLE
20
B
E
DA
C
2
3
10
2
1
Phase 2
M
M
M
M
The lowest-weight edge
of every vertex is metric

28. EXAMPLE
20
B
E
DA
C
2
3
10
2
1
Phase 2
M
M
M
M
The lowest-weight edge
of every vertex is metric
u
v
2
4
2
1
any indirect path
from u to v
would have
larger weight

29. EXAMPLE
20
B
E
DA
C
2
3
10
2
1
Phase 2
?
M
M
M
M
The lowest-weight edge
of every vertex is metric
u
v
2
4
2
1
any indirect path
from u to v
would have
larger weight

30. A 3-PHASE BACKBONE ALGORITHM
21
Find 1st-order semi-metric
edges: only look at triangles!
1.
Identify metric edges in
2-hop paths
2.
Scalable & practical
for large graphs!
Most semi-metric edges
have been removed

31. A 3-PHASE BACKBONE ALGORITHM
21
Find 1st-order semi-metric
edges: only look at triangles!
1.
Identify metric edges in
2-hop paths
2.
Run a BFS for remaining
unlabeled edges.
3.
Scalable & practical
for large graphs!
Most semi-metric edges
have been removed

32. A 3-PHASE BACKBONE ALGORITHM
21
Find 1st-order semi-metric
edges: only look at triangles!
1.
Identify metric edges in
2-hop paths
2.
Run a BFS for remaining
unlabeled edges.
3.
Scalable & practical
for large graphs!
1%-9% edges
Most semi-metric edges
have been removed

33. EXAMPLE
22
B
E
DA
C
2
3
10
2
1
Phase 3
M
M
M
M
BFS

34. EXAMPLE
22
B
E
DA
C
2
3
10
2
1
Phase 3
M
M
M
M
BFS
Explore paths
with shorter
distances only

35. EXAMPLE
22
B
E
DA
C
2
3
10
2
1
Phase 3
M
M
M
M
BFS
Explore paths
with shorter
distances only
If the BFS arrives at
the target, the edge
is semi-metric

36. EXAMPLE
23
B
E
DA
C
2
3
2
1
Metric Backbone

37. DISTRIBUTED IMPLEMENTATION
code available: http://grafos.ml/okapi.html#analytics
24
Implementation in the vertex-centric model

38. EVALUATION

39. EVALUATION GOALS
• How does our algorithm compare to APSP?
• Are large, real-world graphs semi-metric?
• Can we improve graph analysis performance?
26

40. COMPARISONTO APSP
Computing APSP in Giraph
• multiple SSSPs
• multiple MSSPs, i.e. SSSPs from
several sources in parallel
27

41. COMPARISONTO APSP
Computing APSP in Giraph
• multiple SSSPs
• multiple MSSPs, i.e. SSSPs from
several sources in parallel
27
In the order of months
for million-edge graphs

42. COMPARISONTO APSP
Computing APSP in Giraph
• multiple SSSPs
• multiple MSSPs, i.e. SSSPs from
several sources in parallel
27
In the order of months
for million-edge graphs
In the order of days for
million-edge graphs

43. COMPARISONTO APSP
Computing APSP in Giraph
• multiple SSSPs
• multiple MSSPs, i.e. SSSPs from
several sources in parallel
27
In the order of months
for million-edge graphs
In the order of days for
million-edge graphs
Our algorithm is 120-180x faster than SSSP
and 11-14x faster than MSSP:
order of hours for million-edge graphs

44. ALGORITHM PHASES
28
Phase 1 Phase 2 Phase 3

45. ALGORITHM PHASES
28
Phase 1 Phase 2 Phase 3
Very fast
and scalable

46. ALGORITHM PHASES
28
Phase 1 Phase 2 Phase 3
Very fast
and scalable
Removes up to 90%
of semi-metric edges

47. ALGORITHM PHASES
28
Phase 1 Phase 2 Phase 3
Very fast
and scalable
Removes up to 90%
of semi-metric edges
Moderately fast

48. ALGORITHM PHASES
28
Phase 1 Phase 2 Phase 3
Very fast
and scalable
Removes up to 90%
of semi-metric edges
Moderately fast
Labels up to 60%
of the unlabeled edges

49. ALGORITHM PHASES
28
Phase 1 Phase 2 Phase 3
Very fast
and scalable
Removes up to 90%
of semi-metric edges
Moderately fast
Labels up to 60%
of the unlabeled edges
Slow

50. ALGORITHM PHASES
28
Phase 1 Phase 2 Phase 3
Very fast
and scalable
Removes up to 90%
of semi-metric edges
Moderately fast
Labels up to 60%
of the unlabeled edges
Slow
Labels up to 1-9%
of the total edges

51. ALGORITHM PHASES
28
Phase 1 Phase 2 Phase 3
Very fast
and scalable
Removes up to 90%
of semi-metric edges
Moderately fast
Labels up to 60%
of the unlabeled edges
Slow
Labels up to 1-9%
of the total edges
Phase 1 is the fastest and most useful phase

52. PHASE 1 SCALABILITY
29

53. PHASE 1 SCALABILITY
29
<200s on a
billion-edge graph

54. PHASE 1 SCALABILITY
29
almost linear
scalability
<200s on a
billion-edge graph

55. SEMI-METRICITY IN REAL GRAPHS
30
Graph |V| |E| metric semi-metricity
Facebook 190M 49.9B custom 26.5%
Twitter 40M 1.5B jaccard 39%
Tuenti 12M 685M jaccard 59%
Livejournal 4.8M 34M jaccard 40%
NotreDame 0.3M 1.5M jaccard, adamic 45%-29%
DBLP 318K 1M jaccard, adamic 23%-9%
Twitter-ego 81K 1.7M jaccard, adamic 57%-39%
Movielens 1.6K 1.9M jaccard 88%
Facebook 1K 143K
#messages,
message size
78%-77%
US-Airports 0.5K 6K #passengers 72%
C-Elegans 0.3K 2.3K #connections 17%

56. SEMI-METRICITY IN REAL GRAPHS
30
Graph |V| |E| metric semi-metricity
Facebook 190M 49.9B custom 26.5%
Twitter 40M 1.5B jaccard 39%
Tuenti 12M 685M jaccard 59%
Livejournal 4.8M 34M jaccard 40%
NotreDame 0.3M 1.5M jaccard, adamic 45%-29%
DBLP 318K 1M jaccard, adamic 23%-9%
Twitter-ego 81K 1.7M jaccard, adamic 57%-39%
Movielens 1.6K 1.9M jaccard 88%
Facebook 1K 143K
#messages,
message size
78%-77%
US-Airports 0.5K 6K #passengers 72%
C-Elegans 0.3K 2.3K #connections 17%
% 1st-order semi-
metric edges =>
reduction in memory and
communication

57. QUERY SPEEDUP ON NEO4J
31
6.7x speedup

58. APACHE GIRAPH SPEEDUP
32
Including the time to calculate the backbone
4x speedup

59. APACHE GIRAPH SPEEDUP
33
6x speedup

60. COMMUNICATION REDUCTION
34
Up to 70% for highly semi-
metric graphs

61. BEST PRACTICES
When to use the backbone?
• semi-metric weighting schemes, e.g. neighborhood similarity
• we can amortize the overhead: e.g. many algorithms on the same graph,
multiple distance queries
• lossy compression is ok
When not to use the backbone?
• for metric weighting schemes
• we need to run one-off analysis
• we need lossless compression
35

62. RECAP: MAIN CONTRIBUTIONS
36
• An algorithm for computing the metric
backbone without solving APSP
• An open-source distributed implementation
• Graph query and graph analytics speedup on
Neo4j and Apache Giraph

63. THE SHORTEST PATH IS NOT
ALWAYS A STRAIGHT LINE
leveraging semi-metricity in large-scale graph analysis
Vasiliki Kalavri (kalavri@kth.se) KTH Royal Institute of Technology
Tiago Simas (tiago.simas@telefonica.com)Telefonica Research
Dionysios Logothetis (dionysios@fb.com) Facebook