Presented at the openCypher Meetup in New York, USA, on October 2017 @ http://www.opencypher.org/event/2017/10/25/event-oc-meetup/

1. Property graphs with time
Julia Stoyanovich, joint work with Vera Moffitt
Drexel University
Philadelphia, PA USA
stoyanovich.org
openCypher MeetupOctober 25, 2017

2. openCypher MeetupOctober 25, 2017 2
2008 20092007
20112010

3. openCypher MeetupOctober 25, 2017 3
https://www.kenedict.com/apples-internal-innovation-network-unraveled-part-1-evolving-networks/

4. openCypher MeetupOctober 25, 2017 4
https://arxiv.org/abs/1709.06176

5. openCypher MeetupOctober 25, 2017
Exploratory analysis of evolving graphs
• Which nodes are showing an increasing popularity trend?
• Have any changes in network connectivity been
observed?
• At what time scale can interesting trends be observed?
• How can multiple data sources be used jointly to
complement or corroborate information about network
evolution?
5

6. openCypher MeetupOctober 25, 2017
Goal
6
Principled and systematics support for usable,
scalable and extensible analysis of evolving graphs

7. openCypher MeetupOctober 25, 2017
Are Alice and Bill connected?
7
TNGP
… by a path?

8. openCypher MeetupOctober 25, 2017
Snapshot reducibility
8

9. openCypher MeetupOctober 25, 2017
Are Alice and Bill connected?
extended snapshot reducibility
9
… by a journey?
… by a path that persists over >2 time instants

10. openCypher MeetupOctober 25, 2017
TGraph: an evolving property graph
10

11. openCypher MeetupOctober 25, 2017
TGA: Temporal Graph Algebra
• Temporal variants of standard graph operators + novel time-
specific operators
• Compositional: TGraph (or a pair of TGraphs) as input -
TGraph as output
• Operations maintain model integrity
- graph integrity at each time instant: no dangling edges, a
node/edge appears at most once
- temporal integrity: semantics of temporal operations are
automatically enforced (formally: point semantics)
11

12. openCypher MeetupOctober 25, 2017
TGA operations
• trim
• temporal versions of
- vertex-map, edge-map
- subgraph, path
- aggregate messages
- union, intersection, difference - binary
• snapshot analytics
- PageRank, connected components,… - Pregel
12

13. openCypher MeetupOctober 25, 2017
TGA operations
• node creation
• based on temporal window: temporal zoom
• attribute-based: structural zoom
• edge creation
13

14. openCypher MeetupOctober 25, 2017
Structural zoom
14
add university nodes Drexel and CMU,
and edges between students and these universities

15. openCypher MeetupOctober 25, 2017
Structural zoom
15

16. openCypher MeetupOctober 25, 2017
Temporal zoom
16
coarsen taxi trip start-times into 10-min intervals

17. openCypher MeetupOctober 25, 2017
System architecture
17
Portal
Interactive Shell
Query Parser
Spark
Runtime
GraphX
Data Structures
Worker
Spark Runtime
HDFS
Worker
Spark Runtime
HDFS
…
System
Catalog
SparkSQL
Portal Runtime
(optimizer, operators, etc)
Spark 2.0, interoperable with SparkSQL and
with BigDatalog

18. openCypher MeetupOctober 25, 2017
Physical data representation
• On-disk: Apache Parquet
- vertex / edge files
- broken down into snapshot groups
- each file sorted on start time followed by node /edge id
• In-memory:
- nested relational (Vertex-Edge RDDs)
- GraphX-based: RepresentativeGraphs (RG), One
Graph (OG), HybridGraph (HG)
18
1 2 3
BitSet(p1,p2,p3,p4) BitSet(p2,p3,p4,p5)
BitSet(p5)
BitSet(p1,p2,p3,p4,p5)
BitSet(p2,p3)

19. openCypher MeetupOctober 25, 2017
Performance highlights
• 16-node Open Stack cluster
• Apache Spark 2.0
• 4 cores, 16GB / RAM per node
19

20. openCypher MeetupOctober 25, 2017
PageRank on wiki-talk
20

21. openCypher MeetupOctober 25, 2017
PageRank on nGrams
21

22. openCypher MeetupOctober 25, 2017
PageRank on Twitter
22

23. openCypher MeetupOctober 25, 2017
Aggregate messages on wiki-talk
23

24. openCypher MeetupOctober 25, 2017
Vertex-subgraph on wiki-talk
24

25. openCypher MeetupOctober 25, 2017
Portal vs. G*
25
average node degree, wiki-talk

26. openCypher MeetupOctober 25, 2017
Take-aways
• TGraph: a logical model of property graphs with time
• TGA: a compositional temporal graph algebra under
point semantics
• Portal: a library on top of Apache Spark, inter-
operable with SparkSQL
• Ongoing work on a declarative language, multi-
operator query optimization, benchmarking
• Planned open source release this Fall
26

27. openCypher MeetupOctober 25, 2017
References
• Temporal Graph Algebra, Moffitt & Stoyanovich, DBPL
2017.
• Zooming in on NYC taxi data with Portal, Stoyanovich,
Gilbride and Moffitt, DSSG 2017 (arXiv).
• Towards sequenced semantics for evolving graphs,
Moffitt & Stoyanovich, EDBT 2017.
• Towards a distributed infrastructure for evolving graph
analytics, Moffitt & Stoyanovich, TempWeb 2016.
• Vera Moffitt’s Ph.D. thesis.
27

28. openCypher MeetupOctober 25, 2017
Thank you!