Global Graph Summit presentation on the past, present, and near future of Apache TinkerPop. Part of Data Day Texas, January 25th, 2020.

1. TinkerPop 2020Joshua Shinavier, Ph.D.
Global Graph Summit
Austin, Texas - January 25th
, 2020

2. ○ A brief history of Gremlin
○ Open problems
○ What’s next
TinkerPop 2020

3. A brief history of Gremlin

4. Long, long ago...

5. Long, long ago...

6. Long, long ago...

7. TinkerPop 0.x

8. TinkerPop 0.x
○ “Making stuff for the fun of it..."
○ From RDF to the property graph data model
○ A Turing complete path language for graphs
○ Ripple!
○ Oh, and Gremlin
○ Blueprints
○ “JDBC for graphs”
○ RDF ←→ PG support added early on

9. TinkerPop 0.x
○ Rexster
○ Server for Blueprints-enabled graphs
○ Predecessor of Gremlin Server
○ Pipes
○ Pull-based dataflow framework
○ Frames
○ Object-oriented graph interfaces using Java annotations

10. TinkerPop 1.x

11. TinkerPop 1.x
○ Supported graph back-ends as of May 2012:
○ TinkerGraph (in-memory), Neo4j, OrientDB, DEX (Sparksee)
○ Blueprints adapters for
○ Sesame RDF (RDF4j), JUNG

12. TinkerPop 2.x

13. TinkerPop 2.x
○ Furnace
○ Algorithms package built for property graphs
○ Predecessor of graph OLAP in TinkerPop3
○ New language ecosystems
○ Expansion of functionality on top of Blueprints

14. TinkerPop 3.x

15. ○ Complete rewrite of TinkerPop
○ Focus on scale and performance
○ Symmetry between OLTP and OLAP
○ Gremlin becomes more central
○ Git mono-repo
○ Interfaces with not only graph DBs, but
graph processors
TinkerPop 3.x

16. ○ Not-only-JVM
○ Gremlin in native programming languages
○ Now dozens of graph systems implementing TinkerPop
○ Third-party managed libraries and tools
Apache TinkerPop

17. Graph systems
○ Alibaba Graph Database
○ Amazon Neptune
○ ArangoDB
○ Bitsy
○ Blazegraph
○ CosmosDB
○ ChronoGraph
○ DSEGraph
○ GRAKN.AI
○ Hadoop (Spark)
○ HGraphDB
○ Huawei Graph Engine Service
○ IBM Graph
○ JanusGraph
○ Neo4j
○ neo4j-gremlin-bolt
○ OrientDB
○ Apache S2Graph
○ Sqlg
○ Stardog
○ TinkerGraph
○ Titan
○ Titan + Tupl
○ Unipop

18. Query languages, drivers, and GLVs
○ Clojure: ogre
○ Cypher: cypher-for-gremlin
○ Elixir: gremlex
○ Go: grammes, gremgo
○ Haskell: greskell, gremlin-haskell
○ Java: Ferma, gremlin-objects,
Peapod, spring-data-gremlin,
gremlin-driver
○ JavaScript: gremlin-javascript,
gremlin-orm,
gremlin-template-string
○ Kotlin: kotlin-gremlin-ogm
○ .NET: Gremlin.Net, Gremlinq
○ PHP: gremlin-php
○ Python: Goblin, gremlin-python,
gremlin-py, ipython-gremlin,
gremlinclient, gremlin-python, JUGRI,
gremlinrestclient, python-gremlin-rest
○ Ruby: gremlin_client
○ Rust: gremlin-rs
○ Scala: gremlin-scala,
reactive-gremlin,
scalajs-gremlin-client
○ SPARQL: sparql-gremlin
○ SQL: sql-gremlin
○ Typescript: ts-tinkerpop

19. Open problems

20. Escape from the JVM
○ TinkerPop originally 100% Java + Groovy
○ Still very JVM-heavy
○ Gremlin-Server is Java-only
○ How to achieve parity across languages?
○ Ideally: complete Gremlin VM in every language ecosystem
○ Code generation?
○ How to generate both:
○ Clean APIs
○ Efficient runtime code
○ ...that fit together?

21. Making life easier for graph providers
○ Creating TinkerPop implementations
○ Currently a monolithic effort for each language / environment
○ How do we:
○ Ensure consistency across implementations?
○ Reduce the workload?
○ Thoughtful test suite
○ Rigorous in terms of correct operations
○ Does not force functionality that may not fit
○ Types and constraints may help

22. Network serialization formats
○ GraphML (XML)
○ Widely supported
○ Graphs only
○ GraphSON (JSON)
○ TinkerPop-specific
○ Graphs, elements, paths, etc.
○ {1.0, 2.0, 3.0}
○ GraphBinary
○ TinkerPop-specific
○ Graphs, elements, paths, etc.
○ Good forward-compatibility
○ Gryo (Kryo)
○ JVM only

23. Network serialization formats
○ GraphML (XML)
○ Widely supported
○ Graphs only
○ GraphSON (JSON)
○ TinkerPop-specific
○ Graphs, elements, paths, etc.
○ {1.0, 2.0, 3.0}
○ GraphBinary
○ TinkerPop-specific
○ Graphs, elements, paths, etc.
○ Good forward-compatibility
○ Gryo (Kryo)
○ JVM only
○ Bit of a format zoo
○ One format to rule them all?
○ Mappings between formats?
○ Will schemas help?
○ How about common RPC formats
○ Thrift, Protobuf, Avro, etc.

24. ○ Property graphs:
○ Strong on intuitiveness
○ Historically weak on schema
○ Lightweight property graph schemas
○ E.g. in JanusGraph, Neo4j, basic Graph.Features
○ Stronger graph schemas
○ RDF triple stores, hypergraph databases, object databases, etc.
○ Schemas facilitate composability of data and queries
○ ...enabling optimizations, mappings, migration, other good stuff
○ What’s the best fit for TinkerPop?
Schemas in TinkerPop

25. Getting transactions right
○ How to support diverse transactional models?
○ Neo4j is different than JanusGraph is different than...
○ Is there a unified approach to:
○ Threads + queries + transactions?
○ Transactional scope?
○ Transaction failures?
○ Nested transactions?
○ etc.
○ Will functional approaches to concurrency help?

26. Static analysis for traversals
○ Stop supporting opaque traversals
○ Security issues
○ Portability issues
○ Need a replacement for closures/lambdas
○ “Just write Gremlin”
○ What additional features are required?

27. Graph stream processing
○ Much of the world’s data is streaming
○ Much of that data describes entities and relationships
○ Decades of research on relational stream processing
○ 10+ years on continuous SPARQL
○ What is continuous Gremlin?
○ (RDF)-[:betterThan]->(PG) for streaming
○ RDF stream := unbounded sequence of triples
○ Property graph stream := ?
○ Need schemas, global identifiers, set operations on graphs

28. Abstractions
Data models
Query languages
Formal inference
Transformations
Embeddings
Graph +
Relational model
Streams
...
Human and machine knowledge
Knowledge graphs
Enterprise
Personal
Collaborative
Mental representations
Representation learning
Visualization and HCI
...
Processing and performance
Graph...
Ingestion
Generation
Partitioning
Compression
Concurrent systems
Parallel
Distributed
Graph analytics
Hardware acceleration
Benchmarks and metrics
...
The 1010
foot view

29. What’s next

30. From Graph.Features to a real type system
○ No existing standard for property graphs
○ Recent community efforts
○ W3C Workshop on Web Standardization for Graph Data (March 2019)
○ Property Graph Schema Working Group (PGSWG)
○ Graph Query Language (GQL)
○ Don’t forget about external data models
○ Relational model
○ RDF and other graph models
○ Data interchange formats (Protocol Buffers, Thrift, Avro, etc.)
○ OO, ER, and semistructured data models

31. Taming the dragon (connecting 3+ data models)

32. Taming the dragon (connecting 3+ data models)
→

33. Algebraic Property Graphs
○ Last year at Data Day...
○ A Graph is a Graph is a Graph
○ Composable and bidirectional mappings
○ Formal property graph data model
○ Taxonomy of graph elements
○ Use category theory for the model
○ Developed with Ryan Wisnesky (Conexus AI)
○ Implementations in Haskell and CQL
○ Minimal cover for enterprise data
○ Analogous features in graph and non-graph data models

34. EIements, labels, values, and types

35. Graph transformations

36. Building structure APIs
○ Vertices, edges, and properties
○ Special cases that can be derived from the type system
○ Graphs are different
○ Not described in terms of types
○ Graph API is often redundant in TinkerPop3
○ Structure APIs currently written by hand
○ In each language, for each Gremlin Language Variant
○ We can generate consistent interfaces across GLVs
○ Some tooling already exists
○ Build new tools if we want to make it easier

37. Building process APIs
○ Need abstractions for graph processing
○ Steps, constraints, traversals
○ Freebie: every traversal has a graph representation
○ Graph programs as graph data
○ Generate process APIs for each GLV
○ Using a schema; analogous to generating structure APIs
○ Possible to also generate process implementations?
○ That would be great, but... TBD
○ Code gen options: Haskell? Idris? LLVM? Custom code...

38. Abstractions for graph processing
○ Gremlin traversals are “like” monadic composition
○ Let’s make them properly monadic
○ Pure functional encapsulation of:
○ Side-effects, transactions, exception handling
○ Learn from existing functional approaches to Gremlin
○ Gremlin-Scala, Greskell, Gremlin-Haskell

39. Mainstream languages for serialization

40. Transforming graph data and operations
○ Need a language for schema mappings
○ In theory, that gives us:
○ Automated query rewriting
○ Automated data migration
○ Mix-and-match operations
○ Easy, right...?

41. Making a smooth transition
○ (TP3 → TP4) ≠ (TP2 → TP3)
○ Large user base, good support for TinkerPop3
○ Q: how do we:
○ Make new features useful to the current community
○ Make the migration to TinkerPop4 as seamless as possible
○ A: we try stuff out
○ “The revolution will be A/B tested”
○ Get involved!
○ gremlin-users@googlegroups.com
○ dev@tinkerpop.apache.org

42. Thanks!
Joshua Shinavier
joshsh@uber.com
{ }∪{ , , , ...
Stephen Mallette Marko Rodriguez Ketrina Yim Graph community