Apache AGE and the Synergy of Postgres and NoSQL
•Download as PPTX, PDF•
1 like•238 views
This document provides an overview and agenda for a presentation on Apache AGE and the synergy between PostgreSQL and NoSQL databases. The agenda includes introducing the speaker, discussing graph databases and Apache AGE as an open source project, an introduction to Apache AGE including its internal logic and roadmap, advantages of using Apache AGE, and how to contribute to the project. Key points are that Apache AGE is a multi-model graph database built on PostgreSQL that allows users to integrate relational and graph data models, leveraging the features of PostgreSQL and providing a graph query interface using openCypher. The goal of the project is to become an Apache Software Foundation top-level project.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Apache AGE and the Synergy of Postgres and NoSQL
Similar to Apache AGE and the Synergy of Postgres and NoSQL (20)
More from EDB
More from EDB (20)
Recently uploaded
Recently uploaded (20)
Apache AGE and the Synergy of Postgres and NoSQL
- 1. ⓒ 2020 by Bitnine Co, Ltd. All Rights Reserved. December, 2020 Apache AGE and the synergy effect in the combination of Postgres and NoSQL
- 2. Agenda ● Speaker Introduction ● Graph Database ● Apache AGE as an Open Source Project ○ What is Apache AGE ? ○ Why Apache AGE? ○ The meaning of being an ASF TLP and qualification ○ Current status and upcoming activities ○ External Contributors ● Introducing Apache AGE ○ Apache AGE internal logic ○ Apache AGE roadmap ○ Apache AGE Ecosystem ○ Advantage of using Apache AGE ● How to contribute ?
- 4. Graph Database When Connected Data Matters Most Flexibility
- 5. Lorem ipsum congue tempus Lorem ipsum tempus Real-time Recommendation Engines Fraud Detection Master data manageme nt (MDM) Network and IT operations Identity and access manageme nt (IAM) Graph Database Telecomm unication Financial Services
- 7. Apache AGE Project? This open source project is a new generation of a multi-model graph database for the modern complex data environment. Apache AGE is a multi-model database designed to be simple and user-friendly, which supports the relational and graph data model at the same time that enables users to integrate the legacy relational data model and the flexible graph data model in one database. Since AGE is based on the powerful PostgreSQL RDBMS, it is very robust and fully-featured. AGE is optimized for handling complex connected graph data and provides plenty of powerful database features essential to the database environment including ACID transactions, multi-version concurrency control, stored procedure, triggers, constraints, sophisticated monitoring and a flexible data model http://age.apache.org
- 8. A strong need for a cohesive, easy to implement multimodel databases. Apache AGE is an extension of PostgreSQL which supports all the functionalities and features of the PostgreSQL and offers a graph model in addition. Users with a relational background and data model who are in need of having a graph model on top of their existing relational model can use this extension with minimal effort because they can use existing data without migration to enable graph database. Apache AGE?
- 10. Why Apache AGE?
- 11. Why Apache AGE? The R&D team has worked on the edition upon the official announcement of development at PG Vision in Boston in 2019.
- 12. Why Apache AGE? ▪ The users do not need to migrate their data to utilize graph model on the existing relational data that makes profound difference in user adoption. ▪ The extension edition will support multiple versions of PostgreSQL immediately that requires less maintenance. AGE Extension Edition ▪ The alpha version was released in March 2020. Apache announced the incubation podling of AGE. AGE belongs to ASF as an open source. ▪ Full graph data process capability through the implementation of openCypher that is most widely used graph query language.Supports the manifesto of graph query language standardization PostgreSQL Extension ▪ The Apache AGE project is being actively developed and accepts new committers
- 14. ⓒ 2020 by Bitnine Co, Ltd. All Rights Reserved. Numerous Top-Level Projects have successfully raised fund and found acquisition opportunities. ASF Top-Level Project DataStax, the commercial face of Apache Cassandra, announced $106M Series E Funding Couchbase has raised $2 million from Redpoint Ventures ASF Founded1999 2014 Adobe announces acquisition Nitobi, Creator of Apache Cordova 2011 2003 Cloudera announced a $900 million funding round, led by Intel Capital Started Apache HTTP Server, its first project. Jim Jagielski, the original committer of the project is now our champion. Elastic announced raising $70 million in a Series C funding. Elasticsearch is a search engine based on Apache Lucene. 2017 Apache Beam, that was created by Google became a Top-Level Project. Splunk announces acquisition of Streamlio, powered by Apache Pulsar2019 Databricks, the Apache Sparks commercial vendor, Draws $400 Million Series F Investment and $6.2 Billion Valuation
- 15. ⓒ 2020 by Bitnine Co, Ltd. All Rights Reserved. The road to the ASF Top-Level Project Pre-incubation Incubation Top Level Project Jim Jagielski The Cofounder of Apache Project: Kevin Ratnasekera Vice President at Apache Gora Von Gosling Senior Technologist at Alibaba Raphael Bircher President bei Vefko Champion Internal Committers John Gemignani – Core Developer at Bitnine Josh Innis – Core Developer at Bitnine Eya Abdisho – Technical Lead at Bitnine Mentors External Committers Mason Sharp - Principal Engineer at Immuta Jasper Blues - CEO of Liberation Data Aaron Genovia - IT Support Technician at Google Proposal Draft Find a sponsor The mentors are not only targeting to the ‘Top Level Project’ election. Find mentors Call a vote Incubator Community submitted support vote Elected as an Incubation podling project Team up feedback Podling graduation IPMC vote Community Graduation vote External Mentor Dave Fisher Director at Apache Amanda K Moran Software Engineer at Apple Felix Cheung VP of Engineering at SafeGraph
- 16. How Apache AGE works Transforms a Cypher query into a Query tree that will be attached as a subquery node. 2 Parses Cypher queries imbedded in cypher function calls. Here we implement the grammar for openCypher. 1 Understands some graph operations and produces plan nodes that are related to graph operations. 3 Executes plan nodes that are related to graph operations.4 Query Parsing Query Transform Planner/Optimizer Executor Storage (PostgreSQL) Cypher queries work with Postgres’ existing fully transactional system (ACID). 5 Transaction/CacheLayer AGE Architecture
- 17. How AGE works Parses Cypher queries by a function call that uses a parser following the OpenCypher standard. 1 Query Parsing Query Transform Planner/Optimizer Executor Storage (PostgreSQL) Transaction/CacheLayer AGE Architecture
- 18. How AGE works Transforms a Cypher query into a Query tree.2 Query Parsing Query Transform Planner/Optimizer Executor Storage (PostgreSQL) Transaction/CacheLayer AGE Architecture
- 19. How AGE works Understands some graph operations and produces plan nodes that are related to graph operations. 3 Query Parsing Query Transform Planner/Optimizer Executor Storage (PostgreSQL) Transaction/CacheLayer AGE Architecture
- 20. How AGE works Executes plan nodes that related to graph operations.4 Query Parsing Query Transform Planner/Optimizer Executor Storage (PostgreSQL) Transaction/CacheLayer AGE Architecture
- 21. How Apache AGE works Query Parsing Query Transform Planner/Optimizer Executor Storage (PostgreSQL) Cypher queries work with Postgres’ existing fully transactional system (ACID). 5 Transaction/CacheLayer AGE Architecture
- 22. ⓒ 2020 by Bitnine Co, Ltd. All Rights Reserved. ● User Interface ○ Visualization – It takes a connector to use AgensBrowser ○ Graph Modeler framework ● ETL ○ Data ingestion workflow - ingesting data from other RDBMS’s ○ Object Graph Mapper - to support GraphQL and Spring Data ○ Sparql compatibility ● Data Analytics ○ Analytic framework ○ High-level Analytics API ○ Machine Learning framework ● DMBS ○ Distributed Graph ○ Expanding Multi-Graph Features ○ Enhanced HA - Multi Master nodes ○ PL/pgSQL, PL/agCypher, PL/Python - Providing API ○ AGE CLI Wrapper Apache AGE Roadmap Graph Databases Graph Computing Framework Graph Visualization and Business Intelligence Dashboard Graph processing frameworks / engines Graph analytics libraries and toolkits AGE Functions as both a database and a Graph Process Engine. Front-endBack-end
- 23. Advantages Apache AGE Advantages of Apache AGE ● Has the full support and mentorship of the Apache community. ● Eases adoption and system migration to a graph database for PostgreSQL users. ● Supports multiple versions of PostgreSQL that clients may already be using. ● Leverages third party developers and the PostgreSQL community. ● Has better support for other Postgres extensions. ● All of this eases, and take the fear out of adoption and migration to Apache AGE. ● The unique feature of multi-graph queries that no other openCypher graph database currently supports. ● The ability to adapt to other graph query languages: Gremlin, GraphQL, etc. ● Quicker support of new features, performance improvements, and bug fixes from Postgres.
- 24. Challenges that Apache AGE solves Allows hybrid queries between SQL and Cypher. AGE
- 25. Querying Multiple Graphs Many users of Agensgraph complained of not being able to query multiple graphs. Apache AGE offers a solution that will allow users query multiple graphs at the same time. An example from the Healthcare Domain: Find all Long Term Services & Support (LTSS) claims that a doctor has. Challenges that Apache AGE solves
- 26. ⓒ 2020 by Bitnine Co, Ltd. All Rights Reserved. Primary Goal: Have Apache AGE support the core functionality of the openCypher specification, with the exception of Multi-Labels. AGE Timeline at a Glance: AGE Development Timeline - 2020,2021 Quarter 1 (Alpha version 0.1.0): completed! ● Create Agtype to support all datatype requirements in cypher queries ● Basic Match and Create Clause support ● Expression Support Quarter 2 (Alpha version 0.2.0): completed! ● Extend Agtype for better functionality in SQL queries ● Scalar Functions ● Advanced Match and Create Clause support, except for VLE Quarter 3 (Alpha version 0.3.0): completed! ● SET, REMOVE clause support ● Aggregation Support ● Mathematical Computation Functions Quarter 4 (initial beta release): ● VLE support ● Postgres 12 support ● Java Driver Support ● Label Inheritance ● DELETE, MERGE clause
- 27. How To Contribute Instructions on how to contribute to Apache AGE http://age.apache.org/instruction.html Apache AEG Documentation https://github.com/bitnine-oss/AgensGraph-Extension/tree/master/doc Download Apache AGE https://github.com/bitnine-oss/AgensGraph-Extension http://age.apache.org
- 28. ⓒ 2020 by Bitnine Co, Ltd. All Rights Reserved. The ASF is home to a wide range of nearly 200 software product communities, each working with their own collaborative community style to create the open source software products. The first project was HTTP Server, most widely used technology as an Internet protocol. Apache AGE Goal - Top Level Project 338 Top Level Projects ▪ There are total 338 Top Level Projects since 1999. ▪ The Top-Level projects are adapted by numerous organizations and organically build solid communities around them. 45 Incubating Projects Apache Software Foundation Projects ▪ Currently there are 45 projects in the incubation podling stage. To be elected as a Top-Level project. They need to meet strict requirements. ▪ "Apache project" specifically means a top-level project at the ASF. Project using the Apache license alone are not qualified as "Apache projects". ▪ Top level projects are created by the Board. The Incubator Project Management Committee (IPMC) can therefore only recommend to the Board that the project is ready to graduate to a top-level project. ▪ World leading enterprises such as Google supports ASF and sponsors promising projects. For example, Apache Beam, the map- reduce processor, was initiated by Google and elected as a top-level Project.
- 29. ⓒ 2020 by Bitnine Co, Ltd. All Rights Reserved. How To Become a Apache Member Where to start with Apache https://community.apache.org/gettingStarted/101.html How to contribute back to the community https://www.apache.org/foundation/getinvolved.html Apache Incubating projects https://incubator.apache.org/
- 30. Q & A “We do not learn from experience. We learn from reflecting on experience.” ― John Dewey AGE Slack Channel : https://agensgraphsupport.slack.com/archives/C0102328XEJ
- 31. Thank you!
Editor's Notes
- I will be giving a bit of an overview of the AGE architecture and implementation details. AGE is implemented as an extension to PostgreSQL (which is like a plugin to PostgreSQL). AGE is roughly composed of 4 parts - shown in the slide. qp,qt,po,ex And Underneath AGE we have the PostgreSQL database, the transaction/cache layer and the storage layers. AGE implements the components for openCypher query parsing, query transformation, planning and optimizing, and execution.
- John: At the top we have the query parser which implements the tokenizer and grammar for the openCypher language specification. This is where an openCypher command is translated into a parse tree representation. When done, the generated parse tree is fed into the query transform phase.
- John: Once we have the parse tree, we move to the query transform phase and transform the parse tree into a query tree. A query tree is what PostgreSQL will eventually turn into an execution tree to be executed in the backend. At the end of this phase, the generated query tree is passed on to PostgreSQL which will do some additional processing and then pass it to the planner/optimizer phase.
- Josh: In the planner phase, we manipulate the plan that Postgres creates to allow data transformations in SELECT queries. Without being able to manipulate the planner, postgres would not be able to create, update, or delete information in in the middle of the Execution phase.
- Josh: In the execution phase, the cypher statement is executed with all read operations using existing postgres’ nodes, that were set up in the transform phase. All write operations use custom nodes that were set up in the optimizer phase.
- Josh: Cypher’s transformation nodes still conform to ACID principles. All create, update, and delete operations in cypher work within Postgres’ transaction system.