The OpenMetadata Community Meeting was held on Dec 14th, 2023. In this meeting, we discussed and demoed the upcoming features in the 1.3.0 Release. You can refer the meeting slides here and watch the video on Youtube - https://www.youtube.com/@OpenMetadataChannel/
About OpenMetadata:
OpenMetadata is a unified platform for discovery, observability, and governance, powered by a central metadata repository, in-depth lineage, and seamless team collaboration. Based on Open Metadata Standards and APIs, supporting connectors to a wide range of data services, OpenMetadata enables end-to-end metadata management, giving you the freedom to unlock the value of your data assets. OpenMetadata (https://open-metadata.org/) is an open-source project supported by Collate, Inc. (https://www.getcollate.io/). Collate also supports a *SaaS version* (https://cloud.getcollate.io/).
Join us on Meetup to get notified about OpenMetadata Webinars: https://www.meetup.com/openmetadata-meetup-group/
7. How-to Guides
● No Code; Steps using the UI
● Feature-based Documentation
● Quick Start for Admins, and Data Users
● Advanced Guides
● Step-by-Step Guides with Images
● Webinar Recordings
● Short Video Tutorials
● Tips, Use Cases, and Best Practices
8.
9. Release 1.3 Sneak Peek Jan 2024
+380 Commits so far…
● GCS Storage Connector
● Iceberg Connector
● Doris Connector
● Lineage UI Revamp
● Alerts - Data Observability
● Ingestion Performance
● MetaPilot - Gen AI
● Spark Lineage Agent
● Knowledge Center Improvements
● DQ Incidents Manager
● Custom Metrics
10. Data Quality - What we shipped so far
● 30+ Native Tests
● Database & Data Lake support
● Time Series Visualization of Failures
● Resolution Status
● 3rd Party Integration
11. Data Quality - What’s coming
● Actively manage quality failures with OpenMetadata Incident Manager
13. Profiler - What we shipped so far
● 20+ metrics
● Database & Data Lake support
● Fine grain column level profiling
● Sample, partition, filter out metrics
15. Knowledge Center
● Knowledge Page Hierarchy
● Associate Data Assets With Knowledge Page
● QuickLink Improvements
● Search Support for Knowledge Page
16. Lineage - What we shipped so far
Lineage Specification and API
18. Lineage - What we shipped so far
State of the Art - SQL Query Processor
● Contributions to SQLFluff to improve the parsing of different
SQL dialects
● Major contributions to SQL Lineage python package to make
pluggable SQL parsers
● Performance improvements in SQL Lineage
● Stored Procedure support
19. Lineage - What we shipped so far
Lineage UI
● We shipped huge improvements to the Lineage UI
● Expand all to see the column level lineage
● Ability to load 1000s of lineage nodes by using pagination
innovation
● Preview panel to look at the data assets details without leaving
the Lineage UI
● Users can expand the Lineage view to full screen and many more.
20. Lineage - What’s Coming
● Ability to draw lineage for larger depth and larger height for
100s of nodes and more
● Search for a column and find its lineage E2E, even when its
renamed
● Drive root cause analysis through Lineage
● Notify Users and Consumers of upstream failures
● Propagate tags/descriptions through Lineage
21. Lineage - What’s Coming in 1.3.0
● All the Lineage Graph is published to
ElasticSearch/OpenSearch
● API additions to do Graph like queries to Search Indexes
● Lineage UI built with new APIs
● Brings in additional information such as service details of a data
asset
● Showcasing DQ results and Pipeline status right with-in lineage
view
● Ability to search and filter based on service, ownership, tags
etc., right with-in lineage
22. Data Observability & Notifications
What OpenMetadata Provides
● Metadata Change Events: Any changes from your sources and
with-in OpenMetadata are versioned and a Change Event is
generated
● Webhook, Slack, GChat, Email Integrations
● Simple UI to configure alerts using various conditions on
Change Events
● Enables users to build automation based on Change Events.
23. Data Observability & Notifications
What we are shipping in 1.3.0
● Add Durable Queue based off of Database - ISSUE-12764
24. Data Observability & Notifications
What we are shipping in 1.3.0
● Separation of Data Observability and Notifications
● Notifications
○ Data Assets updates - Ownership, Tags, description etc..
○ Task assignment/resolution
● Data Observability Alerts
○ Schema Change - Tables/Topics/Containers
○ DQ Test Case
○ Pipeline Status
○ Data Asset Deletion
25. Data Observability & Notifications
What we are shipping in 1.3.0
● Improvements to the Slack/GChat and activity feed rendering of
messages
● At-least once guarantees of delivering of these notifications
26. Star us on GitHub
https://github.com/open-metadata/OpenMetadata
Join our Slack
https://slack.open-metadata.org/
Follow us on X
@open_metadata
Discover Collate SaaS
https://www.getcollate.io/