In the OpenMetadata Webinar on Custom Connectors we discussed some of the core aspects of OpenMetadata, the ingestion process, and showcased a demo on how you can build your own custom connectors in OpenMetadata. Thanks to the custom connectors, you can integrate your internal business systems with OpenMetadata, and explore their metadata together with the other third-party services crucial to your business.
What we’ll cover in the Mini-Webinar:
* OpenMetadata Foundations: Schemas and APIs
* How this translates to the Python SDK
* How does the ingestion work
* Demo on implementing the Custom Connector
Watch the Webinar here - https://www.youtube.com/watch?v=fDUj30Ub9VE
5. + Flexibility
- Abstraction
API Calls
Python & Java
SDKs
Ingestion
Framework
How do we bring Internal Systems into OpenMetadata?
- Flexibility
+ Abstraction
7. Creating a Table with the API
More info at https://docs.open-metadata.org/swagger.html
PROS ✅ CONS ❌
● Close to no
requirements
● Verbose
● No safeguards
9. Creating a Table with the Python SDK
Java - https://www.jsonschema2pojo.org/
Python - https://github.com/koxudaxi/datamodel-code-generator
Typescript - https://www.npmjs.com/package/quicktype
10. Creating a Table with the Python SDK
More info at https://docs.open-metadata.org/sdk/python
PROS ✅ CONS ❌
● Metadata
standard
● Helper methods
● Language-specific
12. Creating a Custom Connector
What are the Benefits?
● Only focus on extracting metadata
● The Ingestion Framework manages how to schedule
and write the metadata
● Run and schedule the ingestion directly from the UI
13. Creating a Custom Connector
PROS ✅ CONS ❌
● Focus on
business logic
● Integrated with
the full Ingestion
Framework
● Less flexibility
14. Read a CSV
Validate its
contents
Transform to
the OM
standard
Send it to the
next step in the
workflow
Demo: Creating a Custom Connector
15. DEMO:
Creating a Custom Connector
https://github.com/open-metadata/openmetadata-demo/
tree/main/custom-connector
