Data Lineage is a hot topic in Data Governance and Analytics. Everyone wants to know where the data comes from and where it is used. Learn how you can document the flow of your data with Dataedo.

1. Documenting
Data Lineage
with Dataedo
© 2023, Dataedo sp. z o.o.
Chase Summers
Director, Customer Operations
Piotr Kononow
CEO

2. Agenda
1. What is Data Lineage and Impact Analysis?
2. What’s the challenge?
3. Data Lineage in Dataedo – current solution and plans
4. What you need
5. Announcements
6. Q&A

3. What is Data Lineage

4. What is Data Lineage
• Process of understanding, documenting and visualizing of how data
flows throught the ecosystem

5. Data Lineage vs Impact analysis

7. Data Lineage of an object

8. Impact Analysis in an object

9. Object level vs column level lineage

10. Manual vs automatic
• Fully automatic
• Some elements automatic
• Suggestions, human verification
• Fully manual

11. Why understanding a lineage is a challange?
SQL Views Bactch scripts
SQL Stored procedures
Databasetriggers
External tables
BI tools
Different SQL dialects
Different ETL tools
Pipelines
COPY INTO
CREATE TABLE AS
API calls
Webhooks
Manualentry
Application forms

12. Examples of Data Flows

13. Data flows: Stored Procedures
• Stored procedures
• Functions
• Triggers
Destination
Source
My Procedure
crm_mails
crm_customers

14. Data flows: Views

15. Data flows: ETL

16. Data flows: BI tools

17. Data flows: API and integration

18. Data Lineage in Dataedo

19. Under construction

20. Lineage for views from SQL parsing

21. Lineage for views from dependecies
• Object level only
• Connectors that support dependencies

22. SQL parsing

23. BI tools

24. Power BI

25. Power BI
Currently
• Datasets, columns
• Reports
• Object level lineage
• Some source<->dataset lineage
Soon
• Object level lineage
• On prem

26. ETL tools

27. Defining object level lineage manually
Source Object
Processor (object)
Step 1
dbo.incoming_emails dbo.crm_customers
Destination Object
Processes (steps)
Inflow Outflow
dbo.load_customer_data
Source columns Destination columns
Mapping
Step 2
src_column_1
src_ column_2
dest_column_1
dest_column_2

28. Manual lineage
• Create „processor” or use existing object
• Show inflow and outflow objects

29. Defining column level lineage
• For anly lineage
• Inflow first/Outflow first
• Intelligent matching

30. Plans

31. More metadata for processes (steps) [Q2]
• Script
• Description

32. More metadata for column level linage [Q2]
• Desciption
• Transformation
• Custom fields

33. Automatic lineage for saved queries [Q2]

34. Import from external source/Excel [Q2]
• Import through interface tables
• Paste to UI (Later)

35. Change: direct table to table lineage [Q2]
• Anything can be a processor

36. More object types [Q2]
To help document data ecosystem:
• ETL package
• Application form, list
• API
• …

37. Lineage Assistant [2023+]
• Shows tables/datasets without lineage
• Shows lineage without column lineage

38. Tableau [Q2+]
Q2
• Reports
• Databases and Data Sources
• Lineage between
Later
• Automatic lineage for Custom SQL Query

39. Power BI [Q2+]
• Column level lineage [?]
• Power BI On prem [Q2-3]

40. SSIS [Q3+]
Q3
• Other than SQL Server sources
• Column level lineage
• Import SQL sources to catalog
• Parse SQL for sources
Later
• Support for Execute SQL Task – SQL parsing

41. Snowflake [Q2]
• Snowpipes
• Query Log – COPY INTO
• External tables

42. Data Lineage:
What you need

43. Licensing

44. New customers
Contact: Jonathan@dataedo.com

45. Current customers
• Plans Dataedo, Data Datalog: Upgrade to 10.4
• Legacy plans: contact Jonathan@dataedo.com

46. Slack channel – coming soon!
• Product/feature ideas, feedback and discussions
• Best practices
• Community
Who’s interested?

47. Hiring: Data Success Manager
• Help our customers have success with data and documentation
• Soon on: https://dataedo.com/careers

48. Q&A

49. Contact us:
dataedo@dataedo.com
Support: support@dataedo.com
Chase Summers
Director, Customer Operations
Piotr Kononow
CEO