The first version of Data Linage solution if the complexity data pipeline in MOMO - Meet our ENKI...

1. Data Lineage in
MOMO
Mr. Hải Nguyễn
Head of Data Platform

2. MOMO

3. Growths
● Founded date 2007
● Number #1 e-wallet with most connected
banks
● F&B, Transportation, eCommerce, Billpay,
supermarket,...
● ~13,5 million users

4. sources:https://analyticks.wordpress.com/2016/11/14/the-perfect-big-data-platform-my-blueprint/

5. Tech stacks
● Streaming sources: Kafka, Google Pubsub
● Orgirin sources: RDMS, Cassandra.
● Migration tools: Beam on DataFlow, Spark on DataProc.
● Data Lake: GCS, BigQuery.
● ETL tools: Jupyter Notebook, Apache Airflow
● Report tools: Google Data Studio
● Tools: GK8

6. Achievements
● Number of dataset: ~80
● Number of table: ~500
● Number of generated records (daily): 300 mil
● Number of queries (daily): ~40,000
● Data scanned (daily): ~100TB
● Number of ETL: ~500

7. ETL
Source: https://www.astera.com/type/blog/etl-vs-elt-whats-the-difference/

8. Apache Airflow
Airflow is a platform created by community to
programmatically author, schedule and monitor workflows

10. Department A Department B
Department C

11. Department A Department B
Department C

12. What happens if
exceptions occur in
the middle of
nowhere?

13. Department A Department B
Department C

14. ● Downstream job still process => wasting resources
● Based on wrong data => Data correctness
● Preproccess impact downstreams
Challenges

15. Enki
Data Lineage
Source: https://www.ancient.eu/Enki/

16. ● Data lineage includes the data origin, what happens to it and where it
moves over time.[1]
Data lineage gives visibility while greatly
simplifying the ability to trace errors back to the root cause in a data
analytics process
Source:( Hoang, Natalie (2017-03-16). "Data Lineage Helps Drives Business Value | Trifacta".
Trifacta. Retrieved 2017-09-20.)
.
Data Lineage

17. Collecting all information giving us:
- Data source dependencies
- Data source status
- Data source logic
=> Metadata
Our Approach

18. Core information
Data Lineage Service

19. Push rather than Pull

23. Data source status keeper service
Data Heartbeat Service

26. Preprocessing solution

28. Achievements
● No more wrong data sources
● The ability to backfill all data sources with a
step
● Reduce debug time: day => within an hour

29. ● Business information
● Lineage version control
.
What next.

30. Thank you!
We ‘re hiring
hai.nguyen1@mservice.com.vn