The document outlines the role of a data engineering team at Kudo, emphasizing the need for an analytics-friendly database to handle increasing data variety and volume. It discusses the implementation of ETL processes and machine learning for product classification and recommendations. Additionally, it highlights the use of Airflow for scheduling and monitoring data pipelines, as well as the company's recruitment efforts.

1. ETL & Machine Learning
@ Kudo
FemaleGeek & PHP Indonesia
Kudoplex 2, 18 Juni 2016

2. • Three years experienced software
engineer in Java, Android and Python
• Have big interest about Big Data and
Data Science
• Currently working as a Data Engineer
at Kudo
Luthfi Hariz
Email : luthfi@kudo.co.id
Linkedin : https://id.linkedin.com/in/luthfihariz

3. Data Analyst
Predictive Analytics, Reporting
Analysis, Fraud Analysis
Data
Engineer
ETL, Data Infrastructure,
Machine Learning at Scale

4. Why Kudo need Data team ?
● Our data is getting higher,
especially in the variety
● Partnered with many vendor
with different characteristic data
● Unique user (agent) behavior,
not a typical e-commerce user
● Specific user (agent) profile

5. ETL (Extract Transform Load) Machine Learning

6. ETL (Extract Transform Load)
We need ”analytics friendly” database that is single source of all
data in Kudo

7. python package : petl, pandas
Extract Transfor
m
Load
operational
& logs
analytical DB

8. Business Intelligence

9. Tableau UI

10. ETL is all about jobs that run periodically
we need to make sure all jobs run “pretty smooth..”

11. Airflow
a platform to programmaticaly author, schedule and monitor our
data pipelines

12. Support :
• Retries
• Complex
Dependency (DAG)
• Python Operator
• Email on Error/Retry
• Exchange Message
between Task
• Web UI
• etc…
Airflow

13. Airflow
DAG
(Directed Acyclic Graph)
Task A Task B
Task C
Task D
Task E
Email OperatorPython Operator
Retry 5 times
HTTP Operator
SQL Operator
Bash Operator

14. Airflow Web UI

15. Airflow Web UI

16. Machine Learning

17. Product Classification

18. Category
Product

19. Process
“360 Degree
Rotating Quiet
Usb Fan”
Elektronik
0.8
Fesyen
0.15
Perhiasan & Emas
0.05
Model
Product Name
Train Data
0 1 0 1 0 0 0
1 0 1 0 0 0 0
1 1 1 0 1 0 0
1 0 1 1 1 0 0
1 1 0 0 1 1 0
Naive
Bayes

21. Recommendation Engine
Item to Item Similiarity, Collaborative Filtering
User Profilling
Labelling Kudo Agents

22. Thank You!Luthfi Hariz - luthfi@kudo.co.id
Pssst..we are hiring!