The document outlines a presentation by Francois Callewaert, a senior data scientist at Databricks, focusing on data lifecycle and processes involved in data analytics and machine learning. It covers the roles of data engineers, analysts, and scientists in handling datasets, as well as a practical demo leveraging tools like Azure and Spark for data management and analysis. The presentation emphasizes the integration of various technologies to enhance data-driven business decisions.

1. Databricks
Empowering you to do
more with data
Francois Callewaert
Senior Data Scientist

2. Agenda
▪ Intro
▪ Data Lifecycle
▪ Demo

3. My career in data tools
Before Now
EDA-ML EDA++, sharing
Build and refresh
own datasets
Build and refresh
own datasets in
Python
Azure SQL DB Azure Data Factory
Azure VM
Windows Scheduler
Model-serving
and delivery
Azure ML

4. Software (app,
website, IOT…)
User / signals
Raw logs
Clean logs
(de-dup, PII
removal…)
Business data
(aggregates, joins,
metrics, features...)
Analytics
Modelling
Software
Engineer
Data
Engineer
Data
Analyst
Data
Scientist
Recommender
Systems
Business
Decisions
Data lifecycle
Program
Manager
ML Engineer
Bronze Silver Gold
DE/DA/DS

5. Demo
• Dataset = https://www.kaggle.com/mkechinov/ecommerce-behavior-data-from-multi-category-store
• User actions: VIEW, CART, PURCHASE
• Data Engineer: Delta, Auto-loader
• Data Analyst: Spark SQL, Notebooks, Jobs, SQL Analytics
• Data Scientist: pySpark, Notebooks, Jobs
• GitHub: https://github.com/databricks/tech-talks/tree/master/samples/2020-09-
16%20%7C%20eCommerce%20demo

6. Dimension table
(id → name)
Data Engineer
Cloud bucket
(raw logs)
CSV files are appended
every few seconds
Software
Engineer
Auto-loader
+JOIN
eCommerce events
(BUY, VIEW, CART)
Clean event logs
Data
Engineer

7. Data Analyst
Clean event logs
Data
Engineer
Data
Analyst
Notebook
EDA
Job
Product metrics
(Business data)
SQL
Analytics
Dashboard

8. Data Scientist
Clean event logs
Data
Engineer
Data
Scientis
t
Notebook
EDA - Feature engineering
Job
Feature table
MLFlow
experiment
Prod model

9. Contact: francois.callewaert@databricks.com
Do more with data.
Databricks
Conclusion