Download as PDF, PPTX
Uploaded byAlexey Grigorev
Data engineering zoomcamp introduction
The document outlines the plan and syllabus for a Data Engineering Zoomcamp hosted by DataTalks.Club. It introduces the four instructors for the course - Ankush Khanna, Sejal Vaidya, Victoria Perez Mola, and Alexey Grigorev. The 10-week course will cover topics like data ingestion, data warehousing with BigQuery, analytics engineering with dbt, batch processing with Spark, streaming with Kafka, and a culminating 3-week student project. Pre-requisites include experience with Python, SQL, and the command line. Course materials will be pre-recorded videos and there will be weekly live office hours for support. Students can earn a certificate and compete on a
More Related Content
Similar to Data engineering zoomcamp introduction
Data engineering zoomcamp introduction
- 1.
- 2. Plan ● Instructors ● Course ○Is it for me? ○ Syllabus ● Course logistics ● DataTalks.Club slack ● Questions
- 3. Ankush Khanna ● SeniorData Engineer at Wayfair ● Anything Streaming or Batch Picture
- 4. Sejal Vaidya ● Data& ML Engineer ● Course Topics: Platform Infrastructure and Data Ingestion Picture
- 5. Victoria Perez Mola ●Team lead of the Analytics platform at Tier Mobility ● Not a data engineer ● Course topics: Analytics engineering Picture
- 6. Alexey Grigorev ● PrincipalData Scientist at OLX Group ● Not a data engineer =) ● Instructor for ML Zoomcamp Picture
- 7. Plan ● Instructors ● Course ○Is it for me? ○ Syllabus ● Course logistics ● DataTalks.Club slack ● Questions
- 8. Is it forme? ● Pre-requisites ○ Experience with programming (Python) ○ Being comfortable with command line (Git, etc) ○ Exposure to SQL ● Not required ○ Previous experience with data engineering
- 9.
- 10.
- 11.
- 15. Course overview ● Week1: Introduction & Prerequisites ● Week 2: Ingestion and orchestration ● Week 3: Data warehouse (Big query) ● Week 4: Analytics engineering (dbt) ● Week 5: Batch processing (Spark) ● Week 6: Streaming (Kafka) ● Weeks 7-10: Project
- 16. Week 1: Introduction& Prerequisites ● Setting up the environment ○ Google Cloud account ○ Docker ○ Terraform ● Running Postgres in Docker ● Taking a look at the NY taxi dataset ● SQL refresher
- 17. Week 2: Ingestionand orchestration ● Data Lake ○ What is a Data Lake, ELT vs. ETL, Using GCS ● Orchestration ○ What is an Orchestration Pipeline, Data Ingestion, Introducing & Using Airflow ● Demo: ○ Setting up Airflow with Docker ○ Data ingestion DAG ■ Extraction, Pre-processing (parquet, partitioning), Loading, Exploration with BigQuery, etc. ● Best Practices
- 18. Week 3: DataWarehouse ● What is Data warehouse ● BigQuery? ○ Partitioning and Clustering ○ With Airflow ○ Best practices
- 19. Week 4: AnalyticsEngineering ● What is dbt and how does it fit the tech stack? ● Using dbt: ○ Anatomy of a dbt model ○ Seeds ○ Jinja, Macros and tests ○ Documentation ○ Packages ● Build a dashboard in Google data studio
- 20. Week 5: Batchprocessing ● Spark internals ● Broadcasting ● Partitioning ● Shuffling ● Spark + Airflow ● Apache Flink as alternative to Spark
- 21. Week 6: Streamprocessing ● Basics of Kafka ● Consumer-Producer ● Kafka Streams ● Kafka Connect
- 22. Project ● Putting everythingwe learned in practice
- 23. Plan ● Instructors ● Course ○Is it for me? ○ Syllabus ● Course logistics ● DataTalks.Club slack ● Questions
- 24. Course logistics ● Lessons ○Pre-recorded ○ Published on our YouTube channel ● Office hours ○ Live on Mondays 17:00 CET ○ Homework solutions and answering questions ● Project ○ 1-2 weeks - working on the project ○ 3 week - peer reviewing
- 25.
- 26. Course logistics ● Certificate- for passing the project ● Leaderboard ○ Scores for homework ○ Project ○ Learning in public
- 27. Learning in public ●LinkedIn ● Twitter ● Blogs
- 30. Plan ● Instructors ● Course ○Is it for me? ○ Syllabus ● Course logistics ● DataTalks.Club slack ● Questions
- 31. Plan ● Instructors ● Course ○Is it for me? ○ Syllabus ● Course logistics ● DataTalks.Club slack ● Questions