Data Engineering. Basics and getting started. References: The Data Engineering Cookbook by Andreas Kretz

1. DATA ENGINEERING
BASICS & GETTING STARTED

2. DEFINITIONS

3. Data Engineer
 They build and scale the platforms that enable data collection, processing and
storage for data science/business analytics use.
Data Scientist
 They use linear algebra and multivariable calculus to create new insight from
existing data.

4. DATA ENGINEERING

5. Designing, building and scaling systems that organize
data for analytics

6. ETL (EXTRACT,
TRANSFORM, LOAD)

7. Basic architecture of
ETL
Scaling factor

9. DATA CLASSIFICATION

10. Raw data
 Unprocessed data in format used on source e.g JSON
 No schema applied
Processed data
 Raw data with schema applied
 Stored in event tables/destinations in pipelines
Cooked data
 Processed data that has been summarized.

11. BIG DATA PROPERTIES

12. Volume
 How much data you have
Velocity
 How fast data is getting to you
Variety
 How different your data is
Veracity
 How reliable your data is

13. DATA PROCESSING
METHODS

14. BATCH PROCESSING

15. STREAM PROCESSING
Process data on the fly, as it comes in

16. STREAMING METHODS

17. At Least Once
At Most Once
Exactly Once

18. PROCESSING
FRAMEWORKS

19. MAP REDUCE
Key –Value pairing.
Organize the data into keys and values,
Sort by the key,
Combine the data with matching keys
Repeat until you have the final key- value outcome.

21. DATA STORAGE

22. Relational Database (SQL)
Document Store (NoSQL)

23. THANKYOU

24. REFERENCES

25. The Data Engineering Cookbook
https://github.com/andkret/Cookbook