Recommended
More Related Content
Similar to ETL – Everything you need to know
Similar to ETL – Everything you need to know (20)
More from Adi Polak
More from Adi Polak (7)
Recently uploaded
Recently uploaded (20)
ETL – Everything you need to know
- 2. Adi Polak Microsoft ETL – Everything you need to know
- 3. About me – Adi Polak @adipolak https://medium.com/@adipolak https://dev.to/adipolak
- 4. Use cases AD tech Automotive Cyber Security … EVERYTHING WITH DATA! @adipolak
- 5. Agenda • ETL • Data Pipelines • Data Challenges • Big Data • Stream vs Batch processing • Architecture • Tools • Learn More! @adipolak
- 6. E T L Extract Transform Load @adipolak
- 7. Data Pipeline != ETL
- 8. @adipolak
- 9. ETL
- 10. ETL
- 12. Most complex data pipeline demo
- 13. Data Challenges – Collection of data @adipolak
- 14. Data Challenges - Collection of data Audio, video, images. Meaningless without adding some structure Unstructured JSON, XML, sensor data, social media, device data, web logs. Flexible data model structure Semi-Structured Structured CSV, Columnar Storage (Parquet, ORC). Strict data model structure @adipolak
- 15. Data Challenges – Duplication of data @adipolak
- 16. Data Challenges – Inconsistency of data @adipolak Schema Data representation Data value
- 17. Data Challenges – Variety of data @adipolak
- 18. What about Big Data ?
- 19. @adipolak
- 20. Big Data Processing Pipelines Visualize Azure Machine Learning
- 21. Why is processing Big Data challenging ? Variety Velocity Volume @adipolak
- 22. Data Engineers Analytics tools Knowledge of SQL Data Warehousing Data Architecture Coding Skills Machine Learning * @adipolak
- 24. Batch processing
- 25. NRT / Stream processing
- 26. Why not both?
- 28. BATCH LAYER SPEED LAYER SERVICE LAYER
- 29. Batch Layer Apache Spark Azure Batch Speed Layer Spark Streaming Kafka Stream Azure stream analytics Service Layer Prometheus Graphana PowerBI Tools Many more… @adipolak
- 31. What did we learn today: • ETL vs Data Pipelines • Big Data and Data challenges • Data Engineers • Streaming vs Batch processing • Architectures • Tools @adipolak
Editor's Notes
- Simple data pipeline
- Show command line example ps –A | grep java
- Relational databases (RDBMS) work with structured data. Non-relational databases (NoSQL) work with semi-structured data Relational data and non-relational data are data models, describing how data is organized. Structured, semi-structured, and unstructured data are data types
- Relational databases (RDBMS) work with structured data. Non-relational databases (NoSQL) work with semi-structured data Relational data and non-relational data are data models, describing how data is organized. Structured, semi-structured, and unstructured data are data types
- Relational databases (RDBMS) work with structured data. Non-relational databases (NoSQL) work with semi-structured data Relational data and non-relational data are data models, describing how data is organized. Structured, semi-structured, and unstructured data are data types
- Relational databases (RDBMS) work with structured data. Non-relational databases (NoSQL) work with semi-structured data Relational data and non-relational data are data models, describing how data is organized. Structured, semi-structured, and unstructured data are data types
- Relational databases (RDBMS) work with structured data. Non-relational databases (NoSQL) work with semi-structured data Relational data and non-relational data are data models, describing how data is organized. Structured, semi-structured, and unstructured data are data types
- Variety: It can be structured, semi-structured, or unstructured Velocity: It can be streaming, near real-time or batch Volume: It can be 1GB or 1ZB
- Apache Hadoop-based analytics - it is an open source platform for distributed storage and distributed processing of data sets. These services provide data storage, processing, data access, security, governance, and operations. You need to have a good grasp on tools like MapReduce, Hadoop and HBase etc. Knowledge of SQL - you need to have strong knowledge in relational database management system such as SQL to manage data. Data Warehousing - learning how to construct and use a data warehouse is a must. Data warehousing helps you aggregate unstructured data from one or more sources to compare and analyse for better business. Data Architecture - having knowledge of building complex database systems for companies. This term also refers to processes that address the data at rest, data at motion, data sets, and how they relate to data dependent applications and processes. Coding skills - you need to have good coding skills in Python, Java, Perl etc. Machine Learning - Although, it is said that machine learning is an integral part of data science but having some level of understanding of how to put the data into use using statistical analysis and data modelling is a huge advantage. Therefore, knowing is machine learning can be like cherry to the cake. Operating system - extensive knowledge in operating systems such as Linux, UNIX and Solaris etc. can be very helpful since most of the tools will be based on these systems.