Many companies struggle to get some business knowledge from their data. They know they have data, but then what? They even know what to extract from the data, but guess what happens? Garbage in - garbage out. ETL is a known approach for transforming data, but it has a number of drawbacks. One should also know the further reviews of it like ECTL and ELT. We will go through them and also discuss the usual problems with starting data processing in different industries. In this talk, Piotr wants to show the typical mistakes companies make when trying to build their data warehouse, and why small companies fail to even start building their business on their data.

1. Do ECTL not ETL
The art and science of
Data cleansing in data pipelines
#ECTLmeetup

2. Products -> 3 000 000
Categories -> 200
Categorized Products -> 156 000

3. Racing wheel 14’’
aluminum

4. Racing wheel 14’’
aluminum
Watches

5. Apple watch series
4

6. Apple watch series
4
Gym Equipment

7. KINGSTON
MICROSD 128GB
MICRO ADAPTER

8. KINGSTON
MICROSD 128GB
MICRO ADAPTER
Women’s Clothing

9. Accurate prediction -> ~20%
Not categorized -> ~30%
Time spent improving model -> ~2 weeks

10. Hierarchy Of Needs
#ECTLmeetup
#HierarchyOfNeeds

11. Hierarchy of needs
^ Optimize ^

12. Hierarchy of needs
^ Optimize ^
^ Learn ^

13. Hierarchy of needs
^ Optimize ^
^ Learn ^
^ Aggregate ^

14. Hierarchy of needs
^ Optimize ^
^ Learn ^
^ Aggregate ^
^ Transform ^

15. Hierarchy of needs
^ Optimize ^
^ Learn ^
^ Aggregate ^
^ Transform ^
^ Store ^

16. Hierarchy of needs
^ Optimize ^
^ Learn ^
^ Aggregate ^
^ Transform ^
^ Store ^
^ Collect ^

17. Hierarchy of needs
^ Optimize ^
^ Learn ^
^ Aggregate ^
^ Transform ^
^ Store ^
^ Collect ^

18. Hierarchy of needs
^ Transform ^
^ Store ^
^ Collect ^

19. ETL
^ Transform ^
^ Load ^
^ Extract ^

20. ETL vs ELT
#ECTLmeetup
#ETLvsELT

21. Extract Transform Load

22. Extract Load Transform

23. Transform
● Selecting only certain columns to load
● Translating coded values/enumerations
● Encoding free-form values
● Deriving a new calculated value
● Sorting or ordering the data
● Joining data from multiple sources
● De-duplicating the data
● Aggregating
● Generating surrogate-key values
● Transposing or pivoting
● Splitting a column into multiple columns
● Validating the data by referential tables
● Applying any form of data validation

24. Clean Transform
● Encoding free-form values
● Selecting only certain columns to load
● Translating coded values/enumerations
● Joining data from multiple sources
● De-duplicating the data
● Validating the data by referential tables
● Applying any form of data validation
● Deriving a new calculated value
● Sorting or ordering the data
● Aggregating
● Generating surrogate-key values
● Transposing or pivoting
● Splitting a column into multiple columns

25. C is for Cleansing
#ECTLmeetup
#DataCleanning

26. “
Data cleaning is one of the biggest problems in
data warehousing.
“
Ralph Kimball

27. Data quality
● Missing
● Non-valid
● Inaccurate
● Duplicated or Redundant
● Noisy Data

28. Inaccurate Data

29. Duplicated or Redundant Data

30. Final Thoughts
#ECTLmeetup

31. Final Thoughts
● Keep in mind Hierarchy of needs

32. Final Thoughts
● Keep in mind Hierarchy of needs
● Use ELCT rather than ETL

33. Final Thoughts
● Keep in mind Hierarchy of needs
● Use ELCT rather than ETL
● Always:

34. Final Thoughts
● Keep in mind Hierarchy of needs
● Use ELCT rather than ETL
● Always:
○ Deduplicate data

35. Final Thoughts
● Keep in mind Hierarchy of needs
● Use ELCT rather than ETL
● Always:
○ Deduplicate data
○ Detect outliers in data

36. Final Thoughts
● Keep in mind Hierarchy of needs
● Use ELCT rather than ETL
● Always:
○ Deduplicate data
○ Detect outliers in data
○ Gather metrics about data

37. THANK YOU

38. #ECTLmeetup
piotr.wisniowski
@scalac.io
WisniowskiPiotr
Contact me