etl

1. ETL Process in Data
Warehouse
Babjee Reddy
babjee@gmail.com
BI-Gems Technology

2. OLTP
 Objective of OLTP is to process data as
quickly as possible
 Support client Server technoligy
 Support Large Amount of Data
 Data is secure

3. OLAP/DSS/DWH
 Objective of OLAP database to process
data as Quickly as possible with less
complexity
 It is use for decision making purpose
 Used by Management People

4. Difference between OLTP, OLAP
Transactional oriented Analytical Oriented
Normalized tables De-normailzed tables
Used by clerical people Use by management people
Current data Historical data
Insert , Updates, deletes Bulk load
Select query retrieves less no of records Retrieves large no of records

5. ETL
---------------------------------------------------------------------------------------------------------------------
• This Section Covers
• What is ETL is
• Motivation
• Where to Use ETL
• House To Implement ETL
• Key ETL Aspect

6. Motivation
---------------------------------------------------------------------------------------------------------------------
• Is ETL is Interesting area?
70 to 80% BI(DI or DW) projects is reliable ETL process
• Let’s have a look on the DW & DI market size
• In 2003, DI was USD 9.3 billion market
• In 2008, DI was USD 13 billion market
• By 2015, yearly grow estimated to USD 20 Billon
• The more systems in the world, the more work in Data Integration!

7. What is ETL?
• ETL = Extract – Transform – Load
• Extract
• Get the data from source system as efficiently as
possible
• Transform
• Perform calculations on data
• Load
• Load the data in the target storage

8. Why is ETL (System) Important?
 Adds value to data
Removes mistakes and corrects data
Documented measures of confidence in data
Captures the flow of transactional data
Adjusts data from multiple sources to be used together (conforming)
Structures data to be usable by BI tools
Enables subsequent business / analytical data procesing

9. ETL Disambiguation
•ETL = Extract – Transform – Load
›Not tight specifically to DW anymore
•Process/System
›A complete process including
•Data extraction
•Enforcing DQ and consistency standards
•Conforming data from disparate systems
•Delivering data to target
•People, HW, Documentation, Support, etc.
•Tool
›A piece of software implementing the
•three (four) E-(C)-T-L steps.
•A tool designed specifically to perform data transformations

10. ETLProcess

11. ETL Tool:true Datata Intigration

12. ETL Data Integration Solutions

14. Where is ETL used?

15. How to implement ETL system

16. How to implement ETL
•Scripting (shell, perl, python)
•PL/SQL, sqlldr
•Transformation hardcoded in Java, C#
•Develop (universal) ETL tool in-house
•Using off-the-shelf ETL tool

17. ETL tool Key Feture
Extract, Load => flexible on interfaces
›Flat files, DBMS, XML data, XLS,
›MQ, web services, LDAP
›Semi-structured data (emails, web logs, wiki pages)
›Unstructured data (blogs, documents)
›Extensibility with custom connectors
›Local data, remote data FTP(S), SFTP, SCP, http(s)
•Clean
›Lookups, Validations, Filters, Translations
•Transform
›Changing data structure, Joins, (De)Normalization, Aggregation, RollUp,
Sorting, Partitioning, Data De-duplication
›Ability to call external tools

18. •Performance
›Symmetric Multiprocessing (SMP)
•Pipeline processing
•Multithreaded processing
›Massively Parallel Processing (MPP)
•Clustering
•MapReduce
›Load balancing
•User friendliness
›GUI
›Metadata capture
›Training time
•Development
›Reusable components
›Impact Analysis / Data Lineage

19. •Manageability
›Team collaboration
›Transformation repository
›Metadata repository
›Development process (Dev -> Test -> Prod)
›Security
•Runtime
›Scheduler Automation
›Recovery and Restart
›Workflow
•Others
›Vendor stability
›Release cycle
›Support

21. Well Known ETL Tools
•Commercial
›Ab Initio
›IBM DataStage
›Informatica PowerCenter
›Microsoft Data Integration Services
›Oracle Data Integrator
›SAP Business Objects – Data Integrator
›SAS Data Integration Studio
•Open-source based
›Adeptia Integration Suite
›Apatar
›CloverETL
›Pentaho Data Integration (Kettle)
›Talend Open Studio/Integration Suite