Your SlideShare is downloading. ×
Introduction to ETL and Data Integration
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Introduction to ETL and Data Integration

29,185

Published on

This presenation explains basics of ETL (Extract-Transform-Load) concept in relation to such data solutions as data warehousing, data migration, or data integration. CloverETL is presented closely as …

This presenation explains basics of ETL (Extract-Transform-Load) concept in relation to such data solutions as data warehousing, data migration, or data integration. CloverETL is presented closely as an example of enterprise ETL tool. It also covers typical phases of data integration projects.

Published in: Technology, Business
4 Comments
32 Likes
Statistics
Notes
No Downloads
Views
Total Views
29,185
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
1,516
Comments
4
Likes
32
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. ETL March 2011 Mgr. Jan UlrychAll rights reserved Javlin 2011
  • 2. Organizational Matters • Introduction to ETL • More about ETL • CloverETL intro • ETL Projects • Current TrendsAll rights reserved Javlin 2011
  • 3. Presenter • Jan Ulrych › Graduated from Faculty of Mathematics and Physics at Charles University, Prague › Works for Javlin a.s. as ETL Consultant since 2008 › E-mail: jan.ulrych@javlin.eu • Professional experience › DVRA project at DHL IT Services Europe, Prague › ETL Consultant on various data integration projects › Since 2010 is a Pre-sales consultant for CloverETL™All rights reserved Javlin 2011
  • 4. Javlin Overview • Javlin – since 2005 • Javlin is a software developer and services provider › CloverETL platform › Javlin services and ETL consulting › Software development for major clients • Employees: staff of 60+ › Developers and consultants › Service, sales, support › Executive management • Experienced management team › Data and ETL software development legacy › Key industry expertise – finance, health, media, logistics, government • Office locations › Prague › Brno › Greater Washington DCAll rights reserved Javlin 2011
  • 5. Selected CustomersAll rights reserved Javlin 2011
  • 6. Session 1: Origins & MotivationAll rights reserved Javlin 2011
  • 7. Data Warehousing • “A data warehouse is a system that extracts, cleans, conforms, and delivers source data into a dimensional data store and then supports and implements querying and analysis for the purpose of decision making.” Source: Ralph Kimball, Joe Caserta: The Data Warehouse ETL Toolkit; Wiley 2004 • The most visible part is “querying and analysis” • The most complex and time consuming part is “extracts, cleans, conforms, and delivers”All rights reserved Javlin 2011
  • 8. Data Warehousing • The most complex and time consuming part is “extracts, cleans, conforms, and delivers” • How complex is it? › 70-80% of BI (DI or DW) project is reliable ETL processAll rights reserved Javlin 2011
  • 9. Getting data into DW • How to load data into DW? › Scripts in linux shell, perl, python, … › sqlldr + SQL › Hardcoded in Java, C#, C › In-house built ETL tool › Off-the shelf ETL tool • Aspects to be kept in mind › Manageability › Maintainability › Transparency › Scalability › Flexibility › Complexity › Auditing › Job restartability › TestingAll rights reserved Javlin 2011
  • 10. Session 1: Introduction to ETLAll rights reserved Javlin 2011
  • 11. ETL • How to load data into DW in a right way? › Introduce formal ETL process • This section covers › What ETL is › Motivation › Where to use ETL › How to Implement ETL › Key ETL AspectsAll rights reserved Javlin 2011
  • 12. Motivation • Is ETL interesting area? › 70-80% of BI (DI or DW) project is reliable ETL process. • Let’s have a look on the DW & DI market size › In 2003, DI was USD 9.3 billion market › In 2008, DI was USD 13 billion market › By 2010, yearly grow estimated to USD 2.2 billion › TCO of DI can reach USD 509,600 annually • The more systems in the world, the more work in Data Integration!All rights reserved Javlin 2011
  • 13. What is ETL? • ETL = Extract – Transform – Load • Extract › Get the data from source system as efficiently as possible • Transform › Perform calculations on data • Load › Load the data in the target storageAll rights reserved Javlin 2011
  • 14. What is ETL? • ETL = Extract – Transform – Load • Extract › Get the data from source system as efficiently as possible • Clean › Perform data cleansing and dimension conforming • Transform › Perform calculations on data • Load › Load the data in the target storageAll rights reserved Javlin 2011
  • 15. Why is ETL (System) Important? • Adds value to data › Removes mistakes and corrects data › Documented measures of confidence in data › Captures the flow of transactional data › Adjusts data from multiple sources to be used together (conforming) › Structures data to be usable by BI tools › Enables subsequent business / analytical data procesingAll rights reserved Javlin 2011
  • 16. ETL Disambiguation • ETL = Extract – Transform – Load › Not tight specifically to DW anymore • Process/System › A complete process including • Data extraction • Enforcing DQ and consistency standards • Conforming data from disparate systems • Delivering data to target • People, HW, Documentation, Support, etc. • Tool › A piece of software implementing the • three (four) E-(C)-T-L steps. • A tool designed specifically to perform data transformationsAll rights reserved Javlin 2011
  • 17. ETL Process Presentation Dashboards, Reports, Portals Analytics BI Tools (SAP, Cognos), KPI, Data Mining Integrated data value for applications ETL Data Integration ETL Extracting value Data from the database DBMS (MS SQL, MySQL, Oracle), XML, flat files, CSV, mainframeAll rights reserved Javlin 2011
  • 18. ETL Tool: True Data Integration Source A ETL Source B Files, Databases, Read Files, Databases, Message Queues, Apply Logic Message Queues, Web Services Write Web Services True Data Integration is agnostic of source or target application ETL is a bridge for bi-directional flowAll rights reserved Javlin 2011
  • 19. ETL Data Integration Solutions (1) Data Migration Process of transferring data between storage types or formats. An automated migration frees up human resources from tedious tasks. Design, extraction, cleansing, load and verification are done for moderate to high complexity jobs. Data Consolidation Usually associated with moving data from remote locations to a central location or combining data due to an acquisition or merger. Data Integration Process of combining data residing at different sources and providing a unified view. Emerges in both commercial and scientific fields and is focus of extensive theoretical work. Also referred to as Enterprise Information Integration.All rights reserved Javlin 2011
  • 20. ETL Data Integration Solutions (2) Master Data Management Processes and tools to define and manage non-transactional data. Provides for collecting, aggregating, matching, consolidating, quality-assuring, persisting and distributing data to an organization to ensure consistency and control. Data Warehouse Repository of electronically stored data. ETL facilitates populating, reporting and analysis. Includes business intelligence as well as metadata retrieval and management tools. Data Synchronization Process of making sure two or more locations contain the same up-to-date files. Add, change, or delete a file from one location, synchronization will mirror the action at the new location.All rights reserved Javlin 2011
  • 21. Where is ETL used?All rights reserved Javlin 2011
  • 22. How to Implement ETL System (1) Source: Ralph Kimball, Joe Caserta: The Data Warehouse ETL Toolkit; Wiley 2004All rights reserved Javlin 2011
  • 23. How to Implement ETL System (2) • Scripting (shell, perl, python) • PL/SQL, sqlldr • Transformation hardcoded in Java, C# • Develop (universal) ETL tool in-house • Using off-the-shelf ETL toolAll rights reserved Javlin 2011
  • 24. ETL Tool Key Features (1) • Extract, Load => flexible on interfaces › Flat files, DBMS, XML data, XLS, › MQ, web services, LDAP › Semi-structured data (emails, web logs, wiki pages) › Unstructured data (blogs, documents) › Extensibility with custom connectors › Local data, remote data FTP(S), SFTP, SCP, http(s) • Clean › Lookups, Validations, Filters, Translations • Transform › Changing data structure, Joins, (De)Normalization, Aggregation, RollUp, Sorting, Partitioning, Data De-duplication › Ability to call external toolsAll rights reserved Javlin 2011
  • 25. ETL Tool Key Features (2) • Performance › Symmetric Multiprocessing (SMP) • Pipeline processing • Multithreaded processing › Massively Parallel Processing (MPP) • Clustering • MapReduce › Load balancing • User friendliness › GUI › Metadata capture › Training time • Development › Reusable components › Impact Analysis / Data Lineage › DocumentationAll rights reserved Javlin 2011
  • 26. ETL Tool Key Features (3) • Manageability › Team collaboration › Transformation repository › Metadata repository › Development process (Dev -> Test -> Prod) › Security • Runtime › Scheduler Automation › Recovery and Restart › Workflow • Others › Vendor stability › Release cycle › SupportAll rights reserved Javlin 2011
  • 27. ETL Market Source: Ted Friedman, Mark A. Beyer, Eric Thoo: Magic Quadrant for Data Integration Tools; Gartner RAS Core Research Note G00207435; 19 November 2010All rights reserved Javlin 2011
  • 28. Well Known ETL Tools • Commercial › Ab Initio › IBM DataStage › Informatica PowerCenter › Microsoft Data Integration Services › Oracle Data Integrator › SAP Business Objects – Data Integrator › SAS Data Integration Studio • Open-source based › Adeptia Integration Suite › Apatar › CloverETL › Pentaho Data Integration (Kettle) › Talend Open Studio/Integration Suite The list above is not meant to be comprehensiveAll rights reserved Javlin 2011
  • 29. ETL or ELT? • ETL = Extract – Transform – Load › Much more flexible • ELT = Extract – Load – Transform › Pushed forward (mainly) by Oracle • First get the data into database • Then use Oracle DB tools to work with it › Less flexible • Tightly coupled to vendor’s database/solution • Less flexible on output formats • Requires staging area › Possibly better performance • All data are processed in the same database • Nothing is downloaded from database for Transform stepAll rights reserved Javlin 2011
  • 30. Session 2: Dominate Your Data with CloverETL™All rights reserved Javlin 2011
  • 31. What is CloverETL Clover works on all OS • A data integration software platform › Linux › Manages, designs and runs your data › Windows › Embeddable and scalable › HP-UX › Integrates easily with databases, operating › AIX systems and applications › IBM AS/400 › Solaris › Mac OS X • ETL platform that dominates your data Extract – Transform – Load › Reads from one or more data sources › Transforms data in almost any way imaginable › Writes to any number of data targets • Legacy of open source and commitment to commercial use • CloverETL Engine can be used as embedded OEM as well www.cloveretl.comAll rights reserved Javlin 2011
  • 32. CloverETL Key Features Platform independent Java, integration, library support Cost Scalability delivers lowest total Desktop → Enterprise cost of ownership → Cluster OEM Embeddable Usability small footprint, built by ETL experts extensible and for ETL experts customizable Services Performance Clover was built to outstanding at all require minimal production levels servicesAll rights reserved Javlin 2011
  • 33. CloverETL Product Suite CloverETL Server Design Manage • Production Platform • Web app’s (Tomcat, WebSphere, GlassFish) • For Enterprise Integration CloverETL Designer • Visual Designer • Transformation Developer • Eclipse Platform CloverETL Engine • Pure Java 6.0 • Embeddable • Extensible Runtime • Ideal for OEMAll rights reserved Javlin 2011
  • 34. CloverETL Designer Features • Transformation design • Intuitive GUI • Drag & drop • Components Library • DebugAll rights reserved Javlin 2011
  • 35. CloverETL Vision Vision CloverETL enables companies to get more value from their data quickly without massive infrastructure expense or years of project investment. Dominate your data Approach CloverETL is the best value for money. It builds off the open source foundation of the CloverETL Engine and scales from desktop to enterprise to cluster. Get it done quickly Investment CloverETL is an easy to use ETL software that can grow by both core features and user expertise… at a fraction of the cost of larger system vendors. Low cost buy-in for our customersAll rights reserved Javlin 2011
  • 36. CloverETL Approach • Components › Prebuilt algorithms Input • Graph Transform › Processing algorithm in visual form • Data Flow › Edges between components • Process › Build > Connect > Configure › Data processed as structured records with named fields › Components operate on Output record fieldsAll rights reserved Javlin 2011
  • 37. Transformation capabilities • 50+ specialized components available for use › Readers and writers • Text or binary files, CSV, XML • Archives including ZIP or GZIP • Remote transfers over HTTP/FTP(S) • Access to messaging via JMS › Database connectivity • JDBC connectivity, support for bulk loaders • Supports any SQL statements, stored procedure calls › Transformers and aggregators • Variety of data manipulation algorithms: sorting, deduping, joining, arithmetics, aggregating, statistic functions and more • Customizable with user-defined code • Alternative implementations for efficient executionAll rights reserved Javlin 2011
  • 38. Physical architecture • Others: Server-centric with thin clients › Server is necessary for development and execution › Transformations and data are stored remotely on server › Limited use when working on multiple sites or in restricted-access networks • CloverETL: Designer is a standalone application › Development and execution is possible without central server › Transformations and metadata are stored on local machine › Transformations can be also deployed to server for centralized management › Designer available for all major platforms (Linux, Windows, Mac)All rights reserved Javlin 2011
  • 39. Repository • Others: Central repository › With proprietary storage format (binary files, tables) › Often managed with a proprietary version control system › Can be problematic with mass changes or when “hacking” is necessary • CloverETL: XML format files and directories › No proprietary repository, uses plain files and directories › Transformation and metadata stored in human-readable XML › Open to any version management systems • CVS, SVN, IBM Rational ClearCase, git › Integrated with Eclipse VMS clients • Subclipse, EGit, ClearCaseAll rights reserved Javlin 2011
  • 40. Flexibility and expressive power • Others: Expression-based languages only › Expressions and built-in functions for simple data manipulation › No support for programming statements (loops, user functions) › Limited when data manipulation requires complex coding • CloverETL: Scripting and more › Components are customizable with CTL or Java code › CTL: scripting language with simple syntax • Allows simple expressions as well as complex code • Has variables, loops, user-defined functions • Many built-in data validation functions › Java: mature programming language, allow coding • Access to variety of existing libraries • GC-based memory management = rapid developmentAll rights reserved Javlin 2011
  • 41. Extensibility • Others: Custom components › Limited support for developing custom components › Impossible to extend other aspects of data processing • CloverETL: Plugin-based extensible platform › Ready to extend, customize and modify › Plugin-based architecture for easy extension management › Supports custom components, connections, functions › Implemented in Java • Access to libraries • Memory management • DevelopersAll rights reserved Javlin 2011
  • 42. Session 2: CloverETL™ Designer ExamplesAll rights reserved Javlin 2011
  • 43. CloverETL Server Features • Automation and scheduling • File and event triggers • Workflows • Monitoring and logging • User management • Real-time ETL • Clustering • Load balancing • Failover • Distributed processing • Inexpensive buy-inAll rights reserved Javlin 2011
  • 44. Key features • Runtime automation › Allows integration with existing infrastructure › Simplified management and execution of data transformations • Scalability and optimization › Increases transformation performance › Shortens response time in continuous/transactional processing • Security › Controls access to data, transformations and server configuration › Secures communication between server and clients • Clustering › Allows cooperation between multiple processing nodes › Improves scalability, performance and error resiliencyAll rights reserved Javlin 2011
  • 45. Runtime automation • Internal scheduler › Supports one-time or periodic execution › Allows interval-based scheduling, including flexible cron-like rules • Integration with enterprise schedulers › Transformation execution can be started by external scheduler › cron, IBM Tivoli (Maestro), Autosys, UC4 › Scheduler instructs CloverETL Server via one of its interfaces (HTTP,JMX) • Events, tasks and dependencies › Tasks and transformations are started on internal or external events › Internal: transformation finished or failed › External: file arrived or its size changed › Allows creating dependencies between executed tasks › Suitable for logical sequencing of execution or monitoring purposesAll rights reserved Javlin 2011
  • 46. Runtime automation (cont.) • Monitoring › Support alert emails or messages with configurable content › Automatically populated with execution status, log, statistics › Allow integration with ticketing systems, support team • Execution history › Automatically stores performance and statistics about each execution › Stored in database tables and log files › Open for any trend analysis and reporting • Archiving › Configurable cleanup of execution logs and history › Can be further extended by scriptingAll rights reserved Javlin 2011
  • 47. Scalability and optimization • Parallel execution › Executes multiple transformations in parallel › Can execute multiple instances of single transformations (SOA) • Graph pooling › Improves response time › Useful with SOA-architectures and Launch Services • Launch services › Applications implemented as data transformations and deployed as RESTful web services › Schema on next slideAll rights reserved Javlin 2011
  • 48. CloverETL Cluster  Increased processing and throughput - Parallel execution of transformations over multiple servers or nodes - Load balancing based on individual node utilization  Increased fault tolerance - Fail over in case of problems with particular nodes - Data replication  Increased flexibility - Cluster can be dynamically reconfigured by adding or removing nodes Large quantity of data loaded Processed in parallel in Cluster Written out in parallelAll rights reserved Javlin 2011
  • 49. Session 3: Data Integration ProjectsAll rights reserved Javlin 2011
  • 50. Data Integration Projects • This section covers › How to manage DI projects › Phases of DI project › Responsibilities › Typical issuesAll rights reserved Javlin 2011
  • 51. Data Integration Projects • Phases of typical Data Integration Project › Requirements › Planning › Analysis (of ETL steps) › Implementation › Documentation › RTP › SupportAll rights reserved Javlin 2011
  • 52. Requirements • Functional › Input data › Output data › Output data format › Transformation logic • Non-functional › Time restrictions › Availability › Frequency of update › Data Latency › How to handle erroneous records › Security requirementsAll rights reserved Javlin 2011
  • 53. Planning • ETL system implementation to be planned properly › Time for implementation › Correctly prioritized › Thorough data analysis extremely important › Unforeseen data quality issues cause delays › Biggest risk is unexpected data quality issues › Communicate properly • Keep it simple › Do no try to save the world › “If you think it can be done simply, do it simply”All rights reserved Javlin 2011
  • 54. Analysis – Extract • In which source systems is the data we need? • How can we access the system? › Flat files / database access / XML / web service / JMS › Full extract / incremental / change notification / on-request › Local access / ftp(s) / sftp / scp / MQ • Data Syntax › What is a record? › What is record delimiter? › What is field delimiter? › What is the data length / data type / format? • Data Semantics › What are the field names? › What data does the field represent? › Ok, what data does the field really represent? › Are there any duplicates? • What are the limitations / restrictions? › What are the data volumes? › How often can the export be done? › Any impact on network (NIA)? • What is the expected data growth rate?All rights reserved Javlin 2011
  • 55. Analysis – Clean & Transform • Which fields need to be validated? › … against which source? › How to handle erroneous data? • What is the data flow? › Between source and target data › What is the transformation logic? › Action on error? • Stop transformation • Process valid data only • Transformation restartability › Small data volumes – transaction based › Huge data volumes – process in bulk modeAll rights reserved Javlin 2011
  • 56. Analysis – Load • What is the target schema? • What is the target data volume? • What are the history requirements › Usually SCD type 1, 2 or 3 • Data Syntax › What is a record? › What is record delimiter? › What is field delimiter? › What is the data length / data type / format? • Data Semantics › What are the field names? › What data does the field represent? › Ok, what data does the field really represent? • What are the limitations / restrictions? › What are the data volumes? › How often can the export be done? › Any impact on network (NIA)?All rights reserved Javlin 2011
  • 57. Implementation • Development › Enforce standardization • Naming conventions • Best practices • Generating surrogate keys • Looking up keys • Applying default values • Testing › Review › Testing on Production data › Unit TestingAll rights reserved Javlin 2011
  • 58. Implementation • Documentation › Data sources / targets / transformations › Data Lineage • Important to know and publish › Frequency of ETL processes runs › Error handling › Support – Monitoring checklist • RTPAll rights reserved Javlin 2011
  • 59. Planning & Leadership • Contact person for each source system • Contact business person • ETL team responsibilities › Define ETL scope › Perform source system data analysis › Define data quality strategy › Gather & document business rules from business users › ETL Implementation › Defining & executing Unit & QA testing › Implementing productionAll rights reserved Javlin 2011
  • 60. ETL team Roles & Responsibilities • ETL Manager • ETL Architect • ETL Developer • System Analyst • Data-Quality Specialist • Database Administrator (DBA) • Dimension Manager • Fact Table ProviderAll rights reserved Javlin 2011
  • 61. Typical Issues • Project Management • Poor Data Analysis • Data Understanding • Performance • ScalabilityAll rights reserved Javlin 2011
  • 62. Typical Issues – Project Management • ETL takes 70-80% project resources › Be aware of this from beginning › Communicate this to stakeholders › Plan ETL phase properly • Data Quality › Unexpected data quality issues is biggest risk › DQ issues cause delays › DQ issues generate extra workAll rights reserved Javlin 2011
  • 63. Typical Issues – Data Understanding • Source system (data) › Not documented › Documented incorrectly › Represent something else than they should › Data not clean • Transformation/Requirements › Not specified properly › Not specified at all › Initial analysis has not revealed issues/complexity › Requirements being changedAll rights reserved Javlin 2011
  • 64. Typical Issues – Performance & Scalability • Performance › Is the performance ok now? › Will performance be ok in 5 years? • Scalability › What is the data growth rate? › Are we testing on production data volumes? • Change Data Capture › Time consuming task › Issue on old systemsAll rights reserved Javlin 2011
  • 65. Session 4: Current TrendsAll rights reserved Javlin 2011
  • 66. Market Trends • Shift to Semi-structured & Unstructured data › Emails, documents, blogs, … • Real-time processing › CRM, Zero-latency business › SOA, Web Services, ESB, JMS, MQ • Cloud-mania › Cloud, Cluster, Elastic Cluster › MapReduce, Apache Hadoop • Reducing TCO › License costs, Development cost, Maintenance › Emphasis on value; not price • Services for small customers › Require better ROIAll rights reserved Javlin 2011
  • 67. Literature Ralph Kimball, Joe Caserta: The Data Warehouse ETL Toolkit; Wiley Ralph Kimball, Margy Ross: The Data Warehouse Toolkit; Wiley Len Silverston: The Data Model Resource Book; WileyAll rights reserved Javlin 2011
  • 68. Contact Javlin Web www.cloveretl.com US Europe Javlin Inc. Javlin a.s. 8000 Towers Crescent Drive Křemencova 18 Suite 1350 110 00 Praha 1 Vienna, VA 22182 Czech Republic USA Web: www.javlininc.com Web: www.javlin.eu Email: info@javlininc.com E-mail: info@javlin.eu Phone: +1 703 847 3600 Phone: +420 277 003 200All rights reserved Javlin 2011

×