Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ETL Market Webcast


Published on

Short overview of the components of the data integration market, some statistics and trends related to ETL, and how ETL products are evolving.

Published in: Business, Technology
  • The shares depend on how you want to measure and who does the measuring. You can measure by revenue, new license sales, installed licenses, etc. There are no accurate numbers out there for ETL products, just a lot of approximations. Some vendors like IBM don't break out the information in their reports so you don't know what the real share is and of what products. You have to average IDC and Forrester numbers today to get an approximation.

    You can count on Informatica and IBM having the biggest directly purchased share. Oracle and Microsoft as ETL for DW projects would be interesting. Nobody knows but they go out with almost every database install. Therefore you can't do revenue counting share metrics. All depends on what you want to use the information for.
    Are you sure you want to  Yes  No
    Your message goes here
  • i wolud like to knaow about the current ETL market share
    Are you sure you want to  Yes  No
    Your message goes here

ETL Market Webcast

  1. 1. Extract-Transform-Load (ETL) Market Overview and Directions TDWI Webcast Series June, 13 2007 Mark Madsen
  2. 2. Course Outline and Overview <ul><li>Components of the data integration market </li></ul><ul><li>Market data and trends </li></ul><ul><li>What’s happening now </li></ul><ul><li>What to expect </li></ul>
  3. 3. ETL: Extract, Transform and Load ETL Engine The good points Connectivity Transformation Read-write access Metadata The bad points Latency Distributed query limitations Complexity Batch processing focus Database Targets Databases Documents Flat Files XML Services ERP Applications Source Environments
  4. 4. EAI: Enterprise Application Integration <ul><li>Multiple models </li></ul><ul><li>Multiple implementations </li></ul><ul><li>Evolution toward open standards-based services and SOA </li></ul>Hub Point Bus
  5. 5. EDR: Enterprise Data Replication EDR Server Order Entry CRM Fulfillment Inventory EDR EDR
  6. 6. EII: Enterprise Information Integration EII Server Consuming Environments Databases Dashboards OLAP Productivity BAM/BPM Reporting ETL ERP Applications Databases Documents Flat Files XML Queues ERP Applications Source Environments SQL SOAP WS-* REST File Virtual Models
  7. 7. MDM <ul><li>Processes, systems and technologies for managing master reference data and ensuring consistency across the organization. </li></ul>
  8. 8. MDM: Master Data Management <ul><ul><li>Operational / infrastructure </li></ul></ul><ul><ul><ul><li>Distribution and/or synchronization of master reference data to ensure consistency in transactions and daily operations </li></ul></ul></ul><ul><ul><ul><li>More short-term latency and transactional issues </li></ul></ul></ul><ul><ul><li>Analytical / application </li></ul></ul><ul><ul><ul><li>Distribution or synchronization of master data to ensure consistent usage for BI purposes </li></ul></ul></ul><ul><ul><ul><li>More single-definition and long-term tracking issues </li></ul></ul></ul><ul><li>There are two basic product types with different motivations: </li></ul><ul><ul><li>Master data registries </li></ul></ul><ul><ul><li>MDM applications </li></ul></ul>
  9. 9. Data Profiling, Data Quality, Metadata <ul><li>Standalone data profiling, quality and metadata tools have been abandoned by vendors as an application market– it’s all merging into data integration products. </li></ul>
  10. 10. Market data and trends What’s driving the current market and what are the trends?
  11. 11. Data Integration Market Size and Growth Source: IDC
  12. 12. Spending Priorities in IT <ul><li>Great but… </li></ul>Sources: CIO Insight
  13. 13. Vendor Market Share Source: Forrester Research, Inc.
  14. 14. What is the Real Market Share? Sources: Forrester Research, Inc. and TDWI
  15. 15. Diversity of Data Sources Increasing <ul><li>Increased number and format of data sources, countering any gains made by ERP installations. </li></ul>Sources: TDWI, META Group, Inc.
  16. 16. Timeliness of Data Increasingly Important <ul><li>Increased data load frequency </li></ul><ul><li>Decreasing nightly load windows and more on-demand access </li></ul>Sources: TDWI, Gartner Percentage of Respondents
  17. 17. What’s happening now Consolidation, Extension, Coping
  18. 18. Commoditization of ETL Technology
  19. 19. Market Reflects Different Customer Types
  20. 20. Incremental Product Extension <ul><li>i.e. New features </li></ul><ul><ul><li>Text </li></ul></ul><ul><ul><li>Semi-structured data </li></ul></ul><ul><ul><li>Documents </li></ul></ul><ul><ul><li>Predictive analytics </li></ul></ul><ul><ul><li>Search </li></ul></ul>
  21. 21. Finding Other Uses for ETL
  22. 22. Other Uses for ETL One-time Extracts System Migrations System Consolidations Correction / Synchronization
  23. 23. In IT, Data Integration is Still Messy <ul><li>The history of IT has left us with both application silos and integration silos. </li></ul><ul><li>Current state of practice in IT is to integrate the integration software – it’s worse, not better. </li></ul>1960s 1970s 1980s 1990s 2000s
  24. 24. Integration Competency Centers <ul><li>Larger organizations are dealing with integration complexity by creating ICCs </li></ul><ul><ul><li>a centralized group to address integration across systems and projects, rather than dealing with integration project by project in an ad-hoc fashion. </li></ul></ul><ul><li>Split in the organization depending on where ICC starts from. </li></ul>
  25. 25. What we expect to happen
  26. 26. Shift in Data Integration Focus <ul><li>Features being built into products indicate a shift in focus from data and technical features to process and the data management lifecycle. </li></ul><ul><li>Traceability </li></ul><ul><li>Data quality </li></ul><ul><li>Data governance </li></ul><ul><li>Master data management </li></ul>
  27. 27. Multiple Integration Technologies in Suites DB API JDBC/ODBC Files Queues JMS SOAP/REST JSR 170 Databases Documents Flat Files XML Queues ERP Legacy Apps Data Quality Data Profiling Metadata Services EDR EII Adapters / Connectors ETL
  28. 28. ETL Product Evolution <ul><li>ETL tools have been growing into suites which are slowly evolving into data integration platforms. </li></ul><ul><li>The current state of the art in the ETL market is suites with integrated ETL, metadata, data quality and profiling. </li></ul><ul><li>The tools are including new sources: mining output, federation, services, EAI, semi-structured data </li></ul>batch files ftp database EAI ETL SOA
  29. 29. ETL Vendor Positioning and Strategy <ul><li>ETL vendor strategies in the data integration market have been shifting. </li></ul><ul><ul><li>Horizontal – expand to fill all the different types of integration needs, staying within the information management layers </li></ul></ul><ul><ul><li>Vertical – leverage strengths to expand up and down into other layers </li></ul></ul><ul><ul><li>Niche – focus on specific technical or vertical market needs for a single technology </li></ul></ul>
  30. 30. Warehouse Architecture: Traditional View SQL Warehouse Database ETL ODS Mart Databases Documents Flat Files XML Queues ERP Applications Source Environments Data Warehouse Clients Dashboards OLAP Productivity BAM/BPM Reporting DM Data Mining
  31. 31. Warehouse Arch Going Forward Databases Documents Flat Files XML Queues ERP Applications Source Environments Data Consumers Databases Dashboards OLAP Productivity BAM/BPM Reporting ETL Data Mining Applications SQL Warehouse Database ETL ? ? Mart ODS EDR EII Content Store
  32. 32. Expect that… <ul><li>The big vendors continue to get bigger </li></ul><ul><li>Commoditization continues and forces price disruption and more consolidation. </li></ul><ul><li>There will still be new entrants, particularly in low-cost or specialty areas like specific apps or dealing with streaming data </li></ul><ul><li>Performance will continue to be a concern, but there will be many more options to deal with it. </li></ul><ul><li>Suites will accrete more stuff, and the split between focused and platform/stack will broaden. </li></ul><ul><li>Some things won’t catch on like we think. </li></ul>
  33. 33. Creative Commons <ul><li>This work is licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. To view a copy of this license, visit or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA. </li></ul>
  34. 34. Creative Commons Image Attributions <ul><li>The following CC licensed images were used in this presentation: </li></ul><ul><li>Shopping carts: </li></ul>