Extract-Transform-Load (ETL) Market Overview and Directions TDWI Webcast Series June, 13 2007 Mark Madsen http://ThirdNatu...
Course Outline and Overview <ul><li>Components of the data integration market </li></ul><ul><li>Market data and trends </l...
ETL: Extract, Transform and Load ETL   Engine The good points Connectivity Transformation Read-write access Metadata The b...
EAI: Enterprise Application Integration <ul><li>Multiple models </li></ul><ul><li>Multiple implementations </li></ul><ul><...
EDR: Enterprise Data Replication EDR Server Order Entry CRM Fulfillment Inventory EDR EDR
EII: Enterprise Information Integration EII Server Consuming Environments Databases  Dashboards  OLAP  Productivity  BAM/B...
MDM <ul><li>Processes, systems and technologies for managing master reference data and ensuring consistency across the org...
MDM: Master Data Management <ul><ul><li>Operational / infrastructure </li></ul></ul><ul><ul><ul><li>Distribution and/or sy...
Data Profiling, Data Quality, Metadata <ul><li>Standalone data profiling, quality and metadata tools have been abandoned b...
Market data and trends What’s driving the current market and what are the trends?
Data Integration Market Size and Growth Source: IDC
Spending Priorities in IT <ul><li>Great but… </li></ul>Sources: CIO Insight
Vendor Market Share Source: Forrester Research, Inc.
What is the Real Market Share? Sources: Forrester Research, Inc. and TDWI
Diversity of Data Sources Increasing <ul><li>Increased number and format of data sources, countering any gains made by ERP...
Timeliness of Data Increasingly Important <ul><li>Increased data load frequency </li></ul><ul><li>Decreasing nightly load ...
What’s happening now Consolidation, Extension, Coping
Commoditization of ETL Technology
Market Reflects Different Customer Types
Incremental Product Extension <ul><li>i.e. New features </li></ul><ul><ul><li>Text </li></ul></ul><ul><ul><li>Semi-structu...
Finding Other Uses for ETL
Other Uses for ETL One-time Extracts System Migrations System Consolidations Correction / Synchronization
In IT, Data Integration is Still Messy  <ul><li>The history of IT has left us with both application silos  and  integratio...
Integration Competency Centers <ul><li>Larger organizations are dealing with integration complexity by creating ICCs </li>...
What we expect to happen
Shift in Data Integration Focus <ul><li>Features being built into products indicate a shift in focus from data and technic...
Multiple Integration Technologies in Suites DB API JDBC/ODBC Files Queues JMS SOAP/REST JSR 170 Databases  Documents  Flat...
ETL Product Evolution <ul><li>ETL tools have been growing into suites which are slowly evolving into data integration plat...
ETL Vendor Positioning and Strategy <ul><li>ETL vendor strategies in the data integration market have been shifting. </li>...
Warehouse Architecture: Traditional View SQL Warehouse Database ETL ODS Mart Databases  Documents  Flat Files  XML  Queues...
Warehouse Arch Going Forward Databases  Documents  Flat Files  XML  Queues  ERP  Applications Source Environments Data Con...
Expect that… <ul><li>The big vendors continue to get bigger </li></ul><ul><li>Commoditization continues and forces price d...
Creative Commons <ul><li>This work is licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 United...
Creative Commons Image Attributions <ul><li>The following CC licensed images were used in this presentation: </li></ul><ul...
Upcoming SlideShare
Loading in...5
×

ETL Market Webcast

9,831

Published on

Short overview of the components of the data integration market, some statistics and trends related to ETL, and how ETL products are evolving.

Published in: Business, Technology
2 Comments
15 Likes
Statistics
Notes
  • The shares depend on how you want to measure and who does the measuring. You can measure by revenue, new license sales, installed licenses, etc. There are no accurate numbers out there for ETL products, just a lot of approximations. Some vendors like IBM don't break out the information in their reports so you don't know what the real share is and of what products. You have to average IDC and Forrester numbers today to get an approximation.

    You can count on Informatica and IBM having the biggest directly purchased share. Oracle and Microsoft as ETL for DW projects would be interesting. Nobody knows but they go out with almost every database install. Therefore you can't do revenue counting share metrics. All depends on what you want to use the information for.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • i wolud like to knaow about the current ETL market share
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
9,831
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
0
Comments
2
Likes
15
Embeds 0
No embeds

No notes for slide
  • ETL Market Webcast

    1. 1. Extract-Transform-Load (ETL) Market Overview and Directions TDWI Webcast Series June, 13 2007 Mark Madsen http://ThirdNature.net
    2. 2. Course Outline and Overview <ul><li>Components of the data integration market </li></ul><ul><li>Market data and trends </li></ul><ul><li>What’s happening now </li></ul><ul><li>What to expect </li></ul>
    3. 3. ETL: Extract, Transform and Load ETL Engine The good points Connectivity Transformation Read-write access Metadata The bad points Latency Distributed query limitations Complexity Batch processing focus Database Targets Databases Documents Flat Files XML Services ERP Applications Source Environments
    4. 4. EAI: Enterprise Application Integration <ul><li>Multiple models </li></ul><ul><li>Multiple implementations </li></ul><ul><li>Evolution toward open standards-based services and SOA </li></ul>Hub Point Bus
    5. 5. EDR: Enterprise Data Replication EDR Server Order Entry CRM Fulfillment Inventory EDR EDR
    6. 6. EII: Enterprise Information Integration EII Server Consuming Environments Databases Dashboards OLAP Productivity BAM/BPM Reporting ETL ERP Applications Databases Documents Flat Files XML Queues ERP Applications Source Environments SQL SOAP WS-* REST File Virtual Models
    7. 7. MDM <ul><li>Processes, systems and technologies for managing master reference data and ensuring consistency across the organization. </li></ul>
    8. 8. MDM: Master Data Management <ul><ul><li>Operational / infrastructure </li></ul></ul><ul><ul><ul><li>Distribution and/or synchronization of master reference data to ensure consistency in transactions and daily operations </li></ul></ul></ul><ul><ul><ul><li>More short-term latency and transactional issues </li></ul></ul></ul><ul><ul><li>Analytical / application </li></ul></ul><ul><ul><ul><li>Distribution or synchronization of master data to ensure consistent usage for BI purposes </li></ul></ul></ul><ul><ul><ul><li>More single-definition and long-term tracking issues </li></ul></ul></ul><ul><li>There are two basic product types with different motivations: </li></ul><ul><ul><li>Master data registries </li></ul></ul><ul><ul><li>MDM applications </li></ul></ul>
    9. 9. Data Profiling, Data Quality, Metadata <ul><li>Standalone data profiling, quality and metadata tools have been abandoned by vendors as an application market– it’s all merging into data integration products. </li></ul>
    10. 10. Market data and trends What’s driving the current market and what are the trends?
    11. 11. Data Integration Market Size and Growth Source: IDC
    12. 12. Spending Priorities in IT <ul><li>Great but… </li></ul>Sources: CIO Insight
    13. 13. Vendor Market Share Source: Forrester Research, Inc.
    14. 14. What is the Real Market Share? Sources: Forrester Research, Inc. and TDWI
    15. 15. Diversity of Data Sources Increasing <ul><li>Increased number and format of data sources, countering any gains made by ERP installations. </li></ul>Sources: TDWI, META Group, Inc.
    16. 16. Timeliness of Data Increasingly Important <ul><li>Increased data load frequency </li></ul><ul><li>Decreasing nightly load windows and more on-demand access </li></ul>Sources: TDWI, Gartner Percentage of Respondents
    17. 17. What’s happening now Consolidation, Extension, Coping
    18. 18. Commoditization of ETL Technology
    19. 19. Market Reflects Different Customer Types
    20. 20. Incremental Product Extension <ul><li>i.e. New features </li></ul><ul><ul><li>Text </li></ul></ul><ul><ul><li>Semi-structured data </li></ul></ul><ul><ul><li>Documents </li></ul></ul><ul><ul><li>Predictive analytics </li></ul></ul><ul><ul><li>Search </li></ul></ul>
    21. 21. Finding Other Uses for ETL
    22. 22. Other Uses for ETL One-time Extracts System Migrations System Consolidations Correction / Synchronization
    23. 23. In IT, Data Integration is Still Messy <ul><li>The history of IT has left us with both application silos and integration silos. </li></ul><ul><li>Current state of practice in IT is to integrate the integration software – it’s worse, not better. </li></ul>1960s 1970s 1980s 1990s 2000s
    24. 24. Integration Competency Centers <ul><li>Larger organizations are dealing with integration complexity by creating ICCs </li></ul><ul><ul><li>a centralized group to address integration across systems and projects, rather than dealing with integration project by project in an ad-hoc fashion. </li></ul></ul><ul><li>Split in the organization depending on where ICC starts from. </li></ul>
    25. 25. What we expect to happen
    26. 26. Shift in Data Integration Focus <ul><li>Features being built into products indicate a shift in focus from data and technical features to process and the data management lifecycle. </li></ul><ul><li>Traceability </li></ul><ul><li>Data quality </li></ul><ul><li>Data governance </li></ul><ul><li>Master data management </li></ul>
    27. 27. Multiple Integration Technologies in Suites DB API JDBC/ODBC Files Queues JMS SOAP/REST JSR 170 Databases Documents Flat Files XML Queues ERP Legacy Apps Data Quality Data Profiling Metadata Services EDR EII Adapters / Connectors ETL
    28. 28. ETL Product Evolution <ul><li>ETL tools have been growing into suites which are slowly evolving into data integration platforms. </li></ul><ul><li>The current state of the art in the ETL market is suites with integrated ETL, metadata, data quality and profiling. </li></ul><ul><li>The tools are including new sources: mining output, federation, services, EAI, semi-structured data </li></ul>batch files ftp database EAI ETL SOA
    29. 29. ETL Vendor Positioning and Strategy <ul><li>ETL vendor strategies in the data integration market have been shifting. </li></ul><ul><ul><li>Horizontal – expand to fill all the different types of integration needs, staying within the information management layers </li></ul></ul><ul><ul><li>Vertical – leverage strengths to expand up and down into other layers </li></ul></ul><ul><ul><li>Niche – focus on specific technical or vertical market needs for a single technology </li></ul></ul>
    30. 30. Warehouse Architecture: Traditional View SQL Warehouse Database ETL ODS Mart Databases Documents Flat Files XML Queues ERP Applications Source Environments Data Warehouse Clients Dashboards OLAP Productivity BAM/BPM Reporting DM Data Mining
    31. 31. Warehouse Arch Going Forward Databases Documents Flat Files XML Queues ERP Applications Source Environments Data Consumers Databases Dashboards OLAP Productivity BAM/BPM Reporting ETL Data Mining Applications SQL Warehouse Database ETL ? ? Mart ODS EDR EII Content Store
    32. 32. Expect that… <ul><li>The big vendors continue to get bigger </li></ul><ul><li>Commoditization continues and forces price disruption and more consolidation. </li></ul><ul><li>There will still be new entrants, particularly in low-cost or specialty areas like specific apps or dealing with streaming data </li></ul><ul><li>Performance will continue to be a concern, but there will be many more options to deal with it. </li></ul><ul><li>Suites will accrete more stuff, and the split between focused and platform/stack will broaden. </li></ul><ul><li>Some things won’t catch on like we think. </li></ul>
    33. 33. Creative Commons <ul><li>This work is licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/us/ or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA. </li></ul>
    34. 34. Creative Commons Image Attributions <ul><li>The following CC licensed images were used in this presentation: </li></ul><ul><li>Shopping carts: http://flickr.com/photo_zoom.gne?id=238070241&size=o </li></ul>

    ×