Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ugif 04 2011 france ug04042011-jroy_part1


Published on

  • Be the first to comment

  • Be the first to like this

Ugif 04 2011 france ug04042011-jroy_part1

  1. 1. Informix Warehouse AcceleratorJacques RoyIBM, Informix developmentApril 6, 2011 © 2010 IBM Corporation
  2. 2. Agenda■ Data warehouse industry trends■■ Data warehouse on Informix■■ Infomrix warehouse accelerator■ 2 © 2010 IBM Corporation
  3. 3. Sate of Data Warehousing 2011DBMS Market in 2011:■ DBMS market at the close of 2009 was approximately $21.2 billion (2010 data not yet available)■ Data Warehouse DBMS market was approximately 35% of the DBMS market or $7.42 billionKey Findings:■ Data warehouse DBMSs have evolved to a broader analytics infrastructure supporting operational analytics, corporate performance management and other new applications and uses.■ Cost is driving interest in alternative architectures but performance optimization is driving multi-tiered data architectures and a variety of deployment options - notably a strong interest in in-memory data mart deployments. 3 © 2010 IBM Corporation
  4. 4. Sate of Data Warehousing (cont.) Market Dynamics for 2011■ Today, smaller data warehouses, those less than 5 TBs of source system extracted data (SSED) are the only "data warehouse" for the entire organization and are commonly solving organizations analytic needs. Analysis:■ Gartner only rarely encounters an organization which has actually delivered on the Enterprise Data Warehouse (EDW) vision. The EDW remains a design principle, but it is rarely if ever actually deployed. Gartner estimates that between 70% and 75% of all systems referred to as EDW are actually single business departments in nature.■ Optimization techniques such as summaries, aggregates and indexes are simply the result of performance restrictions inherent to normalized data and the way the RDBMS manages rows and columns. 4 © 2010 IBM Corporation
  5. 5. State of Data Warehousing (Cont.) A Glimpse Into the Future ■ Vendor solutions began to focus even more on the ability to isolate and prioritize workload types including strategies for dual warehouse deployments and mixing OLTP and OLAP on the same platform. ■ In-memory DBMS solutions provide a technology which enables OLTP/OLAP combined solutions. Organizations should increase their emphasis on financial viability during 2011 and even into 2012 as well as aligning their analytics strategies with vendor road maps when choosing a solution. 5 © 2010 IBM Corporation
  6. 6. Existing Informix Warehouse Features■ Performance & Scalability – Inherent SMP Multi-threading – Parallel Data Query (PDQ) – Light Scan for fast table scans – Online Index build – Efficient Hash Joins – Auto Fragment Elimination – Memory Grant Manager (MGM) – High Performance Loader – Optimistic Concurrency■■ Easy of Management – Time cyclic data management using Range Partitioning – Sophisticated Query Optimizer for OLTP and Warehousing 7 © 2010 IBM Corporation
  7. 7. Informix Warehousing Moving Forward■ Goal is to provide a comprehensive warehousing platform that is highly competitive in the marketplace – – Incorporating the best features of XPS and Red Brick into IDS for OLTP/Warehousing and Mixed-Workload – – Using the latest Informix technology in: • Continuous Availability and Flexible Grid • Data Warehouse Accelerator using latest industry technology – – Integration of IBM’s BI software stack 8 © 2010 IBM Corporation
  8. 8. BI Tools for Informix The Performance Management Framework Cognos 10 provides a comprehensive set Cognos identifies best-practice decision areas, or of BI tools for: information sweet spots by business function: Reporting Analysis Dashboards Scorecards Performance Management Framework for: Solutions for different areas of the organization 13 © 2010 IBM Corporation
  9. 9. Third Generation of Database Technology According to IDC’s Article (Carl Olofson) – Feb. 2010 1st Generation: - Vendor proprietary databases of IMS, IDMS, Datacom 2nd Generation: - RDBMS for Open Systems, dependent on disk layout, limitations in scalability and disk I/O - Database tuning by adding updating stats, creating/dropping indexes, data partitioning, summary tables & cubes, force query plans, resource governing 3rd Generation: IDC Predicts that within 5 years: ■ Most data warehouses will be stored in a columnar fashion ■ Most OLTP database will either be augmented by an in-memory database (IMDB) or reside entirely in memory ■ Most large-scale database servers will achieve horizontal scalability through clustering 14 © 2010 IBM Corporation
  10. 10. Market Data: Key Drivers of Change 15 © 2010 IBM Corporation
  11. 11. Informix Warehouse Accelerator How is it different?What is it? • Performance: Unprecedented responseThe Informix Warehouse Accelerator (IWA) is a times to enable train of thought analysisworkload optimized, appliance-like, add-on, that enables frequently blocked by poor querythe integration of business insights into operational performance.processes to drive winning strategies. It accelerates • Integration: Connects to IDS through deepselect queries, with unprecedented response times. integration providing transparency to all applications. • Self-managed workloads: queries are executed in the most efficient way • Transparency: applications connected to IDS, are entirely unaware of IWA • Simplified administration: appliance-like hands-free operations, eliminating many database tuning tasks Breakthrough Technology Enabling New Opportunities 16 © 2010 IBM Corporation
  12. 12. Breakthrough Technologies for Performance Extreme Compression Row & Columnar Database Required because RAM is the limiting factor. Row format within IDS for transactional workloads and columnar data access via accelerator for OLAP queries. Multi-core and Vector In Memory Database Optimized Algorithms 3 generation database technology avoids rd Avoiding locking or synchronization 7 1 I/O. Compression allows huge databases to be completely memory resident 6 2 5 3 Predicate evaluation on 4 Frequency Partitioning compressed data Enabler for the effective parallel access of Often scans w/o decompression the compressed data for scanning. during evaluation Horizontal and Vertical Partition Elimination. Massive Parallelism All cores are used within used for queries 17 © 2010 IBM Corporation
  13. 13. IWA: Characteristics • A dedicated SMP system (Linux on Intel x86_64) • No changes to the applications – Applications continue to attach to IDS. – When applicable query needs to be executed, IDS exploits the accelerator transparently to the applications – Fencing and protection of IDS against possible accelerator failures • Order of magnitude performance improvement • Reducing need for tedious tuning of IDS (partitioning, indexes, etc.) • Appliance-like form-factor – Hands free operations • Significantly improved price/performance and TCO as a combined effect of: – Accelerating intensive & complex analytics queries – Orders of magnitude performance improvement for accelerated queries – Reduced DBA effort for tuning accelerated queries 18 © 2010 IBM Corporation
  14. 14. Sample Customer Results: Case Study #1 Query Description Informix Informix w ISAO Notes Improvement 1 Find Top 100 Entities 1:28:22 0:01:28 Fact Table Scan 6023.23% 2 Find Top 100 Members 1:22:32 0:01:05 Fact Table Scan 7640.45% Summarize all transactions by State 3 and County 1:34:37 0:00:14 Fact Table Scan 41708.49% IWA did not support this Summarize the top 10 Commodities subquery 4 by State and County 1:05:03 1:03:35 query 102.29% Detailed Report on Specific Programs, States, Counties and 5 Years 0:00:00 0:00:00 Index Read 83.45% Detailed Report on Specific 6 Programs in a Date Range 0:00:06 0:00:06 Index Read 108.41% Summarize all transactions by State, County, City, State, Zip, Program, Program Year, 7 Commodity and Fiscal Year 1:48:58 0:00:41 Fact Table Scan 15800.89% Failed - I did not configure Find Entities where the payments do Long enough logs to not equal total Member Transac Failed - Long support the 8 Transaction Amounts tion Transaction query Totals 7:19:37 1:07:09 654.69% 19 © 2010 IBM Corporation
  15. 15. Government Agency Datamart Performance expectation goals were up to 20X OLAP-style Queries Tests were done on a Intel x86_64 SMP box running Linux RHEl Microstrategy Report was used, which generates 667 SQL statements 537 are SELECT statements. Datamart for this report has 250 Tables and 30 GB Data size Informix Panther and IWA run this report in 67 seconds. 7 seconds in IWA and 60 seconds in Informix (TEMP table processing, etc) Without IWA, total runtime on Informix 11.70 on the same HW is 40 Minutes! The same report today runs on XPS & SUN HW (Sparc M9000) and takes 90 mins. Performance gain for the customer would be ~90x !!! 20 © 2010 IBM Corporation
  16. 16. IWA Referenced Hardware Configuration Intel(R) Xeon(R) CPU X7560 @ 2.27GH 4 X 8 Memory 512G 6 disks 300 GB SAS hard disk drives each Options: - 4-processor, 4U rack-optimized enterprise server with Intel® Xeon® processors. - 8-core, 6-core and 4-core processor options with up to 2.26 GHz (8- core), 2.66 GHz (six-core) and 1.86 GHz (four-core) speeds with up to 16 MB L3 cache - Scalable from 4 sockets and 64 DIMMs to 8 sockets and 128 DIMMs - Optional MAX5 32-DIMM memory expansion - 16x 1.8" SAS SSDs with eXFlash or 8x 2.5" SAS HDDs 21 © 2010 IBM Corporation
  17. 17. IWA Software Components■ Linux on Intel x86_64 (RHEL 5 or SUSE SLES 11)■■ IDS 11.70 + IWA code modules including IDS Stored Procedures (Informix Ultimate Warehouse Edition)■■ ISAO Studio Plug-in – GUI for Mart definition■■ OnIWA – On Utilities for Monitoring IWA 22 © 2010 IBM Corporation