• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Ugif 12 2011-informix iwa
 

Ugif 12 2011-informix iwa

on

  • 846 views

 

Statistics

Views

Total Views
846
Views on SlideShare
846
Embed Views
0

Actions

Likes
0
Downloads
8
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Ugif 12 2011-informix iwa Ugif 12 2011-informix iwa Presentation Transcript

    • Discover InformixIBM Information Management Informix Ultimate Warehouse Edition - Extreme Performance for Faster Decisions sandor.szabo@de.ibm.com © 2011 IBM Corporation1
    • Discover InformixThe State of Data WarehouseA Glimpse Into the Future Vendor solutions began to focus even more on the ability to isolate and prioritize workload types including strategies for dual warehouse deployments and mixing OLTP and OLAP on the same platform. In-memory DBMS solutions provide a technology which enables OLTP/OLAP combined solutions. Organizations should increase their emphasis on financial viability during 2011 and even into 2012 as well as aligning their analytics strategies with vendor road maps when choosing a solution. Source: The State of Data Warehousing in 2011, 1/31/2011 by Mark Beyter, Roxane Edjlali, Donald Feinberg (ID Number: G00209643) © 2011 IBM Corporation
    • Discover InformixData Warehouse Trends for the CIO, 2011-2012Data Warehouse Appliances: DW appliances are not a new concept…Most vendors have developed an appliance offering or promote certified configurations…Although there are many reasons why organizations consider buying an appliance, the main reason is simplicity.The Resurgence of Data Marts: Data marts can be used to optimize DW by offloading part of the workload, returning greater performance to the warehousing environmentColumn-Store DBMSs CIOs should be aware that their current DBMS vendor may offer a column-store solution. Don’t just buy a column-store-only DBMS because a column store was recommended by your team.In-Memory DBMSs IMDBMS technology also introduces a higher probability that analytics and transactional systems can share the same database. Source: Data Warehousing Trends for the CIO, 2011-2012, 1/27/2011 by Mark Beyter, Roxane Edjlali, Donald Feinberg (ID Number: G00210272) © 2011 IBM Corporation
    • Discover InformixIT & Business Challenges for Analytics & Data Warehouse  Costly for IT  Challenges for Business – Cost for new hardware for – Lack of real-time operational processors and disks information – Administering OLTP and Data – Lack of Insight from lengthy Warehouses concurrently analyses – Expertise to tune databases – Inability to adopt new solutions © 2011 IBM Corporation4
    • Discover InformixWhat’s New in Data Warehousing (and Analytics)? Columnar © 2011 IBM Corporation5
    • In-Memory Computing Technology – Defined © 2011 IBM Corporation6
    • Discover Informix Row Oriented Data Store Each row stored sequentially • Optimized for record I/O • Fetch and decompress entire row, every time • Result – • Very efficient for transactional workloads • Not always efficient for analytical workloads If only few columns are required the complete row is still fetched and uncompressed © 2011 IBM Corporation
    • Discover Informix Columnar Data Store Data is stored sequentially by column• Data is compressed sequentially for column: •Aids sequential scan •Slows random access If attributes are not required for a specific query execution, they are skipped completely. © 2011 IBM Corporation
    • Discover InformixDW Appliance, Columnar and In-Memory Databases DW Appliance Columnar Database DataAllegro (Microsoft) Calpont Dataupia Exasol Infobright Greenplum (EMC) ParAccel Kognito Sand Technology Netezza (IBM) Vertica (HP) Sybase IQ (SAP) In-Memory OLAP Tools QlikTech/QlikView In-Memory Data Warehouse Applix TM-1 (IBM-Cognos) HANA (SAP) PALO ISAO-DB2 Z (IBM) Exalytics (Oracle) IWA (IBM) © 2011 IBM Corporation9
    • Discover InformixInformix Warehouse Accelerator – Breakthrough Technology for Performance Extreme Compression Row & Columnar Database 3 to 1 compression ratio Row format within IDS for transactional workloads and columnar data access via accelerator for OLAP queries. Multi-core and Vector In Memory Database Optimized Algorithms 3 generation database technology avoids rd Avoiding locking or synchronization 7 1 I/O. Compression allows huge databases to be completely memory resident 6 2 5 3 Predicate evaluation on 4 Frequency Partitioning compressed data Enabler for the effective parallel access of Often scans w/o decompression the compressed data for scanning. during evaluation Horizontal and Vertical Partition Elimination. Massive Parallelism All cores are used for each query Comes with Smart Analytics Studio, a GUI tool, for configuring data mart and monitoring IWA © 2011 IBM Corporation
    • Discover InformixInformix Ultimate Warehouse Edition What it is *Informix Warehouse Accelerator requires a Linux Intel system as it is relies on optimizations in that environment © 2011 IBM Corporation
    • Discover Informix Informix Warehouse Accelerator (Key Technologies) values Rare Frequency Partitioning Occurrences Number of 64-bit processor Common Values RAM in TB … … … … 11111 0 &1111 0 01001 0 == 1110 0 Compressed Predicate Evaluation A1 D1 G1 A2 D2 G2 A3 D3 G3 A4 D4 G4 SIMD © 2011 IBM Corporation12
    • Discover InformixCompression: Frequency Partitioning values RareTrade Info (volume, product, Histogram Column Partitions origin country) on Origin Vol Prod Origin Occurrences Number of China GER, USA FRA, … Rest Common Values Origin Top 64 traded goods Cell Cell Cell 4 – 6 bit code 1 3 Product Cell Cell Cell 6 Rest 2 5 Histogram on Product Table partitioned into Cells • Field lengths vary between cells • Higher Frequencies  Shorter Codes (Approximate Huffman) • Field lengths fixed within cells © 2011 IBM Corporation
    • Discover Informix Data is Processed in Compressed Format • Within a Register – Store, several columns are grouped together. • The sum of the width of the compressed columns doesn‘t exceed a register compatible width. This utilizes the full capabilities of a 64 bit system. It doesn‘t matter how many columns are placed within the register – wide data element. • It is beneficial to place commonly used columns within the same register – wide data element. But this requires dynamic knowledge about the executed workload (runtime statistics). • Having multiple columns within the same register – wide data element prevents ANDing of different results. Predicate evaluation is done against compressed data! The Register – Store is an optimization of the Column – Store approach where we try to make the best use of existing hardware. Reshuffling small data elements at runtime into a register is time consuming and can be avoided. The Register – Store also delivers good vectorization capabilities. © 2011 IBM Corporation
    • Discover InformixDefining, What Data to Accelerate • A MART is a logical collection of tables which are related to each other. For example, all tables of a single star schema would belong to the same MART. • The administrator uses a rich client interface or SmartMart to define the tables which belong to a MART together with the information about their relationships. • IDS creates definitions for these MARTs in the own catalog. The related data is read from the IDS tables and transferred to IWA. • The IWA transforms the data into a highly compressed, scan optimized format which is kept locally (in memory) on the Accelerator IDS + IWA Coordinator Worker Process Processes Define © 2011 IBM Corporation
    • Discover InformixInformix IWA in Action At A Retail Company IWA Store Managers & 160 GB data ~ 40GB Home Office compressed RAM Managers across IWA with 24 cores thousands of stores single Linux Intel < 10 secs average want to analyze box response with 500 promotional items users and 10x better Data set is ~200GB price/performance Current database Able to change unable to provide promotional items on quick enough a daily basis turnaround Challenge Solution Result © ۲۰۱۱ IBM Corporation۱۶
    • Discover InformixIWA in Action for Public Sector Long response when Seconds response time police calls dispatcher to queries Informix IWA with 2 Uncoordinated data cpus & 64 GB of Dispatcher can provide from State, County, memory at nominal coordinated data Dept, Specialty price databases No solution offered Challenge Solution Result © 2011 IBM Corporation17
    • Discover InformixPOC with Informix Warehouse Accelerator Data Warehouse query Performance without Perspiration Analysis query run time reduced from 45 minutes to 4 seconds Acceleration from 60x to 1400x – average acceleration of 450x More questions, faster answers, better business insights © 2011 IBM Corporation
    • Discover InformixPOC: Datamart at a Government Agency • Microstrategy report was run, which generates • 667 SQL statements of which 537 were Select statements • Datamart for this report has 250 Tables and 30 GB Data size • Original report on XPS and Sun Sparc M9000 took 90 mins • With IDS 11.7 on Linux Intel box, it took 40 mins • With IWA, it took 67 seconds. © 2011 IBM Corporation
    • Discover Informix Informix Growth Warehouse Edition IUWE IGWE Components Informix Ultimate Edition Informix Growth Edition Compression IWA IWA ISAO Studio ISAO Studio Limits Max memory available Informix Growth No core limits 16 cores, 16 GB Memory max Informix on 4 platforms: Informix on 4 platforms: AIX64, Sol64, HPUX64, Linux- AIX64, Sol64, HPUX64, Linux-Intel64 ntel64 IWA on Linux-Intel 64 IWA on Linux-Intel 64 48 GB Max, 16 core limit Target > 300 GB of raw data < 300 GB of raw data List Price $463 per PVU $150 per PVU © 2011 IBM Corporation20
    • Discover Informix Target Informix clients in the Ultimate Warehouse sweet spot Informix Warehouse EditionsInformix Ultimate WarehouseEdition (IUWE) and GrowthWarehouse Edition (IGWE)means higher performanceand lower costs for Informix Informix, XPS, Red Brickclients < 5 TB Star schema Mixed data mart Workloads "Gemini Systems is extremely excited about the Informix Ultimate Warehouse Edition. Combining deep columnar technology with the super fast performance of in-memory databases solves many problems for both legacy and future warehouse customers. The investment preservation proposition of this offering just cant be beat. No rip-and-replace, no code rewrites, no data migrations, no tuning. Just plug-in and go for immediate business value return." - Michael "Mick" Bisignani , Senior Vice President and CTO ,Gemini Systems LLC © 2011 IBM Corporation 21
    • Discover InformixDo you struggle with… … performance issues on analytics and business reports ? •Reports taking too long to run •Ad-hoc queries with unpredictable response times … cost and flexibility for mixed … ongoing warehouse workloads? maintenance and administration? •Unable to optimize on a single platform •Constant tuning •Building/Maintaining cubes •Constant storage optimization … leaving you at a competitive disadvantage ? This is an example text. Go ahead and replace it with your own text. It is meant to give you … feeling of how the designs looks including text. a Introducing the Informix Ultimate Warehouse Edition © 2011 IBM Corporation
    • Discover InformixNew order of Performance! Take “No” for No Near zero Storage maintenance!! allocation/data administration!! partitioning 10s to 1000s of Index Statistics times faster maintenance maintenance Predictable NO response times Cube Application maintenance changes or summary tables © 2011 IBM Corporation
    • Discover Informix Informix Ultimate Warehouse – Performance, Simplicity, Transparency BI App Configure, offload data mart HPUX-64, AIX-64, SOL-64, Linux-64 Linux-64, Intel Redirect queries Informix env Informix Warehouse Accelerator Query Results Warehouse DataMart © 2011 IBM Corporation
    • Discover InformixInformix Hybrid Engine Overview . . . . © 2011 IBM Corporation
    • Discover Informix IWA Design Studio DB connections Accelerator © 2011 IBM Corporation
    • Workload Advisor for Mart Definition• Takes the guesswork out of defining a data mart for IWA• Run selected queries (presumably the most time- consuming ones) through advisor• Advisor will generate mart definition in XML format to be loaded onto IWA• Can be fully automated
    • Typical Data Warehouse ArchitectureDiscover InformixAll databases as marked above including OLTP, data warehouse/data mart/ODS canrun on Informix © 2011 IBM Corporation
    • Discover InformixWhat Is IWA Ideally Suited For? REGION Star or snowflake schema Complex, OLAP-style queries that typically: MONTH • Need to scan large subset of data (unlike QUARTER CITY OLTP queries) • Involve aggregation function such as COUNT, SUM, AVG. • Look for trends, exceptions to assist in STORE PERIOD making actionable business decisions SALES SELECT PRODUCT_DEPARTMENT, REGION, SUM(REVENUE) FROM FACT_SALES F INNER JOIN DIM_PRODUCT P ON F.FKP = P.PK INNER JOIN DIM_REGION R ON F.FKR = R.PK LEFT OUTER JOIN DIM_TIME T ON F.FKT = T.PK PRODUCT WHERE T.YEAR = 2007 GROUP BY PRODUCT_DEPARTMENT, REGION CATEGORY BRAND © 2011 IBM Corporation
    • Discover Informix Sizing Guidelines Number of Intel cores T-shirt size Raw data * Main Memory (X7560) XL >1.5 TB to 3 TB 1 TB 24-32 L >750 GB to 1.5 TB 512 20-24 M > 400 GB to 750 GB 256 GB 16-20 S > 250 GB to 400 GB 192 GB 12-16 XS ≥ 100 GB to 250 GB 96 GB 8-12 XXS < 100 GB 48 GB 8 XXXS < 50 GB 24 GB 4 * Raw data represents only table data and excludes any indices, temp table space etc Important Considerations T-shirt sizes are a reference guideline only and are not officially available configurations. © 2011 IBM Corporation
    • Discover Informix Configuration Scenarios Alternative 1: Install IWA on a separate Linux box Database Server InformixWarehouse Accelerator RHEL 5,6/SUSE 11 -64 Solaris 10/AIX 6.1/HP-UX 11.31 64 RHEL 5,6/SUSE 11 - Alternative 2: Install Informix and IWA in the same symmetric multiprocessing system Database Server Informix Warehouse Accelerator RHEL 5,6/SUSE 11-64 Note: IWA requires Linux on Intel x64 (64-bit EM64T) Xenon © 2011 IBM Corporation
    • Discover InformixThe Differentiation Deep Columnar Technology In-Memory Data is stored and accessed Entire data set being queried is using columnar approach compressed and in-memory eliminating disk I/O IUWE Run mixed workloads No Maintenance OLTP transactions and OLAP No requirements for indexes, queries can run against the query tuning or MOLAP cubes same system 450 times 330 900 times The Result!! times 1350 times ORDERS OF MAGNITUDE PERFORMANCE IMPROVEMENTS!! © 2011 IBM Corporation
    • Discover InformixMotto for UWE“Everything should be madeas simple as possible, but notsimpler.”―Albert Einstein © 2011 IBM Corporation
    • Discover Informix Questions? contact Sandor Szabo, Sandor.szabo@de.ibm. com © 2011 IBM Corporation