© 2014 IBM Corporation
Sandor Szabo IBM Informix Development Lab – IBM Software Group
May , 2014
BLU Acceleration for SQL ...
© 2014 IBM Corporation
Informix 12.1
22
Agenda: Informix Warehouse Accelerator (IWA)
• Data Warehouse Trends
• Informix Wa...
© 2014 IBM Corporation
Informix 12.1
33
TRENDS
Database and Data Warehousing Industry
© 2014 IBM Corporation
Informix 12.1
44
Data Warehousing Workload & Optimizations
 Data Warehousing/OLAP workload
are inh...
© 2014 IBM Corporation
Informix 12.1
55
Third Generation of Database Technology
According to IDC’s Article (Carl Olofson) ...
© 2014 IBM Corporation
Informix 12.1
66
In-Memory DB: Why Now?
 64-bit processors can address up to16 exabytes of data
 ...
© 2014 IBM Corporation
Informix 12.1
77
“By 2012, 70% of Global 1000 organizations will load detailed data into
memory as ...
© 2014 IBM Corporation
Informix 12.1
88
TECHNOLOGY OVERVIEW
IBM Informix Warehouse Accelerator (IWA)
© 2014 IBM Corporation
Informix 12.1
99
Introducing IBM Informix Warehouse Accelerator (IWA)
Results
Analytic
query
Linux ...
© 2014 IBM Corporation
Informix 12.1
1010
Unique combination of technologies for Speed-of-Thought Analysis
In-Memory Data ...
© 2014 IBM Corporation
Informix 12.1
1111
IWA Benefits
 Extreme performance for Analytics: 100x+ faster response times fo...
© 2014 IBM Corporation
Informix 12.1
1212
You can use IWA’s In-Memory Analytics to Speed Up queries on…
© 2014 IBM Corporation
Informix 12.1
1313
HOW IT WORKS
IBM Informix Warehouse Accelerator (IWA)
© 2014 IBM Corporation
Informix 12.1
1414
IWA Overview and Seamless Integration with Informix/IDS
 Before IWA…
 Informix...
© 2014 IBM Corporation
Informix 12.1
1515
IWA Overview and Seamless Integration with Informix/IDS
 Setting up IWA…
 Info...
© 2014 IBM Corporation
Informix 12.1
1616
IWA Overview and Seamless Integration with Informix/IDS
 Using IWA: Process is ...
© 2014 IBM Corporation
Informix 12.1
1717
IWA Overview and Seamless Integration with Informix/IDS
 Using IWA: Process is ...
© 2014 IBM Corporation
Informix 12.1
1818
CUSTOMERS AND PARTNERS
Informix Warehouse Accelerator (IWA)
© 2014 IBM Corporation
Informix 12.1
1919
Federació Farmacèutica SCCL
50K IWA query requests a day
464 Million rows in big...
© 2014 IBM Corporation
Informix 12.1
2020
Corporate Name: LABCO S.A.
Brand Name: Labco
Life Sciences, Medical and Laborato...
© 2014 IBM Corporation
Informix 12.1
2121
LABCO
20K IWA analytic query requests a day
2 big data marts, with these fact ta...
© 2014 IBM Corporation
Informix 12.1
2222
Europe’s Largest Power company tackles the Smart Meter Big Data
challenge with I...
© 2014 IBM Corporation
Informix 12.1
2323
ROADMAP & NEW FEATURES
Informix Warehouse Accelerator (IWA)
© 2014 IBM Corporation
Informix 12.1
2424
Informix 11.70.FC2
(Mar 2011)
 IWA 1st Release on IUWE
 On SMP
Informix 11.70....
© 2014 IBM Corporation
Informix 12.1
2525
What’s New in IWA 12.10.FC1
 Refreshing Data in IWA
– Automatic Partition Refre...
© 2014 IBM Corporation
Informix 12.1
2626
Automatic Partition Refresh
Automatic IWA synchronization with
Informix, on-dema...
© 2014 IBM Corporation
Informix 12.1
2727
Low Administration
Automated and fast small updates
keep IWA data current
Allows...
© 2014 IBM Corporation
Informix 12.1
2828
Support of UNION [ALL] queries
Benefits…
Fast response for operational and
busi...
© 2014 IBM Corporation
Informix 12.1
2929
Support of standard SQL OLAP functions
In-Database and In-Memory Analytics
Simpl...
© 2014 IBM Corporation
Informix 12.1
3030
Administering IWA from OAT
to Monitor and Manage IWA
Web and Mobile Interface
As...
© 2014 IBM Corporation
Informix 12.1
3131
Informi
x
Informix TimeSeries table Virtual Table Interface (VTI)
representation...
© 2014 IBM Corporation
Informix 12.1
3232
IWA data mart supports External Tables
Direct, fast and flexible way to leverage...
© 2014 IBM Corporation
Informix 12.1
3333
IWA data mart supports synonyms and views
Fast analytic queries on SQL and NoSQL...
© 2014 IBM Corporation
Informix 12.1
3434
IWA data mart supports of views and usage in NoSQL query (1)
 From MongoDB shel...
© 2014 IBM Corporation
Informix 12.1
3535
IWA data mart supports of views and usage in NoSQL query (2)
 From Informix:
– ...
© 2014 IBM Corporation
Informix 12.1
3636
IWA data mart view support to accelerate SQL-NoSQL query (1)
 From Informix:
– ...
© 2014 IBM Corporation
Informix 12.1
3737
IWA data mart view support to accelerate SQL-NoSQL query (2)
 Probe the query w...
© 2014 IBM Corporation
Informix 12.1
3838
 Use workload analysis to find best IWA data
mart to accelerate these 3 queries...
© 2014 IBM Corporation
Informix 12.1
3939
IWA data mart supports of views and usage for Self Join queries (2)
 Deploy IWA...
© 2014 IBM Corporation
Informix 12.1
4040
IWA data mart supports of views and usage for Self Join queries (3)
 From Infor...
© 2014 IBM Corporation
Informix 12.1
4141
sandor.szabo@de.ibm.com
© 2014 IBM Corporation
Informix 12.1
42
Handling Big Data without angst
© 2014 IBM Corporation
Informix 12.1
4343
© 2014 IBM Corporation
Informix 12.1
4444
Logo
© 2014 IBM Corporation
Informix 12.1
4545
Logo
Upcoming SlideShare
Loading in …5
×

Informix warehouse accelerator update

932 views

Published on

Informix Warehouse Accelerator - Update, "BLU Acceleration for SQL / NoSQL databases " Sandor Szabo

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
932
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
29
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Informix warehouse accelerator update

  1. 1. © 2014 IBM Corporation Sandor Szabo IBM Informix Development Lab – IBM Software Group May , 2014 BLU Acceleration for SQL / NoSQL databases Extreme Speed with In-Memory Analytics
  2. 2. © 2014 IBM Corporation Informix 12.1 22 Agenda: Informix Warehouse Accelerator (IWA) • Data Warehouse Trends • Informix Warehouse Accelerator (IWA) • Technology Overview • How it Works • Positioning and Competition • Customers and Partners • Reference Architecture • 12.10 Features & Roadmap • Q&A
  3. 3. © 2014 IBM Corporation Informix 12.1 33 TRENDS Database and Data Warehousing Industry
  4. 4. © 2014 IBM Corporation Informix 12.1 44 Data Warehousing Workload & Optimizations  Data Warehousing/OLAP workload are inherently more complex than OLTP transactions and reasons are well-documented  Ways to overcome that include: – Building Indexes – Partitioning of Data – Building Cubes (MOLAP / ROLAP / HOLAP) – Query Tuning – Appliances that add a new layer of Hardware to perform I/O for DBMS  Mixed-Workload always a challenge  DBMS needs to be built to handle such a workload
  5. 5. © 2014 IBM Corporation Informix 12.1 55 Third Generation of Database Technology According to IDC’s Article (Carl Olofson) – Feb. 2010  1st Generation: –Vendor proprietary databases of IMS, IDMS, Datacom  2nd Generation: –RDBMS for Open Systems –Dependent on disk layout, limitations in scalability and disk I/O  3rd Generation: IDC Predicts that within 5 years: –Most data warehouses will be stored in a columnar fashion –Most OLTP database will either be augmented by an in-memory database (IMDB) or reside entirely in memory –Most large-scale database servers will achieve horizontal scalability through clustering
  6. 6. © 2014 IBM Corporation Informix 12.1 66 In-Memory DB: Why Now?  64-bit processors can address up to16 exabytes of data  DRAM prices drop by 30% every 18 months  1 Gb of NAND flash memory average price is less than US$0,50  Commodity blades provide 1 terabyte of DRAM  Multicore CPUs enable parallel processing of in-memory data  In-memory-enabling software is amply available and proven
  7. 7. © 2014 IBM Corporation Informix 12.1 77 “By 2012, 70% of Global 1000 organizations will load detailed data into memory as the primary method to optimize BI application performance.” - Gartner Conventional Databases Disk Read 5 milliseconds In-Memory Databases Disk Read 5 nanoseconds The idea: Store & Process Data in Memory instead on Disk to analyze data magnitudes Faster Than with traditional systems
  8. 8. © 2014 IBM Corporation Informix 12.1 88 TECHNOLOGY OVERVIEW IBM Informix Warehouse Accelerator (IWA)
  9. 9. © 2014 IBM Corporation Informix 12.1 99 Introducing IBM Informix Warehouse Accelerator (IWA) Results Analytic query Linux on Intel / AMD 64-bit TCP/IP Query Optimizer In-Memory Compressed Columnar Database Partition Bulk Loader Query Processor Yes Analytic query Results Accelerate Query? Most Unix/Linux 64-bit platforms In-Disk [Compressed] Relational / Row-based Database Informix database server Informix Warehouse Accelerator No POWERFUL HYBRID DATABASE PLATFORMPOWERFUL HYBRID DATABASE PLATFORM Extreme Performance Transactions Extreme Performance Analytics
  10. 10. © 2014 IBM Corporation Informix 12.1 1010 Unique combination of technologies for Speed-of-Thought Analysis In-Memory Data storage & Query Processing Multi-Core Parallelism and Vector Optimized Algorithm (No Locking) on Intel 64-bit SIMD technology Massive Parallel Processing of Data Load/Refresh & Query Fast Storage Backup in Disk for recovery purposes IWA combines Breakthrough IBM Research & Development Lab Innovations Row (Informix) & Column (IWA) data storage Deep Data Compression Fits TB of raw data storage Predicate evaluation done directly on Compressed data Intelligent Frequency Partitioning Full, Partial and Incremental Refresh (Insert Only on Delta) IWA IWA Informix IWAInformix IWA IWA File System copy Memory
  11. 11. © 2014 IBM Corporation Informix 12.1 1111 IWA Benefits  Extreme performance for Analytics: 100x+ faster response times for complex BI queries  Leverages existing Informix database, builds on top, to provide instant performance boost  Uses low cost commodity HW: Linux on Intel/AMD 64-bit  Handles Terabytes of data in-memory, thanks to compressed storage and query technology  Works “behind the scenes” in Informix, transparent to client applications  Very simple and flexible installation, configuration and administration  Informix + IWA is a hybrid database platform which provides the best technology and performance for both OLTP and OLAP workloads and support of Big Data solutions  No need to keep doing all this in order to get high performance OLAP queries: – Indexes – Aggregates / summary tables – Materialized query tables/views – Cubes – Decide on best data partition strategies – Keep different database systems for each type of workload: OLTP vs OLAP – Migrate data to another OLAP database – Change your database schema – Change your analytic applications – Tune I/O, memory and CPU for OLAP – Update Statistics – Tune Queries with Optimizer Directives
  12. 12. © 2014 IBM Corporation Informix 12.1 1212 You can use IWA’s In-Memory Analytics to Speed Up queries on…
  13. 13. © 2014 IBM Corporation Informix 12.1 1313 HOW IT WORKS IBM Informix Warehouse Accelerator (IWA)
  14. 14. © 2014 IBM Corporation Informix 12.1 1414 IWA Overview and Seamless Integration with Informix/IDS  Before IWA…  Informix – Receives analytic query from client – Spends some time doing intensive I/O – Returns results back to the client Informix 12.1 Results SQL
  15. 15. © 2014 IBM Corporation Informix 12.1 1515 IWA Overview and Seamless Integration with Informix/IDS  Setting up IWA…  Informix – Determine the database subset used in analytic queries to accelerate (data mart) • Manually or through Workload Analysis – Deploy an IWA data mart based on DB subset • Stream Load the data from Informix into IWA – Informix Optimizer is aware of IWA datamart Informix 12.1  The Accelerator – Install and configure IWA on Linux x86_86 – Connect with Informix using custom protocol – IWA compresses and stores a copy of the Informix DB set into data marts in-memory – IWA data mart is fully loaded, valid and ready for Informix server to use as needed Linux on Intel/AMD 64-bit Bulk Loader Compressed Database Partition TCP/IP Compression In-Memory Columnar Storage Frequency Partitioning Parallelism Predicate evaluation on compressed data Multi-core and Vector optimized algorithms SIMD Compression In-Memory Columnar Storage Frequency Partitioning Parallelism Predicate evaluation on compressed data Multi-core and Vector optimized algorithms SIMD Query Router Query Processor
  16. 16. © 2014 IBM Corporation Informix 12.1 1616 IWA Overview and Seamless Integration with Informix/IDS  Using IWA: Process is transparent to Informix client Results SQL 16 Introduction to IBM Informix Warehouse Accelerator  Informix – Receives analytic query from client – If query uses data matching an IWA datamart and can be accelerated, route/offload it to IWA – Returns results back to the client Informix 12.1  The Accelerator – Processes the routed SQL query extremely fast and returns answer back to Informix Linux on Intel/AMD 64-bit Bulk Loader Compressed Database Partition TCP/IP Compression In-Memory Columnar Storage Frequency Partitioning Parallelism Predicate evaluation on compressed data Multi-core and Vector optimized algorithms SIMD Compression In-Memory Columnar Storage Frequency Partitioning Parallelism Predicate evaluation on compressed data Multi-core and Vector optimized algorithms SIMD Query Router Query Processor SQL Results
  17. 17. © 2014 IBM Corporation Informix 12.1 1717 IWA Overview and Seamless Integration with Informix/IDS  Using IWA: Process is transparent to Informix client Results SQL 17 Introduction to IBM Informix Warehouse Accelerator  Informix – Receives analytic query from client – If query uses data matching an IWA datamart and can be accelerated, route/offload it to IWA – Returns results back to the client – If query is not based on an IWA datamart or cannot be accelerated, Informix will resolve it Informix 12.1  The Accelerator – Processes the routed SQL query extremely fast and returns answer back to Informix Linux on Intel/AMD 64-bit Bulk Loader Compressed Database Partition TCP/IP Compression In-Memory Columnar Storage Frequency Partitioning Parallelism Predicate evaluation on compressed data Multi-core and Vector optimized algorithms SIMD Compression In-Memory Columnar Storage Frequency Partitioning Parallelism Predicate evaluation on compressed data Multi-core and Vector optimized algorithms SIMD Query Router Query Processor
  18. 18. © 2014 IBM Corporation Informix 12.1 1818 CUSTOMERS AND PARTNERS Informix Warehouse Accelerator (IWA)
  19. 19. © 2014 IBM Corporation Informix 12.1 1919 Federació Farmacèutica SCCL 50K IWA query requests a day 464 Million rows in biggest fact table 34x faster Global Sales statistics: From 1hr down to 1min 45sec 2x faster Invoicing systems 30 users, 83 processes using IWA Huge savings in Data Warehouse space in disk: No need for Staging Tables: Size went from 4.5TB to 500GB Informix Storage Optimization Feature (Deep Compression): Size from 500GB to 140GB “Previous DWH needs a lot of staging tables due to performance issues. With IWA we've dropped all Staging requirements and converted our old 4.5 Terabytes DWH to 500GB on disk. With IWA, our invoicing system has dropped dramatically the time required from 24H to just 12H.” Santi Pla, IT Director, FedeFarma “Previous DWH needs a lot of staging tables due to performance issues. With IWA we've dropped all Staging requirements and converted our old 4.5 Terabytes DWH to 500GB on disk. With IWA, our invoicing system has dropped dramatically the time required from 24H to just 12H.” Santi Pla, IT Director, FedeFarma
  20. 20. © 2014 IBM Corporation Informix 12.1 2020 Corporate Name: LABCO S.A. Brand Name: Labco Life Sciences, Medical and Laboratory Services IWA Utilization Country: Belgium, France, Portugal, Spain, Switzerland IBM Business Partner: Deister S.A. Solution components: Hardware: IWA runs on 8 cores IBM X Series Intel Xeon Software: IBM Informix Advanced Enterprise Edition v12 IBM Informix Technologies used: Informix Warehouse Accelerator (IWA) Storage Optimization Feature (Deep Compression)
  21. 21. © 2014 IBM Corporation Informix 12.1 2121 LABCO 20K IWA analytic query requests a day 2 big data marts, with these fact tables: 1150 Million rows fact table one 484 Million rows fact table the other 783GB database (w/ storage optimization) 48x faster response of heavy dashboards: From 24hr down to 30min Fast deep analysis of laboratory data: Average analysis time: 1min 45sec 15 users, 20 processes using IWA; DWH is becoming source of info for new users “Without IWA, this project could not been completed successfully. Delivering accurate information of the status of a pan European group to the decision makers staff in minutes was impossible previously. Now with IWA, a lot of new information is available and could be analyzed at the moment it's required.” Vicente Salvador, Deister S.A. “Without IWA, this project could not been completed successfully. Delivering accurate information of the status of a pan European group to the decision makers staff in minutes was impossible previously. Now with IWA, a lot of new information is available and could be analyzed at the moment it's required.” Vicente Salvador, Deister S.A.
  22. 22. © 2014 IBM Corporation Informix 12.1 2222 Europe’s Largest Power company tackles the Smart Meter Big Data challenge with Informix TimeSeries + In-Memory Accelerator (IWA)  E.ON Metering (EMTG) is the centre of excellence for the development and commercialization of smart energy solutions and technologies and part of Europe’s largest Power and Gas company E.ON  EMTG operates a sophisticated Smart Meter data infrastructure based on IBM Informix TimeSeries technology in combination with Informix In-Memory Warehouse Accelerator  IBM Information Management products currently used: – Informix 11.70 Ultimate Warehouse Edition – Cognos Business Intelligence 10
  23. 23. © 2014 IBM Corporation Informix 12.1 2323 ROADMAP & NEW FEATURES Informix Warehouse Accelerator (IWA)
  24. 24. © 2014 IBM Corporation Informix 12.1 2424 Informix 11.70.FC2 (Mar 2011)  IWA 1st Release on IUWE  On SMP Informix 11.70.FC3 (Jun 2011)  Workload Analysis Tool  Support of more Locales  Support of Data Currency Informix 11.70.FC4 (Oct 2011)  IGWE (Growth ed. IWA)  IWA cluster on Blade Server  Non-disruptive Mart Refresh  New AQT monitoring options  New SQL syntax/functions Roadmap of IWA-Specific Features Informix 11.70.FC5 (May 2012)  Use w/IDS Secondary Servers  New use_dwa options  Multiple DISTINCT’s in IDS+IWA  Additional SQL functions/syntax  Support Solaris Intel 64 on IDS  Partition-level Fact tables refresh  Support col[x,y] substr notation Informix 11.70.FC6 (Oct 2012)  Partition-level refresh on Dimension tables Informix 11.70.FC7 (Dec 2012)  Enhanced query processing on sessions with multiple cursors open Informix 12.10.FC3 (Mar 2014)  IWA support of synonyms and views, allows using new and multiple sources of data in a single data mart and query:  Data from local/remote tables  NoSQL data (mapped views)  Enables Self-Joins (via views)  New function QUARTER in IDS+IWA  Extended LIMIT syntax in IDS+IWA  IWA accelerates NoSQL data (views)  New use_dwa option (uniquecheck) 20112011 20122012 20132013 20142014  Informix 12.10.FC1 (Mar 2013)  Cognos BI [+SPSS] in Advanced ed.  Automatic partition-level refresh  Continuous refresh: Trickle Feed, enabling real-time / right-time analytics  Standard OLAP SQL in IDS+IWA  Support of UNION [ALL] queries  CTE: SELECT…FROM (SELECT…)  Support of INSERT INTO…SELECT  Additional SQL and data types  Admin w/OAT & built-in ifx_ functions  WAREHOUSE security privilege Informix 12.10.FC2 (Sep 2013)  Support of data from External Tables  TimeSeries data (VTI) support: Real- time analytics & Big Data on sensor data  IDS (not IWA): Hybrid SQL + NoSQL (non-structured data: JSON/BSON)
  25. 25. © 2014 IBM Corporation Informix 12.1 2525 What’s New in IWA 12.10.FC1  Refreshing Data in IWA – Automatic Partition Refresh – Trickle Feed (continuous refresh)  Additional SQL support for analytics in IWA – UNION [ALL] queries – New SQL OLAP functions  Additional interfaces for IWA administration – OpenAdmin Tool (OAT) menu for IWA – New SQL built-in routines for administering IWA  Improved Security – New WAREHOUSE privilege required for administering IWA
  26. 26. © 2014 IBM Corporation Informix 12.1 2626 Automatic Partition Refresh Automatic IWA synchronization with Informix, on-demand Let Informix find the changed partitions since last refresh and refresh them in IWA for you Easy adoption and maintenance of Real-Time Analytics With this enhancement… A single command instructs IWA to refresh only changed data partitions from Informix database to IWA Applies to Fact and Dimension tables Benefits… It removes the potentially error-prone process for manual identification of changed partitions in Informix Easier administration for keeping current Informix data in IWA
  27. 27. © 2014 IBM Corporation Informix 12.1 2727 Low Administration Automated and fast small updates keep IWA data current Allows for Real/Right-Time Analytics and Operational BI Continuous Refresh: Trickle Feed ifx_setupTrickleFeed Tracks changes in Dimensions Tracks inserts in Fact tables Automated updates in IWA datamart With this enhancement… Incremental inserts to the Fact tables and changes to Dimensions tables can be continuously updated into IWA Changes can be at row level, which is more granular than at partition level Benefits… We can have “speed of thought” analytics in a real-time data warehouse or mixed workload environment Actionable analytics on operational data
  28. 28. © 2014 IBM Corporation Informix 12.1 2828 Support of UNION [ALL] queries Benefits… Fast response for operational and business analytics workloads calling heavy queries in UNION [ALL] Typical UNION use cases in BI tools:  OLAP/Cube operations: ROLL- UP/ACROSS, AGGREGATE, GROUP  ETL: Extract, Slowly-Changing Dimensions (SCD)  Cross-table queries and reports  Set operations: union/union all Some or all of the queries in an UNION, will be accelerated and run much faster BI tools operations that use UNION behind the scenes will run 100x+ faster With this enhancement… IWA can now accelerate previously long running queries combined in an UNION or UNION ALL operation SELECT i_item_id, avg(cs_quantity) agg1, avg(cs_list_price) agg2, avg(cs_coupon_amt) agg3, avg(cs_sales_price) agg4 FROM catalog_sales, customer_demographics, date_dim, item, promotion WHERE cs_sold_date_sk = d_date_sk and cs_item_sk = i_item_sk and cs_bill_cdemo_sk = cd_demo_sk and cs_promo_sk = p_promo_sk and cd_gender = 'F' and cd_marital_status = 'M' and cd_education_status = 'College' and (p_channel_email = 'N' or p_channel_event = 'N‘) and d_year = 2001 GROUP BY i_item_id UNION ALL SELECT i_item_id, avg(ws_quantity) agg1, avg(ws_list_price) agg2, avg(ws_coupon_amt) agg3, avg(ws_sales_price) agg4 FROM web_sales, customer_demographics, date_dim, item, promotion WHERE ws_sold_date_sk = d_date_sk and ws_item_sk = i_item_sk and ws_bill_cdemo_sk = cd_demo_sk and ws_promo_sk = p_promo_sk and cd_gender = 'F' and cd_marital_status = 'M' and cd_education_status = 'College' and (p_channel_email = 'N' or d_channel_event = 'N') and d_year = 2001 GROUP BY i_item_id ;
  29. 29. © 2014 IBM Corporation Informix 12.1 2929 Support of standard SQL OLAP functions In-Database and In-Memory Analytics Simplified Code for OLAP which can be accelerated, much faster response times Better Platform Integration and Utilization With this enhancement… Both Informix and IWA support ANSI standard SQL On-Line Analytical Processing (OLAP) functions: – Ranking: RANK, DENSE_RANK, DENSERANK, CUME_DIST, PERCENT_RANK, NTILE – Numbering: ROW_NUMBER, ROWNUMBER – Aggregate: SUM, COUNT, AVG, MIN, MAX, STDEV, VARIANCE, RANGE, RATIO_TO_REPORT, RATIOTOREPORT – First/Last: FIRST_VALUE, LAST_VALUE Support of windowed aggregates: Create windows partitions. Apply OLAP function on each row Final Order by Join filters Group by Having Benefits… Reduce SQL code and accelerate to get much faster performance for OLAP or multidimensional analysis Better integration and support for BI tools like Cognos BI, and applications that use standard OLAP SQL calls Increase performance by reducing repeated scans, temporary tables and aggregation needed to do OLAP
  30. 30. © 2014 IBM Corporation Informix 12.1 3030 Administering IWA from OAT to Monitor and Manage IWA Web and Mobile Interface As part of Informix administration tools With this enhancement… You can use the standard graphic administration tool OpenAdmin Tool (OAT), for all Informix administration tasks, including the ones for IWA Benefits… Easy to use graphic tool OAT can be used to easily setup, integrated administration environment OLTP/OLAP No need to remember and run commands or stored procedures to manage IWA
  31. 31. © 2014 IBM Corporation Informix 12.1 3131 Informi x Informix TimeSeries table Virtual Table Interface (VTI) representation of TimeSeries table Real-time Analytics IWA IWA support for Time Series data Benefits… High-performance right-time analytics on big data collected from your sensors, meters, events, GPS/location, RFIDs, to anticipate and improve actions Combine TimeSeries and IWA for operational actionable analytics based on historic and current sensors data Unique platform, flexible, fast and scalable, for the most challenging Big Data and Smart Planet solutions Right-Time Analytics on time-stamped data Big Data solutions on Sensor data Operational Intelligence With this enhancement… You can include Time Series data coming from smart sensors into IWA Data marts in IWA can be defined and loaded from an Informix’ s Virtual Table Interface (VTI) of your TimeSeries data
  32. 32. © 2014 IBM Corporation Informix 12.1 3232 IWA data mart supports External Tables Direct, fast and flexible way to leverage external data for in-memory analytics With this enhancement… We can load data directly from external tables into IWA data marts without having to load it into Informix database first Large external data in ASCII / binary files or network devices –ex: through named pipes– can be used to quickly populate an IWA data mart Benefits… Run extremely fast in-memory analytic queries on operational data from non- Informix external files and devices. Large amount of external data is quickly loaded and made available in IWA, thanks to high performance reads of Informix External Tables and the no need for the external data to be loaded into Informix database first Storage savings and flexibility to do in- memory analytics on large data in file systems or devices and integrate it with other SQL and NoSQL data in Informix.
  33. 33. © 2014 IBM Corporation Informix 12.1 3333 IWA data mart supports synonyms and views Fast analytic queries on SQL and NoSQL data Analytics on data from different sources More queries and datatypes can be accelerated With this enhancement… Until now, an IWA datamart could only contain regular local tables, all in the same database We can now create an IWA datamart that uses remote tables in another Informix DB and accelerate queries using those remote tables – By having a local synonym in the Informix DB of the datamart, which points to the remote Informix table We can include views as part of an IWA datamart definition, and accelerate queries that use views – Views could map to a subset of another Informix table or also to NoSQL data – Use views to accelerate self-joins Benefits… Ability to combine and accelerate queries on local with remote tables, no need to make all tables local Allows to be able to do accelerate self- joins queries by using views Allows to accelerate data in JSON collections by using views Fast analytic queries on views, typically slow in SQL DBs due to on-the-fly view materialization
  34. 34. © 2014 IBM Corporation Informix 12.1 3434 IWA data mart supports of views and usage in NoSQL query (1)  From MongoDB shell: – Create two collections (JSON): comments and users  From Informix: – Create a view on each JSON collection (comments, users) – Deploy an IWA data mart by probing a join between them: – Run the accelerated query. $ mongo demo_database MongoDB shell version: 2.4.9 connecting to: demo_database mongos> db.comments.insert( [ { uid:12345, pid:444, comment:"first" }, { uid:12345, pid:888, comment:"second" }, { uid:99999, pid:444, comment:"third" } ] ) mongos> db.users.insert( [ { uid:12345, name:"john" }, { uid:99999, name:"mia" } ] ) mongos> exit Example: Accelerating NoSQL data (in a JSON collection) using IWA’s view support $ dbaccess demo_database - > create view vcomments(uid,pid,comment) as select bson_value_int(data,'uid'), bson_value_int(data,'pid'), bson_value_varchar(data,'comment') from comments; > create view vusers(uid,name) as select bson_value_int(data,'uid'), bson_value_varchar(data,'name') from users; set environment use_dwa 'probe cleanup'; set environment use_dwa 'probe start'; select {+ avoid_execute} * from vcomments c, vusers u where c.uid=u.uid; set environment use_dwa 'probe stop'; execute procedure ifx_probe2mart('demo_database','noSQL_mart'); execute function ifx_createmart('demo_dwa','noSQL_mart'); execute function ifx_loadmart('demo_dwa','noSQL_mart','NONE');
  35. 35. © 2014 IBM Corporation Informix 12.1 3535 IWA data mart supports of views and usage in NoSQL query (2)  From Informix: – Run the accelerated query on the JSON collection data:  Example: Accelerating NoSQL data (in a JSON collection) using IWA’s view support set environment use_dwa 'accelerate on'; select c.uid,name,comment from vcomments c, vusers u where c.uid=u.uid and pid=444; uid 12345 name john comment first uid 99999 name mia comment third
  36. 36. © 2014 IBM Corporation Informix 12.1 3636 IWA data mart view support to accelerate SQL-NoSQL query (1)  From Informix: – We have an Informix table sqldoc1: – We also have a JSON collection doc1: In Informix using its JSON compatibility or its MongoDB driver: Example: Accelerating a query combining SQL with NoSQL data (in a JSON collection) using IWA’s view support create table sqldoc1 ( name varchar(10), value integer ); insert into sqldoc1 values ("John", 1); insert into sqldoc1 values ("Scott", 2);  Create a view for the NoSQL data: – Run the accelerated query on the JSON collection data: create table doc1 ( c1 serial, data BSON ); Insert into doc1(c1,data) values (0, '{ fname:"John", lname:"Miller", age:21, address: { street:"Informix ave" } }'::JSON ); Insert into doc1(c1,data) values (0, '{ fname:"Scott", lname:"Lashley", age:21.50, address: { street:"Blazer ave" }, job:"Ref" }'::JSON ); create view viewdoc1 (c1,fname,lname,age,street) as SELECT c1, bson_value_lvarchar(data,"fname"), bson_value_lvarchar(data,"lname"), bson_value_double(data,"age"), bson_value_lvarchar(data,"address.street") FROM doc1; > select * from viewdoc1; c1 1 fname John lname Miller age 21.00000000000 street Informix ave c1 2 fname Scott lname Lashley age 21.50000000000 street Blazer ave 2 row(s) retrieved.
  37. 37. © 2014 IBM Corporation Informix 12.1 3737 IWA data mart view support to accelerate SQL-NoSQL query (2)  Probe the query we want to accelerate – Use workload analysis to find datamart for a query joining SQL table with NoSQL data (JSON collection) – Deploy datamart proposed: Example: Accelerating a query combining SQL with NoSQL data select a.value,a.name,b.* from sqldoc1 a, viewdoc1 b where a.value=b.c1  Run the query with acceleration. Works fine: ... <mart name="dm_sqlnosql1"> <table name="sqldoc1" schema="root" isFactTable="true"> <column name="name"/> <column name="value"/> </table> <table name="viewdoc1" schema="root" isFactTable="false"> <column name="age"/> <column name="c1"/> <column name="fname"/> <column name="lname"/> <column name="street"/> </table> <reference referenceType="LEFTOUTER" isRuntimeJoin="true" parentCardinality="n" dependentCardinality="n" dependentTableSchema="root" dependentTableName="sqldoc1" parentTableSchema="root" parentTableName="viewdoc1"> <parentColumn name="c1"/> <dependentColumn name="value"/> </reference> </mart> </dwa:martModel> set environment use_dwa '3'; Environment set. select a.value,a.name,b.* from sqldoc1 a, viewdoc1 b where a.value=b.c1 ; value 1 name John c1 1 fname John lname Miller age 21.00000000000 street Informix ave value 2 name Scott c1 2 fname Scott lname Lashley age 21.50000000000 street Blazer ave Online.log: 01:22:11 SQDWA: select a.value,a.name,b.* from sqldoc1 a, viewdoc1 b where a.value=b.c 01:22:11 SQDWA: Identified 1 candidate AQTs for matching 01:22:11 SQDWA: matched: aqt46221ac7-8eae-4b9a-9275- bd00dcca357c 01:22:11 SQDWA: matching successful (17 msec) aqt46221ac7-8eae-4b9a-9275-bd00dcca357c 01:22:11 SQDWA: offloading successful (3036 msec)
  38. 38. © 2014 IBM Corporation Informix 12.1 3838  Use workload analysis to find best IWA data mart to accelerate these 3 queries: – Self-Join using a view (v1) on the table (employee): – Query on a view to filter rows or columns on the table: – Query on the base table: IWA data mart supports of views and usage for Self Join queries (1)  Create 2 views on a base table: – One of the views (v1) will be used to implement a self join of table ‘employee’ with itself. create table "root".employee ( employeeid integer, lastname varchar(20), country varchar(20), departmentid integer ); create view "root".v1 (employeeid,lastname,country,departmentid) as select x0.employeeid ,x0.lastname ,x0.country ,x0.departmentid from "root".employee x0 ; create view "root".v2 (employeeid,lastname,country,departmentid) as select x0.employeeid ,x0.lastname ,x0.country ,x0.departmentid from "root".employee x0 where (x0.country = 'United States' ) ; Example: Accelerating a self-join query, and queries on a table and a given view SELECT Employee.EmployeeID, Employee.LastName, Employee.EmployeeID, v1.LastName, Employee.Country FROM Employee INNER JOIN v1 -- self-join ON Employee.Country = v1.Country; SELECT EmployeeID, LastName, EmployeeID FROM v2; SELECT EmployeeID FROM Employee;
  39. 39. © 2014 IBM Corporation Informix 12.1 3939 IWA data mart supports of views and usage for Self Join queries (2)  Deploy IWA data mart produced – Contains the base table (employee), views (v1) and (v2), and self-join column relationship (employee, v1)  Run the queries enabling Acceleration: – Run the accelerated query using view v1 for self-join: – Run accelerated queries on the view v2 and base table alone: Example: Accelerating a Self-Join query using Views <?xml version="1.0" encoding="UTF-8" ?> <dwa:martModel xmlns:dwa="http://www.ibm.com/xmlns/prod/dwa" version="1.0"> <mart name="dm_empv1"> <table name="employee" schema="root" isFactTable="true"> <column name="country"/> <column name="employeeid"/> <column name="lastname"/> </table> <table name="v1" schema="root" isFactTable="false"> <column name="country"/> <column name="lastname"/> </table> <table name="v2" schema="root" isFactTable="true"> <column name="employeeid"/> <column name="lastname"/> </table> <reference referenceType="LEFTOUTER" isRuntimeJoin="true" parentCardinality="n" dependentCardinality="n" dependentTableSchema="root" dependentTableName="employee" parentTableSchema="root" parentTableName="v1"> <parentColumn name="country"/> <dependentColumn name="country"/> </reference> </mart> SET ENVIRONMENT use_dwa ‘3’; SELECT Employee.EmployeeID, Employee.LastName, Employee.EmployeeID, v1.LastName, Employee.Country FROM Employee INNER JOIN v1 -- self-join ON Employee.Country = v1.Country; SET ENVIRONMENT use_dwa ‘3’; SELECT EmployeeID, LastName, EmployeeID FROM v2; SELECT EmployeeID FROM Employee;
  40. 40. © 2014 IBM Corporation Informix 12.1 4040 IWA data mart supports of views and usage for Self Join queries (3)  From Informix: – Run the accelerated query using view v1 for self-join: – Run accelerated queries on the view v2 and base table alone: Example: Accelerating a Self-Join query using Views SET ENVIRONMENT use_dwa ‘3’; SELECT Employee.EmployeeID, Employee.LastName, Employee.EmployeeID, v1.LastName, Employee.Country FROM Employee INNER JOIN v1 -- self-join ON Employee.Country = v1.Country; SET ENVIRONMENT use_dwa ‘3’; SELECT EmployeeID, LastName, EmployeeID FROM v2; SELECT EmployeeID FROM Employee;
  41. 41. © 2014 IBM Corporation Informix 12.1 4141 sandor.szabo@de.ibm.com
  42. 42. © 2014 IBM Corporation Informix 12.1 42 Handling Big Data without angst
  43. 43. © 2014 IBM Corporation Informix 12.1 4343
  44. 44. © 2014 IBM Corporation Informix 12.1 4444 Logo
  45. 45. © 2014 IBM Corporation Informix 12.1 4545 Logo

×