Informix Warehouse Accelerator    Query Flow and Matching           Keshava Murthy,Architect, IBM Informix Development    ...
IBM Smart AnalyticsStep 1. Install, configure,                                                                           S...
Connecting to Informix• For data mart design, from ISAO Studio  – Use 11.5 Informix driver  – Protocol tcp/ip (onsoctcp or...
Connection to Informix• For data mart design, from ISAO Studio• ISAO Studio runs on Windows and Linux• Connect from these ...
Connection to Informix• For Applications, connect as usual.• No Application changes/redeployment necessary• Set the enviro...
Connection to Informix• USE_DWASET ENVIRONMENT USE_DWA ‘1’;  – Controls the session behavior of query matching.  – ‘0’ (ze...
Adding Accelerator                                                                           IBM Smart AnalyticsStep 1. In...
Adding Accelerator• Add new accelerator from data studio or  command line interface (CLI)• Need four parameters to add acc...
Adding Accelerator• Informix always talks to IWA Coordinator  – For all data mart operations  – Queries  – To obtain the r...
Design, Validate, Deploy Data mart                                                                           IBM Smart Ana...
Design, Validate, Deploy Data marts                                                                         Step 5. Save t...
Store Sales ER-Diagram from TPC-DS                                                 300GB database     73,049              ...
12
Designing data mart• Start with a good logical and physical design• Typically has Star or Snowflake schema• Data mart itse...
Designing data mart• Design identifies and uses existing PK-FK  relationship between the tables• In warehouse environment,...
Designing data mart• When you don’t have PK-FK relationship  – Identify the keys from logical design  – Identify the keys ...
Designing data mart• Single Data mart with multiple fact tables  – Shares the dimensions with all• Multiple data marts eac...
Designing data mart – Smart mart tool• Simply enable workload analysis• Run the workload• Informix will give you data mart...
Deploying the data mart• Creates and sends the data mart definition to IWA• Verify the fact tables and dimension tables.• ...
Deploying the data mart• IWA returns one or more SQL statements  representing the data mart.• Informix creates Accelerated...
Design, Validate, Deploy Data mart                                                                           IBM Smart Ana...
Loading the data mart•   Load the data mart using Studio•   Load using loadMart command from CLI•   Takes snapshot of the ...
Query Flow                        Step 1. Submit SQL                        DB protocol: SQLI or DRDA                     ...
Query Flow within IWA  Step1  SQL from Informix                                          Step5: Send the results          ...
Life of a querySQL Statement                            Explain File                SemanticSQL Parser                   O...
Optimizer is enhanced to                                                 do the query matchingSQL Statement               ...
26
create view "dwa"."aqt2dbca0d9-509d-434b-9cc9-4a12c6de6b3d""COL20","COL21","COL22","COL23","COL24","COL25","COL26","COL27"...
Content of the view•The data mart schema should be star or snowflake schema   • Single table data mart is fine (e.g. weblo...
Query Matching•Fact table should be used in the query•Dimensions should be joined using the join keys in the data mart•Sup...
Create the table           create table kfact(id int, name varchar(32), amount decimal(9,2));Create the data martDatamart ...
QUERY: (ISAO-Executed)(OPTIMIZATION TIMESTAMP: 05-16-2011 09:10:37)------select count(*) from kfactEstimated Cost: 1Estima...
select id, name, sum(amount) from kfact group by id, nameEstimated Cost: 4Estimated # of Rows Returned: 1Maximum Threads: ...
Thank You            33
Acceleration with Informix Warehouse Accelerator Smart Anlaytics           1. Identify the datamart                       ...
Acceleration with Informix Warehouse Accelerator                        Step 1. Submit SQL                        DB proto...
Upcoming SlideShare
Loading in...5
×

Informix Warehouse accelerator -- design, deploy, use

884

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
884
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
17
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Informix Warehouse accelerator -- design, deploy, use

  1. 1. Informix Warehouse Accelerator Query Flow and Matching Keshava Murthy,Architect, IBM Informix Development 0
  2. 2. IBM Smart AnalyticsStep 1. Install, configure, Studiostart InformixStep 2. Install, configure, Step 3start Accelerator Step 1Step 3. Connect Studio toInformix & add accelerator Step 4 Informix Database ServerStep 4. Design, validate,Deploy Data mart Step 5Step 5. Load data toacceleratorReady for Queries BI Applications Step 2 Ready Informix warehouse Accelerator 1
  3. 3. Connecting to Informix• For data mart design, from ISAO Studio – Use 11.5 Informix driver – Protocol tcp/ip (onsoctcp or ontlitcp) – Use the port with TCP/IP and SQLI protocol• From informix applications, scripts, tools – Supports protocols: tcp/ip, shared memory – Supports all drivers – CSDK, ODBC, JDBC, JCC, .NET, etc, etc. 2
  4. 4. Connection to Informix• For data mart design, from ISAO Studio• ISAO Studio runs on Windows and Linux• Connect from these two platforms to any supported Informix server – Linux64/Intel – HP-UX/Itanium – Power/AIX – Sparc/Solaris 3
  5. 5. Connection to Informix• For Applications, connect as usual.• No Application changes/redeployment necessary• Set the environments (USE_DWA) using sysdbopen() procedure• sysdbopen() procedure is automatically executed when any application connects to a database. 4
  6. 6. Connection to Informix• USE_DWASET ENVIRONMENT USE_DWA ‘1’; – Controls the session behavior of query matching. – ‘0’ (zero) turns off using IWA for query processing – ‘1’ turns on considering IWA – ‘3’ same as 1 with diagnostics – ‘998’ Use IWA only. 5
  7. 7. Adding Accelerator IBM Smart AnalyticsStep 1. Install, configure, Studiostart InformixStep 2. Install, configure, Step 3start Accelerator Step 1Step 3. Connect Studio toInformix & add accelerator Step 4 Informix Database ServerStep 4. Design, validate,Deploy Data mart Step 5Step 5. Load data toacceleratorReady for Queries DRDA over TCP/IP BI Applications Step 2 Ready Informix warehouse Accelerator 6
  8. 8. Adding Accelerator• Add new accelerator from data studio or command line interface (CLI)• Need four parameters to add accelerator – Name of the accelerator (you choose) – IP address of the IWA instance – Port on which IWA is listening to – PIN obtaining after executing ‘ondwa getpin’• Port number is in dwainst.conf file. 7
  9. 9. Adding Accelerator• Informix always talks to IWA Coordinator – For all data mart operations – Queries – To obtain the resultset.• Informix treats IWA Coordinator as remote node 8
  10. 10. Design, Validate, Deploy Data mart IBM Smart AnalyticsStep 1. Install, configure, Studiostart InformixStep 2. Install, configure, Step 3start Accelerator Step 1Step 3. Connect Studio toInformix & add accelerator Step 4Step 4. Design, validate, Informix Database ServerDeploy Data mart Step 5Step 5. Load data toacceleratorReady for Queries BI Applications Step 2 Ready Informix warehouse Accelerator 9
  11. 11. Design, Validate, Deploy Data marts Step 5. Save the Definition Step 1. Design, Validate the data mart. InformixISAO Studio or Step 2. Deploy Data mart AQT CLI Tool AQT Step 6. Return acknowledgement Step 3 Step 4 Send the data mart Return the SQL definition definitions Coordinator Worker Worker Worker Worker Compressed Compressed Compressed Compressed data data data data In memory In memory In memory In memory Memory image Memory image Memory image on disk Memory image on disk on disk on disk 10
  12. 12. Store Sales ER-Diagram from TPC-DS 300GB database 73,049 402 204,000 287,997,024 86,400 1000 1,920,800 1,000,000 7200 20 2,000,000 11
  13. 13. 12
  14. 14. Designing data mart• Start with a good logical and physical design• Typically has Star or Snowflake schema• Data mart itself can contains – One or more fact tables – Available dimensions – Relationship between the fact and dimensions• Relationships – 1:n relationship -- needs unique constraint on PK – n:m relationshp 13
  15. 15. Designing data mart• Design identifies and uses existing PK-FK relationship between the tables• In warehouse environment, it’s typical not to have constraints defined within the schema• Manually create the relationships between the tables.• Always start from the Parent and end with Child – In customer, web_sales relationship, customer is the parent and web_sales is the child. – customer.customer_id will be the primary key, web_sales.customer_id will be the foreign key. 14
  16. 16. Designing data mart• When you don’t have PK-FK relationship – Identify the keys from logical design – Identify the keys from equi-join keys in queries – Identify the parent and child• Type of Relationships between two tables – Single relationship with single key – Single relationship with multiple keys – Multiple relationship with single or multiple keys 15
  17. 17. Designing data mart• Single Data mart with multiple fact tables – Shares the dimensions with all• Multiple data marts each with its own fact table, but same fact tables – Separate copy of dimension tables – Higher memory requirement 16
  18. 18. Designing data mart – Smart mart tool• Simply enable workload analysis• Run the workload• Informix will give you data mart definitions required to run the workload• Design is done for you based on workload• Simply deploy and load the mart using this definition• Useful while generating data mart for standard reports• Use it as guiding tool for identifying tables needed within warehouses. 17
  19. 19. Deploying the data mart• Creates and sends the data mart definition to IWA• Verify the fact tables and dimension tables.• Generate the report and verify when necessary• You can load the data when deploying the data mart• Typically you deploy once and load periodically• Loading can be automated via command line inerface (CLI) 18
  20. 20. Deploying the data mart• IWA returns one or more SQL statements representing the data mart.• Informix creates Accelerated Query Tables (AQT) for those.• AQTs are essentially views used exclusively for query matching• Data mart deployment, enable, disable, drop events are recorded in the system catalog 19
  21. 21. Design, Validate, Deploy Data mart IBM Smart AnalyticsStep 1. Install, configure, Studiostart InformixStep 2. Install, configure, Step 3start Accelerator Step 1Step 3. Connect Studio toInformix & add accelerator Step 4 Informix Database ServerStep 4. Design, validate,Deploy Data mart Step 5Step 5. Load data toacceleratorReady for Queries BI Applications Step 2 Ready Informix warehouse Accelerator 20
  22. 22. Loading the data mart• Load the data mart using Studio• Load using loadMart command from CLI• Takes snapshot of the table• Options – No locking of the tables – Locking of all the tables 21
  23. 23. Query Flow Step 1. Submit SQL DB protocol: SQLI or DRDA Informix Network : TCP/IP,SHMApplications 2. Query matching and BI Tools redirection technology Local Step 5. Return results/describe/error Execution Database protocol: SQLI or DRDA Network : TCP/IP, SHM Step 3 Step 4 offload SQL. Results: DRDA over TCP/IP DRDA over TCP/IP Coordinator Worker Worker Worker Worker Compressed Compressed Compressed Compressed data data data data In memory In memory In memory In memory Memory image Memory image Memory image on disk Memory image on disk on disk on disk 22
  24. 24. Query Flow within IWA Step1 SQL from Informix Step5: Send the results back to Infomrix server Step2 Send the queries to all the Step4: merge intermediate workers Coordinator results, ORDER BY, FIRSTN Worker Worker Worker WorkerCompressed data Compressed data Compressed data Compressed data In memory In memory In memory In memoryStep3: Scan, Filter, Step3: Scan, Filter, Step3: Scan, Filter, Step3: Scan, Filter, join, group join, group join, group join, group 23
  25. 25. Life of a querySQL Statement Explain File SemanticSQL Parser Optimizer Query Plan Analyzer System Table Stats & Executor Catalog Column Information Distribution Query Query stats Results 24
  26. 26. Optimizer is enhanced to do the query matchingSQL Statement Query qualified for Explain File acceleration Semantic GenerateSQL Parser Optimizer Query Plan SQL-- Analyzer Informix IWA System Table Stats & Execution Execution Catalog Column Information Distribution Query Query Results Results Query stats 25
  27. 27. 26
  28. 28. create view "dwa"."aqt2dbca0d9-509d-434b-9cc9-4a12c6de6b3d""COL20","COL21","COL22","COL23","COL24","COL25","COL26","COL27","COL28","COL29","CO 8","COL39","COL40","COL41","COL42","COL43","COL44","COL45","COL46","COL47","COL07"OL48","COL49","COL50","COL51","COL52","COL53","COL54","COL55","COL56","COL57","COL5 ","COL02","COL03","COL04","COL05","COL06","COL62","COL63","COL64…) as select x0.perkey ,x0.storekey ,x0.custkey ,x0.prodkey ,x0.promokey ,x0.quantity_sold ,x0.extended_price ,x0.extended_cost ,x0.shelf_location ,x0.shelf_number ,x0.start_shelf_date ,x0.shelf_height ,x0.shelf_width ,x0.shelf_depth ,x0.shelf_cost ,x0.shelf_cost_pct_of_sale ,x0.bin_number ,x0.product_per_bin ,x0.start_bin_date ,x0.bin_height ,x0.bin_width ,x0.bin_depth ,x0.bin_cost ,x0.bin_cost_pct_of_sale …… from ((((("informix".daily_sales x0 left join "informix".period x1 on (x0.perkey = x1.perkey ) )left join "informix".product x2 on (x0.prodkey = x2.prodkey ) )left join "informix"."store" x3 on (x0.storekey = x3.storekey ) )left join "informix".customer x4 on (x0.custkey = x4.custkey ) )left join "informix".promotion x5 on (x0.promokey = x5.promokey ) ); 27
  29. 29. Content of the view•The data mart schema should be star or snowflake schema • Single table data mart is fine (e.g. weblog, call detail record)•The view created represents the whole data mart •All the selected columns from all tables •The join predicate between the fact and dimension and dimension to dimension. 28
  30. 30. Query Matching•Fact table should be used in the query•Dimensions should be joined using the join keys in the data mart•Supported functions, expressions and aggregates•INNER JOIN, LEFT OUTER JOIN with fact on the dominant side•Cannot reference tables outside the data mart 29
  31. 31. Create the table create table kfact(id int, name varchar(32), amount decimal(9,2));Create the data martDatamart definition in the database… saved as a special viewcreate view "dwa"."aqtf5246230-8cce-42fd-8c3e-f516bbeacca3" ("COL1","COL2","COL3")as select x0.id ,x0."name" ,x0.amount from "keshav".kfact x0 ; 30
  32. 32. QUERY: (ISAO-Executed)(OPTIMIZATION TIMESTAMP: 05-16-2011 09:10:37)------select count(*) from kfactEstimated Cost: 1Estimated # of Rows Returned: 1Maximum Threads: 01) tpcds_100gb@DWAFINAL:dwa.aqtf5246230-8cce-42fd-8c3e-f516bbeacca3: REMOTE PATH Remote SQL Request: {QUERY {FROM dwa.aqtf5246230-8cce-42fd-8c3e-f516bbeacca3} {SELECT {count(*) } } } 31
  33. 33. select id, name, sum(amount) from kfact group by id, nameEstimated Cost: 4Estimated # of Rows Returned: 1Maximum Threads: 0 1) tpcds_100gb@DWAFINAL:dwa.aqtf5246230-8cce-42fd-8c3e-f516bbeacca3: REMOTEPATH Remote SQL Request: {QUERY {FROM dwa.aqtf5246230-8cce-42fd-8c3e-f516bbeacca3} {SELECT {SYSCAST COL1 AS INTEGER NULLABLE} {SYSCAST COL2 AS VARCHAR 32 819} {SUM COL3 } } {GROUP COL1 COL2 } } 32
  34. 34. Thank You 33
  35. 35. Acceleration with Informix Warehouse Accelerator Smart Anlaytics 1. Identify the datamart to offload. Informix Data Studio 4. Create the metadata5. Issue Off-load Datamart command 3. Return the SQL 2. Datamart Definition representation 9. Return ACK 6. Off-load the dataInformixWarehouseAccelerator Coordinator process7. Distribute the data among workers8. Compress the data Worker Processes 34
  36. 36. Acceleration with Informix Warehouse Accelerator Step 1. Submit SQL DB protocol: SQLI or DRDA Applications Network : TCP/IP,SHM BI Tools Informix 2. IDS query matching Local and redirection Execution technology Step 5. Return results/describe/error Database protocol: SQLI or DRDA Network : TCP/IP, SHM Step 3 Step 4 offload SQL. DRDA over tcp/ip Results: DRDA over tcp/ip Coordinator process Worker Processes 35
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×