• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Informix Warehouse Accelerator (IWA) features in version 12.1
 

Informix Warehouse Accelerator (IWA) features in version 12.1

on

  • 684 views

Informix Warehouse Accelerator (IWA) features in version 12.1

Informix Warehouse Accelerator (IWA) features in version 12.1

Statistics

Views

Total Views
684
Views on SlideShare
680
Embed Views
4

Actions

Likes
0
Downloads
0
Comments
0

2 Embeds 4

https://twitter.com 3
https://abs.twimg.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • execute function dropPartMart(’myAccelerator’,’myMart’,’user10’,’tab22’,’part1’); execute function loadPartMart(’myAccelerator’,’myMart’,’user10’,’tab22’,’part1’);
  • execute function dropPartMart(’myAccelerator’,’myMart’,’user10’,’tab22’,’part1’); execute function loadPartMart(’myAccelerator’,’myMart’,’user10’,’tab22’,’part1’);
  • IBM IOD 2011 04/25/13 Prensenter name here.ppt

Informix Warehouse Accelerator (IWA) features in version 12.1 Informix Warehouse Accelerator (IWA) features in version 12.1 Presentation Transcript

  • IWA Enhancements in 12.10Keshava MurthyIBM Informix Development
  • Informix Warehousing Moving Forward Goal is to provide a comprehensive warehousing platform thatis highly competitive in the marketplace Incorporating the best features of XPS and Red Brick intoInformix for OLTP/Warehousing and Mixed-Workload Using the latest Informix technology in: Continuous Availability and Flexible Grid Data Warehouse Accelerator using latest industrytechnology Integration of IBM’s BI software stack
  • 3Informix WarehouseFeature- SQW- Data Modeling- ELT/ETLInformix Warehouse withStorageOptimization/CompressionCognos integration- Native Content Store on InformixSQL MergeInformix Warehouse: RoadmapExternal TablesStar Join OptimizationMulti-index ScanNew FragmentationFragment Level StatsStorage ProvisioningWarehouseAcceleratorOLAPQuery RewriteHash JoinEnhance IWA+++11.5xC311.5xC611.7011.70xC211.5xC411.5xC512.10
  • 4IWA 1stReleaseOn SMPSMB: IGWEScale out: IWAon Blade ServerWorkload Analysis ToolMore LocalesData CurrencyIWA: RoadmapPartition RefreshMACH11 supportSolaris on IntelTrickle FeedUnion queriesDerived tablesOAT IntegrationSQL/OLAP for IWA11.7xC211.7xC512.1xC111.7xC311.7xC42012 IIUG2012 IIUGSupport for InformixTimeseries11.7xC7
  • Informix PublicationsBulletin of the Technical Committee on Data Engineering: March 2012 Vol. 35 No. 1Real Time Business Intelligence. September 2, 2011 - Seattle, United StatesIBM Data management Magazine: Supercharging thedata wharehouse while keeping the costs down.2012 Bloor Report: IBM Informix in hybrid workload environments2012 Ovum Analyst report: Informix Accelerates Analytic Integration into OLTPDBTA Article: Empowering Business Analysts with Faster Insightshttp://youtu.be/xJd8M-fbMI0
  • 6OLAP Functions in Informix09/12/2012Bingjie Miao, bingjie@us.ibm.com
  • What is OLAP?• On-Line Analytical Processing• Commonly used in BusinessIntelligence (BI) tools– ranking products, salesmen, items, etc– exposing trends in sales from historic data– testing business scenarios (forecast)– sales breakdown or aggregates on multipledimensions (Time, Region, Demographics,etc)
  • OLAP Functions in Informix• Supports subset of commonly usedOLAP functions• Enables more efficient query processingfrom BI tools such as Cognos
  • Example query with group byselect customer_num, count(*)from orderswhere customer_num <= 110group by customer_num;customer_num (count(*))101 1104 4106 2110 24 row(s) retrieved.
  • Example query with OLAP functionselect customer_num, ship_date, ship_charge,count(*) over (partition by customer_num)from orderswhere customer_num <= 110;customer_num ship_date ship_charge (count(*))101 05/26/2008 $15.30 1104 05/23/2008 $10.80 4104 07/03/2008 $5.00 4104 06/01/2008 $10.00 4104 07/10/2008 $12.20 4106 05/30/2008 $19.20 2106 07/03/2008 $12.30 2110 07/06/2008 $13.80 2110 07/16/2008 $6.30 29 row(s) retrieved.
  • Supported OLAP Functions• Ranking functions– RANK, DENSE_RANK (DENSERANK)– PERCENT_RANK, CUME_DIST, NTILE– LEAD, LAG• Numbering functions– ROW_NUMBER (ROWNUMBER)• Aggregate functions– SUM, COUNT, AVG, MIN, MAX– STDEV, VARIANCE, RANGE– FIRST_VALUE, LAST_VALUE– RATIO_TO_REPORT (RATIOTOREPORT)
  • Ranking Functions• Partition by clause is optional• Order by clause is required• Window frame clause is NOT allowed• Duplicate value handling is differentbetween rank() and dense_rank()– same rank given to all duplicates– next rank used “skips” ranks already covered byduplicates in rank(), but uses next rank fordense_rank()
  • Where does OLAP function fit?Joins, group by,havingOLAP functionsFinal order by
  • ApplicationsBI ToolsStep 1. Submit SQLDB protocol: SQLI or DRDANetwork : TCP/IP,SHMInformix2. Query matching andredirection technologyStep 3offload SQL.DRDA over TCP/IPStep 4Results:DRDA over TCP/IPLocalExecutionCoordinatorCompresseddataIn memoryWorkerMemory imageon diskCompresseddataIn memoryWorkerMemory imageon diskCompresseddataIn memoryWorkerMemory imageon diskCompresseddataIn memoryWorkerMemory imageon diskStep 6. Return results/describe/errorDatabase protocol: SQLI or DRDANetwork : TCP/IP, SHMQuery Flow5. Feed results to OLAPiterators if it exists
  • QUERY: DWA executed:(OPTIMIZATION TIMESTAMP: 10-02-2012 20:52:56)------select ws_web_page_sk, ws_net_paid, avg(ws_net_paid) over()from web_saleswhere ws_web_page_sk < 10group by ws_web_page_sk, ws_net_paidorder by ws_web_page_sk, ws_net_paidEstimated Cost: 1497990Estimated # of Rows Returned: 286309Temporary Files Required For: Order By Group By1) ds2@BVSRDWA:dwa.aqt5cbe4c46-acdc-463a-a9cb-2c3318cc9164: REMOTE DWA PATHRemote SQL Request:{QUERY {FROM dwa.aqt5cbe4c46-acdc-463a-a9cb-2c3318cc9164} {WHERE {< COL066 10 } } {SELECT {SYSCASTCOL066 AS INTEGER NULLABLE} {SYSCAST COL083 AS DECIMAL 7 2 NULLABLE} } {GROUP COL066 COL083 } }Query statistics:-----------------Table map :----------------------------Internal name Table name----------------------------type rows_prod est_rows time est_cost-------------------------------------------------dwa 307576 0 00:03.62 0type it_count time----------------------------olap 307576 00:04.48type rows_sort est_rows rows_cons time est_cost------------------------------------------------------------sort 307576 286309 307576 00:06.49 192262
  • OLAP Performance Improvements
  • Union Query Supportselect sum(sales_amt) from SALESUNION ALLselect sum(returns_amt) from SALES_RETURN;
  • Derived table Queryselect state, totsales, totreturnsfrom (select state, sum(sale_amt) from sales)as stsales(state, totsales),(select state, sum(return_amt) from sales_returns)as streturns(state, totreturns)Where stsales.state = streturns.state;
  • SELECT d_year,i_brand_id,i_class_id,i_category_id ,i_manufact_id,SUM(sales_cnt) AS sales_cnt ,SUM(sales_amt) AS sales_amtFROM(SELECT d_year ,i_brand_id ,i_class_id ,i_category_id,i_manufact_id,SUM(sales_cnt) AS sales_cnt,SUM(sales_amt) AS sales_amtFROM (SELECT d_year ,i_brand_id ,i_class_id ,i_category_id ,i_manufact_id ,cs_quantity AS sales_cnt,cs_ext_sales_price AS sales_amtFROM catalog_sales JOIN item ON i_item_sk=cs_item_skJOIN date_dim ON d_date_sk=cs_sold_date_skWHERE i_category=MusicUNIONSELECT d_year ,i_brand_id ,i_class_id ,i_category_id ,i_manufact_id,ss_quantity AS sales_cnt,ss_ext_sales_price AS sales_amtFROM store_sales JOIN item ON i_item_sk=ss_item_skJOIN date_dim ON d_date_sk=ss_sold_date_skWHERE i_category=BooksUNIONSELECT d_year,i_brand_id,i_class_id,i_category_id,i_manufact_id,ws_quantity AS sales_cnt,ws_ext_sales_price AS sales_amtFROM web_sales JOIN item ON i_item_sk=ws_item_skJOIN date_dim ON d_date_sk=ws_sold_date_skWHERE i_category=Sports) sales_detailGROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id) as tmpGROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_idORDER BY sales_amt, sales_cnt
  • Query statistics:Table map :----------------------------Internal name Table name----------------------------t1 (Temp Table For Collection Subquery)t2 (Temp Table For Collection Subquery)type rows_prod est_rows time est_cost-------------------------------------------------dwa 1410644 0 00:06.38 0type rows_prod est_rows time est_cost-------------------------------------------------dwa 2749278 0 00:12.53 0type rows_prod rows_cons_1 rows_cons_2 time------------------------------------------------------merge 4159922 1410644 2749278 00:19.38type rows_prod est_rows time est_cost-------------------------------------------------dwa 723063 0 00:08.01 0type rows_prod rows_cons_1 rows_cons_2 time------------------------------------------------------merge 4882985 4159922 723063 00:28.11type rows_sort est_rows rows_cons time-------------------------------------------------sort 4867550 0 4882985 01:22.97type table rows_prod est_rows rows_scan time est_cost-------------------------------------------------------------------scan t1 4867550 4320015 4867550 00:04.59 190744type rows_prod est_rows rows_cons time est_cost------------------------------------------------------------group 77949 1769089 4867550 00:52.63 11610464type table rows_prod est_rows rows_scan time est_cost-------------------------------------------------------------------scan t2 77949 1769089 77949 00:00.08 99698type rows_prod est_rows rows_cons time est_cost------------------------------------------------------------group 77949 724460 77949 00:01.20 4647282type rows_sort est_rows rows_cons time est_cost------------------------------------------------------------sort 77949 724460 77949 00:01.77 709964
  • UNION and Derived QueryPerformance Improvements
  • SQL Enhancements• OLAP Window functions/aggregates• Multiple distinct aggregates• Distinct with CASE expression• NULLS FIRST, NULLS LAST modifier to ORDER BY
  • Support for custom NULL sorting• Informix has NULLS FIRST by default and cannot bechanged.• NULLS FIRST, NULLS LAST are modifiers to ORDER BYclause• Oracle supports both.• Helps avoid sorting in Cognos. Cognos used this for somereports against Oracle.SELECT c1, c2, sum(c3)FROM t1GROUP BY c1, c2ORDER BY c2 NULLS LAST, C1 NULLS FIRST;
  • DISTINCT with CASE expression• Support for CASE expression as argument toaggregates was added in 11.70.• 12.10 adds support for distinct on CASE expressionSELECT sum(T983271.set_avgday_sales_rtl_amt) as c2,count(distinct case when T983271.not_set_cnt > 0then T983271.store_sk_id end ) as c3FROM features_upc_tab T983271;
  • Multiple aggregates with distincts• Long requested by Cognos,Wal-Mart and others.• Wal-Mart had run into this during IWA pilot project.• Design and part of the code taken from XPS.• Long live XPS!select region, sum(distinct cid), avg(distinct salesdt)From sales_tab;
  • Informix Database ServerInformix warehouse AcceleratorBI ApplicationsStep 1. Install, configure,start InformixStep 2. Install, configure,start AcceleratorStep 3. Connect Studio toInformix & add acceleratorStep 4. Design, validate,Deploy Data martStep 5. Load data toacceleratorReady for QueriesIBM Smart AnalyticsStudioStep 1Step 2Step 3Step 4Step 5ReadyInformix Ultimate Warehouse edition
  • Informix Database ServerBI ApplicationsStep 1. Create the Sales-Martand load it. Sales is the facttable -- range partitioned.Step 2. Load jobsupdate the fact table “sales”Only updates existing partitionStep 3. Identify the partition,execute dropPartMart().Step 4. for same partition,execute loadPartMart().Ready for QueriesIBM Smart AnalyticsStudio or storedprocedures orcommand line toolStep 1Step 4Step 2Step 3ReadyCase 1: Partition refresh: Updates to existing PartitionsSales-MartsalescustomerstoresIWAOLTP Appspartitioned fact tableSQL Script: callStored procedureModified partitionINSERT, UPDATE, DELETE
  • Informix Database ServerBI ApplicationsStep 1. Create the Sales-Martand load it. Sales is the facttable -- range partitioned.Need to move the Timewindow to next range.ep 2. DETACH operationExecute dropPartMart()DETACH the partitionep 3. ATTACH operationATTACH the partitionExecute loadPartMart()Ready for QueriesIBM Smart AnalyticsStudio or storedprocedures orcommand line toolStep 1Step 3Step 2ReadyCase 2: Partition refresh: Time Cyclic data managementSales-MartsalescustomerstoresIWAOLTP Appspartitioned fact tableMove the window.
  • dropPartMart() procedure1. Uses the accelerator name, datamart name, tablename and partition name.Partition name can be the name of the partition orpartition number (sysfragments.partn)The partition name or number should be a validpartition for the table.Call dropPartMart() first before doing the DEATCH
  • loadPartMart() procedure1. Uses the accelerator name, datamart name, tablename and partition name.Partition name can be the name of the partition orpartition number (sysfragments.partn)The partition name or number should be a validpartition for the table.ATTACH the partition first, before callingloadPartMart().
  • 31Informix Database ServerInformix warehouse AcceleratorBI ApplicationsStep 1. Install, configure,start InformixStep 2. Install, configure,start AcceleratorStep 3. Connect Studio toInformix & add acceleratorStep 4. Design, validate,Deploy Data martStep 5. Load data toacceleratorReady for QueriesIBM Smart AnalyticsStudioStep 1Step 2Step 3Step 4Step 5ReadyInformix Warehouse Accelerator – In 11.70.FC4
  • Background• Prior to 11.70.FC5, adding accelerator, create, deploy,load, enable, disable datamart, accelerating queries – areall operations officially supported only on Standardserver or Primary node of MACH11/HA environment.• We estimate about 50% of Informix customers use HDRsecondary servers and growing number of customers useMACH11 (SDS secondary) configurations and RSS nodes.MACH11 is the Informix scale out solution.• IWA itself supports a scale out solution (on a cluster)starting with 11.70.FC4.• Reasons to support MACH11 and IWA together.– This feature will enable partitioning a cluster or HA groupbetween OLTP and BI workload.– This feature will give help to off-load the expensive LOADfunctionality to secondary servers– We have customers now requesting support for HDR secondaryto IWA
  • 33Informix PrimaryInformix warehouse AcceleratorBI ApplicationsStep 1. Install, configure,start InformixStep 2. Install, configure,start AcceleratorStep 3. Connect Studio toInformix & add acceleratorStep 4. Design, validate,Deploy Data mart fromPrimary, SDS, HDR, RSSStep 5. Add IWA to sqlhostsLoad data toAccelerator from any node.Ready for QueriesIBM Smart AnalyticsStudioStep 1Step 3Step 4Step 5ReadyInformix Warehouse Accelerator – 11.70.FC5. MACH11 SupportInformixSDS1InformixSDS2InformixHDRSecondaryInformixRSSStep 2
  • Step 1: Install:• Informix and IWA are installed just like before.• Informix can be combination of standard,primary, SDS, HDR secondary and RSS nodes.• IWA can be installed on the same computer as anyone of the nodes or on distinct computer.• IWA can also be installed on a cluster hardwarewith multiple worker nodes for scale outperformance.
  • Step 2: Configure• Informix and IWA are installed just like before.• Informix can be combination of standard,primary, SDS, HDR secondary and RSS nodes.• IWA can be installed on the same computer as anyone of the nodes or on distinct computer.• IWA can also be installed on a cluster hardwarewith multiple worker nodes for scale outperformance.• Note: Informix MACH11 technology works withlogged and ANSI databases only.
  • •The secondary servers should be updatablesecondary servers.•Set this in $ONCONFIGUPDATABLE_SECONDARY 10Step 2: Configure
  • Step 3: Connect• You can connect to IWA from Informix from any of theInformix servers using existing method.– Get the connection details via:# ondwa getpin– The output will be, ip address, port, pin for IWA connection.– Use that information to create the connection.• After successful connection from Informix to IWA, theSQLHOSTS will have something like thisFAST group - -c=1,a=484224232041684420473a283e612f74393e6025757159506a51344a6b4e2f2d2d47455e6b653f2f6c795f287d7b65224d6c3c2f65722e6a2a4245397b3b447d572c3129696b306440FAST_1 dwsoctcp 172.34.22.188 21022 g=FAST• To use this connection on any of Informix nodes, copy these lines AS IS to theSQLHOSTS file of those servers.• Make sure copy ALL the lines within the FAST group.
  • Step 3: Connect..continued• The name of the IWA will be used as the AQT site name insystables.sitename. So, it’s important to have the rightsite name in SQLHOSTS entry for a successful connection.• Changing ANY of the details of this SQLHOSTS entry willresult in connection, query matching and accelerationissues.
  • Step 4. Design, Validate and Deploy• The secondary servers should be updatable secondaryservers.• Set this in $ONCONFIGUPDATABLE_SECONDARY 10• The Design, validate and deploy would be identical.
  • Step 5. Running queries• Once you deploy the data mart from one of the nodes,the data mart definitions in the catalogs are replicated toall the systems.• SQLHOSTS entries should be copied over manually.• After this, queries can be run as usual. Informix doesquery matching and off-loading as it does on Primary.
  • Informix Database ServerInformix warehouse AcceleratorBI ApplicationsStep 1. Install, configure,start InformixStep 2. Install, configure,start AcceleratorStep 3. Connect Studio toInformix & add acceleratorStep 4. Design, validate,Deploy Data martStep 5. Load data toacceleratorReady for QueriesIBM Smart AnalyticsStudioStep 1Step 2Step 3Step 4Step 5Ready
  • Design DM byworkload analysis ormanuallyDeployed datamartDatamartDeletedDatamart in USEDatamart DisabledPartition based refreshTrickle feed refreshDeployLoadDropDisableEnable Drop
  • Administration: Open Admin Tool• Browser based administration tool for Informix• Replaces ISAO studio with most functions• Adds workload analysis and datamart deployment• Adds support for data refresh and setup commands.
  • Data Refresh: RefreshMart Implementation :new stored procedure :ifx_refreshMart(accelerator_name,data_mart_name,locking_mode,NULL);locking_mode is optional : can be NULL4thparameter : not used as of nowif used while new functionality “trickleFeed” is active :ifx_refreshMart() will not refresh fact tables for which trickleFeedis active.
  • Data Refresh: RefreshMart :granularity based on table partitionsdata mart remains available for query accelerationsingle call of stored procedure for ease of usecontrol of execution remains with administratorhandles all data changes, including fragment operationsdata consistency via lock mode parameterprerequisite :sysadmin database accessible for administrator
  • Informix Database ServerStep 1. Create the Sales-Martand load it. Sales is the factTable, customer and storesDimension tables.Step 2 Setup tricklefeed bycalling ifx_setupTrickleFeedp 3. Let application roll.the inserts on fact anddates on any dimensions.ep 4. As the applicationsns, the reports see newta updated on IWAIBM Smart AnalyticsStudio or storedprocedures orcommand line toolStep 1Step 3Step 2Data Refresh: Scenario for Real-time trickle feed.Sales-MartsalescustomerstoresIWAOLTP Appsfact tableSetup the tricklefeedRun the applicationStep 4Reports & BI Apps
  • Data Refresh: Trickle feed (cont.)insert intofact_table ...fact tabledata row triggerdimension table1data rowacceleratordata martdata rowDbschedulertaskifx_loadPartMart()ifx_refeshMart()data rowdimension table2data row
  • “refreshMart” - a New Function for theInformix Warehouse AcceleratorMotivation:data mart is a snapshot,time consuming load required to reflect data changes,manual drop and (re-) load of individual partitions iscumbersome,want ease of use with a single function “doing it all”.IWA refreshMart ( 1 )
  • RefreshMart :refreshes only “data units” that were changed :less data to be moved.“data units” are table data partitions :single data partition for a normal table,single data partition for each fragment of a fragmented tablecontrol of granularity by table fragmentation.IWA refreshMart ( 2 )
  • Data Mart Meta Info :new meta information about data marts needed :to keep track of changesstored in new tables in sysadmin databasesysadmin database must existdata marts must be re-created after upgradeadministrator needs acces rights for sysadmindatabaseexecute function task(grant admin,<user name>,warehouse);IWA refreshMart ( 3 )
  • Data Mart Meta Info :full data load is reference pointchanges registered :insert, update, delete of data records (but no actual data islogged),drop partition, then reload partitiondetach fragmentdrop partitionattach fragmentload partitioncompleted refreshMart is new reference pointIWA refreshMart ( 4 )
  • LoadMart :loads complete data, always and unconditionallyrebuilds compression dictionary from scratchRefreshMart :does not extend or rebuild compression dictionarynew values are placed in “catch-all” containersperiodically do full data load using “ifx_loadMart()”IWA refreshMart ( 5 )
  • RefreshMart and Data Consistency :optional lock mode specifies locking of tables in warehousedatabase :MARTTABLENONEworks like lock mode for loadMartif not specified :lock mode of last loadMart is in effect.IWA refreshMart ( 6 )
  • RefreshMart Implementation :new stored procedure :ifx_refreshMart(accelerator_name,data_mart_name,locking_mode,NULL);locking_mode is optional : can be NULL4thparameter : not used as of nowIWA refreshMart ( 7 )
  • RefreshMart - Summary :granularity based on table partitionsdata mart remains available for query accelerationsingle call of stored procedure for ease of usecontrol of execution remains with administratorhandles all data changes, including fragmentoperationsdata consistency via lock mode parameterprerequisite :sysadmin database accessible for administratorIWA refreshMart ( 8 )
  • Informix WarehouseAccelerator (IWA)Using Trickle Feed feature in12.10 forreal time analysis.
  • Data Transfer from Informix to IWA – First timeReadyfor QueriesReadyfor QueriesDesign the Data Mart-- OAT analysis-- cmdline analysis-- ISAO StudioDesign the Data Mart-- OAT analysis-- cmdline analysis-- ISAO StudioDeploy the Data Mart-- OAT-- Stored procedure-- ISAO StudioDeploy the Data Mart-- OAT-- Stored procedure-- ISAO StudioOptionally lock the tableOptionally lock the tableInsert table data into external tableInsert table data into external tableSend data over to IWASend data over to IWAFact table – split into each workerDim table – copy to each workerFact table – split into each workerDim table – copy to each workerCompression frequency partitioning & encodingCompression frequency partitioning & encodingWrite the memory image to diskWrite the memory image to diskInformixInformixIWAIWALoad the Mart-- OAT-- Stored procedure-- ISAO StudioLoad the Mart-- OAT-- Stored procedure-- ISAO StudioRed: New in v12.10Red: New in v12.10
  • Distributing data from IDS (Fact tables)Data FragmentFact TableUNLOADUNLOADUNLOADUNLOADIDS Stored ProceduresCopyA copy of the IDS data is nowtransferred over to the Workerprocess. The Worker processholds a subset of the data(compressed) in main memoryand is able to execute querieson this subset. The data isevenly distributed (no valuebased partitioning) across theCPUs.CoordinatorProcessWorkerProcessCompressed DataCompressed DataCompressed DataCompressed DataCompressed DataCompressed DataWorkerProcessWorkerProcessData FragmentData FragmentData Fragment
  • Dimension TableDimension TableDimension TableDimension TableDistributing data from IDS (Dimension tables)IDSUNLOADUNLOADUNLOADUNLOADIDS Stored ProcedureDimension TableDimension TableDimension TableDimension TableAll dimension tables aretransferred to the workerprocess.Dimension TableDimension TableDimension TableDimension TableDimension TableDimension TableDimension TableDimension TableCoordinatorProcessWorkerProcessWorkerProcessWorkerProcess
  • • Fact table(s) of a data mart are populated only by new datarows. Data is inserted and not updated or deleted.• Dimension tables are updated.• Need the latest data for analysis• Full refresh can take a long time.• Partitioning/fragmentation scheme could refresh too manypartitions.• Inserted rows shall get loaded to the data mart:• - within a configurable time• - optionally, data mart dimension tables shall get refreshedTrickle feed Use Case
  • Informix Database ServerStep 1. Create the Sales-Martand load it. Sales is the factTable, customer and storesDimension tables.Step 2 Setup tricklefeed bycalling ifx_setupTrickleFeedp 3. Let application roll.the inserts on fact anddates on any dimensions.ep 4. As the applicationsns, the reports see newta updated on IWAUse OAT, Studio orstored procedures.Step 1Step 3Step 2Scenario for Real-time trickle feed.Sales-MartsalescustomerstoresIWAOLTP Appsfact tableSetup the tricklefeedRun the applicationStep 4Reports & BI Apps
  • User interface:ifx_setupTrickleFeed( accelerator_name, data_mart_name, buffertime)accelerator_nameThe name of the accelerator that contains the data mart.data_mart_nameThe name of the data mart.buffertimeAn integer that represents the time interval between refreshes andwhether dimension tables are refreshed.Examples:execute procedure ifx_setupTrickleFeed(salesacc, ‘partsmart, 60);execute procedure ifx_setupTrickleFeed(salesacc, carmart, -300);Trickle feed (cont.)
  • User interface:ifx_removeTrickleFeed( accelerator_name, data_mart_name)accelerator_nameThe name of the accelerator that contains the data mart.data_mart_nameThe name of the data mart.Trickle feed (cont.)
  • Trickle feed (cont.)insert intofact_table ...fact tabledata row triggerdimension table1data rowacceleratordata martdata rowDbschedulertaskifx_loadPartMart()ifx_refeshMart()data rowdimension table2data row
  • Design DM by workloadanalysis or manuallyDeployed datamartDatamart DeletedDatamart in USEDatamart DisabledPartition based refreshTrickle feed refreshDeployLoadDropDisableFull Load/EnableDropComplete view of Data mart state transitions.
  • Summary• Full refresh will recreate the dictionary, but cantake time.• Partition based refresh is very fast and refreshesonly the partitions with new data since the lastrefresh• Trickle feed captures the INSERTs on the facttable and refreshes by sending this data to IWA.It can also refresh dimension tables.• Even when you use partition refresh or tricklefeed, do a full refresh periodically, say, daily orweekly.
  • Deep dive into interval androlling window table partitioning in IBM InformixKeshava Murthy IBM rkeshav@us.ibm.com