Informix IWA: Architectural options
Upcoming SlideShare
Loading in...5
×
 

Informix IWA: Architectural options

on

  • 631 views

 

Statistics

Views

Total Views
631
Views on SlideShare
626
Embed Views
5

Actions

Likes
0
Downloads
2
Comments
0

1 Embed 5

https://twitter.com 5

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Features Informix Warehousing IWA Acceleration Multiple data marts with same definition Bonus IWA as a service
  • execute function dropPartMart(’myAccelerator’,’myMart’,’user10’,’tab22’,’part1’); execute function loadPartMart(’myAccelerator’,’myMart’,’user10’,’tab22’,’part1’);
  • execute function dropPartMart(’myAccelerator’,’myMart’,’user10’,’tab22’,’part1’); execute function loadPartMart(’myAccelerator’,’myMart’,’user10’,’tab22’,’part1’);
  • YouTube tutorial for RFE submit, view, and send out notification  http://www.ibm.com/developerworks/rfe/execute?use_case=tutorials#tut2 Note: Transcript for this video  http://www.ibm.com/developerworks/podcasts/demos/special-RFE-process-2/cm-int-special-RFE-process-2.html What is Different from the Current Requirements System? Requirements submitter interacts directly with Product Management No need to involve Customer Support or Sales rep Requirements go to back-end system already being used by Product Management & Development No separate tracking system that is not “part of the process” Improved ability to monitor and manage requirements Watch lists, “me too”, groups, voting Crisply defined Service Level Agreements Compliance to SLAs will be monitored monthly by Informix team Consistent requirements system for IBM Software Group products

Informix IWA: Architectural options Informix IWA: Architectural options Presentation Transcript

  • Architectural Options with IWAKeshav MurthyIBM Informix Development
  • • Data Warehouse query Performance without Perspiration• Consistent query performance without tuning efforts.• More questions, faster answers, better data driven decisions & business insights• SKECHERS: Acceleration from 60x to 1400x – average acceleration of 450xMotivation
  • Informix Database ServerInformix warehouse AcceleratorBI ApplicationsStep 1. Install, configure,start InformixStep 2. Install, configure,start AcceleratorStep 3. Connect Studio toInformix & add acceleratorStep 4. Design, validate,Deploy Data martStep 5. Load data toacceleratorReady for QueriesIBM Smart AnalyticsStudioStep 1Step 2Step 3Step 4Step 5ReadyInformix Ultimate Warehouse edition
  • 4INTEL/IWA: Breakthroughtechnologies for performance1234567 12345671. Large memory support64-bit computing; System X with MAX5 supports upto 6TB on a single SMP box; Up to 640GB on eachnode of blade center. IWA: Compress large datasetand keep it in memory; totally avoid IO.7. Multi-core, multi-node environmentNehalem has 8 cores and Westmere 10 cores. This trend isexpected to continue. IWA: Parallelize the scan, join, groupoperations. Keep copies of dimensions to avoid cross-nodesynchronization.4. Virtualization PerformanceLower overhead: Core micro-architectureenhancements, EPT, VPID, and End-to-EndHW assist IWA: Helps informix and IWA toseemlessly run and perform in virtualizedenvironment.5. Hyperthreading2x logical processors; increases processorthroughput and overall performance of threadedsoftware. IWA: Does not exploit this since thesoftware is written to avoid pipeline flushing.3. Frequency PartitioningIWA: Enabler for the effective parallel accessof the compressed data for scanning.Horizontal and Vertical Partition Elimination.2. Large on-chip CacheL1 cache 64KB per core, L2 cache is 256KB percore and L3 cache is about 4-12 MB.Additional Translation lookaside buffer (TLB).IWA: New algorithms to avoid pipelineflushing and cache hash tables in L2/L3 cache6. Single Instruction Multiple DataSpecialized instructions for manipulating128-bit data simultaneously. IWA:Compresses the data into deep columnarfashion optimized to exploit SIMD. Used inparallel predicate evaluation in scans.
  • Store_sales data mart
  • Options Optimized PlatformsSource: BI Research, 2013
  • Use Case Application ExampleReal-Time Monitoring & Analytics In-line fraud detection to reducefinancial losses caused bystolen creditcardsNear-Real-TimeAnalytics Next best customer offer to the channelto increase customer satisfaction &reduce churnData Integration Hub Collect and manage all sales-relateddetailed data (POS, web, supply chain)for down stream analysisAnalytics Accelerator Offload & boost the performance ofselected financial analyses to increasesatisfaction/retention of key clientsNew LOB Analytic Application Manage & monitor spot buying onweb advertising exchangesInvestigative ComputingPlatformEvaluate the effectiveness ofdifferent social computing channelsStarting PointSource: BI Research, 2013
  • • Ten different machine configurations• Five ways to sync data• Just combining the two options above
  • Informix Warehouse AcceleratorDeployment Options
  • Ingredients1. Computer hardware2. Informix3. IWA• This is typically part of a larger IT system andworkflow• We’ll be focusing on options for each of thesecomponents.
  • Ingredients: Computer Hardware• Computer hardware– Single SMP system– Multiple SMP systems– Single Cluster systems– Multiple cluster systems
  • Ingredients: Computer Hardware• Single SMP system– Informix and IWA running on the same system– Should be a high-memory system.• IBM System X with MAX5 can go up to 3TB with DIMMsup to 16GB.• http://www-03.ibm.com/systems/data/flash/systemx/hardware/ddr3//– The machine has to be based on Intel Xeon with SSE– Informix can be running OLTP or OLAP workload– Limit the number of CPU VPs and the number ofcores for IWA– Set the SHMTOTAL and memory for IWA
  • Ingredients: Computer Hardware• Multiple SMP systems– Informix and IWA running on the separate systems– IWA machine should be an Intel Xeon basedprocessor with high-memory.– Informix machine can be:• Linux on Intel• Linux or AIX on Power• Solaris on Intel or Sparc• HP/UX on Itanium– Data is transferred from Informix instance to IWA.
  • Ingredients: Computer Hardware• Computer hardware– Single SMP system– Multiple SMP systems– Single Cluster systems– Multiple cluster systemsLet’s discuss Informix and IWA on cluster systems.i.e., Informix MACH11IWA on a multi-node cluster.
  • 16Informix Database ServerInformix warehouse AcceleratorBI ApplicationsStep 1. Install, configure,start InformixStep 2. Install, configure,start AcceleratorStep 3. Connect Studio toInformix & add acceleratorStep 4. Design, validate,Deploy Data martStep 5. Load data toacceleratorReady for QueriesIBM Smart AnalyticsStudioStep 1Step 2Step 3Step 4Step 5ReadyInformix Warehouse Accelerator – In 11.70.FC4
  • Design DM by workloadanalysis or manuallyDeployed datamartDatamart DeletedDatamart in USEDatamart DisabledPartition based refreshTrickle feed refreshDeployLoadDropDisableFull Load/EnableDropComplete view of Data mart state transitions.
  • Background• Prior to 11.70.FC5, adding accelerator, create, deploy, load, enable,disable datamart, accelerating queries – are all operations officiallysupported only on Standard server or Primary node of MACH11/HAenvironment.• We estimate about 50% of Informix customers use HDR secondaryservers and growing number of customers use MACH11 (SDSsecondary) configurations and RSS nodes. MACH11 is the Informixscale out solution.• IWA itself supports a scale out solution (on a cluster) starting with11.70.FC4.• Reasons to support MACH11 and IWA together.– This feature will enable partitioning a cluster or HA group between OLTP andBI workload.– This feature will give help to off-load the expensive LOAD functionality tosecondary servers– We have customers now requesting support for HDR secondary to IWA
  • 19Informix PrimaryInformix warehouse AcceleratorBI ApplicationsStep 1. Install, configure,start InformixStep 2. Install, configure,start AcceleratorStep 3. Connect Studio toInformix & add acceleratorStep 4. Design, validate,Deploy Data mart fromPrimary, SDS, HDR, RSSStep 5. Add IWA to sqlhostsLoad data toAccelerator from any node.Ready for QueriesIBM Smart AnalyticsStudioStep 1Step 3Step 4Step 5ReadyInformix Warehouse Accelerator – 11.70.FC5. MACH11 SupportInformix Warehouse Accelerator – 11.70.FC5. MACH11 SupportInformixSDS1InformixSDS2InformixHDRSecondaryInformixRSSStep 2
  • 1. Machine Summary1. Informix can be in any of the following2. IWA can be running in any of the following• Single SMP system• Multiple SMP systems• Single Cluster systems• Multiple cluster systems3. You can mix and match for scale-out (performance), highavailability, application evolution, migration and any numberof reasons.
  • 1. Hardware Configuration• All in one – single system – All on Linux on Intel• Multiple systems• Homogeneous systems – Linux on Intel• Heterogeneous systems• Informix on Linux on Intel/Power, AIX on Power, Solarison Intel/Sparc• IWA Linux on Intel• Informix on Cluster and IWA on single node• Informix on single system and IWA on cluster• Informix on cluster and IWA on cluster• Informix on combination of Cluster and smp system; IWA oncluster or smp system
  • 1. Informix Topology• Informix single node.• Informix Primary+SDS• Informix Primary + HDR + RSS• Informix Primary +SDS + RSS• Informix Flexible grid1. IWA Topology• Single node• Multiple single node systems• Single cluster• Multiple clusters• combo of single/multi
  • • Features• Informix Warehousing• IWA Acceleration• Multiple data marts with same definition• MACH11 support• Heterogeneous platform support• Data sync – refresh mart• Data sync – trickle feed• Data mart – External table• Data mart Timeseries acceleration
  • Informix Database ServerBI ApplicationsStep 1. Create the Sales-Martand load it. Sales is the facttable -- range partitioned.Step 2. Load jobsupdate the fact table “sales”Only updates existing partitionStep 3. Identify the partition,execute dropPartMart().Step 4. for same partition,execute loadPartMart().Ready for QueriesIBM Smart AnalyticsStudio or storedprocedures orcommand line toolStep 1Step 4Step 2Step 3ReadyCase 1: Partition refresh: Updates to existing PartitionsSales-MartsalescustomerstoresIWAOLTP Appspartitioned fact tableSQL Script: callStored procedureModified partitionINSERT, UPDATE, DELETE
  • Informix Database ServerBI ApplicationsStep 1. Create the Sales-Martand load it. Sales is the facttable -- range partitioned.Need to move the Timewindow to next range.ep 2. DETACH operationExecute dropPartMart()DETACH the partitionep 3. ATTACH operationATTACH the partitionExecute loadPartMart()Ready for QueriesIBM Smart AnalyticsStudio or storedprocedures orcommand line toolStep 1Step 3Step 2ReadyCase 2: Partition refresh: Time Cyclic data managementSales-MartsalescustomerstoresIWAOLTP Appspartitioned fact tableMove the window.
  • Design DM byworkload analysis ormanuallyDeployed datamartDatamartDeletedDatamart in USEDatamart DisabledPartition based refreshTrickle feed refreshDeployLoadDropDisableEnable Drop
  • Data Refresh: RefreshMart Implementation :new stored procedure :ifx_refreshMart(accelerator_name,data_mart_name,locking_mode,NULL);locking_mode is optional : can be NULL4thparameter : not used as of nowif used while new functionality “trickleFeed” is active :ifx_refreshMart() will not refresh fact tables for which trickleFeed is active.
  • Data Refresh: RefreshMart :granularity based on table partitionsdata mart remains available for query accelerationsingle call of stored procedure for ease of usecontrol of execution remains with administratorhandles all data changes, including fragment operationsdata consistency via lock mode parameterprerequisite :sysadmin database accessible for administrator
  • Informix Database ServerStep 1. Create the Sales-Martand load it. Sales is the factTable, customer and storesDimension tables.Step 2 Setup tricklefeed bycalling ifx_setupTrickleFeedp 3. Let application roll.the inserts on fact anddates on any dimensions.ep 4. As the applicationsns, the reports see newta updated on IWAIBM Smart AnalyticsStudio or storedprocedures orcommand line toolStep 1Step 3Step 2Data Refresh: Scenario for Real-time trickle feed.Sales-MartsalescustomerstoresIWAOLTP Appsfact tableSetup the tricklefeedRun the applicationStep 4Reports & BI Apps
  • Data Refresh: Trickle feed (cont.)insert intofact_table ...fact tabledata row triggerdimension table1data rowacceleratordata martdata rowDbschedulertaskifx_loadPartMart()ifx_refeshMart()data rowdimension table2data row
  • User interface:ifx_setupTrickleFeed( accelerator_name, data_mart_name, buffertime)accelerator_nameThe name of the accelerator that contains the data mart.data_mart_nameThe name of the data mart.buffertimeAn integer that represents the time interval between refreshes andwhether dimension tables are refreshed.Examples:execute procedure ifx_setupTrickleFeed(salesacc, ‘partsmart, 60);execute procedure ifx_setupTrickleFeed(salesacc, carmart, -300);Trickle feed (cont.)
  • Deep dive into interval androlling window table partitioning in IBM InformixKeshava Murthy IBM rkeshav@us.ibm.com
  • IBM’s statements regarding its plans, directions, and intent are subject to change orwithdrawal without notice at IBM’s sole discretion.Information regarding potential future products is intended to outline our generalproduct direction and it should not be relied on in making a purchasing decision.The information mentioned regarding potential future products is not a commitment,promise, or legal obligation to deliver any material, code or functionality. Informationabout potential future products may not be incorporated into any contract. Thedevelopment, release, and timing of any future features or functionality described forour products remains at our sole discretion.Please Note:Performance is based on measurements and projections using standard IBMbenchmarks in a controlled environment. The actual throughput or performance thatany user will experience will vary depending upon many factors, includingconsiderations such as the amount of multiprogramming in the users job stream, theI/O configuration, the storage configuration, and the workload processed. Therefore,no assurance can be given that an individual user will achieve results similar to thosestated here.
  • 04/22/13 34Availability. References in this presentation to IBM products, programs, or servicesdo not imply that they will be available in all countries in which IBM operates.The workshops, sessions and materials have been prepared by IBM or the sessionspeakers and reflect their own views. They are provided for informational purposesonly, and are neither intended to, nor shall have the effect of being, legal or otherguidance or advice to any participant. While efforts were made to verify thecompleteness and accuracy of the information contained in this presentation, it isprovided AS-IS without warranty of any kind, express or implied. IBM shall not beresponsible for any damages arising out of the use of, or otherwise related to, thispresentation or any other materials. Nothing contained in this presentation is intendedto, nor shall have the effect of, creating any warranties or representations from IBM orits suppliers or licensors, or altering the terms and conditions of the applicable licenseagreement governing the use of IBM software.Acknowledgements andDisclaimers:
  • Acknowledgements &Disclaimers:© Copyright IBM Corporation 2013. All rights reserved.– U.S. Government Users Restricted Rights - Use, duplication or disclosurerestricted by GSA ADP Schedule Contract with IBM Corp.– Please update paragraph below for the particular product or family brand trademarksyou mention such as WebSphere, DB2, Maximo, Clearcase, Lotus, etcIBM, the IBM logo, ibm.com, [IBM Brand, if trademarked], and [IBM Product, if trademarked]are trademarks or registered trademarks of International Business Machines Corporation inthe United States, other countries, or both. If these and other IBM trademarked terms aremarked on their first occurrence in this information with a trademark symbol (® or ™), thesesymbols indicate U.S. registered or common law trademarks owned by IBM at the time thisinformation was published. Such trademarks may also be registered or common lawtrademarks in other countries. A current list of IBM trademarks is available on the Web at“Copyright and trademark information” at www.ibm.com/legal/copytrade.shtmlIf you have mentioned trademarks that are not from IBM, please update and add thefollowing lines:[Insert any special 3rd party trademark names/attributions here]Other company, product, or service names may be trademarks or service marks of others.
  • Do you have a great presentation topic thatyou’d like to share?•We’re looking for dynamic, innovative and thought-provokingsessions•Whether your proposal aims at sharpening skills, sharing bestpractices, or presenting new ideas and groundbreaking concepts, allproposals are welcome•Visit the conference website to learn moreThe Call for Speakers closes April 30! Hurry to submit your session!
  • Sign Up! Informix Usability Sandbox!Help shape the future of Informix.Influence Informix usability and functionality.Share your experiences and feedback.Usability Sandbox sessions in Santa Fe 3April 22-24th, between 9am and 5pmSign-up at the IBM Information Table or find Justin McDavid.*The first 20 participants will get a free IBM t-shirt!
  • Informix RFE (Request For Enhancement) ProcessAs Simple as 1, 2, 31. Submit from the IM RFE site – simply complete the RFE form and click Submit when ready Many fields will be auto-filled as a convenience for you Note that fields with the ‘key’ field e.g. Company Name and Business Justification will bekept private for confidentiality purposes Provide as much detail as possible in the Description, Use Case, and BusinessJustification fields to help the IBM team understand your requirement2. View via Watchlist Lists all the RFEs that you’re interested in Simple to add an RFE via Search3. Subscribe to email notifications Specify ‘Opting in for email notifications’ Notified when any change occurs to any RFE on your watch listYouTube: http://www.ibm.com/developerworks/rfe/execute?use_case=tutorials#tut2YouTube: http://www.ibm.com/developerworks/rfe/execute?use_case=tutorials#tut2Give it a shot! http://www.ibm.com/developerworks/rfe/
  • Backup
  • ar·chi·tec·ture  /ärkitekCH r/əNoun• The art or practice of designing and constructing buildings.• The style of a building with regard to a specific period,place, or culture.op·tion  NounA benefit in the form of an option given by a company to an employee tobuy stock in the company at a discount or at a stated fixed price.Surely, were neither discussing buildings or stock options.
  • A system architecture or systems architecture is theconceptual model that defines the structure,behavior, and more views of a system.An architecture description is a formal descriptionand representation of a system, organized in a waythat supports reasoning about the structures of thesystem, which comprise system components, theexternally visible properties of those components,the relationships (e.g. the behavior) between them,and provides a plan from which products can beprocured, and systems developed, that will worktogether to implement the overall system.
  • A system architecture or systems architecture is theconceptual model that defines the structure,behavior, and more views of a system.An architecture description is a formal descriptionand representation of a system, organized in a waythat supports reasoning about the structures of thesystem, which comprise system components, theexternally visible properties of those components,the relationships (e.g. the behavior) between them,and provides a plan from which products can beprocured, and systems developed, that will worktogether to implement the overall system.