Data Service Layer for Service-Oriented Architectures (SOA) Master Data Management (MDM)
Maintain consistent view of core business entities
Avoid political issues around data ownership
Reduce data replication
Create data services layer that simplifies SOA
Allow universal info exchange as Web Services
Manage both HTTP and SOAP Web Services
Use Case: Real-Time Reporting Reporting Trade Data Market Data Risk Data Oracle OLTP Sybase OLTP Web Service EII Server Virtual DB
Use Case: Data Warehouse Prototyping Relational Databases Sales Ops Applications/Marts EII Server Virtual DB Data Warehouse Reporting & BI Tools Migrate ETL Prototype Permanent
Use Case: Database Migration - Insulation from Change Reporting Trade Data Trade Data Sybase DB2 EII Server Virtual DB Migrate
Use Case: Migrating to SOA Relational Databases Sales Ops Applications/Marts EII Server Virtual DB
Ipedo Creates a Virtual Data Services Layer from Multiple Remote Data Sources Ipedo XIP Sales Business Unit A Business Unit B Customer System Finance System Operations System Finance System Operations System Mfg. System Finance Marketing Subset Data Distributed Data Mixed Data Query Query Query Query
Cross business boundaries
No data replication
Mix data types in real time
TIBCO SQL Server SAP Oracle SQL BAPI SOAP WSDL JMS SOAP WSDL SQL View 2 Operations View 1 Sales by Region View 3 Financials by Account View 4 Financials Quarterly
Ipedo Provides a Data Services Layer
Views Virtualize Data Access Custom Metadata Data Source Connections Data Mappings Transformations Output Schema View Caching Policy Validation Security View Data Warehouse Relational Database Web Services Packaged Apps XML Documents Message Queues Ipedo Views A View is a metadata construct. It contains no data, unless explicitly cached. When a query is executed against a View, Ipedo returns the data to fulfill the request.
Deploying Ipedo XIP Ipedo XIP Development/QA Ipedo XIP Monitor & Manage Production App Server Reporting Dashboards Web Apps SOA End Users
Views are designed, developed and tested
Views are then deployed to production server
Additional production servers can be added to support more users
Ipedo XIP Solves Integration Challenges Faster Development $ Cheaper Deployment Lower Risk $ Data discovery tools and wizards for RDB, XML and SOA Data source introspection with statistic and key gathering Metadata collection and management $ Drag and Drop View creation $ Automatic SQL and XQuery generation Rule and aggregate tools Automatic XML to RDB bridging $ Automatic query plan generation Cost-based query optimization Policy-driven caching for SQL and XML result sets $ Virtualized data access without replication Customizable result set presentation Adjustable configurations for scaling up, down or out Detailed data access controls Performance thresholds and limits Data lineage and query forensics reporting Automated Discovery Powerful Modeling Intelligent Optimization Flexible Deployment Comprehensive Management Where does our data reside and in what format? How can I quickly combine all the data I need for my application? How can I meet my users’ performance needs and not break existing systems? How can I tailor data to suit the various and changing user needs? How can I make sure we adhere to our security and data governance policies?
Auto-Discovery Simplifies Data Integration XIP > Automated Discovery Lists all tables Displays column names Shows data types Generates SQL DDL Ipedo automatically Introspects remote data sources.
Custom Metadata Annotates Data XIP > Automated Discovery Ipedo Metadata Catalog Add custom metadata to annotate: * data sources * remote tables * columns * views Retrieve metadata from Ipedo system tables using standard SQL system functions
GUI View Builder Simplifies Development XIP > Powerful Modeling ‘ Drag and Drop’ tables into the design pane Build logic using intuitive graphical tools: TotalPrice>$50,000 Ipedo automatically generates SQL, which can be modified
Ipedo Ideal for SOA
Consuming Web Services
Blending multiple web services
Conditional invocation of web services in a specific order
Combining web services with other data sources
Processing Web Services
Business rules checking on web services data
Caching of web services data for reuse
Ipedo uses a query-based approach, preserving the format of the original data
‘ Web Services’ Tables Allows SQL on XML Customer Records (Oracle) Market Exposure (SOAP Web Service) Credit Record (HTTP Web Service) XML XML Rel. Rel. XMLQuery XMLQuery XML Core SQL Core BI/Reporting Client ODBC / JDBC SQL SQL SQL SQL Data Sources Remote Tables Relational View Use familiar SQL syntax with Web Services: Join, Group By, hints XIP > Powerful Modeling
Map Web Services to Relational Tables XIP > Powerful Modeling Select a Web Service Map results to relational columns Modify data types to fit your application’s needs
Ipedo Application Adapters Integrate Data Into Ipedo Framework
Pre-built templates and objects
Accessible via SQL, Web Services, XQuery
Visual drag and drop, API/script access
Broadest range of applications (Librados)
Simple pricing structure ($15K/CPU per adapter)
Application View Templates Application Data Objects Oracle Integration Tables Integration Objects API BAPI SOQL Oracle Siebel SAP R/3 Salesforce.com Common Reports Common Objects
Ipedo Approach to SAP BW Data
Librados adapters for relational data
Simba OLAP adapters for cube data
Best-in-class vendors of standards-based advanced analytical data adaptors
Perform more complex analyses using broader range of data sources
Integrate SAP BW with other data sources
Ipedo Optimizes Data Access Data Virtualization Customizable Presentation Deployment Time Flexible Deployment Intelligent Optimization Query Optimization Query Execution Cache Policy Cost-Based Optimizer Query Plan Advanced Tuning XML View Cache SQL View Cache Schedule Refresh Policy
Each table can contain data from disparate sources (relational or Web Services)
XIP > Flexible Deployment
Performance Optimization
Cost-Based Query Optimization
CBO means knowing how a query will perform; Rules-based optimization means guessing
Examines statistics in remote data sources
Size of database (cardinality)
Indexes - presence, type, selectivity
Allows user to assign cost function to remote procedure
Determines join order and join algorithm based on “cost” of performing query
Minimizes intermediate results to speed performance
CBO greatly enhances EII performance
XIP > Intelligent Optimization
Query Plan Shows Optimization Approach
Automatic Query
Plan Generation
Join Order
Join Algorithms
Pushdowns
Joins
Predicates
Aggregates
XIP > Intelligent Optimization
Advanced Tuning Provides Ultimate Control
Hints
override Ipedo’s query engine with domain-specific knowledge
SQL pass-through
Ipedo sends data source specific SQL to the data source for execution
Polymorphic remote function mapping
XIP > Intelligent Optimization pass sql "dbcc show statistics (customers, zipcode_stat)" to mssqlsvr select a.* from properties joinOrder=fixed orders o, customers c properties joinStrategy=NESTEDLOOP where c.c_customerid = o.o_customerid
Query Execution
Result Set Streaming
Immediate response to user
Query Throttling
Control load on back-end systems
Concurrency
Support large numbers of users with optimal performance
Hybrid memory-disk intermediate result set processing
Manage memory to handle extremely large data sets
Flexible Security Model
Common user credential across data sources
Dynamic session-based user credentials for data sources
View-based access control
XIP > Intelligent Optimization
Intelligent Caching Cuts Response Time, System Load
Speeds Response to Repeated Queries
Reduces Load on Back-End Data Sources
Cache relational or XML data natively
Sub-queries on cache
Scheduled or on-demand cache loading
Automatic cache invalidation
XIP > Intelligent Optimization
Top-Down Modeling Tools Can Create Tables and Views in Ipedo 1. Build Logical Views In enterprise modeling software 2. Export Data Definitions Using standard ODBC 3. Select Options For schemas, tables, columns, indexes 4. Customize DDL for Ipedo Syntax reflects virtual table construct 5. View Model in Ipedo All metadata appears, available for integration
Reuse Ipedo Views in Modeling Tools 1. Build Views in Ipedo Map tables and create relationships across disparate data sources 2. Reverse Engineer Start with blank template inside ERwin 3. Select Options Choose what information to import 4. Model Appears in ERwin Tables, metadata, joins, etc., available to share and reuse
Comprehensive Monitoring and Reporting Simplifies System Management Partners Finance Marketing Management Dashboard
Detailed User and Source Management Session Status XIP > Comprehensive Management Database Availability See Query Plan Kill Session or Cancel Query
Ipedo Facilitates Data Governance Where is the information in this view coming from? How does change to data source affect downstream views?
Dynamic Failover Redirection
Many large customers have database replicas worldwide - want EII Views to failover to replica database
Ipedo XIP will failover after it reaches retry limit
SOA Sybase 1 Oracle 2 Oracle 1 New York London New York Tokyo Sybase 2
Enterprise Class Product
Clustering and Failover - support for standard clustering environment
Scales to manage millions of documents and terabytes of storage
Industry-standard management APIs
Robust archival & backup capabilities
Automated schema migration
Integrated document versioning and journaling
Ipedo XIP Components SQL Engine • Query Optimization XQuery Engine • Query Optimization Federation Engine • Query Orchestration • Statistical Analysis Data Services Manager • Web Services • SQL Data Services • XQuery Data Services • APIs • Web Tag Library • Outbound Pipelines Rules / Intelligence Engine • Visual Rules Wizard • Rules Processing • Compound Aggregate Indexing Integration / Transformation Engine • Relational Views • XML Views • Graphical View Builder • Feeds • Inbound Pipelines Metadata Manager • Sources • Mappings Cache Manager • Relational • XML XML Store • Schema Management
HP - Finance Dashboard
Application
Financial Dashboard
Challenges
Accessing data from disparate sources
Frequently changing data needs
Requirement to keep interface simple
Granular security model
Scalability for 25,000 users
Solution
Use Ipedo to aggregate data from multiple sources, filter using XML, and transform for delivery into HP’s internal portal,
Results
Makes new information available for real-time business decisions
My Dashboard Financial Databases (Oracle) Data Warehouse (Hyperion Essbase) MS Word Analysis Reports Corporate Portal Framework Ipedo XIP Product View Region View Account View Channel View
Assurant - Insurance Policy Management
Application
Insurance Company: next-generation policy management system
Challenges
Manage large transaction volumes of policies in ACORD XML schema
Integrate with complex workflow processes using a service-oriented architecture (SOA)
Maintain uptime in high-availability environment
Solution
Use Ipedo to manage and integrate information from underwriting, accounting, claims processing and billing systems,
Results
Streamlines operations, improves response time to inquiries
Billing Database Ipedo XIP Profitable Accounts Claims Status Expensive Claims Arrears Sales View Workflow View Claims View Acct. View Claims System Acct. System Web Services Policies ACORD Schema
Sun - Customer Service
Application
Customer Service
Challenges
Combining XML messages, Web Services, databases, other data
Connecting with external risk engine
Providing rapid response to inquiries
Managing huge data volumes
Solution
Embed Ipedo into Sun application to deliver risk analysis as a service
Results
Improves system uptime and availability and reduces incident volume and cost
Customer Reference Data (Oracle) Risk Engine (Fair Isaac) System Information (XML) Preventive Services Ipedo XIP Hardware View Software View Account View Risk View Risk Assessment Advanced Diagnostics System Configurations Web Services JDBC
The Ipedo Difference
High Performance
Cost-based Query Optimization
Intelligent Caching
Investment Protection
Choice of data model: Relational (SQL) or XML (XQuery)
Choice of SOA deployment model – WSDL/SOAP or REST
One cohesive product
Seamless Deployment
Fully scriptable with complete SQL DDL support
Integrated Data Lineage and Impact Analysis reporting
Comprehensive APIs: Java, Web Services, .NET
Highly Scalable
Concurrency for hundreds of simultaneous users
Easily accommodates additional data sources
Additional Resources 2005 EII Survey Guide to EII ROI Guide
Background Slides
Enterprise Information Integration is…
… a new kind of virtual data integration technology that makes several remote data sources appear as one local database.
Ipedo XIP: Integration + Intelligence Vision: Intelligence without boundaries: Give business decision makers “information on demand, from anywhere, at any time” Product: Ipedo XIP integrates and manages information from disparate and complex data sources on demand
EII Generates Real Financial Benefits
Developer Productivity
EII allows developers to write to one, common interface to access multiple data sources, reducing the development cost for new applications
What % of your IT budget is spent on data integration development?
Analyst Utilization
EII allows analysts to simplify the data collection process and spend more time producing information and less time gathering data
How much time do your analysts spend collecting data?
Increased Flexibility
EII allows companies to respond quickly to changes in market conditions without incurring a major IT investment
What effort is involved to link new data sources to existing reports or applications?
Data Consolidation
Using EII to create a virtual centralized database can cut the cost of data maintenance, and reduce the need for data replication
How much does it cost to deploy each new data mart?
Integration Platform Built for Developers/Integrators Data and Cache Management • TTL • Performance • Versioning Publishing and API Access • XML • JSP Tag Lib • JDBC/ODBC • .NET • WS/SOA Transformation & Virtualization • XML Schema • CSV • SQL <-> XML • Pipelines Connectivity • Relational Views • XML Views • SOA/WS Views • JMS Feeds • HTTP Views Query Management • XQuery • SQL • Cost-based Optimizer Rules Processor • SQL SP • XQuery Modules • Java Functions Security • ACLs • Roles Management • JMX • JSP • APIs GUI Tools • View Builder • Query Builder • Rules Builder
EII Brings Advantages to New Projects - Create data services layer that simplifies SOA MDM - Maintain consistent view of core business entities - Avoid political issues around data ownership SOA - Allow universal info exchange as Web Services Solution Key Advantages of EII Dashboards - Intelligently combine company-wide information - Provide view customization per context or user Risk Management - Extract key information across company data silos - Reduce cost of compliance BI / Reporting - Enable better reports with up-to-date information - Access diverse data sources via queries, not code
Metadata Slides
Views Use Metadata Extensively Ipedo XIP View Output Schema Transformations Data Mappings Data Source Connections View Definitions of Fields Business Definitions Caching Policies Security Primary / Foreign Keys Security Data Source Metadata Source Repository Ipedo
Metadata Flows Throughout the Framework Metadata Manager Enterprise Data Sources Introspection Ipedo XIP Metadata Repositories Metadata Mgmt/Mod Tools Databases Web Services XML Doc Collections Other Data Sources Supported by Ipedo Modeling Tools BI/Reporting Tools SOA Tools App Dev Tools JDBC/ODBC/SQL WSDL/UDDI JDBC/ODBC/SQL WSDL/UDDI XMI JDBC/ODBC/SQL JDBC/ODBC/SQL XMI/Proprietary
Additional Customer Case Studies
Anhui Electric - Fraud and Risk Management
Application
Power Utility Billing Risk Management Dashboard
Challenges
Forecasting business risk
No aggregated meter monitoring data to prevent rampant theft of power
Lack of credit information on new customers
Solution
Use Ipedo to integrate energy meter collection system, customer databases and financial systems across multiple sites
Results
Significant reduction in fraud and theft; better management reporting
Billing Database Customer Database Energy Meter Collection System Ipedo Credit View Theft View Risk View Forecast View
Application
Ring tone, music, and mobile game digital rights management platform
Challenges
Ability to address diverse content and transaction data types
Adapting rapidly to changing market and customer requirements
Scalability for millions of buyers and millions of content assets
Solution
Use Ipedo to integrate user, payment, and ring tone/game metadata for presentation through Web interface
Results
Allows rapid response to customers and easy integration of new content libraries into client Web sites
Navio Card Navio Merchant Navio Store ASP Applications Report Server Title Manager Content Lockbox Digital Commerce Engine Ipedo XIP Ipedo XIP Ipedo XIP Navio - Rights-Based Commerce Web Storefront P2P Net
HP OpenView - Customer Service Portal
Application
Customer Service Portal
Challenges
Flexible search interface
Frequently changing content
Scalability for global users
High Availability
Solution
Use Ipedo to manage granular content and deliver in response to Web-based user inquiries
Results
Improve customer experience by directing users to relevant software downloads or reference materials
Customer Service Marketing Databases (RDBMS) OpenView Web Site Ipedo XIP Product Info Customer Info Marketing Info
Pharma R&D Data Integration Ipedo Cuts Costs, Speeds Discovery BEFORE IPEDO AFTER IPEDO
Too many data sources
Unable to consolidate genomic data
Data sources in proprietary formats
Unable to analyze all genomic data
Scientists manually research genes
Scientists’ observations are not shared
Ipedo XIP Observations Annotations
Deliver one-stop shop for research data
Capture scientists’ annotations
Keep scientists’ information synchronized
Views Integration
Query Analysis
Pipelining Presentation
“ 10x researcher productivity gain.”
Internal Data Warehouse GenBank BLAST MedLine Internal Data Warehouse GenBank BLAST MedLine Scientists Scientists Research Application Web Site Web-Based Queries Subscription Web Site ROI in Time to Market Research Portal
British Telecom Price Book October 2003
Vertical Market Unique Needs
Financial Services Has Unique Needs
360 degree view of the customer
See across systems in subsidiaries/acquisitions
Integrate data from different databases
Combine live and historical data
Satisfy regulatory requirements
Cross-divisional reporting
Granular security / data ownership
Insurance Has Unique Needs
Integrate underwriting, policy management, billing, claims systems
Perform ad-hoc reports
Leverage industry standard schemas
Produce reports for regulators
360 degree view of the customer
Cross-divisional reporting
Post-merger/acquisition consolidation
Granular security / data ownership
Health Care Has Unique Needs
HIPAA compliance
Protect patient privacy - granular security model
Mix historical and real-time information
Customize reports for different uses: administration, insurance, treatment
360 degree view of the patient
Retail Has Unique Needs
Generate real-time sales information
Track inventory across multiple locations
Share information with supply chain
Spot trends as they occur
360 degree view of the customer
Cross-divisional reporting
Post-merger/acquisition consolidation
Granular security / data ownership
Business Intelligence
Business Intelligence and Reporting
Use Business Objects, Cognos, MicroStrategy, or any BI tools to analyze data from:
Relational databases
Unstructured data sources
Enterprise applications
Web Services
Combine historical information with real-time information from message queues or Web Services
Ipedo Supports BOBJ Three Ways Ipedo XIP BusinessObjects Universe Business Objects BI Platform Data Sources Data Mart
Ipedo Bridges Data Partitions
Application
Ad-hoc reporting across separate databases using BI tools
Challenges
Overcoming size limitation on Sybase servers
Shielding reports from changes in underlying database partitions
Combining data from disparate sources into a single “Universe”
Maintaining high-performance for complex queries
Solution
Use Ipedo to create virtual union views
BI tools connect to Ipedo views
Ipedo optimizes and federates queries to all databases
Business Intelligence / Enterprise Reporting Tools Ipedo XIP All Customers View All Orders View All Sellers View ... Sybase1 Sybase2 Sybase10
Ipedo Appears as a Database to Business Intelligence Applications In this example, Business Objects Web Intelligence sees Ipedo as a single relational database. Ipedo is really pulling data from three different relational sources.
Ipedo Complements BOBJ
Data Integration
Real-time, query-based access to data sources
Combining web services with other data sources
Query optimization
Validation of data in real-time
Web Services
Blending multiple web services
Conditional invocation of web services in a specific order
Business rules checking on web services data
Caching of web services data for reuse
Master Data Management
Master Data Management
Create a master index that leverages Ipedo’s high performance engine
Cost-based query optimization
Union join pushdown
Reduce data replication
Customize departmental subsets of customer data to avoid data politics and ownership issues
Perform data cleansing (through Ipedo partnership)
Help improve the performance of the denormalization process and boost the flexibility in handling underlying schema changes
Ipedo XIP for MDM BI/Reporting Client ODBC / JDBC SQL SQL SQL Data Sources Remote Tables Relational View Join data from multiple RDBMS using Master Customer Key in MDM Ipedo Customer View Join View Union View Master Customer Key SQL SQL CRM Data (DB2) Trading Data (Sybase) Master Data (DB2)
Data Quality
Data Quality Enhances EII Data Source Mappings Virtual View Mappings User Query on View EII Step: Join DQ Call: Matching Join Key EII Step: Filter DQ Call: Matching Other Columns Data Sources Live Views (seconds of latency) User Query on View EII Step: Filter, Other DQ Call: Cleanse or Match Data Cached Views (minutes to hours of latency)
Caching Options:
Scheduled Cache (Push) – Include cleanse/match option for admin
User-driven Cache (Pull) – Include cleanse/match option after initial view
Aligning data between different sources for efficient join operations
Reconciling differences between data from different sources
Cleaning up cached information for subsequent reuse
Risk Management
Risk Management
Present complete set of information for analysis
Real-time feeds from message queues (trading data, interest rates, etc.)
Analyze complex XML documents (such as FpML or other trading schemas) and combine seamlessly with relational data
Cache XML for subsequent query, analysis, and integration
Compound Aggregate Indexes (CAI) track key data fields and automatically compute aggregates across different data feeds
Maintain audit trail for compliance and SOX
Ipedo for Risk Analysis Financial View Ipedo XIP JDBC ODBC SQL Convert to Relational Tables Calculate Aggregates Message Queue Message Queue XBRL FpML XML Caching BI/Reporting Client Trading Data In XML Format Order View Risk View
Enterprise Readiness
Ipedo Delivers Enterprise Readiness
Performance
Cost-based Query Optimization
Result-set Streaming
Policy-based Caching
Advanced Tuning
Scalability
Clustering
Concurrency
Security
Data Governance
Ipedo Joins Huge Data Sets… SELECT CUSTOMER.C_NAME, CUSTOMER.C_CUSTKEY, CUSTOMER.C_ADDRESS, COUNT(ORDERS.O_ORDERKEY), SUM(ORDERS.O_TOTALPRICE ) FROM CUSTOMER, ORDERS WHERE (CUSTOMER.C_CUSTKEY=ORDERS.O_CUSTKEY) AND CUSTOMER.C_NAME = 'Customer#000000250' GROUP BY CUSTOMER.C_NAME, CUSTOMER.C_CUSTKEY, CUSTOMER.C_ADDRESS
Select all ORDERS of a given CUSTOMER
ORDERS and TIME_DIM tables from Oracle (Orders Management App) 750,000 rows
CUSTOMER and NATION tables from SQL Server (CRM) 10 0 , 0 0 0 r o w s
…And Delivers Sub-Second Performance
Query Optimization:
Search predicate is pushed down to remote RDBMS.
Join predicate will be pushed into remote inner table if nested loop join is cost-effective.
Number of rows of the where clause is taken into consideration in Cost-Based Optimization (CBO).
Performance:
average query time = 0.1 sec
Query Plan:
Ipedo Security Protects Data and Views
User and Group Level Hierarchical Access Control List
Integration with external LDAP directories
Document/Attribute-level filtering (node-level)
JAAS-compliant encryption algorithm flexibility for 2-way secure authentication (from client, as well as server)
Low Overhead in IT Environment
Connection Pooling allows you to protect back-end data sources from excessive load
Intuitive setup and administration means no DBA required to run Ipedo
Runs on Windows, Linux, Solaris, HP-UX, AIX - any environment with a JVM
Runs on Tomcat (included), WebLogic, WebSphere, JBoss, JRocket
ORDERS and TIME_DIM tables from Oracle (Orders Management App)
CUSTOMER and NATION tables from SQL Server (CRM)
100,000 rows 750,000 rows
Testing conducted in July 2005 on dual-CPU Intel XEON 2.2GHz with 2.0Gb memory
SQL Query #1 SELECT CUSTOMER.C_NAME, CUSTOMER.C_CUSTKEY, CUSTOMER.C_ADDRESS, COUNT(ORDERS.O_ORDERKEY), SUM(ORDERS.O_TOTALPRICE ) FROM CUSTOMER, ORDERS WHERE (CUSTOMER.C_CUSTKEY=ORDERS.O_CUSTKEY) AND CUSTOMER.C_NAME = 'Customer#000000250' GROUP BY CUSTOMER.C_NAME, CUSTOMER.C_CUSTKEY, CUSTOMER.C_ADDRESS
Select all ORDERS of a given CUSTOMER
Query Plan #1 and Performance
Query Optimization:
Search predicate is pushed down to remote RDBMS.
Join predicate will be pushed into remote inner table if nested loop join is cost-effective.
Number of rows of the where clause is taken into consideration in Cost-Based Optimization (CBO).
Performance:
average query time = 0.1 sec
Query Plan:
SQL Query #2 SELECT TIME_DIM.YEAR, COUNT(ORDERS.O_ORDERKEY), SUM(ORDERS.O_TOTALPRICE ) FROM TIME_DIM, ORDERS, CUSTOMER WHERE (ORDERS.O_ORDERDATE=TIME_DIM.NATIVEDATE) AND CUSTOMER.C_CUSTKEY=ORDERS.O_CUSTKEY) AND ( ( CUSTOMER.C_CUSTKEY ) = CASE WHEN char(hour(current_timestamp)) = '24' THEN 250 ELSE 251 END )) GROUP BY TIME_DIM.YEAR
Conditionally select a customer’s total order count and sum
Query Plan #2 and Performance
Query Optimization
ANSI SQL standard case-when-else conditional expression is pushed down to remote RDBMS.
Column transitivity is supported to allow better query plan.
e.g. (t1.c1 = t2.c2 and t2.c2 = constant) (t1.c1=constant and t2.c2 = constant)
Performance:
average query time = 0.2 sec
Query plan:
SQL Query #3 SELECT V_MERGED_CUSTOMER.C_NATIONKEY, sum(V_MERGED_ORDERS.O_TOTALPRICE) FROM V_MERGED_CUSTOMER, V_MERGED_ORDERS WHERE (V_MERGED_CUSTOMER.C_CUSTKEY=V_MERGED_ORDERS.O_CUSTKEY) AND (V_MERGED_CUSTOMER.C_MKTSEGMENT = 'MACHINERY ‘) GROUP BY V_MERGED_CUSTOMER.C_NATIONKEY
Query efficiently across data partitions in Oracle and SQLServer:
View v_merged_customer unions customer tables in both partitions.
View v_merged_order unions order tables in both partitions.
Query Plan #3 and Performance
Query Optimization
Search predicates on views are pushed down to data sources
Joins across union partitions are regrouped and pushed down to data sources, so that smaller joined result sets are combined in XIP
Performance:
average query time = 2 sec
Query plan:
SQL Query #4
Query Optimization
Shows stability and scalability in handling large result set with high concurrency without running out of memory.
The first record is returned immediately without waiting for the entire result set to be processed.
Ipedo support cursor fetching all the way through from client to Ipedo and to remote RDBMS.
Query can be cancelled without fetching the entire result set, and resources are released immediately.
SELECT ORDERS.O_ORDERKEY, ORDERS.O_ORDERDATE, ORDERS.O_CLERK FROM ORDERS
Select large result set (750,000 rows)
Query Performance #4
Cursor is used throughout the fetch chain to achieve high responsiveness:
Cursor is used from client to Ipedo:
Client obtains first row : 0.174s
Cursor is used from Ipedo to remote RDBMS:
Database returns first row in 0.171 s.
This shows cursor is used in back end when Ipedo fetches data from remote RDBMS.
Ipedo XIP Reference Architectures
Ipedo XIP Sizing Considerations
Number of Concurrent Users
Query Load
Number of queries
Complexity of queries (number of joins)
Amount of information being returned (row count)
Additional Processing
Data transformations (relational - XML)
Rules processing (number and complexity of rules)
High-Availability Requirements
Clustering (configuration)
Replication (writing or reading transaction logs)
Basic EII Server Load Concurrent users: < 75 Queries: < 50/sec Server Configuration CPUs: 1-4 RAM: 2 G Typical Usage Enable departmental reporting across multiple sources
Basic EII with Failover Ipedo XIP Primary Application-aware session pooling Ipedo XIP Secondary Server Load Concurrent users: < 75 Queries: < 50/sec Server Configuration CPUs: 1-4 RAM: 2 G Typical Usage Enable reporting across multiple sources for a business unit Ipedo can also use a clustering server for failover
HA Clustered - “Shared-Disk” Architecture Shared Disk Ipedo XIP Primary Ipedo XIP Secondary LAN Hub Private Network Windows Advanced Server Clustering Server Load Concurrent users: < 75/node Queries: < 50/sec/node Server Configuration CPUs: 1-4 per node RAM: 2 G per node Typical Usage Enable reporting across multiple sources for an enterprise or build an SOA data services layer Ipedo also supports clustering software on Linux, Solaris, HP-UX (HA = High Availability)
HA Clustered - “Shared-Nothing” Architecture SCSI Disk Ipedo XIP Primary Ipedo XIP Secondary LAN Hub Private Network Windows Advanced Server Clustering SCSI Disk Server Load Concurrent users: < 75/node Queries: < 50/sec/node Server Configuration CPUs: 1-4 per node RAM: 2 G per node Typical Usage Enable reporting across multiple sources for an enterprise or region, or build an SOA data layer Ipedo also supports clustering software on Linux, Solaris, HP-UX (HA = High Availability)
EII Server Farm (Clustered for Load Balancing) Ipedo XIP Ipedo XIP LAN Load Balancer Ipedo XIP Server Load Concurrent users: < 75/node Queries: < 50/sec/node Server Configuration CPUs: 1-4 per node RAM: 2 G per node Typical Usage Enable reporting across multiple sources for an enterprise or build an SOA data services layer
0 comments
Post a comment