Grid Wrapper  for  IBM Websphere Information Integrator <ul><li>Patrick Dantressangle  8/2005 </li></ul><ul><li>IBM Hursle...
Agenda <ul><li>The Data GRID: virtualization of data </li></ul><ul><li>OGSA-DAI and DAIS </li></ul><ul><li>Websphere II  &...
The Grid Distributed Computing Over a Network of Heterogeneous Resources Using Open Standards Grid Computing Storage Appli...
Desired World for Customers:  <ul><li>Customers need access to information from anyplace, anytime </li></ul><ul><li>Based ...
Virtualization of data on the Grid EMC IBM XXX TAPE Microsoft Windows VMWare/MSCS IBM AIX HACMP Sun Solaris HP/UX Linux (I...
OGSA-DAI <ul><li>DAIS – The GGF Database Access and Integration Working Group </li></ul><ul><ul><li>http:// cs.man.ac.uk /...
OGSA-DAI - Objective Define open standards and open source based uniform service interfaces for accessing heterogeneous da...
Grid Web The definition of WSRF means that Grid and Web  communities can move forward on a common base  Grid and Web Servi...
Some IBM Grid Data Activities GGF - OGSA OGSA   Data  Architecture UK eScience - OGSA DAI GGF - DAIS Working Group OGSA Da...
OGSA-DAI Project Partners
GDS Internals connection role data data data query perform  document response  document element element element credential...
OGSA-DAI Supported Data Resources        SQLServer PostgreSQL Oracle DB2 Files Xindice MySQL Other XML Relational
Some OGSA-DAI Customers…  more here  http://www.ogsadai.org.uk/projects/ OGSA-DAI (http://www.ogsadai.org.uk) AstroGrid (h...
Websphere Information Integrator
WebSphere Information Integrator Web Services Requests Event Consumption SQL, SQL/XML,  XPath,  XQuery (soon) Event Publis...
Heterogeneous Data Sources Text Sybase Informix SQL Server Oracle WebSphere II SQL, SQL/XML Federation Engine Wrappers and...
WebSphere II Web Services Overview Web Browser Soap  Client HTTP/SOAP HTTP/GET WebSphere  SQL HTTP/SOAP WebSphere II  appl...
Accessing Web Services from DB2 – Through User Defined Functions <ul><li>Can integrate SQL statements and Web Service invo...
DB2 Tables DB2  Tables XML XML 1. Publish 2. Transform & Manipulate 3. Store <ul><li>1. Publish: </li></ul><ul><li>SQL/XML...
Integration of XML & Relational Capabilities soon, beta in 2005 SERVER CLIENT SQL/XML XQuery DB2 Engine XML Interface Rela...
Websphere Information Integration : Wrapping it up! Websphere Information Integrator Wrapper Wrapper Plan  Query Plan  Que...
A Typical Environment, Today HTTP Server <ul><li>The goal: to achieve an Integrated, Service-Based Architecture </li></ul>...
An application produces a Report of the Sales <ul><li>grouping sales, per region, and showing current percentage of quota ...
...the application requires complex logic to implement federation <ul><li>// Merge the data: </li></ul><ul><li>for each cu...
Our Environment, with Information Integration HTTP Server Purchase Orders Reporting & Analysis Unix zOS  Customers Invento...
Our EJB using Information Integration <ul><li>Use SQL to query the virtual database, into which the Excel Spreadsheet appe...
Federation hides the Location and Source of the data behind a nickname <ul><li>A nickname,  for most cases,  is  just like...
With Information Integration the federation chore moves out of the application DB2  CUSTOMERS Oracle  ORDERS 100 rows  Exc...
Variations of the same Report are queried repeatedly: WII can cache it! <ul><li>DB2 will cache query results for Materiali...
WebSphere II: Replication <ul><li>Highlights: </li></ul><ul><li>Release independence (install “new” replication over “old”...
WebSphere II: Event Publishing   WS Classic EP For z/OS JDBC Client ODBC Client Workflow WebSphere Business Integration XM...
Summary: Federated Technology <ul><li>Infrastructure for extracting  information  from  data </li></ul><ul><ul><li>Wrapper...
The Grid wrapper for Websphere Information Integrator V8
What is the Grid Wrapper ? <ul><li>New technology developed here in Hursley </li></ul><ul><li>Bridging Websphere II v8 wit...
OGSA-DAI - today External resources External World OGSA/OGSI Grid External resources  represented as grid services = Grid ...
OGSA-DAI + Grid Wrapper External resources External World = Grid/WS-RF services External resources  represented as grid se...
Data and GRID/Web Services access patterns The Diamond graph. Data Registry Service Access, Performance Orchestration,  da...
Leveraging Both Worlds  OGSA-DAI Wrapper for instance OGSADAI  DBMS Boundary (Firewall or adminstrative,  Technology) Grid...
Grid/OGSADAI and Websphere II  <ul><li>Wrapper = registry which is pretty static less likely to change  </li></ul><ul><li>...
Some Customer Scenarios, leveraging OGSA-DAI & Websphere II <ul><li>SELECT:  Get data from first available or Preferred GR...
Quality of Service: AND 9.20.6.100 9.20.6.101 9.20.6.102 ? 9.20.6.100 9.20.6.101 9.20.6.102 select * from table Result: se...
Quality of Service: PARTIAL AND 9.20.6.100 9.20.6.101 9.20.6.102 ? 9.20.6.100 9.20.6.101 9.20.6.102 select * from table Re...
An example for Partial And: EDiaMoND Scenario (Part I) As it is today. ??? Training Services OGSA DAI Websphere II Data Fe...
An example for Partial And: EDiaMoND Scenario (Part II) with the WebSphere II wrapper. ??? Core API Training Application T...
Quality of Service: PREFERRED 9.20.6.100 9.20.6.101 9.20.6.102 ? 9.20.6.100 9.20.6.101 9.20.6.102 Result: select * from ta...
Quality of Service: OR 9.20.6.100 9.20.6.101 9.20.6.102 ? 9.20.6.100 9.20.6.101 9.20.6.102 Result: select * from table sel...
Quality of Service: Third Party Delivery 9.20.6.100 9.20.6.101 9.20.6.102 ? 9.20.6.100 9.20.6.101 9.20.6.102 Result: URLs ...
Wrapper Security – Components <ul><li>Makes use of: </li></ul><ul><ul><li>Grid Security Infrastructure (GSI)  http://www-u...
Create Wrapper statement <ul><li>CREATE WRAPPER ogsadai LIBRARY ‘db2qgjava.dll’ </li></ul><ul><ul><li>States the wrapper n...
Create Server <ul><li>Simple statement isn’t it? </li></ul><ul><li>CREATE SERVER ogsadai_server WRAPPER ogsadai </li></ul>...
Create Nickname statement <ul><li>CREATE NICKNAME WRAP.basetbl (pkey INT NOT NULL,   chr CHAR(1),   chr20 CHAR(20),   vchr...
DEMO NOW!
A basic grid wrapper XML registry <ul><li><?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?> </li></ul><ul><li><re...
Possibilities of this wrapper <ul><li>Using GRID registry as policy enforcement </li></ul><ul><ul><li>Move DB from one QoS...
Conclusion <ul><li>Websphere II can really be GRID enabled and be the unique simple front end for all your data on the GRI...
Questions/ Comments <ul><li>Patrick Dantressangle </li></ul><ul><ul><li>[email_address] </li></ul></ul>
Upcoming SlideShare
Loading in...5
×

Grid Wrapper for IBM Websphere Information Integrator - isegserv ...

658

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
658
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • What they would really like is depicted here, which is a virtual data store, where you specify to the data store the quality of service requirement and it provides that level of service and feedback on the kind of service that it provided. So this is driven through policy driven cache and replica management so they can specify the policies. Then the data store should make decisions about how that’s done: Autonomic data placement, which determines if I need a cache here or I need a replica here and it manages a virtual view across all of that. This is the ultimate desire of what they want. The future: we will provide you the tools and ways to set the policies so you can integrate your systems without worrying about where you are.
  • All these layers offer potential data virtualization. About SAN Volume Controller: The job of the SAN Volume Controller is to improve the flexibility of you disk storage and delivers specific results in the areas of: Improved Application Availability Eliminate many of the causes of storage-related downtime Create a common platform and API for volume Point-in-time and Remote copy services Optimized Storage Resource Utilization Aggregate smaller islands of spare disk capacity and transparently reallocate to new servers or applications Enhanced Storage Personnel Productivity Create a single point of control, administration and security for disk volumes Move, add or change physical disks without requiring application outages For instance, assuming EMC disks are more expensive than IBM storage which are more expensive than XXX storage and so on, SVC would allow for transparent migration of disk data to the cheapest storage device as the data age and become either obsolete or only used sparingly. About SAN FS: With today&apos;s file systems, you need to manage the file system for each server individually. The storage administrators need to assign storage volumes and monitor the usage for each server. With the SAN File System you will be able to manage the storage and data for all of the servers together. This in and of itself can save storage administration resources. In addition, the SAN File System provides automated file space allocation into storage pools that match the requirements for data. For example, data that needs high performance storage will be put into that storage pool and data that needs lower performance storage will be put into that storage pool. The servers will be able to use all of the types of storage without having to manually assign storage volumes to the individual servers. This can save storage administration resources in reducing tasks for moving data About OGSA-DAI:( OGSA - Data Access and Integration) Define open standards and open source based uniform service interfaces for accessing heterogeneous data sources within the Open Grid Services Architecture (OGSA). OGSA-DAI provides the middleware tools to grid-enable existing databases easily. It wraps data resources without hiding underlying data model. It also leverages all the OGSA Grid attributes (open standards, discovery, GRID registry, …). It’s a project joint funded by the UK DTI eScience programme and industry (IBM, Oracle) and it’s closely associated with the GGF DAIS working group. Project team members drawn from commercial and non-commercial organizations All these virtualization allow for =&gt; no single point of failure (not the same hardware, not the exact same software, disks). =&gt; moving “stuff” from one constructor/manufacturer to another in case of urgency or competitive deals. =&gt; flexibility to deal with unexpected threats/changes in technology
  • Standard based: GGF standards related to data GGF Architecture area working groups: OGSI GGF Data area working groups: DAIS GRID FTP DFDL (data format description language) GGF Data area research groups Replication Persistant archive Transport High performance networking Future GGF initiatives CMM (common management model)
  • OGSA-DAI - Team Members : Edinburgh Parallel Computing Centre UK National eScience Centre Manchester University Newcastle University Southampton University IBM US – Data Management IBM UK – Hursley Services and Technology Oracle UK
  • DB2 has the ability to access other web services.
  • This shows the problems faced by IT organization today. Efforts are underway to deploy a Service Based Architecture. Information Architects must integrate the whole enterprise. Today, application servers provide a unified interface to client applications, bringing huge productivity gains, but it is still a lot of work to deploy these servers: We observe: Multiple Data Sources and Formats structured and unstructured data, also semi-structured(XML) multiple of interfaces to proprietary systems Variety of commercial Applications Islands of DATA Islands of knowledge Product specific experts, DBAs, application programmers For example: How does the Reporting and Analysis function get at the data in the zOS and Unix systems? Today, need to use platform specific connection protocols and interfaces, often outside of the service oriented framework. Microsoft is moving closer to this with .NET and Web Service integration JMS = Java Messaging Service (part of Websphere run-time – one of the interfaces to MQSeries)
  • To generate this report, the application needs to read data from 3 different sources, join it; perform arithmetic operations on it; and aggregate it.
  • The federating application needs to be knowledgeable about each back-end interface (even if and when standard interfaces such as JDBC or ODBC are used). It needs to be able to break down a user query into sub-queries, often having to rewrite it for efficient access. The application needs t o connect to each back-end, open the spreadsheet... Despite the best of efforts, this approach eventually hits limit in the amount of complexity and data it can handle. Increasingly complex logic as queries are more elaborate – leads to heavy processing in application what if 4, or even more data sources? limited scalability It is also required that client code for Oracle and DB2 be installed on the application server.
  • Application developers need be concerned about a single data model . Native APIs for the back-ends are used by the federated DBMS; not by the application. The client code for the different data sources need only to be installed on the Information Integrator, it does not need to reside on the Development platform or the application server. When to use EAI versus ii? The short answer is probably &apos;when you need to merge data&apos; you&apos;d use the MQ integrated messaging. If all you need is application messaging you use EAI. EAI is transaction (record) oriented and does not easily allow you to merge data from multiple places. It is better at communicating with apps - so using EAI in an instead-of trigger to update a remote source might be apropos.
  • The Wrapper deals with each proprietary back-end. The nickname is just another type of table: one that resides elsewhere, and which data source may not be a table at all! The Information Integrator will also deal with differences in back-end capabilities, offering: function compensation – ie., supplying functions that the back-end does not support push-down – ie,. using capabilities of the back-end that are available when it makes sense (eg., sorts, search by index) The ii layer hides the location of the data. EJB developer needs not be aware of where the data actually resides, physically.
  • The Unified Data Model is a SINGLE-SITE IMAGE of the data, wherever it may resides. It leaves the tasks of merging, sorting, aggregating, programming to the proprietary APIs to the database engine. The DBA defines the data model for applications to use. The DBA can create views that join or union data from multiple tables (or other data representations) that reside on multiple data servers produced by different vendors. Views (and Materialized Query Tables (MQTs) can be defined over nicknames. Nicknames can be extended to several other sources, including application interfaces. They can be used in SQL queries just like any other table, in most cases. Some example of restrictions: a source may be read-only. Will need to modify bean code to avoid write attempts, if and when container-managed persistence is selected. DDL Transparency: A nickname may be created over an existing table or with transparent DDL: c reate a table on a remote non-DB2 DBMS, along with its associated local nickname, in a single command.  no need to learn the specific SQL syntax of the source to create a valid table.
  • The application connects to a single database.
  • Differences with WebSphere caching: WS DynaCache can be used to cache the results of JSP and servlets you are caching html, not the data. If you invoke the same servlet with the same parameters, then you can benefit from the cache. With DB cache: you will be able to take advantage of the cache for different queries that share the same result set, of subset of.
  • Capture program stages data in queues Each message represents a transaction One or more data transport queues per source/target database pair Apply is significantly re-architected Highly parallel in how it applies transactions to tables Data is always applied per source transaction units Data is applied such that source commit order is observed where necessary for data integrity Conflict detection very robust, including ability to handle deletes and key changes Data can also be published in XML format for external applications, using the same capture infrastructure Original story was SQL Replication – take the sql query and ship to the mirror side. Today added log-based capture for Q replication… Want to get a capture going for Oracle. Want to keep replication metadata changes in sync…. Need to have way to resolve conflicts (if both sides can be master copies).
  • The change is captured from the logfile – published the MQ queue and anyone that is wants this – they could grab it off the queue – can do it with IDMS, DB2 UDB z/OS and LUW (linux-unix-windows)
  • Decisions and boundaries to communication lead to different patterns, High Performance Requirement, knowledge of Data Location, understanding of data format, direct access permissions to data (Just 1) High Performance Requirement, understanding of data format, direct access permissions, dynamic location (3 then 1) High Performance Requirement, very limited understanding of data (location, access etc) (3 then 2 then 1) No direct access to data permitted, limited understanding of data format, knowledge of service representing/federating data (2) No direct access to data permitted, limited understanding of data format, no knowledge of location some knowledge of virtualised domain specific concepts (3 then 2 then 1) In all interactions along 2 orchestration or calls to other services for functions such as policy, security, QoS can take place. In all interactions 2 and 3 WS-notification can be present
  • Providing an OGSA-DAI wrapper leverage both data virtualization worlds : -&gt; schema integration and federation through Websphere II : single access layer to the GRID, leveraging current SQL knowledge. No need to rewrite applications and they can all benefit from the DB2 optimizer/compiler capabilities. -&gt; transparent access and loose coupling to remote data sources provided by OGSA-DAI middleware would allow for more scalability and flexibility in the choice of data sources. OGSA-DAI registries can be used for domain specific metadata repositories that can be changed without impacting Websphere II configuration. -&gt; Overall : it’s a very good story for huge scalable and easy to use data grid solutions.
  • Grid Wrapper for IBM Websphere Information Integrator - isegserv ...

    1. 1. Grid Wrapper for IBM Websphere Information Integrator <ul><li>Patrick Dantressangle 8/2005 </li></ul><ul><li>IBM Hursley – Emerging Technology Services </li></ul><ul><li>[email_address] </li></ul>
    2. 2. Agenda <ul><li>The Data GRID: virtualization of data </li></ul><ul><li>OGSA-DAI and DAIS </li></ul><ul><li>Websphere II & GRID </li></ul><ul><li>OGSA-DAI Wrapper features: an example of GRID technology with Websphere II. </li></ul><ul><li>Conclusion/questions </li></ul>
    3. 3. The Grid Distributed Computing Over a Network of Heterogeneous Resources Using Open Standards Grid Computing Storage Applications Processing Operating System Data I/O
    4. 4. Desired World for Customers: <ul><li>Customers need access to information from anyplace, anytime </li></ul><ul><li>Based upon Policy-managed quality of service (QoS) </li></ul><ul><li>Which will drive an automatic data placement </li></ul><ul><ul><li>Federation of data stores to utilize existing resources </li></ul></ul><ul><ul><li>Access to all relevant resources – structured, semi-structured, unstructured </li></ul></ul>Virtualized Information Request & QoS Response & Delivered QoS Placement Manager Data Policy App. App. App. App.
    5. 5. Virtualization of data on the Grid EMC IBM XXX TAPE Microsoft Windows VMWare/MSCS IBM AIX HACMP Sun Solaris HP/UX Linux (Intel) VMWare IBM BladeCenter VMWare/Windows/Linux OPM/FCS SAN DB2 MS Oracle MySQL XML DataSource access Over SOA File system access (SAN FS) Disk Access IBM Websphere information integrator OGSA DQP? (open source distributed query processor) Schema integration Federation Data Virtualization Global policy WS J2EE SQL API GRID registry WS OGSA-DAI OQL API OGSA-DAI Global Mngt SAN Volume Controller (SVC) OGSA-DAI (GRID data access middleware)
    6. 6. OGSA-DAI <ul><li>DAIS – The GGF Database Access and Integration Working Group </li></ul><ul><ul><li>http:// cs.man.ac.uk /grid-db </li></ul></ul><ul><li>OGSA-DAI – The UK e-Science funded reference implementation of DAIS </li></ul><ul><ul><li>http://ogsadai.org.uk/ </li></ul></ul><ul><li>IBM Emerging Technology Services : </li></ul><ul><ul><li>http:// tes.hursley.ibm.com / </li></ul></ul>
    7. 7. OGSA-DAI - Objective Define open standards and open source based uniform service interfaces for accessing heterogeneous data sources within the Open Grid Services Architecture (OGSA) Since WS-RF, This is now moving to a full web services architecture.
    8. 8. Grid Web The definition of WSRF means that Grid and Web communities can move forward on a common base Grid and Web Services: Convergence (an aside) WSRF Started far apart in apps & tech OGSI GT2 GT1 HTTP WSDL, WS-* WSDL 2, WSDM Have been converging
    9. 9. Some IBM Grid Data Activities GGF - OGSA OGSA Data Architecture UK eScience - OGSA DAI GGF - DAIS Working Group OGSA Data Services GGF – OGSI Working Group OASIS WSRF 2002 2003 2004 Today 2005
    10. 10. OGSA-DAI Project Partners
    11. 11. GDS Internals connection role data data data query perform document response document element element element credentials Query Activity Transform Activity Delivery Activity Data Resource Implementation Role Mapper The Engine connection
    12. 12. OGSA-DAI Supported Data Resources        SQLServer PostgreSQL Oracle DB2 Files Xindice MySQL Other XML Relational
    13. 13. Some OGSA-DAI Customers… more here http://www.ogsadai.org.uk/projects/ OGSA-DAI (http://www.ogsadai.org.uk) AstroGrid (http://www.astrogrid.org/) BioSimGrid (http://www.biosimgrid.org/) BioGrid (http://www.biogrid.jp/) Bridges (http://www.brc.dcs.gla.ac.uk/projects/bridges/) eDiaMoND (http://www.ediamond.ox.ac.uk/) FirstDig (http://www.epcc.ed.ac.uk/~firstdig/) GeneGrid (http://www.qub.ac.uk/escience/projects.php#genegrid) GEON (http://www.geongrid.org/) IU RGRBench (http://www.cs.indiana.edu/~plale/projects/RGR/OGSA-DAI.html) myGrid (http://www.mygrid.org.uk/) N2Grid (http://www.cs.univie.ac.at/institute/index.html?project-80=80) ODD-Genes (http://www.epcc.ed.ac.uk/oddgenes/) OGSA-WebDB (http://www.gtrc.aist.go.jp/dbgrid/)
    14. 14. Websphere Information Integrator
    15. 15. WebSphere Information Integrator Web Services Requests Event Consumption SQL, SQL/XML, XPath, XQuery (soon) Event Publishing Web Services JDBC, ODBC, XQJ, Web Services, JMS Heterogeneous Data Sources Enterprise Service Bus Enterprise Service Bus Information Virtualization Can mix and match Can mix and match Applications Applications Applications
    16. 16. Heterogeneous Data Sources Text Sybase Informix SQL Server Oracle WebSphere II SQL, SQL/XML Federation Engine Wrappers and functions Teradata WebSphere MQ ODBC IBM Extended Search Excel … WWW, email,… XML DB2 UDB on z/OS Software AG Adabas VSAM CA-IDMS IMS O D B C DB2 UDB Integrated SQL or XML View Web services Packaged Apps CA-Datacom Content Management systems DB2 II Classic Federation
    17. 17. WebSphere II Web Services Overview Web Browser Soap Client HTTP/SOAP HTTP/GET WebSphere SQL HTTP/SOAP WebSphere II applications consume Web Services data WebSphere II provides Web Services data SQL Applications Soap Client DB2 Web Service Provider (WORF) WebSphere II Tables Stored Procedures XML Extender Web Service Providers Web Service Wrapper or UDFs Dynamic Query Service DADX
    18. 18. Accessing Web Services from DB2 – Through User Defined Functions <ul><li>Can integrate SQL statements and Web Service invocations </li></ul><ul><li>Support for generating SQL scalar and table UDFs (User Defined Functions) based on WSDL (web service description) </li></ul><ul><ul><li>Command line version </li></ul></ul><ul><ul><li>Tool support integrated into Web Sphere Studio </li></ul></ul>SELECT city, GetTemperature(city) FROM location Web Airline Fare Language Translate Currency Rate Stock Quote Temperature DB2 Virtual Web Data Service Providers
    19. 19. DB2 Tables DB2 Tables XML XML 1. Publish 2. Transform & Manipulate 3. Store <ul><li>1. Publish: </li></ul><ul><li>SQL/XML Functions </li></ul><ul><li>XML Extender Composition </li></ul><ul><li>DB2 Web Services </li></ul><ul><li>2. Transform & Manipulate: </li></ul><ul><li>Validate (schema and DTD) </li></ul><ul><li>XSL </li></ul><ul><li>Extract (fragments or values through XPath) </li></ul><ul><li>Update </li></ul><ul><li>3. Store: </li></ul><ul><li>Intact as CLOB or as XML Extender Type </li></ul><ul><li>Shredded into relational through XML Extender Decomposition </li></ul><ul><li>DB2 Web Services </li></ul>Today: XML Support in DB2 For references and more information see http://www.ibm.com/developerworks/db2/library/techarticle/0212malaika/0212malaika.html
    20. 20. Integration of XML & Relational Capabilities soon, beta in 2005 SERVER CLIENT SQL/XML XQuery DB2 Engine XML Interface Relational Interface DB2 Storage: DB2 Client / Customer Client Application <ul><ul><li>Native XML data type (server & client side) </li></ul></ul><ul><ul><ul><li>(not Varchar, not CLOB, not object-relational !) </li></ul></ul></ul><ul><ul><li>XML Capabilities in all DB2 components </li></ul></ul><ul><ul><li>Applications combine XML & relational data </li></ul></ul>Relational XML
    21. 21. Websphere Information Integration : Wrapping it up! Websphere Information Integrator Wrapper Wrapper Plan Query Plan Query Planning Query Execution Generic/ODBC, Oracle, SQL Server, Sybase, Informix, Teradata, SQL Anywhere, Flat file, Excel, XML, Documentum, BLAST,… <ul><li>Encapsulate data source information inside a wrapper </li></ul><ul><ul><li>Models data as tables </li></ul></ul><ul><ul><li>Expose unique function </li></ul></ul><ul><li>Wrapper participates in query planning and execution </li></ul><ul><li>Easy to add for unusual data sources </li></ul>Plan request Execution Plan
    22. 22. A Typical Environment, Today HTTP Server <ul><li>The goal: to achieve an Integrated, Service-Based Architecture </li></ul>XML Web Clients Business Partners Enterprise Portals Internal Applications XML or Web Service Data Feeds Windows Purchase Orders Reporting & Analysis Unix zOS Customers Inventory System DB2/MVS IMS Oracle DB2 Excel Content Manager DB2 Digital media Web Server Web Service Web Services EJBs Servlets and JSPs Websphere AS JMS Worflows
    23. 23. An application produces a Report of the Sales <ul><li>grouping sales, per region, and showing current percentage of quota attained. </li></ul><ul><li>The information comes from 3 systems: the EJB needs to federate this data. </li></ul>Purchase Orders Oracle Monthly Sales Report EJBs Servlets and JSPs Application <ul><li>The Application needs connectivity to each system; load and merge the data; and aggregate it for the report. </li></ul>Customers DB2/MVS Sales Quotas Excel Monthly Sales per region San Jose S. Shanghai Orlando ... 100.5 218.0 88.7
    24. 24. ...the application requires complex logic to implement federation <ul><li>// Merge the data: </li></ul><ul><li>for each customer in custs: </li></ul><ul><li>for each order in ords </li></ul><ul><li>if order.custid == customer.custid </li></ul><ul><li>region.sales += </li></ul><ul><li>(order.qty * order.price) </li></ul><ul><li>i=i+1; </li></ul><ul><li>. . . etc ... </li></ul>SELECT custid, qty, price FROM ORDERS ORDER BY custid into ords; SELECT custid, region FROM CUSTOMES ORDER BY custid into custs; Monthly Report Service Federate <ul><li>10 000 orders/month </li></ul><ul><li>2 000 000 customers </li></ul>10 000 rows 2 000 000 rows 100 rows Thousands of lines of Java code! 2 000 records <ul><li>2000 salespersons </li></ul><ul><li>The EJB needs a connection to each back-end ! </li></ul><ul><li>JDBC helps accessing Oracle/DB2 data, but what about Excel? </li></ul><ul><li>Millions of rows loaded into the application, stressing the JVM! </li></ul><ul><li>Joining large sets of data complex and slow! </li></ul>JDBC Java file i/o JDBC MBs of memory required! Only 100 regions with sales Websphere AS DB2 CUSTOMERS Oracle ORDERS Excel SALES Quotas Monthly Sales per region San Jose S. Shanghai Orlando ... 100.5 218.0 88.7
    25. 25. Our Environment, with Information Integration HTTP Server Purchase Orders Reporting & Analysis Unix zOS Customers Inventory System DB2/MVS IMS Oracle DB2 Excel Content Manager DB2 Digital media XML Web Clients Business Partners Enterprise Portals Internal Applications XML or Web Service Data Feeds Information Integrator meta-data Web Services EJBs Servlets and JSPs Unified Data Model Federation Web Server Web Service application Windows <ul><li>All Information Sources appear as if from the same database! </li></ul>
    26. 26. Our EJB using Information Integration <ul><li>Use SQL to query the virtual database, into which the Excel Spreadsheet appears as just another table ! </li></ul>Purchase Orders Unix Oracle Monthly Sales Report EJBs Servlets and JSPs Application <ul><li>Each data source appears as a local table: via a nickname </li></ul><ul><li>Information Integration efficiently merges and aggregate data from all sources, transparently! </li></ul>Information Integrator Unified Data Model Federation Customers DB2/MVS Sales Quotas Excel JDBC ORDERS CUSTOMERS SALESQUOTA Native APIs nicknames:
    27. 27. Federation hides the Location and Source of the data behind a nickname <ul><li>A nickname, for most cases, is just like any other local table ! </li></ul>ORDERS table Oracle CUSTOMERS table DB2/MVS QUOTAS spreadsheet Excel Unified Data Model: CUSTOMERS nickname ORDERS nickname QUOTAS nickname ORDSTATS view Application : SELECT c.region, SUM(o.price * o.qty) FROM CUSTOMERS c, ORDSTATS o WHERE c.custid = o.custid GROUP BY c.region; DBA: CREATE NICKNAME ORDERS FOR ORACLE8.bourbon.ORDERS; CREATE VIEW ORDSTATS ... or, if table does not exist: CREATE TABLE ORDERS(id primary key... ... OPTIONS(REMOTE_SERVER 'ORACLE8' REMOTE_SCHEMA 'bourbon')
    28. 28. With Information Integration the federation chore moves out of the application DB2 CUSTOMERS Oracle ORDERS 100 rows Excel SALES Quotas SELECT region, SUM(o.price * o.qty) FROM SALES WHERE month(sdate) = CURRENT MONTH GROUP BY c.region; A single simple database query Single View Monthly Report Service <ul><li>The application still uses JDBC, but only worries about 1 data model </li></ul><ul><li>Nicknames allow automatic type mappings and transformations </li></ul>ORDERS nickname CUSTOMERS nickname SALESQUOTA nickname SALES view 100 rows Application JDBC A few 100 bytes of memory required! Information Integrator Federation Monthly Sales per region San Jose S. Shanghai Orlando ... 100.5 218.0 88.7
    29. 29. Variations of the same Report are queried repeatedly: WII can cache it! <ul><li>DB2 will cache query results for Materialized Query Table (MQT) </li></ul><ul><li>DB2 optimizer also picks MQT for other similar queries! </li></ul>SELECT region, SUM(o.price * o.qty) FROM SALES WHERE month(sdate) = CURRENT MONTH GROUP BY c.region; Monthly Report Service Application DB2 CUSTOMERS Oracle ORDERS Excel SALES Quotas ORDERS nickname CUSTOMERS nickname SALESQUOTA nickname SALES view JDBC MQT Cache A single simple database query No need to re-fetch the data! Information Integrator Federation Monthly Sales per region San Jose S. Shanghai Orlando ... 100.5 218.0 88.7
    30. 30. WebSphere II: Replication <ul><li>Highlights: </li></ul><ul><li>Release independence (install “new” replication over “old” UDB) </li></ul><ul><li>Add Oracle log-based capture for Q Replication </li></ul><ul><li>Enable Classic sources to participate in Q Replication (VSAM, IMS, IDMS) </li></ul><ul><li>Enable Heterogeneous targets </li></ul><ul><li>Replication of schema changes (starting with Alter Table/Column) </li></ul><ul><li>Expand peer-to-peer conflict resolution schemes (merge, user-defined) </li></ul>Federation Engine WebSphere MQ Heterogeneous Introduction to Replication : ftp://ftp.software.ibm.com/ps/products/db2/info/vr82/pdf/en_US/db2gpe80.pdf Control Control Admin Capture Target Apply DB2 II Classic EP for z/OS VSAM IMS IDMS Log Based Oracle DB2
    31. 31. WebSphere II: Event Publishing WS Classic EP For z/OS JDBC Client ODBC Client Workflow WebSphere Business Integration XML message <ul><li>Asynchronous communication between data and application layers to build powerful solutions </li></ul><ul><li>Key capability of DB2 II for both distributed and “classic” platforms </li></ul><ul><li>New – addition of IDMS sources </li></ul>IDMS WS II Event Publisher DB2 UDB for z/OS VSAM IMS DB2 UDB for LUW
    32. 32. Summary: Federated Technology <ul><li>Infrastructure for extracting information from data </li></ul><ul><ul><li>Wrappers allow access to key data sources </li></ul></ul><ul><ul><li>Query engine provides cross-source queries </li></ul></ul><ul><li>An extension of industrial-strength DB2 technology </li></ul><ul><ul><li>Robust, high function, high performance </li></ul></ul><ul><ul><li>Benefit from R & D in relational DBMS </li></ul></ul><ul><ul><li>Benefit from experience in content management </li></ul></ul><ul><li>Eases application development </li></ul><ul><ul><li>Transparency, heterogeneity, high function </li></ul></ul><ul><ul><li>APIs for modern environments (XML, J2EE, Web Services) </li></ul></ul><ul><li>Application Autonomy </li></ul><ul><ul><li>Does not disrupt existing applications </li></ul></ul>
    33. 33. The Grid wrapper for Websphere Information Integrator V8
    34. 34. What is the Grid Wrapper ? <ul><li>New technology developed here in Hursley </li></ul><ul><li>Bridging Websphere II v8 with the standard based Grid world to hide data heterogeneity on a datagrid as well as introducing new access patterns. </li></ul><ul><li>Released as an IBM Alphaworks technology in August 2005 </li></ul><ul><ul><li>http://www.alphaworks.ibm.com/ </li></ul></ul>
    35. 35. OGSA-DAI - today External resources External World OGSA/OGSI Grid External resources represented as grid services = Grid services Note: The application has to “integrate/federate” all the data from the different GRID services. Grid enabled application !!!! Schema integration and federation of Data grid services were features required by OGSA-DAI users in the 2004 user group meeting in Edinburgh.!!!!!
    36. 36. OGSA-DAI + Grid Wrapper External resources External World = Grid/WS-RF services External resources represented as grid services OGSA-DAI + Websphere II OGSI/WS RF Grid Standard DB2 enabled application registry
    37. 37. Data and GRID/Web Services access patterns The Diamond graph. Data Registry Service Access, Performance Orchestration, data format, federation Dynamic, Discovery Registration, Discovery Virtualisation Arrow indicates interaction Other Services 1 2 3 Websphere II Client
    38. 38. Leveraging Both Worlds OGSA-DAI Wrapper for instance OGSADAI DBMS Boundary (Firewall or adminstrative, Technology) Grid Websphere II OGSA-DAI Reuse DBMS specific wrapper where possible WAS Application Application SQL SQL SQL Schema integration WS/Grid Service Discovery/Creation SQL
    39. 39. Grid/OGSADAI and Websphere II <ul><li>Wrapper = registry which is pretty static less likely to change </li></ul><ul><li>Server = type of Resource knows how to map information from this type of resource, not tied to a specific resource but is tied to the virtualisation of a particular resource </li></ul><ul><li>Nickname tied to particular table schema or set of data which can be returned from a service, ties it to a domain and data set but not specific set of data in a specific location. </li></ul>
    40. 40. Some Customer Scenarios, leveraging OGSA-DAI & Websphere II <ul><li>SELECT: Get data from first available or Preferred GRID Data Resource (GRID-DR) </li></ul><ul><ul><li> OR or PREFERRED QoS </li></ul></ul><ul><li>UNION ALL: </li></ul><ul><ul><li>Select and consolidate all result sets from all GRID-DR </li></ul></ul><ul><ul><li> AND QoS </li></ul></ul><ul><li>UNION ALL WITH PARTIAL RESULT SETS </li></ul><ul><ul><li>Select and consolidate all result sets from GRID-DR Data Resource, and it may not matter if one or many GRID-DR is down, but get a warning back that not all data were fetched. (see EDiaMoND type scenario) </li></ul></ul><ul><ul><ul><li> AND and PARTIAL AND QoS </li></ul></ul></ul><ul><li>Third Party delivery: </li></ul><ul><ul><li>results sets will be delivered to another place (FTP site, WEB Server) and result set holds URL of where to fetch the data from. </li></ul></ul>
    41. 41. Quality of Service: AND 9.20.6.100 9.20.6.101 9.20.6.102 ? 9.20.6.100 9.20.6.101 9.20.6.102 select * from table Result: select * from table select * from table Registry
    42. 42. Quality of Service: PARTIAL AND 9.20.6.100 9.20.6.101 9.20.6.102 ? 9.20.6.100 9.20.6.101 9.20.6.102 select * from table Result: select * from table select * from table Registry
    43. 43. An example for Partial And: EDiaMoND Scenario (Part I) As it is today. ??? Training Services OGSA DAI Websphere II Data Federation (static union all view) Core Services OGSA DAI Data Load Core API DB2 Content Manager CHU Core Services OGSA DAI Core API DB2 Content Manager UCL Data Load Core Services OGSA DAI Core API DB2 Content Manager UED Data Load Core Services OGSA DAI Core API DB2 Content Manager KCL Data Load Core API Training Application Training API Local Files Database OGSA DAI Core & Training API Core & Training API Core & Training API Core & Training API Training Appl Training Appl Training Appl Training Appl
    44. 44. An example for Partial And: EDiaMoND Scenario (Part II) with the WebSphere II wrapper. ??? Core API Training Application Training API Local Files Database Training Services OGSA DAI Websphere II Data Federation (using OGSA-DAI wrapper: fully dynamic : no union all view!! ) Registry Core Services OGSA DAI Data Load Core API DB2 Content Manager CHU Core Services OGSA DAI Core API DB2 Content Manager UCL Data Load Core Services OGSA DAI Core API DB2 Content Manager UED Data Load Core Services OGSA DAI Core API DB2 Content Manager KCL Data Load OGSA DAI Core & Training API Core & Training API Core & Training API Core & Training API Training Appl Training Appl Training Appl Training Appl
    45. 45. Quality of Service: PREFERRED 9.20.6.100 9.20.6.101 9.20.6.102 ? 9.20.6.100 9.20.6.101 9.20.6.102 Result: select * from table Registry
    46. 46. Quality of Service: OR 9.20.6.100 9.20.6.101 9.20.6.102 ? 9.20.6.100 9.20.6.101 9.20.6.102 Result: select * from table select * from table Registry
    47. 47. Quality of Service: Third Party Delivery 9.20.6.100 9.20.6.101 9.20.6.102 ? 9.20.6.100 9.20.6.101 9.20.6.102 Result: URLs select * from table select * from table FTP/GRIDFTP/HTTP Server holding result sets URLs URLs Client Registry
    48. 48. Wrapper Security – Components <ul><li>Makes use of: </li></ul><ul><ul><li>Grid Security Infrastructure (GSI) http://www-unix.globus.org/toolkit/docs/3.2/security.html </li></ul></ul><ul><ul><li>Grid Certificate Service (GCS) http://gcs.globus.org:8080/gcs/index.html </li></ul></ul><ul><ul><li>Java Commodity Grid Kit (Java CoG Kit) http://www.cogkit.org </li></ul></ul>
    49. 49. Create Wrapper statement <ul><li>CREATE WRAPPER ogsadai LIBRARY ‘db2qgjava.dll’ </li></ul><ul><ul><li>States the wrapper name and the wrapper library to be used (part of Websphere II). In this case a Java non-relational wrapper. </li></ul></ul><ul><li>UNFENCED_WRAPPER_CLASS and FENCED_WRAPPER_CLASS </li></ul><ul><ul><li>Directs DB2 to the Java classes implementing the necessary logic. </li></ul></ul><ul><li>REGISTRY_TYPE and FILENAME </li></ul><ul><ul><li>Points to the XML registry in this case. </li></ul></ul><ul><ul><li>Can also point to a relational registry </li></ul></ul><ul><li>CREATE WRAPPER “ogsadai&quot; LIBRARY 'db2qgjava.dll‘ </li></ul><ul><li>OPTIONS( DB2_FENCED 'Y', FENCED_WRAPPER_CLASS 'com.ibm.db2.wrapper.onDemand.OnDemandFencedWrapper', </li></ul><ul><li>FILENAME 'C:gridwrapperin egistry.xml', </li></ul><ul><li>REGISTRY_TYPE 'XMLFILE‘, </li></ul><ul><li>UNFENCED_WRAPPER_CLASS 'com.ibm.db2.wrapper.onDemand.OnDemandUnfencedWrapper‘ </li></ul><ul><li>); </li></ul>
    50. 50. Create Server <ul><li>Simple statement isn’t it? </li></ul><ul><li>CREATE SERVER ogsadai_server WRAPPER ogsadai </li></ul><ul><ul><li>States name of server and which wrapper it belongs to. </li></ul></ul><ul><li>Options about service quality is possible here but we prefer to locate them in the registry. </li></ul><ul><li>CREATE SERVER “ogsadai_server&quot; WRAPPER &quot;THEGRIDWRAPPERBASIC“; </li></ul>
    51. 51. Create Nickname statement <ul><li>CREATE NICKNAME WRAP.basetbl (pkey INT NOT NULL, chr CHAR(1), chr20 CHAR(20), vchr20 VARCHAR(20), intgr INT, flt DOUBLE, mny DECIMAL(19,4), sdt TIMESTAMP, bt SMALLINT) </li></ul><ul><li>FOR SERVER ogsadai_server </li></ul><ul><li>OPTIONS (TABLE_NAME ‘WRAP.basetbl') </li></ul><ul><li>CREATE NICKNAME WRAP.basetbl </li></ul><ul><ul><li>Nickname identifier , here in schema WRAP </li></ul></ul><ul><li>pkey INT NOT NULL, chr CHAR(1),chr20 CHAR(20)… </li></ul><ul><ul><li>Defines the nickname columns, this can be done dynamically. </li></ul></ul><ul><li>FOR SERVER ogsadai_server OPTIONS (TABLE_NAME 'basetbl') </li></ul><ul><ul><li>Indicates which server will be used as well as the logical table name, the one to be matched with information stored in the registry. </li></ul></ul>
    52. 52. DEMO NOW!
    53. 53. A basic grid wrapper XML registry <ul><li><?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?> </li></ul><ul><li><registry version=&quot;0.0.0&quot; expireafter=&quot;0&quot;> </li></ul><ul><li><class id=&quot;DB2&quot; vendor=&quot;IBM&quot; product=&quot;DB2&quot; version=&quot;-&quot;/> </li></ul><ul><li><class id=&quot;MySQL&quot; vendor=&quot;MySQL&quot; product=&quot;MySQL&quot; version=&quot;-&quot;/> </li></ul><ul><li><transform target=&quot;DB2&quot; fname=“/gridwrapper/TransformPatterns/db2.transform“ kind=&quot;patterns&quot;/> </li></ul><ul><li><transform target=&quot;MySQL&quot; fname=“/gridwrapper/TransformPatterns/mysql.transform&quot; kind=&quot;patterns&quot;/> </li></ul><ul><li><connectivity id=&quot;DB2 Test DB JDBC&quot; type=&quot;JDBC&quot; implementation=&quot;com.ibm.db2.jcc.DB2Driver“ </li></ul><ul><li>url=&quot;jdbc:db2://myhost:50000/MYDB“ userid=&quot;db2localuser“ password=&quot;db2localpass“ /> </li></ul><ul><li><connectivity id=&quot;MySQL Test DB OGSADAI“ type=&quot; OGSADAI &quot; </li></ul><ul><li>url=&quot; http://localhost:8080/ogsa/services/ogsadai/DBSGDSF “ /> </li></ul><ul><li><table logicalname=“WRAP.demotest&quot; assemblage=&quot;all-ns&quot;> </li></ul><ul><li><table assemblage=&quot;direct&quot; using=&quot;MySQL Test DB OGSADAI &quot; physicalname=“DB1.demotest“ class=&quot;MySQL“ /> </li></ul><ul><li><table assemblage=&quot;direct“ using=&quot;DB2 Test DB JDBC“ physicalname=“MyDB.demotest“ class=&quot;DB2“ /> </li></ul><ul><li></table> </li></ul><ul><li><table logicalname=“WRAP.basetbl&quot; assemblage=&quot;direct&quot; </li></ul><ul><li>using=&quot;MySQL Test DB JDBC&quot; physicalname=“DB1.basetbl“ class=&quot;MySQL“ /> </li></ul><ul><li></registry> </li></ul>
    54. 54. Possibilities of this wrapper <ul><li>Using GRID registry as policy enforcement </li></ul><ul><ul><li>Move DB from one QoS to another by moving them in and out of registries.(domain specific registries) </li></ul></ul><ul><ul><li>Change data space topology by just interacting with registries. No impact on the applications/SQL logic. No need to recreate nicknames if DB move from one system onto another. </li></ul></ul><ul><ul><li>DBA liked this idea very much. (see demo for GUI) </li></ul></ul><ul><ul><li>Switch between local access (JDBC..) to SOA depending of application requirement/infrastructure needs. see diamond graph in a previous slide. </li></ul></ul><ul><li>Leverage distribution of data by fetching from multiple GRID data sources in parallel. </li></ul><ul><li>Many more scenario where loose coupling is involved… </li></ul>
    55. 55. Conclusion <ul><li>Websphere II can really be GRID enabled and be the unique simple front end for all your data on the GRID </li></ul><ul><li>It is available on IBM AlphaWorks technology for free </li></ul><ul><ul><li>http:// alphaworks.ibm.com/tech/gridwrapper </li></ul></ul><ul><ul><li>A tutorial is provided for easy set-up and training </li></ul></ul><ul><li>Read the article “ Bridging the integration gap, Part 1: Federating grid data “ on IBM DeveloperWorks web site: </li></ul><ul><ul><li>It’s about why we built it and the vision behind it. </li></ul></ul><ul><ul><li>http://www-128.ibm.com/developerworks/grid/library/gr-feddata/ </li></ul></ul><ul><li>We will add more features as customers ask for them. </li></ul>
    56. 56. Questions/ Comments <ul><li>Patrick Dantressangle </li></ul><ul><ul><li>[email_address] </li></ul></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×