BID210: Using Sybase IQ  as a Report Server for Oracle John Barton Principal Systems Consultant [email_address] August 15-...
The Enterprise. Unwired.
The Enterprise. Unwired. Unwire People Unwire Information Manage Information Sybase Workspace Industry and Cross Platform ...
Moving Oracle Reporting into Sybase IQ <ul><li>Migrate DDL </li></ul><ul><li>Initial Data Load </li></ul><ul><li>Increment...
Migrating DDL from Oracle to Sybase IQ <ul><li>The first step is to convert DDL from Oracle to Sybase IQ </li></ul><ul><ul...
Data Type Mapping Oracle to Sybase IQ Use IQ 12.5 ESD 11 or higher with LOB
Using a Text Editor to Migrate DDL <ul><li>sed – UNIX utility for search and replace </li></ul><ul><li>Example: Replace nu...
Using PowerDesigner to Migrate DDL <ul><li>1. Reverse Engineer Oracle Database </li></ul><ul><ul><li>DDL Script </li></ul>...
PowerDesigner DBMS Resource Editor Oracle 8, 9, 10 Sybase IQ 12.5 Source Database (PhysDataType) PowerDesigner Internal Da...
Source Data Type Mapping to Internal Types
Target Data Type Mapping from Internal Types
Other DDL Mapping Considerations <ul><li>Table of data types and mapping </li></ul><ul><li>Trailing space issue </li></ul>...
Extracting Data from Oracle <ul><li>Using Sybase Direct Connect for Oracle </li></ul><ul><li>Using SQLPlus for Data Extrac...
Loading Directly from Oracle <ul><li>Sybase Enterprise Connect Product Family </li></ul><ul><ul><li>DirectConnect for Orac...
Loading Sybase IQ From Oracle DirectConnect  for  Oracle Sybase IQ Oracle Database 1 2 3 4 Client Application <ul><li>Sequ...
Loading From Oracle – Continued Sequence Steps - continued 5.  The DCO Access Service passes the SQL statement to the  Ora...
Sybase IQ Syntax to Load from a Remote Table <ul><li>Connect to Sybase IQ and issue the following command </li></ul><ul><l...
DirectConnect – Load Performance <ul><li>Customer Testimonial </li></ul><ul><li>“We run batch processes at night that use ...
Extracting Data from Oracle Using SQLPlus <ul><li>A SQL Plus script can be used to extract data from Oracle Database </li>...
SQLPlus Extraction Script <ul><li>A SQL Plus script </li></ul><ul><li>--Set the line size to accommodate the longest possi...
Load Processing Techniques <ul><li>Initial Load </li></ul><ul><li>Ongoing Load </li></ul><ul><li>Incremental Load Techniqu...
ETL Examples - Managing Data Loads <ul><li>Tables and Dimensions can be categorized as follows: </li></ul>* Dimensions as ...
Slowly Growing Dimensions or Tables <ul><li>CREATE TABLE CUST </li></ul><ul><li>( </li></ul><ul><li>CUST_ID  INTEGER, </li...
Loading Type 1 Dimension Tables <ul><li>INSERT INTO STAGE_SUPPLIER ( S_NAME,  S_ADDRESS, S_NATIONKEY,   S_PHONE,   S_ACCTB...
Loading Type 2 Dimension Tables UPDATE SUPPLIER SET S_CURR_FLAG = ‘N’ FROM STAGE_SUPPLIER, SUPPLIER WHERE STAGE_SUPPLIER.S...
Generating Sequence Numbers for Keys <ul><li>Adding Sequence Numbers / Generating Warehouse Keys </li></ul>CREATE TABLE ST...
Why Use Sybase IQ over Oracle <ul><li>ANSI SQL </li></ul><ul><li>Reduced Administration </li></ul><ul><li>Reduced Ongoing ...
Changing Tools from Oracle to Sybase IQ <ul><li>Query Tools </li></ul><ul><li>SQL </li></ul><ul><ul><li>Standard SQL 92 En...
Upcoming SlideShare
Loading in …5
×

Business Intelligence

487
-1

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
487
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • As you will see throughout this week, Sybase creates solutions that guide your data from data center to device through the Enterprise - Unwired. To build an Unwired Enterprise, you need to do three things: Manage enterprise information – make that information useful, relevant and valuable Unwire the information across the enterprise – ensuring fast, reliable access in real-time and unwire the people – enabling them to work wherever they need to, securing data across networks, systems and devices.
  • The Unwired Enterprise is an information management strategy. In this session, we’ll be discussing this (POINT OUT YOUR PRODUCT OR SOLUTION AREA) section of Sybase’s Unwired Enterprise solution.
  • Here is how you copy the definition file: * Click on Tools -&gt; Resources -&gt; DBMS * Click on the New icon on the toolbar * Make the name something like &amp;quot;{original name} - modified&amp;quot; * Select the source file from the pull-down list and click OK * When prompted to save the file, find the original file name and append something, like &amp;quot;-mod&amp;quot; * Now you will see your modified version just after the original version in the list of DBMS definitions * After the file is saved, the new definition file will automatically open for editing * Change the name, again by adding &amp;quot; - modified&amp;quot; to match the name used earlier
  • Business Intelligence

    1. 1. BID210: Using Sybase IQ as a Report Server for Oracle John Barton Principal Systems Consultant [email_address] August 15-19, 2004
    2. 2. The Enterprise. Unwired.
    3. 3. The Enterprise. Unwired. Unwire People Unwire Information Manage Information Sybase Workspace Industry and Cross Platform Solutions <ul><li>Adaptive Server Enterprise </li></ul><ul><li>Adaptive Server Anywhere </li></ul><ul><li>Sybase IQ </li></ul><ul><li>Dynamic Archive </li></ul><ul><li>Dynamic ODS </li></ul><ul><li>Replication Server </li></ul><ul><li>OpenSwitch </li></ul><ul><li>Mirror Activator </li></ul><ul><li>PowerDesigner </li></ul><ul><li>Connectivity Options </li></ul><ul><li>EAServer </li></ul><ul><li>Industry Warehouse Studio </li></ul><ul><li>Unwired Accelerator </li></ul><ul><li>Unwired Orchestrator </li></ul><ul><li>Unwired Toolkit </li></ul><ul><li>Enterprise Portal </li></ul><ul><li>Real Time Data Services </li></ul><ul><li>SQL Anywhere Studio </li></ul><ul><li>M-Business Anywhere </li></ul><ul><li>Pylon Family (Mobile Email) </li></ul><ul><li>Mobile Sales </li></ul><ul><li>XcelleNet Frontline Solutions </li></ul><ul><li>PocketBuilder </li></ul><ul><li>PowerBuilder Family </li></ul><ul><li>AvantGo </li></ul>
    4. 4. Moving Oracle Reporting into Sybase IQ <ul><li>Migrate DDL </li></ul><ul><li>Initial Data Load </li></ul><ul><li>Incremental Data Loads </li></ul><ul><li>Reporting SQL </li></ul>
    5. 5. Migrating DDL from Oracle to Sybase IQ <ul><li>The first step is to convert DDL from Oracle to Sybase IQ </li></ul><ul><ul><li>Two recommended methods for migrating DDL </li></ul></ul><ul><ul><li>Oracle DDL Scripts </li></ul></ul><ul><ul><ul><li>Data Type Mapping Oracle to Sybase IQ – next slide </li></ul></ul></ul><ul><ul><ul><li>Search and Replace in a text editor </li></ul></ul></ul><ul><ul><li>PowerDesigner Model Migration </li></ul></ul><ul><ul><ul><li>Use PowerDesigner to reverse engineer the DDL </li></ul></ul></ul><ul><ul><ul><li>Change the Target Database from Oracle to Sybase IQ </li></ul></ul></ul><ul><ul><ul><li>Verify Translation </li></ul></ul></ul>
    6. 6. Data Type Mapping Oracle to Sybase IQ Use IQ 12.5 ESD 11 or higher with LOB
    7. 7. Using a Text Editor to Migrate DDL <ul><li>sed – UNIX utility for search and replace </li></ul><ul><li>Example: Replace number(1) with tinyint </li></ul><ul><li>sed 's/number(1)/tinyint/' ddl_1.sql > ddl_2.sql </li></ul><ul><li>Textpad – http://www.textpad.com/ great text editor for full control over search and replace </li></ul>
    8. 8. Using PowerDesigner to Migrate DDL <ul><li>1. Reverse Engineer Oracle Database </li></ul><ul><ul><li>DDL Script </li></ul></ul><ul><ul><li>ODBC DSN </li></ul></ul><ul><li>2. Validate Model </li></ul><ul><li>3. Change the Target Database </li></ul><ul><ul><li>Current: Oracle </li></ul></ul><ul><ul><li>New: Sybase IQ </li></ul></ul>
    9. 9. PowerDesigner DBMS Resource Editor Oracle 8, 9, 10 Sybase IQ 12.5 Source Database (PhysDataType) PowerDesigner Internal Data Type Target Database (AMCDDataType) <ul><li>Customize DBMS Using Resource Editor </li></ul><ul><li>This provides granular control on how data types are converted between source and target databases. </li></ul><ul><li>DBMS Properties can be customized (edited) </li></ul><ul><ul><li>Click on Tools -> Resources -> DBMS </li></ul></ul>Create a new copy of the DBMS definition file.
    10. 10. Source Data Type Mapping to Internal Types
    11. 11. Target Data Type Mapping from Internal Types
    12. 12. Other DDL Mapping Considerations <ul><li>Table of data types and mapping </li></ul><ul><li>Trailing space issue </li></ul><ul><ul><li>In Oracle Trailing spaces matter </li></ul></ul><ul><li>Data Type mapping issues </li></ul><ul><li>Understanding Data Domains </li></ul><ul><ul><li>Select max(col_x), min(col_x) from table_z </li></ul></ul><ul><ul><li>Select count(distinct col_x) from table_z </li></ul></ul>
    13. 13. Extracting Data from Oracle <ul><li>Using Sybase Direct Connect for Oracle </li></ul><ul><li>Using SQLPlus for Data Extraction </li></ul>
    14. 14. Loading Directly from Oracle <ul><li>Sybase Enterprise Connect Product Family </li></ul><ul><ul><li>DirectConnect for Oracle </li></ul></ul><ul><ul><li>DirectConnect for Other Supported Databases </li></ul></ul>DirectConnect for Oracle Sybase IQ Oracle Database Non-Sybase Database Oracle Informix MS SQL Server DB2 UDB DB2 400 ODBC Mainframe Sources DirectConnect
    15. 15. Loading Sybase IQ From Oracle DirectConnect for Oracle Sybase IQ Oracle Database 1 2 3 4 Client Application <ul><li>Sequence Steps </li></ul><ul><li>A Client application connects to the Sybase IQ Server and issue the insert from location command </li></ul><ul><li>The IQ Server opens an Open Client connection to the DirectConnect for Oracle Access Service </li></ul><ul><li>The DirectConnect Access Service opens a native OCI connection to the Oracle Server </li></ul><ul><li>The IQ Server sends the Oracle SQL Select statement from the insert from location command to the DCO Access Service </li></ul>
    16. 16. Loading From Oracle – Continued Sequence Steps - continued 5. The DCO Access Service passes the SQL statement to the Oracle Server 6. The Oracle Server executes the SQL statement and returns the result set to the DCO Access Service 7. The DCO Access Service translates the Oracle result set into an Open Client result set and forwards it to the IQ Server 8. The IQ Server loads the result set DirectConnect for Oracle Sybase IQ Oracle Database 1 2 3 4 5 6 7 Client Application 8
    17. 17. Sybase IQ Syntax to Load from a Remote Table <ul><li>Connect to Sybase IQ and issue the following command </li></ul><ul><li>INSERT INTO OWNER.TABLE-NAME </li></ul><ul><li>(COLUMN LIST) </li></ul><ul><li>LOCATION '<DCO SERVER>. <DB NAME>' </li></ul><ul><li>{ ORACLE SQL STATEMENT }; </li></ul>The ORACLE SQL STATEMENT inside the curly brackets can contain any valid SQL in native ORACLE syntax which returns a relational result set. This SQL gets passed directly through to the ORACLE Server without modification.
    18. 18. DirectConnect – Load Performance <ul><li>Customer Testimonial </li></ul><ul><li>“We run batch processes at night that use the LOCATION clause to insert data from Oracle into Sybase IQ” </li></ul><ul><li>“We are generally inserting about 1 million rows into tables that contain 22 to 34 columns. The insert for each table takes about 5 minutes. This is on a 2 CPU server with 4 GB memory. We also load very similar data from a file using the IQ LOAD TABLE command which takes about 60 seconds” </li></ul><ul><li>“If the volume of data is low (a couple million rows or less), then using the LOCATION clause to get Oracle data into Sybase IQ is definitely the way to go because of the ease of use and convenience” </li></ul>Remember to consider total time = extract, transfer and load
    19. 19. Extracting Data from Oracle Using SQLPlus <ul><li>A SQL Plus script can be used to extract data from Oracle Database </li></ul><ul><ul><li>The results are written to a file or named pipe in delimited format </li></ul></ul><ul><ul><li>IQ can load from a file or named pipe which is feed by SQLPlus extract </li></ul></ul><ul><ul><li>Limited only by Oracle SQL Capabilities </li></ul></ul><ul><li>Gather Timings by running SQLPlus from local and remote hosts </li></ul><ul><li>Measure Extract Speed of local transfer vs remote transfer </li></ul><ul><li>Network overhead involved in the network transfer </li></ul><ul><li>Network transfer overhead = Remote transfer time - Local transfer time </li></ul>Sybase IQ Oracle Database SQLPlus File System SQLPlus File System
    20. 20. SQLPlus Extraction Script <ul><li>A SQL Plus script </li></ul><ul><li>--Set the line size to accommodate the longest possible line. </li></ul><ul><li>SET LINESIZE 136 </li></ul><ul><li>--Turn off all page headings, column headings, etc. </li></ul><ul><li>SET PAGESIZE 0 </li></ul><ul><li>SET FEEDBACK OFF </li></ul><ul><li>SET TERMOUT OFF </li></ul><ul><li>SPOOL /stage/emp_data.csv </li></ul><ul><li>SELECT TO_CHAR(employee_id) || ‘,’ </li></ul><ul><li>|| employee_name || ‘,’ </li></ul><ul><li>|| TO_CHAR(employee_hire_date,’YYYYMMDD’) || ‘,’ </li></ul><ul><li>|| TO_CHAR(employee_termination_date, ‘YYYYMMDD’) || ‘,’ </li></ul><ul><li>|| TO_CHAR(employee_billing_rate) || ‘,~’ </li></ul><ul><li>FROM employee; </li></ul><ul><li>SPOOL OFF </li></ul>
    21. 21. Load Processing Techniques <ul><li>Initial Load </li></ul><ul><li>Ongoing Load </li></ul><ul><li>Incremental Load Techniques </li></ul><ul><li>Converting SQL Loader Scripts </li></ul>
    22. 22. ETL Examples - Managing Data Loads <ul><li>Tables and Dimensions can be categorized as follows: </li></ul>* Dimensions as defined by Ralph Kimball Dimension Type   Change Processing History Tracking   Static None None   Slowly Growing New records added None Type 1* Slowly Changing, Update with Changes Content updated None Type 2* Slowly Changing, Versioning of Changes Content updated, new records added Full Type 3* Slowly Changing, Keep Current and Previous values Content updated Partial
    23. 23. Slowly Growing Dimensions or Tables <ul><li>CREATE TABLE CUST </li></ul><ul><li>( </li></ul><ul><li>CUST_ID INTEGER, </li></ul><ul><li>FIRST_NAME char(30), </li></ul><ul><li>LAST_NAME char(30), </li></ul><ul><li>ADDRESS char(60), </li></ul><ul><li>CITY char(30), </li></ul><ul><li>STATE char(2), </li></ul><ul><li>ZIP char(5) </li></ul><ul><li>); </li></ul>LOAD TABLE CUST ( CUST_ID '|', FIRST_NAME '|', LAST_NAME '|', ADDRESS '|', CITY '|', STATE '|', ZIP '|‘ ) from 'C:empechwaveust.txt' quotes off escapes off preview on row delimited by ' '; INSERT INTO CUST (CUST_ID, FIRST_NAME, LAST_NAME, ADDRESS, CITY, STATE, ZIP) LOCATION ‘SERVER.DBNAME’ {SELECT CUST_ID, FIRST_NAME, LAST_NAME, ADDRESS, CITY, STATE, ZIP FROM CUST}; Load Table or INSERT LOCATION Used interchangeably Dimension Type   Change Processing History Tracking   Static None None   Slowly Growing New records added None
    24. 24. Loading Type 1 Dimension Tables <ul><li>INSERT INTO STAGE_SUPPLIER ( S_NAME, S_ADDRESS, S_NATIONKEY, S_PHONE, S_ACCTBAL, S_COMMENT) LOCATION ‘SERVER.DBNAME’ </li></ul><ul><li>{ SELECT S_NAME, S_ADDRESS, S_NATIONKEY, S_PHONE, S_ACCTBAL, S_COMMENT FROM SUPPLIER}; </li></ul>UPDATE SUPPLIER SET S_ACCTBAL = STAGE_SUPPLIER.S_ACCTBAL FROM STAGE_SUPPLIER, SUPPLIER WHERE STAGE_SUPPLIER.S_NAME = SUPPLIER.S_NAME AND STAGE_SUPPLIER.S_ADDRESS = SUPPLIER.S_ADDRESS AND STAGE_SUPPLIER.S_PHONE = SUPPLIER.S_PHONE AND STAGE_SUPPLIER.S_ACCTBAL != SUPPLIER.S_ACCTBAL INSERT INTO SUPPLIER SELECT STAGE_SUPPLIER.* FROM STAGE_SUPPLIER WHERE STAGE_SUPPLIER.S_NAME NOT IN (SELECT SUPPLIER.S_NAME FROM SUPPLIER WHERE STAGE_SUPPLIER. S_NAME = SUPPLIER. S_NAME) Update Existing Insert New Load Staging Table Dimension Type   Change Processing History Tracking Type 1 Slowly Changing, Update with Changes Content updated None
    25. 25. Loading Type 2 Dimension Tables UPDATE SUPPLIER SET S_CURR_FLAG = ‘N’ FROM STAGE_SUPPLIER, SUPPLIER WHERE STAGE_SUPPLIER.S_NAME = SUPPLIER.S_NAME INSERT INTO SUPPLIER SELECT STAGE_SUPPLIER.*, ‘Y’ FROM STAGE_SUPPLIER Update Existing Insert Load Staging Table INSERT INTO STAGE_SUPPLIER (S_NAME, S_ADDRESS, S_NATIONKEY, S_PHONE, S_ACCTBAL, S_COMMENT) LOCATION ‘SERVER.DBNAME’ { SELECT S_NAME, S_ADDRESS, S_NATIONKEY, S_PHONE, S_ACCTBAL, S_COMMENT FROM SUPPLIER}; Dimension Type   Change Processing History Tracking Type 2 Slowly Changing, Versioning of Changes Content updated, new records added Full
    26. 26. Generating Sequence Numbers for Keys <ul><li>Adding Sequence Numbers / Generating Warehouse Keys </li></ul>CREATE TABLE STAGE_SUPPLIER ( S_NAME char(25), S_ADDRESS varchar(40), S_NATIONKEY integer, S_PHONE char(15), S_ACCTBAL double precision, S_COMMENT varchar(101) ); CREATE VARIABLE suppkey INTEGER; SELECT isnull(max(S_SUPPKEY),0) INTO suppkey FROM SUPPLIER; INSERT INTO SUPPLIER SELECT number(*)+suppkey, S_NAME, S_ADDRESS, S_NATIONKEY, S_PHONE, S_ACCTBAL, S_COMMENT FROM STAGE_SUPPLIER; FIND MAX KEY VALUE LOAD DATA number(*) Function: Sequential numbers 1 to N
    27. 27. Why Use Sybase IQ over Oracle <ul><li>ANSI SQL </li></ul><ul><li>Reduced Administration </li></ul><ul><li>Reduced Ongoing Tuning </li></ul><ul><li>Data Storage Simplified </li></ul><ul><li>Multi User Performance </li></ul><ul><li>Hardware Costs </li></ul><ul><li>IQ Indexing outperforms Oracle Indexing </li></ul>
    28. 28. Changing Tools from Oracle to Sybase IQ <ul><li>Query Tools </li></ul><ul><li>SQL </li></ul><ul><ul><li>Standard SQL 92 Entry Level Compliant </li></ul></ul><ul><li>ODBC Business Intelligence Tools </li></ul><ul><ul><li>Business Objects </li></ul></ul><ul><ul><li>Crystal Reports </li></ul></ul><ul><ul><li>Cognos </li></ul></ul><ul><ul><li>MicroStrategy </li></ul></ul><ul><ul><li>Brio </li></ul></ul><ul><ul><li>MS Access/Excel </li></ul></ul>

    ×