Benchmarking the SSIS connectors for Oracle, DB2 and SAP<br />Franck Sidi<br />Technology Architect SQL & BI<br />Microsof...
Franck Sidi<br />Microsoft – 2004 to Now<br />Microsoft France – SQL Server PreSales Technical Leader<br />Migrations Spec...
Session Objectives<br />Session Objective(s):  <br />Give overview of the connectivity capabilities with SSIS for Enterpri...
Data Flow Connectivity Options<br />Data Flow Task<br />OLEDBConnector<br />ADO.NETConnector<br />CustomConnector<br />OLE...
SSIS Connectivity for Enterprise Data<br />
DB2<br />
Microsoft OLEDB Provider for DB2<br />Interactive and scriptable Setup program<br />SNA Trace Utility and Trace Viewer<br ...
Extractingfrom DB2 / AS400<br />Objectives:<br /><ul><li>Extract a single Table with 80 columns and analyze the throughput...
Almost 3 Times faster than the current solution XXXX
70% faster than YYYYY
Parameters to check :
Defer Prepare to True
Rowset Cache Size
Work with small Buffers : 1 000 Rows</li></li></ul><li>Loadinginside DB2 / AS400<br />Objectives:<br /><ul><li>Load inside...
Use Microsoft OLEDB Driver for DB2 V3 (Pre-release)
V3 includes Bulk Load Capabilities to DB2 / AS400
Results:
1 000 Rows / Sec  with a single load (Normal Mode)to DB2 / AS400
12 000  rows / Secin singleload in Bulk Mode to DB2 / AS400
50 000 rows / Sec  in parallel load in Bulk Mode to DB2 / AS 400</li></li></ul><li>Transformation<br />Objectives:<br /><u...
Execute 4 Lookups transformation on DB2 to add lookup columns
Execute Global aggregation and generate a flat file with 7 773 702 rows</li></ul>Results:<br /><ul><li>Execute the global ...
Execute 4 lookups on DB2 / AS400 Tables
54 000 Rows / Sec Throughput
4 Times Faster than the current solution : XXXX</li></li></ul><li>SAP<br />
Extractfrom SAP<br />Microsoft Connectors SAP R/3 (MYSAP)  et SAP BW<br />Tables, Query, BAPI, RFC<br />Partner : Theobald...
Microsoft Connector<br />
TheobaldConnector<br /><ul><li>Real transaction-secure delta transfer of data
Support of all master and variable data including own generic extractors
No installation in SAP required
All business contents are available immediately even though no application-specific knowledge</li></li></ul><li>Oracle Con...
Where Attunity Connector for Oracle Fits<br />Wherever there is a need for extracting or loading large volumes of Oracle d...
Oracle Connector by Attunity Architecture<br />Oracle<br />SSIS<br />DB<br />OCI<br />Oracle <br />Client<br />
Oracle Connector by Attunity Supported platforms & versions<br />SQL Server 2008<br />9.2.0.4 and higher<br />Oracle<br />...
Requirements	<br />Oracle client software version 10.x or 11.x, on the computer with SSIS. Use 32-bit client on 32 bit pla...
Oracle Connector by Attunity High Performance Architecture<br />Oracle integration (using OCI):<br />Bulk extract source u...
Oracle connector test setup<br />A generous two-machine configuration<br />Simple SSIS tasks!<br />24 cores @ 2.4 GHz<br /...
Upcoming SlideShare
Loading in …5
×

User Group3009

1,650 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,650
On SlideShare
0
From Embeds
0
Number of Embeds
74
Actions
Shares
0
Downloads
19
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

User Group3009

  1. 1. Benchmarking the SSIS connectors for Oracle, DB2 and SAP<br />Franck Sidi<br />Technology Architect SQL & BI<br />Microsoft Israel<br />
  2. 2. Franck Sidi<br />Microsoft – 2004 to Now<br />Microsoft France – SQL Server PreSales Technical Leader<br />Migrations Specialist<br />Oracle, Sybase, DB2<br />SAP Integration, BI Architecture<br />ATG – 2000 to 2004<br />ATG France – PreSales Technical Manager<br />Keywords : Java, J2EE, Portal, eCommerce, Search, ECM, Mainframe Integration<br />Sybase – 1994 – 2000 <br />SYBASE INC USA 1999 – 2000 – Senior Developer in charge of PeoplesoftInc and Kenan (Telco Billing ISV)<br />Sybase France 1994 – 1999 – Tech Support Engineer & Presales Benchmark Specialist (Sybase, Sybase IQ)<br />1993 - Master Degree Database Systems (6 Years at University)<br />Learn on Teradata & Oracle<br />
  3. 3. Session Objectives<br />Session Objective(s): <br />Give overview of the connectivity capabilities with SSIS for Enterprise databases and solutions<br />Oracle, DB2 and SAP<br />Give some quick feedback about Performance capabilities with Oracle Drivers<br />Tips and Tricks<br />
  4. 4. Data Flow Connectivity Options<br />Data Flow Task<br />OLEDBConnector<br />ADO.NETConnector<br />CustomConnector<br />OLEDBProvider<br />NativeADO.NETAdapter<br />ADO.NETODBC<br />Data SourceSpecificInterface/Protocol<br />ODBC<br />Data Source Specific Interface/ Protocol<br />Data Storage<br />
  5. 5. SSIS Connectivity for Enterprise Data<br />
  6. 6. DB2<br />
  7. 7. Microsoft OLEDB Provider for DB2<br />Interactive and scriptable Setup program<br />SNA Trace Utility and Trace Viewer<br />TCP/IP network connection<br />Execution of dynamic SQL commands (DDL and DML), including CALL statement for stored procedures<br />Customized Data Link property dialog boxes for creating and modifying file-persisted OLE DB data link files.<br />Supports Integration Services, Replication, Analysis Services, Reporting Services, DQP<br />Packaging and licensing<br />Include with Microsoft SQL Server 2008 Enterprise and Developer editions<br />Feature Pack for SQL Server 2008<br />Microsoft download site<br />
  8. 8. Extractingfrom DB2 / AS400<br />Objectives:<br /><ul><li>Extract a single Table with 80 columns and analyze the throughput</li></ul>Results:<br /><ul><li>19 503 Rows / Sec extract from the Development Interface : BI Studio
  9. 9. Almost 3 Times faster than the current solution XXXX
  10. 10. 70% faster than YYYYY
  11. 11. Parameters to check :
  12. 12. Defer Prepare to True
  13. 13. Rowset Cache Size
  14. 14. Work with small Buffers : 1 000 Rows</li></li></ul><li>Loadinginside DB2 / AS400<br />Objectives:<br /><ul><li>Load inside DB2 / AS400
  15. 15. Use Microsoft OLEDB Driver for DB2 V3 (Pre-release)
  16. 16. V3 includes Bulk Load Capabilities to DB2 / AS400
  17. 17. Results:
  18. 18. 1 000 Rows / Sec with a single load (Normal Mode)to DB2 / AS400
  19. 19. 12 000 rows / Secin singleload in Bulk Mode to DB2 / AS400
  20. 20. 50 000 rows / Sec in parallel load in Bulk Mode to DB2 / AS 400</li></li></ul><li>Transformation<br />Objectives:<br /><ul><li>Extract 10 336 125 rows from DB2 / AS400 using a single query
  21. 21. Execute 4 Lookups transformation on DB2 to add lookup columns
  22. 22. Execute Global aggregation and generate a flat file with 7 773 702 rows</li></ul>Results:<br /><ul><li>Execute the global package in 2 Minutes and 24 sec
  23. 23. Execute 4 lookups on DB2 / AS400 Tables
  24. 24. 54 000 Rows / Sec Throughput
  25. 25. 4 Times Faster than the current solution : XXXX</li></li></ul><li>SAP<br />
  26. 26. Extractfrom SAP<br />Microsoft Connectors SAP R/3 (MYSAP) et SAP BW<br />Tables, Query, BAPI, RFC<br />Partner : Theobald <br />Xtract IS<br />Table (Transparent Tables SAP)<br />Query, BAPI, RFC<br />Delta-Q<br />Extractfrom ABAP Programs<br />
  27. 27. Microsoft Connector<br />
  28. 28. TheobaldConnector<br /><ul><li>Real transaction-secure delta transfer of data
  29. 29. Support of all master and variable data including own generic extractors
  30. 30. No installation in SAP required
  31. 31. All business contents are available immediately even though no application-specific knowledge</li></li></ul><li>Oracle Connectivity with Attunity Connectors<br />
  32. 32. Where Attunity Connector for Oracle Fits<br />Wherever there is a need for extracting or loading large volumes of Oracle data!<br />
  33. 33. Oracle Connector by Attunity Architecture<br />Oracle<br />SSIS<br />DB<br />OCI<br />Oracle <br />Client<br />
  34. 34. Oracle Connector by Attunity Supported platforms & versions<br />SQL Server 2008<br />9.2.0.4 and higher<br />Oracle<br />SSIS<br />DB<br />Oracle <br />Client<br />Any platform<br />Windows XP x86, x64<br />Windows Vista x86, x64<br />Windows Server 2003 x86, x64, IA64<br />Windows Server 2008 x86, x64, IA64<br />
  35. 35. Requirements <br />Oracle client software version 10.x or 11.x, on the computer with SSIS. Use 32-bit client on 32 bit platforms, 64-bit on 64-bit platform<br />If using a 64-bit platforms as a designer client (i.e.; using BIDS) both 64 and 32-bit clients should be installed.<br />SQL Server 2008 - Enterprise or Developer Edition<br />Windows 2003,2008<br />
  36. 36. Oracle Connector by Attunity High Performance Architecture<br />Oracle integration (using OCI):<br />Bulk extract source using OCI Array Binding<br />Bulk load destination using OCI Direct Path (fastest, constraints must turned off)<br />Incremental Bulk load using OCI Array Binding<br />Oracle Connection Manager<br />SSIS integration:<br />Direct integration into internal buffering APIs, cutting through .NET and other layers<br />Oracle<br />SSIS<br />DB<br />OCI<br />Oracle <br />Client <br />
  37. 37. Oracle connector test setup<br />A generous two-machine configuration<br />Simple SSIS tasks!<br />24 cores @ 2.4 GHz<br />32 GB RAM<br />HP disk array, partitioned as 17 logical drives<br />64-bit software<br />24 cores @ 2.4 GHz<br />32 GB RAM<br />HP disk array, partitioned as 17 logical drives<br />64-bit software<br />Attunity Oracle connectors<br />SSIS (SQL Server 2008)<br />Windows Server 2008<br />Oracle 11g<br />Windows Server 2008<br />Flat files<br />Oracle DB<br />1 GB Ethernet<br />
  38. 38. Oracle connector test setup<br />Oracle DDL<br />“Mixed data types”<br />create table LINEITEM (<br /> L_SHIPDATE DATE not null,<br /> L_ORDERKEY NUMBER(19,0) not null,<br /> L_DISCOUNT NUMBER(10,4) not null,<br /> L_EXTENDEDPRICE NUMBER(19,4) not null,<br /> L_SUPPKEY NUMBER(10,0) not null,<br /> L_QUANTITY NUMBER(5,0) not null,<br /> L_RETURNFLAG char(1) not null,<br /> L_PARTKEY NUMBER(10,0) not null,<br /> L_LINESTATUS char(1) not null,<br /> L_TAX NUMBER(10,4) not null,<br /> L_COMMITDATE DATE not null,<br /> L_RECEIPTDATE DATE not null,<br /> L_SHIPMODE varchar2(10) not null,<br /> L_LINENUMBER NUMBER(10,0) not null,<br /> L_SHIPINSTRUCT varchar2(25) not null,<br /> L_COMMENT varchar2(44) not null<br />)<br />“String data types”<br />CREATE TABLE LINEITEM (<br /> L_SHIPDATE varchar2(10), <br /> L_ORDERKEY varchar2(10), <br /> L_DISCOUNT varchar2(10), <br /> L_EXTENDEDPRICE varchar2(10), <br /> L_SUPPKEY varchar2(10), <br /> L_QUANTITY varchar2(10), <br /> L_RETURNFLAG varchar2(1), <br /> L_PARTKEY varchar2(10), <br /> L_LINESTATUS varchar2(1), <br /> L_TAX varchar2(10), <br /> L_COMMITDATE varchar2(10), <br /> L_RECEIPTDATE varchar2(10), <br /> L_SHIPMODE VARCHAR2(10), <br /> L_LINENUMBER varchar2(10), <br /> L_SHIPINSTRUCT VARCHAR2(25), <br /> L_COMMENT VARCHAR2(44)<br />)<br />
  39. 39. Test definitions<br />EmptyTable<br />SSIS reads from flat files, writes into Oracle<br />Table is initially empty<br />Attunity connector Fast Load calls Oracle DirectPath API<br />AddToTable<br />SSIS reads from flat files, writes into Oracle<br />Table already has data in it<br />Extract<br />SSIS reads from Oracle, writes flat files<br />Test measurements<br />Measured throughput (rows/sec)<br />Multiply by 133 bytes/row to get bytes/sec<br />
  40. 40. Mixed data types vs. string-only<br />Can we exploit this difference?<br />mixed<br />
  41. 41. CPU usage<br />Fast Load<br />Not Fast Load<br />SSIS Process Oracle Process<br />
  42. 42. Local vs. remote execution<br />Local Remote<br />mixed - local<br />mixed - remote<br />string - local<br />string - remote<br />Difference is generally not significant<br />
  43. 43. 32-bit vs. 64-bit performance<br />64-bit32-bit<br />Probably not enough to worry about<br />
  44. 44. Moving data from Oracle to SQL Server<br />Face it, we want to move data out of Oracle<br />String data extracts are faster than mixed types<br />Real-world data has various data types<br />How to extract as fast as possible?<br />Keep mixed types in Oracle<br />Extract strings by converting in query<br />Convert to SQL Server types in SSIS data flow<br />
  45. 45. Keep mixed types;Extract strings<br />Convert in SSISdata flow<br />select<br /> TO_CHAR(L_SHIPDATE),<br /> TO_CHAR(L_ORDERKEY),<br /> TO_CHAR(L_DISCOUNT),<br /> TO_CHAR(L_EXTENDEDPRICE),<br /> TO_CHAR(L_SUPPKEY),<br /> TO_CHAR(L_QUANTITY),<br /> L_RETURNFLAG,<br /> TO_CHAR(L_PARTKEY),<br /> L_LINESTATUS,<br /> TO_CHAR(L_TAX),<br /> TO_CHAR(L_COMMITDATE),<br /> TO_CHAR(L_RECEIPTDATE),<br /> L_SHIPMODE,<br /> TO_CHAR(L_LINENUMBER),<br /> L_SHIPINSTRUCT,<br /> L_COMMENT<br />from ATTUSER.LINEITEM<br />
  46. 46. Two conversions are better than one? Really? <br />Extract from Oracle with Attunity connector<br />Insert using SQL Server connector<br />
  47. 47. Tips and Tricks<br />
  48. 48. Parallel Mode<br />What does the SSIS Oracle Destination Parallel property mean in Fast Load mode?<br />The Parallel Load option is available in the Oracle Destination component when choosing the Fast Load mode.This setting tells Oracle not to lock the target table exclusively for this Fast Load session. This allows running additional fast load destination components to load the same target table in parallel.<br />Does this mean that the Oracle Destination component will load the target table in parallel sessions?<br />No, it means that the table will not be locked exclusively by the Oracle Destination therefore the user can design multiple data flow tasks that loads the same table.Here is an example of two tasks loading the same table: A Source component reads records 1-500 and the Fast Load Destination component loads them to the target Oracle table.<br />A Source component reads records 501-1000 and the Fast Load Destination component loads them to the target Oracle table in parallel.<br />Note that both Destination components should have the Parallel property checked. <br />
  49. 49. Support : http://www.attunity.com/forums/microsoft-ssis-connectors-attunity/<br />Error Message: Failed to load OCI DLLSUMMARYThe error indicates a failure when loading the Oracle oci.dll. The error could occur when having multiple Oracle homes or permission problems.SCENARIO 1The user is working on Microsoft Windows 64Bit operating system and have installed the following components: Oracle Client 32Bit (Required for Design) <br />Oracle Client 64Bit <br />Microsoft SSIS Oracle Connector 64Bit <br />This is the required list of components for Developing on 64Bit Windows machine. On some machines the following error was reported when running the Oracle component in 32Bit (Design time):<br />Error at Package [Connection manager &quot;Oracle Connector 1&quot;]: Failed to load OCI DLL.SOLUTIONFirst check that the environments are working and configured properly, and that SQLPlus is responding on both installations of the Oracle Client.On the machine which we encountered the error, it was noticed that the Oracle Home for the 64Bit Oracle Client was defined in the 32Bit registry portion (Wow6432Node), this was causing the problem of loading the wrong oci.dll.To workaround the problem, the user should define a dummy registry entry (Z_SSIS) as follows: 1. Open the regedit utility.2. Locate the following Key: HKEY_LOCAL_MACHINESOFTWAREWow6432NodeORACLE.ssis_oracle1.png3. Right-Click on ORACLE node and click on New -&gt; Key.4. Call the new key Z_SSIS (to make sure it&apos;s the last entry).5. Right-Click on the Z_SSIS node and click on New -&gt; String.6. Name the property ORACLE_HOME.7. Double-Click on the ORACLE_HOME and set it to the location of the Oracle 32Bit installation home directory.<br />
  50. 50. Attunity Oracle-CDC for SSIS<br />Complete solution for integrating changes made to Oracle tables using SSIS, efficiently and in real-time<br />Minimal impact on Oracle (log-based CDC, no install on Oracle)<br />Efficient and reduced use or resources (process only changes)<br />Fully integrated into SSIS (2005, 2008)<br />Synchronizes Bulk-Load and CDC<br />Monitoring & control console<br />Change Stream<br />Oracle<br />SQL Server<br />log<br />SSIS<br />34<br />
  51. 51. Additional Resources <br />Download Connectors<br />Oracle SSIS Connector <br />http://www.microsoft.com/downloads/details.aspx?FamilyID=d9cb21fe-32e9-4d34-a381-6f9231d84f1e&DisplayLang=en<br />Teradata Connector <br />http://www.microsoft.com/downloads/details.aspx?FamilyID=d9cb21fe-32e9-4d34-a381-6f9231d84f1e&DisplayLang=en<br />DB2 Connector <br />http://download.microsoft.com/download/0/E/6/0E67502A-22B4-4C47-92D3-0D223F117190/DB2OLEDB.exe<br />SAP BI<br />http://www.microsoft.com/DownLoads/details.aspx?familyid=B33D2C78-1059-4CE2-B80D-2343C099BCB4&displaylang=en<br />SAP Theobald<br />http://www.theobald-software.com/cms/en/haupt/evaluation.html<br />Others Links <br />http://www.codeplex.com/ssisctc<br />35<br />
  52. 52. http://www.codeplex.com/ssisctc<br />
  53. 53. Questions<br />

×