SQLServerDays2012_SSIS_CDC

1,770 views

Published on

Slidedeck of SQL Server Days 2012 session about SSIS and Change Data Capture. Created by Koen Verbeeck.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,770
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
88
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

SQLServerDays2012_SSIS_CDC

  1. 1. SQL SERVER DAYS 2012SSIS & CDC IN SQL SERVER 2012the perfect marriagekoen verbeeckBI consultant
  2. 2. WHO AM I?
  3. 3. OUTLINE• introduction to change data capture + demo• control flow tasks+ demo• data flow components+ demo• Oracle as a source• conclusion
  4. 4. INTRODUCTION• change data capture (CDC) • designed to capture insert/update/delete data o make it available in “easily consumed” relational format o pretty much DIY in TSQL • introduced in SQL Server 2008 o technically hasn’t changed in SQL Server 2012 • alternative for change tracking • typically used in data warehouse scenarios o speed up extraction by incremental loads• why? • more data everyday • less time to process it o full load vs. incremental load
  5. 5. INTRODUCTION• CDC versus change tracking CDC change tracking which changes? insert/update/delete insert/update/delete shows type of DML? yes yes methodology? asynchronous synchronous column changed? yes yes historical data? yes no editions? Enterprise only all
  6. 6. DEMO• showcase “2008” CDC
  7. 7. OUTLINE• introduction to change data capture + demo• control flow tasks+ demo• data flow components+ demo• Oracle as a source• conclusion
  8. 8. CONTROL FLOW TASKS• CDC control task • controls lifecycle of CDC packages o handles initial load synchronization o management of LSN ranges o error scenarios and recovery • stores state in SSIS variable and in database table • two operations o initial load o incremental load
  9. 9. DEMO• initial load package
  10. 10. CONTROL FLOW TASKS• initial load package• also possible from snapshot
  11. 11. OUTLINE• introduction to change data capture + demo• control flow tasks+ demo• data flow components+ demo• Oracle as a source• conclusion
  12. 12. DATA FLOW COMPONENTS• CDC source • reads a range of change data • all / net changes o and everything in between
  13. 13. DATA FLOW COMPONENTS• CDC splitter • basically a conditional split on steroids o … but without a GUI • splits flow of changed data into inserts, updates and deletes • do not combine with ALL CDC processing mode o order between operations is lost
  14. 14. DEMO• incremental load package
  15. 15. DATA FLOW COMPONENTS• incremental load package• remember to run it twice… • use the __$reprocessing column to identify rows in need of attention
  16. 16. OUTLINE• introduction to change data capture + demo• control flow tasks+ demo• data flow components+ demo• Oracle as a source• conclusion
  17. 17. ORACLE AS A SOURCE• Oracle CDC components by Attunity • not installed by default • can be found on installation media /Tools/AttunityCDCOracle CDC Designer CDC Service mirror cdc log miner figure modeled after slide of Matt Masson [16]
  18. 18. ORACLE AS A SOURCE• Oracle CDC Service • prepare SQL Server o creates MSXDBCDC database + objects • local system account o logon as a policy permission • CDC service master password o used to encrypt Oracle credentials • one service per Oracle DB
  19. 19. ORACLE AS A SOURCE• Oracle CDC Designer • create and manage CDC instances o multiple instances per service possible • contains o Oracle DB info o tables and columns being tracked o mirroring info
  20. 20. OUTLINE• introduction to change data capture + demo• control flow tasks+ demo• data flow components+ demo• Oracle as a source• conclusion
  21. 21. CONCLUSION• no new technical enhancements• enhancements are on manageability level • provided by Attunity• new SSIS tasks/components • CDC control task • CDC source • CDC splitter• two types of packages • initial load • incremental load • remember the __$reprocessing column• possible to use Oracle as CDC source • CDC service & designer
  22. 22. RESOURCES• CDC 2008 & Change Tracking • [1] MSDN - Change Data Capture http://msdn.microsoft.com/en-us/library/bb522489(v=sql.105).aspx • [2] MSDN - Comparing Change Data Capture and Change Tracking http://msdn.microsoft.com/en-us/library/cc280519(v=sql.105).aspx • [3] Kendra Little - Change Data Capture vs Change Tracking http://www.littlekendra.com/2010/06/23/cdcvsct/• CDC & SSIS 2012 • [4] MSDN - CDC Control Task http://msdn.microsoft.com/en-us/library/hh758674.aspx • [5] MSDN - CDC Source http://msdn.microsoft.com/en-us/library/hh758686.aspx • [6] MSDN - CDC Splitter http://msdn.microsoft.com/en-us/library/hh758656.aspx • [7] MSDN - CDC Flow Components http://msdn.microsoft.com/en-us/library/hh231087.aspx • [8] MSDN - Define a State Variable http://msdn.microsoft.com/en-us/library/hh758667 • [9] Allan Mitchell - SSIS CDC Processing Mode Not Updating Flag Attributes http://copperblueblog.com/2012/06/06/ssis-cdc-processing-mode-not-updating-flag-attributes/
  23. 23. RESOURCES• CDC & SSIS 2012 (continued) • [10] Allan Mitchell - CDC Package Needs Executing Twice http://copperblueblog.com/2012/06/06/ssis-change-data-capture-cdc-package-needs-executing-twice/ • [11] Allan Mitchell - Changes Made To Source During Full Load http://copperblueblog.com/2012/06/07/ssis-cdc-changes-made-to-source-during-full-load/ • [12] Allan Mitchell - Batch Sizes, Fast Load, Commit Size And The OLE DB Destination http://copperblueblog.com/2012/10/02/batch-sizes-fast-load-commit-size-and-the-ole-db-destination/ • [13] Matt Masson - CDC in SSIS for SQL Server 2012 http://www.mattmasson.com/index.php/2011/12/cdc-in-ssis-for-sql-server-2012-2/ • [14] Matt Masson - CDC Control Task Operations http://www.mattmasson.com/index.php/2012/02/cdc-control-task-operations/ • [15] Matt Masson - Processing Modes for the CDC Source http://www.mattmasson.com/index.php/2012/01/processing-modes-for-the-cdc-source/ • [16] Channel9 - TechEd North America (Matt Masson) - Incremental ETL Using CDC for SQL and Oracle with SQL Server Integration Services (SSIS) 2012 http://channel9.msdn.com/Events/TechEd/NorthAmerica/2012/DBI322
  24. 24. RESOURCES• CDC & Oracle • [17] TechNet Wiki - Installing Microsoft SQL Server 2012 Change Data Capture for Oracle by Attunity http://social.technet.microsoft.com/wiki/contents/articles/7647.installing-microsoft-sql-server- 2012-change-data-capture-for-oracle-by-attunity.aspx • [18] SSIS blog MSDN - CDC for Oracle in SQL Server 2012 http://blogs.msdn.com/b/mattm/archive/2012/03/26/cdc-for-oracle-in-sql-server-2012.aspx • [19] SSIS blog MSDN - Video about CDC for Oracle Databases in SSIS 2012 http://blogs.msdn.com/b/mattm/archive/2012/07/10/video-about-cdc-for-oracle-databases-in- ssis-2012.aspx
  25. 25. Q&A
  26. 26. THANK YOU koen.verbeeck@ordina.be @Ko_Ver http://www.linkedin.com/in/kverbeeckPartner Logo Here 26

×