Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Oracle data capture c dc


Published on

  • Be the first to comment

  • Be the first to like this

Oracle data capture c dc

  1. 1. Document: Oracle Data Integrator 11g ( Working With Change Data CaptureDescription:(This document describes the basic concepts of the ODI CDC.Changed Data Capture (CDC) allows Oracle Data Integrator to trackchanges in source data caused by other applications.)History: Version Description Change Author Publish Date 0.1 Initial Draft Gourav Atalkar 7-Apr-2011 0.1 Review Amit Sharma 7-Apr-2011. Business Intelligence Solution Providers 1
  2. 2. Changed Data CapturePurpose of CDC is to enable applications to process changed data only. CDC enables ODI totrack changes in source data caused by other applications. When running integrationinterfaces, ODI can avoid processing unchanged data in the flow. Loads will process onlychanges since the last load. The volume of data to be processed is dramatically reduced.Reducing the source data flow to only changed data is useful in many contexts, such as datasynchronization and replication. It is essential when setting up an event-orientedarchitecture for integration. In such architecture, applications make changes in the data("Customer Deletion", "New Purchase Order") during a business process. These changes arecaptured by Oracle Data Integrator and transformed into events that are propagatedthroughout the information system.CDC Techniques Trigger based: ODI will create and maintain triggers to keep track of the changes. Logs based: ODI retrieves changes from the database logs (Oracle, AS/400). Time stamp based: Processes written with ODI can filter the data by comparing the time stamp value with the last load time (cannot process deletes) Sequence number: If the records are numbered in sequence, ODI can filter the data based on the last value loaded (cannot process updates and deletes).Changed Data Capture is performed by journalizing models. Journalizing a model consistsof setting up the infrastructure to capture the changes (inserts, updates and deletes) madeto the records of this models datastores.Oracle Data Integrator supports two journalizing modes: • Simple Journalizing tracks changes in individual datastores in a model. • Consistent Set Journalizing tracks changes to a group of the models datastores, taking into account the referential integrity between these datastores. The group of datastores journalized in this mode is called a Consistent Set.The Journalizing ComponentsThe journalizing components are: Business Intelligence Solution Providers 2
  3. 3. • Journals: Where changes are recorded. Journals only contain references to the changed records along with the type of changes (insert/update, delete). • Capture processes: Journalizing captures the changes in the source datastores either by creating triggers on the data tables, or by using database-specific programs to retrieve log data from data server log files. • Subscribers: CDC uses a publish/subscribe model. Subscribers are entities (applications, integration processes, etc) that use the changes tracked on a datastore or on a consistent set. They subscribe to a models CDC to have the changes tracked for them. Changes are captured only if there is at least one subscriber to the changes. When all subscribers have consumed the captured changes, these changes are discarded from the journals. • Journalizing views: Provide access to the changes and the changed data captured. They are used by the user to view the changes captured, and by integration processes to retrieve the changed data.These components are implemented in the journalizing infrastructureSetting up JournalizingThis is the basic process for setting up CDC on an Oracle Data Integrator data model. Eachof these steps is described in more detail below. 1. Set the CDC parameters 2. Add the datastores to the CDC 3. For consistent set journalizing, arrange the datastores in order 4. Add subscribers 5. Start the journalsJournalizing ToolsOracle Data Integrator provides a set of tools that can be used in journalizing to refreshinformation on the captured changes or trigger other processes: • SnpsWaitForData waits for a number of rows in a table or a set of tables. • SnpsWaitForLogData waits for a certain number of modifications to occur on a journalized table or a list of journalized tables. This tool calls SnpsRefreshJournalCount to perform the count of new changes captured. • SnpsWaitForTable waits for a table to be created and populated with a pre- determined number of rows. • SnpsRetrieveJournalData retrieves the journalized events for a given table list or CDC set for a specified journalizing subscriber. Calling this tool is required if using Database-Specific Processes to load journalizing tables. This tool needs to be used with specific knowledge modules. See the knowledge module description for more information. • SnpsRefreshJournalCount refreshes the number of rows to consume for a given table list or CDC set for a specified journalizing subscriber. Business Intelligence Solution Providers 3
  4. 4. Implementing Changed Data CaptureStep:1) Import the appropriate JKM in the project.Click the Projects tab. Expand the Procedure-Demo > Knowledge Modules node, right-clickJournalization (JKM), and select Import Knowledge Modules. Business Intelligence Solution Providers 4
  5. 5. Step:2) In the Models tab, create a new model named Oracle_relational_01. For Technology,enter: Oracle. Select the logical schema Sales_Order. Click the Reverse Engineer tab and setContext to development. Verify the setting, as shown in the following screen. Click theJournalizing tab.Step: 3) In the Knowledge Module menu, select JKM Oracle Simple. Procedure-Demo, asshown in the following screen. Click the Save to save your model and then close the tab. Business Intelligence Solution Providers 5
  6. 6. Step: 4) Reverse-engineer the model Oracle_Relational_01. Expand this model and verify itsstructure as follows.Step: 5) Set up the CDC Infrastructure. You will start the CDC on the EMPLOYEE table in theOracle_Relational_01 model.To add the table to CDC, expand the Oracle_Relational_01 model, right-click the EMPLOYEEtable, and select Change Data Capture > Add to CDC. Click Yes to confirm. Business Intelligence Solution Providers 6
  7. 7. Step: 6) Click the Refresh icon. The small yellow clock icon is added to the table.Step: 7) Right-click the EMPLOYEE table again and select Changed Data Capture > StartJournal. Business Intelligence Solution Providers 7
  8. 8. Step: 8) you use the default subscriber SUNOPSIS. For that reason, you do not have to addanother subscriber. Click OK to confirm that your subscriber is SUNOPSIS. In theInformation window, click OK again. Wait several seconds, then click Refresh and verifythat the tiny clock icon at the EMPLOYEE table is green now. This means that your journalhas started properlyStep: 9) Click the ODI Operator icon to open the Operator. Click Refresh. Select AllExecutions and verify that the EMPLOYEE session executed successfully.Step: 10) View the data and the changed data.In the Designer window, open the Models tab. Right-click the EMPLOYEE datastore andselect Data. Business Intelligence Solution Providers 8
  9. 9. Step: 11) Select the row with Employee_Key = 10. Change the value of the NAME2 columnto “Symond”. Similarly, select the row with Employee_Key = 15, and then change the valueto “jacob”. Save your changes and close the tab.Step: 12) Right-click the table again and select View Data. Scroll down, and verify that therows are modified. Close the tab. Business Intelligence Solution Providers 9
  10. 10. To verify that your changed data is captured, right-click EMPLOYEE, and select Change DataCapture > Journal Data. Find the captured changed records in the journal data. Close thetab. Business Intelligence Solution Providers 10