Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Integrating Oracle Data Integrator with Oracle GoldenGate 12c

2,624 views

Published on

This presentation will show you how to go through all the required steps and configurations to integrate ODI with OGG

Published in: Technology

Integrating Oracle Data Integrator with Oracle GoldenGate 12c

  1. 1. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! ODI and GoldenGate 12c: Beyond Data Integration Edelweiss Kammermann OOW 2014 September 28th 1!
  2. 2. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Edelweiss Kammermann •  Computer engineer, BI and BPM specialist with over 18 years of consulting and project management experience in Oracle technology. •  Co-founder and Vice President of Uruguayan Oracle User Group (UYOUG) •  Business Intelligence articles published on OTN site •  Frequent speaker at international conferences as Collaborate, OTN Tour LA, UKOUG Annual Conference and Rittman Mead BI Forum •  Oracle ACE •  Training Materials Specialist in Rittman Mead email: edel.kammermann@rittmanmead.com Twitter: @EdelweissK
  3. 3. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Uruguay
  4. 4. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Rittman Mead •  Oracle BI and DW Gold partner with offices in USA (Atlanta), Europa, Australia and India •  Winner of five UKOUG Partner of the Year awards 2013 including BI •  World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI •  100+ consultants worldwide included 1 Oracle ACE Director, 3 Oracle ACEs and 1 Oracle ACE Associate •  Skills in broad range of supporting Oracle tools: OBIEE,OBIA, ODI, Essbase, Oracle OLAP, GoldenGate, Exadata, Exalytics, Endeca and BigData •  Blog : http://www.rittmanmead.com/blog/ •  Twitter : @rittmanmead
  5. 5. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Agenda •  What is Oracle Data Integrator? •  What is Oracle GoldenGate? •  How the integration between ODI and OGG works? •  Benefits of this integration •  ODI and OGG integration steps •  Conclusions
  6. 6. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Oracle Data Integrator ODI is a complete and unified data integration solution for data transformation and integration processes. •  Provides reusability, flexibility and extensibility using Knowledge Modules (KM) •  Part of the Oracle Fusion Middleware 12c Stack •  Platform Independent •  Separates the logical from the physical implementation •  Based in an ELT architecture to leverage target platform capabilities: higher performance. •  Combines data movement, data synchronization, data transformation, data quality, data management and data services. •  Supports batch-based, real-time and service-based integration
  7. 7. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! ETL vs ELT
  8. 8. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! ETL vs ELT
  9. 9. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! ETL vs ELT • Reduce network transfer • Reduce hardware costs • Set-based operations • Leverage target platform capabilities • Referential Integrity Check
  10. 10. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! ODI Principal Components •  ODI Studio •  Agents - the runtime components •  Repositories •  Work Repository •  Master Repository •  Management Pack for Oracle Data Integrator •  ODI Console
  11. 11. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Master and Work Repositories •  ODI systems connect to a single Master Repository and one or more Work Repositories •  A Work Repository can be used for: •  Development •  Execution (Execution Repository) •  Development work repositories are full-function and can be developed in •  Execution work repositories can only import scenarios and execute them

  12. 12. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! ODI Studio •  An IDE (Integrated Development Environment) which allows Operators, Developers and Administrators users to access the repositories. •  Four navigators to perform different tasks. •  Designer •  Topology •  Operator •  Security
  13. 13. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! ODI Studio: Topology Navigator •  Defines the physical and logical architecture of the information systems •  Connections through to databases, OLAP servers, applications, files, etc •  Logical aliases for these connections •  Contexts (environments) •  Standalone and JEE agents •  Manage repositories. •  This information is stored in the Master Repository
  14. 14. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! ODI Studio: Designer Navigator •  Models •  Reverse-engineering metadata on data objects from source systems •  Projects •  Used for designing ODI data, transformation and mapping objects •  Define variables, sequences, functions, mappings and packages among other objects •  Import and use Knowledge Modules to check integrity, load and integrate information. •  Generate Scenarios
  15. 15. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! ODI Studio: Operator Navigator •  Monitor the execution of ODI objects -mappings, packages and scenarios- in the sessions •  Drill-into the actual code executed at each KM step •  Shows errors, warnings and execution metrics in the execution logs. •  Import and execute scenarios and load plans •  This information is stored in the Work Repository
  16. 16. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! What is Change Data Capture (CDC)? •  CDC is the process of identify, capture and delivery only the new and modified data from the source to the target tables •  Incremental Load •  Typical requirement of Data Warehouses •  More efficient: less data to move, less data to process •  Extract data in real time.
  17. 17. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! CDC process in ODI •  ODI performs CDC applying a JKM (Journalize Knowledge Module) to a model •  JKM creates the infrastructure to capture change data •  The Journalizing infrastructure is a set of tables, views and triggers to capture, store and purge the changed data •  ODI will create •  A table with the J$ prefix to load the PK of the changed data •  Views that join the J$ table with the original one to get all the columns of the changed data.
  18. 18. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Oracle Golden Gate •  Oracle GoldenGate 12c is a complete and high-performance application for real-time data integration and data replication with minimal overhead. •  OGG 12c provides solutions for: •  High availability •  Real-time data integration using log-based change data capture (CDC) and replication software platform. •  Disaster Recovery •  Data replication
  19. 19. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Oracle Golden Gate
  20. 20. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! How the integration between ODI and OGG works?
 ? •  OGG replicates the source data into staging area. •  OGG capture the changed data (CDC) from the source table, route it and deliver it to •  the replicated table in the staging area •  the J$/JV$ structures required by ODI JKMs •  ODI detects the changed records in its journalizing infrastructure and move the data into the final target table.
  21. 21. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! How the integration between ODI and OGG works?
 ? •  ODI 12c integrates with Oracle GoldenGate through the GoldenGate Jagent •  This allows ODI to communicate with OGG and also to remotely deploy configuration settings into GG installation. •  OGG JAgents are defined in ODI as Data Servers in the Topology Navigator
  22. 22. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Benefits of ODI – OGG integration •  Low impact on the source system •  OGG reads the changes directly from the database redo-logs •  OGG doesn’t require additional activity as triggers •  Great performance •  Integration of changed data in real-time •  Heterogeneous capabilities •  ODI and GoldenGate can work on heterogenous databases allowing more flexibility in the data integration infrastructure.
  23. 23. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! OGG and ODI Integration Steps Configurations in Oracle GoldenGate: 1.  Install OGG twice for the source and target schema. 2.  Database and Environment Variables Configurations 3.  Configure the JAgent for each GG installation 4.  Start both installations of OGG
  24. 24. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 1: Install OGG twice •  We need to install Golden Gate twice •  A source installation is required for capturing the data from the source schema •  A target installation is required for delivering the data to the target schema
  25. 25. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 2: Database configuration •  Oracle GoldenGate uses the redo logs to capture the data that needs to replicate, so they must be configured in the source database. •  Two commands need to be executed to enable Supplemental Logging •  ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; •  ALTER SYSTEM SWITCH LOGFILE; •  Oracle recommends to set also the Database to force logging: •  ALTER DATABASE FORCE LOGGING; •  If your Oracle Database version is 11.2.0.4 you also need to set explicitly a database initialization parameter to enable GoldenGate •  ENABLE_GOLDENGATE_REPLICATION=true
  26. 26. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 2: Environment variables configuration •  In Unix environments, the Library Path variable that contents the path of the OGG shared libraries must be set •  Example: export LD_LIBRARY_PATH=/ggcore_1:$LD_LIBRARY_PATH (*) •  The JAVA_HOME and PATH variables also required to be setup to point to the JDK folder and to the BIN folder of JDK respectively. •  Example: export JAVA_HOME=/u01/app/oracle/product/ogg1/jdk export PATH=$PATH:$JAVA_HOME/bin (*) The command to set variables depends on the type of Unix. Check the Installing and Configuring OGG document
  27. 27. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 3: JAgent Configuration for each GG installation •  To use the OGG JAgent enable monitoring is required. •  Create a parameter file without any extension in the GG home directory and name it GLOBALS. •  Add the line ENABLEMONITORING to enable monitoring in GG.
  28. 28. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 3: JAgent Configuration •  In order to secure the communication between ODI and OGG an Oracle Wallet must be created 1.  Set the JAVA_HOME environment variable to point to the JDK folder under the OGG home. 2.  Run the pw_agent_util utility which is under the OGG installation folder using the –jagentonly option
  29. 29. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 3: JAgent Configuration •  Modify the config.properties file 1.  Agent_type_enabled=OEM. To specify that we are not using the OGG Monitor (OGGMON) which is the default value 2.  As OEM is used for jagent type, the RMI port will be used. Default value 5559
  30. 30. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 4: Start GoldenGate •  Run the executable file ggsci which is under the OGG home folder 1.  Start Manager 2.  Start jagent
  31. 31. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! OGG and ODI Integration Steps Configurations in Oracle Data Integrator: 1.  Create both source and target OGG Data Servers in the Topology Navigator. 2.  Create OGG Physical Schemas from the Model 3.  Start Journal 4.  Develop a mapping for initial and incremental load of the final target table 5.  Automate the journalized process
  32. 32. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 1: Create OGG Data Servers in ODI •  ODI 12c communicates with OGG through the GoldenGate JAgents which are defined as Data Servers in the Topology navigator. •  Create a new Data Server under Oracle GoldenGate Technology. •  Enter values for: host, JMX Port (RMI port), JMI User (root by default), the Password created for the Oracle Wallet and the Installation Path. •  Test the connection.
  33. 33. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 2: Select the OGG JKM on the source Model •  We need to apply the journalizing process to the source model 1.  In the Journalizing tab of the source model, set the Journalizing Mode as Consistent Set •  This option takes into account referential integrity between datastores involved in the CDC process 2.  Select the JKM Oracle to Oracle Consistent (OGG online)
  34. 34. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 3: Create OGG Physical Schemas from the Model •  The Capture and Delivery process are the Physical Schemas in ODI •  They can be defined in the Topology Navigator or directly from the source Model. •  Press the Create button next to the Capture process. Select the source OGG Data Server as Jagent and enter a Name. That will create the Physical and Logical Schemas in ODI. •  Repeat those steps for the Delivery Process and select the target schema
  35. 35. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 3: Create OGG Physical Schemas from the Model •  Once we define the Capture and Delivery process: •  We need to set the ONLINE option as TRUE. •  In the Topology Navigator we have the Physical and Logical Schemas.
  36. 36. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 3: Create OGG Physical Schemas from the Model •  When we create the Capture process from the Model, the Trial File Path isn’t set with the path of the Delivery instance. •  The Remote Trial File Path needs to be modified in the Topology Navigator.
  37. 37. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 4: Start Journal 1.  Once we configured the model, we need to select which datastores (tables) we want to journalize adding the datastores to the CDC process 2.  Define the Subscribers that have to be notified of those changes 3.  Start the journal process. 1 3 2
  38. 38. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 5: Create a mapping to use the journalized data 1.  In the Designer Navigator create a New Mapping to load the data from the source datastore to the final target. 2.  In the Physical tab create a Deployment Specification for the initial load. Use an Insert or Control Append KM. 3.  Create another Deployment Specification for the incremental load. Select the source datastore and tick the Journalized Data Only option. Use an Incremental IKM.
  39. 39. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 6: Automation of the Journalized Process When we use Consistent Set as Journalizing mode there are some required tasks to perform before and after the consumption of the modified data. Previous Tasks: •  Extended Window •  Lock Subscribers Post Tasks: •  Unlock Subscribers •  Purge Journal
  40. 40. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Step 6: Automation of the Journalized Process We need to create a package in ODI with the following steps: •  ODIWaitForLogData Step: waits for changes on the Journalized data •  Journal Model Step: to Extend Window and Lock Subscriber •  Mapping Step: executes the mapping (need to select a Deployment Specification) •  Journal Model Step: to Unlock Subscriber and Purge Journal Data
  41. 41. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! DEMO Oracle Data Integrator and GoldenGate 12c 41!
  42. 42. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Conclusions •  OGG is the best option for CDC in ODI •  Less impact in the source database (OGG reads the redo logs and doesn’t require additional activity in the database as triggers) •  Great performance in delivering the modified data (real time integration). •  More flexibility in the data integration infrastructure (ODI and OGG support a wide range of databases)
  43. 43. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Questions?
  44. 44. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Additional Information •  ODI 12c Documentation: http://docs.oracle.com/middleware/1212/odi/odi-devandrun.htm •  ODI 12c OTN Forum : https://community.oracle.com/community/developer/english/ business_intelligence/system_management_and_integration/data_integrator •  A-Team Blog: http://www.ateam-oracle.com/ •  Data Integration Blog: https://blogs.oracle.com/dataintegration/ •  Rittman Mead Blog: www.rittmanmead.com/blog
  45. 45. info@rittmanmead.com www.rittmanmead.com @rittmanmead ! Thank you! Twitter: @EdelweissK Mail: edel.kammermann@rittmanmead.com

×