GoldenGate Fundamentals Student Guide Version 10.4

9,043 views
8,934 views

Published on

GoldenGate Fundamentals Student Guide Version 10.4

Published in: Technology, Business
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
9,043
On SlideShare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
709
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

GoldenGate Fundamentals Student Guide Version 10.4

  1. 1. Oracle® GoldenGateFundamentalsStudent GuideVersion 10.4October 2009
  2. 2. Oracle GoldenGate Fundamentals Student Guide, version 10.4Copyright © 1995, 2009 Oracle and/or its affiliates. All rights reserved.This software and related documentation are provided under a license agreement containing restrictions on use anddisclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreementor allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute,exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, ordecompilation of this software, unless required by law for interoperability, is prohibited.The information contained herein is subject to change without notice and is not warranted to be error-free. If youfind any errors, please report them to us in writing.If this software or related documentation is delivered to the U.S. Government or anyone licensing it on behalf of theU.S. Government, the following notice is applicable:U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical datadelivered to U.S. Government customers are "commercial computer software" or "commercial technical data"pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such,the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license termsset forth in the applicable Government contract, and, to the extent applicable by the terms of the Governmentcontract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007).Oracle USA, Inc., 500 Oracle Parkway, Redwood City, CA 94065.This software is developed for general use in a variety of information management applications. It is not developedor intended for use in any inherently dangerous applications, including applications which may create a risk ofpersonal injury. If you use this software in dangerous applications, then you shall be responsible to take allappropriate fail-safe, backup, redundancy, and other measures to ensure the safe use of this software. OracleCorporation and its affiliates disclaim any liability for any damages caused by use of this software in dangerousapplications.Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of theirrespective owners.This software and documentation may provide access to or information on content, products, and services from thirdparties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of anykind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not beresponsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, orservices.
  3. 3. ContentsABOUT GOLDENGATE – COMPANY AND SOLUTIONS.................................................................. 5TECHNOLOGY OVERVIEW ................................................................................................................. 18ARCHITECTURE ..................................................................................................................................... 22CONFIGURING ORACLE GOLDENGATE ......................................................................................... 28 STEP 1. PREPARE THE ENVIRONMENT ....................................................................................................... 29 GOLDENGATE COMMAND INTERFACE...................................................................................................... 46 STEP 2. CHANGE CAPTURE ....................................................................................................................... 51 STEP 3. INITIAL LOAD ............................................................................................................................... 64 STEP 4. CHANGE DELIVERY ...................................................................................................................... 71EXTRACT TRAILS AND FILES............................................................................................................. 76 GOLDENGATE DATA FORMAT .................................................................................................................. 79 ALTERNATIVE FORMATS .......................................................................................................................... 85 VIEWING IN LOGDUMP ............................................................................................................................. 92 REVERSING THE TRAIL SEQUENCE ......................................................................................................... 104PARAMETERS ........................................................................................................................................ 107 GLOBALS PARAMETERS....................................................................................................................... 109 MANAGER PARAMETERS ........................................................................................................................ 110 EXTRACT PARAMETERS .......................................................................................................................... 113 REPLICAT PARAMETERS ......................................................................................................................... 122DATA MAPPING AND TRANSFORMATION ................................................................................... 131 DATA SELECTION AND FILTERING .......................................................................................................... 131 COLUMN MAPPING ................................................................................................................................. 138 FUNCTIONS ............................................................................................................................................. 141 SQLEXEC ............................................................................................................................................. 149 MACROS ................................................................................................................................................. 158 USER TOKENS......................................................................................................................................... 163 USER EXITS ............................................................................................................................................ 170 ORACLE SEQUENCES .............................................................................................................................. 177CONFIGURATION OPTIONS .............................................................................................................. 178 BATCHSQL .......................................................................................................................................... 179 COMPRESSION ........................................................................................................................................ 182 ENCRYPTION .......................................................................................................................................... 183 EVENT ACTIONS ..................................................................................................................................... 187 BIDIRECTIONAL CONSIDERATIONS ......................................................................................................... 192 ORACLE DDL REPLICATION ................................................................................................................... 202MANAGING ORACLE GOLDENGATE ............................................................................................. 211 COMMAND LEVEL SECURITY ................................................................................................................. 211 TRAIL MANAGEMENT ............................................................................................................................. 214
  4. 4. PROCESS STARTUP AND TCP/IP ERRORS ............................................................................................... 219REPORTING AND STATISTICS .................................................................................................................. 221MONITIORING ORACLE GOLDENGATE ................................................................................................... 229TROUBLESHOOTING ................................................................................................................................ 235TECHNICAL SUPPORT ............................................................................................................................. 244
  5. 5. Oracle GoldenGate Fundamentals Student GuideAbout GoldenGate – Company and Solutions Our Business We enable real-time, continuous movement of transactional data across Operational and Analytical business systems. Real-Time Access to Real-Time Information Real-Time Access Real-Time Information Mission-Critical Integration: the process of Availability: the degree to Systems combining data from different which information can be sources to provide a unified view. instantly accessed.Oracle GoldenGate provides solutions that enable your mission-critical systems to have continuous availability and access to real-time data.We offer a robust yet easy platform for moving real-time transactional data between operational and analytical systems to enable both:- High Availability solutions, and- Real Time Integration solutions • Real-Time Access -- meaning your critical data is accessible and available whenever you need it, 24x7. At the same time… • Real-Time Information -- meaning that the data available is as current as possible – not 24 hours old, not even 4 hours old.
  6. 6. Oracle GoldenGate Success Company Strength GoldenGate Software Acquired by Global sales and and Service established in 1995 Oracle in 2009 support Rapid Growth in Strategic Partners 500+ customers... 4000+ solutions implemented… in 35 countries Established, Loyal Customer BaseOur partnerships are rapidly increasing with major technology players, includingdatabase and IT infrastructure, packaged applications, business intelligence, andservice providers.And because our software platform supports a variety of solution use cases… Ourmore than 500 customers are using our technology for over 4000 solutions around theworld. What we typically find that once an initial solution is implemented and thebenefits achieved, our customers then find additional areas across the enterprisewhere we can further drive advantages for them.
  7. 7. Oracle GoldenGate Fundamentals Student Guide Transactional Data Management (TDM) Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of database transactions across heterogeneous environments in real time Key Capabilities: Additional Differentiators: Log-based capture moves Real Time thousands of transactions Performance per second with low Moves with sub-second latency impact Heterogeneous Meets variety of customer Extensibility & needs and data Moves changed data across Flexibility environments with open, different databases and platforms modular architecture Transactional Resilient against Reliability interruptions and failures Maintains transaction integrityOur focus is on transactional data management (TDM) – which means delivering aplatform in which that data can be best utilized in real-time enterprise wide.Oracle GoldenGate captures, routes, transforms, and delivers transactional data inreal time – and it works across heterogeneous environments with very low impact andpreserved transaction integrity.Our Key Capabilities in which we architect the product are: • We move data essentially in “real time” – with sub-second speed. • Works in heterogeneous environments – across different database and hardware types • Transactional – we are “transaction aware” and apply read-consistent changed data to maintain its referential integrity between source and target systems.We further Differentiate ourselves from other technologies with: - High performance with low impact – we can move large volumes of data very efficiently while maintaining very low lag times/latency. - Our flexibility – we meet a wide range of customer solution and integration needs, thanks to our open, modular architecture. - Our reliability – our architecture is extremely resilient against potential interruptions; no single point of failure or dependencies, and easy to recover. 7
  8. 8. TRANSACTIONAL DATA INTEGRATIONOracle GoldenGate provides the following data replication solutions:•High AvailabilityLive Standby for an immediate fail-over solution that can later re-synchronize withyour primary source.Active-Active solutions for continuous availability and transaction load distributionbetween two or more active systems.•Zero-Downtime Upgrades and MigrationsEliminate downtime for upgrades and migrations.•Live ReportingFeeding a reporting database so that you don’t burden your source productionsystems.•Operational Business Intelligence (BI)Real-time data feeds to operational data stores or data warehouses, directly or viaETL tools.•Transactional data integrationReal-time data feeds to messaging systems for business activity monitoring (BAM),business process monitoring (BPM) and complex event processing (CEP).Uses event-driven architecture (EDA) and service-oriented architecture (SOA).
  9. 9. Oracle GoldenGate Fundamentals Student Guide Oracle GoldenGate Solutions High Availability & Disaster Real-Time Data Integration Tolerance Live Standby Real-Time Data Warehousing Active-Active Live Reporting Zero-Downtime Operations for: Transactional Data Integration Upgrades Migrations MaintenanceOracle GoldenGate provides two primary solution areas:High Availability/Disaster Tolerance and Real-Time Data Integration.Within High Availability and Disaster Tolerance, we offer:• Active-Active solutions for continuous availability and transaction load distributionbetween two or more active systems•Zero-Downtime Operations that eliminates downtime for planned outages involvingupgrades, migrations, and ongoing maintenance•Live Standby for an immediate fail-over solution that can later re-synchronize withyour primary sourceWithin Real-Time Data Integration, we offer:• Real-Time Data Warehousing which gives you real-time data feeds to datawarehouses or operational data stores•Transactional Data Integration for distributing data in real-time betweentransaction processing systems•Live Reporting is for feeding a reporting database so that you don’t burden yoursource production systems 9
  10. 10. Oracle GoldenGate for High Availability & Disaster Tolerance High Availability & Disaster Tolerance Live Standby Real-Time Access Active-Active Improved Uptime Higher Performance Zero-Downtime Operations for: Faster Recovery Upgrades Minimized Data Loss Migrations Maintenance Lower TCOFor High Availability and Disaster Tolerance solutions, it’s about “real-time” orCONTINUOUS access to your data via your critical applications.The benefits that Oracle GoldenGate drives here include:-Improved uptime and availability (helping you reach aggressive service levelagreements/SLAs)-Higher Performance for your production systems – help to eliminate scalability orresponse time delays that can give users the impression of an availability or accessissue-Faster Recovery and Minimized Data Loss – so you can achieve higher RecoveryTime Objectives (RTOs) and Recovery Point Objectives (RPOs)- and an overall lower Total Cost of Ownership by putting your standby systems towork for other solutions!
  11. 11. Oracle GoldenGate Fundamentals Student Guide High Availability: Live Standby Benefits: Eliminate unplanned downtime Reduce data loss and isolate corrupt data Re-synchronize backup and primary systems Remove distance constraints Automate switchovers Improve ROI with active standby available for reportingLive Standby: helps eliminate unplanned outages to enable continuous availability,with no geographic distance constraints.Oracle GoldenGate moves changed data from primary database to a standby in sub-seconds so that end users have a reliable failover system with up to date data thatthey can immediately switchover.There is no database recovery process required because changed data is queuedoutside of the database in persisted Trail files, and data loss risk is minimized. OracleGoldenGate also isolates corrupt data during movement to make sure the secondarysystem is reliable when it is needed.The customer’s Return on Investment can be further increased by using the livestandby system for reporting or testing – Oracle GoldenGate allows the standbydatabase to be open, so it does not have to sit idle and can be put to work! 11
  12. 12. High Availability: Active-Active Benefits: Achieve continuous availability Enable transaction load distribution (with built-in conflict resolution) Improve performance Lower TCOActive-Active: Oracle GoldenGate enables bidirectional data movement between twoor more databases that actively support an application, with no geographic distanceconstraints.The active-active solution allows data updates and changes (“write” activity) to occuron two or more active databases supporting live applications. Oracle GoldenGatesynchronizes the two active databases by replicating the data between each at alogical level, and allows load distribution to improve system performance. In the caseof an outage of one system, there is no downtime for the end user because the otheractive system continues with operations.Because Oracle GoldenGate is an asynchronous solution, conflict management isrequired to ensure data accuracy in the event that the same row is changed in two ormore databases at (or about) the same time. Oracle GoldenGate provides capabilitiesto detect and resolve conflicts as well.A variety of active-active scenarios can be supported – depending on the desiredimplementations. We have strong experience in active-active solutions for both HighAvailability as well as Zero-Downtime upgrades and migration projects.
  13. 13. Oracle GoldenGate Fundamentals Student Guide Zero-Downtime Upgrades and Migrations Benefits: Eliminate “planned downtime” during hardware, database, OS and/or application upgrades and migrations Minimize risk with fail-back contingency Improve success with phased user migrationsZero Downtime Operations: is for eliminating planned outages during database,application or server upgrades, migrations, and/or maintenance.Oracle GoldenGate captures all the changed data in the primary system while the newsystem is initiated and prepared. Once the second or the new system is upgraded ormigrated Oracle GoldenGate applies all the changed data to the new system. OracleGoldenGate then keeps the two environments synched with our real-time replication.Often with such projects, there is always concern about what will happen once youswitchover to the new environment. Oracle GoldenGate alleviates many of those riskswith our fail-back capabilities -- after switchover Oracle GoldenGate captures thechanges that happen in the new system so that the old system is kept up to date incase there is a need for fail back to the old environment.And currently in Oracle or HP Nonstop environments, our Oracle GoldenGateVeridata product verifies that the data is consistent in both systems before and evenafter switchover. 13
  14. 14. Oracle GoldenGate for Real-Time Data Integration Real-Time Data Integration Real-Time Information Live Reporting Fresher Data Operational Business Intelligence Minimal Overhead No Batch Windows Transactional Data Integration Data Integrity Ease of IntegrationFor our Real-Time Data Integration solutions, it’s about “real-time information” oraccess to CURRENT operational data.The benefits that Oracle GoldenGate drives here include:• Fresher, real-time data available for use and decision-making – remove latency as atechnical constraint.• Minimal overhead and impact on your source systems and overall architecture tocapture and move real-time data• No requirement for batch windows• Transactional data integrity helps improve overall data quality• Ease of integration – Oracle GoldenGate easily fits into existing and desirearchitecture, and is overall easy to maintain over long term.
  15. 15. Oracle GoldenGate Fundamentals Student Guide Data Integration: Live Reporting Benefits: Use real-time data for better, faster decision making Remove reporting overhead on source system Reduce cost-to-scale as user demands and data volumes grow Leverage cost-effective systems for reporting needsOracle GoldenGates Live Reporting enables both real-time reporting capabilitieswhile improving the performance of operational source systems.Oracle GoldenGate feeds real-time data from the source to a secondary reporting-onlydatabase such as an operational data store (ODS). This allows reporting activity to beoff-loaded from the production database. This secondary database can be a differentdatabase and/or platform from the production database, to lower the total cost ofownership and allow organizations to leverage emerging open source technologies.The solution also helps increase scalability as user demands and data volumes grow. 15
  16. 16. Operational Business Intelligence Benefits: Use real-time data for better, faster decision making Eliminate batch window dependency Reduce overhead on source system Maintain referential integrity for data quality Leverage its flexibility for transformations and integration with ETL 18For Real-Time Data Warehousing -- The Oracle GoldenGate Real-Time DataWarehousing solution enables continuous, real-time data feeds for data warehouses oroperational data stores, to improve business intelligence. Our log-based changed datacapture has very minimal impact on the source, no batch windows and moves the datain sub-seconds. Each transaction’s commit boundaries are maintained for dataintegrity.Oracle GoldenGate’s architecture also improves data recoverability in case there is anoutage during the data movement. This is an important requirement as data latencydecreases in feeding the analytical environment. Oracle GoldenGate’s trail files thatstore the changed data are persisted, so if needed they can be reapplied to the targetand also source system without having to capture the data again.Transformations or co-existing with ETL:Oracle GoldenGate out-of-the box can support a number of common datatransformations often required for data integration. However, where complextransformations are needed Oracle GoldenGate can be used to augment an existingETL solution in several ways:1) First, Oracle GoldenGate can deliver transactional data to staging tables in realtime, which then would be used by the ETL to extract from and performtransformations and then load user tables. This method works best when the ETLproduct is optimized to perform the transformations within the target database. Thisis an “ELT” model.2) Second method: Oracle GoldenGate provides the data to the ETL engine as flat filesand in micro-batches. The latency depends on the ETL product and businessrequirements but we typically deliver every few minutes to an hour.3) Third method: Oracle GoldenGate publishes changed data to a messaging systemand the ETL solution (that can subscribes to the queue or topic) receives it in real-time.In each of these architectures combining real-time change data capture with ETLdecreases data latency to real time or near real-time and eliminates the batch windowdependency.
  17. 17. Oracle GoldenGate Fundamentals Student Guide Transactional Data Integration Benefits: Easily integrate large volumes of real-time data between transaction processing systems Reduce overhead; Eliminate batch windows Improve scalability Enhance SOA and EDA environments (delivery to JMS-based messaging systems) 19Oracle GoldenGate provides real-time data integration between OLTP systems non-intrusively and with minimal impact. Distributed databases and the applications theysupport can continuously access, utilize, and act on the most current operational dataavailable. The solution can also integrate with JMS-based messaging systems toenable event driven architecture (EDA) and to support service oriented architecture(SOA). 17
  18. 18. Technology Overview How Oracle GoldenGate Works: Modular “Building Blocks” Capture: Committed changes are captured (and can be filtered) as they occur by reading the transaction logs. Trail files: Stages and queues data for routing. Route: Data is compressed, encrypted for routing to targets. Delivery: Applies data with transaction integrity, transforming the data as required. Source Network Target Capture (TCP/IP) Delivery Trail Trail Source Target Database(s) Target Source Database(s) Delivery Capture Trail Trail Bi-directionalOracle GoldenGate consists of decoupled modules that are combined to create the bestpossible solution for your business requirements. On the source system(s):•Oracle GoldenGate’s Capture (Extract) process reads data transactions as they occur,by reading the native transaction log, typically the redo log. Oracle GoldenGate onlymoves changed, committed transactional data, which is only a % of all transactions –therefore operating with extremely high performance and very low impact on the datainfrastructure.• Filtering can be performed at the source or target - at table, column and/or row level.• Transformations can be applied at the capture or delivery stages.• Advanced queuing (trail files): To move transactional data efficiently and accurately across systems, Oracle GoldenGate converts the captured data into a Oracle GoldenGate data format in “trail” files. With both source and target trail files, Oracle GoldenGate’s unique architecture eliminates any single point of failure and ensures data integrity is maintained – even in the event of a system error or outage.Routing:•Data is sent via TCP/IP to the target systems. Data compression and encryption aresupported. Thousands of transactions can be moved per second, without distancelimitations.On the target system(s):•A Server Collector process (not shown) reassembles the transactional data into atarget trail.
  19. 19. Oracle GoldenGate Fundamentals Student Guide•The Delivery (Replicat) process applies transactional data to the designated targetsystems using native SQL calls.Bi-directional:•In bi-directional configurations/solutions, this process runs the same in reverse, toconcurrently synchronize data between the source and target systems.Manager processes (not shown) perform administrative functions at each node. Oracle GoldenGate Supports Applications Running On… Databases O/S and Platforms Capture: Oracle Windows 2000, 2003, XP DB2 Linux Microsoft SQL Server Sun Solaris Sybase ASE HP NonStop Ingres HP-UX Teradata HP TRU64 Enscribe HP OpenVMS SQL/MP IBM AIX SQL/MX IBM z/OS Delivery: All listed above MySQL, HP Neoview, Netezza, and any ODBC compatible databases ETL products JMS message queuesOracle GoldenGate is ideal for heterogeneous environments – not just supportingdifferent versions of the same database or operation system/hardware, but replicatingand integrating data across vendor systems.We support log-based Capture of changed data from nearly all major databasevendors.We can Deliver that data to an even wider range of targets – including open sourcedatabases, several data warehouse appliances, ETL servers, and JMS message queuesto support Service Oriented Architectures (SOA) and Event-Driven Architectures(EDA). 19
  20. 20. Oracle GoldenGate Advantages Movement Management Integration Speed Transaction Integrity Heterogeneous Data – Subsecond Latency Transparent Capture Sources Volume Guaranteed Delivery Mapping – Thousands of TPS Conflict Detection, Transformation Log-based Capture Resolution Enrichment Native, Local Apply Dynamic Rollback Decoupled Architecture Efficient IO and Incremental TDM Table, Row, Column Bandwidth Usage Initial Data Load Filtering Bidirectional GUI-based Monitoring and XML, ASCII, SQL Formats Group Transactions Configuration Queue Interface Bulk Operations Proactive Alerts Stored Procedures Compression Encryption User Exits One-to-Many, Real-Time Deferred or ETL Integration Many-to-One Batch Java/JMS Integration Cascade Event Markers Oracle GoldenGate Director Manages, defines, configures, and reports on Oracle GoldenGate components Key features: Centralized management of Oracle GoldenGate modules Rich-client and Web-based interfaces Alert notifications and integration with 3rd-party monitoring products Real-time feedback Zero-impact implementationOracle GoldenGate Director is a centralized server-based graphical enterpriseapplication that offers an intuitive way to define, configure, manage, and report onOracle GoldenGate processes.Oracle GoldenGate Director is a value added module to centralize management andimprove productivity.Oracle GoldenGate Director supports all platforms and databases supported by OracleGoldenGate.
  21. 21. Oracle GoldenGate Fundamentals Student Guide Oracle GoldenGate Veridata A high-speed, low impact data comparison solution Identifies and reports data discrepancies between two databases without interrupting those systems or the business processes they support Supports Oracle, Teradata, SQL Server, NonStop SQL/MP and Enscribe Supports homogeneous and heterogeneous compares Benefits: Reduce financial/legal risk exposure Speed and simplify IT work in comparing data sources No disruption to business systems Improved failover to backup systems Confident decision-making and reportingOracle GoldenGate Veridata is a high-speed data comparison solution that identifiesand reports data discrepancies between databases without interrupting ongoingbusiness processes. Using Oracle GoldenGate Veridata, companies can audit andverify large volumes of data across a variety of business applications with certainty,and maintain reliable data synchronization. Oracle GoldenGate Veridata reduces theamount of time and resources required to compare data, it minimizes the impact ofhuman errors, and it ensures that potential problems can be instantly identified andaddressed.Key Veridata Features:•Compares large data volumes with high speed and efficiency•Allows both data sources to be online by handling in-flight transactions•Performs selective, parallel comparison•Offers intuitive Web interface and personalized views•Enables the comparison of databases that are different database versions or ondifferent operating systems•(HP Nonstop only) Supports the comparison of only the data changed since theinitial comparison (delta processing)Why would you need Veridata?Data discrepancies arise even without malicious intent -- due to infrastructureproblems, application errors, operator mistakes, configuration errors, or unexpecteduser behavior. With vigilant verification procedures using Oracle GoldenGateVeridata, companies can eliminate data inconsistencies across different businessapplications and avoid potential operational, financial, or regulatory risks. 21
  22. 22. Architecture Oracle GoldenGate Data Capture and Delivery Oracle GoldenGate Transactional Data Management: Primarily used for change data capture and delivery from database transaction logs Can optionally be used for initial load directly from database tables Especially useful for synchronizing heterogeneous databases Database-specific methods may be preferable for homogeneous configurations Change Data Capture & Delivery Source Database Network (TCP/IP) Target Database Extract Server Replicat Collector Trail Transaction Log Manager ManagerOn the source system: An Extract process captures transactional changes from transaction logs The Extract process sends data across a TCP/IP network to the target system.On the target system: A Server Collector process reassembles and writes the data to a GoldenGate trail. A Replicat process reads the trail and applies it to the target database. This can beconcurrent with the data capture or performed later.
  23. 23. Oracle GoldenGate Fundamentals Student GuideManager processes on both systems control activities such as starting, monitoring andrestarting processes; allocating data storage; and reporting errors and events. Change Data Capture & Delivery using a Data Pump Source Database Network (TCP/IP) Target Database Extract Server Replicat Collector Remote Trail Transaction Log Local Data Trail Pump Manager ManagerOn the source system: An Extract process captures transactional changes from the database transaction log The Extract process writes the data to a local GoldenGate trail. This preserves thecaptured data if the network or target trail fails. A second Extract process (called a Data Pump) sends the data across the network tothe target system.On the target system: A Server Collector process reassembles and writes the data to a GoldenGate trail. A Replicat process reads the trail and applies it to the target database. This can beconcurrent with the data capture or performed later.Manager processes on both systems control activities such as starting, monitoring andrestarting processes; allocating data storage; and reporting errors and events. 23
  24. 24. Initial Load Source Network Database (TCP/IP) Target Tables Database Extract Replicat Or DB Bulk Server Load Utility Collector Files Manager Manager GoldenGate initial load methods: Direct Load (Extract sends data directly to Replicat to apply using SQL) Direct Bulk Load (Replicat uses Oracle SQL*Loader API) File to Replicat (Extract writes to a file that Replicat applies using SQL) File to database utility (Extract writes to a file formatted for a DB bulk load utility)On the source system: An Extract process captures source data directly from tables The Extract process sends data in large blocks across a TCP/IP network to thetarget system.On the target system, one of the following scenarios:1. Direct Load. Replicat reads the data stream and concurrently applies the data to the target database using SQL.2. Direct Bulk Load (Oracle). Replicat can apply the data using the Oracle SQL*Loader API to improve performance.3. File to Replicat. Server Collector reassembles and writes the data to Extract files. Replicat applies the data to the target database using SQL.4. File to database utility. Server Collector reassembles and writes the data to files formatted for a bulk loader, which applies the data to the target database.Manager processes on both systems control activities such as starting, monitoring andrestarting processes; allocating data storage; and reporting errors and events.
  25. 25. Oracle GoldenGate Fundamentals Student Guide Online versus Batch Change data capture & delivery can be run either continuously (online) or as a special run (batch run) to capture changes for a specific period of time. Initial load is always a special run (batch run). Checkpointing - Extract For change data capture, Extract and Replicat save checkpoints to a checkpoint file so they can recover in case of failure Extract maintains: 2 input checkpoints 1 output checkpoint for each trail it writes to Start of oldest uncommitted Last record transaction in log read from log Input: Transaction Log Checkpoints Output: One or more GoldenGate Trails End of last committed transaction written to trailCheckpoints are used during online change synchronization to store the current readand write position of a process. Checkpoints ensure that data changes marked forsynchronization are extracted, and they prevent redundant extractions. They providefault tolerance by preventing the loss of data should the system, the network, or aGoldenGate process need to be restarted. 25
  26. 26. Checkpointing - Replicat Best practice is to create a checkpoint table in the target database Checkpoints are maintained in both the checkpoint table (if it exists) and a checkpoint file Replicat maintains 2 input checkpoints: Start of current Last record uncommitted transaction read from trail Input: GoldenGate Trail CheckpointsParameters, Process Groups and Commands GoldenGate processes are configured by ASCII parameter files. A process group consists of: An Extract or Replicat process Associated parameter file Associated checkpoint file Any other files associated with that process Each process group on a system must have a unique group name. Processes are added and started using the GoldenGate Software Command Interface (GGSCI) with the group name. GGSCI commands also add trails, check process status, etc.
  27. 27. Oracle GoldenGate Fundamentals Student Guide Solutions and Architecture – Discussion Points 1. How is Oracle GoldenGate different from simply replicating database operations? 2. What are some use cases for Oracle GoldenGate software? 3. What is the purpose of checkpointing?1. Log-based change data capture, decoupled from database architecture. Real-time, heterogeneous and transactional.2. (a) High availability – live standby, active-active, zero down-time upgrades and migrations. (b) Real-time data integration – real-time data warehousing (operational business intelligence), live reporting, transactional data integration.3. For recovery if a GoldenGate process, network or system goes down. 27
  28. 28. Configuring Oracle GoldenGate Configuring Oracle GoldenGate Oracle GoldenGate can be deployed quickly and easily in four steps: 1. Prepare the Environment 2. Change Capture 3. Initial Load 4. Change DeliveryNote: You can run log-based change capture after the initial data load if you set the extract begin time to the start of the longest running transaction committed during the initial data load. Configuring Oracle GoldenGate 1. Prepare the Environment Source Database 3. Initial Load (various methods) Target Database Transaction Log Local Data Extract Pump Remote Replicat Trail Trail 4. Change Delivery 2. Change CaptureOracle GoldenGate can be deployed quickly and easily in four steps:• Prepare the environment, e.g. • Install Oracle GoldenGate software on source and target • Enable transaction logging
  29. 29. Oracle GoldenGate Fundamentals Student Guide • (Heterogeneous source/target) Generate source definitions so Replicat can process trail data• Configure and start change capture to GoldenGate trail files (Extract processes – primary and data pump)• Perform initial load to synchronize databases by database-specific or GoldenGate methods• Configure and start change delivery (Replicat process)Step 1. Prepare the Environment Step 1. Prepare the Environment 1. Prepare the Environment Source Database 3. Initial Load (various methods) Target Database Transaction Log Data Extract Local Pump Remote Replicat Trail Trail 4. Change Delivery 2. Change Capture Step 1. Prepare the Environment Set up each system: Install Oracle GoldenGate software on source and target Configure and start GoldenGate Manager on source and target If heterogeneous source/target, generate source definitions and copy to target Prepare the database. For example: Ensure database access by GoldenGate Enable transaction logging 29
  30. 30. Installing Oracle GoldenGate installs all of the components required to run andmanage GoldenGate processing, and it installs the GoldenGate utilities.Manager must be running on each system before Extract or Replicat can be started,and must remain running while those processes are running so that resourcemanagement functions are performed.The source definitions file contains the definitions of the source tables and is requiredon the target system in hetereogeneous configurations. Replicat refers to the file towhen transforming data from the source to the target.To reconstruct an update operation, GoldenGate needs more information than Oracleand SQL Server transaction logs provide by default. Adding supplemental log dataforces the logging of the full before and after image for updates.Install Oracle GoldenGate Software Prepare Environment: Installation – Access the Media Pack Access the product media pack (software and documentation) at edelivery.oracle.com Identify the proper release of GoldenGate for your source and target environments Database and version Operating system and versionA GoldenGate instance is a single installation of GoldenGate.
  31. 31. Oracle GoldenGate Fundamentals Student Guide Prepare Environment: Installation - Windows Download .zip file to C:GGS Unzip .zip file into C:GGS folder Configure a Windows Service Name for Manager process in a GLOBALS parameter file (required only if multiple Managers on the server) C:GGS> INSTALL ADDSERVICE ADDEVENTS GGSCI> CREATE SUBDIRSFor Windows: Do not install Oracle GoldenGate into a folder that contains spaces inits name, for example “GoldenGate Software.” The application references pathnames, and the operating system does not support path names that contain spaces,whether or not they are within quotes. Prepare Environment: Installation – Windows INSTALL Program On Windows, an INSTALL program performs the following functions: Installs GoldenGate event messages into the system registry Installs the Manager as a Windows service Syntax: INSTALL <item> [<item> …] Example: C:GGS> INSTALL ADDEVENTS ADDSERVICE Note: The uninstall command is: INSTALL DELETESERVICE DELETEEVENTSItems (all optional)ADDEVENTS Adds the GoldenGate events to the registry so that event messagesappear in the Windows Event Log.DELETEEVENTS Deletes GoldenGate events from the registry.ADDSERVICE Defines the GoldenGate Manager process as a Windows service(Recommended) Manager can run by a local or domain account. However, when run 31
  32. 32. this way, Manager will stop when the user logs out. By using install, you can installManager as a Windows service so that it can be operated independently of userconnections and can be configured to start either manually or when the system starts.You can configure the Manager service to run as the Local System account or as aspecific named account. The configuration of a service can be changed by using theServices applet of the Windows Control Panel and changing the service Properties.DELETESERVICE Removes the GoldenGate Manager service.AUTOSTART Specifies that the service be started at system boot time (the default).MANUALSTART Specifies that the service be started only at user request (with GGSCIor the Control Panel Services applet).USER Specifies a user name to logon as when executing Manager. If specified, username should include the domain name, a backward slash, and the user name.PASSWORD Specifies the user’s password for logon purposes. This can be changedusing the Control Panel Services applet. Prepare Environment: Installation - Multiple Manager Services GoldenGate supports running multiple Manager services on Windows For two or more GoldenGate instances, or GoldenGate with a Veridata C Agent (which uses a Manager) Each Manager service must be assigned a unique name Before installing the service, you can specify the name Create a GLOBALS parameter file for each Manager Specify the one-word name of the service using the MGRSERVNAME <name> parameter INSTALL ADDSERVICE Reads the GLOBALS MGRSERVNAME for the service name If no GLOBALS setting, uses default service name GGSMGRA GLOBALS file stores parameters that relate to the GoldenGate instance as a whole,as opposed to runtime parameters for a specific process. This file is referenced wheninstalling the Windows service, so that the correct name is registered.
  33. 33. Oracle GoldenGate Fundamentals Student Guide Prepare Environment: Installation - UNIX, Linux or z/OS Download .gz file to /ggs gzip –d {filename}.tar.gz tar -xvof {filename}.tar GGSCI> CREATE SUBDIRSFor UNIX, z/OS, or Linux: Use the gzip and tar options appropriate for yoursystem. If you are installing GoldenGate into a cluster environment, make certainthat the GoldenGate binaries and files are installed on a file system that is availableto all cluster nodes. After installing GoldenGate, make certain to configure theGoldenGate Manager process within the cluster application, as directed by thevendor’s documentation, so that GoldenGate will fail over properly with the otherapplications. The Manager process is the master control program for all GoldenGateoperations.A GoldenGate instance is a single installation of GoldenGate. Prepare Environment: Installation – NonStop SQL/MX For a SQL/MX source, install Oracle GoldenGate on OSS running on the NonStop source system: Download .gz file to /ggs gzip –d {filename}.tar.gz tar -xvf {filename}.tar GGSCI> CREATE SUBDIRS Run the ggmxinstall script For a SQL/MX target, install Oracle GoldenGate Either on OSS running on the NonStop target system (as described above) Or on an intermediate Windows system(as described earlier)The ggmxinstall script SQL compiles the Extract program and installs the VAMSERVprogram in the NonStop Guardian space. 33
  34. 34. The command to run it is: OSS> ggmxinstall /G/<Guardian vol>/<Guardian subvol>where: <Guardian vol>/<Guardian subvol> is the destination NonStop volume andsubvolume in OSS format. Prepare Environment: Installation - GoldenGate Directories Directory Contents dirchk GoldenGate checkpoint files dirdat GoldenGate trail and extract files dirdef Data definitions produced by DEFGEN and used to translate heterogeneous data dirpcs Process status files dirprm Parameter files dirrpt Process report files dirsql SQL scripts dirtmp Temporary storage for transactions that exceed allocated memorydirchkContains the checkpoint files created by Extract and Replicat processes, which storecurrent read and write positions to support data accuracy and fault tolerance. Writtenin internal GoldenGate format. Do not edit these files.The file name format is <group name><sequence number>.<ext> where <sequencenumber> is a sequential number appended to aged files and <ext> is either cpe forExtract checkpoint files or cpr for Replicat checkpoint files. Examples: ext1.cpe,rep1.cprdirdatThe default location for GoldenGate trail files and extract files created by Extractprocesses to store records of extracted data for further processing, either by theReplicat process or another application or utility. Written in internal GoldenGateformat. Do not edit these files.File name format is a user-defined two-character prefix followed by either a six-digitsequence number (trail files) or the user-defined name of the associated Extractprocess group (extract files). Examples: rt000001, financedirdef
  35. 35. Oracle GoldenGate Fundamentals Student GuideThe default location for data definitions files created by the DEFGEN utility to containsource or target data definitions used in a heterogeneous synchronizationenvironment. Written in external ASCII.File name format is a user-defined name specified in the DEFGEN parameter file.These files may be edited to add definitions for newly created tables. If you are unsureof how to edit a definitions file, contact technical support. Example: defs.datdirpcsDefault location for status files. File name format is <group>.<extension> where<group> is the name of the group and <extension> is either pce (Extract), pcr(Replicat), or pcm (Manager).These files are only created while a process is running. The file shows the programname, the process name, the port, and process ID that is running. Do not edit thesefiles. Examples: mgr.pcm, ext.pcedirprmThe default location for GoldenGate parameter files created by GoldenGate users tostore run-time parameters for GoldenGate process groups or utilities. Written inexternal ASCII format.File name format is <group name/user-defined name>.prm or mgr.prm. These filesmay be edited to change GoldenGate parameter values. They can be edited directlyfrom a text editor or by using the EDIT PARAMS command in GGSCI. Examples:defgen.prm, finance.prmdirrptThe default location for process report files created by Extract, Replicat, and Managerprocesses to report statistical information relating to a processing run. Written inexternal ASCII format.File name format is <group name><sequence number>.rpt where <sequence number>is a sequential number appended to aged files. Do not edit these files. Examples:fin2.rpt, mgr4.rptdirsqlThe default location for SQL scripts.dirtmpThe default location for storing large transactions when the size exceeds the allocatedmemory size. Do not edit these files. 35
  36. 36. Oracle GoldenGate Documentation Prepare Environment: Oracle GoldenGate Documentation Quick Install Guide Installation and Setup Guides (by database) Administration Guide Reference Guide Troubleshooting and Tuning Guide Note: You can download the documentation from http://www.oracle.com/technology/documentation/index.htmlWindows and UNIX platforms:● Oracle GoldenGate Quick Install Guide: Describes the structure of the media packand where to find installation instructions.● Oracle GoldenGate Installation and Setup Guides: There is an installation guideand setup guide for each database that is supported by Oracle GoldenGate. Theseinclude database-specific configuration information.● Oracle GoldenGate Administration Guide: Introduces Oracle GoldenGatecomponents and explains how to plan for, configure, and implement Oracle GoldenGateon the Windows and UNIX platforms.● Oracle GoldenGate Reference Guide: Provides detailed information about OracleGoldenGate parameters, commands, and functions for the Windows and UNIXplatforms.● Oracle GoldenGate Troubleshooting and Tuning Guide: Provides suggestions forimproving the performance of Oracle GoldenGate in different situations, and providessolutions to common problems.
  37. 37. Oracle GoldenGate Fundamentals Student GuideConfigure and Start Manager Prepare Environment: Manager - Overview Performs system management and monitoring tasks Starting GoldenGate processes Starting dynamic Server Collector, Replicat, or GGSCI processes Error and lag reporting GoldenGate trail management Parameter file mgr.prm file in GGS ./dirprm directory Event information written to ggserr.log fileThe Manager process performs system management and monitoring tasks onWindows and Unix, including the following. Starting Server Collector processes to collect data from remote Extract processes Threshold reporting (for example, when Extract falls behind GGSLOG) Purging trails Purging of GGSLOG or GGSLOG_HISTORY dataManager ParametersEnter Manager parameters in the dirprm/mgr.prm file, under the GoldenGateinstallation directory. If no mgr.prm file exists, default management parameters areused.Error and Informational ReportingManager reports critical and informational events to the ggserr.log file in theGoldenGate installation directory. 37
  38. 38. Prepare Environment: Manager - Configuration Create the parameter file using GGSCI GGSCI> EDIT PARAM MGR Start the Manager using GGSCI GGSCI> START MGR Note: To determine which port Manager is using GGSCI> INFO MGRStarting ManagerYou must start Manager before most other configuration tasks performed in GGSCI.Use either START MANAGER or START MGR.On Windows systems, you can also start and stop Manager through the standardWindows services control applet (in Control Panels). Prepare Environment: Manager – Sample MGR Parameter File PORT 7809 DYNAMICPORTLIST 8001, 8002, 9500–9520 PURGEOLDEXTRACTS /ggs/dirdat/aa*, USECHECKPOINTS PURGEOLDEXTRACTS /ggs/dirdat/bb*, & USECHECKPOINTS, MINKEEPDAYS 5 AUTOSTART ER * AUTORESTART EXTRACT *, WAITMINUTES 2, RETRIES 5 LAGREPORTHOURS 1 LAGINFOMINUTES 3 LAGCRITICALMINUTES 5This parameter file has the Manager listening on PORT 7809. Ports 8001, 8002, andthose in the range 9500 to 9520 will be assigned to the dynamic processes started byManager.This manager process will recycle GoldenGate trails that match the file name of/ggs/dirdat/aa* and /ggs/dirdat/bb*. It will only recycle the trail once all Extracts
  39. 39. Oracle GoldenGate Fundamentals Student Guideand Replicats have a checkpoint beyond the file (USECHECKPOINTS), however bb* trailswill not be purged until there has been no activity for 5 days.The manager will automatically start any Extract and Replicat process at startup andwill attempt to restart any Extract process that abends after waiting 2 minutes, butonly up to 5 attempts.The manager will report lag information every hour, but only for processes that have3 and 5 minutes of latency. The message will be flagged informational for lags of 3minutes and critical for any process that has a lag greater than 5 minutes.Generate Source Definitions (Heterogeneous Source/Target) Prepare Environment: Source Definitions - Overview The problem Understanding source and target layouts across disparate systems and databases The solution – the DEFGEN utility program DEFGEN produces a file containing layout definitions of the source files and tables This source definition file is used to interpret layouts for data stored in GoldenGate trails - At start up Replicat reads the definition file specified with the SOURCEDEFS parameter - Server Collector uses the –d argument to specify which definition file to read at startup Can also capture target definitions on target system and copy to source system for Extract to useThe ProblemWhen Capturing, Transforming, and Delivering data across disparate systems anddatabases, you must understand both the source and target layouts. Understandingcolumn names and data types is instrumental to GoldenGate’s data synchronizationfunctions.The Solution - The DEFGEN Utility ProgramThe DEFGEN utility program produces a file containing a definition of the layouts ofthe source files and tables. The output definitions are saved in an edit file andtransferred to all target systems in text format. Replicat and Collector read in thedefinitions at process startup and use the information to interpret the data from theGoldenGate trails.When transformation services are required on the source system, Extract can use adefinition file containing the target, rather than source, layouts. 39
  40. 40. Note: The user should never modify the DEFGEN output. Prepare Environment: Source Definitions – Run DEFGEN DEFGEN is initiated from the command prompt: defgen paramfile <paramfile> [ reportfile <reportfile> ] Unix Example: defgen paramfile /ggs/dirprm/defgen.prm reportfile /ggs/dirrpt/defgen.rpt Windows Example: defgen paramfile c:ggsdirprmdefgen.prm reportfile c:ggsdirrptdefgen.rpt Definitions are saved to the file specified in the parameter file This file needs to be transferred to the target system as a text file Prepare Environment: Source Definitions - Sample DEFGEN Parameters DEFSFILE /ggs/dirdef/source.def, PURGE SOURCEDB mydb, USERID ggs, PASSWORD ggs TABLE SALES.ACCOUNT; TABLE SALES.PRODUCT; Parameter Specifies DEFSFILE The output definitions file location and name SOURCEDB The database name (if needed) USERID The user ID and password (if needed) to access the database TABLE The table(s) to be defined
  41. 41. Oracle GoldenGate Fundamentals Student GuidePrepare the Source Database Prepare Environment: Source Database – Overview Set up the database to: Ensure access by GoldenGate Enable transaction logging Note: the exact steps depend on the databaseDatabase accessYou need to assign a database user for each of the GoldenGate processes, unless thedatabase allows authentication at the operating system level. While not required,GoldenGate recommends creating a user specifically for the GoldenGate application.To ensure that processing can be monitored accurately, do not permit other users orprocesses to operate as the GoldenGate user.In general, the following permissions are necessary for the GoldenGate user:• On the source system, the user must have permissions to read the data dictionary orcatalog tables.• On the source system, the user must have permissions to select data against thetables.• On the target system, the user must have the same permissions as the GoldenGateuser on the source system plus additional privileges to perform DML on the targettables. 41
  42. 42. Prepare Environment: Source Database Oracle Add minimal supplemental logging at database level ADD TRANDATA to mark tables for replication DB2 Enter DATA CAPTURE CHANGES at the column for LOB data type ADD TRANDATA to mark tables for replication Sybase Set the secondary truncation point in the logs ADD TRANDATA to mark tables for replication NonStop SQL/MX Special installation steps but no special database preparationOracle logs - On UNIX, GoldenGate reads the online logs by default, or the archivedlogs if an online is not available. On the Windows platform, GoldenGate reads thearchived logs by default, or the online logs if an archive is not available. GoldenGaterecommends archive logging be enabled, and that you keep the archived logs on thesystem for the longest time possible. This prevents the need to resynch data if theonline logs recycle before all data has been processed.DB2 - In addition to enabling logging at a global level, each table to be captured mustbe configured to capture data for logging purposes. This is accomplished by the DATACAPTURE CHANGE clause in the CREATE TABLE statement.Sybase –To capture database operations for tables that you want to synchronize withGoldenGate, each one must be marked for replication. This can be done through thedatabase, but GoldenGate recommends using ADD TRANDATA.GoldenGate uses the secondary transaction log truncation point to identifytransaction log entries that have not been processed by the Extract process. Thesecondary truncation point must be established prior to running the GoldenGateExtract process. The GoldenGate process will manage the secondary truncation pointonce it has been established.NonStop SQL/MXDuring the installation of SQL/MX, the script ggmxinstall sets a pointer to the VAMthat will work with Extract to capture changes from the TMF audit trail.
  43. 43. Oracle GoldenGate Fundamentals Student Guide Prepare Environment: Source Database – SQL Server To prepare the SQL Server source environment for GoldenGate: Create the ODBC data source GoldenGate connects to a SQL Server database through an ODBC connection Extract and Replicat require an established data source name (dsn) Set up transaction logging Log truncation and non-logged bulk copy must be turned off The SQL Server database must be set to full recovery mode Before GoldenGate processes are started, at least one full database backup must be done ADD TRANDATA to mark tables for replicationSupplemental logging is enabled using the GoldenGate command interface GGSCI.The other preparation is done using Windows and SQL Server utilities.ODBC Data Source Administrator is used to configure and define the ODBCconnection and to define the data source.SQL Server Enterprise Manager is used to set full recovery mode and back up thedatabase.SQL Server Query Analyzer is used to access the database to turn off logtruncation and non-logged bulk copy. 43
  44. 44. Prepare Environment: Source Database – SQL Server 2005 Additional considerations for SQL Server 2005 database: Either install Microsoft Cumulative Update package 6 for SQL Server 2005 Service Pack 2 (or later) Set TRANLOGOPTIONS to MANAGESECONDARYTRUNCATIONPOINT Or install SQL Server replication components Create a distribution database Add a replication publication Set transaction retention to zero Disable replication alerts Log full before and after images (no compressed) Set TRANLOGOPTIONS to NOMANAGESECONDARYTRUNCATIONPOINTThe Distributor database must be used only for source databases to be replicated byGoldenGate. One Distributor can be used for all of these databases. GoldenGate doesnot depend on the Distributor database, so transaction retention can be set to zero.Because GoldenGate does not depend on the Distributor database, but rather readsthe logs directly, the GoldenGate extraction process can process at its customary highspeed. For instructions on installing the replication component and creating aDistributor database, see the GoldenGate for Windows and UNIX AdministratorGuide.MANAGESECONDARYTRUNCATIONPOINTRequired TRANLOGOPTIONS parameters control whether or not GoldenGate maintainsthe secondary truncation point. Use the MANAGESECONDARYTRUNCATIONPOINT optionif GoldenGate will not be running concurrently with SQL Server replication. Use theNOMANAGESECONDARYTRUNCATIONPOINT option if GoldenGate will be runningconcurrently with SQL Server replication. Allows SQL Server replication to managethe secondary truncation point.Primary Key Requirement - The requirement that all tables to be captured fromSQL Server 2005 source databases must have a primary key is a requirement of theMicrosoft replication component, which is utilized by GoldenGate as part of the log-based capture process.Connecting - Use SQL Server Management Studio for SQL Server 2005/2000 toconnect to a MS SQL Server 2005 or 2000 database. Use Enterprise Manager only forMS SQL Server 2000.
  45. 45. Oracle GoldenGate Fundamentals Student Guide Prepare Environment – Discussion Points 1. Where do you download Oracle GoldenGate software from? 2. What are the roles and responsibilities of the Manager process?1. edelivery.oracle.com2. Starting and stopping processes; monitoring processes; reporting lag, errors and events; purging trail files. 45
  46. 46. GoldenGate Command Interface GGSCI – Starting and Help Start the command interface from the GoldenGate install directory: Shell> cd <GoldenGate install location> Shell> GGSCI For Help Summary page: GGSCI> HELP For Help on a specific command: GGSCI> HELP <command> <object> For example: GGSCI> HELP ADD EXTRACT Help returns overview, syntax and exampleGolden Gate Software Command Interface (GGSCI) provides on-line help for allcommands. The following is an example of the information returned when you enterHELP STATUS EXTRACT:Use STATUS EXTRACT to determine whether or not Extract groups are running.Syntax: STATUS EXTRACT <group name> [, TASKS] [, ALLPROCESSES]<group name> is the name of a group or a wildcard (*) to specify multiple groups.ALLPROCESSES displays status of all Extract processes, including tasks.TASKS displays status of all Extract tasks.Examples: STATUS EXTRACT FINANCE, STATUS EXTRACT FIN*
  47. 47. Oracle GoldenGate Fundamentals Student Guide GGSCI Commands MANAGER EXTRACT REPLICAT ER EXTTRAIL RMTTRAIL TRANDATA CHECKPOINT TRACE TABLE TABLE ADD X X X X X X X ALTER X X X X CLEANUP X X X DELETE X X X X X X X X INFO X X X X X X X X X KILL X X X LAG X X X REFRESH X SEND X X X X START X X X X STATS X X X STATUS X X X X STOP X X X XObjectsManager, Extract, Replicat GoldenGate processes.ER Multiple Extract and Replicat processes.EXTTRAIL Local trail.RMTTRAIL Remote trail.TRANDATA Transaction data (from transaction logs).CHECKPOINTTABLE Checkpoint table (on target database).TRACETABLE Oracle trace table (on target database).CommandsADD Creates an object or enables TRANDATA capture.ALTER Changes the attributes of an object.CLEANUP Deletes the run history of a process or removes records from a checkpointtable.DELETE Deletes an object or disables TRANDATA capture.INFO Displays information about an object (status, etc).KILL Forces a process to stop (no restart).LAG Displays the lag between when a record is processed by the process and thesource record timestamp.REFRESH Refreshes Manager parameters (except port number) without stoppingManager.SEND Sends commands to a running process.START Starts a process.STATS Displays statistics for one or more processes.STATUS Displays whether a process is running.STOP Stops a process gracefully. 47
  48. 48. GGSCI Commands (cont’d) Commands Parameters SET EDITOR, EDIT PARAMS, VIEW PARAMS Database DBLOGIN, ENCRYPT PASSWORD, LIST TABLES DDL DUMPDDL [SHOW] Miscellaneous !command, CREATE SUBDIRS, FC, HELP, HISTORY, INFO ALL, OBEY, SHELL, SHOW, VERSIONS, VIEW GGSEVT, VIEW REPORTParameter commandsSET EDITOR Changes the default text editor for the current GGSCI session fromNotepad or vi to any ASCII editor.EDIT PARAMS Edits a parameter file.VIEW PARAMS Displays the contents of a parameter file.Database commandsDBLOGIN Establishes a database connection through GGSCI.ENCRYPT PASSWORD Encrypts a database login password.LIST TABLES List all tables in the database that match a wildcard string.DDL commandsDUMPDDL Saves the GoldenGate DDL history table to file.SHOW option displays the DDL information in standard output format.Miscellaneous commands!command Executes a previous GGSCI command without modification.CREATE SUBDIRS Creates default directories within the GoldenGate home directory.FC Edit a previously issued GGSCI command.HELP Displays information about a GGSCI command.HISTORY List the most recent GGSCI commands issued.INFO ALL Displays the status and lag for all GoldenGate processes on a system.OBEY Runs a file containing a list of GGSCI commands.SHELL Run shell commands from within GGSCI.SHOW Displays the GoldenGate environment.VERSIONS Displays OS and database versions.VIEW GGSEVT Displays the GoldenGate event/error log.VIEW REPORT Displays a process report for Extract or Replicat.
  49. 49. Oracle GoldenGate Fundamentals Student GuideGGSCI Examples Start a Manager process GGSCI> START MGR Add an Extract group GGSCI> ADD EXTRACT myext, TRANLOG, BEGIN NOW Add a local trail GGSCI> ADD EXTTRAIL /ggs/dirdat/rt, EXTRACT myext Start an Extract group GGSCI> START EXTRACT myextUsing Obey Files You can use an Obey file to perform a reusable sequence of commands Save the commands in a text file, for example: START MGR ADD EXTRACT myext, TRANLOG, BEGIN NOW ADD EXTTRAIL /ggs/dirdat/rt, EXTRACT myext START EXTRACT myext Then use the GGSCI OBEY command to run the file: GGSCI> OBEY <obey filename>.oby Note. An Obey file can have any file extension or none. 49
  50. 50. Running GoldenGate from the OS Shell You can also start GoldenGate processes from the OS command shell when running a batch job or initial load, for example: Shell> cd <GoldenGate install location> Shell> extract paramfile <filepath> reportfile <filepath> [-p <port>] Shell> replicat paramfile <filepath> reportfile <filepath> This is especially useful to schedule GoldenGate batch jobs to run during off-peak hours using a command-line capable schedulerManager must be running when you issue these commands.<filepath> specifies the fully qualified name of the parameter and report files.paramfile can be abbreviated to pf.reportfile can be abbreviated to rf. GoldenGate Commands – Discussion Points 1. What is GGSCI? 2. Where can you view the GoldenGate command syntax? 3. What is an Obey file and why would you use one?1. GoldenGate Software Command Interface.2. Help or Reference Guide.3. A text file containing a sequence of GoldenGate commands; for easy re-use of common command sequences.
  51. 51. Oracle GoldenGate Fundamentals Student GuideStep 2. Change Capture Step 2. Change Capture 1. Prepare the Environment Source Database 3. Initial Load (various methods) Target Database Transaction Log Local Data Extract Pump Remote Replicat Trail Trail 4. Change Delivery 2. Change Capture Change Capture - Extract Overview Extract can be configured to: Capture changed data from database logs Distribute data from local trails to remote systems (data pump) Capture data directly from source tables for initial data load 51
  52. 52. Change Capture - Tasks On the source system: Add a primary Extract (reading from source transaction logs) with an associated parameter file Optionally, add a local trail and a data pump Extract (reading from the local trail) with an associated parameter file Add a remote trail Start the Extract(s)To configure Extract to capture changes from transaction logs, perform the followingsteps: Set up a parameter file for Extract with the GGSCI EDIT PARAMS command. Set up an initial Extract checkpoint into the logs with the GGSCI ADD EXTRACTcommand. Optionally, create a local trail using the GGSCI ADD EXTTRAIL command and a datapump Extract (and parameter file) reading from the local trail. Set up a remote trail using the GGSCI ADD RMTTRAIL command. Start the Server Collector process on the target system or let the Manager start theServer Collector dynamically. Start Extract using the GGSCI START EXTRACT command. For example: GGSCI>START EXTRACT FINANCEGGSCI sends this request to the Manager process, which in turn starts Extract.Manager monitors the Extract process and restarts it, when appropriate, if it goesdown. Change Capture - ADD EXTRACT Command Add the initial Extract checkpoint with the GGSCI command ADD EXTRACT: ADD EXTRACT <group name> , <data source> , <starting point> [, <processing options>] The components of this command are discussed in subsequent slides.
  53. 53. Oracle GoldenGate Fundamentals Student Guide Change Capture - ADD EXTRACT <data source> <data source> Source (and when used) SOURCEISTABLE Database table (initial data load) TRANLOG Transaction log (change capture) [<bsds name>] [DB2 z/OS] EXTFILESOURCE <file name> Extract file (data pump) EXTTRAILSOURCE <trail name> Trail (data pump)SOURCEISTABLE Creates an Extract task that extracts entire records from thedatabase for an initial load. If SOURCEISTABLE is not specified, ADD EXTRACTcreates an online change-synchronization process, and one of the other data sourceoptions must be specified. When using SOURCEISTABLE, do not specify serviceoptions. Task parameters must be specified in the parameter file.TRANLOG [<bsds name>]Specifies the transaction log as the data source. Use this option for log-basedextraction. TRANLOG requires the BEGIN option.Use the <bsds name> option for DB2 on a z/OS system to specify the BSDS(Bootstrap Data Set) file name of the transaction log. Make certain that the BSDSname you provide is the one for the DB2 instance to which the Extract process isconnected. GoldenGate does not perform any validations of the BSDS specification.EXTFILESOURCE <file name>Specifies an extract file as the data source. Use this option with a secondary Extractgroup (data pump) that acts as an intermediary between a primary Extract group andthe target system. For <file name>, specify the fully qualified path name of the file,for example c:ggsdirdatextfile.EXTTRAILSOURCE <trail name>Specifies a trail as the data source. Use this option with a secondary Extract group(data pump) that acts as an intermediary between a primary Extract group and thetarget system. For <trail name>, specify the fully qualified path name of the trail, forexample c:ggsdirdataa. 53
  54. 54. Change Capture - ADD EXTRACT <starting point> <starting point> Database BEGIN {NOW | <datetime> } Any EXTSEQNO <seqno>, EXTRBA <relative byte address> Oracle, SQL/MX EXTRBA <relative byte address> DB2 z/OS EOF | LSN <value> DB2 LUW LSN <value> SQL Server, Ingres LOGNUM <log number>, LOGPOS <byte offset> c-tree PAGE <data page>, ROW <row> SybaseThe starting point is indicated by one of the following:BEGIN specifies when the Extract begins processing.- For all databases except DB2 LUW, NOW specifies the time at which the ADDEXTRACT command is issued.- For DB2 LUW, NOW specifies the time at which START EXTRACT takes effect.<datetime> specifies the start date and time in the format: yyyy-mm-dd[hh:mi:[ss[.cccccc]]].Several parameters specify the position with the log or trail to begin processing:EXTSEQNO <seqno>, EXTRBA <relative byte address>Valid for a primary Extract for Oracle and NonStop SQL/MX, and for a data pumpExtract. Specifies one of the following:- sequence number of an Oracle redo log and RBA within that log at which to begincapturing data.- the NonStop SQL/MX TMF audit trail sequence number and relative byte addresswithin that file at which to begin capturingdata. Together these specify the location in the TMF Master Audit Trail (MAT).- the file in a trail in which to begin capturing data (for a data pump). Specify thesequence number, but not any zeroes used for padding. For example, if the trail file isc:ggsdirdataa000026, you would specify EXTSEQNO 26. By default, processingbegins at the beginning of a trail unless this option is used.Contact GoldenGate Technical Support before using this option.EXTRBA <relative byte address>Valid for DB2 on z/OS. Specifies the relative byte address within a transaction log atwhich to begin capturing data.

×