4. PROCESS STARTUP AND TCP/IP ERRORS ............................................................................................... 219
REPORTING AND STATISTICS .................................................................................................................. 221
MONITIORING ORACLE GOLDENGATE ................................................................................................... 229
TROUBLESHOOTING ................................................................................................................................ 235
TECHNICAL SUPPORT ............................................................................................................................. 244
5. Oracle GoldenGate Fundamentals Student Guide
About GoldenGate – Company and Solutions
Our Business
We enable real-time, continuous movement of transactional data across
Operational and Analytical business systems.
Real-Time Access
to
Real-Time Information
Real-Time Access Real-Time Information
Mission-Critical Integration: the process of
Availability: the degree to
Systems combining data from different
which information can be
sources to provide a unified view.
instantly accessed.
Oracle GoldenGate provides solutions that enable your mission-critical systems to
have continuous availability and access to real-time data.
We offer a robust yet easy platform for moving real-time transactional data between
operational and analytical systems to enable both:
- High Availability solutions, and
- Real Time Integration solutions
• Real-Time Access -- meaning your critical data is accessible and available
whenever you need it, 24x7.
At the same time…
• Real-Time Information -- meaning that the data available is as current as
possible – not 24 hours old, not even 4 hours old.
6. Oracle GoldenGate Success
Company Strength GoldenGate Software Acquired by Global sales and
and Service established in 1995 Oracle in 2009 support
Rapid Growth in
Strategic Partners
500+ customers... 4000+ solutions implemented… in 35 countries
Established, Loyal
Customer Base
Our partnerships are rapidly increasing with major technology players, including
database and IT infrastructure, packaged applications, business intelligence, and
service providers.
And because our software platform supports a variety of solution use cases… Our
more than 500 customers are using our technology for over 4000 solutions around the
world. What we typically find that once an initial solution is implemented and the
benefits achieved, our customers then find additional areas across the enterprise
where we can further drive advantages for them.
7. Oracle GoldenGate Fundamentals Student Guide
Transactional Data Management (TDM)
Oracle GoldenGate provides low-impact capture, routing, transformation, and
delivery of database transactions across heterogeneous environments in real
time
Key Capabilities: Additional Differentiators:
Log-based capture moves
Real Time thousands of transactions
Performance per second with low
Moves with sub-second latency
impact
Heterogeneous Meets variety of customer
Extensibility & needs and data
Moves changed data across Flexibility environments with open,
different databases and platforms modular architecture
Transactional Resilient against
Reliability interruptions and failures
Maintains transaction integrity
Our focus is on transactional data management (TDM) – which means delivering a
platform in which that data can be best utilized in real-time enterprise wide.
Oracle GoldenGate captures, routes, transforms, and delivers transactional data in
real time – and it works across heterogeneous environments with very low impact and
preserved transaction integrity.
Our Key Capabilities in which we architect the product are:
• We move data essentially in “real time” – with sub-second speed.
• Works in heterogeneous environments – across different database and
hardware types
• Transactional – we are “transaction aware” and apply read-consistent
changed data to maintain its referential integrity between source and target
systems.
We further Differentiate ourselves from other technologies with:
- High performance with low impact – we can move large volumes of data
very efficiently while maintaining very low lag times/latency.
- Our flexibility – we meet a wide range of customer solution and integration
needs, thanks to our open, modular architecture.
- Our reliability – our architecture is extremely resilient against potential
interruptions; no single point of failure or dependencies, and easy to recover.
7
8. TRANSACTIONAL
DATA INTEGRATION
Oracle GoldenGate provides the following data replication solutions:
•High Availability
Live Standby for an immediate fail-over solution that can later re-synchronize with
your primary source.
Active-Active solutions for continuous availability and transaction load distribution
between two or more active systems.
•Zero-Downtime Upgrades and Migrations
Eliminate downtime for upgrades and migrations.
•Live Reporting
Feeding a reporting database so that you don’t burden your source production
systems.
•Operational Business Intelligence (BI)
Real-time data feeds to operational data stores or data warehouses, directly or via
ETL tools.
•Transactional data integration
Real-time data feeds to messaging systems for business activity monitoring (BAM),
business process monitoring (BPM) and complex event processing (CEP).
Uses event-driven architecture (EDA) and service-oriented architecture (SOA).
9. Oracle GoldenGate Fundamentals Student Guide
Oracle GoldenGate Solutions
High Availability & Disaster Real-Time Data Integration
Tolerance
Live Standby Real-Time Data Warehousing
Active-Active Live Reporting
Zero-Downtime Operations for: Transactional Data Integration
Upgrades
Migrations
Maintenance
Oracle GoldenGate provides two primary solution areas:
High Availability/Disaster Tolerance and Real-Time Data Integration.
Within High Availability and Disaster Tolerance, we offer:
• Active-Active solutions for continuous availability and transaction load distribution
between two or more active systems
•Zero-Downtime Operations that eliminates downtime for planned outages involving
upgrades, migrations, and ongoing maintenance
•Live Standby for an immediate fail-over solution that can later re-synchronize with
your primary source
Within Real-Time Data Integration, we offer:
• Real-Time Data Warehousing which gives you real-time data feeds to data
warehouses or operational data stores
•Transactional Data Integration for distributing data in real-time between
transaction processing systems
•Live Reporting is for feeding a reporting database so that you don’t burden your
source production systems
9
10. Oracle GoldenGate for High Availability & Disaster Tolerance
High Availability & Disaster
Tolerance
Live Standby Real-Time Access
Active-Active
Improved Uptime
Higher Performance
Zero-Downtime Operations for:
Faster Recovery
Upgrades
Minimized Data Loss
Migrations
Maintenance
Lower TCO
For High Availability and Disaster Tolerance solutions, it’s about “real-time” or
CONTINUOUS access to your data via your critical applications.
The benefits that Oracle GoldenGate drives here include:
-Improved uptime and availability (helping you reach aggressive service level
agreements/SLAs)
-Higher Performance for your production systems – help to eliminate scalability or
response time delays that can give users the impression of an availability or access
issue
-Faster Recovery and Minimized Data Loss – so you can achieve higher Recovery
Time Objectives (RTOs) and Recovery Point Objectives (RPOs)
- and an overall lower Total Cost of Ownership by putting your standby systems to
work for other solutions!
11. Oracle GoldenGate Fundamentals Student Guide
High Availability: Live Standby
Benefits:
Eliminate unplanned downtime
Reduce data loss and isolate corrupt data
Re-synchronize backup and primary systems
Remove distance constraints
Automate switchovers
Improve ROI with active standby available for reporting
Live Standby: helps eliminate unplanned outages to enable continuous availability,
with no geographic distance constraints.
Oracle GoldenGate moves changed data from primary database to a standby in sub-
seconds so that end users have a reliable failover system with up to date data that
they can immediately switchover.
There is no database recovery process required because changed data is queued
outside of the database in persisted Trail files, and data loss risk is minimized. Oracle
GoldenGate also isolates corrupt data during movement to make sure the secondary
system is reliable when it is needed.
The customer’s Return on Investment can be further increased by using the live
standby system for reporting or testing – Oracle GoldenGate allows the standby
database to be open, so it does not have to sit idle and can be put to work!
11
12. High Availability: Active-Active
Benefits:
Achieve continuous availability
Enable transaction load distribution (with built-in conflict resolution)
Improve performance
Lower TCO
Active-Active: Oracle GoldenGate enables bidirectional data movement between two
or more databases that actively support an application, with no geographic distance
constraints.
The active-active solution allows data updates and changes (“write” activity) to occur
on two or more active databases supporting live applications. Oracle GoldenGate
synchronizes the two active databases by replicating the data between each at a
logical level, and allows load distribution to improve system performance. In the case
of an outage of one system, there is no downtime for the end user because the other
active system continues with operations.
Because Oracle GoldenGate is an asynchronous solution, conflict management is
required to ensure data accuracy in the event that the same row is changed in two or
more databases at (or about) the same time. Oracle GoldenGate provides capabilities
to detect and resolve conflicts as well.
A variety of active-active scenarios can be supported – depending on the desired
implementations. We have strong experience in active-active solutions for both High
Availability as well as Zero-Downtime upgrades and migration projects.
13. Oracle GoldenGate Fundamentals Student Guide
Zero-Downtime Upgrades and Migrations
Benefits:
Eliminate “planned downtime” during hardware, database, OS and/or
application upgrades and migrations
Minimize risk with fail-back contingency
Improve success with phased user migrations
Zero Downtime Operations: is for eliminating planned outages during database,
application or server upgrades, migrations, and/or maintenance.
Oracle GoldenGate captures all the changed data in the primary system while the new
system is initiated and prepared. Once the second or the new system is upgraded or
migrated Oracle GoldenGate applies all the changed data to the new system. Oracle
GoldenGate then keeps the two environments synched with our real-time replication.
Often with such projects, there is always concern about what will happen once you
switchover to the new environment. Oracle GoldenGate alleviates many of those risks
with our fail-back capabilities -- after switchover Oracle GoldenGate captures the
changes that happen in the new system so that the old system is kept up to date in
case there is a need for fail back to the old environment.
And currently in Oracle or HP Nonstop environments, our Oracle GoldenGate
Veridata product verifies that the data is consistent in both systems before and even
after switchover.
13
14. Oracle GoldenGate for Real-Time Data Integration
Real-Time Data Integration
Real-Time Information
Live Reporting
Fresher Data Operational Business Intelligence
Minimal Overhead
No Batch Windows Transactional Data Integration
Data Integrity
Ease of Integration
For our Real-Time Data Integration solutions, it’s about “real-time information” or
access to CURRENT operational data.
The benefits that Oracle GoldenGate drives here include:
• Fresher, real-time data available for use and decision-making – remove latency as a
technical constraint.
• Minimal overhead and impact on your source systems and overall architecture to
capture and move real-time data
• No requirement for batch windows
• Transactional data integrity helps improve overall data quality
• Ease of integration – Oracle GoldenGate easily fits into existing and desire
architecture, and is overall easy to maintain over long term.
15. Oracle GoldenGate Fundamentals Student Guide
Data Integration: Live Reporting
Benefits:
Use real-time data for better, faster decision making
Remove reporting overhead on source system
Reduce cost-to-scale as user demands and data volumes grow
Leverage cost-effective systems for reporting needs
Oracle GoldenGate's Live Reporting enables both real-time reporting capabilities
while improving the performance of operational source systems.
Oracle GoldenGate feeds real-time data from the source to a secondary reporting-only
database such as an operational data store (ODS). This allows reporting activity to be
off-loaded from the production database. This secondary database can be a different
database and/or platform from the production database, to lower the total cost of
ownership and allow organizations to leverage emerging open source technologies.
The solution also helps increase scalability as user demands and data volumes grow.
15
16. Operational Business Intelligence
Benefits:
Use real-time data for better, faster decision making
Eliminate batch window dependency
Reduce overhead on source system
Maintain referential integrity for data quality
Leverage its flexibility for transformations and integration with ETL
18
For Real-Time Data Warehousing -- The Oracle GoldenGate Real-Time Data
Warehousing solution enables continuous, real-time data feeds for data warehouses or
operational data stores, to improve business intelligence. Our log-based changed data
capture has very minimal impact on the source, no batch windows and moves the data
in sub-seconds. Each transaction’s commit boundaries are maintained for data
integrity.
Oracle GoldenGate’s architecture also improves data recoverability in case there is an
outage during the data movement. This is an important requirement as data latency
decreases in feeding the analytical environment. Oracle GoldenGate’s trail files that
store the changed data are persisted, so if needed they can be reapplied to the target
and also source system without having to capture the data again.
Transformations or co-existing with ETL:
Oracle GoldenGate out-of-the box can support a number of common data
transformations often required for data integration. However, where complex
transformations are needed Oracle GoldenGate can be used to augment an existing
ETL solution in several ways:
1) First, Oracle GoldenGate can deliver transactional data to staging tables in real
time, which then would be used by the ETL to extract from and perform
transformations and then load user tables. This method works best when the ETL
product is optimized to perform the transformations within the target database. This
is an “ELT” model.
2) Second method: Oracle GoldenGate provides the data to the ETL engine as flat files
and in micro-batches. The latency depends on the ETL product and business
requirements but we typically deliver every few minutes to an hour.
3) Third method: Oracle GoldenGate publishes changed data to a messaging system
and the ETL solution (that can subscribes to the queue or topic) receives it in real-
time.
In each of these architectures combining real-time change data capture with ETL
decreases data latency to real time or near real-time and eliminates the batch window
dependency.
17. Oracle GoldenGate Fundamentals Student Guide
Transactional Data Integration
Benefits:
Easily integrate large volumes of real-time data
between transaction processing systems
Reduce overhead; Eliminate batch windows
Improve scalability
Enhance SOA and EDA environments (delivery to
JMS-based messaging systems)
19
Oracle GoldenGate provides real-time data integration between OLTP systems non-
intrusively and with minimal impact. Distributed databases and the applications they
support can continuously access, utilize, and act on the most current operational data
available. The solution can also integrate with JMS-based messaging systems to
enable event driven architecture (EDA) and to support service oriented architecture
(SOA).
17
18. Technology Overview
How Oracle GoldenGate Works: Modular “Building Blocks”
Capture: Committed changes are captured (and can be
filtered) as they occur by reading the transaction logs.
Trail files: Stages and queues data for routing.
Route: Data is compressed, encrypted for routing to targets.
Delivery: Applies data with transaction
integrity, transforming the data as
required.
Source Network Target
Capture (TCP/IP) Delivery
Trail Trail
Source Target
Database(s) Target Source Database(s)
Delivery Capture
Trail Trail
Bi-directional
Oracle GoldenGate consists of decoupled modules that are combined to create the best
possible solution for your business requirements.
On the source system(s):
•Oracle GoldenGate’s Capture (Extract) process reads data transactions as they occur,
by reading the native transaction log, typically the redo log. Oracle GoldenGate only
moves changed, committed transactional data, which is only a % of all transactions –
therefore operating with extremely high performance and very low impact on the data
infrastructure.
• Filtering can be performed at the source or target - at table, column and/or row level.
• Transformations can be applied at the capture or delivery stages.
• Advanced queuing (trail files):
To move transactional data efficiently and accurately across systems, Oracle
GoldenGate converts the captured data into a Oracle GoldenGate data format in
“trail” files. With both source and target trail files, Oracle GoldenGate’s unique
architecture eliminates any single point of failure and ensures data integrity is
maintained – even in the event of a system error or outage.
Routing:
•Data is sent via TCP/IP to the target systems. Data compression and encryption are
supported. Thousands of transactions can be moved per second, without distance
limitations.
On the target system(s):
•A Server Collector process (not shown) reassembles the transactional data into a
target trail.
19. Oracle GoldenGate Fundamentals Student Guide
•The Delivery (Replicat) process applies transactional data to the designated target
systems using native SQL calls.
Bi-directional:
•In bi-directional configurations/solutions, this process runs the same in reverse, to
concurrently synchronize data between the source and target systems.
Manager processes (not shown) perform administrative functions at each node.
Oracle GoldenGate Supports Applications Running On…
Databases O/S and Platforms
Capture:
Oracle Windows 2000, 2003, XP
DB2 Linux
Microsoft SQL Server Sun Solaris
Sybase ASE HP NonStop
Ingres HP-UX
Teradata HP TRU64
Enscribe HP OpenVMS
SQL/MP
IBM AIX
SQL/MX
IBM z/OS
Delivery:
All listed above
MySQL, HP Neoview, Netezza, and any
ODBC compatible databases
ETL products
JMS message queues
Oracle GoldenGate is ideal for heterogeneous environments – not just supporting
different versions of the same database or operation system/hardware, but replicating
and integrating data across vendor systems.
We support log-based Capture of changed data from nearly all major database
vendors.
We can Deliver that data to an even wider range of targets – including open source
databases, several data warehouse appliances, ETL servers, and JMS message queues
to support Service Oriented Architectures (SOA) and Event-Driven Architectures
(EDA).
19
20. Oracle GoldenGate Advantages
Movement Management Integration
Speed Transaction Integrity Heterogeneous Data
– Subsecond Latency Transparent Capture Sources
Volume Guaranteed Delivery Mapping
– Thousands of TPS Conflict Detection, Transformation
Log-based Capture Resolution Enrichment
Native, Local Apply Dynamic Rollback Decoupled Architecture
Efficient IO and Incremental TDM Table, Row, Column
Bandwidth Usage Initial Data Load Filtering
Bidirectional GUI-based Monitoring and XML, ASCII, SQL Formats
Group Transactions Configuration Queue Interface
Bulk Operations Proactive Alerts Stored Procedures
Compression Encryption User Exits
One-to-Many, Real-Time Deferred or ETL Integration
Many-to-One Batch Java/JMS Integration
Cascade Event Markers
Oracle GoldenGate Director
Manages, defines, configures, and reports on Oracle GoldenGate
components
Key features:
Centralized management of Oracle GoldenGate modules
Rich-client and Web-based interfaces
Alert notifications and integration with 3rd-party monitoring products
Real-time feedback
Zero-impact implementation
Oracle GoldenGate Director is a centralized server-based graphical enterprise
application that offers an intuitive way to define, configure, manage, and report on
Oracle GoldenGate processes.
Oracle GoldenGate Director is a value added module to centralize management and
improve productivity.
Oracle GoldenGate Director supports all platforms and databases supported by Oracle
GoldenGate.
21. Oracle GoldenGate Fundamentals Student Guide
Oracle GoldenGate Veridata
A high-speed, low impact data comparison solution
Identifies and reports data discrepancies between two databases without
interrupting those systems or the business processes they support
Supports Oracle, Teradata, SQL Server, NonStop SQL/MP and Enscribe
Supports homogeneous and heterogeneous compares
Benefits:
Reduce financial/legal risk exposure
Speed and simplify IT work in
comparing data sources
No disruption to business systems
Improved failover to backup systems
Confident decision-making and
reporting
Oracle GoldenGate Veridata is a high-speed data comparison solution that identifies
and reports data discrepancies between databases without interrupting ongoing
business processes. Using Oracle GoldenGate Veridata, companies can audit and
verify large volumes of data across a variety of business applications with certainty,
and maintain reliable data synchronization. Oracle GoldenGate Veridata reduces the
amount of time and resources required to compare data, it minimizes the impact of
human errors, and it ensures that potential problems can be instantly identified and
addressed.
Key Veridata Features:
•Compares large data volumes with high speed and efficiency
•Allows both data sources to be online by handling in-flight transactions
•Performs selective, parallel comparison
•Offers intuitive Web interface and personalized views
•Enables the comparison of databases that are different database versions or on
different operating systems
•(HP Nonstop only) Supports the comparison of only the data changed since the
initial comparison (delta processing)
Why would you need Veridata?
Data discrepancies arise even without malicious intent -- due to infrastructure
problems, application errors, operator mistakes, configuration errors, or unexpected
user behavior. With vigilant verification procedures using Oracle GoldenGate
Veridata, companies can eliminate data inconsistencies across different business
applications and avoid potential operational, financial, or regulatory risks.
21
22. Architecture
Oracle GoldenGate Data Capture and Delivery
Oracle GoldenGate Transactional Data Management:
Primarily used for change data capture and delivery from database
transaction logs
Can optionally be used for initial load directly from database tables
Especially useful for synchronizing heterogeneous databases
Database-specific methods may be preferable for homogeneous
configurations
Change Data Capture & Delivery
Source
Database
Network
(TCP/IP) Target
Database
Extract Server Replicat
Collector Trail
Transaction
Log
Manager Manager
On the source system:
An Extract process captures transactional changes from transaction logs
The Extract process sends data across a TCP/IP network to the target system.
On the target system:
A Server Collector process reassembles and writes the data to a GoldenGate trail.
A Replicat process reads the trail and applies it to the target database. This can be
concurrent with the data capture or performed later.
23. Oracle GoldenGate Fundamentals Student Guide
Manager processes on both systems control activities such as starting, monitoring and
restarting processes; allocating data storage; and reporting errors and events.
Change Data Capture & Delivery using a Data Pump
Source
Database
Network
(TCP/IP) Target
Database
Extract Server Replicat
Collector Remote
Trail
Transaction
Log
Local Data
Trail Pump
Manager Manager
On the source system:
An Extract process captures transactional changes from the database transaction log
The Extract process writes the data to a local GoldenGate trail. This preserves the
captured data if the network or target trail fails.
A second Extract process (called a Data Pump) sends the data across the network to
the target system.
On the target system:
A Server Collector process reassembles and writes the data to a GoldenGate trail.
A Replicat process reads the trail and applies it to the target database. This can be
concurrent with the data capture or performed later.
Manager processes on both systems control activities such as starting, monitoring and
restarting processes; allocating data storage; and reporting errors and events.
23
24. Initial Load
Source Network
Database (TCP/IP) Target
Tables Database
Extract Replicat
Or DB Bulk
Server Load Utility
Collector Files
Manager Manager
GoldenGate initial load methods:
Direct Load (Extract sends data directly to Replicat to apply using SQL)
Direct Bulk Load (Replicat uses Oracle SQL*Loader API)
File to Replicat (Extract writes to a file that Replicat applies using SQL)
File to database utility (Extract writes to a file formatted for a DB bulk load utility)
On the source system:
An Extract process captures source data directly from tables
The Extract process sends data in large blocks across a TCP/IP network to the
target system.
On the target system, one of the following scenarios:
1. Direct Load. Replicat reads the data stream and concurrently applies the data to
the target database using SQL.
2. Direct Bulk Load (Oracle). Replicat can apply the data using the Oracle
SQL*Loader API to improve performance.
3. File to Replicat. Server Collector reassembles and writes the data to Extract files.
Replicat applies the data to the target database using SQL.
4. File to database utility. Server Collector reassembles and writes the data to files
formatted for a bulk loader, which applies the data to the target database.
Manager processes on both systems control activities such as starting, monitoring and
restarting processes; allocating data storage; and reporting errors and events.
25. Oracle GoldenGate Fundamentals Student Guide
Online versus Batch
Change data capture & delivery can be run either continuously (online)
or as a special run (batch run) to capture changes for a specific period of
time.
Initial load is always a special run (batch run).
Checkpointing - Extract
For change data capture, Extract and Replicat save checkpoints to a
checkpoint file so they can recover in case of failure
Extract maintains:
2 input checkpoints
1 output checkpoint for each trail it writes to
Start of oldest uncommitted Last record
transaction in log read from log
Input:
Transaction Log
Checkpoints
Output: One or more
GoldenGate Trails
End of last committed
transaction written to trail
Checkpoints are used during online change synchronization to store the current read
and write position of a process. Checkpoints ensure that data changes marked for
synchronization are extracted, and they prevent redundant extractions. They provide
fault tolerance by preventing the loss of data should the system, the network, or a
GoldenGate process need to be restarted.
25
26. Checkpointing - Replicat
Best practice is to create a checkpoint table in the target database
Checkpoints are maintained in both the checkpoint table (if it exists) and
a checkpoint file
Replicat maintains 2 input checkpoints:
Start of current Last record
uncommitted transaction read from trail
Input:
GoldenGate Trail
Checkpoints
Parameters, Process Groups and Commands
GoldenGate processes are configured by ASCII parameter files.
A process group consists of:
An Extract or Replicat process
Associated parameter file
Associated checkpoint file
Any other files associated with that process
Each process group on a system must have a unique group name.
Processes are added and started using the GoldenGate Software
Command Interface (GGSCI) with the group name.
GGSCI commands also add trails, check process status, etc.
27. Oracle GoldenGate Fundamentals Student Guide
Solutions and Architecture – Discussion Points
1. How is Oracle GoldenGate different from simply replicating
database operations?
2. What are some use cases for Oracle GoldenGate software?
3. What is the purpose of checkpointing?
1. Log-based change data capture, decoupled from database architecture. Real-time,
heterogeneous and transactional.
2. (a) High availability – live standby, active-active, zero down-time upgrades and
migrations.
(b) Real-time data integration – real-time data warehousing (operational business
intelligence), live reporting, transactional data integration.
3. For recovery if a GoldenGate process, network or system goes down.
27
28. Configuring Oracle GoldenGate
Configuring Oracle GoldenGate
Oracle GoldenGate can be deployed quickly and easily in four steps:
1. Prepare the Environment
2. Change Capture
3. Initial Load
4. Change Delivery
Note: You can run log-based change capture after the initial data load if you set the
extract begin time to the start of the longest running transaction committed
during the initial data load.
Configuring Oracle GoldenGate
1. Prepare the Environment
Source
Database
3. Initial Load (various methods)
Target
Database
Transaction
Log
Local Data
Extract Pump Remote Replicat
Trail Trail
4. Change Delivery
2. Change Capture
Oracle GoldenGate can be deployed quickly and easily in four steps:
• Prepare the environment, e.g.
• Install Oracle GoldenGate software on source and target
• Enable transaction logging
29. Oracle GoldenGate Fundamentals Student Guide
• (Heterogeneous source/target) Generate source definitions so Replicat can
process trail data
• Configure and start change capture to GoldenGate trail files (Extract processes –
primary and data pump)
• Perform initial load to synchronize databases by database-specific or GoldenGate
methods
• Configure and start change delivery (Replicat process)
Step 1. Prepare the Environment
Step 1. Prepare the Environment
1. Prepare the Environment
Source
Database
3. Initial Load (various methods)
Target
Database
Transaction
Log
Data
Extract Local Pump Remote Replicat
Trail Trail
4. Change Delivery
2. Change Capture
Step 1. Prepare the Environment
Set up each system:
Install Oracle GoldenGate software on source and target
Configure and start GoldenGate Manager on source and target
If heterogeneous source/target, generate source definitions and
copy to target
Prepare the database. For example:
Ensure database access by GoldenGate
Enable transaction logging
29
30. Installing Oracle GoldenGate installs all of the components required to run and
manage GoldenGate processing, and it installs the GoldenGate utilities.
Manager must be running on each system before Extract or Replicat can be started,
and must remain running while those processes are running so that resource
management functions are performed.
The source definitions file contains the definitions of the source tables and is required
on the target system in hetereogeneous configurations. Replicat refers to the file to
when transforming data from the source to the target.
To reconstruct an update operation, GoldenGate needs more information than Oracle
and SQL Server transaction logs provide by default. Adding supplemental log data
forces the logging of the full before and after image for updates.
Install Oracle GoldenGate Software
Prepare Environment: Installation – Access the Media Pack
Access the product media pack (software and documentation) at
edelivery.oracle.com
Identify the proper release of GoldenGate for your source and target environments
Database and version
Operating system and version
A GoldenGate instance is a single installation of GoldenGate.
31. Oracle GoldenGate Fundamentals Student Guide
Prepare Environment: Installation - Windows
Download .zip file to C:GGS
Unzip .zip file into C:GGS folder
Configure a Windows Service Name for Manager process in a
GLOBALS parameter file (required only if multiple Managers on
the server)
C:GGS> INSTALL ADDSERVICE ADDEVENTS
GGSCI> CREATE SUBDIRS
For Windows: Do not install Oracle GoldenGate into a folder that contains spaces in
its name, for example “GoldenGate Software.” The application references path
names, and the operating system does not support path names that contain spaces,
whether or not they are within quotes.
Prepare Environment: Installation – Windows INSTALL Program
On Windows, an INSTALL program performs the following
functions:
Installs GoldenGate event messages into the system registry
Installs the Manager as a Windows service
Syntax:
INSTALL <item> [<item> …]
Example:
C:GGS> INSTALL ADDEVENTS ADDSERVICE
Note: The uninstall command is:
INSTALL DELETESERVICE DELETEEVENTS
Items (all optional)
ADDEVENTS Adds the GoldenGate events to the registry so that event messages
appear in the Windows Event Log.
DELETEEVENTS Deletes GoldenGate events from the registry.
ADDSERVICE Defines the GoldenGate Manager process as a Windows service
(Recommended) Manager can run by a local or domain account. However, when run
31
32. this way, Manager will stop when the user logs out. By using install, you can install
Manager as a Windows service so that it can be operated independently of user
connections and can be configured to start either manually or when the system starts.
You can configure the Manager service to run as the Local System account or as a
specific named account. The configuration of a service can be changed by using the
Services applet of the Windows Control Panel and changing the service Properties.
DELETESERVICE Removes the GoldenGate Manager service.
AUTOSTART Specifies that the service be started at system boot time (the default).
MANUALSTART Specifies that the service be started only at user request (with GGSCI
or the Control Panel Services applet).
USER Specifies a user name to logon as when executing Manager. If specified, user
name should include the domain name, a backward slash, and the user name.
PASSWORD Specifies the user’s password for logon purposes. This can be changed
using the Control Panel Services applet.
Prepare Environment: Installation - Multiple Manager Services
GoldenGate supports running multiple Manager services on
Windows
For two or more GoldenGate instances, or
GoldenGate with a Veridata C Agent (which uses a Manager)
Each Manager service must be assigned a unique name
Before installing the service, you can specify the name
Create a GLOBALS parameter file for each Manager
Specify the one-word name of the service using the
MGRSERVNAME <name> parameter
INSTALL ADDSERVICE
Reads the GLOBALS MGRSERVNAME for the service name
If no GLOBALS setting, uses default service name GGSMGR
A GLOBALS file stores parameters that relate to the GoldenGate instance as a whole,
as opposed to runtime parameters for a specific process. This file is referenced when
installing the Windows service, so that the correct name is registered.
33. Oracle GoldenGate Fundamentals Student Guide
Prepare Environment: Installation - UNIX, Linux or z/OS
Download .gz file to /ggs
gzip –d {filename}.tar.gz
tar -xvof {filename}.tar
GGSCI> CREATE SUBDIRS
For UNIX, z/OS, or Linux: Use the gzip and tar options appropriate for your
system. If you are installing GoldenGate into a cluster environment, make certain
that the GoldenGate binaries and files are installed on a file system that is available
to all cluster nodes. After installing GoldenGate, make certain to configure the
GoldenGate Manager process within the cluster application, as directed by the
vendor’s documentation, so that GoldenGate will fail over properly with the other
applications. The Manager process is the master control program for all GoldenGate
operations.
A GoldenGate instance is a single installation of GoldenGate.
Prepare Environment: Installation – NonStop SQL/MX
For a SQL/MX source, install Oracle GoldenGate on OSS running
on the NonStop source system:
Download .gz file to /ggs
gzip –d {filename}.tar.gz
tar -xvf {filename}.tar
GGSCI> CREATE SUBDIRS
Run the ggmxinstall script
For a SQL/MX target, install Oracle GoldenGate
Either on OSS running on the NonStop target system
(as described above)
Or on an intermediate Windows system(as described earlier)
The ggmxinstall script SQL compiles the Extract program and installs the VAMSERV
program in the NonStop Guardian space.
33
34. The command to run it is:
OSS> ggmxinstall /G/<Guardian vol>/<Guardian subvol>
where: <Guardian vol>/<Guardian subvol> is the destination NonStop volume and
subvolume in OSS format.
Prepare Environment: Installation - GoldenGate Directories
Directory Contents
dirchk GoldenGate checkpoint files
dirdat GoldenGate trail and extract files
dirdef Data definitions produced by DEFGEN and used to
translate heterogeneous data
dirpcs Process status files
dirprm Parameter files
dirrpt Process report files
dirsql SQL scripts
dirtmp Temporary storage for transactions that exceed
allocated memory
dirchk
Contains the checkpoint files created by Extract and Replicat processes, which store
current read and write positions to support data accuracy and fault tolerance. Written
in internal GoldenGate format. Do not edit these files.
The file name format is <group name><sequence number>.<ext> where <sequence
number> is a sequential number appended to aged files and <ext> is either cpe for
Extract checkpoint files or cpr for Replicat checkpoint files. Examples: ext1.cpe,
rep1.cpr
dirdat
The default location for GoldenGate trail files and extract files created by Extract
processes to store records of extracted data for further processing, either by the
Replicat process or another application or utility. Written in internal GoldenGate
format. Do not edit these files.
File name format is a user-defined two-character prefix followed by either a six-digit
sequence number (trail files) or the user-defined name of the associated Extract
process group (extract files). Examples: rt000001, finance
dirdef
35. Oracle GoldenGate Fundamentals Student Guide
The default location for data definitions files created by the DEFGEN utility to contain
source or target data definitions used in a heterogeneous synchronization
environment. Written in external ASCII.
File name format is a user-defined name specified in the DEFGEN parameter file.
These files may be edited to add definitions for newly created tables. If you are unsure
of how to edit a definitions file, contact technical support. Example: defs.dat
dirpcs
Default location for status files. File name format is <group>.<extension> where
<group> is the name of the group and <extension> is either pce (Extract), pcr
(Replicat), or pcm (Manager).
These files are only created while a process is running. The file shows the program
name, the process name, the port, and process ID that is running. Do not edit these
files. Examples: mgr.pcm, ext.pce
dirprm
The default location for GoldenGate parameter files created by GoldenGate users to
store run-time parameters for GoldenGate process groups or utilities. Written in
external ASCII format.
File name format is <group name/user-defined name>.prm or mgr.prm. These files
may be edited to change GoldenGate parameter values. They can be edited directly
from a text editor or by using the EDIT PARAMS command in GGSCI. Examples:
defgen.prm, finance.prm
dirrpt
The default location for process report files created by Extract, Replicat, and Manager
processes to report statistical information relating to a processing run. Written in
external ASCII format.
File name format is <group name><sequence number>.rpt where <sequence number>
is a sequential number appended to aged files. Do not edit these files. Examples:
fin2.rpt, mgr4.rpt
dirsql
The default location for SQL scripts.
dirtmp
The default location for storing large transactions when the size exceeds the allocated
memory size. Do not edit these files.
35
36. Oracle GoldenGate Documentation
Prepare Environment: Oracle GoldenGate Documentation
Quick Install Guide
Installation and Setup Guides (by database)
Administration Guide
Reference Guide
Troubleshooting and Tuning Guide
Note: You can download the documentation from
http://www.oracle.com/technology/documentation/index.html
Windows and UNIX platforms:
● Oracle GoldenGate Quick Install Guide: Describes the structure of the media pack
and where to find installation instructions.
● Oracle GoldenGate Installation and Setup Guides: There is an installation guide
and setup guide for each database that is supported by Oracle GoldenGate. These
include database-specific configuration information.
● Oracle GoldenGate Administration Guide: Introduces Oracle GoldenGate
components and explains how to plan for, configure, and implement Oracle GoldenGate
on the Windows and UNIX platforms.
● Oracle GoldenGate Reference Guide: Provides detailed information about Oracle
GoldenGate parameters, commands, and functions for the Windows and UNIX
platforms.
● Oracle GoldenGate Troubleshooting and Tuning Guide: Provides suggestions for
improving the performance of Oracle GoldenGate in different situations, and provides
solutions to common problems.
37. Oracle GoldenGate Fundamentals Student Guide
Configure and Start Manager
Prepare Environment: Manager - Overview
Performs system management and monitoring tasks
Starting GoldenGate processes
Starting dynamic Server Collector, Replicat, or GGSCI
processes
Error and lag reporting
GoldenGate trail management
Parameter file
mgr.prm file in GGS ./dirprm directory
Event information written to ggserr.log file
The Manager process performs system management and monitoring tasks on
Windows and Unix, including the following.
Starting Server Collector processes to collect data from remote Extract processes
Threshold reporting (for example, when Extract falls behind GGSLOG)
Purging trails
Purging of GGSLOG or GGSLOG_HISTORY data
Manager Parameters
Enter Manager parameters in the dirprm/mgr.prm file, under the GoldenGate
installation directory. If no mgr.prm file exists, default management parameters are
used.
Error and Informational Reporting
Manager reports critical and informational events to the ggserr.log file in the
GoldenGate installation directory.
37
38. Prepare Environment: Manager - Configuration
Create the parameter file using GGSCI
GGSCI> EDIT PARAM MGR
Start the Manager using GGSCI
GGSCI> START MGR
Note: To determine which port Manager is using
GGSCI> INFO MGR
Starting Manager
You must start Manager before most other configuration tasks performed in GGSCI.
Use either START MANAGER or START MGR.
On Windows systems, you can also start and stop Manager through the standard
Windows services control applet (in Control Panels).
Prepare Environment: Manager – Sample MGR Parameter File
PORT 7809
DYNAMICPORTLIST 8001, 8002, 9500–9520
PURGEOLDEXTRACTS /ggs/dirdat/aa*, USECHECKPOINTS
PURGEOLDEXTRACTS /ggs/dirdat/bb*, &
USECHECKPOINTS, MINKEEPDAYS 5
AUTOSTART ER *
AUTORESTART EXTRACT *, WAITMINUTES 2, RETRIES 5
LAGREPORTHOURS 1
LAGINFOMINUTES 3
LAGCRITICALMINUTES 5
This parameter file has the Manager listening on PORT 7809. Ports 8001, 8002, and
those in the range 9500 to 9520 will be assigned to the dynamic processes started by
Manager.
This manager process will recycle GoldenGate trails that match the file name of
/ggs/dirdat/aa* and /ggs/dirdat/bb*. It will only recycle the trail once all Extracts
39. Oracle GoldenGate Fundamentals Student Guide
and Replicats have a checkpoint beyond the file (USECHECKPOINTS), however bb* trails
will not be purged until there has been no activity for 5 days.
The manager will automatically start any Extract and Replicat process at startup and
will attempt to restart any Extract process that abends after waiting 2 minutes, but
only up to 5 attempts.
The manager will report lag information every hour, but only for processes that have
3 and 5 minutes of latency. The message will be flagged informational for lags of 3
minutes and critical for any process that has a lag greater than 5 minutes.
Generate Source Definitions (Heterogeneous Source/Target)
Prepare Environment: Source Definitions - Overview
The problem
Understanding source and target layouts across disparate
systems and databases
The solution – the DEFGEN utility program
DEFGEN produces a file containing layout definitions of the
source files and tables
This source definition file is used to interpret layouts for data
stored in GoldenGate trails
- At start up Replicat reads the definition file specified with the
SOURCEDEFS parameter
- Server Collector uses the –d argument to specify which definition file
to read at startup
Can also capture target definitions on target system and copy to
source system for Extract to use
The Problem
When Capturing, Transforming, and Delivering data across disparate systems and
databases, you must understand both the source and target layouts. Understanding
column names and data types is instrumental to GoldenGate’s data synchronization
functions.
The Solution - The DEFGEN Utility Program
The DEFGEN utility program produces a file containing a definition of the layouts of
the source files and tables. The output definitions are saved in an edit file and
transferred to all target systems in text format. Replicat and Collector read in the
definitions at process startup and use the information to interpret the data from the
GoldenGate trails.
When transformation services are required on the source system, Extract can use a
definition file containing the target, rather than source, layouts.
39
40. Note: The user should never modify the DEFGEN output.
Prepare Environment: Source Definitions – Run DEFGEN
DEFGEN is initiated from the command prompt:
defgen paramfile <paramfile> [ reportfile <reportfile> ]
Unix Example:
defgen paramfile /ggs/dirprm/defgen.prm
reportfile /ggs/dirrpt/defgen.rpt
Windows Example:
defgen paramfile c:ggsdirprmdefgen.prm
reportfile c:ggsdirrptdefgen.rpt
Definitions are saved to the file specified in the parameter file
This file needs to be transferred to the target system as a text file
Prepare Environment: Source Definitions - Sample DEFGEN Parameters
DEFSFILE /ggs/dirdef/source.def, PURGE
SOURCEDB mydb, USERID ggs, PASSWORD ggs
TABLE SALES.ACCOUNT;
TABLE SALES.PRODUCT;
Parameter Specifies
DEFSFILE The output definitions file location and name
SOURCEDB The database name (if needed)
USERID The user ID and password (if needed) to access the
database
TABLE The table(s) to be defined
41. Oracle GoldenGate Fundamentals Student Guide
Prepare the Source Database
Prepare Environment: Source Database – Overview
Set up the database to:
Ensure access by GoldenGate
Enable transaction logging
Note: the exact steps depend on the database
Database access
You need to assign a database user for each of the GoldenGate processes, unless the
database allows authentication at the operating system level. While not required,
GoldenGate recommends creating a user specifically for the GoldenGate application.
To ensure that processing can be monitored accurately, do not permit other users or
processes to operate as the GoldenGate user.
In general, the following permissions are necessary for the GoldenGate user:
• On the source system, the user must have permissions to read the data dictionary or
catalog tables.
• On the source system, the user must have permissions to select data against the
tables.
• On the target system, the user must have the same permissions as the GoldenGate
user on the source system plus additional privileges to perform DML on the target
tables.
41
42. Prepare Environment: Source Database
Oracle
Add minimal supplemental logging at database level
ADD TRANDATA to mark tables for replication
DB2
Enter DATA CAPTURE CHANGES at the column for LOB data type
ADD TRANDATA to mark tables for replication
Sybase
Set the secondary truncation point in the logs
ADD TRANDATA to mark tables for replication
NonStop SQL/MX
Special installation steps but no special database preparation
Oracle logs - On UNIX, GoldenGate reads the online logs by default, or the archived
logs if an online is not available. On the Windows platform, GoldenGate reads the
archived logs by default, or the online logs if an archive is not available. GoldenGate
recommends archive logging be enabled, and that you keep the archived logs on the
system for the longest time possible. This prevents the need to resynch data if the
online logs recycle before all data has been processed.
DB2 - In addition to enabling logging at a global level, each table to be captured must
be configured to capture data for logging purposes. This is accomplished by the DATA
CAPTURE CHANGE clause in the CREATE TABLE statement.
Sybase –To capture database operations for tables that you want to synchronize with
GoldenGate, each one must be marked for replication. This can be done through the
database, but GoldenGate recommends using ADD TRANDATA.
GoldenGate uses the secondary transaction log truncation point to identify
transaction log entries that have not been processed by the Extract process. The
secondary truncation point must be established prior to running the GoldenGate
Extract process. The GoldenGate process will manage the secondary truncation point
once it has been established.
NonStop SQL/MX
During the installation of SQL/MX, the script ggmxinstall sets a pointer to the VAM
that will work with Extract to capture changes from the TMF audit trail.
43. Oracle GoldenGate Fundamentals Student Guide
Prepare Environment: Source Database – SQL Server
To prepare the SQL Server source environment for GoldenGate:
Create the ODBC data source
GoldenGate connects to a SQL Server database through an ODBC
connection
Extract and Replicat require an established data source name (dsn)
Set up transaction logging
Log truncation and non-logged bulk copy must be turned off
The SQL Server database must be set to full recovery mode
Before GoldenGate processes are started, at least one full database
backup must be done
ADD TRANDATA to mark tables for replication
Supplemental logging is enabled using the GoldenGate command interface GGSCI.
The other preparation is done using Windows and SQL Server utilities.
ODBC Data Source Administrator is used to configure and define the ODBC
connection and to define the data source.
SQL Server Enterprise Manager is used to set full recovery mode and back up the
database.
SQL Server Query Analyzer is used to access the database to turn off log
truncation and non-logged bulk copy.
43
44. Prepare Environment: Source Database – SQL Server 2005
Additional considerations for SQL Server 2005 database:
Either install Microsoft Cumulative Update package 6 for SQL
Server 2005 Service Pack 2 (or later)
Set TRANLOGOPTIONS to MANAGESECONDARYTRUNCATIONPOINT
Or install SQL Server replication components
Create a distribution database
Add a replication publication
Set transaction retention to zero
Disable replication alerts
Log full before and after images (no compressed)
Set TRANLOGOPTIONS to
NOMANAGESECONDARYTRUNCATIONPOINT
The Distributor database must be used only for source databases to be replicated by
GoldenGate. One Distributor can be used for all of these databases. GoldenGate does
not depend on the Distributor database, so transaction retention can be set to zero.
Because GoldenGate does not depend on the Distributor database, but rather reads
the logs directly, the GoldenGate extraction process can process at its customary high
speed. For instructions on installing the replication component and creating a
Distributor database, see the GoldenGate for Windows and UNIX Administrator
Guide.
MANAGESECONDARYTRUNCATIONPOINT
Required TRANLOGOPTIONS parameters control whether or not GoldenGate maintains
the secondary truncation point. Use the MANAGESECONDARYTRUNCATIONPOINT option
if GoldenGate will not be running concurrently with SQL Server replication. Use the
NOMANAGESECONDARYTRUNCATIONPOINT option if GoldenGate will be running
concurrently with SQL Server replication. Allows SQL Server replication to manage
the secondary truncation point.
Primary Key Requirement - The requirement that all tables to be captured from
SQL Server 2005 source databases must have a primary key is a requirement of the
Microsoft replication component, which is utilized by GoldenGate as part of the log-
based capture process.
Connecting - Use SQL Server Management Studio for SQL Server 2005/2000 to
connect to a MS SQL Server 2005 or 2000 database. Use Enterprise Manager only for
MS SQL Server 2000.
45. Oracle GoldenGate Fundamentals Student Guide
Prepare Environment – Discussion Points
1. Where do you download Oracle GoldenGate software from?
2. What are the roles and responsibilities of the Manager process?
1. edelivery.oracle.com
2. Starting and stopping processes; monitoring processes; reporting lag, errors and
events; purging trail files.
45
46. GoldenGate Command Interface
GGSCI – Starting and Help
Start the command interface from the GoldenGate install directory:
Shell> cd <GoldenGate install location>
Shell> GGSCI
For Help Summary page:
GGSCI> HELP
For Help on a specific command:
GGSCI> HELP <command> <object>
For example: GGSCI> HELP ADD EXTRACT
Help returns overview, syntax and example
Golden Gate Software Command Interface (GGSCI) provides on-line help for all
commands. The following is an example of the information returned when you enter
HELP STATUS EXTRACT:
Use STATUS EXTRACT to determine whether or not Extract groups are running.
Syntax:
STATUS EXTRACT <group name>
[, TASKS]
[, ALLPROCESSES]
<group name> is the name of a group or a wildcard (*) to specify multiple groups.
ALLPROCESSES displays status of all Extract processes, including tasks.
TASKS displays status of all Extract tasks.
Examples:
STATUS EXTRACT FINANCE, STATUS EXTRACT FIN*
47. Oracle GoldenGate Fundamentals Student Guide
GGSCI Commands
MANAGER EXTRACT REPLICAT ER EXTTRAIL RMTTRAIL TRANDATA CHECKPOINT TRACE
TABLE TABLE
ADD X X X X X X X
ALTER X X X X
CLEANUP X X X
DELETE X X X X X X X X
INFO X X X X X X X X X
KILL X X X
LAG X X X
REFRESH X
SEND X X X X
START X X X X
STATS X X X
STATUS X X X X
STOP X X X X
Objects
Manager, Extract, Replicat GoldenGate processes.
ER Multiple Extract and Replicat processes.
EXTTRAIL Local trail.
RMTTRAIL Remote trail.
TRANDATA Transaction data (from transaction logs).
CHECKPOINTTABLE Checkpoint table (on target database).
TRACETABLE Oracle trace table (on target database).
Commands
ADD Creates an object or enables TRANDATA capture.
ALTER Changes the attributes of an object.
CLEANUP Deletes the run history of a process or removes records from a checkpoint
table.
DELETE Deletes an object or disables TRANDATA capture.
INFO Displays information about an object (status, etc).
KILL Forces a process to stop (no restart).
LAG Displays the lag between when a record is processed by the process and the
source record timestamp.
REFRESH Refreshes Manager parameters (except port number) without stopping
Manager.
SEND Sends commands to a running process.
START Starts a process.
STATS Displays statistics for one or more processes.
STATUS Displays whether a process is running.
STOP Stops a process gracefully.
47
48. GGSCI Commands (cont’d)
Commands
Parameters SET EDITOR, EDIT PARAMS, VIEW PARAMS
Database DBLOGIN, ENCRYPT PASSWORD, LIST TABLES
DDL DUMPDDL [SHOW]
Miscellaneous !command, CREATE SUBDIRS, FC, HELP, HISTORY, INFO ALL,
OBEY, SHELL, SHOW, VERSIONS, VIEW GGSEVT, VIEW REPORT
Parameter commands
SET EDITOR Changes the default text editor for the current GGSCI session from
Notepad or vi to any ASCII editor.
EDIT PARAMS Edits a parameter file.
VIEW PARAMS Displays the contents of a parameter file.
Database commands
DBLOGIN Establishes a database connection through GGSCI.
ENCRYPT PASSWORD Encrypts a database login password.
LIST TABLES List all tables in the database that match a wildcard string.
DDL commands
DUMPDDL Saves the GoldenGate DDL history table to file.
SHOW option displays the DDL information in standard output format.
Miscellaneous commands
!command Executes a previous GGSCI command without modification.
CREATE SUBDIRS Creates default directories within the GoldenGate home directory.
FC Edit a previously issued GGSCI command.
HELP Displays information about a GGSCI command.
HISTORY List the most recent GGSCI commands issued.
INFO ALL Displays the status and lag for all GoldenGate processes on a system.
OBEY Runs a file containing a list of GGSCI commands.
SHELL Run shell commands from within GGSCI.
SHOW Displays the GoldenGate environment.
VERSIONS Displays OS and database versions.
VIEW GGSEVT Displays the GoldenGate event/error log.
VIEW REPORT Displays a process report for Extract or Replicat.
49. Oracle GoldenGate Fundamentals Student Guide
GGSCI Examples
Start a Manager process
GGSCI> START MGR
Add an Extract group
GGSCI> ADD EXTRACT myext, TRANLOG, BEGIN NOW
Add a local trail
GGSCI> ADD EXTTRAIL /ggs/dirdat/rt, EXTRACT myext
Start an Extract group
GGSCI> START EXTRACT myext
Using Obey Files
You can use an Obey file to perform a reusable sequence of
commands
Save the commands in a text file, for example:
START MGR
ADD EXTRACT myext, TRANLOG, BEGIN NOW
ADD EXTTRAIL /ggs/dirdat/rt, EXTRACT myext
START EXTRACT myext
Then use the GGSCI OBEY command to run the file:
GGSCI> OBEY <obey filename>.oby
Note. An Obey file can have any file extension or none.
49
50. Running GoldenGate from the OS Shell
You can also start GoldenGate processes from the OS command shell when running
a batch job or initial load, for example:
Shell> cd <GoldenGate install location>
Shell> extract paramfile <filepath> reportfile <filepath>
[-p <port>]
Shell> replicat paramfile <filepath> reportfile <filepath>
This is especially useful to schedule GoldenGate batch jobs to run during off-peak
hours using a command-line capable scheduler
Manager must be running when you issue these commands.
<filepath> specifies the fully qualified name of the parameter and report files.
paramfile can be abbreviated to pf.
reportfile can be abbreviated to rf.
GoldenGate Commands – Discussion Points
1. What is GGSCI?
2. Where can you view the GoldenGate command syntax?
3. What is an Obey file and why would you use one?
1. GoldenGate Software Command Interface.
2. Help or Reference Guide.
3. A text file containing a sequence of GoldenGate commands; for easy re-use of
common command sequences.
51. Oracle GoldenGate Fundamentals Student Guide
Step 2. Change Capture
Step 2. Change Capture
1. Prepare the Environment
Source
Database
3. Initial Load (various methods)
Target
Database
Transaction
Log
Local Data
Extract Pump Remote Replicat
Trail Trail
4. Change Delivery
2. Change Capture
Change Capture - Extract Overview
Extract can be configured to:
Capture changed data from database logs
Distribute data from local trails to remote systems (data pump)
Capture data directly from source tables for initial data load
51
52. Change Capture - Tasks
On the source system:
Add a primary Extract (reading from source transaction logs)
with an associated parameter file
Optionally, add a local trail and a data pump Extract (reading
from the local trail) with an associated parameter file
Add a remote trail
Start the Extract(s)
To configure Extract to capture changes from transaction logs, perform the following
steps:
Set up a parameter file for Extract with the GGSCI EDIT PARAMS command.
Set up an initial Extract checkpoint into the logs with the GGSCI ADD EXTRACT
command.
Optionally, create a local trail using the GGSCI ADD EXTTRAIL command and a data
pump Extract (and parameter file) reading from the local trail.
Set up a remote trail using the GGSCI ADD RMTTRAIL command.
Start the Server Collector process on the target system or let the Manager start the
Server Collector dynamically.
Start Extract using the GGSCI START EXTRACT command. For example: GGSCI>
START EXTRACT FINANCE
GGSCI sends this request to the Manager process, which in turn starts Extract.
Manager monitors the Extract process and restarts it, when appropriate, if it goes
down.
Change Capture - ADD EXTRACT Command
Add the initial Extract checkpoint with the GGSCI command ADD EXTRACT:
ADD EXTRACT <group name>
, <data source>
, <starting point>
[, <processing options>]
The components of this command are discussed in subsequent slides.
53. Oracle GoldenGate Fundamentals Student Guide
Change Capture - ADD EXTRACT <data source>
<data source> Source (and when used)
SOURCEISTABLE Database table (initial data load)
TRANLOG Transaction log (change capture)
[<bsds name>] [DB2 z/OS]
EXTFILESOURCE <file name> Extract file (data pump)
EXTTRAILSOURCE <trail name> Trail (data pump)
SOURCEISTABLE Creates an Extract task that extracts entire records from the
database for an initial load. If SOURCEISTABLE is not specified, ADD EXTRACT
creates an online change-synchronization process, and one of the other data source
options must be specified. When using SOURCEISTABLE, do not specify service
options. Task parameters must be specified in the parameter file.
TRANLOG [<bsds name>]
Specifies the transaction log as the data source. Use this option for log-based
extraction. TRANLOG requires the BEGIN option.
Use the <bsds name> option for DB2 on a z/OS system to specify the BSDS
(Bootstrap Data Set) file name of the transaction log. Make certain that the BSDS
name you provide is the one for the DB2 instance to which the Extract process is
connected. GoldenGate does not perform any validations of the BSDS specification.
EXTFILESOURCE <file name>
Specifies an extract file as the data source. Use this option with a secondary Extract
group (data pump) that acts as an intermediary between a primary Extract group and
the target system. For <file name>, specify the fully qualified path name of the file,
for example c:ggsdirdatextfile.
EXTTRAILSOURCE <trail name>
Specifies a trail as the data source. Use this option with a secondary Extract group
(data pump) that acts as an intermediary between a primary Extract group and the
target system. For <trail name>, specify the fully qualified path name of the trail, for
example c:ggsdirdataa.
53
54. Change Capture - ADD EXTRACT <starting point>
<starting point> Database
BEGIN {NOW | <datetime> } Any
EXTSEQNO <seqno>, EXTRBA <relative byte address> Oracle,
SQL/MX
EXTRBA <relative byte address> DB2 z/OS
EOF | LSN <value> DB2 LUW
LSN <value> SQL Server,
Ingres
LOGNUM <log number>, LOGPOS <byte offset> c-tree
PAGE <data page>, ROW <row> Sybase
The starting point is indicated by one of the following:
BEGIN specifies when the Extract begins processing.
- For all databases except DB2 LUW, NOW specifies the time at which the ADD
EXTRACT command is issued.
- For DB2 LUW, NOW specifies the time at which START EXTRACT takes effect.
<datetime> specifies the start date and time in the format: yyyy-mm-dd
[hh:mi:[ss[.cccccc]]].
Several parameters specify the position with the log or trail to begin processing:
EXTSEQNO <seqno>, EXTRBA <relative byte address>
Valid for a primary Extract for Oracle and NonStop SQL/MX, and for a data pump
Extract. Specifies one of the following:
- sequence number of an Oracle redo log and RBA within that log at which to begin
capturing data.
- the NonStop SQL/MX TMF audit trail sequence number and relative byte address
within that file at which to begin capturing
data. Together these specify the location in the TMF Master Audit Trail (MAT).
- the file in a trail in which to begin capturing data (for a data pump). Specify the
sequence number, but not any zeroes used for padding. For example, if the trail file is
c:ggsdirdataa000026, you would specify EXTSEQNO 26. By default, processing
begins at the beginning of a trail unless this option is used.
Contact GoldenGate Technical Support before using this option.
EXTRBA <relative byte address>
Valid for DB2 on z/OS. Specifies the relative byte address within a transaction log at
which to begin capturing data.
55. Oracle GoldenGate Fundamentals Student Guide
EOF | LSN <value>
Valid for DB2 LUW. Specifies a start position in the transaction logs when Extract
starts.
EOF configures processing to start at the active LSN in the log files. The active LSN
is the position at the end of the log files that the next record will be written to. Any
active transactions will not be captured.
LSN <value> configures processing to start at an exact LSN if a valid log record
exists there. If one does not exist, Extract will abend. Note that, although Extract
might position to a given LSN, that LSN might not necessarily be the first one that
Extract will process. There are numerous record types in the log files that Extract
ignores, such as DB2 internal log records. Extract will report the actual starting LSN
to the Extract report file.
LSN <value>
Valid for SQL Server or Ingres. Specifies the LSN in a SQL Server or Ingres
transaction log at which to start capturing data. The LSN specified should exist in a
log backup or the online log.
- For SQL Server, an LSN is composed of three hexadecimal numbers separated by
colons. The first is the virtual log file number, the second is the segment number
within the virtual log, and the third is the entry number.
- For Ingres, an LSN is two, 4-byte unsigned integers, separated by a colon. For
example, to specify an LSN of 1206396546,43927 (as viewed in an Ingres utility), you
would enter 1206396546:43927.
- An alias for this option is EXTLSN.
LOGNUM <log number>, LOGPOS <byte offset>
Valid for c-tree. Specifies the location in a c-tree transaction log at which to start
capturing data.
<log number> is the number of the c-tree log file.
<byte offset> is the relative position from the beginning of the file (0 based).
PAGE <data page>, ROW <row>
Valid for Sybase. Specifies a data page and row that together define a start position in
a Sybase transaction log.
55
56. Change Capture - ADD EXTRACT <processing options>
<processing options> Specifies
DESC “<description>” Description of Extract group
THREADS <n> Number of redo threads when extracting from an
Oracle RAC clustered database
PARAMS <file name> Alternative parameter file name (fully qualified)
PASSTHRU Used only in Data Pumps. Passes the data through
without any transformation.
REPORT <file name> Alternative report file name (fully qualified)
Change Capture - ADD EXTRACT Examples
Create an Extract group named “finance” that extracts database changes
from the transaction logs. Start extracting with records generated at the
time when you add the Extract group.
ADD EXTRACT finance, TRANLOG, BEGIN NOW
Create an Extract group named “finance” that extracts database changes
from the transaction logs. Start extracting with records generated at 8:00 on
January 31, 2006.
ADD EXTRACT finance, TRANLOG, BEGIN 2006-01-31 08:00
Create a data-pump Extract group named “finance” that reads from the
GoldenGate trail c:ggsdirdatlt.
ADD EXTRACT finance, EXTTRAILSOURCE c:ggsdirdatlt
Create an initial-load Extract named “load”.
ADD EXTRACT load, SOURCEISTABLE