Insync10 goldengate

3,936 views

Published on

Oracle GoldenGate - Real-time Access to Real-time Information

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,936
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
448
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Insync10 goldengate

  1. 1. Oracle GoldenGate <br />Real-time Access to Real-time Information<br />Gavin Soorma<br />Senior Principal Consultant, <br /> OnCall DBA<br />
  2. 2. Introduction<br /><ul><li>Oracle buys GoldenGate -transaction closed September 3, 2009
  3. 3. A private company, founded in 1995 and based in San Francisco, CA
  4. 4. Recognized by industry analysts as a leader in real-time data integration
  5. 5. Fortune 500 customers across several key industries including Financial Services,Communications, Healthcare, Public Sector, Retail and Utilities
  6. 6. Extends Oracle’s data integration capabilities with the fastest, most scalable realtime Heterogeneous data integration solution
  7. 7. Another Oracle High Availability solution
  8. 8. Oracle Streams not widely accepted by the community – a strategic replacement?</li></li></ul><li>Real-Time <br />Data Integration<br />High Availability<br />GoldenGate Oracle Product Positioning<br />Continuous Low Latency Data Movement<br />Oracle <br />GoldenGate<br />Continuous <br />Operations<br />©2009 Oracle Corporation<br />3<br />
  9. 9. GoldenGate is Strategic to Oracle<br /><ul><li>Best-in-class leader in real-time data solutions
  10. 10. Complements existing Oracle products
  11. 11. Continuous Availability for heterogeneous systems
  12. 12. Real-time data access for Reporting & BI/EPM
  13. 13. Zero-downtime migrations / upgrades to Oracle Database and Applications
  14. 14. Over 500 customers with 4,000+ implementations across Fortune 500 companies: Financial Services, Communications, Healthcare, Public Sector, Retail & Utilities industries
  15. 15. Top 3 of 5 largest commercial banks
  16. 16. Top 3 of 3 busiest ATM networks
  17. 17. Top 7 of 10 financial data services companies
  18. 18. Top 4 of 5 telecommunications providers
  19. 19. Top 3 of 5 largest food & drug stores</li></li></ul><li>Some Success Stories<br /><ul><li>Supports 1.6TB of data movement per day to read-only servers
  20. 20. Source– Oracle 10g | Target – MySQL
  21. 21. Real-time data integration from Siebel CRM on Oracle </li></ul>to central TeradataWarehouse<br /><ul><li>GoldenGatemoves 150-200 million records per day with </li></ul>1.5 second latency.<br /><ul><li>5 way Active-Active configuration provides</li></ul>same data across worldwide plants<br /><ul><li>Zero downtime DB migration to new version using </li></ul>phased approach<br />
  22. 22. High Performance<br />Extensibility &<br />Flexibility<br />Reliability<br />What is Oracle GoldenGate?<br />Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of transactional dataacross heterogeneous environments in real-time<br />Why Oracle GoldenGate?<br />Non-intrusive, low-impact, sub-second latency<br />Open, modular architecture - Supports heterogeneous sources and targets<br />Maintains transactional integrity - Resilient against interruptions and failures<br />6<br />
  23. 23. Oracle GoldenGate Heterogeneity<br />Supports Applications running on ….<br />
  24. 24. Oracle GoldenGate Products<br /><ul><li>Oracle GoldenGate
  25. 25. GoldenGate Management Pack
  26. 26. Golden Gate Veridata
  27. 27. Available for download on Oracle E-Delivery web site.
  28. 28. Available on all major platforms for Oracle versions 10g and 11g </li></li></ul><li>Oracle GoldenGate Solutions<br />Enterprise-wide Solution for Real Time Data Needs<br />Disaster Recovery, Data Protection<br />Standby<br />(Open & Active)<br />Microsoft SQL Server<br />Zero Downtime Migration and Upgrades<br />Log Based, Real-Time Change Data Capture<br />Oracle<br />Operational Reporting<br />Reporting<br />Database<br />GoldenGate<br />ETL<br />EDW<br />ODS<br />ETL<br />Sybase<br />Heterogeneous Source Systems<br />Real-time BI<br />EDW<br />Query Offloading<br />Data Distribution<br />
  29. 29. Oracle GoldenGate Topologies<br />Bi-Directional<br />Live Standby or<br />Active-Active for HA<br />Peer-to-Peer<br />Load Balancing, <br />Multi-Master<br />Unidirectional<br />Query Offloading<br />Broadcast <br />Data Distribution<br />Integration/Consolidation<br />Data Warehouse<br />Cascading<br />Data Marts<br />
  30. 30. Integration with Data Warehousing<br />Integration via:<br />Staging tables<br />Flat files<br />Messaging<br />
  31. 31. Architecture<br /><ul><li>Manager process
  32. 32. Capture – Extract process
  33. 33. Delivery – Replicat process
  34. 34. Data Pump
  35. 35. Trail files
  36. 36. Checkpoints
  37. 37. Data definition files
  38. 38. Log files, Discard files</li></li></ul><li>GoldenGate Architecture<br />Network <br />(TCP/IP)<br />Source Database<br />Collector<br />Extract<br />Transaction Logs<br />Replicat<br />Data Pump<br />Remote Trail<br />Local Trail<br />Manager<br />Manager<br />Target Database<br />
  39. 39. Manager<br /><ul><li>Manager process runs on source and target
  40. 40. One manager per GoldenGate installation
  41. 41. Supports multiple Extract and Replicat processes
  42. 42. Starts other GoldenGate processes
  43. 43. Performs trail file management – purge old extracts
  44. 44. Responsible for Event and Error reporting </li></li></ul><li>Extract<br /><ul><li>Extract process runs on the source system
  45. 45. Capture mechanism of GoldenGate
  46. 46. Configure extract for
  47. 47. Initial Data Load
  48. 48. Online Change Synchronization
  49. 49. Multiple Extract processes can operate on multiple database objects</li></li></ul><li>Data Pump<br /><ul><li>Secondary extract group within source GoldenGate configuration
  50. 50. Optional
  51. 51. When not used, Extract sends data over network to remote trail on destination
  52. 52. When used, Extract process writes to local trail. Data Pump reads this trail and writes to the remote trail
  53. 53. Data pump isolates the primary extract process from TCP/IP activity
  54. 54. Performs data filtering, mapping and conversion
  55. 55. Protects against network and target failures</li></li></ul><li>Replicat<br /><ul><li>Runs on target system
  56. 56. Reads and extracts data changes (and DDL changes when configured)
  57. 57. Configured for initial load or online change synchronization.
  58. 58. Multiple replicat processes to increase throughput
  59. 59. Configured with a apply delay if required</li></li></ul><li>Trail Files<br /><ul><li>Series of files stored on disk to enable continuous extraction and propagation of changes
  60. 60. Can exist on the source (local trail), target (remote trail) or on intermediary systems
  61. 61. By default 10 MB in size
  62. 62. Files are named at configuration time with a two letter identifier and are appended with an unique 6 digit serial number
  63. 63. GoldenGate writes to trail files in Universal Data Format – a proprietary format
  64. 64. This proprietary format enables the heterogeneous data integration
  65. 65. Checkpoints provide known position in trail from where to start in case of unexpected failure
  66. 66. Records of checkpoints maintained in file on disk or database checkpoint table </li></li></ul><li>How Oracle GoldenGate Works<br />Capture<br />Deliver<br />Source Trail<br />Target Trail<br />Deliver<br />Capture<br />Source Trail<br />Target Trail<br />Bi-directional<br />Capture: Committed changes are captured (and can be filtered) as they occur by reading the transaction logs.<br />Trail files: Universal data format enables heterogeneity.<br />Route: No distance constraints via TCP/IP. Compression & encryption. <br />Delivery: Applies transactional data with guaranteed integrity.<br />Redo/archived log<br />LAN / WAN / <br />Internet<br />Source Oracle Database <br />Target<br />Database<br />
  67. 67. GoldenGate and Oracle Data Source<br />Capture<br />Commit Ordered Trail<br /><ul><li>Support for Oracle RAC
  68. 68. Support for ASM
  69. 69. Support for index-organized tables (IOTs) with overflow
  70. 70. Support for clustered tables
  71. 71. Support for object tables
  72. 72. Support for object types (UDTs)
  73. 73. Support for DDL operations
  74. 74. Archived log only (ALO) mode
  75. 75. Off-platform capture (LOGSOURCE)
  76. 76. Multi-threaded capture</li></ul>Capture<br />Online Log<br />Archived Log<br />OCI/Oracle Net<br />
  77. 77. Capture<br />Commit Ordered Trail<br />GoldenGate and Microsoft SQL ServerMicrosoft SQL Server Capture <br /><ul><li>Ability to capture from native backups
  78. 78. Coexistence with SQL Server Replication
  79. 79. SQL Server Replication components are not required
  80. 80. Support tables with no unique key (not supported by SQL Server Replication)
  81. 81. Support for computed columns
  82. 82. Support for identity columns</li></ul>Capture<br />Transaction Log<br />Backups<br />ODBC<br />
  83. 83. Zero-Downtime Upgrades, Migrations and Maintenance<br />Eliminate “planned downtime” during:<br /><ul><li>Database Upgrades: Oracle 8i or 9i  11g
  84. 84. Database Migrations (e.g. DB2 to Oracle)
  85. 85. Database Server Upgrades
  86. 86. Database Maintenance (e.g. de-fragmentation, table re-orgs, de-chaining rows, etc.)
  87. 87. Application Upgrades (validated for Siebel CRM)</li></li></ul><li>Oracle 8i or 9i to 10/11g Upgrades/Migrations<br /><ul><li>What GoldenGate TDM offers:
  88. 88. Minimal impact and continuous availability during the project
  89. 89. Technology solution for eliminating database downtime
  90. 90. Improved capabilities for data integrity, accuracy
  91. 91. Fail-back solution in the event of unexpected issues/downtime
  92. 92. Proven – with customers in production</li></li></ul><li>Cross Platform Migration<br />1<br />2<br />3,4<br />9i Clone  10g<br />Solaris<br />9i Solaris<br />Start GoldenGate TDM’s Capture process<br />Set up Clone database, then Upgrade to 10g <br />Cross platform transportable tablespaces metadata export<br />Use a full database NOROWS export (Views, Packages, etc)<br />
  93. 93. Cross Platform Migration<br />1<br />2<br />3,4<br />5<br />6,7<br />9i Clone  10g<br />Solaris<br />9i Solaris<br />10g<br />Linux<br />Start GoldenGate TDM’s Capture process<br />Set up Clone database, then Upgrade to 10g <br />Cross platform transportable tablespaces metadata export<br />Use a full database NOROWS export (Views, Packages, etc)<br />Set up a new 10g vanilla target <br />Cross platform transportable tablespaces metadata import<br />Full import with IGNORE option<br />
  94. 94. Cross Platform Migration<br />1<br />2<br />5<br />8<br />3,4<br />6,7<br />9i Clone  10g<br />Solaris<br />9i Solaris<br />10g<br />Linux<br />Start GoldenGate TDM’s Capture process<br />Set up Clone database, then Upgrade to 10g <br />Cross platform transportable tablespaces metadata export<br />Use a full database NOROWS export (Views, Packages, etc)<br />Set up a new 10g vanilla target <br />Cross platform transportable tablespaces metadata import<br />Full import with IGNORE option<br />Start GoldenGate TDM Apply process at target <br />**SWITCHOVER** (not depicted)<br />
  95. 95. Cross Platform Migration<br />1<br />5<br />8<br />10<br />11<br />2<br />3,4<br />6,7<br />9i Clone  10g<br />Solaris<br />9i Solaris<br />Failback contingency<br />10g<br />Linux<br />Start GoldenGate TDM’s Capture process<br />Set up Clone database, then Upgrade to 10g <br />Cross platform transportable tablespaces metadata export<br />Use a full database NOROWS export (Views, Packages, etc)<br />Set up a new 10g vanilla target <br />Cross platform transportable tablespaces metadata import<br />Full import with IGNORE option<br />Start GoldenGate TDM Apply process at target<br />Verify data (optional)<br />**SWITCHOVER** (not depicted)<br />Start TDM Capture at target (new source)<br />Start TDM Apply at target (old source)<br />
  96. 96. GoldenGate Management Pack<br /> Graphical interface for designing, configuring, managing, monitoring, and reporting on the Oracle GoldenGate components.<br /><ul><li>Golden Gate Director Server
  97. 97. GoldenGate Director Client
  98. 98. GoldenGate Web Director</li></li></ul><li>GoldenGateVeridata™<br />“GoldenGate Veridata is a high-speed, low impact data comparison solution that identifies and reports data discrepancies between two databases, without interrupting those systems or the business processes they support.”<br /><ul><li>High-speed data comparison solution
  99. 99. Identifies and reports on data discrepancies between heterogeneous databases with minimal impact on infrastructure
  100. 100. Companies can audit and verify large volumes of data across different business applications with certainty
  101. 101. It enables users to research root causes of the discrepancies and determine how to remedy the problem and avoid future risk. </li></li></ul><li>GoldenGateVeridata™<br />
  102. 102. GoldenGate Veridata: How it Works<br />The user chooses tables or files on the source and target databases<br />The comparison is initiated from the GUI, command line or batch<br />As the databases continue to change, GoldenGate Veridata reports: <br />Persistent discrepancies <br />In-flight data discrepancies (user configurable)<br />
  103. 103. GoldenGate Director<br />
  104. 104. GoldenGate Director<br />
  105. 105. GoldenGate Director Web<br />
  106. 106. GoldenGate Demo<br />
  107. 107. Create the Manager, Extract and Replicat Process<br />./ggsci<br />GGSCI (linux01.oncalldba.com) 1> edit params mgr<br />PORT 7809<br />USERID ggs_owner, PASSWORD ggs_owner<br />Source – Extract Process<br />GGSCI (linux01.oncalldba.com) 1> ADD EXTRACT ext1, TRANLOG, BEGIN NOW<br />EXTRACT added.<br />GGSCI (linux01.oncalldba.com) 2> ADD RMTTRAIL /home/oracle/goldengate/dirdat/rt, EXTRACT ext1<br />RMTTRAIL added.<br />Target – Replicat Process<br />GGSCI (linux03.oncalldba.com) 1> ADD REPLICAT rep1, EXTTRAIL /home/oracle/goldengate/dirdat/rt<br />REPLICAT added.<br />
  108. 108. Start Manager (Source)<br />[oracle@linux01 goldengate]$ ./ggsci<br />Oracle GoldenGate Command Interpreter for Oracle<br />Version 10.4.0.19 Build 002<br />Linux, x86, 32bit (optimized), Oracle 10 on Sep 17 2009 23:49:42<br />Copyright (C) 1995, 2009, Oracle and/or its affiliates. All rights reserved.<br />GGSCI (linux01.oncalldba.com) 1> start manager<br />Manager started.<br />GGSCI (linux01.oncalldba.com) 2> info all<br />Program Status Group Lag Time Since Chkpt<br />MANAGER RUNNING<br />
  109. 109. Start Manager (Target)<br />[oracle@linux03 goldengate]$ ./ggsci<br />Oracle GoldenGate Command Interpreter for Oracle<br />Version 10.4.0.19 Build 002<br />Linux, x86, 32bit (optimized), Oracle 11 on Sep 29 2009 08:50:50<br />Copyright (C) 1995, 2009, Oracle and/or its affiliates. All rights reserved.<br />GGSCI (linux03.oncalldba.com) 1> start manager<br />Manager started.<br />GGSCI (linux03.oncalldba.com) 2> info all<br />Program Status Group Lag Time Since Chkpt<br />MANAGER RUNNING<br />
  110. 110. Source – Start Extract Process<br />GGSCI (linux01.oncalldba.com) 4> view params ext1<br />EXTRACT ext1<br />USERID ggs_owner@oncalldb.world,passwordggs_owner<br />RMTHOST linux03, MGRPORT 7809<br />RMTTRAIL /home/oracle/goldengate/dirdat/rt<br />DDL INCLUDE ALL<br />TABLE ggs_owner.myobjects;<br />;<br />GGSCI (linux01.oncalldba.com) 5> start extract ext1<br />Sending START request to MANAGER ...<br />EXTRACT EXT1 starting<br />GGSCI (linux01.oncalldba.com) 6> info extract ext1<br />EXTRACT EXT1 Last Started 2010-08-13 13:12 Status RUNNING<br />Checkpoint Lag 03:00:32 (updated 00:00:00 ago)<br />Log Read Checkpoint Oracle Redo Logs<br /> 2010-08-13 10:12:23 Seqno 4, RBA 19823616<br />
  111. 111. Target – Start Replicat Process<br />GGSCI (linux03.oncalldba.com) 3> view params rep1<br />REPLICAT rep1<br />ASSUMETARGETDEFS<br />USERID ggs_owner, PASSWORD ggs_owner<br />discardfile /home/oracle/goldengate/dirout/discard.txt,append<br />DDL INCLUDE MAPPED<br />MAP ggs_owner.myobjects, TARGET ggs_owner.myobjects;<br />GGSCI (linux03.oncalldba.com) 4> start replicat rep1<br />Sending START request to MANAGER ...<br />REPLICAT REP1 starting<br />GGSCI (linux03.oncalldba.com) 5> info replicat rep1<br />REPLICAT REP1 Last Started 2010-08-13 13:15 Status RUNNING<br />Checkpoint Lag 00:00:00 (updated 00:00:04 ago)<br />Log Read Checkpoint File /home/oracle/goldengate/dirdat/rt000010<br /> 2010-08-13 13:12:55.105247 RBA 1030<br />
  112. 112. Test DDL Replication<br />Create a table and index on source database<br />[oracle@linux01 ~]$ sqlplusggs_owner/ggs_owner<br />SQL*Plus: Release 10.2.0.4.0 - Production on Fri Aug 13 13:09:38 2010<br />Copyright (c) 1982, 2007, Oracle. All Rights Reserved.<br />Connected to:<br />Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production<br />With the Partitioning, OLAP, Data Mining and Real Application Testing options<br />SQL> create table myobjects as select * from dba_objects where 1=2;<br />Table created.<br />SQL> alter table myobjects add constraint pk_myobjects<br /> 2 primary key (object_id);<br />Table altered.<br />
  113. 113. Test DDL Replication<br />On Target database MYOBJECTS table and index created<br />SQL> descmyobjects<br /> Name Null? Type<br /> ----------------------------------------- -------- ----------------------------<br /> OWNER VARCHAR2(30)<br /> OBJECT_NAME VARCHAR2(128)<br /> SUBOBJECT_NAME VARCHAR2(30)<br /> OBJECT_ID NOT NULL NUMBER<br /> DATA_OBJECT_ID NUMBER<br /> OBJECT_TYPE VARCHAR2(19)<br /> CREATED DATE<br /> LAST_DDL_TIME DATE<br /> TIMESTAMP VARCHAR2(19)<br /> STATUS VARCHAR2(7)<br /> TEMPORARY VARCHAR2(1)<br />SQL> select index_name from user_indexes where table_name='MYOBJECTS';<br />INDEX_NAME<br />------------------------------<br />PK_MYOBJECTS<br />
  114. 114. Test Online Change Synchronization<br />INSERT rows on source tables<br />SQL> !date<br />Fri Aug 13 13:24:14 WST 2010<br />SQL> insert into myobjects select * from dba_objects where owner='SYSMAN';<br />1341 rows created.<br />SQL> commit;<br />Commit complete.<br />SQL> !date<br />Fri Aug 13 13:27:01 WST 2010<br />SQL> insert into myobjects select * from dba_objects where owner='SYS';<br />22937 rows created.<br />SQL> commit;<br />Commit complete.<br />
  115. 115. Test Online Change Synchronization<br />SQL> !date<br />Fri Aug 13 13:24:44 WST 2010<br />SQL> select count(*) from myobjects;<br /> COUNT(*)<br />----------<br /> 1341<br />SQL> !date<br />Fri Aug 13 13:27:14 WST 2010<br />SQL> select count(*) from myobjects;<br /> COUNT(*)<br />----------<br /> 24278<br />Check target tables to see that all the rows have been inserted<br />
  116. 116. Monitor Replicat Progress<br />GGSCI (linux03.oncalldba.com) 8> stats replicat rep1<br />Start of Statistics at 2010-08-13 13:35:39.<br />DDL replication statistics:<br />*** Total statistics since replicat started ***<br /> Operations 2.00<br /> Mapped operations 2.00<br /> Unmapped operations 0.00<br /> Other operations 0.00<br /> Excluded operations 0.00<br /> Errors 0.00<br /> Retried errors 0.00<br /> Discarded errors 0.00<br /> Ignored errors 0.00<br />Replicating from GGS_OWNER.MYOBJECTS to GGS_OWNER.MYOBJECTS:<br />*** Total statistics since 2010-08-13 13:24:33 ***<br />Total inserts 24278.00<br /> Total updates 0.00<br /> Total deletes 0.00<br /> Total discards 0.00<br /> Total operations 24278.00<br />
  117. 117. Cleanup – some more DDL Replication<br />SOURCE<br />SQL> truncate table myobjects;<br />Table truncated.<br />TARGET <br />SQL> select count(*) from myobjects;<br /> COUNT(*)<br />----------<br /> 0<br />SOURCE<br />SQL> drop table myobjects;<br />Table dropped.<br />TARGET<br />SQL> select count(*) from myobjects;<br />select count(*) from myobjects<br /> *<br />ERROR at line 1:<br />ORA-00942: table or view does not exist<br />
  118. 118. Q<br />&<br />A<br />gavin.soorma@oncalldba.com.au<br />http://www.gavinsoorma.com<br />http://www.oncalldba.com.au<br />

×