13. GoldenGate Microservices Architecture
13
Trail Files
Administration
Service
Metrics
Service
Service
Manager
>REST
Parallel
Replicat
Integrated Remote
Capture
network
DBA/Ops Proxy/Reverse Proxy API Automation
Micro-services Mid-Tier
* Oracle recommends to use Integrated Remote Capture, Micro-Services and Parallel Replicat
Distribution
Service
Receiver
Service
Shared
Storage
14. GoldenGate for Oracle
19.1 New Features
14
Performance and Scalability Improvements
Improved performance and scalability for Parallel Replicat.
Remote integrated capture performance tolerates high latency networks.
Security and Manageability Enhancements for Microservices
Integration with Key Management Systems, Support for DMZ environments.
Defaults changed to TLS1.2, Digest Auth, Strong Password Verifier
Managed profiles for AutoStart, AutoRestart, and Key Management.
Ease of use
Simplified software upgrade procedures. Enhanced heartbeats to simplify system recovery.
Simplify mid-tier & cloud deployments with remote cross-endian Integrated Extract.
Better smart defaults that eliminate need to set parameters for common use cases.
Support for Oracle Database 19.1
Long term support release. Final patchset update for Oracle GoldenGate 12.3 release.
OCI Marketplace Solution
GoldenGate 19.1 available from OCI Marketplace in both Classic and Microservices editions.
Cross endian remote Integrated Extract allows remote capture of all on-premises Oracle databases for easier migration to the cloud.
15. Run Anywhere - Remote Capture
15
*remote transaction capture with zero footprint on Source DB hosts
*heterogeneous support for Mainframe, SQL Server and Postgres *roadmap
*endian normalization inline to GG, Oracle DB 11.2.0.4 and higher
Windows
WAN
GG Mid-Tier Host
Remote Connect
Any GoldenGate
target system
16. Extreme Performance
16
Performance Data Points:
*Note: your experiences will vary
Oracle DB:
-- 4TB REDO/hr, raw performance
seen in Oracle lab setting
-- 1.5TB REDO/hr, actual perf at
large banking customer
-- 1m Tx/second, actual perf at
Fortune 200 manufacturing cust
Non-Oracle DB:
-- 4b Tx/day on DB2 active-active
at Fortune 200 retailer
-- 55k Tx/second on Mainframe to
data lake at APAC mega-bank
• Performance improvements in *every* release
• No other vendor can compare on Oracle Database
replication performance
– Purpose-built APIs for extract and apply
– Integrated (same DB code base) and highly parallel (external
dependency checking)
– Remote capture is network-optimized so that any network faster
than 100ms latency performs similar to local extracts
• Non-Oracle DB performance highly competitive, regularly
win bake-offs w/DB2, Mainframe, SQL Server and other non-
Oracle databases
• Eg: For GoldenGate Big Data, have worked directly with
LinkedIn and Confluent to optimize GGàKafka performance
17. GoldenGate for Big Data
19.1 New Features
17
Transactional Support for MongoDB
MongoDB 4.0+ onwards supports transactions
Stage and Merge to Cloud Data warehouses*
AWS S3 to Redshift
Improvements
Kafka Interceptor support in Kafka Connect Handler *
Before and After Images for Delimited Text Formatter
MAPINVISIBLECOLUMNS functionality for Big Data Targets
Routing key for Elasticsearch shards
Command Event Handler
New event handler to Run external commands after writing a file locally.
Improves performance to DWH loads like Netezza, Greenplum etc
OCI Marketplace Solution
GoldenGate Big Data 19.1 available from OCI Marketplace
Heartbeat
GoldenGate Heartbeat feature
UpToDate Certifications
Hadoop 3.x, HBase 2.x, Elasticsearch 7.x, 6.6, DSE 6.x, Azure DataLake Gen 2, Confluent 5.x, NoSQL 18.1
.
18. • A high-speed, low impact data
comparison and repair solution
•identifies and reports data discrepancies
between heterogeneous databases
•without interrupting their availability
•Certified for integration with Oracle DBCS
• Benefits:
•Reduce financial/legal risk exposure
•Speed and simplify IT work in comparing
data sources
•No disruption to business systems
•Confident decision-making and
reporting
Oracle GoldenGate Veridata
Oracle GoldenGate Veridata
19. • Oracle Preferred monitoring tool,
covers entire Oracle stack – Database,
Storage, FMW etc
• For customers, who prefers to drill into
DB sessions, OS stats, etc, apart from
monitoring GoldenGate process
• Certified for integration with Oracle
DBCS
• Certified with latest OEM 13c
• Certified with GoldenGate
Microservices
Management Pack for Oracle GoldenGate
Enterprise Manager Plug-in
21. 21
Why Stream Analytics?
Streaming Data Automated Decisions
Batch ETL
Decision
Maker Action
DecisionReports
Traditional
AnalyticsData Warehouse
Data Lake
Queries
Event Producers
Business Data
Stream Analytics
Stream Analytics
Batch Analytics
Elapsed time
23. RESTful API for Producers and Subscribers (events are pushed)
Databus
Raw Data
Topics
Schema Event
Topics
Stream
Processing
(ETL)
Prepared
Data Topics
Master Data
Topics
Stream
Processing
(ETL)
10,000’s 100’s 10’s
Lambda/Kappa: Large Batch Mode is Impractical
23
App
DB
App
DB
App
DB
ERP
Operational
Data Store
EDW
Staging Prod
ETL
ETL
ETL
ETL
ETL
Mart
Mart
Mart
ETL
Enterprise BI
Mart
Mart
Mart
ETL
Departmental BIDiscovery
App
DB
App
DB
App
DB
ERP
WebApps
Mobile
EDW
NoSQL
Hadoop / Spark Marts Marts Marts Mobile
Less Governed --------------------------------------------------------------- More Governed
Enterprise BI Departmental BIDiscovery WebApps
Old: Hub and Spoke
• Invasive on Sources
• 24hr+ Latency
• Limited views on data
• Heavy IT process overhead
New: Databus/Kappa
• Low impact on Sources
• ~1 Second Latency
• Access raw or prepared data
• Supports more Agile process
GoldenGate
MDM
Hub
• ODS & ETL Hubs
• EDW/Mart Hubs
• MDM/RDM Hubs
• Big Data Lake Hubs
• Pub/Sub for Staging
• ETL in Pipelines
• Analytics/CEP in Stream
• Static data in “Lake”
26. Oracle Stream Analytics Architecture
26Confidential – Oracle Internal/Restricted/Highly Restricted
OSA Components
OSA Web Application
(Embedded Jetty App Server)
Reference
Input
Oracle DB
Coherence
Oracle DB or MySQL
(Used for Metadata Store)
Apache Hadoop
or Spark
Apache Kafka
Kafka
JMS
GoldenGate
Streaming
Input
Kafka
JMS
REST
Streaming
Output
Analytics
Output
Apache Druid
Oracle DB
File
)In-Memory Cache
(Used for reference data)
27. 6 Differentiators for Oracle Stream Analytics
27
CQL
Interactive
Designer UI
Rich Set of Streaming
Patterns
Predictive Analysis and
Machine Learning
Location and Geospatial
Analysis
Integrated CDC with
Oracle GoldenGate
Robustness, Speed, and
Scalability
32. End State Vision for GoldenGate Offerings
On Premise
World-class Enterprise Software for
Any Data Center
Customer-managed Cloud
Oracle-provided IaaS, World-class
Enterprise Software
Oracle-managed Cloud
Native, Fully-managed Cloud PaaS
Capabilities
Data Replication
àMulti-Active DBs for High Availability
àOracle/Non-Oracle Data Capture
àStreaming Ingest to Data Lakes
ü
- GoldenGate is the most
successful enterprise
integration tool in history
ü
- GoldenGate is provided
with 5 distinct listings
inside Oracle Cloud
Marketplace; providing
automation for 5-minute
provisioning times
ü
- GoldenGate Service will be
a fully managed solution for
customers who do not want
the overhead of admin for
their own Patching,
Upgrades and basic Ops
- All core GoldenGate use
cases will be covered
- All Stream Processing use
cases will come from the
same service (OSA use
cases)
Stream Data Processing
àDataOps on Real-time Event Streams
àETL for Data in Motion (non-batch)
àEvent Correlation across Moving Time Windows
àSpatial and Time-Series Analysis
àSimple plug-in for Machine Learning Scoring
Models
ü
- Oracle Stream Analytics has
10yr history and over 70
patents on core stream
processing technology
ü
- Stream Analytics is
provided in OCI
Marketplace for fast
provisioning and a
simplified Image that is pre-
configured with Kafka,
Spark & MySQL… it is a
turn-key solution
Active investments in leadership across full range of Data Integration capabilities:
35. Additional Information
GoldenGate
OGG-Product Home Page
OGG 19c- Data Sheet
OGG 19c-BD Data Sheet
OGG-Certification Matrix
OGG-Product Download
OGG-BD Demo Image – Big Data Lite VM
OGG-BD 19.1 Documentation
Oracle Stream Analytics
OSA Product Home Page
OSA Data Sheet
OSA Certification Matrix
OSA Product Download
OSA 19.1 Documentation
35
38. About Petroleum Development Oman
Petroleum Development Oman (PDO) is the
leading exploration and production company
in the Sultanate of Oman.
We deliver the majority of the country’s
crude oil production and natural gas supply,
but above all we focus on delivering
excellence, growth and sustainable value
creation within and well beyond our
industry.
39. Background
Petroleum Development Oman (PDO):
PDO Dimensions
•Number of producing oil fields: 209
•Number of producing gas fields: 55
•Number of active wells: 8,000+
•Number of staff: 8,500+
IT Dimensions
• Number of DC: 3 (Main, HA & DR)
• Number of voice lines: 16,000
• Number of Servers: 2,000
• Storage Size: 10 Petabyte
• Number of Applications: 565
• Number of Telecom Rooms:10
40. Health and
Safety
PDO Aspirations
“Comprehensive, robust and reliable technical Data is one of the critical pillars for managing a complex company like PDO; without a healthy Data set it is not
possible to deliver accurate field development plans and/or execute safe and effective operations.”
Raoul Restucci
PDO Managing Director
• Maintain continuous oversight on potential operational risks through access to real time information on hazards, in
order to lower safety incidents and harm to assets and people.
• Accelerate the growth by improving operational performance, reducing the TCO, eliminating operational wastes and
redundancies, increasing throughput from existing assets and maintain equipment integrity.
• Reduce the downtime by improving efficiencies and asset productivity
• Reduce the WE overhead activity
• Reduce the e2e new oil / gas delivery time.
Improved
Production
• Increased collaboration and Integration across businesses and seamless transparent transfer of information across PDO to
take real-time decisions to Improve people productivity.
Operational
Excellence
Integrated
Business
PDO’s high level strategy consists of the following areas of improvements and objectives:
40
41. PDO Data Management Strategy
• Improve decision-making by providing relevant, reliable and up-to-date data to PDO
personnel with Data Profiling and Data Quality technologies
• Batch and real time Data Integration with ELT and Replication technologies
• Optimise Business Processes by introducing Data Governance
• Provide single version of the truth with the support of Master Data Management
technologies
• Enable digitalisation initiatives ML and AI by introducing Data Lake and Advanced
Analytics technologies
42. 42
Data and Analytics as Core Capability
Digitalisation of Support Functions
Mobile Devices IoT Platforms Cloud Computing
Location Detection
Technologies
3D Printing
Digital Oilfield Digital Twin Plant Smart Sensors
Advanced
Human-Machine
Interfaces
IT/Technology
Architecture
Digital Capabilities and
Culture
Partnership and
Alliances
Information Risk
Management
Key Enablers
Data Management,
Governance, Processes
Digitalisation Journey
43. Corporate Information Factory
The PDO Corporate Information Factory programme is an enterprise-wide initiative designed to address and solve the challenges faced by the
business around Data. (Data Governance, Master Data Management, Data Lake, Data Integration,Advanced Analytics Platform, Data Quality)
Corporate Information Factory (CIF)
Business Intelligence
Services
Integration
DataGovernance
DataSecurity
KPI
Hierarchy
Dashboard
and Visual
Self Service
Exception
Based Alerts
Advance
Analytics
Data
Discovery
Analytics
API Manager
Semantic Data
Layer
Batch Transfer Big Data Analytics Automation
Data Platform
Low Latency Data
Store
Enterprise
DW
Data Lake
Data
Discovery
Data Platform
Master Data
Management
Source Systems Production
Applications
Enterprise
Applications
Engineering
Applications
Operation
Applications
External
Applications
Subsurface
Applications
43
Streaming
44. Data Management Journey
ETL Mappings
Implementation
with ODI
Database
Consolidation
Exadata
Enterprise Data
Quality (EDQ)
Implementation
Data
Governance
Data
Integration
Self Service
Analytics
Data Management Road Map
Metadata
Management
Data Lake /
Data Streaming
Advanced
Analytics
Master Data
Management
Not Started
Completed
In Progress
Digitalisation
