SlideShare a Scribd company logo
Microservices Patterns with GoldenGate:
Deep-dive into Microservices Patterns with Replication and Stream Analytics
O R A C L E D E V E L O P M E N T , M A Y - 2 0 2 0
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Today’s Agenda
Microservices and the Challenge of Data
Data Mesh Concept, How GoldenGate Fits
Microservice Patterns with GoldenGate
Anyone Remember “Complex Event Processing”?
Copyright © 2020 Oracle and/or its affiliates.
1
2
3
4
Microservices are Good!
4
Service Mesh Revolution
Emergence and widespread use of microservices have
directly led to revolution in DevOps, massive uptake in
Kubernetes and by 2020 the Service Mesh revolution
Key Benefits:
• Decomposition, of monolithic architecture
• Modularity, smaller services and improved
speed of initial development
• Independence, loosely-coupled systems that can
be created using different languages or data
• With the loose coupling also comes much
greater flexibility around deployment and
upgrades, eliminating complex dependencies
• Flexibility at Scale, deployments may start small,
run locally and later scale very wide, running
across multi-cloud environments and containers
• Sidecar pattern for “Mesh” frameworks are
further revolutionizing microservices
Chris Richardson: https://www.nginx.com/blog/introduction-to-microservices/
Dogma of Received Wisdom at the Data Tier
5
https://docs.microsoft.com/en-us/dotnet/architecture/microservices/architect-
microservice-container-applications/data-sovereignty-per-microservice
• “DB per Service” is a founding concept of Microservices
Architectures
• Some architects are more dogmatic than others:
• “All microservices must have an isolated data store” vs. “Some
microservices may share a data store”
• At scale, usually means that microservice application
must accept “eventual consistency” for data tier
Graph of Microservices vs. Classical App Architecture
“Drawbacks”: https://www.nginx.com/blog/introduction-to-microservices/
https://microservices.io/patterns/data/database-per-service.html
Encapsulate the data to simplify system
dependencies, create loose-coupling and
empower a good CICD/DevOps process!
Does this pattern always make sense?
Mr. Microservice
6
https://en.wikipedia.org/wiki/Rorschach_test
• Service Granularity, DDD offers a weak heuristic for defining the domain
scope of a microservice (decomposition is hard)
• Enterprise Domains are Big, for larger applications (ERP etc) domains are
inherently large. domain decomposition can create artificial, subjective
boundaries that proliferate incompatible business ontologies
• Business Processes are Cross-Functional, a given business process may
cut across many data domains, corporate business units and even 3rd party
entities altogether (shared kernels and domain aggregates are needed)
• Context Matters, reasonable differences of object definitions and
taxonomy will naturally occur across large organizations (context
mappings)
• Legacy Applications, they exist. And, they don’t fit neatly into idealistic
Bounded Context mappings (bubble contexts for Legacy)
Context mapping is more art than science:
http://www.ouarzy.com/2016/07/25/micro-
service-and-bounded-context-clarification/
Context is in the mind of the modeler:
However…
Domain Driven Design (DDD) principles guide developers to create microservices that align to Bounded
Contexts, which “defines tangible boundaries of applicability of some sub-domain”
https://microservices.io/patterns/decomposition/decompose-by-subdomain.html
Challenges with Bounded Context & DDD
Sounds hard!
Ledger
Service
Command
Model
Domain
Event
Hardest Part About Microservices: Your Data
7
Availability & Durability – what happens with the
data when there is a service failure? How is the state
of data saved and restored? (non-volatility)
CAP Theorem & Network Partitions (CP or AP)
• Strong Consistency – which services require guaranteed
consistency, recovery and a single record of the truth?
• Eventual Consistency – if a microservice end-point is
unavailable, what is the retry protocol? Do all the
services in the Saga/Process adhere to the same
transaction semantics for retry and rollbacks?
ACID – what is the business impact of data
transactions ending in an inconsistent state? Will
business processes fail? Are there any regulatory or
compliance impacts? (ACID vs CAP, Google F1)
findOrderHistory()
findOrder()
Order Events
GL EventsShipping
Events
Logistics
Service
Command
Model
Domain
Event
Order
Service
Command
Model
Domain
Event
Order
History Service
Command
Model
Event
Handlers
Event
Handlers
Event
Handlers
Event
Store
many microservice design patterns
have been created specifically to
try and compensate for the data
consistency guarantees that
databases inherently provide
Eg: CQRS and Event Sourcing
DB encapsulation seems
to shift the very hard data
consistency issues to the
App Tier…
just accept “eventual consistency” and “lack of isolation” for transactions, and we’ll all be ok!
Data and Events are First-Class Citizens
8
Service Service Service
Analytics, Data Science and Data Lakes are too important to my
business’ Digital Transformation and Data-Driven initiatives…
we need architecture focus on Data and Events too
Application Microservices
Data Stores
Event Logs
produce events
consume events
produce events
consume events
read write
App
Events
Data
Events
DB Log
Events
Control Plane
• “State of the Truth”
at a point in time
(current or historic)
• Durable storage used
for Data Recovery /
Archives/years of data
• Polyglot, each service
may determine its own
data structures
• “Narrative of the Truth”
sequence of events
(between data snapshots)
• Days/months of event
data available as Time
Series or Messaging
• Strict ordering of events
& Idempotency
• Strong Consistency of
DB logs (eg; when using
GoldenGate)
• “Systems of Record”
at application tier
• APIs, business rules
and business object
semantics
• Not durable storage
The microservice API is king!
Can we take some of the best ideas of a Service Mesh and apply them to
Data and Events, to create a kind of Data Mesh?
Microservice Mesh and Data Mesh
9
Immutable
Raw Data Events Prepared Data Canonical Data
App
Events
Data
Events
DB Log
Events
Data Domain
Projection …n
Data Domain
Projection 1
Application
Microservices
Control Plane
Today’s Agenda
Microservices and the Challenge of Data
Data Mesh Concept, How GoldenGate Fits
Microservice Patterns with GoldenGate
Anyone Remember “Complex Event Processing”?
Copyright © 2020 Oracle and/or its affiliates.
1
2
3
4
What is a Data Mesh?
11
Microservice
Patterns
Log-based
Integrations
Polyglot Data
Movement
Data Mesh is a data-tier architecture to integrate and
govern enterprise data assets across distributed multi-
cloud environments
Microservices-centric:
• For the administration, deployment and monitoring of the core
frameworks of data movement and governance
• Aligns with for Service Mesh frameworks (K8S, Istio, etc)
• “Sidecar Proxy” style pattern for Events and Data
Immutable event-logs for data integrations:
• Messaging and data store events are globally accessible via
immutable event logs
• Logs may be used to drive Streaming or Batch integrations
Distributed data movement of all types of data
• A data mesh moves data: Relational, NoSQL, JSON, Graph…
• Relational data consistency (ACID) during data movement
• Must work reliably with enterprise OLTP data sets
https://en.wikipedia.org/wiki/ACID
Data
Mesh
Event
Streaming
Immutable
Logs
Data
Replication
Polyglot
Persistence
Edge / 5G
Frameworks
Domain
Driven
Design
Service Mesh
“Sidecars”
Data
Mesh
Evolution towards Real-Time Data Mesh
Copyright © 2020 Oracle and/or its affiliates.
Industry 3.0: Hub and Spoke Transitional: Kappa Hub Mature: Distributed Kappa
This data pattern, popularized by Ralph
Kimball and Bill Inmon, has been the
foundation for enterprise data
management since 1993.
It is transaction consistent, can scale up
nicely for most use cases, and is based on
SQL, lingua-franca for most tools.
By 2010, the Lambda (big data) pattern
was common. In 2014, Jay Kreps (of
LinkedIn) questioned the Lambda
Architecture and spawned Kappa.
The Kappa principles consider batch
processing as a special case of stream
processing. Use a historized event log to
process both real-time as well as batch
processing.
By 2020, IT infrastructure has
dramatically changed – networking,
containers, cloud, compute, IoT etc have
all pushed data to the edge.
A mature Kappa architecture is not a
single instance “hub” but rather a
distributed mesh of data logs, stream
data processing, change events, and time
series data.
Kappa: https://www.oreilly.com/radar/questioning-the-lambda-architecture/
https://en.wikipedia.org/wiki/Dimensional_modeling
mesh & microservice controls
12
ETL
ETL
ETL
ETL
Lambda: http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html
Monoliths Distributed
Data Mesh Conceptual View – Data Domains
13Copyright © 2020 Oracle and/or its affiliates.
Enterprise Data
Producers:
ERP Apps, DBs,
Middleware etc.
Data Domain
Consumers
People owners of “Data
Products”, collections of
data sets in various
stages of curation
IoT Data
Producers:
Devices & Things
Raw Data
Prepared Data
Canonical Data Data
Domain A
Data
Domain B
f(x)
f(x)
Data
Domain C
Data Mesh
(distributed Kappa, microservices, cloud agnostic)
Domain-Specific Views of Data
Raw Event Consumers
Automated Devices,
Edge Nodes (5G), Scheduled
Routines (eg; ETL etc)
Data Product-Specific
Storage Choices:
• RDBMS
• Data Lake
• Object Store
• Graph, etc.
Raw data, Time Series & Alerting events are pushed
Direct to Database (high fidelity transaction semantics fully preserved)
Consumer-Driven, Event-Centric Data Mesh
Copyright © 2020 Oracle and/or its affiliates.
Enterprise Data
Producers
Detect
Event
Logical
Change
Records
(LCRs)
App
DB
committed!
CDC Replication
Data Domain
Consumers
Data
Objects
Table
Data
Raw Data
/ Alerts
SQL
Consumers
Raw
Data
Prepared
Data
Canonical
Data
Raw Data (LCR)
Schema Events
(DDL)
Prepared
Data Topics
“Master”
Data Topics
JSON, XML,
Avro, Parquet,
CSV
Prepared data events are pushed
Canonical data events
Speed &
Fidelity
Trusted
Views
Ease of
Consumption
LCR/TFs
Applications,
Data Services
Biz Consumers
Analytics &
Data Marts
Data Science
& Streaming
Applications
DBAs for HA,
DR and OLTP
Data Mesh puts the consumer
needs first – they require data
at different latency, fidelity,
trust levels and views
Data Model
Object Model
System
Of Record
(SoR)
User
Action
App APIs and
system log events
14
Direct to Database (high fidelity transaction semantics fully preserved)
Distributed by Design, Microservices Based
Copyright © 2020 Oracle and/or its affiliates.
Data Domain
Producers
Detect
Event
Logical
Change
Records
(LCRs)
App
DB
committed!
Data Domain
Consumers
Data
Objects
Table
Data
Raw Data
/ Alerts
SQL
Consumers
Data Model
Object Model
System
Of Record
(SoR)
User
Action
CDC Replication
Microservices
Edge Compute
or Cloud for
Raw Data
Events
Prepare
Technical Data
Views
LCRs
Business
Data Views
Raw data, Time Series & Alerting events are pushed
Prepared data events are pushed
Canonical data
Events
(ephemeral or persisted)
Stream
Process
Events
(persisted)
Stream
Process
Events
(persisted)
Applications,
Data Services
Biz Consumers
Analytics &
Data Marts
Data Science
& Streaming
Applications
DBAs for HA,
DR and OLTP
15
Single Pane of Glass for Real-Time Data Mesh
Copyright © 2020 Oracle and/or its affiliates.
connect
DB2/z
Data
Objects
Table
Data
Raw Data
/ Alerts
SQL
Consumers
Applications,
Data Services
Biz Consumers
Analytics &
Data Marts
Data Science
& Streaming
Applications
DBAs for HA,
DR and OLTP
Real-Time Stream
Data Processing
Raw
Data
DBAs &
Data Engineers
Data Owners &
Data Products
16
Data Consumer DrivenEvent Centric Pipelines
Deploys in a Mesh
Across Containers, Public Clouds and 5G Edge Devices
How it Works Today: GoldenGate for Big Data
Copyright © 2020 Oracle and/or its affiliates.
Data Domain
Consumers
Data
Objects
Table
Data
Raw Data
/ Alerts
SQL
Consumers
Applications,
Data Services
Biz Consumers
Analytics &
Data Marts
Data Science
& Streaming
Applications
DBAs for HA,
DR and OLTP
BYOS (Bring Your Own Spark)
* distributed, may run on any combination of containers and clouds
17
Data Engineer Data AnalystDBA/GG Ops
Capture Pipeline Analyze DeliverIngest
GoldenGate Microservices Applications Stream Analytics Application
BYOM
(Bring Your
Own Messaging)All Data Events
& Transactions
What Exactly Does GoldenGate Do?
Copyright © 2020 Oracle and/or its affiliates.
Database
DBA
ETL
Tools
continuous stream of events
Database Events:
• Transaction Commits (DML)
• Schema Changes (DDL)
• Procedure Execution
Any Data Store Target
• Databases (any)
• Non-Relational
• NoSQL, HDFS, etc
• Search (Elastic…)
• Cloud (Obj Store…)
• Messaging
• Kafka, JMS etc.
• Cloud (any)
GoldenGate
Trail (protocol)
Online Redo Logs
18
App
Services
S
Q
L
S
Q
L
S
Q
L
GoldenGate Replicats, Event
Handlers, and Data Formatters:
Core Concept:
DB Event Log
Customer
Data Formats
Business Events:
Committed Events
1
2
3
4
Today’s Agenda
Microservices and the Challenge of Data
Data Mesh Concept, How GoldenGate Fits
Microservice Patterns with GoldenGate
Anyone Remember “Complex Event Processing”?
Copyright © 2020 Oracle and/or its affiliates.
1
2
3
4
First Things First – GoldenGate Microservices
20
data replications
bi-directional
ms/sec updates
consistency guarantees
Cloud, Containers or Edge Devices
Extracts Replicats Client Libraries
Native GUIs and Full REST APIs
API Gateway or Proxy Service
GG GUI
GoldenGate is itself a set of microservices
that human users or other services may
interact with
Embedded User Interface:
• C-based microservices with embedded HTTP client for
native JavaScript based GUI
• Oracle Jet frameworks for intuitive interaction model
REST native APIs:
• Fully REST native
• Also available, a Command Line Interface (CLI) produces
REST calls to the native services
Full GoldenGate Replication Capabilities:
• 100% coverage for all traditional GG replication
patterns; fully capable of HA/DR use cases
GG
Admin
Service
GG
Distro
Service
GG
Metrics
Service
GG
Receiver
Service
GG
Service
Manager
Your
Services
Microservice Design Patterns for Data
21
Patterns for MicroservicesInherent to the Microservice Architecture is the developer
using specific patterns, sometimes the patterns are partially
embodied in a Programming Framework, but typically the
developers must choose to follow certain heuristics while
programming.
This presentation’s focus:
• “Database Patterns” & “Integration Patterns” …using DB Event
Replication (AKA: Change Data Capture) to improve them
• Simplify the pattern, make the microservice application more resilient
and provide better data consistency guarantees
DB Patterns for Discussion:
• Database per Service (covered earlier)
• CQRS – Command Query Responsibility Segregation
• Event Sourcing
• Saga Pattern
• Transactional Outbox
• Aggregates (AKA: Domain Events)
Transaction
Outbox
https://medium.com/@madhukaudantha/microservice-architecture-and-design-patterns-for-microservices-e0e5013fd58a
Transaction Outbox
Copyright © 2020 Oracle and/or its affiliates.
https://microservices.io/patterns/data/transactional-outbox.html
App
Tables outboxApp
TablesApp
Tables
JSON
Payload
Order
Service
Object
Storage
Update DB *and* Send Domain Events:
• Keep the DB Update and Event as atomic
Other Context:
• “Message Relay” should be Idempotent / Exactly Once in order to
assure that the recipient of the events are not receiving duplicates
• “Message Relay” concept may use (1) polling events, or (2) tail
the DB transaction logs (AKA: Changed Data Capture, CDC)
XYZ
Service
Using Oracle GoldenGate:
• …GG is an exactly-once “Message Relay” microservice
• …also adds benefit of maintaining Transaction Consistency
• …millisecond latencies, trusted as a DR solution for OLTP
• …flexible “payloads” and “broker” variations (see next page)
Baseline Tx Outbox Pattern (microservices.io) Outbox Without the Middleman (No Broker)
Outbox Payload is Tables (not Documents) Outbox from Materialized View
Transaction Outbox Variations
Copyright © 2020 Oracle and/or its affiliates.
outboxApp
Tables
JSON
Payload
Order
Service
Message
Broker
outboxApp
Tables
JSON
Payload
Order
Service
XYZ
Service JSON
Payload
JSON
Payload
App
Tables
Order
Service
Doc
Payload
Message
Broker
App
Tables
XYZ
Service
JSON
Payload
Object
Storage
Doc
Payload
• Full fidelity database raw data
is needed
• Sharding, Data Science,
ODS/EDW, etc…
• Data binding becomes shape of
table rather than shape of
Service-defined doc
• GG Formatters put Table data
into Broker as JSON, Avro,
Parquet, XML, etc.
• Payload centric, may be used
with Aggregates pattern
• Developer of microservice
defines Payload signature
• Best for smaller services
• See also: CQRS and Event
Sourcing
• Point-to-point replication rather
than Pub/Sub
• Orchestration-based Sagas
• Singleton use cases; eg; always
copying service events to a data
lake
• Useful for micro-monoliths and
monoliths; DB’s with very large
or very 3NF schema
• Views are “interface tables” for
external data consumers
• Views present business-friendly
data objects
• Views must be maintained by
DBAs
Object
Storage
JSON
Payload
App
Tables
Order
Service
View
Tables
Object
Storage
View
Tables
Doc
Payload
Doc
Payload
XYZ
ABC
Transaction Outbox for the Whole Enterprise
Copyright © 2019 Oracle and/or its affiliates.
DB2/z
Replication of
Real-time Data
Transactions & Events
GoldenGate Stream Analytics
ETL
&ML
Object
Storage
Relational
Non-
Relational
Apps
https://www.oracle.com/middleware/technologies/goldengate.html
Microservices-
Based
DBMS
Cloud
Big Data
NoSQL
Streams
Order
History
Logistics
Service
Order
Service
Ledger
Service
CQRS Pattern
Command Query Responsibility Segregation
Copyright © 2020 Oracle and/or its affiliates.
Query
Model
Command
Model
SQL Select
Read Side
Write Side
Inserts,
Updates etc.
Basic CQRS (https://martinfowler.com/bliki/CQRS.html)
findOrderHistory()
findOrder()
Event
Handlers
Multi-Service Example (https://microservices.io/patterns/data/cqrs.html)
GL Events
Shipping Events
Order Events
• Separation of duties:
• Writes (Commands) – go against Domain Model
• Reads – optimized for client Queries
• Alternative to typical “CRUD” type interfaces for
Microservice API / signatures
• Popular when:
• Complex Domains – require queries against many
domain objects at same time
• High Performance – to support highly optimized read
interfaces
• Complex Domains:
• Multiple services must update common “view”
• Clients (GUIs) may run queries that require many
joins and/or lots of records fetched
write read
User Interface
GUI
Order
History
Order
Service
Simplified CQRS with Transaction Outbox
Copyright © 2020 Oracle and/or its affiliates.
findOrderHistory()
findOrder()
Delegate the data event propagation:
• Instead of having the developers (of the
microservices) write the logic, handlers, and
failure conditions for eventing → leverage data-
tier event replication to propagate the DML
events to the “view” data store
Avoid complicated and costly mistakes:
• CAP Theorem is a hard problem to code for in the
microservice itself:
• What happens if the service fails just after
a local update but before the event is
sent?
• What if the “view” Service receives the
event but the data store fails?
• What if the event volume / workload
spikes and the “view” Service goes down?
Data tier “logical replication” is designed to be very fast, work across
WAN/Clouds and to maintain consistency:
• Relational semantics may involve long-running transactions, foreign
key dependent DML operations, and strict transaction ordering
requirements
write
User Interface
ms/sec updates
consistency guarantees
Order Events
read
Order
Service
Logistics
Service
Ledger
Service
Order
History
Simplified CQRS Pattern
Copyright © 2020 Oracle and/or its affiliates.
findOrderHistory()
findOrder()
Event
Handlers
Order EventsCommand
Model
Command
Model
Command
Model
Event
Store
Event
Handlers
Event
Handlers
GL Events
Shipping
Events
Domain
Event
Domain
Event
Domain
Event
Before After
• Developer responsible for correct
implementation of CAP Theorem
• Data consistency on any outage
(microservice / data tier)
• More code to write and maintain
• Worse performance (latency)
• Impedance, across many services into
view model (semantic misalignment)
✓ Consistency and recoverability
guarantees
✓ Less code to write or maintain
✓ Best possible performance
write read
User Interface User Interface
Order
History
Logistics
Service
Order
Service
Ledger
Service
write read
Event Sourcing Pattern
Copyright © 2020 Oracle and/or its affiliates.
BIG topic in the Microservice community:
• …and, a point of confusion as well
Pattern espoused for:
• An alternate durable data store (instead of a Database) to
store a services’ state, and/or… In combination with a local
Database, where the Event Store is use primarily for
communication of events (and not for saving state)
• A way to record a sequence of immutable app events to be
used later by the service which generated the events,
and/or (when Event Store is shared) by other services who
wish to subscribe to events that other services create
Implementations:
• DIY using your favorite language; purpose-built event
stores like Eventuate, Axon, Akka or Event Store
• Apache Kafka, Apache Pulsar, RabbitMQ, or NoSQL engines
As a Durable Store ++ For Event Communications (only)
Local Events Shared Events
Event Store
➢ ALL events…
➢ Order Created
➢ Order Approved
➢ …
➢ Order Shipped
Event Store
all durable
events
➢ only the
➢ events that
➢ need to be
➢ communicated
➢ ….
Order
Events
Logistics
Events
Ledger
Events
Event Store
Order
Events
Logistics
Events
Ledger
Events
Event Store
Circa 2005: https://martinfowler.com/eaaDev/EventSourcing.html
• https://microservices.io/event-sourcing.html
• https://microsoft.com/patterns/event-sourcing
https://dzone.com/articles/microservices-with-
cqrs-and-event-sourcing
https://www.infoq.com/news/2017/11/eve
nt-sourcing-microservices/
Order
Service
Order
Service
Order
Service
Logistics
Service
Ledger
Service
Order
Service
Logistics
Service
Ledger
Service
Critiques of Event Sourcing
Copyright © 2020 Oracle and/or its affiliates.
Exposing the Persistence Tier:
• Taken too far (Why Event Sourcing is an Anti-Pattern), developers wind up using the
Event Store as a Shared Persistence model, and other microservice now have hard-
coupled binding to the message formats of the originating service
Whole System Fallacy:
• Some microservices leaders (Udi and Greg Reach CQRS Agreement) say to narrow
the aperture on when to use CQRS + Event Sourcing → only within a Business
Component and a Single Bounded Context
• Minimizes utility of pattern for Communications
Forcing Eventual Consistency on Developers:
• The propensity to over-use CQRS & Event Sourcing at the at the whole system level
forces developers to manage eventual consistency in the Application tiers (What
they don’t tell you about event sourcing)
• “…they will make your life a living hell” doing DevOps, debugging and
system recovery when a “Mesh” of services are interacting via Event Store and
message signatures can lead to disaster
Event Sourcing with Transaction Outbox
Copyright © 2020 Oracle and/or its affiliates.
Before After
Service Service Service
Application Microservices
Data Stores
Events
produce events
consume events
produce events
consume events
read write
App
Events
Data
Events
DB Log
Events
Control Plane
Data Mesh
https://dzone.com/articles/microservices-with-cqrs-and-event-sourcing
• Microservice developers must code for consistency
• Data may be stale or incorrect
• Many tiers for failure, difficult to trouble-shoot
• Useful for “communications” and “shared events” requirements
• Delegate data tier consistency to “Events” tier
• Using DB Replication tools (eg; GoldenGate) for ACID
• Events may be used “system-wide” rather than per Context
per ‘bounded context’ and not used system-
wide for communications
Domain z
Domain X
Domain Y
Saga Pattern
Copyright © 2020 Oracle and/or its affiliates.
https://livebook.manning.com/book/microservices-patterns/chapter-4/17
Popular, long-standing pattern:
• First paper back in 1987 (Garcia-Molina and Salem, 1987)
• Used for long-lived transactions in distributed systems, that cannot
be completed using multi-phase commits
In Microservices:
• Frequent requirement due to “one DB per service” pattern
• Multi-phase commits at microservice tier are uncommon
• Typically see either (a) Choreography or (b) Orchestration based
approaches (https://microservices.io/patterns/data/saga.html)
Drawbacks:
• Lack of Isolation (ACD not ACID) and Eventual Consistency lag, system
state can be inconsistent for seconds, minutes or hours
• Complex programming model, depends on “Compensating
Transactions”
• Requires other Patterns (eg; Event Sourcing, Aggregates) for
maintaining DB consistency
Additional logic for
compensation /
rollbacksChoreography or
Orchestration logic
We don’t want to lose any events
and leave the system inconsistent.
How to atomically update the
store and send the event?
Saga Pattern with Transactional Outbox
Copyright © 2020 Oracle and/or its affiliates.
Ticket
Service
Insurance
Service
Payment
Service
Outbox Pattern
Outbox Pattern
Outbox Pattern
Event Hub
Ticket
Topic
Order
Topic
Payment
Topic
Publish Events:
Subscribe Events:
Red Hat Summit 2019: The New Era of Transactions…
Saga Approach:
• Choreography
Outbox with Saga:
• Publish committed events to event hub
(eg; Kafka or other); very low risk of
inconsistency due to network or service
failures (Dzone: Data Consistency…)
• GoldenGate enforces Idempotency /
Exactly Once and may preserve intra-table
Consistency as well
Subscribers:
• All services in the Saga can be assured
that events in the Hub are fully committed
• Compensating Txs happen per normal
Saga pattern guidelines
Broker
Aggregate is a collection of objects grouped together:
• Typically grouped by Transaction boundaries
• Used for local persistence
• *within* a specific Bounded Context (always!)
Motivation:
• A core purpose of this pattern (within a specific context) is to
ensure that invariants and domain rules are always applied to the
whole transaction holistically
Domain Event (Aggregate) vs Integration Event
Copyright © 2020 Oracle and/or its affiliates.
Domain Service
Aggregate 1
Aggregate 2
Root Entity
Entity Value
Domain Event
GUI Domain Context xyz
Order
Events
Chris Richardson: Using Aggregates with Events and CQRS
Integration Event is a collection of objects grouped together:
• Also, typically grouped by Transaction boundaries
• Used mainly for events and external communications
• Usually designed for bindings outside of local Context
See Also:
• This pattern is mainly focused on the organization of data posted
via Event Sourcing, Saga, or Transaction Outbox
Domain Service
Aggregate 1
Aggregate 2
Root Entity
Entity Value
Domain Event
GUI Domain Context xyz
Microsoft Patterns: Using Integration Events (with Event Brokers)
Integration Event
Non-Relational
Integrations
Data Lake Ingest Streaming Ingest Cloud Ingest Messaging Replication NoSQL Replication SaaS Replication
DB Event
Replication
Unidirectional Bi-Directional Peer-to-Peer Broadcast Consolidation Distribution
Event Stream
Processing
Data Pipelines Data Transformation GoldenGate Integrations Time Series Analysis Geo-Fencing Predictive Analytics
Integration Events with GoldenGate
Copyright © 2020 Oracle and/or its affiliates. 34
Today’s Agenda
Microservices and the Challenge of Data
Data Mesh Concept, How GoldenGate Fits
Microservice Patterns with GoldenGate
Anyone Remember “Complex Event Processing”?
Copyright © 2020 Oracle and/or its affiliates.
1
2
3
4
Hype Cycle 2009
Copyright © 2020 Oracle and/or its affiliates.
Complex Event Processing:
• CEP is a kind of computing in which incoming data about
events is distilled into more useful, higher level event data
that provides insight into what is happening. […] CEP is
used for highly demanding, continuous-intelligence
applications that enhance situation awareness and
support real-time decisions. Gartner
20 Years Too Early?
• CEP dates back to the 1990’s (history of CEP engines)
• CEP came before “Event Stream Processing” and generally
has covered more complex use cases (eg; handling of out-
of-order events and more complicated correlation
semantics) (what’s the difference, 2019 and mythbuster
CEP vs ESP, 2008)
• Largely overtaken by Big Data stream processing
technologies that are open-source, massively-parallel, and
widely available as cloud-native
2009!
Stream Processing 2020
Copyright © 2020 Oracle and/or its affiliates.
Time
Series
DB/OLAP
Big Data Event
Stream Processing
Complex
Event
Processing
Becoming more aligned to open source
/ apache frameworks
Becoming more capable of rich windowing functions
and time-clock correlation semantics
Complex Event Processing:
• Traditionally running in “scale-up”
SMP in-memory framework
• Many CEP engines are moving
toward MPP “scale-out”
architectures
• Programming-centric
historically,
some CEP
engines
were
also query-
centric as far
back as 2009
• Time-clock semantics
and advanced cache
may be used for “state
machine” type use cases
Stream Processing:
• Built around MPP frameworks and
typically Apache open-sourced
• Genesis was around simplistic use
cases on high volumes of events
• All SP engines began with
rudimentary windowing and
correlation semantics, but most
frameworks are gradually
becoming more functionally
comparable to classic CEP
• Simplistic windowing and caching
for basic stream-clock events
Time Series Databases:
• Optimized for persistence and historic analytics
on time-ordered events…data is often sourced
from CEP or SP engines
Technical and Functional Differences between CEP and SP: https://complexevents.com/2019/07/15/whats-the-difference-between-esp-and-cep-2/
Stream Processing/CEP for Event Driven Architectures
Copyright © 2020 Oracle and/or its affiliates.
There has been a widespread
awakening to the benefits of Event
Drive Architecture (EDA) for
increasing the scalability and agility of
business systems. […] Stream
analytics is based on the mathematics
of complex-event processing (CEP).
CEP is a computing technique in
which incoming data about what is
happening (event data) is processed
as it arrives (data in motion or
recently in motion) to generate
higher level, more useful, summary
information (complex events).
W. Roy Schulte (of Gartner), March 2020:
EDA is Suddenly Popular Will Stream Analytics be Next?
Event Stream Analytics (& CEP)
Data & Microservice Events
Event/Data
Pipelines
Time-Series
Analysis
Geospatial
Analysis
Real-time
AI/ML
Continious
ETL
Use Cases:
Oracle GoldenGate Stream Analytics
39Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2018
Ingest Events Select Processing Patterns Build Event Pipelines Serve Data Downstream
GoldenGate events from any DB or
NoSQL; also supports events from
Kafka, JMS, AQ and MQTT
Rich set of pre-built patterns will
dramatically improve developer
efficiency and time-to-value
Easily leverage geo-fencing,
machine-learning, and other
reference data within the stream
Data can be delivered out to Kafka,
databases, or easily staged for
external ETL jobs
Data Owners &
Data Products
Significant Intellectual Property
Copyright © 2020 Oracle and/or its affiliates.
More than 70 patents on stream processing
Mature tech stack for Event Processing
Over 10 years of IP investment
Continuous Query Language (CQL)
12.2
12.3
18c
19c
11g
GoldenGate Stream Analytics
Copyright © 2020 Oracle and/or its affiliates.
Interactive, Low-Code Designer Time Series Analytics Event Driven Dashboards
Patterns / Accelerators Geo-Spatial Analysis & Geo-Fencing Predictive Analytics / ML
Upgrade to Spark, Better than Flink or KSQL
Copyright © 2019 Oracle and/or its affiliates.
[best] [worst] GoldenGate
Spark
Streaming
Apache
Flink / Beam
Confluent
KSQL
User Experience
Low Code Development (with built-in patterns/accelerators)    
Interactive/Live Edits (browser based, view changes immediately)    
Built-in Live Dashboards (event-driven charts/graphs)    
Core Streaming Semantics
What is Being Computed (transforms, joins, flatten, statefulness etc)    
Time Windows(global, fixed, sliding, tumbling, custom etc)    
When in Processing Time (triggers – event, time, count, timers, etc)    
How do Refinements Relate (discarding, accumulating, retracting)    
Analytics
Robust CEP Capabilities (complex event correlations, native time clock)    
Geo-Fencing & Spatial (lat/long, built in maps, custom map tiles, etc)    
Machine Learning (native scala, PMML, python support etc)    
Time Series Analysis (built-in interval patterns, thresholding etc)    
Other Features
Backpressure (dynamic ingest per pipeline) Automatic Custom Custom Custom
State Management (automation across streams & native cache) Data Grid N/A RocksDB RocksDB
Data Consistency (OLTP Change Events, Inserts/Updates/Deletes) Automatic Custom Custom Custom
GoldenGate Stream Type (aware of SCN/CSN, transactions, order, etc) Native Custom Custom Custom
Today’s Agenda
Microservices and the Challenge of Data
Data Mesh Concept, How GoldenGate Fits
Microservice Patterns with GoldenGate
Anyone Remember “Complex Event Processing”?
Conclusion!
Copyright © 2020 Oracle and/or its affiliates.
1
2
3
4
5
GoldenGate Stream Analytics for Data Mesh
Copyright © 2020 Oracle and/or its affiliates.
connect
DB2/z
Data
Objects
Table
Data
Raw Data
/ Alerts
SQL
Consumers
Applications,
Data Services
Biz Consumers
Analytics &
Data Marts
Data Science
& Streaming
Applications
DBAs for HA,
DR and OLTP
Real-Time Stream
Data Processing
Raw
Data
DBAs &
Data Engineers
Data Owners &
Data Products
44
Data Consumer DrivenEvent Centric Pipelines
Deploys in a Mesh
Across Containers, Public Clouds and 5G Edge Devices
This is not a Metamorphosis, it is a Paradigm Shift
Copyright © 2020 Oracle and/or its affiliates.
The Success Paradox
Data success factors that did well in
Industry 3.0 will not be the factors
that create success in Industry 4.0
Next Gen Data Architecture
ETL Vendors
1990 – 2010’s Gen1 :
• Replication
• Messaging
• Streaming
• Pipelines
Next-Gen has
new DNA not
tied to old ETL tools
It is impossible to evolve older Batch
Processing tools into a modern Event-
Centric Stream Processing solution; the
underlying paradigms are fundamentally
different
45
46Copyright © 2020 Oracle and/or its affiliates.
Questions?
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2020 Oracle and/or its affiliates.

More Related Content

What's hot

Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Dr. Arif Wider
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
Databricks
 
Moving to Databricks & Delta
Moving to Databricks & DeltaMoving to Databricks & Delta
Moving to Databricks & Delta
Databricks
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks
 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3
Jeffrey T. Pollock
 
Databricks Platform.pptx
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptx
Alex Ivy
 
Oracle dba training
Oracle  dba    training Oracle  dba    training
Oracle dba training
P S Rani
 
Building End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCPBuilding End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCP
Databricks
 
Introduction SQL Analytics on Lakehouse Architecture
Introduction SQL Analytics on Lakehouse ArchitectureIntroduction SQL Analytics on Lakehouse Architecture
Introduction SQL Analytics on Lakehouse Architecture
Databricks
 
Oracle data guard for beginners
Oracle data guard for beginnersOracle data guard for beginners
Oracle data guard for beginners
Pini Dibask
 
Delta lake and the delta architecture
Delta lake and the delta architectureDelta lake and the delta architecture
Delta lake and the delta architecture
Adam Doyle
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Databricks
 
Oracle RAC 19c and Later - Best Practices #OOWLON
Oracle RAC 19c and Later - Best Practices #OOWLONOracle RAC 19c and Later - Best Practices #OOWLON
Oracle RAC 19c and Later - Best Practices #OOWLON
Markus Michalewicz
 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to Mesh
Jeffrey T. Pollock
 
What to Expect From Oracle database 19c
What to Expect From Oracle database 19cWhat to Expect From Oracle database 19c
What to Expect From Oracle database 19c
Maria Colgan
 
Differentiate Big Data vs Data Warehouse use cases for a cloud solution
Differentiate Big Data vs Data Warehouse use cases for a cloud solutionDifferentiate Big Data vs Data Warehouse use cases for a cloud solution
Differentiate Big Data vs Data Warehouse use cases for a cloud solution
James Serra
 
Introducing Azure SQL Database
Introducing Azure SQL DatabaseIntroducing Azure SQL Database
Introducing Azure SQL Database
James Serra
 
[DSC Europe 22] Overview of the Databricks Platform - Petar Zecevic
[DSC Europe 22] Overview of the Databricks Platform - Petar Zecevic[DSC Europe 22] Overview of the Databricks Platform - Petar Zecevic
[DSC Europe 22] Overview of the Databricks Platform - Petar Zecevic
DataScienceConferenc1
 
Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2
Databricks
 
Data Migration to Azure
Data Migration to AzureData Migration to Azure
Data Migration to Azure
Sanjay B. Bhakta
 

What's hot (20)

Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
 
Moving to Databricks & Delta
Moving to Databricks & DeltaMoving to Databricks & Delta
Moving to Databricks & Delta
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3
 
Databricks Platform.pptx
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptx
 
Oracle dba training
Oracle  dba    training Oracle  dba    training
Oracle dba training
 
Building End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCPBuilding End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCP
 
Introduction SQL Analytics on Lakehouse Architecture
Introduction SQL Analytics on Lakehouse ArchitectureIntroduction SQL Analytics on Lakehouse Architecture
Introduction SQL Analytics on Lakehouse Architecture
 
Oracle data guard for beginners
Oracle data guard for beginnersOracle data guard for beginners
Oracle data guard for beginners
 
Delta lake and the delta architecture
Delta lake and the delta architectureDelta lake and the delta architecture
Delta lake and the delta architecture
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
 
Oracle RAC 19c and Later - Best Practices #OOWLON
Oracle RAC 19c and Later - Best Practices #OOWLONOracle RAC 19c and Later - Best Practices #OOWLON
Oracle RAC 19c and Later - Best Practices #OOWLON
 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to Mesh
 
What to Expect From Oracle database 19c
What to Expect From Oracle database 19cWhat to Expect From Oracle database 19c
What to Expect From Oracle database 19c
 
Differentiate Big Data vs Data Warehouse use cases for a cloud solution
Differentiate Big Data vs Data Warehouse use cases for a cloud solutionDifferentiate Big Data vs Data Warehouse use cases for a cloud solution
Differentiate Big Data vs Data Warehouse use cases for a cloud solution
 
Introducing Azure SQL Database
Introducing Azure SQL DatabaseIntroducing Azure SQL Database
Introducing Azure SQL Database
 
[DSC Europe 22] Overview of the Databricks Platform - Petar Zecevic
[DSC Europe 22] Overview of the Databricks Platform - Petar Zecevic[DSC Europe 22] Overview of the Databricks Platform - Petar Zecevic
[DSC Europe 22] Overview of the Databricks Platform - Petar Zecevic
 
Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2
 
Data Migration to Azure
Data Migration to AzureData Migration to Azure
Data Migration to Azure
 

Similar to Microservices Patterns with GoldenGate

Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Igor De Souza
 
Data Services and the Modern Data Ecosystem (ASEAN)
Data Services and the Modern Data Ecosystem (ASEAN)Data Services and the Modern Data Ecosystem (ASEAN)
Data Services and the Modern Data Ecosystem (ASEAN)
Denodo
 
Accelerating a Path to Digital with a Cloud Data Strategy
Accelerating a Path to Digital with a Cloud Data StrategyAccelerating a Path to Digital with a Cloud Data Strategy
Accelerating a Path to Digital with a Cloud Data Strategy
MongoDB
 
#dbhouseparty - Should I be building Microservices?
#dbhouseparty - Should I be building Microservices?#dbhouseparty - Should I be building Microservices?
#dbhouseparty - Should I be building Microservices?
Tammy Bednar
 
Flash session -streaming--ses1243-lon
Flash session -streaming--ses1243-lonFlash session -streaming--ses1243-lon
Flash session -streaming--ses1243-lon
Jeffrey T. Pollock
 
Microservices: Where do they fit within a rapidly evolving integration archit...
Microservices: Where do they fit within a rapidly evolving integration archit...Microservices: Where do they fit within a rapidly evolving integration archit...
Microservices: Where do they fit within a rapidly evolving integration archit...
Kim Clark
 
Technology Overview
Technology OverviewTechnology Overview
Technology Overview
Liran Zelkha
 
Best Practices in the Cloud for Data Management (US)
Best Practices in the Cloud for Data Management (US)Best Practices in the Cloud for Data Management (US)
Best Practices in the Cloud for Data Management (US)
Denodo
 
Webinar future dataintegration-datamesh-and-goldengatekafka
Webinar future dataintegration-datamesh-and-goldengatekafkaWebinar future dataintegration-datamesh-and-goldengatekafka
Webinar future dataintegration-datamesh-and-goldengatekafka
Jeffrey T. Pollock
 
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...
marksimpsongw
 
Data Virtualization: Introduction and Business Value (UK)
Data Virtualization: Introduction and Business Value (UK)Data Virtualization: Introduction and Business Value (UK)
Data Virtualization: Introduction and Business Value (UK)
Denodo
 
Microservices Architecture - Bangkok 2018
Microservices Architecture - Bangkok 2018Microservices Architecture - Bangkok 2018
Microservices Architecture - Bangkok 2018
Araf Karsh Hamid
 
Microservices Architecture & Testing Strategies
Microservices Architecture & Testing StrategiesMicroservices Architecture & Testing Strategies
Microservices Architecture & Testing Strategies
Araf Karsh Hamid
 
MicroServices-Part-1.pdf
MicroServices-Part-1.pdfMicroServices-Part-1.pdf
MicroServices-Part-1.pdf
chanhluc2112
 
How Financial Institutions Are Leveraging Data Virtualization to Overcome the...
How Financial Institutions Are Leveraging Data Virtualization to Overcome the...How Financial Institutions Are Leveraging Data Virtualization to Overcome the...
How Financial Institutions Are Leveraging Data Virtualization to Overcome the...
Denodo
 
Accelerating a Path to Digital With a Cloud Data Strategy
Accelerating a Path to Digital With a Cloud Data StrategyAccelerating a Path to Digital With a Cloud Data Strategy
Accelerating a Path to Digital With a Cloud Data Strategy
MongoDB
 
Microservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SREMicroservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SRE
Araf Karsh Hamid
 
SQL Server 2019 Data Virtualization
SQL Server 2019 Data VirtualizationSQL Server 2019 Data Virtualization
SQL Server 2019 Data Virtualization
Matthew W. Bowers
 
Database@Home : Data Driven Apps - Data-driven Microservices Architecture wit...
Database@Home : Data Driven Apps - Data-driven Microservices Architecture wit...Database@Home : Data Driven Apps - Data-driven Microservices Architecture wit...
Database@Home : Data Driven Apps - Data-driven Microservices Architecture wit...
Tammy Bednar
 
Data Virtualization: An Introduction
Data Virtualization: An IntroductionData Virtualization: An Introduction
Data Virtualization: An Introduction
Denodo
 

Similar to Microservices Patterns with GoldenGate (20)

Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...
 
Data Services and the Modern Data Ecosystem (ASEAN)
Data Services and the Modern Data Ecosystem (ASEAN)Data Services and the Modern Data Ecosystem (ASEAN)
Data Services and the Modern Data Ecosystem (ASEAN)
 
Accelerating a Path to Digital with a Cloud Data Strategy
Accelerating a Path to Digital with a Cloud Data StrategyAccelerating a Path to Digital with a Cloud Data Strategy
Accelerating a Path to Digital with a Cloud Data Strategy
 
#dbhouseparty - Should I be building Microservices?
#dbhouseparty - Should I be building Microservices?#dbhouseparty - Should I be building Microservices?
#dbhouseparty - Should I be building Microservices?
 
Flash session -streaming--ses1243-lon
Flash session -streaming--ses1243-lonFlash session -streaming--ses1243-lon
Flash session -streaming--ses1243-lon
 
Microservices: Where do they fit within a rapidly evolving integration archit...
Microservices: Where do they fit within a rapidly evolving integration archit...Microservices: Where do they fit within a rapidly evolving integration archit...
Microservices: Where do they fit within a rapidly evolving integration archit...
 
Technology Overview
Technology OverviewTechnology Overview
Technology Overview
 
Best Practices in the Cloud for Data Management (US)
Best Practices in the Cloud for Data Management (US)Best Practices in the Cloud for Data Management (US)
Best Practices in the Cloud for Data Management (US)
 
Webinar future dataintegration-datamesh-and-goldengatekafka
Webinar future dataintegration-datamesh-and-goldengatekafkaWebinar future dataintegration-datamesh-and-goldengatekafka
Webinar future dataintegration-datamesh-and-goldengatekafka
 
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...
 
Data Virtualization: Introduction and Business Value (UK)
Data Virtualization: Introduction and Business Value (UK)Data Virtualization: Introduction and Business Value (UK)
Data Virtualization: Introduction and Business Value (UK)
 
Microservices Architecture - Bangkok 2018
Microservices Architecture - Bangkok 2018Microservices Architecture - Bangkok 2018
Microservices Architecture - Bangkok 2018
 
Microservices Architecture & Testing Strategies
Microservices Architecture & Testing StrategiesMicroservices Architecture & Testing Strategies
Microservices Architecture & Testing Strategies
 
MicroServices-Part-1.pdf
MicroServices-Part-1.pdfMicroServices-Part-1.pdf
MicroServices-Part-1.pdf
 
How Financial Institutions Are Leveraging Data Virtualization to Overcome the...
How Financial Institutions Are Leveraging Data Virtualization to Overcome the...How Financial Institutions Are Leveraging Data Virtualization to Overcome the...
How Financial Institutions Are Leveraging Data Virtualization to Overcome the...
 
Accelerating a Path to Digital With a Cloud Data Strategy
Accelerating a Path to Digital With a Cloud Data StrategyAccelerating a Path to Digital With a Cloud Data Strategy
Accelerating a Path to Digital With a Cloud Data Strategy
 
Microservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SREMicroservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SRE
 
SQL Server 2019 Data Virtualization
SQL Server 2019 Data VirtualizationSQL Server 2019 Data Virtualization
SQL Server 2019 Data Virtualization
 
Database@Home : Data Driven Apps - Data-driven Microservices Architecture wit...
Database@Home : Data Driven Apps - Data-driven Microservices Architecture wit...Database@Home : Data Driven Apps - Data-driven Microservices Architecture wit...
Database@Home : Data Driven Apps - Data-driven Microservices Architecture wit...
 
Data Virtualization: An Introduction
Data Virtualization: An IntroductionData Virtualization: An Introduction
Data Virtualization: An Introduction
 

More from Jeffrey T. Pollock

2017 OpenWorld Keynote for Data Integration
2017 OpenWorld Keynote for Data Integration2017 OpenWorld Keynote for Data Integration
2017 OpenWorld Keynote for Data Integration
Jeffrey T. Pollock
 
Flash session -goldengate--lht1053-lon
Flash session -goldengate--lht1053-lonFlash session -goldengate--lht1053-lon
Flash session -goldengate--lht1053-lon
Jeffrey T. Pollock
 
Version Control Training - First Lego League
Version Control Training - First Lego LeagueVersion Control Training - First Lego League
Version Control Training - First Lego League
Jeffrey T. Pollock
 
Oracle Stream Analytics - Developer Introduction
Oracle Stream Analytics - Developer IntroductionOracle Stream Analytics - Developer Introduction
Oracle Stream Analytics - Developer Introduction
Jeffrey T. Pollock
 
GoldenGate and Stream Processing with Special Guest Rakuten
GoldenGate and Stream Processing with Special Guest RakutenGoldenGate and Stream Processing with Special Guest Rakuten
GoldenGate and Stream Processing with Special Guest Rakuten
Jeffrey T. Pollock
 
Stream based Data Integration
Stream based Data IntegrationStream based Data Integration
Stream based Data Integration
Jeffrey T. Pollock
 
Intelligent Integration OOW2017 - Jeff Pollock
Intelligent Integration OOW2017 - Jeff PollockIntelligent Integration OOW2017 - Jeff Pollock
Intelligent Integration OOW2017 - Jeff Pollock
Jeffrey T. Pollock
 
Oracle Data Integration - Overview
Oracle Data Integration - OverviewOracle Data Integration - Overview
Oracle Data Integration - Overview
Jeffrey T. Pollock
 
Oracle Data Integration CON9737 at OpenWorld
Oracle Data Integration CON9737 at OpenWorldOracle Data Integration CON9737 at OpenWorld
Oracle Data Integration CON9737 at OpenWorld
Jeffrey T. Pollock
 
CDO - Chief Data Officer Momentum and Trends
CDO - Chief Data Officer Momentum and TrendsCDO - Chief Data Officer Momentum and Trends
CDO - Chief Data Officer Momentum and Trends
Jeffrey T. Pollock
 
Big Data at Oracle - Strata 2015 San Jose
Big Data at Oracle - Strata 2015 San JoseBig Data at Oracle - Strata 2015 San Jose
Big Data at Oracle - Strata 2015 San Jose
Jeffrey T. Pollock
 
One Slide Overview: ORCL Big Data Integration and Governance
One Slide Overview: ORCL Big Data Integration and GovernanceOne Slide Overview: ORCL Big Data Integration and Governance
One Slide Overview: ORCL Big Data Integration and Governance
Jeffrey T. Pollock
 
Oracle Big Data Governance Webcast Charts
Oracle Big Data Governance Webcast ChartsOracle Big Data Governance Webcast Charts
Oracle Big Data Governance Webcast Charts
Jeffrey T. Pollock
 
Unlocking Big Data Silos in the Enterprise or the Cloud (Con7877)
Unlocking Big Data Silos in the Enterprise or the Cloud (Con7877)Unlocking Big Data Silos in the Enterprise or the Cloud (Con7877)
Unlocking Big Data Silos in the Enterprise or the Cloud (Con7877)
Jeffrey T. Pollock
 
Tapping into the Big Data Reservoir (CON7934)
Tapping into the Big Data Reservoir (CON7934)Tapping into the Big Data Reservoir (CON7934)
Tapping into the Big Data Reservoir (CON7934)
Jeffrey T. Pollock
 
Brief lessons from the greatest product managers
Brief lessons from the greatest product managersBrief lessons from the greatest product managers
Brief lessons from the greatest product managers
Jeffrey T. Pollock
 
Klarna Tech Talk - Mind the Data!
Klarna Tech Talk - Mind the Data!Klarna Tech Talk - Mind the Data!
Klarna Tech Talk - Mind the Data!
Jeffrey T. Pollock
 
Accelerate Return on Data
Accelerate Return on DataAccelerate Return on Data
Accelerate Return on Data
Jeffrey T. Pollock
 
2010.03.16 Pollock.Edw2010.Modern D Ifor Warehousing
2010.03.16 Pollock.Edw2010.Modern D Ifor Warehousing2010.03.16 Pollock.Edw2010.Modern D Ifor Warehousing
2010.03.16 Pollock.Edw2010.Modern D Ifor Warehousing
Jeffrey T. Pollock
 
2009.10.22 S308460 Cloud Data Services
2009.10.22 S308460  Cloud Data Services2009.10.22 S308460  Cloud Data Services
2009.10.22 S308460 Cloud Data Services
Jeffrey T. Pollock
 

More from Jeffrey T. Pollock (20)

2017 OpenWorld Keynote for Data Integration
2017 OpenWorld Keynote for Data Integration2017 OpenWorld Keynote for Data Integration
2017 OpenWorld Keynote for Data Integration
 
Flash session -goldengate--lht1053-lon
Flash session -goldengate--lht1053-lonFlash session -goldengate--lht1053-lon
Flash session -goldengate--lht1053-lon
 
Version Control Training - First Lego League
Version Control Training - First Lego LeagueVersion Control Training - First Lego League
Version Control Training - First Lego League
 
Oracle Stream Analytics - Developer Introduction
Oracle Stream Analytics - Developer IntroductionOracle Stream Analytics - Developer Introduction
Oracle Stream Analytics - Developer Introduction
 
GoldenGate and Stream Processing with Special Guest Rakuten
GoldenGate and Stream Processing with Special Guest RakutenGoldenGate and Stream Processing with Special Guest Rakuten
GoldenGate and Stream Processing with Special Guest Rakuten
 
Stream based Data Integration
Stream based Data IntegrationStream based Data Integration
Stream based Data Integration
 
Intelligent Integration OOW2017 - Jeff Pollock
Intelligent Integration OOW2017 - Jeff PollockIntelligent Integration OOW2017 - Jeff Pollock
Intelligent Integration OOW2017 - Jeff Pollock
 
Oracle Data Integration - Overview
Oracle Data Integration - OverviewOracle Data Integration - Overview
Oracle Data Integration - Overview
 
Oracle Data Integration CON9737 at OpenWorld
Oracle Data Integration CON9737 at OpenWorldOracle Data Integration CON9737 at OpenWorld
Oracle Data Integration CON9737 at OpenWorld
 
CDO - Chief Data Officer Momentum and Trends
CDO - Chief Data Officer Momentum and TrendsCDO - Chief Data Officer Momentum and Trends
CDO - Chief Data Officer Momentum and Trends
 
Big Data at Oracle - Strata 2015 San Jose
Big Data at Oracle - Strata 2015 San JoseBig Data at Oracle - Strata 2015 San Jose
Big Data at Oracle - Strata 2015 San Jose
 
One Slide Overview: ORCL Big Data Integration and Governance
One Slide Overview: ORCL Big Data Integration and GovernanceOne Slide Overview: ORCL Big Data Integration and Governance
One Slide Overview: ORCL Big Data Integration and Governance
 
Oracle Big Data Governance Webcast Charts
Oracle Big Data Governance Webcast ChartsOracle Big Data Governance Webcast Charts
Oracle Big Data Governance Webcast Charts
 
Unlocking Big Data Silos in the Enterprise or the Cloud (Con7877)
Unlocking Big Data Silos in the Enterprise or the Cloud (Con7877)Unlocking Big Data Silos in the Enterprise or the Cloud (Con7877)
Unlocking Big Data Silos in the Enterprise or the Cloud (Con7877)
 
Tapping into the Big Data Reservoir (CON7934)
Tapping into the Big Data Reservoir (CON7934)Tapping into the Big Data Reservoir (CON7934)
Tapping into the Big Data Reservoir (CON7934)
 
Brief lessons from the greatest product managers
Brief lessons from the greatest product managersBrief lessons from the greatest product managers
Brief lessons from the greatest product managers
 
Klarna Tech Talk - Mind the Data!
Klarna Tech Talk - Mind the Data!Klarna Tech Talk - Mind the Data!
Klarna Tech Talk - Mind the Data!
 
Accelerate Return on Data
Accelerate Return on DataAccelerate Return on Data
Accelerate Return on Data
 
2010.03.16 Pollock.Edw2010.Modern D Ifor Warehousing
2010.03.16 Pollock.Edw2010.Modern D Ifor Warehousing2010.03.16 Pollock.Edw2010.Modern D Ifor Warehousing
2010.03.16 Pollock.Edw2010.Modern D Ifor Warehousing
 
2009.10.22 S308460 Cloud Data Services
2009.10.22 S308460  Cloud Data Services2009.10.22 S308460  Cloud Data Services
2009.10.22 S308460 Cloud Data Services
 

Recently uploaded

Orca: Nocode Graphical Editor for Container Orchestration
Orca: Nocode Graphical Editor for Container OrchestrationOrca: Nocode Graphical Editor for Container Orchestration
Orca: Nocode Graphical Editor for Container Orchestration
Pedro J. Molina
 
Kubernetes at Scale: Going Multi-Cluster with Istio
Kubernetes at Scale:  Going Multi-Cluster  with IstioKubernetes at Scale:  Going Multi-Cluster  with Istio
Kubernetes at Scale: Going Multi-Cluster with Istio
Severalnines
 
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdfThe Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
kalichargn70th171
 
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
dakas1
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
ervikas4
 
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
safelyiotech
 
Upturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in NashikUpturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in Nashik
Upturn India Technologies
 
DevOps Consulting Company | Hire DevOps Services
DevOps Consulting Company | Hire DevOps ServicesDevOps Consulting Company | Hire DevOps Services
DevOps Consulting Company | Hire DevOps Services
seospiralmantra
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
Patrick Weigel
 
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
Manyata Tech Park Bangalore_ Infrastructure, Facilities and MoreManyata Tech Park Bangalore_ Infrastructure, Facilities and More
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
narinav14
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
kgyxske
 
Liberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptxLiberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptx
Massimo Artizzu
 
What’s New in Odoo 17 – A Complete Roadmap
What’s New in Odoo 17 – A Complete RoadmapWhat’s New in Odoo 17 – A Complete Roadmap
What’s New in Odoo 17 – A Complete Roadmap
Envertis Software Solutions
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Paul Brebner
 
Using Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query PerformanceUsing Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query Performance
Grant Fritchey
 
Boost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management AppsBoost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management Apps
Jhone kinadey
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
sjcobrien
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
brainerhub1
 
Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...
Paul Brebner
 
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
kalichargn70th171
 

Recently uploaded (20)

Orca: Nocode Graphical Editor for Container Orchestration
Orca: Nocode Graphical Editor for Container OrchestrationOrca: Nocode Graphical Editor for Container Orchestration
Orca: Nocode Graphical Editor for Container Orchestration
 
Kubernetes at Scale: Going Multi-Cluster with Istio
Kubernetes at Scale:  Going Multi-Cluster  with IstioKubernetes at Scale:  Going Multi-Cluster  with Istio
Kubernetes at Scale: Going Multi-Cluster with Istio
 
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdfThe Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
 
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
 
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
 
Upturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in NashikUpturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in Nashik
 
DevOps Consulting Company | Hire DevOps Services
DevOps Consulting Company | Hire DevOps ServicesDevOps Consulting Company | Hire DevOps Services
DevOps Consulting Company | Hire DevOps Services
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
 
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
Manyata Tech Park Bangalore_ Infrastructure, Facilities and MoreManyata Tech Park Bangalore_ Infrastructure, Facilities and More
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
 
Liberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptxLiberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptx
 
What’s New in Odoo 17 – A Complete Roadmap
What’s New in Odoo 17 – A Complete RoadmapWhat’s New in Odoo 17 – A Complete Roadmap
What’s New in Odoo 17 – A Complete Roadmap
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
 
Using Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query PerformanceUsing Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query Performance
 
Boost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management AppsBoost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management Apps
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
 
Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...
 
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
 

Microservices Patterns with GoldenGate

  • 1. Microservices Patterns with GoldenGate: Deep-dive into Microservices Patterns with Replication and Stream Analytics O R A C L E D E V E L O P M E N T , M A Y - 2 0 2 0
  • 2. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
  • 3. Today’s Agenda Microservices and the Challenge of Data Data Mesh Concept, How GoldenGate Fits Microservice Patterns with GoldenGate Anyone Remember “Complex Event Processing”? Copyright © 2020 Oracle and/or its affiliates. 1 2 3 4
  • 4. Microservices are Good! 4 Service Mesh Revolution Emergence and widespread use of microservices have directly led to revolution in DevOps, massive uptake in Kubernetes and by 2020 the Service Mesh revolution Key Benefits: • Decomposition, of monolithic architecture • Modularity, smaller services and improved speed of initial development • Independence, loosely-coupled systems that can be created using different languages or data • With the loose coupling also comes much greater flexibility around deployment and upgrades, eliminating complex dependencies • Flexibility at Scale, deployments may start small, run locally and later scale very wide, running across multi-cloud environments and containers • Sidecar pattern for “Mesh” frameworks are further revolutionizing microservices Chris Richardson: https://www.nginx.com/blog/introduction-to-microservices/
  • 5. Dogma of Received Wisdom at the Data Tier 5 https://docs.microsoft.com/en-us/dotnet/architecture/microservices/architect- microservice-container-applications/data-sovereignty-per-microservice • “DB per Service” is a founding concept of Microservices Architectures • Some architects are more dogmatic than others: • “All microservices must have an isolated data store” vs. “Some microservices may share a data store” • At scale, usually means that microservice application must accept “eventual consistency” for data tier Graph of Microservices vs. Classical App Architecture “Drawbacks”: https://www.nginx.com/blog/introduction-to-microservices/ https://microservices.io/patterns/data/database-per-service.html Encapsulate the data to simplify system dependencies, create loose-coupling and empower a good CICD/DevOps process! Does this pattern always make sense? Mr. Microservice
  • 6. 6 https://en.wikipedia.org/wiki/Rorschach_test • Service Granularity, DDD offers a weak heuristic for defining the domain scope of a microservice (decomposition is hard) • Enterprise Domains are Big, for larger applications (ERP etc) domains are inherently large. domain decomposition can create artificial, subjective boundaries that proliferate incompatible business ontologies • Business Processes are Cross-Functional, a given business process may cut across many data domains, corporate business units and even 3rd party entities altogether (shared kernels and domain aggregates are needed) • Context Matters, reasonable differences of object definitions and taxonomy will naturally occur across large organizations (context mappings) • Legacy Applications, they exist. And, they don’t fit neatly into idealistic Bounded Context mappings (bubble contexts for Legacy) Context mapping is more art than science: http://www.ouarzy.com/2016/07/25/micro- service-and-bounded-context-clarification/ Context is in the mind of the modeler: However… Domain Driven Design (DDD) principles guide developers to create microservices that align to Bounded Contexts, which “defines tangible boundaries of applicability of some sub-domain” https://microservices.io/patterns/decomposition/decompose-by-subdomain.html Challenges with Bounded Context & DDD Sounds hard!
  • 7. Ledger Service Command Model Domain Event Hardest Part About Microservices: Your Data 7 Availability & Durability – what happens with the data when there is a service failure? How is the state of data saved and restored? (non-volatility) CAP Theorem & Network Partitions (CP or AP) • Strong Consistency – which services require guaranteed consistency, recovery and a single record of the truth? • Eventual Consistency – if a microservice end-point is unavailable, what is the retry protocol? Do all the services in the Saga/Process adhere to the same transaction semantics for retry and rollbacks? ACID – what is the business impact of data transactions ending in an inconsistent state? Will business processes fail? Are there any regulatory or compliance impacts? (ACID vs CAP, Google F1) findOrderHistory() findOrder() Order Events GL EventsShipping Events Logistics Service Command Model Domain Event Order Service Command Model Domain Event Order History Service Command Model Event Handlers Event Handlers Event Handlers Event Store many microservice design patterns have been created specifically to try and compensate for the data consistency guarantees that databases inherently provide Eg: CQRS and Event Sourcing DB encapsulation seems to shift the very hard data consistency issues to the App Tier… just accept “eventual consistency” and “lack of isolation” for transactions, and we’ll all be ok!
  • 8. Data and Events are First-Class Citizens 8 Service Service Service Analytics, Data Science and Data Lakes are too important to my business’ Digital Transformation and Data-Driven initiatives… we need architecture focus on Data and Events too Application Microservices Data Stores Event Logs produce events consume events produce events consume events read write App Events Data Events DB Log Events Control Plane • “State of the Truth” at a point in time (current or historic) • Durable storage used for Data Recovery / Archives/years of data • Polyglot, each service may determine its own data structures • “Narrative of the Truth” sequence of events (between data snapshots) • Days/months of event data available as Time Series or Messaging • Strict ordering of events & Idempotency • Strong Consistency of DB logs (eg; when using GoldenGate) • “Systems of Record” at application tier • APIs, business rules and business object semantics • Not durable storage The microservice API is king!
  • 9. Can we take some of the best ideas of a Service Mesh and apply them to Data and Events, to create a kind of Data Mesh? Microservice Mesh and Data Mesh 9 Immutable Raw Data Events Prepared Data Canonical Data App Events Data Events DB Log Events Data Domain Projection …n Data Domain Projection 1 Application Microservices Control Plane
  • 10. Today’s Agenda Microservices and the Challenge of Data Data Mesh Concept, How GoldenGate Fits Microservice Patterns with GoldenGate Anyone Remember “Complex Event Processing”? Copyright © 2020 Oracle and/or its affiliates. 1 2 3 4
  • 11. What is a Data Mesh? 11 Microservice Patterns Log-based Integrations Polyglot Data Movement Data Mesh is a data-tier architecture to integrate and govern enterprise data assets across distributed multi- cloud environments Microservices-centric: • For the administration, deployment and monitoring of the core frameworks of data movement and governance • Aligns with for Service Mesh frameworks (K8S, Istio, etc) • “Sidecar Proxy” style pattern for Events and Data Immutable event-logs for data integrations: • Messaging and data store events are globally accessible via immutable event logs • Logs may be used to drive Streaming or Batch integrations Distributed data movement of all types of data • A data mesh moves data: Relational, NoSQL, JSON, Graph… • Relational data consistency (ACID) during data movement • Must work reliably with enterprise OLTP data sets https://en.wikipedia.org/wiki/ACID Data Mesh Event Streaming Immutable Logs Data Replication Polyglot Persistence Edge / 5G Frameworks Domain Driven Design Service Mesh “Sidecars” Data Mesh
  • 12. Evolution towards Real-Time Data Mesh Copyright © 2020 Oracle and/or its affiliates. Industry 3.0: Hub and Spoke Transitional: Kappa Hub Mature: Distributed Kappa This data pattern, popularized by Ralph Kimball and Bill Inmon, has been the foundation for enterprise data management since 1993. It is transaction consistent, can scale up nicely for most use cases, and is based on SQL, lingua-franca for most tools. By 2010, the Lambda (big data) pattern was common. In 2014, Jay Kreps (of LinkedIn) questioned the Lambda Architecture and spawned Kappa. The Kappa principles consider batch processing as a special case of stream processing. Use a historized event log to process both real-time as well as batch processing. By 2020, IT infrastructure has dramatically changed – networking, containers, cloud, compute, IoT etc have all pushed data to the edge. A mature Kappa architecture is not a single instance “hub” but rather a distributed mesh of data logs, stream data processing, change events, and time series data. Kappa: https://www.oreilly.com/radar/questioning-the-lambda-architecture/ https://en.wikipedia.org/wiki/Dimensional_modeling mesh & microservice controls 12 ETL ETL ETL ETL Lambda: http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html Monoliths Distributed
  • 13. Data Mesh Conceptual View – Data Domains 13Copyright © 2020 Oracle and/or its affiliates. Enterprise Data Producers: ERP Apps, DBs, Middleware etc. Data Domain Consumers People owners of “Data Products”, collections of data sets in various stages of curation IoT Data Producers: Devices & Things Raw Data Prepared Data Canonical Data Data Domain A Data Domain B f(x) f(x) Data Domain C Data Mesh (distributed Kappa, microservices, cloud agnostic) Domain-Specific Views of Data Raw Event Consumers Automated Devices, Edge Nodes (5G), Scheduled Routines (eg; ETL etc) Data Product-Specific Storage Choices: • RDBMS • Data Lake • Object Store • Graph, etc.
  • 14. Raw data, Time Series & Alerting events are pushed Direct to Database (high fidelity transaction semantics fully preserved) Consumer-Driven, Event-Centric Data Mesh Copyright © 2020 Oracle and/or its affiliates. Enterprise Data Producers Detect Event Logical Change Records (LCRs) App DB committed! CDC Replication Data Domain Consumers Data Objects Table Data Raw Data / Alerts SQL Consumers Raw Data Prepared Data Canonical Data Raw Data (LCR) Schema Events (DDL) Prepared Data Topics “Master” Data Topics JSON, XML, Avro, Parquet, CSV Prepared data events are pushed Canonical data events Speed & Fidelity Trusted Views Ease of Consumption LCR/TFs Applications, Data Services Biz Consumers Analytics & Data Marts Data Science & Streaming Applications DBAs for HA, DR and OLTP Data Mesh puts the consumer needs first – they require data at different latency, fidelity, trust levels and views Data Model Object Model System Of Record (SoR) User Action App APIs and system log events 14
  • 15. Direct to Database (high fidelity transaction semantics fully preserved) Distributed by Design, Microservices Based Copyright © 2020 Oracle and/or its affiliates. Data Domain Producers Detect Event Logical Change Records (LCRs) App DB committed! Data Domain Consumers Data Objects Table Data Raw Data / Alerts SQL Consumers Data Model Object Model System Of Record (SoR) User Action CDC Replication Microservices Edge Compute or Cloud for Raw Data Events Prepare Technical Data Views LCRs Business Data Views Raw data, Time Series & Alerting events are pushed Prepared data events are pushed Canonical data Events (ephemeral or persisted) Stream Process Events (persisted) Stream Process Events (persisted) Applications, Data Services Biz Consumers Analytics & Data Marts Data Science & Streaming Applications DBAs for HA, DR and OLTP 15
  • 16. Single Pane of Glass for Real-Time Data Mesh Copyright © 2020 Oracle and/or its affiliates. connect DB2/z Data Objects Table Data Raw Data / Alerts SQL Consumers Applications, Data Services Biz Consumers Analytics & Data Marts Data Science & Streaming Applications DBAs for HA, DR and OLTP Real-Time Stream Data Processing Raw Data DBAs & Data Engineers Data Owners & Data Products 16 Data Consumer DrivenEvent Centric Pipelines Deploys in a Mesh Across Containers, Public Clouds and 5G Edge Devices
  • 17. How it Works Today: GoldenGate for Big Data Copyright © 2020 Oracle and/or its affiliates. Data Domain Consumers Data Objects Table Data Raw Data / Alerts SQL Consumers Applications, Data Services Biz Consumers Analytics & Data Marts Data Science & Streaming Applications DBAs for HA, DR and OLTP BYOS (Bring Your Own Spark) * distributed, may run on any combination of containers and clouds 17 Data Engineer Data AnalystDBA/GG Ops Capture Pipeline Analyze DeliverIngest GoldenGate Microservices Applications Stream Analytics Application BYOM (Bring Your Own Messaging)All Data Events & Transactions
  • 18. What Exactly Does GoldenGate Do? Copyright © 2020 Oracle and/or its affiliates. Database DBA ETL Tools continuous stream of events Database Events: • Transaction Commits (DML) • Schema Changes (DDL) • Procedure Execution Any Data Store Target • Databases (any) • Non-Relational • NoSQL, HDFS, etc • Search (Elastic…) • Cloud (Obj Store…) • Messaging • Kafka, JMS etc. • Cloud (any) GoldenGate Trail (protocol) Online Redo Logs 18 App Services S Q L S Q L S Q L GoldenGate Replicats, Event Handlers, and Data Formatters: Core Concept: DB Event Log Customer Data Formats Business Events: Committed Events 1 2 3 4
  • 19. Today’s Agenda Microservices and the Challenge of Data Data Mesh Concept, How GoldenGate Fits Microservice Patterns with GoldenGate Anyone Remember “Complex Event Processing”? Copyright © 2020 Oracle and/or its affiliates. 1 2 3 4
  • 20. First Things First – GoldenGate Microservices 20 data replications bi-directional ms/sec updates consistency guarantees Cloud, Containers or Edge Devices Extracts Replicats Client Libraries Native GUIs and Full REST APIs API Gateway or Proxy Service GG GUI GoldenGate is itself a set of microservices that human users or other services may interact with Embedded User Interface: • C-based microservices with embedded HTTP client for native JavaScript based GUI • Oracle Jet frameworks for intuitive interaction model REST native APIs: • Fully REST native • Also available, a Command Line Interface (CLI) produces REST calls to the native services Full GoldenGate Replication Capabilities: • 100% coverage for all traditional GG replication patterns; fully capable of HA/DR use cases GG Admin Service GG Distro Service GG Metrics Service GG Receiver Service GG Service Manager Your Services
  • 21. Microservice Design Patterns for Data 21 Patterns for MicroservicesInherent to the Microservice Architecture is the developer using specific patterns, sometimes the patterns are partially embodied in a Programming Framework, but typically the developers must choose to follow certain heuristics while programming. This presentation’s focus: • “Database Patterns” & “Integration Patterns” …using DB Event Replication (AKA: Change Data Capture) to improve them • Simplify the pattern, make the microservice application more resilient and provide better data consistency guarantees DB Patterns for Discussion: • Database per Service (covered earlier) • CQRS – Command Query Responsibility Segregation • Event Sourcing • Saga Pattern • Transactional Outbox • Aggregates (AKA: Domain Events) Transaction Outbox https://medium.com/@madhukaudantha/microservice-architecture-and-design-patterns-for-microservices-e0e5013fd58a
  • 22. Transaction Outbox Copyright © 2020 Oracle and/or its affiliates. https://microservices.io/patterns/data/transactional-outbox.html App Tables outboxApp TablesApp Tables JSON Payload Order Service Object Storage Update DB *and* Send Domain Events: • Keep the DB Update and Event as atomic Other Context: • “Message Relay” should be Idempotent / Exactly Once in order to assure that the recipient of the events are not receiving duplicates • “Message Relay” concept may use (1) polling events, or (2) tail the DB transaction logs (AKA: Changed Data Capture, CDC) XYZ Service Using Oracle GoldenGate: • …GG is an exactly-once “Message Relay” microservice • …also adds benefit of maintaining Transaction Consistency • …millisecond latencies, trusted as a DR solution for OLTP • …flexible “payloads” and “broker” variations (see next page)
  • 23. Baseline Tx Outbox Pattern (microservices.io) Outbox Without the Middleman (No Broker) Outbox Payload is Tables (not Documents) Outbox from Materialized View Transaction Outbox Variations Copyright © 2020 Oracle and/or its affiliates. outboxApp Tables JSON Payload Order Service Message Broker outboxApp Tables JSON Payload Order Service XYZ Service JSON Payload JSON Payload App Tables Order Service Doc Payload Message Broker App Tables XYZ Service JSON Payload Object Storage Doc Payload • Full fidelity database raw data is needed • Sharding, Data Science, ODS/EDW, etc… • Data binding becomes shape of table rather than shape of Service-defined doc • GG Formatters put Table data into Broker as JSON, Avro, Parquet, XML, etc. • Payload centric, may be used with Aggregates pattern • Developer of microservice defines Payload signature • Best for smaller services • See also: CQRS and Event Sourcing • Point-to-point replication rather than Pub/Sub • Orchestration-based Sagas • Singleton use cases; eg; always copying service events to a data lake • Useful for micro-monoliths and monoliths; DB’s with very large or very 3NF schema • Views are “interface tables” for external data consumers • Views present business-friendly data objects • Views must be maintained by DBAs Object Storage JSON Payload App Tables Order Service View Tables Object Storage View Tables Doc Payload Doc Payload XYZ ABC
  • 24. Transaction Outbox for the Whole Enterprise Copyright © 2019 Oracle and/or its affiliates. DB2/z Replication of Real-time Data Transactions & Events GoldenGate Stream Analytics ETL &ML Object Storage Relational Non- Relational Apps https://www.oracle.com/middleware/technologies/goldengate.html Microservices- Based DBMS Cloud Big Data NoSQL Streams
  • 25. Order History Logistics Service Order Service Ledger Service CQRS Pattern Command Query Responsibility Segregation Copyright © 2020 Oracle and/or its affiliates. Query Model Command Model SQL Select Read Side Write Side Inserts, Updates etc. Basic CQRS (https://martinfowler.com/bliki/CQRS.html) findOrderHistory() findOrder() Event Handlers Multi-Service Example (https://microservices.io/patterns/data/cqrs.html) GL Events Shipping Events Order Events • Separation of duties: • Writes (Commands) – go against Domain Model • Reads – optimized for client Queries • Alternative to typical “CRUD” type interfaces for Microservice API / signatures • Popular when: • Complex Domains – require queries against many domain objects at same time • High Performance – to support highly optimized read interfaces • Complex Domains: • Multiple services must update common “view” • Clients (GUIs) may run queries that require many joins and/or lots of records fetched write read User Interface GUI
  • 26. Order History Order Service Simplified CQRS with Transaction Outbox Copyright © 2020 Oracle and/or its affiliates. findOrderHistory() findOrder() Delegate the data event propagation: • Instead of having the developers (of the microservices) write the logic, handlers, and failure conditions for eventing → leverage data- tier event replication to propagate the DML events to the “view” data store Avoid complicated and costly mistakes: • CAP Theorem is a hard problem to code for in the microservice itself: • What happens if the service fails just after a local update but before the event is sent? • What if the “view” Service receives the event but the data store fails? • What if the event volume / workload spikes and the “view” Service goes down? Data tier “logical replication” is designed to be very fast, work across WAN/Clouds and to maintain consistency: • Relational semantics may involve long-running transactions, foreign key dependent DML operations, and strict transaction ordering requirements write User Interface ms/sec updates consistency guarantees Order Events read
  • 27. Order Service Logistics Service Ledger Service Order History Simplified CQRS Pattern Copyright © 2020 Oracle and/or its affiliates. findOrderHistory() findOrder() Event Handlers Order EventsCommand Model Command Model Command Model Event Store Event Handlers Event Handlers GL Events Shipping Events Domain Event Domain Event Domain Event Before After • Developer responsible for correct implementation of CAP Theorem • Data consistency on any outage (microservice / data tier) • More code to write and maintain • Worse performance (latency) • Impedance, across many services into view model (semantic misalignment) ✓ Consistency and recoverability guarantees ✓ Less code to write or maintain ✓ Best possible performance write read User Interface User Interface Order History Logistics Service Order Service Ledger Service write read
  • 28. Event Sourcing Pattern Copyright © 2020 Oracle and/or its affiliates. BIG topic in the Microservice community: • …and, a point of confusion as well Pattern espoused for: • An alternate durable data store (instead of a Database) to store a services’ state, and/or… In combination with a local Database, where the Event Store is use primarily for communication of events (and not for saving state) • A way to record a sequence of immutable app events to be used later by the service which generated the events, and/or (when Event Store is shared) by other services who wish to subscribe to events that other services create Implementations: • DIY using your favorite language; purpose-built event stores like Eventuate, Axon, Akka or Event Store • Apache Kafka, Apache Pulsar, RabbitMQ, or NoSQL engines As a Durable Store ++ For Event Communications (only) Local Events Shared Events Event Store ➢ ALL events… ➢ Order Created ➢ Order Approved ➢ … ➢ Order Shipped Event Store all durable events ➢ only the ➢ events that ➢ need to be ➢ communicated ➢ …. Order Events Logistics Events Ledger Events Event Store Order Events Logistics Events Ledger Events Event Store Circa 2005: https://martinfowler.com/eaaDev/EventSourcing.html • https://microservices.io/event-sourcing.html • https://microsoft.com/patterns/event-sourcing https://dzone.com/articles/microservices-with- cqrs-and-event-sourcing https://www.infoq.com/news/2017/11/eve nt-sourcing-microservices/ Order Service Order Service Order Service Logistics Service Ledger Service Order Service Logistics Service Ledger Service
  • 29. Critiques of Event Sourcing Copyright © 2020 Oracle and/or its affiliates. Exposing the Persistence Tier: • Taken too far (Why Event Sourcing is an Anti-Pattern), developers wind up using the Event Store as a Shared Persistence model, and other microservice now have hard- coupled binding to the message formats of the originating service Whole System Fallacy: • Some microservices leaders (Udi and Greg Reach CQRS Agreement) say to narrow the aperture on when to use CQRS + Event Sourcing → only within a Business Component and a Single Bounded Context • Minimizes utility of pattern for Communications Forcing Eventual Consistency on Developers: • The propensity to over-use CQRS & Event Sourcing at the at the whole system level forces developers to manage eventual consistency in the Application tiers (What they don’t tell you about event sourcing) • “…they will make your life a living hell” doing DevOps, debugging and system recovery when a “Mesh” of services are interacting via Event Store and message signatures can lead to disaster
  • 30. Event Sourcing with Transaction Outbox Copyright © 2020 Oracle and/or its affiliates. Before After Service Service Service Application Microservices Data Stores Events produce events consume events produce events consume events read write App Events Data Events DB Log Events Control Plane Data Mesh https://dzone.com/articles/microservices-with-cqrs-and-event-sourcing • Microservice developers must code for consistency • Data may be stale or incorrect • Many tiers for failure, difficult to trouble-shoot • Useful for “communications” and “shared events” requirements • Delegate data tier consistency to “Events” tier • Using DB Replication tools (eg; GoldenGate) for ACID • Events may be used “system-wide” rather than per Context per ‘bounded context’ and not used system- wide for communications Domain z Domain X Domain Y
  • 31. Saga Pattern Copyright © 2020 Oracle and/or its affiliates. https://livebook.manning.com/book/microservices-patterns/chapter-4/17 Popular, long-standing pattern: • First paper back in 1987 (Garcia-Molina and Salem, 1987) • Used for long-lived transactions in distributed systems, that cannot be completed using multi-phase commits In Microservices: • Frequent requirement due to “one DB per service” pattern • Multi-phase commits at microservice tier are uncommon • Typically see either (a) Choreography or (b) Orchestration based approaches (https://microservices.io/patterns/data/saga.html) Drawbacks: • Lack of Isolation (ACD not ACID) and Eventual Consistency lag, system state can be inconsistent for seconds, minutes or hours • Complex programming model, depends on “Compensating Transactions” • Requires other Patterns (eg; Event Sourcing, Aggregates) for maintaining DB consistency Additional logic for compensation / rollbacksChoreography or Orchestration logic
  • 32. We don’t want to lose any events and leave the system inconsistent. How to atomically update the store and send the event? Saga Pattern with Transactional Outbox Copyright © 2020 Oracle and/or its affiliates. Ticket Service Insurance Service Payment Service Outbox Pattern Outbox Pattern Outbox Pattern Event Hub Ticket Topic Order Topic Payment Topic Publish Events: Subscribe Events: Red Hat Summit 2019: The New Era of Transactions… Saga Approach: • Choreography Outbox with Saga: • Publish committed events to event hub (eg; Kafka or other); very low risk of inconsistency due to network or service failures (Dzone: Data Consistency…) • GoldenGate enforces Idempotency / Exactly Once and may preserve intra-table Consistency as well Subscribers: • All services in the Saga can be assured that events in the Hub are fully committed • Compensating Txs happen per normal Saga pattern guidelines
  • 33. Broker Aggregate is a collection of objects grouped together: • Typically grouped by Transaction boundaries • Used for local persistence • *within* a specific Bounded Context (always!) Motivation: • A core purpose of this pattern (within a specific context) is to ensure that invariants and domain rules are always applied to the whole transaction holistically Domain Event (Aggregate) vs Integration Event Copyright © 2020 Oracle and/or its affiliates. Domain Service Aggregate 1 Aggregate 2 Root Entity Entity Value Domain Event GUI Domain Context xyz Order Events Chris Richardson: Using Aggregates with Events and CQRS Integration Event is a collection of objects grouped together: • Also, typically grouped by Transaction boundaries • Used mainly for events and external communications • Usually designed for bindings outside of local Context See Also: • This pattern is mainly focused on the organization of data posted via Event Sourcing, Saga, or Transaction Outbox Domain Service Aggregate 1 Aggregate 2 Root Entity Entity Value Domain Event GUI Domain Context xyz Microsoft Patterns: Using Integration Events (with Event Brokers) Integration Event
  • 34. Non-Relational Integrations Data Lake Ingest Streaming Ingest Cloud Ingest Messaging Replication NoSQL Replication SaaS Replication DB Event Replication Unidirectional Bi-Directional Peer-to-Peer Broadcast Consolidation Distribution Event Stream Processing Data Pipelines Data Transformation GoldenGate Integrations Time Series Analysis Geo-Fencing Predictive Analytics Integration Events with GoldenGate Copyright © 2020 Oracle and/or its affiliates. 34
  • 35. Today’s Agenda Microservices and the Challenge of Data Data Mesh Concept, How GoldenGate Fits Microservice Patterns with GoldenGate Anyone Remember “Complex Event Processing”? Copyright © 2020 Oracle and/or its affiliates. 1 2 3 4
  • 36. Hype Cycle 2009 Copyright © 2020 Oracle and/or its affiliates. Complex Event Processing: • CEP is a kind of computing in which incoming data about events is distilled into more useful, higher level event data that provides insight into what is happening. […] CEP is used for highly demanding, continuous-intelligence applications that enhance situation awareness and support real-time decisions. Gartner 20 Years Too Early? • CEP dates back to the 1990’s (history of CEP engines) • CEP came before “Event Stream Processing” and generally has covered more complex use cases (eg; handling of out- of-order events and more complicated correlation semantics) (what’s the difference, 2019 and mythbuster CEP vs ESP, 2008) • Largely overtaken by Big Data stream processing technologies that are open-source, massively-parallel, and widely available as cloud-native 2009!
  • 37. Stream Processing 2020 Copyright © 2020 Oracle and/or its affiliates. Time Series DB/OLAP Big Data Event Stream Processing Complex Event Processing Becoming more aligned to open source / apache frameworks Becoming more capable of rich windowing functions and time-clock correlation semantics Complex Event Processing: • Traditionally running in “scale-up” SMP in-memory framework • Many CEP engines are moving toward MPP “scale-out” architectures • Programming-centric historically, some CEP engines were also query- centric as far back as 2009 • Time-clock semantics and advanced cache may be used for “state machine” type use cases Stream Processing: • Built around MPP frameworks and typically Apache open-sourced • Genesis was around simplistic use cases on high volumes of events • All SP engines began with rudimentary windowing and correlation semantics, but most frameworks are gradually becoming more functionally comparable to classic CEP • Simplistic windowing and caching for basic stream-clock events Time Series Databases: • Optimized for persistence and historic analytics on time-ordered events…data is often sourced from CEP or SP engines Technical and Functional Differences between CEP and SP: https://complexevents.com/2019/07/15/whats-the-difference-between-esp-and-cep-2/
  • 38. Stream Processing/CEP for Event Driven Architectures Copyright © 2020 Oracle and/or its affiliates. There has been a widespread awakening to the benefits of Event Drive Architecture (EDA) for increasing the scalability and agility of business systems. […] Stream analytics is based on the mathematics of complex-event processing (CEP). CEP is a computing technique in which incoming data about what is happening (event data) is processed as it arrives (data in motion or recently in motion) to generate higher level, more useful, summary information (complex events). W. Roy Schulte (of Gartner), March 2020: EDA is Suddenly Popular Will Stream Analytics be Next? Event Stream Analytics (& CEP) Data & Microservice Events Event/Data Pipelines Time-Series Analysis Geospatial Analysis Real-time AI/ML Continious ETL Use Cases:
  • 39. Oracle GoldenGate Stream Analytics 39Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2018 Ingest Events Select Processing Patterns Build Event Pipelines Serve Data Downstream GoldenGate events from any DB or NoSQL; also supports events from Kafka, JMS, AQ and MQTT Rich set of pre-built patterns will dramatically improve developer efficiency and time-to-value Easily leverage geo-fencing, machine-learning, and other reference data within the stream Data can be delivered out to Kafka, databases, or easily staged for external ETL jobs Data Owners & Data Products
  • 40. Significant Intellectual Property Copyright © 2020 Oracle and/or its affiliates. More than 70 patents on stream processing Mature tech stack for Event Processing Over 10 years of IP investment Continuous Query Language (CQL) 12.2 12.3 18c 19c 11g
  • 41. GoldenGate Stream Analytics Copyright © 2020 Oracle and/or its affiliates. Interactive, Low-Code Designer Time Series Analytics Event Driven Dashboards Patterns / Accelerators Geo-Spatial Analysis & Geo-Fencing Predictive Analytics / ML
  • 42. Upgrade to Spark, Better than Flink or KSQL Copyright © 2019 Oracle and/or its affiliates. [best] [worst] GoldenGate Spark Streaming Apache Flink / Beam Confluent KSQL User Experience Low Code Development (with built-in patterns/accelerators)     Interactive/Live Edits (browser based, view changes immediately)     Built-in Live Dashboards (event-driven charts/graphs)     Core Streaming Semantics What is Being Computed (transforms, joins, flatten, statefulness etc)     Time Windows(global, fixed, sliding, tumbling, custom etc)     When in Processing Time (triggers – event, time, count, timers, etc)     How do Refinements Relate (discarding, accumulating, retracting)     Analytics Robust CEP Capabilities (complex event correlations, native time clock)     Geo-Fencing & Spatial (lat/long, built in maps, custom map tiles, etc)     Machine Learning (native scala, PMML, python support etc)     Time Series Analysis (built-in interval patterns, thresholding etc)     Other Features Backpressure (dynamic ingest per pipeline) Automatic Custom Custom Custom State Management (automation across streams & native cache) Data Grid N/A RocksDB RocksDB Data Consistency (OLTP Change Events, Inserts/Updates/Deletes) Automatic Custom Custom Custom GoldenGate Stream Type (aware of SCN/CSN, transactions, order, etc) Native Custom Custom Custom
  • 43. Today’s Agenda Microservices and the Challenge of Data Data Mesh Concept, How GoldenGate Fits Microservice Patterns with GoldenGate Anyone Remember “Complex Event Processing”? Conclusion! Copyright © 2020 Oracle and/or its affiliates. 1 2 3 4 5
  • 44. GoldenGate Stream Analytics for Data Mesh Copyright © 2020 Oracle and/or its affiliates. connect DB2/z Data Objects Table Data Raw Data / Alerts SQL Consumers Applications, Data Services Biz Consumers Analytics & Data Marts Data Science & Streaming Applications DBAs for HA, DR and OLTP Real-Time Stream Data Processing Raw Data DBAs & Data Engineers Data Owners & Data Products 44 Data Consumer DrivenEvent Centric Pipelines Deploys in a Mesh Across Containers, Public Clouds and 5G Edge Devices
  • 45. This is not a Metamorphosis, it is a Paradigm Shift Copyright © 2020 Oracle and/or its affiliates. The Success Paradox Data success factors that did well in Industry 3.0 will not be the factors that create success in Industry 4.0 Next Gen Data Architecture ETL Vendors 1990 – 2010’s Gen1 : • Replication • Messaging • Streaming • Pipelines Next-Gen has new DNA not tied to old ETL tools It is impossible to evolve older Batch Processing tools into a modern Event- Centric Stream Processing solution; the underlying paradigms are fundamentally different 45
  • 46. 46Copyright © 2020 Oracle and/or its affiliates. Questions?
  • 47.
  • 48. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. Copyright © 2020 Oracle and/or its affiliates.