Flash session -streaming--ses1243-lon

OpenWorld London:
Streaming Data with GoldenGate
O R A C L E D E V E L O P M E N T , F E B - 2 0 2 0

The Problems of Monolithic Data Architecture
Copyright © 2020 Oracle and/or its affiliates.
People Process Technology
• Business units may have few
incentives work across boundaries
• Hyper-specialization in tech teams
narrow the focus on technology
rather than outcomes or solutions
• Pressure on all stakeholders to
produce value, but org structures
still a carry-over from older EDW
projects
• The classical Lambda/Kappa
“Ingest -> Process -> Serve” design
institutionalizes Batch Processing
into the team processes
• Conceptually it is still “Extract -
> Transform -> Load” but with
other words/syntax
• The monolithic data lake is big and
slow by design, not by accident
• Architecture decomposition
happens at several layers, but Big
Data is inarguably “storage centric”
• From HDFS (Hadoop) to Object
Storage (Cloud) the classical
approach is “the Lake” as a physical
area where we pile up data
• But data is not static, data is in a
constant state of dynamic
equilibrium
Images: https://martinfowler.com/articles/data-monolith-to-mesh.html
2

Evolution towards Real-Time Data Mesh
Industry 3.0: Hub and Spoke Transitional: Kappa Hub Mature: Distributed Kappa
This data pattern, popularized by Ralph
Kimball and Bill Inmon, has been the
foundation for enterprise data
management since 1993.
It is transaction consistent, can scale up
nicely for most use cases, and is based
on SQL, lingua-franca for most tools.
In 2014, Jay Kreps (of LinkedIn)
questioned the Lambda Architecture
and spawned Kappa.
The Kappa principles consider batch
processing as a special case of stream
processing. Use a historized event log
to process both real-time as well as
batch processing.
By 2020, IT infrastructure has
dramatically changed – networking,
containers, cloud, compute, IoT etc
have all pushed data to the edge.
A mature Kappa architecture is not a
single instance “hub” but rather a
distributed mesh of data logs, stream
data processing, change events, and
time series data.
https://www.oreilly.com/radar/questioning-the-lambda-architecture/https://en.wikipedia.org/wiki/Dimensional_modeling
mesh & microservice controls
3

This is not a Metamorphosis, it is a Paradigm Shift
The Success Paradox
Data success factors that did well in
Industry 3.0 will not be the factors
that create success in Industry 4.0
Next Gen Data Architecture
ETL Vendors
1990 – 2010’s Gen1 :
• Replication
• Messaging
• Streaming
• Pipelines
Next-Gen has
new DNA not
tied to ETL tools
It is impossible to evolve older Batch
Processing tools into a modern Event-
Centric Stream Processing solution; the
underlying paradigms are fundamentally
different
4

Widespread Adoption of New Paradigms
Data
Consumers
First
Note: the data science community is a leader in
popularizing the “Data as a Product” concept
(https://hbr.org/2018/10/how-to-build-great-
data-products) in part because most data scientists
are not domain experts but need to work closely
with those who are; this is also a common problem
for data engineers and DBAs
Databases
(& Data Lakes)
Inside Out
Note: the LinkedIn/Confluent/Kafka innovators
should get credit for popularizing this concept
(https://engineering.linkedin.com/distributed-
systems/log-what-every-software-engineer-
should-know-about-real-time-datas-unifying), but
Kafka as a technology still mostly fails at this goal
due to lossy transaction semantics and
unpredictable failure modes (eg; Kafka is not a DB)
Push, Not Pull Note: we’ll take some credit here, for more than
20yrs the GoldenGate team has been a pioneer
(http://www.oracle.com/us/products/middleware
/data-integration/oracle-goldengate-innovations-
wp-5093027.pdf) on the cutting-edge helping
users retire batch processing and shift to push-
style, event-based, real-time data integration in
the most challenging (DB High Availability, Data
Warehousing, Data Lake) use cases.
https://noti.st/rmoff/LSemuJ/embed
Big Idea:
think of data a
continuous
Change Log
rather than State
at a point-in-time
Big Idea:
batch processing
is a special-case
of event streams,
once you have
streams you can
also batch
Big Idea:
work backwards
from the needs of
a data consumer,
instead of from
the Ingest stage or
a dimensional
model
Needs of the consumer for self-service data
5

The Next Evolutionary Paradigm:
An Enterprise Platform for Real-Time Data Mesh
Copyright © 2020 Oracle and/or its affiliates. 6

Real-Time Data Mesh for Industry 4.0
…from Industry 3.0 …to Industry 4.0
Batch Centric, Schedulers Event Centric, Streams
Mostly Relational Data (aka Views) Polyglot Data (via Logs)
Size for Peak Workloads Elastic, Scale on Demand
Kimball / Inmon Architecture Kappa / Pipelines Architecture
Vendor Specific Open Source Foundation
Simplex Processing is Standard Massively Parallel is Standard
Hubs (EDW, Hadoop, Data Lake) Mesh (Edge, Hybrid, Multi-Cloud)
Governance is “Bolt On” Governance is Embedded
7

Significant Intellectual Property
More than 70 patents on stream processing
Mature tech stack for Event Processing
Over 10 years of IP investment
12.2
12.3
18c
19c
11g

Single Pane of Glass for Real-Time Data Mesh
connect
DB2/z
Data Domain
Consumers
Data
Objects
Table
Data
Raw Data
/ Alerts
SQL
Consumers
Data Mesh puts the consumer
needs first – they require data
at different latency, fidelity,
trust levels and views
Applications,
Data Services
Biz Consumers
Analytics &
Data Marts
Data Science
& Streaming
Applications
DBAs for HA,
DR and OLTP
Real-Time Stream
Data Processing
Raw
Data
Data Engineers Data Owners
* distributed, may run on any combination of containers and clouds
9

Platform for Real-Time Data Mesh
Database
Replication
Non-Relational
Replication
Stream
Processing
Pre-Built Patterns / Templates
Communications
Deployment Models
Governance Operations
Key Personas / Single Pane of Glass
10

High Growth Patterns:
Replication in/out
for Non-Relational
Data Lake Ingest Streaming Ingest Cloud Ingest Messaging Replication NoSQL Replication SaaS Replication
Foundation Patterns:
Database
Replication
Unidirectional Bi-Directional Peer-to-Peer Broadcast Consolidation Distribution
Expansionary Patterns:
Stream Processing
Data Pipelines Data Transformation GoldenGate Integrations Time Series Analysis Geo-Fencing Predictive Analytics
Real-Time Data Mesh Platform Capabilities

Insights with Stream
Processing and Oracle
GoldenGate
12

Interactive Browser-based Designer
Accessible to Non-Technical Users
• Empower data analysts to enhance data with
no coding skills required
• Intuitive, always-on data view shows results of
transformations as they are defined
• Filter and correlate streams, apply rules,
aggregate, calculate fields etc.
Function extensibility via Java
• Allow data engineers to provide custom
stages and functions to be used by all team
members
Integrated Visualizations
• Explore your business data live through
various tables, charts and geospatial maps

Rich Set of Streaming Patterns
Simplify Access to Complex Algorithms
• Easy-to-use modules with user assistance in the
designer
• Pre-defined visualizations to provide immediate
feedback
• Accessible to data analysts
Comprehensive Library of Patterns
• Covers diverse areas such as anomaly detection,
stream correlation, trend analysis, spatial functions
• Duplicate, out-of-order, and missing event
detection
• Functions for financial, statistic, and log analytic
operations

Location and Geo-Spatial Capabilities
Interactive Spatial Design and Visualization
• Show live location data on maps as events are
processed
• Track individual objects and highlight them based
on different conditions, e.g. Red for violation
Rich Geospatial Pattern Set
• Correlate multiple objects through their spatial
interaction
• Detect speed, and proximity
• Obtain address and city information from location
and vice versa through Geocoding
Scalable Definition of Areas and Geo-Fences
• Define polygons through drawing borders on a map
• Manage large amounts of shapes through spatial
types in Oracle database.

Time Series Analytics
Anatomy of a Time Series Pattern
Built in Patterns for Anomalies
• Pre-defined visualizations to provide
immediate feedback
• Accessible to data analysts
GoldenGate Supplies High Fidelity Events
• Every database commit, logical change
record, schema and procedure event is
visible in the event stream
• Combine with application logs for full picture
Examples
• Banking, credit card transactions, trades…
• Sales and Marketing Data (eCommerce)
• IoT, Telemetry, Devices, Smart Home
• Monitoring data, data centers, networks etc
• Science/medicine, EEG, ECG, DNA
• Social networks, likes, classification, trends

Predictive Analysis and Machine Learning
Real-time Scoring and Decision Making
• Use Machine Learning models to make business
decisions in real-time
• Predict future outcomes such as equipment failures,
customer behavior, fraud and security breaches
• Re-import refined models for improved predictions
Put Data Science in Production
• Import Predictive Models created by data scientists and
engineers in their own environment.
• Import of PMML models for a variety of algorithms such as
vector machines, association rules, Naive Bayes classifier,
clustering models, text models, decision trees, and different
regression models.
• Hide model complexity for use by data analysts
• Custom stages for access to external scoring systems Oracle R
Enterprise
Notebooks
(Jupyter,
Zeppelin, etc)
Data Scientist
Data Analyst/
Data Engineer

Built-in Dashboards
Visualizations Built-In
•Not intended to be a replacement for
purpose-built Data Visualization tools,
•OSA includes some visualizations to
support building graphs on data that is
streaming in-memory (before writing to
Data Store), including:
• Bar Charts
• Line Charts
• Geo-Spatial (Google Maps)
• Area Charts
• Pie Charts
• Scatter Charts
• Bubble Charts
• Thematic Maps

SaaS/ERP Replication & Stream Processing
Messages,
Events & Alerts
Analytics
Data
{rest}
Data
Lake
connect
Ingest Pipeline Analyze Deliver
Financials
Supply Chain
Human Capital
Customer Engagement
Merchandising
Point of Service
PushTopics API
Enterprise Messaging

Stream Analytics Cloud – Get Started in 5 Minutes
OCI Marketplace
GG Database
Replication
Open
Source
Stream
Analytics
GG for
Big Data
OCI Compute (any shape)
OCI Block Store
Trail
File
Kafka
Topics
MySQL
Store
Messages,
Events & Alerts
Analytics
Data
{rest}
Data
Lake
LOW LATENCY HIGH SPEED LOW CODE

Oracle GoldenGate | Everyday Moments
• Swipe a Visa, Mastercard or American Express credit card
• Use Starbucks loyalty card to buy a Latte
• Pay for a purchase with PayPal
• Withdraw cash at an ATM from Bank of America, or Citibank, or Wells Fargo,
or CapitalOne, or American Express, or JP Morgan Chase, or… most other
banks in the world
• Click on a Google AdWords link
• Select a recommended item from EBay.com
• Ship a package with FedEx, UPS, USPS or DHL
• Use a store coupon from Safeway
• Browse the catalog from Walmart or Overstock.com
• Check flight availability on American Airlines
• Use the Cartwheel App at Target stores
• Take your General Motors car in for service
• Go shopping at Macy’s online store on Black Friday
• Deposit your payroll check issued by Paychex
• Take care of your small business accounting on Quickbooks.com
• Buy a song on Apple iTunes
• Change your profile or add a skill on LinkedIn.com
7,500+ Customers in >180 Countries

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

Flash session -streaming--ses1243-lon

Flash session -streaming--ses1243-lon

More Related Content

What's hot

Similar to Flash session -streaming--ses1243-lon

More from Jeffrey T. Pollock

Recently uploaded

Flash session -streaming--ses1243-lon