Oracle Stream Analytics - Developer Introduction

1
Oracle Stream Analytics
Overview
Speakers
Jeff Pollock, VP Product Development
Copyright © 2019 Oracle and/or its affiliates.

The following is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September 2019
and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor

Industry and Use Cases
Oracle Stream Analytics Key Features
Developer Experience
Streaming Data Pipelines (ETL) with GoldenGate
…for More Information

What is a Streaming Platform?
A Streaming Platform is to build real-time data
processing pipelines in a rapid fashion, at both
small and very large scale, in order to provide
actionable business insights and very fast data
processing services

Key Streaming Use Cases
IT/Data
Management
Financial
Services
Transportation Telecom
• Data Lake / Warehouse Ingest
• Enterprise Data Services
(REST, Pub/Sub)
• Streaming ETL, Data Pipelines
• DataOps (insights on operational
data)
• Fraud Detection
• Risk Management
• Real-time analysis of currency
exchange data or commodities
• Customer retention, realtime
issue intervention
• Tracking Containers, Delivery
Vehicles, and other Assets
• Vehicle Management
• Passenger Alerts
• Logistics and Route Optimization
• Wifi Off-Loading
• Video Analytics
• Network Management
• Security Operations
• Geolocation Marketing
• Mobile Data Processing
Retail Manufacturing
Utilities, Oil &
Gas
Healthcare
• Real-time Personalized Offers
• Markdown optimization
• Dynamic pricing and forecasting
• Shopping cart defections
• Better store and shelf
management
• Customer retention, realtime
issue intervention
• Smart Inventory
• Quality Control
• Building Management
• Logistics and Route Optimization
• Outage Intelligence
• Workforce Management
• Real-time Drilling Analysis
• Telemetry on critical assets
• Medical Device Monitoring
• In-home Patient Monitoring
• Medical Fraud Detection
• Safer Cities

For Building Data Pipelines
Ingest Database Events Select Processing Patterns Build Event Pipelines Serve Data Downstream
Any GoldenGate event is included
free, Kafka native events require
full-use license
Rich set of pre-built patterns can
dramatically improve developer
efficiency and time-to-value
Tool can easily leverage geo-fencing,
machine-learning, and other lookup
data within the data stream
Data can be delivered out to kafka,
databases, or easily staged for
downstream ETL jobs
connect
Messages,
Events & Alerts
Analytics
Data
{rest}
Data
Lake

MPP Data Processing
Raw
Data
A Complete Solution
GG Database
Replication
Open
Source
Stream
Analytics
GG for Big
Data
ETL
Filter
Aggregate
Transform
Correlate/Enrich
Geo-fence
Thresholds
Business Rules
Data Policies
Queries
Time Series
Spatial Analytics
Data Patterns
Anomalies
Classification
Clustering
Statistical Inference
Regression Models
Messages,
Events & Alerts
Analytics
Data
{rest}
Data
Lake
LOW CODE HIGH SPEED LOW LATENCY DATAOPS

Part of the GoldenGate Platform
Replication of
Real-time Data
Transactions & Events Stream Analytics
ETL
&ML
DBMS
Cloud
Big Data
NoSQL
Streams
Object
Storage

10
GG Stream Analytics – Key Feature Areas
Interactive
Designer UI
Rich Set of Streaming
Patterns
Predictive Analysis and
Machine Learning
Location and Geospatial
Analysis
Integrated CDC with
Oracle GoldenGate
Robustness, Speed, and
Scalability

Two User Personas
Data
Engineer
• Creates custom queries and
patterns
• Defines complex windowing
correlations
• Installs and maintains Big Data
and Messaging Environment
• Manages lookup data sources
• Tunes production pipelines for
performance and high
availability
Data
Analyst
• Explores incoming data
content
• Defines visualizations and
dashboards
• Creates and maintains data
rules
• Chooses and applies patterns
• Defines geo fences and
spatial rules

Interactive Browser-based Designer
Accessible to Non-Technical Users
• Empower data analysts to enhance data with
no coding skills required
• Intuitive, always-on data view shows results of
transformations as they are defined
• Filter and correlate streams, apply rules,
aggregate, calculate fields etc.
Function extensibility via Java
• Allow data engineers to provide custom
stages and functions to be used by all team
members
Integrated Visualizations
• Explore your business data live through
various tables, charts and geospatial maps

Predictive Analysis and Machine Learning
Real-time Scoring and Decision Making
• Use Machine Learning models to make business
decisions in real-time
• Predict future outcomes such as equipment failures,
customer behavior, fraud and security breaches
• Re-import refined models for improved predictions
Put Data Science in Production
• Import Predictive Models created by data scientists and
engineers in their own environment.
• Import of PMML models for a variety of algorithms such as
vector machines, association rules, Naive Bayes classifier,
clustering models, text models, decision trees, and different
regression models.
• Hide model complexity for use by data analysts
• Custom stages for access to external scoring systems Oracle R
Enterprise
Notebooks
(Jupyter,
Zeppelin, etc)
Data Scientist
Data Analyst/
Data Engineer

Location and Geo-Spatial Capabilities
Interactive Spatial Design and Visualization
• Show live location data on maps as events are
processed
• Track individual objects and highlight them based
on different conditions, e.g. Red for violation
Rich Geospatial Pattern Set
• Correlate multiple objects through their spatial
interaction
• Detect speed, and proximity
• Obtain address and city information from location
and vice versa through Geocoding
Scalable Definition of Areas and Geo-Fences
• Define polygons through drawing borders on a map
• Manage large amounts of shapes through spatial
types in Oracle database.

GoldenGate Integrations
Process and Analyze Live Data
• Gain insights into business by analyzing live
transactions – or schema events (DDL etc)
• Transform and aggregate events to store into data lake
in real time using filters, joins, rules, aggregations,
splits, unions and other common operations.
• Natively process GoldenGate change records and
keep live aggregates based on change records
Monitor your Database Transactions
• Analyze statistics of ongoing database activity
• Identify hot records with many changes, monitor
sensitive tables or records for activity and exceeding
thresholds.
• Correlate different transactions, for example confirm
that a request is acknowledged.
• Identify unusual or fraudulent activity, such as records
that are created and soon after deleted
Real-time BI
Big Data Lakes
Business
Process
Operational
Dashboards
OLTP
Database
GoldenGate Kafka - Oracle
Enterprise Hub
Oracle Stream
Analytics
How does my data
do right now?
Alert and Act on
critical issues
OLTP
Database
GoldenGate
Kafka - Oracle
Enterprise Hub
Oracle Stream
Analytics
Target
Database

Rich Set of Streaming Patterns
Simplify Access to Complex Algorithms
• Easy-to-use modules with user assistance in the
designer
• Pre-defined visualizations to provide immediate
feedback
• Accessible to data analysts
Comprehensive Library of Patterns
• Covers diverse areas such as anomaly detection,
stream correlation, trend analysis, spatial functions
• Duplicate, out-of-order, and missing event
detection
• Functions for financial, statistic, and log analytic
operations

Robustness, Speed and Scalability
Horizontal Scalability through Spark clusters
• High throughput by using highly parallelized in-memory
processing by Spark
• Efficient event correlations using Oracle’s CQL engine
• Cloud-based elastic Spark clusters through Oracle Big Data
Cloud
• Scale clusters dynamically by adding more nodes
• Distribute multiple web server nodes through load balancing
High Availability through Cluster Redundancy
• Exactly once query semantics
• Recover failing worker nodes in Spark cluster without Data
Loss

Architecture
LBR
OSA
Web-tier
jetty
OSA
Web-tier
jetty
Metadata Store
Web-Tier Cluster
MPP Processing
Pipeline
Deployment &
Data Access
Data Streams From Applications
Results, Alerts, Notifications
• Pipeline
Design
• Interactive
Analytics
• Dashboards
runtimedesign
time

20Confidential – Oracle Internal/Restricted/Highly Restricted
Data Pipeline
GoldenGate Feeds
Sensor Data
Social Media
Click Stream
Geo Location
Filter
Aggregate
Transform
Correlate/Enrich
Geo-fence
Queries
Time Windows
Data Patterns
Spatial Analytics
Anomalies
Classification
Clustering
Statistical Inference
Regression Models
Business Rules
Policies
Conditional Logic
Notify/Publish
Invoke/Execute
Visualize
Persist
Data Ingestion Pre-processing
Analysis
Prediction
Decisions Actions
Ingest Transform and Correlate Act and Deliver

End-to-End Steps to build a Stream Application:
1. Create Connections, Stream, and References for Sources
Kafka, JMS, File,
Database, or REST
Connection Types Message Shape is
detected from
Kafka Topic

2. Create Geographical Areas / Geo Fences
Build Geo Fences
manually or from DB
repository

3. Import Predictive PMML Model
Train and export
PMML models from
common ML tools
such as R, SAS, H2O,
etc.

4. Create New Pipeline
Incoming messages
are displayed
automatically
New Pipeline is
immediately valid
and active

5. Add Joins to Pipeline
Join Stream or
Batch source
Joined events are shown
with color-coded fields

6. Add Patterns to Pipeline
Choose from a library of
vertical patterns
Event locations are shown
on map in real-time

7. Add ML Scoring to Pipeline
Refer to uploaded
PMML model
Map event fields into
PMML model properties

8. Add Target to Pipeline
Send events to Kafka,
JMS, or REST targets

9. Publish Pipeline to Production
One-click deploy into
production Spark cluster

More than 20 Years of Innovation
1000’S OF CUSTOMERS GLOBALLY
1990’s – Database HA/DR
2000’s – OLTP Replication
2010 – Data Warehouse
2015 – Data Lake
& Cloud
KEY USE CASES
& GROWTH PHASES:
http://www.oracle.com/us/products/middleware/data-
integration/oracle-goldengate-innovations-wp-5093027.pdf

GoldenGate Platform Capabilities
Data Replication Data Lake Pipelines Stream Analytics
Data High Availability
• Oracle/Non-Oracle DB
• Low Downtime Migrations
Transaction Replication
• OLTP/Reference Data
Data Warehouse Loading
• Non-invasive Capture
• Realtime Staging
Data Lake Ingest
• High Fidelity Change Stream
• Event-driven Realtime
Pre-Processing
• Filter, Correlate, Enrich
Data Transformations
• Streaming ETL Ops: Query,
Aggregate, Lookup, etc.
Data Operations (DataOps)
• Low code development, iterate
and work with production data
Advanced Analytics
• Time Series Analysis, Machine
Learning, Geo-Spatial
Dashboards
• Active graphs and charts

For Oracle Cloud
Virtual Machine
Database Cloud Service
Bare Metal
Database Cloud
Exadata Cloud Service
Autonomous DB
OCI Object Storage
OCI Streaming Service
Replication of
Real-time Data
Transactions & Events

For the Enterprise
Replication of
Real-time Data
Transactions & Events GoldenGate Stream Analytics
ETL
&ML
DBMS
Cloud
Big Data
NoSQL
Streams
Object
Storage

GoldenGate Marketplace – 3 Steps to Productivity
Step 1: Search OCI Marketplace Step 3: Run GG Web AppsStep 2: Provision GoldenGate on OCI
Navigate to
https://cloudmarketplace.oracle.
com/marketplace/en_US/listing/
58489224 Choose “Launch App” and click
through ~4 pages of forms
(existing OCI account is
required)
Access GoldenGate
Microservices (Web Apps) from
public IP Address given via OCI
confirmation page
FREE Promo Now!

Stream Analytics Marketplace
OCI Marketplace
GG Database
Replication
Open
Source
Stream
Analytics
GG for
Big Data
OCI Compute (any shape)
OCI Block Store
Trail
File
Kafka
Topics
MySQL
Store
Messages,
Events & Alerts
Analytics
Data
{rest}
Data
Lake
LOW LATENCY HIGH SPEED LOW CODE DATAOPS

GoldenGate Integrations
Process and Analyze Live Data
• Gain insights into business by analyzing live
transactions
• Transform and aggregate events to store into data lake
in real time using filters, joins, rules, aggregations,
splits, unions and other common operations.
• Natively process GoldenGate change records and
keep live aggregates based on change records
Monitor your Database Transactions
• Analyze statistics of ongoing database activity
• Identify hot records with many changes, monitor
sensitive tables or records for activity and exceeding
thresholds.
• Correlate different transactions, for example confirm
that a request is acknowledged.
• Identify unusual or fraudulent activity, such as records
that are created and soon after deleted
Real-time BI
Big Data Lakes
Business
Process
Operational
Dashboards
OLTP
Database
GoldenGate Kafka - Oracle
Enterprise Hub
Oracle Stream
Analytics
How does my data
do right now?
Alert and Act on
critical issues
OLTP
Database
GoldenGate
Kafka - Oracle
Enterprise Hub
Oracle Stream
Analytics
Target
Database

Managed GoldenGate Service
GoldenGate for OCI Marketplace GoldenGate Service (GGS)
• Automated provisioning (via Terraform)
• Customer managed software, runs on customer
managed OCI Compute & Storage
• Choose from 5 listings: GG Microservices, GG
Classic, GG for Big Data, GG for Non-Oracle, or
GG for Mainframe
• Bring your own license
(Processor, Term, Named User,
Promotional…)
• OCI native service
(console-based, admin, monitoring, lifecycle, patching etc)
• Oracle managed software
(multi-tenant control plane, secure encryption keys and Trail
Files deployed in customer tenancy)
• Single, integrated customer experience for all
aspects of GoldenGate functionality
• Universal Cloud Credits

GoldenGate Service Key Use Cases
GoldenGate Service GoldenGate Service GoldenGate Service
sync
Database
Use Cases
Data Lake
Use Cases
HA/DR, Active-Active, Multi-Master,
Zero-Downtime,DB Migrations
Replicate transactions, data and
DB events across OLTP DBs
Trickle feed, load and stage data
into Data Warehouse tables
Streaming data ingestion to Object
Storage, Hadoop, Kafka etc.
Transform and process any data
as it in arrives in the Stream
Apply machine learning, geo-
spatial and advanced rules
GoldenGate Service GoldenGate Service GoldenGate Service
ETL
Pipeline AI/MLIoT Time Series
push
down

The preceding is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September
2019 and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor

Integration with GoldenGate
Kafka Cloud Services
Contextual
Data
ML Models
Real-time
BI
Big Data Lakes
Business
Process
Operational
Dashboards
DB Events
Ingest with
GoldenGate
Actions
Oracle
SQL Server
MySQL
IBM DB2 Z
IBM DB2 i
IBM DB2 LUW
HP NonStop
Informix
Sybase
Messaging
Oracle
GoldenGate
Stage in
Kafka
Stream
Analytics
Capture Trail
Files Delivery
Trail
Files

Pattern for Logical Data Zones & Topic Types
43Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
| Oracle OpenWorld 2018
Raw Data (LCR)
Schema Events
(DDL)
Prepared Data Topics
Master Data Topics
ETL ETL
1 Topic : 1 Table
Data Consumers
Applications
Analytics
ODS (Data Store)
Data Marts
Data Warehouses
Stream Data Producers
Apps & DBs:
Staging Trusted Master
ETL
Bulk Data Producers
ETL
Data Science
Events are
Pushed
Batching
Interactive
Queries
OLAP SQL
Bucket 1 Bucket 2 Bucket 3

Oracle Stream Analytics - Developer Introduction

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Oracle Stream Analytics - Developer Introduction

Similar to Oracle Stream Analytics - Developer Introduction (20)

More from Jeffrey T. Pollock

More from Jeffrey T. Pollock (20)

Recently uploaded

Recently uploaded (20)

Oracle Stream Analytics - Developer Introduction