Paolo Kreth - Persistence layers for microservices – the converged database approach

Cloud Platform Architect
Paolo Kreth
DEVOPSDAYS GENEVA 24.02.2020
Persistence layers for microservices –
the converged database approach

Persistence layers for microservices – the converged database approach2

WhoamI?
• * 1968 in Genoa /Italy
• Master Degree in Informatics at Università degli Studi
di Genova
• Many Years Freelancer Oracle-DBA & Developer
• Principal Instructor - Oracle University Switzerland
• 2010 la Mobilière
• Oracle Technical Lead & Project Manager
• Team Lead Data Management
• DB Architecture and Open Source DBs
• 2019 Cloud Platform Architect at Oracle
• Member of Swiss Oracle User Group

… be About
• DevOps Culture
• Use Cases
• Past Experience
• Architecture
• Databases
• Microservices
• Hope we all will have fun
… not be about
• Marketing
• Commercials
• Technical Deep Dive
• Competition
What will this presentation….

I will try to tell You a story…

A long time ago in an insurance far, far away….

Who saw this film when it first came out at cinema ? 1977 Hands Up Please

MainframeOracle
Kessel Run in less than twelve parsecs….

Oracle
Specialists
Foundation: DB-Specialists, less deep more wide knowledge
3 Pillars of the Force
DB2 Windows
Projects
MS SQL-
Specialists
DB/2-
Specialists

Peace in the Galaxis

There’s something going on….

« Oracle and Docker together are difficult to handle. »
« My forecast: PostgreSQL will come. It only depends in which way »
« Step 1 : find a DBA who is willing to support PostgreSQL . »
« Microservices would gain efficiency with PostgreSQL. »
« I know it does not fit with the official strategy, but it will come. If not on premises than in Azure or
Google and probably overnight, but it will come.»
« So again step 1 : find a guy who wants to support it, or hire on, do what it needs….
A pre announcement - PostgreSQL
How would You have approched
such an E-Mail?
Remember I was an OPS-ler!

My approach to this Mail ? Curiosity

« Oracle and Docker together are difficult to handle. »
« My forecast: PostgreSQL will come. It only depends in which way »
« Step 1 : find a DBA who is willing to support PostgreSQL . »
« Microservices would gain efficiency with PostgreSQL. »
« I know it does not fit with the official strategy, but it will come. If not on premises than in Azure or
Google and probably overnight, but it will come.»
« So again step 1 : find a guy who wants to support it, or hire on, do what it needs….
A pre announcement - PostgreSQL Docker
Micro Services
Cloud

The Microservices!

A new feeling
Open Source DBs Oracle

What was it all about ? Bad
Relationship
with DBAs
Automated
Testing
Persitence
Layer for
Docker and
Kubernetes
Container
Databases
Data Stores
or Cache ?
Cloud
Native
Database
Build of
CI/CD
Pipeline
I build it
I run it
I want to
choose it
Not satisfied
with offering
and serviceWork locally
offline
DBAs are old
fashioned –
they do not
understand

The DBA Wookie (Ops)

The Next Thing….

Rocketchat – Relies on MongoDB

• Dev Teams want to introduce a new persistent
messaging system
• The persistence layer relies on MongDB
• Skype for Business is not an Option as not
persistent
• Unfortunately MongoDB is not part of the IT
Strategy and no DBA has knowledge
• JSON in JSON out
What is this about ?

Requirement: A Mongo Database for each Relational DB to export Data in JSON format
And another one – The Data Market

….and even the next thing… Elasticsearch !

Demand for new Data Stores from everywhere

Hilfe, die Open Source-DBs kommen!25
Why so many new demands ?

I build it I run it….I want to chose it

Where are the DBAs in SAFE?
“Because these resources are specialized – often single-sourced
and typically quite busy – each ART and Solution Train must plan
to engage the shared services personnel it needs, when it needs
them”
“Feature teams have to engage us”?

The DBA Wookie (Ops)

Is this the future?
Projects

Microservice Architecture
1. Architecture Pattern, not just putting code into Docker
containers
2. Each microservice can run in a container
Private database and data model
for each microservice
3. Microservices communicate using asynchronous
messaging via some event queuing system
Decoupled for maximum resiliency and scalability
Services do not talk to each other directly – only via event
queuing service
Insulated from slowdown or failure of other microservices
Shipping
Service
Persistence layers for microservices – the converged database
approach
30

Pros
• Freedom in models for each µServices
• Freedom in technologies for each µServices
• No µservices development or deployment
dependencies
• Full service isolation
• Independent µServices scaling
Cons
• Data and model consistency between µServices is
difficult
• Cross service transactions require consistency.
Data aggregation requires ETL
• No common data access roles or security models
• Operational sprawl and management overhead
Separate Data Management Platforms for each µService
Separate Data Management
Platforms
And Schemas for each µService
Product
catalog
API
Financial
data
API
Recommend
engine
API
JSON
approach
31

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Relational
Database
Two Approaches to Polyglot Persistence
Single-model Multi-model
Document store
(JSON/XML)
Relational data
Key / Value data
Documents
(JSON/XML)
Graphs
Free Text and
Docs
(PDF, DOC,…)
Spatial data
Key/Value
Store
Graph Database
Free Text/Docs
(PDF, DOC, …)
Spatial
Database

So You have to choose…
approach
33

µServices and Polyglot Persistence
Product
catalog
API
Financial
data
API
Recommen
d engine
API
Monolingual µServices Polyglot µServices
Product
catalog
API
Financial
data
API
Recommen
d engine
API
JSON
approach
34

Product
catalog
API
Financial
data
API API
Recommen
d engine
Product
catalog
API
Financial
data
API
Recommen
d engine
API
Monolingual µServices Polyglot µServices
approach
35

Product
catalog
API
Financial
data
API API
Recommen
d engine
Product
catalog
API
Financial
data
API
Recommen
d engine
API
Polyglot µServicesMonolingual µServices
approach
36

Product
catalog
API
Financial
data
API
Recommen
d engine
API
JSON
Product
catalog
API
Financial
data
API
Recommen
d engine
API
Polyglot µServicesPolyglot µServices
approach
37

Product
catalog
API
Financial
data
API
Recommend
engine
API
JSON
Product
catalog
API
Financial
data
API
Recommend
engine
API
Polyglot µServicesPolyglot µServices
approach
38

JSON storage
create table emp(
empno number,
first varchar2(50),
last varchar2(50),
salary number,
flex CLOB,
check (flex IS JSON));
insert into emp values(1, 'John', 'Smith’,
2000, '{skills:["Java", "C", "C++"]}');
insert into emp values(2, 'Amanda', 'Jones’,
2500, '{skills:["JavaScript", "nodeJS"],
numPatents:5}');
create table emp(
empno number,
first varchar2(50),
last varchar2(50),
salary number,
flex JSON);
insert into emp values(1, 'John', 'Smith’,
2000, '{skills:["Java", "C", "C++"]}');
insert into emp values(2, 'Amanda', 'Jones’,
2500, '{skills:["JavaScript", "nodeJS"],
numPatents:5}');
NEW

Binary JSON Type
1. No text parsing or serialization required
2. dictionary based encoding scheme
3. compression for reduced IO and
network overhead
4. efficient “in-place” random access on
both server and client
5. partially updateable (less IO)
6. supports extended SQL primitives
(TIMESTAMP, DATE, RAW, etc.)
7. Client-side libraries to encode/decode
Persistence layers for microservices – the converged database approach
NEW
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
10KB 100KB 1MB
JSON Scan Time (s) for
100MB
CLOB JSON
40
40

JSON_TABLE: ‘flatten’ the JSON data
{
"firstName":"John",
"lastName":"Smith",
"age":25,
"address":{
"street":"21 2nd Street“,
"city":"New York",
"state":"NY",
"postalCode":"10021",
"isBusiness":false
},
"phoneNumbers":[
{"type":"home",
"number":"212-555-1234"},
{"type":"mobile",
"number":"646-555-4567"}
],
"bankruptcies":null,
"lastUpdated":"2019-05-13T13:03:35"
}
select id, jt.*
from custData NESTED jcol columns (
firstName,
age NUMBER,
”state" PATH '$.address.state',
NESTED phoneNumbers[*] columns(
"number"
) jt;
ID firstName age state number
--------------------------
1 John 25 NY 212-555-1234
1 John 25 NY 646-555-4567
NEW
41
41

JSON Generation
select
JSON_OBJECTAGG(dname VALUE
select
JSON_ARRAYAGG(JSON {ename})
from emp e
where e.deptno = d.deptno
) from dept d;
JSON_OBJECTAGG(…
-------------------------
{"ACCOUNTING":[
{"eName":"CLARK"},
{"eName":"KING"},
{"eName": "MILLER"}
],
"RESEARCH":[
{"eName": "SMITH"},
{"eName": "JONES"},
42
42

Want to know more ?– Call with the Developer right after session
43 Beda Hammerschmidt – JSON and SODA

Updated licensing as of December 5th 2019
• Machine learning features now included as part of Oracle Database license -
purchase of Advanced Analytics option no longer required.
• Spatial & Graph features now included as part of Oracle Database license -
purchase of Spatial and Graph option no longer required.
• Features are included in all editions of Oracle Database
• Enterprise Edition and SE2
• For all releases under Premier or Active Support, ie. 11.2.0.4. and above
• Key part of „converged database“ strategy
Making developer APIs available free of charge
Copyright © 2019 Oracle and/or its affiliates.
approach
44

Oracle Spatial and Graph
Three major features
Confidential – Oracle Internal/Restricted/Highly Restricted 45
Spatial Property Graph RDF Graph

100’s of SQL
spatial analysis
operators
Filter
Combine
Transform
Measure
Confidential – Oracle Internal/Restricted/Highly Restricted46
Spatial Analysis

Topology for mapping, land
management and cadastre
applications
Confidential – Oracle Internal/Restricted/Highly Restricted47
Advanced Spatial Data Models
Spatial networks for roads, transport,
pipelines, telcos and other
geographically connected analysis

Property Graph Analysis for
Business Insight
48
Identify
Influencers
Discover Graph Patterns
in Big Data
Generate
Recommendations
Confidential – Oracle Internal/Restricted/Highly Restricted

approach
49
Rich set of built-in parallel graph algorithms … and parallel graph mutation operations
Computational Property Graph Analytics: Built-in Package
Detecting Components
and Communities
Tarjan’s, Kosaraju’s,
Weakly Connected
Components, Label
Propagation (w/ variants),
Soman and Narang’s
Spacification
Ranking and Walking
Pagerank, Personalized Pagerank,
Betwenness Centrality (w/ variants),
Closeness Centrality, Degree
Centrality,
Eigenvector Centrality, HITS,
Random walking and sampling (w/
variants)
Evaluating Community
Structures
∑ ∑
Conductance,
Modularity
Clustering Coefficient
(Triangle Counting)
Adamic-Adar
Path-Finding
Hop-Distance (BFS)
Dijkstra’s,
Bi-directional Dijkstra’s
Bellman-Ford’s
Link Prediction SALSA
(Twitter’s Who-to-follow)
Other Classics Vertex Cover
Minimum Spanning-Tree
(Prim’s)
a
d
b e
g
c i
f
h
The original graph
a
d
b e
g
c i
f
h
Create Undirected
Graph
Simplify Graph
a
d
b e
g
c i
f
h
Left Set: “a,b,e”
a d
b
e
g
c
i
Create
Bipartite Graph
ge b d i a f c h
Sort-By-Degree (Renumbering)
Filtered
Subgraph
d
b
g
i
e

approach
50
W3C Standards
• Native support for W3C
standards for semantic data,
ontologies and inferencing
• RDF and RDB2RDF
• Inferencing with RDFS, OWL,
SKOS, and user-defined rules
• OGC GeoSPARQL support
Languages, Tools, and APIs
• SQL query support
• SPARQL query language
SPARQL/update, SPARQL
endpoint
• Ontology-assisted query using
SQL
• Java APIs via Jena, Joseki and
Sesame
• Protégé ontology editing
• Cytoscape visualization plug-in
Enterprise Database
• Scalable to over 54 billion
triples, up to 8 PB
• Fine-grained Security
• RDF Views on relational tables
and Property Graphs
• Supports Property Graph
analysis on RDF data
• Compressed, partitioned
storage
• Multitenant database support
RDF for Knowledge Graph, Linked-Data and
Semantic Data Integration

Oracle Labs
• Global research team
• 220+ PhD’s, MS, BS
51
“Identify, explore, and transfer new technologies that
have the potential to substantially advance Oracle’s
business.”
Mission

Parallel Graph AnalytiX (PGX)
https://docs.oracle.com/cd/E56133_01/latest/
A fast, parallel, in-memory graph analytics framework
Offers 35+ built-in, native graph analytics algorithms and a DSL for
custom algorithms
Provides a graph-specific query language (PGQL)
52
SELECT speaker.Name AS Speaker, count(id(speaker)) AS Popularity
FROM graph
MATCH (speaker)-[presented]->(talk)<-[attended]-(attendee)
WHERE talk.Type = ‘Talk'
AND speaker.Role = ‘Speaker’
AND attendee.Role = ‘Attendee’
GROUP BY id(speaker)
ORDER BY NumberOfAdmissions DESC
LIMIT … OFFSET …
Speaker with the most listeners:

CLASSIFICATION
Naïve Bayes
Logistic Regression (GLM)
Decision Tree
Random Forest
Neural Network
Support Vector Machine (SVM)
Explicit Semantic Analysis
CLUSTERING
Hierarchical K-Means
Hierarchical O-Cluster
Expectation Maximization (EM)
ANOMALY DETECTION
One-Class SVM
TIME SERIES
Forecasting - Exponential Smoothing
Includes popular models
e.g. Holt-Winters with trends,
seasonality, irregularity, missing
data
REGRESSION
Linear Model
Generalized Linear Model (GLM)
Support Vector Machine (SVM)
Stepwise Linear regression
Neural Network
LASSO
ATTRIBUTE IMPORTANCE
Minimum Description Length
Principal Component Analysis (PCA)
Unsupervised Pair-wise KL Div
CUR decomposition for row & AI
ASSOCIATION RULES
A priori/ market basket
PREDICTIVE QUERIES
Predict, cluster, detect, features
SQL ANALYTICS
SQL Windows
SQL Patterns
SQL Aggregates
FEATURE EXTRACTION
Principal Comp Analysis (PCA)
Non-negative Matrix Factorization
Singular Value Decomposition (SVD)
Explicit Semantic Analysis (ESA)
TEXT MINING SUPPORT
Algorithms support text columns
Tokenization and theme extraction
Explicit Semantic Analysis (ESA) for
document similarity
STATISTICAL FUNCTIONS
Basic statistics: min, max,
median, stdev, t-test, F-test,
Pearson’s, Chi-Sq, ANOVA, etc.
R AND PYTHON PACKAGES
Third-party R and Python Packages
through Embedded Execution
Spark MLlib algorithm integration
Oracle Machine Learning Algorithms and Analytics
Copyright © 2019 Oracle and/or its affiliates.
Persistence layers for microservices – the converged database approach

Requirement: A Mongo Database for each Relational DB to export Data in JSON format
Back to the Use Case: The Data Market
Really no need to introduce new data stores !

Back to the use case Rocketchat – Keep Mongo

Which infrastructure for the Persistence ?

Product
catalog
API
Financial
data
API
Recommen
d engine
API
JSON
DevOps on Database - Oracle DB Multitenant
1 Container DB, and a Pluggable DB for each µService
Self-contained PDB for each µService
• Portability (via pluggability)
• Rapid provisioning (via hot clones)
• µServices isolation
• Model consistency and shared reference
data
• Scaling independence for PDBs
Shared memory and background processes
• More applications per server
Common operations performed at Root level
• Manage many as one (patch, upgrade, backups, HA)
• Granular control when appropriate

Multitenant Architecture
PDBs
Root
CDB
Multitenant Container Database
DB Link
A PDB is a portable collection of schemas, schema objects, and non-schema objects that appears to an
Oracle client as a non-CDB

Multitenant
• Container managed database virtualization
• Manage Many as one
• Patching, Backup, Security, Online Cloning, Online
Relocation
• Software as Service
• Shared metadata, Data location transparency
59
12.1
12.2, 18c, 19c
• 12.2
• Online cloning & relocation
• Incremental refresh of test/dev master
• Application containers
• 18c
• Transportable backups
• Snapshot carousel
• Refreshable PDB switchover
• 19c
• RAT and ADDM at PDB level
Confidential - Oracle Internal Only
approach

New Solutions were born
approach
60

DBaaS architecture at Swiss Mobiliar
t = 2 min
t = 1 min
Continuous
integration
Development
registerdetect build
Get imagePush
Start Image
Docker registry
Kubernetes
order get state
OUD connect
t = 4...6 min
t = 8...10 min
Fully automated application deployment
including Oracle database in less than 10 minutes
trigger deployment
PDB
Version control
approach
Alain Fuhrer - Oracle Open World 2018 – Multitenant Microservices
in Swiss Mobiliar
61

Conclusion - The DBaaS Interface is a great success
We satisfy the requirements of all stakeholders
We delivered more than 1200 PDB’s over the Interface in less than two years
With this service we can cover today and in the near future the requirements, which an agile software
development places on the DB management - and Oracle Multitenant is the right architecture for it
177
65
5
27
16
73
57
40
34
58
33
49
64
31
49
83
98
24
139
26
16
51
1
29
16
9
15 11 14 13 9
28
2 5 9
51
11 14
29 34
48
21
101
2
35
0
20
40
60
80
100
120
140
160
180
200
Dez
16
Jan
17
Feb
17
Mär
17
Apr
17
Mai
17
Jun
17
Jul 17 Aug
17
Sep
17
Okt
17
Nov
17
Dez
17
Jan
18
Feb
18
Mär
18
Apr
18
Mai
18
Jun
18
Jul 18 Aug
18
Sep
18
Okt
18
DB created DB deleted
created +1200 PDB’s
deleted + 500 PDB’s
approach
Alain Fuhrer - Oracle Open World 2018 – Multitenant Microservices
in Swiss Mobiliar
62

You do not need to build it yourself
approach
63

Oracle Rest Data Services
Oracle REST Data Services (ORDS) bridges HTTPS and Oracle Database.
• Provides data access consistent with modern App Dev frameworks
• Serving JSON results from relational data
• Data stored in standard relational tables and columns
• Oauth2 integration
• Fully provisioned and functional in all cloud editions
Oracle Database
SQL
HTTP(S) client
Oracle REST Data Services
JSON
Map & BindURI
Transform to JSON SQL Result
approach
64

I want a new Database System
The IT Architecture Jedi Council
approach
65

Which aspects have to be considered when introducing a new Database
System?
approach
66
Security
Monitoring
and Logging
Backup &
Restore Availability
Support
Operations
Lifecycle Usage
Scalability

Don’t Underestimate Security Hardening
approach
67

Keep the use case in mind – Porgs Nest
approach
68
Architecture DesignThe Use Case

approach
69

Use Case in the mind of chief architect
approach
70
Postgres Instance and
Data in Container
Application in Container
• PostgreSQL DB for Config Data of Microservice.
• Config Data may be manipulated at runtime.
• If container crashes it will load original config.
• If original config needs to be changed a new container is build and
deployed.
Is here PostgreSQL the
correct solution ?

Who is responsible for operations?
approach
71

Autonomous Database
Self-Driving
• Scale-out database with
fault-tolerance and DR
• Runs on enterprise-
proven Exadata platform
• Full compatibility with
existing enterprise
databases
Self-Repairing
• Recovers automatically
from any failure
• 99.995% uptime
including maintenance
• Elastically scales
compute or storage as
needed
Self-Securing
• Automatically applies
security updates online
• Secure configuration
with full database
encryption
• Sensitive data hidden
from Oracle or customer
admins

COP
approach
73

SAFe: CoP – Community of Practice

So what are the takeaways ?
• If some technologies are already present in Your environment
think about exploring their capabilities before choosing new ones
– You may reach the goal much faster.
• When designing new architectures keep the use case in mind –
don’t oversize the architecture.
• If You want to introduce new technologies keep in mind that
there will be some initial overhead.
• Talk together and be open minded – communication and
collaboration is the key to success.

76
New Always Free Services
Get in touch with me
after my presentation
Build, test, and deploy applications
on Oracle Cloud —for free.
• Databases, Analytics, Compute, and Container Engine for Kubernetes
• Up to eight instances across all available services
• Up to 5 TB of storage

77 Confidential – © 2019 Oracle Internal/Restricted/Highly Restricted
Swissquote, State of Geneva and Energy Web Foundation

Q&A

Thank You
Paolo.kreth@oracle.com
@PaoloKreth

Paolo Kreth - Persistence layers for microservices – the converged database approach

More Related Content

What's hot

Similar to Paolo Kreth - Persistence layers for microservices – the converged database approach

More from matteo mazzeri

Recently uploaded

Paolo Kreth - Persistence layers for microservices – the converged database approach