Analytics meets Integration – Modern Development mit Data APIs

© OPITZ CONSULTING 2022 / Öffentlich
Analytics meets Integration – Modern Development mit Data APIs 1
Modern Development mit
Data APIs
21.09.2022
Sven Bernhardt, Fabian Hardt
ANALYTICS MEETS
INTEGRATION

AGENDA
MOTIVATION
01
WHY DOES INTEGRATION AND
ANALYTICS BELONG TOGETHER?
02
DEMO
04
REPORT FROM PRACTICE
03
DATA APIS - BASIS FOR
DATA MESH
05
CONCLUSION
06

MOTIVATION
01

¢ Fast time to market is becoming more and more important
¢ Classic software development shows how it's done
¢ CI/CD, Containerization, high degree of automation
¢ Departments are often unhappy because the BI-Team
cannot keep up with the implementation
¢ Departments want self-service offers
¢ Important to avoid „shadow IT“
¢ Faster time to market through decentralization of systems
¢ Data Platform vs. Integration Platform
¢ Grows together more and more
NEW WORLD PROBLEMS
ANALYTICS
MODERN

¢ Decentralization requires more governance
¢ Metrics & Logging
¢ Access permissions & policies
¢ Metadata
¢ API-Management can help to implement these solutions
¢ Creates standardization
¢ Avoids duplicate development efforts
¢ A high degree of automation can help to control decentralized
systems
¢ Self service platform & ready to use components
¢ Development templetes for recurring tasks
SECURITY & OBERSERVABILITY
SECURITY
OBSERVABILITY

WHY DO INTEGRATION AND
ANALYTICS BELONG TOGETHER?
02

INTERFACES IN ANALYTICS
Analytics meets Integration – Modern Development mit Data APIs
¢ Classic „API“ in the form of
direct database access
¢ Database Link, Views, etc.
¢ Database authentication and
authorization is used
¢ Missing standardization and
documentation
¢ Consumer usually has to
deal with the JDBC protocol
7

TRADITIONAL VS. CLOUD-NATIVE SYSTEMS
¢ Single Deployment unit
¢ Single execution process
¢ Shared database
¢ Local, app-internal communication only
¢ Multiple Deployment units
¢ Exclusive execution process per service
¢ Database per Service (usually)
¢ Remote, inter-service communication
UI
Business
Logic
Persistence

ACHIEVE LOOSE COUPLING THROUGH API DESIGN-FIRST APPROACH
¢ Synchronous APIs (it’s not just REST!)
¢ REST (OpenAPI)
¢ GraphQL
¢ gRPC
¢ Asynchronous APIs (preferred)
¢ Async API
¢ Data description with
¢ Protobuf
¢ Avro
¢ JSON Schema

APIS SHOULD DELIVER BUSINESS VALUE!
¢ Focus on potential consumers
¢ How will a consumer use my API?
¢ What kind of resources and operations are
needed by the consumers?
¢ …
¢ APIs should be consumer-centric and thus
be consumable
¢ Well-designed, secured and compliant
¢ Documented, discoverable and accessible
¢ Performant, reliable and scalable
Source: https://konghq.com/blog/what-is-apiops/
Compliant
Functional Findable
The intersection of the three
characteristics defines
consumable APIs!

BUSINESS DOMAINS
¢ Domain Driven Design (DDD) as a basis
¢ Loose coupling between systems
¢ Changes to modules should have as little
impact as possible on other modules
¢ Puts domain at the center of modeling /
architectural work
¢ Often in the microservice environment -
classic software development
¢ Domain
¢ Depicts the technicality / technical logic
of a business area
¢ HR department, CRM department, SCM, etc.

MODERNIZATION WITH DATA APIS
¢ Existing interfaces are expanded with "real" API services
¢ Contract defined in API-Specification
¢ Overview of the available APIs in central developer portal
¢ Other services can reuse the new API
¢ Flexible platforms such as Kubernetes can be used

REPORT FROM PRACTISE
03

Data IN Data OUT
CUSTOMER SCENARIO: API ENABLED DATA LAKE
*ADLS = Azure Data Lake Storage
DWH /
Data
Lake
Stream/
Events
Files
A
P
I
L
o
g
i
c
A
p
p
,
o
.
ä
.
A
D
L
S
L
a
n
d
i
n
g
Z
o
n
e
Logic
App, o.ä.
AD
LS
Export
Zone
API
API
D
BM
S
A
P
I
F
i
l
e
L
i
n
k
Extern
Intern
A
P
I
T
o
p
i
c
Pow
er
API
K
a
f
k
a
AD
LS
L
o
g
i
c
A
p
p
,
o
.
ä
.
Logic
App, o.ä.
A
D
L
S
L
a
n
d
i
n
g
Z
o
n
e
A
D
L
S
A
D
L
S
E
x
p
o
r
t
Z
o
n
e
API
T
o
p
i
c
Kafka
IoT-H
ub

CUSTOMER SCENARIO: SDLC

DEMO

DEMO: ARCHITECTURE & DATA MODEL

DEMO PLATFORM: ORACLE CONTAINER ENGINE FOR KUBERNETES
(OKE)
Based on IaaS Oracle
Compute Cloud Service
Worker Nodes: VM
Master Node:
•Managed and maintained by Oracle
•Not visible for the end user
•Master nodes are free of charge
Auto-scaling capabilities
using Worker-Node Pools
Can be provisioned using
OCI Cloud Console
OCI Cloud Shell
OCI CLI
Terraform (OCI Resource Manager)

DEMO PLATFORM: CONTAINER REGISTRY
¢ Open standards-based, Oracle-managed Docker registry service for securely storing and
sharing container images (Docker v2 compliant)
¢ Data protection through integration with OCI object storage
¢ Automatic clean up old docker images (Retention policies)
¢ Integrates with different OCI services and 3rd party DevOps and dev tools (Jenkins, Gitlab)
¢ Container Engine for Kubernetes (OKE)
¢ Identity and Access Management (IAM)
¢ Visual Builder Studio
¢ Docker containers can be pushed/pulled by Docker CLI and API

DEMO PLATFORM: DEVOPS
¢ End-to-end, continuous integration and continuous delivery (CI/CD) platform
¢ Service provides:
¢ Private Git repositories (allows connections to external code repos)
¢ Build pipelines
¢ Deployment pipelines
Code
Repository
External
Connection
Trigger
Build
Pipeline
Container
Registry
Artifact
Registry
Artifact
Deployment
Pipeline
OKE
Functions
Compute

DEMO PLATFORM: RESOURCE MANAGER
¢ Automate the process of provisioning Oracle Cloud Infrastructure resources
¢ Infrastructure as Code (IaC) with HashiCorp Terraform using specific OCI Terraform provider
¢ Managing OCI resources in a declarative way
¢ Ensure consistent infrastructure management by supporting
¢ Immutable infrastructure approach
¢ GitOps

DATA APIS AS THE BASIS FOR DATA
MESH
05

DATA PRODUCTS
¢ In direct connection with microservices from the classic SD environment
¢ Operational Applications vs. Analytical Applications

CROSS-DOMAIN DATA PRODUCT USAGE

4. FEDERATED GOVERNANCE

ANWENDUNGSBEISPIEL

CONCLUSION
06

KEY TAKEAWAYS
¢ Software development practises are making their way in the world of Analytics
¢ Data API and DDD are essential for implementing Data Mesh
¢ Data APIs help you to
¢ Decouple data stores, structures and technologies
¢ Gain insights into data usage (Observability)
¢ Control data access consistently (AuthN & AuthZ)
¢ APIs must be compliant, functional & findable (Consumable APIs)
¢ APIs != REST

Q & A

www.opitz-consulting.com
KONTAKT
Sven Bernhardt
Senior Manager Corporate Development
Sven.Bernhardt@opitz-consulting.com
+49 172 2193529
https://www.xing.com/profile/Sven_Bernhardt/
https://www.linkedin.com/in/sven-bernhardt-0570b823/
Fabian Hardt
Solution Architect
Fabian.Hardt@opitz-consulting.com
https://twitter.com/fabian_hardt
https://www.xing.com/profile/Fabian_Hardt
https://www.linkedin.com/in/fabian-hardt-0956b1b1

Analytics meets Integration – Modern Development mit Data APIs

Analytics meets Integration – Modern Development mit Data APIs

Recommended

More Related Content

Similar to Analytics meets Integration – Modern Development mit Data APIs

Similar to Analytics meets Integration – Modern Development mit Data APIs(20)

More from Fabian Hardt

More from Fabian Hardt(8)

Recently uploaded

Recently uploaded(20)

Analytics meets Integration – Modern Development mit Data APIs