Talk was delivered at DOAG 2022 with Fabian Hardt Abstract: Since analytics platforms are now an integral part of corporate IT landscapes, the networking of these central data platforms with other systems is becoming increasingly important. In order to be able to design these securely and comprehensibly (API gateway), as well as individually (API modelling), data APIs are a possible solution. In this presentation, we will show you what this means and what an example implementation with a corresponding API gateway could look like.

1. © OPITZ CONSULTING 2022 / Öffentlich
Analytics meets Integration – Modern Development with Data APIs 1
Modern Development with
Data APIs
21.09.2022
Sven Bernhardt, Fabian Hardt
ANALYTICS MEETS
INTEGRATION

2. © OPITZ CONSULTING 2022 / Öffentlich
AGENDA
Analytics meets Integration – Modern Development with Data APIs 2
MOTIVATION
01
WHY DOES INTEGRATION AND
ANALYTICS BELONG TOGETHER?
02
DEMO
04
REPORT FROM PRACTICE
03
DATA APIS - BASIS FOR
DATA MESH
05
CONCLUSION
06

3. © OPITZ CONSULTING 2022 / Öffentlich
Analytics meets Integration – Modern Development with Data APIs 3
MOTIVATION
01

4. © OPITZ CONSULTING 2022 / Öffentlich
Analytics meets Integration – Modern Development with Data APIs 4
¢ Fast time to market is becoming more and more important
¢ Classic software development shows how it's done
¢ CI/CD, Containerization, high degree of automation
¢ Departments are often unhappy because the BI-Team
cannot keep up with the implementation
¢ Departments want self-service offers
¢ Important to avoid „shadow IT“
¢ Faster time to market through decentralization of systems
¢ Data Platform vs. Integration Platform
¢ Grows together more and more
NEW WORLD PROBLEMS
ANALYTICS
MODERN

5. © OPITZ CONSULTING 2022 / Öffentlich
Analytics meets Integration – Modern Development with Data APIs 5
¢ Decentralization requires more governance
¢ Metrics & Logging
¢ Access permissions & policies
¢ Metadata
¢ API-Management can help to implement these solutions
¢ Creates standardization
¢ Avoids duplicate development efforts
¢ A high degree of automation can help to control decentralized
systems
¢ Self service platform & ready to use components
¢ Development templetes for recurring tasks
SECURITY & OBERSERVABILITY
SECURITY
OBSERVABILITY

6. © OPITZ CONSULTING 2022 / Öffentlich
Analytics meets Integration – Modern Development with Data APIs 6
WHY DO INTEGRATION AND
ANALYTICS BELONG TOGETHER?
02

7. © OPITZ CONSULTING 2022 / Öffentlich
INTERFACES IN ANALYTICS
Analytics meets Integration – Modern Development with Data APIs
¢ Classic „API“ in the form of
direct database access
¢ Database Link, Views, etc.
¢ Database authentication and
authorization is used
¢ Missing standardization and
documentation
¢ Consumer usually has to
deal with the JDBC protocol
7

8. © OPITZ CONSULTING 2022 / Öffentlich
TRADITIONAL VS. CLOUD-NATIVE SYSTEMS
Analytics meets Integration – Modern Development with Data APIs 8
¢ Single Deployment unit
¢ Single execution process
¢ Shared database
¢ Local, app-internal communication only
¢ Multiple Deployment units
¢ Exclusive execution process per service
¢ Database per Service (usually)
¢ Remote, inter-service communication
UI
Business
Logic
Persistence

9. © OPITZ CONSULTING 2022 / Öffentlich
ACHIEVE LOOSE COUPLING THROUGH API DESIGN-FIRST APPROACH
Analytics meets Integration – Modern Development with Data APIs 9
¢ Synchronous APIs (it’s not just REST!)
¢ REST (OpenAPI)
¢ GraphQL
¢ gRPC
¢ Asynchronous APIs (preferred)
¢ Async API
¢ Data description with
¢ Protobuf
¢ Avro
¢ JSON Schema

10. © OPITZ CONSULTING 2022 / Öffentlich
APIS SHOULD DELIVER BUSINESS VALUE!
Analytics meets Integration – Modern Development with Data APIs 10
¢ Focus on potential consumers
¢ How will a consumer use my API?
¢ What kind of resources and operations are
needed by the consumers?
¢ …
¢ APIs should be consumer-centric and thus
be consumable
¢ Well-designed, secured and compliant
¢ Documented, discoverable and accessible
¢ Performant, reliable and scalable
Source: https://konghq.com/blog/what-is-apiops/
Compliant
Functional Findable
The intersection of the three
characteristics defines
consumable APIs!

11. © OPITZ CONSULTING 2022 / Öffentlich
BUSINESS DOMAINS
Analytics meets Integration – Modern Development with Data APIs 11
¢ Domain Driven Design (DDD) as a basis
¢ Loose coupling between systems
¢ Changes to modules should have as little
impact as possible on other modules
¢ Puts domain at the center of modeling /
architectural work
¢ Often in the microservice environment -
classic software development
¢ Domain
¢ Depicts the technicality / technical logic
of a business area
¢ HR department, CRM department, SCM, etc.

12. © OPITZ CONSULTING 2022 / Öffentlich
MODERNIZATION WITH DATA APIS
Analytics meets Integration – Modern Development with Data APIs 12
¢ Existing interfaces are expanded with "real" API services
¢ Contract defined in API-Specification
¢ Overview of the available APIs in central developer portal
¢ Other services can reuse the new API
¢ Flexible platforms such as Kubernetes can be used

13. © OPITZ CONSULTING 2022 / Öffentlich
Analytics meets Integration – Modern Development with Data APIs 13
REPORT FROM PRACTISE
03

14. © OPITZ CONSULTING 2022 / Öffentlich
Analytics meets Integration – Modern Development with Data APIs 14
Data IN Data OUT
CUSTOMER SCENARIO: API ENABLED DATA LAKE
*ADLS = Azure Data Lake Storage
DWH /
Data
Lake
Stream/
Events
Files
A
P
I
L
o
g
i
c
A
p
p
,
o
.
ä
.
A
D
L
S
L
a
n
d
i
n
g
Z
o
n
e
Logic
App, o.ä.
AD
LS
Export
Zone
API
API
D
BM
S
A
P
I
F
i
l
e
L
i
n
k
Extern
Intern
A
P
I
T
o
p
i
c
Pow
er
API
K
a
f
k
a
AD
LS
L
o
g
i
c
A
p
p
,
o
.
ä
.
Logic
App, o.ä.
A
D
L
S
L
a
n
d
i
n
g
Z
o
n
e
A
D
L
S
A
D
L
S
E
x
p
o
r
t
Z
o
n
e
API
T
o
p
i
c
Kafka
IoT-H
ub

15. © OPITZ CONSULTING 2022 / Öffentlich
CUSTOMER SCENARIO: SDLC
Analytics meets Integration – Modern Development with Data APIs 15

16. © OPITZ CONSULTING 2022 / Öffentlich
Analytics meets Integration – Modern Development with Data APIs 16
DEMO

17. © OPITZ CONSULTING 2022 / Öffentlich
DEMO: ARCHITECTURE & DATA MODEL
Analytics meets Integration – Modern Development with Data APIs 17

18. © OPITZ CONSULTING 2022 / Öffentlich
DEMO PLATFORM: ORACLE CONTAINER ENGINE FOR KUBERNETES
(OKE)
Analytics meets Integration – Modern Development with Data APIs 18
Based on IaaS Oracle
Compute Cloud Service
Worker Nodes: VM
Master Node:
•Managed and maintained by Oracle
•Not visible for the end user
•Master nodes are free of charge
Auto-scaling capabilities
using Worker-Node Pools
Can be provisioned using
OCI Cloud Console
OCI Cloud Shell
OCI CLI
Terraform (OCI Resource Manager)

19. © OPITZ CONSULTING 2022 / Öffentlich
DEMO PLATFORM: CONTAINER REGISTRY
Analytics meets Integration – Modern Development with Data APIs 19
¢ Open standards-based, Oracle-managed Docker registry service for securely storing and
sharing container images (Docker v2 compliant)
¢ Data protection through integration with OCI object storage
¢ Automatic clean up old docker images (Retention policies)
¢ Integrates with different OCI services and 3rd party DevOps and dev tools (Jenkins, Gitlab)
¢ Container Engine for Kubernetes (OKE)
¢ Identity and Access Management (IAM)
¢ Visual Builder Studio
¢ Docker containers can be pushed/pulled by Docker CLI and API

20. © OPITZ CONSULTING 2022 / Öffentlich
DEMO PLATFORM: DEVOPS
Analytics meets Integration – Modern Development with Data APIs 20
¢ End-to-end, continuous integration and continuous delivery (CI/CD) platform
¢ Service provides:
¢ Private Git repositories (allows connections to external code repos)
¢ Build pipelines
¢ Deployment pipelines
Code
Repository
External
Connection
Trigger
Build
Pipeline
Container
Registry
Artifact
Registry
Artifact
Deployment
Pipeline
OKE
Functions
Compute

21. © OPITZ CONSULTING 2022 / Öffentlich
DEMO PLATFORM: RESOURCE MANAGER
Analytics meets Integration – Modern Development with Data APIs 21
¢ Automate the process of provisioning Oracle Cloud Infrastructure resources
¢ Infrastructure as Code (IaC) with HashiCorp Terraform using specific OCI Terraform provider
¢ Managing OCI resources in a declarative way
¢ Ensure consistent infrastructure management by supporting
¢ Immutable infrastructure approach
¢ GitOps

22. © OPITZ CONSULTING 2022 / Öffentlich
Analytics meets Integration – Modern Development with Data APIs 22
DATA APIS AS THE BASIS FOR DATA
MESH
05

23. © OPITZ CONSULTING 2022 / Öffentlich
DATA PRODUCTS
Analytics meets Integration – Modern Development with Data APIs 23
¢ In direct connection with microservices from the classic SD environment
¢ Operational Applications vs. Analytical Applications

24. © OPITZ CONSULTING 2022 / Öffentlich
CROSS-DOMAIN DATA PRODUCT USAGE
Analytics meets Integration – Modern Development with Data APIs 24

25. © OPITZ CONSULTING 2022 / Öffentlich
FEDERATED GOVERNANCE
Analytics meets Integration – Modern Development with Data APIs 25

26. © OPITZ CONSULTING 2022 / Öffentlich
DATA MESH EXAMPLE
Analytics meets Integration – Modern Development with Data APIs 26

27. © OPITZ CONSULTING 2022 / Öffentlich
Analytics meets Integration – Modern Development with Data APIs 27
CONCLUSION
06

28. © OPITZ CONSULTING 2022 / Öffentlich
KEY TAKEAWAYS
Analytics meets Integration – Modern Development with Data APIs 28
¢ Software development practises are making their way in the world of Analytics
¢ Data API and DDD are essential for implementing Data Mesh
¢ Data APIs help you to
¢ Decouple data stores, structures and technologies
¢ Gain insights into data usage (Observability)
¢ Control data access consistently (AuthN & AuthZ)
¢ APIs must be compliant, functional & findable (Consumable APIs)
¢ APIs != REST

29. © OPITZ CONSULTING 2022 / Öffentlich
Analytics meets Integration – Modern Development with Data APIs 29
Q & A

30. © OPITZ CONSULTING 2022 / Öffentlich
Analytics meets Integration – Modern Development with Data APIs 30
www.opitz-consulting.com
KONTAKT
Sven Bernhardt
Senior Manager Corporate Development
Sven.Bernhardt@opitz-consulting.com
+49 172 2193529
https://www.xing.com/profile/Sven_Bernhardt/
https://www.linkedin.com/in/sven-bernhardt-0570b823/
Fabian Hardt
Solution Architect
Fabian.Hardt@opitz-consulting.com
https://twitter.com/fabian_hardt
https://www.xing.com/profile/Fabian_Hardt
https://www.linkedin.com/in/fabian-hardt-0956b1b1