DeveloperWeekEnterprise2023 - Introduction to Kubernetes Operators for Databases

1. Introduction to Kubernetes Operators for
Databases
DeveloperWeek Enterprise 2023
Juarez Barbosa Junior - @juarezjunior
November 2023
Copyright © 2023, Oracle and/or its affiliates

2. Juarez Barbosa Junior
Sr. Principal Java Developer Evangelist - Oracle
• Coming from Dublin, Ireland
• Developer since 1995 (Java ☕️ 1997)
• 28 years of experience in SW Engineering & DevRel
• Microsoft, Oracle, IBM, Nokia, Unisys, Accenture, startups
• Microsoft Azure Developer Relations Lead
• IBM Watson Tech Evangelist & Cloud Rockstar
• IBM Mobile Tech Evangelist & Global Thought Leader
• Nokia Developers Global Champion
• Java, Python, Cloud, DevOps, SRE, Cloud-native, IoT, AI,
Blockchain, Rust
• Speaker at conferences
• Oracle CloudWorld, Oracle Code, Microsoft Ignite, Microsoft TechX,
jPrime, JCON, GeeCon, DevConf.cz, DevOpsDays, DeveloperWeek,
DevOps Institute, CloudLand, DWX, The Developer’s Conference
(TDC), Sec4Dev, JSNation, NodeConf, Conf42, Shift Conf, Global
Azure, Open-Source Lisbon, CodeFrenzy, Mêlée Numérique, React
Summit, Test.js Summit, Pyjamas, JUGs, meetups, hackathons.
@juarezjunior
@juarezjunior

3. Copyright © 2023, Oracle and/or its affiliates
Cloud Native Computing: Kubernetes at the Center
Stage
Kubernetes
• Management of containerized applications, components and services
• Declarative platform definition: YAML
• Non-terminating Control loops
• Service discovery and load balancing
• State Maintenance
• Self-healing & automation robustness
• Storage orchestration
• Automated rollouts and rollbacks
• Secret and configuration management
Observe
Analyze
Act

4. 4
Kubernetes Popularity
Source: https://www.statista.com/statistics/1233945/kubernetes-adoption-level-organization/ Source: https://www.datadoghq.com/container-report/
Copyright © 2023 Oracle and/or its affiliates

5. 5
Kubernetes deployment challenges to address
Kubernetes alternatives
1. RedHat OpenShift
2. Rancher
3. Docker Swarm
4. OKE
Source: https://www.redhat.com/en/resources/kubernetes-adoption-security-market-trends-overview
Copyright © 2023 Oracle and/or its affiliates

6. 6
Container images running in Kubernetes
Copyright © 2023 Oracle and/or its affiliates

7. Oracle’s Strategy
for Cloud Native
Application
Development
7
Copyright © 2023 Oracle and/or its affiliates

8. Governance
OKE Cluster
Data Plane
Worker nodes
Container images
Kubernetes Persistent Volume Claims
Identity and Access
Networking Encryption
Seamless integration with other OCI services
8 Copyright © 2023, Oracle and/or its affiliates
Bare Metal
Compute
Virtual
Machine
OCI VCN OCI VCN Security
Lists/Groups
OCI Container
Registry
OCI VCN
Route Table Auditing
Logging
OCI IAM Groups
OCI Key Vault
Monitoring
OCI File Storage
Container Engine
For Kubernetes
Kubernetes Services type Load Balancer
OCI Block
Storage
OCI IAM Policies
3rd party identity system
OCI Load
Balancer
Application users
federation
Control Plane
Kubernetes API Server
etcd
Node
kubelet
kube-proxy
OS
• X86
• Flex
• Arm
• GPU
OCI Vulnerability
Scanning
CI/CD and Automation
OCI DevOps
OCI Bastion
Resource Manager
(Terraform)
controller
scheduler CCM

9. OCI container-based platform for app development
SECURITY & GOVERNANCE (Secure)
OBSERVABILITY & MANAGEMENT (Operate)
Bastions Certificates Vault VCN NSG Max Security Zones
Identity & Access
Mgmt.
Cloud Guard Data Guard
Logging Log Analytics Monitoring APM Java Mgmt. OS Mgmt. Operations Insights Cloud Advisor
3. Application Runtime (Run)
STORAGE
Block
Storage
Object
Storage
Caching
APPLICATION RUNTIMES
Functions
Container Engine for
Kubernetes (OKE)
APEX
Containerized Runtimes Low Code
TRAFFIC MANAGEMENT
API Gateway
Load
Balancer
1. Languages & Frameworks (Develop)
2. DevOps (Build & Deploy)
DATA & ANALYTICS
ATP ADW
Oracle Analytics
Cloud
Argo
Resource Manager
GitHub
DevOps
Service Mesh
9 Copyright © 2023, Oracle and/or its affiliates | Oracle Confidential: Highly Restricted

10. Container Runtimes for Oracle Database
10
Docker
Running Containers on Docker
Engine
Build Images (SI, Sharding, RAC &
CMAN)
Standalone Environments
OL7 Support only
Podman
Alternative to Docker
Secure than Docker
Build Images (SI, Sharding, RAC &
CMAN)
Standalone Environments
OL8 Support
Kubernetes
Container Orchestration solution
Uses Software Defined
Networking
Provide Observability and Scaling
SI & Sharding
DB Operator
(OraOperator)
DB Specific Controller
Software extension of the
Kubernetes APIs
ADB, BaseDB, SIDB, Sharding &
Multitenant
Database Observability
Ansible
Go
Helm
Podman Compose
Docker Compose
Configuration scripts
Provisioning scripts
Copyright © 2023 Oracle and/or its affiliates

11. Make Oracle Database Container-native and Kubernetes-native - Kubernetes is a popular
• Modern app dev deploys applications in containers for rapid CI/CD pipelines, scale, resilience
• Deploy database in containers for the same advantages
Container-native
• Deploy Oracle Database in container runtime environments (docker, podman, Kubernetes)
• Easy deployment using pre-packaged scripts in the image available for each environment
Kubernetes-native
• Database Operator for Kubernetes (OraOperator) simplifies and automates the
management of Oracle Database lifecycle using Kubernetes commands - kubectl
• Oracle Database running in Kubernetes cluster (containers) or standalone configs
11 Copyright © 2023 Oracle and/or its affiliates
Oracle Database App Simple Mission

12. Container images Docker (OL7) Podman (OL8) Oracle Container Engine for Kubernetes
(OKE), Oracle Cloud Native Env (OCNE)
Single Instance DB (Includes XE, SE,
and EE)
19.16c, 21.3c 19.3c, 21.3c 19.3c, 21.3c
Sharded DB 19.16c, 21.3c 19.3c, 21.3c 19.3c, 21.3c
Real Application Clusters (RAC) 19.16c, 21.3c 19.16c, 21.3c OCNE Planned for 23c GA
Connection Manager (CMAN) 19.3c, 21.3c 19.3c, 21.3c
Oracle Container Based Runtime Environments
Production Database Release
Copyright © 2023 Oracle and/or its affiliates

13. Oracle Database is embracing the use of Containers and Kubernetes
• Supported container images
• Helm chart for configuring these images in Kubernetes
• Operator to allow Kubernetes to manage Oracle databases (in-container or external)
• Adapters for Kubernetes open-source observability and tracing frameworks
All of these are open-source (available on GitHub)
Kubernetes-native helps build:
• MicroTx: Support high-end OLTP with microservices
• Spring Boot Backend: Manage database within Kubernetes
Full Support for Kubernetes
13
Copyright © 2023 Oracle and/or its affiliates

14. Why a Kubernetes Operator for Stateful Applications?
• Stateful Application
• Replicas have own/unique state and identity
• Configuration steps before deployment
• Be in some special state for patching/upgrading
• Complexity and order of special maintenance tasks
(backup, restore, compression, and so on)
• More handholding through the entire lifecycle
• Helm charts are good for deploying stateless apps, templating/reuse; can’t
handle the lifecycle of stateful applications
• In-house scripts cannot fully automate stateful applications lifecycle
• State management in Kubernetes is complex
Oracle CloudWorld Copyright © 2022, Oracle and/or its affiliates | Confidential: Restricted

15. Copyright © 2022, Oracle and/or its affiliates
15
Pod Specification
volumeMounts
• name
• mountPath
volumes
• name
• PVC reference
PVC Specification
Storage Class
Access Mode (rw/ro)
Size File Storage Block Storage
Physical Volume
allocate
Kubernetes: Managing States

16. Oracle Database
Operator for
Kubernetes:
Architecture
16
Copyright © 2023 Oracle and/or its affiliates

17. Kubernetes Built-in Objects and Controllers
• Deployment
• StatefulSet
• Endpoint
• Service
• ReplicaSet
• Job
• CronJob
• DaemonSet
• Node
• Route
• Authorization
• Service Account
• Tokens
• …
Declarative Spec of
Deployments,
Pods, Services,
Volumes
etc
YAML
Controller monitors objects
and reconciles against Specs
Deployments, Pods
Copyright © 2023, Oracle and/or its affiliates

18. Master Node
Kubernetes Cluster
Kubernetes APIs Extended with Custom Objects and
Controllers
API
Server
Operator
(Extends K8s API)
Kubectl
Custom Resources
Specifications
YAML
Custom
Resources
Oracle CloudWorld Copyright © 2022, Oracle and/or its affiliates | Confidential: Restricted

19. Oracle Database Operator for Kubernetes
Oracle CloudWorld Copyright © 2023, Oracle and/or its affiliates
19
Automation to manage an Oracle Database in a Kubernetes Cluster
Supported Database Configurations
1. Containerized Database in Kubernetes
2. Multitenant Databases
3. Autonomous Database
4. BaseDB Service
Supported Lifecycle Operations
Provision / Bind / Start / Stop / Terminate
Backup / Restore / Patch / Upgrade / Scale
Support major K8s Distributions and Clouds
Oracle DB Operator
Kubernetes Cluster
kubectl
Autonomous DBs
Multitenant DB
Containerized Oracle DBs
1
3
2
Base DBs
Oracle Cloud Infrastructure
Controllers
4

20. Why a Kubernetes Operator for the Oracle Database?
1. The Oracle database images as supported on Containers (Docker, Podman) and
deployable on Kubernetes (using Helm Chart and/or scripts)
2. We want to go further and make Oracle Database natively part of the Kubernetes
infrastructure
• Start, stop, monitor, patch, upgrade, and other life-cycle management operations from Kubernetes
3. Databases are Stateful Kubernetes Applications; Helm Charts and Kubernetes
Deployments are not appropriate for managing states
4. Provide these capabilities whether the Oracle database is in the cloud, on-premises, in
Kubernetes, or even outside of Kubernetes
Oracle CloudWorld Copyright © 2022, Oracle and/or its affiliates | Confidential: Restricted

21. OraOperator: Why Should You Care?
• Kubernetes is primordial for your DevOps & GitOps CI/CD pipeline
• You have adopted the Kubernetes operator pattern for automating the lifecycle
management of complex and/or stateful applications e.g., databases
• You are using or plan to use Oracle databases in your DevOps
• OraOperator eliminates the dependency on a human operator or
administrator for the majority of Oracle database operations.
• Oracle is resolute in making the Oracle database Kubernetes-native
• The Oracle database Operator for Kubernetes is open-source
Oracle CloudWorld Copyright © 2022, Oracle and/or its affiliates | Confidential: Restricted

22. Supported DBs Features / Lifecycle Operations
Containerized DB on K8s clusters
Single Instance DB
(SE, EE, XE, 23c Free)
Provision DB Clone DB Patch DB
(in-place/
out-of-place)
Delete DB Update DB config
(FlashBack,
Archivelog,
Forcelog
Update DB Init
Params
Setting up Data
Guard
Data Guard
(Preview)
Create Standby
for SIDB
Add DBs in
DG Config
Manual Switch
over
Connecting to
the Primary
Patching Primary
and Secondary
Sharded DB Deploy Shard
Topology
Add a Shard Delete a Shard Clone DB Shard
OCI-Only DBs
Autonomous DB
(Shared & Dedicated)
Provision ADB Bind ADB Start/ Stop
ADB
Terminate
(Soft/Hard)
Scale
(Up/Down)
Network access
options
On-demand
Backup/Restore
Base DB Provision Bind Scale shape
Up/Down
Scale UP the
storage
On Demand
backup
Terminate a
Base DB
Update License
On-Premises and OCI DBs
Multitenant DB
(Exadata, PDB/CDB)
Bind to a CDB Create a PDB Plug a PDB Unplug a PDB Delete a PDB Clone a PDB Open/Close PDB
Oracle CloudWorld Copyright © 2023, Oracle and/or its affiliates
22
OraOperator v1.0.0 – Production (on July 6)

23. Oracle CloudWorld Copyright © 2023, Oracle and/or its affiliates
23
Roadmap – Next 12 Months
• Integration in DevOps CI/CD Pipelines including support for database
or schema change management tools (Flyway, Liquibase)
• Oracle RAC Controller
• OpenShift certification of v 1.0.0 (done for v0.2.x)
• VMWare Tanzu certification
• Verrazzano integration
• Workflow deployment Integration on OKE Console
• Operator image on GitHub Container Image Registry
• Publishing on operatorhub.io
• Testing on other major Kubernetes distributions/Clouds (AWS, Azure
already done)
• Operator Lifecycle Manager (OLM) support

24. A simplified example of the Oracle DB
Operator integrated into feature-
branching strategies.
• The DB Operator is used to create
and manage a Single-Instance
Databases (SIDB)
Looking into the integration with
database/schema change frameworks
e.g., Liquibase, Flyway
24 Oracle CloudWorld Copyright © 2023, Oracle and/or its affiliates
Oracle Database Operator in DevOps Pipeline

25. Certified and Supported by Red Hat
Copyright © 2023, Oracle and/or its affiliates 25
`
Oracle DB Operator certified on OpenShift
Joint effort with Oracle Product
Management and Product Development
Operator managed from OpenShift
registry and console
All database flavours
Autonomous Exadata
On-Premise PaaS
https://catalog.redhat.com/software/operators/detail/630dd7ead2868f23242cefea
Copyright © 2023 Oracle and/or its affiliates

32. 32
OraOperator Add-on to OKE Console deployment
Copyright © 2023 Oracle and/or its affiliates

33. Demo - steps
33
Copyright © 2023 Oracle and/or its affiliates

34. Use Cases shown
• Binding to an existing ADB
• Provisioning an ADB
• Scaling up a database
• Stopping a database
• Terminating a database
Managing the Autonomous Database (ADB) Lifecycle
Copyright © 2023, Oracle and/or its affiliates
34
`

35. Bind to an existing ADB
Copyright © 2023, Oracle and/or its affiliates
35
kubectl client
machine
Prepare .yaml file for
binding to an ADB
Login cloud console and click
Autonomous Transaction Processing
Click on the database name Copy database OCID
ADB bound
Use the OCID from above
Apply the .yaml file
kubectl apply –f bind_adb.yaml
Verify the resource is available using kubectl
kubectl get adb

36. Scale up an ADB
Copyright © 2023, Oracle and/or its affiliates
36
kubectl client
machine
Scale request sent
Set CPU count in the
earlier .yaml file for
scaling up/down ADB
Scaling in progress
Apply the .yaml file
kubectl apply –f bind_adb.yaml
Verify CPU count of the
database using oci cli
oci db autonomous-database list
--compartment-id $COMPARTMENT_ID
--output table
--query "data [*].{dbname:"display-name", ocpus:"cpu-core-count"}”

37. Stop an ADB
Copyright © 2023, Oracle and/or its affiliates
37
kubectl client
machine
Stop request sent
Set lifecycle state in
the earlier .yaml file for
stopping ADB
Stopping in progress
Apply the .yaml file
kubectl apply –f bind_adb.yaml
To start/terminate the database, repeat the
steps above with lifecycle state
AVAILABLE/TERMINATED
Verify state of the database
using oci cli
oci db autonomous-database list
--compartment-id $COMPARTMENT_ID
--output table
--query "data [*].{dbname:"display-name", state:"lifecycle-state"}”

38. Terminate an ADB using HardLink
Copyright © 2023, Oracle and/or its affiliates
38
kubectl client
machine
Terminate
request sent
Terminating in progress
Delete ADB from the
Kubernetes cluster
kubectl delete –f bind_adb.yaml
Set hard link to true
in the earlier .yaml
file for terminating
ADB
ADB
configured
Apply the .yaml file
kubectl apply –f bind_adb.yaml
Verify state of the database
using oci cli
oci db autonomous-database list
--compartment-id $COMPARTMENT_ID
--output table
--query "data [*].{dbname:"display-name", state:"lifecycle-state"}”

39. Cloud Native Operator Observability
ANY Oracle
Database
MetricsExport
Controller
&
LogExporter Controller
MetricsExporter
ConfigMap, Deployment,
Service, and ServiceMonitor
LogExporter
ConfigMap, Deployment
Grafana, Prometheus,
Loki, Promtail
• MetricsExporter Controller generates all resources needed to export DB metrics in Prometheus for visualization in Grafana
• LogExporter Controller generates all resources needed to export DB logs in Loki for visualization in Grafana
• Custom Resource Definitions simply reference secrets/bindings.
generates
generates
metrics scraped
logs pushed
Copyright © 2023 Oracle and/or its affiliates

40. Copyright © 2021, Oracle and/or its affiliates
40
Observability with EM DB Express console

41. Containers:
Can we run Oracle Database in Docker Containers? Yes, with 19c, and 23c Beta
Can we run Oracle Database in Podman Containers? Yes, with 19c and 23c Beta
Is Oracle Database supported in production use with Kubernetes containers? Yes, various configurations
• https://www.oracle.com/database/kubernetes-for-container-database/
Kubernetes Operator:
https://github.com/oracle/oracle-database-operator
Can we manage Oracle Database using Kubernetes? Yes, OraOperator developer preview available; production coming soon (end of May)
Licensing:
What are the licensing guidelines/restrictions when running Oracle Database in Docker/Podman containers?
None, same pricing per OCPU in containers as in standalone database configurations
Support:
What are support guidelines/restrictions when running Oracle Database in Docker/Podman containers?
Oracle Database using the underlying hardware configuration needs to be supported by Oracle
41
FAQ & References | Production releases 19c (and 23c Beta)
Copyright © 2023 Oracle and/or its affiliates

42. Oracle LiveLabs
Showcasing how Oracle’s solutions can
solve your business problems
500+
free workshops,
available or in
development
3.5 million
people have already visited
LiveLabs
developer.oracle.com/livelabs
learn something new …at your pace!
600+
events run
using LiveLabs
workshops
Copyright © 2023 Oracle and/or its affiliates

43. Oracle
Database
Oracle Database 23c Free – Developer Edition
available for download now
• Oracle Database 23c accelerates Oracle’s
mission to make it simple to develop and
run all data-driven apps
• Provides developers easy early access to
23c app dev features
• Limited database size, memory, CPU
threads (same limits as Express Edition)
Free Developer Release
http://oracle.com/23cFree
http://oracle.com/23cFree

44. Create your FREE
Cloud Account
• Go to
https://signup.cloud.oracle.com/
Copyright © 2023, Oracle and/or its affiliates

45. 3 membership tiers
Connect: @oracleace facebook.com/OracleACEs
aceprogram_ww@oracle.com
500+ technical experts &
community leaders helping peers globally
The Oracle ACE Program recognizes & rewards individuals for
their technical & community contributions to the Oracle community
Nominate
yourself or a candidate:
ace.oracle.com/nominate
Learn more - ace.oracle.com
blogs.oracle.com/ace

46. Juarez Junior
@juarezjunior