Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Pivotal Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value.
Apache Kafka® is providing developers a critically important component as they build and modernize applications to cloud-native architecture.
This talk will explore:
• Why cloud-native platforms and why run Apache Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Demo: Running Apache Kafka as a Streaming Platform on Kubernetes
3. Changing Model for Application Delivery
Development Deployment
Sparingly at
designated times
Ready for prod at
any time
Architecture
Abstraction
Layer
“Day 2” Ops
App Server on Machine
App on “disposable”
infrastructure
App and Data
Monolith
Microservices /
Composite app
Linear / Sequential
Short cycles, test
driven, iterative
Many tools, ad hoc
automation
Manage services,
not servers
4. Driven by Modern Cloud Native Platform
Cont
inuous Del
ivery
An idea in the morning can ship by evening
Microservices
Release #1
Microservices
Release #2
7. Hardware
IaaS
CaaS
PaaS
FaaS
Strategic goal: Push as many workloads as
technically feasible to the top of the platform hierarchy
Higher flexibility and
less enforcement of
standards
Lower development
complexity and higher
operational efficiency
10. Goal: Operational consistency AND multiple
abstractions
FREEDOM: Run Your App on Any Cloud
AUTOMATION: Lowers Ops Cost and Increases Velocity
RESILIENCY: Auto-Scale, Blue-green Deploys, Health Mgmt.
GOVERNANCE: Common Developer-Self-Serviced Frameworks
VISIBILITY: Full Stack Logging, Monitoring & Auditability
SPEED & SCALE: Cut Cycle Time from Months to Days/Hours
Built for Cloud
Native
New Initiatives
Existing Workloads
Modernize to
Cloud-Native?
11. Outcomes trump technology bingo
Needs High Dev Efficiency
Needs Low Dev Efficiency
App Rarely changes App Frequently changes
• Traditional
apps
• Leave as-is
• Re-architect
• Replatform
• CI/CD
• Evaluate
• Evaluate
12. Evaluating the Enterprise App Portfolio
Business
Technical Economic
Score
• Business criticality
• Risk tolerance
• Change frequency
• Lifecycle stage
• Domain expert availability
• Licensing costs (HW / SW)
• Time-to-market implications
• Revenue impact-fullness
• Codebase
• Suitable framework / runtime
• Relatively lightweight footprint
• No “hard wired” file system
dependencies
• Usage / workload
For each app
13. Application Transformation Strategies
Re-Host
Containerize aka
“lift and shift” to
CaaS and gain
infrastructure +
platform ops benefits
Re-Platform
Minimal readiness
effort to run on PaaS
(4 -7 factor apps +
data service) that
can run on PaaS
Re-Factor
Convert to cloud-
native/microservice
(12-15 factor app +
data service) runs
well on PaaS/FaaS
Re-Build
Decompose and
rewrite from the
ground up
Application Portfolio
Funneling Process
Learn More: Tools and Recipes to Replatform Monolithic Apps to Modern Cloud Environments
14. Evolution of the enterprise application portfolio
Learn More: Microservices, Events, and Breaking the Data Monolith with Kafka
Microservices
or Dist. Monolith
16. “The world’s most valuable resource is no longer oil, but data” - The
Economist, May 6 2017
17. Apache Kafka is a Distributed Event
Streaming Platform
Process streams of events In real time, as they occur
110101
010111
001101
100010
Publish and subscribe to
streams of events
Similar to a message queue or
enterprise messaging system
110101
010111
001101
100010
Store streams of events In a fault tolerant way
110101
010111
001101
100010
19. What is a company?
A business is a series of events and
reacting to those events.
20. Events enable Microservice Architectures -
Let’s refactor!
Submit
Order
Orders
Service
Shipping
Service
Customer
Service
Webserver
E-Commerce
Microservices (with RPC)
- Orders Service calls
Shipping Service to tell it
to ship item.
- Shipping service looks
up address to ship to
(from Customer Service)
- No Kafka 😢shipOrder() getCustomer()
21. Refactoring Orders and Shipping
Message Broker (Kafka)
Submit
Order
Order
Created
getCustomer()
RPC
Orders
Service
Shipping
Service
Customer
Service
Webserver
- Orders Service no longer
knows about the Shipping
service (or any other service).
- Events are fire and forget.
25. Why run Kafka on
Kubernetes?
33% of Confluent
Customers plan to
run Production
Kafka on Kubernetes
26. Microservices - driving container and
container orchestration adoption
Breaking down the monolith - enables dev teams to work and
scale independently
µ
Containers allow services to be independently deployable services
● Each service has its own purpose & implementation language
● Each service has its own dependencies
Orchestration - running containers at scale
● Microservices = container deployments at scale
● Service discovery
● Scale services independently
● High Availability, Self Healing of services
● Hardware Abstraction
28. There are some challenges
Translating an existing architecture to Kubernetes: K8s
pods, nodes, stateful sets, custom resources
External access: What do we do with advertised listeners
on Kubernetes?
Managing configuration of Kafka, Zookeeper and all
Confluent Components
Persistent Storage options on prem and clouds
Security Configuration and Upgrades
29. Confluent Operator enables you to:
Automate provisioning of Kafka
pods in minutes
Monitor SLAs through Confluent
Control Center or Prometheus
Scale Kafka elastically &
Automate rolling updates
Built on our first hand knowledge of
running Confluent at scale
Confluent Operator
is our Kubernetes
Operator and our
first step towards
becoming cloud
native
CONFIDENTIAL
30. Helm - The Confluent Operator Package
Manager ● Confluent Operator leverages Helm Charts to
deploy, upgrade and uninstall Confluent
Platform custom resources and pods
● Configuration front end for users to specify
how a Confluent Platform Cluster is deployed:
○ # of replicas for Kafka, Zookeeper
○ Security and Authentication configuration
○ Persistent Storage configuration
● Cluster configuration edits are also performed
using Helm
Operator Helm
Charts - yaml
32. Confluent Operator - Scale Horizontally
Automate Scaling:
Spin up new brokers, connect workers
easily
Distribute partitions to new brokers:
Determine balancing plan
Execute balancing plan
Monitor Resources
33. Confluent Operator - Rolling Upgrade of all
components
Automated Rolling Upgrades of all
components - Kafka Brokers,
Zookeeper, Connect, Control Center
Kafka Broker Upgrades:
1. Stop the broker, upgrade Kafka
2. Wait for Partition Leader reassignment
3. Start the upgraded broker
4. Wait for zero under-replicated partitions
5. Upgrade the next broker
34. Confluent Operator - Automated Security
Configuration
SASL PLAIN and Mutual TLS
Authentication
Automate configuration of truststores
and keystores with secret objects
Automate configuration of Kafka and
all Confluent Platform Components
35. Confluent Operator GA Plans
We are in private Preview Release now
24 customers testing the Operator in Preview:
● Global customers
● Banks, Fin Tech, Retailers, Consumer Tech
We are in the final stages of Preview and about to launch soon.
38. Demo – Provision a Complex Cluster in K8s
with Confluent Operator
● Provision:
○ 3 Kafka Brokers, 3 Zookeeper Nodes
○ Confluent Control Center
○ Provisioned on GKE (7 node cluster)
● Mutual TLS Authentication, External Access via Service Endpoints, DNS
● Pub/Sub example (remote access to Kafka Cluster)