Microservices became the new black in enterprise architectures. APIs provide functions to other applications or end users. Even if your architecture uses another pattern than microservices, like SOA (Service-Oriented Architecture) or Client-Server communication, APIs are used between the different applications and end users. Apache Kafka plays a key role in modern microservice architectures to build open, scalable, flexible and decoupled real time applications. API Management complements Kafka by providing a way to implement and govern the full life cycle of the APIs. This session explores how event streaming with Apache Kafka and API Management (including API Gateway and Service Mesh technologies) complement and compete with each other depending on the use case and point of view of the project team. The session concludes exploring the vision of event streaming APIs instead of RPC calls.

1. Kai Waehner
Field CTO
contact@kai-waehner.de
linkedin.com/in/kaiwaehner
@KaiWaehner
www.confluent.io
www.kai-waehner.de
Event Streaming vs. API Gateway / API Management
Friends, Enemies, or Frenemies?

2. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
1. APIs
2. Event Streaming and API Tools
3. Streaming Data Exchange
4. Streaming Service Mesh
Agenda

3. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
1. APIs
2. Event Streaming and API Tools
3. Streaming Data Exchange
4. Streaming Service Mesh
Agenda

4. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
What is an API?
• API = Application Programming Interface
• Defines interactions between multiple
software applications
• Defines the calls or requests, data formats,
conventions, extension mechanisms
• Information hiding enables modular
programming, and usage of the interface
independently of the implementation
• Not related to any specific technology
• Not always point-to-point

5. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
API Business Models
5
https://www.infoq.com/presentations/API-Business-Models/

6. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
What is a Standard API?
• Often powered by an organization
or industry consortium
• Vendor-agnostic
• Slow evolution
• Inflexible

7. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
The Evolution of Business to Business (B2B) Communication
Traditional B2B
• Complex interfaces
• Generic: EDI, RosettaNet, XBRL
• Industry specific: Swift (Banking), PNRs (Aviation),
HL7 (Healthcare), etc.
• Proprietary, inflexible, limited scale, batch
7
Modern B2B
• API- and data-driven
• Open, flexible, agile, scalable, real-time
• Valueable ecosystem and partnering

8. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Electronic Data Interchange (EDI)
EDI (including X12, EDIFACT, ODETTE, etc.) exists since the early 70s (until today)
Electronically communicating information that was communicated on paper
8

9. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
XML Schema
XPath
SOAP
WSDL
XML
The World of XML-based Web Service Standards
Often called “WS-* Hell” due to its complexity
</XML>
WS-Security
WS-Addressing
WS-Policy
Many more… L

10. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
What is an De Facto Standard API?
• Originates from an existing
successful solution
• Driven by a single vendor or by an
open community
• Practical processes and rules to
ensure good quality and consensus
• No complex, formal, long-running
standard processes

11. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
== de facto standard API for OBJECT STORAGE
Proprietary REST-based De Facto Standard API
https://www.kai-waehner.de/blog/2021/05/09/kafka-api-de-facto-standard-event-streaming-like-amazon-s3-object-storage/

12. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Is there a technology standard for APIs?
12
Use HTTP/REST
for every API!

13. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
1. APIs
2. Event Streaming and API Tools
3. Streaming Data Exchange
4. Streaming Service Mesh
Agenda

14. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
API Communication Paradigms
Request-Response
• Low latency
• Typically synchronous
• Point-to-point
• “Bespoke API”
• e.g. HTTP, SOAP, gRPC, Amazon S3
14
Event Streaming
• Messaging / Pub Sub (sending data from A to B and C)
• Continuous data processing (filtering, transformations,
aggregations, business logic)
• Asynchronous with real decoupling
• Event-driven, supporting patterns like Event Sourcing and CQRS
• General-purpose events
• e.g. Apache Kafka, Amazon Kinesis
Most enterprise architectures require
event streaming and request-response!

15. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Open Source De Facto Standard API
== de facto standard API for EVENT STREAMING
https://www.kai-waehner.de/blog/2021/05/09/kafka-api-de-facto-standard-event-streaming-like-amazon-s3-object-storage/

16. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
API Development Workflow
16
https://www.slideshare.net/KaiWaehner/a-new-front-for-soa-open-api-and-api-management-as-game-changer
X
Any API

17. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
API Gateway and API Management Tools
Common features:
• API Portal for creating and publishing APIs
• Enforcing usage policies and controlling access
• Technical features for data transformations
• Nurturing the subscriber community
• Collecting and analyzing usage statistics
• Reporting on performance
• Monetization and billing
17
API == REST/HTTP
for most of these products

18. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Business Digitalization Trends are Driving the Need to Process
Events at a whole new Scale, Speed and Efficiency
The World has Changed
Mobile Cloud Microservices Internet of Things Machine Learning

19. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
This is a fundamental paradigm shift...
19
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

20. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Apache Kafka is a Platform for Data in Motion
REST/HTTP is not compatible with the scale and speed of Event Streaming
MES
ERP
Sensors
Mobile
Customer 360
Real-time Alerting
System
Data warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
20

21. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Kafka Makes Your Business Real-time and Scalable.
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB

22. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Kafka Connect
Kafka Cluster
Salesforce CRM
Kafka Makes Your Microservices really decoupled from each other.
Mainframe
Custom
Application
3rd Party
Middleware
Java / C++ / Go /
JavaScript / ksqlDB /
etc.
Schema Registry
Event Streaming Platform
CRM Domain Payment Domain Fraud Domain

23. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Kafka and API Gateway / API Management
complement your Enterprise Architecture.
Orders Customers
Payments
Stock
REST
JMS
ESB
REST
CRM
Mainframe
SOAP
…
Kafka
Kafka
Kafka
API Management
HTTP
https://www.kai-waehner.de/blog/2020/05/25/api-management-gateway-apache-kafka-comparison-mulesoft-kong-apigee/
SOAP

24. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Kafka and API Gateway / API Management
complement your Enterprise Architecture.
(Mulesoft Whitepaper)

25. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Overlapping Features between Kafka and API Tools
• Protocol conversion: JSON, Avro, Protobuf, XML, …
• ETL (Extract Transform Load): Transformations, filtering,
sorting and similar tasks.
• Connectivity: Integration with back-end systems like
databases, data warehouses, data lakes, messaging
systems, business applications.
• Gateway: Routing, public endpoints, single entry point,
access control, encryption, throttling…
Separation of Concerns!
Avoid complex monoliths!
Choose the right tool for the job!

26. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
1. APIs
2. Event Streaming and API Tools
3. Streaming Data Exchange
4. Streaming Service Mesh
Agenda

27. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Streaming Data Exchange
Streaming Replication and API Management
MirrorMaker 2
Confluent Replicator
Confluent Cluster Linking
Tier 1 Mobility
Service
Streaming integration
between companies
API Management
with REST/HTTP is
not appropriate
for streaming data
OEM

28. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Event Streaming becomes the standard for many APIs
https://engineering.salesforce.com/how-apache-kafka-inspired-our-platform-events-architecture-2f351fe4cf63 https://medium.com/salesforce-architects/the-new-salesforce-event-bus-f82165cb0585

29. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Here Technologies
Majority-owned by a consortium of German automotive companies (namely Audi, BMW, Daimler) and Intel
Captures location content such as road networks, buildings, parks and traffic patterns
Sells or licenses mapping content, along with map related navigation and location services to other businesses
https://developer.here.com/documentation/data-client-library/dev_guide/client/direct-kafka.html

30. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Seldon
Machine Learning and Model Deployment
Kafka-native Streaming Model Server
Streams
Input Event
Prediction
Model Serving
Kafka Protocol
Streaming Communication
Streaming Application
Broker
https://www.kai-waehner.de/blog/2020/10/27/streaming-machine-learning-kafka-native-model-server-deployment-rpc-embedded-streams/

31. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Most API Tools (somehow) integrate with Kafka
== Streaming API Management…
31

32. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
32
HTTP(S)
API Management
Cluster
REST API
Client App with
HTTP Kafka Producer /
HTTP Kafka Consumer
HTTP(S)
REST API for Kafka
Workaround for the Integration of Kafka and API Management Tools

33. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Confluent REST API for Kafka
Workaround for the Integration of Kafka and API Management Tools
33
Self-managed
Dedicated node
.
Self-managed
Broker plugin
Managed
Confluent Cloud
Confluent
Cloud
REST
Confluent
Server
REST
Kafka
REST
Proxy
HTTP(S)
API Management
Cluster
REST API
Client App with
HTTP Kafka Producer /
HTTP Kafka Consumer
HTTP(S)

34. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
1. APIs
2. Event Streaming and API Tools
3. Streaming Data Exchange
4. Streaming Service Mesh
Agenda

35. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Service Mesh – The Next Generation API Gateway
A microservice pattern to move visibility, reliability, and security
primitives for service-to-service communication into the
infrastructure layer, out of the application layer.
https://www.infoq.com/articles/linkerd-v2-production-adoption/
35

36. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Service Mesh Features
• Metrics without instrumenting apps
• Trace flow of requests across services
• One stable URI for each service
• Service discovery
• Monitor request latency
• Routing - A/B testing, green/blue deployments
• Circuit breaking
• Protocol translation (HTTP, gRPC, Kafka Protocol, etc.)
• Mutual TLS (mTLS)
• SSL Termination
• Integrate with 3rd party tools like Prometheus, Grafana, Zipkin, etc.
• Much more…
Observability
“is by far the most important thing that a Proxy and the Service Mesh
provide in a distributed Microservice architecture!” Matt Klein
36

37. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Kafka Connect
Kafka Cluster
Salesforce CRM
Kafka and Service Mesh are complementary.
Mainframe
Custom
Application
3rd Party
Middleware
Java / C++ / Go /
JavaScript / ksqlDB /
etc.
Schema Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Control
Plane
37
https://www.kai-waehner.de/blog/2019/09/24/cloud-native-apache-kafka-kubernetes-envoy-istio-linkerd-service-mesh/

38. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Example: Kafka + Envoy + Confluent REST Proxy
Envoy
Proxy
I am using REST too!
Kafka?
Never heard of her.
I’m using REST to
talk to a service
I’m proxying
REST.
And also logging
stuff to Kafka
Confluent
REST Proxy
I only support TCP!
HTTP
HTTP
38

39. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Example: Kafka + Envoy Kafka Protocol Filter
Envoy
Proxy
I am using REST too!
Kafka?
Never heard of her.
I’m using REST to
talk to a service
I’m proxying
REST.
And also logging
stuff to Kafka
HTTP
TCP
(Kafka Protocol)
39

40. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Key Takeaways
• Agility is key for businesses in 202X.
• There is no silver bullet - choose the right APIs and technologies.
• Apache Kafka decouples (all) services, including event streams and request-response.
• A Streaming Data Exchange is the future of event-based data transfer.
40

41. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
The Rise of Data in Motion
à Next: A Streaming Data Exchange
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
43

42. Apache Kafka and API Gateways / API Management – @KaiWaehner - www.kai-waehner.de
Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.
https://www.kai-waehner.de/blog/2021/04/20/comparison-open-source-apache-kafka-vs-confluent-cloudera-red-hat-amazon-msk-cloud/

43. Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!