(Jeff Sharpe + Alex Srisuwan, Capital One) Kafka Summit SF 2018 Using Kafka as a platform messaging bus is common, but bridging communication between real-time and asynchronous components can become complicated, especially when dealing with serverless environments. This has become increasingly common in modern banking where events need to be processed at near-real-time speed. Serverless environments are well-suited to address these needs, and Kafka remains an excellent solution for providing the reliable, resilient communication layer between serverless components and dedicated stream processing services. In this talk, we will examine some of the strengths and weaknesses of using Kafka for real-time communication, some tips for efficient interactions with Kafka and AWS Lambda, and a number of useful patterns for maximizing the strengths of Kafka and serverless components.

1. Confidential
Jeff Sharpe & Alex Srisuwan
Capital One
Kafka for Real Time Event Processing in
Serverless Environments

2. 2Public
Who we are and what we hope to get to
Jeff Sharpe
Alex Srisuwan
Software Engineers for Capital One
Retail Bank, Data Strategy & Architecture
Tysons Corner, VA
• Laying out the foundation
• The example use case
• Serverless Kafka
• Request-Response pattern
• Lessons (ie: failures)
• Other options

3. 3Public
Realism
real time: adj.
1. Happening in the exact moment that
real life is currently occurring.
2. Now (or maybe now, no … now)
2. (computing) Being processed within
milliseconds of a triggering event, give or
take a minute or so.
Kafka: n.
1. A streaming platform for quickly and
dependably accepting and distributing
messages between applications that may
be neither quick nor dependable.

4. 4Public
Your software, and it doesn’t matter who owns the hardware
Serverless
but also serverful, or containerous.

5. 5Public
We did say it was a use case presentation
Catching Fraud at the Teller

6. 6Public
About 80% of what we’re allowed to explain

7. 7Public
{λ Background}
Function Execution
Generic
Worker
Initialize
Execution
Environment
Load Code Execute
Execute
Frozen
Function
Worker

8. 8Public
{Kafka & lambda}
Serverless Kafka
Pretty much just Kafka, except:
• Maintaining connections in AWS Lambda isn’t as simple as you’d like.
• Committing offsets is suddenly more important

9. 9Public
{Kafka & lambda}
Optimizing for Serverless
• AWS Lambdas are short lived, but maybe not as short as you expect
• Connections can live across invocations, but might not
• Producers can be simple, but...
• Consumers need thoughtful configuration

10. 10Public
{Kafka & lambda}
(probably) Non-Obvious Hangups
• After completion, Lambdas are frozen in a Linux STOP-like state
• Frozen Lambdas do not respond to any messaging from the brokers
• When Lambdas thaw, you need to check if connectivity still exists
• Remember to commit offsets before sending the Lambda response

11. 11Public
There and Back Again, a Message’s Tale
Request-Response Pattern
aka: Kafka as an entire message bus

12. 12Public
What the Request-Response pattern feels like
λ Data Service

13. 13Public
Service
What the Request-Response actually is
Data
Data
Data
λ
Request Topic
Data Data Data Data Data Data
Response Topic
Data Data Data Data
Data
Data
Data

14. 14Public
Making things a bit smoother
• Connect the Consumer before setting up the
Producer.
• Perform a read on the Consumer before submitting
your request via the Producer.
• Explicitly assign Consumer partitions

15. 15Public
What the Request-Response pattern does when things aren’t perfect
Service
Data
Data
λ
Request Topic
Data Data Data Data Data
Data
Response Topic
Data Data Data
DataData
Data
Data
Data
Data
Data
Data

16. 16Public
An interesting but ultimately completely fabricated bar chart

17. 17Public
No one reads titles up here
Request-Response Pattern ≅ Background Job Pattern

18. 18Public
Solution: Stop using Kafka?
Alternative approach: Kafka Write-Through/Tap

19. 19Public
Not Always Better, but Sometimes Much Better
Write Before / Write Through
• Deliver to Kafka before/during
a real-time request.
• This works well for producing a
record of processing; not so
well if you need the results
from stream processing.
λ
Real-Time
Service
Stream
Processing
or Storage
Kafka
REST/gRPC
REST/gRPC

20. 20Public
Sometimes “More Complex” actually is an improvement
Write After / Tap
• Offload stream processing
behind a topic. Retrieve results
from a cache.
• Predictable consistency so long
as you can respond without
stream processing results.
λ
Real-Time
Service
Stream
Processing
REST/gRPC
REST/gRPC
Kafka
Cache

21. 21Public
404: Next Slide Not Found
Jeff Sharpe
Alex Srisuwan
Software Engineers for Capital One
Retail Bank, Data Strategy & Architecture
Tysons Corner, VA
This space left intentionally blank.
Feel free to fill in with questions,
comments, or corrections.