Events are driving a paradigm shift in application development from commands to reactions to events. Event-driven architectures using streaming platforms allow building real-time, scalable applications. Apache Kafka is an event streaming platform that can be used to build event-driven microservices and data pipelines to integrate various data sources. This enables building applications that deliver value to customers through real-time processing of event streams. The public sector can leverage these approaches to enable digital transformation through software-defined, event-driven applications across various domains.

1. Events Everywhere
Enabling Digital Transformation in the Public
Sector

2. 2

3. What’s the big idea?

4. Events vs Commands

5. I walk into a room, flip the light switch,
and the light turns on
This is a Command

6. I walk into a room, generate an “entered room” event
and the light turns on
This is a reaction to an event

7. It is a fundamental paradigm shift of how applications
are developed

8. Event Driven Architectures

9. Aren’t new…
…but…
An Architecture for Well-Conditioned,
Scalable Internet Services
- white paper
Focusing on Events blog post
- Martin Fowler
2001
2006
Your coffee shop doesn’t use two-phase commit
- Gregor Hohpe
2005

10. The world is changing.

11. What is an Event?
SOMETHING
HAPPENED!

12. Events
Something happened
Immutable
Related
- When
Behavioural

13. Business Digitization Trends are Revolutionizing your
Data Flow

14. How did we get here?

15. LinkedIn (pre 2011)

16. Pre-Streaming

17. LinkedIn (post 2011)

19. The Rise of Event Streaming
99%
19
1%
- Non-Transactional “Event Data”
- Transactional “Database State”

21. Event driven apps look like this

22. A series of related events
..is a stream..

23. A stream captures behavior

24. The events are the data…
persisted, in order, in Kafka

25. In a log
like a commit-log in a database

26. Streaming Correctness
delivery guarantees, timestamp,
temporal consistency, scale model

27. Streams?

28. Driven by Events
•
•
•
•
{Survey:100}
{IVR:150}
{Twitter:190}
Stream
processor
Data driven
logic goes here!
{Sat_Score:90}
{Sat_Score:60}{Sat_Score:40}

29. Using external processing systems leads to
complicated architectures
DB CONNECTOR
APP
APP
DB
STREAM
PROCESSING
APPDB
CONNECTOR
CONNECTOR

30. We can put it back together in a simpler way
DB
APP
APP
DB
APP
PULL
PUSH
CONNECTORS
STREAM PROCESSING
STATE STORES
ksqlDB

31. CREATE STREAM unhappy_users AS
SELECT * FROM survey
WHERE sat_score < 40;
With ksqlDB -
Not a single line of Java or Scala code needed.
Stream processing with Kafka

32. Evolvability
user experience?
how many
users affected?
has it happened
before?
new old
supports data change, logic change, logic extension, schema
evolution, loose coupling, add processors, A/B path

33. Case Study: Life as a Stream of Events
https://www.confluent.io/online-talks/how-apache-kafka-helps-to-create-data-culture/
https://www.confluent.io/kafka-summit-sf18/life-is-a-stream-of-events/
Parent
information
Healthcare
provider
Child
benefits
Child birth
registration
Parental
leave

34. OUTCOME:
Build Apps that Deliver Value
To our customers and our business
(Often, this involves data pipelines)

35. Apache Kafka Connect API:
Import and Export Data In & Out of Kafka
Kafka Connect API
Kafka Pipeline
Sources Sinks

36. Instantly Connect Popular Data Sources & Sinks
Data Diode
100+
pre-built
connectors
80+ Confluent Supported 20+ Partner Supported, Confluent Verified

37. Patterns that start to emerge
● Data pipelines (ETL, integration)
● Monitoring and alerting, including log collection and aggregation
● Event-driven microservices
● Enterprise wide event-driven microservices
● IoT
● Customer 360
● Central nervous system/digital nervous system

38. Change Data Capture (CDC)
• Table/Row updates detected from commit log
{
eventKey: user-100
username: felix
address: blah
membership: 1234
}
table: users
connector topic: users

39. Internet of Things
• MQTT
• Device centric
{
eventKey: device-100
lat: 144.123
long: 120.133
value: 0.0007
}
device: geo-sensor
MQTT topic: geo-sensors

40. Microservices
• Events
• Transactional
• Plus some magic with local-state
{
eventKey: user-100
username: felix
address: london
membership: 123
}user-enrichment
topic: enriched-users
Stream
processor

41. New Streaming-first Paradigm
• Scalable
• Real-time
• Stream processing
Real-Time
Inventory
Real-Time
Fraud
Detection
Real-Time
Customer 360
Machine
Learning
Models
Real-Time
Data
Transformation
...
Contextual Event-Driven Applications
Universal Event Pipeline
Data Stores Logs 3rd Party Apps Custom Apps/Microservices
EVENT STREAMING
PLATFORM
ksqlDB

42. Every Industry is Moving from Batch/Manual
to Software-Defined
Auto / Transport
Monolithic Applications Software-defined
Spreadsheet-driven driver schedule Real-time ETA
Banking Nightly credit-card fraud checks Real-time credit card fraud prevention
Retail Batch inventory updates Real-time inventory management
Healthcare Batch claims processing Real-time claims processing
Oil and Gas Batch analytics Real-time analytics
Manufacturing Scheduled equipment maintenance Automated, predictive maintenance
Cybersecurity Reactive cyber-security forensics Automated SIEM and Anomaly Detection
U.S. Defense
Agencies
42

43. How ‘Events Everywhere’ can
Enable Digital Transformation in
the Public Sector!

44. Free eBooks
Kafka: The Definitive Guide
Neha Narkhede, Gwen Shapira, Todd
Palino
Making Sense of Stream Processing
Martin Kleppmann
I ❤ Logs
Jay Kreps
Designing Event-Driven Systems
Ben Stopford
http://cnfl.io/book-bundle

45. Interacting with Confluent
45
Join the Confluent Slack Channel
https://launchpass.com/confluentcommunity
Local meetups
https://www.confluent.io/community/
Try a hands-on demo
https://github.com/confluentinc/examples
Developer resources
https://developer.confluent.io/

46. Project Metamorphosis
Unveiling the next-gen event
streaming platform
Listen to replay and
Sign up for updates
cnfl.io/pm
Jay Kreps
Co-founder and CEO
Confluent

47. Thank you!
More talks at
www.confluent.io/events/
Kafka Summit
www.kafka-summit.org