Ben Stopford is an engineer and architect working on the Apache Kafka Core Team at Confluent (the company behind Apache Kafka). A specialist in data, both from a technology and an organizational perspective, Ben previously spent five years leading data integration at a large investment bank, using a central streaming database. His earlier career spanned a variety of projects at Thoughtworks and UK-based enterprise companies. He writes at Benstopford.com.

1. The Data Dichotomy:
Rethinking Data and Services
with Streams
Ben Stopford
@benstopford

3. What are
microservices
really about?

4. GUI
UI
Service
Orders
Service
Returns
Service
Fulfilment
Service
Payment
Service
Stock
Service
Splitting the Monolith?
Single Process
/Code base
/Deployment
Many Processes
/Code bases
/Deployments

5. Orders
Service
Autonomy?

6. Orders
Service
Stock
Service
Email
Service
Fulfillment
Service
Independently
Deployable
Independently
Deployable
Independently
Deployable
Independently
Deployable

7. Independence is
where services get
their value

8. Allows Scaling

9. Scaling in terms of people
What happens when we grow?

10. Companies are inevitably a
collection of applications
They must work together to some degree

11. Interconnection is an
afterthought
FTP / Enterprise Messaging etc

12. Internal World
External world
External world
External World
External world
Service
Boundary
Services Force Us To Consider The External World

13. Internal World
External world
External world
External World
External world
Service
Boundary
External World is something we should Design For

14. But this
independence
comes at a cost
$$$

15. Orders
Object
Statement
ObjectgetOpenOrders(),
Less Surface Area = Less Coupling
Encapsulate
State
Encapsulate
Functionality
Consider Two Objects in one address space
Encapsulation => Loose Coupling

16. Change 1
Change 2
Redeploy
Singly-deployable
apps are easy

17. Orders Service Statement ServicegetOpenOrders(),
Independently Deployable Independently Deployable
Synchronized
changes are painful

18. Services work best
where requirements
are isolated in a single
bounded context

19. Single Sign On Business Serviceauthorise(),
It’s unlikely that a Business
Service would need the
internal SSO state /
function to change
Single Sign On

20. SSO has a tightly
bounded context

21. But business
services are
different

22. Catalog Authorisation
Most business services share the
same core stream of facts.
Most services
live in here

23. The futures of
business services
are far more
tightly intertwined.

24. We need encapsulation to hide
internal state. Be loosely coupled.
But we need the freedom to slice
& dice shared data like any other
dataset

25. But data systems
have little to do
with encapsulation

26. Service Database
Data on
inside
Data on
outside
Data on
inside
Data on
outside
Interface
hides data
Interface
amplifies
data
Databases amplify the data they hold

27. The data dichotomy
Data systems are about exposing data.
Services are about hiding it.

28. Service Database
Data on
inside
Data on
outside
Data on
inside
Data on
outside
Encapsulation
protects us
against future
change.
Databases
provide ultimate
flexibility for the
now
These approaches serve different goals

29. This affects
services in one of
two ways

30. getOpenOrders(
fulfilled=false,
deliveryLocation=CA,
orderValue=100,
operator=GreatherThan)
Either (1) we constantly add to the interface,
as datasets grow

31. getOrder(Id)
getOrder(UserId)
getAllOpenOrders()
getAllOrdersUnfulfilled(Id)
getAllOrders()
(1) Services can end up looking like
kookie home-grown databases

32. ...and DATA amplifies
this “God Service”
problem

33. (2) Give up and move whole datasets
en masse
getAllOrders()
Data Copied
Internally

34. This leads to a
different problem

35. Data diverges over time
Orders
Service
Stock
Service
Email
Service
Fulfill-
ment
Service

36. The more
mutable copies,
the more data will
diverge over time

37. Nice neat
services
Service
contract too
limiting
Can we change
both services
together, easily?
NO
Broaden Contract
Eek $$$
NO
YES
Is it a
shared
Database?
Frack it!
Just give me
ALL the data
Data diverges.
(many different
versions of the same
facts)
Lets encapsulate
Lets centralise
to one copy
YES
Start
here
Cycle of inadequacy:

38. These forces compete in
the systems we build
Divergence
Accessibility
Encapsulation

39. Shared
database
Service
Interfaces
Better
Accessibility
Better
Independence
No perfect solution

40. Is there a better
way?

41. Data on the Inside
Data on the Outside
Data on the Outside
Data on the Outside
Data on the Outside
Service
Boundary

42. Make
data-on-the-outside
a 1st class citizen

43. Separate Reads &
Writes with
Immutable
Streams

44. Orders
Service
Stock
Service
Kafka
Fulfilment
Service
UI
Service
Fraud
Service
Event Driven + CQRS +
Stateful Stream Processing
Commands go to
owning services
Queries are interpreted
entirely inside in each
bounded context

45. Event Driven + CQRS +
Stateful Stream Processing
Orders
Service
Stock
Service
Orders
Fulfilment
Service
UI
Service
Fraud
Service
• Events form a central Shared
Narrative
• Writes/Commands => Go to “owning”
service
• Reads/Queries => Wholly owned by
each Service
• Stream processing toolkit

46. Kafka is a Streaming Platform

47. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

48. The Log

49. Kafka is a type of messaging system
Writers Kafka Readers
Important Properties:
• Petabytes of data
• Scales out linearly
• Built in fault tolerance
• Log can be replayed

50. Scaling becomes a
concern of the
broker, not the
service

51. Services Naturally Load
Balance

52. This provides Fault Tolerance

53. Reset to any point in the shared
narrative
Rewind & Replay

54. Compacted Log
(retains only latest version)
Version 3
Version 2
Version 1
Version 2
Version 1
Version 5
Version 4
Version 3
Version 2
Version 1

55. Service Backbone
Scalable, Fault Tolerant, Concurrent, Strongly Ordered, Retentive
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

56. A place to keep the
data-on-the-outside
as an immutable
narrative

57. Now add Stream
Processing

58. Kafka’s Streaming API
A general, embeddable streaming engine
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

59. What is Stream Processing?
A machine for combining and
processing streams of events

60. Features: similar to database query engine
Join Filter
Aggr-
egate
View
Window

61. Stateful Stream Processing adds
Tables & Local State
stream
Compacted
stream
Join
Stream Data
Stream-Tabular
Data
Domain Logic
Tables
/
Views
Kafka
Event Driven Service

62. Avg(o.time – p.time)
From orders, payment, user
Group by user.region
over 1 day window
emitting every second
Stateful Stream Processing
Streams
Stream Processing Engine
Derived “Table”
Table
Domain Logic

63. Tools make it easy to join streams
of events (or tables of data) from
different services
Email Service
Legacy App
Orders Payments Stock
KSTREAMS
Kafka

64. So we might join two event
streams and create a local
store in our own Domain Model
Orders
Service
Payment
Events
Purchase
Events
PurchasePayments

65. Make the projections queryable
Orders
Service
Kafka
Queryable State API
Context
Boundary

66. Blend the concepts of services
communicating and services sharing state
customer orders
catalogue
Pay-
ments

67. Very different to sharing a database
customer orders
catalogue
Pay-
ments
Services communicate and
share data via the log
Slicing & Dicing is embedded in each
bounded context

68. But sometimes you have
to move data

69. Easy to drop out to an
external database if needed
DB of choice
Kafka
Connect
APIFraud
Service
Context
Boundary

70. Oracle
Connector API & Open Source Ecosystem
make this easier
The Log
ConnectorConnector
Producer Consumer
Streaming Engine
Cassandra

71. WIRED Principals
• Wimpy: Start simple, lightweight and fault tolerant.
• Immutable: Build a retentive, shared narrative.
• Reactive: Leverage Asynchronicity. Focus on the now.
• Evolutionary: Use only the data you need today.
• Decentralized: Receiver driven. Coordination avoiding. No
God services

72. So…

73. Architecture is really about
designing for change

74. The data dichotomy
Data systems are about exposing data.
Services are about hiding it.
Remember:

75. Shared data is a reality for
most organisations
Catalog

76. Embrace data that lives and flows
between services

77. Canonical Immutable Streams
Embed the ability to slice and dice into each
bounded context
Distributed
LogEvent Driven
Service

78. Shared
database
Service
Interfaces
Better
Accessibility
Better
Independence
Balance the Data Dichotomy

79. Create a canonical
shared narrative

80. Use stream processing
to handle data-on-the-
outside whilst prioritizing
the NOW

81. Event Driven Services
The Log
(streams & tables)
Ingestion
Services
Services with
Polyglotic
persistence
Simple Services
Streaming Services

82. Twitter:
@benstopford
Blog of this talk at
http://confluent.io/blog