Presentation for JAX London 2016 on Kafka, Streaming and Microservices.

2. Microservices
in a Streaming
World

4. What are
microservices
really about?

5. GUI
UI
Service
Orders
Service
Returns
Service
Fulfilment
Service
Payment
Service
Stock
Service
Collaboration?

6. Orders
Service
Autonomy?

7. Internal World
External world
External world
External World
External world
Service
Boundary

8. User-Orders StatementgetOpenOrders(),
Less Surface Area = Less Coupling
Encapsulate
State
Encapsulate
Functionality
Encapsulation => Loose Coupling

9. Independence
gives services
their value

10. Orders
Service
Stock
Service
Email
Service
Fulfillment
Service
Independently
Deployable
Independently
Deployable
Independently
Deployable
Independently
Deployable

11. It’s hard to scale individual applications
What happens when we grow?

12. Companies are inevitably a
collection of applications
They must work together to some degree

13. Inverse Conway Maneuver
The 'Inverse Conway Maneuver' recommends evolving your
team and organizational structure to promote your desired
architecture. !
Org Structure
Software Architecture

14. But independence
comes at a cost
$$$

15. Change 1
Change 2
Redeploy
Singly-deployable
apps are easy

16. User-Orders StatementgetOpenOrders(),
Independently Deployable Independently Deployable
Synchronized
changes are painful

17. Single Sign On Business Serviceauthorise(),
It’s unlikely that a Business
Service would need the
internal SSO state /
function to change
Single Sign On

18. SSO has a tightly
bounded context

19. Business services
are different

20. Catalog Authorisation
Most business services share the
same core stream of facts.
Most services
live in here

21. We need encapsulation to hide
internal state
But we need the freedom to slice
& dice shared data like any other
dataset

22. But data systems
have little to do
with encapsulation

23. Service Database
Data on
inside
Data on
outside
Data on
inside
Data on
outside
Interface
hides data
Interface
amplifies
data
Databases amplify the data they hold

24. The data dichotomy
Data systems are about exposing data.
Services are about hiding it.

25. This affects
services in one of
two ways

26. getOpenOrders(
fulfilled=false,
deliveryLocation=CA,
orderValue=100,
operator=GreatherThan)
Either (1) we constantly add to the interface,
as datasets grow

27. getOrder(Id)
getOrder(UserId)
getAllOpenOrders()
getAllOrdersUnfulfilled(Id)
getAllOrders()
Services can end up looking like kookie
home-grown databases

28. ...and data amplifies
this service
boundary problem

29. (2) Give up and move whole
datasets en masse
getAllOrders()
Data Copied
Internally

30. This leads to a
different problem

31. Data diverges over time
Orders
Service
Stock
Service
Email
Service
Fulfill-
ment
Service

32. The more
mutable copies,
the more data will
diverge over time

33. Nice neat
services
Service
contract too
limiting
Can we change
both services
together, easily?
NO
Broaden Contract
Eek $$$
NO
YES
Is it a
shared
Database?
Frack it!
Just give me
ALL the data
Data diverges.
(many different
versions of the same
facts)
Lets encapsulate
Lets centralise
to one copy
YES
Start
here

34. There is competition
between these forces
DivergenceAccessibilityEncapsulation

35. Shared
database
Service
Interfaces
Better
Accessibility
Better
Independence
No perfect solution

36. Is there a better
way?

37. Data on the Inside
Data on the Outside
Data on the Outside
Data on the Outside
Data on the Outside
Service
Boundary

38. Make
data-on-the-outside
a 1st class citizen

39. Use a Stream
Processing tool-kit

40. Kafka is a Streaming Platform

41. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

42. The Log?

43. Shard on the way in
Producing
Services
Kafka
Consuming
Services

44. Each shard is a queue
Producing
Services
Kafka
Consuming
Services

45. Consumers share load
Producing
Services
Kafka
Consuming
Services

46. Load Balanced Services

47. Fault Tolerant Services

48. Build ‘Always On’ Services
Rely on Fault
Tolerant Broker

49. Services can “Rewind & Replay” the
log
Rewind & Replay

50. Compacted Log
(retains only latest version)
Version 3
Version 2
Version 1
Version 2
Version 1
Version 5
Version 4
Version 3
Version 2
Version 1

51. Two primitives:
(a) Log (append only)
(b) Compact-Log (update)

52. Service Backbone
Scalable, Fault Tolerant, Concurrent, Strongly Ordered, Stateful
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

53. A place to keep
data-on-the-
outside

54. Now add Stream
Processing

55. KStreams: A database engine for data-in-flight
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

56. Max(price)
From orders
where ccy=‘GBP’
over 1 day window
emitting every second
Continuously Running Queries

57. Features: similar to database query
engine
JoinFilter
Aggr-
egate
View
Window

58. What is Stream Processing?
Table
Index
Query
Engine
Query
Enginevs
Database
Finite source
Stream Processor
Infinite source

59. What is Stateful Stream Processing?
Table
Index
Query
Engine
Query
Engine
vs
Database
Finite source
Stateful Stream Processor
Infinite & Finite source
Table
Compacted Stream
KStreams
Kafka

60. Stateful Stream Processing
stream
Compacted
stream
Join
Stream Data
Stream-Tabular
Data
Windowed
Stream
Locally Cached
Table
(disk resident)
KafkaKafka Streams

61. Email Confirmation Example
Orders
Customer
(Compacted)
Join
Customer Stream
Create Confirmation
Emails by Joining
Order “Events” with
Customer
information
Kafka
Email Service
Orders Stream

62. Scales Out

63. Embeddable
Orders Service
Kafka
Orders Service
Orders Service

64. Analytic function, Event
Driven, all in one.
Analytics Event Driven
Business
Services

65. Join shared datasets without putting
pressure on source services
Email Service
Legacy App
Orders Payments Stock
KSTREAMS

66. Shared State is only cached in the service,
so there is no way to diverge
Email Service
Orders Payments Stock
Lives here!
Cached here!
KSTREAMS

67. Services work “Event Driven”, but with the
power to join, filter and aggregate embedded in
each service
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

68. Services work “Event Driven”, but with
the power to join, filter and
aggregate embedded in each service
customer orders
catalogue
Pay-
ments

69. But sometimes you have
to move data

70. Replicate it, so both copies
are identical
Keep Immutable

71. Iterate via view
regeneration

72. Regenerate
from the
Log
Fix at
source
If you have to move data en masse fix at
source and regenerate from the log

73. Oracle
Connectors make this easier
The Log
ConnectorConnector
Producer Consumer
Streaming Engine
Cassandra

74. So…

75. When building
services consider
more than just
REST

76. The data dichotomy
Data systems are about exposing data.
Services are about hiding it.
Remember:

77. Shared data is a reality for
most organisations
Catalog

78. Embrace data that lives
in between services

79. Avoid data services that
have complex / amplifying
interfaces

80. Share Data via Immutable Streams
Embed Function into each service
Distributed
LogEmbedded
KStreams

81. Always on, Event-Driven Services
The Log
(streams & tables)
Ingestion
Services
Services with
Polyglotic
persistence
Simple Services
Streaming Services

82. Keep it simple,
Keep it moving
Twitter:
@benstopford
Slides at
benstopford.com
Blog series arriving
end of the month at
http://confluent.io/blog