Talk given to both Confluent and IRM conferences on the advantages of moving to a Microservices architecture using Kafka

1. London
18th November 2019
David M. Walker
Data
Management &
Warehousing

2. The next 100 milliseconds might define my day …

3. … it’s all OK …
“Cappuccino for
David”
“Thanks”
“Have A Nice Day!”

4. But what if it’s not?
• Do I have another
card with me?
• Do I have time for
this right now?
• Did I pay the card
bill last month?
Or
• Is it the banks fault?

5. By 9am this morning I can easily have made 3 card transactions,
created 5 social media events, and any number of other digital events
e.g. FitBit, Ring, Nest, Flight Check-In, etc.
Millions of people are now creating billions of digital events
about every aspect of their lives every second of every day

6. … but hang on, that’s not even me …
Cyber-crime is a threat
to the customer:
• Fraud
• Identity Theft
and the business:
• Hacking
• DDoS
• etc.
And we need to modify
our platforms to
protect our customers
and out business

7. And even if they don’t
want it now the
expectation bar is being
raised by their
experiences elsewhere!
• Customers Want More Information
• Do you actually have it in stock – can I get it
right now?
• What’s my up-to-date balance?
• Customers Want Faster Information
• Can I see that on my mobile?
• Has that deposit cleared yet?
• Customers Want To Be Notified
• Here’s where I’ve put the parcel that I’ve just
delivered
• There’s been a gate change – you need to be
at Gate H7
• Customers Need To Be Protected
• Is This Fraud ? – but they have a low
tolerance for false positives

8. Can My Organisation Adapt To Deliver ?
• Innovators, Start Up and
Digital Native companies
have been built in a different
way to both generate and
meet these customer
demands
• Traditional organisations are
encumbered with an existing
infrastructure of on-premise
monolithic platforms

9. In traditional organisation we need to evolve our approach to
building new platforms and upgrading existing platforms …
Micro Services Architecture
underpinned by Streamed Events
Data
Governance
Agile
Development
Automation &
Self-Service
Data Science
& AI

10. … to meet the demands from the business …
More For Less
Greater productivity when
creating or enhancing
technology solutions and
delivered by fewer people and a
shorter time to market
Cost Reduction
Paying less for the technology
stack and support teams
required to run the business
solutions
Risk Management
The reduction and/or mitigation of
risks that the business faces from
technology including fraud
protection, legislative and industry
standard compliance, data security,
platform availability, delivery, etc.
Better Analytics & Reporting
The ability for decision makers to access the data
that they need in order to make informed
decisions and execute strategically rather than
operate tactically. Where possible the operational
must be as automated as possible.
Differentiated Product with a
Reduced Time to Market
Technology that helps create a competitive
advantage in the commercial world or
demonstrates value add in a non-profit
environment
… whilst also meeting the customers expectations

11. Micro Services & Event Streaming
Rest API
Operational User Interface – Any Channel (Web, Mobile App, Third Party, etc.)
Microservice Database
Micro-Application
(e.g. Admin)
Data
Capture
Event
Handlers
Event Streaming based on Distributed, Replicated Event Log – (e.g. Kafka)
Rest API
Microservice Database
Micro-Application
(e.g. Orders & Booking)
Data
Capture
Event
Handlers
Rest API
Search Engine
Micro-Application
(e.g. Search)
Data
Capture
Event
Handlers
User Interface
Corporate Business Platform
Enterprise Database
(e.g Oracle, MS SQL Server)
Monolithic Architecture Microservices Architecture
Analytic & Data
Science Tooling
Analytic Database(s)
Analytical
User Interface
Event
Handlers
Rest
API
StreamAnalyticTooling
Data
Capture

12. The Micro-Service Architectural Approach …
• A microservice architecture is a style that structures an application as a
collection of services that are:
• Highly Maintainable and Testable
• Owned by a Small, Largely Self-Sufficient Team
• Independently Deployable
• Organized Around Business Capabilities
• Loosely Coupled
• These are designed to facilitate the rapid, frequent and reliable delivery of
large, complex applications, whilst allowing the organization to evolve its
technology stack.
https://microservices.io/

13. The Benefits (1)
• Does one job well
• Manages one aspect of the process e.g. Payment Beneficiaries
• A well-defined versioned API that is easy to integrate with
• Has a GDPR benefit of being able to identify and audit all interactions with
data
• Limits exposure of data outside the service
• Security benefits
• Can secure data at rest and data in motion, tokenisation, authentication and
access control
• Owned by a single team using a technology stack of their choice
and architect the service to handle specific requirements
• For example distributed or local database, CAP requirements, etc.
• Note: There shouldn’t be unbounded choice of technologies
Rest API
Operational User
Interface
Microservice Database
Beneficiaries
Micro-Application
Data
Capture
Event
Handlers
Event Stream

14. The Benefits (2)
• Easy to build an automated Continuous Deployment (CD) and
Continuous Integration (CI) process that operates asynchronously
from other micro-services release cycles
• Can be deployed in a scalable way on premise or in the cloud
• Can be used by multiple legacy platforms
• Can aid performance on the legacy platforms by off-loading work
Rest API
Operational User
Interface
Microservice Database
Beneficiaries
Micro-Application
Data
Capture
Event
Handlers
Event Stream

15. Example: Beneficiaries In A Payment System
• Creates new beneficiary
• POST /v1/customer/beneficiaries
• List all beneficiaries in customer environment
• GET /v1/customer/beneficiaries
• Update existing beneficiary
• PUT /v1/customer/beneficiaries/{beneficiary_id}
• Fetch the beneficiary entity
• GET /v1/customer/beneficiaries/{beneficiary_id}
• Creates new account for beneficiary
• POST /v1/customer/beneficiaries/{beneficiary_id}/accounts
• Return all accounts for given for beneficiary
• GET /v1/customer/beneficiaries/{beneficiary_id}/accounts
• Return account for beneficiary
• GET /v1/customer/beneficiaries/{beneficiary_id}/accounts/{account_id}
• Update existing account for beneficiary
• PUT /v1/customer/beneficiaries/{beneficiary_id}/accounts/{account_id}
• Change default account of beneficiary
• PUT /v1/customer/beneficiaries/{beneficiary_id}/accounts/{account_id}/make-default
https://docs.railsbank.com/api/beneficiaries
Rest API
Operational User
Interface
Microservice Database
Beneficiaries
Micro-Application
Data
Capture
Event
Handlers
Event Stream

16. Communicating between Services
Transaction Service
TransactionService
CreateTransaction
Saga
Transaction
Create Transaction
Account Service
Account
CreditLimit
CreditReservations
create()
create()
approve()
Event Stream:
Account Command Channel
Event Stream:
CreateOrder Saga Reply Channel
*This is an orchestration pattern, other patterns e.g. choreography exist
Reserve Credit
Credit Reserved
• The channel provides a method of communication between services
• Serialises events to ensure that they happen in the right order
• Added benefit that it is recording the business events as they happen

17. Kafka Event Streams Change Everything …
Long Data
Persistence
Durable
Fault
Tolerant
Distributed
Real Time
Handling
Batch
Handling
Capability
Scalable
Low
Latency
Increases
Data
Accessibility
High
Concurrency
Reduces
Data
Integration
High
Throughput
Data is
Replay-able
Cloud/
On-Premise
Data
Consumer
Friendly

18. Micro-service
Kafka Event Stream
Services can log all
their events and
read the events
from other services
Transaction

19. Micro-service
Kafka Event Stream
Services can
communicate via
the event stream to
process an events in
order
Micro-service
Services can
communicate via
the event stream to
process an events in
order
Transaction
Account

20. Micro-service
Kafka Event Stream
Micro-service
A single event can
trigger multiple
other actions
Micro-service Micro-service
Transaction
Account Fraud
Score
Address
Check
A single event can
trigger multiple
other actions

21. Micro-service
Kafka Event Stream
Micro-service
A single event can
trigger multiple
other actions
Micro-service Micro-service
Transaction
Account Fraud
Score
Address
Check
And it is easy to
disable/remove or
add new actions
Micro-service
Loyalty
Actions

22. What about Analytics & Data Science capabilities?
Rest API
Operational User Interface – Any Channel (Web, Mobile App, Third Party, etc.)
Microservice Database
Micro-Application
(e.g. Admin)
Data
Capture
Event
Handlers
Event Streaming based on Distributed, Replicated Event Log – (e.g. Kafka)
Rest API
Microservice Database
Micro-Application
(e.g. Orders & Booking)
Data
Capture
Event
Handlers
Rest API
Search Engine
Micro-Application
(e.g. Search)
Data
Capture
Event
Handlers
Analytic & Data
Science Tooling
Analytic Database(s)
Analytical
User Interface
Event
Handlers
Rest
API
StreamAnalyticTooling
• Events can be routed directly
and in real-time to analytical
platforms
• Move from trying to
understand what was
happening after the event
(data warehousing) to
analysing events as they
happen (streaming analytics)
• Opportunity to offer your
analytics as service to others
(true data monetisation)
Data
Capture

23. From the Data Warehousing to Streaming Analytics
• A data warehouse provides a
series of snapshots over time,
often with significant latency
from which you try to
understand the events that
occurred
• If stock level of an item drops
by 6 did you sell 6 items or
did you sell 10 items and
have 4 items returned?
• A streaming event tells you what
is happening right now
• An item was sold, an item
was sold, an item was
returned, etc.
• A data warehouse defines a data
model and then tries to fit the
data from multiple sources into
that model
• This often results in long and
complex analysis of source
systems to determine what
the data is and how its used
• Streaming Analytics provides a
way to act on an event within
context
• The data model of useful
information is defined by the
event data passed in the API

24. Micro-service
Kafka Event Stream
Micro-service Micro-service Micro-service
Transaction
Account Fraud
Score
Address
Check
And we want to
feed our Data
Scientists with all
the data they need
Micro-service
Loyalty
Actions
Micro-service
Analytics

25. Micro-service
Kafka Event Stream
Micro-service Micro-service Micro-service
Transaction
Account Fraud
Score
Address
Check
So that we can
deploy new and
innovative AI based
Services
Micro-service
Loyalty
Actions
Micro-service
Analytics
Micro-service
New AI
Based
Service

26. Micro-service
Kafka Event Stream
Micro-service Micro-service Micro-service
Transaction
Account Fraud
Score
Address
Check
Micro-service
Loyalty
Actions
Micro-service
Analytics
Micro-service
New AI
Based
Service
Micro-service
Third Party
Service
We can also expedite
our development by
making use of third
party services rather
than developing our
own solutions

27. Micro-service
Kafka Event Stream
Micro-service Micro-service Micro-service
Transaction
Account Fraud
Score
Address
Check
If our services are
useful to others we
can also monetise
them
Micro-service
Loyalty
Actions
Micro-service
Analytics
Micro-service
New AI
Based
Service
Micro-service
Third Party
Service
₤$€¥

28. Micro Services Architecture
underpinned by Streamed Events
Data
Governance
Agile
Development
Automation &
Self-Service
Data Science
& AI
I stated that we need to evolve our approach to building new
platforms and upgrading existing platforms to turn our ideas into
reality

29. Micro Services helps Data Governance
Data Security by creating multiple
loosely coupled layers each of which
is protected in multiple ways
Data Privacy by only holding
confidential data in a small number of
well protected places and restricting
access via APIs
Data Quality by tightly defining where
and how data is entered and
prohibiting side routes to updating
that information
Compliance e.g. GDPR by being able
to define where data is held, how if
flows and is retained in the
organisation and the data models (as
defined by the APIs) used

30. Micro Services help Agile Development
Self organising teams take ownership of well
defined products and can held made responsible
for delivery of working software
Makes change is
significantly easier,
and changes can be
made without having to
co-ordinate massive
releases across a
monolithic platform
Facilitates an environment that has short cycles
and sustainable development and is focused on
technical excellence

31. Micro Services help Automation and Self Service
In Development
Smaller systems are easier to
build continuous development
(CD) and continuous
integration (CI) environments
that also allow continuous
Automated environment builds
allow developers to sand box
and experiment rapidly – fail
fast
In Production
Scale out solutions can be ramped up and down on demand,
reducing the operating costs and increasing platform availability
and resilience
In Analytics
Quicker access to data for the analytics and data science team
Multiple concurrent experiments to be carried out on isolated
platform capacity that doesn’t impact operational systems

32. Micro Services helps Data Science & Artificial Intelligence
Access to the business events as they
happen
Rather than using the data artefacts
after the event
Easier access to the right data rather
than exploring everything
Facilitates deploying AI tools into
production
Reduces the scope of where they have
to be connected to the platform
Allows multiple intelligent (AI) services
to be deployed quickly and easily into
the business process

33. Why do this?
• Put simply – you can’t afford not to, your customers will expect the
features and your competition will be delivering them.
• Bringing together micro-services with the other techniques creates
a deployment momentum that is greater than the sum of the parts
and enhances your chances of succeeding
• It is a journey of several years of constant iteration but also has
significant incremental benefits along the way
• However - it’s not a Silver Bullet – you have to work at it

34. David M. Walker
davidw@datamgmt.com
+44 7990 594 372
Scan Me