The technology world is rapidly moving towards microservices and there are well documented best practices on how to do so. However, data persistence has always been a challenge for most brownfield or greenfield microservices initiatives. Concepts such as ACID properties need to be considered when moving to a decentralized model. Data consistency is often a challenge that affects the overall service. This presentation takes a pragmatic look at a decentralized data architecture and how it aides a move towards a true microservices model. We also look at some of the latest data initiatives such as streaming data for persistence

1. Vice President - Solutions Architecture, WSO2
De-centralised Data Architectures
Mifan Careem

2. Source: https://xkcd.com/327/

3. ● Centralised Data
● Towards de-centralisation and microservices
● Business domain and data boundaries
● Transaction boundaries
● Data caching and micro data
● Streaming data and Persistence
● Summary
Agenda

4. Centralised data principles
● ACID (Atomicity, Consistency,
Isolation, Durability) properties
● Single place to manage and
lookup
● Mutable
● Global
● Simpler management of
technology
● Single point of enforcement
Source:
https://www.cazena.com/blog/my-first-database

5. RESTful APIs backed
by a central data
store Schemas: Drivers, Riders, Registration,
Traffic, Queues, Complaints,...
Middleware, API Gateway
REST API REST API
Global, Mutable,
Shared, Highly
Available, Single
lookup, Simpler
management
Ride Sharing Platform UI

6. Data in a microservices
world

7. ● Teams working independently,
at different speeds, with
different technology
● No dependencies
● Autonomous decisions on
design, methodology and
technology
● React to business requirements
independently
Why Microservices

8. “Each microservice controls
and owns its own data
source or data base and
services do not share a
database”
Microservice data
architecture mantra

9. Towards Data
De-centralisation

10. Rides
REST API
Aggregated
Service
Registration
REST API
Rides Drivers Riders
Rides
REST API
Driver Reg
REST API
Rides Drivers Riders
Rider Reg
REST API
1. Identify business
domains
Data Model is driven by our
domain requirements (think REST!)
2. Identify boundaries

11. Rides
REST API
Driver Reg
REST API
Rides Drivers Riders
Rider Reg
REST API
Rides
REST API
Driver Reg
REST API
Rides Drivers Riders
Rider Reg
REST API
Source of truthCopy
Riders RO
Rider Reg
RO
REST API
Read Copy
3. Understand
transactions and their
boundaries
Identify granular
transactions
4. Identify
micro-services and
transactions that cross
boundaries
Design for failure

12. ● Eventual consistency is key! Identify
locations where strong consistency is
absolutely needed
● If consistency is required among multiple
services, identify one of them as the
source of truth - others can be eventually
consistent
● Using DDD, store only data that is needed
locally by a (micro) service or API
● Use an Event Driven Architecture where
possible - publish changes to the model
which can be subscribed to by interested
parties
● Ensure services aren’t too chatty - that
might mean the model can be improved
Detailed design guidelines
Rides
REST API
Driver Reg
REST API
MySQL MSSQL
Elastic
Search
Rider Reg
REST API
Event Bus
pub/
sub
pub/
sub
pub/
sub

13. ● Use events and streams as your source and persist required
data
● Data storage is simply the persistence of streaming data at
different points in time
● Database is the storage of the current state of data
● Replay data anytime for debugging or migration
● Use and switch to any data persistence technology as
needed when needed
● Kafka, Apache Samza, WSO2 Stream Processor
Streaming Data as your (persistent) data store

14. Advantages
● Resilient, highly available, fault
tolerant data stores
● Polyglot persistence - use
whatever persistence
technology that suits a service
● Pave the way for independent
development teams that avoid
data dependency
● Designed for failures
● Avoid complex transactions
across boundaries
Pros and Cons
Disadvantages
● Complex to operationalize
● Cannot guarantee consistency
● Nightmare to debug

15. Summary - De-centralised data architecture
● Since data sources are
independant, select
the persistence layer
depending on the
requirement and
speed
-- RDBMS
-- NoSQL
-- Cached data
- Introduce caching or
indexing as needed
- Micro-data lakes

16. THANK YOU
wso2.com