This document discusses data microservices and related concepts. It introduces data microservices as lightweight services that input or output data and can be composed into pipelines. Spring Cloud Stream is presented as a framework that simplifies developing data microservices through annotations like @EnableBinding, @StreamListener, and @SendTo. The document also covers topics like communication models, transactions, event sourcing, and CQRS as patterns for ensuring data consistency across distributed systems of microservices.

1. Data Microservices
with Spring Cloud

2. Speaker
• Orkhan Gasimov, Software Engineer
– 15 years of software engineering;
– variety of technologies (languages & frameworks);
– solution design and implementation;
• Teaching training courses.
– Architecture.
– Java.
– JavaScript / TypeScript.
• Author of training courses.
– Spring Cloud.
– Akka for Java.

3. Speaker
• Overview:
– Communication Model
– Spring Cloud Stream
– Transactions
– Data Consistency

4. Data Microservices

5. Data Microservices
• Why do we think about applications while designing applications?
– Applications come and go.
– They change from time to time.
– Only data stays.
– It’s about data, not apps.

6. Data Microservices
• What should we think about while designing an application?
– Most of the applications are data centric.
– The role of most applications is to input/output/process the data.

7. Data Microservices
• What is a Data Microservice?
– Lightweight;
– Has an input and/or output;
– Composable into a pipeline;
AppIn Out

8. Data Microservices
• Why do we need Data Microservices?
– Is it just about the design?
– What are the benefits and drawbacks?
AppIn Out

9. Contract

10. Contract
• Do not tell what to do.
• Just do it.
• Provide/accept data.
AppIn Out AppIn Out AppIn Out

11. Communication

12. Communication
• Consider an application using direct communication between services.
• When all services work flawlessly…

13. Communication
• What if a service gets unstable or unavailable at all?
– Client requests are not fulfilled.
– Data loss?
• What are the available solutions?

14. Communication
• Use a shared storage between services.
– Store request data in database.
– Notify other service when a request is available for processing.

15. Communication
• Persistent queue is a storage too.
– Publish request data to the queue.
– Other services subscribe to the queue.

16. Communication
• Benefits:
– No direct connection between services.
– Services are decoupled, composition and orchestration is easier.
– If a service gets down, it may process requests later when gets up.

17. Communication
• Drawbacks:
– Additional cost of message broker maintenance.
– Queues should be highly available.

18. Spring Cloud Stream

19. Spring Cloud Stream

20. Spring Cloud Stream

21. Spring Cloud Stream
• The basic channel abstractions are provided by three interfaces:
– Sink – input message channel.
– Source – output message channel.
– Processor – extends Sink and Source.

22. Spring Cloud Stream
public interface Sink {
String INPUT = "input";
@Input(Sink.INPUT)
SubscribableChannel input();
}
public interface Source {
String OUTPUT = "output";
@Output(Source.OUTPUT)
MessageChannel output();
}
public interface Processor extends Source, Sink {
}

23. Spring Cloud Stream
• Development is simplified down to three simple annotations:
– @EnableBinding - activate connectivity to a message broker.
– @StreamListener - receive data from a channel.
– @SendTo - stream data to a channel.

24. Spring Cloud Stream
• A simple processor application looks like:
@SpringBootApplication
@EnableBinding(Processor.class)
public class WordNumFilter {
@StreamListener(Processor.INPUT)
@SendTo(Processor.OUTPUT)
private String filter(String word) {
//filter numbers, return only words
}
//main method...
}

25. Spring Cloud Stream
• Configuration for the channels looks like:
spring:
application.name: WordNumFilter
cloud.stream.bindings:
input:
destination: wordNumFilter
output:
destination: words

26. Spring Cloud Stream
• Consumer Groups:
– Processing of messages is balanced between several consumers;
– Durability enabled;
– Anonymous subscriptions are non-durable;

27. Spring Cloud Stream
• Configuration for the channels looks like:
spring:
application.name: WordNumFilter
cloud.stream.bindings:
input:
destination: wordNumFilter
group: WordNumFilters
consumer:
instanceCount: 2
instanceIndex: 0
output:
destination: words

28. Spring Cloud Stream
• Partitioning:
– Is critical, for either performance or consistency reasons;
– Can be used whether the broker itself is naturally partitioned or not;
– Requires configuration at both, producer and consumer ends;

29. Spring Cloud Stream
• Configuration for the channels looks like:
spring:
application.name: WordNumFilter
cloud.stream.bindings:
input:
destination: wordNumFilter
group: WordNumFilters
consumer:
partitioned: true
instanceCount: 2
instanceIndex: 0
output:
destination: words
producer:
partitionCount: 2

30. Spring Cloud Stream
• Common configuration options available include:
– Properties for use of Spring Cloud Stream;
– Binding properties for Consumer;
– Binding properties for Producer;
– Binder specific properties.

31. Data Consistency
It’s all about data

32. Data Consistency

33. Transactions

34. Transactions
• Transactions:
– A unit of work performed within the system;
– Allow to update several records as part of one transaction;
– If one or more updates fails, we can roll the entire transaction back;

35. Transactions
• In monolithic applications, transactions are simple.
– One process is updating and creating records.
– Same process can either commit or roll back the transaction.

36. Transactions
• In microservices, transactions are complex.
– Several processes are involved in one transaction.
• The transaction is distributed.
– It is complex to observe and solve problems.
– It is more complex to roll back.

37. Transactions
• How to handle failed transactions?
– If one of microservices fail in a transaction, we will need to rollback the entire
transaction.

38. Transactions
• Option 1: Try again later.
– The part of the transaction that failed is put into a queue.
– Another service can pick it up and process later.
• Benefits:
– Transaction will eventually be completed.
• Drawbacks:
– It relies on other instances not failing with the same part of transaction.

39. Transactions
• Option 2: Abort the entire transaction.
– We detect our transaction has failed.
– We issue an Undo transaction (to all the microservices involved).
– The microservices undo any creates or updates.
• Problems:
– Who issue the undo transaction?
– What happens when the undo transaction fails itself?
• One of the possible solutions:
– Use a transaction manager with two phase commit.

40. Transactions: Two Phase Commit
• Phase 1:
– All microservices involved report to transaction manager if they are fine to
commit to their part of the transaction.
• Phase 2:
– If they are fine, then transaction manager tells all participating microservices
to commit the transaction.

41. Transactions
• If a microservice doesn’t respond or respond with a “no to committing”,
transaction manager tells to all the participants to rollback the
transaction.

42. Transactions
• Problems using transaction manager?
– We heavily depend on it.
– It delays the processing of our transactions (potential bottleneck).
– Complex to implement.
– More complex when we need to communicate with another system.

43. Transactions
• When dealing with transactions:
– Think about different approaches for different business processes;
– A transaction manager with two phase commit is not always required;
– Actor/event oriented approaches are available as well;

44. Transactions
• Think about the process of ordering, paying, preparing and receiving the
coffee you order.
– Two phase commit model suggests, to wrap the whole process (all steps) in a
single transaction, until you receive your coffee.
– All employees involved would wait and stop working until you get your coffee.

45. Transactions
• Following the "best effort" model and compensating errors in the process
is more productive.
– First, they make sure that you pay!
– Then, your order is sent to the message queues attached to the cup.
– If something goes wrong, we enter into the compensation process.
– Get what you want or refund.
• Sometimes, they are wasting coffee, but, the overall process is efficient.
– This is the most efficient model for increased productivity.

46. Event Sourcing

47. Event Sourcing
• Event Sourcing uses an event-centric approach to persist business entities.
– Store the sequence of state-changing events instead of the state of an entity.
– Reconstruct an entity’s state by replaying events.
– Saving an event is a single operation, it is inherently atomic.

48. Event Sourcing
• Events are persisted by the Event Store.
– The Event Store behaves like a message broker.
– It provides an API that enables services to subscribe to events.
– Guarantees delivery of events to all interested subscribers.
– It is the backbone of an event-sourced architecture.

49. Event Sourcing
• Audit trail.
– Events are immutable and store the full history of the state of the system.
– They provide a detailed audit trail of what happened within the system.

50. Event Sourcing
• Integration with other subsystems.
– Event store can publish events and notify other subsystems about the changes
to the application’s state.
– The event store provides a complete record of all the events that it published
to other systems.

51. Event Sourcing
• Time Travel.
– Determine the state of the system at any previous point in time.
– Answer historical questions about the system.

52. Event Sourcing
• Drawbacks?
– State is stored as series of events.
– Have to replay the events to rebuild state of an entity.
– Retrieval of the current state is time consuming.

53. Event Sourcing
• Solution is to maintain two separate models:
– write model — the event store.
– read model — the database.
– A different handler is responsible for updating the read model, once an event
is persisted in the event store (write model).
Read
Model
Write
Model
Events

54. CQRS
Command & Query Responsibility Segregation

55. CQRS
• CQRS states that
– every method should either be a command that performs an action,
– or a query that returns data to the caller,
– but not both.

56. CQRS
• Methods should return a value only if they possess no side effects.
– If you have a return value you cannot mutate state.
– If you mutate state your return type must be void.

57. CQRS + Event Sourcing
• Split into two separate services:

58. CQRS + Event Sourcing
• Consistency:
– Write model is consistent, events are ordered, transactions are simple.
– Most systems can be eventually consistent on the Query side.

59. CQRS + Event Sourcing
• Data Storage:
– Normalized data on the Command side (e.g. a relational structure).
– Denormalized data on the Query side (e.g. to minimize the number of joins).

60. CQRS + Event Sourcing
• Scalability:
– Scale Command and Query sides separately.

61. Summary

62. Summary
• Communication
– Direct
– Decoupled

63. Summary
• Spring Cloud Stream
– Sink, Source, Processor
– Enable, Listen, Send
– Consumer Groups
– Partitioning

64. Summary
• Data Consistency
– Transactions
– Event Sourcing
– CQRS + Event Sourcing

65. Summary
• Data Microservices:
– Lightweight;
– Data-oriented;
– Composition;
– Orchestration;
AppIn Out

66. Thank you!
Data Microservices with Spring Cloud