The document discusses event streaming architecture, emphasizing its advantages for real-time processing and analytics over traditional ETL and microservices methods. It explores how event-driven communication reduces dependencies between services, illustrated through an online shopping scenario. Key concepts include events as immutable facts, stream joins for data analysis, and practical insights into utilizing Azure Event Hubs for anomaly detection.

1. EVENT STREAMING
ARCHITECTURE - DEEP
DIVE
KARTHIKEYAN VK
@KARTHIK3030

2. WHY EVENT STREAMING?
Realtime
processing
Realtime
analytics
Live
dashboarding
Solves
Microservices
drawbacks
Faster
developme
nt

3. PROBLEMS WITH CURRENT ANALYTICS
Not Realtime
ETL was created during a
period of monolithic
architectures, data warehouses
and relational databases.

4. PROBLEMS WITH CURRENT MICROSERVICES
ARCHITECTURE
• Client more details about our
service
• Too many details to be
exposed
• Chatty communication

5. PROBLEMS WITH CURRENT MICROSERVICES
ARCHITECTURE
• Gateway can take care of
authentication.
• Rate limiting possible
• Things seems ok.

6. ANTIPATTERN – GOD

7. SIDECAR PATTERN

8. CHANGES ARE STILL HARD
• Adding a new service needs to
communicated to other services of
its presence.
• Chatty communication between
services
• REST+JSON is SLOW
• GRPC better, but not a complete
solution
• Adding services adds new changes
to the other services.

9. EVENT DRIVEN/STREAMING

10. COMPARISON

11. EVENT STREAMING PATTERN

12. ONLINE SHOP
1. No Knowledge of other services
2. Communication at services will.
3. Rest calls reduced.
4. Broadcast your events and can be
ignored by other services or
consumed.

13. ONLINE SHOPPING USING R.E.S.T
1.Order should know about
shipment services
2.Shipment should get details
of the customer from the
customer service
3.What if there is customer
details are changed ?

14. EVENT ARE BOTH FACTS AND TRIGGERS

15. PRODUCT ORDERED - FACT
1.Order submitted
2.Event sent to Kafka
3.Shipment reacts to the order
being placed
4.No REST communication
between order and shipment
5.Product updates itself based
on process result

16. ADDRESS CHANGED - TRIGGER
1.Customer changes address
2.Event sent to Kafka
3.Shipment reacts to changes
4.Order does not have to care
about any changes once it is
placed.
5.Shipment updates its database
6.Other services will not react if it
is not pertained to them,
IGNORE

17. DEMO
AZURE EVENT HUBS - SETUP

18. DEMO
EVENT STREAMING -
DEEPDIVE

19. BOOKS TO READ

20. BASICS OF EVENT STREAMING
• Databases vs streams
• Command vs Events
• Immutable Events
• State, Streams and Immutability
• Stream Joins

21. DATABASE VS STREAMS
• Concept wise it is same
• Every action takes places is stored as logs
• Restore happens from reading the logs again and creating
the database

22. COMMAND VS EVENTS VS IMMUTABLE
EVENTS
• Any request is a command.
• After integrity check it becomes an Event.
• Booking an air ticket is command, if there is ticket
available, it becomes an event. It also becomes a fact.
• When the ticket is cancelled, it is still an event, that is
immutable and recorded in our system.

23. STATE, STREAMS AND IMMUTABILITY
• When the ticket is booked, the State is saved.
• When the ticket date is changed and ticket class is
upgraded, there is a Stream of events that changes the
state.
• Ticket can even be cancelled but the history of the ticket
booking will remain and recorded as logs and each event
is Immutable.

24. STREAM JOINS
• Joining different datasets to perform analysis
• Stream-Stream Join
• Stream-Table Join
• Table-Table Join

25. STREAM-STREAM JOINS
• User searches for a book and he clicks the search result.
• You need to map the book searched and clicked book.
• Now the search is recorded as one event and click is
another event.
• If you want to analyse the click through rate you need to
bring together the search and click action, which can be
connected by Session ID

26. STREAM-TABLE JOINS
• User searches for a book
• You need to show some recommendation based on his
search
• Now you need to join the search which is an event and his
search history which is saved as table.
• You can copy the table to local cache and join the search.
But cache can become stale. Better is to subscribe the
cache to the stream of user history

27. DEMO
AZURE EVENT HUBS – ANOMALY
DETECTION ON STREAMING DATA

28. REFERENCES
• https://www.infoq.com/presentations/microservices-streams-state-
scalability/
• https://docs.microsoft.com/en-in/azure/event-hubs/event-hubs-about
• https://docs.microsoft.com/en-in/azure/event-hubs/event-hubs-tutorial-
visualize-anomalies
• https://github.com/Azure/azure-event-
hubs/tree/master/samples/DotNet/Azure.Messaging.EventHubs

29. Q&A