1. Why Event streaming
2. What is Event streaming
3. What is Transmitting Event Streams
4. What are message brokers
5. Types of message brokers available in market
6. Databases and streams
7. Event Sourcing
8.Command vs Events
9.State, Streams and Immutability
10. Immutable Events
11. Stream Events
12. Stream Joins
13. Demo on Azure Event Hub
14. Visualize data anomalies in real-time events sent to Azure Event Hubs
3. PROBLEMS WITH CURRENT ANALYTICS
Not Realtime
ETL was created during a
period of monolithic
architectures, data warehouses
and relational databases.
4. PROBLEMS WITH CURRENT MICROSERVICES
ARCHITECTURE
• Client more details about our
service
• Too many details to be
exposed
• Chatty communication
5. PROBLEMS WITH CURRENT MICROSERVICES
ARCHITECTURE
• Gateway can take care of
authentication.
• Rate limiting possible
• Things seems ok.
8. CHANGES ARE STILL HARD
• Adding a new service needs to
communicated to other services of
its presence.
• Chatty communication between
services
• REST+JSON is SLOW
• GRPC better, but not a complete
solution
• Adding services adds new changes
to the other services.
12. ONLINE SHOP
1. No Knowledge of other services
2. Communication at services will.
3. Rest calls reduced.
4. Broadcast your events and can be
ignored by other services or
consumed.
13. ONLINE SHOPPING USING R.E.S.T
1.Order should know about
shipment services
2.Shipment should get details
of the customer from the
customer service
3.What if there is customer
details are changed ?
15. PRODUCT ORDERED - FACT
1.Order submitted
2.Event sent to Kafka
3.Shipment reacts to the order
being placed
4.No REST communication
between order and shipment
5.Product updates itself based
on process result
16. ADDRESS CHANGED - TRIGGER
1.Customer changes address
2.Event sent to Kafka
3.Shipment reacts to changes
4.Order does not have to care
about any changes once it is
placed.
5.Shipment updates its database
6.Other services will not react if it
is not pertained to them,
IGNORE
20. BASICS OF EVENT STREAMING
• Databases vs streams
• Command vs Events
• Immutable Events
• State, Streams and Immutability
• Stream Joins
21. DATABASE VS STREAMS
• Concept wise it is same
• Every action takes places is stored as logs
• Restore happens from reading the logs again and creating
the database
22. COMMAND VS EVENTS VS IMMUTABLE
EVENTS
• Any request is a command.
• After integrity check it becomes an Event.
• Booking an air ticket is command, if there is ticket
available, it becomes an event. It also becomes a fact.
• When the ticket is cancelled, it is still an event, that is
immutable and recorded in our system.
23. STATE, STREAMS AND IMMUTABILITY
• When the ticket is booked, the State is saved.
• When the ticket date is changed and ticket class is
upgraded, there is a Stream of events that changes the
state.
• Ticket can even be cancelled but the history of the ticket
booking will remain and recorded as logs and each event
is Immutable.
24. STREAM JOINS
• Joining different datasets to perform analysis
• Stream-Stream Join
• Stream-Table Join
• Table-Table Join
25. STREAM-STREAM JOINS
• User searches for a book and he clicks the search result.
• You need to map the book searched and clicked book.
• Now the search is recorded as one event and click is
another event.
• If you want to analyse the click through rate you need to
bring together the search and click action, which can be
connected by Session ID
26. STREAM-TABLE JOINS
• User searches for a book
• You need to show some recommendation based on his
search
• Now you need to join the search which is an event and his
search history which is saved as table.
• You can copy the table to local cache and join the search.
But cache can become stale. Better is to subscribe the
cache to the stream of user history