Event Driven-Architecture from a Scalability perspective

12,822 views
12,488 views

Published on

Event Driven-Architecture from a Scalability perspective

Published in: News & Politics, Technology
1 Comment
66 Likes
Statistics
Notes
No Downloads
Views
Total views
12,822
On SlideShare
0
From Embeds
0
Number of Embeds
181
Actions
Shares
0
Downloads
529
Comments
1
Likes
66
Embeds 0
No embeds

No notes for slide

Event Driven-Architecture from a Scalability perspective

  1. 1. Building loosely coupled and scalable systems using Event-Driven Architecture Jonas Bonér Patrik Nordwall Andreas Källberg
  2. 2. Why is EDA Important for Scalability?
  3. 3. What building blocks does EDA consists of?
  4. 4. OutlineConcepts Patterns ChallengesHighly Scalable Web Sites
  5. 5. Concepts
  6. 6. MessagingPublish-Subscribe Point-to-Point Store-forward Request-Reply
  7. 7. Publish-Subscribe
  8. 8. Point-to-Point
  9. 9. Store-Forward
  10. 10. Request-Reply
  11. 11. StandardsAMQP JMS
  12. 12. Some Products ...
  13. 13. Domain Events“Its really become clear to me in the lastcouple of years that we need a new buildingblock and that is the Domain Events” -- Eric Evans, 2009
  14. 14. Domain Events“State transitions are an important part of ourproblem space and should be modeled withinour domain.” -- Greg Young, 2008
  15. 15. Domain Events Something that hashappened in the past CustomerRelocated CargoShippedInventoryLossageRecorded
  16. 16. Domain EventsUniquely identifiable Self containedObservable   Time relevant
  17. 17. Patterns
  18. 18. Event Stream Processingselect  *  from  Withdrawal(amount>=200).win:length(5)
  19. 19. Actors• Share NOTHING• Isolated lightweight processes• Communicates through messages• Asynchronous and non-blocking• No shared state … hence, nothing to synchronize.• Each actor has a mailbox (message queue)
  20. 20. ActorsEasier to reason aboutRaised abstraction level Easier to avoid Race conditions Deadlocks Starvation Live locks
  21. 21. ActorsTransparent remoting • Client-managed • Server-managedPub-Sub • Redis • ZeroMQGuaranteed deliveryPersistent mailbox • File-based • Network-based
  22. 22. Command and Query Responsibility Segregation“A single model cannot be appropriatefor reporting, searching andtransactional behavior.” -- Greg Young, 2008
  23. 23. CQRS• Aggregate roots receive Commands and publish Events• All state changes are represented by Domain Events• Reporting module is updated as a result of the published Events• All Queries go directly to the Reporting, the Domain is not involved
  24. 24. The traditional way... DB Data access Domain Service Update Query DTO DTO Client
  25. 25. The CQRS way... DB DBData access publish subscribe Domain Data access Service Service QueryCommand DTO Client
  26. 26. The CQRS way... DB DB Data access Service DBData access publish Data access Domain Service DB Data access Service Service Client
  27. 27. CQRS Benefits• Separation of concern• Fully encapsulated domain that only exposes behavior• Queries do not use the domain model• Easy integration with external systems• Performance and scalability• Testability
  28. 28. Event Sourcing• Every state change is materialized in an Event• All events are stored in an Event Log• System can be reset and Event Log replayed• Many different Listeners can be added
  29. 29. Storing Structure * Line Item Purchase Order Shipping Information
  30. 30. Event Sourcing - Storing Deltas Shipping Cart Added 2 Added 2 InfoCreated Socks Shirts Added
  31. 31. Aggregates are tracking events as to what has changed within them Current state is constructed by replaying all eventsData is not persisted in a structure but as a series of transactions No ORM is needed
  32. 32. Event Sourcing - Replaying Events1 2 3 4 5 6 7
  33. 33. Event Sourcing - Rolling Snapshot Snapshot1 2 ... 100 101 102 103 104
  34. 34. Event Sourcing - Benefits• No object-relational impedance mismatch• Bullet-proof auditing and historical tracing• Support future ways of looking at data• Performance and scalability• Testability• Reconstruct production scenarios
  35. 35. Simple CQRS Samplehttp://github.com/patriknw/ sculptor-simplecqrs/
  36. 36. Challenges
  37. 37. Clustering of BrokersActiveMQ• Master-Slave• Store and Forward Network of BrokersRabbitMQ• Cluster of Erlang nodesZeroMQ• Brokerless - point-to-point or pub-sub
  38. 38. ZeroMQNetwork protocol - thin layer above TCP/IPTransports Text• INPROC• IPC• MULTICAST• TCP
  39. 39. ZeroMQForwarding devices• QUEUE• FORWARDER• STREAMER
  40. 40. ZeroMQPatternsREQUEST/REPLY (load-balanced)PUB/SUBUPSTREAM/DOWNSTREAM (pipelining)PAIR (exclusive)
  41. 41. Wire FormatsJava serialization (binary, schema, runtime)Protobuf (binary, schema, compiled)Avro (binary, schema, compiled & runtime)Thrift (binary, schema, compiled) TextMsgPack (binary, schema, compiled)Protostuff (binary, schema, compiled)Kryo (binary, schema-less, runtime)BERT (binary, schema-less, runtime)Hessian (binary, schema-less, compiled)XML (text, schema)JSON (text, schema-less)
  42. 42. Guaranteed DeliveryDo I really need it?Persistence increasesreliability at the expenseof performance  
  43. 43. Competing Consumers Pattern for solving: • Load balancing • Concurrency • Failover Only works with Point-to-Point ChannelChallenge• ordering• duplicates (idempotent receiver)
  44. 44. Duplicate Messages What do I need? • Once-and-only-once • At-least-once • At-most-once QOS keep history of processed idsUnique message identifier Business semantics
  45. 45. How to get back on track?Point-to-point: no problem, justmake the queue persistentPub/sub: well, not so straightforward Problem: only active subscribers Solution: durable subscriber Problem: failover and load balancing
  46. 46. Producer Flow ControlWhat to do when producers flood youwith messages?Running low on broker resources, slowconsumersGraceful degradation • caller run (in process only)• block• abort• discard
  47. 47. Behind the Scenes of Highly Scalable Web Sites
  48. 48. caching is important, but also...
  49. 49. Minimize latencyFlickr: Do The Essential Work Up-Front And Queue The Rest Amazon: ~99% of content is staticReddit: Precompute everythingand cache it
  50. 50. Changes - pull or pushFacebook: Pull on Demand Digg: Push on ChangeTwitter: Push tweets
  51. 51. Truly event-driven web clientsRequest-response doesntfit collaborative systems WebSockets enable real event- driven web
  52. 52. Why is EDA Important for Scalability?• Scale out and up• Load balance• Parallel execution• Non-blocking• Loosely coupled components can scale more independent of each other
  53. 53. ?
  54. 54. thanksfor listening

×