How to build more reliable, robust and scalable distributed systems

2,456 views
2,178 views

Published on

Slides from my lightning talk at Teknologihuset in Oslo on how to build more reliable, robust and scalable distributed systems.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,456
On SlideShare
0
From Embeds
0
Number of Embeds
21
Actions
Shares
0
Downloads
50
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

How to build more reliable, robust and scalable distributed systems

  1. 1. How to build more reliable, robust and scalable distributed systems Lars-Erik Kindblad Senior Consultant Twitter: @kindblad E-mail: le@kindblad.com
  2. 2. The Book Shop  Place order should: 1. Save the order in a database 2. Charge the customer’s credit card 3. Send a confirmation e-mail to the customer
  3. 3. The typical way – request/response Place order Browser 1. Add order OrderService OrderController 2. Charge credit card PaymentService 3. Send mail NotificationService Also called synchronous remote procedure call
  4. 4. Problem #1 – the order is lost Failures: Network Webservice Database++ Place order 1. Add order Browser OrderController Error page Error OrderService Exception 2. Charge credit card PaymentService 3. Send mail NotificationService  Order is lost  User will receive an error  User might leave – lost sale  User might retry
  5. 5. Problem #2 – no transactional management Place order 1. Add order Browser OrderService OrderController Error page Exception 2. Charge credit card PaymentService 3. Send mail Error  Order is stored  Payment is processed  E-mail is not sent  User receives an error and might retry:  Order might be stored twice in the database  Credit card might be charged multiple times NotificationService
  6. 6. Solution – one-way messaging Add PlaceOrder message Place order Browser OrderController Queue Asynchronous The message: PlaceOrder message 4 First-In Queue PlaceOrder message 3 PlaceOrder message 2 PlaceOrder message 1 First-Out
  7. 7. The message must also be processed Queue 1. Connect 2. Receive message PlaceOrder PlaceOrder Worker 3. PlaceOrder Message Handler 4. Add order OrderService 5. Charge credit card PaymentService 6. Send mail NotificationService
  8. 8. Benefits  Very fast on the frontend – put the message on the queue  The order is never lost  Automatically retries during errors 4. Rollback. Put the message back on the queue and retry Queue 1. Connect 2. Receive message PlaceOrder PlaceOrder Worker 3. PlaceOrder Error Message Handler
  9. 9. We still have poor transactional management 1. Add order Handle PlaceOrder OrderService 2. Charge credit card PaymentService 3. Send mail Exception Error NotificationService  Order is created  Credit card is charged  E-mail sending fails  The message is put back on the queue and will be retried  Order is duplicated and credit card will be charged twice
  10. 10. Solution  Split into many messages - one message per transactional boundary Place order OrderController Browser 1. Add PlaceOrder message All the services must be idempotent to be 100% reliable PlaceOrder message 2. Handle PlaceOrder Add PayOrder message PayOrder message Queue OrderService 3. Handle PayOrder Add SendMail SendMail message 4. Handle SendMail NotificationService If this fails the message is put back on the queue and will be retried PaymentService
  11. 11. Scaling  Request/Response  Needs to process unpredictable many requests  Messaging  Needs to store unpredictable number of messages  The workers only process a predictable number of messages, even during peaks  Scale up  Concurrently process multiple messages  Scale out  Use multiple workers
  12. 12. Messaging challenges 1. ID generation  The order ID is first available in the worker 2. Eventual consistency  The PlaceOrder message might not have been picked up yet
  13. 13. NServiceBus  Lightweight messaging framework for .NET  Great choice for implementing one-way messaging + publish/subscribe  Open Source but not free  Available at http://particular.net/
  14. 14. About Capgemini With more than 120,000 people in 40 countries, Capgemini is one of the world's foremost providers of consulting, technology and outsourcing services. The Group reported 2011 global revenues of EUR 9.7 billion. Together with its clients, Capgemini creates and delivers business and technology solutions that fit their needs and drive the results they want. A deeply multicultural organization, Capgemini has developed its own way of working, the Collaborative Business ExperienceTM, and draws on Rightshore ®, its worldwide delivery model. Rightshore® is a trademark belonging to Capgemini www.capgemini.com The information contained in this presentation is proprietary. © 2012 Capgemini. All rights reserved.

×