Slides from my lightning talk at Teknologihuset in Oslo on how to build more reliable, robust and scalable distributed systems.

1. How to build more reliable,
robust and scalable distributed
systems
Lars-Erik Kindblad
Senior Consultant
Twitter: @kindblad
E-mail: le@kindblad.com

2. The Book Shop
 Place order should:
1. Save the order in a database
2. Charge the customer’s credit
card
3. Send a confirmation e-mail to
the customer

3. The typical way – request/response
Place order
Browser
1. Add order
OrderService
OrderController
2. Charge credit card
PaymentService
3. Send mail
NotificationService
Also called synchronous remote procedure call

4. Problem #1 – the order is lost
Failures:
Network
Webservice
Database++
Place order
1. Add order
Browser
OrderController
Error page
Error
OrderService
Exception
2. Charge credit card
PaymentService
3. Send mail
NotificationService
 Order is lost
 User will receive an error
 User might leave – lost sale
 User might retry

5. Problem #2 – no transactional management
Place order
1. Add order
Browser
OrderService
OrderController
Error page
Exception
2. Charge credit card
PaymentService
3. Send mail
Error
 Order is stored
 Payment is processed
 E-mail is not sent
 User receives an error and might retry:
 Order might be stored twice in the database
 Credit card might be charged multiple times
NotificationService

6. Solution – one-way messaging
Add PlaceOrder
message
Place order
Browser
OrderController
Queue
Asynchronous
The message:
PlaceOrder message 4
First-In
Queue
PlaceOrder message 3
PlaceOrder message 2
PlaceOrder message 1
First-Out

7. The message must also be processed
Queue
1. Connect
2. Receive message PlaceOrder
PlaceOrder
Worker
3. PlaceOrder
Message Handler
4. Add order
OrderService
5. Charge credit card
PaymentService
6. Send mail
NotificationService

8. Benefits
 Very fast on the frontend – put the message on the queue
 The order is never lost
 Automatically retries during errors
4. Rollback. Put the message
back on the queue
and retry
Queue
1. Connect
2. Receive message
PlaceOrder
PlaceOrder
Worker
3. PlaceOrder
Error
Message Handler

9. We still have poor transactional management
1. Add order
Handle PlaceOrder
OrderService
2. Charge credit card
PaymentService
3. Send mail
Exception
Error
NotificationService
 Order is created
 Credit card is charged
 E-mail sending fails
 The message is put back on the queue and will be retried
 Order is duplicated and credit card will be charged twice

10. Solution
 Split into many messages - one message per transactional boundary
Place order
OrderController
Browser
1. Add PlaceOrder message
All the services must
be idempotent to be
100% reliable
PlaceOrder message
2. Handle PlaceOrder
Add PayOrder message
PayOrder message
Queue
OrderService
3. Handle PayOrder
Add SendMail
SendMail message
4. Handle SendMail
NotificationService
If this fails the
message is put back
on the queue and will
be retried
PaymentService

11. Scaling
 Request/Response
 Needs to process unpredictable many requests
 Messaging
 Needs to store unpredictable number of messages
 The workers only process a predictable number of messages, even during peaks
 Scale up
 Concurrently process multiple messages
 Scale out
 Use multiple workers

12. Messaging challenges
1. ID generation

The order ID is first available in the worker
2. Eventual consistency

The PlaceOrder message might not have been picked up yet

13. NServiceBus
 Lightweight messaging framework for .NET
 Great choice for implementing one-way messaging + publish/subscribe
 Open Source but not free
 Available at http://particular.net/

14. About Capgemini
With more than 120,000 people in 40 countries, Capgemini is one
of the world's foremost providers of consulting, technology and
outsourcing services. The Group reported 2011 global revenues
of EUR 9.7 billion.
Together with its clients, Capgemini creates and delivers
business and technology solutions that fit their needs and drive
the results they want. A deeply multicultural organization,
Capgemini has developed its own way of working, the
Collaborative Business ExperienceTM, and draws on Rightshore ®,
its worldwide delivery model.
Rightshore® is a trademark belonging to Capgemini
www.capgemini.com
The information contained in this presentation is proprietary.
© 2012 Capgemini. All rights reserved.