Sending a purchase receipt may seem simple, and yet it can you deep down into distributed systems design and concurrency.

1. How to Send a Receipt
Topics in Concurrency and Distributed Systems | @pascallouisperez

2. The Situation
‣ It’s Black Friday on an e-commerce site.
‣ When a customer makes a purchase,
‣ We send said customer a purchase receipt.

3. Not That Simple
Sending a purchase receipt via email after a purchase is a
concrete product requirement which takes us down into
distributed systems design.
Generalizable Problematic
Many problems we encounter can be rephrased as the
canonical ‘sending a receipt’ problem. And hence,
understanding it well is a powerful tool.
How to Send a Receipt
Topics in Concurrency and Distributed
Systems

4. Bad Solution purchase_handler(…) {
err = charge_cc(cc, order.amount);
if err != nil { return err }
err = mark_as_paid(db, order);
if err != nil { return err }
err = send_receipt(order.customer);
if err != nil { return err }
err = mark_receipt_sent(db, order);
if err != nil { return err }
return nil
}

5. Bad Solution
Mo Money Mo Problems
‣ Not easy to charge a credit
card.
‣ Auth / Capture model? Single
Charge?
‣ Gateway Idempotency?
‣ Handling of timeouts? TCP
interruptions?
‣ Database crash?
purchase_handler(…) {
err = charge_cc(cc, order.amount);
if err != nil { return err }
err = mark_as_paid(db, order);
if err != nil { return err }
err = send_receipt(order.customer);
if err != nil { return err }
err = mark_receipt_sent(db, order);
if err != nil { return err }
return nil
}

6. ‣ Failure to mark as paid leads
to dangling payment without a
confirmed order!
‣ Or a paid order without an
email receipt.
‣ Retrying the handler would
double charge.
‣ (Many other bad things.)
Bad Solution
Some of the Failure Modes
purchase_handler(…) {
err = charge_cc(cc, order.amount);
if err != nil { return err }
err = mark_as_paid(db, order);
if err != nil { return err }
err = send_receipt(order.customer);
if err != nil { return err }
err = mark_receipt_sent(db, order);
if err != nil { return err }
return nil
}

7. (Shorthand) purchase_handler(…) {
err = charge_cc(cc, order.amount); on err return
err = mark_as_paid(db, order); on err return
err = send_receipt(order.customer); on err return
err = mark_receipt_sent(db, order); on err return
return nil
}

8. Let’s Fix Charging

9. Charging
Single Call Model
‣ Creating a charge debits the
customer’s credit card.
‣ Charges are uniquely
identified, and the payment
provider guarantees
idempotency.
‣ Makes it easy to charge,
simply retry until success.
However, requires care to
avoid dangling payments.

10. Charging
Auth / Capture Model
‣ Authorize creates a pending
transaction, and reserves
funds. Auth are uniquely
identified… And providers
expose pseudo idempotency.
‣ Capture confirms the
transactions.
‣ Makes it hard to charge.
However, avoids dangling
payments in failure cases.
(Let’s ignore this model for now.)

11. Charging
Let’s Fix It
purchase_handler(…) {
err = run_transaction(db, func(tx) error {
order.cc_charge_uuid = new_uuid(…);
save(order);
}); on err return
charge, err = stripe.create_charge(order); on err retur
err = run_transaction(db, func(tx) error {
order.cc_charge = charge;
save(order);
}); on err return
err = send_receipt(order.customer); on err return
err = mark_receipt_sent(db, order); on err return
return nil
}

12. Charging
The Pattern
Prepare, Do, Propagate
‣ Tell them what you are going
to tell them;
‣ Tell them;
‣ Then tell them what you told
them.
purchase_handler(…) {
err = run_transaction(db, func(tx) error {
order.cc_charge_uuid = new_uuid(…);
save(order);
}); on err return
charge, err = stripe.create_charge(order); on err retur
err = run_transaction(db, func(tx) error {
order.cc_charge = charge;
save(order);
}); on err return
err = send_receipt(order.customer); on err return
err = mark_receipt_sent(db, order); on err return
return nil
}

13. Charging
Let’s Fix It
purchase_handler(…) {
err = run_transaction(db, func(tx) error {
order.cc_charge_uuid = new_uuid(…);
save(order);
}); on err return
charge, err = stripe.create_charge(order); on err retur
err = run_transaction(db, func(tx) error {
order.cc_charge = charge;
save(order);
}); on err return
err = send_receipt(order.customer); on err return
err = mark_receipt_sent(db, order); on err return
return nil
}
‣ No ‘cc_charge_uuid’: failed
early, can retry.
‣ No ‘cc_charge’: retry
payment, Stripe guarantees
idempotency.
‣ Did we send the receipt?

14. How are Emails Sent
Anyways?

15. $ telnet smtp.thatsanexample.com 25
HELO thatsanexample.com
MAIL from: <pascal@thatsanexample.com>
RCPT to: <pascallouisperez@gmail.com>
DATA
From: pascal@thatsanexample.com
To: pascallouisperez@gmail.com
Subject: SMTP is low level and simple
This is an example.
.
QUIT
Sending Emails
Quick SMTP Primer
‣ RFC2821 for SMTP protocol;
and
‣ RFC2822 for message format.

16. Sending Emails
Avoiding Duplicates
‣ Great. How do you avoid
duplicate delivery of
messages?

17. Sending Emails
Avoiding Duplicates
‣ Only four mentions of
‘duplicate’ in RFC2821:
1. Avoiding duplicates for
mailing lists, not relevant;
2. Specification of a ‘text line’
which has a ‘dot duplicate’,
not relevant;
3. Page 62 represented here,
simplistic recommendation;
4. Reference 28 which points to
an 1988 article “Duplicate
messages and SMTP”, which
became RFC1047.

18. Sending Emails
Avoiding Duplicates
‣ Oh, and RFC1047 is not useful.
Good Description of the Problem
“If the communications link fails during this synchronization gap,
then the message has been duplicated. Both mailers have active
copies of the message that they will try to deliver.”
Simplistic Suggestion
“The best way to avoid the synchronization problem is to
minimize the length of the synchronization gap. In other words,
receiving mailers should acknowledge the final dot as soon as
possible and do more complex processing of the message later.”

19. Sending Emails
Avoiding Duplicates
‣ RFC2821 replaced RFC821 in
2001;
‣ In particular, it saw the
introduction of a ‘message-id’.
‣ One can hope most clients
correctly populate this field
(just kidding); and
‣ One can hope most SMTP
relays use this field to de-
duplicate messages.

20. Sending Emails
Email Delivery Services
‣ Your friendly REST speaking
email delivery services!
‣ Except…

21. Sending Emails
Email Delivery Services
‣ SendGrid doesn’t expose the
message-id.

22. Sending Emails
Email Delivery Services
‣ MailChimp doesn’t expose the
message-id;
‣ Though it allows custom
headers to be passed, hence
likely making it possible to
pass a message-id.

23. Sending Emails
Email Delivery Services
‣ SailThru has a ‘send_id’, yeah!
(Note: Haven’t tested if this is used
to populate the message-id. Seems
to imply as much.)

24. ‣ Impossible to avoid duplicates.
‣ Hence, we have
‣ No delivery; or
‣ At least once delivery.
(It sucks. SMS is worse. So is the post office frankly.)
Sending Emails
The Conclusion

25. Let’s Fix It

26. Fix It, How?
Back to the Subject at Hand
purchase_handler(…) {
err = run_transaction(db, func(tx) error {
order.cc_charge_uuid = new_uuid(…);
save(order);
}); on err return
charge, err = stripe.create_charge(order); on err retur
err = run_transaction(db, func(tx) error {
order.cc_charge = charge;
save(order);
}); on err return
err = send_receipt(order.customer); on err return
err = mark_receipt_sent(db, order); on err return
return nil
}
‣ If we are unable to record the
charge, don’t send the receipt.
‣ If we are able to record the
charge, we must guarantee
that the receipt will be
delivered.
‣ Given what we learned about
email delivery, our choice is
between no delivery, or at
least once delivery.

27. Fix It, How?
Back to the Subject at Hand
purchase_handler(…) {
err = run_transaction(db, func(tx) error {
order.cc_charge_uuid = new_uuid(…);
order.receipt_uuid = new_uuid(…);
save(order);
}); on err return
charge, err = stripe.create_charge(order); on err retur
err = run_transaction(db, func(tx) error {
order.cc_charge = charge;
order.receipt_state = PENDING;
save(order);
}); on err return
return nil
}
Prepare
‣ Record intent to send receipt.
‣ The intent includes the
identifier of the receipt: it is
this specific email we intend
to send.
Do
‣ Charge
Propagate
‣ In the background, process
the email queue and retry
until confirmation of delivery.

28. Generalizing

29. Inventoried Item (e.g. a seated ticket to a show)
Prepare: check availability, and ‘soft reserve’ to avoid
double bookings.
Propagate: confirm reservation.
Activating Gift Cards
Prepare: soft creation of the card.
Propagate: activate and fund.
Coupons
Prepare: verify the validity, and ’soft redeem’ to avoid
double use.
Propagate: actually redeem it.
Similar Problematics
Hard Across µServices Boundaries

30. The Order as Distributed Transaction Coordinator
Distributed transaction processing, and two-phase commit.
Order ManagementOrder Management
Purchase

31. The Order as Distributed Transaction Coordinator
Distributed transaction processing, and two-phase commit.
Order ManagementOrder Management
Prepare: “Soft Reserve” Prepare: “Create Gift Card”
“The Commit Request Phase”

32. The Order as Distributed Transaction Coordinator
Distributed transaction processing, and two-phase commit.
Order ManagementOrder Management
Do: “Charge”
“The Commit Phase”

33. The Order as Distributed Transaction Coordinator
Distributed transaction processing, and two-phase commit.
Order ManagementOrder Management
Propagate: “Confirm” Propagate: “Activate”
Propagate: “Send”
“The Success Phase”

34. The Order as Distributed Transaction Coordinator
Distributed transaction processing, and two-phase commit.
Order ManagementOrder Management
Propagate: “Cancel” Propagate: “Dispose”
“The Failure Phase”

35. Propagate: “Reconcile to
Cancel (or Refund) Dangling
Payment”
The Order as Distributed Transaction Coordinator
Distributed transaction processing, and two-phase commit.
Order ManagementOrder Management
Propagate: “Expiry” Propagate: “Garbage Collect”
“The (Unofficial) Cleanup Post Failure Phase”

36. In Closing
‣ RTFM (“Read The Fucking Manual”)
‣ It's turtles all the way down, follow the thread
‣ Correctness at app level requires understand of low level details
‣ Distributed Systems are Hard
‣ No need to re-invent the wheel, most of CS was invented in the 1970s, and we’re just
repurposing known techniques
‣ Design Early, Design Often
‣ You can’t iterate your way to correctness
‣ Think through your systems early, write it down, rinse, repeat