Steve Smith from Atlassian discusses improvements made to Atlassian's order processing system. The system was originally manual but now processes $700 million in orders with only 3 developers. To prepare for growth and compliance requirements, the system was redesigned to have atomic, idempotent operations with asynchronous processing and two-phase transactions. To handle credit card charges reliably despite potential failures, the system uses credit card pre-authorizations that guarantee payment but don't capture funds, addressing the risk of failing systems by ensuring orders can rollback without impacting customers.

1. Robust Systems
or:
Abusing credit card features to not fuck
the customer

2. Who Am I?
• Steve Smith!
• An Atlassian for 7+ years!
• Founded the sysadmin team!
• Developer for the last 4

3. Background
•
•
•
•
•
Atlassian built its own order system.!
Replaced a purely manual system.!
Has processed $700 million to date.!
3 developers for most of its life.!
Devs are also the admins.

4. The problem
•
•
•
•
Atlassian is growing.!
Amount of manual intervention growing with it.!
IPO and SOX compliance looming.!
We’re lazy, we want an ops-free system.

5. The problem
•
•
Parts of the system were non-transactional.!
Partially processed orders required manual
cleanup.

6. The plan
•
•
•
An atomic ordering system.!
Full commit or full roll-back (as far as possible).!
“Don’t fuck the customer”!

7. The design
•
•
•
Break into smallest possible operations.!
Make these operations idempotent.!
Make asynchronous throughout.

8. The implementation
•
•
•
Asynchronous events persist across failures.!
All asynchronous and DB operations are bound.!
Use 2-phase/XA transactions for everything.

9. The credit card
problem
•
•
•
When do I charge the credit card?!
Single operation, hard to roll-back.!
Remember, you have to assume your systems
will fail catastrophically.

10. Credit cards and failure
•
•
To early, failure fucks the customer.!
Too late, we don’t get paid, and we need to
clean-up.

11. Abusing credit card
features
You know that pre-authorisation thing hotels do?

12. Abusing credit card
features
•
Ensure funds are available, but don’t charge.

13. Making charges 2-phase
•
•
They have automatic roll-back on timeout.!
Credit card pre-auths look a lot like the XA 2phase commit spec.

14. Phase-1
•
•
During order setup perform a pre-auth for the
order amount.!
The credit card provider then guarantees the
payment, but doesn’t capture the funds.

15. Phase-2
•
•
Final capture operation is asynchronous.!
But only occurs once we have committed to
completing the order.

16. Failure modes
•
On any error we force a cancellation event.

17. Failure modes
•
On catastrophic failure the authorisation times
out.

18. Failure is always an
option
•
You have to assume your systems will fail…

19. Failure is always an
option
•
… and when they fail they can’t fuck the
customer (or you).

20. Steve Smith
@tarkasteve
ssmith@atlassian.com