Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Active/Active Payments Processing
at Square
Ted Mao and Jiang-Ming Yang
October 2014
Active/Active
!
What
• Resilient to datacenter-level failure
• Resilient to Internet routing
problems
• Transparent to the...
Challenges
!
Inconsistent state between
datacenters
Datacenters can’t tell if a transaction
has already been processed
els...
Concepts
Client idempotence key
Concepts
Client idempotence key Server transaction
Concepts
Client idempotence key Server transaction Transaction progression
Card Processing
Multi-DC resolution
Multi-Tender
Multi-DC challenge
Scenario
When Merchant try to sell items/products to customers, customers will
have the op...
Multi-Tender
Multi-DC resolution
Multi-Tender
Multi-DC resolution
State Machine
Tender state machine
!
!
!
!
Bill state machine
!
!
!
Correctness
1. 1. A f...
Caveats
Eventually consistent
Asynchronous, eventually consistent
systems are harder to reason about.
!
Complex
Active/act...
Future Plans
!
We want a storage solution with the
following properties:
1. Horizontally scalable
2. Tolerant to DC failur...
Q&A
Upcoming SlideShare
Loading in …5
×

Active_Active Payments Processing

420 views

Published on

  • Be the first to comment

  • Be the first to like this

Active_Active Payments Processing

  1. 1. Active/Active Payments Processing at Square Ted Mao and Jiang-Ming Yang October 2014
  2. 2. Active/Active ! What • Resilient to datacenter-level failure • Resilient to Internet routing problems • Transparent to the merchant • No human intervention ! Why • Every second of uptime matters to our merchants. Goal is 5 9s. • Much easier and safer to perform datacenter-level maintenance.
  3. 3. Challenges ! Inconsistent state between datacenters Datacenters can’t tell if a transaction has already been processed elsewhere. ! Limited idempotence Payment networks can’t reliably guarantee idempotence on retries. ! Real-time latency requirements We can’t just wait until our datacenters get in sync. ! !
  4. 4. Concepts Client idempotence key
  5. 5. Concepts Client idempotence key Server transaction
  6. 6. Concepts Client idempotence key Server transaction Transaction progression
  7. 7. Card Processing Multi-DC resolution
  8. 8. Multi-Tender Multi-DC challenge Scenario When Merchant try to sell items/products to customers, customers will have the option to pay with multiple tenders. ! APIs 1. 1. CreateBill 2.2. AddTender 3.3. CompleteBill / CancelBIll ! Challenges 1. 1. Each time we receive a tender request, we need to process this tender immediately. Thus different tenders for the same bill may be processed at different data centers. 2.2. When receiving the CompleteBill request, we may need to wait for the tender information from remote data center.
  9. 9. Multi-Tender Multi-DC resolution
  10. 10. Multi-Tender Multi-DC resolution State Machine Tender state machine ! ! ! ! Bill state machine ! ! ! Correctness 1. 1. A formal proof 2.2. Simulate all the possible operational combinations and verify the results
  11. 11. Caveats Eventually consistent Asynchronous, eventually consistent systems are harder to reason about. ! Complex Active/active systems are harder to design, implement, and test. ! Data Loss If the original data center is down and never comes back, we may not be able the perform the capture due to the loss of original auth. ! Downstream effects Not all downstream effects are reversible.
  12. 12. Future Plans ! We want a storage solution with the following properties: 1. Horizontally scalable 2. Tolerant to DC failure 3. Transactional ! CockroachDB: a Scalable, Geo- Replicated, Transactional Datastore ! ! ! ! ! http://cockroachdb.org/
  13. 13. Q&A

×