6/18/14 Billing & Payments Engineering Meetup I

65,419 views

Published on

06/18/2014 - Billing & Payments Engineering Meetup @ Netflix

For this Meetup, we have invited speakers from several tech companies to give a series of lightening talks on challenges related to billing & payments systems.

This event is for engineers who are interested in learning more about billing & payments systems. No previous experience with this kind of system is required to attend.

Presenters:
- Mathieu Chauvin - Engineering Manager for Payments @ Netflix
- Taylor Wicksell - Sr. Software Engineer for Billing @ Netflix
- Jean-Denis Greze - Engineer @ Dropbox
- Alec Holmes - Software Engineer @ Square
- Emmanuel Cron - Software Engineer III, Google Wallet @ Google
- Paul Huang - Engineering Manager @ Survey Monkey
- Anthony Zacharakis - Lead Engineer @ Lumos Labs
- Shengyong Li / Feifeng Yang - Dir. Engineering Commerce / Tech Lead Payment @ Electronic Arts

6/18/14 Billing & Payments Engineering Meetup I

  1. 1. Billing & Payments Engineering Meetup June 18th, 2014
  2. 2. • Mathieu Chauvin – Netflix Validating Payment Information • Taylor Wicksell – Netflix Flexible Billing Integrations Using Events • Jean-Denis Greze – Dropbox Mistakes (and Wins) Building Payments at Dropbox • Anthony Zacharakis – Lumosity Billing Migrations: Maintaining Your (Data) Sanity • Alec Holmes – Square Scaling Merchant Payouts at Square === Break === • Paul Huang – SurveyMonkey Billing Globalization and Auto Renewal Optimization • Emmanuel Cron – Google Wallet Moving from SE to Host Card Emulation • Feifeng Yang / Michael Chen – Electronic Arts Ecommerce at EA • Krishnan Sridhar – LinkedIn Real-Time Analytics and Smart Routing
  3. 3. Validating Payment Information Mathieu Chauvin @matchauvin – linkedin/matchauvin
  4. 4. Signing-Up for Netflix • Subscription Based • 30 Days Free Trial • Payment Information Collection
  5. 5. • Pro Forma • Luhn Check (Mod10) • Card Type vs Card# RegEx • Authorization / Reversal Payment Validation 101
  6. 6. But Wait! You Can Do More!
  7. 7. Authorization Amount • $0.00, $1.00, or Full Price Authorization • Don’t Forget the Reversal! • Constraints by Country or by Card Type
  8. 8. Response Types • Hard Declines • Soft Declines
  9. 9. MCC • What is an MCC? • Figure out the Best MCC Fit
  10. 10. BINs • What is a BIN? • Card Types: Credit, Debit, Prepaids • What About Non-Reloadable Prepaid Cards?
  11. 11. Forking Traffic • Do You Use Several Payment Processors? • Why Don’t You Test Which One Performs Better?
  12. 12. Use Cases • Sign-Up • Updating Payment Information • Returning Customer
  13. 13. Country Specific Behavior • Adapt To Your Market
  14. 14. The Feedback Loop • A/B Tests • Analyze, Project & Evaluate Risks • Test Again
  15. 15. Conclusion • Try Everything • Build Your Application Dynamically • The Feedback Loop! Thank You!
  16. 16. Flexible Billing Integrations Using Events Taylor Wicksell
  17. 17. The Billing Team • Signups/Cancellations • Recurring Billing • Price and Tax Calculations • Discounts / Gifts • Financial Reporting
  18. 18. Current Architecture • Netflix Data Center • Batch Processing and Customer Requests • Oracle (Single-Master) • Data driven
  19. 19. Cloud Migration - Primary Goals • Join Netflix in the Cloud • High Availability • Multi-Regional Active/Active • Scalability
  20. 20. New Design • Amazon EC2 • Multi-Regional • Cassandra • Event Driven
  21. 21. Order Flow
  22. 22. Cross-Cutting Concerns • Publishing Data To Interested Parties – Hold Messaging – Financial Reporting • Metrics/Analytics • Velocity Checks • Debugging
  23. 23. Wiretap to Common Event Pipeline
  24. 24. Common Event Pipeline Current Implementation • SQS Entry Point • Custom routing service • Custom code for each endpoint integration • Weak ordering of events Future Implementation • Suro / Kafka Entry Point • Configurable Routing and Transformation • Pluggable endpoint integration • Option for strong ordering of events
  25. 25. Sting – Broad Analytics
  26. 26. Druid – Event Aggregation Metrics
  27. 27. ElasticSearch – Event Level Debugging
  28. 28. Mistakes (and Wins) Building Payments At Dropbox Jean-Denis Grèze & Dan Wheeler June 18th, 2014
  29. 29. Backend Tips • Not about increasing conversion • Not about pricing • Not about plan and feature optimizations • Not about upselling • Not about consumer SaaS at scale • Not about self-serve in SMB/SME/Enterprise
  30. 30. Pains of Scaling Payments • Thousands of customers to millions of customers • SMB to Enterprise – Custom flows! • International expansion – Fraud – New payment methods (delayed settlements) – Different price points
  31. 31. Out Of The Dark Ages • For a long time, only 0.5 engineers worked on payments and billing • March 2013: consult w/ leading payment engineers, PMs and executives on how to build an amazing payments team – 15+ in 1 year
  32. 32. Advice • Build a payments + billing backend that is: – Flexible • Migrations (sadface) • Requirements will change – often – Auditable • Always know why and state changed – Append Only • Never lose data
  33. 33. Team Ca$h
  34. 34. Team Ca$h • Engineers – Some finance background – A lot of systems/analytics background • PMs – Some finance/accounting background – Some product/marketing background • Advisors – Payments experts – Tight feedback loop w/ finance • Designers
  35. 35. Migrations
  36. 36. Migrations • You will have to migrate – 3rd-party vaulting to self vaulting – New markets = new processors – If you are a growing company, your internals will require migrations • Stakes are high – Double-billing? Forgetting to charge some users? Inadvertently moving users from one pricing category to another? • Old way: – Ad hoc – Tons of tests
  37. 37. I. Leverage Existing Code: Equivalence • Write equivalence between old and new implementations (database, API, 3rd party providers, tests, etc.) • Run everything through both systems at once, with equivalence being tested • Every step of the way check that equivalence relations hold (e.g., old-style invoice has a new-style invoice equivalent) • Turn off old system when everything works for X amount of time
  38. 38. Migration Pro-Tip • If you can migrate in both directions at will on a per-endpoint basis, your life will be awesome and people will love you.
  39. 39. Moneytree
  40. 40. Logging • Dark Ages = tons of logging • Very comprehensive, but ad-hoc = too much effort to re-create state • Human error
  41. 41. II. Automated Logging • Automatically log – Any DB write (graph, relational, etc.) – Any 3rd party API call (and some internal calls like email) • Pre-log – Any incoming 3rd party payload • Can recreate past actions if we introduced a regression • LOG A REASON (and code version) – 1 year is a long time
  42. 42. Automated Logging = Time Machine
  43. 43. Not too much data • Dropbox = large scale for SaaS • Hundreds of millions of users (provisioning is hard) • Tons of data, but still payments data << file storage data – Although we do have the benefits of amazing infrastructure
  44. 44. III. States, Not Deltas • Generally # states << # deltas • States – Pro 100 + Packrat – DfB + Harmony Enabled • Deltas – Add 5 licenses, 6 licenses, etc. – Add 20 licenses and switch from monthly to yearly • Use states and let the system figure out how to get from start to end
  45. 45. IV. Possibility & Transitions Use Same Code Paths • No difference between: – entity.is_valid_transition(end_state) – entity.perform_transition(end_state) • Except that writes are turned off for the former. • No change for “is something possible” logic to be different than “do the thing” logic.
  46. 46. States Are Nice transition_space = MoneytreeTransitionsSpace.build_cross_product( entity=me, gateways=ALL_GATEWAYS, plans=[Pro100, Pro200, Pro500, DfB, DfBTrial], schedules=[Monthly, Yearly], currencies=ALL_CURRENCIES, features=ALL_FEATURES, tax_profile=[NoTax, SimpleTax, ComplexTax], ) # … If transition_space.supports(FEATURE_PACKRAT): # …
  47. 47. Write Protection • Seems dumb, but need to be careful not to accidentally change values in payments world. Have clearly-defined code paths that can touch state, talk to 3rd party components, etc.
  48. 48. V. One More Lesson • Payments + Billing != Finance • Business requirements don’t always translate to what’s best/easiest in the world of accounting. You need to flexibly work in both worlds – can’t risk the user experience to make your finance dep’t happy (and vice-versa) • Get a great PM
  49. 49. VI. Why Payments Are Cool? • Infrastructure? – Payments service – Provisioning service • Product? – Upsells? (+50% increase in revenue per user) – Gating features? • Product Infrastructure! – Build a successful structure by emphasizing hard problems in both worlds!
  50. 50. Now + The Future • Other Cool Projects – ML for risk/anomaly detection (e.g., for payment methods that don’t settle immediately) – Price AB testing (*) – Cross-platform upsell framework • Questions? – dan@dropbox.com – jeandenis@dropbox.com • Hiring – Get in touch!
  51. 51. Team Ca$h
  52. 52. Lumos Labs, Inc. Data Sanity A short treatise on rebuilding a payment system
  53. 53. Lumos Labs, Inc. the background
  54. 54. Lumos Labs, Inc. Circa 2012 Decided to rewrite our payment system
  55. 55. Lumos Labs, Inc. Reasons Old payment system has a lot of limitations
  56. 56. Lumos Labs, Inc. Payment system limitations • Hard to add additional gateways • Models don’t reflect business reality • Not built for reporting • Code is brittle
  57. 57. Lumos Labs, Inc. 3 months later…
  58. 58. Lumos Labs, Inc. New payment system features • Trivial to add new payment methods • Subscriptions are the core model • Built with reporting in mind • Separate, well encapsulated library
  59. 59. Lumos Labs, Inc. Works like a charm!
  60. 60. Lumos Labs, Inc. wait… just one problem
  61. 61. Lumos Labs, Inc. Having two parallel payment systems is not sustainable Surprise! As it turns out
  62. 62. Lumos Labs, Inc. Just deprecate the old one
  63. 63. Lumos Labs, Inc. Just deprecate the old one • Don't migrate anyone, let old users churn out naturally (will take forever)
  64. 64. Lumos Labs, Inc. Just deprecate the old one • Don't migrate anyone, let old users churn out naturally (will take forever) • Migrate everyone, but only most critical/current subscription info (loses history)
  65. 65. Lumos Labs, Inc. Just deprecate the old one • Don't migrate anyone, let old users churn out naturally (will take forever) • Migrate everyone, but only most critical/current subscription info (loses history) • Migrate everyone + full history (tricky, lots of edge cases)
  66. 66. Lumos Labs, Inc. so, what’d we choose? (drum roll)
  67. 67. Lumos Labs, Inc. We decided to migrate everyone and their entire billing history
  68. 68. Lumos Labs, Inc. Why? One system One history One source of truth
  69. 69. Lumos Labs, Inc.
  70. 70. Lumos Labs, Inc. enter the sanity check
  71. 71. Lumos Labs, Inc. enter the sanity check Just make a sanity check before and after the migration to ask questions
  72. 72. Lumos Labs, Inc. enter the sanity check Both models should answer certain questions the same way, e.g: • How much did the user pay for a subscription? • How many total transactions did the user make? • Was auto-renewal enabled on X date?
  73. 73. Lumos Labs, Inc. class SanityCheck Methods = [:subscriber?, :transaction_count] # etc. def initialize(record) @record = record @before_values = SanityCheck.values(record) end def self.values(record) Methods.map { |m| [m, record.send(m)] }.to_h end end enter the sanity check
  74. 74. Lumos Labs, Inc. class SanityCheck ... def check @after_values = SanityCheck.values(record) @diff = diff(@before_values, @after_values) end def diff(a, b) a.delete_if { |k, v| b[k] == v } .merge!(b.dup.delete_if { |k, v| a.has_key?(k) }) end end enter the sanity check
  75. 75. Lumos Labs, Inc. enter the sanity check User.each do |user| sanity_check = SanityCheck.new(user) user.migrate! sanity_check.check if sanity_check.diff.any? # sanity check failed -- log an error, rollback, etc. raise ActiveRecord::Rollback else # woo hoo, success! user.update_attributes(:migrated => true) end end config/initializers/user_auth.rb
  76. 76. Lumos Labs, Inc. the result
  77. 77. Lumos Labs, Inc. A resounding success! the result
  78. 78. Lumos Labs, Inc. (Mostly) the result
  79. 79. Lumos Labs, Inc. Did not work in all cases ● Payment system behavior changed over time
  80. 80. Lumos Labs, Inc. Did not work in all cases ● Payment system behavior changed over time ● Some concepts did not map between systems
  81. 81. Lumos Labs, Inc. Handling the edge cases ● Replayed history as it would play out in the new system ● Still kept most critical information the same (e.g. transaction timestamps)
  82. 82. Lumos Labs, Inc. Skip ahead to today Migrated 99.9994%* of all our users successfully to the new system *The remaining .0006% live on for business, not technical reasons
  83. 83. Lumos Labs, Inc. Thanks Anthony Zacharakis anthony@lumoslabs.com @azacharax
  84. 84. Scaling Merchant Payouts at Square Alec Holmes
  85. 85. ‣ Infrastructure team that moves money ‣ Card processing (money in) ‣ Settlements (money out) Payments Team
  86. 86. ‣ Monolithic Rails app ‣ Database storage and IO limits ‣ Controls around accounting data Disaster Ahead
  87. 87. Monorail Database Payments Bills Deposits Where We Started
  88. 88. Ledger Database Payments Bills Deposits Monorail Database API Current System
  89. 89. Challenge: Migrating Live Data ‣ Stricter money-moving event model ‣ Same event streams to all systems ‣ Smart queueing of events
  90. 90. Challenge: Moving to APIs ‣ Replace direct data access with API calls ‣ Hadoop for analytics queries
  91. 91. That's it. All pennies accounted fo
  92. 92. Billing Globalization and Auto- Renewal Optimization • Paul Huang • Engineering Manager • SurveyMonkey • June 18th, 2014
  93. 93. Our mission is to help people make better decisions
  94. 94. We help people get answers to their questions customers create surveys distribute to others recipients respond customers analyze and gain insight
  95. 95. Our customers have invented many ways to use our tool 98
  96. 96. 90MUnique visitors every month 17K New sign-ups daily “SurveyMonkey turns online surveys into a hot business.” “ Start-up companies using ‘freemium’ business models, including SurveyMonkey, are thriving as the cost of computer power and storage falls.” One of the hottest startups to watch. 2.4MSurvey responses are generated daily SurveyMonkey is the world’s largest survey company
  97. 97. Billing Globalization In 2010... Currencies:1 currency (USD). Payment Methods: Credit Card, Invoice. Pricing: one price per package. Revenue: $ MM
  98. 98. Billing Globalization In 2014... Currencies: 39 international currencies. Payment Methods: Credit Card, PayPal, Debit Card, Bank Transfer, iTunes, Invoice. Pricing: each package with different price per currency, multiple prices allowed for price testing. Revenue: $$$$ MM
  99. 99. Auto-Renewal Optimization Retry Logic: retry failed payments at different intervals, ~60% retry success rate Account Updater: get users’ updated credit card data (number / expiration date) before next renewal date, ~4% users at 96% success rate Immediate Renewal: charge pending invoices when users update payment accounts, ~4% retry success rate improvement
  100. 100. The Road Ahead
  101. 101. Upcoming Billing Projects in 2014 VAT 2015: preparations for VAT rule changes in Europe in 2015 Brazil Payment Processing: set up local entity, integrate with a new Payment Service Provider in Brazil Continue Improving Retry Logic: increase retry frequencies
  102. 102. The End (BTW, we’re hiring…)
  103. 103. 112 * Jun 1st to Oct 17th

×