• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
PayPal under the hood

PayPal under the hood



This is a peek at PayPal’s inner workings. Take a broad look at the technologies and foundations of the PayPal system, and the history and evolution behind its design. Take a tour of PayPal’s ...

This is a peek at PayPal’s inner workings. Take a broad look at the technologies and foundations of the PayPal system, and the history and evolution behind its design. Take a tour of PayPal’s service-oriented architecture, the techniques the engineering team uses to achieve such secure financial transaction processing, and how PayPal innovates at scale.



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds


Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • former engineercaptures our philosophy
  • what paypal tech doesI picked some problemsideas influencing devsampler platetechnical, dense12:30-1:15 Thurs 10/13 Room 2018(40m, 5m Q&A)
  • startuprelease nights, stayed up late, ate eggrolls, and crossed fingers that the site wouldn’t crashdb pw
  • daughter, (new reason to stay up all night)still eat eggrolls for release, but now it happens during the day (don’t sleep under desks)even if I wanted, customer data with 10 foot polewhat’s inside?
  • early engineerswere smart, motivated - take over the worldno problem unsolveable, no technique off-limitsinnovation (GL – impl of captcha)huge copy/paste, “magic” communication
  • voila! inside kind of looks like this.
  • spend money on reliable data storehttpd & geronimo
  • 1998 ecosystemfounding culture of “we can build it better” – different nowC++ ecosystem is weak compared to what you see in Java, Pythonwhat does all this stuff do?
  • trend: API box covering more
  • tech problems, our solutions
  • 3 themes for problems
  • why do we consider this “reliable” at this point?redo log for raw datastandby – fast failoveroffsite – recover from disasterwhat about pieces in payments that have to work together
  • lots of different systems involved in fulfilling a payment, working together reliablyif you wonder about delay
  • ensure that a payment reaches the end stateinfra technique used many domainsas a business that deals with money, how do we build trust that nothing fishy is going on?
  • trustworthy!how do you prevent or and detect tampering?
  • responsibleexamples in the industrypreso: Bill Corry info secwhat about people that are allowed to touch these things?
  • Two ways to answer the same questionchain of comparisons ultimately takes you to border between PP and external financial systemor penny-slicing, like insuperman III?round-off slicing doesn’t really apply – PP is in the middle of fxtxns, and round-off would have to be a txn
  • tricky word because it can apply to a lot of things
  • Does your codebase have “room”?tech organization that experienced huge, continuous growth, all of these dims have had “scaling” challenges
  • payment processing capacitymem managementzombie boxes in/out rotationpush (connections)less eff, but SIMPLE to debug/operateput on read-only instances of DB’shorizontal scaling, indefiniteisolate problems of state-mgt
  • our strategy for scaling readsauthentication, customer historybut what about state?
  • monster box, could take 128 CPUs, started with 48
  • too much work sync between cpuswork not totally independent (indices, etc.)
  • business functionpartition by domain, independent machinesone machine lot of CPUs couldn’t do it
  • dependencies!points of failure!work gets to be too big for one?
  • don’t need all partitions to serve requestlocalize customer data to an in-country datacenterwhat about work spanning users (later)
  • hiring more peoplelogisticsgetting people into a roommake room, keep small scope
  • domainsexercise at first forced us to define “what paypalis”given these buckets, next question was how to ensure ensure dependencies don’t just turn it into a black holesmall scope, SOAbut something else that’s proved effective is…
  • files that specify dependenciestopology and securitymaven has thistools that tell us, constrain, what talks to whatenforces boundaries – keeps things apartbut how do you get composition without coupling?
  • some principles that we’re working with, to ensure scales and is reliablesum it up in one phrase
  • ACID propertiesdo work, spanning lots of systems, together, in a way that ensures a consistent outcomeincomplete financial activityhow to classify “work” or transactions?
  • how to keep these transactions consistent at scale?first two are easy – replicas, work on single partitions
  • entities are customersother happens eventuallydecouples how they get their job done, contentionconsistent state change across entitiescloud storage APIs – this constraint in txnswhat’s the difficulty with this model?
  • RPC/http/soapreliable, trustworthy - “unknown” is the worst possible answer!all systems with RPC-style interaction, with remote stateas you partition state, more places where this can happencoding this is a mess – have infra help you
  • constraints, freedoms of: data access, RPC, memory/process model. your public APIs.Constraints define what happen when you scalewe underestimated the weight/cost of this, and didn’t invest enough early in engineering solutionsimplications can’t be completely hidden from app; rely on infrayour core competency, differentiator
  • motivating designconstantly looking for ways to push this into the infrastructures.t.devs don’t have to worry about scale, throughput, reliability, correctness – our differentiator
  • good read because:-short, with lots of pictures; appear intelligent without having to read a library- covers fundamental issues in large-scale, reliable, distributed systemsweakened ACIDPP Wars Eric Jackson, beginnings, war stories

PayPal under the hood PayPal under the hood Presentation Transcript

  • “Lose a penny and everybody goes %#!’ing crazy!”
  • Introduction to PayPal technology Challenges & solutions Design Concepts
  • Technology
  • http://ajsmarowsky.tripod.com/WEB260_test2/images/img_online_crime.jpg
  • web/API C++/Java/Python Linuxservices C++/Java Linux permanenttransient “mayfly” store Oraclestore (database) IBM/AIX, Sun Linux
  • web API paymentauth wallet processing
  • foreign risk exchange fees models limitsreceiving prefs. refunds IPN item info
  • Challenges/Solutions• reliability• security• scale
  • app 1 work 2 work 3 COMMITRedo log standby offsite failover
  • app A1 work2 work3 insert message 5 read message app B4 COMMIT 6 if message isn’t “done” 7 work 8 mark message as “done” 9 COMMIT
  • 5 sync call app A1 work2 work3 insert message 10 read message app B4 COMMIT 6 if message isn’t “done” 7 work 8 mark message as “done” 9 COMMIT
  • physical security machine access controls firewallsservice-level access control encryption on the wire encryption at resthardware encryption (HSM)
  • balance log balance: $100 Account ActivityOct 1 open: $0 Oct 12 Add Funds $15Oct 12 +$150 from Bank 0Oct 13 - $50 Oct 13 Balance- $50 funded Payment
  • 2002 2011payments/sec tens hundredsSLOC ~1M ~10Mapplications tens hundredsdevelopers tens thousandsrelease LTS monthly dailycomplexity some lots
  • server child process
  • server server server machine machine machine read read replica replica
  • app 1 app 2 app 3 app 4 database
  • app 1 app 2 app 3 app 4 database
  • app 1 app 2 app 3 app 4 cs risk transaction profile s
  • app 1 app 2 app 3 app 4 US EU profile txn profile txn risk cs risk cs
  • Account Servicing Financial Product Mobile RiskCompliance Help Money SecurityConsumer History Money Market ShippingCustomer Service Incentive Notification Skype™Dropbox Infrastructure Payflow Trinity™FinOps/Recon Marketing Platform UserFinSys Marketplaces Presentation WebFinancial Instrument Merchant
  • app_A: app A: CONNECTS_TO app_B DEPENDS_ON foo.a CONNECTS_TO app_C library foo.a:… INCLUDES foo.o DEPENDS_ON bar.a ONLY_WITH payment_app
  • Design Concepts
  • History pageChangepasswordPayments
  • reliable orchestrationcustomer 1 customer 2 eventually consistent
  • paymentprocessing payment customerprocessing balance recovery idem- potent
  • Wrap-up
  • “Lose a penny and everybody goes%#!’ing crazy!” “It’s all about the [transactions], [baby]”
  • Life Beyond Distributed TransactionsThe PayPal Wars
  • Thanks!Q&A