• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
PayPal under the hood
 

PayPal under the hood

on

  • 803 views

This is a peek at PayPal’s inner workings. Take a broad look at the technologies and foundations of the PayPal system, and the history and evolution behind its design. Take a tour of PayPal’s ...

This is a peek at PayPal’s inner workings. Take a broad look at the technologies and foundations of the PayPal system, and the history and evolution behind its design. Take a tour of PayPal’s service-oriented architecture, the techniques the engineering team uses to achieve such secure financial transaction processing, and how PayPal innovates at scale.

Statistics

Views

Total Views
803
Views on SlideShare
803
Embed Views
0

Actions

Likes
1
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • former engineercaptures our philosophy
  • what paypal tech doesI picked some problemsideas influencing devsampler platetechnical, dense12:30-1:15 Thurs 10/13 Room 2018(40m, 5m Q&A)
  • startuprelease nights, stayed up late, ate eggrolls, and crossed fingers that the site wouldn’t crashdb pw
  • daughter, (new reason to stay up all night)still eat eggrolls for release, but now it happens during the day (don’t sleep under desks)even if I wanted, customer data with 10 foot polewhat’s inside?
  • early engineerswere smart, motivated - take over the worldno problem unsolveable, no technique off-limitsinnovation (GL – impl of captcha)huge copy/paste, “magic” communication
  • voila! inside kind of looks like this.
  • spend money on reliable data storehttpd & geronimo
  • 1998 ecosystemfounding culture of “we can build it better” – different nowC++ ecosystem is weak compared to what you see in Java, Pythonwhat does all this stuff do?
  • trend: API box covering more
  • tech problems, our solutions
  • 3 themes for problems
  • why do we consider this “reliable” at this point?redo log for raw datastandby – fast failoveroffsite – recover from disasterwhat about pieces in payments that have to work together
  • lots of different systems involved in fulfilling a payment, working together reliablyif you wonder about delay
  • ensure that a payment reaches the end stateinfra technique used many domainsas a business that deals with money, how do we build trust that nothing fishy is going on?
  • trustworthy!how do you prevent or and detect tampering?
  • responsibleexamples in the industrypreso: Bill Corry info secwhat about people that are allowed to touch these things?
  • Two ways to answer the same questionchain of comparisons ultimately takes you to border between PP and external financial systemor penny-slicing, like insuperman III?round-off slicing doesn’t really apply – PP is in the middle of fxtxns, and round-off would have to be a txn
  • tricky word because it can apply to a lot of things
  • Does your codebase have “room”?tech organization that experienced huge, continuous growth, all of these dims have had “scaling” challenges
  • payment processing capacitymem managementzombie boxes in/out rotationpush (connections)less eff, but SIMPLE to debug/operateput on read-only instances of DB’shorizontal scaling, indefiniteisolate problems of state-mgt
  • our strategy for scaling readsauthentication, customer historybut what about state?
  • monster box, could take 128 CPUs, started with 48
  • too much work sync between cpuswork not totally independent (indices, etc.)
  • business functionpartition by domain, independent machinesone machine lot of CPUs couldn’t do it
  • dependencies!points of failure!work gets to be too big for one?
  • don’t need all partitions to serve requestlocalize customer data to an in-country datacenterwhat about work spanning users (later)
  • hiring more peoplelogisticsgetting people into a roommake room, keep small scope
  • domainsexercise at first forced us to define “what paypalis”given these buckets, next question was how to ensure ensure dependencies don’t just turn it into a black holesmall scope, SOAbut something else that’s proved effective is…
  • files that specify dependenciestopology and securitymaven has thistools that tell us, constrain, what talks to whatenforces boundaries – keeps things apartbut how do you get composition without coupling?
  • some principles that we’re working with, to ensure scales and is reliablesum it up in one phrase
  • ACID propertiesdo work, spanning lots of systems, together, in a way that ensures a consistent outcomeincomplete financial activityhow to classify “work” or transactions?
  • how to keep these transactions consistent at scale?first two are easy – replicas, work on single partitions
  • entities are customersother happens eventuallydecouples how they get their job done, contentionconsistent state change across entitiescloud storage APIs – this constraint in txnswhat’s the difficulty with this model?
  • RPC/http/soapreliable, trustworthy - “unknown” is the worst possible answer!all systems with RPC-style interaction, with remote stateas you partition state, more places where this can happencoding this is a mess – have infra help you
  • constraints, freedoms of: data access, RPC, memory/process model. your public APIs.Constraints define what happen when you scalewe underestimated the weight/cost of this, and didn’t invest enough early in engineering solutionsimplications can’t be completely hidden from app; rely on infrayour core competency, differentiator
  • motivating designconstantly looking for ways to push this into the infrastructures.t.devs don’t have to worry about scale, throughput, reliability, correctness – our differentiator
  • good read because:-short, with lots of pictures; appear intelligent without having to read a library- covers fundamental issues in large-scale, reliable, distributed systemsweakened ACIDPP Wars Eric Jackson, beginnings, war stories

PayPal under the hood PayPal under the hood Presentation Transcript

  • “Lose a penny and everybody goes %#!’ing crazy!”
  • Introduction to PayPal technology Challenges & solutions Design Concepts
  • Technology
  • http://ajsmarowsky.tripod.com/WEB260_test2/images/img_online_crime.jpg
  • web/API C++/Java/Python Linuxservices C++/Java Linux permanenttransient “mayfly” store Oraclestore (database) IBM/AIX, Sun Linux
  • web API paymentauth wallet processing
  • foreign risk exchange fees models limitsreceiving prefs. refunds IPN item info
  • Challenges/Solutions• reliability• security• scale
  • app 1 work 2 work 3 COMMITRedo log standby offsite failover
  • app A1 work2 work3 insert message 5 read message app B4 COMMIT 6 if message isn’t “done” 7 work 8 mark message as “done” 9 COMMIT
  • 5 sync call app A1 work2 work3 insert message 10 read message app B4 COMMIT 6 if message isn’t “done” 7 work 8 mark message as “done” 9 COMMIT
  • physical security machine access controls firewallsservice-level access control encryption on the wire encryption at resthardware encryption (HSM)
  • balance log balance: $100 Account ActivityOct 1 open: $0 Oct 12 Add Funds $15Oct 12 +$150 from Bank 0Oct 13 - $50 Oct 13 Balance- $50 funded Payment
  • 2002 2011payments/sec tens hundredsSLOC ~1M ~10Mapplications tens hundredsdevelopers tens thousandsrelease LTS monthly dailycomplexity some lots
  • server child process
  • server server server machine machine machine read read replica replica
  • app 1 app 2 app 3 app 4 database
  • app 1 app 2 app 3 app 4 database
  • app 1 app 2 app 3 app 4 cs risk transaction profile s
  • app 1 app 2 app 3 app 4 US EU profile txn profile txn risk cs risk cs
  • Account Servicing Financial Product Mobile RiskCompliance Help Money SecurityConsumer History Money Market ShippingCustomer Service Incentive Notification Skype™Dropbox Infrastructure Payflow Trinity™FinOps/Recon Marketing Platform UserFinSys Marketplaces Presentation WebFinancial Instrument Merchant
  • app_A: app A: CONNECTS_TO app_B DEPENDS_ON foo.a CONNECTS_TO app_C library foo.a:… INCLUDES foo.o DEPENDS_ON bar.a ONLY_WITH payment_app
  • Design Concepts
  • History pageChangepasswordPayments
  • reliable orchestrationcustomer 1 customer 2 eventually consistent
  • paymentprocessing payment customerprocessing balance recovery idem- potent
  • Wrap-up
  • “Lose a penny and everybody goes%#!’ing crazy!” “It’s all about the [transactions], [baby]”
  • Life Beyond Distributed TransactionsThe PayPal Wars
  • Thanks!Q&A