Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Load Testing at Scale - Grace Hopper Celebration 2018

141 views

Published on

Load testing and disaster recovery strategies at LinkedIn

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Load Testing at Scale - Grace Hopper Celebration 2018

  1. 1. LOAD TESTING AT SCALE L O A D T E S T I N G A N D D I S A S T E R R E C O V E R Y S T R A T E G I E S #GHC 18 Zaina Afoulki | zaina@linkedin.com
  2. 2. PAGE 2 | GRACE HOPPERCELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY Site ReliabilityEngineerrole at LinkedIn Scalechallengesat LinkedIn LinkedIn’straffic tier architecture Preventingdisasterswith traffic shifts Load testing Q&A TODAY’S AGENDA #GHC 18
  3. 3. PAGE 3 | GRACE HOPPERCELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY Develop solutions to design,build and run large-scalesystems that arereliable, scalableand efficient Guidesystem architectureby operatingat the intersection of softwaredevelopment and systemsengineering We approachour work with a spiritof constructivepessimism:we hope for the best, but plan for the worst THE (SRE) SITE RELIABILITY ENGINEER ROLE AT LINKEDIN IN A NUTSHELL #GHC 18
  4. 4. PAGE 4 | GRACE HOPPERCELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY HardwareFailure HumanError Releasing BadCode Service Degradation #GHC 18 WHAT CAN GO WRONG?
  5. 5. PAGE 5 | GRACE HOPPERCELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY 575Mmembers 2 new members per second 15Mactive job listings #GHC 18 SCALE CHALLENGES AT LINKEDIN
  6. 6. PAGE 6 | GRACE HOPPERCELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY Fabric/Colo: Data Center with full application stack deployed PoP/Edge:Entry point to LinkedIn network (TCP/SSL Termination) QPS: Queries Per Second TERMINOLOGY #GHC 18
  7. 7. PAGE 7 | GRACE HOPPERCELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY #GHC 18 WHEN A USER VISITS LINKEDIN.COM, THEIR REQUESTS ARE SENT TO ONE OF 18 POPS
  8. 8. PAGE 8 | GRACE HOPPERCELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY LINKEDIN TRAFFIC ARCHITECTURE #GHC 18 Edge Fabric Sticky Routing Service Border Router Internet IPVS ATS ATS Front End
  9. 9. PAGE 9 | GRACE HOPPERCELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY LINKEDIN TRAFFIC ARCHITECTURE #GHC 18 Edge Fabric Sticky Routing Service ATS DC1 DC2 Request
  10. 10. PAGE 10 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY LINKEDIN TRAFFIC ARCHITECTURE #GHC 18 Buckets DC1 Fabric Sticky Routing
  11. 11. PAGE 11 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY WHEN A DISASTER STRIKES, WE REDIRECT LIVE TRAFFIC TO ANOTHER DATA CENTER #GHC 18
  12. 12. PAGE 12 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY #GHC 18 PREVENTING DISASTERS WITH TRAFFIC SHIFT
  13. 13. PAGE 13 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY #GHC 18 DURING A LOAD TEST, EXTRA TRAFFIC IS RE-ROUTED FROM OTHER COLOS DC1 DC2 Target Traffic percentage 60% BucketsBuckets
  14. 14. PAGE 14 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY #GHC 18 WE REGULARLY LOAD TEST OUR DATA CENTERS TO IDENTIFY CAPACITY BOTTLENECKS
  15. 15. PAGE 15 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATIONFORCOMPUTINGMACHINERY • Capacity Planning • Leverage Production traffic to stress test services • Identify bugs in production • Confidence in disaster recovery BENEFITS OF LOAD TESTING #GHC 18
  16. 16. Thank You #GHC 18 Zaina Afoulki | zaina@linkedin.com

×