Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

From a monolith to microservices + REST: The evolution of LinkedIn's architecture

5,983 views

Published on

This is the story of LinkedIn's journey from a monolithic web application to a microservice based architecture, some of the challenges they faced along the way, and the tools they built to make this transition possible, including the Rest.li and Deco frameworks.

Published in: Software
  • It changed my life. After ten long years of being bulimic I am now bulimia free for six months. All because of this bulimia recovery program. ▲▲▲ http://t.cn/A6Pq6KF6
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download Full EPUB Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download Full doc Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download PDF EBOOK here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download EPUB Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download doc Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download Full EPUB Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download Full doc Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download PDF EBOOK here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download EPUB Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download doc Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • The 3 Secrets To Your Bulimia Recovery ■■■ http://scamcb.com/bulimiarec/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

From a monolith to microservices + REST: The evolution of LinkedIn's architecture

  1. 1. From a monolith to microservices + REST The evolution of LinkedIn’s service architecture by Steven Ihde and Karan Parikh (LinkedIn)
  2. 2. Leo ● Our original codebase ● Java, Servlets, JSP, JDBC Leo Oracle 4
  3. 3. Remote Graph ● Graph: member-to-member connection graph ● Complex graph traversal problems not suited to SQL queries ● RPC was used to keep it separate from Leo ● Our first service 5
  4. 4. Remote Graph Leo Oracle Graph RPC JDBC Databus 6
  5. 5. Mid/Back Tier Services ● “Back” tier services encapsulate data domains ● “Mid” tier services provide business logic ● We applied the service pattern to many domains, e.g. member profiles, job postings, group postings 7
  6. 6. Front Tier Services ● “Front” tier services aggregate data from many domains ● Transform the data through templates to present to the client ● Should be stateless for scaling purposes 8
  7. 7. Service Explosion ● Over 100 services by 2010 ● Most new development occurring in services, not Leo ● Site release every two weeks 9
  8. 8. Architectural Challenges ● Test failures ● Incompatibilities ● Complex orchestration ● Rollback difficult or impossible ● Complex dependencies between services 10
  9. 9. Microservices? ● Services were fine grained ● But monolithic build and release process did not allow us to realize the benefits of microservice architecture 11
  10. 10. Solutions ● Continuous delivery ● Break apart the code base ● Devolution of control ● Strict backwards compatibility ● Better defined boundaries between tiers 12
  11. 11. Continuous Delivery ● Shared trunk ● Pre- and post-commit automated testing ● Easy promotion of builds to production environment 13
  12. 12. Decentralize Codebase ● Separate, independently buildable repositories ● Shared trunk within each repository ● Versioned binary dependencies between repositories 14
  13. 13. Devolution of Control ● Service owners control release schedule, release criteria ● Service owners are responsible for backwards compatibility ● Services must release independently 15
  14. 14. Backwards compatibility ● Insulates teams from each other at runtime ● Allows service owners to deploy on their own schedule without impacting clients 16
  15. 15. Boundaries Between Tiers ● Limit aggregation to the front tier ● Limit crosstalk in the back tier: “superblocks” 17
  16. 16. Boundaries Between Tiers 18
  17. 17. Java RPC ● Difficult to maintain backwards compatibility ● Verb-centric APIs ● Use case specific APIs ● Difficult to navigate the proliferation of APIs 19
  18. 18. Rest.li plus Deco equals Microservices at LinkedIn
  19. 19. What is Rest.li? “Rest.li is an open source REST framework for building robust, scalable RESTful architectures using type-safe bindings and asynchronous, non-blocking I/O.” Primarily JSON over HTTP. 21
  20. 20. Why Rest.li? ● Polyglot (frontend) ecosystem - Java, Scala, Python, Node.js, Objective-C ● Uniform service interfaces (REST) 22
  21. 21. The Rest.li stack Rest.li Data layer and RESTful operations D2 Dynamic discovery and load balancing R2 Network Communication 23
  22. 22. Request Response (R2) ● REST abstraction that can send messages over any application layer protocol (HTTP, PRPC (old custom LinkedIn protocol)) ● Client - fully asynchronous Netty ● Server - Jetty, Netty (experimental) Rest.li D2 R2 24
  23. 23. Dynamic Discovery (D2) ● Apache ZooKeeper ● Dynamic server discovery ● Client side software load balancing ● D2 service Rest.li D2 R2 25
  24. 24. Rest.li ● Data using PDSCs (Pegasus Data Schemas) ● RESTful API that developers use to build services ● CRUD + finders + actions ● API and data backwards compatibility checking Rest.li D2 R2 26
  25. 25. 830 Rest.li resources. 90 billion Rest.li calls/day across multiple datacenters. 65% service-to-service calls.
  26. 26. What is deco? 28
  27. 27. Aside: Normalized Domain Models ● Links over inclusion (denormalization) ● URNs are fully qualified foreign keys InfluencerPost Long id String title String content URN author Member Long id String firstName String lastName String summary URN company 29
  28. 28. What is Deco? ● URN resolution library ● What data you want, not how you want it Time 1 2 3 30
  29. 29. Deco Example: Influencer Post Dummy Post by Karan Parikh at LinkedIn Hi QCon! /influencerPosts /profiles /companies /influencerPosts 31
  30. 30. Deco Example: Influencer Post deco://influencerPosts/123?projection=(title, content, author~(firstName, lastName, company~(companyName))) 32
  31. 31. Three services. One client call. Deco.
  32. 32. Rest.li plus Deco equals Microservices at LinkedIn
  33. 33. How Rest.li enables Microservices ● Rest.li + D2 facilitate domain specific services ● Services can easily configure clients via D2 ● D2 helps us scale the architecture 35
  34. 34. How Deco enables Microservices ● Deals with service explosion ● Abstracts away services from clients 36
  35. 35. Challenges ● Coordinating a massive engineering effort. (LiX to the rescue!) ● Ensuring uniform RESTful interfaces ● Performance 37 Rest.li API Hub
  36. 36. Wins ● All languages talk to the same service ● Developer productivity ● Reduction of hardware load balancers ● Ability to expose APIs directly to third parties 38
  37. 37. LinkedIn Microservices
  38. 38. Questions?
  39. 39. References and links ● Rest.li: http://rest.li/ ● Rest.li API Hub: https://github.com/linkedin/rest.li-api-hub ● Rest.li user guide: https://github.com/linkedin/rest.li/wiki/Rest.li-User-Guide ● Modeling resources with Rest.li: https://github.com/linkedin/rest.li/wiki/Modeling-Resources-with-Rest.li ● LinkedIn engineering blog posts about Rest.li: http://engineering.linkedin.com/architecture/restli-restful-service-architecture- scale http://engineering.linkedin.com/restli/linkedins-restli-moment ● LinkedIn’s GitHub projects http://linkedin.github.io/ 43
  40. 40. Thanks! ● Steven Ihde: http://www.linkedin.com/in/stevenihde ● Karan Parikh: http://www.linkedin.com/in/parikhkaran https://twitter.com/karanrparikh http://karanparikh.com/ 44

×