Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scaling LoL Chat to 70M Players

10,663 views

Published on

Would you ever play an online game if you were not able to communicate with your teammates? Isn’t it fun if you can make new friends, arrange pre-made games and celebrate your victories with people you like to play with?

Riot Games’ League of Legends handles millions of online players at any given time. Each chat server is responsible for routing over 1 billion real time events a day. In order to support the overwhelming user base and be prepared future growth, as well as pave the road for the upcoming features, chat infrastructure had to be designed and built with the utmost care, so that it would never fail the players.

In this talk I would like to present how we achieved linear scalability, improved the overall fault tolerance, created a framework for real time code upgrades and got ready for the new features we want to ship. I will also discuss in detail why we chose to use Erlang as a foundation for the system, and why we migrated our data from MySQL to Riak.

Published in: Engineering
  • If we are speaking about saving time and money this site ⇒ www.WritePaper.info ⇐ is going to be the best option!! I personally used lots of times and remain highly satisfied.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THE BOOK INTO AVAILABLE FORMAT (New Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THE can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THE is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBOOK .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookBOOK, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, EBOOK, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THE Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THE the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THE Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi there! I just wanted to share a list of sites that helped me a lot during my studies: .................................................................................................................................... www.EssayWrite.best - Write an essay .................................................................................................................................... www.LitReview.xyz - Summary of books .................................................................................................................................... www.Coursework.best - Online coursework .................................................................................................................................... www.Dissertations.me - proquest dissertations .................................................................................................................................... www.ReMovie.club - Movies reviews .................................................................................................................................... www.WebSlides.vip - Best powerpoint presentations .................................................................................................................................... www.WritePaper.info - Write a research paper .................................................................................................................................... www.EddyHelp.com - Homework help online .................................................................................................................................... www.MyResumeHelp.net - Professional resume writing service .................................................................................................................................. www.HelpWriting.net - Help with writing any papers ......................................................................................................................................... Save so as not to lose
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THE BOOK INTO AVAILABLE FORMAT (New Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THE can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THE is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBOOK .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookBOOK, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, EBOOK, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THE Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THE the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THE Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THE BOOK INTO AVAILABLE FORMAT (New Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THE can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THE is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBOOK .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookBOOK, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, EBOOK, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THE Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THE the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THE Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Scaling LoL Chat to 70M Players

  1. 1. SCALING LoL CHAT TO 70 MILLION PLAYERS Michal Ptaszek, @michalptaszek Riot Games
  2. 2. WHAT’S PLANNED 1 2 3 4 GAME CHAT TECH LESSONS LEARNED 5 Q&A
  3. 3. WHAT IS LEAGUE OF LEGENDS? 2009 LAUNCH TEAM ORIENTED 100+ CHAMPS MODERN FANTASY
  4. 4. MESSAGING SERVICE Private player chat and group chats. PRESENCE SERVICE Friend lists, availability and status. SOCIAL GRAPH SERVICE Internal service for store, match history, leagues. CHAT WHAT IS IT?
  5. 5. CHAT WHAT IS IT?
  6. 6. CHAT BY THE NUMBERS 67 million monthly players 27 million daily players 7.5 million concurrent players 1 billion events routed per server, per day
  7. 7. CHAT AT 10K FEET STABLE, SCALABLE CHAT SERVICE PROTOCOL DATA SERVER STORE
  8. 8. CHAT AT 10K FEET STABLE, SCALABLE CHAT SERVICE DATA PROTOCOL SERVER STORE
  9. 9. PROTOCOL: XMPP Decentralized Architecture Openness Extensibility Availability of Client Libraries Security Wide Adoption
  10. 10. CHAT AT 10K FEET STABLE, SCALABLE CHAT SERVICE DATA PROTOCOL SERVER STORE
  11. 11. SERVER: EJABBERD ‣ Open source Jabber/XMPP server ‣ Relatively nice scalability and performance with default configuration ‣ Wide adoption and active, helpful community ‣ Very good as a starting point for our own server solution ▾ We were aware that one day we would need to start customizing it ‣ Written in Erlang programming language
  12. 12. TECHNOLOGY: ERLANG/OTP Erlang is... Which gives us... A functional language Built with concurrency and distribution in mind Able to scale extremely well Capable of reloading code on the fly A declarative style of programming An easier way to build our distributed applications More time to focus on coding Less downtime
  13. 13. SERVER: EJABBERD - PHILOSOPHY Share nothing approach; enables massive, near linear horizontal scalability. ARCHITECTURE Implementation of self-healing properties, which bring the system to a well-known, stable state. FAULT TOLERANCE When something is massively broken - do not fix it! LET IT CRASH
  14. 14. SERVER: EJABBERD - ARCHITECTURE ETL Queries Secondary Riak Cluster External Traffic (5223) Internal Traffic Riak Riak Ejabberd Server Ejabberd LB Server
  15. 15. SERVER: EJABBERD - IMPLEMENTATION PHASE 1 - MAKE IT WORK ‣ Over time mostly rewritten ‣ Removed unwanted and unneeded parts ‣ Optimized certain flow paths ‣ Make it compatible with industry standards ‣ Wrote over 600 tests to cover it Invite Alice Bob Accept Alice Bob Invite Alice Bob Accept Alice Bob Alice Bob
  16. 16. SERVER: EJABBERD - IMPLEMENTATION PHASE 1 - MAKE IT WORK ‣ Over time mostly rewritten ‣ Removed unwanted and unneeded parts ‣ Optimized certain flow paths ‣ Make it compatible with industry standards ‣ Wrote over 600 tests to cover it Invite Alice Bob Accept Alice Bob Alice Bob
  17. 17. SERVER: EJABBERD - IMPLEMENTATION PHASE 2: MAKE IT RIGHT ‣ Removed clear bottlenecks ‣ Avoid shared, mutable state ‣ “Make it work, make it right, make it fast” MUC router user sesussioenr sesussioenr session MUC room user sesussioenr sesussioenr session user sesussioenr sesussioenr session MUC room MUC room
  18. 18. SERVER: EJABBERD - IMPLEMENTATION PHASE 2: MAKE IT RIGHT ‣ Removed clear bottlenecks ‣ Avoid shared, mutable state ‣ “Make it work, make it right, make it fast” user sesussioenr sesussioenr session MUC room user sesussioenr sesussioenr session user sesussioenr sesussioenr session MUC room MUC room
  19. 19. SERVER: EJABBERD - IMPLEMENTATION PHASE 2: MAKE IT RIGHT ‣ Removed clear bottlenecks ‣ Avoid shared, mutable state ‣ “Make it work, make it right, make it fast” Session Table: JID -> Session Handler session table Alice Bob Charlie
  20. 20. SERVER: EJABBERD - IMPLEMENTATION PHASE 3 - MAKE IT FAST ‣ Patched VM and stdlibs ‣ Sacrificing generic nature of Erlang/OTP framework in favor of better scalability and fault tolerance ‣ Better traceability and profiling functions ‣ More visibility into the system ‣ Improved logging for code reloading and real time system upgrades
  21. 21. CHAT AT 10K FEET STABLE, SCALABLE CHAT SERVICE PROTOCOL SERVER DATA STORE
  22. 22. NOSQL DATA STORE: RIAK SCALE Linearly scalable No growth headaches FAULT Higher TOLERANCE No SPoF uptime SCHEMA-LESS Faster feature iterations More shipped features ‣ Distributed, fault-tolerant, key-value store ‣ Masterless, fully peer-to-peer architecture ‣ AP in CAP theorem, with eventual consistency ‣ Low, predictable latency ‣ Extreme scalability ‣ Multi data center replication
  23. 23. LESSONS LEARNED UNDERSTAND YOUR SYSTEM ‣ Over 500 real-time counters, rates, histograms collected each minute ‣ Make sure to know counter values for “correct” and “abnormal” conditions ‣ Alerts and logs for long running operations ‣ Integration with Graphite, Zabbix and Nagios
  24. 24. IMPLEMENT FEATURE TOGGLES LESSONS LEARNED ‣ Safety valve for things that might cause problems ‣ Partial deployments allowing features to be enabled only for certain groups of people Alice Bob Charlie group reordering feature whitelist: Bob Bob
  25. 25. SUPPORT CODE RELOADING ‣ Patching bugs on the fly ‣ Changing server configuration ‣ Collecting data for future analysis ‣ No downtime deploys LESSONS LEARNED buggy code fixed code server restart buggy code fixed code
  26. 26. GET YOUR LOGGING RIGHT LESSONS LEARNED ‣ Proper logging and tracing facilities ‣ Debug modes for selected users ‣ Tools for analysis of the collected data Alice ejabberd.log slow_db.log trace_alice.log roster_audit.log muc_audit.log Honu
  27. 27. ALWAYS LOAD TEST YOUR CODE ‣ Automatic verification of the latest builds ‣ Collecting historical results for comparison ‣ Measuring the impact of new features and changes to the code ‣ Simulating various failures LESSONS LEARNED
  28. 28. THINGS WILL FAIL LESSONS LEARNED ‣ Prepare for the worst ‣ It’s just a matter of time for crash to happen ‣ It’s not only our code that fails ‣ Unlikely events happen every second under given scale
  29. 29. CHAT IS DOING GREAT! The quality uptime is over 99% each month, and is increasing, with hundreds of servers deployed all over the world. SCALE AND PERFORMANCE Each server offer reliable, low latency to the players, routing over 1B events a day with low resource utilization. CHAT IS EVOLVING Rolling out Riak worldwide, making LoL Chat available outside of the client, explore possibilities around using social graph data, and more... CURRENT SITUATION
  30. 30. THANK YOU! ANY QUESTIONS?

×