Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Video Games at Scale: Improving the gaming experience with Apache Spark

2,501 views

Published on

Spark Summit 2016 talk by Colin Borys (Riot Games) and Xiaoyang Yang (Riot Games)

Published in: Data & Analytics
  • Be the first to comment

Video Games at Scale: Improving the gaming experience with Apache Spark

  1. 1. VIDEO GAMES AT SCALE Improving the gaming experience with Spark
  2. 2. Choose From over 120 champions, each having a unique backstory and abilities Compete With your team to complete objectives and battle the enemy team. Win Take down defenses and destroy the enemy nexus
  3. 3. What can data tell us? How do we interact with these data? WHY SPARK? USE CASES FUTUREDATA@RIOTAGENDA Data
  4. 4. Game Balance How is Draven’s damage output since the patch?
  5. 5. Network Performance Player noted lag during gameplay. Is his ISP having trouble?
  6. 6. Customization Are there customizations similar to Star Guardian Lux this player may enjoy?
  7. 7. Players & Data W O R L D W I D E Statistics released Jan 2014 67+ million monthly active players 500+ billion data points per day 26 petabytes data collected since beta
  8. 8. THE DATA SCIENCE TOOLKIT ● Scheduled ETLs ● Ad Hoc Queries ● Data pulls ● Recent ● 106 events/sec ● Monitoring ● Fraud/Anomaly ● Desktops ● R/Python ● Tableau
  9. 9. THE DATA SCIENCE TOOLKIT ● Scheduled ETLs ● Ad Hoc Queries ● Data pulls ● Recent ● 106 events/sec ● Monitoring ● Fraud/Anomaly ● Desktops ● R/Python ● Tableau
  10. 10. Our data and ecosystem are scaling fine, but our analytic tools were not.
  11. 11. WHY SPARK? USE CASES FUTUREDATA@RIOTAGENDA Data Science At Scale
  12. 12. Why ?
  13. 13. Spark SQL
  14. 14. Spark Streaming Spark SQL
  15. 15. Spark Streaming Spark ML Spark SQL
  16. 16. Spark Streaming Spark ML Spark SQL Empowerment
  17. 17. WHY SPARK? USE CASES FUTUREDATA@RIOTAGENDA How did we apply this?
  18. 18. 1How to Play (with data) SparkSQL
  19. 19. Iterative query development Pain PointsFOR DATA EXPLORATION + REPORTING
  20. 20. Iterative query development Data pulls are slow Pain PointsFOR DATA EXPLORATION + REPORTING
  21. 21. Workflow is inefficient. Data is separated from tools.
  22. 22. Integrated Interactive Faster Data, easier.
  23. 23. Head-to-head tests between EMR and SparkSQL . (equal cost clusters) Minutes
  24. 24. Data exploration efficiency increased Performant SQL queries Self managed cluster deploys Managed/SparkSQL TAKEAWAY
  25. 25. 2 Winning the war on lag With Spark Streaming
  26. 26. Gameplay highly dependent on network connection.
  27. 27. The existing network limits our players.
  28. 28. So we built our own!
  29. 29. Riot Direct
  30. 30. Peering
  31. 31. But we need to find them! We have this awesome tool to fix problems.
  32. 32. Scope of Data W O R L D W I D E 17,000+ Unique ISPs 171,000+ City/ISP combinations 250,000+ Network stat messages per second
  33. 33. Model this PlayersLatency Days
  34. 34. Model this ...so we can detect this Days PlayersLatency
  35. 35. Model this ...so we can detect this ...or this. Days PlayersLatency
  36. 36. Model Building/ Evaluation HIVE (stores aggregated data) Kafka Consume/ Aggregate Alerts Spark Elasticsearch Dashboards
  37. 37. Works Intuitive Will be a lot easier in 2.0! Spark Streaming TAKEAWAY
  38. 38. 3The Recommendation System Secret Store
  39. 39. Why personalized recommendations?
  40. 40. Why personalized recommendations?
  41. 41. Why personalized recommendations?
  42. 42. 67+ millionmonthly active players 500+ billiondata points per day ~1000champion/skin combo Players Champ/Skins Champ/Skins Players
  43. 43. Modeling/ Evaluation HIVE Explore/ Feature engineering Recommendation Game Server Data Feature SparkSQL MLlib Spark
  44. 44. Fast prototyping Works on big matrix! Easy automation Recommendation System TAKEAWAY
  45. 45. WHY SPARK? USE CASES FUTUREDATA@RIOTAGENDA Now what?
  46. 46. With Spark, we think that we’re just getting started.
  47. 47. Wanna learn more? (we’re hiring!) Colin Borys CBORYS @ RIOTGAMES.COM Xiaoyang Yang XYANG @ RIOTGAMES.COM

×