Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging


Published on

How do you balance the dynamic needs of millions of concurrent users, support hundreds of developers, optimize a distributed cloud environment, push code releases daily, and stay sane? It starts with visibility into all your data and real-time insight from your log files.

In this session at the 2014 Game Developers Conference, Albert Ho provided a step-by-step outline of how Rumble Entertainment maximizes revenue in online games by:

Instrumenting its games and platform to facilitate global troubleshooting
Gathering insight from log files to identify, quantify and solve operational issues
Conducting transactional tracing starting from player feedback on open social networks to isolate the source, server and all users affected by the issue for remediation
Baking Loggly into the DNA of its games to serve as a single source of truth and extended it to how we work with partners
Correlating real-time visibility directly to revenue to speed decision making and planning

Published in: Software, Technology, Business
  • Be the first to comment

  • Be the first to like this

Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging

  1. 1. Maximizing Revenue Through Logging Albert Ho
  2. 2. About Rumble Funded by: Mission: ●  Bring AAA quality to F2P ●  Play together cross-device/platform
  3. 3. Rumble’s Platform ●  Played in >200 countries ●  Six data centers worldwide ●  Hybrid infrastructure ●  Play together on:
  4. 4. What this talk is about… User Acquisition Funnel Loss Retention Loss Your Game Paid Users Free Users
  5. 5. “Logging” Distinctions ●  User behavior ●  Game design, retention, virality, monetization ●  System behavior ●  Maximize potential ●  Lower CPA
  6. 6. Thesis “You really aren’t maximizing revenue unless you can measure system behavior”
  7. 7. Challenges ●  24/7 operations ●  Game teams ship code daily ●  Being “up” is not good enough
  8. 8. The Long Tail of Revenue Lost: Daily Site Problems Affect Weeks of Revenue Outage Problem Recovery New Installs/Sales Revenue Days
  9. 9. How to Detect Problems ●  Solving is Harder than it sounds Signal to noise ratio + Indexing and search + Multiple game teams shipping code daily = BIG Headache and world’s biggest chore The answers are in the logs!
  10. 10. Rumble Logging 1.0 ●  Graylog, Elasticsearch ftw? ●  1 day to setup ●  Collecting for 2 game servers ●  Few hundred MB a day / 7-day retention time ●  Scaling is easy… right? - Bigger hard disks, more ES nodes
  11. 11. Rumble Logging 1.5 ●  7 days to setup ES cluster ●  > 20 servers ●  Logs @ TB a day / 3-day retention time ●  Log noise incomprehensible ●  Very expensive!!! ●  Logging cluster broke every day
  12. 12. Key Learnings Log Management is really… “Undifferentiated Heavy Lifting” We were spending more on LOGGING than HOSTING There must be an expert that does it better, faster, and cheaper
  13. 13. Rumble Logging 2.0 Requirements ●  Must support JSON ●  Easy to integrate ●  Can scale up and meet retention time goals ●  Cost-effective
  14. 14. Enter……. ●  No more agents! ●  REST API – 21st century please/no syslogs ●  Scalable and Responsive ●  Superb partner relationship
  15. 15. Sample Log
  16. 16. Loggly Use Cases ●  Log clean-up ●  Release monitoring and debugging ●  Customer service requests ●  Trend analysis ●  3rd-party developers
  17. 17. Log Clean-up ●  1000’s errors per minute noise ●  DEBUG, INFO, WARN, ERROR? ●  Reduce TB’s of logs to GB’s
  18. 18. Release Monitoring ●  Log snapshots before and after
  19. 19. Release Monitoring ●  Game teams own process ●  All logs in one place ●  Easy for PM’s
  20. 20. Customer Service Reactive problem solving ●  “Why is customer x slow?” ●  “Why did I not get my gems?” ! Proactive debugging ●  “These 20 customers were affected by today’s issue.”
  21. 21. Trend Analysis Game performance analysis over time Client Lag Days
  22. 22. 3rd Party Development ●  Bake logging into games at the beginning ●  Including 3rd party ones…
  23. 23. Loggly Live Demo
  24. 24. Loggly Before And After Before After Log Retention 3 days 15 days Log Availability 50% uptime >99.9% Debugging Speed Days Minutes Happiness Factor
  25. 25. Questions? KingsRoad Ballistic