Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DeNA West & BigQuery

2,026 views

Published on

Introduction of BigQuery at DeNA West

Published in: Internet
  • Be the first to comment

DeNA West & BigQuery

  1. 1. DeNA West & BigQuery Yoshiki Izawa
  2. 2. Who is DeNA? Japan China West
  3. 3. Who is DeNA West? • 1st Party: developed in house • 2nd-3rd Party: Developed externally with/without our help and published by DeNA • JP 1st Party: Import hits made by DeNA in Japan
  4. 4. What data do we care about? Standard Game KPIs Marketing Data Custom Game Insight
  5. 5. What’s our data like? of raw logs per minute ~60MB of raw logs per day ~50GB
  6. 6. How were we dealing with it? Log Table Logs from over 100 titles and multiple studios 42TB raw log data from May 2011 to Jan 2015 Accessed via HiveQL
  7. 7. What were our data woes? Data sources ETL Table Storage Data access for ad hoc analysis Visualization solutions Visualization workflows
  8. 8. Issue #1 - 3 Hour Data Delay Production Collector (add rats, scrip) Amazon S3 (for backup) ETL (Validation, Normalization) PS Logs GS Logs GC/PC Logs
  9. 9. Issue #2 - Too Many Cooks Hadoop Marketing Finance Game Analysts User Retention Clipart credit: iconka.com, webdesignhot.com
  10. 10. Issue #3 - Slow Queries
  11. 11. What was our solution? Simplify common tech using AppEngine and BigQuery
  12. 12. Solution #1 - BigQuery Instantly Ingestion Google API Ad Data PS / PC Logs GC/GC Logs PF Data
  13. 13. Solution #1 - BigQuery Instantly Ingestion (original) Gateway module PS / PC Logs GC/GC Logs Pull Task Queue Game Project Common Project App Engine Big Query streaming insert Game ProjectGame ProjectGame Project
  14. 14. Solution #1 - BigQuery Instantly Ingestion (current) Gateway module PS / PC Logs GC/GC Logs Cloud Storage Game Project Common Project App Engine Big Query Game ProjectGame ProjectGame Project Log Common Project TEMP dataset batch load every 3min create log timestamp base files by 3min cron job bq query every 3min
  15. 15. Solution #2 - BigQuery Scaling Scaling + support = happier analysts! Wh ya gonna call?
  16. 16. Solution #2 - BigQuery Permissions
  17. 17. Solution #3 - BigQuery is fast
  18. 18. Lessons Learned • Quotas • Different cost structure • New(ish) Product - some features may not work as expected
  19. 19. Is BigQuery for you? • Cutting edge tech • High control of your analysis • Zero maintenance
  20. 20. Let’s see some data!
  21. 21. Let’s see some data! Dropped entry price, increased conversion Reduced overall revenue at start of funnel but increase overall
  22. 22. Let’s see some data! Ruby Medal Balance over Time Tutorial Completion RateTCP Latency Distribution

×