Holding Effective Data Meetings
Ben Weber
Director of BI & Analytics
April 30, 2015
Analytics Summit Goals
Learn how to build an effective data team
Discover new techniques and tools for mining data
insights
Learn how to effectively interact with game teams to
act on data insights
Questions
What data to share with game teams?
When and how to share data?
How to operationalize your data?
Barriers to Using Data
Little communication between the data and game teams
Automated reports not providing actionable data
Data team is not working on actionable issues
Team structure separates game developers from analysts
Daybreak’s Approach
Integrate analysts into game teams
Develop consistent reporting across the portfolio and
teach teams how to use the data
Frequently meet with game teams to identify areas of
opportunity
Overview
Team Structure
Sharing Data
Meetings
Acting on Data
BI Team Structure
Centralized Roles
– Director of BI & Analytics
– Data Architect
– Ecommerce Analyst
Embedded Analysts
– H1Z1
– PlanetSide 2
– DC Universe Online
Database Operations
Centralized Analyst Role
Performs data analysis across the portfolio
Supports the marketing and exec teams by building
portfolio-wide KPIs and reports
Types of Analysis
– User acquisition funnels
– Email campaign ROI
– Ad-hoc analysis for centralized teams
Embedded Analyst Role
Co-located and considered part of the game team
Supports the producer and creative director with game
design and marketplace insights
Types of Analysis
– Game balance
– In-game retention funnels
– Ad-hoc analysis for the game team
What Data to Share?
All game-specific KPIs are provided to all members of
the team
Ad-hoc analysis is first shared with the leads and
communicated to relevant team members
Portfolio KPIs are shared across the company
Reports
Automated Reports
– Daily KPIs
– Hourly KPIs
– Monthly KPIs
– Self-Service Portal
Data Pipeline
– Vertica, Tableau Server, and Excel
Meetings
Analytics 101
Data Scrum Meetings
Data Insights Meetings
Monthly Business Reviews
Analytics 101
Goal
– Explain our automated reports to game teams
Attendees
– Everyone!
Agenda
– Define all of those fun acronyms
– Provide context for each of the metrics being tracked
– Team members ask questions and provide feedback
Data Scrum Meetings
Goal
– Meet with game team weekly to track KPIs
Attendees
– Game team leads, brand manager, eComm team
Agenda
– Discuss KPI trends and targets
– Identify ad-hoc analysis to perform
– Evaluate the impact of promotional events
Data Insights Meetings
Goal
– Share insights from portfolio or game-specific analysis
– Scheduled as necessary
– Example: item drop rates in a game are too high
Attendees
– Game team leads
Agenda
– Explain the results of the analysis
– Indentify how to respond to the findings
Monthly Business Reviews
Goal
– Monthly meeting to review game performance
Attendees
– Senior management and game team leads
Agenda
– Review KPIs for the past and current months
– Present substantial data insights
– Try to get executives to become data evangelists
Acting on Data Insights
How can we get game teams to act on data insights?
Interaction Approaches
– Hand-off the results
– Own a data model
– Share a data model
– Show ROI results
The Hand-Off Approach
Overview
– The data team performs an analysis and hands-off the results to
the game team
– The game team decides how to use the data
Examples
– DCUO Character Traits
– PlanetSide 2 Starter bundles
– PlanetSide 2 Implant Drops
Improving Hand-Offs
Hand-offs can be improved by incorporating the game team
earlier and following up
Approach
– Share preliminary data insights
– Schedule a “Data Insights” meeting
– Perform additional analysis
– Deploy the change
– Schedule a follow up “Data Insights” meeting
Owning a Data Model
Overview
– The data team develops a model that provides input to an
in-game mechanism
– On a daily or weekly basis, the game team ingests a targeted
user list
Examples
– DCUO Nudge System
– PlanetSide 2 Tutorial Targeting
Improving Owned Models
Issues
– Manual data hand-off
– Multiple failure points
– Model is a black box to the game team
Recommendation
– Automate as much of the process as possible and allow
game teams to run competing approaches
Sharing a Data Model
Overview
– The data team builds and validates the models
– The game team implements the models in-game
Examples
– EverQuest Landmark Recommendation System
– PS2 Nudge System
Landmark’s Recommendation System
Improving Shared Models
Work with the game team to make lots of parameters
available for the models
Implement fall-back approaches in case the models fail
Use a modular approach for implementing the model
rules and parameters
Showing ROI Results
Overview
– The data team performs analysis that does not require work
from the game team
– The results are shared with the game team and used to plan for
future campaigns
Examples
– A/B testing email campaigns
– Advertising on PSN
Summary
Meeting Types
– Analytics 101
– Data Scrum Meetings
– Data Insights Meetings
– Monthly Business Reviews
Interaction Approaches
– Hand-off the results
– Own a data model
– Share a data model
– Show ROI results
Questions
What data to share with game teams?
– All the KPIs and more detailed analysis as needed
When and how to share data?
– Regularly through automated reports, scrum meetings, and data
insights meetings
How to operationalize your data?
– Meet early with preliminary data, develop an action plan, and follow
up with more data
Thank You
Ben Weber
Director of BI & Analytics
Daybreak Game Company
DaybreakGames.com
@bgweber
Holding Effective Data Meetings
Ben Weber
Daybreak Game Company

Holding Effective Data Meetings

  • 1.
    Holding Effective DataMeetings Ben Weber Director of BI & Analytics April 30, 2015
  • 2.
    Analytics Summit Goals Learnhow to build an effective data team Discover new techniques and tools for mining data insights Learn how to effectively interact with game teams to act on data insights
  • 3.
    Questions What data toshare with game teams? When and how to share data? How to operationalize your data?
  • 4.
    Barriers to UsingData Little communication between the data and game teams Automated reports not providing actionable data Data team is not working on actionable issues Team structure separates game developers from analysts
  • 5.
    Daybreak’s Approach Integrate analystsinto game teams Develop consistent reporting across the portfolio and teach teams how to use the data Frequently meet with game teams to identify areas of opportunity
  • 6.
  • 7.
    BI Team Structure CentralizedRoles – Director of BI & Analytics – Data Architect – Ecommerce Analyst Embedded Analysts – H1Z1 – PlanetSide 2 – DC Universe Online Database Operations
  • 8.
    Centralized Analyst Role Performsdata analysis across the portfolio Supports the marketing and exec teams by building portfolio-wide KPIs and reports Types of Analysis – User acquisition funnels – Email campaign ROI – Ad-hoc analysis for centralized teams
  • 9.
    Embedded Analyst Role Co-locatedand considered part of the game team Supports the producer and creative director with game design and marketplace insights Types of Analysis – Game balance – In-game retention funnels – Ad-hoc analysis for the game team
  • 10.
    What Data toShare? All game-specific KPIs are provided to all members of the team Ad-hoc analysis is first shared with the leads and communicated to relevant team members Portfolio KPIs are shared across the company
  • 11.
    Reports Automated Reports – DailyKPIs – Hourly KPIs – Monthly KPIs – Self-Service Portal Data Pipeline – Vertica, Tableau Server, and Excel
  • 12.
    Meetings Analytics 101 Data ScrumMeetings Data Insights Meetings Monthly Business Reviews
  • 13.
    Analytics 101 Goal – Explainour automated reports to game teams Attendees – Everyone! Agenda – Define all of those fun acronyms – Provide context for each of the metrics being tracked – Team members ask questions and provide feedback
  • 14.
    Data Scrum Meetings Goal –Meet with game team weekly to track KPIs Attendees – Game team leads, brand manager, eComm team Agenda – Discuss KPI trends and targets – Identify ad-hoc analysis to perform – Evaluate the impact of promotional events
  • 15.
    Data Insights Meetings Goal –Share insights from portfolio or game-specific analysis – Scheduled as necessary – Example: item drop rates in a game are too high Attendees – Game team leads Agenda – Explain the results of the analysis – Indentify how to respond to the findings
  • 16.
    Monthly Business Reviews Goal –Monthly meeting to review game performance Attendees – Senior management and game team leads Agenda – Review KPIs for the past and current months – Present substantial data insights – Try to get executives to become data evangelists
  • 17.
    Acting on DataInsights How can we get game teams to act on data insights? Interaction Approaches – Hand-off the results – Own a data model – Share a data model – Show ROI results
  • 18.
    The Hand-Off Approach Overview –The data team performs an analysis and hands-off the results to the game team – The game team decides how to use the data Examples – DCUO Character Traits – PlanetSide 2 Starter bundles – PlanetSide 2 Implant Drops
  • 19.
    Improving Hand-Offs Hand-offs canbe improved by incorporating the game team earlier and following up Approach – Share preliminary data insights – Schedule a “Data Insights” meeting – Perform additional analysis – Deploy the change – Schedule a follow up “Data Insights” meeting
  • 20.
    Owning a DataModel Overview – The data team develops a model that provides input to an in-game mechanism – On a daily or weekly basis, the game team ingests a targeted user list Examples – DCUO Nudge System – PlanetSide 2 Tutorial Targeting
  • 21.
    Improving Owned Models Issues –Manual data hand-off – Multiple failure points – Model is a black box to the game team Recommendation – Automate as much of the process as possible and allow game teams to run competing approaches
  • 22.
    Sharing a DataModel Overview – The data team builds and validates the models – The game team implements the models in-game Examples – EverQuest Landmark Recommendation System – PS2 Nudge System
  • 23.
  • 24.
    Improving Shared Models Workwith the game team to make lots of parameters available for the models Implement fall-back approaches in case the models fail Use a modular approach for implementing the model rules and parameters
  • 25.
    Showing ROI Results Overview –The data team performs analysis that does not require work from the game team – The results are shared with the game team and used to plan for future campaigns Examples – A/B testing email campaigns – Advertising on PSN
  • 26.
    Summary Meeting Types – Analytics101 – Data Scrum Meetings – Data Insights Meetings – Monthly Business Reviews Interaction Approaches – Hand-off the results – Own a data model – Share a data model – Show ROI results
  • 27.
    Questions What data toshare with game teams? – All the KPIs and more detailed analysis as needed When and how to share data? – Regularly through automated reports, scrum meetings, and data insights meetings How to operationalize your data? – Meet early with preliminary data, develop an action plan, and follow up with more data
  • 28.
    Thank You Ben Weber Directorof BI & Analytics Daybreak Game Company DaybreakGames.com @bgweber
  • 29.
    Holding Effective DataMeetings Ben Weber Daybreak Game Company