Building a Data-Driven Culture by Olof Hoverfält discusses how to build a data-driven culture at Sanoma Games. Key points include: 1) Being data-driven requires an organization that supports lean development, a data-driven culture with accessible tools, shared goals, and management that fosters self-direction. 2) A data-driven culture is built through intrinsic motivation by making the benefits of data visible, not through coercion. Transparency, autonomy and ownership are important. 3) Continuous hypothesis-driven testing should be the standard approach across functions to gain insights and steer development initiatives toward business goals.

1. Building a Data-Driven Culture
Olof Hoverfält
Director, Sanoma Games
September 2, 2015

2. @Hoverfalt

3. Sanoma Games is a diverse yet compact and lean unit
B2C & B2B, web & mobile Autonomous unit Key functions “in-house”
Business development
Marketing
Design
Development
Sales

4. DESIGN
BUSINESS ANALYTICS
UX 
Business

5. Being data-driven requires much more than the right tools
http://toolmonger.com/category/85/computer/page/2/
Handheld
USB Snake
Scope

6. ...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts

7. ...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts

8. http://theleanstartup.com/

9. http://www.startupolic.com/launching-your-service-think-about-ux/; https://en.wikipedia.org/wiki/Lean_startup
An MVP is “a version
of a new product
which allows a team
to collect the
maximum amount of
validated learning
about customers with
the least effort"
The MVP approach is useful at all levels of development, from
single product feature to new business concepts
Adopt MVP thinking in all development

10. http://www.yesware.com/blog/how-to-close-more-deals-with-lean-email/
A continuous Build-Measure-Learn cycle enables fast learning
but is really sensitive to friction in workflow and cooperation
Ensure all key functions are fully dedicated and the workflow is frictionless

11. ...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts

12. A data-driven culture cannot be created by coercion, but must
be built through intrinsic motivation from its visible benefits
Transparency Autonomy Ownership

13. xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
Improvement actions
xxx
Revenue
ROS&packag.
campaign vol.
Video
campaign vol.
Pricing
Discounts
Campaign
volume
Optimization
Direct
Referral
SEO
Ret. visitors
Potential
issues
Site problems
Ad blockers
New visitors
Reach
Monetization
Visits
Premium
fill rate
Premium
eRPM
Remnant
eCPM
Unique
visitors
Media sales breakdown by contributing component
Impression
success rate
nn%
nn%
nn% 
mm%
nn%
nn%
nn%
lost
Ensure overall understanding of how the business works and of the contribution of and the
dependencies between its components. Make deep data available
In order for a team to develop and fine tune an engine,
everyone must know how it works

14. Adoption of a data-driven approach requires for people to see
its benefits; how they could kick ass even more at what they do
Image: http://blog.crazyegg.com/lessons/show-benefits/
Make the personal benefits of being data-driven visible.
Show how its autonomous pursuit enhances mastery.
Super You!
Data-drivenness
You

15. Ownership
is not about doing what I tell you to do
is not about not doing what I tell you not to do
is about doing what I don’t tell you to do
Ensure ownership of benefits, not deliverables
Focus on why, not what

16. ...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts

17. Data-drivenness, at both individual and team level, is a
capability that can and should be developed
http://skepdic.com/tencommands.html
+
Encourage autonomous reflection but ensure conceptual guidance
Reflection Key concepts
Being data-driven is not only about
sensibly answering questions, but
about finding them in the first place
It cannot itself be standardized
into rules or processes, but it
can be guided by high-level key
concepts

18. A core principle of learning is to be hypothesis-driven. Random
experimentation will yield various outcomes but little insight
Make continuous hypothesis-driven split testing the standard approach in all development activities
across functions
Hypothesis:
[A statement that can
be shown true or false]
Design a test and a
metric that confirms
the hypothesis at a
certain pre set limit
Test in a real setting
with real usersPay attention also to
what is NOT
happening
Confirm or reject the
hypothesis based on
the chosen metric and
pre set limit
Add the learnings to
your body of
knowledge, also the
invalidated hypotheses
Build on the
knowledge to form the
next hypothesis

19. Data without insight can be destructive as it may lead
development the wrong way look like improvement
DAU
(daily active users)
Downloads
(i.e. not even activations)
Activity delta
(change in user engagement)
Churn
(users who quit)
>
Vanity metric
Indicator of
actual use
>Leading indicator of churn
Actionable
Lagging indicator
Not actionable
Put a lot of effort into designing KPIs (Key Performance Indicators) that are both sensible and
actionable; such that steer development initiatives toward your ultimate business goal

20. Main KPIs should be static but reflect the current growth phase,
while tactical tools can be whatever helps improve the KPIs
High-level (growth-stage-specific) KPIs to
continuously optimize
Tactical tools do explore user behavior and to
measure impact from change on KPI(s)
http://lanesplitter.jalopnik.com/x-ray-artist-reveals-the-inner-beauty-of-classic-motorc-1698564514

21. ...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts

22. A simple tool that is actually used provides infinitely more value
than a sophisticated one that is not
https://xplr.com/xplr-umbrella-dataviz-on-top-of-unsupervised-machine-learning/
>
Make sure everyone has easy access to all tools they might need and know how to use them

23. ...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts

24. Goals set at function level can often be unaligned or conflicting,
breaking joint effort and leading to sub-optimal performance
UX
Set high level common goals to ensure everyone works and cooperates to optimize the whole as
opposed to sub-optimizing their own components
Business Manager goal:
Visit eRPM
(effective revenue per session)
Product Manager goal:
Avg. DAU
(daily active users)

25. ...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts

26. ...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts

27. Example:
Continuous improvement and A/B testing

28. A/B testing is about continuous and systematic gradual
improvements that over time build increased performance
2.5x+
Improvement

29. Seemingly small but sustained improvements can have a
very large impact on overall performance
49%
Improvement
min to max
25%
Improvement
mean to max

31. Example:
Retention (how well you keep your users)

34. 6.1%
Retention D1

37. Count of users by level

38. Count of users by level

39. Count of users by level

40. 6.1%
Retention D1
47%
Retention D1
25%
Retention D7
6.2%
Retention D90
Compare with
6.1% D1 Retention
in the previous example

41. Example:
LTV ((Customer) Life Time Value)

42. User profitability is basically determined by LTV* and CAC*.
Being able to even roughly predict LTV can be very insightful.
KPIs
DAU
Monthly rev.
Retention D1 43 %
Retention D7 24 %
Retention D14 19 %
Retention D30 12 %
Conversion
ARPDAU
ARPPDAU
LTV
CAC
ARPDAU nn € nn €
Extr. tot. days (≤120th) nn nn
LTV nn € nn €
* LTV = (Customer) Life Time Value ; CAC = Customer Acquisition Cost

43. Example:
Liigapörssi (KPIs and tools)

46. Direct
Org. search
Email
SEM
Display
Other
Our main time unit of comparable comparison is a game
“phase”. The key desired user behavior varies within the phase
F
r
e
e
Trades
1
Pool
F
r
e
e
Trades
2
Pool
F
r
e
e
Trades
6
Pool
Direct
Org. search
Email
SEM
Display
Other
Pay
(Revenue)
Join pool
(Activation)
Sign up
(Acquisition)
Play the
whole phase
(Retention)
Continue to
next phase
(Retention)
Sign up or
reactivate
(Acquisition)
Desired user
behavior
Create pool
& invite
(Referral)
Invite friends
(Referral)
Make trades
(Activation)
Buy trades
(Revenue)
Create team
(Acquisition)
Phase 1 Phase 2 Phase 3-6
Season
Acquisition
Activation
Retention
Revenue
Referral
Acquisition
Referral
Revenue
Retention
Activation Focus area

47. We use various tools to identify development opportunities, run
tests and measure impact on the main high-level KPIs set
Area KPI
Acquisition
Signed up (%)
Created team (%)
CAC (€)
Activation
Active user (%)
In pool (%)
Retention
In-phase retention (%)
Phase change retention (%)
Revenue
LTV
ARPPU
Conversion
Referral Invited friend (%)
High-level (growth-stage-specific) KPIs to
continuously optimize
Tactical tools do explore user behavior and to
measure impact from change on KPI(s)
… …
Funnels
Emails
Anything!
Custom Dimensions
using hashed user
IDs enables user
level analysis, which
is VERY powerful

48. Some tools we are currently using
Tool Domain Sought benefits
Email Improve open rate and CTR
Email (transactional) Identify issues in use and funnels
Funnels Improve user onboarding and payments
Ecommerce Research revenue by product, user type, etc. (Enh. Ecom. Trac.)
Engagement Research anything at the user level!
Engagement Retention, User level progression
User Acquisition Volume and CPI (Cost Per Install)
Landing pages Optimize CTA (Call To Action) conversion
UI heat maps Research behavior and improve UI
KPI dashboard Communication and awareness

49. Some suggested further reading
* http://www.slideshare.net/dmc500hats/startup-metrics-for-pirates-long-version
Slideshare*

50. ...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts

51. @Hoverfalt