0
evaluating long-term
audience effect
how to know the impact of changes
on audience reach
The speakers
Dmitry Kharitonov

Digital Director, RBC

Andrey Sverdlov

Digital Analytics
Consultant, AT Internet
what’s the problem?
The problem
• You’re a digital publisher having earnings
from advertising.
• You’re addicted to audience volume in terms
o...
How can you be sure?
• if you change your site will it affect the
audience metrics and how?
How can you trust the data?
• if the difference is based on product change
or just noise?
Common approach – A/B test
There’s number of ways to test direct response
as page views per visit, CTR, clicks and other
a...
towards the solution
with Andrey
Audience-centric approach in testing
Key idea: look at development of daily share of
returning visitors.
Spoiler: share of...
Idea background
Day

Audience A total

Audience B total

new

new

returning

9 500

returning

Share of B
new

500

retur...
Idea background
Day

Audience A total

Audience B total

new

new

returning

9 500

returning

Share of B
new

500

retur...
mathematics behind
you can’t trust the numbers without it
Fundamentals
Share of B is a random value.
Share_of_B = B / (B + A) / Base_proportion – 100%
Example:
Day 1
B = 500
B+A = ...
Fundamentals
Share of B is a random number.
and We

don’t know it’s distribution function.
Still we can apply methods of s...
Hypothesis
Zero hypothesis: tested option B is the same as
A = no effect on reach.
Right-hand alternative: option B attrac...
Metric to test hypothesis
Daily share of visitors that have seen option B
10,0%
8,0%
6,0%
4,0%
2,0%
0,0%
1

3

5

7

9 11 ...
Hypothesis in terms of metric
Zero hypothesis: proportion of A and B has not
changed = no effect on reach.
Right-hand hypo...
What we are testing now?
We’re testing A and B against 1 number –
share of B.
We can make decision based on this number
co...
we’ve got data, so what?
Can we trust the numbers?
Ok, now we know B is X% better than A.
But can we trust this value?
We need to prove statistical...
Statistical significance proof
Build sample distribution.
It’s very much like normal distribution.
If result is out of 3*s...
Statistical significance proof: bonus
Empirical Rambler values:
significant results start from 2-3% in audience
reach.
Dis...
Technical means
What we used
A/B test – nginx w/testing module
new visitors have constant proportion
returning visitors see what they have...
Conclusion
Pros&Cons
+ can understand an impact on an audience reach
+ can set up GO / NO GO constraints for changes
+ can mix audien...
Outcome








Evaluation of immediate effect is not enough for
publishers.
Traditional test metrics (page views, cli...
Example of metrics mix
Audience reach and Impressions
10,0%
8,0%
6,0%
4,0%
2,0%

0,0%
1

3

5

7

9 11 13 15 17 19 21 23 2...
Q&A
Dmitry, Andrey
Upcoming SlideShare
Loading in...5
×

How to know the impact of changes on audience reach - User and partner conference 2013

654

Published on

Evaluating long-term audience effect - how to know the impact of changes on audience reach

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
654
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "How to know the impact of changes on audience reach - User and partner conference 2013"

  1. 1. evaluating long-term audience effect how to know the impact of changes on audience reach
  2. 2. The speakers Dmitry Kharitonov Digital Director, RBC Andrey Sverdlov Digital Analytics Consultant, AT Internet
  3. 3. what’s the problem?
  4. 4. The problem • You’re a digital publisher having earnings from advertising. • You’re addicted to audience volume in terms of reach and page views. • One day you’re going to update the site significantly.
  5. 5. How can you be sure? • if you change your site will it affect the audience metrics and how?
  6. 6. How can you trust the data? • if the difference is based on product change or just noise?
  7. 7. Common approach – A/B test There’s number of ways to test direct response as page views per visit, CTR, clicks and other actions. We have a lack of tools to measure impact on audience reach.
  8. 8. towards the solution with Andrey
  9. 9. Audience-centric approach in testing Key idea: look at development of daily share of returning visitors. Spoiler: share of visitors of tested option B as a metric shows this development.
  10. 10. Idea background Day Audience A total Audience B total new new returning 9 500 returning Share of B new 500 returning 5.00% 1 1 200 8 300 63 9 300 437 4.98% 400 5.00% 4.12% 2 1 100 8 200 58 9 700 342 5.01% 700 4.00% 6.73% 3 1 500 8 200 79 621 5.00% 7.04% New visitors always have base proportion, returning shares play a key role.
  11. 11. Idea background Day Audience A total Audience B total new new returning 9 500 returning Share of B new 500 returning 5.00% 1 1 200 8 300 63 9 300 437 4.98% 400 5.00% 4.12% 2 1 100 8 200 58 9 700 342 5.01% 700 4.00% 6.73% 3 1 500 8 200 79 621 5.00% 7.04% New visitors always have base proportion, returning shares play key role.
  12. 12. mathematics behind you can’t trust the numbers without it
  13. 13. Fundamentals Share of B is a random value. Share_of_B = B / (B + A) / Base_proportion – 100% Example: Day 1 B = 500 B+A = 10 000 (total) Base proportion = 5% Share_of_B = 500 / 10 000 / 5% – 100% = 0% Day 2 B = 400 B+A = 9 700 (total) Base proportion = 5% Share_of_B = 400/ 9 700 / 5% – 100% = -17,6%
  14. 14. Fundamentals Share of B is a random number. and We don’t know it’s distribution function. Still we can apply methods of statistical analysis.
  15. 15. Hypothesis Zero hypothesis: tested option B is the same as A = no effect on reach. Right-hand alternative: option B attracts more returning visitors than A = B is better than A.
  16. 16. Metric to test hypothesis Daily share of visitors that have seen option B 10,0% 8,0% 6,0% 4,0% 2,0% 0,0% 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 -2,0% -4,0% Audience 7 Moy. mobile sur pér. (Audience)
  17. 17. Hypothesis in terms of metric Zero hypothesis: proportion of A and B has not changed = no effect on reach. Right-hand hypothesis: proportion has skewed to option B = B is better than A.
  18. 18. What we are testing now? We’re testing A and B against 1 number – share of B. We can make decision based on this number combined with range of other metrics: page views number, clicks on links, revenue, CTR, etc.
  19. 19. we’ve got data, so what?
  20. 20. Can we trust the numbers? Ok, now we know B is X% better than A. But can we trust this value? We need to prove statistical significance of result.
  21. 21. Statistical significance proof Build sample distribution. It’s very much like normal distribution. If result is out of 3*sigma interval it is significant. Otherwise it’s just noise. We can also use Student’s T-test.
  22. 22. Statistical significance proof: bonus Empirical Rambler values: significant results start from 2-3% in audience reach. Disclaimer: values may vary for your sites. It is not possible to calculate these border values with only AT Internet tools.
  23. 23. Technical means
  24. 24. What we used A/B test – nginx w/testing module new visitors have constant proportion returning visitors see what they have seen first time “murmurhash32” algorithm Measurement – ATI Multivariate Testing tag Proof – own server logs and processing
  25. 25. Conclusion
  26. 26. Pros&Cons + can understand an impact on an audience reach + can set up GO / NO GO constraints for changes + can mix audience and direct response metrics in tests - no out-of-the-box tool - complex computations - mathematics involved
  27. 27. Outcome     Evaluation of immediate effect is not enough for publishers. Traditional test metrics (page views, clicks, CTR, etc.) could be combined with audience metrics. It makes possible to evaluate test effect on audience reach in mid-term. Not the easiest approach, still very useful.
  28. 28. Example of metrics mix Audience reach and Impressions 10,0% 8,0% 6,0% 4,0% 2,0% 0,0% 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 -2,0% -4,0% -6,0% -8,0% -10,0% Audience Impressions 7 Moy. mobile sur pér. (Audience) 7 Moy. mobile sur pér. (Impressions)
  29. 29. Q&A Dmitry, Andrey
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×