0
New Media Measurement
                  sean zehnder   socialesque 09




Wednesday, April 29, 2009                       ...
Outline
                            old-school Media
                            New-School Interactive
                  ...
old-school Media

                            How do they size...                  Why do they size...

                  ...
old-school Media
                            May 1996              May 1997                    Oct 1997                   ...
old-school Media

           Web 1.0
                      Most user’s didn’t have broadband
                      Flash e...
old-school Media
            Web 1.0 Measurement
                      Impressions
                      Clicks
          ...
old-school Media




                               page 61 of July 5, 1993 issue of The New Yorker, (Vol.69 (LXIX) no. 20...
New-School Interactive
                        April 2001              March 2004                    Feb 2005             ...
New-School Interactive
                               INTERACTIVE | SOCIAL | MOBILE | UBIQUITOUS | VIRAL | HYBRID




    ...
New-School Interactive
                               INTERACTIVE | SOCIAL | MOBILE | UBIQUITOUS | VIRAL | HYBRID
        ...
The Interactive Revolution
                                                                                     Let’s say ...
The Interactive Revolution
      A new online paper opens in your market...                                               ...
The Interactive Revolution
                                                                     So your competitor decides...
The Interactive Revolution
                                          More Reasons to Measure

              I measure...  ...
(How) are video games different?
                      1. Games are *highly* interactive                               2. ...
(How) are video games different?
                                                                       4. Games require m...
(How) are video games different?
                                 5. Many behaviors are emergent from this complex system....
(How) are video games different?
                                         Eve Online




         www.socialesque.com     ...
(How) are video games different?
                                         Eve Online




         www.socialesque.com     ...
(How) are video games different?
                                         Eve Online




         www.socialesque.com     ...
(How) are video games different?
                                         Eve Online




         www.socialesque.com     ...
(How) are video games different?
                                         Eve Online




         www.socialesque.com     ...
New-School Measurement

                                 Will Wright -- Creator of The Sims franchise

                   ...
New-School Measurement

                     Measurement at Massive Scale
                     Metrics that can’t often be...
New-School Measurement
                                     Socialesque’s “Rules of Engagement”




                      ...
New-School Measurement
                                       Case Study: Attrition in Social Network
                    ...
New-School Measurement
                                                                     Case Study: Attrition in Socia...
New-School Measurement
                                    Case Study: Attrition in Social Network

                      ...
New-School Measurement
                                    Case Study: Attrition in Social Network
            losing a fr...
New-School Measurement
                                  Case Study: Attrition in Social Network

                        ...
New-School Measurement
                                 Case Study: Avenue Q Facebook Application
                        ...
New-School Measurement
                                              Case Study: Navigation in an FPS
                    ...
New-School Measurement
                                          Case Study: Navigation + Survey Data in an FPS
          ...
Upcoming SlideShare
Loading in...5
×

Metrics and Analytics for new media

1,216

Published on

Evolution of metrics and analytics for new media and video games

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,216
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
31
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "Metrics and Analytics for new media"

  1. 1. New Media Measurement sean zehnder socialesque 09 Wednesday, April 29, 2009 1
  2. 2. Outline old-school Media New-School Interactive The Interactive Revolution (How) are video games different? New‐School Measurement www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 2
  3. 3. old-school Media How do they size... Why do they size... Newspaper Audiences Radio Audiences TV Audiences What is the smallest unit of an ‘audience’ in each? What does each metric imply about the behavior of the audience member -- meaning what are they doing? www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 3
  4. 4. old-school Media May 1996 May 1997 Oct 1997 June 1998 Portals Virtual Community New Measurement New Busines Opportunities June 1999 July 1999 Sept 1999 Aug 1999 New Skills in Demand Disruptive Technologies Old Industries In Flux Dreaming of Mobile www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 4
  5. 5. old-school Media Web 1.0 Most user’s didn’t have broadband Flash existed, but it was mostly for animations There was dynamic content, but it required a lot of custom scripting and code to do relatively simple things Discourse around offline vs. online www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 5
  6. 6. old-school Media Web 1.0 Measurement Impressions Clicks (later) Time on Screen Google Analytics “breakthrough” -- finally it is easy to gather and chart basic stats Data was primarily HTTP/Referrer, Cookie, and JavaScript based www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 6
  7. 7. old-school Media page 61 of July 5, 1993 issue of The New Yorker, (Vol.69 (LXIX) no. 20) www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 7
  8. 8. New-School Interactive April 2001 March 2004 Feb 2005 Nov 2005 Dealing w/ Massive Scale Wisdom of Crowds Open Source Mobile Revolution Dec 2006 July 2006 April 2006 Aug 2006 Social Networks Emerge Games Grow Up Cult of Personality User-Generated Content www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 8
  9. 9. New-School Interactive INTERACTIVE | SOCIAL | MOBILE | UBIQUITOUS | VIRAL | HYBRID www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 9
  10. 10. New-School Interactive INTERACTIVE | SOCIAL | MOBILE | UBIQUITOUS | VIRAL | HYBRID flash | ajax | rails | django | openid | opensocial ... www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 10
  11. 11. The Interactive Revolution Let’s say you have some data in an newspaper... “The Chronicle” 1.0 Unit Metric Offline Newspaper AVERAGE RAINFALL (INCHES/MONTH) A Sale # of Sales JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC San Fran 4.35 3.17 3.06 1.37 0.19 0.11 0.03 0.05 0.20 1.22 2.86 3.09 Seattle 5.35 4.03 3.77 2.51 1.84 1.59 0.85 1.22 1.94 3.25 5.65 6.00 Chicago 1.53 1.36 2.69 3.64 3.32 3.78 3.66 4.22 3.82 2.41 2.92 2.47 New York 3.17 3.02 3.59 3.90 3.80 3.65 3.80 3.41 3.30 2.88 3.65 3.42 Total Subscriptions A Subscription Miami 2.01 2.08 2.39 2.85 6.21 9.33 5.70 7.58 7.63 5.64 2.66 1.83 Total New Subscriptions You might consider sprucing it up to make it more readable... “The Chronicle” 2.0 Offline Newspaper Rationale Costs ROI Average Rainfall (inches/month) Jan Feb Mar Apr May June July Aug Sep Oct Nov Dec More Sales San Francisco 3.17 3.06 1.37 0.19 0.03 0.06 0.05 0.20 1.22 2.86 3.09 4.35 You believe readability Seattle 5.35 4.03 3.77 2.51 1.84 1.59 0.85 1.22 1.94 3.25 5.65 6.00 increases the attractiveness of Layout & Graphic Chicago 1.53 1.36 2.69 3.64 3.32 3.78 3.66 4.22 3.82 2.41 2.92 2.47 your paper, and attractiveness Design New York 3.17 3.02 3.59 3.90 3.80 3.65 3.80 3.41 3.30 2.88 3.65 4.42 More New will drive sales. Miami 2.01 2.08 2.39 2.85 6.21 9.33 5.70 7.58 7.63 5.64 2.66 1.83 Subscriptions www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 11
  12. 12. The Interactive Revolution A new online paper opens in your market... Unit Metric theHerald.com 1.0 A Sale # of Sales Online Newspaper A Subscription Total Subscriptions, Total New Subscriptions Average Rainfall (inches/month) 4+ 2-3 1-2 <1 An Impression Time On-Screen , # of Impressions Jan Feb Mar Apr May June July Aug Sep Oct Nov Dec San Francisco Rationale Costs ROI Seattle Layout & Graphic Chicago They can reach a new audience (e.g. Did we reach a Design New York City a younger one that gets their info younger demographic? Miami online). Can we learn more Website creation They hope to learn more about their about our readers ‘readers’ by using new metrics. now? Website hosting & support So you update your own look-and-feel... 3.0 “The Chronicle” Offline Newspaper Rationale Costs ROI Layout & Graphic Did we reach a younger Design You believe your readership is more demographic? interested in getting a sense of how the environment is changing, rather Are we keeping our Updating your than using raw rainfall data. subscribers or losing them printing press to to the competitor? do color graphics Your notion of an the ‘average reader’ demographic has changed. Note Your notion of what the ‘average reader’ is DOING has also changed. www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 12
  13. 13. The Interactive Revolution So your competitor decides to make their graphic interactive... theHerald.com 2.0 Unit Metric Online Newspaper A Sale # of Sales Average Rainfall (inches/month) Choose cities... A Subscription Total Subscriptions, Total New Subscriptions Jan Feb Mar Apr May June July Aug Sep Oct Nov Dec An Impression Time On-Screen , # of Impressions Seattle Chicago A Click Most Popular Cities New York A Drag Most Popular Months San Francisco An Error Number of times the city was changed A Behavior (generally) Number of Times Shared w/ a Friend Miami An Event (generally) Number of Times the Server Crashed Rationale Costs ROI Can we track how often they share Layout & Graphic this page with a friend? Design They believe that more interactive content is more engaging for their ‘readers.’ Can we track how long they typically spend with the content? Flash Designer / They believe that users will share the link to this Scripter website with friends if they find the content really Can we tell if they spend more time cool (or fun, or useful, etc.). with this content than they did DB Engineer reading our static table (from Hosting & Support previous iteration)? Take-Away: Behind Every Good Metric is a Model of Behavior www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 13
  14. 14. The Interactive Revolution More Reasons to Measure I measure... In order to... Time Assess Effectiveness of site on gaining and holding user’s attention Impressions System Behavior Save money. Stop bad behavior. Keep system out of user’s way. User Behavior Ensure they are having good experience. Identify new business opportunities. www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 14
  15. 15. (How) are video games different? 1. Games are *highly* interactive 2. Games involve simulations Dynamic Interactive Static Systems Model System Display Systems Systems 3. Game Simulations enable Designed Experiences Mechanics Dynamics Aesthetics Mechanics - logic and data that makes the system work (e.g. code) Dynamics - the rules of the game world (e.g. physics) Aesthetics - the game as experienced/perceived by the user www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 15
  16. 16. (How) are video games different? 4. Games require mastery Perceives the Plans Some Builds a Acts in the The Player (Agent)... World Action Theory World Refines their idea of the world... 5. Since key behaviors differ across games, standardization can be extremely difficult. Example: “Dead Reckoning” An expert behavior common in an FPS where a player moves in an arc relative to a target such that it makes it harder for the target to hit the player while the player is still able to fire on the target. www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 16
  17. 17. (How) are video games different? 5. Many behaviors are emergent from this complex system. For one, artificial intelligence is an increasingly important aspect of video games. It is being used to improve the ‘performances’ of the non- player characters, and to simulate complex dynamics in real-time such as crowd behavior. Also, massively multiplayer games have dramatically increased the SCALE of data needing to be measured and processed. In other words, behavior is emergent. www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 17
  18. 18. (How) are video games different? Eve Online www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 18
  19. 19. (How) are video games different? Eve Online www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 19
  20. 20. (How) are video games different? Eve Online www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 20
  21. 21. (How) are video games different? Eve Online www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 21
  22. 22. (How) are video games different? Eve Online www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 22
  23. 23. New-School Measurement Will Wright -- Creator of The Sims franchise “With Sims Online, we’re trying to take a lot of the community dynamics that we’ve learned from The Sims off-line and reinterpret them in an online world. And we study the online community all the time. It’s a very interesting community -- it’s over half female, which for an online game is totally different. And it turns out that a community of 55% females behaves very differently than a community that’s 75% males...In fact we’re capturing very detailed information. I can tell you how many people are kissing more today than they were yesterday, or how that’s correlated to other things.” (From Laurel, B. 2003, p. 255). www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 23
  24. 24. New-School Measurement Measurement at Massive Scale Metrics that can’t often be standardized New ways of thinking about engagement New reasons to measure Cross-Media Measurement www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 24
  25. 25. New-School Measurement Socialesque’s “Rules of Engagement” analyze model instrument refine What do you expect will be What might fun look like in your Did people do what you What can be done to improve the target engaging about your application? game? expected? behaviors? What might frustrate users? What might frustration look like? Were there things that you notice Can the model of engagement be refined in the data that you didn’t think to based on the new understanding? What do you hope they do while What measures might capture include in the model? they use the application? engagement? Which items and features were most Which measures are the strongest popular, and might some features need to What is practical to measure in your model? be removed or improved to enhance from this application? engagement? www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 25
  26. 26. New-School Measurement Case Study: Attrition in Social Network “A friend leaves the network” model expectations basic model Social measures will be most influential. Amount of Social Number of Friends Quality of Friendship Interaction + + + The effect will be cumulative: the more friends you have Game Satisfaction & Enjoyment lost, the more each loss is felt. - Will have a greater impact with Likelihood of Attrition a closer friend. www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 26
  27. 27. New-School Measurement Case Study: Attrition in Social Network model Examination of Context Aggregate User Behavior Social Network SESSIONS Number of Friends How often do people play? Amount of Interaction Quality of Friendship How long do people play? Quality of Interaction Network Density What does a play session look like? Network Centralization USERS Network Homogeneity Financial Investment In this example, the user behaviors are seen as proxies of social network qualities, Brand Loyalty or Favour since we do not have direct measures for those variables. Time Investment www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 27
  28. 28. New-School Measurement Case Study: Attrition in Social Network Statistical analyze Filter Query Analysis if not in avatars Distributed table Processing using ‘cloud’ computing If they don’t have friends 11 node Cluster If they never 10 hour query whispered reduced to 30 seconds random reduced Survival 95% Confidence sample processing time Std. Error Time Interval Mean 76.32 0.69 74.97, 77.66 by ~100X www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 28
  29. 29. New-School Measurement Case Study: Attrition in Social Network losing a friend *does* increase the analyze however chance that a user will leave. this effect can be reduced when users have strong social networks and communicate frequently. Regression Analysis of Social Factors in Game Attrition Predictor B Std. Error Wald Sig. Exp(B) Model 1 Friendship Size –.034 .004 64.381 .000 .966 Cohesiveness –.010 .005 5.229 .022 .990 Outgoing Whispers –.001 .001 5.269 .022 .999 Model 2 Friendship Size –.035 .004 66.599 .000 .966 Cohesiveness –.001 .001 4.604 .032 .999 Incoming Whispers –.010 .005 5.208 .022 .990 www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 29
  30. 30. New-School Measurement Case Study: Attrition in Social Network Was the data timely enough to effect change? refine If users aren’t doing what you expected, should you general be trying to redirect them or reward their actual behavior? After implementing a fix, did you observe the desired outcome? Additional user-level variables would improve analysis: demographic, account creation/ cancellation, etc. It is possible, from this type of model, to implement Specific a monitoring system to alert Disney of high-risk clusters or even individual users. The content of the messages can also be an important source of information about the network. www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 30
  31. 31. New-School Measurement Case Study: Avenue Q Facebook Application analyze www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 31
  32. 32. New-School Measurement Case Study: Navigation in an FPS analyze Animated heatmap-style visualization Contact Sheet for Frame-by-Frame Study Useful for level balancing & helping users find their way www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 32
  33. 33. New-School Measurement Case Study: Navigation + Survey Data in an FPS analyze Pearson Correlations: Navigational Behaviors and Player Reactions # Cells Visited Freq. of map use .188* Max Time For A Cell Frustration .208* Distance Traveled Understanding of Map .194* # Cells Visited Understanding of Map .316** Speed Understanding of Map .263** # Cells Visited Frustration -.202* Max Time For A Cell Understanding of Map -.311** Dead Reckoning Count Intensity -.191* **. Correlation is significant at the 0.01 level (2-tailed) *. Correlation is significant at the 0.05 level (2-tailed) www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog Wednesday, April 29, 2009 33
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×