Measurement and analytics for social apps 18 may - speakernotes


Published on

Presentation to London Facebook Developer Garage, May 18 2011, by Heather Stark.

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • I like to make sense of stuff. I’m an analyst. Pretty much everything I’ve done has an ‘analyst’ flavour - data analysis, discourse analysis, user behaviour analysis, technology analysis, market analysis, business model analyst…. (not yet ‘psychoanalyst’). What I’ve been doing recently is trying to make sense of the options available for measuring and analysing social apps. I’m here to share a bit of my enjoyment with you today. The reason I’ve been researching this is that I think it’s the most interesting thing going on at the moment. Specifically, I think relationship between [characteristics of the social graph] and [application design and content optimisation] is the most interesting thing going on. So I want to understand it better. What I’m going to be talking about is work in progress. An alpha version. A demo with no code. Your role in this is to throw tomatoes. Ideally, not ‘til the end of the talk. But afterwards – yes, please. That’s how I make stuff better.
  • I promised you a poem about elephants. It’s a 19thC poem by John Godfrey Saxe about an ancient Hindu, and later Buddhist, and Muslim legend. Everyone keeps borrowing the story. Why? Because it’s very good. Here are the first few verses: IT was six men of Indostan  To learning much inclined, Who went to see the Elephant  (Though all of them were blind), That each by observation  Might satisfy his mind. ii. The First approached the Elephant,  And happening to fall Against his broad and sturdy side,  At once began to bawl: "God bless me!—but the Elephant  Is very like a wall!" iii. The Second , feeling of the tusk,  Cried:"Ho!—what have we here So very round and smooth and sharp?  To me 't is mighty clear This wonder of an Elephant  Is very like a spear!" And so it goes.
  • The situation with social apps analytics strikes me as being quite similar. There are a lot of different aspects to it, and only grabbing on to one is misleading.
  • Of course what kind of elephant you want might be a matter of what you want it to DO. Lift logs, eat popcorn, make more elephants.
  • I’d like to start the serious bit by putting up a fence up between measurement and analytics. What I’m talking about is the difference between WHAT and SO WHAT. Measurement is about the WHAT. How many elephants, how big are they? Analytics I see as being about the SO WHAT. Do we have enough elephants? Are they different from the ones we had last week? What will they do next? So the domain of analytics is about thresholds, trends, associations, inference, and prediction. Analytics is more of a process than a product, though for sure there are products can save you from having to add it all up on your toes.
  • At its simplest, we are talking about actions performed by people, using applications, in a universe which encodes their relationships to objects, and to other people. Notice I haven’t used the F-word. This isn’t just about Facebook. But Facebook is particularly fun because of the way it makes relationships explicit and discoverable, and weaves those relationships into their own content provision. One thing that’s absolutely fundamental for insight is the ability to track users through sessions, and identify when they recur. This isn’t always possible, but it’s always worth striving for. If you don’t do this you end up relying on measures like the stickiness ratio (DAU/MAU). This is a bit like used chewing gum. Perhaps better than nothing. Cheap and cheerful. It tells you sweet-all about retention. You need to be able to tell whether you are pouring marketing money into a leaking ship or not.
  • "If you wish to bake an apple pie from scratch, you must first invent the universe” Carl Sagan Another distinction to get a grip on is the difference between applications and media. I view media as being a specific type of application – one that is preoccupied with perfoming actions on content. In fact media – which is to say Content applications are getting smarter – as content is customised for presentation using rules that take advantage of what the apps knows about the user, and the user’s social graph. Facebook is the 800 pound gorrilla here, but they aren’t the only one in the jungle. One startup worth watching is Cognitive Match, who do AI-based real-time content optimisation for ads. Another is Playnomics, which is so stealthy it’s a bit hard to tell exactly what they do, except that it’s very on-trend.
  • Here are some dimensions you can use to tell what bit of the elephant you’ve got hold of. I think these dimensions are fairly straightforward. There’s some magic involved in the more complicated measures, when you look at the detail. Where it gets interesting is in how they relate to each other . I’d say that’s where analytics plays best. So, for instance, you can track how messages are transformed as they propagate through networks, and how this transformation varies depending on what kind of network neighbourhoods they go through. (Kind of like a ‘regional accent’….;-) Or you can look at how changing some aspects of an app affects the extent to which people invite other people to share it, and the degree of success these invites have. Or on how ‘connected’ an individual is affects how influential they are. The answer isn’t always obvious. There’s a fair amount of published research on this, which is very interesting, but the best generalisation I can make about it is that it is worth measuring for yourself, for your own app and your own ecosystem. Usually if I stare at something like this for long enough it will resolve into a set of dimensions I can put on a graph. I’m still waiting on this one…. any ideas?
  • “ a terminological jungle, in which any newcomer may plant a tree” J. Barnes There’s a fascinating sub-world out there which is associated with measures for looking at social network structure. [QUOTE] Whatever you want to measure there’s at least one way to do it, and you don’t need a special license to make stuff up. One quick point: it can can pay to look at more than just local structure. Let’s think about ‘A’ in the graph. A is not highly connected but she links different groups. In my own social graph, I’ve got a few of those. You can see how they could be even more valuable than people with lots of connections. Another quick point: how interconnected a group is can affect how easy it is to address. Media6 Degrees has come up with something they call ‘Tribal Brand Index’ – I don’t know the algorithm details but it measures how interconnected and similar customers are. This predicts how well that community responds to some types of social media-based campaigns. More homogenous and interconnected communities are potentially easier to reach. Regarding dynamic processes, traditional models of diffusion of innovation, or ‘virality’ assume a ‘fully mixed’ population model. However it is clear that network structure affects the dynamics of how signals can be propagated. As a stochastic process it isn’t so analytically tractable as the old-fashioned models. There’s lots of research work going on on this topic which at the moment is fairly brute force empirical – based on observation and simulation.
  • I’d say that in terms of where we’re at, at the moment, the real hot spots are in doing analysis on social action dynamics, social network structure, and media/content evaluation. It’s by combining those with the relatively well understood disciplnes of application design testing and customer lifecyle tracking that companies are going to get the most traction.
  • Here’s the really scary bit. The demo. Here’s a test-drive of how to use the framework I’ve established in order to tell what kind of elephant is in the room with you. For sure, it’s wrong in some details. But that’s easy to fix. I’d be interested in hearing whether you find it a useful starting point for your own thinking.
  • Measurement and analytics for social apps 18 may - speakernotes

    1. 1. Measurement and analytics for social apps Heather Stark @HAStark Investigate Analyse Interpret May 18 2011
    2. 2. The blind men and the elephant
    3. 3. The blind men and the elephant Virality coefficient Bag of words Stickiness ratio Betweenness centrality Influencer identification Cohort analysis Multivariate testing Customer lifetime value Funnel optimisation Community detection Message cut-through
    4. 4. The blind men and the elephant
    5. 5. Measurement vs analytics Measurement How many, how big? Analytics Enough? Reliably different from X? What will happen next? Thresholds Trends Classification/segmentation Comparison Inference Prediction WHAT SO WHAT
    6. 6. What kind of ‘what’ are we talking about? Action ( , [ ], [ ]) P1 Pn On People Objects Actions Relations
    7. 7. Social apps and social media Context-specific social actions Content-generic social actions Applications Domain-specific App-specific Media Domain-specific Objects Actions Share Recommend Create Modify Display Delete Platform-specific Objects Content Content - Generic attributes Distribution: p2p, controlled, public … Source: user, user role, organisation … Referent: person, object, other content, … Relationship:self (versioning), other content (includes…) Persistence: real-time, asynch; transient, trace, … App-specific Platform-specific
    8. 8. Dimensions of difference: the ‘what’ Media/content evaluation Social action dynamics Social network structure Application design testing Customer lifecycle tracking Impact, Favourability, Message cut-through… Activity zone Sample measures (direct and derived) Virality, Influencer/Opinion leader scoring… Connectivity, Community identification, Assortative mixing… Funnel health indices, App-specific measures Acquisition channel tracking, Engagement, Retention…
    9. 9. Social network structure & dynamics Structure Dynamics Centrality Diffusion Connectivity Clustering
    10. 10. Dimensions of difference: the ‘so what’ Media/content evaluation Social action dynamics Social network structure Application design testing Customer lifecycle tracking Activity zone ‘ So what’ secret sauce?    } } (relatively) well-understood ‘ Wild west’ – unexplored (possibly difficult, possibly rewarding…)     
    11. 11. There IS an elephant in the room! THIS IS AN ALPHA VERSION FEEDBACK WELCOME THIS IS WRONG in at least some details….maybe more. BUT IS IT THE RIGHT WAY TO UNDERSTAND THE LANSCAPE? But what kind of elephant? Here’s one way to start to think about it.
    12. 12. Tomatoes? @HAStark