GSummit SF 2014 - Recognizing Behavior with Big Data + Gamification by Ross Smith @42projects


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • You, the audience are in 1964 B&W and I am a visitor from the future – 50 years from now in 2014 and I’m going to share some stats
  • Source UN
  • Pew Research
  • Cisco
  • World Bank
  • Source: EEDAR market research
  • Education – is a VERY good place to do games. Many serious games exist here from childrens learning games through military or civilian simulators, etc.

    But we are most excited about the OCB/Core Work Skll space – where you can get as many players as possible, and they do something outside their normal job.

  • Build a story one sentence at a time, using instant messaging feature in Lync
    Start with participants sitting in a circle
    Each person is added to meeting sequentially, and are asked to add a sentence to the story
    Each person can only see the sentence written by the previous person when they add their own sentence
    After the last person finishes, the moderator reads the story
    Instant Messaging is the “bread and butter” of Lync, and one of our product goal was that IM need to be 100% reliable
    People were reporting issues with messages not being sent – so our game helped to track down this problem
    Game made it easy to identify when message wasn’t received by one party, or when it was sent out of order
    Able to connect information to debug issues that would have been difficult to catch in real life scenario

    Additional Points:
    Games allowed us to get usability feedback on the IM feature
    Automation could only simulate narrow user configuration
    Automation does not allow for unpredictability of user interaction (Phillip from EA – All software works fine until users use it)
  • Game specific to Windows 8 tablets
    Purpose of the game is to determine the user perceived performance on various tablets.
    Goal for the players is to identify the landmark that the moderator shows and respond back with IM quickly.
    The catch is the content and the IM are on different screens and players have to be able to transition between the screens
    We can determine the actual performance programmatically – but if the perceived performance is bad, then we need to change the design to accommodate
    Each participant had a different device/tablet so we were also able to cover many form factors
    Used data to improve the responsiveness of Lync

  • Another game specific to Windows 8
    Feature call Ink – allows users to draw images and send over IM
    Made this game more educational by helping participants learn traffic sign
    Moderator calls out traffic signs and players would draw it; best drawing wins
    The game also served as a simple adhoc usability study
    For example, brush stroke was too thin for finger, and by increasing the size of brush stroke helped improve the drawing experience
    Traditionally this would have been done with a user researcher conducting a usability study with external users
    Usability studies are usually after the development team have moved on to other features – this gives us immediate feedback on a feature
    You only need so many users to try something before a feedback becomes repetitive and we had enough internal users to get this feedback
  • “elevation of privilege” and “hackers, inc.”
    tool for learning secure development techniques
    emulate a white-hat security analysis company
    imagine and identify possible security threats

    Adam Shostack
  • Authentic Transactions
    A/B Testing
    put out multiple version and gather information on which is better
    requires ability to give different groups of users a different product
    passively gather data on actual usage
    Synthetic Transactions
    Test in Production (TiP)
    continually run known tests using various types of automation
    expected outcomes provide known anticipated product responses
    Gamified Transactions
    Crowd Sourced Testing
    direct users to atypical usage patterns
    provide interactions outside of the program under test
    note: always calculate the effect of the game itself (observer effect)
  • Say: A subset of experimentation is *controlled* experimentation, known as AB Testing
    Explain how AB Testing works
    Describe why it is controlled – how it proves causality of your observed results

    cluster analysis for user groups and scenario determination

    The unique user numbers are from a 2009 MSN experiment
    Note that these are likely not all MSN users, just those in the experiment over the experiment duration
  • This is by no means a comprehensive list.

    Descriptive analytics

    The Lync Active Monitoring Wiki site contains more examples, this slide merely highlights some of the big buckets:
  • Gamified Transactions generate data based upon experimental needs rather than actual customer usage. Customers are incentivized to use the product in atypical manners in order to either stress the product, test specific functionality, or provide training in how to use certain features. Data are gathered that provide the ability to predict attributes of usage for a broader group.
  • GSummit SF 2014 - Recognizing Behavior with Big Data + Gamification by Ross Smith @42projects

    1. 1. + Gamification and Big Data Ross Smith Microsoft Using Game Mechanics to Improve Data Science
    2. 2. + about me
    3. 3. + cultural change
    4. 4. + our world is changing new workforce global demographics technology revolution social connectivity ‘user’ experience is key
    5. 5. + IMAGINE 1964 San Francisco event The Beatles arrive in the US San Francisco - Cow Palace – 19 Aug 1964
    6. 6. + 7.2 billion humans 1804 – 1 billion 1927 – 2 billion 1960 – 3 billion 2025 – 8 billion 2050 – 9.6 billion 2081 – 10 billion Population growth rate is slowing !
    7. 7. + the internet 87% of American adults now use the internet 99% of households earning $75,000 or more 97% of young adults 97% with college degrees 68% of adults connect via mobile
    8. 8. + internet of things
    9. 9. + mobile video
    10. 10. > 50% of all global growth by 2025 will come from BRIC countries plus South Korea and Indonesia Global Shift: Arrival of Emerging Economies
    11. 11. + China 1.6 gamers in China for every American citizen 517 million 28% play more than 1 hour a day
    12. 12. In a global village of 100 61 would be Asian, ~80 would have mobile, 11 would be from Europe… and 70 would be gamers… Global Shift: Diverse and Distributed Workforces
    13. 13. + why is big data so hyped? “big data” was added to Merriam-Webster in 2014 edition… an accumulation of data that is too large and complex for processing by traditional database management tools
    14. 14. + “bigdataUX” – user research
    15. 15. + game design
    16. 16. + testing
    17. 17. + marketing
    18. 18. + productivity games using games to get the data you need
    19. 19. +where games work best Skills- Behaviors Matrix Core Work Skills Unique Work Skills Expanding Work Skills In-Role Behaviors Organizational Citizenship Behaviors
    20. 20. +Lync Test Games  Build a Story  Landmarks  Road Signs  Mobile Fest
    21. 21. + languagequalitygame
    22. 22. + Results Significant Quality Improvements for Windows 7 Positive Impact on Ship Schedule Team Morale and Subsidiary Engagement Total Screens Reviewed: Over 500,000 Total Number of Reviewers: Over 4,500 Screens per Reviewer: Average 119
    23. 23. Significant Quality Improvements for product Positive Impact on Ship Schedule Team Morale and Dogfood User Engagement Players Over 1,000 Feedback increase > 16x Feedback received: 10,000+ Players vs. non-players 67% of players participate vs. 3% of non Results
    24. 24. + data science using games to get the data you need
    25. 25. + ‘given enough eyeballs, all bugs are shallow’ Eric Raymond, The Cathedral and the Bazaar
    26. 26. +Gamified Transactions Data Authenticity Precision HighLow HighLow Gamified Transactions Synthetic Transactions “Authentic” Transactions A/B Tests
    27. 27. + passive monitoring and inferential analytics Visitors Randomly Distributed 50% CTR Version A (Control) 1.2% of users with a Page View clicked on Signup 2.8% of users with a Page View clicked on Signup Version B (Treatment) 50% Is the observed difference statistically significant? CTR User interactions instrumented, analyzed and compared Page Title Signup Here Title of Page Signup Here authentic transactions
    28. 28. +synthetic transactions - examples  test in production methods  outage detection  call quality  slow performance  video frame rate
    29. 29. + gamified transactions (GT’s)  use game mechanics to direct crowd activity  carefully designed not to impact service load  cluster analysis for user groups and scenario determination  generate A/B comparative studies  predictive analytics  beware of the observer effect
    30. 30. + types of elicited information what can we use GT’s for?
    31. 31. + usage studies  games that get users to try new features  teach best practice of product use  measure before and after usage patterns to determine stickiness of feature  can impact the long-term product use
    32. 32. + scenario coverage studies  direct crowd to use features that require more testing  use early adopters willing to accept lower quality to prepare for late majority that require high quality  requires knowing the impact to the user population  measure the observer effect – are gamers acting differently?
    33. 33. +crowd reputation scoring the value of gamified transactions hinges on our ability to trust user’s feedback Good Fair Needs work Poor Reputation Scores
    34. 34. + data science and feedback quality  especially important when rewards are used  use the crowd to test each others results!  develop a probability function for each gamer indicating the chance of a correct item of input  reputation score is the Bayesian prior for correctness calculation  reputation is categorized using percentile  provide games that train to improve reputation
    35. 35. + the opportunity GT’s bring to cloud  gamified monitoring  experience tuning and optimization  scenario feedback  performance  marketing data  crowdsourced testing
    36. 36. + marriage of gamification and big data is the future
    37. 37. +42Projects  Collaborative Play  Trust  Management Innovation
    38. 38. + thank you ross smith