Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University

557 views

Published on

The goal of social science research is to discover fundamental features of human behavior. One of the best environments to accomplish this task are "games." Scientific behavioral experiments are a particular type of games, where the players' preferences are revealed through strategic interactions. "Gamification" -- the general technique of applying behavioral economics insights to understand any type of problems through games -- is increasingly been used in corporate business. This talk introduces the open-source project nodeGame (nodegame.org), a framework in Node.js to create and manage real-time and discrete-time synchronous behavioral experiments.

The basic API for creating and running behavioral experiments in a stand-alone server or on the cloud is presented, and techniques to recruit participants from the Amazon Mechanical Turk labor market and other sources such as Volunteer Science are discussed. The talks also peeks into the world of academia, showing how the availability of data and new quantitative methodologies has changed the way social scientists do research. Their current and future needs for better, more reliable software, simple APIs, and down-to-Earth documentations are discussed.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University

  1. 1. Online Behavioral Experiments with NodeGame Stefano Balietti Network Science Institute, Northeastern University Harvard Institute for Quantitative Social Science D'Amore McKim School of Business, Northeastern University @balietti | @nodegameorg | stefanobalietti.com | nodegame.org
  2. 2. 1. Become confident about the principles and the methodology behind social science scientific experiments 2. Get a quick-start with the software nodeGame to conduct experimental research online Goals of the Workshop
  3. 3. Workshop Instructors
  4. 4. Workshop Instructors Me Social Scientist Open Source Developer
  5. 5. Workshop Instructors Me Social Scientist Open Source Developer
  6. 6. What is an experiment?
  7. 7. An experiment is a methodological procedure carried out with the goal of verifying, falsifying, or establishing the validity of a hypothesis. A test under controlled conditions that is made to demonstrate a known truth, examine the validity of a hypothesis, or determine the efficacy of something previously untried. An experiment is an empirical method that arbitrates between competing hypotheses. What is an experiment?
  8. 8. Internal validity: The extent to which a causal inference based on a study is warranted by an experiment. What is an experiment?
  9. 9. Control 3 Pillars of Experimental Design RandomizationIndependence
  10. 10. Control 3 Pillars of Experimental Design RandomizationIndependence Deliberate choice of the variables that will be kept constants and those that will be varied: treatment variables
  11. 11. Control 3 Pillars of Experimental Design Randomization Independence Treatment variables must not be correlated
  12. 12. Control 3 Pillars of Experimental Design Randomization Independence Participants are randomly assigned to conditions Conditions are executed in random order Stratification might be required
  13. 13. Internal validity: The extent to which a causal inference based on a study is warranted by an experiment. External validity: The extent to which the results of a study can be generalized or extended to others. What is an experiment?
  14. 14. Internal validity: The extent to which a causal inference based on a study is warranted by an experiment. External validity: The extent to which the results of a study can be generalized or extended to others. What is an experiment?
  15. 15. External Validity
  16. 16. Galileo's hand-made drawing of the moon surface. Sidereus Nuncius (1610) External Validity
  17. 17. Galileo's hand-made drawing of the moon surface. Sidereus Nuncius (1610) External Validity Telescope: From Land to Sky
  18. 18. Galileo's hand-made drawing of the moon surface. Sidereus Nuncius (1610) External Validity Telescope: From Land to Sky Experiments: From Lab to Real Life
  19. 19. Super-brief history of scientific experiments
  20. 20. • Novum Organum (1620) • Scientific induction • Torturing Nature to extract her secrets Super-brief history of scientific experiments (1/3)
  21. 21. • Novum Organum (1620) • Scientific induction • Torturing Nature to extract her secrets Super-brief history of scientific experiments (1/3) Ever heard of Phlogistons??
  22. 22. “There is a property common to almost all the moral sciences, and by which they are distinguished from many of the physical; that is, that it is seldom in our power to make experiments with them” John Stuart Mill, 1836 Super-brief history of scientific experiments (2/3)
  23. 23. “There is no laboratory in which economists can test their hypothesis” , 1993 Super-brief history of scientific experiments (2B/3)
  24. 24. Nobel prize for Vernon Smith “for having established laboratory experiments as a tool in empirical economic analysis, especially in the study of alternative market mechanisms” 2002 Super-brief history of scientific experiments (3/3)
  25. 25. The Puzzle of Cooperation
  26. 26. The Ultimatum Game
  27. 27. The Ultimatum Game Rules: • 2 Players (A,B) • 100 $
  28. 28. The Ultimatum Game Rules: • 2 Players (A,B) • 100 $ • Player A makes a proposal
  29. 29. The Ultimatum Game Rules: • 2 Players (A,B) • 100 $ • Player A makes a proposal • Player B can accept or refuse the split.
  30. 30. The Ultimatum Game Rules: • 2 Players (A,B) • 100 $ • Player A makes a proposal • Player B can accept or refuse the split. • If B accepts each one gets his/her share.
  31. 31. The Ultimatum Game Rules: • 2 Players (A,B) • 100 $ • Player A makes a proposal • Player B can accept or refuse the split. • If B accepts each one gets his/her share. • If B refuses both players get nothing.
  32. 32. The Ultimatum Game: How to Split 100$ Rules: • 2 Players (A,B) • 100 $ • Player A makes a proposal • Player B can accept or refuse the split. • If B accepts each one gets his/her share. • If B refuses both players get nothing.
  33. 33. The Ultimatum Game: How to Split 100$ A B
  34. 34. The Ultimatum Game: How to Split 100$ Proposer Respondent RationalRational
  35. 35. The Ultimatum Game: How to Split 100$ Proposer Respondent I say 99 for me and 1 for you. RationalRational
  36. 36. The Ultimatum Game: How to Split 100$ Proposer Respondent I say 99 for me and 1 for you. RationalRational I say OK.
  37. 37. The Ultimatum Game: How to Split 100$ Human Being Human Being I say 99 for me and 1 for you.
  38. 38. The Ultimatum Game: How to Split 100$ Human Being Human Being I say…$!!@** I say 99 for me and 1 for you.
  39. 39. Nowak (2006) 5 rules for Cooperation
  40. 40. Traditional Laboratory Research • Decision making tools, heuristics, models of human behavior
  41. 41. Traditional Laboratory Research • Decision making tools, heuristics, models of human behavior • BUT
  42. 42. Traditional Laboratory Research • Decision making tools, heuristics, models of human behavior • BUT • Small groups of individuals, generally 4-12 (bounded by uni. Lab)
  43. 43. Traditional Laboratory Research • Decision making tools, heuristics, models of human behavior • BUT • Small groups of individuals, generally 4-12 (bounded by uni. Lab) • Behavior of individuals in social networks under-researched
  44. 44. Traditional Laboratory Research • Decision making tools, heuristics, models of human behavior • BUT • Small groups of individuals, generally 4-12 (bounded by uni. Lab) • Behavior of individuals in social networks under-researched • Limited in the number of hypotheses that can be tested
  45. 45. Traditional Laboratory Research • Decision making tools, heuristics, models of human behavior • BUT • Small groups of individuals, generally 4-12 (bounded by uni. Lab) • Behavior of individuals in social networks under-researched • Limited in the number of hypotheses that can be tested • Observations are limited in time
  46. 46. Traditional Laboratory Research • Decision making tools, heuristics, models of human behavior • BUT • Small groups of individuals, generally 4-12 (bounded by uni. Lab) • Behavior of individuals in social networks under-researched • Limited in the number of hypotheses that can be tested • Observations are limited in time • Limited(?) external validity
  47. 47. Traditional Laboratory Research • Decision making tools, heuristics, models of human behavior • BUT • Small groups of individuals, generally 4-12 (bounded by uni. Lab) • Behavior of individuals in social networks under-researched • Limited in the number of hypotheses that can be tested • Observations are limited in time • Limited(?) external validity • Generally high costs for an academic budget
  48. 48. The Present/Future of Behavioral Research
  49. 49. The Present/Future of Behavioral Research
  50. 50. Workshop Instructors Me Social Scientist Open Source Developer
  51. 51. Workshop Instructors Me Social Scientist Open Source Developer
  52. 52. nodeGame 3.x
  53. 53. Install nodeGame 3.x 1. Go to http://nodegame.org 2. Click on the “Get Started” button 3. Follow instructions to install Development version 4. Keep following my talk!
  54. 54. nodeGame 3.x Architecture Overview
  55. 55. nodeGame 3.x Architecture Overview
  56. 56. nodeGame 3.x Architecture Overview
  57. 57. nodeGame 3.x Architecture Overview ~46.000 lines (including comments)
  58. 58. Launch nodeGame 3.x Open a terminal and browse to the nodegame folder Start the server with the command: node launcher.js Open a browser and access the default ultimatum game at the url: localhost:8080/ultimatum Start an automated player (bot) at the url: localhost:8080/ultimatum?clientType=autoplay Access the administrator interface at the url: localhost:8080/ultimatum/monitor/
  59. 59. Create a New Game cd nodegame/bin/ node nodegame create-game gamename <author> <email> Creates a new game in the games/ folder (usually nodegame/games/) Important! Windows users must install the nodegame-generator package manually from the git repository (or npm) git clone https://github.com/nodeGame/nodegame-generator.git cd nodegame-generator npm install
  60. 60. Anatomy of a Game
  61. 61. Anatomy of a Game Features: - Define game variables and treatments - Define the game sequence with the Stager API - Implement the game sequence into client types
  62. 62. The Game Sequence and Client Types Game Sequence Client Types stager.stage("instructions") .step("instructions_1") .step("instructions_2") .step("instructions_3"); stager.stage("quiz"); stager.repeat("game", 3) stager.stage("questionnaire"); stager.extendStage("game", { steps: ["offer", "respond", . "display_results"] }); Code Snippet
  63. 63. The Game Sequence and Client Types Game Sequence Client Types stager.stage("instructions") .step("instructions_1") .step("instructions_2") .step("instructions_3"); stager.stage("quiz"); stager.repeat("game", 3) stager.stage("questionnaire"); stager.extendStage("game", { steps: [ "offer", "respond", . "display_results" ] }); Code Snippet
  64. 64. The Game Sequence and Client Types Game Sequence Client Types Client types implement the game sequence by extending its steps and stages.
  65. 65. The Game Sequence and Client Types Game Sequence Client Types
  66. 66. The Game Sequence and Client Types Game Sequence Client Types
  67. 67. The Game Sequence and Client Types Game Sequence Client Types Code Snippet stager.setDefaultProperty("coins",100); stager.extendStage("game", { coins: 200 }); stager.extendStep("offer", { coins: 50 });
  68. 68. The Game Sequence and Client Types Game Sequence Client Types Code Snippet stager.setDefaultProperty("coins",100); stager.extendStage("game", { coins: 200 }); stager.extendStep("offer", { coins: 50 }); node.game.getProperty("coins") returns 50 in step 'offer', 200 in steps 'respond' and 'display results' inside the 'game' stage, and 100 in any other step. Step/Stages Properties are hierarchical
  69. 69. Some Built-in Step-Properties Property Description Type init Executed upon entering the step/stage function exit Executed upon exiting the step/stage function timer Sets the maximum execution time for step (ms) number timeup Executed if the timer expires function done Executed when the step terminates function cb The actual step execution function
  70. 70. Logic Client Type Executed on the server, governs a game room - Makes groups - Keep track of game history - Handles disconnections and reconnections - Access to: - internal Database NDDB: node.game.memory - player list: node.game.pl - channel and registry object
  71. 71. Player Client Types Executed on a client, e.g. a browser - Display/manipulate information on screen - Exchange messages with the server and other players - Respond to user events - Determine when the current step is finished (“done”) W (Window): manipulates information on screen and the behavior of user interface, e.g. disabling right click, locking screen, etc. At the end of each step, the game developer must call the method node.done, which triggers a procedure that advances the client to the next step.
  72. 72. Inside a Step Callback Function Instructions Step - Player client type - No interaction with other players - No interaction with the page - No time limit - Simply click “Done” when finished reading and advance to next step
  73. 73. Inside a Step Callback Function Instructions Step - Player client type - No interaction with other players - No interaction with the page - No time limit - Simply click “Done” when finished reading and advance to next step
  74. 74. Inside a Step Callback Function stager.extendStep('instructions_1', { frame: "instr1.htm" }); Instructions Step - Player client type - No interaction with other players - No interaction with the page - No time limit - Simply click “Done” when finished reading and advance to next step
  75. 75. Inside a Step Callback Function stager.extendStep('instructions_1', { frame: "instr1.htm" }); Widget Step - Player client type - No interaction with other players (at the moment) - Standardized interaction - With or without time limit - Register the mood of the participant using the PANAS method. Instructions Step - Player client type - No interaction with other players - No interaction with the page - No time limit - Simply click “Done” when finished reading and advance to next step
  76. 76. Inside a Step Callback Function stager.extendStep('instructions_1', { frame: "instr1.htm" }); Widget Step - Player client type - No interaction with other players (at the moment) - Standardized interaction - With or without time limit - Register the mood of the participant using the PANAS method. Instructions Step - Player client type - No interaction with other players - No interaction with the page - No time limit - Simply click “Done” when finished reading and advance to next step
  77. 77. Inside a Step Callback Function stager.extendStep('instructions_1', { frame: "instr1.htm" }); Widget Step - Player client type - No interaction with other players (at the moment) - Standardized interaction - With or without time limit - Register the mood of the participant using the PANAS method. Instructions Step - Player client type - No interaction with other players - No interaction with the page - No time limit - Simply click “Done” when finished reading and advance to next step stager.extendStep('instructions_2', { widget: "MoodGauge" });
  78. 78. Roles (version 3.5 dev) stager.extendStep('bidder', { matcher: { // Available roles. roles: [ 'BIDDER', 'RESPONDENT'], // Each player is matched with a // partner and is assigned a role. match: 'roundrobin', // How to continue matching after // all combinations are exhausted cycle: 'repeat_invert' } }); Logic Client Type
  79. 79. Roles (version 3.5 dev) stager.extendStep('bidder', { roles: { BIDDER: { frame: 'bidder.htm', cb: function() { // Bidder stuff. } }, RESPONDENT: { frame: 'resp.htm', cb: function() { // Respondent stuff. }}} }); Player Client Type stager.extendStep('bidder', { matcher: { // Available roles. roles: [ 'BIDDER', 'RESPONDENT'], // Each player is matched with a // partner and is assigned a role. match: 'roundrobin', // How to continue matching after // all combinations are exhausted cycle: 'repeat_invert' } }); Logic Client Type
  80. 80. Inside the Step Callback Function // The respondent receives the offer, // displays it, and terminates the step. // On incoming data labeled "offer", // executes the function. node.on.data("offer", function(msg) { // Display offer using W. W.setInnerHTML("offer", msg.data); // End step. node.done(); }); // Player made a choice // and clicked the submit button. submit.onclick = function() { // Send offer to respondent. node.say("offer", "respondent", 1); // End step, store offer in server. node.done({ offer: 1 }); }; I say 99 for me and 1 for you. Proposer Respondent
  81. 81. NDDB: Internal Database // The respondent receives the offer, // displays it, and terminates the step. // On incoming data labeled "offer", // executes the function. node.on.data("offer", function(msg) { // Display offer using W. W.setInnerHTML("offer", msg.data); // End step. node.done(); }); // Player made a choice // and clicked the submit button. submit.onclick = function() { // Send offer to respondent. node.say("offer", "respondent", 1); // End step, store offer in server. node.done({ offer: 1}); }; I say 99 for me and 1 for you. Proposer Respondent Logic Client Type NDDB database node.game.memory
  82. 82. NDDB: Internal Database node.done({ offer: 1 }); // Set an object in memory explicitly. node.set({ foo: "bar" });
  83. 83. NDDB: Internal Database { stage: { stage: 2, step: 1, round: 1 }, player: "respondent", timestamp: 1480336721939 offer: 1, // foo: "bar" // node.done specific. timeup: false, time: 4321, // from step. done: true }; node.done({ offer: 1 }); // Set an object in memory explicitly. node.set({ foo: "bar" });
  84. 84. NDDB: Internal Database var m = node.game.memory; // NDDB instance. // Lazy loading. m.select("player", "=", "respondent"); m.and("stage.step", "=", 1) .and("offer") .fetch()[0]; // Use indexes. m.player["respondent"].last(); m.stage["2.1.1"].select("offer").fetch()[0]; node.done({ offer: 1 }); // Set an object in memory explicitly. node.set({ foo: "bar" }); { stage: { stage: 2, step: 1, round: 1 }, player: "respondent", timestamp: 1480336721939 offer: 1, // foo: "bar" // node.done specific. timeup: false, time: 4321, // from step. done: true };
  85. 85. NDDB: Internal Database // Modify database items upon inserting them. node.game.memory.on("insert", function(item) { // Decorate item or save it somewhere else. }); // Disable internal database completely. // Remove default event listener. node.off('in.set.DATA'); // Add new one. node.on('in.set.DATA', function(msg) { // Save msg somewhere else, e.g. Mongo. }); Event Listener Manager node.game.events
  86. 86. NDDB: Internal Database // Modify database items upon inserting them. node.game.memory.on("insert", function(item) { // Decorate item or save it somewhere else. }); // Disable internal database completely. // Remove default event listener. node.off('in.set.DATA'); // Add new one. node.on('in.set.DATA', function(msg) { // Save msg somewhere else, e.g. Mongo. }); Event Listener Manager node.game.events
  87. 87. Events in nodeGame 3.x Internal: generated by the game or by the nodeGame engine. Incoming: triggered by incoming messages, 'in.[target].[action]', ‘in.set.DATA’
  88. 88. Events in nodeGame 3.x Internal: generated by the game or by the nodeGame engine. Incoming: triggered by incoming messages, 'in.[target].[action]', ‘in.set.DATA’ // Registering an internal event listener. node.on('LOCAL_EVENT', function(a, b) { // This is local event. }); // Triggering a local event. node.emit('LOCAL_EVENT', 1, 2);
  89. 89. Events in nodeGame 3.x Internal: generated by the game or by the nodeGame engine. Incoming: triggered by incoming messages, 'in.[target].[action]', ‘in.set.DATA’ // Registering an internal event listener. node.on('LOCAL_EVENT', function(a, b) { // This is local event. }); // Listening for incoming DATA msgs. node.on('in.say.DATA', function(msg) { // I got a data msg! }); // Listening for incoming "foo" DATA msgs. node.on.data('foo', function(msg) { // I got a data msg labeled "foo"! }); // Triggering a local event. node.emit('LOCAL_EVENT', 1, 2); // Triggering a local event. node.say('foo', "bar");
  90. 90. Events in nodeGame 3.x Events are hierarchically organized, where you register your event matters!
  91. 91. Events in nodeGame 3.x Events are hierarchically organized, where you register your event matters! // Listening for incoming "foo" DATA msgs. node.on.data('foo', function(msg) { // I got a data msg labeled "foo"! });
  92. 92. Events in nodeGame 3.x Events are hierarchically organized, where you register your event matters! // Listening for incoming "foo" DATA msgs. node.on.data('foo', function(msg) { // I got a data msg labeled "foo"! }); Valid only in Quiz stage
  93. 93. Events in nodeGame 3.x Events are hierarchically organized, where you register your event matters! // Listening for incoming "foo" DATA msgs. node.on.data('foo', function(msg) { // I got a data msg labeled "foo"! }); Registered in init or cb function of Step 3.3 Valid only in step 3.3 (any round).
  94. 94. Events in nodeGame 3.x Events are hierarchically organized, where you register your event matters! // Listening for incoming "foo" DATA msgs. node.on.data('foo', function(msg) { // I got a data msg labeled "foo"! }); Registered in the init function of “Game” Valid throughout all steps of “Game” stage
  95. 95. Events in nodeGame 3.x Events are hierarchically organized, where you register your event matters! How to register a listener valid throughout all the game?
  96. 96. Events in nodeGame 3.x Events are hierarchically organized, where you register your event matters! stager.setInitCb(function() { // Listening for incoming "foo" DATA msgs. node.on.data('foo', function(msg) { // I got a data msg labeled "foo"! }); }); How to register a listener valid throughout all the game? In the init function of the game!
  97. 97. Anatomy of a Game
  98. 98. Topics Covered and To Cover
  99. 99. Topics Covered and To Cover
  100. 100. Recruiting Participants Free Platforms Psychological Research on the Net http://psych.hanover.edu/research/exponnet.html The Web Experiment List http://www.wexlist.net Online Social Psychology Studies http://www.socialpsychology.org/expts.htm Volunteer Science http://www.volunteerscience.com
  101. 101. Recruiting Participants Commercial Platforms ODesk, Freelancer, Elance, Guru, CrowdFlower, Innocentive, SurveySampling, UpWork … Most used in research:
  102. 102. How To Run a Game on AMT: The Big Picture 1. Upload nodeGame server + game to a server e.g. Heroku cloud: https://github.com/nodeGame/nodegame-heroku 2. Create a new HIT on Amazon Mechanical Turk (AMT), e.g. using the requester interface or other service API. 3. Use the nodegame-mturk package https://github.com/nodeGame/nodegame-mturk/ to handle: - Connection and authentication from AMT HIT page to nodeGame server. - Opening/expiring/extending HIT while running it. - Payments, bonuses and qualifications after HIT is done.
  103. 103. Known Issues Conducting Synchronous Experiments on AMT 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform
  104. 104. Known Issues Conducting Synchronous Experiments on AMT 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many
  105. 105. Known Issues Conducting Synchronous Experiments on AMT 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many 3. Waiting time in the waiting room should be limited and homogenous across participants
  106. 106. Known Issues Conducting Synchronous Experiments on AMT 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many 3. Waiting time in the waiting room should be limited and homogenous across participants 4. Demographics should be stratified as much as possible
  107. 107. Known Issues Conducting Synchronous Experiments on AMT 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many 3. Waiting time in the waiting room should be limited and homogenous across participants 4. Demographics should be stratified as much as possible 5. Participants should not re-take the same experiment again
  108. 108. Known Issues Conducting Synchronous Experiments on AMT 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many 3. Waiting time in the waiting room should be limited and homogenous across participants 4. Demographics should be stratified as much as possible 5. Participants should not re-take the same experiment again 6. Dropouts rate (non-completing the HIT) signigicantly affects the whole experiment, increasing costs $$ and time to complete it
  109. 109. Known Issues Conducting Synchronous Experiments on AMT 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform Pay minimum wage (at least). Make bonus dependent on performance
  110. 110. Known Issues Conducting Synchronous Experiments on AMT 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many Rule of thumb: 2.8 times desidered number of participants Expire HIT as soon as desired number of participant entered the experimental pipeline (+small buffer) 3x
  111. 111. Known Issues Conducting Synchronous Experiments on AMT 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many 3. Waiting time in the waiting room should be limited and homogenous across participants Rule of thumb: 2.8 times desidered number of participants Expire HIT as soon as desired number of participant entered the experimental pipeline (+small buffer) Dispose participants who waited too long (compensated)
  112. 112. Known Issues Conducting Synchronous Experiments on AMT 4. Demographics should be stratified as much as possible Waiting room can group players based on features Collect some demographics features in a pre-experimental part Start 2 (or more) groups at the same time
  113. 113. Known Issues Conducting Synchronous Experiments on AMT 5. Participants should not re-take the same experiment again Create one HIT (no group) with 3x rule Keep track of turker ids (nodegame-mturk) Assign qualification via AMT API (nodegame-mturk)
  114. 114. Known Issues Conducting Synchronous Experiments on AMT 6. Rate of dropouts (non-completing the HIT) significantly affects the whole experiment, increasing costs $$ and time to complete it • Make sure waiting time is not too long (previous advices)
  115. 115. Known Issues Conducting Synchronous Experiments on AMT 6. Rate of dropouts (non-completing the HIT) significantly affects the whole experiment, increasing costs $$ and time to complete it • Make sure waiting time is not too long (previous advices) • Inform about required time BEFORE and DURING experiment
  116. 116. Known Issues Conducting Synchronous Experiments on AMT 6. Rate of dropouts (non-completing the HIT) significantly affects the whole experiment, increasing costs $$ and time to complete it • Make sure waiting time is not too long (previous advices) • Inform about required time BEFORE and DURING experiment • Design experiments that are robust to dropout (as much as possible). For example, use shares instead of fixed quantities.
  117. 117. Known Issues Conducting Synchronous Experiments on AMT 6. Rate of dropouts (non-completing the HIT) significantly affects the whole experiment, increasing costs $$ and time to complete it • Make sure waiting time is not too long (previous advices) • Inform about required time BEFORE and DURING experiment • Design experiments that are robust to dropout (as much as possible). For example, use shares instead of fixed quantities. • Use requirements checkings.
  118. 118. Known Issues Conducting Synchronous Experiments on AMT 6. Rate of dropouts (non-completing the HIT) significantly affects the whole experiment, increasing costs $$ and time to complete it • Make sure waiting time is not too long (previous advices) • Inform about required time BEFORE and DURING experiment • Design experiments that are robust to dropout (as much as possible). For example, use shares instead of fixed quantities. • Use requirements checkings. • Use “seriousness checks” and “warm-up” phases.
  119. 119. How To Run a Game on AMT
  120. 120. AMT How To Run a Game on AMT AMT HIT
  121. 121. Not Just Experiments, also Gamification • Industry: lots of A/B testing and customer surveys • Gamification: applies elements of game playing to solve complex and simple problems • Increasing trend • Improve engagement and understanding • Fun • Generate More and Better Ideas • Train Machine Learning Algorithms (http://quickdraw.withgoogle.com)
  122. 122. Contributions Welcome! • Using it, request features, report bugs • Help developing • Extremely well documented source code
  123. 123. Contributions Welcome! • Using it, request features, report bugs • Help developing • Extremely well documented source code
  124. 124. Thank You! @balietti | @nodegameorg | stefanobalietti.com | nodegame.org

×