Synchronous Online Experiments with NodeGame

1. Synchronous Online Experiments nodeGame.org Stefano Balietti Microsoft Research Computational Social Science Symposium Cologne 5th November 2018 @balietti @nodegameorg stbaliet@microsoft.com

2. Postdoc Researcher Microsoft Research - Computational Social Science New York City Previously Postdoc Network Science Institute, Northeastern University Fellow IQSS, Harvard University PhD Computational Social Science, ETH Zurich http://stefanobalietti.com Your Instructor: Stefano Balietti

3. Main Research Interests:  Incentives schemes for peer review systems  Consensus formation and social influence  Inequality, redistribution, and preferences thereof  Optimal experimental design  Philosophy of science, in particular Paul Feyerabend http://stefanobalietti.com Your Instructor: Stefano Balietti

4. Some Stats: • 5 Undergrad / 3 PhD / 3 Postdoc / 4 Professor / 1 Other • 8 Female / 8 Male • 5 Computer Science / 4 Media&Comm. / 3 Sociology / 1 Econ / 1 Political Science / 1 Other Your Fellow Participants

5. 1. Review fundamental concepts of experimental design 2. Understand challenges of opportunities for online synchronous experiments 3. Acquire enough knowledge of JavaScript programming 4. Learn the how to design, implement, and run a basic experiment with nodeGame (and contribute to it?) 5. Learn more advanced nodeGame features OR implement your own game OR Amazon Mechanical Turk Goals of the Workshop

6. 1. Review fundamental concepts of experimental design 2. Understand challenges of opportunities for online synchronous experiments 3. Acquire enough knowledge of JavaScript programming 4. Learn the how to design, implement, and run a basic experiment with nodeGame (and contribute to it?) 5. EXTRA: advanced nodeGame features, implement own game, Amazon Mechanical Turk, JavaScript, … Goals of the Workshop

7. 1. Git Latest https://git-scm.com (Windows users install and use Git Bash) 2. Node.JS 10.x https://nodejs.org/en 3. NodeGame v4.x https://nodegame.org Additionally, • Make sure to have a Text Editor with syntax highlighting associated with .js files • Simple editor: Notepad++ (Windows), TextMate, TextWrangler (Mac) • More advanced: Atom, Sublime, Emacs, Vim • Might want to setup a free account on Heroku https://www.heroku.com • Might want to signup to the nodeGame’s forum For Hands On: Install

8. 1. Git Latest https://git-scm.com (Windows users install and use Git Bash) 2. Node.JS 10.x https://nodejs.org/en 3. NodeGame v4.x https://nodegame.org Additionally, • Make sure to have a Text Editor with syntax highlighting associated with .js files • Simple editor: Notepad++ (Windows), TextMate, TextWrangler (Mac) • More advanced: Atom, Sublime, Emacs, Vim • Might want to setup a free account on Heroku https://www.heroku.com • Might want to signup to the nodeGame’s forum For Hands On: Install

9. What Is an Experiment?

10. • An experiment is a methodological procedure carried out with the goal of verifying, falsifying, or establishing the validity of a hypothesis. • A test under controlled conditions that is made to demonstrate a known truth, examine the validity of a hypothesis, or determine the efficacy of something previously untried. • An experiment is an empirical method that arbitrates between competing hypotheses. What Is an Experiment?

11. • It is an experiment where participants • (i) interact in a common environment, • (ii) at the same time, • (iii) and their actions have an immediate effect on the decision process, and experimental outcome (e.g., monetary payoff) • Synchronous experiments can be turn-based or real-time What Is a Synchronous Experiment?

12. The extent to which a causal inference based on a study is warranted by an experiment. Internal Validity

13. 3 Pillars of Experimental Design Control RandomizationIndependence

14. 3 Pillars of Experimental Design Control Deliberate choice of the variables that will be kept constants and those that will be varied: treatment variables

15. 3 Pillars of Experimental Design Independence Treatment variables must not be correlated

16. 3 Pillars of Experimental Design • Participants are randomly assigned to treatments • Treatments are executed in controlled order • Stratification might be required Randomization

17. Short History of Experimental Research “There is a property common to almost all the moral sciences, and by which they are distinguished from many of the physical; that is, that it is seldom in our power to make experiments with them” John Stuart Mill, 1836

18. Short History of Experimental Research Friedman, Cassar (2004) Economists go to the laboratory. Who, what, when and why.

19. Short History of Experimental Research “Economics unfortunately cannot perform the controlled experiments of chemists or biologists because [it] cannot easily control other important factors. Like astronomers or meteorologists, [it] generally must be content largely to observe.” Samuelson and Nordhaus 1985, Economics Textbook (italics added) “There is no laboratory in which economists can test their hypotheses” 1993

20. Short History of Experimental Research Nobel prize for Vernon Smith “for having established laboratory experiments as a tool in empirical economic analysis, especially in the study of alternative market mechanisms” 2002

21. Induced Value Theory “The key idea of induced value theory is that proper use of a reward medium allows an experimenter to induce pre-specified characteristics, and the subjects innate characteristics become largely irrelevant. ” (Vernon Smith, 1976) Monotonicity: Participants always prefer more of the reward medium (m) Salience: Changes in m are directly related to a participant’s actions Dominance: changes in a subject’s utility from participating in the experiment are primarily due to changes in m.

24. Induced Value Theory “The key idea of induced value theory is that proper use of a reward medium allows an experimenter to induce pre-specified characteristics, and the subjects innate characteristics become largely irrelevant. ” (Vernon Smith, 1976) Monotonicity: Participants always prefer more of the reward medium (m) Salience: Changes in m are directly related to a participant’s actions Dominance: changes in a subject’s utility from participating in the experiment are primarily due to changes in m. Can you think of examples where failing to induce values can affect experimental results?

25. Some Limitations of Traditional Experimental Research

26. Some Limitations of Traditional Experimental Research • Small groups of individuals, generally 4-12 (bounded by size uni. Lab) • Behavior of individuals in social networks under-researched • Limited in the number of hypotheses that can be tested • Observations are limited in time • Generally high costs for an academic budget • Limited external validity: • Sample WEIRD student • Artificial conditions

27. Goals for Present and Future Behavioral Research

28. Goals for Present and Future Behavioral Research

29. • Survey Experiments • Game-Based Asynchronous Experiments • Game-Based Synchronous Experiments Examples of Online Experiments

30. Survey Experiments Examples of Online Experiments Kuziemko et al. (2015) How Elastic Are Preferences for Redistribution Insert a new block of questions, or provide additional info Measure preferences for outcome variable at the individual level Relatively simple to implement

31. Game-Based Asynchronous Experiments Examples of Online Experiments Create a common environment where a single variable is changed (e.g network structure) Observe variation on outcome variable at the group level Generally complex design Can extend for long periods of time Centola (2010) The Spread of Behavior in Online Social Networks

32. Game-Based Synchronous Experiments Examples of Online Experiments Create a common environment where a single variable is changed (e.g., group size or payoff) Observe variation on outcome variable at the group level Can be really complex or stylized Mao et al. (2016) An Experimental Study of Team Size and Performance on a Complex Task

33. Examples of Online Experiments COMPLEXITY OF EXECUTION Survey Experiments Game-Based Asynchronous Experiments Game-Based Synchronous Experiments

34. Challenges in Online Experiments • Recruitment • Payment • Anonymity Dropouts • Data Quality • Farming • Cheating • Technical Barriers for implementation • Cross-Device Compatibility • Accessibility (impaired users) •Fairness • Interference

36. Recruiting Platforms • Amazon Mechanical Turk https://www.mturk.com • TurkPrime https://www.turkprime.com • Figure Eight (ex Crowd Flower, ML) https://www.figure-eight.com • Prolific.ac https://www.prolific.ac • YouGov (country-specific) https://yougov.co.uk • PollFish (also built-in surveys) https://www.pollfish.com • Psychological Research on the Net http://psych.hanover.edu/research/exponnet.html • The Web Experiment List http://www.wexlist.net • Online Social Psychology Studies http://www.socialpsychology.org/expts.ht m • Reddit (for bursty access) https://www.reddit.com/r/WebGames

37. Citizen Science Initiatives • Volunteer Science https://volunteerscience.com • Science@Home https://www.scienceathome.org • Zoouniverse https://www.zooniverse.org • Lab in the Wild https://labinthewild.org • CitizenLab https://www.citizenlab.co

40. Ming et al. (2016) “The Communication Network Within the Crowd”

43. Challenges in Online Experiments • The same person uses the same computer to participate repeatedly. • The same person uses different computers to participate repeatedly. • Different persons use the same computer to participate. • Anonymity might increase deviant behavior • (but get access to specialized populations)

46. Improve Data Quality and Attention Content responsive faking: (a) responses influenced by the item content but (b) are not completely accurate. purposeful faking (e.g., malingering in medical surveys) socially desirable response of both intentional and nonintentional varieties Content nonresponsivity: responding without regard to item content. Large variance estimation (2.5 – 50%; 10.6 for students) Inattentive only on some questions Meade, A. W. and S. B. Craig (2012). Identifying Careless Responses in Survey Data. Psychological methods 17(3), 437.

47. Improve Data Quality and Attention • Engagement is key ui design, interactive elements, colors, length, feedback… • Nonsensical or “bogus” items “30th February” • Instructed response items “To monitor quality, please respond with a two for this item” • Unlikely sets “Select all the good qualities you possess” • Consistency indexes Evaluating differences among responses to items highly similar in content • Too quick response times and other obvious patterns in responses Meade, A. W. and S. B. Craig (2012). Identifying Careless Responses in Survey Data. Psychological methods 17(3), 437.

48. Commitment Device • Ask participants if they paid attention to the previous set of questions • It prompts participants to pay attention to the subsequent set of questions • Its purpose is fulﬁlled regardless of whether the respondents answer honestly

49. Commitment Device Example Before proceeding to the next set of questions, we want to ask for your feedback about the responses you provided so far. It is vital to our study that we only include responses from people who devoted their full attention to this study. This will not aﬀect in any way the payment you will receive for taking this survey. In your honest opinion, should we use your responses, or should we discard your responses since you did not devote your full attention to the questions so far? - Yes, I have devoted full attention to the questions so far and I think you should use my responses for your study - No, I have not devoted full attention to the questions so far and I think you should not use my responses for your study Alesina, Miano, and Stancheva (2018) “Immigration and Redistribution” NBER 24733

50. AMT Bot Scare TurkPrime Blog https://blog.turkprime.com/after-the-bot-scare-understanding-whats-been-happening-with-data-collection-on-mturk-and-how-to-stop-it https://blog.turkprime.com/moving-beyond-bots-mturk-as-a-source-of-high-quality-data https://github.com/MAHDLab/rIP https://itaysisso.shinyapps.io/Bots Tools to filter Farmers’ IP addresses Dennis et al. (2018) “MTurk Workers' Use of Low-Cost 'Virtual Private Servers' to Circumvent Screening Methods: A Research Note” https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3233954 Ahler et al. (2018) The Micro-Task Market for “Lemons”: Collecting Data on Amazon’s Mechanical Turk http://gsood.com/research/papers/turk.pdf

52. Fairness for Crowd Workers • IRB, IRB, IRB (Institutional Review Board) • Crowd worker have flexibility, but almost no rights • Low pay, menial tasks, no accrual of sick days, or contributions to pension, no unions, etc. http://www.ghostwork.info http://www.inthecrowd.org Mary Gray & Sid Suri

55. Dropouts

56. Dropouts (Attrition) Arechar et al. (2018) “Conducting interactive experiments online” “The most severe problem for online interactive studies, and the largest discrepancy with laboratory experiments, is attrition (participant dropout).” Major challenge to internal validity, if dropout rates vary with treatment, selection bias may arise

57. Dropouts (Attrition) Arechar et al. (2018) “Conducting interactive experiments online” Just a little bit more than 50% of the groups that started finished without dropouts (18%) overall dropout rate As dropouts are spread across experimental groups, the fraction of the data set affected by these dropouts is considerably larger.

58. What Keeps Participants In? Intrinsic Motivation Volunteer Science, Galaxy Zoo, Science@Home (Deci, 2005) Monetary incentives about 75% of US Turkers reported that MTurk was their primary or secondary source of income (Paolacci et al, 2009) Curiosity Solving and unrelated, but intriguing puzzle (Law et al., 2016) Personalized Feedback Lab-in-the-Wild (Ye et al., 2017)

59. Two Simple Techniques To Limit Dropouts • Implement warm-up phase (nodeGame levels) • Implement seriousness checks Reips (2002) Standards for Internet-Based Experimenting

60. What to Do When Dropouts Happen?

61. What to Do When Dropouts Happen? • Terminate experiment immediately • Continue with less participants (notify? Does behavior change?) • Replace with bots (notify? Behavior is different) • Replace with other humans (difficult to implement technically and humanly)

62. What to Do When Dropouts Happen? • Terminate experiment immediately • Continue with less participants (notify? Does behavior change?) • Replace with bots (notify? Behavior is different) • Replace with other humans (difficult to implement technically and humanly)  Try to design games that are dropout-robust, Make Leaving Harder, Inform about total time for experiment, Inform about waiting times during the experiment, Wait for reconnections, Notify

64. List of Platforms for Online experiments Group-Behavior • nodeGame  https://nodeGame.org • Wextor (pioneer) https://www.wextor.eu • Otree (large base) https://www.otree.org • Lioness (new) https://lioness-lab.org • Breadboard (networks) http://breadboard.yale.edu Individual • JSPsych (many plugins) https://www.jspsych.org • PsiTurk (groups possible) https://psiturk.org Other Resources • Browser Stack https://www.browserstack.com • CSS Frameworks https://getbootstrap.com • Empirica (very new) https://empirica.ly • TurkServer (discontinued) https://turkserver.readthedocs.io

65. nodeGame: the Good Parts • Powerful API to customize experiments • Access to low-level details • Integrated JS database • Fast and highly scalable • Game Levels • Modular design (games, widgets, window) • Well-documented (and active Forum) • Integration with Amazon Mechanical Turk

66. Who Develops NodeGame? Me! (shakty) B. Vartok, L. Miserez, J. Dorrie, P.Kueng, Ewen Wang, and others. (thanks!)

67. It comes from Node.JS, in which it is implemented, particularly suited for real-time interactive games. Why Is It Called NodeGame? Why Node.JS ? Built on Chrome's V8 JavaScript engine, it uses an event-driven, non- blocking I/O model that makes it lightweight and efficient. Node.js' package ecosystem, npm, is the largest ecosystem of open source libraries in the world.

68.  JavaScript was developed in May 1995 by Brendan Eich for Netscape Communications Corp  Was created in 10 days in order to accommodate the Navigator 2.0 Beta release  Initially called Mocha, later renamed LiveScript in September, and later JavaScript in the same month JavaScript

69.  Microsoft introduced JScript as reverse-engineered implementation of Netscape's JavaScript in 1996 in Internet Explorer 3  In 1996 Netscape submitted JavaScript to European Computer Manufacturers Association (ECMA) to create and industry standard  In 1997 ECMAScript was released  Between 1997 and 2009 5 standard have been released.  July 2015 ECMASCRIPT V6 released. JavaScript

70.  Node.JS was invented in 2009 by Ryan Dahl and other developers working at Joyent  Combination of Google's V8 JavaScript engine, an event loop, and a low-level I/O API  npm, the node package manager, in 2011  Versions: 0.10, 0.12, 4.0 … 10.0! Node.JS

71. JavaScript is #1 Language on Github http://githut.info

72. Hands On 1 Go to Google.com and open the JavaScript console of your browser: (ctrl+shift+I or Right Click/Inspect Element)

73. Some methods are immediately available to manipulate the page elements: Hands On 1 // Fetches the HTML element with given id. var a = document.getElementById("hplogo"); // Changes one of its attributes. a.src = "https://upload.wikimedia.org/wikipedia/commons/e/e9/Bing_logo.svg"; // Defines an onclick event-handler. a.onclick = function() { // Redirect to a new page. window.location.href = "http://bing.com"; };

74. var a = 1; // numeric var b = 'Hello world!’; // string var c = { key: 'value’ }; // object var d = [ 'value1’, “value2”, 3 ]; // array var e = function(p) { return p+1; }; // function  Variables are loosely typed  Variables are scoped within the function in which they are declared  Functions and array are objects  All the variables declared above are global Variable Declaration in JS • ES6 introduced let const

75. How to make variables private ? Declare them inside an own function! This technique is called: closure function foo(value) { var a = value; } foo(10); console.log(a); // undefined Variable Declaration in JS

76. Self-executing anonymous function (function(value) { var a = value; })(10); console.log(a); // undefined Variable Declaration in JS

77. Self-executing anonymous function (function(value) { var a = value; })(10); console.log(a); // undefined Variable Declaration in JS What is an advantage of closures?

78. Self-executing anonymous function (function(value) { var a = value; })(10); console.log(a); // undefined Variable Declaration in JS What is an advantage of closures? • They do not pollute the global space • Needed when you create functions in a loop

79. Variable Declaration in JS function foo() { var a = 1; } foo(); console.log(a); // undefined What happens is we do not use the var keyword?

80. Variable Declaration in JS function foo() { var a = 1; } foo(); console.log(a); // undefined JS will try to access the global variable a What happens is we do not use the var keyword?

81. Variable Declaration in JS function foo() { var a = 1; } foo(); console.log(a); // undefined JS will try to access the global variable a What if there is no global variable a? What happens is we do not use the var keyword?

82. Variable Declaration in JS function foo() { var a = 1; } foo(); console.log(a); // undefined 1 What happens is we do not use the var keyword? JS will try to access the global variable a What if there is no global variable a? Variable leaking into the global scope

83. Avoid Programming Mistakes There are ways to avoid these types of mistakes in JS  Use a linting tool: JSHint/JSLint/ESLint/JSCS https://www.sitepoint.com/comparison-javascript-linting-tools  JS Strict Mode “use strict”;

84.  Open the Node.JS terminal and launch nodeGame Hands On 2: nodeGame

85.  Open the Node.JS terminal and launch nodeGame Hands On 2: nodeGame cd = change directory (cd .. = dir up) ls = list node = type to launch Node.JS programs Hint! Use TAB to autocomplete the name of files

86.  Open the Node.JS terminal and launch nodeGame  Take a look at the different components of nodeGame  Home Page, Game Interface, Requirement Room, Waiting Room, Game, EndScreen Widget, The Monitor Interface, Get Saved Data Hands On 2: nodeGame

87. Configuration and game files are separated in different folder Game’s Anatomy

88. Configuration and game files are separated in different folder Game’s Anatomy • Define treatments • Define the sequence of stages and steps of the experiment • Implement the rules of interactions among participants • General setup • Define the graphical contents of the game • Define the rules about how to assign treatments to game rooms, and participants to treatments

89. Hands On 3: Folder waitroom/ // Display buttons to play with bots and select treatment ALLOW_PLAY_WITH_BOTS: true ALLOW_SELECT_TREATMENT: true // How many clients must connect before groups are formed POOL_SIZE: 2 // The size of each group GROUP_SIZE: 2 Waiting-Room-v4  Change waiting room options  Start 2 groups simultaneously with random treatment assignment (hint: POOL_SIZE),  Remove Select Treatments button

90. Folder game/ Define the sequence of stages and steps of the experiment Define game variables and group them into treatments

91. Stages Definition stager .next('id_of_stage’) stager .step(‘id_of_first_step_within_stage’) .step(‘id_of_second_step_within_stage’) stager .repeat('stage_to_repeat’, 3) We use the stager API to define the sequence A sequence contains stages, and stages contain steps Stager-API-v4

92. Stages Definition Game Sequence Code Snippet stager.stage("instructions") .step("instructions_1") .step("instructions_2") .step("instructions_3"); stager.stage("quiz"); stager.repeat("game", 3) stager.stage("questionnaire"); stager.extendStage("game", { steps: [ "offer", "respond", "display_results" ] }); Stager-API-v4

93. Hands On 4 // Skip stages from the sequence. stager.skip('precache'); stager.skip('selectLanguage'); stager.skip('quiz'); stager.skip('instructions'); stager.skip('mood');  Skip some stages of the game sequence (very useful for debugging)

94. Settings and Treatments Settings-and-Treatments-v4

95. Settings and Treatments Settings-and-Treatments-v4 Multiline Comments Game variables (these two are game-specific) Node.JS makes the content of this object available outside of the file

96. Settings and Treatments Settings-and-Treatments-v4 • Variable TIMER is read by nodeGame’s engine • It defines the max duration (in milliseconds) of each step • The names of the properties are the names of the steps in the sequence

97. Settings and Treatments Settings-and-Treatments-v4 • Variable treatments is read by nodeGame’s engine • It is an object indexed by the names of the treatments • Each treatment contains all the variables defined outside of the treatments object, plus what it is defined inside

98. Hands On 5 // Change the number of repetition to 1. REPEAT: 1, • Change the number of repetition of the ultimatum stage

99. Hands On 5 // Change the number of repetition to 1. REPEAT: 1, module.exports = function(stager, settings) { stager .next('precache') .next('selectLanguage') .next('instructions') .next('quiz') .next('mood') .repeat('ultimatum', settings.REPEAT) .next('questionnaire') .next('endgame') .gameover();

100. Hands On 6: Create a New Game cd bin node nodegame create-game gamename  Open a terminal and start the utility to create new games  Choose a name and follow instructions on screen

101. Hands On 6: Create a New Game cd bin node nodegame create-game gamename  Open a terminal and start the utility to create new games  Choose a name and follow instructions on screen • Personalize the name of the game as you wish

102. Hands On 6: Create a New Game  Open a terminal and start the utility to create new games  Choose a name and follow instructions on screen

103. Folder game/client_types/ • Implement the sequence • The same sequence can be implemented differently, depending by who is playing or where the code is going to be executed Human, web browser Computer (orchestrator), server

104. Client Type player.js  Contains code that will be executed on the client (web browser)  Has access to all standard JS objects AND nodeGame client API  However, it is “compiled” on the server, so it also has access to some server objects

105. Information Flow for Client Type • Decide a treatment and • Load a client type with correct settings Read 3 1 2 Read 1 23

106. How To Implement a Sequence  Use Stager API stager.[method]  To initialize game stager.setOnInit(func);  To add properties to steps and stages stager.extendStep("stepId", { property: value }); stager.extendStage(…)  frame  What to show (interface)  cb  How to handle user clicks  … and many others

107.  Implement Game Sequence: change contents of header Hands On 6

108. Header • It is created at initialization • It stays throughout the whole game

109. Header Widget Widget Widget • It is created at initialization • It stays throughout the whole game • Widgets such as timers, stage counters, and done button are generally added here

110. Frame

111. Frame • It is created at initialization • Its content is updated at every step

112. Hands On 7 module.exports = function(treatmentName, settings, stager, setup, gameRoom) { stager.setOnInit(function() { // Initialize the client. var header, frame; header = W.generateHeader(); frame = W.generateFrame(); // … • Anonymous initialization function passed as parameter: it will be executed on the client • Variable W (stands for Window) contains methods to manipulate the interface of the game • Function setOnInit is executed on the server • It adds a property to the stager • At the end, the stager is sent to the client  Implement Game Sequence: change contents of header

113. Hands On 7 • Variable node contains the entire nodeGame client API • Variable node.game contains all game-related methods and objects, including sequence, database, player list, etc. • You can add your own variables to node.game • Variable node.widgets contains methods to create and manipulate widgets • Widgets are reusable code portions, usually something visible on the interface node.game.visualTimer = node.widgets.append('VisualTimer', header);  Implement Game Sequence: change contents of header

114.  Implement Game Sequence: Change contents of frame  Go inside folder public/  Copy public/instructions.html into public/instructions_mod.html  Make a change to the HTML content Hands On 8 A

115.  Implement Game Sequence: Change contents of frame  Rename to instructions_mod.htm  Extend instruction step to load new page Hands On 8 B stager.extendStep('instructions', { frame: 'instructions_mod.htm' }); • Method extendStep adds/overwrites a property into a specific step • Property frame controls which page is loaded from public/ • Property frame can also be a function or an object • Property frame could also load a template from views/

116.  Executed immediately after the frame has fully loaded  The scope is the main body (not the frame, which must be accessed via W) 1. Bind page element to actions (W.gid, W.setInnerHTML, onclick) 2. Send messages to server or other players (node.say) 3. Handle incoming messages from server or other players (node.on.data) 4. Decide when the step is finished (node.done) Step Callbacks Step-Callback-Functions-v4 stager.extendStep('instructions', { cb: function() { // Do something. } });

117. Step Callbacks Step-Callback-Functions-v4 node.on.data("offer", function(msg) { // Display offer using W. // msg.data = 1 W.setInnerHTML("decision", "The dictator offered: " + msg.data); // End step. node.done(); }); var submit = W.getElementById("submitOffer"); var v = W.gid("offer").value; // 1 submit.onclick = function() { // Send offer to server, it will store it // and forward it to observer. node.done({ offer: v }); }; I say 99 for me and 1 for you. Dictator Observer

118. Step Callbacks Step-Callback-Functions-v4 var submit = W.getElementById("submitOffer"); var v = W.gid("offer").value; // 1 submit.onclick = function() { // Send offer to server, it will store it // and forward it to observer. node.done({ offer: v }); // Send offer to observer. node.say("offer", "partner_id", v); }; I say 99 for me and 1 for you. Dictator Observer • This is what server does, we will see it later in details node.on.data("offer", function(msg) { // Display offer using W. // msg.data = 1 W.setInnerHTML("decision", "The dictator offered: " + msg.data); // End step. node.done(); });

119. Step Callbacks Step-Callback-Functions-v4 var submit = W.getElementById("submitOffer"); var v = W.gid("offer").value; // 1 submit.onclick = function() { // Send offer to server, it will store it // and forward it to observer. node.done({ offer: v }); }; I say 99 for me and 1 for you. Dictator Observer • W.gid = W.getElementById • Returns HTML element from frame node.on.data("offer", function(msg) { // Display offer using W. // msg.data = 1 W.setInnerHTML("decision", "The dictator offered: " + msg.data); // End step. node.done(); });

120. Step Callbacks Step-Callback-Functions-v4 var submit = W.getElementById("submitOffer"); var v = W.gid("offer").value; // 1 submit.onclick = function() { // Send offer to server, it will store it // and forward it to observer. node.done({ offer: v }); }; I say 99 for me and 1 for you. Dictator Observer • W.gid = W.getElementById • Returns HTML element from frame When both players are DONE the game advances to next step node.on.data("offer", function(msg) { // Display offer using W. // msg.data = 1 W.setInnerHTML("decision", "The dictator offered: " + msg.data); // End step. node.done(); });

121.  Modify game step  Observer replies a message to the split to approve or disapprove (need to change HTML also) Hands On 9

122.  Modify game step  Observer replies a message to the split to approve or disapprove (need to change HTML also) Hands On 9 <div id="observer" style="display: none"> <h4 id="waitingFor"> Waiting for the decision of the dictator<span id="dots"></span> </h4> <span id="decision"></span> <p style="display: none" id="reply_p"> Tell the dictator what you think <input type="text" id="observer_reply" /> <button id="submit_reply">Submit</button> </p> <br/> </div>

123. Hands On 9 Observer node.on.data("offer", function(msg) { // Display offer using W. W.setInnerHTML("decision", "The dictator offered: " + msg.data); // Show hidden part. W.show("reply_p"); // Add listener that sends the text back to dictator. W.gid("submit_reply").onclick = function() { // Say message directly to dictator. node.say("reply", "dictator_id", W.gid("observer_reply"); // End step. node.done(); }); });

124. Hands On 9 Dictator submit.onclick = function() { // We send offer directly to observer. node.say("offer", "partner_id", v); // End step, // store offer in server. node.on.data("reply", function(msg) { W.writeln("Your partner thinks: " + msg.data || "nothing"); node.timer.randomDone({ offer: v }); }); };

125. Hands On 9 Dictator submit.onclick = function() { // We send offer directly to observer. node.say("offer", "partner_id", v); // End step, // store offer in server. node.on.data("reply", function(msg) { W.writeln("Your partner thinks: " + msg.data || "nothing"); node.timer.randomDone({ offer: v }); }); }; Same labels as in Observer: “offer” and “reply”

126. Hands On 9 Dictator submit.onclick = function() { // We send offer directly to observer. node.say("offer", "partner_id", v); // End step, // store offer in server. node.on.data("reply", function(msg) { W.writeln("Your partner thinks: " + msg.data || "nothing"); node.timer.randomDone({ offer: v }); }); }; Same labels as in Observer: “offer” and “reply” Randomly ends step

127. Hands On 9 Dictator submit.onclick = function() { // We send offer directly to observer. node.say("offer", "partner_id", v); // End step, // store offer in server. node.on.data("reply", function(msg) { W.writeln("Your partner thinks: " + msg.data || "nothing"); node.timer.randomDone({ offer: v }); }); }; Same labels as in Observer: “offer” and “reply” Randomly ends step Logical OR, shortcut to assign a default value, if the first term is falsy

128. https://dorey.github.io/JavaScript-Equality-Table/ === (strict equality)== (involves type conversion)

129. Folder game/client_types/ • Implement the sequence • The same sequence can be implemented differently, depending by who is playing or where the code is going to be executed Human, web browser Computer (orchestrator), server

130. Client Type logic.js  Contains code that will be executed on the server  Has access to the nodeGame client API AND to all server objects  What does it do?  It waits for all players to finish a step, if there is nothing better to do  Make sure the timers are respected  Can push/kick out non-responsive players (Matcher API)  It receives incoming messages from players and stores them (node.game.db)  It matches partners, and assigns roles

131. Client Type logic.js  Contains code that will be executed on the server  Has access to the nodeGame client API AND to all server objects  What does it do?  It waits for all players to finish a step, if there is nothing better to do  Make sure the timers are respected  Can push/kick out non-responsive players (Matcher API)  It receives incoming messages from players and stores them (node.game.memory)  It matches partners, and assigns roles

132. Client Type logic.js stager.extendStep('bidder', { matcher: { // Available roles. roles: [ 'BIDDER', 'RESPONDENT' ], // Each player is matched with a // partner and is assigned a role. match: 'roundrobin', // How to continue matching after // all combinations are exhausted cycle: 'repeat_invert' } }); Logic Client Type Matching-Roles-Partners-v4

133. Client Type logic.js stager.extendStep('bidder', { roles: { BIDDER: { frame: 'bidder.htm', cb: function() { // Bidder stuff. } }, RESPONDENT: { frame: 'resp.htm', cb: function() { // Respondent stuff. }}} }); Player Client Type stager.extendStep('bidder', { matcher: { // Available roles. roles: [ 'BIDDER', 'RESPONDENT' ], // Each player is matched with a // partner and is assigned a role. match: 'roundrobin', // How to continue matching after // all combinations are exhausted cycle: 'repeat_invert' } }); Logic Client Type Matching-Roles-Partners-v4

134. Client Type logic.js stager.extendStep('bidder', { roles: { BIDDER: { frame: 'bidder.htm', cb: function() { // Bidder stuff. } }, RESPONDENT: { frame: 'resp.htm', cb: function() { // Respondent stuff. }}} }); Player Client Type stager.extendStep('bidder', { matcher: { // Available roles. roles: [ 'BIDDER', 'RESPONDENT' ], // Each player is matched with a // partner and is assigned a role. match: 'roundrobin', // How to continue matching after // all combinations are exhausted cycle: 'repeat_invert' } }); Logic Client Type Logic lists roles Player implements Matching-Roles-Partners-v4

135. Memory Database node.game.memory Memory-Database-v4

136. Memory Database node.game.memory Memory-Database-v4 node.done({ offer: 1 }); { stage: { stage: 2, step: 1, round: 1 }, player: "id12345678", timestamp: 1480336721939 offer: 1, timeup: false, time: 4321, //From step done: true };

137. Memory Database node.game.memory Memory-Database-v4 // Set an object in memory explicitly. node.set({ foo: "bar" }); { stage: { stage: 2, step: 2, round: 1 }, player: "id12345678", timestamp: 148033845673 foo: "bar" , };

138. Memory Database Memory-Database-v4 var m = node.game.memory; // Lazy evaluation: add a bunch of statements // and evaluates all of them at the end. m.select("player", "=", "id12345678"); m.and("stage.step", "=", 1) .and("offer") // Must have property “offer” .fetch()[0]; // Returns array, get first element // Use indexes. // By sender. m.player["id12345678"].last(); // By stage. m.stage["2.1.1"].select("offer").fetch()[0];

139.  Explore this RunKit on selecting and fetching items from node.game.memory  https://runkit.com/shakty/gamedb Hands On 10

140. What’s Left Out  Levels  Widget Steps  Authorization  Get Help / Contribute  Dropout & Reconnections  Upload Your Game to the Cloud  Amazon Mechanical Turk

141. What More?  Technical Topics  Levels  Widget Steps  Authorization  Requirements  Get Help / Contribute  Dropout & Reconnections  Go Live  Upload Your Game to the Cloud  Amazon Mechanical Turk

142. Extend

143. Extend Sent to client as part nodegame-full.js

144. Extend Core components for the client Sent to client as part nodegame-full.js After editing a core component, rebuild nodegame-server/ public/javascripts/nodegame-full.js node launcher -b widgets,window (nodegame-client automatically rebuilt with –b)

145. Extend SASS CSS framework https://sass-lang.com/ Do changes to: public/css/stylesheets/nodegame.scss node launcher -b css Builds nodegame.css

146. Amazon Mechanical Turk (AMT)

147. AMT 1. Upload/install nodeGame + game to the cloud e.g., Digital Ocean or Heroku cloud: https://github.com/nodeGame/nodegame- heroku 2. Create a new HIT on Amazon Mechanical Turk (AMT), e.g. using the requester interface or other service API. 3. Use the nodegame-mturk package https://github.com/nodeGame/nodegame-mturk/ to handle: - Connection and authentication from AMT HIT page to nodeGame server. - Opening/expiring/extending HIT while running it. - Payments, bonuses and qualifications after HIT is done.

148. AMT HIT Page Check the boilerplate in the public folder of the nodegame-mturk package

149. AMT HIT Page • The important things: • Inform about the task (in general terms) • Make it really clear it is a SYNCHRONOUS task • Explain expected duration and payoff structure • Explain technical requirements, if any • Optionally, have a Trouble Ticket • Exit code input

150. AMT HIT Page • Display link to game server only after acceptance • Get turker information, upon acceptance, from the page and send it to the server along the query string or via a separate AJAX request HITID = turkGetParam('hitId’); WORKER_ID = turkGetParam('workerId’); ASSIGNMENT_ID = turkGetParam('assignmentId'); • These functions are not officially supported by AMT, so they might disappear in the future • These info is stored in the page as hidden input

151. AMT Issues for Synchronous Games 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many 3. Waiting time in the waiting room should be limited and homogenous across participants 4. Demographics should be stratified as much as possible 5. Participants should not re-take the same experiment again 6. Dropouts rate (non-completing the HIT) significantly affects the whole experiment, increasing costs $$ and time to complete it

152. AMT Issues for Synchronous Games 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform Pay minimum wage (at least). Make bonus dependent on performance

153. AMT Issues for Synchronous Games 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many Rule of thumb: 2.8 times desired number of participants Expire HIT as soon as desired number of participant entered the experimental pipeline (+small buffer) 3x

154. AMT Issues for Synchronous Games 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many Rule of thumb: 2.8 times desired number of participants Expire HIT as soon as desired number of participant entered the experimental pipeline (+small buffer) Beware of “HIT Grabbers”: use scripts to accept HITs very quickly, and then take it when they have time; Often, they only see the title, so make sure to write in title something like: “MUST TAKE IMMEDIATELY”

155. AMT Issues for Synchronous Games 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many 3. Waiting time in the waiting room should be limited and homogenous across participants Dispose participants who waited too long (compensated)

156. AMT Issues for Synchronous Games 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many 3. Waiting time in the waiting room should be limited and homogenous across participants 4. Demographics should be stratified as much as possible Waiting room can group players based on features Collect some demographics features in a pre-experimental part (level 1) Start 2 (or more) groups at the same time

157. AMT Issues for Synchronous Games 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many 3. Waiting time in the waiting room should be limited and homogenous across participants 4. Demographics should be stratified as much as possible 5. Participants should not re-take the same experiment again Create one HIT (no group) with 3x rule Keep track of turker ids (nodegame-mturk) Assign qualification via AMT API (nodegame-mturk)

158. AMT Issues for Synchronous Games 1. Not a survey or classification task, need to have DIFFERENT people at the same time on your platform 2. Btw, not too few, not too many 3. Waiting time in the waiting room should be limited and homogenous across participants 4. Demographics should be stratified as much as possible 5. Participants should not re-take the same experiment again 6. Dropouts rate (non-completing the HIT) signigicantly affects the whole experiment, increasing costs $$ and time to complete it • Hey, we talked about this already! (see previous slides)

159. AMT Dropouts and Relative Cumulative Payoff • Tested combinations of demographics and other game characteristics • The only significant difference is between dropouts and participants who stay is their relative payoff.

160. Keep a Good Requester Score https://turkerview.com https://turkopticon.ucsd.edu (will be discontinued)

161. External Validity

162. External Validity External validity: The extent to which the results of a study can be generalized or extended to others.

163. External Validity

164. External Validity Galileo's hand-made drawing of the moon surface. Sidereus Nuncius (1610)

165. External Validity Galileo's hand-made drawing of the moon surface. Sidereus Nuncius (1610) Telescope: From Land to Sky

166. External Validity Galileo's hand-made drawing of the moon surface. Sidereus Nuncius (1610) Telescope: From Land to Sky Experiments: From Lab to Real Life

167. JS Exercises

168.  To access variables between files must be explicitly exported and required // File A. var a = 1; module.exports = a; // File B. var a = require('./A.js') How to export multiple variables? // File A. module.exports = { a: 1, b: 2 }; // File B. var a = require('./A.js').a; Node.JS Export Variables

169.  JS functions are objects  Can be passed as parameters to other functions  Treat differently different input parameters  Can have properties  Can be executed with different contexts  Two types exists: declaration and expressions  Always remember the scope of creation JS Functions

170. JS Functions Create an array of 10 functions returning the index in which they are inserted in the array. Is the code below correct? var i, len, obj; len = 10, obj = []; for (i = 0 ; i < len ; i++) { obj[i] = function() { return i; } }

171. JS Functions Create an array of 10 functions returning the index in which they are inserted in the array. Is the code below correct? var i, len, obj; len = 10, obj = []; for (i = 0 ; i < len ; i++) { obj[i] = function() { return i; } } • Creates 10 functions all returning the value 10, because they are all referencing variable i, which has value 10 when the loop ends

172. JS Functions Create an array of 10 functions returning the index in which they are inserted in the array. You need a closure! for (i = 0 ; i < len ; i++) { obj[i] = (function(i) { return function() { return i; } })(i); }

173. JS Function Input Parameters // Internally modifies input. function a(obj, num, str) { obj.a = 10; num = 1; str = 'a'; } var obj = {}, num = 0, str = ''; a(obj, num, str); console.log(obj); console.log(num); console.log(str); ?

174. JS Function Input Parameters // Internally modifies input. function a(obj, num, str) { obj.a = 10; num = 1; str = 'a'; } var obj = {}, num = 0, str = ''; a(obj, num, str); console.log(obj); // { a: 10 } console.log(num); // 0 console.log(str); // ''

175. JS Function Input Parameters console.log(obj); // { a: 10 } console.log(num); // 0 console.log(str); // ''  Objects are passed as reference, while numbers and strings as copy  Modifying a copy does not affect the value outside the function, modifying the reference does.

176. JS this function a() { return this.a; } a(); // undefined // Create a context. var foo = { a: 1}; // call and apply set the this value a.call(foo, 1, 2, 3); // 1; a.apply(foo, [1, 2, 3]); // 1;  The value of this is dynamic in JavaScript  It is determined when function is called, not when it is declared.

177. JS this What will be printed to console? function A() { this.a = 1; (function() { console.log(this.a); })(); } // Create a new object. var a = new A();

178. JS this What will be printed to console? function A() { this.a = 1; (function() { console.log(this.a); })(); } // Create a new object. var a = new A(); • It will print undefined • How to adapt to print 1?

179. JS this What will be printed to console? function A() { this.a = 1; var that = this; (function() { console.log(that.a); })(); } • It will print 1 • The reference to this is stored in another variable

180. JS this What will be printed to console? function A() { this.a = 1; var that = this; (function() { console.log(that.a); })(); } • It will print 1 • The reference to this is stored in another variable Why is “this” solution better than call or apply?

181. JS this What will be printed to console? function A() { this.a = 1; var that = this; (function() { console.log(that.a); })(); } • It will print 1 • The reference to this is stored in another variable because you can reuse that multiple times! Why is “this” solution better than call or apply?

182.  In JS, each object inherits methods and properties from a parent object called prototype  In turn, also the prototype object can have an own prototype, and all the properties are are inherited through the prototype chain  It is possible to extend an object by extending its prototype or the prototype of its prototype...  This pattern is called prototypical inheritance, and it is extremely powerful – if well understood JS Inheritance

183. function A() { this.a = 1; } A.prototype.printA = function() { console.log(this.a); } var a = new A(); a.printA(); // 1; JS Prototypical Inheritance

184. function A() { this.a = 1; } A.prototype.printA = function() { console.log(this.a); } var a = new A(); a.printA(); // 1; JS Prototypical Inheritance  What is the difference with defining define the method printA inside the constructor? (this.printA = function … )

185. // Create a second object. var a2 = new A(); // Assign property to method printA. a2.printA.foo = 1; // Property is also on object a // because it is the prototype // to be modified. console.log(a.print.foo); // 1 JS Prototypical Inheritance

186. JS Prototypical Inheritance  Extending the prototype of the function leads to faster object creations because all the methods are already existing and only need to be referenced instead of being created  However sometimes you need to have a clear separation between methods of objects of the same class

187. Looping in Objects (For In)  Javascript does not guarantee clear separation between variables of the prototype and of the object itself  Therefore, when looping through the properties of an object it is necessary to invoke the method .hasOwnProperty

188. Looping in Objects (For In) var triangle = { a: 1, b: 2, c: 3 }; function ColoredTriangle() { this.color = "red"; } ColoredTriangle.prototype = triangle; var obj = new ColoredTriangle(); for (var prop in obj) { if (obj.hasOwnProperty(prop)) { console.log(obj[prop]); } }

189. Linting Tool: JSHint Try it on : http://jshint.com

190.  Really easy to install and use  Install with the node package manager  What is the option -g doing? Linting Tool: JSHint  You might be interested in one of the plugins for editors (vim, emacs, sublime...) jshint your_file.js --show-non-errors npm install -g jshint

191. "use strict"; (function(value) { a = value; })(10) a = value; ^ ReferenceError: a is not defined at /home/stefano/Documents/myworkshops/nodegame_CESS/scripts/A.js:4:7 at Object.<anonymous> (/home/stefano/Documents/myworkshops/nodegame_CESS/scripts/A.js:5:3) at Module._compile (module.js:460:26) at Object.Module._extensions..js (module.js:478:10) at Module.load (module.js:355:32) at Function.Module._load (module.js:310:12) at Function.Module.runMain (module.js:501:10) at startup (node.js:129:16) at node.js:814:3 Strict Mode on

Synchronous Online Experiments with NodeGame

futurdorko

Synchronous Online Experiments with NodeGame