Metacognition in ELEKTRA


Published on

How metacognition was trained in ELEKTRA project of game based learning of physics

Published in: Education, Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Metacognition in ELEKTRA

  1. 1. Metacognition in ELEKTRA∗ Jean-Loup Castaigne LabSET — IFRES — University of Liege January 24, 20071 Why certitude degrees in learning?In an evaluation based only on identifying correct and incorrect answers thereis little information available for both teacher and learner other than right orwrong. Adding certitude degree to evaluation brings a new dimension, allowingmore precise conclusions. For instance what conclusion should raise a teacher when 95 % of his studentssucceed answering a question? What other conclusion if those 95 % successfulstudents only have a mean confidence of 10% in their correct answer? Maybethe teacher will consider this task as not achieved by his students despite 95%of correct answer. The way a test will be corrected will also be different if a majority of studentsare failing a question with a mean certitude degree of 10% for that question orwith a mean certitude degree of 90%. In the first case doubt can be expressedregarding the question itself. In the second case it appears students are confidentin their believe which is an incorrect information. This last situation may beconsidered as dangerous as students will trust what they think they know. Learning does not move someone from total ignorance to perfect knowledge.Often people will already have some knowledge or representation about what isteached, even if these representation or knowledge might be erroneous. Evalua-tion should not be limited to either knowledge or correct answer and ignoranceor incorrect answer. Information is what reduces doubt. “Partial informationexists. To detect it is necessary and feasible” (De Finetti, 1965)1 . certitudedegrees give a teacher a more detailed image of the knowledge of a student:they reveal partial information. ∗ ELEKTRA : “Enhanced Learning Experience and Knowledge TRAnsfer” is an Europeanresearch project that aims at revolutionising technology-enhanced learning. 1 De Finetti, B. (1965). Methods of discriminating levels of partial knowledge concerninga test item. Bristish Journal of Math. & Statist. Psychol., 18, 87-123. 1
  2. 2. 2 What is metacognition?2.1 Metacognition in generalMetacognition has been defined by Flavell (1976)2 : “Metacognition refers toone’s knowledge concerning one’s own cognitive processes or anything relatedto them, e.g. , the learning-relevant properties of information or data” . Manyinterpretations were made from the original definition of metacognition butmetacognition is agreed to involve both • knowledge about one’s own knowledge and • knowledge about one’s own cognitive processes.Metacognition has to do with the active monitoring and regulation of cognitiveprocesses. Metacognition is also relevant to work on cognitive styles and learningstrategies in so far as the individual has some awareness of their thinking orlearning processes. Within metacognition “knowledge about memory” is defined as metamem-ory (Flavell & Wellman, 1977)3 . Metamemory itself is declined into declar-ative metamemory (e.g. knowledge about strategy variables) and proceduralmetamemory with its two components: the monitoring component and the con-trol and self-regulated component. Metacognition • Metamemory 1. Declarative metamemory 2. Procedural metamemory (a) monitoring component (b) control and self-regulated component Schraw and Dennison (1994)4 separate the knowledge and the regulation ofcognition, each one being defined into different parts. 1. The knowledge of cognition is composed of: (a) Declarative knowledge: knowledge about one’s skills, intellectual re- sources, and abilities as a learner. (b) Procedural knowledge: knowledge about how to implement learning procedures (e.g., strategies). (c) Conditional knowledge: knowledge about when and why to use learn- ing procedures. 2. The regulation of cognition refers to: 2 Flavell, J.H. (1976). Metacognitive aspects of problem solving. In Resnick (Ed.), Thenature of intelligence. (pp. 231-235). New Jersey : Lawrence Erlbaum Associates. 3 Flavell, J.H., & Wellman, H. (1977). Metamemory. In R. V. Kail Jr. & J. Hagen (Eds.),Perspectives on the development of memory and cognition (pp. 3–33). Hillsdale, NJ: Erlbaum 4 Schraw, Gregory, and Rayne Sperling Dennison, 1994. Assessing Metacognitive Aware-ness? In Contemporary Educational Psychology 19, 460-475. 2
  3. 3. (a) Planning: planning, goal setting, and allocating resources prior to learning. (b) Information management: skills and strategy sequences used on-line to process information more efficiently (e.g., organizing, elaborating, summarizing, selective focusing). (c) Monitoring: assessment of one’s learning or strategy use. (d) Debugging: strategies used to correct comprehension and perfor- mance errors. (e) Evaluation: analysis of performance and strategy effectiveness after a learning episode.The EARLI 5 metacognition special interest group often refers to the maincomponents of metacognitive monitoring and metacognitive control that wasdefined from the works of Nelson (1996) 6 and Nelson & Narens (1994) 7 – seefigure 1. Figure 1: Main components of metacognitive monitoring and metacognitive control definedby Nelson (1996) and Nelson & Narens (1994) Leclercq & Poumay (2004) 8 propose an “operational definition” of metacog-nition: “Observable (or not) judgments, analysis and /or regulations effectuated 5 EARLI: European Association for Research on Learning and Instruction 6 Nelson, T. O. (1996). Gamma is a measure of the accuracy of predicting performanceon one item relative to another item, not of the absolute performance on an individual item.Applied Cognitive Psychology, 10, 257–260. 7 Nelson, T. O., & Narens, L. (1994). Why investigate metacognition? In J. Metcalfe &A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 1–25). Cambridge, MA:Bradford Books. 8 personal translation of “les jugements, analyses et/ou r´gulations observables effectu´s par e el’apprenant sur ses propres performances (processus ou produits d’apprentissages), ceci dansdes situations de PRE, PER ou POST performance” from Leclercq, D. & Poumay, M. (2004).Une d´finition op´rationnelle de la m´tacognition et ses mises en oeuvre. 21e Conf´rence e e e einternationale de l’AIPU, Morocco: Marrakech. 3
  4. 4. by a learner on his/her own performances (learning processes or products), insituations of pre, per or post performance (mainly testing or learning).”2.2 Certitude degrees and metacognitionEbel (1965)9 defined confidence weighting as “a special mode of responding toobjective test items, and special mode of scoring those responses. In generalterms, the examinee is asked to indicate not only what he believes to be thecorrect answer to a question, but also how certain he is of the correctness of hisanswer”. certitude degrees are confidence weighting. In Nelson & Narens’s maincomponents of metacognitive monitoring and metacognitive control certitudedegrees with testing take place in the monitoring side, retrieval of knowledgecategory, in the Confidence in retrieved answers box. In Leclercq & Poumay’soperational definition certitude degree with testing will be observable judgmentseffectuated by a learner on his/her own performances (products), in situationsof per performance (testing).2.3 Metacognition in the demonstratorOne can ask if metacognition is usable as such for teenagers 14 years old. Inbrief, publications that were founded consider tested components of metacogni-tive monitoring and control not significantly different between adults and as lowas 10 years old children. Below the age of 10, components significance evolveswith age. Flavell, Friedrichs & Hoyt (1970)10 have showed significant correla-tion between predicted and actual memory span in children from the 4th gradeand no significant correlation was founded below that age, including at nurseryand kindergarden. Young children show unrealistic performances prediction.Schneider (2006)11 offers three theoretical reasons: 1) insufficient metacognitiveknowledge (young children not monitoring their memory activities or lacking inunderstanding about the interplay among relevant factors), 2) predominance ofwishful thinking over expectations thus predictions reflecting their desires) and3) belief in the power of effort. Duell (1986)12 confirm that as children get olderthey demonstrate more awareness of their thinking processes. In the Elektra demonstrator the use of certitude degrees so far is neitherrelevant of a “judgment of learning” nor a confidence in retrieved answer. Judg-ments of learning (jols), are defined as judgments that “occur during or afteracquisition and are predictions about future test performance on recently stud-ied items” (Nelson & Narens, 1994). 9 Ebel, R. (1965). Confidence weighting and test reliability. Journal of Educational Mea-surement, pages 49–57. 10 Flavell, J. H., Friedrichs, A., & Hoyt, J. (1970). Developmental changes in memorizationprocesses. Cognitive Psychology, 1, 324–340. 11 Schneider 2006, The development of metacognition in childhood and adolescence, keyn-note of the 2nd international biennial conference of the Metacognition Special Interest Group(SIG 16) of the European Association of Research in Learning and Instruction (EARLI), July2006, Cambridge 12 Duell, O.K. (1986). Metacognitive skills. In G. Phye & T. Andre (Eds.), CognitiveClassroom Learning. Orlando, FL: Academic Press. 4
  5. 5. 2.3.1 What is a JOL?Imagine a learner studying pairs of linked words13 such as game–elektra.The learner receives the instructions that each pair should be learned so thatsubsequently when prompted with the cue (e.g., “game–?”), the learner wouldrecall the target (e.g., “elektra”). An interval of time elapses between thetermination of studying a given item and the onset of the jol for that item.This interval could be • extremely brief (e.g., as in an immediate jol that occurs as close in time as possible to the offset of the studied item) or • could be delayed for a more lengthy amount of time (as in a delayed jol) filled with other activity and/or other to-be-learned items.Then the first jol occurs, prompted by a cue that usually consisted of onlythe cue from the studied item (e.g., “game–?”). The learner generates a jolby choosing the predicted likelihood (e.g., on a Likert-type rating scale or on ascale ranging from 0% to 100% in steps of 20%) of remembering the item on aneventual criterion test that might occur later (e.g., 10–20 minutes later). Otherpairs of words are studied, and jols are made for each of them, until everypair had been studied and had received a jol. Finally, following an intervalof perhaps 10–20 minutes (filled with other items) from the time of studying agiven item, the person received the eventual memory test on that item. Thememory test is self-paced, usually asking the person to recall the target whenprompted by the cue (e.g., when prompted by “game–?”, the person attemptto recall “elektra”). With this example we demonstrate that what is asked in the demonstratoris neither true jols nor confidence in retrieved answers.2.3.2 JOL or confidence in retrieved answerWe suggest that metacognition in elektra is in a first time limited to the useof certitude degrees with the objective of improving the learner metacognitiveself assessment. Kruger & Dunning (1999) 14 showed that “learners whose skillsor knowledge bases are weak in a particular area tend to overestimate theirability in that area”. In other words, they don’t know enough to recognizethat they lack sufficient knowledge for accurate self-assessment. In contrast,learners whose knowledge or skills are strong may underestimate their ability.These high-ability learners don’t recognize the extent of their knowledge or skills.Kruger and Dunning’s research also shows that “it is possible to teach learners atall ability levels to assess their own performance more accurately”. It is in thatperspective that certitude degrees will be implemented in elektra. With theuse of certitude degree in elektra we can at least give the learner/player somefeedback about their cognitive strategies with the confidence in their retrievedanswers. 13 adapted form Nelson, Narens & Dunlosky (2004). A Revised Methodology for Research onMetamemory: Pre-judgment Recall And Monitoring (PRAM). Psychological Methods, Vol.9, No. 1, 53–69. 14 Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties inrecognizing one’s own incompetence lead to inflated self-assessment. Journal of Personalityand Social Psychology, 77, 1121-1134. 5
  6. 6. Consolidated Research Report M1-6 specifies that metacognition within elek-tra should: • support and facilitate the development of metacognitive abilities and strate- gies and • support the assessment of knowledge.We recommend to be very cautious in identifying what is assessment of knowl-edge and what is support to metacognition. Feedbacks should always be veryclear in regard to what they refer to, either cognition–knowledge or metacognition–knowledge about knowledge. With elektra we can provide teachers a tool tobegin a reflection with their students about metacognition. The distinctionbetween cognitive and metacognitive strategies is important, partly because itgives some indication of which strategies are the most crucial in determiningthe effectiveness of learning. It seems that “metacognitive strategies, that allowstudents to plan, control, and evaluate their learning, have the most central roleto play in this respect, rather than those that merely maximize interaction andinput . . . Thus the ability to choose and evaluate one’s strategies is of centralimportance” (Graham, 1997) 15 . Learners who are metacognitively aware knowwhat to do when they don’t know what to do; that is, they have strategies forfinding out or figuring out what they need to do. “The use of metacognitivestrategies ignites one’s thinking and can lead to more profound learning and im-proved performance, especially among learners who are struggling” (Anderson,2002) 16 .3 Metacognition in gamesPrakash (1999) 17 reports that he heard on the radio a sixth grader explainingwhat she was learning from playing the Stock Market Game. This game was anactivity designed to help children become familiar with how the stock marketfunctions. She said, “This game makes me think how to think”. What thisstatement reveals is that this young learner was beginning to understand thereal key to learning; she was engaged in metacognition using a game. Salda˜a (2004) 18 reports using a Master Mind c modified (order was not nimportant) to assess metacognitive processes use to solve the enigma. Thesolution could be found independently or with three levels of assistance 1. focusing on the metacognitive processes present in the task such as • planning aims / strategies, • supervision and control of aims / strategies, • revision of aims / strategies, 15 Graham, S. (1997). Effective language learning. Clevedon, England: Multilingual Mat-ters. 16 Neil J. Anderson, The Role of Metacognition in Second Language Teaching and Learning,Brigham Young University 17 Prakash, S. (Reporter). (1999, March 19). Market games [Radio series episode]. Allthings considered. Washington: National Public Radio. 18 Salda˜ a, D. (2004) Dynamic Master Mind: interactive use of a game for testing metacog- nnition. School Psychology International, Vol 25(4): 422-438. 6
  7. 7. • meta knowledge of possibilities and limitations • meta knowledge of tasks / strategies; 2. scaffolding each and every one of the selfregulatory steps included in the task; 3. modelling of the task solution processOn top of working metacognitive process this experiment also showed diversifi-cation in methods for adaptivity. Additional literature research is still under process. Please keep this docu-ment for private use within elektra.4 About the demonstrator, 2006 versionThese comments only cover some metacognitive aspects of the demonstrator.4.1 A surpriseWhen Galileo says that the most important thing is to first define what youwant to do, he is asking George to identify the objective. This is a metacognitiveactivity and looks like what is described by Salda˜a (2004) as “planning aims” nin his Master Mind c experiment. It is a pleasant surprise to see that theformulation of that question turned it into a metacognitive activity. Figure 2: Cut scene from the elektra demonstrator, 2006 7
  8. 8. 4.2 General comments 1. The asked question should not have been ‘Are you sure?” but something like “With what you know and what you have learned, how confident are you that you will succeed opening the door in this try?” Asked the way it was, it was confusing as we mentioned as soon as we had the opportunity to look at the demonstrator. But it was too late to change as the validation in ORT Marseilles had already begun. UG document about metacognition 19 mentioned that “As shown for example in the first validation results, the current formulation of the confidence rating is not understood by most individuals of the target group”. We argue that (a) the formulation of the question, especially that lack of “sure of what?” (b) the fact that the question was neither a jol nor a confidence in retrieved answer, (c) the lack of explanations related to the metacognitive aims and the absence of tutorial explaining how to use certitude degree, (d) the lack of training of students in general regarding expression of their confidence, (e) the use of verbal expression e.g., “sure” is confusing by itself and should not be used when retrieving certitude (see Shuford, Albert & Massengill (1966)20 recommendations for the use of subjective prob- abilities instead of verbal expressions) are all explanations of the confusion. As mentioned here above, in collect- ing jol Nelson & Narens themselves are using a probabilistic scale from 0% to 100% by steps of 20%. In the validation report21 ORT mentions that: “It seemed that most pupils didn’t really understand the meaning behind the confidence degree, or the questions that was asked.” We to- tally agree with their recommendation that suggest: “It seems that the question related to the confidence degree wasn’t very clear, it is therefore recommended to formulate the question in a clearer way (adapted to 13-14 years old pupils)”. To avoid confusion we propose section 7 some sugges- tion about what to explain in the tutorial. Shuford (1966) and Leclercq (198322 , 198623 , 199324 ) defined 4 rules when using certitude degrees: (a) the use of probabilistic value instead of words; (b) if points are awarded tariff should respect the decision theory; (c) feedbacks concerning the good or not good use of certitude degree provided to students; 19 Kickmeier-Rust, Albert & Linek (2007) A psychological perspective on metacognition andthe usage of confidence ratings. Draft for internal use of elektra consortium 20 Shuford, E. H., Albert, A., and Massengill, H. E. (1966). Admissible probability mea-surement procedures. Psychometrika, 31(2):125–45. 21 Hirschberg, G. (2007) DELIVERABLE Name: Phase 1 – Validation Report 22 Leclercq, D. (1983). Confidence marking, its use in testing. Postlethwaite, Choppin (eds)Evaluation in Education, Oxford : Pergamon, 1982, vol. 6, 2, pp. 161-287. 23 Leclercq, D. (1986). La conception des questions ` choix multiple, Bruxelles, Ed. Labor. a 24 Leclercq, D. (1993). Validity, reliability and acuity of self-assessment in educational test-ing. In Leclercq, D. and Bruno, J., editors, Item Banking : Interactive Testing and Self-Assessment, number F 112 in NATO ASI Series, pages 114–131. Berlin: Springer Verlag. 8
  9. 9. (d) students must be trained to use certitude degrees. The respect of these rules is mandatory for anyone willing to use certitude degrees and avoid confusion or deception. 2. Chosen certitude and chosen answer are not recorded into the game. Both teacher and learner should be able to view the results, success and failure, during tries and tasks. These data should influence the feedbacks. As such they can be part of micro-adaptivity. 3. The feedback should be more precise. Galileo might say something like: When I asked you how confident you were about succeeding this task, you told me you were XX% confident succeeding it. If (FAILURE) then If (chosen certitude degree > 50%) then You did not succeed. But you were very confident you would succeed. So you’ve made a bad assumption. You should be more careful in your estimations. Try again and be more modest with your confidence. else You did not succeed. But you confidence in your ability to succeed was low. So your assumption was good: you have failed but you knew it was risky so you associated doubt with your try. I congratulate you for that. Give it another try. end If else If (chosen certitude degree ≤ 50%) then You’ve succeeded. Congratulations. But your confidence in your ability was too low. You should be more confident in your estimations. else You’ve succeeded. Congratulations. And your estimation was a good assumption: you were sure and you’ve succeeded. That’s great. end If end IfNotice how feedbacks about cognition and metacognition are in different sen-tences as suggested in section 2.3.2, page 5. Knowledge feedbacks are clearlydifferent than support of metacognition. When a few tries have been made, thefeedback can even turn to be more precise. This will be described in the lastpart of this document.4.3 Experiment in world 2 and metacognitionWe also suggest that, in world 2, the player should be able to turn on and offthe flashlight. He should only be allowed to move the blinds and the screen 9
  10. 10. when the torch is turned off. After he has placed all the objects the way hewants, he will have to turn the flashlight on to see the results. Even if the player does not find the correct solution he will see the result ofwhat he built when turning the flashlight on. He will see it in 3D and will also seethe associated top view. When the player turns on the flashlight Galileo mightalso ask a question like “How sure are you that this construction will produce anarrow beam of light?” When turning it off Galileo will give a feedback (see theprevious algorithm). Instead of Galileo always asking the same question aboutconfidence, we suggest that the switch of the light might be something like off on with certitude degree of 0% 20% 40% 60% 80% 100% Table 1: Flashlight on–off switch enhanced with certitude degree4.4 The metal-wooden door and the blindsImagine that the player succeed placing the blinds and screen in LeS1.2a inworld 2 with a very good confidence. Then he/she fails in the real world withthe metal-wooden door. What kind of feedback can we give about what he/shehas learned in world 2? What kind of feedback can we give about his/herconfidence in succeeding the task based on what he/she has experimented inworld 2? We have no answer for these questions because the problem, from ourpoint of view, is that the metal-wooden door does not evaluate what might havebeen learned with the blinds and screen in world2. This seems impossible to bechanged. So we recommend not to use certitude degrees at the moment withthe opening of the metal–wooden door situation.5 Deeper with certitude degreeThe demonstrator is working on one single expression of the player’s jol. It isdifficult for him to gain any metacognitive knowledge with a single measurement.The only feedback we can actually give with the actual state of development ofthe demonstrator is a goodbad performance associated with a confidentnonconfident jol as showed in table 2. good bad performance performance confident Best Worst good knowledge wrong knowledge good confidence bad confidence non confident good knowledge lack of knowledge but but lack of confidence good confidence Table 2: Feedback class associated with the combination of true-false and confidence 10
  11. 11. 5.1 Theoretical backgroundThe following is based on the works of Hunt (1993), Jans & Leclercq (1999) andLeclercq (2003) and gives a much more detailed measure of knowledge with theassociation of a confidence chosen out of a scale of 6 degrees of certitude (seefigures 3 to 5, pages 11 et 12). All figures are spectral distributions of knowledge.On the left hand side are the incorrect answers. They are distributed by thelearner chosen confidence, ranking from left to right from 100% down to 0%. Inthe middle is the grey area of not answered questions. On the right hand sideare the correct answer, also distributed by confidence but ranking from left toright from 0% to 100%. Figure 3 showed Darwin Hunt (1993) 25 suggestion todistinguish between three types of knowledge situations in which a person canbe in relation to a piece of content : “misinformed, uninformed, informed”. Figure 3: Spectral distributions of Hunt’s three situations of knowledge (1993) These situations were redefined –see figure 4– by Jans & Leclercq (1999) 26as: “dangerous knowledge, unusable knowledge, usable knowledge”. Unhappy with the “unusable knowledge” unable to make the difference be-tween a student making a mistake with a low confidence and a student hav-ing a low confidence in a correct answer, Leclercq (2003) 27 divided it into“unawareness” and “mid knowledge” as showed in figure 5. This is an impor-tant distinction as mid knowledge needs working on metacognitive judgment, a 25 Hunt, D. (1993). Human self-assessment : Theory and application to learning and testing.In Leclercq, D. and Bruno, J., editors, Item Banking : Interactive Testing and Self-Assessment,volume F 112 of NATO ASI Series, pages 177–189. Berlin: Springer Verlag 26 Jans, V. and Leclercq, D. (1999) Mesurer l’effet de l’apprentissage ` l’aide de l’analyse aspectrale des performances. In Depover, C. and No¨l, B., editors, L’´valuation des e ecomp´tences et des processus cognitifs. Mod`les pratiques et contextes, pages 303-317. Brux- e eelles : De Boeck Universit´e 27 Leclercq, D. (2003). Un diagnostic cognitif et m´tacognitif au seuil de l’universit´. Le e eprojet MOHICAN men´ par les 9 universit´s de la Communaut´ Fran¸aise Wallonie Bruxelles. e e e cLi`ge : Editions de l’Universit´ de Li`ge. e e e 11
  12. 12. Figure 4: Spectral distribution of knowledge kinds by Jans & Leclercq (1999) based on Hunt’sthree situation of knowledgemetacognitive part, and unawareness needs to first work on knowledge, a cogni-tive part. Again, as mentioned in section2.3.2 page 5, do not confuse assessmentof knowledge and support of metacognition with certitude degrees. It is alreadyFigure 5: Leclercq’s (2003) spectral splitting of Jans & Leclercq’s (1999) unusable knowledgemuch better than a simple correct or not correct feedback. But certitude degreesallows to give much more refined feedbacks. 12
  13. 13. 5.2 Application in the gameCertitude degrees will add a dimension in storytelling as we define a new “socio-affective” goal to the game: George is to gain the trust of Galileo Galilei tobecome the new trustfully leader of the Galileans. Galileo’s trust in George willbe based on his good use on certitude degrees, mainly based on confidence inretrieved answers. The point is to give a general feedback on how the player isusing certitude degree. Not only immediatly after after a single judgment as itis in the actual demonstrator but with enough measure to pin tendencies likeover– and under–estimation, see section 2.3, page 4.5.3 Technical implementationSo on top of what was suggested here above, the program should computetwo values and communicate with the player about confidence (mcdca) andimprudence (mcdia) as defined by Leclercq & Poumay (2004): 1. mcdca is the mean certitude degree when the tries are successful or n (n · pn ) Conf idencce = CDcA = (1) NCDcA where • CDcA stands for mean certitude degree for correct answers; • n is a certitude degree (0%, 20%, 40%, 60%, 80% or 100%) • pn is the number of use of the n certitude degree associated with correct answers; • NCDcA is the number of use of the certitude degrees in correct an- swers; 2. mcdia is the mean certitude degree when the tries are unsuccessful or n (n · qn ) Imprudence = CDiA = (2) NCDiA • CDiA stands for mean certitude degree for incorrect answers; • n is a certitude degree (0%, 20%, 40%, 60%, 80% or 100%) • qn is the number of use of the n certitude degree associated with incorrect answers; • NCDiA is the number of use of the certitude degrees in incorrect answers.Exemple : Out of a test of 7 questions a student has 5 correct answers and2 incorrect answers. All 7 answers were associated with a certitude degree asshowed in table 3 page 14. To compute confidence we focus on the correct answers only: first we countthe number of use of certitude degree 0% (only for correct answer!). So n is 0%.In our exemple, there is 1 use of certitude degree 0% with a correct answer. Sop0 = 1. We multiply those two numbers by each other: n · pn giving 0 · 1 = 0. 13
  14. 14. Question Answer Certitude degree 1 C 80 2 C 20 3 C 100 4 C 0 5 C 80 6 I 60 7 I 0 Table 3: Exemple for the computation of confidence and imprudence (C for correct answerand I for Incorrect answer).Then we do the same for n = 20% which means that we count the number ofuse of certitude degree 20%, only for correct answer! In our exemple there is 1use of certitude 20% within all correct answers: p20 = 1. We multiply those twonumbers by each other: n · pn giving 20 · 1 = 20. Same thing for n = 40% whichmeans that we count the number of use of certitude degree 40%, remember onlyfor correct answer! In our exemple there is no use of certitude 40% within allcorrect answers: p40 = 0. We multiply those two numbers by each other: n · pngiving 40·0 = 0. Same thing for each certitude degree (60%, 80%) until we reachcertitude of 100% (n = 100%) with one use of certitude 100% making p100 = 1.We multiply those two numbers by each other: n · pn giving 100 · 1 = 100. Wesum all n · pn into [(0 · 1) + (20 · 1) + (40 · 0) + (60 · 0) + (80 · 2) + (100 · 1)]and divide by NCDcA or the number of use of certitude degree associated withcorrect answers, in our case 5. Final equation being [0+20+0+0+160+100] = 56 5 To compute Imprudence, it is the same process but focusing only on incorrectanswers. So we finish with [(0 · 1) + (20 · 0) + (40 · 0) + (60 · 1) + (80 · 0) + (100 · 0)]and divide by NCDcA or the number of use of certitude degree associated withincorrect answers, in our case 2. Final equation being [0+0+0+60+0+0] = 30. So 2in our exemple confidence is 56% and imprudence 30%. It is in the computation of those mean values that the use of a scale with6 levels makes sense. First Miller’s (1956) 28 works demonstrated that peopleare able to work with 7 plus or minus two certitude degrees to choose from.And the 6 levels are very easy to remember as they are from 0% to 100% stepof 20%. Shuford et al.(1966)29 demonstrated the necessity of using subjectiveprobabilities instead of verbal expressions. Finally it is after a number of answersthat mean values are computed taking into account every choice made by thestudent. In the case of elektra, after collecting certitude degrees during thestudent’s experimentations it is expected that the mean certitude of correctanswers (mcdca) –that is to say for the successful tries, should above 50%.Mixing well the use of all 6 levels of the certitude degree scale, students shouldlearn to weight their certitude in a correct way when they are about to succeed.A student will be considered as confident when his confidence is above 50%.On the other side the mean certitude of incorrect answers (mcdia) –that is to 28 Miller, G. (1956). The magical number seven, plus or minus two: Some limits on ourcapacity for processing information. Psychological Review, (63):81–97 29 Shuford, E. H., Albert, A., and Massengill, H. E. (1966). Admissible probability mea-surement procedures. Psychometrika, 31(2):125–45 14
  15. 15. say for the unsuccessful tries, should be below 50%. Students should learn tobe modest in confidence using low value on the 6 level scale. An accident mayalways occur like using a high certitude degree and failing but the tendencyshould be that when about to fail student learn to be prudent. An imprudentstudent will be a student with an imprudence above 50%. The best performanceis • a confidence of 100% or a mean certitude degree for the successful answers of 100% and • an imprudence of 0% or a mean certitude degree for the unsuccessful answers of 0%For gaming reasons, LMR asked that instead of trying to reach 100% on onescale and 0% on another scale the student should always try to maximize hisscore (like gaining points) or to minimize his level (like note losing live pointswhen fighting). To comply we suggest to use prudence instead of imprudence prudence = 100 − imprudence (3)Imprudence or mean certitude degree of incorrect answer should ideally be aslow as possible. Over 50% is the limit when the student becomes imprudent.With the inversion presented in equation (3) goals are exactly the opposite.Prudence should ideally be as close to 100% as possible. A student will bequalified as prudent if his prudence is over 50%.5.4 Graphic interfacePrudence and confidence will be the indicators of Galileo’s trust gained by theplayer. We suggest to represent prudence and confidence to use the certimeterthat was invented by Galileo and suggested by Frederic Pourbaix. There willbe 2 certimeters, one for the confidence (mean certitude of correct answers) andone for prudence (100 − mean certitude of incorrect answers), see figure 6. Bothcertimeter will be graduated from 0% to 100%. Suggested colors might be fromred to light red between 0% and 50% and from light green to green from 50%to 100%. Including prudence and confidence we offer a possible answer to ORT rec-ommendation30 : “The idea of confidence degree [. . . ] and may have the formof points in the game, which will bring the pupils to think further about theiranswer and will bring additional anticipation.” The aim of the player will be toreach green level score on both certimeter meaning that Galileo can trust himwhen he says that he is confident or when he says that he doubt. Of course tomake the “eclipse machine” work the player will have to gain knowledge andcomprehension of optic physics and astronomy. But the player should also payattention to some metacognitive aspects: confidence and prudence. If the playerfails to gain Galileo’s trust but manage to get the “eclipse machine” workinghe will gain access to the Galilean’s secret headquarters and to the book. Butmaybe Galileo will be a bit reluctant to help the player using the book. . . andgive him a feedback in that way, unless the player starts the game all over againand tries to improve his score on both certimeter. Success in the game could 30 Hirschberg, G. (2007) DELIVERABLE Name: Phase 1 – Validation Report 15
  16. 16. Figure 6: Rough design of a possible way to communicate to the player the level of Galileo’s trust gained: the cursor on each certimeter takes the color of the position in the color grada- tion. Both cursor must be at least light green to gain Galileo’s trust. then be achieved by reaching the Galilean’s HQ. But another level of success will be to reach HQ and to gain Galileo’s trust. 5.5 Realistic use of certitude degrees The feedbacks could take the form described in this algorithm here after. The limit values of 50% used in this alogrithm should be variables. Four limits are to be created and defined at a default level of 50conf-lim-ccd the limit to judge a single judgement with a correct answer or the limit to be declared confident in a single judgmentprud-lim-ccd the limit to judge a single judgment with an incorrect answer or the limit to be declared prudent in a single judgment mcdca-lim the limit to judge the mean certitude degree for correct answer as good or not or when is the player confident or not in the game so far mcdia-lim the limit to judge the mean certitude degree for incorrect answer as good or not or when is the player prudent or not in the game so far Every situation is symbolized by a group in between brackets (e.g., F + -). • First with a letter F for failure in case of an unsuccessful try S for success in case of a successful try 16
  17. 17. • a plus (+) or (-) sign + if the chosen confidence degree is above 50% - if the chosen confidence degree is below 50%• a plus (+) or (-) sign + if the mean confidence degree is above 50% - if the mean confidence degree is below 50%ccd (chosen certitude degree) : matrix;mcdca (mean certitude degree of correct answers) : integer;mcdia (mean certitude degree of incorrect answers) : integer;print When I asked you how confident you were about succeedingthis task, you told me you were {ccd}% confident succeeding it.If (FAILURE) then If (ccd > 50) then If (mcdia > 50) then print Situation A (F + +) else print Situation B (F + -) end If else If (mcdia ≤ 50) then print Situation C (F - -) else print Situation D (F - +) end If end Ifelse If (ccd ≤ 50) then If (mcdca > 50) then print Situation E (S - +) else print Situation F (S - -) end If else If (mcdca ≤ 50) then print Situation G (S + -) else print Situation H (S + +) end If end Ifend If 17
  18. 18. 5.5.1 Situation A (F + +)Situation: Test unsuccessful, confident in answer, tend to be overconfidentwhen wrong.Feedback: You did not succeed. And you were very confident you wouldsucceed. So you’ve made a bad assumption. You should be more careful in yourestimations. When you failed you tend to choose a too high certitude degree.So when you fail your average prudence is {100-mcdia}%. I consider thatit should be over {mcdia-lim}%. You should try to be less confident whenyou choose a certitude degree. You should learn to be a bit more modest andhumble. You tend to believe you know when you’re still learning. So give itanother try. A try to be more modest if you doubt, even a little.5.5.2 Situation B (F + -)Situation: Test unsuccessful, confident in answer, tend to be prudent whenwrong.Feedback: You did not succeed. You were very confident you would succeed.So you made a bad assumption but I consider this assumption as an accident.Usually you seem prudent in your own estimations. Every time you’ve failedyou’ve chosen before a low certitude degree. So when you fail your prudenceis {100-mcdia}%. I consider that it should be over {mcdia-lim}% so I trustyour judgement. Keep going like that when you choose a certitude degree andmaybe go back to the table for some more experimentation.5.5.3 Situation C (F - -)Situation: Test unsuccessful, prudent in answer, tend to be prudent whenwrong.Feedback: You did not succeed. But you certitude in your ability to succeedwas low: you were prudent. So your assumption was good: you have failed butyou knew it was risky so you associated doubt with your try. I congratulateyou for that. It is important for me to know if I can trust you. In case case ofcourse you’ve failed but before failing you told me that you were not sure. Andas usual when you say that you’re not sure you do not succeed. So this failureis not important at all. It seems you’re quite good at being prudent: usuallywhen you’ve failed you’ve chosen before a low certitude degree. So when youfail your overall prudence is {100-mcdia}%. I consider that it should be over{mcdia-lim}%. That is a good performance. Continue to do so and I will keepmy trust in your judgments. But you still need to succeed this task: give itanother try.5.5.4 Situation D (F - +)Situation: Test unsuccessful, prudent in answer, tend to be overconfident whenwrong.Feedback: You did not succeed. But this time your assumption was good: youhave failed but you knew it was risky so you associated doubt with your try. Icongratulate you for that. You should continue to behave like that because itdoes not appear you’re very skilled at this on the long run. For instance, whenyou fail your average prudence is {100-mcdia}%. I consider that it should be 18
  19. 19. over {mcdia-lim}%. That is why I said you’re not very skilled. With such avalue I will wait for you to improve before fully trusting your judgement. Maybeyou should go back to the table for some more experimentation before giving itanother try. And do not forget to keep going like you just did in giving yourconfidence.5.5.5 Situation E (S - +)Situation: Test successful, prudent in answer, tend to be confident when right.Feedback: To be completed5.5.6 Situation F (S - -)Situation: Test successful, prudent in answer, tend to be under confident whenright.Feedback: To be completed5.5.7 Situation G (S + -)Situation: Test successful, confident in answer, tend to be under confidentwhen right.Feedback: To be completed5.5.8 Situation H (S + +)Situation: Test successful, confident in answer, tend to be confident whenright.Feedback: To be completed5.6 Even more complex feedbacksIt will really be enjoying for the player to hear the same long feedback. Wesuggest that all these information should be delivered not at once but over anumber of tries. Counter of try and counters of times being in the same situation(e.g., F + - or S + +) should be implemented into the if–then–else loops. So thefirst time the player is in situation A he will received a more simple feedbackbut the second time he will receive a different feedback. It will be only afternumber of correct or incorrect answers that the computed mean values will haveany meaning and so can be given to the player. Prudence and confidence canalready have significant values computed on 55 value of certitude chosen by theplayer if we had the suggested game situation into the tutorial.6 Certitude degrees in LU1.2bThe Labset proposition for Elektra learning situation 1.2b (the slope) is similarto what we propose for learning situation 1.2a (the blinds) : every time Georgewants to release a marble he is asked his confidence in reaching the target on ascale from 0% to 100% by step of 20%. We suggest that beneath to the cursorsfor the magnet and the fan a third cursor with “confidence to reach the target”record the confidence associated with each try. Feedbacks will be less systematic 19
  20. 20. as the player/learner should be more aware of his confidence score showed in thecertimeter. When the player successfully dropped all 3 balls in the basket Galileowill ask him a transfer question. The player will have to chose out of a numbera drawings the one that will represent the trajectory of light when turned on.He will also be asked about his confidence. Imagine they are only 4 drawing tochoose from and the player knows that if he fails he can immediately try again.Without confidence degrees the player may try any of the 4 propositions untilhe finds the right one. But with confidence degrees he knows that if he choosesa wrong one and is confident in his guessing he will loose some of Galileo’s trust.So he might know the right solution, be confident and then will answer witha high confidence degree. He also may guess the right solution but then thechosen certitude should be lower. If guessing all proposition until he finds thecorrect one, all confidence will be low31 . So teacher will have information onhow did the student reached transfer.7 The tutorial for certitude degreesORT recommended32 that “The idea of confidence degree should be explained inthe beginning of the game (tutorial)”. The player will receive some explicationsabout the Galileans. It is at that moment that certitude degrees should be ex-plained. We propose a theoretical information based on transmission–receptionand a gaming situation similar to experimentation–reactivity. The tutorial should include 1. the player should gain the trust of Galileo Galilei 2. Galileo’s trust is based on the fact the when someone says “I know” he can trust that knowledge. And when someone doubt that person is right to doubt. 3. To gain Galileo trust everytime the player will experiment to solve an enigma he will be asked how confident he his that he will succeed resolving the enigma at this exact moment, on this precise try. 4. To indicate his self-confidence the player will chose one certitude degree between 0%, 20%, 40%, 60%, 80% and 100%. Points will be gained and lost based on • if what the player just tried is successful and – if he has chosen a high certitude degree he will gain confidence points – if he has chosen a low confidence degree he will loose confidence points • else if what the player just tried is unsuccessful and – if he has chosen a high certitude degree he will loose prudence points – if he has chosen a low confidence degree he will gain prudence points 31 with the limitation that certitude will go up as the number of possible answers goes down. 32 Hirschberg, G. (2007) DELIVERABLE Name: Phase 1 – Validation Report 20
  21. 21. 5. In other word if the player bet 100% on a try and if that try is • successful then he will gain a lot of confidence points • unsuccessful then he will loose a lot of prudence points 6. On the other hand if the player bet 0% on a try and if that try is • successful then he will loose a lot of confidence points • unsuccessful then he will gain a lot of prudence pointsWe suggest that this “theoretical” explanation is to be followed with a GamingSituation based on betting certitude degrees. This might be something like theShannon Guessing Game (1951)33 or another gaming situation that UL willprovide.7.1 Shannon’s guessing game“The new method of estimating entropy exploits the fact that anyone speakinga language possesses, implicitly, an enormous knowledge of the statistics of thelanguage. Familiarity with the words, idioms, cliches and grammar enableshim to fill in missing or incorrect letters in proof-reading, or to complete anunfinished phrase in conversation. An experimental demonstration of the extentto which English is predictable can be given as follows; Select a short passageunfamiliar to the person who is to do the predicting, He is then asked to guessthe first letter in the passage. If the guess is correct he is so informed, andproceeds to guess the second letter. If not, he is told the correct first letterand proceeds to his next guess. This is continued through the text. As theexperiment progresses, the subject writes down the correct text up to the currentpoint for use in predicting future letters.” Gilles & Leclercq (1994) 34 haveadapted this game to train student to use certitude degrees: each time a letteris chosen a certitude for it being the right next letter in the sentence is asked.Metacognitive indices like realism are computed and a graphical representationof the confidence in retrieved answer is drawn. The student tries to find thetext and to score well in metacognitive judgment. In elektra the passage might be a simple phrase like The secret book ofthe Galileans is in the third drawer of the desk. The player will face an emptyparchment that will be filled with letters written by a 3D feather when theplayer types a letter and a certitude degree. If the letter is incorrect it willburn, disappear and be replaced by the correct letter. Prudence and confidencewill be computed and a metacognitive feedback will be generated. Then theplayer is to guess the next letter and so on.7.2 Tutorial for teacherWe also suggest that informations about metacognition and certitude degreesshould be available for teachers. These informations should cover a wide domain,from objectives in the elektra game to proposition of reflection path about 33 Shannon, C. E. (1951) Prediction and entropy in printed English, Bell Syst. Tech. J., 30,1951, 50-64 34 Leclercq, D. & Gilles, J.-L. (1994). GUESS, un logiciel pour entraˆ ıner ` l’auto-estimation ade sa comp´tence cognitive. Actes du colloque QCM et questionnaires ferm´s, Paris: ISIEE e e 21
  22. 22. metacognition in classroom and support to further metacognitive activities forthe students.8 ConclusionKnowing how to learn, and knowing which strategies work best, are valuableskills that differentiate expert learners from novice learners. Metacognition, orawareness of the process of learning, is a critical ingredient to successful learning.“Metacognition is an important concept in cognitive theory. It consists of twobasic processes occurring simultaneously: monitoring your progress as you learn,and making changes and adapting your strategies if you perceive you are notdoing so well” (Winn, W. & Snyder, D., 1998).35Contents1 Why certitude degrees in learning? 12 What is metacognition? 2 2.1 Metacognition in general . . . . . . . . . . . . . . . . . . . . . . . 2 2.2 Certitude degrees and metacognition . . . . . . . . . . . . . . . . 4 2.3 Metacognition in the demonstrator . . . . . . . . . . . . . . . . . 4 2.3.1 What is a JOL? . . . . . . . . . . . . . . . . . . . . . . . 5 2.3.2 JOL or confidence in retrieved answer . . . . . . . . . . . 53 Metacognition in games 64 About the demonstrator, 2006 version 7 4.1 A surprise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.2 General comments . . . . . . . . . . . . . . . . . . . . . . . . . . 8 4.3 Experiment in world 2 and metacognition . . . . . . . . . . . . . 9 4.4 The metal-wooden door and the blinds . . . . . . . . . . . . . . . 105 Deeper with certitude degree 10 5.1 Theoretical background . . . . . . . . . . . . . . . . . . . . . . . 11 5.2 Application in the game . . . . . . . . . . . . . . . . . . . . . . . 13 5.3 Technical implementation . . . . . . . . . . . . . . . . . . . . . . 13 5.4 Graphic interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 5.5 Realistic use of certitude degrees . . . . . . . . . . . . . . . . . . 16 5.5.1 Situation A (F + +) . . . . . . . . . . . . . . . . . . . . . 18 35 Winn, W. & Snyder D. (1996). Cognitive perspectives in pyschology. In D.H. Jonassen,ed. Handbook of research for educational communications and technology, 112-142. NewYork: Simon & Schuster Macmillan 22
  23. 23. 5.5.2 Situation B (F + -) . . . . . . . . . . . . . . . . . . . . . 18 5.5.3 Situation C (F - -) . . . . . . . . . . . . . . . . . . . . . . 18 5.5.4 Situation D (F - +) . . . . . . . . . . . . . . . . . . . . . 18 5.5.5 Situation E (S - +) . . . . . . . . . . . . . . . . . . . . . . 19 5.5.6 Situation F (S - -) . . . . . . . . . . . . . . . . . . . . . . 19 5.5.7 Situation G (S + -) . . . . . . . . . . . . . . . . . . . . . 19 5.5.8 Situation H (S + +) . . . . . . . . . . . . . . . . . . . . . 19 5.6 Even more complex feedbacks . . . . . . . . . . . . . . . . . . . . 196 Certitude degrees in LU1.2b 197 The tutorial for certitude degrees 20 7.1 Shannon’s guessing game . . . . . . . . . . . . . . . . . . . . . . 21 7.2 Tutorial for teacher . . . . . . . . . . . . . . . . . . . . . . . . . . 218 Conclusion 22List of Tables 1 Flashlight on–off switch enhanced with certitude degree . . . . 10 2 Feedback class associated with the combination of true-false and confidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3 Exemple for the computation of confidence and imprudence (C for correct answer and I for Incorrect answer). . . . . . . . . . . . 14List of Figures 1 Main components of metacognitive monitoring and metacognitive control defined by Nelson (1996) and Nelson & Narens (1994) . . 3 2 Cut scene from the elektra demonstrator, 2006 . . . . . . . . . 7 3 Spectral distributions of Hunt’s three situations of knowledge (1993) 11 4 Spectral distribution of knowledge kinds by Jans & Leclercq (1999) based on Hunt’s three situation of knowledge . . . . . . . . . . . 12 5 Leclercq’s (2003) spectral splitting of Jans & Leclercq’s (1999) unusable knowledge . . . . . . . . . . . . . . . . . . . . . . . . . 12 6 Rough design of a possible way to communicate to the player the level of Galileo’s trust gained: the cursor on each certimeter takes the color of the position in the color gradation. Both cursor must be at least light green to gain Galileo’s trust. . . . . . . . . . . . 16 23