The document describes a research project that designed a tool to visualize verbal protocols of programming experts solving problems. The tool aims to help transfer experts' valid mental models, problem-solving strategies, and metacognitive abilities to novice programmers. An evaluation of the tool involved undergraduate students and found those using the tool performed better on a programming test compared to the control group. Future work is needed to further evaluate the tool's effectiveness over longer periods.

1. A Tool to Visualize Verbal Protocols to Enhance Strategic and Metacognitive Abilities in Basic Programming Dr. Carlos Argelio Arévalo Mercado M. en C. Lizeth Itziguery Solano Romo TAMK Opening IGUAL Project meeting. February 14 – 18, 2011

2. UAA IGUAL Proposed Project Staff Academic Staff Dr. Carlos Argelio Arévalo Mercado (Local Coordinator) M.C. Lizeth Itziguery Solano Romo (Student coordinator and research support) Dra. Lizbeth Muñoz Andrade (Experiment design and execution) Dra. Laura A. Garza González (Financial support) M.C. Jorge E. Macías Luévano (Infrastructure support) Student Staff Francisco Javier García Pilar (programmer) Bertha Alicia Cobos (protocol recording) Student 3 (programmer) Student 4 (protocol recording)

3. Aguascalientes. Population 1.065.416

4. Autonomous University of Aguascalientes History and Background Different names: 1887 Agriculture School 1871 to 1875 Scientific and Literature Institute 1985 State Science Institute 1906 State High School Before 1973 Autonomous Science and Technology Institute Since June 19, 1973 Autonomous University of Aguascalientes 36 years since foundation 9 University presidents 54 Under graduate programs, Graduate programs 38, 7 technical programs, 2 high school programs

5. University Population Professors: 1,985 Students: 14, 310 Section Total Graduate 235 Under graduate 12,456 High School 1,619

6. Internal UAA Programming Statistics

7. Cognitive aspects of programming (Literature Review) Cognitive Aspect Description References Implications Mental Models A mental model is an internal representation of a complex system or task, whose construction enables the learner to understand and predict that system's behavior. In the context of programming, this aspect is sometimes called "The Notional Machine " (George 2000); (Bayman 1983); Johnson-Laird (1983); Winslow (1996 ); Du Boulay (1989); . (Fixx, Wiedenbeck, y Scholtz 1993; Hegarty 1993; Ma et al. 2007; Ramalingam 2004) Valid mental models have to be taught explicitly. Problem occurs when novice creates an invalid mental model. Schemas and Strategies A strategy (or plan, or cliché) is a predefined solution within the programmers knowledge structure, used when a recurring problem pattern is recognized. Writing a program is not a straightforward process. It is incremental and hypothesis driven. A key part of a program (called focal point,) is used by an expert, and the solution is built around that point. (Fixx, Wiedenbeck, y Scholtz 1993); Brooks (1990); Rist (1995, 514); Soloway y Ehrlich (1984); Davies (1993) Rist… Strategic knowledge of program construction is hard to teach, and not sufficiently emphasized in traditional courses. Not enough instruction is given to novices on how to "put the pieces together“. Text books often only show the final state of a problem/program, but not the process of construction. Metacognition Metacognition is defined as "awareness about how a person learns or solves a problem". Flavell (1979) Through Metacognition, an individual can define the nature of a problem, select a useful mental representation, select an effective strategy, monitor the progress of problem resolution and adjust strategies during the task. Flavell (1979); Gourgey (1998); Williams y Upchurch (2001) ; Roth (2004), Metacognitive aspects of program design are included within strategic knowledge. These Metacognitive knowledge must also be taught explicitly

8. Instructional aspects related to programming Instructional Aspect Description References Implications Learning Style According to a previously selected model, an individual can have different preferences or characteristics that facilitate his/her learning process Kolb (1984) Felder (1996) (Briggs y Myers, 1995) There are no statistically significant evidence about some particular learning style being a predictor of success in the first programming course. Previous Experience A previous programming experience can occur in the context of high school education (Fauxx 2006); (Hagan y Markham 2000); (Holden y Weeden 2005). There is a positive effect on performance if the student has had a previous programming experience. This positive effect tends to disappear following undergraduate semesters. Self-Efficacy How well one can execute courses of action required to deal with prospective situations (Bandura 1982); (Brosnan 1998); (Byrne 2001); (Ramalingam 2004); (Heggestad 2005) It has been identified as a positive influence on first programming course performance. Some have stated that self-efficacy its just a residual variable of previous course performances. Math Proficiency The abstract and logical reasoning ability required to solve mathematical problems, is similar to the ability required to program computers (Byrne 2001); (Hu 2006); (White 2003); (Jenkins 2002) Has a strong correlation with positive performance. But ethical and practical curricular issues arise.

9. Components of an information system design theory (ISDT) (Adapted from Walls, et. al, 1992)

10. Doctoral Research General Objective Apply the ISDT framework to produce a design theory that outlines a kind of software artifact whose properties include: visualizations of valid mental programming models , strategic problem solving knowledge and Metacognitive reasoning of programming experts on the act of writing programs, to foster meaningful learning and better knowledge transfer to novice programming students.

11. Specific Objectives Propose an Information Systems Design Theory for a kind of software artifacts that promotes the transfer of valid programming mental models, problem solving strategies and Metacognitive abilities to novice students . Design and build de artifact. Empirically test the effectiveness of the artifact.

12. ISDT Design Product Kernel Theories Kernel Theories Description Evidence / Principles References Dual Coding Theory When information is registered both verbally and visually, it's said that be "Dually Coded". Information is better retrieved if it is dually stored, because if one part is lost (visual or verbal) the other is available and can be retrieved. Empirical data suggests that students using study materials designed to promote dual coding, seem to have better performance (Kuo y Hooper 2004; Meyer y Sims 1994; Sadoski y Paivio 2004) Verbal Protocols Verbal protocols where developed to study the short term memory process of people (to what things the pay attention and in what order) in the process of solving a problem. They document the mental behavior of a person, by asking him or her to "Think aloud" while doing a certain task Verbal data are a valid source of comprehension and study of human cognitive processes. A phrase is the representation of an idea and can be used to identify different kinds of information y different types of cognitive processes. Used in the context of basic programming to analyze common solution patterns. Also used in usability studies. Ericsson y Simon (1980), Nielsen, Clemmensen, y Carsten (2002) , ( Détienne 1995; Burkhardt, Détienne, y Wiedenbeck 1997; Rist 1996; Rist 1989) ; Worked examples and completion problem effect Worked examples can facilitate the construction of schemas and strategy transfer more efficiently than solving the problems themselves. This prediction is explained in terms of "cognitive load reduction". It has the limitation that worked examples do not force low motivated students to study them. Worked examples can be complemented with the "completion problem effect" pedagogical strategy. Empirical studies have demonstrated the effectiveness of worked examples in fields such as algebra, and heuristic knowledge domains . Important: they should be used only during the first stages of learning , to avoid the so called "expertise reversal effect" and they should not have to be used excessively to avoid the transfer of too much stereotyped solution patterns (J. Sweller, van Merrienboer, y Paas 1998; John Sweller 1994) ; (Gerjets, Scheiter, y Catrambone 2004; Renkl, Hilbert, y Schworm 2009; Renkl, Atkinson, y Große 2004; M. Ward y J. Sweller 1990; J. Sweller y G. Cooper 1985)

13. Verbal Protocols editing and recording process Select and Categorize a problem (instructor) Transcribe and Edit the protocol in the system As a Worked Example (Instructor) Add Visual Aids (Animated Code) (Instructor) Add additional notes to the protocol (i.e Focal Point, general strategy) (Instructor) Visualize and Study the Problem Solving Protocol (Student) Record the problem solving protocol for the selected problem (Programmer & Interviewer)

14. Development of the Artifact

15. Evolution of the user interface Flash Access Web (PHP)

16. User interface for verbal protocol visualization Protocol Verbalization Dual Coding Programming Problem Instructor aditional Notes Protocol Navigation

17. Modified user interface Problem search by keywords The interface was subject to several tests to refine its usability. For example a feature was added to allow a student to “jump” directly to a specific step of the protocol

18. Modified user interface Instructor annotations (i.e. focal point) Animated code Go to a specific protocol step Navegación Verbalizations

19. Measurement instrument To measure the tool’s capability to transfer strategic knowledge to novice undergraduate programmers, we designed a standard test, consisting of three basic programming problems, to assess the student’s skills

20. Cuasi-experimental conditions To test the effectiveness of the tool, a semi-experimental setting was designed, using two groups of programming students: one from 2 nd semester Computer Science students, and another from 2 nd semester Electronic Engineering students, both from Autonomous University of Aguascalientes (UAA), México. Selection of participants was not random. Complete groups were invited, given that the experiment was conducted during a period of normal classes. The Electronic Engineering group (n=20) was selected as the control group (TRAD). The computer science group (n=18) was to serve as the experimental group (EXP). 55% of the (TRAD) group had previous programming experience from high school, while 41% of the (EXP) group had previous programming experience. Two days before the application of the test, four exercises were given to both groups for them to study and practice. This exercises were the same ones loaded in the Protocol Visualizer Tool.

21. Cuasi-experimental conditions Controlled conditions for both groups were: Explanatory lecture sessions had a one hour duration for both groups. The given time to answer the instrument was limited to an hour for both groups. A “motivation” factor was introduced for both groups in form of “extra points”. Characteristics of the lab computers used from both groups were the same Previous programming experience was similar in both groups (55% for TRAD and 41% for EXP). At the time of the experiment, the instructional content given (during normal classes) to both groups was the same. Both groups where studying basic data structures in C Language.

22. Cuasi-experimental conditions Uncontrolled conditions were: Teachers from both groups were different, but they came from the same programming academy, taught the same content, and had the same experience in teaching. It was not possible to record the individual answering time of the test participants. Selection of participants was not random. In both cases, complete groups where invited to participate in the study.

23. Results

24. Statistical data

25. Correlation Analysis

26. Limitations of the study Given the uncontrolled conditions reported, the positive results obtained by the EXP group using the Protocol Visualizer Tool, cannot be generalized to be valid for all the population of first year undergraduate programming students. An uncontrolled variable was teaching style : that is, even though both teachers were part of the same academy and had similar background and experience, the two groups having different teachers could have had an unmeasured effect on the final results. Also, duration of the actual study was limited to three days , given the limited availability of the students who voluntarily participated.

27. Current state: Interface Redesign

28. Current state: Interface Redesign

29. Current state: Interface Redesign

30. Current state: Interface Redesign

31. Current state Protocols being recorded by social service students Problem selection Quality control Editing Transcription / System mounting

32. Future activities Future studies need to be longitudinal in nature, so that the effect of a longer exposure of the students to the tool can be measured. A randomly selected sample of participants is desirable, but this kind of scenario is not always possible (or practical) given the nature of every day lectures. It is also possible to load the tool with protocols that involve problems related to other programming languages or paradigms such as Java, C#. Also, the graphical user interface can still be improved through usability test, in order to use with other Web-enabled platforms (such as mobile devices). Extend the functionality of the tool, by adding pedagogical features such as completion problems , in selected segments of the protocols. Combine the protocol visualizer with other learning technologies were possible.