1. A Tool to Visualize Verbal Protocols to Enhance Strategic and Metacognitive Abilities in Basic Programming Dr. Carlos Argelio Arévalo Mercado M. en C. Lizeth Itziguery Solano Romo TAMK Opening IGUAL Project meeting. February 14 – 18, 2011

3. Aguascalientes. Population 1.065.416

6. Internal UAA Programming Statistics

7. Cognitive aspects of programming (Literature Review) Cognitive Aspect Description References Implications Mental Models A mental model is an internal representation of a complex system or task, whose construction enables the learner to understand and predict that system's behavior. In the context of programming, this aspect is sometimes called "The Notional Machine " (George 2000); (Bayman 1983); Johnson-Laird (1983); Winslow (1996 ); Du Boulay (1989); . (Fixx, Wiedenbeck, y Scholtz 1993; Hegarty 1993; Ma et al. 2007; Ramalingam 2004) Valid mental models have to be taught explicitly. Problem occurs when novice creates an invalid mental model. Schemas and Strategies A strategy (or plan, or cliché) is a predefined solution within the programmers knowledge structure, used when a recurring problem pattern is recognized. Writing a program is not a straightforward process. It is incremental and hypothesis driven. A key part of a program (called focal point,) is used by an expert, and the solution is built around that point. (Fixx, Wiedenbeck, y Scholtz 1993); Brooks (1990); Rist (1995, 514); Soloway y Ehrlich (1984); Davies (1993) Rist… Strategic knowledge of program construction is hard to teach, and not sufficiently emphasized in traditional courses. Not enough instruction is given to novices on how to "put the pieces together“. Text books often only show the final state of a problem/program, but not the process of construction. Metacognition Metacognition is defined as "awareness about how a person learns or solves a problem". Flavell (1979) Through Metacognition, an individual can define the nature of a problem, select a useful mental representation, select an effective strategy, monitor the progress of problem resolution and adjust strategies during the task. Flavell (1979); Gourgey (1998); Williams y Upchurch (2001) ; Roth (2004), Metacognitive aspects of program design are included within strategic knowledge. These Metacognitive knowledge must also be taught explicitly

8. Instructional aspects related to programming Instructional Aspect Description References Implications Learning Style According to a previously selected model, an individual can have different preferences or characteristics that facilitate his/her learning process Kolb (1984) Felder (1996) (Briggs y Myers, 1995) There are no statistically significant evidence about some particular learning style being a predictor of success in the first programming course. Previous Experience A previous programming experience can occur in the context of high school education (Fauxx 2006); (Hagan y Markham 2000); (Holden y Weeden 2005). There is a positive effect on performance if the student has had a previous programming experience. This positive effect tends to disappear following undergraduate semesters. Self-Efficacy How well one can execute courses of action required to deal with prospective situations (Bandura 1982); (Brosnan 1998); (Byrne 2001); (Ramalingam 2004); (Heggestad 2005) It has been identified as a positive influence on first programming course performance. Some have stated that self-efficacy its just a residual variable of previous course performances. Math Proficiency The abstract and logical reasoning ability required to solve mathematical problems, is similar to the ability required to program computers (Byrne 2001); (Hu 2006); (White 2003); (Jenkins 2002) Has a strong correlation with positive performance. But ethical and practical curricular issues arise.

9. Components of an information system design theory (ISDT) (Adapted from Walls, et. al, 1992)

12. ISDT Design Product Kernel Theories Kernel Theories Description Evidence / Principles References Dual Coding Theory When information is registered both verbally and visually, it's said that be "Dually Coded". Information is better retrieved if it is dually stored, because if one part is lost (visual or verbal) the other is available and can be retrieved. Empirical data suggests that students using study materials designed to promote dual coding, seem to have better performance (Kuo y Hooper 2004; Meyer y Sims 1994; Sadoski y Paivio 2004) Verbal Protocols Verbal protocols where developed to study the short term memory process of people (to what things the pay attention and in what order) in the process of solving a problem. They document the mental behavior of a person, by asking him or her to "Think aloud" while doing a certain task Verbal data are a valid source of comprehension and study of human cognitive processes. A phrase is the representation of an idea and can be used to identify different kinds of information y different types of cognitive processes. Used in the context of basic programming to analyze common solution patterns. Also used in usability studies. Ericsson y Simon (1980), Nielsen, Clemmensen, y Carsten (2002) , ( Détienne 1995; Burkhardt, Détienne, y Wiedenbeck 1997; Rist 1996; Rist 1989) ; Worked examples and completion problem effect Worked examples can facilitate the construction of schemas and strategy transfer more efficiently than solving the problems themselves. This prediction is explained in terms of "cognitive load reduction". It has the limitation that worked examples do not force low motivated students to study them. Worked examples can be complemented with the "completion problem effect" pedagogical strategy. Empirical studies have demonstrated the effectiveness of worked examples in fields such as algebra, and heuristic knowledge domains . Important: they should be used only during the first stages of learning , to avoid the so called "expertise reversal effect" and they should not have to be used excessively to avoid the transfer of too much stereotyped solution patterns (J. Sweller, van Merrienboer, y Paas 1998; John Sweller 1994) ; (Gerjets, Scheiter, y Catrambone 2004; Renkl, Hilbert, y Schworm 2009; Renkl, Atkinson, y Große 2004; M. Ward y J. Sweller 1990; J. Sweller y G. Cooper 1985)

13. Verbal Protocols editing and recording process Select and Categorize a problem (instructor) Transcribe and Edit the protocol in the system As a Worked Example (Instructor) Add Visual Aids (Animated Code) (Instructor) Add additional notes to the protocol (i.e Focal Point, general strategy) (Instructor) Visualize and Study the Problem Solving Protocol (Student) Record the problem solving protocol for the selected problem (Programmer & Interviewer)

14. Development of the Artifact

15. Evolution of the user interface Flash Access Web (PHP)

16. User interface for verbal protocol visualization Protocol Verbalization Dual Coding Programming Problem Instructor aditional Notes Protocol Navigation

17. Modified user interface Problem search by keywords The interface was subject to several tests to refine its usability. For example a feature was added to allow a student to “jump” directly to a specific step of the protocol

18. Modified user interface Instructor annotations (i.e. focal point) Animated code Go to a specific protocol step Navegación Verbalizations

19. Measurement instrument To measure the tool’s capability to transfer strategic knowledge to novice undergraduate programmers, we designed a standard test, consisting of three basic programming problems, to assess the student’s skills

23. Results

24. Statistical data

25. Correlation Analysis

27. Current state: Interface Redesign

28. Current state: Interface Redesign

29. Current state: Interface Redesign

30. Current state: Interface Redesign