Making Expert Design Knowledge Useful for NovicesYael Kali ( Ronen-Fuhrmann (tamarrf@gmail....
about design, usually representing a socio-constructivist approach) and their “applied epistemologies about learning” (ide...
(a) Design artefact showing level 4 of maturity   (b) Design artefact showing level 6 of maturity                       Fi...
Outcomes and DiscussionFollowing an in-depth analysis of each of the 14 case-studies, in which we used both the MODA and t...
ReferencesAlexander, C. (1979). The timeless way of building. New York: Oxford University Press.Collins, A., Joseph, D., &...
Upcoming SlideShare
Loading in …5

Asld2011 kali ronen-fuhrman


Published on

Yael Kali and Tamar Ronen-Fuhrman: Making Expert Design Knowledge Useful for Novice

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Asld2011 kali ronen-fuhrman

  1. 1. Making Expert Design Knowledge Useful for NovicesYael Kali ( Ronen-Fuhrmann ( 21) University of Haifa2) Technion – Israeli Institute of TechnologyIn the past decades, much design knowledge has been gained by expert educational technology designers, and accumulated viadesign research projects. Learning scientists have sought various ways to make this knowledge available and useful for othereducational technology designers, and particularly for novices in this field. These efforts are part of a trajectory which viewseducation as a design-science (Collins et al. ,2004), or even more broadly, as one of the sciences of the artificial (Simon, 1969). Insuch fields generalizations of common examples are often articulated, to enable their application in other settings. Following thework of Alexander (1979), and his vision for articulating a “design pattern language” in architecture, learning scientists havedeveloped several manners to articulate general design guideline for curricular design. Three main types of guidelines that havebeen developed are: (a) design narratives (Hoadley, 2002; Linn & Hsi, 2000, Mor & Noss, 2008), (b) design principles(Herrington, 2006; Kali, 2006, 2008; Linn, Bell, & Davis, 2004; Merrill, 2002; van den Akker, 1999), and (c) design patterns(Goodyear, 2005; Goodyear & Retalis, 2010; Linn & Eylon, 2006; Mor & Winters, 2007; Retalis, Georgiakakis, & Dimitriadis,2006). Unfortunately, efforts to translate expert tacit knowledge into practical design guidelines, such as those mentioned above,have often failed to serve as useful aids for novice designers (McAndrew & Goodyear, 2007). It appears that in order for novicesto take advantage of such guidelines, a pedagogical framework is required. To overcome this challenge, we developed, in aprevious study (Kali & Ronen-Fuhrmann, 2011), a pedagogical model aimed at assisting graduate students in education to designtechnology-enhanced curriculum modules, which utilizes a set of design guidelines called the Design Principles Database (DPD)(Kali 2006; 2008). The model was developed in a designed-based research methodology with three iterations. In each iteration, a course thatwas based on the pedagogical model was implemented with students. Data was collected and analyzed, and design decisions weremade to improve the model for the next iteration (Kali & Ronen-Fuhrmann, 2011). The two authors of this paper served asteachers in the three implementations (as often the case in design-based research projects).A Pedagogical Model for Teaching Educational Technology DesignThe pedagogical model was embedded in a design course which combined theoretical and practical aspects of educationaltechnology design. It’s final version includes three main elements, and reflects a unique application and integration of threeframeworks: (a) the well-known Analysis, Design, Development, Implementation, Evaluation (ADDIE) model (Dick, Carey, andCarey 2001), (b) the studio approach to design teaching (Hoadley & Kim, 2003; Schon, 1983), and (c) the use of the DPD (Kali2006; 2008) as a framework of design guidelines.a) The ADDIE model was used as follows: In the Analysis stage students selected contents from a scientific discipline they knew well and had teaching experience with. They conducted a needs-analysis and a content-analysis to focus on a specific pedagogical challenge in this area. The Design stage was expanded to include three additional non-linear iterative stages: Brainstorm Activities, Build Flow, and Design Features. Students brainstormed ideas for activities that would potentially assist learners1 gain skills and knowledge required for understanding the contents of the module they designed. Then, they built a flow of activities, and designed features showing in detail how each activity would be viewed by a learner, including a screen layout, interactive elements, and instructions. Instead of the Develop stage of the ADDIE model, students were required to design a detailed mockup of their module. For the Evaluation stage students were required to present their modules in class and provide extensive feedback to each other. Based on this feedback, and additional comments from the instructors, they conducted a second design cycle.b) All course meetings took place as ‘design studio’ sessions. Students worked in groups of two or three students. At key stages each group presented their latest version of the artefact, and received feedback from peers and instructors.c) The DPD was embedded into students’ work process. This Web-based infrastructure, was designed to support researchers and technology-based curriculum designers share and synthesize their design knowledge (Kali, 2006; 2008). The shared design knowledge is accumulated in the DPD in the form of general design principles that are connected to example instantiations in various pedagogical settings (elementary, secondary and tertiary educational settings, in various subject matters). Students in the course were required to use the DPD at four points in the process: Analysis, Brainstorm Activities, Build Flow, and Design Features. Research GoalThis research builds on two earlier studies that explored student learning with the pedagogical model described above. The first,(Ronen-Fuhrmann, Kali, & Hoadley, 2008), showed that there is an important added value in engaging graduate students indesigning their own technology-based curriculum modules; while working on their design projects, students became more awareof gaps between what was defined as their “theoretical epistemologies about learning” (ideas expressed during general discussions1 We use the term Students to refer to the graduate students who designed the modules, and Learners to refer to potential users of those modules.  1
  2. 2. about design, usually representing a socio-constructivist approach) and their “applied epistemologies about learning” (ideasreflected in artifacts they created, which tended to apply more transmissionist approaches), and were able to reduce these gaps. Inthis manner, students’ epistemologies about learning became more coherent – an important outcome for students in education,whether or not they intend to design curriculum materials. The second study (Kali & Ronen-Fuhrmann, forthcoming), showed thatin the technology-enhanced educational modules they designed, students tended to stay at an abstract level and had a difficulty totranslate their pedagogical rationales and design ideas into concrete features. Thus, by the end of the course many artefacts stayedat an immature level. As the pedagogical model was refined to attend to student challenges (such as coping with the open-endednature of the task and making complex design-decisions with limited peer-feedback), it also better supported them in developingthe skill to concretize their design ideas and translate these ideas into features in mature learning environments. Concretizationwas described as a crucial skill for novices to progress in a design knowledge novice-expert continuum. In order to betterunderstand how expert design knowledge, such as the knowledge in the DPD can assist novice educational technology designers,the goal of the current research was to explore the relationship between students’ development of concretization skills and theirability to reduce their epistemological gaps, in the context of the educational technology course explored.Methodology We used a case-study methodology to examine students’ learning processes and their development of design knowledgethroughout the course. A collective case-study approach–often referred to as “multiple case study” (Stake, 1994)—wasimplemented. This approach is aimed at providing insights into an issue or problem or to refine a theory by exploring similaritiesand patterns between several case-studies. In this research, each group of 2-3 students, who worked on one design project duringthe course, was defined as a case-study. The study was conducted with 14 groups (33 graduate students) who participated in three enactments of the DesigningEducational Technologies course. Most students had some experience in teaching or were active science teachers. They had someexperience in designing curricula but most of them had no experience in designing technology-based learning modules. In order tocharacterize student learning in each of the iterations student documents were collected at various stages of the course. Thesedocuments included formal design artifacts students were required to create (including their final mockup), as well as informalnotes and sketches students created to discuss their ideas with peers and with us. These artefacts were analyzed using two rubrics;the first, entitled a Maturity Of Design Artefact (MODA) rubric (Kali & Ronen-Fuhrmann, forthcoming), was used to evaluate thedegree to which students were able to translate their design ideas into design artifacts (Table 1 and Figure 1); the second, entitled“epistemology rubric” (Table 2) was used to examine the epistemological changes that students went through during the course.Table 1: Maturity Of Design Artefact (MODA) rubric (Kali & Ronen-Fuhrmann, forthcoming) Stage in Degree of Maturity Required in the Design Representative Artefacts or Expressions Design Artefact ProcessAnalysis 1. Only general pedagogical ideas about “It’s very important to build activities that would be relevant and interesting to the the module should be expressed in learner” this stage (Except from a discussion of one of the groups in the analysis stage)Brainstorm 2. A collection of design ideas for the “Learning throughout the whole module should follow a specific inquiry question”.Activities module. The ideas should only (Excerpt from a discussion of one of the groups regarding their design of a biology generally refer to the way a learner module. They planned to design the activities around an inquiry question but were not might act in the module. concerned at this stage about the nature of this question).Build Flow 3. Graphical or verbal description of a set “First we should show them [the learners] the story about the family tree, then have of activities, with an indication of which them review the algorithm for scanning the tree, and then use the simulation” activity should take place before or (Excerpt from a discussion regarding the design of a module for high-school computer- after another. science learners) Figure 1a shows a sketch of the way students envisioned an activity they planned for aDesign 4. Ideas should be translated to actual module in genetics. Learners in this activity were required to decide whether they canFeatures features and presented in a way that donate blood to a kid with cystic fibrosis. shows how a learner might interact with the module. (As reference, see Figure 1b showing stage 6 – Mockup iteration 2).Mockup: 5. Initial learning environment – AIteration 1 mockup of the module showing some of the activities, with instructions for learners. An initial navigation scheme should be present. Figure 1b shows a sketch of one screen (from about 20 screens of the mockup whichMockup: 6. Mature learning environment – A were developed by this group with a similar level of detail) of a module designed forIteration 2 mockup of the module showing most teaching logical thinking for middle school math students. The buttons at the top and of the activities with clear instructions side of the screen are part of the whole learning environment’s navigating scheme. for learners. A clear navigating scheme should be represented. 2
  3. 3. (a) Design artefact showing level 4 of maturity (b) Design artefact showing level 6 of maturity Figure 1. Examples of artefacts showing levels 4 and 6 of maturity in the MODA rubric. Table 2: Epistemology Rubric Dimension Low Medium HighLearner activity Passive: e.g. learner Active: e.g. learner E.g. learner clicks onThe degree to which students expressed ideas that support active reads or views manipulates links.engagement of learners within a technology-based learning environment. information. variablesCollaboration Group work is not Collaboration isThe degree to which the students supported using technology in ways that Individual learning supported by intrinsic to the activityenable learners to learn from each other technologyContent accessibility No effort to connect Motivational aspects Motivational aspectsThe degree to which students expressed views that support making the contents to student are extrinsic to are intrinsic tocontents of a learning environment accessible to learners. world activities activitiesCombining the Two Rubrics to Map FindingsInitial analysis of the findings showed that using each of the rubrics described above, we can distinguish between two patterns oflearning. Using the MODA rubric, we found that one pattern was represented by groups who had difficulties in translating theirdesign ideas into concrete artefacts (they were slow indeveloping concretization skills). The maturity level oftheir artifacts at various stages of the design process waslower than the level required at that stage (see Table 1). Onthe other hand there were groups whose pattern did notshow any difficulty with the concretization and weresometimes even ahead of the required level in the designprocess. This enabled us to refer to the dichotomy: Lowversus High pace of concretization skills acquisition. Using the epistemology rubric, we were able toclearly distinguish between one pattern, in which groups ofstudents showed a gap at beginning stages of the semester,as described above, versus another pattern of those whoshowed a coherent epistemology throughout the semester.This enabled us to refer to the dichotomy: Coherent versusNon-coherent pattern of group epistemology. Using thesetwo dichotomies, we developed a four-quadrant matrix(Figure 2) to map our findings regarding the relationsbetween maturity/concretization and epistemology. Figure 2. The four-quadrant concretization/epistemology matrix. 3
  4. 4. Outcomes and DiscussionFollowing an in-depth analysis of each of the 14 case-studies, in which we used both the MODA and the epistemology rubricsusing all the data sources, we were able to map the cases into the four-quadrant matrix. Thus, two of the cases were mapped inquadrant 1, another two in quadrant 2, three more in quadrant 3, and 7 cases—more than half of the students—were mapped inquadrant 4. Additionally, the anaysis of each of the cases’ patterns of learning revealled that groups that were classified asbelonging to quadrant 4 significantly reduced their episteomogical gaps throughout the semester, whereas groups that belonged toquadrant 3 only did so to a small extent. We argue that the high pace of their acquisition of concretization skills (expressed in thematurity of their artifacts) was an important factor in enabling groups in quadrant 4 to reduce their epistemological gaps. Tosupport this claim, we describe in detail one case-study representing and illustrating the learning processes of groups that wereclassified as belonging to quadrant 4.Illustrating Learning Processes in Quadrant 4: The case of I,S&EI,S&E designed a technology-based module designated for high-school computer-science learners. Their module focused onrecursive algorithms for scanning data-structure trees. One of the features they designed, from very early stages of the designprocess was an animation that demonstrates a certain algorithm for scanning a tree. Their (potential) learners were required tosolve problems that utilize the demonstrated algorithm. Analysis of the design artifacts they produced at various stages of the design process using the MODA rubric (left graphin Figure 3) indicates that this group’s acquisition of concretization skills was of a high pace (high pace was defined as a slopethat is higher or equals to 0.75, where each stage of the design course was numbered consecutively starting with “Analysis=1”).I,S&E had come up with the idea of the animation as early as the Analysis stage (in which they were still not required to suggestideas for activities). They continued at a “normal”, or “required” pace (see dotted line in Figure 3 - left graph) in the BrainstormActivities and Build Flow stages. When required to design features, they were still struggling with their flow of activities, but theygradually progressed until they reached level four of concretization in their final mockup. The analysis of IS&Es’ learning process using the epistemology rubric (Figure 3, right side) revealed that at thebeginning of the semester, in general discussions about educational technologies, each of these students expressed ideas that weclassified as high level of sophistication with regards to epistemology (level 3 in each of the dimensions of the epistemologyrubric). However, as can be seen in figure 3, there was a large drop at the Analysis stage, with respect to the Learning Activity andContent Accessibly dimensions, which continued with a drop of the Collaboration dimension at the Build Flow stage. These dropsrepresent the gap described earlier, between “theoretical” and “applied” epistemologies. Specifically, when IS&E began to designtheir animation, it required learners only to passively watch the animation, and there was no attempt to make the contents moreaccessible. Collaborative aspects were minimal (a forum was designed for Q&A). Gradually, as this feature was revised followingdiscussions with peers and instructors, and following the use of the Design Principles Database, this feature became a manipulabletool, which enabled learners to solve problems by exploring various ways to scan given trees, as well as their own trees. Ouranalysis of their final mockup, using the epistemology rubric was as follows: Learning Activity = 3 (learner manipulatesvariables); Content Accessibility = 2 (motivational aspects were eventually at an intermediate level); Collaboration = 3 (activitiesthat required learners to solve problems in tasks created by their peers were designed). Thus - we interpreted their learning processas representing a major reduction of their groups’ epistemological gap. The dotted line in left graph of Figure 3, which representsthe average between the three dimensions, illustrates this decrease of the epistemological gap. 5 3 Concratization skill Activity Expected Pace Collboration Accessibility 4 Epistemology avrage Level of Concratization Epistemology 3 2 2 1 0 1 Theory Analysis Brainstorm Flow Features Mockup‐1 Final Theory Analysis Brainstorm Flow Features Mockup‐1 Final Design course stages Design course stagesFigure 3. The four-quadrant concretization/epistemology matrix. Our findings illustrate that as students concretized their design ideas and represented them in sequences of activities, theyexposed their pedagogical way of thinking to others. This enabled them to negotiate and reexamine their thinking with peers andinstructors and to compare the design solutions they came up with, with those of experts, as represented in the DPD. The exposureof ideas, induced by the concretization, brought to identification of gaps between students’ views about how people learn, andpedagogical notions expressed in artifacts they designed at initial stages of the course (Ronen-Fuhrmann et al., 2008). As students’artifacts became more concrete, they also represented more advanced pedagogical views of learning. The gaps were reduced as aresult of refinements students made throughout the design process. Thus, in the context of educational technology design, we viewconcretization as: (a) an essential skill in the process of gaining design knowledge, and (b) a way to assist students to reflect andreduce gaps in their understanding about learning theory. Our pedagogical model proved as a productive manner for novices touse expert design knowledge, in the form of design principles and feature in the DPD to guide their design process. 4
  5. 5. ReferencesAlexander, C. (1979). The timeless way of building. New York: Oxford University Press.Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. Journal of the Learning Sciences, 13(1), 15-42.Goodyear, P. (2005). Educational design and networked learning: Patterns, pattern languages and design practice. Australasian Journal of Educational Technology, 21(1), 82-101.Goodyear, P & Retalis, S. (2010). Learning, technology and design. In Goodyear, P & Retalis, S (Ed.), Technology-enhanced learning: design patterns and pattern languages. (pp. 1–27), Rotterdam: Sense.Herrington, J. A. (2006). Authentic e-learning in higher education: design principles for authentic learning environments and tasks. In T. C. Reeves & S. Yamashita (Eds.), Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2006 (pp. 3164-3173). Chesapeake, VA: AACE.Hoadley, C. (2002). Creating context: Design-based research in creating and understanding CSCL. In G. Stahl (Ed.), Computer Support for Collaborative Learning 2002 (pp. 453-462). Mahwah, NJ: Lawrence Erlbaum Associates.Hoadley, C., & Kim, D. (2003). Learning, design and technology: The creation of a design studio for educational innovation. In A. P. d. Reis & P. Isaías (Eds.), Proceedings of the IADIS international conference e-Society 2003 (pp. 510-519). Lisbon, Portugal: IADIS Press.Kali, Y. (2006). Collaborative knowledge building using the Design Principles Database. International Journal of Computer Support for Collaborative Learning, 1(2), 187-201.Kali, Y. (2008). The Design Principles Database as means for promoting design-based research. In A. E. Kelly, R. A. Lesh & J. Y. Baek (Eds.), Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. Mahwah, NJ: Lawrence Erlbaum Associates.Kali, Y., & Ronen-Fuhrmann, T. (2011). Teaching to design educational technologies. The International Journal of Learning Technology (IJLT), 36(2), 129-149.Kali, Y., & Ronen-Fuhrmann, T., (forthcoming). Concretization of design ideas in the context of educational technology design. The Journal of Instructional Science.Linn, M. C., Bell, P., & Davis, E. A. (2004). Specific design principles: Elaborating the scaffolded knowledge integration framework. In M. C. Linn, E. A. Davis & P. Bell (Eds.), Internet Environments for Science Education (pp. 315-340). Mahwah, NJ: Lawrence Erlbaum Associates.Linn, M. C., & Eylon, B.-S. (2006). Science education: Integrating views of learning and instruction. In P. A. Alexander & P. H. Winne (Eds.), Handbook of educational psychology (2nd ed., pp. 511-544). Mahwah, NJ: Lawrence Erlbaum Associates.Linn, M. C., & Hsi, S. (2000). Computers, Teachers, Peers: Science Learning Partners. Mahwah, NJ: Lawrence Erlbaum Associates.McAndrew, P., & Goodyear, P. (2007). Representing practitioner experiences through learning design and patterns. In H. Beetham & R. Sharpe (Eds.), Rethinking pedagogy for a digital Age: Designing and Delivering E-learning (pp. 92-102). Oxon: Routledge.Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43-59.Mor, Y., & Winters, N. (2007). Design approaches in technology enhanced learning. Interactive Learning Environments, 15, 61- 75.Mor, Y., & Noss, R. (2008). Programming as mathematical narrative. International Journal of Continuing Engineering Education and Life-Long Learning (IJCEELL) 18 (2), 214-233.Retalis, S., Georgiakakis, P., & Dimitriadis, Y. (2006). Eliciting design patterns for e-learning systems. Computer Science Education, 16(2), 105-118.Ronen-Fuhrmann, T., Kali, Y., & Hoadley, C. M. (2008). Helping education students understand learning through designing. Educational Technology, 48(2), 26-33.Schon, D. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.Simon, H. A. (1969). The sciences of the artificial. Cambridge, Massachusetts: MIT.Stake, R. E. (1994). Case studies. In N. K. Densin & Y. S. Lincoln (Eds.), Handbook of Qualitative Research (pp. 236-247). Thousand Oaks, CA: SAGE Publications.van den Akker, J. (1999). Principles and methods of development research. In J. v. d. Akker, N. Nieveen, R. M. Branch, K. L. Gustafson & T. Plomp (Eds.), Design methodology and developmental research in education and training (pp. 1-14). Dordrecht, the Netherlands: Kluwer. 5