Metis project deliverable D3.2: Draft of pilot workshop


Published on

This deliverable represents the analysis of best practices and workshop design from the first cycle of the METIS project methodology. Alongside this report a prototype is provided to allow access to the package of resources representing a workshop structure developed from the preliminary analysis of best practices in teacher training reported in Deliverable D3.1. Section ‎2 provides an account of the review of best practices, the process, current status and outcomes, and plans for the future. It also lists risks and challenges and implications to and from WP 2 and 4.

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Metis project deliverable D3.2: Draft of pilot workshop

  1. 1. Project Number: 531262-LLP-2012-ES-KA3-KA3MPMETIS - Meeting teachers co-design needs by means ofIntegrated Learning EnvironmentsD3.2: Draft of pilot workshopWP3: Workshop DesignWPLeader: OUAuthor(s)/Editor(s): Andrew Brasher,Chris Walsh, Patrick McAndew, Yishay Mor(OU)
  2. 2. Project Number: 531262-LLP-2012-ES-KA3-KA3MPProject informationProject acronym: METISProject title: Meeting teachers co-design needs by means ofIntegrated Learning EnvironmentsProject number: 531262-LLP-1-2012-1-ES-KA3-KA3MPSub-programme or KA: KA3 Multilateral projectsProject website: http://www.metis-project.orgReporting period: From 31/1/13To 31/5/13Report version: 0.1Date of preparation: 10/5/13Beneficiary organisation: University of Valladolid (UVa), SpainProject coordinator: Prof. Yannis DimitriadisProject coordinator organisation: University of Valladolid (UVa), SpainProject coordinator telephone number: +34 983 423696Project coordinator email address: info@metis-project.orgWP Leader: Yishay Mor (OU)WP Leader email address: historyDate Version Author(s) Description10/5/13 01 Andrew Brasher, ChrisWalsh, PatrickMcAndrew, Yishay MorFinal draft for formal internal review16/5/13 0.1.1 Gráinne Conole Review20/5/13 0.1.2 Michael Derntl Review29/5/13 02 Andrew Brasher Edited in response to reviewers’comments.6/6/13 03 Andrew Brasher, YishayMorFinal versionThis work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.This project has been funded with support from the European Commission. Thispublication reflects the views only of the author(s), and the Commission cannot beheld responsible for any use which may be made of the information containedtherein.
  3. 3. Project Number: 531262-LLP-2012-ES-KA3-KA3MPExecutive SummarySeveral decades of research in Technology-Enhanced Learning (TEL) have clearlydemonstrated the potential of digital technology to transform education. Yet theimpact of TEL research on daily teaching-learning practices is still far from fulfillingthis potential (Mellar, Oliver, & Hadjithoma-Garstka, 2009). Teachers lack thenecessary digital literacy skills (Jenkins, 2009) to harness the potential of newtechnologies. Arguably, this is a gap in the capacity for learning design1: educatorsneed the tools and competencies which would allow them to identify educationalchallenges, describe the context in which they arise, identify the opportunitiesafforded by technology, project the insights derived from research, and devise newlearning experiences. To address this gap, educators need tools and practices. Toolsthat would support them through the cycle of learning design – from conception todeployment and evaluation of techno-educational innovations. Professionalpractices that use such tools to ensure the robustness and effectiveness of theirinnovations and make learning design a daily habit and part of their professionalidentity. The METIS project ( aims to contribute to this aim,by providing educators with an Integrated Learning Design Environment (ILDE) and aworkshop package for training educators in using the ILDE to support effectivelearning design.Work Package 3, led by the OU (UK), is concerned with the design and developmentof the workshop package.Deliverable D3.2 represents the analysis of best practices and workshop design fromthe first cycle of the METIS project methodology. Alongside this report a prototype isprovided to allow access to the package of resources representing a workshopstructure developed from the preliminary analysis of best practices in teachertraining reported in Deliverable D3.1. (This prototype is included in appendix 1).Section 2 provides an account of the review of best practices – the methodologyused for this review, its current status and outcomes, and plans for the future.Section 3 provides an account of the design of the METIS workshop – the process,current status and outcomes, and plans for the future. It also lists risks andchallenges and implications to and from WP 2 and 4.The METIS partners responsible for designing the workshops have extensiveexperience of running and evaluating learning design workshops. This includes the1. Typically “Learning Design” (with capital letters) is used to refer to specific tools or projects e.g. ‘IMS Learning Design’, and the ‘Integrated Learning Design Environment’. The phrase “learning design”(all lower case) is used to refer to general practices and instantiations e.g. ‘enactment of innovativelearning designs’ .
  4. 4. Project Number: 531262-LLP-2012-ES-KA3-KA3MPwork undertaken by the OULDI project at the OU, the Carpe Diem workshops run bythe University of Leicester, the 7Cs of Learning Design framework (developed byUniversity of Leicester), the recent OLDS MOOC (led by the OU) and the JISC-fundedSPEED project.The workshop structure presented here is a “meta-design”, which needs to becustomized and specified for each user group. The structure provides a flexible basisfor developing the ready-to-run workshops, and is the input for subsequent tasks inWP3/WP4 in the second cycle. The partner methodologies (provided in Appendices)show some of the activities envisioned that can be placed within the structure todevelop the different workshops.This report consists of an account of the workshop design process and criticaldecisions, an overview of the design principles and outline of the workshopstructure, and a review of the pedagogical framework and best practices informingthe design. The current version of the design prototype, as well as some of the bestpractices data, are includes as appendices.
  5. 5. METIS PROJECTD3.2: Draft of pilot workshop Page 5 of 64Contents1 Introduction .............................................................................................................................72 Review of best practices ..........................................................................................................92.1 Method..............................................................................................................................92.2 Results .............................................................................................................................113 Outline of draft design of the METIS Workshops ..................................................................164 Next steps and implications for other work packages...........................................................174.1 Risks.................................................................................................................................174.2 Implications for and from WP 2 ......................................................................................185 Conclusions ............................................................................................................................196 Appendices.............................................................................................................................207 References..............................................................................................................................20Appendix 1. Workshop design..................................................................................................22Appendix 2. Methodology descriptions ...................................................................................34a. Learning Design Workshop Methodologies from the ICOPER Best Practice Network..........34b. 7Cs of learning design framework .........................................................................................35c. Design-Practice.......................................................................................................................37d. Learning Design Studio...........................................................................................................39e. Collage / LdShake workshops.................................................................................................41f. Participatory Pattern Methodology.......................................................................................42g. Design Challenge....................................................................................................................44h. OULDI Learning Design Training Module...............................................................................45Appendix 3. Methodology design narratives and design patterns ..........................................48a. “Design Narratives” task presented to partners....................................................................48b. Design narrative template .....................................................................................................49Appendix 4. Sample of Methodology Design Narratives..........................................................51a. Meta-Pyramid design.............................................................................................................51b. Role-playing on problematic situations and technology .......................................................56c. Mini-focus-groups..................................................................................................................59Appendix 5. Personas template................................................................................................63Appendix 6. Survey of user groups...........................................................................................64
  6. 6. METIS PROJECTD3.2: Draft of pilot workshop Page 6 of 64
  7. 7. METIS PROJECTD3.2: Draft of pilot workshop Page 7 of 641 IntroductionThe last few decades have seen tremendous progress in the use of ICT for education andtraining across Europe. However, effective integration of ICT should go beyond replacing,streamlining or accelerating current practices. It must also support “pedagogical andorganisational innovation” (EACEA, 2009). The current gap between research and practice inTechnology-Enhanced Learning (TEL) highlights the need for a shift in attention from thedevelopment of specific tools and resources to the support for their integration, e.g. in teacherpractice for the design of ICT-based learning activities. This integration into practice needssupport for the whole design and implementation life-cycle, from (co-)design to enactment(Kelly, Lesh, & Baek, 2008; Laurillard, 2008).The METIS project ( aims to provide this kind of support, synthesisingthe achievements of the design paradigm and making them available to a broad circle ofpractitioners across multiple educational sectors such as adult education, vocational trainingand higher education. This is to be achieved mainly through a practitioner-centred approach,which combines a) technological support for the whole learning design life-cycle (in the LearningDesign Environment, or ILDE); b) professional development support in the form of ready-to-useworkshop packages; and c) the dissemination of these project outcomes to a wide communityof practitioners.Work Package 3 of the METIS project is concerned with the design of the professionaldevelopment workshops. This work package is led by the OU (UK), in collaboration with otherpartners. We will work closely with the projects’ user groups to ensure that the workshopdesign addresses their constraints and concerns, and with the evaluation partners to ensurethat the design is robust and effective. The workshop design will be evaluated by the user grouppartners (task T3.4), and eventually offered as an open educational resource on the projectwebsite (as part of deliverable D3.4, task T3.9).Deliverable D3.2 represents the analysis of best practices and workshop design from the firstcycle of the METIS project methodology. Alongside this report a prototype is provided to allowaccess to the package of resources representing a workshop structure developed from thepreliminary analysis of best practices in teacher training reported in Deliverable D3.1.The workshop structure presented here is a “meta-design”, which needs to be customized andspecified for each user group. The structure provides a flexible basis for developing the ready-to-run workshops, and is the input for subsequent tasks in WP3/WP4 in the second cycle. Thepartner methodologies (provided in Appendices) show some of the activities envisioned thatcan be placed within the structure to develop the different workshops.This report consists of an account of the best practices review process and its outcomes (section2), an overview of the design principles (section 2.2.1), an outline of the workshop structure andits rationale (section 3), and a discussion of the next steps in the process (section 4) and theperceived risks (section 4.1) and implications (section 4.2). The current version of the design
  8. 8. METIS PROJECTD3.2: Draft of pilot workshop Page 8 of 64prototype, as well as some of the best practices data, are includes as appendices (indexed insection 6).
  9. 9. METIS PROJECTD3.2: Draft of pilot workshop Page 9 of 642 Review of best practices2.1MethodD3.1 (Brasher & Mor, 2013) identified a set of workshop aims for participants, a selection ofsuccessful approaches and a set of design principles. These were derived from a survey oflearning design experts, a series of 4 semi-structured interviews of representatives of usergroups and a workshop held on 9thJanuary 2013 in which representatives of user groups andother project partners were guided in describing their context and characterising theirprospective workshop participants. These aims, approaches and principles have guided thedevelopment of the draft design of METIS workshops below. In parallel to working on thedesign, we have continued the process of reviewing past and current practices, which informsour design work and will continue to feed into it as we proceed into greater detail.Our aim is to capture these best practices in the form of transferable design knowledge, whichcan then be applied to the core generic workshop design, used in the process of customising thisdesign to the needs of the user groups, and could also be shared with anyone who wishes todesign their own learning design workshops.To achieve this aim, we choose to represent this design knowledge using a combination ofoverview descriptions, design narratives, design principles and design patterns. This approach isbased on the SNaP! Methodology (Mor, 2013)2and the PPW methodology (Mor, Warburton andWinters, 2012)3.A design narrative is an “account of critical events in a design experiment from a personal,phenomenographic perspective”4. A design pattern “describes a recurring problem, or designchallenge, the characteristics of the context in which it occurs, and a possible method ofsolution”5. A design principle is “…an intermediate step between scientific findings, which mustbe generalized and replicable, and local experiences or examples that come up in practice.” (Bellet al, 2004, p. 83, in Kali, 2009)6. Together with the overview descriptions, these cover a full arcfrom the pedagogical framework, through the high-level design, and down to specific activities.The design principles will be declared upfront in our workshop design documentation and serveas a pedagogical contract between the workshop designers, facilitators and participants:defining a set of mutual expectations regarding the roles and interactions between them. We2Mor, Yishay (2013). SNaP! Re-using, sharing and communicating designs and design knowledge using scenarios, narratives andpatterns. In: Luckin, Rosemary; Puntambekar, Sadhana; Goodyear, Peter; Grabowski, Barbara L.; Underwood,Joshua and Winters, Niall eds. Handbook of Design in Educational Technology. London, UK: Routledge, (In press).3Mor, Yishay; Warburton, Steven and Winters, Niall (2012). Participatory pattern workshops: a methodology for openlearning design inquiry. Research in Learning Technology, 204
  10. 10. METIS PROJECTD3.2: Draft of pilot workshop Page 10 of 64will review the general flow of the workshop as well as the individual activities to ensure thatthey are compliant with these principles.The design patterns will be used to mold specific activities, or even elements of activities. Theywill also be offered as a resource for workshop facilitators as an aid in customizing the workshopdesign, and as a resource for participants as an illustration of an exemplar learning designprocess.The end product of this process with be deliverable D3.4 “Final workshops packages: workshopsfor different educational levels and education contexts”. This will contain instructions for thetrainers as to how to run the workshop; a sequence of activities for the trainer and the trainees;a description of the rationale and pedagogical methodology on which the workshop is based;and attached learning resources to be used in the workshop.The review process consists of the following phases:Phase 1. Collating overviews of the workshop methodologies, with example resources andlinks and references to detailed descriptions (completed, see Appendix 2)Phase 2. Collation of design narrative of successful workshop practices (completed, seeAppendix 3)Phase 3. Identifying design principles in the design principles database and in the overviews(completed, see section Results’).Phase 4. Extraction of initial design patterns from the design narratives, to be achieved byanalysis and cross-comparison of narratives.Phase 5. Refinement, elaboration and substantiation of the design patterns and designprinciplesPhase 6. Identifying the implications of the design patterns and principles for WP 2 (ILDE),WP 4 (enactment) and WP 5 (evaluation).Phase 7. Applying the patterns and principles to the workshop design.Originally, it was planned to complete this process before elaborating the workshop design.However, our discussions with WP2 and WP5 suggested that they need an early draft of thedesign in order to align their work with it. Furthermore, it has become evident that the usergroups need to be engaged early on with the general framework of the workshop design toverify that they can commit to its structure. Consequently, we are proceeding with themethodology review and the workshop design in parallel rather than sequentially.Initial results from phases 1 to 4 of the review process are presented in section Results’. Thenext steps required to support this process through phases 5-7 are:Step 1. WP3: Provide guidance to partners to embark on a more robust and critical of thenew design patterns using the additional guidance provided (completed June 4,2013);
  11. 11. METIS PROJECTD3.2: Draft of pilot workshop Page 11 of 64Step 2. ALL: Elicit/evoke additional and refined design narratives;Step 3. All: Elicit/evoke additional and more refined design patterns into the ILDE byrefining/building on the design patterns that have already been generated. We willdo this by providing high quality and relevant examples from workshops conductedby the OU to work as a catalyst to evoke the additional examples;Step 4. All: Migrate the rest of the original design narratives into the ILDE;Step 5. WP3: Author a design principles document drawing on the work already completedand the expected work to be completed in 1-5 above;Step 6. WP3 and User groups: Use the design principles, patterns and narratives to informfuture versions of the generic as well as the user-group specific workshop designs;Step 7. WP3 and WP2: review the patterns and principles, and consider their implicationsfor the ILDE design and implementation;Step 8. OU and WP4: review the patterns and principles, and consider their implications forthe workshop enactment (WP4);Step 9. OU and WP5: review the patterns and principles, and consider their implications forthe Metis evaluation strategy;2.2 ResultsAppendix 2 lists the methodology descriptions. Appendix 3 presents the procedure for collatingthe design narratives (including our design narrative template). Appendix 4 offers a sample ofdesign narratives of successful workshop activities. In section 2.2.1 we describe the designprinciples which have been extracted so far.We initiated the process using the project internal collaborative work space( We collected 12 design narratives there (Figure 1).
  12. 12. METIS PROJECTD3.2: Draft of pilot workshop Page 12 of 64Figure 1: screen shot of design narrative repository on the project internal work space.In parallel, ILDE was launched for internal beta, and we decided to shift our activity there. Webegan migrating the design narratives to ILDE, and in parallel began extracting patterns in ILDE.Currently, we have 10 design narratives in ILDE (Figure 2).Figure 2: Design narratives in ILDESo far, we have identified 6 design patterns (Figure 3)
  13. 13. METIS PROJECTD3.2: Draft of pilot workshop Page 13 of 64Figure 3: Design patterns in ILDEFigure 4 shows an example of a design pattern in ILDE. This pattern is not yet available for publicviewing. Another pattern, which has been published and is available for public viewing, can befound at:
  14. 14. METIS PROJECTD3.2: Draft of pilot workshop Page 14 of 64Figure 4: example design pattern in ILDE2.2.1 Design principlesTable shows a first set of design principles extracted during phase 3. This table shows the sourceof each principle, the principle itself, and some remarks on how it has or could be applied in theworkshop design presented in Appendix 1.Source Principle ApplicationPrinciples fromthe Educationalprinciples databasethat UVA havefound useful intheir workshopsand can be appliedto the METISworkshops1. Build on student ideas( Reuse student artifacts as resourcefor learning ( Integrate online with offlineactivities ( METIS workshops will be on aspecific theme relevant toparticipants’ context, and willfocus on participants owndesign problems. During the workshopparticipants will be encourageto share resources producedduring a workshop boththrough the ILDE and face-to-face. This principle isembodied in activity 9
  15. 15. METIS PROJECTD3.2: Draft of pilot workshop Page 15 of 64il.php?prKey=330)4. Connect to personally relevantcontexts (‘Evaluate’ in which participantswill evaluate and learn fromothers’ designs. The workshop design includesactivities which use bothonline and offline tools. METIS workshops will be on aspecific theme relevant toparticipants’ context, and willfocus on participants owndesign problems.Additionalprinciples fromthe Educationalprinciplesdatabase that UPFhave found usefulin their workshopsand can be appliedto the METISworkshops5. Encourage learners to learn fromothers ( Employ multiple social activitystructures ( This principle is used explicitlyin activities 9, 10, and underliesall the other activities as theyare collaborative in nature. The workshop design includesmultiple social activitystructures. The emphasis is onworking in small groups, withindividual activity and wholeclass activity occurringoccasionally (e.g. activity 2,activity 4).Principles fromreviewing the LDSmethodology:8. Continued work on achallenge/design project: "the mainactivity of a course is the studentscontinued work on designchallenges in a defined domain ofpractice" (akin to project-basedlearning)9. Public review of group artifacts:"classroom sessions are mostlydedicated to group work and publicreview of design artefacts"10. Iterate!: implicit in "continued workon design challenges in a defineddomain of practice" METIS workshops will be on aspecific theme relevant toparticipants’ context, and willfocus on participants owndesign problems. This occurs in activities 4 and 9. Iteration is implicit in theworkshop design (see figure 2),but time limitations may limitthe iteration that is possibleduring the workshop itself.Some principlesfrom CARDETsDesign-Practicemethodology:11. Include an early "how to ruin X"activity to get people startedthinking about the topic X12. Break down a (part of a) learningdesign to a full detail level -- this isan important activity to getteachers started thinking in detailabout design decisions andimplications of those decisions (seealso this paper)13. Pitching results -- Important to letpeople pitch their results and havepeers discuss those. In the CARDET Included as activity 2. Creating a fully detailed designoccurs in activity 8 ‘Prototype’. This occurs in activity 9,‘Evaluate’.
  16. 16. METIS PROJECTD3.2: Draft of pilot workshop Page 16 of 64Table 1: design principles, their source, and comments on their application to the workshop design3 Outline of draft design of the METIS WorkshopsBased on the aims, approaches and principles outlined in D3.1 (section 3, Brasher & Mor, 2013,pp. 8-11), we have proceeded to iteratively develop the design for the METIS workshops. Earlysketches of this design have been presented to partners and user groups for their review andfeedback. The current state of the draft design is presented in Appendix 1.This is a generic, or “template” design, in the sense that it needs to be customized for each ofthe user groups to ensure it meets local needs and contexts. The process of customization willinvolve further discussions and negotiations with each user group so as implement contextspecific versions of the activities described in the template during WP3’s work towards D3.3 (thethree ready to run workshop structures). The design decisions taken in this process will berecorded by WP3 and used to produce guidance for customisations for application of thetemplate to other contexts. However, it sets a common framework which we see as necessaryfor an effective workshop:A METIS workshop consists of a sequence of collaborative activities, spanning 6-8 hours. Thesecan be conducted as one full day event or two half day events. The workshop requires a spacewhich is set up for group work. It is optimized for 4-5 groups of 3-6 participants each, overall 15-25 participants. Ideally, it would need a facilitator per every 5 participants. To be effective, aworkshop will require participants to invest 1-2 hours pre-event and 1-2 hours post event.methodology it soundsasynchronous, but also veryrelevant in a synchronous/f2fsetting.Some principlesfrom ICOPERmethodologies:14. Solving a proposed task (e.g.,providing a narrative example tosolve) -- helpful when trainingspecific skills associated toimproving proficiency around theelements of a design modellinglanguage / tool.15. Improving participants previousdesigns -- participantsreconceptualise their actual designs(courses) using theoretical inputprovided gradually. Examples are provided inactivity 5 ‘Evidence andexamples of ‘X’’, and alsothrough patterns in activity 7.However, decisions need to betaken about how thesepatterns and examples areinstantiated and present to thedifferent user groups. Iteration is implicit in theworkshop design (see figure 2),but time limitations may limitthe iteration that is possibleduring the workshop itself.Theoretical input is providedgradually (e.g. activities 5 and7).
  17. 17. METIS PROJECTD3.2: Draft of pilot workshop Page 17 of 64The current draft of the workshop design is presented as a word document (Appendix 1) and aCompendiumLD7archive. This format is easy to share, manipulate and edit.4 Next steps and implications for other work packagesThe next phases of the WP 3 design and development process will be:1. Discussing this draft with partners and user groups, and reviewing their feedback(This work is part of part of Task T3.4, face to face discussion to occur at the METISmeeting in Barcelona, July 2013; preliminary online discussion already underway).2. Work with tool providers for CADMOS, OpenGLM, and WebCollage on integratingdetailed and meaningful activities using these tools into the current workshop structure.3. Specifically, consider ways of incorporating the collaborative learning flow patterns(Hernández-Leo, Asensio-Pérez, Dimitriadis, & Villasclaras, 2010) both into the workshopdesign and as meaningful resources for workshop participants.4. Incorporate the outcomes of the “best practices” cross-review process, described insection 2.5. Based on the feedback and the “best practices” review, deploying a revised design to ashared Virtual Learning Environment (VLE). (This work is part of Task T3.5, to occur inproject months 8 – 10, June –August 2013).6. Run an open survey of potential participants to assess their interest in the topics andstructure which emerged so far (an initial survey of the 3 user partners representativeswithin METS highlighted the topics of formative assessment, collaborative learning andproject based learning). (This work is part of Task T3.4, the survey will be launched inJune 2013, project month 8).7. Work with user groups to customize this design per their specific needs, concern andconstraints (coordinated by WP4). The workshop design draft may be modified inresponse to the requirements for customisations, so that it remains a generic templatefor the customisations that are produced. The customised instances of this meta designwill be delivered as METIS Deliverable D3.3 (month 11). (This work is part of Task T3.5, tooccur in project months 8 – 10, June –August 2013).8. Work with WP5 on evaluation procedures for the design. On-going communication withand reporting to WP5 about the evolution of the workshop design.4.1RisksIn the course of the work on the workshop design we have identified several risks which need tobe mitigated. We are collaborating with WP 4 (‘Report on the pilot workshops and LDenactment’) on addressing these:7CompendiumLD is a a software tool for designing learning activities using a flexible visual interface that can bedownloaded from This site also provides a ‘getting started’ guide for new users.
  18. 18. METIS PROJECTD3.2: Draft of pilot workshop Page 18 of 641. User groups need to commit to one full day / two half day workshops. Initialdiscussions with user groups suggest this may be difficult for some of them. However,we are convinced that it would be ineffective to run shorter workshops.2. User groups need to recruit sufficient participants and facilitators for each workshop.In order to properly evaluate and revise the design between the two cycles, we will need3 workshops (1 per user group) of 10-20 participants in the first and second rounds.These workshops will require 2-3 facilitators each. In addition, we note that after theworkshops least one participant from each workshop must use the ILDE in their ownteaching practice, applying techniques they have learnt through the workshops.3. Input from user groups required to customize and add content to their instance of theworkshop design. Some of the activities are common, some are specified in a genericmanner, and need to be elaborated per the specific topics, context and concerns ofevery workshop. Furthermore, user groups may want to translate the workshopsmaterials to their local language.4. User groups need to accommodate diverse technical skills. The workshop designassumes a basic level of computer and internet proficiency. The workshops aresupported by a VLE, the ILDE and a variety of computational tools. Some of theparticipants may not have the required skills, and the user groups will need toaccommodate them by prior training or by grouping them with other participants whocan assist them.4.2 Implications for and from WP 2Table 10 in D2.1 (Hernández-Leo, Asensio, Chacón, & Prieto, 2013) describes the WP2 reviseduse cases and requirements stressed by the user groups. These revised use cases are listedbelow, along with a comment indicating how the workshop structure reported herein relates tothe use case. We also make a note of any specific requirements for WP2 to bear in mind. Someof the activities require specific resources to be provided in the ILDE. We will coordinate thesewith WP2 after the next round of reviews and edits.(1) Choose a learning design tool between several available. The skill to select anappropriate design tool will be developed through several of the activities that occurwithin the workshop structure. For example in Activity 7 participants chose one or moredesign patterns appropriate to the requirements of their design vision. In this activitythey are also introduced to the ILDE tools with the appropriate affordances to producerunnable designs from these patterns.(2) Produce a learning design by both adapting reused designs and starting from scratch.In the workshop structure we have described, we envisage that workshop participantswill produce many design artefacts during the design process, prior to and during thecreation of the runnable design. These artefacts will be sketches, notes, evaluationchecklists and so on.
  19. 19. METIS PROJECTD3.2: Draft of pilot workshop Page 19 of 64(3) Co-produce a learning design. The workshop structure is built around team work andcoproduction.(4) Share a learning design. The ILDE tools facilities for sharing will be used in Activity 9.Designs produced during the workshop will be shared amongst all the teamsparticipating in the workshop and participants will be would encouraged to share themmore widely afterwards.(5) Instantiate a learning design. Instantiating a learning design occurs during Activity 8‘Prototype’.(6) Deploy an instantiated learning design into chosen VLEs. Deploying a learning designoccurs during Activity 8 ‘Prototype’.(7) Provide feedback and reflections. In Activity 9 participants evaluate each other’s designsusing the checklists developed during earlier activities.(8) Explore designs, instantiations and feedback. This is the focus of Activities 9 and 10 inthe workshop structure.5 ConclusionsWe have completed the first cycle of the workshop design and development. This deliverable isthe end product of that cycle, and consists of package of resources representing the workshopstructure developed from the best practices identified in Deliverable D3.1 (Brasher & Mor,2013). This package includes: a sequence of activities for the trainer and the trainees; adescription of the rationale and pedagogical methodology on which the workshop is based; andattached learning resources to be used in the workshop.This package provides a meta-design for the Metis workshops. Over the next few months wewill use this as the basis for developing the detailed designs of the actual workshops to be runby and for the user groups.In the course of our work towards this deliverable, we identified a need to deepen and broadenour review of best practices. Due to the high level of interdependencies between the workshopdesign and other work packages, we decided to prioritise progress on the workshop design, andcontinue the review of best practices in parallel.Also, during the work on the design and review of best practices, the ILDE had moved intoprivate beta phase. Once we verified that it is stable and the basic functionality is in place, wedecided to shift our work from the project’s internal workspace to the ILDE. This entailed atransition cost, but will allow us to develop our expertise in using ILDE, and will provide WP2with “real world” testing and evaluation.Our next goals would be to synchronise the workshop design with the ILDE development, anduse the generic design here to develop the structures for the three pilot workshops (D3.3). Wewill continue our review of best practices, and these will feed into the design processcontinuously.
  20. 20. METIS PROJECTD3.2: Draft of pilot workshop Page 20 of 646 AppendicesAppendix 1: WorkshopdesignAppendix 1The sequence of activities, rationale and pedagogicalframework, and associated resources. This will form the basisfor the structure of the three pilot workshops.Appendix 2: MethodologydescriptionsHigh-level descriptions of the workshop methodologiesconsidered in the review of best practices.Appendix 3: Methodology designnarratives and design patternsProcess and template used for collecting the methodologydesign narratives.Appendix 4: Sample ofMethodology Design NarrativesExample of design narratives collected in the review of bestpractices.Appendix 5: Personas template An example resource used in a workshop activity. Also used byus as an aid in designing the workshops.Appendix 6: Survey of usergroupsSummary of data collected to ensure the workshops addressuser concerns and constraints.7 ReferencesBrasher, A., & Mor, Y. (2013). METIS deliverble D3.1: Report 2 on meetings with user groups:Early feedback on candidate best practices for teacher training on learning design.Cross, S., Galley, R., Brasher, A., & Weller, M. (2012). OULDI-JISC Project Evaluation ReportRetrieved 3/8/2012, from (2009). Lifelong Learning Programme Key Action 3: Information and CommunicationTechnologies (ICT) Retrieved 22/5/2013, from, R. (2010). Activity: 30 mins: Learning Outcomes view Retrieved 9/5/2013, fromández-Leo, D.; Asensio-Pérez, J.I.; Dimitriadis, Y.; & Villasclaras, E.D. Generating CSCLScripts: From a Conceptual Model of PAttern Languages to the Design of Real Scripts. In:Goodyear P.; Retalis, S. (eds.). Technology-Enhanced Learning, Design patterns and patternlanguages, Sense Publishers, Series Technology-Enhanced Learning; 2010. p. 49-64. Appendixavailable atández-Leo, D., Asensio, J. I., Chacón, J., & Prieto, L. P. (2013). METIS deliverble D2.1: Report1 on meeting with stakeholders: early feedback on ILDE requirements.Jenkins, H. (2009). Confronting the Challenges of Participatory Culture: Media Education for the21st Century The John D. and Catherine T. MacArthur Foundation Reports on Digital Media andLearning (pp. 146). Retrieved from
  21. 21. METIS PROJECTD3.2: Draft of pilot workshop Page 21 of 64Kelly, A. E., Lesh, R. A., & Baek, J. Y. (2008). Handbook of Design Research Methods in Education.New York: Routledge.Laurillard, D. (2008). The teacher as action researcher: using technology to capture pedagogicform. Studies in Higher Education, 33(2), 139-154. doi:, H., Oliver, M., & Hadjithoma-Garstka, C. (2009). The role of research in institutionaltransformationTransforming Higher Education through Technology-Enhanced Learning. York,UK: Higher Education Academy. Retrieved from
  22. 22. METIS PROJECTD3.2: Draft of pilot workshop Page 22 of 64Appendix 1. Workshop designMETIS Workshop Design (version 0.7)SummaryDespite the abundance of high-quality educational technology, and the wealth of researchdemonstrating its value, educators are still struggling to make effective use of thesetechnologies and the associated pedagogies in their daily practices. Arguably, this is a gap in thecapacity for learning design: educators need the tools and competencies which would allowthem to identify educational challenges, describe the context in which they arise, identify theopportunities afforded by technology, project the insights derived from research, and devisenew learning experiences. To address this gap, educators need tools and practices. Tools thatwould support them through the cycle of learning design – from conception to deployment andevaluation of techno-educational innovations. Professional practices that use such tools toensure the robustness and effectiveness of their innovations and make learning design a dailyhabit and part of their professional identity. The METIS project ( aimsto contribute to this aim, by providing educators with an Integrated Learning DesignEnvironment (ILDE) and a workshop package for training educators in using the ILDE to supporteffective learning design.This document describes the aims, organisational requirements and activity structure for METISlearning design workshops. The aims are specified in terms of outcomes for participants and theMETIS project. The organisational requirements describe the human and other resourcesrequired to run a workshop. The activity structure describes a reusable structure of activities toenable participants to reach the learning outcomes specified. It includes a description of thetools and the information resources for both participants and facilitators for each activity.Workshop aims and organisation1 Learning and other outcomesOur research of user concerns suggests that practitioners have limited interest in training onlearning design in general, but are have much higher interest in learning design for specificthemes, such as collaborative learning, formative assessment or project-based learning (Brasher& Mor, 2013). Hence, the workshop design presented here is a meta-design that is flexible andcan be customised to a specific theme. In this document, this theme is noted as ‘X’.The intended learning and other outcomes for a METIS workshop on learning design for ‘X’ areshown in figure 1. The learning design is intended to be applicable across a range of topics.
  23. 23. METIS PROJECTD3.2: Draft of pilot workshop Page 23 of 64Figure 5: Learning and other outcomes of the workshop2 Workshop organisation2.1 DurationThe total duration of the activities in the workshop(s) is 7.5 hours (Table 2). The activities needto run in the order they are presented in the ‘Workshop activity structure’ section, but they canbe split into two sessions ( 4 hour workshop including activities 1 to 6 followed by one3.5 hour workshop consisting of activities 7 to 11).Activity Hours1 Introduction 0.32 How to ruin a course / pedagogical features 0.333 Personas 0.54 Barriers and challenges 0.665 Evidence and examples of X 0.56 Initiate, Ideate, Investigate: produce your Vision 1.57 Connect: gather tools and resources 0.758 Prototype 1.5
  24. 24. METIS PROJECTD3.2: Draft of pilot workshop Page 24 of 649 Evaluate 0.7510 Reflect 0.511 Wrap up 0.25Total 7.74Activities 1 to 6 (half day workshop 1: Context and vision,plus wrap up) 3.79Activities 7 to 11 (half day workshop 2: Prototype andevaluate, plus intro and ice breaker) 3.25Table 2: activity durations2.2 People: Participants, facilitators and othersParticipants work in teams composed of 3 to 6 members. This team size encourages andfacilitates each team member to be fully involved in design discussions throughout theworkshop. A team size of 3 to 6 members is optimal because it allows for a diverse range ofviews to be debated. Teams of more than 6 often split up into smaller teams and lose focus; thisshould be avoided if possible.It is highly advisable to allocate a ‘critical friend’ to each team to challenge design thinking andstimulate focused and informed discussion (Cross, Galley, Brasher, & Weller, 2012, p. 28). This‘critical friend’ role could be played by the workshop facilitators circulating amongst the teams,but if there are many more teams than facilitators it is advisable to recruit additional ‘criticalfriends’ to ensure that each team can benefit from their input and challenges. A learning designworkshop facilitator must have detailed knowledge of the both the workshop topic and theIntegrated Learning Design Environment (ILDE) so he/she can answer any question that mightarise during the workshop. A distinction needs to be made between a critical friend and afacilitator. A critical friend is required to have knowledge of the topic in order to prompt andencourage a focused and robust discussion. Obviously, some awareness of the ILDE will beuseful for critical friends. A learning design workshop facilitator is an expert on both the topicand the ILDE.2.3 ResourcesEach team needs access to one or more laptop or desktop computers with internet access tointeract with the ILDE and other online resources. Prior to the workshop starting, an emptylearning design for each team will be created in the ILDE, and all team members will be givenediting rights for that learning design. If other resources such as printed material and specificonline resources are required for a particular activity, they are described within the relevantactivity in the activity structure that follows.
  25. 25. METIS PROJECTD3.2: Draft of pilot workshop Page 25 of 64Workshop activity structureA diagram of the structure of the participants’ activities during the workshop is shown in Figure2. This illustrates the relationships between the activities, the tools to be used, the resources tobe produced.
  26. 26. METIS PROJECTD3.2: Draft of pilot workshop Page 26 of 64Figure 6: Participants activities, tools used, and resources produced1 Introduction (0.30 hour)Facilitators introduce themselves, describe the structure of workshop, the organisation in termsof the available human resources, tools and information and the intended learning outcomes.They also introduce the critical friends and explain their role in workshop.Facilitators’ resource: slideshow with notes, ILDE.2 How to ruin a learning experience (0.33 hour)(Note: this activity reuses the OULDI activity icebreaker activity will focus on the key issues and strategies that impact on the success (orotherwise) of learning and teaching within their context. The output of this activity will be adesign checklist which can be used as one of the design evaluation tools in a mid-way designreview and at the end of the workshop.Resources (all to be available online and in printed form at the workshop)Participants’ resources: Instructions1. List the 10 best ways to ensure that the learning experience you are designing willfail! (or: 10 ways technology can ruin the learning experience you are designing)2. Share these with the others by uploading your lists to the ILDE.3. What are the key themes?OutputA first version of a design checklist which can be used as one of the design evaluation tools in amid-way design review and at the end of the workshop. The checklist will be uploaded to theILDE.3 Personas (0.5 hour)“Personas are a tool for sharing our understanding of our expected users, as a starting point fordesign” (Mor, 2013). In this activity the participants will create descriptions of 2 or 3 personasrelevant to the context in which they teach. This activity is a first step towards a detailedspecification of the context in which the “X” learning activity will occur.Resources
  27. 27. METIS PROJECTD3.2: Draft of pilot workshop Page 27 of 64 Printed and online versions of the Personas template (appendix x) will be provided forparticipants. (The online version will be in the ILDE).OutputThe personas created will be uploaded or linked to the ILDE.4 Barriers and challenges (0.66 hour)20 minutes team work, 20 minutes plenary (assuming 3 or 4 teams)In this activity the participants are asked to describe what they see as the barriers andchallenges with respect to designing and running ‘X’ learning activities in their context.Facilitators refer to the activity “How to ruin a course”, where they will have already created alist that is likely to be applicable to any type of course or activity. This activity promptsparticipants to relate their current understanding of the workshop topic (X) to the context inwhich they teach. It also allows the facilitators and critical friends to gain an understanding ofeach team’s context. The participants will use the personas created in the “Personas” activity todiscuss and answer the following questions for a learning activity on ‘X’ (e.g. collaborativelearning)a) What are the barriers and challenges from a learners’ perspective of ‘X’?b) What are the barriers and challenges in implementing ‘X’ in your course or topicarea?This will be a collaborative mapping exercise: Each participant writes down 3-4 barriers and 3-4 motivations, each one on a separatepost-it (5 minutes). Participants place their post-its on an A1 paper and arrange them in some order or map(15 minutes). Each group presents its map to the whole workshop (10 minutes) Each participant thinks about their own view, discusses them with one other memberand works collaboratively towards a team list of at least 5 barriers/challenges forlearners and 5 for barriers/challenges for implementation. (20 minutes)Each team shares their list with the wider group. These will be shared with the other teams bydisplaying them in the workshop and each team will describe one from the learners’ perspectiveand on from the implementation perspective in the plenary (one only to limit the time spent onsharing). Facilitators foster focused discussion of remaining barriers/challenges betweendifferent teams to occur during breaks.Resources (all to be available online and in printed form at the workshop)
  28. 28. METIS PROJECTD3.2: Draft of pilot workshop Page 28 of 64 Facilitators’ resources: slideshow of questions (as above). Participants’ resources: slideshow with questions as above with notes indicating themesthat they might discuss (e.g. technical, motivational, temporal challenges). One A1 sheetand several post-it packs per team(As an alternative to post-its, participants can use aconcept mapping tool of their choice.) Instructions1. As individuals, write down 3-4 barriers and 3-4 motivations, each one on a separatepost-it. Use green post it notes for barriers/challenges from learners’ perspective,yellow post-it notes for barriers/challenges for implementation (5 minutes).2. Share these with the others in your team by placing your post-its on an A1 paper andcollaborate to arrange them in some order or map (15 minutes)3. Each team should present its map to the whole workshop; focus on describing up to5 barriers/challenges for learners, and up to 5 barriers/challenges to implementationfrom the map (20 minutes) Critical friends’ resources: document describing suggestions for questions to ask.OutputEach team should produce a list of up to 5 barriers/challenges for learners, and a list of up to 5barriers/challenges to implementation. These will be added to the output produced in the firstactivity ‘How to ruin a course’ to produce a new version of the team’s evaluation checklist5 Evidence and examples of ‘X’ (0.5 hour)The facilitators present a few examples of X that are chosen so as to be relevant to theparticipants. (four or five examples should be sufficient). Evidence demonstrating that eachexample is effective is also presented.Each team selects one or two for review, and notes design features that may be transferred totheir context.Facilitators and critical friends will support the discussion by pointing out particular aspects ofthe example designs critical to the success of the example.Resources Slides showing the example designs, descriptions of their use, and evidence of theirsuccess. Copies of academic papers will also be provided. The focus of the presentation will varydepending on the participants, and will vary from workshop to workshop.
  29. 29. METIS PROJECTD3.2: Draft of pilot workshop Page 29 of 64OutputNotes on potentially useful design features added to the ILDE.6 Initiate, Ideate, Investigate: produce your Vision (1.5 hours)In this activity each team will describe their vision for a ‘X’ activity. This is a first draft, and it maybe modified during the workshop. This vision should focus on describing the effects the activityis intended to have on the learners (not on how these effects will be achieved which is the focusof activity 8 ‘Prototype’).Resources Printed templates for the Course Map (see appendix B.5 in Hernández-Leo, Asensio,Chacón, & Prieto, 2013) need to be provided to participants and they will be able tocomplete this within the ILDE. For the Learning Outcomes View (Galley, 2010) A3 sheets of paper and CompendiumLDicon post-it notes need to be provided, along with a guide showing participants how tocreate a Learning Outcomes View in CompendiumLD should participants wish toproduce a digital version..OutputThe vision should be described in terms of Learning outcomesOther outcomes (e.g. affective outcomes such as individual motivation, confidence, teambuilding) Leaners’ outputsAn initial description of the evidence required to indicate that the learners have reachedthe learning outcomes. Examples include a written piece of work, and an observablebehaviour.Participants will produce representations such as Course Map and Learning Outcomes View.The Learning Outcomes View is probably most appropriate as that view is very simple toproduce, and gives a specification that a single activity can be evaluated heuristically against. Incontrast the Course Map is intended to describe complete courses as a whole such as a courseof 100 or more hours of study time. The Course Map could still be applied to show the contextfor the ‘X’ learning activity that the participants will design. Other representations of contextcould also be introduced (e.g. factors and concerns table) to help focus participants’ attentionon consideration of the context where the design will be run.
  30. 30. METIS PROJECTD3.2: Draft of pilot workshop Page 30 of 647 Connect: gather tools and resources (0.75 hour)By this stage of the workshop participants will have an idea of the kind of behaviours they wanttheir learners to demonstrate, as well as the kind of products/competencies they desire learnersto produce/become proficient in during the learning activity they are collaboratively designing.These need to be described in the learning outcomes view.In this activity the learners are shown a set of ‘X’ learning patterns. The examples used in the “5Evidence and examples of ‘X’” activity will provide concrete examples of some or all of thepatterns that are now made available to the participants.Each team will select one or more patterns which best suit their articulated Vision as defined bythe Course Map, Learning Outcomes view, Personas and the barriers and challenges that theydeveloped during the preceding activities. If more than one pattern is chosen it could bebecause there are alternatives, or because the team thinks that a sequence of ‘X’ activities isnecessary.They now begin the detailed design. Questions that should be used as prompts for this sessioninclude: Which parts of the activity should be synchronous, and which should be asynchronous? Which tools have the right affordances for your activity?This could be carried out further using a Think-Pair-Share activity.Resources A set of design patterns for X. These should be provided both online and in printedform.OutputThe selected pattern(s), a set of resources that are to be used within the activity and anannotated diagram of the pattern(s) showing: asynchronous and synchronous stages within the design where particular tools and resources are used.8 Prototype (1.5 hour)Participants work together to produce a prototype of an ‘X’ activity. The prototypes illustratetasks the facilitator and learners may choose to undertake. Prototyping would use a suitableILDE tool or be carried out using the technology of the team’s choice (including paper).Resources
  31. 31. METIS PROJECTD3.2: Draft of pilot workshop Page 31 of 64Participants’ resources: a guide to the affordances of the ILDE instantiation and deploymenttools which explains which tools may be best suited to instantiate and run patterns of type X.9 Evaluate (0.75 hour)An appropriate summary view of each team’s design is selected by the team to be shared andevaluated. The summary view to be used will depend on ‘X’ (e.g. if ‘X’ is Collaborative Learningthe appropriate summary view will be generated by Web Collage).Designs are swapped amongst each group and evaluated heuristically ( and critical friends advise on how to apply the teams evaluation checklist to thedesign in question.The final step is to share the result of the evaluation back to group as a whole.ResourcesParticipants’ resources: the design of another team, along with the same team’s evaluationchecklist (i.e. the output from Activity 4).10 Reflect (0.5 hour)In the last activity (‘Evaluate’), each group’s design was evaluated by another group, and theresults of the evaluation shared. Now everyone should have an understanding of all the designs.Participant groups should now reflect on if and how their activity design could be connected toother groups’ designs to produce a course.11 Wrap up (0.25 hour)The facilitators conclude by describing how participants can find out more about learning designin general, and the ILDE in particular.ResourcesFacilitators’ resource: slideshow with notes, ILDE.ReferencesCross, S., Galley, R., Brasher, A., & Weller, M. (2012). OULDI-JISC Project Evaluation Report.
  32. 32. METIS PROJECTD3.2: Draft of pilot workshop Page 32 of 64Retrieved 3/8/2012, from, Y. (2013). Personas - The Learning Design Grid, Retrieved 9/5/2013, from
  33. 33. METIS PROJECTD3.2: Draft of pilot workshop Page 33 of 64
  34. 34. METIS PROJECTD3.2: Draft of pilot workshop Page 34 of 64Appendix 2. Methodology descriptionsThis appendix lists the high-level descriptions of methodologies which formed thebasis for the review of best practices. For each methodology, we compiled anoverview and an index of key references.a. Learning Design Workshop Methodologies from the ICOPER BestPractice NetworkSummaryTwo methodologies were developed and then delivered several times during theICOPER eContentplus project8by the people in the Instructional Design workpackage. ICOPER was about interoperability and the alignment of all phases in theinstructional analysis, design and deployment lifecycle based on intended andachieved learning outcomes.(1) The first series of workshops was targeted at propelling the learning designproficiency (in particular IMS LD) at partner institutions. We gave them a briefnarrative description of a concrete unit of learning. We presented the idea oflearning design and the set of elements that are used in LD to abstractly describea teaching/learning process [1]. We then asked them to describe the unit oflearning using the IMS LD level A + B concepts (i.e., role, activity, activitystructure, environment, role-part, property, condition). For results see [2].(2) The second series of workshops was to give teachers at partner institutions ahands-on feeling of how to think of their teaching / courses in terms of conceptsthat are demanded by sound instructional design (alignment of learningoutcomes, teaching methods, and assessment) as well as specified in theEuropean Qualification Framework for lifelong learning (EQF). The latter wasused in particular to foster understanding of the concept of learning outcomesand how to formulate those. The procedure: we gave them some background onlearning outcomes (from EQF perspective), then some theoretical input on howto align intended outcomes, methods and assessment [3]. Then we had them"re-conceptualize" one of their actual courses using the provided input.Both workshops were heavily based on hands-on work based on concrete problemsfrom the participants context. While we had workshop procedure (1) in bothsoftware-based and paper-based settings, we conducted workshop procedure (2)mostly paper based using simple, uniform templates.8
  35. 35. METIS PROJECTD3.2: Draft of pilot workshop Page 35 of 64Key references[1] M. Derntl (2010). ICOPER IMS LD Learning Design Hands-on Workshop.Presentation recording: andslides:[2] M. Derntl, S. Neumann, D. Griffiths, P. Oberhuemer (2012). The ConceptualStructure of IMS Learning Design Does Not Impede Its Use for Authoring.Transactions on Learning Technologies, 5(1): 74-86[3] M. Derntl, S. Kabicher, P. Oberhuemer (2010). Workshop on outcome basededucation. Workshop material. Available at METIS internalsite: 7Cs of learning design frameworkSummaryThe 7Cs of learning design framework (Figure 1) illustrates the key stages involved inthe design process, from initial conceptualisation of a learning intervention throughto trialling and evaluating it in a real learning context.
  36. 36. METIS PROJECTD3.2: Draft of pilot workshop Page 36 of 64Figure 1: The 7Cs of Learning Design FrameworkThe framework consists of the following stages:1. Conceptualise: Vision for the course, including:a. Why, who and what you want to designb. The key principles and pedagogical approachesc. The nature of the learners.2. Capture: Finding and creating interactive materials, including:a. Undertaking a resource audit of existing OERb. Planning for creation of additional multimedia such as interactivematerials, podcasts and videosc. Mechanism for enabling learners to create their own content3. Communicate: Designing activities that foster communication, such as:a. Looking at the affordances of the use of different tools to promotecommunicationb. Designing for effective online moderating4. Collaborate: Designing activities that foster collaboration, such as:a. Looking at the affordances of the use of different tools to promotecollaborationb. Using CSCL (collaborative) Pedagogical Patterns such as JIGSAW, Pyramid,etc.5. Consider: Including three elements:a. Designing activities that foster reflectionb. Mapping Learning Outcomes (LOs) to assessmentc. Designing assessment activities, including diagnostic, formative,summative assessment and peer assessment6. Combine: Combining the learning activities into the following:a. Course map, providing a holistic overview of the nature of the courseb. Activity profile, showing the amount of time learners are spending ondifferent types of activitiesc. Storyboard, creating a temporal sequence of activities mapped toresources and toolsd. Learning pathway, providing a temporal sequence of the learning designs7. Consolidate: Putting the completed design into practice in the following ways:a. Implementation in the classroom, through a VLE or using a specialisedLearning Design toolb. Evaluation of the effectiveness of the designc. Refinement based on the evaluation findings
  37. 37. METIS PROJECTD3.2: Draft of pilot workshop Page 37 of 64d. Sharing with peers through social media and specialised sites likeCloudworksFor each of the seven stages a series of conceptual design tools have beendeveloped, including the Course Features card, Course Map, Activity Profile,Storyboard, Resource Audit tool, and E-tivity template.The framework and its tools and resources have been trailed in the HE context,including four UK universities and overseas institutions such as SAIDE (South AfricanInstitute of Distance Education). In addition, we have run a series of 7Cs workshopsat a number of international conferences, with participants from a variety ofdifferent educational sectors. In METIS, we think that the 7Cs methodology offersthe potential to be applied to other educational sectors, such as adult learning andvocational training. The conceptual design tools developed from the 7Cs enableteachers at all level of education to rethink their design practice and to create moreengaging learning experience for their learners.Key referencesConole, G. (forthcoming). Innovative approaches to learning design – harnessing newtechnologies for learning. In T.D. Bilham (eds.) For the Love of Learning: innovationsfrom outstanding university teachers. Palgrave MacMillan.Conole, G. (2013). Current thinking on the 7Cs of learning design (blog post). project. 7Cs toolkit:, and SPEED blog: Design-PracticeSummaryDesign-Practice project (http:// was aimed at preparingthem in integrating ICT in their teaching and advancing their lifelong learning skills bybuilding a community of teachers for sharing, discussing, debating, and improvinginstructional activities and learning designs. During the project an online portal withcommunity of teachers and face-to-face and online teacher training modules weredeveloped.The workshops were held in Cyprus, Greece and United Kingdom and were based onthe work of the Open University UK (OU) on Learning Design, which was first
  38. 38. METIS PROJECTD3.2: Draft of pilot workshop Page 38 of 64developed in the context of the OULDI (Open University Learning Design Initiative)project. A detailed description of the workshops is available at: workshops included the following 7 activities:Activity 1 - How to ruin a course: The main goal of this activity was to pinpoint keyissues and strategies that can affect the success (or otherwise) of learning andteaching within an educator’s context.Activity 2 - Comparing four web 2.0 tools: This activity was aimed at introducingparticipants to four web 2.0 tools and to motivate them to think of various ideas onhow to use these tools in the classroom.Activity 3 - Affordances: Tools in use: The aim was to explore the ‘affordances’ of oneor more technological tools. The goal was to encourage participants to developcritical thinking and to make judgements as to which tools are the right ones to usein their teaching based on their affordances.Activity 4 - Course Map: The goal of the activity was to propose a way to representlessons which can help the design and implementation of a lesson and it can alsofacilitate the sharing of designs amongst educators.Activity 5 - Mapping a design using learning design notation: The goal was tointroduce a method for designing learning activities to the participants.Activity 6 - Breaking-down a design into the micro-level: This activity was aimed atfurther analyze the lesson representation designed in Activity 5.Activity 7 - Sharing and discussing designs: The goal was to develop an environmentof cooperation and sharing, which can be facilitated with the use of socialnetworking tools such as Cloudworks.Key referencesVrasidas, C., Conole, G., Retalis, S. (2010). Usable representations of Learning Designfor Educators & Instructional Designers. Workshop at Online EDUCA Berlin,December, 2010. (, C., Theodoridou, K., Theodoulou, F., Aravi, C., Pattis, I. (2010). DesignPractice: A Framework for Preparing Teachers to Teach with ICT. International VisualLiteracy Association Conference.
  39. 39. METIS PROJECTD3.2: Draft of pilot workshop Page 39 of 64d. Learning Design StudioSummaryThe learning design studio is a collaborative, blended, project based framework fortraining educators in effective and evidence-based use of educational technology.This approach is modelled after the tradition of studio-instruction in arts and designdisciplines (such as architecture). In this model, the main activity of a course is thestudents continued work on design challenges in a defined domain of practice.Students typically work in groups. They identify an educational challenge, research it,and devise innovative means of addressing it. The course instructor guides thestudents through the process, and classroom sessions are mostly dedicated to groupwork and public review of design artefacts.The Learning Design Studio manifests a model of teaching a Design Inquiry ofLearning. Design, in this context, is the informed creative practice of devising“courses of action aimed at changing existing situations into desired ones” [0Simon,1996, p 129]. Inquiry-based learning attempts to shape educational experiences inthe model of scientific investigation. Similarly, an inquiry approach to the training ofeducational practitioners should mimic the form of design research in education.Thus, the learning design studio mimics the structure of a design experiment (Mor &Winters, 2007), with the exception that students do not have the resources or thetime to conduct several iterations, scaling up from a conceptual prototype to anextensive deployment.
  40. 40. METIS PROJECTD3.2: Draft of pilot workshop Page 40 of 64Key referencesMor, Yishay and Mogilevsky, Orit (2012). A Learning Design Studio in MobileLearning. In: The 11th World Conference on Mobile and Contextual Learning (mLearn2012), 16-18 October, Helsinki.Mor, Y, & Mogilevsky, O. (submitted to EC TEL) The Learning Design Studio:Educational Practice as Design Inquiry of Learning referencesAnastopoulou, S.; Sharples, M.; Ainsworth, S.; Crook, C.; OMalley, C. & Wright, M.(2012), Creating Personal Meaning through Technology-Supported Science InquiryLearning across Formal and Informal Settings, International Journal of ScienceEducation 34 (2) , 251-273Cox, C.; Harrison, S. & Hoadley, C. (2008), Applying the "studio model" to learningtechnology design,Educating learning technology designers: guiding and inspiringcreators of innovative educational tools, 145Kali, Y, & Ronen-Fuhrmann, T. (2011). Teaching to design educational technologies.International Journal of Learning Technology, 6(1), 4–23.
  41. 41. METIS PROJECTD3.2: Draft of pilot workshop Page 41 of 64Mor, Yishay and Winters, Niall (2007). Design approaches in technology enhancedlearning.Interactive Learning Environments, 15(1) pp. 61–75.Mor, Yishay (2013). SNaP! Re-using, sharing and communicating designs and designknowledge using scenarios, narratives and patterns. In: Luckin,Rosemary; Puntambekar, Sadhana; Goodyear, Peter; Grabowski, BarbaraL.; Underwood, Joshua and Winters, Niall eds. Handbook of Design in EducationalTechnology. London, UK: Routledge, (In press).Ronen-Fuhrmann, T, & Kali, Y. (2010). The role of concretization in acquiring designknowledge. Proceedings of the 9th International Conference of the Learning Sciences-Volume 1 (pp. 468–475).Ronen-Fuhrmann, T., Kali, Y., & Hoadley, C. (2008). Helping Education StudentsUnderstand Learning Through Designing. Educational Technology, 48, 26–33.Simon, H. A. (1996), The Sciences of the Artificial - 3rd Edition , The MIT Press ,Cambridge, MAe. Collage / LdShake workshopsSummaryAt the UVA team (and later, also at UPF as Davinia moved from Valladolid toBarcelona), we have hosted several teacher workshops related with our researchefforts in the field of Computer-Supported Collaborative Learning and LearningDesign. These workshops were mainly directed at university teachers (although theapproach can be applied to practitioners at other levels – as long as thetechnological tools proposed make sense in the actual context). The workshopsaimed to promote participants’ awareness of collaborative learning techniques ( the form of collaborative patterns such as the Jigsaw). The main focus was onproviding practical guidance about how to design such learning experiences (bothconceptually by using and combining patterns at different levels, and usingtechnological support such as the Web Collage authoring tool). In the last editionsthere has also been an emphasis on how to implement those collaborative designsusing VLEs (such as Moodle) and external ICT tools (e.g. wikis, Google apps, etc.),again conceptually and through the use of the GLUE!-PS technological system.An example of workshop can be found in Hernández-Leo et al. (2011), in whichLdShake (the precursor to METIS’s ILDE) is shown and used by teachers. Only very
  42. 42. METIS PROJECTD3.2: Draft of pilot workshop Page 42 of 64recently we have made efforts in reflecting on the workshop approach that hasemerged through these years of research and professional developmentinterventions. Prieto et al. (accepted) describe one instance of the last round ofworkshops that have been held (focusing on design, implementation andenactment), and tries to extract some of the main guiding principles and patterns ofthe approach: Focusing on multiple aspects of the activities’ “orchestration” (beyond puredesign, also management, adaptation, pragmatic restrictions, etc.) Focusing on multiple phases of the activities’ lifecycle (beyond design,towards implementation, enactment) Use of pedagogical patterns (e.g. design patterns like Jigsaw, but alsodeployment/enactment patterns like “on-the-fly monitoring of activities”) Providing hands-on practical experience not only in the concepts of design,but also the use of technologies to design and implement the activities Modeling: the workshops themselves are designed and modelled using thetools and strategies being taught at the workshop Authentic/meaningful problems: Participants work on design/enactmentproblems that are meaningful for their own concrete coursesKey referencesDescription of one workshop instance and main characteristics of the approach:Luis P. Prieto, Yannis Dimitriadis, Juan I. Asensio-Pérez, Sara Villagrá-Sobrino, Iván M.Jorrín-Abellán (accepted). Fostering CSCL adoption: an approach to professionaldevelopment focused on orchestration. Accepted at the International Conference onComputer-Supported Collaborative Learning (CSCL 2013).Example of LdShake workshop with teachers: Davinia Hernández-Leo, LaurenRomeo, Miguel A. Carralero, Jonathan Chacón, Mar Carrió, Pau Moreno, Josep Blat(2011). LdShake: Learning design solutions sharing and co-edition, Computers &Education, 57(4), 2011, p. 2249-2260. (see sp. Table 1 and section 3.1)f. Participatory Pattern MethodologySummaryThe Participatory Methodology for Practical Design Patterns is a process by whichcommunities of practitioners can collaboratively reflect on the challenges they faceand the methods for addressing them. The outcome of the process is a set of
  43. 43. METIS PROJECTD3.2: Draft of pilot workshop Page 43 of 64Design narratives, design patterns and design scenarios situated in a particulardomain of practice. This pattern is an “envelope” for the rest of the patterns in thispaper, and the context described here is the baseline for all the others.At the heart of the methodology is the PARTICIPATORY PATTERN WORKSHOPSpattern, which describes the interrelation between three COLLABORATIVEREFLECTION WORKSHOPS: a DESIGN NARRATIVES WORKSHOP, a DESIGN PATTERNSWORKSHOP and a DESIGN SCENARIOS WORKSHOP.Apart from these, the methodology includes a “toolkit” of support patterns, whichaddress critical points in the process or specific recurring needs.The methodology is based on two fundamental assumptions: we are all experts, andwe are all designers. This methodology utilises narrative epistemology: practitionersare prompted to recount their experiences as design narratives, and discuss thesewith their peers. The construction and discussion of these narratives are scaffoldedby a set of tools and activities to extract transferable and verifiable elements ofdesign knowledge in the form of design patterns.Key referencesMor, Yishay; Warburton, Steven and Winters, Niall (2012). Participatory patternworkshops: a methodology for open learning design inquiry. Research in LearningTechnology, 20.
  44. 44. METIS PROJECTD3.2: Draft of pilot workshop Page 44 of 64Mor, Yishay (2010). Embedding design patterns in a methodology for a designscience of e-Learning.In: Kohls, Christian and Wedekind, Joachim eds. ProblemsInvestigations of E-Learning Patterns: Context Factors, Problems andSolutions. Hershey, PA, USA: IGI, pp. 107–134. - Part 1Video - Part 2g. Design ChallengeSummaryThis is one example of a series of related workshop designs conceived, delivered andevaluated by the Open University Learning Design Initiative (OULDI). This one-dayworkshop was developed to introduce course teams to using a learning designapproach at the curriculum level as opposed to lower level detailed design ofspecific activities. The example below (originally developed in collaboration with theOpen University’s Faculty of Education and Language Studies (FELS) in summer 2009)uses the OULDI learning tools and activities but tools and activities from otherprojects (especially the Viewpoints project) have been used successfully and theformat holds well. For a list of other activities and tools you could use try thelearning design toolbox.Designed to be fun and engaging, the event gives an awareness of the latest inthinking innovatively about curriculum design and is designed to be proactive ratherthan being composed around an uncontextualised set of one-to-many presentations.The time-limited challenge enables participants to make a preliminary assessment ofwhich of the ideas, tools and resources are useful and gives them some feel for whatmight be possible in a longer, term real course production process. The challenge isintended to open avenues for participants to pursue this further.The OULDI team delivered almost twenty workshops during the project pilotsincluding fourteen directly associated with the pilots. Post-workshop questionnairesand later impact surveys reveal a wide range of reaction from participants - eventhose present at the same workshop. Overall, feedback has been overwhelminglypositive (Cross, et al., 2012).With respect to application to METIS, the following issues should be considered.
  45. 45. METIS PROJECTD3.2: Draft of pilot workshop Page 45 of 64(1) The OULDI workshops were designed for Higher Education practitioners. Allof the activities and tools are transferable to other sectors such as thoserepresented by KEK and Agora, but customisation will be necessary.(2) In general, the OULDI tools and workshops focus on curriculum level designwhereas METIS is focused on activity design. This means that a selection ofthe tools and activities can be usefully applied within METIS (for example, tospecify the educational context of a particular learning activity), they must becomplemented by workshop activities and tools targeted at activity leveldesign.Key referencesA schedule for the workshop including a description of the activities and tools:The Open University Learning Design Initiative. (2012). Workshop template: DesignChallenge Retrieved 9/5/2013, from of the workshops is contained in:Cross, S., Galley, R., Brasher, A., & Weller, M. (2012). OULDI-JISC Project EvaluationReport Retrieved 3/8/2012, from OULDI Learning Design Training ModuleSummaryAs part of the EU Design-Practice project the OU Learning Design Initiative deliveredd a learning design training module to around 40 OU Associate Lecturer participantsin the UK using Cloudworks. This module has a higher theoretical content than the“Design Practice” workshops (see appendix 2.c; these “Design Practice” workshopswere modelled on the OULDI’s “LD Lite” template which uses more of a practicebased approach).The learning outcomes of the training module are to improve participants’knowledge and understanding of: The design process for learning activities Choices that practitioners make about the ways of applying technologies fora variety of learnersand to improve participants’ ability to :
  46. 46. METIS PROJECTD3.2: Draft of pilot workshop Page 46 of 64 Use and evaluate particular technologies and tools for individual andcollaborative learning Share and collaborate learning designs.The training had three components:(1) A five hour (approximately) self-paced introduction to learning designmodule which was held online. This provided the background to (2) below.(2) A five hour face-to-face ‘Using Technologies in Teaching’ workshop.(3) A five hour online self-paced activity, which draws on the work undertaken inthe workshop, and is based on the collaboration and sharing of learningdesigns.The training module provides an overview of the methodology for learning designdeveloped by the Open Universitys Institute of Educational Technology. The moduleis aimed at participants with an interest in using technology in their teaching practiceand/or are involved in the design of learning activities at all scales.With respect to application to METIS, the following issues should be considered.(1) The OULDI workshops were designed for Higher Education practitioners ( requires the participant to read academic papers).(2) The duration of the complete module is 15 hours. However, activities fromwithin the face-to-face and second online components could be customisedto suit METIS participants.(3) In general, the OULDI tools and workshops focus on curriculum level designwhereas METIS is focused on activity design. However, activities from withinthe face-to-face and second online components of this module are targetedat activity design and sharing and so could be customised to suit METISparticipants.Key referencesA schedule for the workshop including a description of the activities and tools:
  47. 47. METIS PROJECTD3.2: Draft of pilot workshop Page 47 of 64
  48. 48. METIS PROJECTD3.2: Draft of pilot workshop Page 48 of 64Appendix 3. Methodology design narratives and design patternsa. “Design Narratives” task presented to partnersThe following instructions where provided to project partners asking them toprovide design narratives of successful workshop practices:METIS Methodology Design Narratives: telling a ‘good’ storyTask: Each METIS partner authors 2-3 Methodology Design Narratives(see, based on an activity, from your own methodology (the methodologydescriptions you authored are located here[1]).The idea behind the methodology design narrative is simple: narrate a ‘good’ storyabout a workshop activity (e.g. icebreaker, task, wrap-up, etc.) that was successful.This can be from a workshop you either led or attended.The narrative should describe what participants were asked to do “X” (e.g.participate in an activity and/or complete a task, etc.) and what was the intendedoutcome “Y”.In writing the narrative please describe the “how” in as much detail as possible (e.g.steps A, B, which led them to the intended outcome Y).You can do this by providing a “thick description” of the activity and sequence ofevents participants (protagonists) were given, the steps (challenge), their choicesand the results (positive ones). If you have a sample of what materials were used inthe activity or photos this is also useful.Rationale: We require these Methodology Design Narratives for WP3. Once we havethem, we will guide you in extracting patterns from them. Our goal is to extractapplicable patterns from existing Methodology Design Narratives, with a view toapplying some (or all) of the applicable patterns to the METIS workshop design.LD Grid: Please have a look at The Learning Design Grid’s[2] "Healthy Eating[3]"as a Design Narrative example. This example is far more extensive than whatwere asking for here, but if anyone wants a broader view it may be useful.Process: Please download the file below metis-methodology design narratives-template_template.docx (Sample Methodology Design Narrative Template) andmetis-methodology design narratives_Sample.docx (Sample Methodology DesignNarrative). Then using the template (and sample) author 1 to 3 Methodology
  49. 49. METIS PROJECTD3.2: Draft of pilot workshop Page 49 of 64Design Narratives and upload your narratives (each one in a separate file) on thispage by May 15, 2013.To assist you in this task and for consistency across the project, we are providing thefollowing template and example in the word document below. The example is ofone activity, out of eight, from a workshop that lasted 2 hours.[1] See[2][3] Design narrative templateSituationA Tagline for your project or workDescribe the user group and the work contextDescribe your technological setupTaskDescribe what you are trying to achieveActions (How did you try to address the issue?)Activity .Activity goal(s) or intendedoutcome (“X”)Activity title & description Title:Activity materials
  50. 50. METIS PROJECTD3.2: Draft of pilot workshop Page 50 of 64The “how” (steps A, B, C, thatlead to Y or the activityoutcomes)...Activity outcomes (“Y”)(positive)The activity was successful because it achieved theintended outcomes as well as:Results (What were the results of the actions you took?)Activity outcomes(“Y”) (positive)The activity was successful because it achieved the intended outcomesas well as:ReflectionObservation and what you have learned.
  51. 51. METIS PROJECTD3.2: Draft of pilot workshop Page 51 of 64Appendix 4. Sample of Methodology Design Narrativesa. Meta-Pyramid design1. SituationThe GSIC-EMIC research group at the UVA has conducted several workshops in the last twoyears, for professional development of academic teaching staff of the university. The mainaim of these workshops was to promote the adoption of computer-supported collaborativelearning in university teachers’ everyday practice, including both the conceptual side ofdesigning and running collaborative activities, and the technological one of using commonlyavailable ICTs (Moodle, web apps), but also specific ones (e.g. the Web Collage authoringtool).This narrative is extracted from a workshop held in February 2012, which made moreemphasis on the conceptual side of designing collaborative activities, and deciding about ICTtools to use to support such learning. The workshop included 2 face-to-face sessions and afew online activities, for a total of about 12 hours of workshop. There were 25 universityteachers from a variety of backgrounds and levels of teaching experience in the workshop.The technological setup was based around a Moodle course with embedded Google Docsdocuments for collaborative work; also, extensive use of pen and paper was also part of theworkshop philosophy.2. TaskThe main aim of the workshop was to provide teachers with a few best practices forcollaborative learning and its implementation using ICTs, at different levels, including high-level patterns such as the Jigsaw9, and lower level design and implementation “tricks” (whichwe called “routines” or “atomic patterns”10) extracted from successful collaborative learningpractice of other teachers. The ultimate goal of the workshop was that, after the workshop,each teacher had created a collaborative scenario for his/her own practice, as a necessarystep for applying such techniques in practice. It is also important to note that the workshopactivities themselves were designed and structured around these best practices, so thatteachers could experience what collaborative learning “felt like” as an student (which noteveryone had done in their own previous education).3. Actions (How did you try to address the issue?)9A very common collaborative strategy for addressing complex problems by dividing them in parts.See for a more complete description.10See Prieto et al.’s “Recurrent routines: Analyzing and supporting orchestration in technology-enhanced primary classrooms” at
  52. 52. METIS PROJECTD3.2: Draft of pilot workshop Page 52 of 64Activity Designing a collaborative learning scenario.Activitygoal(s) orintendedoutcome(“X”) Engage teachers in their first collaborative learning design Encourage collaboration among participants Make a number of best practices available to teachers (patterns,routines) Show the value of iteration in learning designActivitytitle &descriptionTitle: Design a Pyramid activity using a PyramidThis was a lengthy activity that encompassed three main phases, of 30-45minutes each, where they had to address a fictitious but realistic scenario (theneed for a zero-course on bibliographic search and collaborative report writingfor undergraduates) that required them to design (and later, enact) a set ofcollaborative learning activities. These three phases introduced progressivelycollaborative best practices of different kinds (overall strategies or patterns,such as the Pyramid11; more concrete design “routines”; and even moreconcrete deployment/implementation “routines”), which were explained brieflyby facilitators and provided to participants in the form of cards (see below,translated from the original Spanish).Also, this activity led participants to review their designs and those of theirpeers, and iterate over them, going from the initial, rather abstract idea, upuntil they had a clearer idea of how they would use ICTs to implement it.Later in the workshop, this design was taken further, and parts of it wereenacted through a role-playing, to discuss yet another set of enactment“routines” that might be useful when running such collaborative activities.Activitymaterials Large white paper surfaces Pens, highlighters, pencils A digital camera Scenario description (see an example in Spanish here).11See, Hernández-Leo, D.; Asensio-Pérez, J.I.;Dimitriadis, Y.; & Villasclaras, E.D. Generating CSCL Scripts: From a Conceptual Model of PAtternLanguages to the Design of Real Scripts. In: Goodyear P.; Retalis, S. (eds.). Technology-EnhancedLearning, Design patterns and pattern languages, Sense Publishers, Series Technology-EnhancedLearning; 2010. p. 49-64. Appendix available at:
  53. 53. METIS PROJECTD3.2: Draft of pilot workshop Page 53 of 64 Pattern, design routine and deployment routine cards (examples inSpanish are available here, here, and here, respectively), and an Englishtranslation of the titles of such cards can be found here (see appendixB)The“how”(steps A,B, C, thatlead to Yor theactivityoutcomes)This activity took place after a short introduction by the facilitators to theworkshop methodology and goals, and about collaborative learning in general.A. Explanation of the scenario: Although participant teachers had had toread the scenario previously, the facilitators remind participants aboutthe scenario (zero course in bibliographic search and joint reportwriting) and the scope of the design (10 hours of blended studentwork).B. First design iteration (with patterns): facilitators present very briefly theidea of collaborative design patterns, and describe two of them(Pyramid and Jigsaw). Then, participants are divided in heterogeneousgroups of three people (they had already been seated in such groups atthe beginning of the workshop to minimize hassle) and are providedwith a paper description of several such design patterns. Then, for 30min, they are asked to design the learning activities, applying one of thepatterns (the Pyramid) to the provided scenario. The design is to bedone in paper, in whatever format they desire (textual, graphical,diagrammatic, tabular...). Below we can see the result from one of thegroups:C. Second design iteration (with design routines): Facilitators then brieflypresent other set of best practices, of a lower granularity (designroutines), as elements of blended CSCL practice that have beenobserved in successful practice by other practitioners, that can be usedto further flesh out the overall structure denoted by the design pattern.Facilitators give participants a set of these routines (each on a yellowcard) and, in the same groups as in phase B, participants are asked tore-iterate their design, using such cards to make the activities andstrategies to use more concrete. Again, 30 min are allocated to this
  54. 54. METIS PROJECTD3.2: Draft of pilot workshop Page 54 of 64task. Although it was not explicitly said, many teachers at this pointglued the yellow cards over different parts of the previous design, tomark when a certain design routine would be used. At the end of thisphase, a short break is scheduled, which facilitators use to make photosof the designs so far, uploading them to the workshop’s Moodle.D. Cross-review of designs (with implementation routines): Now,facilitators provide yet another set of routines (“deployment routines”,represented by green cards), which are also lower-level, and which dealwith implementation and deployment details of a CSCL activity usingthe workshop’s concerned ICTs (Moodle, Google Docs, etc.). Then, for15 min, participants are asked to look at the design of another group(and vice-versa), and provide three aspects they liked, three problemsthey saw in the design, and suggestions of deployment routines thatmight be useful in the context of that design. These suggestions are tobe written in Google Docs that have been set up beforehand byfacilitators (one per group), and are accessible through the workshop’sMoodle.E. Third design iteration: Participants now are asked to work in 6-persongroups by joining the two groups that have cross-reviewed each other(thus, following the Pyramid pattern that is being used in the designs).For another 30 min, participants are asked to produce a completedesign using all the elements (by re-iterating either of the groups’original designs, or doing an entirely new one that takes advantage ofthe strategies of both). Below, one of the final products is reproduced: