Presentation at ICTMT10 in Portsmouth, July 2011

1. Benchmarking and Mastery: Integrating Assessment into Teaching and Learning roy.williams@port.ac.uk Dept. Mathematics Univ. of Portsmouth Paper presented at ICTMT, Portsmouth, July 2011

3. Can we relegate assessment to the certification of students’ own benchmarking?

4. What is the best resourcing model: how do we balance and manage:

5. Investment, Outsourcing, Crowd-sourcing or: Fixed, Contracted or Shared costs?

6. What’s the Question? A Digitally Enhanced Learning Ecology ? (a new DELE?) Not ... What does it mean to take technology into the classroom? But rather ... What does it mean to situate teaching and learning within a digital learning ecology?

8. Creating algorithms

9. New proofs

10. Alternative proofs (e.g. by Reductio ad absurdum)

11. More elegant proofs

12. More elegant solutions

13. New/more elegant descriptions/ applications.

14. Informing new strategies (biology, finance, logistics)

15. Facilitating new laws

17. Shifting from external requirements for compliance >> self-organised mastery (strategic behaviour?). 6. Log books: for writingmathematics.

18. Lectures Student Mathematics Tutorials Textbook “Feedback” Assessment Data Figure 1: Basic Graduate

20. ppt Practice Tests … (Same/ New Values) Benchmarking: (New Values) Tips Solutions Links Answers Competence Figure 2: Wiley Certifying Benchmarking ?

21. Workshops Student Lectures Links R: resources A: apps N: networks W: wikis M/B: micro-/blogging PRS W A R Mathematics Passwords N M/B e/textbook Authoring, Customising Tutorials Feedback Log books Benchmarking: (New Values) Certifying Benchmarking Practice Tests … (Same/ New Values) Assessment DIMLE 1 DIMLE 2 Summative Assessment Formative Assessment DIMLE 3 DIMLE 4 Reports Data Mathematician Solutions Next step Tips Links Debugging Answers Figure 3: Design Palette

22. Mathematics PRS DIMLE 3 Student Lectures DIMLE 1 Workshops Links R: resources A: apps N: networks W: wikis M/B: micro-/blogging A W N R A R e/textbook M/B Tutorials Reports W Log books M/B N Authoring, Customising DIMLE 4 Practice Tests … (Same/ New Values) Data Benchmarking: (New Values) Next step Tips Passwords Solutions DIMLE 2 Mathematician Links Certifying Benchmarking Debugging Answers Figure 4: Design # 1 Formative Assessment Assessment Summative Assessment Feedback

23. Narrative 0. This is a basic outline of the presentation, which will be worked up into something more substantial, a paper, after the conference. 1. Introduction This presentation raises some questions about assessment, about the use of technology in teaching and learning, and about how all of these may be integrated, within an approach that shifts away from traditional ‘assessment’ towards ‘benchmarking and mastery’. It also provides a rather basic, but open and interactive strategy and design template for having conversations about past, present, and future approaches to resolve some of these questions. 2. Some things to think about ... In exploring these issues, we need to be mindful of the following: 2. 1 What’s the difference between ‘doing mathematics’ and ‘being a mathematician’? Richard Noss, in his keynote address, said that rather than thinking about how technology can help us to do what we already do (although I am sure he has no objection to that), we should focus on how technology can help us to explore previously unlearnable ideas. Several other papers picked up on the way that a dynamic and interactive mathematics’ is that we want students to learn. Noss spoke mathematical learning environments – DIMLE, or DILE, (Novotna & Karadag) can be used enable students to learn by exploring, trying out, and refining in these digital environments. 2.2 But we first need to ask ourselves what But first we need to ask ourselves what this ‘mathematics’ is that we want students to learn. Noss spoke simply of: transformation and generalisation. We might extrapolate this, for instance by linking it to theory and/ or application.

25. New, alternative, or more elegant proofs / solutions.

26. New / more elegant descriptions / applications.

27. Informing new strategies (biology, physics, finance, logistics).

28. Providing powerful ideas for articulating theory in other disciplines.

29. Creating new ideas and tools: Tau/Pi, i, infinity, fractals, statistical tendency, calculus.2.3 We then we need to look at the data, and ask ourselves: What’s happening when students come back for an extra 5-7 hours of exams, after they have already finishing the scheduled 7 hours of exams? This is consistent over two semesters now, so it’s not the novelty of the experience, and it was the result of the changes that we put in place when we started using the Wileyplus platform (see below). 2.4 This leads to an important challenge: can we relegate assessment to the certification of the student’s own benchmarking? To some extent the answer is yes, this emerges from the previous point, although to some extent it might be a rather idealistic goal that still needs to be further explored (in research and in practice). 2.5 Lastly, what are the implications for resourcing good teaching and learning: how do we balance and manage: investment, outsourcing and crowd/peer-sourcing, or: fixed, contracted-out and shared costs? This arises from many years of e-assessment development and practice, and our recent research on using an ASP (Application Service Provider), i.e. Wileyplus (McCabe, Williams & Pevy, 2011). The issue is the design, development and management of an integrated, sustainable, and flexible teaching and learning environment, and whether we go down the path of internal investment (fixed in-house costs), outsourcing (via commercial ASP’s), or what might be called “peer-sourcing” (a multi-institutional variant of crowd-sourcing or open-sourcing, or what Mike McCabe calls open re-sourcing). All of this leads us to a different question from the one posed early on in the conference, namely: “What does it mean to take technology into the classroom?” We should rather turn the question around: “What does it mean to take teaching and learning into a dynamic, digitally enhanced learning environment?”

30. 3. Eighteen month large classes “Live pilot” We are coming to the end of what could be called an 18- month large classes “live pilot”, using Wileyplus for the 150 or so student taking the 1st year courses in calculus and linear algebra. The key aspects of this were the following: 3. 1 Two-week start-up We literally got the course up and running, live, with 150+ students, in two weeks, with help from the Wileyplus team in Chichester. To be fair, it was based largely on a text book that was already used for this course, published by Wiley, and we had other e-assessment resources available too (both additional and back-up). Nevertheless, it points to two things: i) given a suitable, well known core text, it is possible to mount an integrated facility for a course in two weeks; ii) this therefore also applies, albeit with slightly more lead time, to possibly switching to an alternative ASP. The digitally based educational resource market is more flexible and more competitive than its print (only) predecessors. 3.2 ASP (Application Service Provider) An external provider, Wileyplus, was used which provided a full e-text book (with paper text book available for purchase), algorithmic practice tests, the facility to compile exams, ppt materials, PRS questions, a symbolic palette for writing answers ‘in mathematics’, basic graphic interaction in providing graphs as answers, and hints/ links/ answers/ solutions that can be set to link to particular attempts (1st attempt, 2nd, etc). 3.3 Assessment // Benchmarking These courses use a hybrid mode of formative/ summative assessment: seven tests are set over a 12 week semester, designed to be completed in 7 one hour sessions, with an additional 7 hours of testing optionally available. Students can repeat individual questions (3 times, with new values), rather than the previously used model, in which they had to repeat a whole assessment (up to 3 times). Students can also practice, in similar tests, using either the same values, or new values each time, 24/7, accessible from anywhere.

31. 3.4 Logbooks, lab sessions, tutorials, Maths Café. It is possible to capture ‘working’ in Wileyplus, but it was decided to ask students to write up solutions for selected practice questions in a hard-cover workbook which each of them was given. This logbook served as a resource for their own learning, and for tutorials, for lab sessions, and discussions with peers or lecturers in the open space ‘Maths Café’. Students were required to hand their logbooks in at the end of the semester, and completion of a minimum amount of work contributed a small amount to their grade for the semester. 3.5 Data Date from initial research (McCabe, Williams & Pevy, 2011) as well as from more recent semesters shows that students spend substantial amounts of time doing practice tests online. Many of them put considerable effort into writing up solutions in their logbooks, and roughly a third of the students spend up to 14 hours in the exam sessions, i.e. 100% over and above what is required (i.e. a total of 14 hours rather than 7 over the semester and exam period). 4. Discussion From observation of students and a rough survey of the data in Wiley, as well as from a focus group workshop, it is clear that many students are working well beyond the minimum requirements. This is true for nearly all students using the ‘repeat individual questions’ options in Wileyplus, but is hardly true at all for other test material provided in Maple TA, in which the only option they had was to repeat the whole assignment, regardless of how much they achieved in their previous attempts. This is unusual behaviour for students, particularly as it applies across the board – to struggling, average and good students. E.g. student A did ...., student B did ...., student C did ... In broad terms, what seems to be happening is that students are: 1. First of all, complying with the course’s threshold requirements (25% per test, 40% aggregate for the course). 2. Continuing to benchmark their own ability, based on internal motivation to master more of the material, based on what they would like to try to achieve. 3. Accumulating competencies, not collecting grades.

32. It is clear that the specific affordances of providing opportunities to repeat questions (rather than whole tests) are crucial in motivating students to go beyond the minimal compliance, and start to benchmark themselves against their own targets. This takes us back to the question of what it is that they are benchmarking and mastering: doing mathematics, or becoming mathematicians? To put it in Noss’s terms (above), are they simply mastering specific transformations, or are they also grasping the underlying principles of generalisation, as well as its use, in theoretical and applied problems? In practical terms, it might be useful to consider the following: 1. What are the limits of doing this in large classes? Is benchmarking and mastery applicable to smaller classes, and if so, would it be done differently? What are the limits of compliance and mastering transformations? How should mathematical thinking and working, and specifically the processes of generalisation be accommodated, and how can it usefully be benchmarked and mastered? 5. What are the limits of mastery, particularly in large classes? How far is it desirable and possible to ‘relegate assessment to the certification of the student’s own benchmarking and mastery’? 5. Design and Strategy Once we have provisional answers to these issues, we can explore how we might situate teaching and learning in a digital learning ecology. One way to do this is to visually explore the way teaching, learning, assessment, benchmarking, mastery, and technology can fit together in a (digital) learning ecology. To do this we need the following: 5.1 Basic Model Figure 1 outlines the basic elements that are provided in Higher Education to enable a student to pass through ‘mathematics’ to graduate.

33. 5.2 Wileyplus Figure 2 outlines the additional, and digitally integrated, facilities offered by Wileyplus. Wileyplus has some drawbacks, in terms of formal ‘summative’ assessment (though it is possible to ‘work-around’ them). But crucially for our purposes here, Wileyplus is designed primarily as a learning/ learning and teaching platform, rather than as an assessment platform. This is a useful place to start. 5.3 Design Template To take the Basic Model forward, we first of all need to decide what we need in our design ‘palette’ – what features need to play a part in creating a ‘digital learning ecology’ that will be suitable for taking our student through ‘mathematics’ to becoming a mathematician? Figure 3 includes some elements of such a palette: Workshops Several speakers at the conference, from the first to the last keynote, showed how digital tools and platforms, (Papert’s) ‘microworlds’, digital interactive (mathematical) learning environments (DILE’s or DIMLE’s), etc can not merely literally ‘facilitate’ teaching and learning (make it easier and more efficient), but can make the unthinkable, or the previously unteachable, possible (and fun). All of these can, for the purposes of this palette, be included in the title ‘workshops’ – which could also take non-digital forms, such as dance and movement, as well as writing about mathematics, which emerged in one or two papers and discussions. Benchmarking Crucial to DILE’s is, obviously, interaction. Several speakers talked of interaction in terms of iterative cycles of trying out, exploring, and refining. There are several elements that we might like to add to our palette to make this possible and to start to integrate more interaction into benchmarking. Many of these may help us to transform ‘assessment’ (or at least shift it) towards benchmarking and mastery. These include: practice (tests and microworlds); testing or benchmarking transformations (and generalisation?); enriched interaction (links to text, tips, solutions, answers, next steps, links to debugged examples, etc).

34. Flexibility and Context Good teaching and learning is responsive to learners and to context (as far as is possible). It also needs to be relevant to the teachers and learners and their specific contexts, which are of course changeable. All this means quite simply that although it is useful to have large amounts of standardised material available, it is essential that flexibility allows for other material (both new and imported from elsewhere) to be integrated into the learning environment. Data and Reports Most digital platforms and tools provide data, often lots of it. What teachers (and learners) need, however, are specific, targeted reports, to answer specific questions that arise in their practice. This is often lacking, and where it is provided, it is driven not by learning, but rather by administration. Teachers and learners could use far more sophisticated reports (and ‘feedback’), for which most of the data is already available (just not in useful outputs). Certifying Benchmarking Some outputs, like ‘summative assessment’, which might be transformed into ‘certifying benchmarking’ need additional layers of security. This is best done by password-controlled access, and is essential to eliminating the unnecessary administrative (and technical) overload that is otherwise required. (Note: this should not, and does not have to shift the ‘centre of gravity’ of the learning environment back from teaching and learning into formal assessment or, even worse, into administration). Links Teachers and learners already make extensive use of materials, networks and contacts available through the Internet. There are several tools and platforms that make it possible to link and share all of these. In principle it would be more useful if they could be integrated to a greater extent with teaching and learning. The current trend is away from institutional VLE’s (virtual learning environments, many of which are no more than student management systems), towards PLE’s (personal learning environments). This is an advance, but there is still much to be done to link up PLE’s across communities of inquiry (such as courses). Our palette needs to have many type of links, so that we can decide where they need to connect within our digital learning environment, and how this is to be done.

35. Design Draft #1 Figure 4 is an example of how the Design palette in Figure 3 could be used, to design an integrated, flexible, digital learning environment. A number of elements of the design have been ‘binned’ at the bottom left: assessment, formative and summative assessment, and feedback. These are all displaced by: practice tests, benchmarking, tips, next step, solutions, links, answers. ‘Debugging’ has been brought in too – it can capture, tag, and display common errors, and point up other errors for discussion. ‘Workshops’ now overlaps with lectures, tutorials and even the e-textbook, as it can shift the mode of all of these activities to something more interactive, particularly if other tools (DIMLE’s and Apps) are linked in here too. ‘Data’ is relegated into the background, to be replaced by ‘Reports’ (on demand) for lecturers and students. ‘Passwords’ is an essential addition, to link Benchmarking to Certifying Benchmarking. Links are added too, to wikis, networks, resources, and to other DIMLE’s and Apps. The domain of ‘mathematics’ is expanded to include all of the core elements provided by the university, as well as the links to external resources and networks. Resourcing Resourcing is a key part of design and strategy. In broad terms, three there are three options: outsourcing, in-house investment, and sharing. (See McCabe,. Williams and Pevy 2011). In principle, all three options, and combinations of them, can deliver what is required. However, to achieve integration across all of the above areas in our design palette, considerable organisational and financial resources are required. This can be done in the shared (or ‘peer-sourced’) option, see for instance Linux, but the ‘text book’ (electronic or otherwise) still seems to require a ‘publisher’. (There are interesting questions about whether the relationship between core text and core activities could be inverted, but that’s speculative, and it’s too large a question to go into here). Commercial ASP’s already provide this option, although without much of the desired flexibility. In-house options can and do provide quite a lot, but the management and systems requirements for a comprehensive facility exceed that of individual organisations, many of which are now required to compete against each other, rather than co-operate.

36. 6. Conclusion This paper has raised several questions about the way in which assessment can be integrated into teaching and learning, and in fact become just a confirmation of learning that has already happened, which is more student centred, student driven, and which markedly increases student effort and motivation. One of the interesting issues that arose in the conference was the impact of e-assessment. It was asked whether e-assessment, however much the students enjoyed doing it, actually had an impact on their understanding and learning in the next years of their courses. It probably goes without saying that multiple choice e-assessment is just (and only) that – it assesses what the student knows at a particular point in time, and not much more. The impact of quite a different form of e-assessment (where the answers are written ‘in mathematics’), and quite a different mode of assessment (personal benchmarking and mastery) might be different, and could be an interesting area to research. The ‘live pilot’ experience of using Wileyplus points in the direction of a different mode of assessment, teaching and learning, based on student driven mastery (however modest that may be). But if this can be achieved, the next question would be whether mastery could be opened up, and (slightly) expanded, so that motivated students could take up further challenges, beyond the confines of a particular course. The design and strategy palette is a very basic, but quite flexible, template for trying out, exploring, and refining some of the ways in which we might ‘situate teaching and learning in a digital learning ecology’. The question of resourcing remains, as none of the three options (above) currently yields all that is required. What is evident, though, is that providing the resources and the affordances for students to repeat each question up to three times (each time with different values) does lead to quite different behaviour across the board, and does open up the possibility of ‘relegating assessment to the confirmation of the student’s own benchmarking and mastery’. McCabe, M. Williams, R.T. & Pevy, L.(2011) ASPs: Snakes or Ladders for Mathematics? Forthcoming, MSOR proceedings.