Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Lemons, Catastrophic Climate Change, and e-Portfolios: Implementing sustainable course-wide assessments


Published on

In a recent paper co-authored with Sarah Timbs, Glenn Anderson introduced a “simple, non-intrusive and efficient way of measuring English language proficiency that is independent of the commitment of lecturers and tutors, or even the interest level of students” that would “provide a foundation for measuring the impact of ELP strategies as well as encouraging students to take an interest in their ELP from entry to exit”. In this video, Glenn takes a closer look at the incentives of course-wide assessments such as e-Portfolios for student engagement, graduate recruiters, and critically the engagement of academic staff in its implementation.

Published in: Education
  • Login to see the comments

  • Be the first to like this

Lemons, Catastrophic Climate Change, and e-Portfolios: Implementing sustainable course-wide assessments

  1. 1. Lemons, Catastrophic Climate Change, and e-Portfolios: Implementing sustainable course-wide assessments Dr Glenn Anderson and Sarah Timbs CRICOS Provider Code 00301J
  2. 2. “How(do you)know(your) graduateshave attained thresholdlevelsoforaland writtencommunicationskills upongraduation?”) Arkoudis, 2014). CRICOS Provider Code 00301J “Although someacademicstaffclaimed tohavebeenteachingcertainacademic literacies,itwasdifficulttoverifysuch claimsorestablishhoweffectivelythey weredoingsoandwhethertheywere scaffoldingtheteachingandlearning processinthewayintendedwithinthe proposedmodel.”(Murray &Nallaya, 2014)
  3. 3. Overview • Information, Thresholds & Cooperation • Two informational uncertainties undermine cooperation • Uncertainty: teaching quality • Uncertainty: graduate quality • Collective action dilemma: 2 cases • Pilot Project: Assessing and Developing English Language Proficiency • Further research CRICOS Provider Code 00301J
  4. 4. Information, Thresholds & Cooperation
  5. 5. Information, Thresholds & Cooperation
  6. 6. Information, Thresholds & Cooperation What are the incentives to free-ride? • Costs of reducing CO2 emissions are born by each country, but the benefits are shared • If every country complies with the agreement, the reduction in costs by any one country defecting would outweigh the reduction in individual benefit • There is an incentive to free-ride  “Prisoners’ Dilemma” But wouldn’t breaching 350 ppm result in catastrophic climatic change? • Any one country breaking the agreement would result in catastrophic climate change • Now costs would far outweigh the benefits for each country • Coordination is a realistic possibility But the actual threshold is unknown. • 350 ppm is an average. Is the threshold for irreversible climate change 250 or 450? • The uncertainty over the threshold reduces the expected impact and makes free-riding a possibility once more.  “Prisoners’ Dilemma” Barrett, S., & Dannenberg, A. (2014). On the sensitivity of collective action to uncertainty about climate tipping points. St. Louis: Federal Reserve Bank of St Louis.
  7. 7. Information, Thresholds & Cooperation • Lemons problem (Akerloff, 1970) • Two types of 2nd hand cars • 50% Good quality cars worth $80,000 for the owners • 50% are poor quality cars (“lemons”) worth $20,000 for the owners • If consumers pay the average price of $50,000 only poor quality cars are offered • Students look to signal high quality (Spence, 1973) • High achievers pay a premium for a university with a good reputation • Low achievers not willing to pay the premium (don’t expect to earn as much) • Two problems • Access does not depend on graduate’s expected earnings alone (the rationale for HECS) • Link between reputation and quality is “loose” (Podolny, 1993)
  8. 8. Two informational uncertainties undermine collaboration Uncertainty about teaching quality • “Although some academic staff claimed to have been teaching certain academic literacies, it was difficult to verify such claims..” (Murray & Nallaya, 2014) Uncertainty about graduate quality • “How (do you) know (your) graduates have attained threshold levels?”(Arkoudis, 2014)
  9. 9. Uncertainty: Teaching Quality “Although some academic staff claimed to have been teaching certain academic literacies, it was difficult to verify such claims..” (Murray & Nallaya, 2014) • I get on well with the language specialists and we have worked together to redesign the syllabus integrating literacy skills into learning tasks and assessment of my first year unit • Everyone in the program are very supportive • But I’ve become the ‘go-to person’ when there are any issues with the student literacy • No-one else is willing to commit and the current arrangement is unsustainable
  10. 10. Uncertainty: Teaching Quality Why does the agreement break down? Assume two types of students • Intrinsically motivated attend literacy workshops • Most other students are weakly motivated • Lack intrinsic interest • Working Faculty members come together to design a program that will improve generic skills • Every hour that an instructor spends on literacy, returns an extra $50 for the program • 10 teachers agree to spend 1 hour each • Cost of an hour for each teacher is $10 (opportunity cost: research/publications)
  11. 11. Uncertainty : Teaching Quality If everyone cooperates • Total benefit= $500 & everyone receives $50 each The last instructor has an incentive to break the agreement • If the first 9 instructors contribute, payoff per instructor is at least 450/10 =$45. • If the last instructor contributes: 500/10 =$50 each • If not, the last instructor receives $45 + $10 = $55, while the others receive $45 • Total benefit= $450 (i.e., lower) So the 2nd last also has an incentive to break the agreement • The 2nd last instructor expects last to break the agreement • Keep agreement: $45. Defecting yields $40 +$10 = $50. Incentive to defect • Total benefit $400 (i.e., lower again) Iterate and no-one has an incentive to keep to the agreement • Total benefit = $100 & every receives only $10 each Cannot monitor & enforce agreement  “Prisoners’ Dilemma”
  12. 12. Uncertainty: Teaching Quality Bonus if $500 threshold is achieved • Suppose there is a bonus of $15 for each teacher if the threshold yielding $500 is achieved (or a penalty of $15 if not achieved) • The last teacher. If everyone contributes, the last teacher receives $50 +15 =65 (bonus for meeting threshold). If the last teacher does not contribute, when everyone else contributes, $45 +10 =55. • If everyone else contributes, it pays for the last teacher to contribute. If no-one else contributes it pays the teacher not to contribute • So as long as the bonus is large enough a prisoners dilemma game is converted into a coordination game in which agreements are more easily maintained (Barrett, 2013)
  13. 13. Uncertainty: Teaching Quality Summary so far • Reputation is based on graduate opportunities. We assumed reputation is a reliable indicator of quality – graduate competencies. • However, the contribution of each instructor is not directly observed. This makes monitoring and enforcement of agreement problematic  prisoners’ dilemma • Nevertheless, linking a bonus/penalty to achievement/non- achievement of a threshold standard means there is no incentive for a teacher to defect if all other instructors are keeping to the agreement • Bonus/penalty: when an institution offers students superior/inferior opportunities via signalling with their reputation • Competition between HEIs can yield the optimal outcome
  14. 14. Uncertainty: Teaching Quality Institution B Improve quality Don’t improve quality Institution A Improve quality 500, 500 650, 100 Don’t improve quality 100 ,650 100, 100
  15. 15. Uncertainty: Graduate Quality But concerns remain that competition may not drive quality improvements • “To my knowledge very few universities confront the English language problem by requiring a formal English test and then mandating supplementary English courses for students who are deficient. There is widespread recognition of the English problem. But if a particular university takes unilateral action to require remedial courses this would add to the fees the overseas students must pay and put the institution at a competitive disadvantage relative to the competition.” (Birrell, 2007, p.62-3, emphasis added) Why wouldn’t offering a competency with strong employer demand be seen as an opportunity to gain competitive advantage?
  16. 16. Uncertainty: Graduate Quality Is the link between graduate competencies (hence, teacher effort) and graduate opportunities reliable? • Employers look to institutional prestige & grades to improve the odds of selecting the ‘right candidate’ • institutional reputation, rankings, degree 2:1 or over, entry scores • Validity & reliability of selection process questionable: • 75 per cent of vacancies registered with the Graduate Recruitment Bureau (UK) required 2:1 or above. • After a 18 month study, Earnst & Young found the 2:1 criteria failed to predict performance. • So how will my collaboration with the language specialist impact on the students graduate opportunities, when I know many employers will only select from the “top 10” institutions? • I teach in a “top 10” university. Will my students be any worse off if I don’t collaborate? • Any bonus or penalty is far from assured, at least within the tenure of many academics
  17. 17. Uncertainty: Graduate Quality “How (do you) know (your) graduates have attained threshold levels?”(Arkoudis, 2014) •Climate Change: uncertainty about the threshold •Higher Education: uncertainty about the link between graduate competencies & graduate opportunities
  18. 18. Uncertainty: Graduate Quality Link between graduate capabilities & opportunities is tenuous (Martin, 2011) “The importance of academic reputations means administrators and governing boards are narrowly focused on public relations and they scrupulously avoid controversies. As conditions change, they do not attempt to reallocate resources; instead, they seek to solve all campus problems by raising more money. The public relations mindset adversely selects for lower academic quality. Administrators divert resources from real quality because those programs are slow to develop, the benefits are difficult to measure, and the results are most likely to be realised during the next president’s tenure. Instead, administrators prefer projects with immediate public relations value and obvious tangible results, such as facilities competition and media events.” Martin, R. E. (2011) The College Cost Disease. Edward Elgar, pviii
  19. 19. Collective Action Dilemma Individualism, unknown teacher quality & unknown graduate quality …. “The prevailing norm of academic individualism may therefore impede the systematic monitoring or measuring of student achievement that is crucial to the improvement of academic quality. Without public information about the value-added by an academic program there are insufficient incentives for individual faculty members to enter into the coordinated activity necessary to produce academic programs with the academic coherence and structure research suggests is associated with student learning. The improvement of academic quality thereby represents a classic dilemma of collective action” (Dill & Soo, 2004, pp. 16-17) … nexus between graduate competencies & opportunities result in a collective action dilemma
  20. 20. Collective Action Dilemma CASE 1: University of South Australia: An attempt to embedded academic literacies from above… Another highly significant challenge that impeded the embedding process, and which relates to the kind of attitudinal problems discussed above, was that of compliance among those tasked with implementing it on the ground. Although it was mandated by the Deputy Vice Chancellor: Teaching and Learning that the embedding of academic literacies should be implemented in targeted programmes, there were no clear directives from the university’s Senior Management Group or the Heads of Schools about the process and no punitive measures in place should those responsible for implementation fail to bring about – or attempt to bring about the required change. Given the scale and ambition of this institution-wide innovation, clear directives from senior management regarding the significance of undertaking the embedding process, deadlines for compliance and the consequences of failure to implement would have provided the project with greater credibility and momentum. (Murray & Nallaya, 2014, p. 11) ….undermined by lack of cooperation
  21. 21. Collective Action Dilemma Failure to cooperate • In the event, it was left almost exclusively to the Language and Learning team within the identified schools to oversee the embedding process and the impression created was that it was the sole responsibility of the English language tutors to facilitate that process. The consequence of this was that Course Coordinators perceived the initiative as unimportant and unnecessary. This in turn had a somewhat demoralising effect on language tutors who at times felt as though they were swimming against the tide and that attempts to bring about change were futile.
  22. 22. CASE 2: Coastal Carolina University: Using badges to assure learning in an English composition course • Challenge: “numerous sections of first-year composition courses are taught by a diverse number of tenured, tenure-track, lecturer and adjunct faculty; during the current academic year (Fall 2015 - Spring 2016), 4,364 students enrolled in 237 sections of English 101 and English 102 courses were taught by 79 different faculty members of varying ranks.” • “the badges provide a baseline instructional consistency — but not a standardization — as they ensure learning outcomes are addressed across all sections. The digital badge program is tied directly to academic credit, which increased the traditionally three- credit-hour composition courses by one credit hour. Badges account for 18 to 24 percent of the final course grade. “ Reid, Alan, and Denise Paster. "A Digital Badge Initiative: Two Years Later -- Campus Technology". N.p., 2016. Web. 18 Apr. 2016. Collective Action Dilemma
  23. 23. CRICOS Provider Code 00301J Badges offer an effective measure of learning outcomes…. “However, one thing is certain: This initiative supports collaboration and pedagogical innovation throughout our first-year composition program, and it makes visible our commitment to student learning and success in our composition courses.” ….allowing the success of individual initiatives to be monitored A key challenge … “The digital badge initiative at Coastal Carolina University still faces some big questions. At the forefront of our minds are philosophical issues such as how to translate the value of these badges into a real- world currency for our students ……” …is converting gains in quality into improved graduate opportunities Collective Action Dilemma
  24. 24. Pilot Project: Assessing and Developing English Language Proficiency • Good Practice Principles for English language proficiency: • “Principle 7. Students’ English language development needs are diagnosed early in their studies and addressed, with ongoing opportunities for self-assessment.” • The Higher Education Standards Framework (2011) requires English language proficiency to be • “incorporated into course design (Section 1.3) and assessment” (Section 5.6), as well as • “supported through entry requirements” (Section 3.2). CRICOS Provider Code 00301J
  25. 25. Pilot project Issues in implementing embedding Embedding = Academic staff + learning advisors support students in specific context; however: issues of time, commitment and sustainability “Academics are concerned about their students’ communications skills, but do not believe they have the time and expertise to address these concerns within the disciplinary curriculum… (learning) advisers have expertise in developing student’s academic language and literacy, but developing collaborative approaches within disciplinary curricula is often hit and miss” (Arkoudis, 2014, p. 12). How can we overcome this impasse? CRICOS Provider Code 00301J
  26. 26. Pilot Project English language specialists assess existing assignments for ELP with the results and feedback carried forward to benchmark future assignments. • Incentives to achieve threshold standards are enhanced by language specialists assessing ELP using assignments from the curriculum • Sustainability is more likely since ELP assessments are independent of the commitment of academics (from F/T to casual) • Students/staff are able to take a developmental approach since assessments are administered on an ongoing basis over the degree CRICOS Provider Code 00301J
  27. 27. Pilot Project • Conducted at Curtin University Sydney, July to October 2015 • Three selected units + written assignments • Assessment Tool based on IELTS writing criteria and simplified into everyday language • Students used Assessment Tool to self-assess their ELP • Learning advisor assessed students using the same Tool • Results were given to students to compare • Students completed opinion surveys CRICOS Provider Code 00301J
  28. 28. Assessment Tool Three parts: coherence + cohesion; lexical resource; grammatical range + accuracy Student adds total marks, calculates percentage Example of results: Basic writer 50-65% You have a basic command of written English, coping with overall meaning in most situations, though you are likely to make many errors. You may be unable to use more complex language. To improve: read in English as often as you can. Keep lists of unfamiliar vocabulary and revise them regularly. Practice using new words in sentences. CRICOS Provider Code 00301J Pilot Project
  29. 29. Pilot Project Findings and discussion • Students assessed themselves reasonably accurately • Lexical resources were under-estimated • Grammatical range and accuracy was over-estimated • The students’ assessments mostly aligned overall with the learning advisor’s assessments • 80% of students felt it was important to have their English language ability assessed in their assignment • 73.4% of students felt the Assessment Tool was easy to use and understand • As for the costs, it is estimated that it takes an hour to mark four assignments (approx. 2000 words) CRICOS Provider Code 00301J
  30. 30. Further Research 1. Does benchmarking against a previous ELP assessment improve ELP and assignment outcomes? • return to students in our current sample and ask them to repeat the exercise • compare their average scores and ability to self- assess with other students 2. What are the challenges, in terms of design, management and monitoring, for a program-wide approach to embedding? 3. The role badging and e-Portfolios for supporting threshold standards and strenthening the link between graduate competencies & opportunities CRICOS Provider Code 00301J
  31. 31. References • Akerlof, G. A. (1970). "The Market for 'Lemons': Quality Uncertainty and the Market Mechanism". Quarterly Journal of Economics (The MIT Press) 84 (3): 488–500 • Arkoudis, S. (2014). Integrating English Language communication Skills into Disciplinary Curricula: Options and Strategies. Sydney: Office of Learning and Teaching, Department of Education. • Arkoudis, S., & Tran, L. (2010). Writing Blah, Blah, Blah: Lecturers’ Approaches and Challenges in Supporting International Students. International Journal of Teaching and Learning in Higher Education 2010, 22(2), 169-178. • Barrett, S., & Dannenberg, A. (2014). On the sensitivity of collective action to uncertainty about climate tipping points. St. Louis: Federal Reserve Bank of St Louis. • Campbell, J., & Li, M. (2008). Asian Students’ Voices: An Empirical Study of Asian Students’ Learning Experiences at a New Zealand University. Journal of Studies in International Education, 12(4), 375-396. • Dill, D. D. (2005). Are Public Research Universities Effective Communities of Learning? The Collective Action Dilemma of Assuring Academic Standards. Paper Presented at the Symposium: Future of the American Public Research University, 25 February. CRICOS Provider Code 00301J
  32. 32. References • Department of Education, Employment and Workplace Relations. (2009). Good Practice Principles for English language proficiency for international students in Australian universities. Retrieved from Good_Practice_Principles2009.pdf • Harris, A. J., & Ashton, J. (2011). Embedding and integrating language and academic skills: An innovative approach. Journal of Academic Language and Learning, 5(2), 73- 87. • Martin, R. E. (2011) The College Cost Disease. Edward Elgar • Murray, N. & Nallaya, S. (2014): Embedding academic literacies in university programme curricula: a case study, Studies in Higher Education, 1 – 17. • Podolny, J. M., 1993. A Status-Based Model of Market Competition. American Journal of Sociology, pp. 829-872. • Reid, A. J. & Paster, D.(2016). Digital badges in undergraduate composition courses: Effects on intrinsic motivation. Journal of Computers in Education, 2(4), 377-398. • Reid, A., and Paster, D.. "A Digital Badge Initiative: Two Years Later -- Campus Technology". N.p., 2016. Web. 18 Apr. 2016. • Spence, M. (1973). "Job Market Signaling". Quarterly Journal of Economics 87 (3): 355–374. CRICOS Provider Code 00301J
  33. 33. Findings and discussion N=23 Total score (max 65) Coherence & cohesion (max 25) Lexical resource (max 20) Grammatical range & accuracy (max 20) Assessment: Ave (St. Dev) 40.91 (7.954) 18.09 (4.709) 12.13 (3.217) 8.39 (3.381) Self-Assessment: Ave (St. Dev) 42.30 (9.192) 17.22 (3.655) 9.79 (3.415) 12.96 (3.574) Shapiro-Wilk Normality Test on Diff. (p-values) .956 .103 .185 .666 Ave. Differences (Self-Assess. Less Assess). 1.391 -.870 -2.345 4.565 95% CI [-1.998,4.781] [-2.673, .934] [-4.01,-.685] [2.960,6.179] t-value (p-value) (df =22) .851 (.404) -1.000 (.328) -2.929 (.008) 5.893 (.000)
  34. 34. 1)Afterthisstudy,IammoreawareofhowmyEnglish languageabilityaffectsmyassignment 1 0 1 8 5 Strongly Disagree Disagree Unsure Agree Strongly Agree
  35. 35. 2)IlikebeingabletoassessmyownEnglishability 1 0 3 7 4 Strongly Disagree Disagree Unsure Agree Strongly Agree
  36. 36. 3)Ithinkit'simportantthatmyEnglishlanguageabilityis assessedinmyassignment 1 0 2 7 5 Strongly Disagree Disagree Unsure Agree Strongly Agree
  37. 37. 4)ThemarkingguideIwasgiventoassessmyassignmentwas easytouseandunderstand. 0 0 4 8 3 Strongly Disagree Disagree Unsure Agree Strongly Agree
  38. 38. 5)ItisimportanttometoimprovemyEnglishwritingskills. 1 0 1 5 8 Strongly Disagree Disagree Unsure Agree Strongly Agree