Your SlideShare is downloading. ×
The assessment of 'wicked' competences: post-conference version
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

The assessment of 'wicked' competences: post-conference version

2,132

Published on

A report of an enquiry into the ways in which 'wicked' competences are assessed, incorprorating comments made during the presentation at the SEDA conference, 22 November 2006

A report of an enquiry into the ways in which 'wicked' competences are assessed, incorprorating comments made during the presentation at the SEDA conference, 22 November 2006

Published in: Technology, Health & Medicine
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,132
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. The assessment of ‘wicked’ competences: December version Peter Knight and Anna Page Institute of Educational Technology
  • 2. Outline
    • What are ‘wicked’ competences?
    • What’s the problem?
    • The enquiry and its findings
    • Implications for us
    • Finding out more
  • 3. Explanation
    • By ‘wicked’ competences we mean ones that are hard to define, know how to teach and to assess.
        • Derived from the idea of ‘wicked’ problems which are:
            • not understood until after formulation of a solution
            • reflect radically different world views and different frames for understanding the problem.
            • shape-shifting.
            • never solved.
  • 4. The problem
    • ‘ Wicked’ competences are often the soft skills and other complex achievements that graduate employers say they value.
    • What’s a university to do about:
        • fostering
        • assessing
    • them?
  • 5. AWC study design
    • Ask key informants (n=30) in six professions to identify two ‘wicked’ competences’
    • Choose nine most-cited ‘wicked’ competences
    • Construct questionnaire for each subject area and use all available networks to get respondents (83 usable responses by 15/11)
    • Follow-up phone interviews with sub-sample
    • Literature review runs throughout
  • 6. The nine competences
  • 7. Is there likely to be a problem here?
    • Practice falls short of good social science research standards.
      • Data from UK Student Assessment and Classification Working Group on inconsistent and variable assessment practices
    • Standards are vague and incomplete
    • Lack of clear, agreed criteria or interpretations
  • 8. How seriously is AWC taken? Yes Yes Yes Yes Yes Priority Yes Yes Yes Yes Yes Yes Yes Yes Valued Take onwards Self-manage Relate Know Oral Listen Group EI Support
  • 9. How is AWC done? 5 3 12 13 5 8 15 2 20 n=83 3.6 3.6 3.5 1.87 Coursework 3.8 3.8 3.54 4.2 3.5 3.53 4.3 Tests 4.4 3.87 Portfolios 4.0 4.0 3.58 3.69 3.6 0.5 4.16 Simulations 2.0 1.92 3.8 3.8 1.5 Workplace 2.33 3.5 2.38 4.5 Criteria 2.8 2.0 2.4 0.5 Indirect 2.0 2.17 1.85 2.4 2.25 3.5 Direct Take onwards Self-manage Relate Know Oral Listen Group EI Support
  • 10. In your institution, is this competence mainly assessed at programme or at module level? 65 9 15 5 17 3 16 Mainly module, course or unit 18 3 3 5 3 1 3 Mainly programme / award All (n = 83) Youth work (n = 12) Social work (n = 18) Secondary School teaching (n = 10) Nursing (n = 20) Early years teaching (n = 4) Accountancy (n = 19)
  • 11. Cause for concern?
    • Note the argument that ‘wicked’ competences grow slowly and should therefore be assessed across whole programmes.
    • Did informants identify great assessment difficulties?
        • Following slides suggest not
            • Data organised around six areas of potential difficulty: poor training, high cost, time-intensive, fuzzy criteria, meaning of competence not clear, reliable assessment methods lacking.
  • 12. Develop supportive relationships
    • Secondary education & Youth work
    • 1 = Strongly agree; 2 = Agree; 3 = Neutral; 4 = Disagree; 5 = Strongly disagree
    • Blue means ‘not a problem’; Red means ‘problematic’
    Not a problem, yet Q24 says may be a problem for Youth work 2.4 3.8 3.3 3.1 3.5 3.7 Not a problem for Secondary education 2.8 3.3 2.9 4.0 3.7 3.5 Not a problem 2.6 3.6 3.1 3.6 3.6 3.6 Conclusion Reliability Meaning Criteria Time Cost Training
  • 13. Emotional intelligence
    • Youth work
    • 1 = Strongly agree; 2 = Agree; 3 = Neutral; 4 = Disagree; 5 = Strongly disagree
    • Blue means ‘not a problem’; Red means ‘problematic’
    Problem 1.5 2.5 2.3 2.0 1.5 1.5 Conclusion Reliability Meaning Criteria Time Cost Training
  • 14. Group work
    • Accounting
    • 1 = Strongly agree; 2 = Agree; 3 = Neutral; 4 = Disagree; 5 = Strongly disagree
    • Blue means ‘not a problem’; Red means ‘problematic’
    Neutral, bordering on not problematic, yet Q24 says may be a problem 2.4 2.8 3.0 3.1 3.0 3.1 Conclusion Reliability Meaning Criteria Time Cost Training
  • 15. Listening and assimilating
    • Nursing
    • 1 = Strongly agree; 2 = Agree; 3 = Neutral; 4 = Disagree; 5 = Strongly disagree
    • Blue means ‘not a problem’; Red means ‘problematic’
    Not a problem 2.5 3.5 3.0 3.9 4.1 3.1 Conclusion Reliability Meaning Criteria Time Cost Training
  • 16. Oral communication
    • Accounting & Early years
    • 1 = Strongly agree; 2 = Agree; 3 = Neutral; 4 = Disagree; 5 = Strongly disagree
    • Blue means ‘not a problem’; Red means ‘problematic’
      Not a problem except for time, Q24 says is a problem for Early years 2.0 4.0 3.3 2.0 5.0 4.0 Neutral, bordering on problematic, Q24 says is a problem for Accounting 1.5 2.5 2.5 2.5 3.0 2.3 Neutral, bordering on problematic, Q24 says is a problem 1.6 2.8 2.7 2.4 3.4 2.6 Conclusion Reliability Meaning Criteria Time Cost Training
  • 17. Professional subject knowledge
    • Social work
    • 1 = Strongly agree; 2 = Agree; 3 = Neutral; 4 = Disagree; 5 = Strongly disagree
    • Blue means ‘not a problem’; Red means ‘problematic’
    Not a problem 2.7 3.3 3.3 3.5 3.9 3.6 Conclusion Reliability Meaning Criteria Time Cost Training
  • 18. Relating to clients
    • Nursing & Secondary education*
    • 1 = Strongly agree; 2 = Agree; 3 = Neutral; 4 = Disagree; 5 = Strongly disagree
    • Blue means ‘not a problem’; Red means ‘problematic’
    • *There were zero returns for Secondary School 'Relating to clients'
    Not a problem , yet Q24 says may be a problem 2.2 3.3 2.6 3.0 3.9 3.2 Conclusion Reliability Meaning Criteria Time Cost Training
  • 19. Self-management
    • Early years
    • 1 = Strongly agree; 2 = Agree; 3 = Neutral; 4 = Disagree; 5 = Strongly disagree
    • Blue means ‘not a problem’; Red means ‘problematic’
    Not a problem 3.0 3.7 3.6 3.7 3.3 4.0 Conclusion Reliability Meaning Criteria Time Cost Training
  • 20. ‘ Taking it onwards’
    • Social work
    • 1 = Strongly agree; 2 = Agree; 3 = Neutral; 4 = Disagree; 5 = Strongly disagree
    • Blue means ‘not a problem’; Red means ‘problematic’
    Not a problem , yet Q24 says may be a problem 2.0 3.2 2.5 4.0 3.4 3.0 Conclusion Reliability Meaning Criteria Time Cost Training
  • 21. SEDA discussion
    • Discussion at SEDA on 22 November explored three readings of the data:
        • The data can be trusted. There’s not an assessment problem here and no implications for SEDA members
        • The data are an artefact of the enquiry methods. There may be a problem here – more work needed
        • There is a problem of false consciousness here: people don’t realise that there is a big problem with assessing these competences. Serious implications for SEDA members
  • 22. There’s not a problem
    • This takes the survey data at face value. Since the survey was carefully designed on the basis of responses from subject communities, no reason to doubt the results.
        • Worth suspending judgment until the phone interviews are done?
  • 23. Faulty enquiry methods
    • Survey approach was wrong
    • Some questions and phrases may have been open to varying interpretations
    • Phone interviews should create different data.
    • Close examination of actual assessment data should show whether there are reliability problems
    • Health sciences & practice subject centre willing to help in new round of enquiries
    • No immediate implications for SEDA members
  • 24. False consciousness
    • Other work on summative assessment and degree classification suggest people tend to take practice for granted and not to see the problems unless prompted.
    • Literature suggests there ought to be a problem here
    • Argument that people are desensitised to a problem that’s serious.
    • Implications for SEDA:
        • Raise awareness of problem
        • Offer advice on resolving it.
  • 25. Contact us
    • [email_address]
    • [email_address]
    • http:// web.mac.com/knightpeter/iWeb/Site/Welcome.html (Peter’s site. In late December he will post suggestions on dealing with the problems – if they really do exist!)
    • The project report as a work in progress at: http:// kn.open.ac.uk/document.cfm?docid =8933
    • The sponsoring CETL: http://cetl.open.ac.uk/pbpl/p2_1.shtml

×