Competence & portfolios: how can we relate them? Simon Grant JISC Centre for Educational Technology and Interoperability Standards (CETIS) NORDLET 2009-09-18

Overview “ This presentation traces the connections between competence and portfolio across learning, development, assessment and presentation, and suggests some consequences for interoperable and portable representation of skills, competences and frameworks of them.” Needs a workable concept of competence, which comes first.

“ This presentation traces the connections between competence and portfolio across learning, development, assessment and presentation, and suggests some consequences for interoperable and portable representation of skills, competences and frameworks of them.”

Needs a workable concept of competence, which comes first.

Starting points We are talking about learning, education and training Different areas are often distinguished: learning and teaching assessment or evaluation evidence assembly and presentation towards next stage personal and professional development The different aspects of what is learned: explicit, verbalisable knowledge basic capabilities of the individual across contexts competence – contextual, above knowledge and capability

We are talking about learning, education and training

Different areas are often distinguished:

learning and teaching

assessment or evaluation

evidence assembly and presentation towards next stage

personal and professional development

The different aspects of what is learned:

explicit, verbalisable knowledge

basic capabilities of the individual across contexts

competence – contextual, above knowledge and capability

Explicit, verbalisable knowledge “ Do you know...” e.g. do you know about the world; people; society can you say how to do things (explicit “know-how”) can you describe causes and effects in the world Easily testable quizzes, multiple choice tests a traditional aspect of examinations Knowledge can be about competence but that knowledge is not the same as being competent

“ Do you know...” e.g.

do you know about the world; people; society

can you say how to do things (explicit “know-how”)

can you describe causes and effects in the world

Easily testable

quizzes, multiple choice tests

a traditional aspect of examinations

Knowledge can be about competence

but that knowledge is not the same as being competent

Basic capability of the individual “ Can you do it? Show me here and now!” lift this weight thread this needle read this text solve this puzzle make this machine do something... Testable on demand, anywhere given equipment Traditional practical tests, face-to-face evaluation BUT explicit knowledge and basic capability still do not account for on-the-job effectiveness What is missing to make up competent performance?

“ Can you do it? Show me here and now!”

lift this weight

thread this needle

read this text

solve this puzzle

make this machine do something...

Testable on demand, anywhere given equipment

Traditional practical tests, face-to-face evaluation

BUT explicit knowledge and basic capability still do not account for on-the-job effectiveness

What is missing to make up competent performance?

Competence involves choices Competence depends on combination of parts explicit knowledge about what needs to be done and how range of basic capabilities for action on-the-spot choice of adequate actions in real contexts Competence = knowledge + capability + good choices The “good choices” part could be something like the disposition to make adequate choices in real situations so that the outcomes meet some agreed quality criteria Certain sorts of choices relate to ethics when they affect other people in certain ways (but that is another presentation)

Competence depends on combination of parts

explicit knowledge about what needs to be done and how

range of basic capabilities for action

on-the-spot choice of adequate actions in real contexts

Competence = knowledge + capability + good choices

The “good choices” part could be something like

the disposition to make adequate choices in real situations so that the outcomes meet some agreed quality criteria

Certain sorts of choices relate to ethics

when they affect other people in certain ways

(but that is another presentation)

E.g.: football skills Knowledge, e.g. Do you know the rules of the game? Do you know how to recognise good space to move into? Basic capabilities, e.g. Can you kick a ball accurately to a chosen place? Can you keep up a suitable activity rate for 90 minutes? Can you dribble a ball at a certain speed? Choice of adequate actions in real contexts, e.g. Do you keep the ball or pass it at appropriate times in a match? Do you choose well between shooting at goal or playing on? Do you make good choices of where to move to in good time? Do you tackle opponents fairly?

Knowledge, e.g.

Do you know the rules of the game?

Do you know how to recognise good space to move into?

Basic capabilities, e.g.

Can you kick a ball accurately to a chosen place?

Can you keep up a suitable activity rate for 90 minutes?

Can you dribble a ball at a certain speed?

Choice of adequate actions in real contexts, e.g.

Do you keep the ball or pass it at appropriate times in a match?

Do you choose well between shooting at goal or playing on?

Do you make good choices of where to move to in good time?

Do you tackle opponents fairly?

E.g.: diplomatic communication Knowledge, e.g. Do you know the required words / phrases of that language? Do you know about interpersonal communication and diplomacy (e.g. listening, tactfulness)? Basic capabilities, e.g. Can you pronounce the words understandably? Can you string them together meaningfully? Choice of adequate actions in real contexts, e.g. Do you choose words that are effective? Do you choose well between speaking and listening? Do you balance tact with clarity effectively? Do your actions result in successful conclusions?

Knowledge, e.g.

Do you know the required words / phrases of that language?

Do you know about interpersonal communication and diplomacy (e.g. listening, tactfulness)?

Basic capabilities, e.g.

Can you pronounce the words understandably?

Can you string them together meaningfully?

Choice of adequate actions in real contexts, e.g.

Do you choose words that are effective?

Do you choose well between speaking and listening?

Do you balance tact with clarity effectively?

Do your actions result in successful conclusions?

Relating back to the areas of … Learning and teaching Assessment or evaluation Evidence assembly and presentation towards next stage Personal and professional development

Learning and teaching

Assessment or evaluation

Evidence assembly and presentation towards next stage

Personal and professional development

Learning and teaching Learn fact, theory through self-study or being taught Learn basic capabilities through training in lab, on the job, on training ground, on simulator But how do you learn choice part of competence? must be on the job needs feedback ideally self-directed Invites tool to keep track of choices made and their consequences gather evidence for assessment and feedback help communicate about feedback

Learn fact, theory through self-study or being taught

Learn basic capabilities through training

in lab, on the job, on training ground, on simulator

But how do you learn choice part of competence?

must be on the job

needs feedback

ideally self-directed

Invites tool to

keep track of choices made and their consequences

gather evidence for assessment and feedback

help communicate about feedback

Assessment or evaluation Assess explicit knowledge by test, exam, etc. Assess basic capability by practical test, e.g. medical “OSCE” (objective structured clinical exam) driving tests (timed) problem solving for cognitive skills But how is the choice part of competence evaluated? An expert can watch you doing a real job You can make records while performing a job Tool needed to: keep track of expert assessments; or organise the evidence you record for assessment

Assess explicit knowledge by test, exam, etc.

Assess basic capability by practical test, e.g.

medical “OSCE” (objective structured clinical exam)

driving tests

(timed) problem solving for cognitive skills

But how is the choice part of competence evaluated?

An expert can watch you doing a real job

You can make records while performing a job

Tool needed to:

keep track of expert assessments; or

organise the evidence you record for assessment

Evidencing and presentation Evidence for knowledge can be from test results where the tests can be traditional examinations Also for basic capability except the tests may be practical tests CV backed up with certificates can present these But how to evidence and present that choice part? testimonials and witness statements, certainly but also give give audience the same kind of evidence that was assembled for assessment because they might use some of that extra richness Again, needs similar tools

Evidence for knowledge can be from test results

where the tests can be traditional examinations

Also for basic capability

except the tests may be practical tests

CV backed up with certificates can present these

But how to evidence and present that choice part?

testimonials and witness statements, certainly

but also give give audience the same kind of evidence that was assembled for assessment

because they might use some of that extra richness

Again, needs similar tools

Development (Personal, professional or vocational) We could take this as essentially about increasing effectiveness / competence in real life situations all parts of competence, including the good choices Thus, closely connected to the tools which can support learning, assessment and presentation of competence as a whole (including the choice part) You can call these tools anything you like, but over the last few years we have tended to refer to them as electronic portfolios

(Personal, professional or vocational)

We could take this as essentially about increasing effectiveness / competence in real life situations

all parts of competence, including the good choices

Thus, closely connected to the tools which can support learning, assessment and presentation of competence as a whole (including the choice part)

You can call these tools anything you like, but over the last few years we have tended to refer to them as electronic portfolios

Portfolios can aid formative and summative assessment A tool to gather, organise and link evidence may also help with feedback and other communication usually presented like any other assignment could also use blog, wiki, website, etc. as an e-portfolio A tool to help manage the assessment process designed to help assessors secure submission; “signing off” by assessors may also help moderation commonly used for vocational assessment Portfolio tools can be seen as a key part of a wider competence architecture Portfolios to assess competence

Portfolios can aid formative and summative assessment

A tool to gather, organise and link evidence

may also help with feedback and other communication

usually presented like any other assignment

could also use blog, wiki, website, etc. as an e-portfolio

A tool to help manage the assessment process

designed to help assessors

secure submission; “signing off” by assessors

may also help moderation

commonly used for vocational assessment

Portfolio tools can be seen as a key part of a wider competence architecture

Competence architecture We need to be able to refer to relevant competence information across any tool or system Definitions need to be public should have a URI Framework structure expressing designed relationships between definitions Relationships with usage which actually give meaning Cross-references with other frameworks for practical usefulness

We need to be able to refer to relevant competence information across any tool or system

Definitions

need to be public

should have a URI

Framework structure

expressing designed relationships between definitions

Relationships with usage

which actually give meaning

Cross-references with other frameworks

for practical usefulness

Definitions Need to be public and impersonal should have URI for any automatic use Ideally defined outside – referred to from – portfolios though sometimes people want to define their own UK National Occupational Standards are good examples They do not fit very well into IEEE RCD or HR-XML 2.* Much better as XHTML+RDFa, human readable fully formatted text RDF easily extracted for machine processing Could actually be in any format (including RCD or HR-XML) Need to persuade bodies to put them on Web with URIs

Need to be public and impersonal

should have URI for any automatic use

Ideally defined outside – referred to from – portfolios

though sometimes people want to define their own

UK National Occupational Standards are good examples

They do not fit very well into IEEE RCD or HR-XML 2.*

Much better as XHTML+RDFa,

human readable fully formatted text

RDF easily extracted for machine processing

Could actually be in any format (including RCD or HR-XML)

Need to persuade bodies to put them on Web with URIs

Framework structure Definitions usually occur in structured sets There are broader and narrower terms Multiple broader terms possible How about using W3C's SKOS? skos:broader for relating to a fuller knowledge, skill, competence kicking ball accurately skos:broader football skills skos:narrower for relating to some part diplomatic competence skos:narrower knowledge of French Refine to any useful level

Definitions usually occur in structured sets

There are broader and narrower terms

Multiple broader terms possible

How about using W3C's SKOS?

skos:broader for relating to a fuller knowledge, skill, competence

kicking ball accurately skos:broader football skills

skos:narrower for relating to some part

diplomatic competence skos:narrower knowledge of French

Refine to any useful level

Relationships with usage Requirements for them in jobs Personal aspirations to acquire them Personal claims to have acquired them Courses that have them as intended outcomes Evidence that a person has them, including assessments (and results) relating to them qualifications, certificates that formalise assessment results In each case, systems should be able to refer to the definition URI, and not have to rely on text labels Then processing can be automatic where possible

Requirements for them in jobs

Personal aspirations to acquire them

Personal claims to have acquired them

Courses that have them as intended outcomes

Evidence that a person has them, including

assessments (and results) relating to them

qualifications, certificates that formalise assessment results

In each case, systems should be able to refer to the definition URI, and not have to rely on text labels

Then processing can be automatic where possible

Cross-references Bodies are going to define their own competencies Be realistic ... yes they are! This would lead to a Babel of meaninglessness, unless... ...they cross-refer to other definitions Software needs to process these cross-references So how about again using SKOS? skos:exactMatch – trust other's equivalences skos:closeMatch – not sure about other's equivalences skos:narrowMatch – ours covers more than theirs skos:broadMatch – ours covers some but not all of theirs Would be vital model / method / tool extending range of cross-linked definitions

Bodies are going to define their own competencies

Be realistic ... yes they are!

This would lead to a Babel of meaninglessness, unless...

...they cross-refer to other definitions

Software needs to process these cross-references

So how about again using SKOS?

skos:exactMatch – trust other's equivalences

skos:closeMatch – not sure about other's equivalences

skos:narrowMatch – ours covers more than theirs

skos:broadMatch – ours covers some but not all of theirs

Would be vital model / method / tool

extending range of cross-linked definitions

Conclusions Competence = knowledge + capability + good choices Choice between things you notice, things you can do Portfolio-like tools good for dealing with that extra part for learning, assessment, presentation, development as well as representing more traditional evidence Thus portfolios and competence are closely linked Wider competence architecture includes definitions – use any format, but make sure they have URIs framework structures – could use SKOS ability to relate to portfolio information etc. via URI cross-references between definitions to extend range of URIs

Competence = knowledge + capability + good choices

Choice between things you notice, things you can do

Portfolio-like tools good for dealing with that extra part

for learning, assessment, presentation, development

as well as representing more traditional evidence

Thus portfolios and competence are closely linked

Wider competence architecture includes

definitions – use any format, but make sure they have URIs

framework structures – could use SKOS

ability to relate to portfolio information etc. via URI

cross-references between definitions to extend range of URIs

Thanks... Thanks for your attention I look forward to creative discussion Find me on the web

Thanks for your attention

I look forward to creative discussion

Find me on the web

END of main slides A few supplementary ones follow...

A few supplementary ones follow...

Learner competence? How much sense does this make in schools? EQF recognises “study situations”, but what exactly is competence in those situations? Knowledge: of meanings in languages of instruction what else? Basic capabilities – yes – all those key skills that people teach and write about anything else? Is learning competence about e.g. choosing to pay attention to the teacher rather than a disruptive pupil? Later, is it e.g. about managing one's own time?

How much sense does this make in schools?

EQF recognises “study situations”, but what exactly is competence in those situations?

Knowledge: of meanings in languages of instruction

what else?

Basic capabilities – yes –

all those key skills that people teach and write about

anything else?

Is learning competence about e.g. choosing to pay attention to the teacher rather than a disruptive pupil?

Later, is it e.g. about managing one's own time?

EQF definitions Knowledge : the outcome of the assimilation of information through learning. Knowledge is the body of facts, principles, theories and practices that is related to a field of work or study ... knowledge is described as theoretical and/or factual Skill : the ability to apply knowledge and use know-how to complete tasks and solve problems ... skills are described as cognitive (involving the use of logical, intuitive and creative thinking) or practical (involving manual dexterity and the use of methods, materials, tools and instruments) Competence : the proven ability to use knowledge, skills and personal, social and/or methodological abilities, in work or study situations and in professional and personal development ... described in terms of responsibility and autonomy

Knowledge : the outcome of the assimilation of information through learning. Knowledge is the body of facts, principles, theories and practices that is related to a field of work or study ... knowledge is described as theoretical and/or factual

Skill : the ability to apply knowledge and use know-how to complete tasks and solve problems ... skills are described as cognitive (involving the use of logical, intuitive and creative thinking) or practical (involving manual dexterity and the use of methods, materials, tools and instruments)

Competence : the proven ability to use knowledge, skills and personal, social and/or methodological abilities, in work or study situations and in professional and personal development ... described in terms of responsibility and autonomy

My involvement – from 1997 Employability skills in LUSID (& related e.g. RAPID) Janet Strivens, Adam Marshall, University of Liverpool Portfolio approach, now increasingly common Requirements in practice include Noting desired competences; plans to work towards them Cross-linking competences and experiences etc. these experiences may or may not serve as evidence Claiming competence Noting evidence of a claim qualifications or other achievements learning outcomes for completed course test or assessment results

Employability skills in LUSID (& related e.g. RAPID)

Janet Strivens, Adam Marshall, University of Liverpool

Portfolio approach, now increasingly common

Requirements in practice include

Noting desired competences; plans to work towards them

Cross-linking competences and experiences etc.

these experiences may or may not serve as evidence

Claiming competence

Noting evidence of a claim

qualifications or other achievements

learning outcomes for completed course

test or assessment results

To claim a competency... I write a claim, or (not so good) make implicit claim There is no inherent predefined structure to these claims Just a piece of text I assemble evidence for the claim Evidence is for the claim, not for the competency definition The evidence can in principle be of any kind Those who want to see the evidence (e.g. employers) may specify the kind of evidence they want to see I present the claim, with appropriate evidence, to interested people

I write a claim, or (not so good) make implicit claim

There is no inherent predefined structure to these claims

Just a piece of text

I assemble evidence for the claim

Evidence is for the claim, not for the competency definition

The evidence can in principle be of any kind

Those who want to see the evidence (e.g. employers) may specify the kind of evidence they want to see

I present the claim, with appropriate evidence, to interested people

So what might be transported? The claim The less clearly defined the competency definition, the more work the claim has to do to in clarification Including a reference to the competency claimed Should be same as that referred to in requirement (job etc.) Ideally, common and recognised URI but are there common URIs? The relevant and appropriate evidence All this fits comfortably into the LEAP2 approach LEAP2A for agreed Atom-based format other LEAP2 formats possible, e.g. XHTML+RDFa

The claim

The less clearly defined the competency definition, the more work the claim has to do to in clarification

Including a reference to the competency claimed

Should be same as that referred to in requirement (job etc.)

Ideally, common and recognised URI

but are there common URIs?

The relevant and appropriate evidence

All this fits comfortably into the LEAP2 approach

LEAP2A for agreed Atom-based format

other LEAP2 formats possible, e.g. XHTML+RDFa