The PISA framework for assessment of ICT literacy Wouter van Joolingen University of Amsterdam

ICT Expert Panel Panel Members Irwin Kirsch (Chair), United States Robert Kozma, United States Chul-Hwan Lee, Korea Jörgen Nissen, Sweden Catherine Régnier, France Jean-Paul Reeff, Luxembourg Arnold Spee, The Netherlands Wouter van Joolingen, The Netherlands Project Staff and Counsel Tracy Gray, United States Marylou Lennon, United States Jay Moskowitz, United States Maria Stephens, United States Claudia Tamassia, France Universiteit van Amsterdam Graduate school of Teaching and Learning

Panel Members

Irwin Kirsch (Chair), United States

Robert Kozma, United States

Chul-Hwan Lee, Korea

Jörgen Nissen, Sweden

Catherine Régnier, France

Jean-Paul Reeff, Luxembourg

Arnold Spee, The Netherlands

Wouter van Joolingen, The Netherlands

Project Staff and Counsel

Tracy Gray, United States

Marylou Lennon, United States

Jay Moskowitz, United States

Maria Stephens, United States

Claudia Tamassia, France

Why measure ICT literacy? One goal of PISA is to measure how well young adults are prepared to meet the challenges of the future. For many, if not most, that future will include ICT Therefore, PISA should include a measure of ICT skills as one of its cross-disciplinary competencies. ICT Panel members felt that the rationale for such an assessment is grounded in a number of issues of concern to policy makers and others that include: Universiteit van Amsterdam Graduate school of Teaching and Learning

One goal of PISA is to measure how well young

adults are prepared to meet the challenges of the future.

For many, if not most, that future will include ICT

Therefore, PISA should include a measure of ICT

skills as one of its cross-disciplinary

competencies.

ICT Panel members felt that the rationale for such

an assessment is grounded in a number of issues of concern to policy makers and others that include:

Why measure ICT literacy? ICT is changing the very nature and value of knowledge and information ICT literacy in its highest form has the potential to change the way we live, learn and work ICT literacy cannot be defined primarily as the mastery of technical skills There is a lack of information about the current levels of ICT literacy both within and among countries Universiteit van Amsterdam Graduate school of Teaching and Learning

ICT is changing the very nature and value of knowledge and information

ICT literacy in its highest form has the potential to change the way we live, learn and work

ICT literacy cannot be defined primarily as the mastery of technical skills

There is a lack of information about the current levels of ICT literacy both within and among countries

Defining ICT Literacy Universiteit van Amsterdam Graduate school of Teaching and Learning ICT literacy is the interest, attitude and ability of individuals to appropriately use digital technology and communication tools to access, manage, integrate and evaluate information , construct new knowledge, and communicate with others in order to participate effectively in society.

ICT processes Universiteit van Amsterdam Graduate school of Teaching and Learning Access Knowing about and knowing how to collect and/or retrieve information. Manage Organising information into existing classification schemes. Integrate Interpreting, summarising, comparing and contrasting information using similar or different forms of representation. Evaluate Reflecting to make judgments about the quality, relevance, usefulness, or efficiency of information. Construct Generating new information and knowledge by adapting, applying, designing, inventing, representing or authoring information. Communicate Conveying information and knowledge to various individuals and/or groups.

Test setup for PISA Computer-based testing Students will work on a notebook computer Closed web environment with test scenario’s Simulated “real ICT-world” Small web Simulated responses from the system and the external world Universiteit van Amsterdam Graduate school of Teaching and Learning

Computer-based testing

Students will work on a notebook computer

Closed web environment with test scenario’s

Simulated “real ICT-world”

Small web

Simulated responses from the system and the external world

Building ICT Tasks Universiteit van Amsterdam Graduate school of Teaching and Learning SCENARIOS TASKS Technology Environments Web Desktop E-learning simulations Contexts Personal Public Educational Occupational Processes Access Manage Integrate Evaluate Construct Communicate Functionality Representations Word Processing E-mail Presentation Database E-commerce Spreadsheet Concept Mapping Browser (includes search engines) Text Images Numbers Graphics

A possible scenario Your friend, Jim, has a new hobby, collecting stamps. He just started it, and is still a beginner on this topic. His birthday is approaching and you decide you want to give Jim a book on stamp collecting, that matches his level of expertise. You don’t know anything about stamps, so you start searching for a suitable book. Universiteit van Amsterdam Graduate school of Teaching and Learning

Your friend, Jim, has a new hobby, collecting stamps. He just started it, and is still a beginner on this topic. His birthday is approaching and you decide you want to give Jim a book on stamp collecting, that matches his level of expertise. You don’t know anything about stamps, so you start searching for a suitable book.

Some considerations To find good books, learn something on the topic yourself Use smart search strategies to identify central topics Evaluate reliability of reviews and different bookshops (including 2 nd hand) and compare deals. Need to make more explicit than in “natural” settings. Universiteit van Amsterdam Graduate school of Teaching and Learning

To find good books, learn something on the topic yourself

Use smart search strategies to identify central topics

Evaluate reliability of reviews and different bookshops (including 2 nd hand) and compare deals.

Need to make more explicit than in “natural” settings.

Scoring Universiteit van Amsterdam Graduate school of Teaching and Learning Access Search strategies – key words used Recycling key words, sites used. Go halfway in the buying process Manage Organization of reviews Integrate Linking reviews, books and other knowledge Evaluate Assess trust in reviews and shops – sources used in task products Construct Build ordered lists of findings Communicate Make deal with on-line shop Present findings to others

The framework Allows for the construction of an assessment instrument that Does justice to many aspects of ICT literacy Allows for testing in realistic situations Goes beyond self assessment Provides an objective account of an individual’s ICT skills and knowledge. Universiteit van Amsterdam Graduate school of Teaching and Learning

Allows for the construction of an assessment instrument that

Does justice to many aspects of ICT literacy

Allows for testing in realistic situations

Goes beyond self assessment

Provides an objective account of an individual’s ICT skills and knowledge.

Plans Feasibility study Try-out with small number of test scenarios USA, Australia, Japan Data collected – analysis in progress Test development, field trial (2004-2005) Inclusion in PISA 2006 At least 15 OECD countries Universiteit van Amsterdam Graduate school of Teaching and Learning

Feasibility study

Try-out with small number of test scenarios

USA, Australia, Japan

Data collected – analysis in progress

Test development, field trial (2004-2005)

Inclusion in PISA 2006

At least 15 OECD countries