Evaluating a Programme in Distance Education Preparing Tools for Data Collection

Session Objectives After this session, you are expected to be able to: Identify appropriate tools for data collection; List the characteristics of an appropriate tool; Prepare tools for data collection to evaluate a programme in Open and Distance Learning.

After this session, you are expected to be able to:

Identify appropriate tools for data collection;

List the characteristics of an appropriate tool;

Prepare tools for data collection to evaluate a programme in Open and Distance Learning.

What is Programme Evaluation? A systematic gathering, analysis and reporting of data about a programme to assist decision-making. A process of measuring the worth of a programme.

A systematic gathering, analysis and reporting of data about a programme to assist decision-making.

A process of measuring the worth of a programme.

Mesurement in Programme Evaluation What? How? Learning, Impact of the programme, cost-effectiveness, opinions, etc. Require some tools; quantification of data.

What?

How?

Range of Data Collection Tools Machine Recording of data (video, computer) Standardized Tests Questionnaires Checklists Interview Schedules Direct Observation Sheets

Characteristics of a Tool Validity: Reliability: Does the tool measure what it is supposed to measure? How consistent the tool is in measuring the phenomenon?

Validity:

Reliability:

Types of Questions in a Tool Behaviour/ Action-based: Attitude/ Opinion-based: What the student does? Ownership of electronic gadgets, attendance in counselling. Classification-based: What the student thinks? Writing of assignments, reading of SLM What the student is? Age, gender, and othe demographic variables

Behaviour/ Action-based:

Attitude/ Opinion-based:

Classification-based:

How Questions might be responded Open-ended/ Close-ended Questions Types of Data and precision sought Nominal/ Categorical, eg. RC/SC Ordinal, eg. Rank order of choice of technology Interval, eg. Likert scale, assume equal difference between two points in a scale Ratio, eg. Distance of SC from home, has an absolute zero

Open-ended/ Close-ended Questions

Types of Data and precision sought

Nominal/ Categorical, eg. RC/SC

Ordinal, eg. Rank order of choice of technology

Interval, eg. Likert scale, assume equal difference between two points in a scale

Ratio, eg. Distance of SC from home, has an absolute zero

Steps in Preparation of Tools Objectives of data collection Collection of an item pool Validation of the items Pre-test for reliability Modification and finalization

Objectives of data collection

Collection of an item pool

Validation of the items

Pre-test for reliability

Modification and finalization

Issues in Tool Preparation Consider the language and comprehension level of your respondents Be brief and focused Think about the flow of the questions (logical) Avoid ambiguous words Avoid questions with a negative in them Avoid leading questions, eg. Do you agree that assignments are necessary in Distance Education? Yes  No  Avoid loaded questions, eg. Do you prefer conventional education, because there is widespread corruption in distance education? Yes  No 

Consider the language and comprehension level of your respondents

Be brief and focused

Think about the flow of the questions (logical)

Avoid ambiguous words

Avoid questions with a negative in them

Avoid leading questions, eg. Do you agree that assignments are necessary in Distance Education? Yes  No 

Avoid loaded questions, eg. Do you prefer conventional education, because there is widespread corruption in distance education?

Yes  No 

Issues in Tool Preparation Do not suggest the range of acceptable response, eg. How many hours have you spent in reading this Block? (a) < 2 hours  (b) 2-5 hours  (c) 6-10 hours  (d) 11-15 hours  Avoid questions that are irrelevant, eg. At what time of the day you prefer to attend counselling? (a) Morning  (b) Afternoon  (c) Evening  (d) Night  Do not mix more than one question, eg. How do you find the counselling sessions helpful in understanding Course 01/02/04/05? (a) Excellent  (b) Good  (c) Satisfactory  (d) Unsatisfactory 

Do not suggest the range of acceptable response, eg. How many hours have you spent in reading this Block?

(a) < 2 hours  (b) 2-5 hours  (c) 6-10 hours  (d) 11-15 hours 

Avoid questions that are irrelevant, eg. At what time of the day you prefer to attend counselling?

(a) Morning  (b) Afternoon  (c) Evening  (d) Night 

Do not mix more than one question, eg. How do you find the counselling sessions helpful in understanding Course 01/02/04/05?

(a) Excellent  (b) Good  (c) Satisfactory  (d) Unsatisfactory 

What Tools to Use? A Programme Coordinator wants to gather feedback every year. A Course-writer/ Coordinator wants to check the instructional design quality of SLM An Assistant Regional Director wants to observe a counselling session/ practical for monitoring. An evaluator wants feedback on counselling sessions from counsellors. A Course Coordinator wants Expert Opinion on the quality of a course. An evaluator wants feedback on the quality of assignment evaluation and two-way communication.

A Programme Coordinator wants to gather feedback every year.

A Course-writer/ Coordinator wants to check the instructional design quality of SLM

An Assistant Regional Director wants to observe a counselling session/ practical for monitoring.

An evaluator wants feedback on counselling sessions from counsellors.

A Course Coordinator wants Expert Opinion on the quality of a course.

An evaluator wants feedback on the quality of assignment evaluation and two-way communication.

Credits Thanks to the Participants of this Workshop and earlier workshop on Programme Evaluation Thanks to the University for giving such opportunities Thanks to the Workshop Coordinator, Prof. Basanti Pradhan A STRIDE-IGNOU PRESENTATION THANK YOU