How to Design and Evaluate Research in Education

Seminar in English Research
Presented by: Raghda Al-Turani
Supervised by: Prof. Dina Al-Jamal
Second semester: ( 2020-2021)

OBJECTIVES Studying this chapter should
enable you to:
 Distinguish between a sample and a population.
 Explain what is meant by the term “representative
sample.”
 Explain how a target population differs from an accessible
population.
 Explain what is meant by “random sampling,” and
describe briefly three ways of obtaining a random sample.
 Use a table of random numbers to select a random sample
from a population.
 Explain how stratified random sampling differs from
cluster random sampling.

OBJECTIVES Studying this chapter should
enable you to:
 Explain what is meant by “systematic sampling,”
“convenience sampling,” and “purposive sampling.”
 Explain how the size of a sample can make a
difference in terms of representativeness of the
sample.
 Explain what is meant by the term “external validity.”
 Distinguish between population generalizability and
ecological generalizability and discuss when it is (and
when it is not) appropriate to generalize the results of
a study.

Main points
-* SAMPLES AND SAMPLING
 The term sampling refers to the process of selecting the individuals who will
participate (e.g., be observed or questioned) in a research
study.
 A sample is any part of a population of individuals on whom information is
obtained. It may, for a variety of reasons, be different from the sample originally
selected.
SAMPLES AND POPULATIONS
 The term population refers to all the members of a particular group. It is the group
of interest to the researcher, the group to whom the researcher
would like to generalize the results of a study.
 A target population is the actual population to whom the researcher would like to
generalize; the accessible population is the population to whom the researcher is
entitled to generalize.
 A representative sample is a sample that is similar to the population on all
characteristics.

Main points ……………………….
RANDOM VERSUS NONRANDOM SAMPLING
*Sampling may be either random or nonrandom.
• Random sampling methods include:
• 1: simple random sampling
• 2: stratified random sampling
• 3: cluster random sampling, and
• 4: two-stage random sampling.
*Nonrandom sampling methods include:
• 1: systematic sampling
• 2: convenience sampling, and
• 3: purposive sampling.

Main points……………..
RANDOM SAMPLING METHODS
1: A simple random sample is a sample selected from a
population in such a manner that all members of the population
have an equal chance of being selected.
2: A stratified random sample is a sample selected so that
certain characteristics are represented in the sample in the same
proportion as they occur in the population.
3: A cluster random sample is one obtained by using groups as
the sampling unit rather than individuals
4: A two-stage random sample selects groups randomly and
then chooses individuals randomly from these groups.
* A table of random numbers lists and arranges numbers in no
particular order and can be used to select a random sample.

Main points
NONRANDOM SAMPLING METHODS
1: A systematic sample is obtained by selecting every
nth;( unspecified item typically the last one) name in a
population.
2: A convenience sample is any group of individuals
that is conveniently available to be studied.
3: A purposive sample consists of individuals who have
special qualifications of some sort or are deemed
representative on the basis of prior evidence.

Main points …………………………………..
**SAMPLE SIZE
Samples should be as large as a researcher can obtain with a
reasonable expenditure of time and energy. A recommended
minimum number of subjects is 100 for a descriptive study, 50 for a
correlational study, and 30 in each group for experimental and
causal comparative studies.
EXTERNAL VALIDITY (GENERALIZABILITY)
* The term external validity, as used in research, refers to the extent that
the results of a study can be generalized from a sample to a population.
* The term population generalizability refers to the extent to which the
results of a study can be generalized to the intended population.
* The term ecological generalizability refers to the extent to which the
results of a study can be generalized to conditions or settings other than
those that prevailed in a particular study.
REPLICATION: When a study is replicated, it is repeated with a new
sample and sometimes under new conditions.

BREAK THE ICE ……..
What do we mean by the following:
 Sample?......
Sampling?...... and
Population?..............

Sample, sampling and population
* population * Sample
 The larger group from
individuals are selected
to participate in a study
 The smaller (but hopefully
representative) collection of
units/ individuals from a
population used to determine
truths about that population
 * Sampling
• refers to the process of
selecting these individuals
for a study.

Examples about samples selected
from populations
 A researcher is interested in studying the effects of diet on the
attention span of third-grade students in a large city. There are
1,500 third-graders attending the elementary schools in the city.
The researcher selects 150 of these third-graders, 30 each in
five different schools, as a sample for study.
 An administrator in a large urban high school is interested in
student opinions on a new counseling program in the district.
There are six high schools and some 14,000 students in the
district. From a master list of all students enrolled in the district
schools, the administrator selects a sample of 1,400 students
(350 from each of the four grades, 9–12) to whom he plans to
mail a questionnaire asking their opinion of the program.

Examples about the population
 All fifth-grade classrooms in Delaware (the
hypothesis might be that classrooms in which teachers
display a greater number and variety of student
products have
higher achievement)
 All high school gymnasiums in Nevada (the
hypothesis might be that schools with “better”
physical facilities produce more winning teams)

Target versus accessible population
 The actual population (called the target population) to
which a researcher would really like to generalize is
rarely available.
• The population to which a researcher is able to
generalize, therefore, is the accessible population.
**The former is the researcher’s ideal choice; the latter,
his or her realistic choice.

Target versus accessible population
The
accessible
population

Examples about target v.s accessible population
 Research problem to be investigated: The effects of
computer-assisted instruction on the reading
achievement of first- and second-graders in California.
 Target population: All first- and second-grade children
in California.
 Accessible population: All first- and second-grade
children in the Laguna Salada elementary school
district of Pacifica, California.
 Sample: Ten percent of the first- and second-grade
children in the Laguna Salada district in Pacifica,
California

Representative sample
If researchers need to draw a conclusion
valid for the whole study population, they
should draw a sample in a way that it is
representative of that population

A non
representative
sample: 2
women & 10 men
A
representative
sample: 6
women & 6
men

Sampling methods
**Two main types of
sampling:
1. Probability
sampling/ Random
--------------------------
2. Non probability
sampling
(purposive/
nonrandom)

Probability sampling
 Probability sampling is generally held to be the most
precise type of sampling, but it is largely
inappropriate for qualitative research.
4. Multi-stages random

EXAMPLE ABOUT THE RANDOM
SAMPLING
 Random: A researcher wishes to conduct a survey of all
social studies teachers in a mid western state to
determine their attitudes toward the new state guidelines
for teaching history in the secondary schools. There are
a total of 725 social studies teachers in the state. The
names of these teachers are obtained and listed
alphabetically. The researcher then numbers the names
on the list from 001 to 725. Using a table of random
numbers, which he finds in a statistics textbook, he
selects 100 teachers for the sample.

Types of probability sampling
(Random)
 Simple random sampling
 Stratified random sampling
 Cluster random sampling
 Multi stage random sampling (stages
random sampling)

Selecting Simple Random Samples (process)

A table of random numbers
 A table of random numbers lists and arranges
numbers in no particular order and can be used
to select a random sample.

Simple random sampling
 The researcher wants to
select a sample of 200
from a population of 2000.
 Notice that 2000 contains
4 digits
 We go to the first column
and count the first 4 digits
for each number then we
decide whether the
number is representative
or not .

The Simple Random Sampling: (merits
and demerits)

The stratified random sampling

Stratified sample consists of more than one
stratum

The Stratified Random Sampling( process)

:The Stratified Random Sampling: (Merits
and Demerits)

Simple. R. S is
more effective
with largers
numbers of
individuals
Cluster. R.S is
more effective
with largers
numbers of
clusters

) The Cluster Random Sampling( Process)

The Cluster Random Sampling: (Merits and
Demerits)

Stratified v.s cluster sampling

BREAK THE ICE
 When the division depends on the geographical areas
its called ………….. .
( ………………………… ) ???

MULTI-STAGES RANDOM SAMPLING
(TWO-STAGES)
 Two-stage random sample selects groups
randomly and then chooses individuals
randomly from these groups.
 Example: combining cluster random
sampling with individual random sampling
 In each stage we choose randomly

EXAMPLE ABOUT NONRANDOM
SAMPLING
 Nonrandom: The manager of the campus bookstore at a
local university wants to find out how students feel
about the services the bookstore provides. Every day for
two weeks during her lunch hour, she asks every person
who enters the bookstore to fill out a short questionnaire
she has prepared and drop it in a box near the entrance
before leaving. At the end of the two week period, she
has a total of 235 completed questionnaires.
 **Notice that all bookstore users did not have an equal
chance of being included in the sample,

TYPES OF NON RANDOM SAMPLING
1: Systematic sampling
2: Convenience
sampling
3: Purposive sampling

The Systematic Random Sampling
(Merits and Demerits)
Systematic

The Systematic Sampling(process)
How to select a systematic sample?

Example about systematic
sampling
 The principal of a large middle school (grades 6–8)
with 1,000 students wants to know how students feel
about the new menu in the school cafeteria. She
obtains an alphabetical list of all students in the school
and selects every tenth student on the list to be in the
sample. To guard against bias, she puts the numbers 1
to 10 into a hat and draws one out. It is a 3. So she
selects the students numbered 3, 13, 23, 33, 43, and so
on until she has a sample of 100 students to be
interviewed.
 it`s called((Systematic sampling with a random start))

Two terms referring to systematic
sampling
 The sampling interval is the distance in the list between
each of the individuals selected for the sample.
 A simple formula to determine it is:
population size
desired sample size
 The sampling ratio is the proportion of individuals in the
population that is selected for the sample.
 A simple way to determine the sampling ratio is:
sample size
population size

Examples about the convenience
sampling
 To find out how students feel about food service in the
student union at an East Coast university, the manager
stands outside the main door of the cafeteria one
Monday morning and interviews the first 50 students
who walk out of the cafeteria.
 A high school counselor interviews all the students
who come to him for counseling about their career
plans
**In each of the above examples, a certain group of
people was chosen for study because they were available.

The convenience sample(demerits)
* It has a major disadvantage in that the sample will quite
likely be biased
convenience samples cannot be considered representative
of any population and should be avoided if at all possible

Example about purposive sampling
 An eighth-grade social studies teacher chooses the two
students with the highest grade point averages in her
class, the two whose grade point averages fall in the
middle of the class, and the two with the lowest grade
point averages to find out how her class feels about
including a discussion of current events as a regular
part of classroom activity. Similar samples in the past
have represented the viewpoints of the total class quite
accurately.
**The sample selected would be representative of
the population

Example about purposive
sampling
 A researcher is asked to identify the unofficial power
hierarchy in a particular high school. She decides to
interview the principal, the union representative, the
principal’s secretary, and the school custodian because she
has prior information that leads her to believe they are the
people who possess the information she needs.
 ** it is not expected that the persons chosen are themselves
representative of the population, but rather that they possess
the necessary information about the population.

THE PURPOSIVE SAMPLING
*the researcher’s judgment may be in error—he or she
may not be correct in estimating the representativeness of
a sample or their expertise regarding the information
needed

SAMPLE SIZE
SAMPLE SIZE
Samples should be as large as a researcher can
obtain with a reasonable expenditure of time and
energy. A recommended minimum number of
subjects is 100 for a descriptive study, 50 for a
correlational study, and 30 in each group for
experimental and causal comparative studies.

EXTERNAL VALIDITY (GENERALIZABILITY)
• The term external validity, as used in research, refers to the
extent that the results of a study can be generalized from a
sample to a population. (the extent to which the results of a
study can be generalized determine the external validity of a
study)
• The term population generalizability refers to the extent to
which the results of a study can be generalized to the intended
population; ( the degree to which a sample represents the
population of interests)
* The term ecological generalizability refers to the extent to
which the results of a study can be generalized to conditions or
settings other than those that prevailed in a particular study.

Bias in sampling(control for bias
and error)
 What is bias in sampling?
 Bias in sampling is a systematic error in
sampling procedures that lead to a distortion in
the result of the study.
* The sample doesn’t accurately represent the
population.
To sum up…

WeaknessesStrengthsTechnique
Difficult to construct sampling
frame, expensive, lower precision,
no assurance of representativeness
Easily understood, results
projectable
Simple
random
sampling
Difficult to select relevant
stratification variables, not
feasible
to stratify on many variables,
expensive
Includes all important
subpopulation, precision
Stratified
sampling
Imprecise, difficult to compute an
interpret results
Easy to implement, cost-effectiveCluster
sampling
Can decrease representativenessCan increase representativeness,
easier to implement than simple
random sampling, sampling frame
not always necessary
Systematic
sampling
Selection bias, sample not
representative, not recommended
by descriptive or casual research
Least expensive, least time
consuming, most convenient
Convenience
sampling

REPLICATION
REPLICATION: When a study is replicated,
it is repeated with a new sample and
sometimes under new conditions.
* The researcher/ or other researchers repeats
the same study using different groups of
subjects and in different situations.

INSTRUMENTATION
PAGE: 116- 154

OBJECTIVES
**Studying this chapter will enable you to:
 Explain what is meant by the term “data.”
 Explain what is meant by the term “instrumentation.”
 Name three ways in which data can be collected by
researchers.
 Explain what is meant by the term “data collection instrument.”
 Describe five types of researcher completed instruments used
in educational research.
 Describe five types of subject-completed instruments used in
educational research

OBJECTIVES
 Explain what is meant by the term “unobtrusive
measures” and give two examples of such measures.
 Name four types of measurement scales and give an
example of each.
 Name three different types of scores used in educational
research and give an example of each.
 Describe briefly the difference between norm-referenced
and criterion-referenced instruments.
 Describe briefly how to score, tabulate, and code data for
analysis.

SO, WHAT IS IT?
An
anemometer
which
is a device for
measuring wind
speed.

What is meant by RESEARCH
METHODS?
This is it!
RESEARCH
METHODS are
the instruments
used to gather data
for analysis in a
research study.

WHILE…
A Research Methodology addresses
the question “How does the researcher
answer the questions stated in any
study.

What are DATA?
 The term data refers to the kinds of information researchers
obtain on the subjects of their research.
* Examples:
1: Demographic information, such as age, gender, ethnicity,
religion, and so on.
2: scores from a commercially available or researcher-prepared
test
3: Responses to the researcher’s questions in an
oral interview or written replies to a survey questionnaire
4: Essays written by students, grade point averages obtained
from school records, performance logs kept by coaches,
anecdotal records maintained by teachers or counselors and so
on.

INSTRUMENTS and INSTRUMENTATION
 The device (such as a pencil and-paper test, a
questionnaire, or a rating scale) the researcher
uses to collect data is called an instrument.
 the whole process of preparing to collect data is
called instrumentation. It involves selection
or design of the instruments, the procedures
and the conditions under which the instruments
will be administered.

Choosing the instrument is limited to
these questions:
1: Where will the data be collected? ( location).
2: Where will it be? in a classroom? a schoolyard? a private home?
on the street?
3: When will the data be collected? (time of collection.)
4: When is it to take place? in the morning? afternoon? evening?
5: How often are the data to be collected? ( the frequency of
collection.)
6: How many times are the data to be collected? only once? Twice?
7: Who is to collect the data? (Administration of the instruments)
8: Who is to do this? the researcher? someone selected and trained
by the researcher?
* answering these questions is very important in order to make accurate
inferences about the individuals.

VALIDITY, RELIABILITY, AND OBJECTIVITY
 Three keywords that should be taken into consideration
when choosing any instrument: validity, reliability and
objectivity!
 The valid instrument is that it measures what it is supposed
to measure; (Validity).
 A reliable instrument is one that gives consistent results. If a
researcher repeats the test at two or more different times, he
or she will obtain , almost, the same results; (Reliability), it
makes the researcher more confident about the results.
 Objectivity: it refers to the absence of subjective judgments
about the achievement, performance, or characteristics of
subjects.

USABILITY
 A number of practical considerations face every researcher
such as:
* how easy it will be to use any instrument he or she designs
or selects.
* How long will it take to administer?
* Are the directions clear? Is it appropriate for the ethnic or
other groups to whom it will be administered?
* How easy is it to score? How much does it cost? Do
equivalent forms exist? Have any problems been reported by
others who used it? Does evidence of its reliability and
validity exist?

WHO PROVIDES THE INFORMATION?
 In educational research, THREE general methods are
available for obtaining information. *Researchers can
get the information through:
 (1) Themselves, with little or no involvement of other
people; (Researcher instruments)
 (2) Directly from the subjects of the study; (Subject
instruments)
 (3) From others, frequently referred to as informants, who
are knowledgeable about the subjects; (Informant
instruments)

Examples
 A researcher wishes to test the hypothesis that inquiry
teaching in history classes results in higher-level thinking
than does the lecture method.
1: The researcher may observe students in the classroom,
noting the frequency of oral statements indicative of higher-
level thinking,
* she may examine existing student records that may include
test results and/or anecdotal material(administer tests or
request student products (essays, problem sheets) for
evidence.

Examples
 She may also decide to interview students
using questions designed to reveal their
thinking about history (or other topics);
interview persons (teachers, other students) or
ask them to fill out rating scales in which the
interviewees assess each student’s thinking
skills based on their prior experience with the
student.

QUIZ: Choose the right answer:
• A researcher interested in the concept of
mutual attraction describes in ongoing field
notes how the behaviors of people who work
together in various settings have been
observed to differ on this variable.
 A: Subject instruments
 B: Researcher instruments
 C: Informant instruments

• A researcher asks parents to keep anecdotal
records describing the TV characters their
preschoolers spontaneously role-play.
 A: Researcher instruments
 B: Subject instruments

*A researcher in an elementary school
administers a weekly spelling test that
requires students to spell correctly the new
words learned in class during the week.
 A: Researcher instruments
 B: Subject instruments

WHERE DID THE INSTRUMENT
COME FROM?
 There are essentially two basic ways for a researcher to
acquire an instrument:
1:Developing a new instrument; Designing one’s own
instrument is time-consuming,
2: Selecting an already developed instrument: here, we take
the instrument nearly verbatim, as it is; ( adopt) OR adding/
removing or modifying some items in the instrument (
adapt) which links your study to all other research studies
that have used the same instrument and save much time and
effort.

Another way to classify instruments
 1:Written-response instruments include objective (e.g.,
multiple-choice, true-false, matching, or short-answer)
tests, short-essay examinations, questionnaires, interview
schedules, rating scales, and checklists.

Another way to classify instruments
2: Performance instruments include any device designed
to measure either a procedure ; ways of doing things( e.g.
writing a letter, solving a puzzle, mixing a chemical
solution) or a product: the end results of procedures, such as
the correct chemical solution, the
or a properly typed letter.
*Performance instruments are designed to:
 see whether and how well procedures can be followed
 assess the quality of products.

Think and justify
Performance instruments are generally
preferred over written-response
instruments.
A: true
B: false

Types of data collection instruments based on
whether they are completed by researchers
 RESEARCHER-COMPLETED INSTRUMENTS
1: Rating Scales; measured judgments intended to convey the
rater’s judgment about an individual’s behavior or product.
A- Behavior Rating Scales.(asking the observer to circle or mark
a point , (e.g. numerical rating scale, graphic rating scale)
EX: Explains course material clearly
1 2 3 4 5
EX: Listens to teacher’s instructions
1 || | || Always Frequently Occasionally Seldom Never
*Demerits: The problem with this rating scale is that different
observers are quite likely to have different ideas about the
meaning of the terms that the numbers represent (excellent,
average, etc.)

Rating Scales; Product Rating Scales
B: Product Rating Scales: Whereas behavior ratings must be
done at a particular time (when the researcher can observe
the behavior).
 Examples: book reports, essays, maps and charts,
diagrams, drawings, notebooks. (handwriting scale used
in the California Achievement tests)

 2: Interview Schedules ; (a set of questions to be answered by
the subjects of the study similar to a questionnaire.
* Interviews are conducted orally, and the answers to the
questions are recorded by the researcher or a focus group(around
5-10) using audiotapes or videotapes or smartphones while
the questionnaire is written.
 Merits: the interviewer can clarify any questions and also can
ask the respondent to expand on answers that are particularly
important or revealing.
 Demerits: it takes much longer than the questionnaire to
complete.

TYPES OF INTERVIEW
a. Unstructured: in the form of normal conversations or
freewheeling of ideas (opened-ended).
b. Structured: following a particular sequence in the
conduct of questioning and with a well-defined
content(guided/ closed-ended).
c. Semi-structured: there are specific set of questions, but
are also additional probes that may be done in an open-
minded or closed-ended.
-The researcher can gather data from a respondent to add
depth and significance to the findings.

 3: Observation Forms: this involves the researcher to
participate actively in the conducted research using the
observation guide or observation checklist. The researcher
observes and evaluates at the same time using paper-and-
pencil observation forms ;observation schedules.
 Merits: easy to conduct, the simpler the instrument, the
better.
 Demerits: the observer is asked to record more behaviors
than can be done accurately (or to watch too many
individuals at the same time).

4: Tally Sheets. it is a device often
used by researchers to record the
frequency of student behaviors,
activities, or remarks.
• Examples:
* How many high school students
follow instructions during fire drills?
* How often do students in English
class ask questions?
* How often do they ask inferential
questions?

5: Flowcharts. It is a special type of tally
sheet (participation flowchart).
 Flowcharts are particularly helpful in
analyzing class discussions. Both the
number and direction of student remarks
can be charted to gain some idea of the
quantity and focus of students’ verbal
participation in class.
 One of the easiest ways to do this is to
prepare a seating chart on which a box is
drawn for each student in the class being
observed. A tally is then placed in the
box of a particular student each time he
or she makes a verbal comment.

6: Performance Checklists Sample about performance
checklist
 It consists of a list of
behaviors that make up a
certain type of
performance .
 * It is used to determine
whether an individual
behaves in a certain way
when asked to complete
a particular task.

7: Anecdotal Records. It is a record of observed
behaviors written down in the form of anecdotes.
• The observers are free to record any behavior they
think is important.
• There is no need to focus on the same behavior for all
subjects .
• The observers should be specific and factual as much
as possible.

Four types of anecdotes
** The first three are to be avoided. Only the fourth type is
desired.
1: Evaluative statement; it evaluates or judges the behavior of the
child( for instance) as good or bad, desirable or undesirable.
2: Interpretive statement; it explains the child’s behavior
3: Generalized statement; it describes certain behavior in general
terms, as happening frequently, or as characterizing the child
4: Specific or concrete descriptive statements.it tells exactly what
the child or other persons did or said and describe the exact
situation.
*Example: The weather was so cold that we did not go on the
playground today.

8: Time-and-Motion Logs: It is the observation and detailed
recording over a given period of time of the activities of one
or more individuals (for example, during a 15-minute
laboratory demonstration).
* Observers try to record everything an individual does as
objectively as possible and at brief, regular intervals (such
as every 3 minutes).
* Example: a fourth-grade teacher who believed that her
class’s considerable slowness was due to the fact that they
were extremely meticulous in their work. To check this out,
she decided to conduct a detailed time-and-motion study of
one typical student.

Types of data collection instruments based
on whether they are completed by subjects.
 SUBJECT-COMPLETED INSTRUMENTS
1: A questionnaire, the subjects respond to the questions by
writing or by marking an answer sheet
* Selection items include multiple choice, true-false,
matching, or interpretive-exercise questions. While supply
items include short-answer or essay questions.
Merits: it can be mailed or given to large numbers of people
at the same time.
Demerits: unclear or ambiguous questions cannot be
clarified, and the respondent has no chance to expand on or
react verbally to a question of particular interest or
importance.

WHAT’S THE NEXT?
Are
these?
Look…

2: Self-Checklists. The individuals
are asked to study a list of several
activities or characteristics and then
to place a mark opposite the
characteristics they possess or the
activities in which they have engaged
for a particular length of time.
* Self-checklists are often used when
researchers want students to
diagnose or to appraise their own
performance.

 3: Attitude Scales. a set of statements to which an individual
responds for the sake of discovering their attitudes. It
represents how they feel about the topics included in the
questions or statements in the scale.
* A commonly used attitude scale in educational research is: the
a- Likert scale. which tries to assess the subject’s agreement/
disagreement or approval/ disapproval on a five-point scale-
* Another type that is used in classroom research is:
b- a semantic differential; It allows a researcher to measure a
subject’s attitude toward a particular concept. Subjects are
presented with a continuum of several pairs of adjectives (good-
bad, cold-hot, priceless- worthless, and so on) and asked to place
a check mark between each pair to indicate their attitudes.

Example of Attitude Scale;
Pictorial scale
• Other scale that is used for determining the attitudes of young
children using simply drawn faces is:
c- Pictorial Attitude Scale.
* They can be asked to place an X under a face.
* example: How do you feel about English?
• Pictorial Attitude Scale for Use with Young Children
00 00 00

4: Personality (or Character) Inventories.
* They are designed to measure certain traits of individuals or to
assess their feelings about themselves like the Piers-Harris
Children’s Self-Concept Scale (How I Feel About Myself), and the
IPAT Anxiety Scale.
Example: * Instructions: Check the option that most correctly
describes you
SELF-ESTEEM often Sometimes never
_ _____ ______ _______
1. Do you think your friends are
smarter than you? _____ _ _____ __ ______
2. Do you avoid meeting new people? _____ _______ ______
STRESS
1. Do you have trouble sleeping? _______ _______ ______

 5: Achievement, ability, Tests. They measure an
individual’s knowledge or skill in a given area or subject,
especially, in schools to measure learning or the
effectiveness of instruction.
 Two types of tests are presented:
 1: general achievement test; batteries of tests like
Sequential Tests of Educational Progress (STEP) that
measure such things as vocabulary, reading ability,
language usage, math, and social studies.
 2: specific achievement test; it measures an individual’s
ability in a specific subject, such as English, biology… .

 6: Aptitude, intelligence, Tests. which assess
intellectual abilities that are not, in most cases, taught
in school, some measures can be used as either an
independent or a dependent variable to assess the
effects of different instructional programs.
 The same test may be either an aptitude or an
achievement test, depending on the purpose and
content. Aptitude test usually includes a wider variety
of skills or knowledge for which it is used.

Examples**
 For example, to measure the effectiveness of an instructional
program designed to increase problem-solving ability in
mathematics, a researcher might decide to use an aptitude test to
adjust for initial differences in ability.
 Aptitude test may be administered to individuals or groups.
* Merits and demerits of group-administered tests:
 Merits: they are more convenient to administer which save s
considerable time.
 Demerits: it is difficult for those taking the test to have test
instructions clarified or to have any interaction with the examiner
(which sometimes can raise scores).
 Also, the range of possible tasks on which the student can be
examined is much less with a group-administered test than with an
individually administered test.

Examples**
* Examples of group tests:
1: The California Test of Mental Maturity (CTMM).
2: The Otis-Lennon.
* Examples of the individual aptitude tests:
1: Stanford Binet Intelligence Scale.
2: Wechsler scales; (gives IQ scores)
**Many of these intelligence tests provide reliable and valid
evidence when used with certain kinds of individuals and for
different purposes.
*for example: 1-predicting the college grades of middle-class.
2-identifying members of certain minority groups to be placed
in special classes.

 7: Performance Tests. They measure an individual’s
performance on a particular task.
• For example: A typing test, in which individual scores are
determined by how accurately and how rapidly people
type.

• 8: Projective Device. It is any sort of instrument with a
vague stimulus that allows individuals to project their
interests, preferences, anxieties, prejudices, needs, and so
on through their responses to it. It has no right answer and
allows an individual to express something of his or her
own personality.
• Examples: the Thematic Apperception Test (TAT), in which
pictures of events are presented and individuals are asked to make up
a story about each picture.
* Also, Picture Situation Inventory, a series of cartoonlike drawings,
each picture ha a certain situation in which a child has to enter the
response of the teacher for each one.

Sample Items from the project
inventory; Picture Situation Inventory
Do I have to
do this? You
didn’t make
Tom.

 9: Sociometric Devices. Individuals are asked to rate their
peers in some way.
 Examples: The sociogram and the group play.
• Asociogram is a visual representation, usually by means of
arrows, that represents choices people make about other
individuals to assess the climate and structure of interpersonal
relationships within a classroom.
• Each student is usually represented by a circle(if female) or a
triangle (if male), and arrows are then drawn to indicate
different student choices with regard to a particular question.
• For example: Students may be asked, for example, to list three
students whom they consider leaders of the class.

Example of a Sociogram
*No. of Choices received
*No. of Mutual choices
1_1
2_2
4_3
1-1
3_2
1_0
2-1
Name
1 - 1
female
male

Sociometric devices; Role Play
 The group play. Students are asked to cast different
members of their group in various roles in a play to
illustrate their interpersonal relationships.
 The roles are listed on a piece of paper, and then the
members of the group are asked to write in the name of
the student they think each role best describes.
 The casting choices that individuals make often shed
considerable light on how some individuals are viewed by
others .

Types of items or questions used in
subject-completed instruments
 Items Formats. The types of items can take a form of:
a selection item or a supply item
1: A selection item (You choose the most appropriate
answer from a set of possible responses).
 Examples: True-false items, Multiple-choice items,
Matching items, and the interpretive exercises ; ( in which
there are true-false items or multiple-choice items that are
required to a response depending on the introductory
picture, a paragraph, map… in the question) to get at more
complex learning outcomes.

Types of items or questions used in
subject-completed instruments
 2: A supply item( respondents are asked to formulate and then
supply their own answers).
 Examples: a- Short-answer items( a word, phrase, number, or
symbol that best/ necessary to complete a statement).
* It is usually difficult to be answered.
 Question: When the test measures what is intended to measure,
the test seems to be ………………. .
 B- Essay questions(respondents are asked to write about at
length). They are useful for assessing and measuring an
individual’s ability to organize, integrate, analyze, and
synthesize information;( higher-level learning outcomes).

UNOBTRUSIVE MEASURES****
 The reaction of respondents to the instrumentation
process( to being tested, observed, or interviewed) often
will affect the nature of the information researchers
obtain. To reduce this effect, researchers attempt to use
unobtrusive measures; data-collection procedures that
involve no intrusion into the naturally occurring course of
events. No instrument is required, only some form of
record keeping.

Examples**
 For example:
 The degree of fear induced by a ghost-story telling session
can be measured by noting the shrinking diameter of a
circle of seated children.
 Student attitudes toward various topics can be noted
by observing the amount of graffiti about those topics
written on school walls.
 Racial attitudes in two elementary schools might be
compared by noting the degree of clustering of
members of different ethnic groups in the lunchroom
and on the playground.

Types of scores
 Quantitative data are usually reported in the form of
scores as follows: (Raw score , derived score)
1: Raw score; (initial score obtained).
* It includes the number of questions answered correctly
on a certain test.
* The number of times a certain behavior is tallied.
* The rating given by a teacher, and so forth.
*Demerits: an individual raw score is difficult to
interpret.

Example of Raw Score
* (A student received a score of (62/ 100) on a test).
You don’t know whether 62 is an extremely high (or
extremely low) based on the degree of difficulty or ease the
test has.
*Note: We often want to know how one individual’s raw
score compares to those of other individuals taking the same
test, and at other times, raw scores are difficult to interpret
individual scores, so they are often converted to “derived
scores”.

Derived scores
 2:Derived scores: they are obtained by taking raw scores and converting
them into more useful scores to say how well the individual has performed
compared to all others taking the same test.
 Examples of derived scores:
a: Age- and grade-level equivalents; tell us of what age or grade an
individual score is typical.
b: Percentile ranks. They refer to the percentage of individuals scoring at or
below a given raw score; (converting raw scores to percentile ranks (Pr).
Number of students ₊ All students
below score at score
Pr = x x 100%
Total number in group

Example of derived scores/
Percentile ranks
 Suppose a total of 100 students took an examination,
and 18 of them received a raw score above 85, while
two students received a score of 85. Eighty students,
then, scored somewhere below 85.
 Q: What is the percentile rank of the two students
who received the score of 85?
 80+2
Pr= Х 100 = 82
100

Derived scores/ standard scores
 c: Standard scores. They indicate how far a given raw
score is from a reference point.
 They are particularly helpful in comparing an individual’s
relative achievement on different types of instruments.
 For example: comparing a person’s performance on a
chemistry achievement test with an instructor’s rating of
his work in a laboratory.
 *Note: the two most commonly used and reported in
educational research are z scores and T scores.

Norm-referenced instruments versus
Criterion-referenced instruments
 Norm-referenced instruments; (instruments that
provide scores comparing individual scores to the
scores of an appropriate reference group for the
sake of evaluating learner progress ).
 The group used to determine derived scores is
called the norm group, and instruments that
provide such scores are referred to as norm-
referenced instruments.

Examples about Norm-referenced
instruments
 For example:
 Comparing a girl’s score on a grammar test to a group of
girls’ scores on that test.
 scored above 90 percent of all the students in the
class.
 received a higher grade point average in English
literature than any other student in the school.

Criterion-referenced tests
 Criterion-referenced tests focus more directly on instruction
rather than evaluating learner progress through gain in scores.
They are built based on a specific goal, or target (called a
criterion).
* For example:
 spelled every word in the weekly spelling list correctly.
 solved at least 75 percent of the assigned problems.
 achieved a score of at least 80 out of 100 on the final exam.
 Merits: They give both teacher and students a clear-cut goal to
work toward

Criterion-referenced tests
* Demerits:
• Teachers seldom set or reach the ideal of individualized
student goals;( not all of students will reach the criterion).
• It is difficult to establish.
• Note: A criterion-referenced test at any grade level will
almost certainly be easier than a norm-referenced test
• In criterion-referenced tests, researchers must try to write
items that will be answered correctly by 80 percent of the
students. while 50 percent of sts in norm-referenced test
since it is more difficult.

Measurement scales
 There are two basic types of variables; (quantitative and
categorical), each uses a different type of analysis and
measurement, requiring the use of different measurement scales.
 Four types of measurement scales are presented:
1: Nominal( using numbers to report frequencies or percentages).
2: Ordinal(using numbers only to indicate ranking).
3: Interval(using numbers to indicate ranking that represents
equal distances from unknown zero point).
4: Ratio (using numbers to represent equal distances from a
known zero point).

Measurement scales/ Nominal scale
• It is the simplest form of measurement researchers can
use in in which they assign numbers to different
categories in order to show differences.
* Example:
• a researcher concerned with the variable of gender
might group data into two categories, male and
female, and assign the number 1 to females and the
number 2 to males.
• * Merits: it facilitates computer analysis .

Measurement scales/ Ordinal Scale
• In ordinal scale, data may be ordered from high to
low or least to most; (ranking data).
* Example:
• A researcher might rank-order student scores on a
science test from high to low.

Measurement scales/ Interval scale
• Assumes that equal differences between scores really
mean equal differences in the variable measured.
• Example:
• In English language achievement tests, the distance
between scores of 70 and 80 is considered to be the
same( equal) as the distance between scores of 80 and
90; (10 points).
• No true zero point. E.g. The zero degree in a
Thermometer doesn’t indicate that there is no
temperature.

Measurement scales/ Ratio Scale
• A ratio scale is an interval scale that does possess an
actual, or true, zero point; (using numbers to represent
equal distances from a known zero point).
• * Example: a researcher designed a scale to measure
height or income since the zero point on the scale
represents the absence of height (that is, no height).
* Note: It is uncommon in educational research;(when a
student receives a zero on a test this does not mean that
whatever is being measured is totally absent in the
student).

Preparing data for analysis
1: Collected data must be scored accurately and consistently.
* e.g., of consistently : coding the choices of (a), (b), (c), (d)
as 1, 2, 3, or 4.
* The scoring of a self-developed test can produce
difficulties it is usually advisable to have a second person
also score the results.
2: Once scored, data must be tabulated and coded.
* Example:
If the highest value for any variable being analyzed involves
three digits (e.g., 100), then every individual code number
must have three digits (e.g., the first individual to be
numbered must be 001, not 1).

How to Design and Evaluate Research in Education

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How to Design and Evaluate Research in Education

Similar to How to Design and Evaluate Research in Education (20)

Recently uploaded

Recently uploaded (20)

How to Design and Evaluate Research in Education