The SOLO framework is explained and how to use it to construct multiple choice questions is explained.

1. Assessment Using the
SOLO Framework
Carlo Magno
Center for Learning and Assessment Development-
Asia/ USAID
@carlo2020feb

2. Objectives
• Explain the SOLO Framework
• Prepare Table of Specifications using the SOLO
framework
• Craft questions aligned with levels of SOLO using
the validated table of specifications

3. Topics
• International Large-Scale Assessments (ILSAs)
• Short Answer type in PISA items
• SOLO Framework

4. PISA
• The Philippines ranked 79th among the 79
participating countries in PISA 2018 in reading and
2nd from the bottom in mathematics and science.
• educational reform introduced in the Philippines
starting 2013 does not show yet what it has
envisioned as making the Filipino students globally
competitive and equipped with 21st century skills

5. PISA Levels
PISA
Proficiency
Levels
Performance
Level
Mathematics
Competencies/
Tasks
(see Philippine 2018
Report pages 25-26)
Reading
Competencies/
Tasks
(see Philippine
2018 Report
pages 8-11)
Science
Competencies/
Tasks
(see Philippine
2018 Report
pages 34-35)
Number
of Sets
6
High
Performers
Test 7 good for 1-
hour
Test 7 good for 30
minutes
Test 7 good for 30
minutes
3
5 Test 6 good for 1-
hour
Test 6 good for 30
minutes
Test 6 good for 30
minutes
3
4
Middle
Performers
Test 5 good for 1-
hour
Test 5 good for 30
minutes
Test 5 good for 30
minutes
3
3 Test 4 good for 1-
hour
Test 4 good for 30
minutes
Test 4 good for 30
minutes
3
2 Test 3 good for 1-
hour
Test 3 good for 30
minutes
Test 3 good for 30
minutes
3
1a
Low
Performers
Test 2 good for 1-
hour
Test 2 good for 30
minutes
Test 2 good for 30
minutes
3
1b Test 1 good for 1-
hour
Test 1 good for 30
minutes
Test 1 good for 30
minutes
3
Total 21

6. Sample item coding for Math
Item Code
Proficien
cy Level
Content Category Process Context Type
Item
Writer No.
1b_CR_R_PC_CO_01 1b Change and Relationships Reasoning Personal Constructed Response 01
1b_CR_E_OC_CR_01 1b Change and Relationships Employing Occupationa
l
Constructed Response 01
1a_CR_F_SOC_CR_01 1a Change and Relationships Formulating Social Constructed Response 01
1a_CR_I_SCC_CR_01 1a Change and Relationships Interpreting Scientific Constructed Response 01
2_CR_R_PC_MC_03 2 Change and Relationships Reasoning Personal Multiple Choice 03
2_SS_E_OC_MC_03 2 Space and Shapes Employing Occupationa
l
Multiple Choice 03
3_SS_F_SOC_MC_32 3 Space and Shapes Formulating Social Multiple Choice 32
3_SS_I_SCC_MC_32 3 Space and Shapes Interpreting Scientific Multiple Choice 32
4_QT_R_PC_MC_45 4 Quantity Reasoning Personal Multiple Choice 45
4_QT_E_OC_CR_45 4 Quantity Employing Occupationa
l
Constructed Response 45
5_QT_F_SOC_CR_56 5 Quantity Formulating Social Constructed Response 56
5_UD_I_SCC_CR_56 5 Uncertainty and Data Interpreting Scientific Constructed Response 56
6_UD_R_PC_CR_70 6 Uncertainty and Data Reasoning Personal Constructed Response 70
6_UD_E_OC_MC_70 6 Uncertainty and Data Employing Occupationa
l
Multiple Choice 70

8. Sample Item coding for Science
ITEM WRITER CODE Bio_Teacher Name Initials_Region
Subject Area Living Systems
Content Topic (to be copied from PISA Table 4.6) your choice
Knowledge Domain Procedural
Context (Personal / Local / National / Global) your choice
Proficiency Level 1b
Major Scientific Literacy Explaining Phenomena Scientifically
Scientific Literacy
Indicator
Recalling and applying appropriate scientific knowledge.
Item Code PreSet L1
Item Type Multiple Choice
Item Answer Key
Item Stimulus / Scenario
Item Question / Task
Remark/s for the item
-maybe intended for illustrator for some graphic arts inclusion/revision
-maybe intended for IT assistant for some computer manipulations

9. Sample TOS Math
Proficiency Level
Change and Relationships (Writer 01)
Mathematical
Reasoning
Problem Solving Processes
Formulating Situations
Mathematically
Employing Mathematical
Concepts, Facts,
Procedures and
Reasoning
Interpreting,
Applying,
and
Evaluating
Mathematic
al Outcomes
1b constructed response constructed response
1a constructed response
constructed
response
2 constructed response multiple choice
3 multiple choice
multiple
choice
4 multiple choice multiple choice
5 multiple choice
multiple
choice
6 multiple choice multiple choice
Context Personal Occupational Social Scientific
Number of Items 3 4 3 4

10. Science TOS

11. Science TOS

12. TOS Reading

13. Guidelines for interpretive
exercise
 The PISA items are structured as interpretative
exercise. An interpretive exercise has two parts:
The stimuli and the items. The items require
responses based on a common set of stimuli.
 The data may in the form of written materials,
tables, charts, graphs, maps, or pictures.
 The series of related test items take various forms
in PISA but are most commonly multiple-choice,
short answer and extended response.

14. Guidelines for interpretive
exercise
• Selecting the stimuli
• The content of the stimuli are based on the themes
provided by PISA.
 Relevant to the PISA skills, themes, and correct complexity
 Appropriate for students’ knowledge
 Simple reading level (not complex words or sentence
structure)
 Brief but meaningful (abbreviate without removing
essential content)
 Revise for clarity, conciseness, and more precise purpose
 Revise content, as necessary, when developing questions

15. PISA 2009 themes or units
Reading Math Science
Lake Chad
Flu
Graffitti
Labour
Plan International
Police
Runners
Gift
Amanda and the Duchess
New Rules
Moreland
Warranty
A Just Judge
In Poor Taste
Bullying
Bees
Farms
Walking
Apples
Cubes
Continent Area
Growing Up
Speed of Racing Car
Triangles
Robberies
Carpenter
Internet Relay Chat
Exchange Rate
Exports
Coloured Candies
Science Tests
Bookshelves
Litter
Semmelweis’ Diary
Ozone
Daylight
Cloning
Greenhouse
Clothes
The Grand Canyon
Sunscreens
Mary Montagu
Acid Rain
Physical Exercise
GM Crops Biodiversity

16. Guidelines for interpretive
exercise
Stimuli
written
materials
Dialogue, news article, contract, short story,
quotation,
tables Schedule, receipt,
charts
graphs Bar graphs,
maps Topographical maps
pictures Floor plan, infographics, photographs,
diagram, time

17. Guidelines for interpretive
exercise
• Selecting the items
• Require analysis and interpretation of stimuli
(material)
 Don’t ask for answers that directly answered in the
stimuli.
 Don’t ask questions that don’t require the stimuli
(material)
 Keep number proportional to or greater than length of
stimuli (material)
 Follow all pertinent suggestions for objective items
 Atleast one multiple choice and atleast one
constructed response is made for each stimuli

18. Example for PISA Reading

19. Example for PISA Reading

20. Example for PISA Math

21. Example for PISA Math

22. Example for PISA Science

23. Example for PISA Science

24. A. Guide for Short Answer Response
• 1. Make instructions explicit
 Short-answer items usually have simple instructions.
 It is advisable that you give the necessary prompt
before examinees respond to the test items.
 Set clear instructions even for short-answer items,
which should indicate the content area, the task, and
scoring.
 In directing examinee on the task you expect them to
do,
 specify if they answer the question,
 indicate what is described, or complete the sentences,

25. A. Guide for Short Answer Response
• 2. Decide on the item’s format
• Decide if all your items should come in questions,
descriptions, or incomplete sentences.
• Whichever you decide to use to format your items,
maintain consistency of the format for all your short-
answer items.
 Never mix up direct question items with descriptions
or incomplete sentences.
 One important criterion for choosing what format to
use is the age of the student. For younger learners, it is
usually preferable to use direct questions than
descriptions or incomplete sentences.

26. A. Guide for Short Answer
Response
• 3. Structure the items accordingly
• Make sure you structure every item in a way that it requires only a brief
answer (i.e., a word, a symbol, a number [or a set of numbers], a phrase, or
a short sentence).
• This is achieved by formulating very clear, specific, explicit, and error-free
statements in your items.
• A clear and specific question calls for a specific answer.
• Describe clearly and explicitly that the response refers to a specific word,
symbol, or phrase.
 If your items are incomplete sentences, structure every item so that the
missing word or phrase is a keyword or a key idea.
 Ordinarily, an incomplete sentence has only one blank which corresponds
to one missing keyword.
 You may want to remove 2 keywords as long as it does not distort the key
idea of the incomplete sentence which should guide the students in
figuring out the missing words. Never go more than 2 blanks.

27. A. Guide for Short Answer Response
• 4. Provide the blanks in appropriate places.
• Blanks are spaces in the items where examinees supply their answers by writing
a word, a symbol, a number, a phrase, or a sentence.
• If your items are all in a direct question format where each question begins with
an item number, place the blanks below (PISA format).
• If you are using the incomplete sentence format for your items, place the blank
near the end of the sentence. This means that you take out a keyword that is
found near the end of the sentence so that it becomes an incomplete sentence.
• Never take out a keyword from the beginning of a sentence. The reason for this
is that you need to first establish the key idea of the sentence so that students
immediately know what is missing in the sentence right after one reading.

28. Example of short answer from
PISA
• Reading
• Math
• Science

29. Example of short answer from
PISA

30. Example of short answer from
PISA

31. B. Guide for extended response
• 5. Communicate the extensiveness of
expected response
• By reading the constructed response item,
examinees must know exactly how brief or
extensive their responses should be. Make your
item clearly convey the degree of extensiveness
you expect from their response.
• To determine the degree of complexity you
desire to assess: The restricted-response and
extended-response items.
 If you wish to measure examinees’ ability to
understand, analyze, or apply certain concepts

32. B. Guide for extended response
• 6. Prime and prompt students through the
item.
• The constructed response already includes the
context, assessment task, and assessment focus
standards, altogether.
 Prime: The statement of context that provides
a background of the subject matter in focus,
and primes the students’ thinking of that
subject matter. The prime helps examinees to
be selectively attentive to a subject matter that
is relevant to the assessment task.
 Prompt: The assessment task is what the

33. B. Guide for extended response
• 7. Provide clear assessment standards
• The assessment standards inform the students
about what specific aspects of their responses you
will give merit, and what aspects will earn more
credit than the others.
 If, for example, you give credit to their argument if
they can provide an evidence, then you need to
categorically ask for in the item.

34. Example for PISA Reading

35. Example for PISA Reading
Prime We can talk about the content of a piece of writing
(what it says).
We can talk about its style (the way it is
presented).
Fiona wanted the style of this information sheet to
be friendly and encouraging.
Prompt Do you think she succeeded?
Standard Explain your answer by referring in detail to the
layout, style of writing, pictures of other graphics.

36. Example for PISA Math

37. Example for PISA Math
Prime There are two formulae you can use to calculate
the number of apple trees and the number of
conifer
trees for the pattern described on the previous
page:
Number of apple trees = n2
Number of conifer trees = 8n
where n is the number of rows of apple trees.
Prompt Find the value of n
Standard show your method of calculating this.

38. Example for PISA Science

39. Example for PISA Science
Prime Suppose you were Semmelweis.
Prompt (based on the data Semmelweis
collected) why puerperal fever is unlikely
to be caused by earthquakes.
Standard Give a reason

40. Multiple Choice-Making the
distracters
• Which is the shortest route for the driver
to take? (2 Points)
A. The street parallel to CF, then the street
perpendicular to BD, then the street CA.
B. The street intersecting CD whose angle
is as acute as E, then the street parallel
to BD.
C. The street parallel to CD, then the street
that intersects it, then the street
perpendicular to it.
D. The street parallel to CF, then the street
perpendicular to CD, then the street BA.

41. Choices Feedback to the Student SOLO
CODE/SCO
RE
Coach (Teacher or Tutor)
Which is the shortest route for the driver to take?
A. The street parallel to CF,
then the street
perpendicular to BD, then
the street CA.
A. Option A is a longer route
than B as it takes 3 streets
for the driver to reach home
from the restaurant unlike B
where the diver takes two
streets only.
1 Point Explain to the student what
parallel and perpendicular lines
mean. Indicate where in the given
figure are examples of parallel
and perpendicular lines.
B. The street intersecting CD
whose angle is as acute
as E, then the street
parallel to BD.
B. Option B is the shortest
route than A as it takes 2
streets only for the driver to
reach home from the
restaurant unlike A where
the diver takes three streets.
2 points Explain to the student what
parallel and intersecting lines
mean including acute angle.
Indicate where in the given figure
are examples of parallel and
intersecting lines and the acute
angle.
C. The street parallel to CD,
then the street that
intersects it, then the
street perpendicular to it.
C. Option C is a longer route
than B as it takes 3 streets
for the driver to reach home
from the restaurant unlike B
where the diver takes two
streets only.
1 Point Explain to the student what
parallel intersecting, and
perpendicular lines mean. Indicate
where in the given figure are
examples of these lines.
D. The street parallel to CF,
then the street
perpendicular to CD, then
the street BA.
D. Option D is a longer route
than B as it takes 3 streets
for the driver to reach home
from the restaurant unlike B
where the diver takes two
streets only.
1 Point Explain to the student what
parallel and perpendicular lines
mean. Indicate where in the given
figure are examples of parallel
and perpendicular lines.

42. Multiple Choice-Making the
distracters
Levels of Understanding
based on the SOLO
Framework
Code for the
Level
1. Pre-structural 0 No idea
2. Uni-structural 1 1 idea
3. Multi-Structural 2 2 or more ideas that are
disconnected
4. Relational 3 2 or more ideas linked
together to see the big
picture
5. Extended Abstract 4 looking at the idea in a new
or different way

43. SOLO Framework
• Structure of Observed Learning Outcomes
• Solo Taxonomy is a systematic way that describes
how learners' understanding build from easy to
difficult while learning different tasks or subjects.
• The Solo Taxonomy can be used to enhance the
quality of learning within the classroom teaching
and provide a systematic way of developing deep
understanding (Damopolii, 2020).
• Motivates students to ponder where they are
presently in terms of their level of understanding,
and what they must do to progress

44. Application of the SOLO
Framework
• 1. By including pertinent activities and creative lesson delivery
methods that are in tune with the subject, we may improve the
quality and complexity of cognition.
• 2. To set learning objectives in order to provide students with
precise instructions for accomplishing academic objectives.
• 3. To develop evaluation criteria that ultimately direct students in
exam preparation and test administration using standardized
procedures.
• 4. To develop and assess educational programmes that would
assist students to become more motivated and self-assured,
ultimately leading to success

45. Levels of Understanding

47. Levels of Understanding
• 1. Pre-Structural - the learner is unable to comprehend
the material and shows no signs of having done so.
• 2. Uni-structural - the student has a fundamental
knowledge of a concept or activity and only learns one
pertinent part of the work or subject. He or she is
unaware of the significance of the relationships.
• 3. Multi-structural - Students learn about a variety of
important independent aspects at this point. Despite
being aware of the connections between many
elements, the relationship to the whole is still unclear.

48. Levels of Understanding
• 4. Relational - The learner can now comprehend the
significance of several components of the whole. They
can link thoughts and ideas, which results in a
comprehensive understanding of the subject. They can
explain how each component benefits the overall.
• 5. Extended - They learn how to apply and generalize
ideas and concepts from one topic to another in a given
field. Their reaction, suggests that they have
conceptual capacity above and beyond what has been
taught. Depending on how well they comprehend the
assignment or the subject being taught, they might
provide fresh notions and ideas.

49. Breaking down the objective into
levels of understanding

50. exercise
• Select one learning competency from a subject in
one level.
• Provide subtasks following the levels of
understanding:
• Uni-structural
• Multi-structural
• Relational
• Extended

51. Multiple Choice-Making the
distracters
Levels of Understanding
based on the SOLO
Framework
Code for the
Level
1. Pre-structural 0 No idea
2. Uni-structural 1 1 idea
3. Multi-Structural 2 2 or more ideas that are
disconnected
4. Relational 3 2 or more ideas linked
together to see the big
picture
5. Extended Abstract 4 looking at the idea in a new
or different way

52. Multiple Choice-Making the
distracters
• Which is the shortest route for the driver
to take? (2 Points)
A. The street parallel to CF, then the street
perpendicular to BD, then the street CA.
B. The street intersecting CD whose angle
is as acute as E, then the street parallel
to BD.
C. The street parallel to CD, then the street
that intersects it, then the street
perpendicular to it.
D. The street parallel to CF, then the street
perpendicular to CD, then the street BA.

53. Choices Feedback to the Student SOLO
CODE/SCO
RE
Coach (Teacher or Tutor)
Which is the shortest route for the driver to take?
A. The street parallel to CF,
then the street
perpendicular to BD, then
the street CA.
A. Option A is a longer route
than B as it takes 3 streets
for the driver to reach home
from the restaurant unlike B
where the diver takes two
streets only.
1 Point Explain to the student what
parallel and perpendicular lines
mean. Indicate where in the given
figure are examples of parallel
and perpendicular lines.
B. The street intersecting CD
whose angle is as acute
as E, then the street
parallel to BD.
B. Option B is the shortest
route than A as it takes 2
streets only for the driver to
reach home from the
restaurant unlike A where
the diver takes three streets.
2 points Explain to the student what
parallel and intersecting lines
mean including acute angle.
Indicate where in the given figure
are examples of parallel and
intersecting lines and the acute
angle.
C. The street parallel to CD,
then the street that
intersects it, then the
street perpendicular to it.
C. Option C is a longer route
than B as it takes 3 streets
for the driver to reach home
from the restaurant unlike B
where the diver takes two
streets only.
1 Point Explain to the student what
parallel intersecting, and
perpendicular lines mean. Indicate
where in the given figure are
examples of these lines.
D. The street parallel to CF,
then the street
perpendicular to CD, then
the street BA.
D. Option D is a longer route
than B as it takes 3 streets
for the driver to reach home
from the restaurant unlike B
where the diver takes two
streets only.
1 Point Explain to the student what
parallel and perpendicular lines
mean. Indicate where in the given
figure are examples of parallel
and perpendicular lines.

54. Multiple Choice
1. Salvador Dali is
a. a famous Indian.
b. important in international law.
c. known for his surrealistic art.
d. the author of many avant-garde plays.
•It is recommended that the stem be a direct question.
•The stem should pose a clear, define, explicit, and singular
problem.
Why is the item faulty?

55. Multiple Choice
IMPROVED: With which one of the fine arts is
Salvador Dali associated?
a. surrealistic painting
b. avant-garde theatre
c. polytonal symphonic music
d. impressionistic poetry

56. Multiple Choice
2. Milk can be pasteurized at home by
a. heating it to a temperature of 130o
b. Heating it to a temperature of 145o
c. Heating it to a temperature of 160o
d. Heating it to a temperature of 175o
•Include in the stem any words that might otherwise be
repeated in each response.
Why is the item faulty?

57. Multiple Choice
IMPROVED: What is the minimum temperature that
can be used to pasteurize milk at home?
a. 130oC
b. 145oC
c. 160oC
d. 175oC

58. Multiple Choice
3. Although the experimental research, particularly that by
Hansmocker must be considered equivocal and assumptions
viewed as too restrictive, most testing experts would
recommend as the easiest method of significantly improving
paper-and-pencil achievement test reliability to
a. increase the size of the group being tested.
b. increase the differential weighting of items.
c. increase the objective of scoring.
d. increase the number of items.
e. increase the amount of testing time.
Items should be stated simply and understandably, excluding
all nonfunctional words from stem and alternatives.
Why is the item faulty?

59. Multiple Choice
IMPROVED: Assume a 10-item, 10-minute paper-
and-pencil multiple choice achievement test has a
reliability of .40. What is the easiest way of
increasing the reliability to .80? Increased…
a. group size
b. scoring objectivity
c. differential item scoring weights
d. the number of items
e. testing time

60. Multiple Choice
4. None of the following cities is a state capital
except
a. Bangor
b. Los Angeles
c. Denver
d. New Haven
•Avoid negatively stated items
Why is the item faulty?

61. Multiple Choice
IMPROVED: Which of the following cities is a state
capital?
a. Bangor
b. Los Angeles
c. Denver
d. New Haven

62. Multiple Choice
5. Who wrote Harry Potter and the Goblet of Fire?
a. J. K. Rowling
b. Manny Paquiao
c. Lea Salonga
d. Mark Twain
•If possible the alternatives should be presented in some logical,
numerical, or systematic order.
•Response alternatives should be mutually exclusive.
Why is the item faulty?

63. Multiple Choice
IMPROVED: Who wrote Penrod?
a. J. K. Rowling
b. J. R. R. Tolkien
c. V. Hugo
d. L. Carrol

64. Multiple Choice
6. Which of the following statements makes clear the
meaning of the word “electron”?
a. An electronic tool
b. Neutral particles
c. Negative particles
d. A voting machine
e. The nuclei of atoms
•Make all responses plausible and attractive to the less
knowledgeable and skillful student.
Why is the item faulty?

65. Multiple Choice
IMPROVED: Which of the following phrases is a
description of an “electron”?
a. Neutral particle
b. Negative particle
c. Neutralized proton
d. Radiated particle
e. Atom nucleus

66. Multiple Choice
7. What is the area of a right triangle whose sides
adjacent to the right angle are 4 inches long
respectively?
a. 7
b. 12
c. 25
d. None of the above
•The response alternative “None of the above” should be used
with caution, if at all.
Why is the item faulty?

67. Multiple Choice
IMPROVED: What is the area of a right triangle
whose sides adjacent to the right angle are 4 inches
and 3 inches respectively?
a. 6 sq. inches
b. 7 sq. inches
c. 12 sq. inches
d. 25 sq. inches

68. Multiple Choice
8. As compared with the American factory worker in
the early part of the 19th century, the American
factory worker at the close of the century
a. was working long hours.
b. received greater social security benefits.
c. was going to receive lower money wages
d. belonging to a labor union.
e. less likely to have personal contact with employers
Make options grammatically parallel to each other and consistent
with the stem.
Why is the item faulty?

69. Multiple Choice
IMPROVED: As compared with the American
factory worker in the early part of the century, the
American factory worker at the close of the century
a. worked longer hours.
b. had more social security.
c. received lower money wages.
d. was less likely to belong to a labor union
e. had less personal contact with his employer

70. Multiple Choice
9. The “standard error of estimate’ refer to
a. the objectivity of scoring.
b. the percentage of reduced error
variance.
c. an absolute amount of possible error.
d. the amount of error in estimating
criterion scores.
Avoid such irrelevant cues as “common elements” and “pat
verbal associations.”
Why is the item faulty?

71. Multiple Choice
IMPROVED: The “standard error of estimate” is
most directly related to which of the following test
characteristic?
a. Objectivity
b. Reliability
c. Validity
d. Usability
e. Specificity

72. Multiple Choice
10. What name is given to the group of complex
organic compounds that occur in small quantities in
natural foods that are essential to normal nutrition?
a. Calorie
b. Minerals
c. Nutrients
d. Vitamins
In testing for understanding of a term or concept, it is generally
preferable to present the term in the stem and alternative
definitions in the options.
Why is the item faulty?

73. Multiple Choice
IMPROVED: Which of the following statements is
the best description of a vitamin?

74. Multiple Choice
• Arrange options from shortest to longest

75. Workshop
• Select a learning competency and create an
interpretive exercise with 2 multiple choice
questions.
• The options should follow the levels of
understanding. Label the options according to the
levels of understanding.