Talk presented during GIREP Conferenze 2022

1. A Two Tier
Astronomy
Questionaire
Silvia Galano
I.C. "A. Gemelli" Sant'Agnello (NA), Italy
Department of physics "E. Pancini",
University Federico II of Napoles, Italy

2. Introduction
Multiple choice questionnaire for assessment
• reliable method to detect common students' difficulties;
• a way to receive information about the students’ learning progress.
Students' assessment results:
• inform teachers' strategies and educational approach;
• are useful to compare different teaching approaches' effectiveness.

3. Introduction
• Two-tier instruments are multiple-choice instruments that consist of
items with two sub-questions (tiers):
a. The first tier consists of a content question
b. The second tier, or reason tier, asks students for the alternative best
representing their reasoning in the first tier. In other words the second tier,
evaluates the “explanatory knowledge” or the “mental models” of the
students.
(Treagust, 1988; Tamir, 1989; Tsai & Chou, 2002, p. 158; Scotti di Uccio et al. 2020)

4. Introduction
From two-tier to three tier test
Two-tier cannot differentiate mistakes due to lack of knowledge from
mistakes due to existence of alternative conceptions
(Hasan, Bagayoko, & Kelley, 1999, Scotti di Uccio et al. 2020)
Three-tier diagnostic tests add a Confidence Tier to the two-tier
(Al-Rubayea, 1996; Franklin, 1992; Hill, 1997; Kaaltakęi & Didi, 2007; Caleon & Subramaniam, 2010,
ge Arslan, H., Cigdemoglu, C. & Moseley, C., 2012)

5. Introduction
The use of three-tier is increasing.
Researcher are working to develop this kind of instrument on many different subjects:
• Gravity (Kaaltakęi & Didi 2007)
• Waves (Caleon & R. Subramaniam, 2010)
• Global Warming, Greenhouse Effect, Ozone Layer Depletion, and Acid Rain (Peşman & Eryılmaz,
2010)
• Acids and bases (Cetin-Dindar & Gebanc, 2011)
• Atmosphere-related environmental problems diagnostic test (AREPDiT) - Global warming (GW),
greenhouse effect (GE), ozone layer depletion (OLD), and acid rain (AR) (Ozge Arslan et al., 2012)
• Ionic and covalent bonds (Prodjosantoso et al., 2019)
We focused on Astronomical topics - Change of Seasons; Eclipses and Moon Phases

6. Theoretical
Framework
Misconceptions on
changes of seasons
• Tilt of Earth’s axis (Atwood & Atwood,
1996)
• Earth-Sun distance Changes in Earth-
Sun distance cause changes in seasons.
It is summer when the Earth is closer to
the Sun (Baxter, 1989; Nazé &
Fontaine, 2014)
• Weather
A Year of Sunrises Image Credit & Copyright: Luca Vanzella

7. Theoretical
Framework
Misconception on Moon Phases
• Students often confuse lunar eclipses with Moon
phases, in particular they are unable to articulate the
difference between a full moon and a lunar eclipse
and how the role the Earth plays in the occurrence of
a lunar eclipse (Trumper, 2000; Barnett, 2002; Barnett & Morran
J., 2002).
• Moon phases are due to the Sun periodically blocking
the view of the moon from earth - ‘‘Sun blocking’’.
(Trundle, Atwood, Christopher, 2007)
• Moon phases are explained only in terms of the
shadows of other planets. (Baxter, 1989)
Phases of the Moon Image Credit & Copyright: Jean-Francois Gout, Tom Polakis

8. Theoretical
Framework
Misconception on Eclipses
• Many students thought that the Moon
must be in its Full phase for there to be a
total solar eclipse (Trumper, 2001)
• Many students think that during lunar
eclipses the Sun is between the Earth and
the Moon so that it is not possible to see
the Moon. (Testa et al., 2013)
• Students are not able to justify period of
eclipses. (Testa et al., 2013, Testa et al.,
2015)
Eclipse on the Water Image Credit & Copyright: Elliot Severn

9. Research Questions
RQ1: what are the respondents’
conceptions about the change of
seasons, eclipses and Moon phases?
RQ2: what are the psychometric
properties of TT-Astro?
RQ3: to what extent do TT-Astro
psychometric properties depend on the
adopted scoring method?

10. Instrument
15 Two-Tier items
• 15 Qi_a First tier ask for a fact or astronomical event
• 15 Qi_b Second tier ask for justification to the first tier
5 Two-tier questions for each of the targeted astronomical
phenomena
15 items (Ci) asking for confidence (4 Likert scale)
4 items (Ii) asking for Interest in Astronomy (5 Likert scale)
5 items (Ui) asking for Astronomy Utility Value (5 Likert scale)
5 items (RSi) asking for Astronomy Role in Society (5 Likert scale)
For preliminary
analysis we analyze
only answers to the
two tiers Qi

11. Instrument
Two-Tier questions were developed starting from our previous
questionnaire used to develop and validate a Learning Progression
on astronomical phenomena (Testa et.al, 2015) were designed
starting from previous studies (Chastenay & Riopel, 2020) or
were newly designed
Q15_a) Starting from the images, you can
say that:
a. Seasons are due to change in Earth -
Sun distance
b. Seasons are due to the motion of the
Earth around the Sun and to the tilt
of the Earth's axis
c. Seasons are due to the Earth rotation
d. Seasons are due to the tilt of the Earth'
rotation axis
Q15_b) The reason for your answer is:
a) Because of the tilt of the axis, during the revolution motion
some parts of the Earth are closer to the Sun and it changes
how they are lighted up;
b) During the revolution motion the distance between the Earth
and the Sun changes significantly;
c) Due to the tilt of the axis, during the revolution motion, the
angle of incidence between Sun rays and the Earth's surface
changes significantly;
d) During the rotation motion of the Earth, the Earth's
hemisphere lighted up by the Sun changes
Item adapted from Testa et al. (2015)

12. Instrument
Q7_a) The image shows the Moon
partially lighted up. Imagine to go in
space using a spacecraft so you can turn
around the Moon. Which part of the
Moon you will see lighted up?
a) The same;
b) A bigger portion;
c) None;
d) The whole surface;
Q7_b) The reason for your answer is:
a) The Sun lights up always half Moon surface;
b) The lighted up part of the Moon surface that we can
see, does not change according to our position in the
space;
c) The lighted up part of the Moon surface that we can
see, is bigger because of the Moon rotation;
d) From space we can see also the non lighted up part of
Moon surface;
Item adapted from Chastenay & Riopel (2020)

13. Sample
Sample of 197 students from secondary schools in Naples' area
134 first year 68,4% - 23 second year 11,7% - 2 third year 1,0% - 19 fourth year 9,6 % - 17 Fifth year 8,7%
Female Students: 85 (43,1%) - Male Students: 75 (38,1%) - Non declared: 37 (18,8%)

14. Methods - Scoring
Qi_a Qi_b Score assignment
M1
paired
M2
combined
0 0 0 0
0 1 0 1
1 0 0 1
1 1 1 2
M3 – ordered multiple choice score:
Qi_a and Qi_b
2 - correct
1 - partially correct
0 - wrong
Qi = Qi_a + Qi_b
M0 - 30 independent dichotomic items

15. Methods – M3
Item Question Answer Score (M3)
Q15_a
(S5a)
Starting
from the
images,
you can
say that:
Seasons are due to change in Earth - Sun distance 0
Seasons are due to the motion of the Earth around the Sun and to the tilt of the
Earth's axis
2
Seasons are due to the Earth rotation 0
Seasons are due to the tilt of the Earth' rotation axis 1
Q15_
b
(S5b)
The
reason for
your
answer is:
Because of the tilt of the axis, during the revolution motion some parts of the Earth
are closer to the Sun and it changes how they are lighted up;
1
During the revolution motion the distance between the Earth and the Sun changes
significantly;
0
Due to the tilt of the axis, during the revolution motion, the angle of incidence
between Sun rays and the Earth's surface changes significantly;
2
During the rotation motion of the Earth, the Earth's hemisphere lighted up by the
Sun changes
0

16. Methods – M3
Item Question Answer Score (M3)
Q4_a
(P1a)
Consider the following images. We
can have the "New Moon" phase
“Luna Nuova” when:
Only in position A and E 0
Only in position E 1
Only in position A 1
In one of the position labelled with a letter 2
Q4_b
(P1b)
The reason for your answer is: The New Moon phase must occur in different points of the Moon's orbit
otherwise observer from different point of the Earth will be not able to
see it
0
In the images the Sun is not visible, and the Moon Phases depends on it.
In fact, MP depends on how an observer from the Earth, can see how the
Moon is lighted up from the Sun.
2
"New Moon" occurs only once a month so it cannot occur in more than
one point of the Moon's orbit
1
In the images the Sun is not visible, and Moon Phases depend on the
observer's position on the Earth
0

17. Results M3
Qtot_pp could range from 0 to 60
• Mean 30,1 ± 0,5 (Std. Err. of mean)
• Std. Dev. 6,97
• Skewness ,005 ± ,173 (Std. Err.)
• Kurtosis: - 0,345 ± 0,345 (Std. Err.)
• Cronbach's Alpha ,505
(15 items Qi)
• Cronbach's Alpha ,598
(30 items Qi_a and Qi_b)

18. Results M3
Season Q1_a Q1_b Q2_a Q2_b Q3_a Q3_b Q9_a Q9_b Q15_a Q15_b
S1a S1b S2a S2b S3a S3b S4a S4b S5a S5b
Wrong (%) 10,2 57,4 36,5 14,2 15,7 32 19,8 24,4 44,2 39,6
Partially correct (%) 27,4 21,3 51,8 46,2 58,4 19,3 42,1 45,2 20,8 19,8
Correct (%) 62,4 21,3 11,7 39,6 25,9 48,7 38,1 30,5 35 40,6
Eclipses Q5_a Q5_b Q11_a Q11_b Q12_a Q12_b Q13_a Q13_b Q14_a Q14_b
E1a E1b E2a E2b E3a E3b E4a E4b E5a E5b
Wrong (%) 15,2 15,7 19,3 54,3 65 39,1 34,5 35 42,6 44,2
Partially correct (%) 31 33 57,9 23,9 16,8 44,7 26,4 38,1 31 25,9
Correct (%) 53,8 51,3 22,8 21,8 18,3 16,2 39,1 26,9 26,4 29,9
Moon Phases Q4_a Q4_b Q6_a Q6_b Q7_a Q7_b Q8_a Q8_b Q10_a Q10_b
P1a P1b P2a P2b P3a P3b P4a P4b P5a P5b
Wrong (%) 29,4 37,1 65 31 42,1 43,1 27,4 53,8 57,9 41,6
Partially correct (%) 33 25,9 - 38,6 25,9 29,9 16,2 - 14,7 17,3
Correct (%) 37,6 37,1 35 30,5 32 26,9 56,3 46,2 27,4 41,1

19. Methods – M3
Item Question Answer Score
(M3)
Answers (%)
Q1_a
(S1a)
The mains reason
for the changes of
seasons is
The energy arriving o the Earth from the Sun changes during the
year
0 10,2
The angle of incidence between the sun-rays and the Earth
changes during the year
2 62,4
Day hours changes during the Year 1 13,2
Average temperature on the Earth changes 1 14,2
Q1_
b
(S1b)
The reason for
your answer is:
The position of the earth along its orbit changes during the year 2 21,3
The direction of the Earth's rotation axis changes during the year 0 8,6
The angle between the Earth's rotation axis and the plan of Earth's
orbit changes during the year
0 48,7
Distance between the Earth and the Sun changes 1 21,3

20. Methods – M3
Item Question Answer Score
(M3)
Answers
(%)
Q2_a
(S2a)
If the Earth's
orbit was a
perfect circle
there would no longer be the change of seasons 0
36,5
There would always be alternation of the seasons but the difference
between them would be less perceptible
1
28,4
There would always be alternation of the seasons and the difference
between them would be more perceptible
1
23,4
There would be no differences for our perception 2 11,7
Q2_
b
(S2b)
The reason for
your answer is
that the
seasons are
mainly due to
the
the Earth's revolution around the Sun and the variation of the Earth-Sun
distance
1
20,3
revolution of the Earth around the Sun and the inclination of the
Earth's axis with respect to the plane of the orbit
2
39,6
variation of the Earth-Sun distance and the inclination of the Earth's axis
with respect to the plane of the orbit
1
25,9
Changes in distance between the Earth and the Sun 0
14,2

21. Results M3 – Identified misconception
Weather
10,2%
Tilt of the axis
29,15%
Distance
27,6%
Relative position
of M. E. S.
19,7%
Geographic
position
21,6%
Shadow
29,4%
Observer
position
33,8%
Rotation of M.
and E.
13,7%

22. Results: average frequencies of misconceptions in the second tier vs. first tier
Rotation M. E. (phases) Earth's axix tilt (seasons)
Distance (seasons)
Relative position of E. S. M.
(phases/eclipses)
Geographic position (phases)
Shadows (phases/eclipses)
Observer's position (phases)
Rotation M. E. (phases)
0
35
0
5
10
15
20
25
30
35
0 5 10 15 20 25 30 35
Second
Tier
(%)
First Tier (%)
Weather (seasons) Earth's axix tilt (seasons)
Distance (seasons) Relative position of E. S. M. (phases/eclipses)
Geographic position (phases) Shadows (phases/eclipses)
Observer's position (phases) Rotation M. E. (phases)
Bisettrice Lineare (Bisettrice)

23. Conclusion
RQ1: what are the respondents’ conceptions about the change of seasons, eclipses and Moon phases?
• In agreement with previous research studies all the known misconception on basic astronomical
phenomena emerged.
• The most common misconceptions were those related to Moon Phases: misconception related to the
observer's position and to shadows. This result confirms what emerged from our previous study (Testa et
al., 2015) on "Learning Progression" about basic astronomical phenomena as this misconception are related
to the higher level of the Learning Progression.

24. Future steps
• We will analyse student's answers also using Rasch analysis
• To deepen understand student's reasoning, we will pair differently answers from first and second
tier in order to understand which students were coherent in their reasoning in both the tier. We
will link these new score to the level of the learning progression on basic astronomical
phenomena.
• We will analyse Correlations, Interest in Astronomy, Astronomy Utility Value and Astronomy Role
in Society items
• We will reward some questions as students reported us that some words were too difficult to
understand for them.
• We will extend our sample to validate the revised questionnaire

25. silvia.galano@unina.it Thank you for the
attention

26. Results – M0 scoring
Qtot_M1 could range from 0 to 30
• Mean 10,41 ± 0,31 (Std. Err. of mean)
• Std. Dev. 4,29
• Skewness ,239 ± ,173 (Std. Err.)
• Kurtosis: - 0,108± 0,345 (Std. Err.)
• Cronbach's Alpha ,598

27. Results M0- Difficulty
S1
S2
S3
S4
S5
E1
E2
E3
E4
E5
F1
F2
F3
F4
F5
0
65
0
10
20
30
40
50
60
70
-5 5 15 25 35 45 55 65
Second
Tier
-
Correct
%
First Tier - Correct %
Difficulty
Seasons
Eclipses
Phases
bisettrice
Lineare (bisettrice)

28. Results – M0 scoring
Qtot_M1 could range from 0 to 30
• Mean 10,41 ± 0,31 (Std. Err. of mean)
• Std. Dev. 4,29
• Skewness ,239 ± ,173 (Std. Err.)
• Kurtosis: - 0,108± 0,345 (Std. Err.)
• Cronbach's Alpha 0,588

29. Results M0- Difficulty
S1
S2
S3
S4
S5
E1
E2
E3
E4
E5
F1
F2
F3
F4
F5
0
65
0
10
20
30
40
50
60
70
-5 5 15 25 35 45 55 65
Second
Tier
-
Correct
%
First Tier - Correct %
Difficulty
Seasons
Eclipses
Phases
bisettrice
Lineare (bisettrice)

30. Results – M0 Point-Bi Serial Correlation
Cronbach's Alpha 0,56
S1
S2
S3
P1
E1
P3
P4
S4
P5
E2
E3
E4
E5
S5
-0,05
0,00
0,05
0,10
0,15
0,20
0,25
0,30
0,35
-0,05 0,00 0,05 0,10 0,15 0,20 0,25 0,30 0,35
Second
Tier
First Tier
Point-Bi Serial Correlation (M1)

31. Results – M0
S1_a_dich
S1_b_dich
S2_a_dich
S2_b_dich
S3_a_dich
S3_b_dich
P1_a_dich
P1_b_dich
E1_a_d1ch
E1_b_dich
P2_a_dich
P2_b_dich
P3_a_dich
P3_b_dich
P4_a_dich
P4_b_dich
S4_a_dich
S4_b_dich
P5_a_dich
P5_b_dich
E2_a_dich
E2_b_dich
E_a_dich
E3_b_dich
E4_a_dich
E4_b_dich
E5_a_dich
E5_b_dich S5_a_dich
S5_b_dich
0,00
0,10
0,20
0,30
0,40
0,50
0,60
0,70
-0,10 0,00 0,10 0,20 0,30 0,40 0,50
Difficulty
Point-Bi Serial Correlation
Separate Scoring - Difficulty vs Point-Bi Serial Correlation

32. Results – M1 scoring
Qtot_M1 could range from 0 to 15
• Mean 2,9 ± 0,16 (Std. Err. of mean)
• Std. Dev. 2,19
• Skewness ,668 ± ,173 (Std. Err.)
• Kurtosis: - 0,026 ± 0,345 (Std. Err.)
• Cronbach's Alpha ,588
(15 items Qi_M1)

33. Results – M2 scoring
Qtot_M2 could range from 0 to 30
• Mean 10,32 ± 0,30 (Std. Err. of mean
• Std. Dev. 4,22
• Skewness ,323 ± ,173 (Std. Err.)
• Kurtosis: - 0,003 ± 0,345 (Std. Err.)
• Cronbach's Alpha ,561
(15 items Qi_M2)

34. Results
Discrimination Separate vs. Paired Scoring
S1
S2
S3
MP1
ECL1
MP2
MP3
MP4
S4
MP5
ECL2
ECL3
ECL4
ECL5
S5
-0,10
-0,05
0,00
0,05
0,10
0,15
0,20
0,25
0,30
0,35
0,40
0,00 0,05 0,10 0,15 0,20 0,25 0,30 0,35 0,40 0,45
Separate
Scoring
Paired Scoring
Discrimination Separate vs. Paired Scoring

35. References
• Al-Rubayea, A. A. M. (1996). An analysis of Saudi Arabia high school students’ misconceptions about physics concepts.
Dissertation Abstracts International, 57(04), 1462 (UMI No. 9629018).
• Atwood, R. K. & Atwood, V. A. (1996). Preservice elementary teachers' conceptions of the causes of seasons. J. Res. Sci.
Teach., 33: 553–563.
• Barnett. M. & Morran, J., (2002). Addressing children's alternative frameworks of the Moon's phases and eclipses. Int. J.
of Sci. Educ. Vol. 24 , Iss. 8
• Baxter, J., (1989). Children’s understanding of familiar astronomical events. International Journal of Science Education,
11, 502–513
• Caleon, I. & Subramaniam, R. (2010) Development and Application of a Three‐Tier Diagnostic Test to Assess Secondary
Students’ Understanding of Waves, International Journal of Science Education, 32:7, 939-961,
https://doi.org/10.1080/09500690902890130
• Cetin-Dindar, A., Gebanc, O.(2011) Development of a three-tier test to assess high school students’ understanding of
acids and bases. Procedia - Social and Behavioral Sciences, Vol 15, pp 600-604
https://doi.org/10.1016/j.sbspro.2011.03.147
• Franklin, B. J. (1992). The development and application of a two-tier diagnostic instrument to detect misconceptions in
the area of force, heat, light and electricity. Dissertation Abstracts International, 53(12), 4186 (UMI No. 9301049).
• Haki Peşman & Ali Eryılmaz (2010) Development of a Three-Tier Test to Assess Misconceptions About Simple Electric
Circuits, The Journal of Educational Research, 103:3, 208-222, https://doi.org/10.1080/00220670903383002

36. References
• Hasan, S., Bagayoko, D., & Kelley, E. L. (1999). Misconceptions and the Certainty of Response Index (CRI).
Physics Education, 34(5), 294–299.
• Hill, G. D. (1997). Conceptual change through the use of student-genrated analogies of photosynthesis and
respiration by college non-science majors. Dissertation Abstracts International, 58(6), 2142 (UMI No.
9735480).
• Ivanjek, L., Morris, L., Schubatzky, T., Hopf, M., Burde, J., Haagen-Schützenhöfer, C., Dopatka, L., Spatz, V. &
Wilhelm, T. (2021). Development of a two-tier instrument on simple electric circuits. Phys. Rev. Phys. Educ.
Res. 17, 020123 – Published 24 September 2021
• Kaaltakęi, D., & Didi, N. (2007). Identification of pre-service Physics teachers’ misconceptions on gravity
concepts: A study with a three-tier misconception test. AIP Conference Proceedings, 899(1), 499–500
• Nazé, Y. & Fontaine S., (2014). An astronomical survey conducted in Belgium. Phys. Educ. 49, 151–63
• Ozge Arslan, H., Cigdemoglu, C. & Moseley, C. (2012) A Three-Tier Diagnostic Test to Assess Pre-Service
Teachers’ Misconceptions about Global Warming, Greenhouse Effect, Ozone Layer Depletion, and Acid
Rain, International Journal of Science Education, 34:11, 1667-1686, DOI: 10.1080/09500693.2012.680618

37. References
• Prodjosantoso, A. K., Hertina, A. M., Irwanto (2019) The Misconception Diagnosis on Ionic and Covalent Bonds Concepts with Three Tier Diagnostic
Test. International Journal of Instruction 12(1):1477-1488 DOI: 10.29333/iji.2019.12194a
• Scotti di Uccio, U., Colantonio, A, Galano, S., Marzoli, I., Trani, F., Testa, I. (2020). Development of a construct map to describe students’ reasoning
about introductory quantum mechanics. Phys. Rev. Phys. Educ. Res. vol. 16 (1) https://doi.org/10.1103/PhysRevPhysEducRes.16.010144.
• Tamir,P. (1989). Some issues related to the use of justifications to multiple-choice answers, J. Biol. Educ. 23, 285.
• Testa, I., Cretella, G., Leccia, S. & Puddu, E., (2013). Concezioni alternative degli studenti di scuola superiore su argomenti di base di Astronomia.
Parte 1: Ruolo dell'Istruzione formale. Giornale di fisica della Società Italiana di Fisica, Vol. 54 (3), 195-215
• Testa I, Galano S, Leccia S, Puddu E (2015). Development and validation of a learning progression for change of seasons, solar and lunar eclipses, and
moon phases. Physical Review Special Topics. Physics Education Rresearch, vol. 11, ISSN: 1554-9178.
http://dx.doi.org/10.1103/PhysRevSTPER.11.020102
• Treagust, D. F. (1988). Development and use of diagnostic tests to evaluate students’ misconceptions in science. International Journal of Science
Education, 10, 159–170.
• Trumper, R. (2000). University students’ conceptions of basic astronomy concepts. Phys. Educ. 35 9–15
• Trundle, K. C., Atwood, R. K., Christopher, J. E., (2007). A Longitudinal Study of Conceptual Change: Preservice Elementary Teachers’ Conceptions of
Moon Phases. J. of Res. in Sci. Teach. Vol. 44, No. 2, PP. 303–326
• Tsai, C. C., & Chou, C. (2002). Diagnosing students’ alternative conceptions in science. Journal of Computer Assisted Learning, 18, 157–165.