Authors:Testa I.(1), Colantonio A. (2), Galano S. (1) 1 Department of Physics “E. Pancini”, University Federico II, Naples (Italy) 2 School of Advanced Studies, University of Camerino (Italy)

1. PRINCIPAL COMPONENT ANALYSIS OF
LEARNER-GENERATED DRAWINGS AS A
STRATEGY TO INVESTIGATE STUDENTS’
MENTAL MODELS: EXAMPLES FROM
ASTRONOMY EDUCATION
ITALO TESTA,ARTURO COLANTONIO, & SILVIA GALANO

2. Overview of the talk
• Literature review
• Theoretical Framework
• Research Questions
• Methods
• Results
• Discussion and Implications
• Conclusions
italo.testa@unina.it;
@italo_testa
More information can be found in:
I. Testa, S. Leccia, and E. Puddu, Astronomy textbook
images: do they really help students? Phys. Ed. 49, 332 2198
(2014)
S. Galano, A. Colantonio, S. Leccia, I. Marzoli, E. PudduI & I.
Testa, Developing the use of visual representations to
explain basic astronomy phenomena Phys. Rev. ST Phys.
Educ. Res. Focused Collection on Astronomy Education
Research (2018)
Contacts:

3. Review of literature
To involve students in generating drawing may:
• promote memory, observation, and imagination (Neu & Berglund, 1991; Stein & Power, 1996)
• be beneficial for explanation (Adoniou, 2013)
• help construct patterns by selecting real world features (Cox, 2005)
• promote science learning (Britton & Wandersee, 1997; Quillin & Thomas, 2015)
• assess learning (White & Gunstone, 2000)
• help in uncovering reasoning strategies, attitudes and mental models (Vosniadou & Brewer,
1992; Waldrip et al. 2010; Hsieh & Tsai, 2017)
Renewed emphasis in science education research on the use of drawings as effective tool to access
students’ reasoning strategies and mental models (Ainsworth, Prain, & Tytler, 2011).

4. Review of literature
 It is of essential then to:
 Address methodological issues in analyzing students’ drawings to push further the field
The case of Draw-A-Scientist test (Symington & Spurling, 1990; Losh et al., 2008; Reinisch et al., 2017)
• One or more scientist?
• Appearance
• Activity
• Symbolic representations
• Context in which the scientist is drew
How reliable is the Draw-A-Scientist test to elicit students’ mental models about a scientist?
 Literature results are often inconsistent (Van Meter & Garner, 2005)

5. Theoretical Framework
Paivio (1991) graphical modalities encompass first the construction of an internal representation of the
concept and then the effort to externalize it in conventional form through referential links
Van Meter and Garner (2005) drawing involves constructive learning processes that engage nonverbal
representational modalities and requires integration
Ehrlén (2008) drawings are products of pictorial conventions in cultural contexts and, therefore, the
conception that is represented in a drawing depends on the convention chosen by the student for the
representation.

6. Research questions
RQ1. To what extent drawings may reliably uncover students’ mental
models about a given phenomenon?
RQ2. What is the relationship between drawings and knowledge about the
target phenomenon?
RQ3. To what extent do textbook visualizations and curriculum instruction
influence students’ drawings?

7. Content area
• ASTRONOMY
• astronomy education and popularization are historically based on visualizations
• planetariums and science centers regularly offer to visitors realistic simulations of space travels
and explorations based on computer elaboration of photographs
• multimedia visual representations support students in conceptualizing and representing complex
phenomena that cannot be experienced by first-hand
• Images are fundamental tools for astronomers and professional researchers in astrophysics
• Few studies focused on students’ difficulty with the interpretation of diagrams and iconic
representations in astronomy (Kikas, 1998; Barab et al., 2000; Hansen et al., 2004)
• Astronomical representations require formal knowledge and are not necessarily linked to real
life experience (Vosniadou, 2010)

8. Content area

9. Methods – Analysis of students’ drawings
• Typical textbook images about seasonal changes feature iconic elements difficult to
intepret
• may lead students to interpret incorrectly the mechanism underlying the represented phenomenon
• Pre-existing scoring schemes focused on
• accuracy and fidelity
• number or types of iconic elements that are present in the drawings
cannot be used to analyze the students’ drawings
• highly prescriptive notion of what should be considered as a “successful drawing”
• better fit to pre-determined “expert” instances
• do not allow to cluster emerging students’ models according to visual features

10. Methods – Exploratory Factor Analysis (EFA)
Applying PCA to drawings:
 Identify initial variables  grounded list of all the iconic elements in the drawings
 Reduce initial variables  redundant or “constant” iconic elements were eliminated
 Score of the variables  1 or 0 according to the presence or absence of each element in the
drawing
Two main approaches: Principal Component Analysis (PCA) and Factor Analysis (FA)
Main aim of FA uncover structures underlying the set of original variables
Main aim of PCA to reduce the dimensionality of a data set consisting of a large number of interrelated
variables, while retaining as much as possible of the variation present in the data set

11. Methods – Principal Component Analysis (PCA)
Example of initial identified variables - Seasonal changes
Label Element Description
S1 Orbit Presence of the Earth’s orbit
S2 Elliptical orbit Presence of Earth’s orbit represented as elliptical
S3 Earth Presence of the Earth
S4 One Earth Presence of only one Earth
S5 Multiple Earth Presence of more than one Earth
S6 Multiple Earth and orbit Presence of the Earth’s orbit and of more Earths along this orbit
S7 Multiple Earth and no orbit Presence of more Earths but of no orbit
S8 Sun Presence of the Sun
S9 Sun and orbit Presence of the Sun and the Earth’s orbit
S10 Sun and no orbit Presence of the Sun but of no Earth’s orbit
S11 Rays Presence of some kind of rays
S12 Rays hitting the Earth Presence of some kind of rays hitting the Earth’s surface. It should be possible that the source of the rays is not represented
S13 Converging rays Presence of rays drawn as convergent. It is not important who is the source of the rays
S14 Converging and inclined rays Presence of convergent rays and of their inclination with respect to the hit surface. It is not important who is the source of the rays
S15 Distance Presence of elements (segments, arrows ec.) representing the distance between the Earth and the Sun
S16 Axis Presence of the Earth’s axis
S17 Moon Presence of the Moon
S18 Weather Presence of elements related to clime and weather (rain, cloud, snow etc.)
S19 N/S Presence of cardinal points
S20 Angle Presence of the angle between the Earth's axis and its orbit or between solar rays and the Earth’s surface.
S21 INT Presence of two or more conceptually related images
S22 VER Presence of verbal elements to be read as an important part of the image, such as captions
S23 SEL Presence of elements selected or conceptually highlighted in relation to textual/graphical features, which do or do not make them salient
S24 SYM Presence of elements that require appropriate readings of symbols

12. Methods – Principal Component Analysis (PCA)
Example of selection of variables - Seasonal changes
Label Element Description
S1 Orbit Presence of the Earth’s orbit
S2 Elliptical orbit Presence of Earth’s orbit represented as elliptical
S3 Earth Presence of the Earth
S4 One Earth Presence of only one Earth
S5 Multiple Earth Presence of more than one Earth
S6 Multiple Earth and orbit Presence of the Earth’s orbit and of more Earths along this orbit
S7 Multiple Earth and no orbit Presence of more Earths but of no orbit
S8 Sun Presence of the Sun
S9 Sun and orbit Presence of the Sun and the Earth’s orbit
S10 Sun and no orbit Presence of the Sun but of no Earth’s orbit
S11 Rays Presence of some kind of rays
S12 Rays hitting the Earth Presence of some kind of rays hitting the Earth’s surface. It is possible that the source of the rays is not represented
S13 Converging rays Presence of rays drawn as convergent. It is not important which or where is the source of the rays
S14 Converging and inclined rays Presence of convergent rays and of their inclination with respect to the hit surface. It is not important which or where is the source of the rays
S15 Distance Presence of elements (segments, arrows ec.) representing the distance between the Earth and the Sun
S16 Axis Presence of the Earth’s axis
S17 Moon Presence of the Moon
S18 Weather Presence of elements related to climate and weather (rain, cloud, snow etc.)
S19 N/S Presence of cardinal points
S20 Angle Presence of the angle between the Earth's axis and its orbit or between solar rays and the Earth’s surface.
S21 INT Presence of two or more conceptually related images
S22 VER Presence of verbal elements to be read as an important part of the image, such as captions
S23 SEL Presence of elements selected or conceptually highlighted in relation to textual/graphical features, which do or do not make them salient
S24 SYM Presence of elements that require appropriate readings of symbols, and that contain examples of synonymy, homonymy and/or polysemy of
symbols

13. PCAVS. FACTOR ANALYSIS OF DRAWING
 MODELS of
SEASONAL
CHANGES
Presence of the Earth’s orbit
Presence of the Earth
Presence of more than one Earth
Presence of the Sun
Presence of rays
Presence of Rays hitting the Earth’s surface
Presence of Convergent Rays hitting the Earth’s surface
Presence of Convergent and Inclined Rays hitting the Earth’s surface
Presence of Convergent Rays hitting the Earth’s surface
Presence of elements representing the distance between the Earth and the Sun
Presence of the Earth’s axis
Presence of elements related to clime and weather
Presence of cardinal points
Presence of an angle between the Earth's axis and its orbit
Models are not
known in
advance
We «measure»
the presence of
iconic elements
Variability in the
iconic elements
causes the
variance of the
models

14. PCAVS. FACTOR ANALYSIS OF DRAWING
 MODELS of
SEASONAL
CHANGES
Presence of the Earth’s orbit
Presence of the Earth
Presence of more than one Earth
Presence of the Sun
Presence of rays
Presence of Rays hitting the Earth’s surface
Presence of Convergent Rays hitting the Earth’s surface
Presence of Convergent and Inclined Rays hitting the Earth’s surface
Presence of Convergent Rays hitting the Earth’s surface
Presence of elements representing the distance between the Earth and the Sun
Presence of the Earth’s axis
Presence of elements related to clime and weather
Presence of cardinal points
Presence of an angle between the Earth's axis and its orbit
Models are
considered as a
latent variable
We «measure»
the presence of
iconic elements
Variability in the
iconic elements
is caused by
existence of
different models

15. JUSTIFICATION FORTHE USE OF PCA OF DRAWINGS
 Models emerge from the “data” (the iconic elements) and can be interpreted
ex-post
 Possibility of introducing models from factor scores and not only from
eigenvalue-based criteria
 While similar, PCA is not a technique to perform FA, which assumes the
existence of a “latent trait”, in our case, a “latent” model
 While models precede drawings, they cannot be limited a-priori to those
foreseen by literature  emerging models could be not “stable” enough
 As such, mental models cannot be considered technically as a “latent trait”

16. METHODS - RELIABILITY
 Using the models obtained from PCA, a researcher again classified the
students’ drawings
 Inter-rater reliability between the categorization based on factors scores
and the classification of the researcher

17. METHODS - RELATIONSHIPS BETWEEN DRAWINGS & KNOWLEDGE
 Written task in two modalities: with/without a side textbook picture
 Written questionnaire with 6 T/F and 2 multiple choice questions

18. METHODS – RUBRICS
WRITTEN TASK ABOUT SEASONAL CHANGES
0 points: Unclear answer;
1 point: incorrect answer (e.g., distance
misconception)
2 point: partial answer (e.g., reasoning is
incomplete)
3 point: correct answer
T/F and MULTIPLE CHOICE QUESTIONS
0,5 points: for each correct T/F
1 point: partial answer choice in the M/C
2 point: correct answer choice in the M/C
Max score: 7  0-1 incorrect knowledge (0);
2-3 naïve (1); 4-5 partial (2); 6-7 correct (3)

19. METHODS - RELATIONSHIPS BETWEEN DRAWINGS AND (I) TEXTBOOK
VISUALIZATIONS AND (II) CURRICULUM INSTRUCTION
 Textbook image shown/not shown before drawing
 Curriculum vs. no formal instruction about astronomy

20. SAMPLE
 736 students
 339 6th - 8th grade (no curricular instruction about astronomy)
 151 9th – 10th grade (curricular instruction about astronomy)
 108 9th – 10th grade (textbook image shown before drawing)
 138 9th – 10th grade (drawing first)
 6th - 8th grade  middle school (compulsory)
 9th – 10th grade  first two years high school (compulsory)
 Average age: 13.0±0.1
 Contents taught in curriculum instruction (6-10 hs): Earth motions around the Sun,
solar system

21. RESULTS

22. RQ1. To what extent drawings may reliably uncover students’
mental models about a given phenomena?
 PCA of drawings is effective in identifying students’ mental models
from iconic elements
 Few «more relevant» iconic elements explain about 60% of the variance
in the original drawings
Sufficient agreement (60%) between factor scores and researcher’s
rating

23. RQ2. What is the relationship between drawings and knowledge
about the target phenomenon?
Students who produced naïve and distance-based drawings were more
likely to give unclear explanations and choose incorrect answers to
aggregate T/F and MC questions
Students who produced a distance-based drawing were more likely to
choose a distance-based answer choice and to give incorrect
explanations
Naïve drawings correspond incorrect accounts about sun emission of
energy

24. RQ2. What is the relationship between drawings and knowledge
about the target phenomenon?
Students who produced an inclination-based drawing were more likely
to
 choose correct answer choices
 give correct explanations
Inclined-based drawings more likely correspond to correct notions
about the Earth’s axis

25. RQ3. To what extent do textbook visualizations and curriculum
instruction influence students’ drawings?
Textbook images lead to more “detailed” student generated drawings
• Inclination-based drawing less frequent in “textbook images” group
More correct drawings were generated by students exposed to
curriculum teaching about astronomy
• Distance-based drawing more common in “no instruction” group

26. SUPPORTING EVIDENCE

27. PCA OF STUDENTS’ DRAWINGS - STATISTICS
KMO (Keiser Meyer Olkin) 0,759
Bartlett's test of
sphericity
χ2 2310,023
df 120
p-value 0,000
Total variance explained: 57.3%

28. PCA OF STUDENTS’ DRAWINGS – FACTOR LOADINGS
Principal Component
1 2 3 4
Presence of the Earth’s orbit ,467
Presence of the Earth ,788
Presence of more than one Earth ,643
Presence of the Sun ,834
Presence of some kind of rays -,629
Presence of some kind of rays hitting the Earth’s surface ,837
Presence of rays drawn as convergent. ,916
Presence of convergent rays and of their inclination with
respect to the hit surface.
,804
Presence of elements representing the distance
between the Earth and the Sun
,454 ,353
Presence of the Earth’s axis ,778
Presence of elements related to climate and weather -,855
Presence of cardinal points ,588
Presence of the angle between the Earth's axis and its
orbit
,748
Presence of two or more related images -,701
Presence of verbal elements ,815
Presence of elements that require appropriate readings
of symbols
,343

29. PCA OF STUDENTS’ DRAWINGS - MODELS
Principal Component
1
Presence of the Earth ,788
Presence of the Sun ,834
Presence of elements that require appropriate readings
of symbols
,343
Distance 1

30. PCA OF STUDENTS’ DRAWINGS - MODELS
Principal Component
1 2 3 4
Presence of some kind of rays hitting the Earth’s surface ,837
Presence of rays drawn as convergent. ,916
Presence of convergent rays and of their inclination with
respect to the hit surface.
,804
Rays

31. PCA OF STUDENTS’ DRAWINGS - MODELS
Principal Component
1 2 3 4
Presence of the Earth’s orbit ,467
Presence of more than one Earth ,643
Presence of elements representing the distance
between the Earth and the Sun
,454
Presence of verbal elements ,815
Distance 1I

32. PCA OF STUDENTS’ DRAWINGS - MODELS
Principal Component
1 2 3 4
Presence of the Earth’s axis ,778
Presence of cardinal points ,588
Presence of the angle between the Earth's axis and its
orbit
,748
Tilt

33. PCA OF STUDENTS’ DRAWINGS - MODELS FROM FACTOR SCORES
Weather

34. PCA OF STUDENTS’ DRAWINGS - MODELS FROM FACTOR SCORES
Orbit & Tilt

35. DISTRIBUTION OF MODELS (N = 494)
11.9%
55.1%
25.5% 7.5%
Factor score - researcher
Cohen’s kappa (469) = 0.58

36. WRITTENTASK RESULTS
0 5 10 15 20 25 30 35 40
Unclear
Incorrect
Partial
Correct
% of students
With textbook image (N = 438) No image (N= 485)
N = 369, Cohen’s kappa = 0.51

37. T/F AND MC QUESTIONS
0 10 20 30 40 50 60
Incorrect
Naive
Partial
Correct
% of students
N= 381

38. CORRELATION BETWEEN MODELS AND OPEN ANSWERS (N = 365)
No image
Unclear 82 41 30 22
Incorrect 18 40 9 25
Partial 0 18 56 48
Correct 0 1 5 5
100 100 100 100
χ2 = 63,725; df = 9; p<10-4

39. CORRELATION BETWEEN MODELS AND OPEN ANSWERS (N = 327)
With image
Unclear 36 30 21 14
Incorrect 50 47 7 36
Partial 14 22 65 42
Correct 0 1 7 8
100 100 100 100
χ2 = 56,671; df = 9; p<10-4

40. CORRELATION BETWEEN MODELS AND AGGREGATET/F & MC QUESTIONS
(N = 219)
Unclear 0 1 0 0
Incorrect 24 19 13 0
Partial 48 66 37 47
Correct 28 13 50 53
100 100 100 100
χ2 = 33,263; df = 9; p<10-4

41. CORRELATION BETWEEN MODELS AND MC QUESTIONS
(N = 438)
The change of seasons is
mainly due to:
a) Earth-Sun changing distance 35 49 8 20
b) The Earth rotating around itself 15 15 15 17
c) The changing duration of the day 12 5 6 0
d)The changing inclination of
the sunrays hitting the Earth*
38 31 71 63
χ2 = 56,228; df = 9; p<10-4

42. CORRELATION BETWEEN MODELS AND MC QUESTIONS
(N = 455)
Why in summer it is hotter
than in winter
a) Because the Sun emits more
energy
27 9 4 8
b) Because how the Earth is hit
by sunlight changes*
44 34 83 56
c) Because we are farther from the
Sun and the Earth slows down
3 8 2 6
d) Because the Earth is closer to
the Sun
26 49 11 29

43. CORRELATION BETWEEN MODELS ANDT/F QUESTIONS
(N = 401)
During Summer the Sun emits
more energy
False 79 48 36 49
True 21 52 64 51
χ2 = 9,198; df = 3; p = 0.027

44. CORRELATION BETWEEN MODELS ANDT/F QUESTIONS
(N = 394)
Energy absorbed by a surface is
maximum when light hits
normally the surface
False 50 36 20 38
True 50 64 80 62
χ2 = 6,580; df = 3; p = 0.087

45. CORRELATION BETWEEN MODELS ANDT/F QUESTIONS
(N = 396)
Sunrays incidence on Earth
changes during the year
False 22 23 27 22
True 78 77 73 78
χ2 = 0,350; df = 3; p = 0.950

46. CORRELATION BETWEEN MODELS ANDT/F QUESTIONS
(N = 395)
Earth surface asorbs energy
from the Sun
False 33 24 18 27
True 67 76 82 73
χ2 = 2,070; df = 3; p = 0.558

47. CORRELATION BETWEEN MODELS ANDT/F QUESTIONS
(N = 387)
The Earth axis is inclined w.r.t.
the orbit’s plane
False 25 26 2 18
True 75 74 98 82
χ2 = 13,684; df = 3; p = 0.003

48. CORRELATION BETWEEN MODELS ANDT/F QUESTIONS
(N = 380)
The Earth axis remains parallel
to itself during the year
False 53 52 56 51
True 47 48 44 49
χ2 = 0,204; df = 3; p = 0.977

49. CORRELATION BETWEEN MODELS ANDTEXTBOOK IMAGES
(N = 194)
GROUP
Textbook image before drawing 0 48,6 13,0 82,6
Drawing first 100 51,4 87,0 17,4
χ2 = 27.825; df = 3; p < 10-4

50. CORRELATION BETWEEN MODELS AND INSTRUCTION
(N = 276)
GROUP
No instruction 100 60,1 69,7 44,0
Instruction 0 39,9 30,3 56,0
χ2 = 22.952; df = 3; p < 10-4

51. DISCUSSION AND IMPLICATIONS

52. Discussion
Drawing were good predictors for explanations in open questions
Drawings were good predictors for answers to MC questions
Drawings were good predictors for answers to some T/F questions

53. Discussion
Using Paivio’s theory of representations:
 PCA can help clarify the relationships between students’ drawings and their
conceptions of the represented phenomenon
 Trough PCA, the internal representation of the concept can be “reconstructed”
from the referential links used

54. Implications
Use of PCA helps investigate
which iconic elements students are more often used in visual representations
(Van Meter and Garner, 2005)
how such models correlate to explanation categories of increasing complexity
the influence of conventional representations used in instructions (Ehrlén,
2008)

55. Implications
Main advantages of PCA with respect to typical indexing schemes and self-
made rubrics (Stieff, 2011; Lundin & Jakobson, 2014; Bowker, 2007)
to identify common patterns from the data
to focus on the conception expressed in the drawing rather than on superfluous
or difficult to represent symbols

56. CONCLUSIONS
 Findings suggest that drawings are effective tools to elicit mental models
and to negotiate evidence-based accounts of familiar astronomical
phenomena
 PCA could be extended to drawing analysis in other areas
 However, more research is needed to investigate relationships wirth:
written explanations and questionnaire
curricular instruction

57. THANKS FORYOUR ATTENTION
QUESTIONS?
ITALO.TESTA@UNINA.IT