2. Background
There are many factors that affect the
comprehension of a text:
Traditional methods include the number of
syllables, words, and sentences in a text.
The concept of a reader’s prior knowledge also plays a
role.
Stahl, Hare, Sinatra, & Gregory (1991) showed that previous
knowledge can affect the extent that a reader could make
inferences and focus on the important information in a text.
A text with more inferences and active processing needed to
comprehend it would have low-cohesion, and can be
measured using Coh-Metrix. Although reducing the number
of inferences needed in a text would make it easier for a
reader with less background
knowledge, McNamara, Kintsch, Songer, & Kintsch (1996)
pointed out that the resulting knowledge is only on a
3. Background
Other factors include:
The idea density of a text is the number of ideas
divided by the number of words in a text that can
be measured using the Computerized
Propositional Idea Density Rater (CPIDR).
Kintsch & Keenan (1973) define propositions, or
ideas, as units in processing a text. Therefore, texts
with a higher number of propositions would generally
be written for experts because of the higher level of
processing needed (Covington, 2009).
4. Purpose
The purpose of this project was to focus on how
the prior knowledge of a reader and the idea
density of a text interact in comprehension.
To test this, two articles were chosen with varying
difficulty (low/high idea density). Participants
were split up according to their background
knowledge (non-expert/expert in subject area) and
assigned either article to read. Afterwards, the
percent of questions right on the corresponding
quiz was used to gauge comprehension and the
results between groups compared.
5. Hypothesis
If participants with varying levels of knowledge
in a subject are tested after reading a specific
text in that area, then the participants’
background knowledge will be a better
predictor of their comprehension than the idea
density of the text.
For example,
Idea density would be a better predictor if
comprehension scores varied between the two
articles, regardless of participant prior knowledge.
Prior knowledge would be a better predictor if
comprehension scores varied within non-experts
but not experts between the two articles.
6. Experimental Design: Articles
Articles’ subject area: the role of nitrogen-fixing bacteria in the
nitrogen cycle
2 informational texts were found from the Internet and chosen
based on their intended audience levels and approximate length:
“The Nitrogen Cycle”- easy article written for non-
experts/children
(http://www.realtrees4kids.org/sixeight/cycles.htm#nitrogen)
“Biological Nitrogen Fixation”- difficult article written for
experts/adult
(http://www.nature.com/scitable/knowledge/library/biological-nitrogen-fixation-
23570419)
8 multiple-choice questions were written for each article (4
questions based directly from the text and 4 inference questions)
in order to measure a reader’s comprehension after reading the
article.
7. Article Statistics
After analyzing the two articles using CPIDR, Coh-Metrix, and
an online word counter, the article written for experts/adults
was measured to be actually more difficult than the article
written for non-experts/kids.
For example, the difficult article has a higher idea density
(which requires more processing by the reader) and a lower
cohesion (which means there are more inferences needed to
be made by the reader).
Article
Type
Name Audience Idea Density:
# of ideas/
# of words
Referential
Cohesion:
argument
overlap,
adjacent
sentences,
mean
LSA Overlap:
adjacent
sentences,
mean
Word
Frequency: #
of primary
words/
total # of
words
Easy "The Nitrogen
Cycle"
kid/non-expert 0.466 0.706 0.36 0.534
Difficult "Biological
Nitrogen
Fixation"
adult/expert 0.531 0.619 0.347 0.616
8. Experimental Design:
Participants
Population: college students, ages between 18-40
I found participants by emailing professors and
contacting possible interested students
beforehand. I also asked students at different
locations (such as at the student center) on the
days that I collected data.
Participants were asked to self-rate their
background knowledge on a scale of 1-10 with 1-
no knowledge to 10-high knowledge.
They were then assigned either the easy/difficult
article and had an unlimited amount of time to
read the passage and answer the questions
without referring back to the article
9. Sample Self-Rating
Questionnaire
People have a range of knowledge about the cycles in nature. The following
passage is going to be about the nitrogen cycle and the role of nitrogen-fixing
bacteria. On a scale of 1-10, how much do you think you know about this
topic? A rating of 1 means that you feel that you know little or nothing about
the role of nitrogen-fixing bacteria in this cycle and a rating of 10 means that
you have a good understanding of this topic.
1 2 3 4 5 6 7 8 9 10
What is your gender?
male _________ female_________
Are you between 18 to 40 years old?
yes_________ no_________
12. Data Analysis: Population
Sample
62 participants total
23 males, 39 females (although background
knowledge, rather than gender, should play a bigger
role)
52 participants rated as non-expert (self-rating: 1-5)
10 participants as expert (self-rating 6-10)
The two articles were assigned by
counterbalancing so that half of non-experts and
experts read either article.
The data was split into four groups:
non-expert/easy (non-experts who read the easy
article)
non-expert/difficult
expert/easy
13. Average Comprehension Scores in Each
Group
Comprehension scores were calculated using
number of questions correct divided by total
number of questions.
*Note: After I started collecting data, one inference
question for each article was found to be faulty
and omitted from the final results.
Non-
Experts/
Easy
Article
Non-
Experts/
Difficult
Article
Experts/
Easy
Article
Experts/
Difficult
Article
mean 70.33% 73.63% 88.57% 88.57%
standard
deviation
23.8772% 19.2512% 11.9523% 11.9523%
14. Histogram of Participants’
Results
Overall, exp
erts
generally
had higher
comprehensi
on scores for
both
articles, alth
ough there
was also a
much
smaller
sample size
of experts.
0
1
2
3
4
5
6
7
8
9
10
28.57 42.86 57.14 71.43 85.71 100
Comprehension Score (% Correct)
Frequency of Comprehension Scores for
Non-Experts/Experts
Non-Expert/Easy
Non-
Expert/Difficult
Expert/Easy
15. Variation in Mean Comprehension
Scores
There was
also an
unexpected
increase in
the mean of
comprehensi
on scores
from the easy
article to the
difficult article
for non-
experts, whic
h did not
occur in the
experts’
mean
comprehensi
on scores.
70.33
73.62
88.57 88.57
60
65
70
75
80
85
90
MeanPercentageCorrecton
CorrespondingQuiz
Article Difficulty
Variation in Mean Comprehension Scores of
Non-Expert and Expert for Easy/Difficult Article
Non-Expert
Expert
Easy Article Difficult Article
16. Breakdown of Question Type in
Mean Comprehension Scores
Mean Comprehension Scores of
Non-expert/Expert and Easy
Article
Mean Comprehension Scores of
Non-Expert/Expert and Difficult
Article
0
10
20
30
40
50
60
70
80
90
100
Direct Inference
MeanQuestionsCorrect(%)
Question Type
Non-
Expert/Easy
0
10
20
30
40
50
60
70
80
90
100
Direct Inference
MeanQuestionsCorrect(%)
Question Type
Non-Expert/Difficult
Expert/Difficult
17. Results
Based on a 2-way ANOVA between
groups, the P-value for the interaction between
non-experts and experts for the easy/difficult
articles was 0.8238, which is not low enough
to be statistically significant (p<0.05).
18. Conclusion
The easy/difficult articles used in this study were
written for different audience levels and also had
corresponding idea densities (low/high).
The experts’ comprehension scores had a much
smaller range, higher mean, and lower standard
deviation than the non-experts’ comprehension scores
across both articles. In addition, while the non-
experts score varied with the idea density of the
article, the experts score stayed consistent across the
easy/difficult article.
In general, the participants’ prior knowledge tended to be a
better predictor of comprehension.
However, the sample size for experts was also much
smaller than the non-experts sample size and the
sample did not have an equal number of
males/females. Finally, the results for both experts
19. Application
While my results were not significant, reading
comprehension is an important component in
many fields, including
education, science, business, and the
government. Understanding of
comprehension will facilitate more efficient
communication so others can learn and benefit
from what they read.
20. Future Studies
In the future, I would like to improve my study by getting a larger sample
size with a more equal number of non-experts/experts and
males/females. Moreover, it would be better if all of the participants
were able to read the article in a quiet, focused room to control for
outside variables.
Since some of my participants had been recruited on the day of the
experiment, they may not have taken the article/questions as
seriously, especially the easy article which did not seem very challenging.
This may account for the surprising increase in score from the easy to the
difficult article for non-experts.
I initially thought that the nitrogen cycle would be a basic topic that
participants would have a wide range of knowledge. Instead, I would
probably choose a more general topic next time, and use previous class
experience or interest rather than a self-rating, to measure prior
knowledge.
Finally, the texts used could also be formatted to share more of the same
content but still be written for different audiences. That way, the same
set of questions could be used for both articles to control for any other
variables between texts or additional question sets.
21. Bibliography
Benjamin, R. G. (2012). Reconstructing readability: Recent developments and recommendations in the analysis of text difficulty.
Educational Psychology Review, 24(1), 63-88. doi:10.1007/s10648-011-9181-8
Brown, C., Snodgrass, T., Kemper, S. J., Herman, R., & Covington, M. A. (2008). Automatic measurement of propositional idea
density from part- of-speech tagging. Behavioral Research Methods, 40(2), 540-545. doi:10.3758/BRM.40.2.540
Covington, M. A. (2009). Idea density- A potentially informative characteristic of retrieved documents. Proceedings of the IEEE
SoutheastCon 2009, 201-205. doi:10.1109/SECON.2009.5174076
Cromley, J. G., & Azevedo, R. (2007). Testing and refining the direct and inferential mediation model of reading comprehension.
Journal of Educational Psychology, 99(2), 311-325. doi:10.1037/0022-0663.99.2.311
Kalyuga, S., & Renkl, A. (2010). Expertise reversal effect and its instructional implications: Introduction to the special issue.
Instructional Science, 38(3), 209-215. doi:10.1007/s11251-009-9102-0
Kintsch, W., & Keenan, J. (1973). Reading rate and retention as a function of the number of propositions in the base structure of
sentences. Cognitive Psychology, 5(3), 257-274. doi:10.1016/0010-0285(73)90036-4
McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better? Interactions of text
coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14(1), 1-43.
doi:10.1207/s1532690xci1401_1
McNamara, D. S., Louwerse, M. M., McCarthy, P. M., & Graesser, A. C. (2010). Coh-Metrix: Capturing linguistic features of
cohesion. Discourse Processes, 47(4), 292-330. doi:10.1080/01638530902959943
Pitler, E. & Nenkova, A. (2008). Revisiting readability: A unified framework for predicting text quality. Proceedings of the 2008
Conference on Empirical Methods in Natural Language Processing, 186-195. Retrieved from
http://www.aclweb.org/anthology//D/D08/D08-1020.pdf
Schwanenflugel, P. J., Kuhn, M. R., & Ash, G. (2010). Sharing the stage: Using oral and silent wide reading to develop proficient
reading in the early grades. In E. H. Hiebert, & D. Reutzel (Eds.). Revisiting Silent Reading (pp. 181-197).
Newark, DE: International Reading Association.
