Selecting A topic/Job Position For the first part of the project, you will select a topic position for either a Release of Information Associate For the position you have selected, address the following in 2 paragraphs · Brief description of the position (2 -3 sentences) · Why did you select the position? · What are the 3 major duties of the position that are most essential to the role? Article Corresponding author: John Heilmann, Department of Communication Sciences and Disorders, Mail Stop #668, East Carolina University, Greenville, NC 27858, USA E-mail: [email protected] Language Testing 27(4) 603–626 © The Author(s) 2010 Reprints and permission: sagepub.co.uk/journalsPermission.nav DOI: 10.1177/0265532209355669 http://ltj.sagepub.com Sensitivity of narrative organization measures using narrative retells produced by young school-age children John Heilmann East Carolina University, USA Jon F. Miller University of Wisconsin – Madison, USA Ann Nockerts University of Wisconsin – Madison, USA Abstract Analysis of children’s productions of oral narratives provides a rich description of children’s oral language skills. However, measures of narrative organization can be directly affected by both developmental and task-based performance constraints which can make a measure insensitive and inappropriate for a particular population and/or sampling method. This study critically reviewed four methods of evaluating children’s narrative organization skills and revealed that the Narrative Scoring Scheme (NSS) was the most developmentally sensitive measure for a group of 129 5–7-year-old children who completed a narrative retell. Upon comparing the methods of assessing narrative organization skills, the NSS was unique in its incorporation of higher-level narrative features and its scoring rules, which required examiners to make subjective judgments across seven aspects of the narrative process. The discussion surrounded issues of measuring children’s narrative organization skills and, more broadly, issues surrounding sensitivity of criterion referenced assessment measures. Keywords assessment, ceiling effects, narrative, language development, oral language, psychometrics 604 Language Testing 27(4) Introduction Oral narrative skills in children Assessment of children’s oral narratives is of significant interest to researchers and practitioners, as being a proficient narrator is an important skill in the life of young children. Oral narrative skills are a key component of most school curricula. In the USA, each state is required to develop guidelines describing skills that need to be incor- porated into the general curriculum. While oral language skills are often not formally assessed in the general classroom, they are a key component in each state’s guidelines. For example, the state of North Carolina requires teachers to facilitate mastery of nar- rative comprehension as well as effective ability to produce narr.

1. Selecting A topic/Job Position
For the first part of the project, you will select a topic position
for either a Release of Information Associate
For the position you have selected, address the following in 2
paragraphs
· Brief description of the position (2 -3 sentences)
· Why did you select the position?
· What are the 3 major duties of the position that are most
essential to the role?
Article
Corresponding author:
John Heilmann, Department of Communication Sciences and
Disorders, Mail Stop #668, East Carolina
University, Greenville, NC 27858, USA
E-mail: [email protected]
Language Testing
27(4) 603–626
© The Author(s) 2010
Reprints and permission:
sagepub.co.uk/journalsPermission.nav
DOI: 10.1177/0265532209355669
http://ltj.sagepub.com
Sensitivity of narrative
organization measures using
narrative retells produced by

2. young school-age children
John Heilmann
East Carolina University, USA
Jon F. Miller
University of Wisconsin – Madison, USA
Ann Nockerts
University of Wisconsin – Madison, USA
Abstract
Analysis of children’s productions of oral narratives provides a
rich description of children’s oral
language skills. However, measures of narrative organization
can be directly affected by both
developmental and task-based performance constraints which
can make a measure insensitive
and inappropriate for a particular population and/or sampling
method. This study critically
reviewed four methods of evaluating children’s narrative
organization skills and revealed that the
Narrative Scoring Scheme (NSS) was the most developmentally
sensitive measure for a group
of 129 5–7-year-old children who completed a narrative retell.
Upon comparing the methods
of assessing narrative organization skills, the NSS was unique
in its incorporation of higher-level
narrative features and its scoring rules, which required
examiners to make subjective judgments
across seven aspects of the narrative process. The discussion
surrounded issues of measuring
children’s narrative organization skills and, more broadly,
issues surrounding sensitivity of criterion
referenced assessment measures.

3. Keywords
assessment, ceiling effects, narrative, language development,
oral language, psychometrics
604 Language Testing 27(4)
Introduction
Oral narrative skills in children
Assessment of children’s oral narratives is of significant
interest to researchers and
practitioners, as being a proficient narrator is an important skill
in the life of young
children. Oral narrative skills are a key component of most
school curricula. In the
USA, each state is required to develop guidelines describing
skills that need to be incor-
porated into the general curriculum. While oral language skills
are often not formally
assessed in the general classroom, they are a key component in
each state’s guidelines.
For example, the state of North Carolina requires teachers to
facilitate mastery of nar-
rative comprehension as well as effective ability to produce
narratives and the complex
language associated with a literate speaking style (North
Carolina Department of Public
Instruction, 2004). Oral narrative skills have been incorporated
into children’s curricula
for good reasons. Being a proficient narrator is a skill needed to
express one’s intentions
and effectively participate in classroom activities. In addition,
several decades of
research have documented the strong link between children’s

4. oral narrative skills and
broader curricular requirements.
Research on monolingual English-speaking children has
documented that chil-
dren’s oral narrative skills are predictive of later reading
outcomes (Bishop &
Edmundson, 1987; Dickinson & McCabe, 2001; Griffin,
Hemphill, Camp, & Wolf,
2004; Hemphill & Snow, 1996; Roth, Speece, Cooper, & de la
Paz, 1996; Snow, 1983;
Snow, Dickinson, Jennings, & Purves, 1991; Tabors, Snow, &
Dickinson, 2001).
Additional studies have documented that children’s early
narrative competence is
related to broader academic outcomes (e.g., Fazio, Naremore, &
Connell, 1996;
O’Neill, Pearce, & Pick, 2004). Fazio and colleagues found that
oral narrative skills
were one of the strongest predictors for whether or not a child
later required academic
remediation, while O’Neill et al. identified a strong relationship
between young
children’s oral narrative skills and later mathematical ability.
While these studies
documenting the relationship between oral narratives and
broader reading and aca-
demic outcomes are correlational and a causal relationship
cannot be assumed, there
is a general consensus in the field that oral narrative skills may
play a key role in
developing the foundation for higher level academic tasks.
It is only natural that oral narrative skills are of interest to
those working with
children who have language impairments (LI). Approximately

5. 7% of monolingual
English-speaking children experience significant deficits in
their oral language skills
despite normal cognitive skills (Tomblin, Records, Buckwalter,
Zhang, Smith, &
O’Brien, 1997). Children with LI have substantial difficulty
producing fully coherent
oral narratives (Boudreau & Hedberg, 1999; Fey, Catts, Proctor-
Williams, Tomblin, &
Zhang, 2004; Gillam & Johnston, 1992; Newman & McGregor,
2006; Pearce, McCormack,
& James, 2003; Reilly, Losh, Bellugi, & Wulfeck, 2004).
Assessment of children’s
narratives can be an effective method of identifying the
presence of LI (Allen, Kertoy,
Sherblom, & Petit, 1994; Paul & Smith, 1993) and provides a
functional description of
children’s performance, which may assist in the development of
treatment goals (Miller,
Gillam, & Peña, 2001).
Heilmann et al. 605
Oral narratives in bilingual children
Clinicians, teachers, and researchers are also interested in better
understanding the oral
narrative skills of children learning a second language. Just as
with monolingual chil-
dren, there is a significant predictive relationship between oral
narrative skills and read-
ing outcomes in children learning a second language (August &
Shanahan, 2006; Miller,
Heilmann, Nockerts, Iglesias, Fabiano, & Francis, 2006; Oller
& Pearson, 2002). Miller

6. et al. identified that measures from oral narratives predicted
reading scores within and
across languages in young school-age English language learners
(ELL). Given the strong
relationship between oral narratives and reading, it is important
to ensure that young
ELLs have sufficient oral narrative skills. Because a
disproportionately high number of
ELLs have poor reading outcomes when compared to their
monolingual counterparts
(August & Hakuta, 1997), there is a need for a better
understanding of the relationship
between oral narrative skills and reading outcomes.
Numerous speech and language scholars have recommended
using oral narratives in
clinical assessments of bilingual children, given the naturalness
of the task (e.g., Fiestas
& Peña, 2004; Munoz, Gillam, Peña, & Gulley-Faehnle, 2003;
Uccelli & Páez, 2007).
The act of telling stories is universal across cultures and is an
important instrument for
transmitting information (Mandler, Scribner,Cole, & DeForest,
1980); the format of tell-
ing a story may be more familiar to children from different
cultures than formal language
testing. Analysis of oral narratives across multiple languages
can provide an estimate of
relative proficiency in each language spoken and may assist in
differentiating language
differences, where a children may have limited proficiency in
one of their languages
spoken, from global language disorders, where children
demonstrate marked deficits in
all languages spoken (Gutiérrez-Clellen, 2002; Rojas & Iglesias,
2009).

7. Criterion referenced assessment
Oral narrative analysis falls under the broader umbrella of
criterion referenced (CR)
assessment. The goal of CR assessment is to generate a detailed
description of a child’s
performance in a target domain (e.g., narrative organization)
and to identify if the child’s
performance meets developmental expectations (Linn &
Gronlund, 2000). While the
purpose of standardized tests is to maximize differences
between individuals, CR assess-
ments afford clinicians the opportunity to obtain more detailed
data on specific skill
areas (Bachman, 2000). While standardized language tests
commonly evaluate limited
sets of skills that have been stripped of their communicative
context (Marquardt &
Gillam, 1999), most CR oral language assessments, including
narrative language sample
analysis, examine real-life communication situations and do not
rely on assessment of
decontextualized language skills.
Because CR assessments can generate rich descriptive data, they
are particularly use-
ful when describing the profile of a child exhibiting language
learning difficulties. While
children with LI share the same underlying characteristic (i.e.,
substantial difficulty with
language), there is considerable heterogeneity in the patterns of
difficulties experienced
by these children (Rapin & Allen, 1983; Tomblin, Zhang,
Weiss, Catts, & Ellis Weismer,

8. 606 Language Testing 27(4)
2004). Most standardized language tests provide subtest scores
that would appear to be
useful for describing children’s profiles (e.g., receptive
vocabulary; expressive syntax).
However, the measurement properties of subtests prohibit use
for providing accurate
descriptions of performance patterns; subtest scores are
generated from a relatively small
number of items, making them unreliable for interpretation on
their own (McCauley &
Swisher, 1984). In order to fully identify a child’s pattern of
language performance,
evaluations should use detailed CR assessments to identify
children’s strength and weak-
ness. For example, comprehensive assessment of children’s oral
narrative skills may
reveal relative strengths and weaknesses in using appropriately
complex vocabulary and
syntax, correctly referencing the characters, providing sufficient
description of the major
events, organizing the events of the story, and making the story
interesting to the listener.
Such assessment can provide a clearer understanding of the
nature of a child’s language
difficulties and may assist in the development of treatment
goals.
While CR assessment tasks have many admirable properties,
they can be difficult to
implement, given the unstandardized nature of the task. With
the increased call for use of
naturalistic and CR assessment tasks, there is a need to
understand better the measure-

9. ment properties of these assessment tasks.
Difficulties associated with developing CR assessments
As stated by Laing and Kamhi (2003, p. 46), ‘criterion-
referenced measures are only as
good as the developmental data on which they are based.’ In the
field of language devel-
opment, decades of research have provided a rich and expansive
literature documenting
the complexities associated with acquiring human language.
Language use is complex
and difficult to measure, as it encompasses multiple aspects of
form, content, and use
(Bloom & Lahey, 1978). At any stage of development, different
aspects of children’s
vocabulary, grammar, and pragmatic skills are at very different
stages of development.
To complicate matters further, language use is highly influenced
by the broader speaking
context; a behavior that appears to be immature in one situation
can appear to be mas-
tered in another situation (Elman, 1995). When identifying
appropriate CR assessment
measures, clinicians and researchers must critically evaluate the
assessment tasks and
sampling procedures to determine whether they are
developmentally appropriate and
will generate sensitive measures for the target population.
Ollendick, Grills, and King
(2001) stated that developmentally appropriate measures should
measure meaningful
behaviors that are sensitive to the children’s developmental
levels and be responsive to
changes in the context that may affect the sensitivity of the
measure.

10. Measures from children’s productions of oral narratives are
highly sensitive to
changes in the sampling context and discourse demands of the
task. Peña et al. (2006)
described how increased amounts of support provided to the
child, or scaffolding, facili-
tates children’s production of oral narratives. Factors affecting
the complexity and over-
all quality of a child’s narrative production include whether or
not the child had heard the
story before (Ripich & Griffith, 1988; Schneider & Dubé,
2005), how many times the
child had heard the story (Goodsitt, Raitan, & Perlmutter, 1988;
Martinez & Roser,
1985), familiarity with the events depicted in the story (Fivush,
1984; Hudson & Shapiro,
Heilmann et al. 607
1991), complexity of the story (Heilmann, Miller, Iglesias, &
Francis, 2009), and the
types of directions and expectations provided to the child (de
Temple, Wu, & Snow,
1991). For example, if a child is unfamiliar with a story, he or
she may have substantial
difficulty in organizing the complex events and including more
nuanced points of view.
If, however, a child is familiar with a story and is asked to
relate the story in a complex
manner, he or she will likely produce a longer, more detailed
story. Such variations can
present a problem for measurement of children’s narrative
organization skills.

11. Measuring children’s narrative organization skills
The ability to produce a coherent narrative is a complex
linguistic task that requires nar-
rators to plan and execute their production of the story’s
plotline by using appropriate
vocabulary, grammar, and syntax. Studies examining the
development of narrative have
identified that all stories possess the same underlying
components, or story grammar
(Stein & Glenn, 1979). The story grammar literature has
proposed that all stories contain
a setting and episode system, which includes an initiating event
or problem, a reaction to
that problem, various attempts at resolving the problem, a
conclusion, and resolution. All
stories use some kind of combination of these story grammar
components. Developmental
studies have revealed that the acquisition of narrative
proficiency is a slow process, which
emerges in the preschool years and is not fully developed until
adulthood, with some
adults never becoming fully proficient narrators (Berman &
Slobin, 1994). Analysis of
children’s narrative organization skills has been described in
several clinical texts (Hughes,
McGillivray, & Schmidek, 1997; Strong, 1998), but the
measurement properties of the
various scoring procedures have not been empirically tested
with large samples.
Upon developing a series of narrative databases, the Language
Analysis Lab at the
University of Wisconsin-Madison was interested in identifying
a developmentally sensi-
tive narrative organization measure that that could be used on a
corpus of narrative retells

12. produced by children using the wordless picture book Frog,
Where are You? (Mayer,
1969). We first looked to the literature for sensitive narrative
organization measures for
this corpus and identified studies measuring narrative
organization in children who pro-
duced Frog, Where are You? The review was limited to this
specific story because it has
been extensively reported in the literature, follows a
prototypical story sequence, and
was to be used for several projects completed in the Language
Analysis Lab, including
the data summarized in this study.
The first major method for measuring children’s narrative
organization skills identi-
fied whether or not children included specific plotlines and
themes (i.e., plot and theme
analysis). When applying plot and theme analyses, examiners
developed a coding scheme
using the key story grammar elements for the target story. This
approach allowed exam-
iners to develop binary decision schemes that identified the
presence or absence of spe-
cific story grammar components. Children who produced a
greater number of plotlines
and themes were thought to have more advanced narrative skills
(e.g., Berman, 1988;
Botting, 2002; Boudreau & Hedberg, 1999; Miles & Chapman,
2002; Norbury & Bishop,
2003; Reilly, Losh, Bellugi, & Wulfeck, 2004). An example of a
plot and theme analysis
is provided in Appendix A.

13. 608 Language Testing 27(4)
The second major class of narrative organization measures
relied on holistic judgments
of children’s narrative proficiency (Applebee, 1978; Hedberg &
Westby, 1993; Stein,
1988). While the text-level analyses also documented children’s
production of story gram-
mar components, they were not measured by counting the
presence or absence of specific
plotlines and themes. Rather, these measures required holistic
judgments by the examiner
to rate the quality and developmental level of the narrative. Two
contemporary studies
used the text-level measures of narrative organization for
analysis of Frog, Where are
You? Manhardt and Rescorla (2002) converted Applebee’s
categorical levels to an ordinal
scale to assess narratives produced by 8–9-year-old children
with histories of language
delay, while Pearce, McCormack, and James (2003) used Stein’s
scoring scheme to assess
narratives produced by children of 5 ½ years of age. A summary
and brief description of
these two holistic narrative scoring procedures is provided in
Appendix A.
Several of the reported studies compared performance on the
story grammar measures
across clinical groups with varying results. Reilly et al. (2004)
documented that their
story grammar analysis was sensitive to both age and group
differences in four groups of
children of between 7 and 9 years of age (children with specific
language impairment,
early focal brain injury, Williams syndrome, and typical

14. developmental histories). Pearce
et al. (2003) found significant differences in performance
between children with lan-
guage impairment and typically developing children at 5 ½
years of age. Boudreau and
Hedberg (1999), on the other hand, did not find any differences
between typically devel-
oping children and children with specific language impairment
on their narrative organi-
zation measure despite significant differences on each
additional measure collected from
the children’s narrative productions. McCabe and Rollins (1994)
also noted that the nar-
rative organization skills of children with language impairment
varied widely across
studies and attributed this variability to ‘the use of insensitive
means of scoring narra-
tives’ (p. 47). Further examination of the literature revealed that
these existing story
grammar measures may be too easy and potentially insensitive
for preschool and young
school-age children. For the plot and theme measures, Reilly et
al. documented that typi-
cally developing 7–9-year-old children, on average, produced
95% (11.4/12) of the plot-
lines and themes, while Boudreau and Hedberg documented that
their group of 5-year-old
children produced 78% (4.7/6) of the plotlines and themes. On
average, high success
rates were also noted for the text-level measures, with a group
of 8-year-old children
scoring 3.3/5 on the Applebee measure (Manhardt & Rescorla,
2002) and 5–6-year-old
children scoring 9/11 on the Stein measure (Pearce et al., 2003).
In identifying the best story grammar measure for analyzing

15. retells of a wordless
picture book, the potential sensitivity issue was compounded by
the sampling context.
Our corpus of narratives was collected using the retell
procedure, whereas most studies
from the literature did not provide an initial model of the target
story. Having children
retell a story provides a model story and assists children in
developing an understanding
of the story’s structure. Narrative retells have been shown to
contain more information
and incorporate a greater number of episodes than productions
of spontaneous narratives
(Ripich & Griffith, 1988; Schneider & Dubé, 2005). Our goal in
using the retell procedure
was to ensure that the samples represented the children’s best
narrative productions.
However, we anticipated that a simple story grammar analysis
would be ineffective for this
corpus of retells given that children include a greater amount of
information in their oral
Heilmann et al. 609
productions in the retell condition. Thus, we decided that
existing narrative organization
measures may not be appropriate and developed a new scale to
allow for more sensitive
measurement of narrative proficiency.
The Narrative Scoring Scheme: A comprehensive measure
of narrative organization skills
The Narrative Scoring Scheme (NSS) was developed by the
Language Analysis Lab at the

16. University of Wisconsin – Madison as a sensitive measure of
children’s overall narrative
organization skills. To make the NSS more sensitive for a wider
range of ages and for sam-
pling contexts that provide scaffolding (i.e., the narrative retell
procedure), two modifica-
tions were made from the other narrative organization measures.
First, the literature was
reviewed to identify later developing narrative organization
features that go beyond simple
story grammar analyses. Second, scoring procedures were
critiqued to identify methods
that may be more sensitive when evaluating more advanced
narrative productions.
In addition to incorporating basic story grammar features, fully
proficient narrators use
specific types of language features that define a literate style of
speaking (Bamberg &
Damrad-Frye, 1991). One way narrators use a literate style of
speaking is through the use of
metacognitive verbs, which include verbs used to describe the
characters’ thoughts and men-
tal states (e.g., think and know) and metalinguistic verbs, which
include words used to
describe characters’ speech and dialogue (e.g., say and talk; see
Nippold, 2007 and Westby,
2005 for a review). In their analysis of oral productions of Frog,
Where are You?, Bamberg
& Damrad-Frye contrasted productions from novice narrators
(i.e., young children) to those
from experienced narrators (i.e., older children and adults) and
found that use of abstract
language, including metacognitive and metalinguistic verbs,
were the key features that char-
acterized the more sophisticated narrative productions and

17. assisted the older narrators in
organizing the hierarchical relationships between the events in
the story. They found that
these high-level language skills were first seen around five
years of age, were not consistently
used until the later school-age years, and continued to develop
through adulthood. Additional
studies have documented that these abstract language features,
collectively termed literate
language, emerged during the preschool years (Curenton &
Justice, 2004), were more consis-
tently used as children progressed through adolescence
(Greenhalgh & Strong, 2001;
Nippold, 2007; Pelligrini, Galda, Bartini, & Charak, 1998), and
were not consistently or
appropriately used by children with language impairment
(Greenhalgh & Strong, 2001).
Another high-level skill that is positively related to children’s
narrative organization
skills is the effective use of cohesive devices. Halliday and
Hasan (1976) characterized the
various ways that children maintain concepts across utterances
through use of sophisticated
linguistic procedures, including referential cohesion, which is
the way that speakers main-
tain appropriate reference throughout the story with correct use
of nouns and pronouns,
conjunctive cohesion, which is the way that speakers combine
phrases and sentences with
conjunctions and conjunctive phrases, and lexical cohesion,
which is the way that speakers
choose appropriate words to link concepts across phrases and
sentences. Children with
language impairment have much more difficulty producing
cohesive features than their

18. typically developing peers (Hedberg & Westby, 1993; Liles,
1985; Strong & Shaver, 1991).
610 Language Testing 27(4)
In addition to incorporating higher level narrative skills into the
NSS, we attempted to
increase the sensitivity of the measure by critically reviewing
the scoring procedures and
scaling of the measure. McFadden and Gillam (1996) compared
two methods of scoring
children’s narrative organization skills: discrete coding schemes
based on presence or
absence of story features and holistic ratings of children’s
narrative proficiency. They
found that the holistic ratings were superior to the discrete
coding schemes in discrimi-
nating between children who were typically developing and
children with language
impairment. These children were between the ages of 9;0 and
11;7. McFadden and
Gillam’s study demonstrated that holistic ratings can effectively
assess the inter-
utterance concepts and qualitative aspects of the story,
including the story’s sparkle (Peterson
& McCabe, 1983). To incorporate holistic ratings into a detailed
and descriptive rating
scale, the NSS requires examiners to make broad judgments
across the seven aspects of
narrative organization included in the scale.
The goal of this paper was to compare the measurement
properties of the NSS with
other methods of operationalizing narrative organization from

19. the literature. This study
provided a direct comparison of four separate scoring
techniques using the same pool of
transcripts and identified the measures that were most
developmentally appropriate. The
literature review revealed that many of the existing narrative
organization measures may
focus too much on early developing narrative skills, such as the
inclusion of key story
grammar components. We felt that incorporating higher level
narrative skills would
make the scale developmentally appropriate for young school-
age children who pro-
duced narratives with scaffolding from the retell procedure. In
addition, the NSS required
examiners to provide a level of holistic judgment to assess the
qualitative aspects of the
story. To assess the sensitivity of the measures, the distribution
of scores was evaluated
for each narrative scoring technique to determine which
measures were developmentally
appropriate for the participants and sampling context used in the
present study. This
study addressed the following question: Is the distribution of
scores from the NSS less
skewed than scores from three traditional narrative organization
measures when applied
to narrative retells produced by 5–7-year-old children?
Method
Narrative language samples were collected from 129 typically
developing children
between 5;0 and 7;0 years of age. The children were recruited
from public schools in the
San Diego County and El Cajon County school districts. The
school speech-language

20. pathologist (SLP) and the children’s classroom teachers
confirmed that the children were
typically developing by reviewing all academic records and
summative assessments. To
be included in the study, children had to be performing at grade
level and could not be
receiving special education or speech/language services. See
Table 1 for a summary of
the participants’ demographic data. The narrative samples were
collected by a practicing
SLP working in the schools under the direction of a project
coordinator. The SLPs met
with the project coordinator for three sessions to learn the
protocol, where they had the
opportunity to develop an understanding of the protocol, ask for
clarification, and prac-
tice administering the assessment.
Heilmann et al. 611
The children completed the narrative task in the retell
condition, where the examiner
read the target story aloud to the child, cued the child to follow
along with the pictures in
a wordless picture book (Frog, Where are You?; Mayer, 1969),
and then asked the child to
retell the story to the examiner. This procedure was adapted
from the Strong Narrative
Assessment Procedure (Strong, 1998). The children’s language
samples were digitally
recorded and later transcribed by research assistants (RA) at the
University of Wisconsin
– Madison. Each RA had at least 10 hours of training in
completing transcription of chil-

21. dren’s oral language samples. The narrative samples were first
transcribed using the
Systematic Analysis of Language Transcripts (SALT; Miller &
Iglesias, 2008). After tran-
scribing the sample, the RA reviewed the written transcript and
completed the NSS scor-
ing (see Appendix B for scoring rules).
The NSS is divided into seven sections that assess seven
different aspects of narrative
organization. Three of the sections were modeled after the
original story grammar pro-
posals: introduction, conflict resolution, and conclusion. Use of
literate language skills
were evaluated with the mental states and character
development sections. The final two
sections evaluated children’s cohesion skills and included
referencing and cohesion.
Within each section, scores of 1 were administered if the child
demonstrated immature
performance, scores of 3 if the child demonstrated emerging
skills, and scores of 5 if
proficient performance was noted. After completing each
section of the NSS, the scores
were added together to generate a total NSS score, which could
range from 0 to 35.
To compare performance on the different narrative organization
measures, the NSS
scores were compared to scores using three additional narrative
organization measures
from the literature. For the plot and theme approach, the
protocol described in Reilly
et al. (2004) was chosen because it provided a balanced
combination of the core plot
components and key embedded episodes. For the text-level

22. measures, we used Manhardt
and Rescorla’s (2002) ordinal adaptation of Applebee’s (1978)
narrative levels and
Pearce et al.’s (2003) ordinal adaptation of Stein’s (1988)
narrative levels. See Appendix
A for a description of the scoring procedures.
Each of the narrative organization analyses was scored by a
trained research assistant.
To document inter-rater agreement, 20% of the narrative
organization analyses were inde-
pendently coded by the first author. Krippendorff alpha values
(Krippendorff, 1980) were
calculated with ordinal scaling to determine the level of
agreement between transcribers.
Table 1. Demographic information for the 129 children
completing the narrative retell task
Grade (n) Gender (n) Maternal education (in years)
Race/ethnicity (n)
Preschool: 3 Female: 69 M = 14.4 White: 87
Kindergarten: 79 Male: 60 SD = 2.5 Hispanic: 16
1st Grade: 47 Range = 9–20 Othera: 15
African American: 7
No data: 4
Note: Ethnicity data collected for children who were Hispanic
or Latino. Race data provided for all children
who were non-Hispanic or Latino.
aOther races/ethnicities include Arabic (2), Chinese (3),
Japanese (2), Korean (1), Filipino (5), Portuguese (1),
and Samoan (1).

23. 612 Language Testing 27(4)
Krippendorff established the following benchmarks for alpha
values: ≥0.80 is adequate and
values between 0.67 and 0.80 are acceptable for exploratory
research and drawing tentative
conclusions. The following alpha values were calculated from
samples used in the present
study: NSS = 0.79, Plot & Theme = 0.79, Applebee = 0.61, and
Stein = 0.69.
Results
The normality of the distributions was compared using skewness
and kurtosis statistics
(see Coolican, 2004, for a review). The skewness statistic
identifies if there is an unequal
distribution that goes towards the floor or ceiling. Skewness
measures of zero indicate a
perfectly normal distribution while skewness values exceeding
|0.8| have been described
as ‘noticeably skewed’ (Bourque & Clark, 1992, p. 69). If the
distribution of scores is
negatively skewed, then most of the scores are approaching
mastery levels, reflecting
that the measure was too easy for the child. The kurtosis
statistic identifies how closely
variables are bunched together; platykurtic distributions have
scores that are spread out
and leptokurtic distributions have scores that are bunched
closely together (Coolican,
2004). Skewness and kurtosis statistics were generated for all
four scoring procedures
using SPSS. The traditional kurtosis statistics were converted so
that they were on the
same scale as the skewness statistics, with zero indicating a

24. perfectly normal distribu-
tion. While these analyses do not generate tests of statistical
significance, they do allow
comparisons to be drawn regarding the relative skewness and
kurtosis across different
measures. A final method for documenting the relative
distribution of scores was to
establish a score representing near-mastery performance and
identify the percentage of
the participants that scored at or above that criteria, similar to a
procedure employed by
Helms et al. (2004). For the present study, achieving at or above
90% correct was con-
sidered near-mastery performance and therefore a ceiling effect.
Table 2 summarizes the scores acquired from each scoring
scheme, showing the full
range of possible scores, the range acquired from this initial
sample, the sample means
and standard deviations, the skewness and kurtosis statistics,
and the percentage of the
sample who scored greater than 90% correct on the measure.
The skewness statistics for
all of the narrative measures were negative, demonstrating that
the distribution of scores
were more concentrated towards the ceiling. Skewness for the
NSS was considerably
lower than the additional measures from the literature.
Furthermore, skewness statistics
exceeded |0.8| for the plot and theme, Applebee, and Stein
measures, demonstrating that
these measures were ‘noticeably skewed’ (Bourque & Clark,
1992). The kurtosis values
for each of the narrative measures were greater than zero,
documenting that the narrative
measures were generating leptokurtic distributions that had a

25. relatively large proportion
of scores surrounding the mean. Again, the kurtosis values were
noticeably greater for
the three measures from the literature (0.9–1.1) than scores
from the NSS (0.5), showing
that there was a more restricted distribution for the plot and
theme, Applebee, and Stein
measures. The final analysis identified the number of
participants who scored close to
ceiling, with the criteria set at a score equal to or greater than
90% correct. None of the
children scored above 90% on the NSS, while 30–35% of the
children scored above 90%
on the three measures from the literature. Histograms were
generated for each of the
Heilmann et al. 613
narrative measures to visualize the distribution for each
narrative organization measure
(see Figure 1). Examination of Figure 1 illustrates the heavy
weighting of scores near
ceiling and around the mean for the three measures from the
literature and the relatively
normal distribution of the NSS.
Sample Narratives
To further illustrate differences across the scoring techniques,
three samples were
selected and are available in Appendix C. The samples were
modified from their original
coded format to assist with readability. All codes required for
SALT analyses were

26. Table 2. Children’s performance across four separate narrative
organization measures
Possible range Sample range Sample Mean (SD) Skewness
Kurtosis > 90%
Plot & Theme 0–12 4–12 8.8 (2.0) −1.0 1.1 35%
Applebee 0–5 2–6 4.0 (1.0) −1.1 0.9 30%
Stein 0–11 2–10 6.8 (2.4) −1.0 0.9 30%
NSS 0–35 11–26 20.1 (3.2) −0.5 0.5 0%
Note: > 90% signifies the proportion of the sample who scored
above 90% correct on the measure.
0
5
10
15
20
25
30
35
40
45
0 1 2 3 4 5 6 7 8 9 10 11 12

27. Plot & Theme
F
re
q
u
e
n
c
y
0
10
20
30
40
50
60
70
0 1 2 3 4 5
Applebee
F
re

28. q
u
e
n
c
y
0
10
20
30
40
50
60
0 1 2 3 4 5 6 7 8 9 10 11
Stein
F
re
q
u
e
n

29. c
y
0
5
10
15
20
25
30
0 3 6 9 12 15 18 21 24 27 30 33
NSS
F
re
q
u
e
n
c
y
Figure 1. Histograms depicting the distribution of scores using
four narrative structure coding
schemes

30. 614 Language Testing 27(4)
removed, including word errors, utterance errors, mazes (which
include reduplications,
reformulations, and false starts), and slashes used to identify
bound morphemes and
contractions. While these codes are required for syntactic and
semantic analysis of the
narrative samples, they are not necessary to accurately analyze
narrative structure.
Samples 1, 2, and 3 reflect a range of performance on the
narrative retell task.
By simply reading the transcripts, even an untrained reader with
no experience in
analyzing children’s oral language can identify that sample 3 is
much poorer than the
other two samples. It is considerably shorter and more difficult
to follow than samples 1
and 2. Many readers will also note that sample 1 is more
advanced than sample 2. In
story 1, the narrator provided more detail, better described each
of the events, and used
complex language to make the story interesting for the listener.
Upon comparing the dif-
ferent narrative organization measures, each measure was
effective in identifying that
story 3 was less complete and more poorly organized than
stories 1 and 2. However, the
plot and theme and two holistic narrative organization measures
did not distinguish sto-
ries 1 and 2 from each other. Rather, the NSS was the only
measure sensitive enough to
reveal differences between the productions.

31. Discussion
Sensitivity analyses
This study compared the measurement properties of the NSS to
three other narrative
organization measures from the literature. Each of the 129
transcripts was scored using
the four scoring schemes summarized in Appendix A. Twenty
percent of the transcripts
were recoded by a second member of the research team to
document the level of agree-
ment between transcribers. Krippendorff alpha coefficients were
equivalent for the NSS
and plot and theme measures (α = 0.79), while agreement values
for the Applebee and
Stein measures (α = 0.61 and 0.69, respectively) were notably
lower. We anticipated that
the highest level of inter-rater agreement would occur with the
plot and theme measures,
as the coding task was limited to identification of specific story
components. We were
impressed that the coding procedures of the NSS facilitated
comparable agreement val-
ues when compared to the plot and theme measures and
strikingly higher agreement
levels than the two text-level scoring scales.
Upon comparing the distribution of scores across the four
scoring schemes, scores from
the plot and theme and text-level narrative organization
measures were noticeably more
skewed and leptokurtic than scores from the NSS,
demonstrating that the majority of the
scores were closely bunched together near the ceiling.

32. Furthermore, approximately one
third of the children demonstrated near-mastery performance
and were approaching ceiling
on the measures from the literature, while none of the children
approached ceiling on the
NSS. Upon scoring the narratives with the NSS, we observed a
wider distribution of scores,
a relatively normal distribution across participants, and ample
room for measurement of
children with higher and lower narrative skills. We hypothesize
that there were two reasons
for the increased sensitivity of the NSS. First, by incorporating
children’s use of literate
language and cohesion, the NSS measured skills that were later
developing and/or were
Heilmann et al. 615
present when children receive scaffolding. Second, by utilizing
examiner judgment, the
NSS was able to tap into the perceptual aspects of the narrative
process (i.e., ‘sparkle’) that
are missed by discrete scoring schemes. Use of a more sensitive
scale, such as the NSS,
may address the sensitivity issue described by McCabe and
Rollins (1994) and provide a
more sensitive measure that can distinguish between typically
developing children and
children with language impairment (cf., Boudreau and Hedberg,
1999).
Because children’s narrative skills continue to develop through
the school years, a
more sensitive measure that assesses higher level narrative

33. features allows clinicians and
researchers to document narrative organization skills in older
school-age children. In the
present study, the child with the strongest narrative
organization skills received a score
of 26 on the NSS, while a perfect score would be 35. Based on
these data, there are an
additional 9 points that can be earned on the NSS for even the
best narrators. We predict
that NSS scores continue to increase in older children with
stronger language skills and
continue to develop additional reference databases to evaluate a
wider range of ages.
It is important to note that the data from the present study were
potentially constrained
by the limited sampling context. The data were acquired from a
relatively short and sim-
ple story. The extensive use of Frog, Where are You? in the
literature has greatly increased
our understanding of how children develop narrative
proficiency. However, additional
work is needed to document the effect of different stories on
measures of narrative orga-
nization. The plot and theme and text-level narrative
organization measures may be more
sensitive when children have the opportunity to produce longer
and more complex narra-
tives. From a clinical standpoint, however, increasing the story
length and complexity
increases the time requirements for elicitation and analysis of
the language samples. Our
goal was to use a story that could be collected and analyzed
quickly and efficiently.
Use of the NSS in research, educational, and clinical contexts

34. Research examining the mastery of oral narrative skills in
children can assist in better
understanding of the development of broader cognitive skills.
Studies of children’s
developing narrative have improved our understanding of the
development of cognitive
schemas; Berman and Slobin (1994) completed the most
comprehensive description of
the relationship between linguistic form and function. The study
of narrative develop-
ment has also shaped theories about the relationship between
socio-linguistic factors and
developing narrative competence (Eaton, Collis, & Lewis, 1995;
McCabe, 1997; Peterson
& McCabe, 2004; Quasthoff, 1997) as well as the relationship
between oral narratives
and general cognitive skills, such as working memory (e.g., van
den Broek, 1997).
Having a sensitive measure of children’s narrative organization
skills, such as the NSS,
is essential for documenting children’s narrative organization
skills and will facilitate
further advancements of our understanding of human cognition.
A sensitive and feasible measure of narrative organization, such
as the NSS, has
important educational policy implications. While oral language
skills are a key part of
most schools’ curricula, curriculum based assessment in the
general classroom has pri-
marily focused on documentation of children’s reading skills
(see Reschly, Busch, Betts,
Deno, & Long, 2009). Continuing to recognize the important
relationship between

35. 616 Language Testing 27(4)
oral narrative skills and broader reading and academic
achievements will encourage
greater advocacy for regular assessment of children’s oral
narrative skills in the general
curriculum. However, for this to occur, teachers and
practitioners need to have sensitive
and efficient measures, such as the NSS.
The NSS can also be an important tool for clinicians working
with children with lan-
guage impairment and for those working with second language
learners. The composite
NSS score provides a single estimate of children’s overall
narrative competence that is
sensitive for young children producing narrative retells. Given
the flexibility of the mea-
sure, it is likely also appropriate to use with children producing
more complex and less
complex narratives than those produced by the children in this
study. NSS composite
scores may also be used to assist with the identification of
children experiencing lan-
guage learning difficulties. Several databases are available with
the SALT software that
allows clinicians to compare their clients’ performance to a
sample of typically develop-
ing speakers. One database summarizes NSS scores collected
from monolingual English-
speaking children producing narrative retells; the samples used
in this study are included
in that database. The second major database with NSS scores
was collected on a large
group of English language learners who produced narrative

36. retells in both English and
Spanish. In addition to a composite narrative score, the NSS
provides examiners with
estimates of children’s performance within seven different
aspects of the narrative pro-
cess. Examiners can examine the profile of performance across
each aspect of the NSS
to identify areas of relative strength and relative weakness,
which may assist with further
determining the nature of a child’s language impairment and
assist with the development
of treatment goals. Upon initiating treatment, the NSS can be
used to monitor children’s
progress during the intervention program.
Conclusion
Decades of research and clinical practice literature have
identified the limitations of norm
referenced testing and have identified CR assessment as a viable
alternative for overcom-
ing many of these shortcomings. CR assessments provide a
detailed description of chil-
dren’s performance when completing naturalistic and
meaningful tasks. With the call for
increased use of CR assessments, more rigorous testing of the
properties of these tasks
must be completed. When evaluating assessment measures,
factors influencing the devel-
opmental sensitivity of the measure should be considered,
including the linguistic features
under study and the scaling of the assessment measure. In this
paper, we reviewed four
different methods of measuring children’s narrative
organization skills and found that the
NSS was more sensitive than the three other measures for young
school-age children

37. completing a narrative retell procedure. These analyses
identified features that allowed for
more sensitive analysis of narrative organization skills,
including assessment of higher
level narrative concepts and incorporation of examiner
judgment across multiple narrative
features. These analyses also provided an example of methods
that can be used to evaluate
the measurement properties of additional CR assessment
procedures. Such analyses and
discussion are increasingly important as more clinicians are
implementing the CR assess-
ment into clinical practice.
Heilmann et al. 617
Acknowledgements
We would like to thank Claudia Dunaway and the SLPs in the
San Diego and El Cajon school
districts for their assistance in collecting the narrative retell
data. We would also like to thank
Karen Andriacchi and all of the past and present members of the
language analysis lab who have
been dedicated to accurately transcribing and efficiently
organizing all of the language samples.
References
Allen, M. S., Kertoy, M. K., Sherblom, J. C., & Pettit, J. M.
(1994). Children’s narrative productions:
A comparison of personal event and fictional stories. Applied
Psycholinguistics, 15(2), 149–176.

38. Applebee, A. (1978). The child’s concept of story: Ages two to
seven. Chicago, IL: University of
Chicago Press.
August, D., & Hakuta, K. (1997). Improving schooling for
language-minority children. Washing-
ton, DC: National Academy Press.
August, D., & Shanahan, T. (Eds.) (2006). Developing literacy
in second-language learners:
Report of the National Literacy Panel on Language-Minority
Children and Youth. Mahwah,
NJ: Lawrence Erlbaum.
Bachman, L. (2000). Modern language testing at the turn of the
century: Assuring that what we
count counts. Language Testing, 17(1), 1–42.
Bamberg, M., & Damrad-Frye, R. (1991). On the ability to
provide evaluative comments: Further
explorations of children’s narrative competencies. Journal of
Child Language, 18(3), 689–710.
Berman, R. (1988). On the ability to relate events in narrative.
Discourse Processes, 11(4), 469–497.
Berman, R., & Slobin, D. (Eds.) (1994). Relating events in
narrative: A crosslinguistic develop-
mental study. Hillsdale, NJ: Lawrence Erlbaum.
Bishop, D., & Edmundson, A. (1987). Language-impaired 4-
year-olds: Distinguishing transient
from persistent impairment. Journal of Speech & Hearing
Disorders, 52(2), 156–173.
Bloom, L., & Lahey, M. (1978). Language development and
language disorders. New York: Wiley.

39. Botting, N. (2002). Narrative as a tool for the assessment of
linguistic and pragmatic impairments.
Child Language Teaching & Therapy, 18(1), 1–22.
Boudreau, D. M., & Hedberg, N. (1999). A comparison of early
literacy skills in children with
specific language impairment and typically developing peers.
American Journal of Speech-
Language Pathology, 8(3), 249–260.
Boudreau, D., & Hedberg, N. (1999). A comparison of early
literacy skills in children with specific
language impairment and their typically developing peers.
American Journal of Speech-Language
Pathology, 8(3), 249–260.
Bourque, L., & Clark, V. (1992). Processing data: The survey
example. Thousand Oaks, CA: Sage
Publications.
Coolican, H. (2004). Research Methods and Statistics in
Psychology. London: Hodder & Stoughton.
Curenton, S., & Justice, L. (2004). African American and
Caucasian preschoolers’ use of decon-
textualized language: Literate language features in oral
narratives. Language, Speech, and
Hearing Services in the Schools, 35(3), 240–253.
de Temple, J. M., Wu, H.-f., & Snow, C. E. (1991). Papa Pig
just left for Pigtown: Children’s oral and
written picture descriptions under varying instructions.
Discourse Processes, 14(4), 469–495.

40. 618 Language Testing 27(4)
Dickinson, D. K., & McCabe, A. (2001). Bringing it all
together: The multiple origins, skills,
and environmental supports of early literacy. Learning
Disabilities Research & Practice,
16(4), 186–202.
Eaton, J., Collis, G., & Lewis, V. (1999). Evaluative
explanations in children’s narratives of a video
sequence without dialogue. Journal of Child Language, 26(3),
699–720.
Elman, J. (1995). Language as a dynamical system. In R. Port &
T. van Gelder (Eds.), Mind as
motion: Exploring the dynamics of cognition (pp. 195–223).
Cambridge, MA: MIT Press.
Fazio, B., Naremore, R., & Connell, P. (1996). Tracking
children from poverty at risk for specific
language impairment: A 3-year longitudinal study. Journal of
Speech and Hearing Research,
39(3), 52–63.
Fey, M. E., Catts, H. W., Proctor-Williams, K., Tomblin, J. B.,
& Zhang, X. (2004). Oral and
written story composition skills of children with language
impairment. Journal of Speech,
Language, and Hearing Research, 47(6), 1301–1318.
Fiestas, C. E., & Peña, E. D. (2004). Narrative discourse in
bilingual children: Language and task
effects. Language, Speech, and Hearing Services in Schools,
35(2), 155–168.

41. Fivush, R. (1984). Learning about school: The development of
kindergartners’ school scripts.
Child Development, 55(5), 1697–1709.
Gillam, R. B., & Johnston, J. R. (1992). Spoken and written
language relationships in language/
learning-impaired and normally achieving school-age children.
Journal of Speech & Hearing
Research, 35(6), 1303–1315.
Goodsitt, J., Raitan, J. G., & Perlmutter, M. (1988). Interaction
between mothers and preschool chil-
dren when reading a novel and familiar book. International
Journal of Behavioral Development,
11(4), 489–505.
Greenhalgh, K., & Strong, C. (2001). Literate language features
in spoken narratives of children
with typical language and children with language impairment.
Language, Speech, and Hearing
Services in Schools, 32(2), 114–125.
Griffin, T. M., Hemphill, L., Camp, L., & Wolf, D. P. (2004).
Oral discourse in the preschool years
and later literacy skills. First Language, 24(71), 123–147.
Gutiérrez-Clellen, V. (2002). Narratives in two languages:
Assessing performance of bilingual
children. Linguistics and Education, 13(2), 175–197.
Halliday, M., & Hasan, R. (1976). Cohesion in English. London:
Longman.
Hedberg, N., and Westby, C. (1993). Analyzing storytelling
skills: Theory to practice. Tucson,
AZ: Communication Skill Builders.

42. Heilmann, J., Miller, J., Iglesias, A., & Francis, D. (2009,
June). Differences in oral narratives as a
function of story in bilingual children. Paper presented at the
Symposium on Research in Child
Language Disorders, Madison, WI.
Helms, J., Weichbold, V., Baumann, U., von Specht, H., Schon,
F., Muller, J., et al. (2004). Analysis
of ceiling effects occurring with speech recognition tests in
adult cochlear-implant patients.
Journal for Oto-Rhino-Laryngology and Its Related Specialties,
66(3), 130–135.
Hemphill, L., Snow, C., Olson, D. R., & Torrance, N. (1996).
Language and literacy development:
Discontinuities and differences. In D. R. Olson, and N.
Torrance (Eds.), The handbook of
education and human development: New models of learning,
teaching and schooling
(pp. 173–201). Malden, MA: Blackwell Publishing.
Hudson, J. A., Shapiro, L. R., McCabe, A., & Peterson, C.
(1991). From knowing to telling: The
development of children’s scripts, stories, and personal
narratives. In A. McCabe & C. Peterson
(Eds.), Developing narrative structure (pp. 89–136). Hillsdale,
NJ: Lawrence Erlbaum.
Heilmann et al. 619
Hughes, D., McGillivray, L., & Schmidek, M. (1997). Guide to
narrative language. Austin, TX:
PRO-ED.

43. Krippendorff, K. (1980). Content analysis: An introduction to
its methodology. Thousand Oaks,
CA: Sage Publications.
Laing, S., & Kamhi, A. (2003). Alternative assessment of
language and literacy in culturally and linguis-
tically diverse populations. Language, Speech, and Hearing
Services in the Schools, 34(1), 44–55.
Liles, B. (1985). Cohesion in the narratives of normal and
language-disordered children. Journal
of Speech & Hearing Research, 28(1), 123–133.
Linn, R. L., & Gronlund, N. E. (2000). Measurement and
assessment in teaching (8th ed.). Upper
Saddle River, NJ: Prentice Hall.
Mandler, J. M., Scribner, S., Cole, M., & DeForest, M. (1980).
Cross-cultural invariance in story
recall. Child Development, 51(1), 19–26.
Manhardt, J., & Rescorla, L. (2002). Oral narrative skills of late
talkers at ages 8 and 9. Applied
Psycholinguistics, 23(1), 1–21.
Marquardt, T., & Gillam, R. (1999). Assessment in
communication disorders: Some observations
on current issues. Language Testing, 16(3), 249–269.
Martinez, M., & Roser, N. (1985). Read it again: The value of
repeated readings during storytime.
The Reading Teacher, 38(8), 782–786.
Mayer, M. (1969). Frog, where are you? New York: Dial Press.
McCabe, A. (1997). Developmental and cross-cultural aspects

44. of children’s narration. In G. Bamberg
(Ed.), Narrative development: Six approaches (pp. 137–174).
Mahwah, NJ: Lawrence Erlbaum.
McCabe, A., & Rollins, P. (1994). Assessment of preschool
narrative skills. American Journal of
Speech-Language Pathology, 3(1), 45–56.
McCauley, R., & Swisher, L. (1984). Use and misuse of norm-
referenced tests in clinical assess-
ment: A hypothetical case. Journal of Speech and Hearing
Disorders, 49(4), 34–42.
McFadden, T., & Gillam, R. (1996). An examination of the
quality of narratives produced by children
with language disorders. Language, Speech, and Hearing
Services in Schools, 27(1), 48–56.
Miles, S., & Chapman, R. S. (2002). Narrative content as
described by individuals with Down
syndrome and typically developing children. Journal of Speech,
Language, and Hearing
Research, 45(1), 175–189.
Miller, L., Gillam, R., & Peña, E. (2001). Dynamic assessment
and intervention: Improving chil-
dren’s narrative skills. Austin, TX: Pro-ed.
Miller, J. F., Heilmann, J., Nockerts, A., Iglesias, A., Fabiano,
L., & Francis, D. (2006). Oral language
and reading in bilingual children. Learning Disabilities
Research and Practice, 21(1), 30–43.
Miller, J. F. & Iglesias, A. (2008). Systematic Analysis of
Language Transcripts (Research Version

45. 9.1) [Computer software]. Madison, WI: Language Analysis
Lab.
Muñoz, M. L., Gillam, R. B., Peña, E. D., & Gulley-Faehnle, A.
(2003). Measures of language
development in fictional narratives of Latino children.
Language, Speech, and Hearing Ser-
vices in Schools, 34(4), 332–342.
Newman, R. M., & McGregor, K. K. (2006). Teachers and
laypersons discern quality differences
between narratives produced by children with or without SLI.
Journal of Speech, Language,
and Hearing Research, 49(5), 1022–1036.
Nippold, M. (2007). Later language development: School-age
children, adolescents, and young
adults (3rd ed.). Austin, Texas: Pro-ed.
Norbury, C., & Bishop, D. (2003). Narrative skills of children
with communication impairments.
International Journal of Communication Disorders, 38(3), 287–
313.
620 Language Testing 27(4)
North Carolina Department of Public Instruction (2004).
English language arts: Standard course
of study and grade level competencies, K – 12. Raleigh, NC:
North Carolina Department of
Public Instruction (ERIC Document Reproduction Service No.
ED496004).
Oller, D. & Pearson, B. (2002). Assessing the effects of

46. bilingualism: A background. In D. Oller &
R. Eilers (Eds.), Language and literacy development in bilingual
children (pp. 3–21). Tonawanda,
NY: Multilingual Matters.
Ollendick, T., Grills, A., & King, N. (2001). Applying
developmental theory to the assessment and
treatment of childhood disorders: Does it make a difference?
Clinical Psychology & Psycho-
therapy, 8, 304–315.
O’Neill, D. K., Pearce, M. J., & Pick, J. L. (2004). Preschool
children’s narratives and performance
on the Peabody Individualized Achievement Test–Revised:
Evidence of a relation between
early narrative and later mathematical ability. First Language,
24(71), 149–183.
Paul, R. & Smith, R. (1993). Narrative skills in 4-year-olds with
normal, impaired, and late-developing
language. Journal of Speech and Hearing Research, 36(3), 592–
598.
Pearce, W., McCormack, P, & James, D. (2003). Exploring the
boundaries of SLI: Findings from mor-
phosyntactic and story grammar analyses. Clinical Linguistics &
Phonetics, 17(4–5), 325–334.
Pelligrini, A., Galda, L., Bartini, M., & Charak, D. (1998). Oral
language and literacy learning in
content: The role of social relationships. Merrill-Palmer
Quarterly, 44(1), 38–54.
Peña, E., Gillam, R., Malek, M., Ruiz-Felter, R., Resendiz, M.,
Fiestas, C., & Sabel, T. (2006). Dynamic
assessment of school-age children’s narrative ability: An

47. experimental investigation of classifica-
tion accuracy. Journal of Speech, Language, and Hearing
Research, 49(5), 1037–1057.
Peterson, C., & McCabe, A. (2004). Echoing our parents:
Parental influences on children’s nar-
ration. In Pratt, M. & Fiese, B. (Eds.), Family stories and the
life course: Across time and
generations (pp. 27–54) Mahwah, NJ: Lawrence Erlbaum.
Quasthoff, U. (1997). An interactive approach to narrative
development. In Bamberg, M. (Ed.),
Narrative development: Six approaches (pp. 51–83). Mahwah,
NJ: Lawrence Erlbaum.
Rapin, I., & Allen, D. (1983). Developmental language
disorders: Nosological considerations. In
U. Kirk (Ed.), Neuropsychology of language, reading and
spelling (pp. 155–184). New York:
Academic Press.
Reilly, J., Losh, M., Bellugi, U., & Wulfeck, B. (2004). Frog,
Where are you? Narratives in children
with specific language impairment, early focal brain injury and
Williams Syndrome. In
B. Wulfeck & J. Reilly (Eds.), Plasticity and development:
Language in atypical children. Spe-
cial issue, Brain & Language, 88(2), 229–247.
Reschly, A., Busch, T., Betts, J., Deno, S., & Long, J. (2009).
Curriculum-Based Measurement
Oral Reading as an indicator of reading achievement: A meta-
analysis of the correlational
evidence. Journal of School Psychology 47(6), 427–469.
Ripich, D. N., & Griffith, P. L. (1988). Narrative abilities of

48. children with learning disabilities and
nondisabled children: Story structure, cohesion, and
propositions. Journal of Learning Disabilities,
21(3), 165–173.
Rojas, R., & Iglesias, A. (2009, March). Making a case for
language sampling. The ASHA Leader,
14, 10–11.
Roth, F. P., Speece, D. L., & Cooper, D. H. (2002). A
longitudinal analysis of the connection
between oral language and early reading. Journal of Educational
Research, 95(5), 259–272.
Roth, F. P., Speece, D. L., Cooper, D. H., & De La Paz, S.
(1996). Unresolved mysteries: How do
metalinguistic and narrative skills connect with early reading?
The Journal of Special Education,
30(3), 257–277.
Heilmann et al. 621
Schneider, P., & Dubé, R. (2005). Story presentation effects on
children’s retell content. American
Journal of Speech-Language Pathology, 14(1), 52–60.
Snow, C. (1983). Literacy and language: Relationships during
the preschool years. Harvard Edu-
cational Review, 53(2), 165–189.
Snow, C. E., Dickinson, D. K., Jennings, E. M., & Purves, A. C.
(1991). Skills that aren’t basic in a
new conception of literacy. In E. Jennings & A. Purves (Eds.),
Literate systems and individual

49. lives: Perspectives on literacy and schooling (pp. 179–191).
Albany, NY: State University of
New York Press.
Stein, N. L. (1988). The development of storytelling skill. In M.
Franklin & S. Barten (Eds.), Child
language: A reader. New York: Oxford University Press.
Stein, N., & Glenn, C. (1979). An analysis for story
comprehension in elementary school. In R. Free-
dle (Eds.), New directions in discourse processing, Vol. 2. (pp.
53–119). Norwood, NJ: Ablex.
Strong, C. (1998). The Strong Narrative Assessment Procedure.
Eau Claire, WI: Thinking Publications.
Strong, C., & Shaver, J. (1991). Stability of cohesion in the
spoken narratives of language-impaired and
normally developing school-aged children. Journal of Speech &
Hearing Research, 34(1), 95–111.
Tabors, P., Snow, C., & Dickinson, D. (2001). Homes and
schools together: Supporting language
and literacy development. In D. Dickinson & P. Tabors (Eds.),
Beginning literacy with lan-
guage (pp. 313–334). Baltimore, MD: Brookes.
Tomblin, J. B., Zhang, X., Weiss, A., Catts, H., & Ellis
Weismer, S. (2004). Dimensions of indi-
vidual differences in communication skills among primary grade
children. In M. Rice & S.
Warren (Eds.), Developmental language disorders: From
phenotpyes to etiologies (pp. 53–76).
Mahwah, NJ: Lawrence Erlbaum.
Tomblin, J., Records, N., Buckwalter, P., Zhang, X., Smith, E.,

50. & O’Brien, M. (1997). Prevalence
of specific language impairment in kindergarten children.
Journal of Speech, Language, and
Hearing Research, 40(6), 1245–1260.
Uccelli, P., & Páez, M. M. (2007). Narrative and vocabulary
development of bilingual children
from kindergarten to first grade: Developmental changes and
associations among English and
Spanish skills. Language, Speech, and Hearing Services in
Schools, 38(3), 225–236.
van den Broek, P. (1997). Discovering the cement of the
universe: The development of event
comprehension from childhood to adulthood. In P. van den
Broek, P. Bauer, & T. Bourg (Eds.),
Developmental spans in event comprehension: Bridging
fictional and actual events (pp. 321–342).
Mahwah, NJ: Lawrence Erlbaum.
Westby, C. (2005). Assessing and facilitating text
comprehension problems. In H. Catts & A. Kamhi
(Eds.), Language and reading disabilities (pp. 157–232).
Boston, MA: Allyn & Bacon.
Appendix A: Narrative Organization Scoring Procedures
1. Plot and Theme (Reilly et al., 2004)
Plotline (8 points possible):
Setting – 1 point
Instantiation (frog escapes) – 1 point
5 main search episodes – 0–5 points
Resolution – 1 point

51. 622 Language Testing 27(4)
Theme (4 points possible):
Mention that the frog was missing and the boy was looking for
him – 0–2 points
Search theme mentioned through the story: 0 = no additional
mentions, 1 = one
or two mentions, 2 = three or more mentions
2. Ordinal adaptation of Applebee’s Narrative Maturity Scale
(adapted from Manhardt & Rescorla, 2002 and Hughes et al.,
1997)
1. Heap: events described with no central theme
2. Sequence: events related to a single theme; no causal links
between story
concepts
3. Primitive narrative: stories have a concrete core
4. Focused chain: story revolves around a central character
going through a series
of events, story well connected
5. True narrative: well-developed story that has a central theme
or moral
3. Ordinal adaptation of Stein’s story levels (adapted from
Pearce et al.,
2003 and Hughes et al., 1997)
1. Isolated description: description of random characters and
actions

52. 2. Descriptive sequence: describe characters and actions, but no
causal
relationships
3. Action sequence: actions described in correct chronological
order, but no causal
relationships
4. Reactive sequence: series of actions with some causal
relationships, no goal-
directed descriptions
5. Abbreviated episode: story is goal-directed, but characters’
intent not explicitly
stated
6. Incomplete episode: characters’ intent is explicitly stated but
one of the following
episode components is missing: initiating event, attempt, or
consequence
7. Complete episode: story contains all three aspects of a
complete episode: initiating
event, attempt, and consequence
8. Complex episode: full episode is elaborated by including an
obstacle to obtaining
the goal
9. Multiple episode: story contains more than one episode
(either complete or
incomplete)
10. Embedded episodes: one episode embedded within another
11. Interactive episodes: use multiple perspectives to describe
events; multiple charac-

53. ters and multiple goals mutually influence each other
Heilmann et al. 623
A
p
p
e
n
d
ix
B
:
T
he
N
ar
ra
ti
ve
S
co
ri
ng

54. S
ch
em
e
C
ha
ra
ct
er
is
ti
c:
Pr
o
fic
ie
nt
Em
er
gi
ng

55. M
in
im
al
/I
m
m
at
ur
e
In
tr
o
du
ct
io
n
1)
S
et
ti
ng
:
1)

56. S
et
ti
ng
:
•
L
au
nc
he
s
in
to
s
to
ry
w
it
h
•
S
ta
te

57. s
ge
ne
ra
l p
la
ce
a
nd
p
ro
vi
de
s
so
m
e
•
S
ta
te
s
ge
ne

58. ra
l s
et
ti
ng
b
ut
p
ro
vi
de
s
n
o
a
tt
em
pt
t
o
p
ro
vi

59. de
d
et
ai
l a
bo
ut
t
he
s
et
ti
ng
(
e.
g.
, r
ef
er
en
ce

60. n
o
d
et
ai
l.
t
he
s
et
ti
ng
.
t
o
t
he
t
im
e
o
f

61. th
e
se
tt
in
g,
d
ay
ti
m
e,
•
D
es
cr
ip
ti
o
n
o
r
el
em

62. en
ts
o
f
st
o
ry
a
re
b
ed
ti
m
e,
s
ea
so
n)
.
g
iv
en
in

63. te
rm
it
te
nt
ly
t
hr
o
ug
h
st
o
ry
.
•
S
et
ti
ng
e
le

64. m
en
ts
a
re
s
ta
te
d
at
•
M
ay
p
ro
vi
de
d
es
cr
ip
tio

65. n
o
f s
pe
ci
fic
e
le
m
en
t
a
pp
ro
pr
ia
te
p
la
ce
in
s
to

66. ry
.
o
f
se
tt
in
g
(e
.g
.,
th
e
fr
o
g
is
in
t
he
ja
r)

67. .
O
R
2)
C
ha
ra
ct
er
s:
2)
C
ha
ra
ct
er
s:
•

68. M
ai
n
ch
ar
ac
te
rs
a
re
in
tr
o
du
ce
d
w
it
h
•
C
ha
ra
ct

69. er
s
o
f
st
o
ry
a
re
m
en
ti
o
ne
d
w
it
h
s
o
m
e
de

70. sc
ri
pt
io
n
o
r
de
ta
il
pr
o
vi
de
d.
n
o
d
et
ai
l o
r
de

71. sc
ri
pt
io
n.
C
ha
ra
ct
er
•
M
ai
n
ch
ar
ac
te
r(
s)
a
nd

72. a
ll
su
pp
o
rt
in
g
•
B
o
th
m
ai
n
an
d
ac
ti
ve
s
up
po
rt

73. in
g
•
I
nc
o
ns
is
te
nt
m
en
ti
o
n
o
f
de
ve
lo
pm
en
t
c

74. ha
ra
ct
er
(s
)
ar
e
m
en
ti
o
ne
d.
c
ha
ra
ct
er
s
ar
e
m

75. en
ti
o
ne
d.
i
nv
o
lv
ed
o
r
ac
ti
ve
•
T
hr
o
ug
ho
ut

76. s
to
ry
it
is
c
le
ar
c
hi
ld
c
an
•
M
ai
n
ch
ar
ac
te
rs

77. a
re
n
o
t
cl
ea
rl
y
c
ha
ra
ct
er
s.
d
is
cr
im
in
at

78. e
be
tw
ee
n
m
ai
n
an
d
d
is
ti
ng
ui
sh
ed
f
ro
m
s
up
po
rt

79. in
g
ch
ar
ac
te
rs
.
•
C
ha
ra
ct
er
(s
)
ne
ce
ss
ar
y
s
up

80. po
rt
in
g
ch
ar
ac
te
rs
(
e.
g.
, m
o
re
f
o
r
ad
va
nc
in

81. g
th
e
pl
o
t
d
es
cr
ip
ti
o
n
o
f,
em
ph
as
is
u
po
n

82. a
re
n
o
t
pr
es
en
t.
m
ai
n
ch
ar
ac
te
r(
s)
).

83. •
C
hi
ld
n
ar
ra
te
s
in
f
ir
st
p
er
so
n
us
in
g
c
ha
ra

84. ct
er
v
o
ic
e
(e
.g
., ‘
Yo
u
ge
t
o
ut
o
f
m
y
t
re
e’

85. , s
ai
d
th
e
o
w
l.)
.
M
en
ta
l s
ta
te
s
•
M
en
ta
l s
ta
te
s

86. o
f
m
ai
n
an
d
su
pp
o
rt
in
g
•
S
o
m
e
us
e
o
f
ev
id

87. en
t
m
en
ta
l s
ta
te
•
N
o
u
se
o
f
m
en
ta
l s
ta
te

88. c
ha
ra
ct
er
s
ar
e
ex
pr
es
se
d
w
he
n
ne
ce
ss
ar
y
w
o

89. rd
s
to
d
ev
el
o
p
ch
ar
ac
te
r(
s)
.
w
o
rd
s
to
d
ev
el
o

90. p
f
o
r
pl
o
t
de
ve
lo
pm
en
t
an
d
ad
va
nc
em
en
t.
c

91. ha
ra
ct
er
(s
).
•
A
v
ar
ie
ty
o
f
m
en
ta
l s
ta
te
w
o

92. rd
s
ar
e
us
ed
.
R
ef
er
en
ci
ng
•
P
ro
vi
de
s
ne
ce
ss
ar

93. y
an
te
ce
de
nt
s
•
I
nc
o
ns
is
te
nt
u
se
o
f
re
fe
re
nt
s/

94. an
te
ce
de
nt
s.
•
E
xc
es
si
ve
u
se
o
f
t
o
p
ro
no
un
s.

95. p
ro
no
un
s.
•
R
ef
er
en
ce
s
ar
e
cl
ea
r
th
ro
ug
ho

96. ut
s
to
ry
.
•
N
o
v
er
ba
l c
la
ri
fie
rs
u
se
d.
•
C

97. hi
ld
is
u
na
w
ar
e
l
is
te
ne
r
is
c
o
nf
us
ed
.
(C
on

98. tin
ue
d)
624 Language Testing 27(4)
A
p
p
e
n
d
ix
B
:
(C
o
nt
in
ue
d)
C
ha
ra

99. ct
er
is
ti
c:
Pr
o
fic
ie
nt
Em
er
gi
ng
M
in
im
al
/I
m
m

100. at
ur
e
C
o
nf
lic
t
•
C
le
ar
ly
s
ta
te
s
al
l c
o
nf
lic
ts
a

101. nd
•
U
nd
er
d
ev
el
o
pe
d
de
sc
ri
pt
io
n
o
f
co
nf
lic
ts

102. •
R
an
do
m
r
es
o
lu
ti
o
n(
s)
re
so
lu
ti
o
n
r
es
o
lu
ti

103. o
ns
c
ri
ti
ca
l t
o
a
dv
an
ci
ng
t
he
a
nd
r
es
o
lu
ti
o
ns

104. c
ri
ti
ca
l t
o
a
dv
an
ci
ng
t
he
s
ta
te
d
w
it
h
no
m
en
ti

105. o
n
p
lo
t
o
f
th
e
st
o
ry
.
p
lo
t
o
f
th
e
st
o

106. ry
.
o
f
ca
us
e
o
r
co
nf
lic
t.
O
R
O
R
•
N

107. o
t
al
l c
o
nf
lic
ts
a
nd
r
es
o
lu
ti
o
ns
c
ri
ti
ca
l
•
C
o

108. nf
lic
t
m
en
ti
o
ne
d
t
o
a
dv
an
ci
ng
t
he
p
lo
t
ar

109. e
pr
es
en
t.
w
it
ho
ut
r
es
o
lu
ti
o
n.
O
R
•
M
an
y

110. co
nf
lic
ts
a
nd
r
es
o
lu
ti
o
ns
c
ri
ti
ca
l t
o
a

111. dv
an
ci
ng
t
he
p
lo
t
ar
e
n
o
t
pr
es
en
t.
C
o
he
si
o

112. n
•
E
ve
nt
s
fo
llo
w
a
lo
gi
ca
l o
rd
er
.
•
E
ve
nt
s
fo

113. llo
w
a
lo
gi
ca
l o
rd
er
.
•
N
o
u
se
o
f
sm
o
o
th
•
C

114. ri
ti
ca
l e
ve
nt
s
ar
e
in
cl
ud
ed
w
hi
le
le
ss
•
E
xc
es

115. si
ve
d
et
ai
l o
r
em
ph
as
is
p
ro
vi
de
d
o
n
t
ra
ns
it
io

116. ns
.
e
m
ph
as
is
is
p
la
ce
d
o
n
m
in
o
r
ev
en
ts
.

117. m
in
o
r
ev
en
ts
le
ad
in
g
th
e
lis
te
ne
r
as
tr
ay
.
•

118. S
m
o
o
th
t
ra
ns
it
io
ns
a
re
p
ro
vi
de
d
O
R
b
et
w
ee

119. n
ev
en
ts
.
•
T
ra
ns
it
io
ns
t
o
n
ex
t
ev
en
t
un
cl
ea

120. r.
O
R
•
M
in
im
al
d
et
ai
l g
iv
en
fo
r
cr
it
ic
al

121. e
ve
nt
s.
O
R
•
E
qu
al
e
m
ph
as
is
o
n
al
l e
ve

122. nt
s.
C
o
nc
lu
si
o
n
•
S
to
ry
is
c
le
ar
ly
w
ra
pp
ed
u

123. p
us
in
g
ge
ne
ra
l
•
S
pe
ci
fic
e
ve
nt
is
c
o
nc
lu
de
d,
b

124. ut
n
o
•
S
to
ps
n
ar
ra
ti
ng
a
nd
c
o
nc
lu
di
ng
s
ta

125. te
m
en
ts
s
uc
h
as
‘a
nd
t
he
y
g
en
er
al
s
ta
te
m
en
t

126. m
ad
e
as
t
o
t
he
l
is
te
ne
r
m
ay
n
ee
d
to
a
sk

127. w
er
e
to
ge
th
er
a
ga
in
h
ap
py
a
s
co
ul
d
be
’.
c
o
nc
lu

128. si
o
n
o
f
th
e
w
ho
le
s
to
ry
.
i
f
th
at
is
t
he
e
nd

129. .
N
ot
es
:
Sc
o
ri
ng
: E
ac
h
ch
ar
ac
te
ri
st
ic
r
ec
ei
ve
s

130. a
sc
al
ed
s
co
re
0
–5
. P
ro
fic
ie
nt
c
ha
ra
ct
er
is
ti
cs
=
5

131. , E
m
er
gi
ng
=
3
, M
in
im
al
/
Im
m
at
ur
e
=
1
. S
co
re
s
in
b

132. et
w
ee
n
(e
.g
.,
2,
4
)
ar
e
un
de
fin
ed
, u
se
ju
dg
m
en
t.

133. Sc
o
re
s
o
f 0
, N
A
a
re
d
ef
in
ed
b
el
o
w
. A
c
o
m
po
si

134. te
is
s
co
re
d
by
a
dd
in
g
th
e
to
ta
l o
f t
he
c
ha
ra
ct
er
is

135. ti
c
sc
o
re
s.
H
ig
he
st
s
co
re
=
3
5.
A
s
co
re
o
f 0
is
g

136. iv
en
fo
r
C
hi
ld
E
rr
o
rs
(
i.e
.,
te
lli
ng
t
he
w
ro
ng
s
to

137. ry
, c
o
nv
er
si
ng
w
it
h
ex
am
in
er
, n
o
t
co
m
pl
et
in
g/
re
fu

138. si
ng
t
as
k,
u
si
ng
w
ro
ng
la
ng
ua
ge
c
re
at
in
g
in
ab
ili
ty

139. o
f s
co
re
r
to
c
o
m
pr
eh
en
d
st
o
ry
in
t
ar
ge
t
la
ng
ua

140. ge
, a
ba
nd
o
ne
d
ut
te
ra
nc
es
, u
ni
nt
el
lig
ib
ili
ty
, p
o
o
r
pe

141. rf
o
rm
an
ce
, c
o
m
po
ne
nt
s
o
f r
ub
ri
c
ar
e
in
im
it
at
io
n-

142. o
nl
y)
.
A
s
co
re
o
f N
A
(
no
n-
ap
pl
ic
ab
le
)
is
g
iv
en

143. fo
r
M
ec
ha
ni
ca
l/E
xa
m
in
er
/O
pe
ra
to
r
Er
ro
rs
(
i.e
.,
in

144. te
rf
er
en
ce
fr
o
m
b
ac
kg
ro
un
d
no
is
e,
is
su
es
w
it
h
re

145. co
rd
in
g
(c
ut
-o
ffs
, i
nt
er
ru
pt
io
ns
),
ex
am
in
er
q
ui
tt
in

146. g
be
fo
re
c
hi
ld
d
o
es
, e
xa
m
in
er
n
o
t
fo
llo
w
in
g
pr

147. o
to
co
l,
ex
am
in
er
a
sk
in
g
o
ve
rl
y
sp
ec
ifi
c
o
r
le
ad

148. in
g
qu
es
ti
o
ns
r
at
he
r
th
an
o
pe
n-
en
de
d
qu
es
ti
o
ns
o

149. r
pr
o
m
pt
s)
.
Heilmann et al. 625
Appendix C: Sample Narratives
Sample 1:
A little boy went out one day and caught a frog. He put the frog
in a jar and stared at him
when he got in his room again. The dog looked in the jar and
saw the frog too. While the
boy was sleeping the frog jumped down and ran out the window,
while he was sleeping.
The next morning the boy woke up and looked at the jar. The
frog was not inside the jar.
The boy looked everywhere. The dog put his head inside the jar.
His head got stuck. Then
they looked out the window and called for the frog to come
back. The dog fell off the thing
and the jar broke. The boy picked him up to see if he was OK.
And the dog licked him for
that. The dog and the boy went out in the backyard and went in
the forest. They searched
and searched and searched. But they still couldn’t find the frog.
The little boy crawled in a

150. gopher’s hole. The gopher popped out and started running away.
The dog was barking at a
beehive. And the beehive fell down. And a swarm of bees came
out that minute while the
boy was looking inside a tree. The bees swarmed and chased the
dog. The owl creeped out
and scared the boy away. After that he climbed up a tall rock
and leaned on some branches.
They started moving upwards and turning. This was a deer and
not real branches. The dog
ran with the deer and started barking at the deer. The deer
stopped in a sudden moment.
And the boy and the dog fell down. They landed in a warm pond
and heard the sound of a
croak. The boy told the dog to be quiet because if he wasn’t
then the frog would hop away
because of the babyies over a dead log. They found his old frog
with a mama frog. And they
had eight tiny babies. One jumped up to his hand to greet the
boy. The boy liked him. And
the frog liked him also. So he took that frog for a new pet and
left the other frogs together
so the family would be safe. And they had seven tiny babies to
take care of, not one.
Plot & Theme: 11 (search theme reiterated only twice)
Applebee: 5
Stein: 10
NSS: 26
• Introduction:4
• CharacterDev:4
• MentalStates:2
• Referencing:4
• ConflictRes:4
• Cohesion:4

151. • Conclusion:4
Sample 2:
Once there was a little boy named Tom. He had a little frog and
a dog. One night, when Tom
and his dog were sleeping, the frog crept away. When Tom and
his dog leaned over the next
morning, the frog was gone. He looked everywhere for the frog.
He called out the window.
When he looked into the jar, his head got stuck in the jar. And
when he leaned over, he fell out
the window. And Tom jumped out and picked him up to see if
he was OK. He gave him a lick.
And then Tom spent the rest of the day looking for his frog. He
called down the hole but there
626 Language Testing 27(4)
was a gopher. The dog barked at a tree and some bees scared
them. Tom looked into a tree hole.
There’s an owl. He climbed up a big rock. And he leaned onto
some big branches. But they
weren’t branches. They were a deer’s antlers. And the deer ran
with Tom on his head. He put
the dog and Tom in the water. And they snuck up very quietly.
And they found his frog and lots
of other frogs. He took the baby frog as his new pet. And he
waved bye to his old frogs.
Plot & Theme: 11 (search theme reiterated only once)
Applebee: 5
Stein: 10
NSS: 21

152. • Introduction:3
• CharacterDev:3
• MentalStates:2
• Referencing:3
• ConflictRes:3
• Cohesion:4
• Conclusion:3
Sample 3:
The boy was looking for his frog. All day he looked for the frog
and couldn’t find him.
Finally a beehive, the dog barked at a beehive in the tree. And
the dog got in trouble. And
so did the boy because the gopher and the owl. And ran away.
And then he chased the
dog. And he looked out from it. And then he climbed on the
branch. But it wasn’t
branches. It was a deer. And the deer shoved him off of the
cliff. And then he went to the
frog. And he had a family. And then a frog jumped out to get
him. And then he took that
frog home and left his old frog where his old frog was.
Plot & Theme: 5 (Plot: 2 search episodes (beehive and deer
scenes), resolution; Theme: initial
mention that the boy was looking for the frog, one additional
mention of searching for the frog)
Applebee: 3
Stein: 5 (Abbreviated Episode): Story is goal-directed (i.e.,
searching for the frog). But,
there are no complete episodes in the story.
NSS: 14

153. • Introduction: 1
• CharacterDev: 3
• MentalStates: 1
• Referencing: 3
• ConflictRes: 1
• Cohesion: 2
• Conclusion: 3
Copyright of Language Testing is the property of Sage
Publications, Ltd. and its content may not be copied or
emailed to multiple sites or posted to a listserv without the
copyright holder's express written permission.
However, users may print, download, or email articles for
individual use.
The Development of Narrative Identity in Late Adolescence and
Emergent
Adulthood: The Continued Importance of Listeners
Monisha Pasupathi
University of Utah
Timothy Hoyt
University of New Mexico
Research on narrative identity in late adolescence and early
adulthood has not extensively examined how
conversational storytelling affects the development of narrative

154. identity. This is a major gap, given the
importance of this age period for narrative identity development
and the clear importance of parent– child
conversations in the development of narrative identity. The
authors present a series of 3 studies (n � 220)
examining how late adolescents and early adults construct
narrative identity in ways that are shaped by
their listeners. The findings suggest that late adolescents and
early adults construct more meaning-laden,
interpretive accounts of their everyday experiences when they
converse with responsive friends. Further,
even within this sample’s abbreviated age range, the authors
found evidence for age-related increases in
the factual content of personal memories. Such findings
illuminate the importance of friends in the
construction of narrative identity during this key developmental
period.
Keywords: narrative identity development, autobiographical
memory development, parent– child remi-
niscing, social construction
Late adolescence and early adulthood are accorded special sig-
nificance for the development of self and identity (Arnett, 2000;
Dusek & McIntyre, 2003; Erikson & Erikson, 1997; Harter,
1998;
Kroger, 2003), narrative identity (Habermas & Bluck, 2000;
McLean, 2005), and autobiographical memory (Holmes & Con-
way, 1999; Rubin & Schulkind, 1997). In fact, work on the
reminiscence bump suggests that adolescence and early
adulthood
are a privileged developmental period for the encoding of auto-
biographical memories, which in narrative terms constitute the
stuff of which selves are made.
But despite the widespread acknowledgment of the importance

155. of late adolescence and early adulthood for identity
development,
there are some important gaps in existing research. Perhaps
most
notable among these is a relative lack of attention to the micro-
processes by which identity in general and narrative identity in
particular develop during this period. In the present article, our
focus is particularly on the microprocess of conversational
story-
telling as a critical process by which narrative identity more
broadly develops. Further, we show, across three studies, that
having responsive, attentive friends as listeners for
conversational
storytelling helps further narrative identity development in late
adolescence and early adulthood. Next, we outline the
unaddressed
issues in identity development work that can be better examined
with the narrative and, more specifically, conversational
storytell-
ing frameworks.
Identity Development in Adolescence
and Early Adulthood
Erikson and Erikson (1997) originally defined ego identity as
entailing a sense of uniqueness or individuality, an emerging
commitment to a place in society, and a sense of continuity over
time. Much of the extant work on identity development, as well
as work on self-development, has emphasized the uniqueness or
individuality aspect of self and identity by focusing on explicit
self-descriptive statements that participants do or do not en-
dorse. Research in this arena has suggested that identity devel-
opment in terms of exploring alternative identities and commit-
ting to some rather than others is a nonlinear process that
appears to occur primarily in late adolescence and early adult-

156. hood (Constantinople, 1969; Meilman, 1979; Waterman, 1982).
Over adolescence and early adulthood, self-descriptions be-
come increasingly multifaceted and complex, and changes in
the content of self-descriptions also occur into early adulthood
(Dusek & McIntyre, 2003; Harter, 1998; Sutin & Robins, 2005).
The ability to recognize contradictions in the self emerges by
midadolescence, but the capacity for resolving them is a phe-
nomenon of later adolescence (Harter & Monsour, 1992), con-
sistent with increasing cognitive and epistemic abilities across
this period (e.g., Kitchener, Lynch, Fischer, & Wood, 1993).
Contextual factors such as parent– child relationships, peer re-
lationships, and exposure to college and larger sociopolitical
contexts are linked to identity development over this age range
(Dusek & McIntyre, 2003; Hair, Moore, Garrett, Ling, & Cleve-
land, 2008; McNulty & Swann, 1994; Waterman, 1982).
This literature underscores the importance of late adolescence
and early adulthood for identity development but is relatively
Monisha Pasupathi, Department of Psychology, University of
Utah;
Timothy Hoyt, Department of Psychology, University of New
Mexico.
These studies were supported by National Institute of Mental
Health
Grant 1R03MH64462 awarded to Monisha Pasupathi. Our
gratitude is due
to Benjamin Rich, for his extensive efforts in data collection, as
well as to
Martin Cryer and Leslie Rheinhold, who assisted with data
coding. In
addition, Carol Sansone, Kate McLean, and Frank Drews
provided valu-
able comments on earlier drafts of this article.

157. Correspondence concerning this article should be addressed to
Monisha
Pasupathi, Department of Psychology, University of Utah, 390
South 1530
East, Beh-502, Salt Lake City, UT 84112. E-mail:
[email protected]
Developmental Psychology © 2009 American Psychological
Association
2009, Vol. 45, No. 2, 558 –574 0012-1649/09/$12.00 DOI:
10.1037/a0014431
558
uninformative about the microprocesses by which identity is
con-
structed in that age period and does not capture the way in
which
people construct a sense of themselves as continuous across
time.
As an alternative, narrative approaches to identity development
capture both individuality and the continuity aspects of identity,
and their focus on the construction of identity is one step closer
to
an examination of the processes of identity formation.
Narrative Identity in Late Adolescence and Early
Adulthood: Closer to Process
Narrative approaches to identity development examine how
people construct meaning in relation to their experiences and
thus
further their sense of self and identity (McAdams, 1996). The
construction of narratives about past experiences also involves
the

158. creation of a sense of continuity over time—as the past self is
represented and interpreted by the present self. Thus, narrative
conceptualizations of identity hold the promise of integrating
Erikson and Erikson’s (1997) aspects of identity as well as illu-
minating the processes by which individuals develop identity.
For
example, narratives construct personal continuity over time by
linking past to the present and future. They also highlight
individ-
ual uniqueness and, at the same time, make use of cultural
scripts
and schemas for organizing one’s experience in narrative form.
In
fact, proponents of narrative identity research (McLean,
Pasupathi,
& Pals, 2007; Pasupathi, 2001; Thorne, 2000) have argued that
the
process of constructing narratives about the personal past is the
paramount process by which narrative identity in particular, and
self and identity more broadly, develop. In making this case,
they
specifically suggest that narrative identity develops via the
micro-
process of constructing specific narratives in specific situations.
That is, narrative identity emerges out of countless actions of
narrative construction. The study of narrative identity
development
consequently requires the exploration of processes that
influence
narrative identity creation in the moment.
It is important to note that narrative and narrative identity are
not synonymous. Although all stories must provide an account
of
what happened—the setting and actions that occurred— each
sto-

159. ryteller may have a unique sense of how the actions were con-
nected, which were important, and what the broader
implications
and associations of the experience may entail. These
interpretive,
meaning-laden features of a narrative render a set of facts
uniquely
reflective of the individual whose story they compose, and it is
these features that both reflect and construct narrative identity.
Moreover, it is these features that across various operationaliza-
tions are related to macrolevel development of self and identity
in
early childhood (e.g., Bird & Reese, 2006; Bohanek, Marin, &
Fivush, 2006; Fivush, 1991; Fivush, Bohanek, Robertson, &
Duke,
2004; Harley & Reese, 1999; Howe, Courage, & Peterson, 1994)
and in adolescence and early adulthood (McLean & Pratt, 2006;
see also Sutin & Robins, 2005), and to well-being and maturity
throughout adulthood (King & Patterson, 2000; McAdams,
Reyn-
olds, Lewis, Patten, & Bowman, 2001; Pals, 2006a, 2006b).
The overall elaboration of narratives, including both factual and
interpretive information, increases across childhood (e.g., P. J.
Bauer, 2006; Fivush & Nelson, 2004). Little is known about
changes in general elaboration thereafter. Researchers focusing
on
adolescence and adulthood have emphasized broader, global
types
of meanings; these are conceptually connected to interpretive
elaboration. The prevalence of those larger identity-related ele-
ments in narratives increases with age into middle adulthood (J.
J.
Bauer & McAdams, 2004a, 2004b, 2005; Bluck & Glueck, 2004;
McAdams et al., 2006; McLean, 2005; McLean & Thorne, 2003;

160. Pasupathi & Mansour, 2006), and they become more positive
and
more nuanced (McAdams et al., 2006). This work suggests, but
does not explicitly demonstrate, increases in the prevalence and
sophistication of interpretive elaboration in narratives from ado-
lescence into middle adulthood. Moreover, this work is largely
mute about just how such macrolevel increases might come
about;
that is, it does not necessarily investigate the processes by
which
people come to vary in their construction of narrative identity.
Conversational Storytelling: The Process of Constructing
Narrative Identity
Conversations about the personal past with caregivers are a
major force in shaping young children’s ways of constructing
narratives and developing narrative identity (see, e.g., P. J.
Bauer,
2006; Fivush & Nelson, 2004). Some parents engage children in
highly responsive, elaborative conversation about the past,
whereas others are more focused and repetitive (Cleveland &
Reese, 2005; Fivush & Nelson, 2004; Harley & Reese, 1999;
Reese & Fivush, 1993; Wang, 2004). Children with more elabo-
rative mothers subsequently engage in more elaborative remem-
bering with other listeners and about other events. Maternal lis-
tening, then, influences the story created within that
conversation
but over time exerts cumulative, long-term effects on children’s
style of remembering. Converging findings from experimental
(Peterson, Jesso, & McCabe, 1999) and prospective longitudinal
designs (e.g., Haden, Haine, & Fivush, 1997) provide strong
evidence for a causal role for parent– child remembering. For
very
young children like those studied in this work, parental respon-
siveness is important for both factual and interpretive

161. elaboration
in narratives. Further, the development of basic capacities for
narration and the development of narrative identity are tightly
intertwined during early childhood.
Within a burgeoning literature on narrative identity in late
adolescence and early adulthood (e.g., McLean et al., 2007),
however, we know surprisingly little about the role played by
conversational storytelling and the listeners for such stories.
This
is in spite of the acknowledged importance of identity develop-
ment in this age range and the fact that this period coincides
with
a shift from parents as the primary audience for personal story-
telling to friends as an additional, increasingly important
audience
for such stories (McLean, 2005; see also Buhrmeister, 1996;
Cooper, 1999; Daddis, 2008; Hartup, 1996; Hazan & Zeifman,
1994; Updegraff, McHale, & Whiteman, 2006).
Some tiny pieces of evidence from studies of adolescents and
emergent adults suggest that listeners are important in the con-
struction of narrative identity in this age range. For example,
conversational interactions with parents do matter. More open
and
free-ranging conversations with parents are associated with
greater
autonomy and identity development in adolescents (Cooper &
Grotevant, 1985). By early adulthood, friends and parents
appear
to serve similar purposes as listeners for personal events,
according
to retrospective reports (McLean, 2005). Further, conversations
with friends during adolescence are quite strikingly concerned
with identity (Korobov & Bamberg, 2004; Parker & Gottman,

162. 559NARRATIVE IDENTITY DEVELOPMENT
1989), and work with late adolescents suggests that listeners
prefer
narratives that entail more sophisticated identity-related
insights—
indicative of a listener preference for more interpretive
elaboration
(Thorne, McLean, & Lawrence, 2004). Some work with young
adults has linked the action of telling a personal story to
changes
in more traditionally assessed self-conceptions or event percep-
tions and has shown that variations in the way listeners respond
to
that story, or variations in the goals with which the story is told,
correspond to immediate self-perception changes (e.g.,
McGregor
& Holmes, 1999; McLean, 2005; Pasupathi, Alderman, & Shaw,
2007; Pasupathi & Rich, 2005; Thoman, Sansone, & Pasupathi,
2007).
A handful of studies have explicitly looked at variations in
listener behavior and storytelling among young adults (Bavelas,
Coates, & Johnson, 2000; Dickinson & Givon, 1995; Pasupathi,
Stallworth, & Murdoch, 1998). These studies examined story-
telling by young adults to a stranger and experimentally varied
the stranger’s behavior during storytelling. That variation cre-
ated either distracted or disagreeable listeners; the comparison
conditions always involved a responsive, agreeable listener.
Generally, responsive and agreeable listeners concurrently
elicit more elaborative stories than do distracted or disagreeable
listeners, whether about brief film stimuli (Dickinson & Givon,
1995; Pasupathi et al., 1998) or about personal experiences
(Bavelas et al., 2000). In fact, listeners in the responsive and

163. agreeable conditions behave more elaboratively than do listen-
ers in the distracted conditions, in these studies. That is, they
ask questions, make sympathetic exclamations, demonstrate
that they have understood the speaker’s point, and follow
the speaker’s interests rather than impose their own agenda.
These behaviors are quite similar to those observed among
more elaborative mothers in work on parent– child reminiscing.
One goal of the present studies was to demonstrate that the
development of narrative identity is affected by the process of
storytelling, by showing that the extent to which late adoles-
cents and early adults construct elaborative, richly detailed
personal stories is connected to the behavior of their listening
friends.
But as we noted earlier, not all elaborations are equally
relevant for narrative identity. Interpretive information in par-
ticular makes memory into narrative identity by tying recalled
experiences to the individual’s goals, feelings, thoughts, and
beliefs. In childhood, when memory conversations involve
learning how to remember, responsive listening is clearly linked
to elaboration of all types of information, although for self-
development the elaboration of interpretive or evaluative con-
tent is more important. By contrast, in adulthood, responsive
listening may be of particular importance for interpretive infor-
mation. In fact, Bavelas et al. (2000) showed that distracted
(and therefore unresponsive) listeners were selectively impaired
in their capacity to respond to the meaning of what storytellers
were saying, although they did not examine whether this cor-
responded to a selective suppression of storytellers’ inclusion of
such meanings. Thus, our second aim was to examine whether
responsive friends were especially important in facilitating the
construction of richly interpretive accounts of personal experi-
ences, that is, in helping late adolescents infuse memory with
narrative identity.
Overview of the Present Studies

164. The major goal of the present studies was to provide laboratory-
based evidence for the microprocess. Critically, these studies
were
not designed to address the macrodevelopmental process of nar-
rative identity process but, rather, to address the critical
micropro-
cess building block for narrative identity—the process of
conver-
sational storytelling. Other work suggests that if responsive
listening has immediate importance for narrative identity in the
short term, it will also have long-term implications for
macrolevel
self-development, an issue to which we return in the discussion
of
our findings.
The present studies examine in vivo storytelling about personal
experiences to close, same-gender friends by late adolescents
and
early adults (defined as the age range from 18 to 35 years). We
chose to examine storytelling to same-gender close friends
because
narrative identity suggests that close friends and family are the
primary audience for personal storytelling in adulthood and be-
cause close friends and parents appear to serve similar roles as
listeners among late adolescents and early adults (McLean,
2005).
Restricting the focus to same-gender friendships allowed us to
reduce the possible impact of different relationships between
lis-
teners and speakers (i.e., romantic vs. primarily platonic
relations).
We compared responsive and unresponsive (distracted) friends
during storytelling across three studies, examining how
variations

165. in responsiveness influenced the elaboration of factual and
inter-
pretive aspects of storytelling.
We chose experimental methods because exploratory work in
our laboratory suggested that under normal conditions, most
friends are fairly responsive listeners in a laboratory setting in
which there are few distractions and their defined task is to
listen.
Experimentally creating an unresponsive listener via distraction
ensured that some of our participants would not function like
highly elaborative mothers by asking questions or expressing
strong interest. Experimental methods also permit stronger con-
clusions about causality and link the present studies to work on
young adults.
In addition to examining our major goals, we also examined two
other issues in the present studies. First, although age
differences,
particularly in this restricted age range, were not a primary
focus
of our study, we examined whether age was associated with
increases in the elaboration of factual or interpretive
information,
given other findings on autobiographical memory development
during this age period. Second, by late adolescence and beyond,
people also have the capacity to engage in deliberate, strategic
self-presentation (e.g., Baumeister, Stilman, & Wotman, 1990;
Tice, 1992). Such deliberate self-presentation would suggest
that
people strategically change their stories, and strategic changes
in
narration imply a different type of identity-related implication
than
do changes in narrative identity construction that occur without
a