Public lecture given in Sydney Australia on May 4, 2018 critical of the Australian NAPLAN Writing Task

1. TOWARDS A NEW NAPLAN:
TESTING TO THE TEACHING
Les Perelman, PhD
Massachusetts Institute of Technology
TOWARDS A NEW NAPLAN:
TESTING TO THE TEACHING

3. It is the responsibility of intellectuals to speak the
truth and to expose lies.

4. Topics
1. Briefly identify fundamental differences between speech and
writing to identify some fundamental features of the writing
construct.
2. Summarize 5 other Anglophone writing assessments, 3 at similar
Years and having similar purposes to the NAPLAN writing task and 2
Year 12 with interesting features
3. Analyse the NAPLAN Writing Task
a) In comparison to these other tests
b) In terms of its own Marking Guide
4. Offer some suggestions for a process for developing a new NAPLAN
writing task and, possibly, the other components of a new NAPLAN

5. Nature of Speech & Writing
Speech
Innate human characteristic
Writing
Relatively recent technology

6. Acquisition
Speech
• Exposure to language
Reading / Writing
• Instruction

7. Simultaneous production & reception
Speech
Yes
Writing
No

8. Opportunity for invisible revision
Speech
• No
Writing
• Yes

9. Permanence
Speech
• Transitory
Writing
• Relatively permanent

10. Communication channels
Speech
• Multiple
Writing
• Single

11. Opportunity for audience
to respond to communicator
Speech
• Yes
Writing
• No

12. Differences between speech and writing help identify
some principal characteristics of the writing construct
• Writing employs a single communication channel
• There is the expectation of more care in planning and
revision as well as in adherence to conventions
• Revision is invisible to the reader
• The writer must construct the audience and anticipate
possible reactions to elements in the text
• Still, although humans develop speech and writing in very
different ways, the analogy between listening : speaking /
reading : writing both exists and is useful

13. BABCOCK & WILCOX COMPANY
I R GENERATION GROUP
August 3, 1978
TO: B.A. Karrasch, Manager, Plant Integration
FROM: D.F. Hallman, Manager, Plant Performance Services Section (1149)
SUBJECT: Operator Interruption of High Pressure Injection(HPI)
References:
(1) B.M. Dunn to J. Taylor, Operator Interruption of High Pressure Injection, 9
February 1978
(2) B.M. Dunn to J. Taylor, Operator Interruption of High Pressure Injection, 16
February 1978
References 1 and 2 (attached) recommend a change in Babcock and Wilcox's philosophy
for HPI system use during low-pressure transients. Basically, they recommend leaving
the HPI pumps on, once HPI has been indicated, until it can be determined that the hot
leg temperature is more than 50~F below Tsat for the RCS pressure.
Nuclear Service believes this mode can cause the RCS (including the pressurizer) to be
solid. The pressurizer reliefs will lift, with a water surge through the discharge piping
into the quench tank.
We believe the following incidents should be evaluated:
If the pressurizer goes solid with one or more HPI pumps continuing to operate, would
there be a pressure spike before the reliefs open which could cause damage to the RCS? 2.
What damage would the water surge through the relief valve discharge piping and
quench tank cause?
To date, Nuclear Service has not notified our operating plants to change HPI policy
consistent with References 1 and 2 because of our above-stated questions. Yet, the
references suggest the possibility of uncovering the core if present HPI policy is
continued. We request that Integration resolve the issue of how the HPI system should be
used. We are available to help as needed.
D.F. Hallman
cc: E.R. Kane
J.D. Phinney
B.W. Street
B.M. Dunn
J.F. Walters
Conveying meaning as the essential function of writing

14. Three-mile Island memorandum 1
Babcock & Wilcox Company
I R Generation Group
August 3, 1978
TO: B. A. Karrasch, Manager, Plant Integration
FROM: D.F. Hallman, Manager, Plant Performance Services Section (1149)
SUBJECT: Operator Interruption of High Pressure Injection (HPI)
References:
(1) B.M. Dunn to J. Taylor, Operator Interruption of High Pressure Injection, 9 February 1978
2) B.M. Dunn to J. Taylor, Operator Interruption of High Pressure Injection, 16 February 1978

15. Three-mile Island memorandum 2
References 1 and 2 (attached) recommend a change in Babcock and Wilcox's philosophy for HPI
system use during low-pressure transients. Basically, they recommend leaving the HPI pumps on,
once HPI has been indicated, until it can be determined that the hot leg temperature is more than
50~F below Tsat for the RCS pressure.
Nuclear Service believes this mode can cause the RCS (including the pressurizer) to be solid. The
pressurizer reliefs will lift, with a water surge through the discharge piping into the quench tank.
We believe the following incidents should be evaluated:
If the pressurizer goes solid with one or more HPI pumps continuing to operate, would there be a
pressure spike before the reliefs open which could cause damage to the RCS? 2. What damage
would the water surge through the relief valve discharge piping and quench tank cause?

16. Three-mile Island memorandum 3
To date, Nuclear Service has not notified our operating plants
to change HPI policy consistent with References 1 and 2
because of our above-stated questions. Yet, the references
suggest the possibility of uncovering the core if present HPI
policy is continued. We request that Integration resolve the
issue of how the HPI system should be used. We are available
to help as needed.
D.F. Hallman

17. Writing is not as much a body of knowledge as
it is an art, a craft, a technē
Homo Faber rather than Homo Sapiens
Skills develop in spurts, rather than linearly

18. Anglophone Writing Tests

19. Criteria for selection of six other Anglophone
Writing Assessments
• Information publicly available on web
• Representative of some of the various Anglophone educational systems
• Three K-12 assessments selected as performing one or more of the
following functions
1. as an indicator of school performance for use by school administrators in
improving schools
2. as a snapshot of national student achievement that can be used for longitudinal
comparison
3. as formative feedback to students and parents on student writing skills
• Three Year 12 “Leaving” Assessments that possessed one or more features
of interest (two will be discussed today)

20. National Assessment of Educational Progress
(NAEP) The Nation’s Report Card USA
NAEP Writing Test
• Population samples
• Two 30 minutes tests
• Marked holistically
• Two out of three genres
• to persuade, in order to change the reader’s point of view or affect
the reader’s action
• to explain, in order to expand the reader’s understanding
• to convey experience, real or imagined, in order to communicate
individual and imagined experience to others.
• Mix favors narrative in lower Years and persuasive and
informative writing in higher Years
• Developed by teachers and educational experts
• Students can use dictionaries
• Audience and task defined

21. Smarter Balanced Assessment Consortium
Smarter Balanced
• 20 US States members or affiliates
• 3 Traits for composite score
• Organization/Purpose
• Evidence/Elaboration
• Conventions ( ½ weight of other 2 traits)
• 120 Minutes
• Audience defined
• Online dictionary and thesaurus
• Reading test consists of short written responses
• Reading questions prewriting for extended writing
task

22. Foundation Skills Assessment
British Columbia
• Choice of reading topic
• Classroom paired exercise precedes reading test
• Short written answers involving abstraction and complex
thought
• Extended writing task
• Short general prompt
• 45 minutes (but more time if needed)
• Composite of holistic and 4 trait marks
• Meaning
• Style
• Form
• Conventions
• Dictionary allowed
• Reflective writing at end of test

23. New Optional SAT Essay
• Rhetorical and stylistic analysis
• 3 independent marks
• Reading
• Analysis
• Writing
• Conventions a component of mark
• 50 minutes
• No dictionaries

24. Australian Capital Territory Scaling Test
Writing Section
• Open-ended prompt
• 600 word maximum
• Significant reading
• Holistic scale
• Multiple markers
• Dictionary allowed

25. Common features (especially first three tests)
• Significant connection between writing and reading
• Significant time given
• Audience definition
• Dictionary available
• Prompts and scoring rubrics matched to Year
• Conventions considered but not predominant
• Conveying meaning considered essential function

26. Essential Process Features
• Transparency
• Inclusion
• Alignment

28. One size fits all --Almost
• One prompt for all!
• Two prompts for all!
• One rubric for all years
• Year 3 is marked on the same criteria as Year 9

30. No transparency in NAPLAN Design

31. Relative Weight of Categories
0 1 2 3 4 5 6
Spelling
Punctuation
Sentence…
Paragraphing
Cohesion
Vocabulary
Persuasive…
Ideas
Text Structure
Audience
Meaning Uncertain Conventions

32. 10 Categories: 4 Scales
Audience 0-6
Text Structure 0-4
Ideas 0-5
Persuasive Devices 0-4
Vocabulary 0-5
Cohesion 0-4
Paragraphing 0-3
Sentence Structure 0-6
Punctuation 0-5
Spelling 0-6

33. Estimate markers need to make a mark every
27 seconds

34. My favourite challenging spelling words
• accelerating
• adrenaline
• annihilate
• appreciate
• belligerence
• biodegradable
• blasé
• brusque
• camouflage
• debris
• effervescent
• euphoric
• fluorescent
• guillotine
• haemoglobin
• litigious
• mathematician
• narcissist
• psychiatrist
• resuscitate
• ricochet
• zephyr

35. The effervescent, blasé, and euphoric psychiatrist
resuscitated the camouflaged and florescent
guillotine to annihilate the belligerent and litigious
mathematician whose accelerating adrenaline
ricocheted the biodegradable haemoglobin.

36. Undefined Audience
?

37. The exemplary script
It is important for human beings to set aside time for leisure and recreational
activities in order to relax and enjoy themselves. However, it is not abnormal
for people to become obsessed by such activities and spend too much time
on them. As a teenager / adolescent, the reality is, a lot of time and money
will often be spent on video games or toys for younger children. I believe
that money spent on such things should be regulated. . .
In conclusion, I believe it is important to allocate time and money for toys
and games, however, everything must be done in moderation, and it is an
important role of parents / guardians to ensure that time and money spent
on these activities is regulated.

38. Annotations [excerpts]
Category Marks Comments
Audience 6 of 6 Text establishes strong, credible voice. Values and attitudes are revealed. Writing is crafted to
influence reader through sustained use of persuasive techniques. Control of language choices
enables the reader to concentrate on the issues being presented.
Persuasive Devices 4 of 4 Effectively uses and maintains a range of persuasive devices appropriate to the style of
argument: authoritative statements (It is important), personal opinions and experiences (I
still believe humans...; This is ironic because I ...), modality (everything should be done in
moderation), paired conjunctions for emphasis (not only… but also), repetition (cost money,
and that money is money gone…).
Vocabulary 5 of 5 Language choice is well matched to the reasoned style of argument: leisure and recreational
activities, more productive activities, expand your social network, incur, used in moderation,
frame of mind, develop bad habits, allocate time and money, ensure, regulated.
Spelling 6 of 6 Correct spelling of all words. Text meets requirements for Category 6.
Difficult words correct: recreational, activities, reality, regulated, mentioned, participate,
guardians, purchase, ironic, culprit, productive, healthy, ability, incur, moderation, allocate,
develop, facet, conclusion, ensure. Challenging words correct: leisure, adolescent, obsessed,
physically, unnecessary.

39. Dr. Perelman’s Guide
1. Memorize the attached list of Difficult and Challenging Spelling Words and sprinkle them
throughout paper. Feel free to repeat them, and do not worry very much about meaning.
2. If you are not sure about how to spell a word, do not use it.
3. Repeat the language and ideas in the Writing Task throughout the paper.
4. Begin at least one sentence with the structure, “Although x(sentence), y(sentence).” For example:
“Although these instructions are stupid, they will produce a high mark on the NAPLAN.”
5. Master the 5-paragraph form
a. Have a minimum of four paragraphs, preferably five.
b. Each paragraph, except for the last one, should have a minimum of four sentences. Do not
worry about repeating ideas.
c. The first paragraph should end with your thesis sentence.
d. The next-to-last paragraph should modify your thesis sentence by taking the other side of the
issue in special cases.
e. The last paragraph should begin with “In conclusion” and then repeat the thesis sentence from
the first paragraph. Then just repeat two or three ideas from the other paragraphs.
6. Increase your score on the “Audience” and “Persuasive Devices” score by addressing the reader by
using “you” and ask questions. For example: “So you think you wouldn’t mind writing a stupid
essay?”
7. Use connective (Velcro) words like “Moreover, “However,” “In addition,” “On the other hand” at
the beginning of sentences.

40. Dr. Perelman’s Guide (continued)
8. Begin sentences with phrases like “In my opinion,” “I believe that,” “I think that”
etc.
9. Repeat words and phrases throughout the paper.
10. Employ the passive voice frequently throughout your paper.
11. Use referential pronouns, such as this, without a reference noun following it.
For example, “This will make the marker think you are a coherent writer.”
12. Make arguments using forms such “We all believe that we should do X” or “we
all know that Y is harmful.”
13. Always have at least one, preferably two adjectives next to nouns. Thus, not
“the dog” but “the frisky and playful dog.”
14. If you are writing a narrative essay, think quickly if there is a television program,
movie, or story that you know that fits the requirements of the narrative
writing task. If there is one, use it as your narrative, embellishing it or changing
it as much as you want. Markers are explicitly instructed to ignore any stories
or plots they recognize and mark the script on its own merits as if it was
original.

41. Dr. Perelman’s Guide (conclusion)
15. Never write like this except for writing tests
like the NAPLAN.

42. Towards a New Naplan
• Transparency
• Inclusion
• Alignment

43. Transparency

44. Inclusion

45. Non-Alignment
Assessment
Curriculum
Syllabus-
Based
Teaching

46. Alignment
Syllabus-
based
teaching
AssessmentCurriculum

47. Teacher / Student / Curriculum Centered
Assessment

48. Decentralized Development and Grading
Opportunities for Professional Development

49. Online Grading Transcends Space

50. Test to the Teaching
• Different tests and testing communities for different
approaches and populations
• Technology enabled
• Diverse and linked group of readers
• Opportunity to address problems of low-performing
minorities
• Show students that their writing will be read by a diverse
group of readers

51. Towards a New Naplan
• Transparency
• Inclusion
• Alignment