SlideShare a Scribd company logo
Comparative Revision Writing:
Quantitative & Qualitative Research on Students’
Perspectives of Multimodal Instructor Feedback
Way Jeng
Tialitha Macklin
Mark Triana
Stacy Wittstock
10:50-11:05 – Introductions & Ti Macklin
11:05-11:10 – Questions
11:10-11:30 – Mark Triana & Stacy
Wittstock
11:30 -11:35 – Questions
11:35-11:50 – Way Jeng
11:50-11:55 – Questions
11:55-12:05 – Overall Questions
Panel Overview
Study
Context
Response du Jour
Methods
o Fall 13 - Spring 14
o First-Year Composition
o 143 Student
Participants
o 3 Teacher Participants
o 9 Sections
o Written Control Groups
o Same Assignments
o Same Response Style
o Free Software for
Teachers & Students
o Adobe Reader
o Dropbox
o iAnnotate
o Pre- & Post-Study
Survey
o Collected One Set of
Essays
Student Response Preferences
11%
3%
55%
32%
0%
17%
2%
40%
26%
14%
10%
3%
64%
23%
0%0
0.175
0.35
0.525
0.7
Margins End Combo Conferences Audio
Pre-Study (N=120) Post-Study Audio (N=42) Post-Study Written (N=78)
Teacher Influence on Preferences
10
%
30
%
10
%
50
%
9%
63
%
23
%
5%
40
%
10
%
50
% Audio Group
21
%
79
%
21
%
65
%
14
%
4%
4%
60
%
32
%
Written Group
Teacher 2Teacher 1 Teacher 3
Student Narrative Feedback
Reasons for
Rejecting Audio
Response
o Unfamiliar
o Learning Styles
o Prevalence of Global
Commentary
o Time
o Technical Difficulties
Reasons for Selecting
Audio Response
o Clarity
o Learning Styles
o Prevalence of Global
Commentary
Overall Finding
But What About Revision?
Methods
o Fall 15 - Spring 16
o Same Data Set as Previous
Study (minus a few)
o 3 Comp Teachers (not part
of data set)
o 90 Essay Sets (redacted,
txt files)
o Removed Citation Pages
o Inter- Rater Reliability (4
essay sets)
o Training
o Google Docs
o Text Compare
o CWPA Grant
o Faigley, L., & Witte, S.
(1981). Analyzing Revision.
College Composition and
Communication, 32(4),
400–414.
Meaning Classifications
• Surface Change: No change to meaning at all. All Formal change types are
Surface changes.
• "You pay two dollars" → "You pay a dollar entrance fee.”
• "Because the horse lost his shoe the rider was lost." → "Because the
horse lost his shoe, the rider was lost."
• Local Change: These changes are fairly small. While they add, remove, or alter
information, the basic idea of the text remains the same. A summary
description of the text would not change, based on this revision.
• Global Change: These changes are substantial. They may have an effect on the
text globally. A summary of the text would change, based on this revision.
Type Classifications
- Addition: Make explicit what could earlier be inferred or was unknown
- Removal: Make implicit what was earlier explicit or make missing entirely
- Substitutions: Re-phrasing by using different words
- Permutations: Re-phrasing by using the same words
- Distributions: Material in one text unit is passed into more than one
- Consolidations: Material in multiple text units is compressed into fewer units
- Move: Material is moved whole
- Formal: Changes that affect only the form of text, and do not have the
potential to affect fundamental meaning. Often, these will be textual
features associated with a style guide or grammar rule
- Grammar: Grammatical elements (e.g., clause structure, punctuation) is
changed – syntactic structure
Coding Guide
Surface Local Global
Addition SA LA GA
Removal SR LR GR
Substitution SS LS GS
Permutation SP LP GP
Distribution SD LD GD
Consolidation SC LC GC
Move SM LM GM
Grammar SG LG GG
Formal SF
Rater Experience
&
Pedagogical
Implications
Coding Methodology
- Skimmed through each essay in the set
- Understanding the general gist of each essay helped with detecting
Global changes
- Worked in small, manageable samples, going paragraph by
paragraph in Text-Compare
- Worked consistently on the project as much as possible
- Long gaps between working on the project made it more likely that I
might change how I coded something
- Explained difficult coding decisions
- I felt that this might help the researchers understand coding
differences between raters
Text-Compare
Inter-Rater Reliability
- Similar institutional backgrounds
- All raters taught in the same program at WSU
- All raters were either current or former graduate students at WSU
- All raters experienced the same professional and teacher
development efforts at WSU
- Training also fostered uniformity
- Practiced coding sample essay sets together
- Encouraged discussion of disagreements over coding decisions
Impact on Assessment
- Feedback during the semester—increased focus on Global
revision through screencast feedback
- Less is more!
- Teaching revision—increased focus on helping students to
understand the difference between Global vs. Local vs. Surface
changes
- Quality vs. Quantity in revision work
- Assessment—looking for Global vs. Local vs. Surface changes
- Did the student actually revise? Or did he/she proofread?
- What did I give feedback on? Basing assessment on what I actually
commented on.
Cultivating Peer Review Literacies
- Establishing a complex yet manageable vocabulary
- What is the value (or potential) of asking students to itemize,
compartmentalize, and classify their individual revisions as well as
those made by their peers?
Cultivating Peer Review Literacies
- Putting distinctions between individual revisions and
personalized revision strategies into practice
- How might we go about designing more focused, formative, and
diverse writing and peer review activities that target particular kinds
of revision (global, local, and surface)?
Cultivating Peer Review Literacies
- Constructing peer review ecologies predicated on reflection and
meta-cognition
- To what extent can we frame peer review as a complex genre of
writing, governed as much by rigid conceptual frameworks as
subjective or holistic responses to writing as a whole?
Statistical Analysis
&
Overall Findings
A Brief Note About Inter-Rater
Reliability
This study used 3 different coders who looked at essays to produce
data
- What if they disagreed?
- Inter-rater reliability (agreement) was measured by having all 3
raters look at several essays, then comparing their coding to each
other
- Numerically, this is reported as Cronbach’s Alpha, .962
- Generally consider a very high value
- Intraclass correlation measured at .895
- Suggests that there were patterns to the coding of each paper, meaning that
it is like that we are observing a true construct
- Because Inter-rater reliability is so high, this study treats the coders
as effectively identical for purposes of statistical analysis
The Elephant in the Room
What about comparative
statistics?
- T-tests?
- Statistical significance?
The short version: There are
no statistically significant
differences between
conditions on any variable
we have tested so far
Case Study: Word Count
How much did word count change from first draft to second?
- Looking at all students, across all teachers
- Written condition: Mean=120.93, SD=277.64, N=40
- Verbal condition: Mean=219.42, SD =290.85, N=50
- t stat=1.672, t-critical (2-tail)=1.987
- p value (alpha)=0.098
- There is an almost 10% chance that the difference in the means is purely
the result of chance
But How Can that Be Chance?
The word count change was almost double in the verbal condition!
- But variance (standard deviation) was SUPER high
- Essay 3299: 2390 -> 3097 (+707 words added total)
- Unusual, but not unheard of for a final portfolio
- Essay 3329: 2971 -> 2235 (-736 words removed total)
- Probably an essay that had a lot of unnecessary parts
- Removing these words might have indeed been a good idea!
What does this mean?
- Sometimes an observed difference can be the product of random
chance
- Even a fair coin will have streaks
- There’s almost a 10% chance that the difference in these data is an
illusion
What is a Standard Deviation?
A measure of variance
- How much data vary
- “3, 3, 3, 3” vs “1, 1, 5, 5”
- 2/3s of all samples will fall within 1 standard deviation of the
mean
- Plus or minus
Why are they important?
- A high standard deviation indicates that students are revising in
very different ways
So, What Does This Mean?
Which form of feedback is better?
- Right now, there is no clear answer
- Student revisions are roughly equivalent
- With the high variance in the data, small effects are difficult to
detect
- Further research with larger data collection is advised
- Teachers can/should use whatever they like better
- Teachers with a preference aren’t clearly disadvantaging students
Do student preferences line up with improved performance?
- Numbers of students with a clear preference for verbal (audio)
feedback were simply too rare in this sample to say either way
So, What Does This Mean?
Why are revisions so varied?
- Probably because the reasons to revise are so varied
- No single kind of revision accounts for
- No single pattern of revision can (or should) be seen as “right”
This matches the writing process itself
- If the writing process is highly contextual and individualistic,
then the revision process will be similarly contextual and
individualistic
But Didn’t You Find Anything?
So far, only 1 statistically significant relationship has been found
- Instructor 3’s students exhibited lower word count change in
the audio condition, as compared to the written
- Mean audio: 6.17% (N=22, SD= 12.4)
- Mean written: 22.2% (N=20, SD= 8.6)
Were there any other effects for Instructor 3’s students?
- So far, no
- Analysis of Instructor 3’s student revision for type of revision has
not shown any other statistically significant differences
- The kinds of revision were similar, but the size of revisions was
smaller
What Does That Mean?
Possibly nothing
- Instructor 3’s students may have been outliers
- There’s still almost a 3% chance this is not a real effect
- A larger data collection would really help
Possibly, it means instructors shouldn’t be forced to use a certain
style
- Instructor 3 disliked audio feedback
- Maybe the students made smaller revisions as a result
- Maybe aversion has more of an effect on pedagogy than
preference
What Does This Mean for Future
Research?
Revision research requires massive data collection
- This study collected 90 essays from 3 instructors, yet samples
were still too small
- Probably 400-500 essays is the minimum for a study of this
scope
Institutional support is needed
- Coding this quantity of data is very labor intensive
- Getting enough instructors and students to participate is
difficult
Overall Finding
Comparative Revision Writing:
Quantitative & Qualitative Research on
Students’ Perspectives of Multimodal
Instructor Feedback
Ti Macklin – Tialitha.Macklin@csus.edu
Mark Triana – Mark.Triana@wsu.edu
Stacy Wittstock – Snwittstock@ucdavis.edu
Way Jeng – Wjeng@pierce.ctc.edu

More Related Content

What's hot

1.2 principles of test design: plenary CTS-Academic
1.2 principles of test design: plenary CTS-Academic1.2 principles of test design: plenary CTS-Academic
1.2 principles of test design: plenary CTS-Academic
SeltAcademy
 
Language testing the social dimension
Language testing  the social dimensionLanguage testing  the social dimension
Language testing the social dimension
Amir Hamid Forough Ameri
 
Reliability in Language Testing
Reliability in Language Testing Reliability in Language Testing
Reliability in Language Testing
Seray Tanyer
 
Test standardization and norming
Test standardization and normingTest standardization and norming
Test standardization and norming
Hannah Grace Gilo
 
Day 10 prediction and regression
Day 10 prediction and regressionDay 10 prediction and regression
Day 10 prediction and regression
Elih Sutisna Yanto
 
Testing in language programs (chapter 8)
Testing in language programs (chapter 8)Testing in language programs (chapter 8)
Testing in language programs (chapter 8)
Tahere Bakhshi
 
LENGUAGE TESTING (II Bimestre Abril Agosto 2011)
LENGUAGE TESTING (II Bimestre Abril Agosto 2011)LENGUAGE TESTING (II Bimestre Abril Agosto 2011)
LENGUAGE TESTING (II Bimestre Abril Agosto 2011)
Videoconferencias UTPL
 
Characteristics of a good test
Characteristics of a good testCharacteristics of a good test
Characteristics of a good testcyrilcoscos
 
Standards based classroom assessments of english proficiency
Standards based classroom  assessments of english proficiencyStandards based classroom  assessments of english proficiency
Standards based classroom assessments of english proficiency
Amir Hamid Forough Ameri
 
Testing for Language Teachers Arthur Hughes
Testing for Language TeachersArthur HughesTesting for Language TeachersArthur Hughes
Testing for Language Teachers Arthur HughesRajputt Ainee
 
Day 9 hypothesis and correlation for students
Day 9 hypothesis and correlation for studentsDay 9 hypothesis and correlation for students
Day 9 hypothesis and correlation for students
Elih Sutisna Yanto
 
CHARACTERISTICS OF A GOOD INSTRUMENT
CHARACTERISTICS OF A GOOD INSTRUMENTCHARACTERISTICS OF A GOOD INSTRUMENT
CHARACTERISTICS OF A GOOD INSTRUMENT
Musfera Nara Vadia
 
How to make tests more reliable
How to make tests more reliableHow to make tests more reliable
How to make tests more reliableNawaphat Deelert
 
7 measurement & questionnaires design (Dr. Mai,2014)
7 measurement & questionnaires design (Dr. Mai,2014)7 measurement & questionnaires design (Dr. Mai,2014)
7 measurement & questionnaires design (Dr. Mai,2014)
Phong Đá
 
Reliability and validity of Research Data
Reliability and validity of Research DataReliability and validity of Research Data
Reliability and validity of Research Data
Aida Arifin
 
20100820080859 lecture 7,8,9,10( steps of proposal0
20100820080859 lecture  7,8,9,10( steps of proposal020100820080859 lecture  7,8,9,10( steps of proposal0
20100820080859 lecture 7,8,9,10( steps of proposal0peningla
 
Reliability and validity
Reliability and validityReliability and validity
Reliability and validity
shobhitsaxena67
 
UTPL-LANGUAGE TESTING-II-BIMESTRE-(OCTUBRE 2011-FEBRERO 2012)
UTPL-LANGUAGE TESTING-II-BIMESTRE-(OCTUBRE 2011-FEBRERO 2012)UTPL-LANGUAGE TESTING-II-BIMESTRE-(OCTUBRE 2011-FEBRERO 2012)
UTPL-LANGUAGE TESTING-II-BIMESTRE-(OCTUBRE 2011-FEBRERO 2012)
Videoconferencias UTPL
 
LANGUAJE TESTING
LANGUAJE TESTINGLANGUAJE TESTING
LANGUAJE TESTING
Videoconferencias UTPL
 

What's hot (20)

1.2 principles of test design: plenary CTS-Academic
1.2 principles of test design: plenary CTS-Academic1.2 principles of test design: plenary CTS-Academic
1.2 principles of test design: plenary CTS-Academic
 
Language testing the social dimension
Language testing  the social dimensionLanguage testing  the social dimension
Language testing the social dimension
 
Reliability in Language Testing
Reliability in Language Testing Reliability in Language Testing
Reliability in Language Testing
 
Test standardization and norming
Test standardization and normingTest standardization and norming
Test standardization and norming
 
Day 10 prediction and regression
Day 10 prediction and regressionDay 10 prediction and regression
Day 10 prediction and regression
 
Testing in language programs (chapter 8)
Testing in language programs (chapter 8)Testing in language programs (chapter 8)
Testing in language programs (chapter 8)
 
LENGUAGE TESTING (II Bimestre Abril Agosto 2011)
LENGUAGE TESTING (II Bimestre Abril Agosto 2011)LENGUAGE TESTING (II Bimestre Abril Agosto 2011)
LENGUAGE TESTING (II Bimestre Abril Agosto 2011)
 
Characteristics of a good test
Characteristics of a good testCharacteristics of a good test
Characteristics of a good test
 
Standards based classroom assessments of english proficiency
Standards based classroom  assessments of english proficiencyStandards based classroom  assessments of english proficiency
Standards based classroom assessments of english proficiency
 
Testing for Language Teachers Arthur Hughes
Testing for Language TeachersArthur HughesTesting for Language TeachersArthur Hughes
Testing for Language Teachers Arthur Hughes
 
Day 9 hypothesis and correlation for students
Day 9 hypothesis and correlation for studentsDay 9 hypothesis and correlation for students
Day 9 hypothesis and correlation for students
 
CHARACTERISTICS OF A GOOD INSTRUMENT
CHARACTERISTICS OF A GOOD INSTRUMENTCHARACTERISTICS OF A GOOD INSTRUMENT
CHARACTERISTICS OF A GOOD INSTRUMENT
 
How to make tests more reliable
How to make tests more reliableHow to make tests more reliable
How to make tests more reliable
 
Test validity
Test validityTest validity
Test validity
 
7 measurement & questionnaires design (Dr. Mai,2014)
7 measurement & questionnaires design (Dr. Mai,2014)7 measurement & questionnaires design (Dr. Mai,2014)
7 measurement & questionnaires design (Dr. Mai,2014)
 
Reliability and validity of Research Data
Reliability and validity of Research DataReliability and validity of Research Data
Reliability and validity of Research Data
 
20100820080859 lecture 7,8,9,10( steps of proposal0
20100820080859 lecture  7,8,9,10( steps of proposal020100820080859 lecture  7,8,9,10( steps of proposal0
20100820080859 lecture 7,8,9,10( steps of proposal0
 
Reliability and validity
Reliability and validityReliability and validity
Reliability and validity
 
UTPL-LANGUAGE TESTING-II-BIMESTRE-(OCTUBRE 2011-FEBRERO 2012)
UTPL-LANGUAGE TESTING-II-BIMESTRE-(OCTUBRE 2011-FEBRERO 2012)UTPL-LANGUAGE TESTING-II-BIMESTRE-(OCTUBRE 2011-FEBRERO 2012)
UTPL-LANGUAGE TESTING-II-BIMESTRE-(OCTUBRE 2011-FEBRERO 2012)
 
LANGUAJE TESTING
LANGUAJE TESTINGLANGUAJE TESTING
LANGUAJE TESTING
 

Viewers also liked

Comparative essay practice classical civs
Comparative essay practice classical civsComparative essay practice classical civs
Comparative essay practice classical civsAshley Birmingham
 
Writing a Comparative Essay
Writing a Comparative EssayWriting a Comparative Essay
Writing a Comparative Essay
mburdett10
 
Descriptive writing
Descriptive writingDescriptive writing
Descriptive writing
cafeharmon
 
Types of writing
Types of writingTypes of writing
Types of writing
pdbrans
 
Argumentative essay ppt
Argumentative essay pptArgumentative essay ppt
Argumentative essay ppt
jdsmithii
 
Descriptive writing 1
Descriptive writing 1Descriptive writing 1
Descriptive writing 1mzickefoose2
 
Introduction to descriptive writing
Introduction to descriptive writingIntroduction to descriptive writing
Introduction to descriptive writingenglishteacher-tnt
 
Compare and-Contrast Essay Writing
Compare and-Contrast Essay WritingCompare and-Contrast Essay Writing
Compare and-Contrast Essay WritingUniversity of York
 
Compare and Contrast Analysis Writing
Compare and Contrast Analysis WritingCompare and Contrast Analysis Writing
Compare and Contrast Analysis Writingmmerrin2
 
Comparison and contrast essay
Comparison and contrast essayComparison and contrast essay
Comparison and contrast essayUTPL UTPL
 

Viewers also liked (11)

Comparative essay practice classical civs
Comparative essay practice classical civsComparative essay practice classical civs
Comparative essay practice classical civs
 
Writing a Comparative Essay
Writing a Comparative EssayWriting a Comparative Essay
Writing a Comparative Essay
 
Descriptive writing
Descriptive writingDescriptive writing
Descriptive writing
 
Types of writing
Types of writingTypes of writing
Types of writing
 
Argumentative essay ppt
Argumentative essay pptArgumentative essay ppt
Argumentative essay ppt
 
Descriptive writing 1
Descriptive writing 1Descriptive writing 1
Descriptive writing 1
 
Introduction to descriptive writing
Introduction to descriptive writingIntroduction to descriptive writing
Introduction to descriptive writing
 
Descriptive Essay Writing
Descriptive Essay WritingDescriptive Essay Writing
Descriptive Essay Writing
 
Compare and-Contrast Essay Writing
Compare and-Contrast Essay WritingCompare and-Contrast Essay Writing
Compare and-Contrast Essay Writing
 
Compare and Contrast Analysis Writing
Compare and Contrast Analysis WritingCompare and Contrast Analysis Writing
Compare and Contrast Analysis Writing
 
Comparison and contrast essay
Comparison and contrast essayComparison and contrast essay
Comparison and contrast essay
 

Similar to Cwpa 2016 comparative revision writing

Pg dissertations writing up your findings and discussion webinar
Pg dissertations   writing up your findings and discussion webinarPg dissertations   writing up your findings and discussion webinar
Pg dissertations writing up your findings and discussion webinar
RhianWynWilliams
 
Fostering Curriculum Development Through Artesol 2010
Fostering Curriculum Development Through Artesol 2010Fostering Curriculum Development Through Artesol 2010
Fostering Curriculum Development Through Artesol 2010
shierl
 
Student Evaluations in Physics - A Study on Gender Bias at the University of ...
Student Evaluations in Physics - A Study on Gender Bias at the University of ...Student Evaluations in Physics - A Study on Gender Bias at the University of ...
Student Evaluations in Physics - A Study on Gender Bias at the University of ...Joshua Brothers
 
Scoring Guide for Rhetorical Analysis (10 of grade; 100 po.docx
 Scoring Guide for Rhetorical Analysis (10 of grade; 100 po.docx Scoring Guide for Rhetorical Analysis (10 of grade; 100 po.docx
Scoring Guide for Rhetorical Analysis (10 of grade; 100 po.docx
aryan532920
 
Combining General and Genre-Specific Approaches to L2 Writing Instruction
Combining General and Genre-Specific Approaches to L2 Writing InstructionCombining General and Genre-Specific Approaches to L2 Writing Instruction
Combining General and Genre-Specific Approaches to L2 Writing Instruction
guest05424
 
Pg dissertations writing up your findings and discussion webinar
Pg dissertations   writing up your findings and discussion webinarPg dissertations   writing up your findings and discussion webinar
Pg dissertations writing up your findings and discussion webinar
RhianWynWilliams
 
School of Accounting Trimester 3A 2013 Information Sheet Tes.docx
School of Accounting Trimester 3A 2013 Information Sheet Tes.docxSchool of Accounting Trimester 3A 2013 Information Sheet Tes.docx
School of Accounting Trimester 3A 2013 Information Sheet Tes.docx
kenjordan97598
 
Assessment 15 Annotated
Assessment 15 AnnotatedAssessment 15 Annotated
Assessment 15 Annotated
James Atherton
 
1 Social Science Statistics Project 1 Global Issu.docx
 1 Social Science Statistics  Project 1 Global Issu.docx 1 Social Science Statistics  Project 1 Global Issu.docx
1 Social Science Statistics Project 1 Global Issu.docx
ShiraPrater50
 
1 Social Science Statistics Project 1 Global Issu.docx
1 Social Science Statistics  Project 1 Global Issu.docx1 Social Science Statistics  Project 1 Global Issu.docx
1 Social Science Statistics Project 1 Global Issu.docx
poulterbarbara
 
TESTA, Universtiy of Warwick SCAP Conference (July 2013)
TESTA, Universtiy of Warwick SCAP Conference (July 2013)TESTA, Universtiy of Warwick SCAP Conference (July 2013)
TESTA, Universtiy of Warwick SCAP Conference (July 2013)
TESTA winch
 
Ug dissertations writing up your findings and discussion
Ug dissertations   writing up your findings and discussionUg dissertations   writing up your findings and discussion
Ug dissertations writing up your findings and discussion
RhianWynWilliams
 
Ug dissertations writing up your findings and discussion
Ug dissertations   writing up your findings and discussionUg dissertations   writing up your findings and discussion
Ug dissertations writing up your findings and discussion
RhianWynWilliams
 
Designing Qualitative Research.pdf
Designing Qualitative Research.pdfDesigning Qualitative Research.pdf
Designing Qualitative Research.pdf
sdfghj21
 
Causal comparative n survey
Causal comparative n surveyCausal comparative n survey
Causal comparative n surveyDhiya Lara
 
Aaa assessment 2Assessment description.pdfCritical Thinki.docx
Aaa assessment 2Assessment description.pdfCritical Thinki.docxAaa assessment 2Assessment description.pdfCritical Thinki.docx
Aaa assessment 2Assessment description.pdfCritical Thinki.docx
bartholomeocoombs
 
Bc week 9
Bc week 9Bc week 9
Bc week 9
Beth Carey
 
clinical research writing services
clinical research writing servicesclinical research writing services
clinical research writing services
Pubricahealthcare
 
clinical research writing services
clinical research writing services clinical research writing services
clinical research writing services
Pubricahealthcare
 
clinical research writing services
clinical research writing services clinical research writing services
clinical research writing services
Pubricahealthcare
 

Similar to Cwpa 2016 comparative revision writing (20)

Pg dissertations writing up your findings and discussion webinar
Pg dissertations   writing up your findings and discussion webinarPg dissertations   writing up your findings and discussion webinar
Pg dissertations writing up your findings and discussion webinar
 
Fostering Curriculum Development Through Artesol 2010
Fostering Curriculum Development Through Artesol 2010Fostering Curriculum Development Through Artesol 2010
Fostering Curriculum Development Through Artesol 2010
 
Student Evaluations in Physics - A Study on Gender Bias at the University of ...
Student Evaluations in Physics - A Study on Gender Bias at the University of ...Student Evaluations in Physics - A Study on Gender Bias at the University of ...
Student Evaluations in Physics - A Study on Gender Bias at the University of ...
 
Scoring Guide for Rhetorical Analysis (10 of grade; 100 po.docx
 Scoring Guide for Rhetorical Analysis (10 of grade; 100 po.docx Scoring Guide for Rhetorical Analysis (10 of grade; 100 po.docx
Scoring Guide for Rhetorical Analysis (10 of grade; 100 po.docx
 
Combining General and Genre-Specific Approaches to L2 Writing Instruction
Combining General and Genre-Specific Approaches to L2 Writing InstructionCombining General and Genre-Specific Approaches to L2 Writing Instruction
Combining General and Genre-Specific Approaches to L2 Writing Instruction
 
Pg dissertations writing up your findings and discussion webinar
Pg dissertations   writing up your findings and discussion webinarPg dissertations   writing up your findings and discussion webinar
Pg dissertations writing up your findings and discussion webinar
 
School of Accounting Trimester 3A 2013 Information Sheet Tes.docx
School of Accounting Trimester 3A 2013 Information Sheet Tes.docxSchool of Accounting Trimester 3A 2013 Information Sheet Tes.docx
School of Accounting Trimester 3A 2013 Information Sheet Tes.docx
 
Assessment 15 Annotated
Assessment 15 AnnotatedAssessment 15 Annotated
Assessment 15 Annotated
 
1 Social Science Statistics Project 1 Global Issu.docx
 1 Social Science Statistics  Project 1 Global Issu.docx 1 Social Science Statistics  Project 1 Global Issu.docx
1 Social Science Statistics Project 1 Global Issu.docx
 
1 Social Science Statistics Project 1 Global Issu.docx
1 Social Science Statistics  Project 1 Global Issu.docx1 Social Science Statistics  Project 1 Global Issu.docx
1 Social Science Statistics Project 1 Global Issu.docx
 
TESTA, Universtiy of Warwick SCAP Conference (July 2013)
TESTA, Universtiy of Warwick SCAP Conference (July 2013)TESTA, Universtiy of Warwick SCAP Conference (July 2013)
TESTA, Universtiy of Warwick SCAP Conference (July 2013)
 
Ug dissertations writing up your findings and discussion
Ug dissertations   writing up your findings and discussionUg dissertations   writing up your findings and discussion
Ug dissertations writing up your findings and discussion
 
Ug dissertations writing up your findings and discussion
Ug dissertations   writing up your findings and discussionUg dissertations   writing up your findings and discussion
Ug dissertations writing up your findings and discussion
 
Designing Qualitative Research.pdf
Designing Qualitative Research.pdfDesigning Qualitative Research.pdf
Designing Qualitative Research.pdf
 
Causal comparative n survey
Causal comparative n surveyCausal comparative n survey
Causal comparative n survey
 
Aaa assessment 2Assessment description.pdfCritical Thinki.docx
Aaa assessment 2Assessment description.pdfCritical Thinki.docxAaa assessment 2Assessment description.pdfCritical Thinki.docx
Aaa assessment 2Assessment description.pdfCritical Thinki.docx
 
Bc week 9
Bc week 9Bc week 9
Bc week 9
 
clinical research writing services
clinical research writing servicesclinical research writing services
clinical research writing services
 
clinical research writing services
clinical research writing services clinical research writing services
clinical research writing services
 
clinical research writing services
clinical research writing services clinical research writing services
clinical research writing services
 

More from macktial

CWPA 2015_ Sustaining Teacher and Student Sanity
CWPA 2015_ Sustaining Teacher and Student Sanity CWPA 2015_ Sustaining Teacher and Student Sanity
CWPA 2015_ Sustaining Teacher and Student Sanity
macktial
 
ATL Conference 2015 Voice
ATL Conference 2015 VoiceATL Conference 2015 Voice
ATL Conference 2015 Voice
macktial
 
ATL Conference - Conversational Partnerships
ATL Conference - Conversational PartnershipsATL Conference - Conversational Partnerships
ATL Conference - Conversational Partnerships
macktial
 
CCCC 2015 Presentation - Complicating Risk: First-Generation Self-Identificat...
CCCC 2015 Presentation - Complicating Risk: First-Generation Self-Identificat...CCCC 2015 Presentation - Complicating Risk: First-Generation Self-Identificat...
CCCC 2015 Presentation - Complicating Risk: First-Generation Self-Identificat...
macktial
 
Ti Macklin 2015 C's Presentation
Ti Macklin 2015 C's PresentationTi Macklin 2015 C's Presentation
Ti Macklin 2015 C's Presentation
macktial
 
Macklin 501 presentation ppt
Macklin 501 presentation pptMacklin 501 presentation ppt
Macklin 501 presentation ppt
macktial
 
Beyond Best Practices: The New Frontier of Responding to Student Work
Beyond Best Practices: The New Frontier of Responding to Student WorkBeyond Best Practices: The New Frontier of Responding to Student Work
Beyond Best Practices: The New Frontier of Responding to Student Work
macktial
 
Macklin_CC2014
Macklin_CC2014Macklin_CC2014
Macklin_CC2014
macktial
 

More from macktial (8)

CWPA 2015_ Sustaining Teacher and Student Sanity
CWPA 2015_ Sustaining Teacher and Student Sanity CWPA 2015_ Sustaining Teacher and Student Sanity
CWPA 2015_ Sustaining Teacher and Student Sanity
 
ATL Conference 2015 Voice
ATL Conference 2015 VoiceATL Conference 2015 Voice
ATL Conference 2015 Voice
 
ATL Conference - Conversational Partnerships
ATL Conference - Conversational PartnershipsATL Conference - Conversational Partnerships
ATL Conference - Conversational Partnerships
 
CCCC 2015 Presentation - Complicating Risk: First-Generation Self-Identificat...
CCCC 2015 Presentation - Complicating Risk: First-Generation Self-Identificat...CCCC 2015 Presentation - Complicating Risk: First-Generation Self-Identificat...
CCCC 2015 Presentation - Complicating Risk: First-Generation Self-Identificat...
 
Ti Macklin 2015 C's Presentation
Ti Macklin 2015 C's PresentationTi Macklin 2015 C's Presentation
Ti Macklin 2015 C's Presentation
 
Macklin 501 presentation ppt
Macklin 501 presentation pptMacklin 501 presentation ppt
Macklin 501 presentation ppt
 
Beyond Best Practices: The New Frontier of Responding to Student Work
Beyond Best Practices: The New Frontier of Responding to Student WorkBeyond Best Practices: The New Frontier of Responding to Student Work
Beyond Best Practices: The New Frontier of Responding to Student Work
 
Macklin_CC2014
Macklin_CC2014Macklin_CC2014
Macklin_CC2014
 

Recently uploaded

The basics of sentences session 5pptx.pptx
The basics of sentences session 5pptx.pptxThe basics of sentences session 5pptx.pptx
The basics of sentences session 5pptx.pptx
heathfieldcps1
 
Instructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptxInstructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptx
Jheel Barad
 
Overview on Edible Vaccine: Pros & Cons with Mechanism
Overview on Edible Vaccine: Pros & Cons with MechanismOverview on Edible Vaccine: Pros & Cons with Mechanism
Overview on Edible Vaccine: Pros & Cons with Mechanism
DeeptiGupta154
 
Lapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdfLapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdf
Jean Carlos Nunes Paixão
 
The Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptxThe Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptx
DhatriParmar
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
siemaillard
 
Synthetic Fiber Construction in lab .pptx
Synthetic Fiber Construction in lab .pptxSynthetic Fiber Construction in lab .pptx
Synthetic Fiber Construction in lab .pptx
Pavel ( NSTU)
 
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup   New Member Orientation and Q&A (May 2024).pdfWelcome to TechSoup   New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
TechSoup
 
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
EugeneSaldivar
 
Biological Screening of Herbal Drugs in detailed.
Biological Screening of Herbal Drugs in detailed.Biological Screening of Herbal Drugs in detailed.
Biological Screening of Herbal Drugs in detailed.
Ashokrao Mane college of Pharmacy Peth-Vadgaon
 
The Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official PublicationThe Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official Publication
Delapenabediema
 
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...
Levi Shapiro
 
The French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free downloadThe French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free download
Vivekanand Anglo Vedic Academy
 
BÀI TẬP BỔ TRỢ TIẾNG ANH GLOBAL SUCCESS LỚP 3 - CẢ NĂM (CÓ FILE NGHE VÀ ĐÁP Á...
BÀI TẬP BỔ TRỢ TIẾNG ANH GLOBAL SUCCESS LỚP 3 - CẢ NĂM (CÓ FILE NGHE VÀ ĐÁP Á...BÀI TẬP BỔ TRỢ TIẾNG ANH GLOBAL SUCCESS LỚP 3 - CẢ NĂM (CÓ FILE NGHE VÀ ĐÁP Á...
BÀI TẬP BỔ TRỢ TIẾNG ANH GLOBAL SUCCESS LỚP 3 - CẢ NĂM (CÓ FILE NGHE VÀ ĐÁP Á...
Nguyen Thanh Tu Collection
 
The Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdfThe Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdf
kaushalkr1407
 
Palestine last event orientationfvgnh .pptx
Palestine last event orientationfvgnh .pptxPalestine last event orientationfvgnh .pptx
Palestine last event orientationfvgnh .pptx
RaedMohamed3
 
The geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideasThe geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideas
GeoBlogs
 
Polish students' mobility in the Czech Republic
Polish students' mobility in the Czech RepublicPolish students' mobility in the Czech Republic
Polish students' mobility in the Czech Republic
Anna Sz.
 
Honest Reviews of Tim Han LMA Course Program.pptx
Honest Reviews of Tim Han LMA Course Program.pptxHonest Reviews of Tim Han LMA Course Program.pptx
Honest Reviews of Tim Han LMA Course Program.pptx
timhan337
 
Supporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptxSupporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptx
Jisc
 

Recently uploaded (20)

The basics of sentences session 5pptx.pptx
The basics of sentences session 5pptx.pptxThe basics of sentences session 5pptx.pptx
The basics of sentences session 5pptx.pptx
 
Instructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptxInstructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptx
 
Overview on Edible Vaccine: Pros & Cons with Mechanism
Overview on Edible Vaccine: Pros & Cons with MechanismOverview on Edible Vaccine: Pros & Cons with Mechanism
Overview on Edible Vaccine: Pros & Cons with Mechanism
 
Lapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdfLapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdf
 
The Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptxThe Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptx
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 
Synthetic Fiber Construction in lab .pptx
Synthetic Fiber Construction in lab .pptxSynthetic Fiber Construction in lab .pptx
Synthetic Fiber Construction in lab .pptx
 
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup   New Member Orientation and Q&A (May 2024).pdfWelcome to TechSoup   New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
 
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
 
Biological Screening of Herbal Drugs in detailed.
Biological Screening of Herbal Drugs in detailed.Biological Screening of Herbal Drugs in detailed.
Biological Screening of Herbal Drugs in detailed.
 
The Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official PublicationThe Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official Publication
 
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...
 
The French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free downloadThe French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free download
 
BÀI TẬP BỔ TRỢ TIẾNG ANH GLOBAL SUCCESS LỚP 3 - CẢ NĂM (CÓ FILE NGHE VÀ ĐÁP Á...
BÀI TẬP BỔ TRỢ TIẾNG ANH GLOBAL SUCCESS LỚP 3 - CẢ NĂM (CÓ FILE NGHE VÀ ĐÁP Á...BÀI TẬP BỔ TRỢ TIẾNG ANH GLOBAL SUCCESS LỚP 3 - CẢ NĂM (CÓ FILE NGHE VÀ ĐÁP Á...
BÀI TẬP BỔ TRỢ TIẾNG ANH GLOBAL SUCCESS LỚP 3 - CẢ NĂM (CÓ FILE NGHE VÀ ĐÁP Á...
 
The Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdfThe Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdf
 
Palestine last event orientationfvgnh .pptx
Palestine last event orientationfvgnh .pptxPalestine last event orientationfvgnh .pptx
Palestine last event orientationfvgnh .pptx
 
The geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideasThe geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideas
 
Polish students' mobility in the Czech Republic
Polish students' mobility in the Czech RepublicPolish students' mobility in the Czech Republic
Polish students' mobility in the Czech Republic
 
Honest Reviews of Tim Han LMA Course Program.pptx
Honest Reviews of Tim Han LMA Course Program.pptxHonest Reviews of Tim Han LMA Course Program.pptx
Honest Reviews of Tim Han LMA Course Program.pptx
 
Supporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptxSupporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptx
 

Cwpa 2016 comparative revision writing

  • 1. Comparative Revision Writing: Quantitative & Qualitative Research on Students’ Perspectives of Multimodal Instructor Feedback Way Jeng Tialitha Macklin Mark Triana Stacy Wittstock
  • 2. 10:50-11:05 – Introductions & Ti Macklin 11:05-11:10 – Questions 11:10-11:30 – Mark Triana & Stacy Wittstock 11:30 -11:35 – Questions 11:35-11:50 – Way Jeng 11:50-11:55 – Questions 11:55-12:05 – Overall Questions Panel Overview
  • 5.
  • 6. Methods o Fall 13 - Spring 14 o First-Year Composition o 143 Student Participants o 3 Teacher Participants o 9 Sections o Written Control Groups o Same Assignments o Same Response Style o Free Software for Teachers & Students o Adobe Reader o Dropbox o iAnnotate o Pre- & Post-Study Survey o Collected One Set of Essays
  • 7. Student Response Preferences 11% 3% 55% 32% 0% 17% 2% 40% 26% 14% 10% 3% 64% 23% 0%0 0.175 0.35 0.525 0.7 Margins End Combo Conferences Audio Pre-Study (N=120) Post-Study Audio (N=42) Post-Study Written (N=78)
  • 8. Teacher Influence on Preferences 10 % 30 % 10 % 50 % 9% 63 % 23 % 5% 40 % 10 % 50 % Audio Group 21 % 79 % 21 % 65 % 14 % 4% 4% 60 % 32 % Written Group Teacher 2Teacher 1 Teacher 3
  • 9. Student Narrative Feedback Reasons for Rejecting Audio Response o Unfamiliar o Learning Styles o Prevalence of Global Commentary o Time o Technical Difficulties Reasons for Selecting Audio Response o Clarity o Learning Styles o Prevalence of Global Commentary
  • 11. But What About Revision?
  • 12. Methods o Fall 15 - Spring 16 o Same Data Set as Previous Study (minus a few) o 3 Comp Teachers (not part of data set) o 90 Essay Sets (redacted, txt files) o Removed Citation Pages o Inter- Rater Reliability (4 essay sets) o Training o Google Docs o Text Compare o CWPA Grant o Faigley, L., & Witte, S. (1981). Analyzing Revision. College Composition and Communication, 32(4), 400–414.
  • 13. Meaning Classifications • Surface Change: No change to meaning at all. All Formal change types are Surface changes. • "You pay two dollars" → "You pay a dollar entrance fee.” • "Because the horse lost his shoe the rider was lost." → "Because the horse lost his shoe, the rider was lost." • Local Change: These changes are fairly small. While they add, remove, or alter information, the basic idea of the text remains the same. A summary description of the text would not change, based on this revision. • Global Change: These changes are substantial. They may have an effect on the text globally. A summary of the text would change, based on this revision.
  • 14. Type Classifications - Addition: Make explicit what could earlier be inferred or was unknown - Removal: Make implicit what was earlier explicit or make missing entirely - Substitutions: Re-phrasing by using different words - Permutations: Re-phrasing by using the same words - Distributions: Material in one text unit is passed into more than one - Consolidations: Material in multiple text units is compressed into fewer units - Move: Material is moved whole - Formal: Changes that affect only the form of text, and do not have the potential to affect fundamental meaning. Often, these will be textual features associated with a style guide or grammar rule - Grammar: Grammatical elements (e.g., clause structure, punctuation) is changed – syntactic structure
  • 15. Coding Guide Surface Local Global Addition SA LA GA Removal SR LR GR Substitution SS LS GS Permutation SP LP GP Distribution SD LD GD Consolidation SC LC GC Move SM LM GM Grammar SG LG GG Formal SF
  • 17. Coding Methodology - Skimmed through each essay in the set - Understanding the general gist of each essay helped with detecting Global changes - Worked in small, manageable samples, going paragraph by paragraph in Text-Compare - Worked consistently on the project as much as possible - Long gaps between working on the project made it more likely that I might change how I coded something - Explained difficult coding decisions - I felt that this might help the researchers understand coding differences between raters
  • 19. Inter-Rater Reliability - Similar institutional backgrounds - All raters taught in the same program at WSU - All raters were either current or former graduate students at WSU - All raters experienced the same professional and teacher development efforts at WSU - Training also fostered uniformity - Practiced coding sample essay sets together - Encouraged discussion of disagreements over coding decisions
  • 20. Impact on Assessment - Feedback during the semester—increased focus on Global revision through screencast feedback - Less is more! - Teaching revision—increased focus on helping students to understand the difference between Global vs. Local vs. Surface changes - Quality vs. Quantity in revision work - Assessment—looking for Global vs. Local vs. Surface changes - Did the student actually revise? Or did he/she proofread? - What did I give feedback on? Basing assessment on what I actually commented on.
  • 21. Cultivating Peer Review Literacies - Establishing a complex yet manageable vocabulary - What is the value (or potential) of asking students to itemize, compartmentalize, and classify their individual revisions as well as those made by their peers?
  • 22. Cultivating Peer Review Literacies - Putting distinctions between individual revisions and personalized revision strategies into practice - How might we go about designing more focused, formative, and diverse writing and peer review activities that target particular kinds of revision (global, local, and surface)?
  • 23. Cultivating Peer Review Literacies - Constructing peer review ecologies predicated on reflection and meta-cognition - To what extent can we frame peer review as a complex genre of writing, governed as much by rigid conceptual frameworks as subjective or holistic responses to writing as a whole?
  • 25. A Brief Note About Inter-Rater Reliability This study used 3 different coders who looked at essays to produce data - What if they disagreed? - Inter-rater reliability (agreement) was measured by having all 3 raters look at several essays, then comparing their coding to each other - Numerically, this is reported as Cronbach’s Alpha, .962 - Generally consider a very high value - Intraclass correlation measured at .895 - Suggests that there were patterns to the coding of each paper, meaning that it is like that we are observing a true construct - Because Inter-rater reliability is so high, this study treats the coders as effectively identical for purposes of statistical analysis
  • 26. The Elephant in the Room What about comparative statistics? - T-tests? - Statistical significance? The short version: There are no statistically significant differences between conditions on any variable we have tested so far
  • 27. Case Study: Word Count How much did word count change from first draft to second? - Looking at all students, across all teachers - Written condition: Mean=120.93, SD=277.64, N=40 - Verbal condition: Mean=219.42, SD =290.85, N=50 - t stat=1.672, t-critical (2-tail)=1.987 - p value (alpha)=0.098 - There is an almost 10% chance that the difference in the means is purely the result of chance
  • 28. But How Can that Be Chance? The word count change was almost double in the verbal condition! - But variance (standard deviation) was SUPER high - Essay 3299: 2390 -> 3097 (+707 words added total) - Unusual, but not unheard of for a final portfolio - Essay 3329: 2971 -> 2235 (-736 words removed total) - Probably an essay that had a lot of unnecessary parts - Removing these words might have indeed been a good idea! What does this mean? - Sometimes an observed difference can be the product of random chance - Even a fair coin will have streaks - There’s almost a 10% chance that the difference in these data is an illusion
  • 29. What is a Standard Deviation? A measure of variance - How much data vary - “3, 3, 3, 3” vs “1, 1, 5, 5” - 2/3s of all samples will fall within 1 standard deviation of the mean - Plus or minus Why are they important? - A high standard deviation indicates that students are revising in very different ways
  • 30. So, What Does This Mean? Which form of feedback is better? - Right now, there is no clear answer - Student revisions are roughly equivalent - With the high variance in the data, small effects are difficult to detect - Further research with larger data collection is advised - Teachers can/should use whatever they like better - Teachers with a preference aren’t clearly disadvantaging students Do student preferences line up with improved performance? - Numbers of students with a clear preference for verbal (audio) feedback were simply too rare in this sample to say either way
  • 31. So, What Does This Mean? Why are revisions so varied? - Probably because the reasons to revise are so varied - No single kind of revision accounts for - No single pattern of revision can (or should) be seen as “right” This matches the writing process itself - If the writing process is highly contextual and individualistic, then the revision process will be similarly contextual and individualistic
  • 32. But Didn’t You Find Anything? So far, only 1 statistically significant relationship has been found - Instructor 3’s students exhibited lower word count change in the audio condition, as compared to the written - Mean audio: 6.17% (N=22, SD= 12.4) - Mean written: 22.2% (N=20, SD= 8.6) Were there any other effects for Instructor 3’s students? - So far, no - Analysis of Instructor 3’s student revision for type of revision has not shown any other statistically significant differences - The kinds of revision were similar, but the size of revisions was smaller
  • 33. What Does That Mean? Possibly nothing - Instructor 3’s students may have been outliers - There’s still almost a 3% chance this is not a real effect - A larger data collection would really help Possibly, it means instructors shouldn’t be forced to use a certain style - Instructor 3 disliked audio feedback - Maybe the students made smaller revisions as a result - Maybe aversion has more of an effect on pedagogy than preference
  • 34. What Does This Mean for Future Research? Revision research requires massive data collection - This study collected 90 essays from 3 instructors, yet samples were still too small - Probably 400-500 essays is the minimum for a study of this scope Institutional support is needed - Coding this quantity of data is very labor intensive - Getting enough instructors and students to participate is difficult
  • 36. Comparative Revision Writing: Quantitative & Qualitative Research on Students’ Perspectives of Multimodal Instructor Feedback Ti Macklin – Tialitha.Macklin@csus.edu Mark Triana – Mark.Triana@wsu.edu Stacy Wittstock – Snwittstock@ucdavis.edu Way Jeng – Wjeng@pierce.ctc.edu