WASC ARC ConferenceApril 8, 2011, San Francisco, CAImproving Student Writing and Strengthening Writing Programs –Technolog...
California State University, Fresno             (est. 1911)                 •23-campus CSU system                 •20,932 ...
Fresno State Demographics         Copyright © April 2011   3
Fresno State Demographics§   First generation    college: 68%§   English proficiency:    63% freshmen require    English r...
Multi-pronged Approach toTeaching Writing on our Campus¨   English Composition¨   Upper Division Writing Exam    ¡   Gradu...
The Dilemma¨   High percentage of remedial students.¨   Perception that surface errors distract    from content.¨   Effect...
INSTRUCTORS KEEP     ASKING…         “Where Is        the Time to        Respond To        All of Those         Essays?”
TECHNOLOGY AS ONE    SOLUTION?        •Professor   Kim Morin          •“E-scholar”   Program          •UpperDivision onlin...
A DISCOVERY…Computer EssayScoring ProgramsSeveral College-levelEssay Scoring andWriting programsavailable…
COMPUTER ESSAY SCORING      PROGRAMS •   ETS® Criterion® •   http://www.ets.org/criterion/higher_ed/about •   IEA Intellig...
ETS CRITERION® SELECTED   ON A TRIAL BASIS…        First Trial (Spring 2009):        30 students – 1 essay        assignme...
About ETS Criterion®•   Students write and revise essays    online•   Diagnostic Feedback / Holistic Score    within 20 se...
ABOUT ETS CRITERION®•   No instructor Fees•   Simple Registration     • Students purchase Access codes through       Books...
ABOUT ETS CRITERION®•   Bilingual feedback available         • Spanish, Japanese, Simplified           Chinese, Korean, EL...
ETS CRITERION® Online Tourhttp://www.ets.org/Media/Products/Criterion/tour2/critloader.html
Initial Presentation to Faculty     “Computer Essay Scoring has had a       positive effect so far.     I spend more time ...
Early Student Comments¡   “I like the instant feedback.”¡   “I like that it goes into depth about why you    didnt get the...
Facts About College Student WritingFACT # 1:Student improvements in writing increase with more   practice¨   specific, con...
FACT # 2:If students do not continue to practice,their writing performance may actually deteriorate.IMPLICATION:ü Examine ...
FACT # 3:Teaching students to writeeffectively can be time-consuming and labor-intensive.IMPLICATION:ü Identify effective ...
FACT # 4:Writing instruction involves faculty who are NOT  trained as writing teachers.IMPLICATION:ü Implement Criterion t...
FACT # 5:Assessing & documenting student learning outcomes in  writing performance are key elements to success.IMPLICATION...
Next Came…ISWI Pilot Launch      Spring 2010               Copyright © April 2011   23
ISWI  Improving Student Writing          InitiativeNovember, 2009 - Campus email announcement  calling for faculty partici...
Campus-Wide Initiative              By The Numbers...              Year 1 (2 Semesters)            Spring 2010 / Fall 2010...
Campus-Wide Initiative              By The Numbers...               Year 2 (1 Semester)                 Spring 2011•   231...
Faculty Participation- Year 2Out of 93 instructors opting to use Criterion, only20 received Professional Development funds...
Assessments from Spring 2010..1.    Analyzed Criterion® automatically generated      data for trends.2.    CLA scores for ...
Automatically-Generated Data¡   Students who used Criterion® for revision    with more than one submission, scored    bett...
Collegiate Learning Assessment             (CLA) ¡   A significant difference was found     between Criterion® users and n...
Faculty Scoring PanelFirst and last papers submitted by students   on Criterion® scored by independent   faculty panel.¨  ...
Faculty Feedback –         Positive CommentsWhen using Criterion®, most faculty felt that:1 . Papers were easier to grade....
Faculty Feedback - ChallengesMany faculty felt that Criterion®  1.  Did not reduce their workload.  2 . Did not do as much...
Student Survey Response        Copyright © April 2011   34
Student Survey HighlightsWhat students liked about Criterion®:1.   It was available on-line 24 hours a day.2.   It allowed...
What students found challenging: 1.   Criterion® identified technical terms or      citations as errors. 2.   The program ...
Student Survey ResponseFreshmen/sophomores were  significantly more likely to agree:     - Criterion has helped me improve...
ESL Student Survey Response          Copyright © April 2011   38
English as a Second Language        Student ResponsesESL students were more likely to respondthat Criterion® helped them i...
English as a Second Language      Student Response    http://www.youtube.com/watch?v=pP6feJotLVM                 Copyright...
Positive Comments From Students¨   “Awesome program. I find it really useful.”¨   “I like that it is really convenient and...
Ongoing Research¨   Studies with Control Groups¨   Continue Collecting Data¨   Determine Best Practices / Uses            ...
ConclusionSo Far, Evidence suggests-¨   Criterion® helps improve student writing    when combined with effective instructo...
Next Steps¨   Analyze and refine assessment data.¨   Provide online video tutorials.¨   Expand Professional Development & ...
What Have We Learned?¨   Students generally found Criterion® to be easy    to use and cost effective.¨   ESL students resp...
ISWI Broadly Endorsed & Supported By:¨   Writing Competency Subcommittee (subcommittee of Senate’s GE Committee)¨   Profes...
¨   William Covino, Provost                        ¨     Kim Morin, Professor¨   wcovino@csufresno.edu                    ...
Contact Information- ETS®                                                            Arthur RuzzanoETS® Criterion®        ...
Upcoming SlideShare
Loading in …5
×

Wasc arc conf final, iswi, april 8 2011.pptx

819 views

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
819
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Wasc arc conf final, iswi, april 8 2011.pptx

  1. 1. WASC ARC ConferenceApril 8, 2011, San Francisco, CAImproving Student Writing and Strengthening Writing Programs –Technology and Techniques That Work Ellen Junn, Associate Provost Jennifer Ivie, Assistant Professor of Psychology Kim Morin, Professor of Theatre Arts William Covino, Provost California State University, Fresno
  2. 2. California State University, Fresno (est. 1911) •23-campus CSU system •20,932 students •1,197 faculty Copyright © April 2011 2
  3. 3. Fresno State Demographics Copyright © April 2011 3
  4. 4. Fresno State Demographics§ First generation college: 68%§ English proficiency: 63% freshmen require English remediation Copyright © April 2011 4
  5. 5. Multi-pronged Approach toTeaching Writing on our Campus¨ English Composition¨ Upper Division Writing Exam ¡ Graduation Requirement ¡ Substitution of Designated “W” courses¨ Writing Requirement for ALL General Education Courses ú Requires “Iterative” writing assignments Copyright © April 2011 5
  6. 6. The Dilemma¨ High percentage of remedial students.¨ Perception that surface errors distract from content.¨ Effective writing instruction requires innovative pedagogies. Copyright © April 2011 6
  7. 7. INSTRUCTORS KEEP ASKING… “Where Is the Time to Respond To All of Those Essays?”
  8. 8. TECHNOLOGY AS ONE SOLUTION? •Professor Kim Morin •“E-scholar” Program •UpperDivision online GE courses •Faculty interest across campus
  9. 9. A DISCOVERY…Computer EssayScoring ProgramsSeveral College-levelEssay Scoring andWriting programsavailable…
  10. 10. COMPUTER ESSAY SCORING PROGRAMS • ETS® Criterion® • http://www.ets.org/criterion/higher_ed/about • IEA Intelligent Essay Assessor http://www.knowledge-technologies.com/prodIEA.shtml • SAGrader • https://www.sagrader.com/sgm/features • Pearson MyWriting Lab • http://www.mywritinglab.com/whatis.html
  11. 11. ETS CRITERION® SELECTED ON A TRIAL BASIS… First Trial (Spring 2009): 30 students – 1 essay assignment First Pilot (Fall 2009) 1 Faculty member 100 students- 5 essay assignments
  12. 12. About ETS Criterion®• Students write and revise essays online• Diagnostic Feedback / Holistic Score within 20 seconds.• Topics Library provides prompts.• Faculty can create topics/prompts.• Errors are highlighted but not corrected. Copyright © April 2011 12
  13. 13. ABOUT ETS CRITERION®• No instructor Fees• Simple Registration • Students purchase Access codes through Bookstore • Approximately $11.00 per student • One fee provides student use in all classes per term/semester• Technical Support from ETS®
  14. 14. ABOUT ETS CRITERION®• Bilingual feedback available • Spanish, Japanese, Simplified Chinese, Korean, ELL• Advanced levels available • College–1st & 2nd year, TOEFL, GRE• No prior essays required
  15. 15. ETS CRITERION® Online Tourhttp://www.ets.org/Media/Products/Criterion/tour2/critloader.html
  16. 16. Initial Presentation to Faculty “Computer Essay Scoring has had a positive effect so far. I spend more time assessing content, less on grammatical errors. Students spend more time revising. However, it still misses errors and does not grade for content.” Copyright © April 2011 16
  17. 17. Early Student Comments¡ “I like the instant feedback.”¡ “I like that it goes into depth about why you didnt get the max score or why you did well.”¡ “It is easy to use - just copy and paste!”¡ “I love the fact that I can revise my work for a better score.”¡ “It is very helpful and I can see my growth as a writer.” Copyright © April 2011 17
  18. 18. Facts About College Student WritingFACT # 1:Student improvements in writing increase with more practice¨ specific, constructive feedback from faculty¨ opportunity for revisions.IMPLICATION:ü Students need to write or revise more frequentlyü Examine policy requiring “iterative” writing in GE or W courses Copyright © April 2011 18
  19. 19. FACT # 2:If students do not continue to practice,their writing performance may actually deteriorate.IMPLICATION:ü Examine all writing programs across campusü Identify key courses with writing requirementü Target specific faculty teaching those courses Copyright © April 2011 19
  20. 20. FACT # 3:Teaching students to writeeffectively can be time-consuming and labor-intensive.IMPLICATION:ü Identify effective technology-related tools ü reduce faculty workload ü provide specific, timely feedback to students.ü Target part-time faculty who teach writing intensive courses. Copyright © April 2011 20
  21. 21. FACT # 4:Writing instruction involves faculty who are NOT trained as writing teachers.IMPLICATION:ü Implement Criterion training for faculty.ü Offer Writing Across the Curriculum workshops.ü Provide professional development funds for all faculty during Year 1 Pilot. Copyright © April 2011 21
  22. 22. FACT # 5:Assessing & documenting student learning outcomes in writing performance are key elements to success.IMPLICATION:ü Ongoing data collected for past 3 semestersü Share results with faculty learning community to determine Best Practices.ü Modify training based on feedback and assessment. Copyright © April 2011 22
  23. 23. Next Came…ISWI Pilot Launch Spring 2010 Copyright © April 2011 23
  24. 24. ISWI Improving Student Writing InitiativeNovember, 2009 - Campus email announcement calling for faculty participants§ Criterion® Training: § Faculty ISWI Coordinator § Faculty Learning Community (FLC)§ Writing Across the Curriculum Workshops§ Assessment of Criterion® Copyright © April 2011 24
  25. 25. Campus-Wide Initiative By The Numbers... Year 1 (2 Semesters) Spring 2010 / Fall 2010• 349 Classes Involved• 173 Instructors Used Criterion® • 134 Spring 2010 • 68 Fall 2010• 5,920 Students (Spring 2010)• 3,756 Students (Fall 2010)   Numbers may include duplicates Copyright © April 2011 25
  26. 26. Campus-Wide Initiative By The Numbers... Year 2 (1 Semester) Spring 2011• 231 Classes Involved• 93 Instructors Used Criterion®• 5442 Students (Spring 2011)• 44,080 Essays Submitted (by March 15)   Numbers may include duplicates Copyright © April 2011 26
  27. 27. Faculty Participation- Year 2Out of 93 instructors opting to use Criterion, only20 received Professional Development funds asan incentive in Year 2. Copyright © April 2011 27
  28. 28. Assessments from Spring 2010..1. Analyzed Criterion® automatically generated data for trends.2. CLA scores for students with & without Criterion®3. First and last papers submitted by students on Criterion® scored by independent faculty panel4. Collected student and April 2011 surveys Copyright © faculty 28
  29. 29. Automatically-Generated Data¡ Students who used Criterion® for revision with more than one submission, scored better than those who did not.¡ Students with a larger number of assignments and more submissions on Criterion® increased their holistic score on average by approximately 1 level. Copyright © April 2011 29
  30. 30. Collegiate Learning Assessment (CLA) ¡ A significant difference was found between Criterion® users and non- users on the CLA performance task. ¡ No significant differences were found between the two groups on the analytic writing task. Copyright © April 2011 30
  31. 31. Faculty Scoring PanelFirst and last papers submitted by students on Criterion® scored by independent faculty panel.¨ Overall, student writing improved in classes that used ETS Criterion®. ¡ The greatest improvements were made when instructors engaged students in substantial discussions of writing and how to use Criterion®. Copyright © April 2011 31
  32. 32. Faculty Feedback – Positive CommentsWhen using Criterion®, most faculty felt that:1 . Papers were easier to grade.2 . The program improved their students’ writing skills.3 . Students spent more time revising written assignments.4 . Creating assignments was easy. Copyright © April 2011 32
  33. 33. Faculty Feedback - ChallengesMany faculty felt that Criterion® 1. Did not reduce their workload. 2 . Did not do as much as they had hoped. 3 . Was not worth the cost to the students. Copyright © April 2011 33
  34. 34. Student Survey Response Copyright © April 2011 34
  35. 35. Student Survey HighlightsWhat students liked about Criterion®:1. It was available on-line 24 hours a day.2. It allowed them to correct grammatical or mechanical errors before turning in a paper.3. It gave immediate feedback. Copyright © April 2011 35
  36. 36. What students found challenging: 1. Criterion® identified technical terms or citations as errors. 2. The program identified errors but did not correct them. 1. The program did not grade for content. Copyright © April 2011 36
  37. 37. Student Survey ResponseFreshmen/sophomores were significantly more likely to agree: - Criterion has helped me improve my writing. - Criterion should suggest less and correct more. - I was able to apply what I learned to other writing assignments. Copyright © April 2011 37
  38. 38. ESL Student Survey Response Copyright © April 2011 38
  39. 39. English as a Second Language Student ResponsesESL students were more likely to respondthat Criterion® helped them improve theirwriting úPreferred Criterion® feedback to instructor feedback. úSubmitted their documents more frequently. - I like the kind of feedback Criterion® gives. - I spent more time improving my writing with Criterion®. - I wish I could use Criterion® for other classes. Copyright © April 2011 39
  40. 40. English as a Second Language Student Response http://www.youtube.com/watch?v=pP6feJotLVM Copyright © April 2011 40
  41. 41. Positive Comments From Students¨ “Awesome program. I find it really useful.”¨ “I like that it is really convenient and I found myself using the program for all my classes.”¨ “I started noticing themes in my writing that could use improvement.” Copyright © April 2011 41
  42. 42. Ongoing Research¨ Studies with Control Groups¨ Continue Collecting Data¨ Determine Best Practices / Uses Copyright © April 2011 42
  43. 43. ConclusionSo Far, Evidence suggests-¨ Criterion® helps improve student writing when combined with effective instructor practice.¨ Criterion® appears useful for editing grammar and mechanics.¨ Students who use Criterion® are more engaged with the writing process. Copyright © April 2011 43
  44. 44. Next Steps¨ Analyze and refine assessment data.¨ Provide online video tutorials.¨ Expand Professional Development & training.¨ Establish ISWI committee ú analyze writing instruction across campus.¨ Recognize and thank participating faculty. Copyright © April 2011 44
  45. 45. What Have We Learned?¨ Students generally found Criterion® to be easy to use and cost effective.¨ ESL students responded more positively to Criterion® feedback.¨ Faculty found Criterion® helpful when used to complement instruction.¨ ISWI shows the high degree of faculty interest in improving student writing across campus. Copyright © April 2011 45
  46. 46. ISWI Broadly Endorsed & Supported By:¨ Writing Competency Subcommittee (subcommittee of Senate’s GE Committee)¨ Professional Development Subcommittee (subcommittee of Senate’s Personnel Committee)¨ Chair of the GE Committee¨ Office of Undergraduate Studies¨ Division of Graduate Studies¨ Institutional Research & Assessment Planning Director leading faculty ISWI Assessment Team¨ Provost’s Office as implemented by the Associate Provost through the Center for the Scholarly Advancement of Learning and Teaching (CSALT) & Technology Innovations for Learning and Teaching (TILT) Copyright © April 2011 46
  47. 47. ¨ William Covino, Provost ¨ Kim Morin, Professor¨ wcovino@csufresno.edu ¨ Department of Theatre Arts¨ 559-278-2636 ¨ Artistic Director, Theatre for Young Audiences; English/Drama¨   Credential Advisor¨ Ellen Junn, Associate Provost ¨ ISWI Faculty Coordinator¨ Interim Director, Center for the ¨ kimm@csufresno.edu Scholarly Advancement of Learning ¨ 559-278-4342 and Teaching (CSALT) ¨  ¨ Interim Senior Academic ¨ Jennifer Ivie, Assistant Professor Technology Officer (SATO) ¨ Department of Psychology and Technology Innovations for ¨ Interim Director, Center for Learning and Teaching (TILT) the Scholarly Advancement of¨ ejunn@csufresno.edu Learning and Teaching (CSALT)¨ 559-278-2636 Thank You! jivie@csufresno.edu ¨ ¨ 559-278-2842 Copyright © April 2011 47
  48. 48. Contact Information- ETS® Arthur RuzzanoETS® Criterion® Western Regional Manager¨ http://www.ets.org/criterion Higher Education Assessment Solutions Phone: 310.944.4034¨ Susan L. Yetman Fax: 609.683.2040¨ Criterion Account Manager Email: aruzzano@ets.org¨ Higher Education, ETS¨ Rosedale Road MS 51-L Bill Wynne¨ Princeton, NJ 08541 Product Manager - ETS Proficiency Profile¨ Direct line:  609.683.2675 ETS Programs and Services¨ Toll free:      866.717.1915 Division¨ Fax:                609.683.204 Telephone (609) 683-2006 0 E-mail wwynne@ets.org¨ Email: syetman@ets.org Copyright © April 2011 48

×