A comprehensive presentation about using SAGrader in large college classes.

1. Enhance Learning in
Large College Classes with SAGrader

2. Overview
• PART 1: WHY AUTOMATED ESSAY GRADING?
• PART 2: WHAT IS SAGRADER?
• PART 3: HOW DOES SAGRADER WORK?
• PART 4: SAGRADER AS A LEARNING ENVIRONMENT
• PART 5: WHAT DOES SAGRADER DO FOR
INSTRUCTORS?
• PART 6: WHAT ASSIGNMENTS DOES IT SUPPORT?
• PART 7: MONITORING STUDENT PROGRESS
• PART 8: HOW DO STUDENTS LIKE IT?
• PART 9: HOW DO INSTRUCTORS LIKE IT?
• PART 10: HOW DO I GET STARTED?
OVERVIEW

3. PART 1
WHY AUTOMATED ESSAY
GRADING?

4. Challenges of Large Classes
• Engaging students with the course
• Limited time/sanity for instructors and TAs
• Most exercises use multiple choice tests
with little or no writing
• Writing is time-consuming, slow feedback
to students, grading consistency
• Difficult to assess higher-level reasoning
or critical thinking
WHY AUTOMATED ESSAY GRADING?

5. A Solution?
Writing Across the Curriculum (WAC)
Benefits
– Students write to learn
– Higher level assessments
– Clearer assessment of student understanding
Limitations for large classes
– Expensive, requiring extensive staff to grade
– Slow grading response time from days to weeks
– Minimal timely feedback for students
WHY AUTOMATED ESSAY GRADING?

6. A Better Solution
• SAGrader makes writing across the
curriculum practical for large classes
• Imagine what it would be like to be able to
have students write as much as you
thought they should with very little time
required to grade their work…
WHY AUTOMATED ESSAY GRADING?

7. Learning Benefits
• Writing helps students learn
– Immediate detailed feedback helps them
revise to improve their score and learn
• Writing can assess higher level
reasoning
– Use concepts, theories, facts to reason about
realistic problems, interpret, and critique
WHY AUTOMATED ESSAY GRADING?

8. Learning Benefits (cont.)
• Frequent feedback
– Grades student essays automatically
– Provides immediate feedback
– Encourages students to revise and learn
• Increased student engagement
– Students frequently interact with course
content out of class
WHY AUTOMATED ESSAY GRADING?

9. Master
Content
Facilitate
Active
Ability to
Learning
Think
Writing
Improve
Ownership
Skill at
&
Expressing
Confidence
Ideas
Critical
Thinking

10. Teaching Benefits
• More writing – higher-level learning
even in large classes
• Reduces grading time – instructors can
spend more time teaching
• Systematic data collection – learn how
to improve your course, or publish
research
WHY AUTOMATED ESSAY GRADING?

11. PART 2
WHAT IS SAGRADER?

12. What is SAGrader?
SAGrader™ is an online learning
environment that automatically grades
student essays by assessing substantive
knowledge and reasoning.
WHAT IS SAGRADER?

13. A Unique Approach
• SAGrader works differently than other
programs by ETS, Pearson, and Vantage
• Other programs focus on writing style
rather than content
• Other programs use statistical models to
identify a “good” essay without being able
to say why
WHAT IS SAGRADER?

14. Statistical Model Rule-Based Expert
System System
WHAT IS SAGRADER?

15. Advantages of SAGrader
• Better fit for writing in content-area classes
• Less cost and time to develop custom
assignments
• Provides detailed, personalized feedback
to students rather than generic ratings
• Customizable for different disciplines and
topics
• Objective and unbiased
WHAT IS SAGRADER?

16. PART 3
HOW DOES SAGRADER
WORK?

17. How SAGrader Grades
• Desired knowledge for an assignment
(the instructor’s rubric) is represented as
a semantic network
• Student essays are examined as
alternative expressions of underlying
knowledge
HOW DOES SAGRADER WORK?

19. What Does SAGrader Grade?
• Assesses students’ expression of course-
specific learning objectives
• Can handle low-level knowledge through
high-level knowledge
• Not just key terms…
– Detects multiple expressions of concepts
– Considers relationships among concepts
– Phrases are examined in context
– Handles negations
HOW DOES SAGRADER WORK?

20. Semantic Network
The desired knowledge for an assignment is
represented as a semantic network.
This example identifies some of the features
distinguishing Mead’s stages.
HOW DOES SAGRADER WORK?

21. Detect Surface Features in Student
Text
Then text submitted by students is analyzed to
detect surface-level features consistent with that
underlying knowledge structure.
“Mead’s stages include the
game stage and the play
stage.”
HOW DOES SAGRADER WORK?

22. Fuzzy Logic Detects Variations in
Expression
SAGrader uses fuzzy logic to recognize complex
combinations of key terms.
Key phrases Text
Take the role of the other “…take the role of the other”
Take the role “…understand how other
Other people feel”
Role …look at it from the other
Point of view person’s point of view”
Viewpoint “…consider what the other
Expects person expects”
expectations
HOW DOES SAGRADER WORK?

23. Includes Relationships
SAGrader does not just look for keywords, but
looks at the relationships among concepts
predicted by the knowledge structure
“Mead’s stages include the
game stage and the play
stage.”
HOW DOES SAGRADER WORK?

24. Words in Context
Words must appear in the appropriate
context to be deemed correct. This permits
more sophisticated reasoning such as
“compare and contrast” questions.
In the preparatory stage children In the adult stage children often
often mimic adults without
understanding, while in the adult
stage they are able to take the
≠ mimic adults without
understanding, while in the
preparatory stage they are able to
role of the generalized other. take the role of the generalized
other.
HOW DOES SAGRADER WORK?

25. Handles Negation
SAGrader recognizes negations when detecting
surface structures.
“Mead’s stages include the
game stage and the play
≠
stage.”
“Mead’s stages do NOT
include the game stage and
the play stage.”
HOW DOES SAGRADER WORK?

26. PART 4
SAGRADER AS A LEARNING
ENVIRONMENT

27. A Tool to Enhance Learning
• SAGrader is much more than an
assessment tool
• SAGrader’s detailed feedback and
opportunity for revision permit students
to:
– significantly improve their scores
– learn more in the process
A LEARNING ENVIRONMENT

28. The “Teachable Moment”
Feedback and Revision Cycle
• Students have just submitted and
received immediate feedback
• They are motivated to improve their
grade
• The information is fresh in their minds
• They have the opportunity to revise and
learn
A LEARNING ENVIRONMENT

30. Students Improve by 20%
By using feedback and revising their work,
students improved their grade by 20% -- before
the instructor examined their essay.
First Draft Final Draft
87 87 91 90
100
69
80 61
60
40
20
0
Students who do not Students who revise All students (100%)
revise (29%) (71%)
A LEARNING ENVIRONMENT

31. Improvement via Revision
Submission 1 Excerpt (Score: 50%) Submission 3 Excerpt (Score: 90%)
Material culture is everything that belongs Material culture is art and material objects
to culture that is tangable. Nonmaterial that belong to a culture. Nonmaterial
culture would be the values, beleifs, and culture would be the
behavior accepted in culture. There are two symbols, language, knowledge, beliefs, val
types of norms in culture. Folkways govern ues, attitudes, and norms accepted in
everyday behavior but are not strictly culture. Norms are the expected behavior
enforced. Mores are more serious, carring in a society. There are two types of norms
greater moral gravity and are strictly in culture. Folkways govern everyday
enforced. behavior, are not morally important, and
are not strictly enforced. Mores are more
serious, carring greater moral
importance, and are strictly enforced.
Values are standards of importance and
rightness in society. Language is a abstract
system that allows people of a society to
communicate. Symbols are arbitrary signs
that stand for something.
A LEARNING ENVIRONMENT

32. Students Revise until their Grade
is Acceptable
A LEARNING ENVIRONMENT

33. Initially At-Risk Students
Can Sometimes “Catch Up”
No HS Course HS Course
100 92 92
Students who had a
90 high school course
80 70 in sociology perform
67
70 significantly better
60 on first drafts
50
(t=1.96, p=.05).
40
30
20 But by final
10 drafts, there is
0 virtually no
First Draft Final Draft
t=1.96, p=.05 t=0.0, p=.997 difference
(t=0.0, p=.997).
A LEARNING ENVIRONMENT

34. Helps Retention and Graduation
Rates
• Whether and how much students learn
determines whether they persist and graduate
from college
• One-half to three-quarters of students who drop
out of college do so during or after the freshman
year. (McClanahan, 2004)
• Improving retention rates in the first year has
institutional payoffs over every year students
remain in the institution. (Levitz et al., 1999)
A LEARNING ENVIRONMENT

35. PART 5
WHAT DOES SAGRADER DO
FOR INSTRUCTORS?

36. Monitor Progress and
Encourage Learning
• Easily compare student submissions to see how
students revise
• Monitor the impact of revisions
• Identify students performing poorly
• Track student or classwide performance on
specific learning objectives
• View individual performance reports
• Track systematic data for improving your
teaching or publishing research
FOR INSTRUCTORS

37. Instructors can also…
• Set up assignment and track student progress
anywhere in the world, 24/7
• View and comment on any student submission
• Respond to student questions or challenges
• Review and override any grade
• Export grades to CSV or Excel
FOR INSTRUCTORS

38. Unexpected Benefits
• Logistics of monitoring course much less difficult
• Revisions permit students to learn
• Helps level the playing field for disadvantaged
students
• Cost-effective even with modest sized classes
• Students take more active role through
challenges
FOR INSTRUCTORS

39. PART 6
WHAT ASSIGNMENTS DOES IT
SUPPORT?

40. SAGrader Assignments…
• Can range from a single short question to entire
term papers
• Can have multiple correct answers
• Can assess more abstract rhetorical goals along
with specific substantive knowledge
• Are suitable for different levels of reasoning
• Can be adapted for different texts
• Are suitable for many disciplines
ASSIGNMENTS

41. Intermediate Constraint Tasks
Analyze historic events during the 20th
Unconstrained
century that most affected the rise of the
constructed responses suburbs
From this article, identify three factors that
Intermediate constraint affected the rise of the suburbs and provide
task evidence for and against the importance of
each.
Which of these factors encouraged the rise
of suburbs because of increased ability to
Multiple choice purchase houses?
constrained (A) – the automobile
(B) – movement of jobs to the suburbs
(C) – the GI Bill
ASSIGNMENTS

42. Possible Answers Must
Constrained and Definable
If students are asked to write about their home town:
• SAGrader can recognize broad issues related to
communities (e.g., issues such as population,
governance, geographic location)
• But the program will not be able to judge whether the
information is correct for any specific community
(unless the knowledgebase includes knowledge about
that town.)
ASSIGNMENTS

43. Question Types that Work Well
• Describe a typology
• Describe a concept
• Describe two or more concepts and their relationship to each
other
• Summarize a theory
• Cover broad issues
• Interpret a passage theoretically
• Summarize a particular researcher or scientist
• Interpret a passage from two or more different theoretical
perspectives
• Summarize a study
• Summarize what is known about a particular concept or issue
ASSIGNMENTS

44. Examples
U.S. History (Steam Power) – Factual Recall
What is the history of steam power? Who built the first steam locomotive? ….
Sociology (College Admissions Argument) – Argument
You’re the head of a college admissions committee and must choose between
three candidates. First, read what the other members have to say and then
make your own argument. You should support one of the candidates and
explain why…..
Psychology (REM Sleep) – Comparisons and Typologies
Describe the stages of the sleep cycle that occur before REM sleep. What
distinguishes REM sleep from other sleep stages?
ASSIGNMENTS

45. Examples
Psychology (Sensation) – Analyze a Chain of Events
Suppose you are looking at a ball. A rich chain of events starts within your
eyes. Explain the details of this chain of events, identify each of its components
and the roles they play, and finally say which components of light contribute to
which components of color.
Biology (Zebra Evolution) - Synthesize
Using you understanding of the processes of mutation and natural
selection, describe how a population of striped zebras might have evolved from
a population of zebras without stripes.
ASSIGNMENTS

46. Question Types that Don’t Work
Well
SAGrader is not useful for grading broadly defined essays
with no specific content focus.
• Loosely Restricted Essays
– What do you think is the most important challenge facing teens
today? Why?
• Expository Short Essays
– If you could change places with another person for a whole
day, who would you change places with and why would you
choose that person?
• Creative Writing Essays
– Freewrite for 5 minutes using this opening line: “Behind her, the
noise escalated...”
ASSIGNMENTS

47. PART 7
MONITORING STUDENT
PROGRESS

48. Monitor Student Progress
• Comparing Submissions Tool
• Monitor Impact of Revisions
• Identify Students in Trouble
• Track Student Performance on Learning
Objectives
• Monitor Timeliness
• Gradebook
• Individual Performance Reports
MONITOR STUDENT PROGRESS

50. Monitor Impact of Revisions
MONITOR STUDENT PROGRESS

51. Identify Students in Trouble
MONITOR STUDENT PROGRESS

52. Track Student Performance on
Learning Objectives
MONITOR STUDENT PROGRESS

53. Monitor Timeliness
MONITOR STUDENT PROGRESS

55. Individual Performance Reports
MONITOR STUDENT PROGRESS

56. PART 8
HOW DO STUDENTS LIKE IT?

57. Student Opinions
strongly agree agree
I like the opportunity to challenge my grade (67%) 32 35
I prefer SAGrader over multiple choice tests (72%) 48 24
SAGrader generally grades my essays fairly (77%) 20 57
Writing essays with SAGrader helps me learn (77%) 28 49
I prefer SAGrader over hand-graded essays (84%) 59 25
I like the detailed, personalized feedback (87%) 58 29
SAGrader grades everyone's essays without bias 47 47
(94%)
I like the immediate feedback (97%) 73 24
I like the opportunity to redo my work (100%) 90 10
0 10 20 30 40 50 60 70 80 90 100
HOW DO STUDENTS LIKE IT?

58. I would rather use SAGrader in a course
than hand-graded assignments.
5.00
4.50
4.00
3.50
3.00
2.50
Fall 1 Fall 2 Spring 1 Spring 2 Summer Fall 1 Fall 2 Spring 1
2009 2009 2010 2010 1 2010 2010 2010 2011
HOW DO STUDENTS LIKE IT?

59. Quality Control
• SAGrader provides very detailed feedback, far more
detailed than other essay grading programs
• But SAGrader, particularly for new assignments, is
an imperfect tool, and may not recognize everything.
• Students tend to expect it to perform perfectly and
are quicker to criticize the program than they would
be to criticize an instructor.
• So it includes quality control features that encourage
students to point out any problems to instructors in
the form of grade challenges
HOW DO STUDENTS LIKE IT?

60. Challenges
• Students are encouraged to challenge if they
believe they were graded wrong.
– If students are right the program can be overridden or
revised to grade correctly and applied to everyone’s
submission
– If students are wrong, instructors and TAs can give them
helpful feedback to get on track.
• When first used, assignments have more
challenges…typically around 5-10% of submissions.
• As program is refined challenges drop to 1-5% and
90% of those are cases where the student
misunderstands
HOW DO STUDENTS LIKE IT?

61. Limitations
• Students think the program is wrong instead of them and
challenge the program more than they would instructors
• Students don’t always realize their revisions are not just
make-work but help them learn and refine knowledge
(analogous to math homework)
• On first use, you need to monitor assignment and
student challenges
• Initial effort to construct essay assignments pays off
sooner for larger classes and assignments that are re-
used
• The more specific the question the more knowledge
SAGrader can use to grade
HOW DO STUDENTS LIKE IT?

62. Managing Student Expectations
• Student challenges offer an important way to
improve the program, particularly for new untested
assignments
• Challenges also provide a way for the program and
the instructor to be responsive to student concerns
• However, in many challenges, just as when students
question their grade from an instructor, students are
often wrong
• It is important to answer those challenges and either
use them to improve the program or explain the
student’s error to help them gain confidence in the
program.
HOW DO STUDENTS LIKE IT?

63. Managing Instructor Expectations
• Instructors need to monitor the grading
process, paying particular attention to challenges.
• New assignments typically require some revision
before they grade effectively.
• Once assignments have been thoroughly tested, the
grading process should need very little intervention.
HOW DO STUDENTS LIKE IT?

64. PART 9
HOW DO INSTRUCTORS
LIKE IT?

65. Ande Johnson
Professor of Psychology at Park University
“I use SAGrader not only to reinforce and
improve student writing, but also to help a
segment of students who are challenging to
reach -- the underprepared students. My
students have also admitted that answering
the questions have forced them to open their
textbooks and to read the material.”
HOW DO INSTRUCTORS LIKE IT?

66. “My less skill-developed students have
reported frustration at the onset of using
SAGrader, and have reported that the
writing gets easier and they have to submit
fewer times across the semester. I've also
observed that their responses over the
semester are becoming more concise yet
more dense/ meaningful. Students have also
reported that their writing in other classes
has improved and knowledge/ skill
transference is important in education.”
HOW DO INSTRUCTORS LIKE IT?

67. “The biggest benefit of using SAGrader rests
with improving student writing and thinking.
The SAGrader system gives the less
prepared students more learning
opportunities without taxing the instructor.
I like the automated scoring and immediate
feedback adapted to the students'
schedule.”
HOW DO INSTRUCTORS LIKE IT?

68. Lloyd Chia
Sociology, University of Missouri
Being able to keep a big class of students
writing throughout a course is
invaluable, something that typically would
not be possible with an instructor and two
TAs for a big class of 290 students like I’m
teaching now.
HOW DO INSTRUCTORS LIKE IT?

69. I also appreciate being able to track which
students are not doing well, or who are
consistently late submitting assignments. It
has given me the opportunity to contact
those students, express concern about their
performance, and in a few cases to
eventually help them get back on track with
their grades.
HOW DO INSTRUCTORS LIKE IT?

70. It provided specific writing instructions for
the students that is more detailed than I
normally have time to [give]. It allowed
students to develop better writing skills
without my needing to be an English
teacher.
Students by and large found SAGrader
really helped them to focus on detail, be
more thorough and precise, master the
information, improve their spelling, etc.
HOW DO INSTRUCTORS LIKE IT?

71. PART 10
HOW DO I GET STARTED?

72. Getting Started…
1. Adopt SAGrader
– order through your bookstore along with any textbooks.
2. Set up assignments
– Develop rubrics for assignments.
– SAGrader staff convert rubric to semantic network and implement in
SAGrader
– Review implementation of assignments with SAGrader staff.
– Schedule assignments, setting deadlines, points, late penalties, etc.
3. Track student performance
– Monitor student performance on assignments.
– Review student challenges
• Provide feedback to students to get them on-track.
• Forward any program problems to SAGrader staff.
– Upload student grades from SAGrader to local course management system
gradebook.
GETTING STARTED

73. FOR MORE INFORMATION VISIT
www.SAGrader.com