Enhance Learning in Large College Classes with SAGrader
Enhance Learning inLarge College Classes with SAGrader
Overview• PART 1: WHY AUTOMATED ESSAY GRADING?• PART 2: WHAT IS SAGRADER?• PART 3: HOW DOES SAGRADER WORK?• PART 4: SAGRADER AS A LEARNING ENVIRONMENT• PART 5: WHAT DOES SAGRADER DO FOR INSTRUCTORS?• PART 6: WHAT ASSIGNMENTS DOES IT SUPPORT?• PART 7: MONITORING STUDENT PROGRESS• PART 8: HOW DO STUDENTS LIKE IT?• PART 9: HOW DO INSTRUCTORS LIKE IT?• PART 10: HOW DO I GET STARTED? OVERVIEW
Challenges of Large Classes• Engaging students with the course• Limited time/sanity for instructors and TAs• Most exercises use multiple choice tests with little or no writing• Writing is time-consuming, slow feedback to students, grading consistency• Difficult to assess higher-level reasoning or critical thinking WHY AUTOMATED ESSAY GRADING?
A Solution? Writing Across the Curriculum (WAC)Benefits – Students write to learn – Higher level assessments – Clearer assessment of student understandingLimitations for large classes – Expensive, requiring extensive staff to grade – Slow grading response time from days to weeks – Minimal timely feedback for students WHY AUTOMATED ESSAY GRADING?
A Better Solution• SAGrader makes writing across the curriculum practical for large classes• Imagine what it would be like to be able to have students write as much as you thought they should with very little time required to grade their work… WHY AUTOMATED ESSAY GRADING?
Learning Benefits• Writing helps students learn – Immediate detailed feedback helps them revise to improve their score and learn• Writing can assess higher level reasoning – Use concepts, theories, facts to reason about realistic problems, interpret, and critique WHY AUTOMATED ESSAY GRADING?
Learning Benefits (cont.)• Frequent feedback – Grades student essays automatically – Provides immediate feedback – Encourages students to revise and learn• Increased student engagement – Students frequently interact with course content out of class WHY AUTOMATED ESSAY GRADING?
Master ContentFacilitate ActiveAbility to Learning Think Writing Improve Ownership Skill at &Expressing Confidence Ideas Critical Thinking
Teaching Benefits• More writing – higher-level learning even in large classes• Reduces grading time – instructors can spend more time teaching• Systematic data collection – learn how to improve your course, or publish research WHY AUTOMATED ESSAY GRADING?
What is SAGrader?SAGrader™ is an online learningenvironment that automatically gradesstudent essays by assessing substantiveknowledge and reasoning. WHAT IS SAGRADER?
A Unique Approach• SAGrader works differently than other programs by ETS, Pearson, and Vantage• Other programs focus on writing style rather than content• Other programs use statistical models to identify a “good” essay without being able to say why WHAT IS SAGRADER?
Statistical Model Rule-Based Expert System System WHAT IS SAGRADER?
Advantages of SAGrader• Better fit for writing in content-area classes• Less cost and time to develop custom assignments• Provides detailed, personalized feedback to students rather than generic ratings• Customizable for different disciplines and topics• Objective and unbiased WHAT IS SAGRADER?
How SAGrader Grades• Desired knowledge for an assignment (the instructor’s rubric) is represented as a semantic network• Student essays are examined as alternative expressions of underlying knowledge HOW DOES SAGRADER WORK?
What Does SAGrader Grade?• Assesses students’ expression of course- specific learning objectives• Can handle low-level knowledge through high-level knowledge• Not just key terms… – Detects multiple expressions of concepts – Considers relationships among concepts – Phrases are examined in context – Handles negations HOW DOES SAGRADER WORK?
Semantic NetworkThe desired knowledge for an assignment isrepresented as a semantic network.This example identifies some of the featuresdistinguishing Mead’s stages. HOW DOES SAGRADER WORK?
Detect Surface Features in Student TextThen text submitted by students is analyzed todetect surface-level features consistent with thatunderlying knowledge structure. “Mead’s stages include the game stage and the play stage.” HOW DOES SAGRADER WORK?
Fuzzy Logic Detects Variations in ExpressionSAGrader uses fuzzy logic to recognize complexcombinations of key terms. Key phrases Text Take the role of the other “…take the role of the other” Take the role “…understand how other Other people feel” Role …look at it from the other Point of view person’s point of view” Viewpoint “…consider what the other Expects person expects” expectations HOW DOES SAGRADER WORK?
Includes RelationshipsSAGrader does not just look for keywords, butlooks at the relationships among conceptspredicted by the knowledge structure “Mead’s stages include the game stage and the play stage.” HOW DOES SAGRADER WORK?
Words in ContextWords must appear in the appropriatecontext to be deemed correct. This permitsmore sophisticated reasoning such as“compare and contrast” questions.In the preparatory stage children In the adult stage children oftenoften mimic adults withoutunderstanding, while in the adultstage they are able to take the ≠ mimic adults without understanding, while in the preparatory stage they are able torole of the generalized other. take the role of the generalized other. HOW DOES SAGRADER WORK?
Handles NegationSAGrader recognizes negations when detectingsurface structures. “Mead’s stages include the game stage and the play ≠ stage.” “Mead’s stages do NOT include the game stage and the play stage.” HOW DOES SAGRADER WORK?
A Tool to Enhance Learning• SAGrader is much more than an assessment tool• SAGrader’s detailed feedback and opportunity for revision permit students to: – significantly improve their scores – learn more in the process A LEARNING ENVIRONMENT
The “Teachable Moment” Feedback and Revision Cycle• Students have just submitted and received immediate feedback• They are motivated to improve their grade• The information is fresh in their minds• They have the opportunity to revise and learn A LEARNING ENVIRONMENT
Students Improve by 20%By using feedback and revising their work,students improved their grade by 20% -- beforethe instructor examined their essay. First Draft Final Draft 87 87 91 90100 69 80 61 60 40 20 0 Students who do not Students who revise All students (100%) revise (29%) (71%) A LEARNING ENVIRONMENT
Improvement via RevisionSubmission 1 Excerpt (Score: 50%) Submission 3 Excerpt (Score: 90%)Material culture is everything that belongs Material culture is art and material objectsto culture that is tangable. Nonmaterial that belong to a culture. Nonmaterialculture would be the values, beleifs, and culture would be thebehavior accepted in culture. There are two symbols, language, knowledge, beliefs, valtypes of norms in culture. Folkways govern ues, attitudes, and norms accepted ineveryday behavior but are not strictly culture. Norms are the expected behaviorenforced. Mores are more serious, carring in a society. There are two types of normsgreater moral gravity and are strictly in culture. Folkways govern everydayenforced. behavior, are not morally important, and are not strictly enforced. Mores are more serious, carring greater moral importance, and are strictly enforced. Values are standards of importance and rightness in society. Language is a abstract system that allows people of a society to communicate. Symbols are arbitrary signs that stand for something. A LEARNING ENVIRONMENT
Students Revise until their Grade is Acceptable A LEARNING ENVIRONMENT
Initially At-Risk Students Can Sometimes “Catch Up” No HS Course HS Course100 92 92 Students who had a 90 high school course 80 70 in sociology perform 67 70 significantly better 60 on first drafts 50 (t=1.96, p=.05). 40 30 20 But by final 10 drafts, there is 0 virtually no First Draft Final Draft t=1.96, p=.05 t=0.0, p=.997 difference (t=0.0, p=.997). A LEARNING ENVIRONMENT
Helps Retention and Graduation Rates• Whether and how much students learn determines whether they persist and graduate from college• One-half to three-quarters of students who drop out of college do so during or after the freshman year. (McClanahan, 2004)• Improving retention rates in the first year has institutional payoffs over every year students remain in the institution. (Levitz et al., 1999) A LEARNING ENVIRONMENT
Monitor Progress and Encourage Learning• Easily compare student submissions to see how students revise• Monitor the impact of revisions• Identify students performing poorly• Track student or classwide performance on specific learning objectives• View individual performance reports• Track systematic data for improving your teaching or publishing research FOR INSTRUCTORS
Instructors can also…• Set up assignment and track student progress anywhere in the world, 24/7• View and comment on any student submission• Respond to student questions or challenges• Review and override any grade• Export grades to CSV or Excel FOR INSTRUCTORS
Unexpected Benefits• Logistics of monitoring course much less difficult• Revisions permit students to learn• Helps level the playing field for disadvantaged students• Cost-effective even with modest sized classes• Students take more active role through challenges FOR INSTRUCTORS
SAGrader Assignments…• Can range from a single short question to entire term papers• Can have multiple correct answers• Can assess more abstract rhetorical goals along with specific substantive knowledge• Are suitable for different levels of reasoning• Can be adapted for different texts• Are suitable for many disciplines ASSIGNMENTS
Intermediate Constraint Tasks Analyze historic events during the 20thUnconstrained century that most affected the rise of theconstructed responses suburbs From this article, identify three factors thatIntermediate constraint affected the rise of the suburbs and providetask evidence for and against the importance of each. Which of these factors encouraged the rise of suburbs because of increased ability toMultiple choice purchase houses?constrained (A) – the automobile (B) – movement of jobs to the suburbs (C) – the GI Bill ASSIGNMENTS
Possible Answers Must Constrained and DefinableIf students are asked to write about their home town:• SAGrader can recognize broad issues related to communities (e.g., issues such as population, governance, geographic location)• But the program will not be able to judge whether the information is correct for any specific community (unless the knowledgebase includes knowledge about that town.) ASSIGNMENTS
Question Types that Work Well• Describe a typology• Describe a concept• Describe two or more concepts and their relationship to each other• Summarize a theory• Cover broad issues• Interpret a passage theoretically• Summarize a particular researcher or scientist• Interpret a passage from two or more different theoretical perspectives• Summarize a study• Summarize what is known about a particular concept or issue ASSIGNMENTS
ExamplesU.S. History (Steam Power) – Factual RecallWhat is the history of steam power? Who built the first steam locomotive? ….Sociology (College Admissions Argument) – ArgumentYou’re the head of a college admissions committee and must choose betweenthree candidates. First, read what the other members have to say and thenmake your own argument. You should support one of the candidates andexplain why…..Psychology (REM Sleep) – Comparisons and TypologiesDescribe the stages of the sleep cycle that occur before REM sleep. Whatdistinguishes REM sleep from other sleep stages? ASSIGNMENTS
ExamplesPsychology (Sensation) – Analyze a Chain of EventsSuppose you are looking at a ball. A rich chain of events starts within youreyes. Explain the details of this chain of events, identify each of its componentsand the roles they play, and finally say which components of light contribute towhich components of color.Biology (Zebra Evolution) - SynthesizeUsing you understanding of the processes of mutation and naturalselection, describe how a population of striped zebras might have evolved froma population of zebras without stripes. ASSIGNMENTS
Question Types that Don’t Work WellSAGrader is not useful for grading broadly defined essayswith no specific content focus.• Loosely Restricted Essays – What do you think is the most important challenge facing teens today? Why?• Expository Short Essays – If you could change places with another person for a whole day, who would you change places with and why would you choose that person?• Creative Writing Essays – Freewrite for 5 minutes using this opening line: “Behind her, the noise escalated...” ASSIGNMENTS
Student Opinions strongly agree agree I like the opportunity to challenge my grade (67%) 32 35 I prefer SAGrader over multiple choice tests (72%) 48 24 SAGrader generally grades my essays fairly (77%) 20 57Writing essays with SAGrader helps me learn (77%) 28 49 I prefer SAGrader over hand-graded essays (84%) 59 25 I like the detailed, personalized feedback (87%) 58 29 SAGrader grades everyones essays without bias 47 47 (94%) I like the immediate feedback (97%) 73 24 I like the opportunity to redo my work (100%) 90 10 0 10 20 30 40 50 60 70 80 90 100 HOW DO STUDENTS LIKE IT?
I would rather use SAGrader in a course than hand-graded assignments.5.004.504.003.503.002.50 Fall 1 Fall 2 Spring 1 Spring 2 Summer Fall 1 Fall 2 Spring 1 2009 2009 2010 2010 1 2010 2010 2010 2011 HOW DO STUDENTS LIKE IT?
Quality Control• SAGrader provides very detailed feedback, far more detailed than other essay grading programs• But SAGrader, particularly for new assignments, is an imperfect tool, and may not recognize everything.• Students tend to expect it to perform perfectly and are quicker to criticize the program than they would be to criticize an instructor.• So it includes quality control features that encourage students to point out any problems to instructors in the form of grade challenges HOW DO STUDENTS LIKE IT?
Challenges• Students are encouraged to challenge if they believe they were graded wrong. – If students are right the program can be overridden or revised to grade correctly and applied to everyone’s submission – If students are wrong, instructors and TAs can give them helpful feedback to get on track.• When first used, assignments have more challenges…typically around 5-10% of submissions.• As program is refined challenges drop to 1-5% and 90% of those are cases where the student misunderstands HOW DO STUDENTS LIKE IT?
Limitations• Students think the program is wrong instead of them and challenge the program more than they would instructors• Students don’t always realize their revisions are not just make-work but help them learn and refine knowledge (analogous to math homework)• On first use, you need to monitor assignment and student challenges• Initial effort to construct essay assignments pays off sooner for larger classes and assignments that are re- used• The more specific the question the more knowledge SAGrader can use to grade HOW DO STUDENTS LIKE IT?
Managing Student Expectations• Student challenges offer an important way to improve the program, particularly for new untested assignments• Challenges also provide a way for the program and the instructor to be responsive to student concerns• However, in many challenges, just as when students question their grade from an instructor, students are often wrong• It is important to answer those challenges and either use them to improve the program or explain the student’s error to help them gain confidence in the program. HOW DO STUDENTS LIKE IT?
Managing Instructor Expectations• Instructors need to monitor the grading process, paying particular attention to challenges.• New assignments typically require some revision before they grade effectively.• Once assignments have been thoroughly tested, the grading process should need very little intervention. HOW DO STUDENTS LIKE IT?
Ande Johnson Professor of Psychology at Park University“I use SAGrader not only to reinforce andimprove student writing, but also to help asegment of students who are challenging toreach -- the underprepared students. Mystudents have also admitted that answeringthe questions have forced them to open theirtextbooks and to read the material.” HOW DO INSTRUCTORS LIKE IT?
“My less skill-developed students havereported frustration at the onset of usingSAGrader, and have reported that thewriting gets easier and they have to submitfewer times across the semester. Ive alsoobserved that their responses over thesemester are becoming more concise yetmore dense/ meaningful. Students have alsoreported that their writing in other classeshas improved and knowledge/ skilltransference is important in education.” HOW DO INSTRUCTORS LIKE IT?
“The biggest benefit of using SAGrader restswith improving student writing and thinking.The SAGrader system gives the lessprepared students more learningopportunities without taxing the instructor.I like the automated scoring and immediatefeedback adapted to the studentsschedule.” HOW DO INSTRUCTORS LIKE IT?
Lloyd Chia Sociology, University of MissouriBeing able to keep a big class of studentswriting throughout a course isinvaluable, something that typically wouldnot be possible with an instructor and twoTAs for a big class of 290 students like I’mteaching now. HOW DO INSTRUCTORS LIKE IT?
I also appreciate being able to track whichstudents are not doing well, or who areconsistently late submitting assignments. Ithas given me the opportunity to contactthose students, express concern about theirperformance, and in a few cases toeventually help them get back on track withtheir grades. HOW DO INSTRUCTORS LIKE IT?
It provided specific writing instructions forthe students that is more detailed than Inormally have time to [give]. It allowedstudents to develop better writing skillswithout my needing to be an Englishteacher.Students by and large found SAGraderreally helped them to focus on detail, bemore thorough and precise, master theinformation, improve their spelling, etc. HOW DO INSTRUCTORS LIKE IT?
Getting Started…1. Adopt SAGrader – order through your bookstore along with any textbooks.2. Set up assignments – Develop rubrics for assignments. – SAGrader staff convert rubric to semantic network and implement in SAGrader – Review implementation of assignments with SAGrader staff. – Schedule assignments, setting deadlines, points, late penalties, etc.3. Track student performance – Monitor student performance on assignments. – Review student challenges • Provide feedback to students to get them on-track. • Forward any program problems to SAGrader staff. – Upload student grades from SAGrader to local course management system gradebook. GETTING STARTED