Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on

Published in: Education
  • Be the first to comment


  1. 1. Will Humphries, Senior Developer @ Tufts Brian Broniak, Director of Online Learning & Collaboration Services (OLCS) @ Virginia Tech June 10-15, 2012Growing Community;Growing Possibilities
  2. 2.  Background information ◦ Both schools and Sakai The VT story from idea through full production The Tufts story from mandate to current state Goal: Share ideas for a successful implementation of EvalSys at your university 2012 Jasig Sakai Conference 2
  3. 3.  VT is in Blacksburg, VA – a college town 30,000 students globally, 7000 courses Most students in B’burg, but students global 4 semesters each year – F/S, Summer I and II Very decentralized, hard to do “enterprise” Sungard Banner as our SIS Sakai 2.8.1 as of Spring 2012  Scholar Sakai school fully since 2010, informally for much longer 2012 Jasig Sakai Conference 3
  4. 4.  Tufts is around Boston, MA 10,000 students, approx. 5,000 undergrad Sakai for 1 year, 3 of 8 colleges ◦ it’s called Trunk Moving to PeopleSoft Campus Solutions 9.0 Sakai 2.8.1, eval 1.4 2012 Jasig Sakai Conference 4
  5. 5.  Using paper OpScan forms since 70s ColdFusion based system, first try online ◦ Decided it would not meet needs at scale so started a tool for Sakai  EvalSys today Task Force(s) on the process and form in parallel to work on the system Another system for distance learning (IDDL) at VT (developed in-house) 2012 Jasig Sakai Conference 5
  6. 6.  Decentralized – some online, some paper A&S and Eng. motivated to leave paper ◦ Cost savings from centralized service Evals for all Tufts Trunk adopters Fall 2012 ◦ 30,000 potential responses from 2,500 courses Evals in LMS a selling point for Trunk project 2012 Jasig Sakai Conference 6
  7. 7.  Provide “one” system to unify this data – replace ColdFusion, paper, IDDL system Provide robust reporting Garner 70%+ response rates sustained Provide a “good” student and instructor experience Our approach: ◦ Ramp up slowly to allow for departments to adjust (a 5 semester ramp-up) ◦ Communicate, communicate, and communicate 2012 Jasig Sakai Conference 7
  8. 8.  EvalSys 1.3 with local changes We use the hierarchy with 3 levels – VT-wide, college, and department Email reminders each course section every 2 days All scaled questions on 6 point scale We have an external data warehouse and Jaspersoft reporting interface We purge the data each semester after putting copies of reports into Scholar personal workspaces and in our warehouse 2012 Jasig Sakai Conference 8
  9. 9.  In Spring 2012 we finally ran this VT wide! ◦ 4500 courses, 125,000 requests with 85,000 completed = 69% response rate Students like it far better, faculty feedback has been mixed Response rates in high 67-70% range each semester + a definite “reminder effect” (see chart) 2012 Jasig Sakai Conference 9
  10. 10.  Asked two questions to random sampling of students over all 5 semesters ◦ “Compared to the evaluation form used in the past, the items on this form allow me to provide a better evaluation of this course/instructor.” 22,697 responses from students here – with 87% favorable (somewhat agree, agree, or strongly agree), 61% = agree or strongly agree ◦ “I prefer filling out the evaluation form on-line rather than filling out the paper-and- pencil form in class.” 23,017 responses from students here – with 87% favorable (somewhat agree, agree, or strongly agree), 77% = agree or strongly agree 2012 Jasig Sakai Conference 10
  11. 11.  Need to get chart here from data Steve is working on 2012 Jasig Sakai Conference 11
  12. 12.  Major org challenges, especially with decentralized universities ◦ Get an “owner” outside of IT – our Office of Assessment and Evaluation has been a great partner Focus on the form and go ahead and change scales earlier rather than later Ramp up slowly, gives you time to refine system and everyone to adjust Build a warehouse (lots of valuable data) – do your best to gather report requirements Work through change curve – myths, rationale, etc 2012 Jasig Sakai Conference 12
  13. 13.  Centralized administration Identify edge cases early ◦ Cross-listed ◦ Team taught ◦ TA’s / non-official instructors ◦ Studentless/instructorless courses External reporting? Still in the works 2012 Jasig Sakai Conference 13
  14. 14.  Gradual pilot vs. rapid deployment CM API (evalgroup) vs. Site-based (evalproviders) One template vs. several Separate instance vs. primary LMS instance 2012 Jasig Sakai Conference 14
  15. 15.  Consolidated emails Exposing response rates real-time Deal with edge cases – non-traditional schedules and GAs Get back to the Sakai baseline 2012 Jasig Sakai Conference 15
  16. 16. Contact information: Brian Broniak – Will Humphries – June 10-15, 2012Growing Community;Growing Possibilities