Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

HCCS - Evaluation for Greater Learning Student System


Published on

  • Be the first to comment

  • Be the first to like this

HCCS - Evaluation for Greater Learning Student System

  1. 1. EGLS3EGLS3 - Evaluation for Greater Learning Student Survey System Implementation at Houston Community CollegeDr. Alan Ainsworth Dr. Mac AdkinsFormer HCC Faculty President Senate President SmarterServices LLC Mario Heredia Jeff Worford Director Research & Lead Developer Support Services SmarterSurveys™ Product February 23, 2012 1
  2. 2. EGLS3 Houston Community CollegeA comprehensive community college serving the Houston areaOver 75,000 students credit, non credit in Fall 2011One of highest number of international students of all community colleges in the country 2
  3. 3. EGLS3 Student Evaluation of Instruction – History Paper-Based Student Evaluation of Instruction (SEOI) survey has been in place for 27years, revised incrementally. For Face to face classes, the SEOI was the HCC’s paper method to administer the student evaluation For online classes Distance Education used their own online survey Structure. Four part-time staff plus a supervisor working for two months. Large work room Given the current technology, Institutional Research sought -Online administration methodologies -New Instrument 3
  4. 4. EGLS3 HCC’s SEOI Survey Paper Distribution System IR manually prepared and routed more than 5,000 packets and 100,000 SEOI instruments eachsemester. Each packet contains the SEOI instrument and instructions for faculty and proctors. The packets were sent to their respective campuses administrators. Campuses administrators were responsible for distributing the materials to the Instructors and collecting from the proctors (who administer the survey to the students). Once IR received the packets, entered them manually in the system to track the return rate. IR cleaned the packets, scanned the surveys and generated data file. The report was run in SPPS. The final report was saved on disks and sent to the College Administrators for manual distribution.Students’ hand written comments were sent back to faculty. Process is cumbersome, labor intensive and costly. SEOI’s means report results were not used systematically in the evaluation process. Return rates have been an issue, but in Spring 2010 the rate of return increased to 95% and the response rate was of 73%. 4
  5. 5. EGLS3Student Evaluation of Instruction (SEOI)– Paper Base Instrument 5
  6. 6. EGLS3EGLS3 Student Evaluation Survey Instrument Houston Community College EGLS3 – Evaluation for Greater Learning Student Survey System Organization and Explanation of Materials Strongly Disagre Strongly Agree Agree e Disagree 1. My Instructor explains difficult material clearly.     1. My Instructor communicates at a level that I can     understand. 1. My Instructor makes requirements clear.     1. My Instructor identifies relationships between and     among topics. Learning Environment Strongly Disagre Strongly Agree Agree e Disagree 1. My Instructor establishes a climate of respect.     1. My Instructor is available to me on matters pertaining     to the course. 1. My Instructor respects diverse talents.     1. My Instructor creates an atmosphere in which ideas     can be exchanged freely. Self-Regulated Learning Strongly Disagre Strongly Agree Agree e Disagree 1. My Instructor gives assignments that are stimulating     to me. 1. My Instructor encourages me to develop new     viewpoints. 1. My instructor arouses my curiosity.     1. My Instructor stimulates my creativity.     Overall Opinions Strongly Disagre Strongly Agree Agree e Disagree 1. I like this instructor.     1. I am interested in this subject.     1. I think the classroom was appropriate for this class.     1. I would recommend a course taught by this     instructor. General Information 1. This class is:  A Required Course  An Elective  I am not sure 1. What grade do you expect to earn in this course? A B C D F 6
  7. 7. EGLS3 Important EGLS3 Features from UNT’s SETE These are the factors or elements of teaching effectiveness that are being measured:Factor 1: Organization and explanation of materialsFactor 2: Learning EnvironmentFactor 3: Self-regulated Learning Four questions are used in the EGLS3 validity analysis. Scale scores (the weighted responses) are reported for each of the three domains. An overall score is also computedThe following are sample scale ranges for EGLS3 7
  8. 8. EGLS3 SEOI-EGLS3 Survey Instrument Comparison Student Evaluation of Instruction(SEOI) EGLS3 - Survey Instrument• Paper-Based • Is implemented online to allow• Home Grown Survey flexibility, transparency, and• Included non-instructional Questions accountability• Library and course content questions • Provides a measure of teaching• No Validated effectiveness as perceived by students.• Means report • Allows college-wide comparison in key• IR cleaned the packets, scanned the areas surveys and generated data file. The • Facilitates student evaluations of their report was run in SPPS. instructors (not the course or the• The final report was saved in a disk and content) sent to the College Administrators. Hand- • The EGLS3 scale scores can be used to written comments returned to faculty. make assessments for continual• Process was cumbersome, labor intensive improvement of the institution and costly. • Results available on line for individual• The results were not used in a systematic faculty. evaluation process. • User built their own reports 8
  9. 9. EGLS3 Early Stages Vice Chancellor of Planning and Institutional Effectiveness has Faculty and OIR representativesvisit University of North Texas to discuss SETE. HCC Administration and Faculty Senate decide to move forward with assessing SETE for possible online evaluations. 9
  10. 10. EGLS3 Faculty Governance and Endorsement Faculty Senate President requests the evaluation of SETE for replacement of SEOI byEducational Affairs Committee Faculty Senate sees reports of SETE and votes unanimously to support pilot program for Spring 2011 10
  11. 11. EGLS3 Seeking Additional EndorsementApproval of Pilot from Vice Chancellor of Instruction, Deans’ Councils.Forums held for information and feedback 11
  12. 12. EGLS3 Student’s evaluation process, the transition from paper to onlineIR Partnership with Faculty SenateIR Responsibility: Develop and implement the Operational planFaculty Senate President: Navigating Faculty Governance and EndorsementPilot Implementation Spring 2011 . Two Locations were selectedOutcome: Response Rate 20%Follow up: Student Focus GroupsFindings: Login process cumbersome 12
  13. 13. EGLS3 Solutions for Log-In Process found with PS IntegrationHCC - SmarterServices People Soft login Integration of EGLS3 - Student Center 13
  14. 14. EGLS3 Student’s evaluation process, the transition from paper to onlineHCC - SmarterServices People Soft login Integration of EGLS3 - Faculty Center 14
  15. 15. EGLS3 Integration and Responsibility of IR in EGLS3 Implementation Institutional Research liaison between SmarterServices and the HCC departments participating in the EGLS3. Develop timeline and monitor progress Develop and review documentation with SmarterServices, IT/Costumer Support Develop and review communication plan with Communications Department. Several media channels used to notify the institution: meetings, email, paper, social media,web page, plasma screens, posters, etc. 15
  16. 16. EGLS3 Fall 2011 - Full Implementation Facts Only Regular Term and Second Start Classes surveyed, including Distance Education and Dual Credit Data submitted: 6,460 classes with 140,800 enrollment. System runs smoothly 23,005 responses with a Response Rate of 16%Technical Issues Pop –ups blocked in College Computer Labs and other computers. System Slow due to lack of services running on the web server.Reporting Available on our Web Site Faculty, Administrator access Administrators can Build their Own ReportsFollow up Faculty Focus Groups 16
  17. 17. EGLS3 FALL 2011, ImplementationFull Implementation Fall 2011. District-Wide Steering Committee in place Teams and strategy are developed Operations Strategy : How do we implement? What processes and procedures need to be developed Communication Strategy: How do we communicate? What and where? What are the benefits? How is it different? How does it relate to our vision? Feedback Strategy : How will this improve student learning? How will it improve teaching effectiveness? 17
  18. 18. EGLS3 Spring 2012-Next Steps Spring 2012 Next Steps Immediate: Increase response rate As survey reports became available, faculty and administrators will be more involved inthe process Develop committees to work to integrate this students evaluation with HCC Faculty/Staffevaluation system PEP. Redevelop PEP Research Project 18
  19. 19. EGLS3 About SmarterServices Incorporated in 2002. Served over 500 educational institutions including K-12, technical colleges, career colleges, communitycolleges, private colleges, universities, and corporations. Some of the other solutions provided by SmarterServices include SmarterMeasure, SmarterFaculty,and SmarterProctors.
  20. 20. EGLS3 About UNT’s SETE™ InstrumentBenefit from extensive research at the University of North Texas.Control for bias of 24-variables Student grades Time of course Delivery systemTrack improvement with a standardized effectiveness score (1-1000 scale)Useful for benchmarking and longitudinal analysis of faculty effectivenessMeet state reporting guidelines
  21. 21. EGLS3 Student’s Perspective within SmarterSurveys™
  22. 22. EGLS3 Student’s Perspective within SmarterSurveys™18 Standard SETE Questions16 Likert scale questions (4 per Category)2 General Information questions
  23. 23. EGLS3 Student’s Perspective within SmarterSurveys™Custom Question can be added to SETE.Custom questions can be Likert, open ended, multiple choice, etc.HCC choose to add 5 open ended questions to their evaluation
  24. 24. EGLS3 Student’s Perspective within SmarterSurveys™Confirmation pop-up is presented to Student’s for verification upon submission of a survey.
  25. 25. EGLS3 Student’s Perspective within SmarterSurveys™After completing a survey, the course is marked as “Completed”Any remaining surveys can be taken immediately or at another time
  26. 26. EGLS3 Faculty’s Perspective within SmarterSurveys™HCC choose to suppress results until the end of the survey period.Faculty can still see their completion percentages, survey status, and results release date.
  27. 27. EGLS3 SETE™ Data Processing FlowFaculty and Student Attributes + Grades EGLS3 Scores Generated Data Processed Faculty ID, Course ID & Scores Survey Responses
  28. 28. EGLS3 Faculty’s Perspective within SmarterSurveys™Once survey results open, faculty can view course responses and also pull aggregate reports for the term.
  29. 29. EGLS3 Faculty Reports within SmarterSurveys™Individual Course Response Distribution Report – Per Course Header and Likert Responses
  30. 30. EGLS3 Faculty Reports within SmarterSurveys™Individual Course Response Distribution Report – Per Course Open Ended Responses
  31. 31. EGLS3 Faculty Reports within SmarterSurveys™ EGLS3 Score Report – Per Term
  32. 32. EGLS3 Dean Reports within SmarterSurveys™
  33. 33. EGLS3 Dean Reports within SmarterSurveys™
  34. 34. EGLS3 34
  35. 35. EGLS3 Questions…. Contact Information:Alan Ainsworth 713.718.8709; Mario Heredia 713.718.8627; SmarterSurveys™ Info 877.499.SMARTER (7627); 35