Uploaded byTanya Joosten
Determining the Effectiveness of Your Faculty Development Program
The document outlines a workshop led by Tanya Joosten, Dylan Barth, and Nicole Weber on evaluating the effectiveness of faculty development programs at the University of Wisconsin-Milwaukee. It discusses key steps in assessing online and blended courses, the elements necessary for a successful faculty development program, and the evaluation strategies to be employed at different institutional levels. The focus is on defining quality courses, facilitating success for instructors, and documenting the effectiveness of programs through various evaluation tools and methods.
More Related Content
Similar to Determining the Effectiveness of Your Faculty Development Program
Determining the Effectiveness of Your Faculty Development Program
- 1. Determining the Effectivenessof Your Faculty Development Program March 17th, 2014 Tanya Joosten, Dylan Barth, and Nicole Weber Learning Technology Center, LTC@uwm.edu University of Wisconsin - Milwaukee
- 2. Overview Introduction to UWM Breakoutactivity, questions to consider in planning to determine faculty development effectiveness Sharing of UWM’s model, faculty development, evaluation strategies, and evaluation tools as well as other institutions Conclusions
- 3. University of Wisconsin–Milwaukee Learning Technology Center (LTC) Tanya Joosten, tjoosten@uwm.edu, @tjoosten Dylan Barth, djbarth@uwm.edu , @dylanbarth Nicole Weber, nicolea5@uwm.edu, @nwebs LTC.uwm.edu | LTC@uwm.edu | @UWMLTC Introductions
- 4. Outcomes Identify the characteristicsof a good blended and online course, including the pedagogical model Determine what elements and formats should be considered in designing a faculty development program
- 5. Outcomes Share strategies forevaluating the effectiveness of your faculty development program at the course, program, and institutional levels from multiple perspectives, including students, colleagues, researchers, and administration Understand how these steps fit into a model of evaluation for learning technologies and pedagogical innovation
- 6. Tech Enhanced Blended Online Traditional Self-paced MOOCs Flex Faculty development programs andpedagogical consultation Technology training and support Evaluation and research About us
- 7.
- 8. What is blended andonline?
- 9. A Backwards Design Approach toEvaluation
- 10. Evaluation planning Input • Medium(F2F or Online) • Instructor demographics • Discipline • Course level Process • Defining good blended and online course • Defining pedagogical model • Designing faculty development program Output • Course level • Program and Institutional level
- 11. F2F is notthe gold standard Focus on the process…the pedagogy Inputs
- 12. Define a good blendedand online course Define pedagogical model Design a faculty development program Process
- 13. What are we doingtoday?
- 14. Discuss: Steps toevaluating effectiveness 1.) Break into discussion groups using Adobe Connect. You will be able to use audio or text chat. 2.) Access the Google Docs to document your activities at: ELIOnlineSeminar.wikispaces.com 3.) Discuss, share, and brainstorm responses to the following series of questions on the 3 steps to planning evaluation. Don’t forget to identify the member of your group and document your group’s response in the Google Doc.
- 15.
- 16. 1. What isa “good” blended or online course? What pedagogical model facilitates a “good” blended or online course?
- 17.
- 18. 2. What elementsand format should be considered in designing and developing a faculty development program? What opportunities and experiences should be available to help instructors learn effective practices in design, delivery, and teaching online and blended courses?
- 19.
- 20. 3. How willyou know when faculty are providing quality online and blended courses? What tools or services could be provided for evaluating the effectiveness of online and blended courses? How will quality be communicated to others?
- 21. DISCUSS! 1.) Break intodiscussion groups using Adobe Connect. You will be able to use audio or text chat. 2.) Access the Google Docs to document your activities at: ELIOnlineSeminar.wikispaces.com 3.) Discuss, share, and brainstorm responses to the following series of questions on the 3 steps to planning evaluation. Don’t forget to identify the member of your group and document your group’s response in the Google Doc.
- 22.
- 23.
- 24. 1. What isa “good” blended or online course? What pedagogical model facilitates a “good” blended or online course?
- 25. Good course Higher retention Bettergrades on assessment • Quizzes and exams • Projects and papers • Overall Satisfied students • Future enrollments Learning Engaging Enrollments
- 26. Pedagogical model Content • Text •Images • Audio • Video Interactivity • Discussions • Groups • Feedback Assessment • Written and oral examination • Discursive • Portfolio
- 27.
- 28.
- 29.
- 30.
- 31. 2. What elementsand format should be considered in designing and developing a faculty development program? What opportunities and experiences should be available to help instructors learn effective practices in design, delivery, and teaching online and blended courses?
- 32.
- 33. • Ten questions •Designing learning modules • Online vs. F2F - Integration • Decision rubric for content choices • Learning objects Content • Progressive/summative • Before, during, and after • Self evaluation • Peer evaluation • Student evaluation Course Evaluation • Rubrics • CATs • Templates • Traditional formats Assessment • Synchronous/asynchronous • Establishing voice • Discussion forums • Small groups Interactivity • Managing expectations • Time management • Technology support Helping Your Students • Staying organized • Managing workload • Avoiding course and a half Course Management Course Redesign Transitioning to online and blended teaching Pedagogy and more
- 34.
- 35. F2F 1 3 hrs Online1 F2F 2 3 hrs Online 2 F2F 3 3 hrs Showcase 3 hrs Post- Program Blended
- 36.
- 37.
- 38.
- 39.
- 40.
- 41. 3. How willyou know when faculty are providing quality online and blended courses? What tools or services could be provided for evaluating the effectiveness of online and blended courses? How will quality be communicated to others?
- 42. Did it work? CCFlickr katherine.a
- 43.
- 44.
- 45. Student Evaluation Data Courseevaluation
- 46.
- 47. Who is youraudience? What variables will be examined? How will the data be collected? analyzed? How (and where) will the data be presented? Who will be involved in the evaluation process? When will the evaluation be completed? Evaluation plan
- 48.
- 49. Input Process Output StudentsInteractivity Students/Faculty Demographics Communication Learning Age Engagement Satisfaction Gender Social Presence Performance Ethnicity Content Student Status Assessment Course/Institutional Data Full time/Part time Retention Employment Status Drop/Withdrawal Rate Zip Code Grade Course/Instructor Evaluation Course Discipline Course Level Instructor Mode of Delivery Variables
- 50.
- 51.
- 52.
- 53. Human Resources CC FlickrVandy CFT
- 54.
- 55.
- 56. Check out someresources: ELIOnlineSeminar.wikispaces.com
Editor's Notes
- #2 As the demand for blended and online learning opportunities increases, so does the need to ensure the quality of online education through faculty development programming. And with the increase in the diffusion of blended and online programming across higher education institutions, stakeholders are looking for ways to ensure the quality of the student experience and better understand the impact on student outcomes. Recently, many of us have been asked to provide evidence of the effectiveness of our faculty development programming: administrators are looking for a return on investment in faculty development to ensure quality in blended and online programming, as we are seeing decreases in state funding and enrollments, which leads to cut budgets. In order to for us to determine the effectiveness of our programming using a backwards design approach, we need to first understand what is a good online or blended course as well as what competencies are required of faculty to teach blended and online courses and how those can be best facilitated in a faculty development program. Then we can consider how to evaluate the impact on student outcomes.This workshop will offer a collaborative and interactive opportunity to connect with colleagues to consider and construct how the effectiveness of faculty development programming can be determined and disseminated. A model of evaluation for a faculty development program will be shared.
- #3 ScheduleAgenda12:00-12:30 Welcome to session Intro presentation of UWMLTC, stats of blended and online at UWM, history of blended at UWM (Tanya)12:30-1:15 Break out (Answer 3 questions in groups)1:15-1:45 Question 1 Report out, LTC summary of UWM blended and online course pedagogical model (last 5 minutes) (Nicole)1:45-2:15 Question 2 Report out, LTC summary of UWM faculty development program (last 5 minutes) (Dylan)2:15-3:00 Question 3 Report out, LTC summary of UWM evaluation efforts (last 5 minutes) (Tanya)
- #7 The LTC provides faculty development and pedagogical consultation, technology training and support, and evaluation and research of an array of course delivery modes, including tech enhanced, blended, and online.
- #8 In addition to facts, data, and numbersPossibly that quality online and blended courses are the depts, schools, and colleges responsibilityWe are not the police
- #9 Blended course definition: An Institutional DefinitionTanya can discuss UWM - From an institutional perspective, there are several ways to define blended courses; the example given is that of University of Wisconsin – Milwaukee, though it is not dissimilar to that of other campuses. The definition relies on an assessment of the impact of blended pedagogical strategies. Up to a certain point, about 20% of seat time placed online, the effect of blended learning is roughly equivalent to that of Web enhancement. This isn’t necessarily a bad thing: we often recommend to faculty that they start in the 0-20% range until they feel more comfortable assigning and assessing online work Between 20-50%, a course requires significant redesign work if it is to be taught successfully. The reduction of a third of the face time of a course, for instance, means that the instructor must find new learning activities and new forms of assessment, as well as different ways to deliver course content. Beyond 50%, the online work begins to dominate the pedagogical design of the course, and the instructor must explicitly work with considerable care to resolve the three key issues of blended learning: “closing the loop,”“course-and-a-half,” and “hearing voices.” Finally, when a course reaches 80% or more online, a question becomes increasingly acute: why is this course not being delivered fully online? What is the rationale for having face-to-face meetings at all? Often, there is such a rationale, but it becomes urgent to identify that rationale at this stage of course redesign.
- #10 Your faculty development program is
- #11 How do we know it is a good online or blended course?What does it look like?How do we help faculty design and develop these types of courses?
- #12 Stop comparing F2F to Online (or Blended)
- #13 Process is the magic!
- #16 Your faculty development program is
- #17 What is effective? What is success? What are you trying to accomplish by having a faculty development program or instructional improvement effort? What is your desired result?What makes for an effective online and blended learning model?
- #23 Your faculty development program is
- #24 Your faculty development program is
- #25 Summarize what groups didWhat is effective? What is success? What are you trying to accomplish by having a faculty development program or instructional improvement effort? What is your desired result?What makes for an effective online and blended learning model?
- #27 Active learning, student-centerEngaged learningResearch driven effective practicesNICOLEFor each delivery mode, there are pedagogical considerations to be made with regard to content delivery, interactivity, and assessment. The UWMLTC faculty development program and pedagogical consultations with our team guide instructors in making decisions about these considerations. Content Yes - text+images, current, video clips (under 10 minutes) Avoid video lectures, don’t spend to much time on audio lectures eitherInteractivityCreate opportunities for participation, community, collaboration, and connectedness through online class discussions, group discusionss, team projects, team synchronous meetings Assessment Frequent low-stakes feedback, opportunity to fail, make corrections to learning Avoid catastrophic assessment Provide audio feedback, if appropriate Provide varied forms of assessments from cognitive, behaviors, affective domains
- #28 Text plus images (Mean study)Rich, current (Web clips, videos, see Ginkgotree study)
- #29 Social presence, group activities, thought-provoking discussion boards, interactive Introductions and IcebreakersSupport and Sharing ScholarshipDiscussionsGroup ProjectsFeedback
- #30 Frequent and low stakesAvoid Catastrophic Assessment Valid assessmentshttp://assessment.uconn.edu/primer/taxonomies1.html
- #31 Summarize what groups did
- #32 Summarize what groups did
- #35 Who runs the faculty development program? Experienced instructors in online and blended with terminal degrees in our respective fields, represented by the social sciences, natural sciences, humanities, professions, and arts. We hope to continue to bring in more instructors who have been successful. We bring both the practice and the research elements to the program.
- #36 Blended approachThe structure focuses around small F2F sessions with online sessions in-between. The work is centered on manageable chunks with concrete deliverables.Taught in a blended format and in multiple formats during the academic yearFace-to-face meetings and online assignmentsModel good blended practicesExperience blended course as a studentEffective teaching modelExperienced blended teachers are program facilitatorsThe faculty development program that we offer is taught itself in a blended mode, since we take seriously the adage that the best way to learn how to teach a blended course is to take one. We offer the faculty development program multiple times during the year, once during each UWM semester (Fall and Spring) and once during the summer.The program comprises both face-to-face and online learning activities, in which our goal is to model good blended teaching practices, to encourage participants to experience blended learning as their students will, and to demonstrate that blended learning is an effective teaching model. To this end, we ensure that all of the presenters in our faculty development program are themselves experienced blended instructors, since that ensures that the presenters speak not only from theory, but also from lengthy experience as practitioners of blended learning.
- #37 We use backward design theories of Wiggins and McTighe to focus the redesign process of participants. This also illustrates the importance of studying research in online and blended learning to better inform our practices in the faculty development program. We also use backward design as the model for reconfiguring the program during the lifecycle of evaluation.
- #38 We facilitate our program in an active learning classroom so that we can both model effective classroom practices for blended instructors but also to promote active learning in our own program.
- #39 Practice-based: Develop Useable MaterialsParticipantsfocus on developing useable course materials so that they have developed chunks of their course when they leave. The deliverables all reinforce this aspect.A practical approach Get startedRedesign courseDevelop course materialAcquire teaching skills The purpose of our faculty development program is to support and encourage faculty who wish to redesign their courses for blended learning. We have adopted a practical rather than a theoretical approach – although our notion of “practice” is supported by current theory – because we know that faculty are busy, and that they are more interested in what works than in abstractions. A practical approach to blended course redesign identifies how to get started, how to redesign a course for blended teaching and learning, how to develop blended course material, and how to acquire the teaching skills needed for a blended course. We assume that faculty are already experienced teachers from a traditional face-to-face perspective, and that their principal task is to learn how to redeploy their skills in a somewhat different context.
- #40 Dylan – we should share the evaluation forms and other tools, CATS, etc., that we use. Can you post to the wiki resource page? Thanks!We have 5 main program outcomes. We determine whether these outcomes were met by ask for feedback about our program in several ways. We provide frequent self-reflection on the part of the participants to determine what they still need help with. We also ask for more quantitative feedback through ongoing online surveys through the registration system. As facilitators, we reflect on the progress regularly to make improvements. We also ask for a more thorough, summative evaluation at the end of the program. We use this data to make continuous improvements to our program.Main Program OutcomesStart of a redesigned courseNew teaching skills and knowledgeFaculty know what to expectFaculty get their questions answeredIn blended, re-examine both face-to-face and online component First, faculty start with concrete development of both broad-perspective elements of their new course, such as a course redesign plan and a syllabus, and the development of more specific features of the course such as learning modules. This is especially appropriate in disciplines or areas where “learning modules” are not typically the unit of course redesign, thus requiring faculty to re-examine their conceptual basis for instruction from the bottom up. Second, faculty must acquire new teaching skills and knowledge appropriate to a blended learning frame, such as how to facilitate a peer learning community both face-to-face and online, and how to assess student learning in a manner that is quite distinct from the usual “three exams and a term paper” of a traditional face-to-face course. In the latter context, it is important to note that faculty are often committed to the use of summative assessments such as exams and term papers without ever having questioned previously their efficacy. The prospect of a blended course redesign thus holds out the possibility of a paradigm shift from modes of instruction that once seemed satisfactory to a model of teaching and learning that problematizes everything that has once been taken for granted. This is a source of considerable skepticism and anxiety for faculty who have been teaching for many years in a more traditional manner, often unreflectively.
- #43 Evaluating the impact of the program on course, program, and institutional level
- #45 Course levelCourse evaluations by instructor ChecklistCourse evaluations by dept or institutionPeer evaluation guideQuality mattersUC Chico RubricStandard student evaluation of course and instructor How does it change when blended or online, if at all? Does it capture an evaluation of the pedagogical model? new skills of teaching? Understand course materials Engaging course activities Collaborative course activities Connected to instructor and classmates Receive frequent low-stakes feedback Opportunities to make corrections in learningStudent survey (engagement, learning, satisfaction, performance)
- #46 Nicole—using student evaluation data to ensure course quality. Students have firsthand knowledge of the course that was provided and often have taken other blended courses. Two ways of gathering student perceptions regarding the course is through mid-semester feedback (as pictured) and at the end of the semester. Tanya - I can talk about using CATs as a way to guage course improvements needed
- #47 Course levelCourse evaluations by instructor ChecklistCourse evaluations by dept or institutionPeer evaluation guideQuality mattersUC Chico RubricStandard student evaluation of course and instructor How does it change when blended or online, if at all? Does it capture an evaluation of the pedagogical model? new skills of teaching? Understand course materials Engaging course activities Collaborative course activities Connected to instructor and classmates Receive frequent low-stakes feedback Opportunities to make corrections in learningStudent survey (engagement, learning, satisfaction, performance)The certificate programs allows us to transform the culture of our campus to promote online and blended learning, to emphasize the importance of quality teaching and sound pedagogy, and to create a community of online teachers. Components: successful completion of OBTP, delivery of a blended course at UWM, peer course evaluation by LTC staff and/or peer mentor/evaluator, 2-3 page reflection on how the instructor’s pedagogy has changed, and “paying it forward” (e.g., guidance on the listserv, syllabus added to resource repository). Over 70 faculty and instructors have been awarded the certificate represented in the Colleges of Letters and Science, Peck School of the Arts, School of Education, College of Nursing, School of Information Studies, Lubar School of Business, and College of Health SciencesCan discuss how the certificate program came to be and philosophical debates around its development
- #56 Incentivizing facultyTime and focusIRBData collection (data warehouses, grade and retention, survey tools)Data analysis (lack of expertise or staff)