Long and winding road


Published on

Acumen website

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • This is an extended examination of the process of evaluating one project objective. The purpose of the presentation is to illustrate that even the best thought-out logic models or evaluation plans may need adjustments in order to provide data that can truly contribute to understanding process and outcomes.
  • The US Department of Education Teacher Quality Enhancement grants have eight required objectives. We will look at one of those objectives in the “evaluation case study”. The goal of the project was to recruit, prepare, and retain high quality K-12 science and math teachers for high needs urban school districts. To develop the logic model for this objective, I consulted with project senior personnel to determine how they planned to provide the training and practice to achieve this objective.
  • Here is an abbreviated version of the logic model that focuses on technology. There are 3 major inputs that all fall under the umbrella of coursework. The outputs are gathered from three sources—(1) student grades on the technology course; (2) a student survey given to students at the conclusion of their teacher preparation program that asked them to report on their confidence using a variety of technology tools as specified by the International Society for Technology Education IISTE), in which courses faculty model technology integration was observed, and how they integrated technology into their student teaching; and (3) an examination of student teaching portfolios.
  • Here are the results. There were disparities in two major areas. Students reported that they saw little modeling of the integration of technology and instruction outside of the required Educational Technology course. They also scored below expected levels in confidence using technology tools—especially tools that are content-specific like probes, graphing calculators, simulated labs, and math programs. Their self-reported used of technology during student teaching showed innovative and frequent integration. However, their student teaching portfolios (the artifact of student teaching that is graded) did not support this. So now we needed to reflect. Were the inputs we assumed really there? To investigate, we began with the premise that faculty understanding of the integration of technology may not coincide with ISTE standards. We needed to explore faculty perceptions as to the role of technology in teaching and in the teacher preparation program.
  • So we conducted an in-depth, qualitative study to explore faculty perceptions of the role technology should and does play in the courses they teach. It also examined how they themselves use technology when teaching (specific examples), how they expect their students to use technology when teaching, and where in the teacher education program the use of technology as an instructional tool should or does occur. We used interviews and qualitative research analyses to explore common themes.
  • The interviews provided insight into why students were competent yet not confident. The major theme was that the pre-service teachers needed to be able to “use” the technology so that they did not appear to be incompetent. There was little discussion of the value of integrating technology into teaching. Here are some quotes from the interviews:Yeah, we do have conversations about when it's appropriate to do different things. One of the things that we actually talk about a lot is how to use the space that you have, like your board, or your, when you're using a SMARTboard and how to help make things visual for students. So we, sometimes we'll stop and talk about is this really the best way to do this or not. But we do that really broadly across a lot of things that we, not just in terms of technology. But, like when we're using graphing software, we talk about where's the appropriate place. You know, where does this go in the curriculum? You know, where does it fit, and when's the best time to use it and how you might use it with one group of learners vs. another.  “I don't [model the integration of technology] because I don't think of it as a thoughtful process. I've, I just think of it as, as a, just an opportunity, as just something you do. It's not something you necessarily planned for.” And one faculty member commented after an extended example of how thought technology could be integrated into instruction: “. . .but I would have a problem with trying to demand that they use that [technology]. If you can do it better without the, you know, plugging anything in, well, I say more power to you, just go for it.” The interviews provided insight into why two outputs were not being met—why faculty were not modeling the use of technology and why students were competent but not confident.
  • The other dilemma was to determine why portfolios were not supporting students’ self-reported use of technology. First, the examples students gave in the survey were extended and it seemed unlikely that they were making their examples up. So, working on a hunch, we decided to look more carefully at the requirements for the student teaching portfolio.
  • The College guide to students as well as the requirements did not include the integration of technology. While it encouraged technology integration and required a lesson plan that integrated technology, there was no requirement to implement the technology integration. Similarly, there was nothing in the performance assessment rubric used by faculty to assess student teaching portfolios as far as the integration of technology tools in their instruction.Finally, there was no consensus among faculty as far as whether technology integration was required during student teaching. 50% faculty said no requirement, 25% said they weren’t sure, 25% said there is a requirement suggesting that the faculty are not part of determining the hallmarks of excellent student teaching or the student teaching experience.
  • To measure the quality of teaching that our graduates provided, we conducted classroom observations once they became teachers. During these observations I noticed that all of the former students, now teachers were using technology to enhance instruction. So I began to wonder whether the student teaching portfolios provided a slanted view of pre-service teacher technology skills.
  • I sent our graduates a follow up survey about the use of technology in teaching. It looked at attitudes about using technology and actual teacher and student use of technology beyond just learning to use the tool (e.g., learning how to use a spreadsheet for the sake of learning how to use it as opposed to using a spreadsheet to graph a chart that compares data on a local stream collected at three different times of the year).
  • Results of the survey were quite revealing and contradicted data collected during student teaching. The novice teachers felt using technology to enhance instruction to be important and the majority of teachers used technology to provide instruction on a weekly basis. Most students used technology at least once a month to complete assignments and to learn concepts with those at higher grade levels using it more frequently. Also, the student use of technology was limited by classroom/school resources.
  • Using the original logic model, it was concluded that the project was not achieving its objective and that only one output was realized—grades on the Educational Technology course. Further investigation into why faculty modeling and student implementation of technology into teaching was not being achieved showed that: Faculty were not modeling the use of technology as an instructional tool because they did not have an understanding of ISTE standards, they expected all technology training to take place in the Educational Technology course, and they themselves did not value the use of technology as a planned tool to enhance instruction. Student portfolio requirements did not reflect the assumed integration of technology needed to validate student self-reported use of technology. In fact, it was discovered that students were reluctant to include technology use in their portfolio video because they were concerned it might not work correctly and therefore affect their grade.
  • Classroom observations needed to examine overall teacher quality provided the clue that perhaps the technology objective was actually being achieved in spite of the fact that faculty did not model the use of technology and the student portfolios did not require examples of implementation. Clear evidence of technology implementation was observed during classroom visits. The follow up survey validated the observations and provided data that illustrated that the project participants, once in their own classrooms, use technology on a regular basis to enhance instruction.
  • Long and winding road

    1. 1. The Long and Winding Road: How the Integration of Evaluation Performance Measures and Results Can Lead to Better Quality Evaluations Gale A. Mentzer, PhD Director of Evaluation Services The University of Toledo, Ohio
    2. 2. Required Objective Increase the number of K-12 teachers using educational technology themselves and with their students.
    3. 3. Original Logic Model Inputs Outputs Outcomes Teacher Preparation Program: Course in Educational Technology Minimum 3.0 on required technologycourse. Students are trained in the use of a varietyof educational technologytools. FacultyModeling of use of technology Student reporting on frequency of facultymodeling. All facultymodel the integration of technologyin teaching Student practice through Methods and Student Teaching Student reporting on confidence using specific technologytools 11 on 15 pt scale); on application of technologytools to student teaching (score of 11 on 15 pt. scale); review of student teaching portfolios show evidence of both planned and actual integration of technology. Student are confident using technologytools, are confident using them to enhance instruction, and portifios show the use of technologyduring student teaching .
    4. 4. Actual Results Inputs Outputs Findings Outcomes Teacher Preparation Program: Course in Educational Technology Minimum 3.0 on required technologycourse. Average grade 3.17. Students are trained in the use of a varietyof educational technology tools. FacultyModeling of use of technology Student reporting on frequency of facultymodeling. Students report little faculty modeling of technologyto enhance instruction. Facultynot modeling technology- Are faculty tech-savvy? Student practice through Methods and Student Teaching Student reporting on confidence using specific technologytools; on application of technology tools to student teaching (score of 11 on 15 pt. scale); review of student teaching portfolios. Average confidence score 10 with little confidence using content- specific tools; student report using technologyduring student teaching in a varietyof innovative ways; student portfolios show no or rudimentaryuse of technology Student are below expected confidence levels; students report the use of technologyduring student teaching but artifacts (portfolios) do not support this.
    5. 5. Faculty Modeling • Explore faculty perceptions about the role of technology in their courses • Expectations as to pre-service teacher use of technology • Ask where in program is technology as an instructional tool taught
    6. 6. Outputs • Purpose of technology education is to enhance instruction • Modeling of technology as an instructional tool takes place in all courses • Faculty consciously include assignments and activities that illustrate the use of technology to enhance instruction • Faculty will emphasize the benefits of using technology when teaching Findings • Purpose of technology education is to provide competency • Modeling of technology as an instructional tool takes place in Educational Technology and Methods courses only • Faculty do not plan to use technology but rather do so when convenient to their purposes • Faculty seldom or never share the benefits of using technology when teaching with their students vs
    7. 7. Self-Reporting vs. Portfolios • Review official guidelines and requirements for student teaching portfolios • Examine student teaching portfolio performance assessment rubrics • Ask faculty whether integrating technology as an educational tool is required of student teaching
    8. 8. Outputs • Guide and requirements include evidence of using technology to enhance teaching • Performance assessment rubrics include the evaluation of a technology-enhanced lesson • Faculty agree that integrating technology as an instructional tool is part of student teaching experience Findings • Questions as to how technology might be integrated but no requirement that it actually be integrated • Performance assessment does not include technology integration • Faculty were split as to whether technology integration in student teaching was required or not vs
    9. 9. Wait a minute. . .
    10. 10. UT3 Graduate Technology Survey • Sent to all in-service teachers • Explore attitudes towards using technology in teaching • Determine extent to which teachers and their students use technology to teach and learn
    11. 11. Outputs • Teachers will value the integration of technology into teaching. • Most (80%) of teachers will use technology to provide instruction at least twice a week. • Students will use technology to complete assignments or learn concepts at least once a month. Findings • 68% felt is was essential; 28% felt it was important (96% total) • 94% of the teachers use technology at least twice a week to provide instruction (45% used it 4 or 5 days/week). • 76% use technology at least once a month to complete assignments and 77% use it to learn concepts. vs.
    12. 12. Summative Results
    13. 13. Summative Results