1. The Long and Winding Road:
How the Integration of
Evaluation Performance
Measures and Results Can
Lead to Better Quality
Evaluations
Gale A. Mentzer, PhD
Director of Evaluation Services
The University of Toledo, Ohio
3. Original Logic Model
Inputs Outputs Outcomes
Teacher Preparation
Program:
Course in Educational
Technology
Minimum 3.0 on required
technologycourse.
Students are trained in the use of
a varietyof educational
technologytools.
FacultyModeling of use of
technology
Student reporting on frequency
of facultymodeling.
All facultymodel the integration of
technologyin teaching
Student practice through
Methods and Student
Teaching
Student reporting on confidence
using specific technologytools
11 on 15 pt scale); on
application of technologytools
to student teaching (score of 11
on 15 pt. scale); review of
student teaching portfolios show
evidence of both planned and
actual integration of
technology.
Student are confident using
technologytools, are confident
using them to enhance
instruction, and portifios show the
use of technologyduring student
teaching .
4. Actual Results
Inputs Outputs Findings Outcomes
Teacher Preparation
Program:
Course in Educational
Technology
Minimum 3.0 on required
technologycourse.
Average grade 3.17. Students are trained in the
use of a varietyof
educational technology
tools.
FacultyModeling of use of
technology
Student reporting on frequency
of facultymodeling.
Students report little faculty
modeling of technologyto
enhance instruction.
Facultynot modeling
technology- Are faculty
tech-savvy?
Student practice through
Methods and Student
Teaching
Student reporting on confidence
using specific technologytools;
on application of technology
tools to student teaching (score
of 11 on 15 pt. scale); review of
student teaching portfolios.
Average confidence score 10 with
little confidence using content-
specific tools; student report
using technologyduring student
teaching in a varietyof innovative
ways; student portfolios show no
or rudimentaryuse of technology
Student are below
expected confidence
levels; students report the
use of technologyduring
student teaching but
artifacts (portfolios) do not
support this.
5. Faculty Modeling
• Explore faculty perceptions about
the role of technology in their
courses
• Expectations as to pre-service
teacher use of technology
• Ask where in program is
technology as an instructional tool
taught
6. Outputs
• Purpose of technology
education is to enhance
instruction
• Modeling of technology as an
instructional tool takes place
in all courses
• Faculty consciously include
assignments and activities
that illustrate the use of
technology to enhance
instruction
• Faculty will emphasize the
benefits of using technology
when teaching
Findings
• Purpose of technology
education is to provide
competency
• Modeling of technology as an
instructional tool takes place in
Educational Technology and
Methods courses only
• Faculty do not plan to use
technology but rather do so
when convenient to their
purposes
• Faculty seldom or never share
the benefits of using technology
when teaching with their
students
vs
7. Self-Reporting vs. Portfolios
• Review official guidelines and
requirements for student teaching
portfolios
• Examine student teaching portfolio
performance assessment rubrics
• Ask faculty whether integrating
technology as an educational tool
is required of student teaching
8. Outputs
• Guide and requirements
include evidence of using
technology to enhance
teaching
• Performance assessment
rubrics include the evaluation
of a technology-enhanced
lesson
• Faculty agree that integrating
technology as an
instructional tool is part of
student teaching experience
Findings
• Questions as to how technology
might be integrated but no
requirement that it actually be
integrated
• Performance assessment does
not include technology
integration
• Faculty were split as to whether
technology integration in
student teaching was required
or not
vs
10. UT3 Graduate Technology Survey
• Sent to all in-service teachers
• Explore attitudes towards using
technology in teaching
• Determine extent to which
teachers and their students use
technology to teach and learn
11. Outputs
• Teachers will value the
integration of technology into
teaching.
• Most (80%) of teachers will
use technology to provide
instruction at least twice a
week.
• Students will use technology
to complete assignments or
learn concepts at least once
a month.
Findings
• 68% felt is was essential; 28%
felt it was important (96% total)
• 94% of the teachers use
technology at least twice a
week to provide instruction
(45% used it 4 or 5 days/week).
• 76% use technology at least
once a month to complete
assignments and 77% use it to
learn concepts.
vs.
This is an extended examination of the process of evaluating one project objective. The purpose of the presentation is to illustrate that even the best thought-out logic models or evaluation plans may need adjustments in order to provide data that can truly contribute to understanding process and outcomes.
The US Department of Education Teacher Quality Enhancement grants have eight required objectives. We will look at one of those objectives in the “evaluation case study”. The goal of the project was to recruit, prepare, and retain high quality K-12 science and math teachers for high needs urban school districts. To develop the logic model for this objective, I consulted with project senior personnel to determine how they planned to provide the training and practice to achieve this objective.
Here is an abbreviated version of the logic model that focuses on technology. There are 3 major inputs that all fall under the umbrella of coursework. The outputs are gathered from three sources—(1) student grades on the technology course; (2) a student survey given to students at the conclusion of their teacher preparation program that asked them to report on their confidence using a variety of technology tools as specified by the International Society for Technology Education IISTE), in which courses faculty model technology integration was observed, and how they integrated technology into their student teaching; and (3) an examination of student teaching portfolios.
Here are the results. There were disparities in two major areas. Students reported that they saw little modeling of the integration of technology and instruction outside of the required Educational Technology course. They also scored below expected levels in confidence using technology tools—especially tools that are content-specific like probes, graphing calculators, simulated labs, and math programs. Their self-reported used of technology during student teaching showed innovative and frequent integration. However, their student teaching portfolios (the artifact of student teaching that is graded) did not support this. So now we needed to reflect. Were the inputs we assumed really there? To investigate, we began with the premise that faculty understanding of the integration of technology may not coincide with ISTE standards. We needed to explore faculty perceptions as to the role of technology in teaching and in the teacher preparation program.
So we conducted an in-depth, qualitative study to explore faculty perceptions of the role technology should and does play in the courses they teach. It also examined how they themselves use technology when teaching (specific examples), how they expect their students to use technology when teaching, and where in the teacher education program the use of technology as an instructional tool should or does occur. We used interviews and qualitative research analyses to explore common themes.
The interviews provided insight into why students were competent yet not confident. The major theme was that the pre-service teachers needed to be able to “use” the technology so that they did not appear to be incompetent. There was little discussion of the value of integrating technology into teaching. Here are some quotes from the interviews:Yeah, we do have conversations about when it's appropriate to do different things. One of the things that we actually talk about a lot is how to use the space that you have, like your board, or your, when you're using a SMARTboard and how to help make things visual for students. So we, sometimes we'll stop and talk about is this really the best way to do this or not. But we do that really broadly across a lot of things that we, not just in terms of technology. But, like when we're using graphing software, we talk about where's the appropriate place. You know, where does this go in the curriculum? You know, where does it fit, and when's the best time to use it and how you might use it with one group of learners vs. another. “I don't [model the integration of technology] because I don't think of it as a thoughtful process. I've, I just think of it as, as a, just an opportunity, as just something you do. It's not something you necessarily planned for.” And one faculty member commented after an extended example of how thought technology could be integrated into instruction: “. . .but I would have a problem with trying to demand that they use that [technology]. If you can do it better without the, you know, plugging anything in, well, I say more power to you, just go for it.” The interviews provided insight into why two outputs were not being met—why faculty were not modeling the use of technology and why students were competent but not confident.
The other dilemma was to determine why portfolios were not supporting students’ self-reported use of technology. First, the examples students gave in the survey were extended and it seemed unlikely that they were making their examples up. So, working on a hunch, we decided to look more carefully at the requirements for the student teaching portfolio.
The College guide to students as well as the requirements did not include the integration of technology. While it encouraged technology integration and required a lesson plan that integrated technology, there was no requirement to implement the technology integration. Similarly, there was nothing in the performance assessment rubric used by faculty to assess student teaching portfolios as far as the integration of technology tools in their instruction.Finally, there was no consensus among faculty as far as whether technology integration was required during student teaching. 50% faculty said no requirement, 25% said they weren’t sure, 25% said there is a requirement suggesting that the faculty are not part of determining the hallmarks of excellent student teaching or the student teaching experience.
To measure the quality of teaching that our graduates provided, we conducted classroom observations once they became teachers. During these observations I noticed that all of the former students, now teachers were using technology to enhance instruction. So I began to wonder whether the student teaching portfolios provided a slanted view of pre-service teacher technology skills.
I sent our graduates a follow up survey about the use of technology in teaching. It looked at attitudes about using technology and actual teacher and student use of technology beyond just learning to use the tool (e.g., learning how to use a spreadsheet for the sake of learning how to use it as opposed to using a spreadsheet to graph a chart that compares data on a local stream collected at three different times of the year).
Results of the survey were quite revealing and contradicted data collected during student teaching. The novice teachers felt using technology to enhance instruction to be important and the majority of teachers used technology to provide instruction on a weekly basis. Most students used technology at least once a month to complete assignments and to learn concepts with those at higher grade levels using it more frequently. Also, the student use of technology was limited by classroom/school resources.
Using the original logic model, it was concluded that the project was not achieving its objective and that only one output was realized—grades on the Educational Technology course. Further investigation into why faculty modeling and student implementation of technology into teaching was not being achieved showed that: Faculty were not modeling the use of technology as an instructional tool because they did not have an understanding of ISTE standards, they expected all technology training to take place in the Educational Technology course, and they themselves did not value the use of technology as a planned tool to enhance instruction. Student portfolio requirements did not reflect the assumed integration of technology needed to validate student self-reported use of technology. In fact, it was discovered that students were reluctant to include technology use in their portfolio video because they were concerned it might not work correctly and therefore affect their grade.
Classroom observations needed to examine overall teacher quality provided the clue that perhaps the technology objective was actually being achieved in spite of the fact that faculty did not model the use of technology and the student portfolios did not require examples of implementation. Clear evidence of technology implementation was observed during classroom visits. The follow up survey validated the observations and provided data that illustrated that the project participants, once in their own classrooms, use technology on a regular basis to enhance instruction.
