1. Student Voices & Data-Driven Decision Making Improving Your Academy Dr. Ann G. Bessell (University of Miami) Dr. Cathleen Armstead (University of Miami) National Academy Foundation Staff Development Institute Philadelphia, PA. July, 2010
2. Agenda Overview and Purpose Innovative Techniques: Photolanguage Doing Photolanguage Exploring Data-in-a-Day Analyzing data sets Understanding Data Making Data-Driven Decisions
3. Overview & Purpose To Introduce Innovative Techniques for Self-Assessment & Evaluation – Photolanguage & Data-in-a-Day To Explore and Practice Analyzing Data for Decision-Making Purposes
7. 1 5 Arrange Photos Share Selections & Significance 2 3 4 P H O T O L A N G U A G E Present Focus Question View Photos Select Photos
8. Question #1 Choose a picture that best represents your image of a NAF student when compared to the general population of high school students. Choose a picture that best represents your image of a NAF teacher or leader when compared to the general population of high school teachers. Question #2
9. Challenges & Obstacles Lack of Longitudinal Data Different Data Points Measuring Outcomes Lack of Integrated Data System What’s Next Lack of Comparable Data Teachers: Not Researchers Fear of Retribution Class Time
12. Collect data from multiple sources - surveys - questions - observations -
13. Involve multiple stakeholders: Input from students, faculty & administrators
14. Facilitate a dialogue about school improvement strategies (Ginsberg & Kimball, 2008)
15. Most Important Value Listening Teamwork Encourages Critical Reflection Important Aspects Student Input Promotes Dialogue Representative Teams
16. Our Goal Desired Course We are Here Actual Course Where are we? What is our goal? What course should we follow? How are we doing? 2002 Assessment Committee 12
17. Principal Involvement Teachers & NAF Lead Teachers Involvement Guidance Counselors Explain the process Encourage participation on each team Review schedule From the Start
18. Classroom Observations Review checklist with district and principals at beginning of year Review with principals and teachers at the meet and greet in morning 10-15 minutes of classroom observation
22. 1 5 Review data sets and look for patterns Review & Summarize Areas to Improve 2 Aggregate Quantitative Data 3 Explore Qualitative Data 4 Review & Summarize Strengths G U I D E L I N E S
26. “It sort of makes you stop and think, doesn’t it.”
27. Deciding on Themes More than three students mention in a classroom. There is widespread agreement through nodding or shaking heads Verbal agreement (That’s the truth). Mentioned in more than one classroom with different observers. There are positive and negative corollaries expressed.
29. Students’ Voice What NAF promises: Personalization Empowerment Differentiated Instruction Hands-On Activities Relevant Learning Internships Interdisciplinary projects Connections to the world of work and high school Learning takes place beyond the school What students want: Teachers who know them. Teachers who listen to their opinions. Hands-on Activities Projects and Problem Based Learning Connected to the real world Field trips with assigned academic work
30. Begin with Questions Data Collection Data Analysis Critical Reflection Student Outcomes Multiple Observers Multiple Sources Stakeholder Concerns DECISIONS
31. Data-in-a-Day Approach to data collection and analysis actively engages those involved in the reform,and facilitates critical dialogue about future steps. Through the collaborative process, evaluators and schools work to determine program effectiveness. Consensus after gathering data leads to ongoing dialogue to determine areas on concern, potential strategies for improvement, and next steps. Working collaboratively, the data-in-a-day process culminates in critical conversations about secondary school reform.
32. Thank You! Dr. Ann G. Bessell Dr. Cathleen Armstead agbessell@miami.edu carmstead@miami.edu
Editor's Notes
The University of Miami Education and Evaluation Team (UMEET) is the independent evaluator for Smaller Learning Communities in the Miami-Dade County School District. Our office provides interim reports based on mid-year site visits (which incorporates Data-in-a-Day) and an annual report for each cohort of grantees. UMEET evaluates the grant’s measurable outcomes related to student achievement, school factors, personnel roles and responsibilities, professional development for SLC implementation, continuous program improvement of SLCs along with facilitators and barriers to implementing SLCs. Multiple methods of evaluation were conducted with key stakeholders in the process: administrators, teachers, staff, and students. The multi-method approach to the evaluation provides both depth and breadth to the evaluation, as each method captures a different dimension.For our purposes today we will be focusing on collecting, compiling, and interpreting student perspectives on smaller learning communities – which can be easily translated into research/evaluation on NAF academies.
There are two main goals to our presentation: 1) To introduce two innovative evaluation techniques – Photolanguage and Data-in-a-Day and the 2) is how to understand data, and then how to use the data to make decisions.Throughout the workshop you will be asked to participate – first experiencing Photolanguage, then analyzing sample data sets compiled from our Data-in-a-dayobservations, and discussing what is important to report and how do we make decisions from these.
School wide improvements in student outcomes (student achievement, student attendance, graduation rates and closing the achievement gap) are typically incremental; substantial gains may not be evident until 5 years or more into the process. It is possible that a school could make significant progress towards implementing SLCs that are not yet evident in student outcomes. These specific evaluations concentrated on an assessment of the progress schools are making towards best practices towards implementing smaller learning communities and NAF academies.Student outcomes are more likely to be achieved by schools that are making the most progress towards successful implementation -- Schools that exhibit the characteristics of high impact SLC and NAF programs, together with desirable teaching characteristics can be expected to be schools with positive cultures that offer quality instruction and support systems to ensure that students’ academic, social, and emotional needs are met.
Sometimes it’s not what you collect but how you collect it.Facilitates interactions among individuals and encourages creative and critical thinkingAn unobtrusive probe for gathering information from sensitive populations
Discuss the significance of the picture to the participant, not the picture per se.
Participants in Photolanguage enjoy the experience and we have gotten some very important insights on NAF students and academies. There are still challenges & obstacles to collecting data – includingDifferent data points are required for different entities (District, State, NAF requirements)Teachers in general feel that taking time to collect data – takes time away from teaching and learningTeachers are also not trained researchers – leading to questions about what to collect, what is important, how to be systematic & how to record the dataLast, there is a fear of retribution from the administration – imagined or not Once the data is collected – there are even more difficulties in what to do with itLack of comparable dataLack of longitudinal dataNo user-friendly and integrated data systems.Most of all – is the question of what to do with it next. I’ve had more than one teacher tell me about their data binder – all their students and all their data – and how they carry it around.
During the 2009-2010 academic year, the UMEET team used the participatory approach, Data-in-a-Day (Ginsberg, 2001). A Data-in-a-Day approach allowed us to observe multiple classrooms in a single day, combining short classroom observations, a faculty survey, and a fifteen minute structured student questionnaire, with time for students’ unstructured feedback. Data-in-a-Day is a research tool developed with the dual purposes of involving multiple stakeholders in the self-evaluation process, and facilitating critical dialogue about strategies for school improvement (Ginsberg & Kimball, 2008). Data-in-a-Day is a quick and efficient way of collecting, analyzing, and synthesizing a broad spectrum of data that can be utilized to evaluate the implementation of educational reforms in large schools. It provides timely and effective feedback as schools attempt to improve their practices. District personnel, school faculty and administration were encouraged to conduct classrooms observations and survey students alongside our team. Our experience of conducting evaluation of educational reforms indicates that the Data-in-a-Day process greatly enhances the evaluation. Our previous evaluation of large urban schools consisted of surveys, telephone interviews with SLC design team, and focus group interviews with faculty, administration and students. For each of these methods, schools invariably put their best foot forward, providing extraordinarily positive claims that were unverifiable.The Data-in-a-Day approach to data collection and analysis actively engages those involved in the reform, facilitates critical dialogue about future steps, and enhances the credibility of an external evaluation. No longer were we, as evaluators the designated bearer of bad news, instead we were working together to determine the effectiveness of the program. When the external reviewers and those from the school had similar conclusions after gathering data, there was an immediate discussion on how to enhance the reform and adjust the process. When our conclusions were different, it led the way to fruitful dialogue regarding what was observed, what was missing, and what were the consequences. By working collaboratively, the Data-in-a-day process culminated in a critical conversation about the implementation of secondary school reform.
There are some key important aspects that make data in a day different from classroom walk-throughs: 1) teams with representatives of students, teachers and administrators on each time 2) Promotes a dialogue – as the team is walking from class to class & in the de-briefing 3) Sends a message that the school values student input in addition to teacher & administration’s perspectives 4) Most important, this process stimulates critical reflection and discussion about best practices in and out of the classroom.Best practices includeDesigning user-friendly instrumentsParticipation of the students in data in a dayParticipation of the administration in data in a dayThere is a climate of trustThere is a receptiveness to student opinion
These are the types of questions where we ask students to raise their hands.
What you have in your packets are five classroom observation worksheets which are combined with the structured and unstructured questions we ask of students. Students raise their hands for structured questions and we count hands.During the unstructured questions, students are free to make any comments they feel are useful. And they do. Often, it is how something is said that leads to a clear analysis and that is missing from the worksheets.However, much of what is said is captured verbatim and used in the interim reports.The data sets you have are not actual worksheets, rather they are compilations of typical worksheets.Your task is to decide what are the important variables from a student’s perspective, how important and why? What is the picture of smaller learning communities or academies at this particular school?HINT #1: Pay attention to the classroom as well as the students.HINT #2: Count and tabulate and look for patterns in students words
What you have in your packets are five classroom observation worksheets which are combined with the structured and unstructured questions we ask of students. Students raise their hands for structured questions and we count hands.During the unstructured questions, students are free to make any comments they feel are useful. And they do. Often, it is how something is said that leads to a clear analysis and that is missing from the worksheets.However, much of what is said is captured verbatim and used in the interim reports.The data sets you have are not actual worksheets, rather they are compilations of typical worksheets.Your task is to decide what are the important variables from a student’s perspective, how important and why? What is the picture of smaller learning communities or academies at this particular school?HINT #1: Pay attention to the classroom as well as the students.HINT #2: Count and tabulate and look for patterns in students words
Questions:How would you calculate positive experience in an academy? For example, in the first class, a required class, the percentage of positive experience could be 100% (6 out of 6) or it could be 32% - 6 out of 19. Which one better reflects the reality in the classroom.Why are the numbers higher for academy classes than for other classes?How do you explain the numbers for the early childhood academy?Note what is going on in the classroomNote also the “choice” aspectWhy would 50% of students in the NAF Finance Academy @ this particular school willing to leave if external constraints were removed?What is the most important area to improve? Why? How?
I would stay for the academy classes – Saturday school Academies (Business)University OutreachMedical magnet (Academy)Extracurricular activities (related to academy)Some [teachers] help more than othersHow do you decide on themes?Hands-on activitiesField tripsCaring teachersToo much on FCATMore programs like the health academyWe need to study what we’re interested inMore academies More than three students mention in class or there is affirmation when said (head nodding, verbal agreement) More than one class mention (note different observers) There are positive & negative corollaries to themes: e.g. we need more caring teachers similar to so me help more than others.Having administrators and teachers participate in this research drives the point home. It is one thing to hear guidance counselors are not accessible from an outside observer, it is a very different thing to hear spontaneous laughter when a question is asked.Students, when asked, are both serious and thoughtful about good classes, good teachers, what they want to learn. Yes, students mentioned the cafeteria food often, and the strict cell phone rules. But most students, most of the time, focused on academic issues such as learning, hands-on activities, tutoring (such as Saturday school)
Having administrators and teachers participate in this research drives the point home. It is one thing to hear guidance counselors are not accessible from an outside observer, it is a very different thing to hear spontaneous laughter when a question is asked.Uniforms: consistency about the rulesCell phones: especially during lunch is a huge issue and fight with students. Students, when asked, are both serious and thoughtful about good classes, good teachers, what they want to learn. Yes, students mentioned the cafeteria food often, and the strict cell phone rules. But most students, most of the time, focused on academic issues such as learning, hands-on activities, tutoring (such as Saturday school)
