Imran oomer sample project impact measurement in education
Upcoming SlideShare
Loading in...5
×
 

Imran oomer sample project impact measurement in education

on

  • 539 views

 

Statistics

Views

Total Views
539
Views on SlideShare
539
Embed Views
0

Actions

Likes
0
Downloads
6
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Imran oomer sample project impact measurement in education Imran oomer sample project impact measurement in education Presentation Transcript

  • Lend-A-Hand India: PLAN100 IMPACT ASSESSMENT PILOT STUDY: METHODOLOGY AND PRELIMINARY FINDINGS BY IMRAN OOMER
  • Background Lend-A-Hand India (LAHI) is developing a robust impactmeasurement system to track the short-, medium-, and long term impact on the students’ lives it touches through Plan100.Current measurement focuses on New measurements willquantitative measures of understand behavioralstudent achievement: outcomes of its program, such as:Student attendance • Increases in student confidenceExamination pass rates • Personal aspirationsGraduation rates • Entrepreneurial spirit
  • Project Objectives LAHI is at the initial stages of developing an impact measurement system. This study will lay the groundwork for it, and specifically accomplish the following: • …a set of student outcomes expected from participation in Define the IBT (Introduction to Basic Technology) Program • …an assessment tool to test hypothesized outcomes with Develop students through one-on-one interviews • …a pilot study testing the assessment tool across a sample Execute of IBT and non-IBT students (N>100) • …preliminary findings from pilot study assessing areas of high, Review moderate and low impact ; make recommendations for future impact measurement activity
  • METHODOLOGY
  • The Big Picture• An example of what impact assessment aims to measures and achieve Constant Feedback Loop Institution- System- Individual Input Output Wide wide Impact Impact Impact Newly Well- More Improved Increase in the revamped trained , engaged scores on population’s teacher energized students, standardized job-readiness training and resulting in tests as a result of a program motivated higher higher teachers attendance proportion of high school Impact assessment is a method of gaining constant feedback graduates continuing on on the design of your organization’s programs with the aim of to tertiary refining them in real-time to achieve the results you want education
  • Defining Outcomes and Metrics The following expected student outcomes (near- and long- term) as a result of participation in the IBT program were identified during the initial phase of the study (slide 1 of 2): Educational Achievement: Skills Attainment:• Improved school attendance • Writing skills• Improved performance in • Organizational skills regular curriculum • Project management skills• Improved performance in 10th • Problem-solving skills standard examination • Basic understanding of• Heightened interest in STEM- economics (e.g. costs, demand) related learning (science, • Vocational skills (e.g. technology, engineering, math) engineering, electric wiring)• Heightened interest in pursuing further education
  • Defining Outcomes and Metrics Behavioral Changes: Employability / Value to Society:• Confidence in self • Career aspirations• Increase in self-worth / feeling of self- • Ability to apply skills outside of school reliance • Aspirations to give back to one’s• Sense of accomplishment community• Pride in work • Employment status after 2 /4 / 6 years• Reduced stress related to school / • Increase in family involvement and examinations commitment to children’s education• Increased comfort in asking questions • Increase in community involvement and• Increased comfort in working with commitment to local education groups (including mixed gender groups) institutions• Increase in creativity • Improved local economy with better job• Increase in entrepreneurial spirit opportunities and a well-educated community The pilot study has illuminated preliminary findings to support several expected outcomes across the four defined categories. The analysis, however, will focus on behavioral outcomes – the least studied outcomes of the IBT program
  • The Assessment The assessment begins with each student answering the following four unaided, open-ended questions in written form, anonymously. The purpose is to achieve: a) an unbiased view of student preferences of school subjects, specifically in favor of or against STEM-related courses, and b) students’ unbiased evaluation of their school and IBT (where applicable)  Q1: What is your favorite subject in school? Why?  Q2: What is your least favorite subject in school? Why?  Q3: What do you like most about your school*? Why?  Q4: What would you change about your school*? Why?*IBT students were asked specific about the IBT program in Q3 and Q4 to gain a deeper understanding of studentperceptions of the program and its benefits
  • The Assessment One-one-one interviews with each student and ethnographic observation are the core of the assessment. The interviews combine qualitative and quantitative measures (complete assessment tool can be found in the appendix) The following is an excerpt from the tool where the students rate the importance of specific skills learned at school and within the IBT program. The purpose is to measure where IBT and non-IBT students differ in their perception of skills attainment, and which specific skillsets emerge as the most significant for students in the IBT programQ1: Please read the following 14 skill areas. Which FIVE do you feel arethe most important skills you are learning at school?Study skills Creativity Reading/writing MathScience Communication skills Computer skills Career planning skillsTime mgmt. skills Business skills Organizational skills Problem solving skillsJob-specific skills Teamwork skills
  • The Assessment The tool also includes an 11-question self-esteem battery, adapted from the widely accepted self-esteem scale developed by Dr. Morris Rosenberg*. Students rate their level of agreement with each statement (below) on a 4- point scale from strongly agree to strongly disagree. The questions aim to gain insight into key behavioral changes among IBT students. I am proud of my projects and homework I enjoy working in groups I am proud of my marks at school I enjoy working in mixed groups (boys and girls, combined) I feel positive that I will pass the 10th standard I like to ask questions to my teachers examination I am excited about my future I feel that I make my parents / family proud I feel that I will have great opportunities after I If there is a problem, I believe I can fix it finish school I feel that I am gaining many good skills at school*Source: http://www.bsos.umd.edu/socy/research/rosenberg.htm
  • Sampling and Administration During the pilot study, data was collected from 109 students across four secondary schools in rural Maharashtra. Survey counts by gender and category are below: Survey Sample: Impact Assessment Pilot Study Total sample: 109 students IBT: 60 students (55%) Non-IBT (control group): 49 students (45%) 9th Standard: 57 students (52%) 10th Standard: 52 students (48%) Male: 61 students (56%) Female: 48 students (44%) The surveys were administered in Marathi, the students’ local language with a duration of 5 -10 minutes.
  • PRELIMINARY FINDINGS
  • Preliminary Findings Most important skills learned through school / IBT Top 5 (in order of frequency) Most Selected Skills IBT Students Non-IBT Students Study skills Study skills Teamwork skills Reading/writing Problem-solving skills Problem-solving skills Career-planning skills Science Business skills Computer skills IBT students are more likely to prioritize outcomes-focused, non- academic skills such as business skills, teamwork, and career- planning skills – all core foci of the IBT curriculum
  • Preliminary Findings Effect of IBT Program on Importance Placed on Skill Attainment (slide 1 of 2)Skill Control % IBT % % Effect Selected SelectedCareer planning skills 20% 43% 112%Time management skills 27% 18% (-31%)Business skills 33% 42% 28%Organizational skills 24% 13% (-46%)Problem solving skills 42% 43% 1%Job-specific skills 30% 33% 9%Teamwork skills 41% 45% 10% Highlighted skills are those that IBT students chose more often than non-IBT students as being a Top 5 important skill learned at school.
  • Preliminary Findings Effect of IBT Program on Importance Placed on Skill Attainment (slide 2 of 2)Skill Control % IBT % % Effect Selected SelectedStudy skills 71% 70% (-2%)Creativity 14% 22% 52%Reading/Writing 53% 37% (-31%)Math 41% 38% (-6%)Science 43% 40% (-7%)Communication skills 18% 15% (-18%)Computer skills 43% 38% (-11%)Negative effect does not indicate reduced impact on specific skill areas; instead, it is a reflection on the prioritization given by students on other skills
  • Preliminary Findings By overlaying the self-esteem question set with the proposed outcomes of the IBT program, we identified 6 key behavioral success factors to assess: I like to I am ask excited I am questions about my proud of future If there is my I feel a problem, projects positive I am I can fix it that I Do you proud of will want to my grades I feel like I pass I like working start your I have have many great in groups (and own learned opps. mixed-gender business? good skills groups) Measures and indicators to assess factors
  • Preliminary FindingsMeasured Outcome Control IBT %on a 100 ptscale Score Score Difference Entrepreneurial Spirit 65.3% 71.7% 9.7% Factors Most Affected Among Self-Pride / 3.61 3.67 1.6%Select Survey Questions Self-Worth Self-Esteem Battery + IBT Students Self-Confidence 3.58 3.66 2.1% Personal Aspirations 3.51 3.40 -3.1% Self-Reliance 3.51 3.58 2.1% Interpersonal Skills 3.41 3.58 4.9%
  • Preliminary Findings Learnings / Recommendations • Pride in one’s work • Group work and practicals, “learning byAreas of • Recognition of one’s doing,” has had the biggest impact on High achievements students’ confidence and interpersonal skills • Identify how these areas can be leveraged to Impact • Entrepreneurial spirit positively influence students’ long-term view • Gender equality and development • Confidence in test • Self-reliance (the ability to solve problems Areas of taking and think independently) is a core expectedModerate • Confidence in asking outcome of IBT and should be emphasized Impact questions more during the program; same for asking questions, where many students are still • Self-reliance hesitant • Family pride in child’s • Connect more with parents and familiesAreas of accomplishments about what IBT is offering their children.Low/No This will not only positively affect the • Perception of a bright Impact future families and the students’ self-esteems, but will help in getting families to contribute fees
  • Preliminary Findings Student Aspirations for Continuing Education Q: After 10th standard, what are you interesting in doing? Control IBT Go to ITI or another technical 18% 22% school to earn my diploma Attend junior college (11th and 12th) 78% 74% Go work with my family 0% 2% Go work somewhere else 0% 0% I don’t know 4% 2%While IBT students are slightly more interested than their peers in pursuing technical education after 10th standard, the majority are still interested in junior college. Recommendation:LAHI should evaluate its position as a feeder into diploma programs and how this incentive is being communicated to IBT students and families
  • WRAP-UP
  • Key Findings From Pilot Study LAHI has created positive impact for IBT students across a number of measures. In this study, the most notable outcomes are: Transmitting the Inspiring Future Facilitating Strong Importance of Life Entrepreneurs Interpersonal Skills Skills Learning• IBT students are twice • Specifically, an IBT • IBT students are more as likely as their peers student is 10% more likely than their peers to view career likely to be interested to feel comfortable planning as a in starting his or her working in groups, fundamental skill own shop or business particular mixed learned during their • IBT students’ ideas range gender groups (7% secondary education from tailoring and welding effect)• Business, problem-solving, shops to computer • This is a significant and job-specific skills are institutes and medical achievement in schools also perceived to be of clinics where rote learning and higher importance among individual achievement are IBT students the norms
  • Recommendations For Lend-A-Hand India In areas of moderate and In areas of high impact… low impact… Identify drivers (e.g. teaching Pinpoint gaps between expected styles, level of practicals) of and actual outcomes (e.g. students’ student achievement that can comfort in asking questions) and be leveraged towards improving determine programmatic changes on other measures that can be tested to minimize achievement gapsQuestions to consider: What outcomes exceeded expectations? Which measures were less impacted by the program than what was expected? How can LAHI leverage areas of high impact (e.g. entrepreneurial drive) to positively affect areas of moderate (e.g. creativity) and low impact (e.g. family pride)
  • Recommendations For Further Impact WorkImpact measurement is not a static, annual process to appease funders; it is a dynamic, continuous effort aimed at understanding, refining and improving upon an organization’s programs and outcomes Be aware of measurement pitfalls*, such as:  Trying to assess a whole program instead of specific components  Aspiring for scientific-level design that is appropriate for large scale populations when striving for generalizable trends but not for context-specific learning where a finer, more customized and qualitative approach often yields richer results  Focusing on uncontrollable , long-term community impact metrics as a measure of a program’s success over tangible, measureable, near- term outcomes *R&D: A New Form of Evaluation, TCC Group
  • Recommendations For Further Impact Work Expanding on the pilot study, future steps should include:  Communicating expected outcomes and measurement goals to IBT schools to onboard key stakeholders (e.g. headmaster, instructors) to a more robust assessment system  Tracking outcomes by standard to understand when and how outcomes evolve over program duration  Scaling assessment across all IBT programs with the aim of identifying and replicating star school strategies for maximizing outcomes  Expanding the next phase of the study to include more control variables such as family involvement, family income level, academic achievement, etc.
  • APPENDIX
  • Student Verbatims – Positive Reflections on IBT “I will always have the opportunity to do something because of IBT” “Because of IBT, we can get jobs anywhere from nursing to engineering” “I like IBT because we get knowledge of different things” “I chose IBT because if I don’t get a proper job, I can start my own business” “We get to learn many different and new things; IBT taught me how to make use of waste” “I get to know the benefits of IBT because in our future, we can do wiring, prepare chikki, and first aid solutions on our own. IBT is going to help us a lot more in our future.” “Even if I cannot take further education, I can do something or the other and make a living because of what I’ve learned through IBT” With the help of IBT, you learn how to solve problems and start a small-scale business in the future. “ “We get to learn many things that can help our family.” “It’s very beneficial and it’s the only place where girls and boys can work together”
  • Student Verbatims – Suggestions for IBT “We should be taught about computers as well” “Internet should be added as the 5th section of IBT” “Each student should be required to do each practicals because some don’t do it” “Practicals should be increased” “IBT should double its syllabus and newer things should be taught” “Sometimes materials are in shortage and when tools get broken, it takes long to get them fixed” “An electronic switchboard should be provided in all four sections” “There should be more developed machineries for the practicals so we can learn more things”
  • IBT Impact Assessment Tool (slide 1 of 4)Pre-screener questions:  School  Standard  Gender  IBT Status (IBT or Non-IBT)Self-reported questionnaire:  Q1: What is your favorite subject in school? Why?  Q2: What is your least favorite subject in school? Why?  Q3: What do you like most about your school*? Why?  Q4: What would you change about your school*? Why?*IBT students were asked specific about the IBT program in Q3 and Q4 to gain a deeper understanding of studentperceptions of the program and its benefits
  • IBT Impact Assessment Tool (slide 2 of 4)Interview questions:  Q5: Please read the following 14 skill areas. Which FIVE do you feel are the most important skills you are learning at school? A) Study skills B) Creativity C) Reading/Writing D) Math E) Science F) Communication skills G) Computer skills H) Career planning skills I) Time management skills J) Business skills K) Organizational skills L) Problem solving skills M) Job-specific skills N) Teamwork skills  Q6: After 10th standard, what do you want to do? A) Go to ITI or other school to get a diploma B) Go to Junior College C) Go work with my family D) Go work somewhere else E) I don’t know
  • IBT Impact Assessment Tool (slide 3 of 4) Q7: What is your dream job?__________________ Q8: Why?_______________________________ Q9: Do you ever want to start your own shop or business? A) Yes B) No Q10: What kind of shop or business?________________________________ Q11: Please reach each statement carefully. For each statement, tell me if you: A) Strongly agree B) Agree C) Disagree D) Strongly Disagree  I am proud of my projects and homework  I am proud of my marks at school  I feel positive that I will pass the 10th standard examination
  • IBT Impact Assessment Tool (slide 4 of 4) Q11 (continued): Please reach each statement carefully. For each statement, tell me if you: A) Strongly agree B) Agree C) Disagree D) Strongly Disagree  I am excited about my future  I feel that I can have great opportunities after I finish school  I feel that I have gained many good skills at school  I enjoy working in groups  I enjoy working with mixed groups (boys and girls)  I like to ask questions to my teachers  I feel that I make my parents / family proud  If there is a problem, I believe I can fix it