Published on

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Accountability has to do with showing our programs make a difference We need to sow this through the data we produce and the comparisons we make, not just people saying it helps
  • Research used in everyday speech to cover a broad spectrum of meaning Multiple, misleading meanings Giving and recording known facts and the opinions of others; looking at existing data
  • Formal definition What it really means Opinion gathering versus research Ex. Stories of how students feel TRiO has helped them Vs. TRiO students versus others--change in grade between junior and senior year
  • World filled with unanswered questions, unresolved problems Observe things that cause us to wonder, to speculate, to ask questions Inquisitive mind is beginning of research What’s the cause of that? What does it mean? With questions like these, research begins 2. A clear unambiguous statement of the problem is critical 3. Carefully planned attack, search and discover mission explicitly planned in advance 4. Whole is composed of sum of it’s parts Main Problem: How do I get from Town A to Town B Subproblems: 1. What is the most direct route? 2. How far do I travel on the freeway? 3.What is the number of the exit I take? Subproblems keep main problem from becoming cumbersome and unwieldy 5. Each subproblem is a hypothesis After the hypothesis comes the data Reach to turn on lamp and it doesn’t turn on Hypotheses burnt out bulb (put in new bulb) not plugged in (plugged in) wire from lamp to wall is defective (lift the cord, lights briefly) forgot to pay electric bill
  • Equivalent to axioms in geometry - self-evident truths Example: do students more quickly acquire the ambience of a foreign language (native accent , comprehension of the expressive characteristics of grammatical, idiomatical accentual, etc)by learning only one foreign language at a time or by learning two languages at a time What assumptions underlie such a problem Teacher competent to teach the language and mastered linguistic ambience Student capable of hearing subtleties of accent and alert to other unique characteristics of language Languages selected would have distinguishable ambience characteristics that could be recognized, learned and practiced 8. A. Questioning b. Formally stated as a problem c. Data gathered d. Data point to tentative solution - hypothesis is made e. Quest for more data f. Data processed and interpreted g. Conclusion is reached h. Hypothesis supported or not supported i. Cycle complete
  • F1 = quantitative - objective F2 = qualitative - subjective GPA - What is your GPA vs do you get good or bad grades?
  • Rigorous or strong research
  • Random assignment assumes that attributes of both groups are equal; need to double check Tries to remove any subjective influences
  • A randomized controlled trial of Even Start - a federal program designed to improve the literacy of disadvantaged families--found that the program had no effect on improving the school readiness of participating children at the 18th month follow-up. Specifically, there were no significant differences between young children in the program and those in the control group on standard measures of school readiness. If a pre-post design rather than a randomized design had been used in this study, the study would have concluded erroneously that the program was effective in increasing school readiness. This is because both the children in the program and those in the control group showed improvement in school readiness during the course of the study (both groups of children improved substantially in their national percentile ranking on the tests). A pre-post study would have attributed the participants’ improvement to the program whereas in fact it was the result of other factors, as evidenced by the equal improvement for children in the control group. Example: A randomized controlled trial of the Summer Training and Education Program -- a Labor Department pilot program that provided summer remediation and work experience for disadvantaged teenagers--found that program’s short term impact on participant’s reading ability was positive. Specifically, while the reading ability of the control group members eroded by a full grade level during the first summer of the program, the reading ability of participants in the program eroded by only half a grade level. If a pre-post design rather than a randomized design had been used in this study, the study would have concluded erroneously that the program was harmful. That is, the study would have found a decline in participants’ reading ability and attributed it to the program. In fact, however, the participants’ decline in reading ability was a result of other factors--such as the natural erosion of reading ability during the summer vacation months--as evidenced by the even greater decline for members of the control group.
  • Example from medicine: Over the past 30 years, more than two-dozen comparison group studies have found hormone replacement therapy for postmenopausal women to be effective in reducing the women’s risk of coronary heart disease, by about 35-50 percent. But when hormone therapy was finally evaluated in two large scale randomized controlled trials--it was actually found to do the opposite In most cases well-matched comparison groups ( prior test scores, demographics, time period in which they are studied and methods used to collect outcome data) seem to yield correct overall results about whether an intervention is effective, ineffective, or harmful However, their estimates of the size of the interventions impact are still often inaccurate In medicine, subjectively give most aggressive therapies to best patients Group needs to be large enough to be comparable in all other relevant features
  • Large enough samples to get randomization
  • Gives you a comparison group
  • Next I’ll talk about why prior experience and why two stages
  • Research

    1. 1. An Introduction to Research & WESTOP Data Collection
    2. 2. Lee Fulmer WESTOP Research and Technology Committee
    3. 3. Agenda <ul><li>Why do research? </li></ul><ul><li>What is research? </li></ul><ul><li>Types of research </li></ul><ul><li>What does DOE say about research? </li></ul><ul><li>Exercise </li></ul><ul><li>WESTOP data collection </li></ul>
    4. 4. Why do Research? <ul><li>To find out what is effective </li></ul><ul><li>Accountability </li></ul><ul><li>Mathematica </li></ul>
    5. 5. What is Research? <ul><li>Definitions </li></ul><ul><ul><li>Information gathering </li></ul></ul><ul><ul><li>Collecting and analyzing new information in order to increase our understanding </li></ul></ul>
    6. 6. What Research is <ul><li>Attempt to achieve systematically and with the support of data the answer to a question, the resolution to a problem, or the greater understanding of a phenomenon </li></ul><ul><li>Generation of new information and testing of ideas </li></ul>
    7. 7. Characteristics of Formal Research <ul><li>Starts with a question or problem </li></ul><ul><li>Requires a clear articulation of a goal </li></ul><ul><li>Follows a specific plan or procedure </li></ul><ul><li>Often divides the principal problem into more manageable subproblems </li></ul>
    8. 8. Characteristics of Research Cont’d <ul><li>6. Specify assumptions you’re making </li></ul><ul><li>7. Collection and interpretation of data </li></ul>
    9. 9. Types of Research <ul><li>A man from remote village is the first in his village to fly </li></ul><ul><li>Friend 1: How far did you go? How high did you fly? </li></ul><ul><li>Friend 2: How did you feel moving so fast? What was it like being above the clouds? </li></ul>
    10. 10. Types of Research <ul><li>Quantitative: answer questions about data that can be measured in order to explain and predict </li></ul><ul><li>Qualitative: answer questions about nature of phenomena in order to describe phenomena and understand it from the participant’s point of view </li></ul>
    11. 11. What the Dept. of ED says <ul><li>No Child Left Behind </li></ul><ul><li>Requires that teachers use techniques and methods that have been shown by “rigorous” research to be effective </li></ul><ul><li>Randomized control trial </li></ul>
    12. 12. Randomized Control Trial <ul><li>Definition : people randomly assigned to an intervention group or to a control group, in order to measure the effects of the intervention </li></ul><ul><li>Unique feature : Best chance of objectively demonstrating cause and effect </li></ul><ul><li>Superior to other designs in measuring intervention’s effect </li></ul>
    13. 13. Potentially Misleading Designs <ul><li>Pre-post study: Examine whether participants improve or regress during intervention </li></ul><ul><ul><li>Even Start </li></ul></ul><ul><ul><li>Summer Training and Education Program </li></ul></ul>
    14. 14. Potentially Misleading Designs <ul><li>Comparison Group:Comparison group chosen by other than randomization </li></ul><ul><li>Hormone replacement therapy </li></ul><ul><li>Well-matched comparison groups can suggest trends, but gold standard is control group </li></ul>
    15. 15. Responses to DOE <ul><li>American Evaluation Association </li></ul><ul><ul><li> </li></ul></ul><ul><li>American Educational Research Association </li></ul><ul><ul><li> </li></ul></ul><ul><li>National Education Association </li></ul><ul><ul><li> </li></ul></ul>
    16. 16. Uses of WESTOP Data <ul><li>Illustrate program effectiveness </li></ul><ul><ul><li>Within your institution or agency </li></ul></ul><ul><ul><li>To potential funders </li></ul></ul><ul><ul><li>In grant proposals </li></ul></ul><ul><li>Research on program effectiveness </li></ul>
    17. 17. Data Collection <ul><li>Two stages </li></ul><ul><ul><li>Initially collect data on prior experience </li></ul></ul><ul><ul><ul><li>illustrate program effectiveness </li></ul></ul></ul><ul><ul><li>Eventually collect additional data that may be relevant for research </li></ul></ul>
    18. 18. Why Prior Experience? <ul><li>Aggregate data </li></ul><ul><li>Speaks to legislative intent </li></ul>
    19. 19. Why Two Stages? <ul><li>Prior experience data a modest data set and perhaps the most important </li></ul><ul><li>Work out bugs </li></ul><ul><li>Gauge participation level </li></ul><ul><li>Second stage will require specifying which data to collect </li></ul>
    20. 20. Discussion <ul><li>Is the collection of regional prior experience data useful to you? </li></ul><ul><li>What other types of regional data would you like to see collected? </li></ul>
    21. 21. References <ul><li>Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide . 2003. US. Dept of Education Institute of Education Sciences National Center for Education Evaluation and Regional Assistance. Downloaded 1/5/05. </li></ul>
    22. 22. References <ul><li>Practical Research Planning and Design . (1997) Leedy, Paul with contributions by Timothy J. Newby and Peggy A. Ertmer. Prentice Hall, Inc. Sixth Edition. </li></ul>