Downloaded 16 times
Uploaded byJahanzeb Jahan
Research
The document discusses research and data collection by the WESTOP organization. It defines research and describes types of research such as quantitative and qualitative research. It discusses why research is important for accountability and effectiveness. It also discusses the US Department of Education's emphasis on randomized controlled trials as the most rigorous research design. The document outlines WESTOP's plan to initially collect prior experience data from participants and potentially collect additional longitudinal data to analyze program effectiveness.
More Related Content
Research
- 1. An Introduction toResearch & WESTOP Data Collection
- 2. Lee Fulmer WESTOPResearch and Technology Committee
- 3. Agenda Why doresearch? What is research? Types of research What does DOE say about research? Exercise WESTOP data collection
- 4. Why do Research?To find out what is effective Accountability Mathematica
- 5. What is Research?Definitions Information gathering Collecting and analyzing new information in order to increase our understanding
- 6. What Research isAttempt to achieve systematically and with the support of data the answer to a question, the resolution to a problem, or the greater understanding of a phenomenon Generation of new information and testing of ideas
- 7. Characteristics of FormalResearch Starts with a question or problem Requires a clear articulation of a goal Follows a specific plan or procedure Often divides the principal problem into more manageable subproblems
- 8. Characteristics of ResearchCont’d 6. Specify assumptions you’re making 7. Collection and interpretation of data
- 9. Types of Research A man from remote village is the first in his village to fly Friend 1: How far did you go? How high did you fly? Friend 2: How did you feel moving so fast? What was it like being above the clouds?
- 10. Types of ResearchQuantitative: answer questions about data that can be measured in order to explain and predict Qualitative: answer questions about nature of phenomena in order to describe phenomena and understand it from the participant’s point of view
- 11. What the Dept.of ED says No Child Left Behind Requires that teachers use techniques and methods that have been shown by “rigorous” research to be effective Randomized control trial
- 12. Randomized Control TrialDefinition : people randomly assigned to an intervention group or to a control group, in order to measure the effects of the intervention Unique feature : Best chance of objectively demonstrating cause and effect Superior to other designs in measuring intervention’s effect
- 13. Potentially Misleading DesignsPre-post study: Examine whether participants improve or regress during intervention Even Start Summer Training and Education Program
- 14. Potentially Misleading DesignsComparison Group:Comparison group chosen by other than randomization Hormone replacement therapy Well-matched comparison groups can suggest trends, but gold standard is control group
- 15. Responses to DOEAmerican Evaluation Association http://www.eval.org/doestatement.htm American Educational Research Association http://www.eval.org/doeaera.htm National Education Association http://www.eval.org/doepage.htm
- 16. Uses of WESTOPData Illustrate program effectiveness Within your institution or agency To potential funders In grant proposals Research on program effectiveness
- 17. Data Collection Twostages Initially collect data on prior experience illustrate program effectiveness Eventually collect additional data that may be relevant for research
- 18. Why Prior Experience?Aggregate data Speaks to legislative intent
- 19. Why Two Stages?Prior experience data a modest data set and perhaps the most important Work out bugs Gauge participation level Second stage will require specifying which data to collect
- 20. Discussion Is thecollection of regional prior experience data useful to you? What other types of regional data would you like to see collected?
- 21. References Identifying andImplementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide . 2003. US. Dept of Education Institute of Education Sciences National Center for Education Evaluation and Regional Assistance. http://www.ed.gov/rschstat/research/pubs/rigorousevid/index.html Downloaded 1/5/05.
- 22. References Practical ResearchPlanning and Design . (1997) Leedy, Paul with contributions by Timothy J. Newby and Peggy A. Ertmer. Prentice Hall, Inc. Sixth Edition.
Editor's Notes
- #5 Accountability has to do with showing our programs make a difference We need to sow this through the data we produce and the comparisons we make, not just people saying it helps
- #6 Research used in everyday speech to cover a broad spectrum of meaning Multiple, misleading meanings Giving and recording known facts and the opinions of others; looking at existing data
- #7 Formal definition What it really means Opinion gathering versus research Ex. Stories of how students feel TRiO has helped them Vs. TRiO students versus others--change in grade between junior and senior year
- #8 World filled with unanswered questions, unresolved problems Observe things that cause us to wonder, to speculate, to ask questions Inquisitive mind is beginning of research What’s the cause of that? What does it mean? With questions like these, research begins 2. A clear unambiguous statement of the problem is critical 3. Carefully planned attack, search and discover mission explicitly planned in advance 4. Whole is composed of sum of it’s parts Main Problem: How do I get from Town A to Town B Subproblems: 1. What is the most direct route? 2. How far do I travel on the freeway? 3.What is the number of the exit I take? Subproblems keep main problem from becoming cumbersome and unwieldy 5. Each subproblem is a hypothesis After the hypothesis comes the data Reach to turn on lamp and it doesn’t turn on Hypotheses burnt out bulb (put in new bulb) not plugged in (plugged in) wire from lamp to wall is defective (lift the cord, lights briefly) forgot to pay electric bill
- #9 Equivalent to axioms in geometry - self-evident truths Example: do students more quickly acquire the ambience of a foreign language (native accent , comprehension of the expressive characteristics of grammatical, idiomatical accentual, etc)by learning only one foreign language at a time or by learning two languages at a time What assumptions underlie such a problem Teacher competent to teach the language and mastered linguistic ambience Student capable of hearing subtleties of accent and alert to other unique characteristics of language Languages selected would have distinguishable ambience characteristics that could be recognized, learned and practiced 8. A. Questioning b. Formally stated as a problem c. Data gathered d. Data point to tentative solution - hypothesis is made e. Quest for more data f. Data processed and interpreted g. Conclusion is reached h. Hypothesis supported or not supported i. Cycle complete
- #11 F1 = quantitative - objective F2 = qualitative - subjective GPA - What is your GPA vs do you get good or bad grades?
- #12 Rigorous or strong research
- #13 Random assignment assumes that attributes of both groups are equal; need to double check Tries to remove any subjective influences
- #14 A randomized controlled trial of Even Start - a federal program designed to improve the literacy of disadvantaged families--found that the program had no effect on improving the school readiness of participating children at the 18th month follow-up. Specifically, there were no significant differences between young children in the program and those in the control group on standard measures of school readiness. If a pre-post design rather than a randomized design had been used in this study, the study would have concluded erroneously that the program was effective in increasing school readiness. This is because both the children in the program and those in the control group showed improvement in school readiness during the course of the study (both groups of children improved substantially in their national percentile ranking on the tests). A pre-post study would have attributed the participants’ improvement to the program whereas in fact it was the result of other factors, as evidenced by the equal improvement for children in the control group. Example: A randomized controlled trial of the Summer Training and Education Program -- a Labor Department pilot program that provided summer remediation and work experience for disadvantaged teenagers--found that program’s short term impact on participant’s reading ability was positive. Specifically, while the reading ability of the control group members eroded by a full grade level during the first summer of the program, the reading ability of participants in the program eroded by only half a grade level. If a pre-post design rather than a randomized design had been used in this study, the study would have concluded erroneously that the program was harmful. That is, the study would have found a decline in participants’ reading ability and attributed it to the program. In fact, however, the participants’ decline in reading ability was a result of other factors--such as the natural erosion of reading ability during the summer vacation months--as evidenced by the even greater decline for members of the control group.
- #15 Example from medicine: Over the past 30 years, more than two-dozen comparison group studies have found hormone replacement therapy for postmenopausal women to be effective in reducing the women’s risk of coronary heart disease, by about 35-50 percent. But when hormone therapy was finally evaluated in two large scale randomized controlled trials--it was actually found to do the opposite In most cases well-matched comparison groups ( prior test scores, demographics, time period in which they are studied and methods used to collect outcome data) seem to yield correct overall results about whether an intervention is effective, ineffective, or harmful However, their estimates of the size of the interventions impact are still often inaccurate In medicine, subjectively give most aggressive therapies to best patients Group needs to be large enough to be comparable in all other relevant features
- #16 Large enough samples to get randomization
- #17 Gives you a comparison group
- #18 Next I’ll talk about why prior experience and why two stages