An Introduction to a NewResearch Paradigm: ‘Design-      Based Research’       Michael K. Barbour         Assistant Profes...
It all began last spring when I read two queries from doctoral students on theQualitative Research for the Human Sciences ...
• Vast resources  going into  education research  are wasted• We [educational  researchers]  employ weak  research methods...
• Too much useless  work is done  under the  banner of  qualitative  research• Qualitative research…. [yields]  ….little t...
Problems with Educational       Technology Research1.Misunderstanding about basic and applied  research.2.Poor quality of ...
Problems with Educational        Technology Research1.Misunderstanding about basic and applied  research.2.Poor quality of...
Basic and Applied Research• Basic (aka fundamental or pure)  research is driven by a scientists  curiosity or interest in ...
Basic and Applied Research
Problems with Educational       Technology Research1.Misunderstanding about basic and applied  research.2.Poor quality of ...
Poor Quality of Educational        Technology Research“The best current evidence is thatmedia are mere vehicles thatdelive...
• Abundant technology  has not led to extensive  use of computers for  “tradition-altering  classroom instruction.”• The s...
What Does the Secondary E-LearningResearch Say About Student Performance?
Student Performance• performance of virtual  and classroom students  in Alberta were similar  in English and Social  Studi...
Student Performance• over half of the students who  completed FLVS courses  scored an A in their course and  only 7% recei...
Student Performance• FLVS students performed  better on a non-mandatory  assessment tool than students  from the tradition...
Let’s look a little closer...
Students and Student PerformanceBallas &      performance of virtual and    participation rate in theBelyk, 2000   classro...
Students and Student PerformanceCavanaugh et FLVS students performed      speculated that the virtualal., 2005    better o...
Student Performance and StudentsBut are we reallycomparing apples toapples?
The Students• the vast majority of VHS  Global Consortium students  in their courses were  planning to attend a  four-year...
The StudentsThe preferred characteristicsinclude the highly motivated,self-directed, self-disciplined,independent learner ...
The Students• “only students with a high  need to control and structure  their own learning may  choose distance formats  ...
The Students• the typical online student  was an A or B student  (Mills, 2003)• 45% of the students who  participated in e...
Problems with Educational       Technology Research1.Misunderstanding about basic and applied  research.2.Poor quality of ...
Disappointing Research SynthesisKannapel and DeYoung (1999) found that rural schoolscontained a strong sense of community ...
Disappointing Research SynthesisIn their review of key literature over the past 25 years,Kannapel and DeYoung (1999) found...
Disappointing Research SynthesisIn their review of key literature over the past 25 years,Kannapel and DeYoung (1999) found...
QuickTime™ and a          decompressorare needed to see this picture.
Examining Effect Sizes                Teacher                Effects      Zone of                             Desired Effe...
Primary & Secondary E-Learning          Meta-Analysis• Cavanaugh (2001) - 16 studies  – +0.147 in favor of K-12 distance e...
Results of Interest• Second and third chance programs (d=0.50)• Matching style of learning (d=0.40)• Computer assisted ins...
Results to Consider•   Providing formative evaluation (d=0.90)•   Micro teaching (d=0.88)•   Teacher clarity (d=0.75)•   P...
Maybe The Problem Is How We    Conduct Research?
Randomized Controlled Trials1. Is there a control group?2. Are the control and   experimental groups   assigned randomly?3...
http://ies.ed.gov/ncee/wwc/
What Works ClearinghouseIronically, the WWC personnel have been able to identifyvery few educational programs and practice...
QuickTime™ and a          decompressorare needed to see this picture.
QuickTime™ and a          decompressorare needed to see this picture.
More Problematic ResearchOnline         7 principles of    Interviews with teachers and courseCourse         effective onl...
There Must Be A Better Way!
Design-Based Research                   Reeves (2006)
Design-Based Research• an emphasis on  conducting research in  authentic, natural  educational contexts,  rather than labo...
Design-Based Research• an insistence on theory-guided  educational intervention:  ‘interventions embody specific  theoreti...
Design-Based Research• the use of an iterative  design and evaluation  cycle: interventions are  adjusted as the research ...
Virtual High School Global Consortium • first annual evaluation   – Kozma, Zucker &     Espinoza, 1998 • focused specifica...
Virtual High School Global Consortium • second annual   evaluation   – Espinoza, Dove, Zucker     & Kozma, 1999 • again fo...
Virtual High School Global Consortium • third annual evaluation   – Kozma, Zucker,     Espinoza, McGhee,     Yarnall & Zal...
Virtual High School Global Consortium • content-specific   investigations   – Yamashiro & Zucker, 1999 • examined quality ...
Virtual High School Global Consortium • content-specific   investigations   – Elbaum, McIntyre &     Smith, 2002 • sevente...
Virtual High School Global Consortium • final evaluation   – Zucker & Kozma, 2003 • examined students,   teachers,   admin...
YourQuestions  andComments
Assistant Professor     Wayne State University, USA        mkbarbour@gmail.com   http://www.michaelbarbour.comhttp://virtu...
Sabbatical (Massey University) - An Introduction to a New Research Paradigm: Design-Based Research
Sabbatical (Massey University) - An Introduction to a New Research Paradigm: Design-Based Research
Sabbatical (Massey University) - An Introduction to a New Research Paradigm: Design-Based Research
Upcoming SlideShare
Loading in …5
×

Sabbatical (Massey University) - An Introduction to a New Research Paradigm: Design-Based Research

631 views

Published on

Barbour, M. K. (2011, April). An introduction to a new research paradigm: Design-based research. An invited presentation to the National Centre for Teaching and Learning at Massey University, Palmerston North, New Zealand.

Published in: Education
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
631
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Sabbatical (Massey University) - An Introduction to a New Research Paradigm: Design-Based Research

  1. 1. An Introduction to a NewResearch Paradigm: ‘Design- Based Research’ Michael K. Barbour Assistant Professor Wayne State University
  2. 2. It all began last spring when I read two queries from doctoral students on theQualitative Research for the Human Sciences listserv. Both students came fromlarge public institutions of higher education, one in the USA and the other inCanada. The first student wrote that she intended to focus her dissertationresearch on the quality of "discourse" that takes place in cafes and coffee shopslocated inside bookstores. She complained that she had found no "literature" onthis topic and asked the listserv participants for some guidance. The secondstudent announced that he was preparing a dissertation prospectus centered on thequestion of how people learned about opportunities to take SCUBA divinglessons and what motivated them to register for such courses. He also soughtdirections to relevant literature and advice from the listserv membership.After pondering these queries, I posted a message asking whether facultymembers at taxpayer-supported universities have a moral responsibility to guidetheir students toward "socially responsible" research questions. In my posting, Isuggested that in the face of problems such as adult illiteracy, attacks on publiceducation, "at-risk" students, homelessness, AIDS, and the like, faculty membersshould attempt to inspire in students a dedication to research that would "make adifference."Thomas Reeves, University of GeorgiaPeter Dean Lecture at the 1995 Association for Educational Communications
  3. 3. • Vast resources going into education research are wasted• We [educational researchers] employ weak research methods, write turgid prose, and issue
  4. 4. • Too much useless work is done under the banner of qualitative research• Qualitative research…. [yields] ….little that can be generalized beyond the classrooms in which it is conducted
  5. 5. Problems with Educational Technology Research1.Misunderstanding about basic and applied research.2.Poor quality of educational technology research.3.Disappointing research synthesis.
  6. 6. Problems with Educational Technology Research1.Misunderstanding about basic and applied research.2.Poor quality of educational technology research.3.Disappointing research synthesis.
  7. 7. Basic and Applied Research• Basic (aka fundamental or pure) research is driven by a scientists curiosity or interest in a scientific question. The main motivation is to expand mans knowledge, not to create or invent something.• Applied research is designed to solve practical problems of the modern world, rather than to acquire knowledge for knowledges sake. One might say that the goal of the applied scientist is to improve the human condition.
  8. 8. Basic and Applied Research
  9. 9. Problems with Educational Technology Research1.Misunderstanding about basic and applied research.2.Poor quality of educational technology research.3.Disappointing research synthesis.
  10. 10. Poor Quality of Educational Technology Research“The best current evidence is thatmedia are mere vehicles thatdeliver instruction but do notinfluence student achievementany more than the truck thatdelivers our groceries causeschanges in our nutrition.” Richard Clark Review of Educational Research 1983
  11. 11. • Abundant technology has not led to extensive use of computers for “tradition-altering classroom instruction.”• The small percentage of computer-using instructors only use it to maintain existing classroom practices.
  12. 12. What Does the Secondary E-LearningResearch Say About Student Performance?
  13. 13. Student Performance• performance of virtual and classroom students in Alberta were similar in English and Social Studies courses, but that classroom students performed better overall in all other subject areas (Ballas & Belyk, 2000)
  14. 14. Student Performance• over half of the students who completed FLVS courses scored an A in their course and only 7% received a failing grade (Bigbie & McCarroll, 2000)• students in the six virtual schools in three different provinces performed no worse than the students from the three conventional schools (Barker & Wendel, 2001)
  15. 15. Student Performance• FLVS students performed better on a non-mandatory assessment tool than students from the traditional classroom (Cavanaugh et al., 2005)• FLVS students performed better on an assessment of algebraic understanding than their classroom counterparts (McLeod et al., 2005)
  16. 16. Let’s look a little closer...
  17. 17. Students and Student PerformanceBallas & performance of virtual and participation rate in theBelyk, 2000 classroom students similar assessment among virtual in English & Social Studies students ranged from 65% to courses, but classroom 75% compared to 90% to students performed better 96% for the classroom-based in all other subject areas studentsBigbie & over half of the students between 25% and 50% ofMcCarroll, who completed FLVS students had dropped out2000 courses scored an A in of their FLVS courses over their course and only 7% the previous two-year received a failing grade period
  18. 18. Students and Student PerformanceCavanaugh et FLVS students performed speculated that the virtualal., 2005 better on a non- school students who did mandatory assessment take the assessment may tool than students from have been more the traditional classroom academically motivated and naturally higher achieving studentsMcLeod et FLVS students performed results of the studental., 2005 better on an assessment performance were due to of algebraic understanding the high dropout rate in than their classroom virtual school courses counterparts
  19. 19. Student Performance and StudentsBut are we reallycomparing apples toapples?
  20. 20. The Students• the vast majority of VHS Global Consortium students in their courses were planning to attend a four-year college (Kozma, Zucker & Espinoza, 1998)• “VHS courses are predominantly designated as ‘honors,’ and students enrolled are mostly college bound” (Espinoza et al., 1999)
  21. 21. The StudentsThe preferred characteristicsinclude the highly motivated,self-directed, self-disciplined,independent learner whocould read and write well,and who also had a stronginterest in or ability withtechnology (Haughey &Muirhead, 1999)
  22. 22. The Students• “only students with a high need to control and structure their own learning may choose distance formats freely” (Roblyer & Elbaum, 2000)• IVHS students were “highly motivated, high achieving, self-directed and/or who liked to work independently” (Clark et al., 2002)
  23. 23. The Students• the typical online student was an A or B student (Mills, 2003)• 45% of the students who participated in e-learning opportunities in Michigan were “either advanced placement or academically advanced” students (Watkins, 2005)
  24. 24. Problems with Educational Technology Research1.Misunderstanding about basic and applied research.2.Poor quality of educational technology research.3.Disappointing research synthesis.
  25. 25. Disappointing Research SynthesisKannapel and DeYoung (1999) found that rural schoolscontained a strong sense of community and wereregularly the focus of the cultural and social aspects ofthe community. They also found that “extracurricularand non-academic activities are often valued as much ormore than academics, and a higher proportion ofstudents participate in extracurricular activities than inurban schools” (p. 170).
  26. 26. Disappointing Research SynthesisIn their review of key literature over the past 25 years,Kannapel and DeYoung (1999) found that rural schoolscontained a strong sense of community and wereregularly the focus of the cultural and social aspects ofthe community. They also found that “extracurricularand non-academic activities are often valued as much ormore than academics, and a higher proportion ofstudents participate in extracurricular activities than inurban schools” (p. 170).
  27. 27. Disappointing Research SynthesisIn their review of key literature over the past 25 years,Kannapel and DeYoung (1999) found that rural schoolscontained a strong sense of community and wereregularly the focus of the cultural and social aspects ofthe community. They also found that “extracurricularand non-academic activities are often valued as much ormore than academics, and a higher proportion ofstudents participate in extracurricular activities than inurban schools” (p. 170). It should be noted that three ofthe eleven “key” pieces of literature were written byDeYoung, one of the two authors of this review.
  28. 28. QuickTime™ and a decompressorare needed to see this picture.
  29. 29. Examining Effect Sizes Teacher Effects Zone of Desired EffectsDevelopmentalEffectsReverseEffects
  30. 30. Primary & Secondary E-Learning Meta-Analysis• Cavanaugh (2001) - 16 studies – +0.147 in favor of K-12 distance education• Cavanaugh et al. (2004) - 14 studies – -0.028 for K-12 distance education• Means et al. (2009) - 46 studies (5 on K-12) – +0.24 favoring online over face-to-face* – +0.35 favoring blended over face-to-face*
  31. 31. Results of Interest• Second and third chance programs (d=0.50)• Matching style of learning (d=0.40)• Computer assisted instruction (d=0.37)• Decreasing disruptive behavior (d=0.34)• Programmed instruction (d=0.24)• Individualized instruction (d=0.23)• Class size (d=0.21)• Charter schools (d=0.20)• Web-based learning (d=0.18)• Home-school programs (d=0.16)• Teacher training (d=0.11)• Teacher subject matter knowledge (d=0.09)• Distance education (d=0.09)• Student control over learning (d=0.04) 33 33
  32. 32. Results to Consider• Providing formative evaluation (d=0.90)• Micro teaching (d=0.88)• Teacher clarity (d=0.75)• Providing feedback (d=0.73)• Teacher-student relationships (d=0.72)• Teaching strategies (d=0.60)• Cooperative vs. individualistic learning (d=0.59)• Study skills (d=0.59)• Direct instruction (d=0.59)• Mastery learning (d=0.58)• Worked examples (d=0.57)• Concept mapping (d=0.57)• Goals (d=0.56)• Peer tutoring (d=0.55)• Cooperative vs. competitive learning (d=0.54)
  33. 33. Maybe The Problem Is How We Conduct Research?
  34. 34. Randomized Controlled Trials1. Is there a control group?2. Are the control and experimental groups assigned randomly?3. If it is a matched study, are the groups extremely similar?4. Is the sample size large enough?5. Are the results statistically significant?
  35. 35. http://ies.ed.gov/ncee/wwc/
  36. 36. What Works ClearinghouseIronically, the WWC personnel have been able to identifyvery few educational programs and practices that have theevidence that is sufficiently rigorous according to theirown criteria to warrant their inclusion in the What Worksdatabase. For example, a review of over 1,300 studies thatexamined the effect of teacher professional developmenton student achievement found that only nine met WWCstandards for rigorous evidence (Yoon, Duncan, Lee, Scarloss, & Shapley, 2007).Can Educational Research Be Both Rigorous and Relevant?Thomas Reeves,2011http://www.educationaldesigner.org/ed/volume1/issue4/article13/index.htm
  37. 37. QuickTime™ and a decompressorare needed to see this picture.
  38. 38. QuickTime™ and a decompressorare needed to see this picture.
  39. 39. More Problematic ResearchOnline 7 principles of Interviews with teachers and courseCourse effective online developers at a single virtual school,Design course content with no verification of whether the for adolescent interviewees’ perceptions were actuallyBarbour learners effective (or any student input for that(2005; 2007) matter)Online 37 best Interviews with teachers at a singleTeaching practices in virtual school selected by the virtual asynchronous school itself. Their teachers’ beliefsDiPietro et online teaching were not validated through observational. (2008) of the teaching or student performance.
  40. 40. There Must Be A Better Way!
  41. 41. Design-Based Research Reeves (2006)
  42. 42. Design-Based Research• an emphasis on conducting research in authentic, natural educational contexts, rather than laboratories• the desire for research to have a practical impact, by having clear relevance for the improvement of education (Walker, nd)
  43. 43. Design-Based Research• an insistence on theory-guided educational intervention: ‘interventions embody specific theoretical claims about teaching and learning’• a pluralist approach with respect to theories, research designs, methods, and procedures - includes mixed methods where qualitative and quantitative techniques are used in combination (Walker, nd)
  44. 44. Design-Based Research• the use of an iterative design and evaluation cycle: interventions are adjusted as the research proceeds• a focus on how the intervention worked (Walker, nd)
  45. 45. Virtual High School Global Consortium • first annual evaluation – Kozma, Zucker & Espinoza, 1998 • focused specifically on the seven goals set by VHS • identified five areas to focus on for future practice
  46. 46. Virtual High School Global Consortium • second annual evaluation – Espinoza, Dove, Zucker & Kozma, 1999 • again focused specifically on the seven goals set by VHS • identified three areas to focus on for future practice
  47. 47. Virtual High School Global Consortium • third annual evaluation – Kozma, Zucker, Espinoza, McGhee, Yarnall & Zalles, 2000 • re-examined status of last year’s evaluation finding • focused upon only one of the seven goals set by VHS
  48. 48. Virtual High School Global Consortium • content-specific investigations – Yamashiro & Zucker, 1999 • examined quality of netcourses offered by VHS • developed standards for future course development
  49. 49. Virtual High School Global Consortium • content-specific investigations – Elbaum, McIntyre & Smith, 2002 • seventeen essential elements for online teaching • written by VHS staff
  50. 50. Virtual High School Global Consortium • final evaluation – Zucker & Kozma, 2003 • examined students, teachers, administrators perceptions of the program • outlined successes and areas to focus on for future years
  51. 51. YourQuestions andComments
  52. 52. Assistant Professor Wayne State University, USA mkbarbour@gmail.com http://www.michaelbarbour.comhttp://virtualschooling.wordpress.com

×