Cue Forum2008


Published on

CUE Forum presented at JALT 2008 (Tokyo, Japan). Gives an overview of research design issues for Second Language Acquisition. For further details, visit

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Cue Forum2008

    1. 1. Research design: The backbone of academic inquiry Peter Neff – Doshisha University Matthew Apple – Nara National College of Technology David Beglar – Temple University Japan CUE Forum 2008 Olympic Memorial Youth Center November 1, 2008, 1:15 - 2:50 p.m.
    2. 2. <ul><li>Introduction: The importance of good research design </li></ul><ul><li>Approaching the study </li></ul><ul><li>Developing the study design </li></ul><ul><li>Break for Q&A </li></ul><ul><li>Designing the right instrument </li></ul><ul><li>Implementing your design </li></ul><ul><li>Conclusion </li></ul><ul><li>Further Q&A </li></ul>Overview
    3. 3. Introduction: The importance of good research design
    4. 4. Poor design vs. Good design <ul><li>Poorly-designed study </li></ul><ul><ul><li>Hit on an idea, dive right in </li></ul></ul><ul><ul><li>No background research </li></ul></ul><ul><ul><li>Throw together a survey, give to a group of unwary participants </li></ul></ul><ul><ul><li>Collect data, then ponder how to analyze </li></ul></ul><ul><ul><li>Run to a colleague for help </li></ul></ul><ul><ul><li>Fish around for most “interesting” findings </li></ul></ul><ul><ul><li>Pray to get published </li></ul></ul>
    5. 5. Poor design vs. Good design <ul><li>Well-designed study </li></ul><ul><ul><li>Hit on an idea, do background research </li></ul></ul><ul><ul><li>Formulate relevant, specific, practical RQs </li></ul></ul><ul><ul><li>Consider participants, context, data analysis in advance </li></ul></ul><ul><ul><li>Decide/develop instrument; pilot and revise it </li></ul></ul><ul><ul><li>Decide on appropriate pre/post-test instruments </li></ul></ul><ul><ul><li>Plan stages and structure of data collection </li></ul></ul><ul><ul><li>Prepare participants adequately </li></ul></ul><ul><ul><li>… Then carry out the study </li></ul></ul>
    6. 6. The importance of good design <ul><li>A well-designed study provides many benefits: </li></ul><ul><ul><li>Demonstrates researcher knowledge </li></ul></ul><ul><ul><li>Ties the study to an underlying philosophy </li></ul></ul><ul><ul><li>Provides a clear path for the researcher(s) </li></ul></ul><ul><ul><li>Helps avoid mishaps of previous studies </li></ul></ul>
    7. 7. The importance of good design <ul><li>Other benefits </li></ul><ul><ul><li>Leads to more concrete results, more definitive conclusions </li></ul></ul><ul><ul><li>Improves chances of publication </li></ul></ul><ul><ul><li>Raises the status of SLA as a field of inquiry </li></ul></ul>
    8. 8. A word about mixed methods designs <ul><li>The great quan-qual debate </li></ul><ul><li>Mixed methods – the “best” of both worlds </li></ul><ul><li>Add a qualitative component to a quantitatively-oriented study: </li></ul><ul><ul><li>Participant interviews </li></ul></ul><ul><ul><li>Observational, audiovisual data </li></ul></ul><ul><ul><li>Open-ended survey questions </li></ul></ul><ul><li>Plan for qualitative analyses (text analysis, response coding) </li></ul>
    9. 9. Part 1 Approaching the Study
    10. 10. Approaching the study <ul><li>Hitting upon research ideas </li></ul><ul><li>Review of the literature </li></ul><ul><li>Formulating research questions </li></ul>
    11. 11. Hitting upon research ideas <ul><li>Identify the topic in a few words </li></ul><ul><li>Reflect on “doability” of research </li></ul><ul><ul><li>Can I research this? </li></ul></ul><ul><ul><li>Should I research this? </li></ul></ul><ul><ul><li>Am I interested in researching this? </li></ul></ul><ul><li>Review of the literature can help redefine and revise ideas </li></ul>
    12. 12. Identifying the topic: Hints for starting to narrow <ul><li>Pose a short question using “what” or “how” </li></ul><ul><li>Write a short title that consists of one sentence under 12 words </li></ul><ul><li>Ask a friend or colleague to read your topic and gauge their reactions </li></ul><ul><li>Draft research questions to see if the topic can be adequately explored </li></ul>
    13. 13. A “researchable” topic <ul><li>“ Can I do this in my current situation?” </li></ul><ul><li>“ Does this concern people at other institutions?” </li></ul><ul><li>“ Does this add to the current body of research related to this topic?” </li></ul><ul><li>“ Does this study contribute something from a unique perspective?” </li></ul>
    14. 14. Filtering “probably not so good” ideas: <ul><li>To boldly go where no research has gone before… (The “Star Trek” idea) </li></ul><ul><li>My theory is clearly better than X (The “Steven Krashen is so wrong” idea) </li></ul><ul><li>My classroom is totally unique (The “I don’t need theory” idea) </li></ul><ul><li>This is a really cool technology / methodology / text book (The “I am primarily a teacher” idea) </li></ul>
    15. 15. Filtering ideas: A few hints <ul><li>Review research designs and statistical techniques </li></ul><ul><li>Review teaching methods and overall SLA research results </li></ul><ul><li>Evaluate access to potential study participants </li></ul><ul><li>Plan time for material creation, study design, and implementation </li></ul>
    16. 16. Review of the literature <ul><li>Relate the study to continuing “dialogue” in current research </li></ul><ul><li>Finding a “gap” in the literature </li></ul><ul><li>Provide a framework for the importance of the study </li></ul>
    17. 17. Review of the Literature: Finding a “gap” in knowledge <ul><li>“ We do not enough about X…” </li></ul><ul><li>“ This way of looking at X has never been done…” </li></ul><ul><li>“ This way of learning about X has not been duplicated in my context” </li></ul><ul><li>“ Previous research has inadequately explored X…” </li></ul>
    18. 18. Finding literature: Some hints <ul><li>Google Scholar using key words or researcher names </li></ul><ul><li>Scour recent literature review articles </li></ul><ul><li>Check for “cited” numbers online </li></ul><ul><li>Get access to university databases </li></ul><ul><li>Refer to recently published articles </li></ul><ul><ul><li>After the year 2000 </li></ul></ul><ul><ul><li>During the previous 2 to 3 years </li></ul></ul><ul><li>Examine “outside the field” articles </li></ul>
    19. 19. Finding literature: Separating the wheat… <ul><li>“ Top tier” journal articles </li></ul><ul><li>Most-often-cited articles </li></ul><ul><li>Recent articles </li></ul><ul><li>Research articles (not reviews) </li></ul><ul><li>Books / Edited book-articles </li></ul><ul><li>Major international conference papers </li></ul><ul><li>Dissertations / dissertation abstracts </li></ul>
    20. 20. … from the chaff <ul><li>“ In-house” journal articles </li></ul><ul><li>Articles from “proceedings” books </li></ul><ul><li>Online journal articles with only .html versions </li></ul><ul><li>Unedited books from small publishers </li></ul><ul><li>Newspaper and magazine articles </li></ul><ul><li>Web pages </li></ul><ul><li>Anecdotal evidence </li></ul><ul><li>Your own previous papers for an MA course </li></ul>
    21. 21. Research questions: A few useful guidelines <ul><li>Naturally flow from the literature review </li></ul><ul><li>Strongly connected to the topic </li></ul><ul><li>At least two or three (not one)… </li></ul><ul><li>… but not five or six or more </li></ul><ul><li>As specific as possible </li></ul><ul><li>Directly concern variables in the study </li></ul><ul><li>Do not contain yes/no question words </li></ul>
    22. 22. RQs: What not to ask <ul><li>“ Is X true/false?” </li></ul><ul><li>“ Will X happen if…?” </li></ul><ul><li>“ Does X cause Y?” </li></ul><ul><li>“ What do participants think of X?” </li></ul><ul><li>“ Why does X happen?” </li></ul>
    23. 23. RQs: What to ask <ul><li>“ What differences exist between…” </li></ul><ul><li>“ Compared to X, how does Y…?” </li></ul><ul><li>“ To what degree do X and Y differ…?” </li></ul><ul><li>“ When X is controlled for Y…, how does Z…?” </li></ul><ul><li>“ What are underlying patterns among…?” </li></ul><ul><li>“ To what degree does X predict Y?” </li></ul>
    24. 24. Part 2 Developing the Study Design
    25. 25. Research design <ul><li>Cross-sectional design: A design in which data are collected from a sample at only one point in time. </li></ul><ul><li>Longitudinal design: A design in which data are collected at more than one point in time. </li></ul>
    26. 26. Randomized Control-Group Pretest-Posttest Design <ul><li>Experimental Group 1 T 1 Xa (Method a ) T 2 </li></ul><ul><li>Experimental Group 2 T 1 Xb (Method b ) T 2 </li></ul><ul><li>Control Group T 1 T 2 </li></ul>
    27. 27. Randomized Control-Group Pretest-Posttest Design <ul><li>Reasonably strong conclusions can be reached about the effects of the treatments. </li></ul><ul><li>Problem 1: Within session variation (e.g., different teachers or room conditions) may intervene. </li></ul><ul><li>The solution? Randomly assigning participants, times, and places to the experimental and control conditions. </li></ul><ul><li>Problem 2: The pretest may interact with the treatment. This potential problem is dealt with in the next design. </li></ul>
    28. 28. Randomized Solomon Six-Group Design <ul><li>Pretested (Random assignment) T 1 Xa (Method a ) T 2 </li></ul><ul><li>Pretested (Random assignment) T 1 Xb (Method b ) T 2 </li></ul><ul><li>Pretested (Random assignment) T 1 T 2 </li></ul><ul><li>Unpretested (Random assignment) Xa (Method a ) T 2 </li></ul><ul><li>Unpretested (Random assignment) Xb (Method b ) T 2 </li></ul><ul><li>Unpretested (Random assignment) T 2 </li></ul>
    29. 29. Randomized Solomon Six-Group Design <ul><li>This design amounts to doing the experiment twice –once with and once without pretesting. </li></ul><ul><li>It is possible to know what effects, if any, are associated with pretesting. </li></ul><ul><li>If the results of the “two experiments” are consistent, greater confidence can be placed in the findings. </li></ul>
    30. 30. Counterbalanced Design <ul><li>This design is useful when randomization is not possible and intact groups must be used. </li></ul>C B C D 4 B D A C 3 A A D B 2 D C B A 1 Xd Xc Xb Xa Replication
    31. 31. Counterbalanced design <ul><li>The counterbalanced design rotates out the participants’ differences (e.g., one group has more aptitude or motivation than the other groups) by exposing each group to all variations of the treatment. </li></ul><ul><li>Order-of-presentation effects are controlled. </li></ul><ul><li>Primary weakness: The possibility of carryover effects from one treatment to the next exists. Allowing time between treatments can alleviate this problem. </li></ul>
    32. 32. Control-Group Time-Series Design Experimental Group 1 T 1 T 2 T 3 T 4 Xa (Method a ) T 5 T 6 T 7 T 8 Experimental Group 2 T 1 T 2 T 3 T 4 Xb (Method b ) T 5 T 6 T 7 T 8 Control Group T 1 T 2 T 3 T 4 T 5 T 6 T 7 T 8
    33. 33. Control-Group Time-Series Design <ul><li>This design allows the researcher to determine growth over time, and the effect of an intervention. </li></ul><ul><li>The presence of a control group increases the trustworthiness of the results because the possibility of a contemporary event causing any gains can be determined. </li></ul>
    34. 34. Control-Group Time-Series Design <ul><li>This design can be extended by exposing the participants to the intervention on multiple occasions. </li></ul><ul><li>This approach is more sensitive to partial gains in knowledge and tests the strength of the intervention more than once, thus giving the researcher a more accurate understanding of the effectiveness of the intervention. </li></ul><ul><li>Experimental Group 1 T 1 T 2 Xa T 3 T 4 Xa T 5 T 6 </li></ul><ul><li>Experimental Group 2 T 1 T 2 Xb T 3 T 4 Xb T 5 T 6 </li></ul><ul><li>Control Group T 1 T 2 T 3 T 4 T 5 T 6 </li></ul>
    35. 35. Q&A Break
    36. 36. Part 3 Designing the Right Instrument
    37. 37. Instrument Design <ul><li>Commonly used instruments in SLA research </li></ul><ul><ul><li>Scored tests </li></ul></ul><ul><ul><li>Rater scores </li></ul></ul><ul><ul><li>Surveys </li></ul></ul><ul><ul><li>Interviews </li></ul></ul><ul><li>Consider your eventual data analysis when developing instruments </li></ul>
    38. 38. Instruments - Scored tests <ul><li>Pluses </li></ul><ul><li>Quantitative items (M/C, Cloze/C-tests) </li></ul><ul><ul><li>Simple to score large # of participants </li></ul></ul><ul><ul><li>Easier to analyze </li></ul></ul><ul><li>Qualitative items (short answer, timed essays) </li></ul><ul><ul><li>Good complement to quantitative scores </li></ul></ul><ul><ul><li>Can provide more in-depth assessment of participants’ abilities </li></ul></ul><ul><li>Minuses </li></ul><ul><li>Quantitative items </li></ul><ul><ul><li>Limited to one type of data </li></ul></ul><ul><li>Qualitative items </li></ul><ul><ul><li>Take more time/effort to score </li></ul></ul><ul><ul><li>Rater bias </li></ul></ul>
    39. 39. Instruments – Performance ratings <ul><li>An assessment of participants’ performance in an assigned task </li></ul><ul><li>Tasks may include presentations, interviews, written essays </li></ul><ul><li>Performances can be scored using a Likert-scale, a rubric, or holistically </li></ul><ul><li>Usually scored by at least two “expert” raters; sometimes also by peers </li></ul>
    40. 40. Performance ratings <ul><li>Rating criteria should be concretely established with little ambiguity </li></ul><ul><li>Avoid including too many (or too few) criteria for one performance task </li></ul><ul><li>All raters should undergo a “normative” training session prior to assessment </li></ul><ul><li>Use models to train raters </li></ul><ul><li>Avoid single-score holistic ratings </li></ul>
    41. 41. Instruments - Surveys <ul><li>Often used for: </li></ul><ul><ul><li>Collecting learner history data (L2 study experience, other background info) </li></ul></ul><ul><ul><li>Assessing participants’ attitudes towards a predetermined construct (language learning motivation, anxiety using the L2) </li></ul></ul><ul><ul><li>Determining reactions to an experimental treatment (teaching methods, innovative learning tasks) </li></ul></ul>
    42. 42. Survey making <ul><li>For non-advanced learners – surveys should be in their L1 </li></ul><ul><li>Build in redundancy - Include multiple questions for each concept area </li></ul><ul><li>Questions should be simply worded – avoid negative or confusing wording </li></ul><ul><li>Depending on the purpose of the survey, 20-40 items/session is a good range to shoot for </li></ul>
    43. 43. Survey making <ul><li>Any survey used in a serious study should be piloted in advance </li></ul><ul><li>It is acceptable to make adjustments to an existing instrument </li></ul><ul><li>Likert-scale items should usually have between 4 and 6 choices </li></ul><ul><li>A few qualitative questions can provide a nice complement to quantitative instruments </li></ul>
    44. 44. Instruments - Interviews <ul><li>Interviews can provide an excellent qualitative component to a larger study </li></ul><ul><li>It is not necessary to interview all participants </li></ul><ul><ul><li>a subsample as small as 10-20% can be acceptable </li></ul></ul><ul><li>Use your best judgment on participants’ language ability </li></ul><ul><ul><li>For intermediate-and-above learners, L2 interviews are often fine </li></ul></ul>
    45. 45. Conducting interviews <ul><li>Inform students they are being interviewed, obtain consent </li></ul><ul><li>Record unobtrusively </li></ul><ul><li>“ Warm up” the participants before getting into the heart of the interview </li></ul><ul><li>Collect more data than you need </li></ul>
    46. 46. Validating Instruments
    47. 47. Instrument Validity <ul><li>The construct = The heart of the matter </li></ul><ul><li>What construct do you wish to measure? </li></ul><ul><li>How do you define the construct? </li></ul><ul><li>What are its component parts? Do they form a unified whole? </li></ul>
    48. 48. Operationalizing the construct: The items <ul><li>Conceptualize the construct as a continuum: easy—difficult items and less able—more able persons. </li></ul><ul><li>How have other researchers measured the construct? </li></ul><ul><li>Write original or adapted items. </li></ul><ul><li>Cover the estimated range of your participants. </li></ul><ul><li>Write 50% more items than you intend to use. This will allow you to “cherry pick” the best items as well as items at various levels of difficulty. </li></ul>
    49. 49. Operationalizing the Construct: The Items <ul><li>More able | More difficult </li></ul><ul><li>persons | items </li></ul><ul><li>| </li></ul><ul><li>x | item 1 </li></ul><ul><li>xx | item 2 item 3 </li></ul><ul><li>xxx | item 4 item 5 </li></ul><ul><li>xxxx | item 6 item 7 item 8 </li></ul><ul><li>xxxx | item 9 item 10 item 11 </li></ul><ul><li>xxx | item 12 item 13 </li></ul><ul><li>xx | item 14 </li></ul><ul><li>x | item 15 </li></ul><ul><li>| </li></ul><ul><li>Less able | Less difficult </li></ul><ul><li>Persons | items </li></ul>
    50. 50. Operationalizing the Construct: The Items <ul><li>After piloting the items, statistically analyze the results. </li></ul><ul><li>Examine dimensionality, item difficulty, and item content. </li></ul><ul><li>Select the best items to make an efficient, highly reliable instrument. </li></ul>
    51. 51. Part 4 Implementing Your Design
    52. 52. Implementing the design <ul><li>Including other researchers </li></ul><ul><li>Practical issues </li></ul><ul><li>Handling ethical concerns </li></ul>
    53. 53. Including other researchers in the study <ul><li>The nature of the researchers involved </li></ul><ul><ul><li>Main researcher plus “helpers” </li></ul></ul><ul><ul><li>One researcher plus “other participants” </li></ul></ul><ul><li>The nature of the instructors involved </li></ul><ul><ul><li>Teaching methods </li></ul></ul><ul><ul><li>Students taught </li></ul></ul><ul><ul><li>Course goals </li></ul></ul><ul><ul><li>University program goals </li></ul></ul>
    54. 54. Working with other researchers <ul><li>Work with people you know and trust </li></ul><ul><li>Establish a schedule early </li></ul><ul><li>Define clear roles for each researcher </li></ul><ul><li>Decide definite research goals prior to data collection </li></ul><ul><li>Keep in regular contact </li></ul><ul><li>The “band practice time” principle </li></ul>
    55. 55. Example research roles <ul><li>Head researcher / contact person </li></ul><ul><li>Data entry specialist </li></ul><ul><li>Statistician </li></ul><ul><li>Interviewer </li></ul><ul><li>Literature analyst </li></ul><ul><li>Editor / proofreader </li></ul>
    56. 56. Heading off potential problems <ul><li>Explain study commitments prior to starting the study </li></ul><ul><li>Agree on “ownership” prior to data collection and data entry </li></ul><ul><ul><li>Who will keep the data? </li></ul></ul><ul><ul><li>Whose name comes first, second, etc.? </li></ul></ul><ul><li>Keep everyone aware of deadlines </li></ul><ul><li>Include everyone in decision-making processed and data analysis </li></ul>
    57. 57. Things to avoid in group research <ul><li>People you don’t know </li></ul><ul><li>Research groups larger than four or five members </li></ul><ul><li>Using someone for language skills </li></ul><ul><li>Involving others just to get more participants </li></ul><ul><li>Forgetting to thank others for their assistance </li></ul>
    58. 58. Improving relations with research helpers <ul><li>Write clear instructions </li></ul><ul><li>Thank profusely for their time and effort </li></ul><ul><li>Offer to send copies of final research papers and/or results </li></ul><ul><li>Offer to assist in future research </li></ul><ul><li>Keep in touch after research ends </li></ul>
    59. 59. Practical Issues <ul><li>Timing of implementation </li></ul><ul><li>Learning and research context </li></ul><ul><li>Participant consent </li></ul><ul><li>Financial considerations </li></ul>
    60. 60. Timing of the implementation <ul><li>Beginning, middle, or end of semester </li></ul><ul><li>Day of the week </li></ul><ul><li>Time of day </li></ul><ul><li>Exams and exam preparation periods </li></ul><ul><li>“ Culture Festivals” or other club-related events </li></ul><ul><li>“ Open classes” or “parents’ day” </li></ul>
    61. 61. Learning and research context <ul><li>Differing course goals (I.e., listening class vs. reading class) </li></ul><ul><li>Different major field of study </li></ul><ul><li>Gender, age, year in school </li></ul><ul><li>Number of class meetings </li></ul><ul><li>Perception of the value of research by institution heads </li></ul>
    62. 62. Participant consent <ul><li>Always allow for “non-participation” choice from potential participants </li></ul><ul><li>Write clear instructions for participants asking for their cooperation </li></ul><ul><li>Ask co-researchers or helpers to briefly inform participants about their choice </li></ul>
    63. 63. Financial considerations <ul><li>Copies for questionnaires, exams, etc. </li></ul><ul><li>Computer analysis software </li></ul><ul><li>Mailing costs </li></ul><ul><li>Interview travel costs </li></ul><ul><li>Recording equipment </li></ul><ul><li>Reference books </li></ul><ul><li>Journal article costs </li></ul>
    64. 64. Heading off lack of cooperation problems <ul><li>Review requirements of the study </li></ul><ul><ul><li>How many items in the questionnaire? </li></ul></ul><ul><ul><li>How many treatments? </li></ul></ul><ul><li>Recognize that students are busy, tired, etc. </li></ul><ul><li>Plan to get between 30-50% more data than you need </li></ul><ul><li>Try to get more data at a later date </li></ul>
    65. 65. Ethical Considerations
    66. 66. Ethical considerations <ul><li>Students should not be exploited just because they are there </li></ul><ul><li>In theory, they should derive benefit (directly or indirectly) from the research </li></ul><ul><li>Explain the basic purpose of your research before collecting data </li></ul><ul><ul><li>1-2 sentences should be fine </li></ul></ul><ul><li>It is also good to briefly explain this at the beginning of any survey instrument </li></ul>
    67. 67. Ethics - Consent <ul><li>Provide students with the chance to opt out of participation </li></ul><ul><li>Verbal consent is usually acceptable for surveys, ratings, and test scores </li></ul><ul><li>A written consent form may be necessary for more involved forms of participation (interviews, essay passages) </li></ul><ul><li>When in doubt, check recent articles in well-known journals for guidance </li></ul>
    68. 68. Ethics – Other considerations <ul><li>The role of this study within your institution </li></ul><ul><li>Potential gender, proficiency or other issues that may affect your data or conclusions </li></ul><ul><li>Have a plan for anonymizing the data (and consider making this conspicuous on your instruments) </li></ul>
    69. 69. Conclusion
    70. 70. In conclusion… <ul><li>There are many factors to consider when embarking on a serious study </li></ul><ul><li>Some points to take away… </li></ul><ul><li>Most studies should fill a place (a “gap”) within the current academic dialogue </li></ul><ul><li>Research questions should reflect continuity with the literature and should be specific </li></ul><ul><li>Carefully consider the design setup which will work best with the participant groups and the research and analytical goals </li></ul>
    71. 71. In conclusion <ul><li>Further points… </li></ul><ul><li>Different instruments work better in different circumstances. Choose those which best reflect your aims. </li></ul><ul><li>Plan your analyses at the same time you are developing your instruments </li></ul><ul><li>Develop and pilot instruments which can cover responses from a range of participants </li></ul>
    72. 72. In conclusion <ul><li>Even more points… </li></ul><ul><li>Work with others who will be serious and committed, and then be an organized and conscientious leader </li></ul><ul><li>Consider how practical, pedagogical, and institutional concerns may affect your study </li></ul><ul><li>Do not forget current ethical guidelines for carrying out participant research </li></ul>
    73. 73. Good luck with your research! Thank you for listening
    74. 74. Q&A
    75. 75. For a copy of this presentation: