外国語教育メディア学会（LET）関西支部メソドロジー研究部会 2013年度第2回研究会 @ 大学コンソーシアムあきた（秋田市） 浦野 研（北海学園大学）・亘理 陽一（静岡大学） 「英語教育研究における追試（replication）の必要性 」

1. LET関西支部メソドロジー研究部会
2013年度第2回研究会 in 秋田 2013年10月26日
@ 大学コンソーシアムあきた
英語教育研究における
追試（replication）の必要性
浦野 研（北海学園大学）
亘理 陽一（静岡大学）

2. メタ分析
最近多い
META-ANALYSIS IN L2 RESEARCH
16
14
14
Number of meta-analyses
•
12
10
10
8
6
4
2
2
2
1999-2001
2002-2004
1
0
1996-1998
2005-2007
2008-in press
Year of meta-analysis
Fig. 1. Growth of meta-analysis in L2 research.
Oswald & Plonsky (2010, p. 87)
magnitudes and patterns of relationships as well as the circumstances that
affect them.
87

3. Norris & Ortega (2000)
•
Effectiveness of L2 instruction: A research
synthesis and quantitative meta-analysis.
Language Learning, 50, 417–528.

4. Norris & Ortega (2000)
•
Effectiveness of L2 instruction: A research
synthesis and quantitative meta-analysis.
Language Learning, 50, 417–528.

5. Norris & Ortega (2000)
•
Effectiveness of L2 instruction: A research
synthesis and quantitative meta-analysis.
Language Learning, 50, 417–528.

6. メタ分析
•
Research synthesis の重要性
•
•
•
拡散する一方ではよくない
知見を集約し、方向性を提示したい
車輪の再発明を避けたい

7. メタ分析とは

8. メタ分析とは
•
「過去に行われた複数の独立な研究
結果を統合するための（統合できる
か否かの検討も含めた）統計解析」
（丹後, 2002, p. 1）

9. 簡単に言うと

10. 先行研究のまとめ方の
ひとつ

11. Review
Narrative Review
subjective & qualitative
Systematic Review
objective & qali/quanti-tative
Meta-Analysis
quantitative

12. メタ分析の手順

13. メタ分析の手順
1. 課題設定
2. 先行研究の収集
3. 文献の吟味
4. 分析と解釈
5. 発表

14. 実例（医学）

15. Griesdale et al. (2009)
•
Intensive insulin therapy and mortality
among critically ill patients: A metaanalysis including NICE-SUGAR study data.
Canadian Medical Association Journal, 180,
821–827.

16. Griesdale et al. (2009)
•
Intensive insulin therapy and mortality
among critically ill patients: A metaanalysis including NICE-SUGAR study data.
Canadian Medical Association Journal, 180,
821–827.

17. Griesdale et al. (2009)
•
危篤状態の患者に対するインスリンの
集中投与（強化インスリン療法）が推
奨されている
•
ただし、最近の研究にはその効果を否
定するものもある

18. そこでメタ分析

19. 先行研究の収集と吟味

20. Selection Criteria
1. The study was a randomized controlled trial
2. The study participants were adults
3. A critical care setting was used
4. The intensive insulin therapy was defined
by a target blood glucose concentration of
83 mmol/L or less
5. The study documented mortality

23. Results (1)

24. Results (1)
•
強化インスリン療法と死亡率の関係

27. Results (1)
•
強化インスリン療法は、統制群と比べ
て死亡率リスクに変化をもたらしてい
ない
•
ただし、外科系集中治療室において強
化インスリン療法は死亡率リスク減少
の効果がある

28. Results (2)

29. Results (2)
•
強化インスリン療法と高血糖の関係

31. Results (2)
•
通常の治療と比べ、強化インスリン療
法は高血糖のリスクを高める

32. 結論

33. 結論
•
In our updated meta-analysis of
randomized trials of intensive insulin
therapy in critically ill patients, we found
that such therapy had no effect on the
overall risk of death.

34. 結論
•
In our updated meta-analysis of
randomized trials of intensive insulin
therapy in critically ill patients, we found
that such therapy had no effect on the
overall risk of death.

35. 結論
•
Our findings do not support the
guidelines of organizations such as the
American Diabetes Association, … and
other organizations, … who recommend
intensive insulin therapy for all critically ill
patients.

36. 結論
•
Our findings do not support the
guidelines of organizations such as the
American Diabetes Association, … and
other organizations, … who recommend
intensive insulin therapy for all critically ill
patients.

37. 結論
•
We suggest that policy-makers reconsider
recommendations promoting the use of
intensive insulin therapy in all critically ill
patients.

38. 結論
•
We suggest that policy-makers reconsider
recommendations promoting the use of
intensive insulin therapy in all critically ill
patients.

39. 他の研究の結論の例

40. Gooßen & Gräber (2012)
•
Longer term safety of dipeptidyl
peptidase-4 inhibitors in patients with
type 2 diabetes mellitus: Systematic
review and meta-analysis. Diabetes,
Obesity and Metabolism, 14, 1061–1072.

41. Gooßen & Gräber (2012)
•
DPP-4 inhibitors are a safe treatment
option for patients with type 2 diabetes
mellitus. Concerns over immune effects of
gliptins were dispelled and should not
influence future prescribing behaviour.

42. Miller et al. (2005)
•
Meta-analysis: High-dosage vitamin E
supplementation may increase all-cause
mortality. Annual Internal Medicine, 142,
37–46.

43. Miller et al. (2005)
•
High-dosage vitamin supplementation is
often assumed to be at worst innocuous. In
view of the increased mortality associated
with high dosages of β-carotene and now
vitamin E, use of any high-dosage vitamin
supplements should be discouraged until
evidence of efficacy is documented from
appropriately designed clinical trials.

44. Reid et al. (2005)
•
Oxycodone for cancer-related pain: Metaanalysis of randomized controlled trials.
Archives of Internal Medicine, 166, 837–843.

45. Reid et al. (2005)
•
In this review, we did not find any
important differences between oxycodone
and morphine. Oxycodone is almost 4
times more expensive than morphine in
England, and there is less general
experience of its use. Thus, there is no
reason to challenge the recommendation
to use morphine as a first-line agent for
cancer pain.

46. 何が言いたいかというと

47. 何が言いたいかというと
•
•
•
治療法が明確に定義されており
メタ分析の結果
何らかの提案がなされている

48. 進む蓄積

49. 進む蓄積
•
•
累積メタ・アナリシス
新しい報告がでる度にメタ・アナリシ
スを繰り返しその結果を図示していく
（丹後, 2002, p. 16）

50. コクラン共同計画

52. The Cochrane Collaboration is an international network of
more than 31,000 dedicated people from over 120
countries. We work together to help healthcare
practitioners, policy-makers, patients, their advocates and
carers, make well-informed decisions about health care,
by preparing, updating, and promoting the accessibility
of Cochrane Reviews—over 5,000 so far, published online
in the Cochrane Database of Systematic Reviews, part of
The Cochrane Library. We also prepare the largest
collection of records of randomised controlled trials in the
world, called CENTRAL, published as part of The Cochrane
Library.
Our work is internationally recognised as the benchmark
for high quality information about the effectiveness of
health care.

53. 英語教育では？

54. 英語教育では？
•
Norris & Ortega (2000)
•
•
•
のちの研究に多大な影響
メタ分析が一気に増えた
方法論上の問題点も指摘されている

55. 英語教育では？
•
N & O (2000)に対する批判 by Shin (2010)
•
•
•
データ収集の問題
コード化の問題
統計上の問題

56. とりわけ問題なのが
•
N & O (2000) に対する批判 by Shin (2010)
•
データ収集の問題
•
•
一次研究の測定法の妥当性・一貫性
コード化の問題: 過度の単純化
•
トリートメントの性質

57. でケンカ売ってみた

58. でケンカ売ってみた
•
Norris & Ortega (2000)
•
•
1980-98に出版された250研究→46(49)研究
38(40)研究の「明示的指導」（n = 71）
•
+ Rule explanation (deductive/metalinguistic)
(n = 47)
•
+ direction to attend to forms and arrive at
rules (explicit induction) (n =24)

59. でケンカ売ってみた
•
Norris & Ortega (2000)
•
•
1980-98に出版された250研究→46(49)研究
38(40)研究の「明示的指導」（n = 71）
•
効果量d = 95% CI [0.93, 1.33] ＞ 暗示的指
導
•
その中身が気になる！

60. Watari (2013)

61. Watari (2013)
•
N & O (2000)の再検討
•
•
•
•
(a) 目標言語と文法項目の分布
(b) 規則提示のタイミング
(c) 焦点を当てている機能的側面
(d) 文法項目の選択・教材の配列の根拠

62. and (d) rationale for selection, sequence, and examples.
Table 1. Target languages & grammatical items (number of treatments)
(a)目標言語と文法項目の分布
English
Articles
2
Locative suffixes/Location phrase
Artificial
6
Japanese
French
5
2
Finnish
1
Number/Person/Gender
Spanish
Case
Pronouns/Relativization
Present/past participal
2
4
1
3
10
1
3
Dative alternation/SVO-order
1
12
1
2
Whether/if clause/Conditional
2
Implicature/Politeness
Derewiank
Cumm
York: S
3
3
6
2
1
" [5] “Yo
Refe
10
Particles/Honorifics
(Passive) voice
" [3] [2]
" [4] Yoi
{ stand
8
Interrogative/Imperative/Negative
" [1] Yoi
" [2] “Yo
3
Tense/Time phrases
Som
gave th
Besides
for exp
hypoth
e.g., tea
2
1
*This study doesn’t go into the methodological limitations of their meta-analysis (See
Doughty, C
H. Lon
Norris, J., &
analys
Shin, H. W.

63. Watari (2013)
•
N & O (2000)の再検討
•
(a) 目標言語と文法項目の分布
•
•
ムラが多い
一部の形態統語的カテゴリーに偏って
いる

64. 0.46
n.t.
1.85
n.a.
(dDoughty (1991) re-calculation shows target language is a potential and possibly important No. 1
= -0.01). This College, English University Working Papers in TESOL & Applied Linguistics, Vol. 10,
Teachers
Columbia
Another Look at Norrisn.t. Ortega (2000)
and
moderating variable. This example shows that the potential effect of moderators may be critical
Ha (2005)
English
0.07
n.t.
0.26
n.t.
in Izumi (2002) treatment effectiveness through meta-analysis. 0.67
exploring the
English
0.02
n.t.
(dLee (2007)This re-calculation shows target language is a potential and possibly -0.73
= -0.01).
important
English
1.05
n.t.
1.11
moderating variable. This example shows that the potential effect of moderators may be critical
White (1998)
English
0.26 TABLE 1
-0.01
0.97
n.t.
in exploring the re-calculation of the effect sizemeta-analysis.
treatment effectiveness through in Lee and Huang’s (2008) study
A
Mean
0.37
-0.01
0.97
-0.73
Effect size (d)
SD
0.42 TABLE 1
n.a.
0.59
n.a.
K
5 Grammar Learning 5
1
1
A re-calculation of the effect size Delayedand Pre-to-post
in Lee
Huang’s (2008) study
Target
Immediate
Meaning
95%CI upper
0.74
1.49
size (d)
Study
Language
posttest
posttestEffect contrast
comprehension
95%CI lower
0.01 Grammar Learning 0.46
Doughty (1991)
English
0.46
n.t.
1.85
n.a.
Cf. N & O (2000)に対する批判 by Shin (2010)
Target
Immediate Delayed
Pre-to-post
Meaning
Target
Immediate Delayed
Pre-to-post
Meaning
n.t.
Ha (2005)
English
0.07
n.t.
0.26
Study
Language
posttest
posttest
contrast
comprehension
Study
Language
posttest
posttest
contrast
comprehension
n.t.
Izumi (2002) (1998)
English
0.02
n.t.
0.67
n.a.
Jourdenais 1
Spanish
-0.04
-0.1
-0.04
Doughty (1991)
English
0.46
n.t.
1.85
n.a.
Lee (2007) 2 (1998)
English
1.05
n.t.
1.11
-0.73
n.a.
Jourdenais
Spanish
-0.02
-0.1
-0.19
n.t.
Ha (2005)
English
0.07
n.t.
0.26
White (1998) (1998)
English
0.26
-0.01
0.97
n.t.
n.a.
Jourdenais 3
Spanish
-0.15
0.09
0.01
n.t.
Izumi (2002)
English
0.02
n.t.
0.67
Mean 1 (2000)
0.37
-0.01
0.97
-0.73
n.a.
n.t.
Kubota
Spanish
-0.37
-0.45
Lee (2007)
English
1.05
n.t.
1.11
-0.73
SD
0.42
n.a.
0.59
n.a.
n.a.
n.t.
Kubota 2 (2000)
Spanish
-0.1
-0.21
White (1998)
English
0.26
-0.01
0.97
n.t.
KLeow (1997)
5
1
5
1
n.t.
Spanish
-0.06
0.86
0.28
Mean upper
0.37
-0.01
0.97
-0.73
95%CI (2001)
0.74
1.49
n.a.
n.t.
n.a.
Leow
Spanish
n.a.
SD
0.42
n.a.
0.59
n.a.
95%CI et al (2003)
0.01
0.46
n.a.
n.t.
n.a.
Leow lower
Spanish
n.a.
K
5
1
5
1
Target
Immediate
Delayed
Pre-to-post
Meaning
n.t.
Overstreet 1 (1998)
Spanish
0.07
0.07
-0.94
95%CI upper
0.74
1.49
Study
Language
posttest
posttest
contrast
comprehension
n.t.
n.t.
Overstreet 2 (2002)
Spanish
0.61
-0.34
n.a.
95%CI lower(1998)
0.01
0.46
Jourdenais 1
Spanish
-0.04
-0.1
-0.04
n.t.
n.t.
Overstreet 1 (2002)
Spanish
-0.05
-0.13
Target
Immediate Delayed
Pre-to-post
Meaning
n.a.
Jourdenais 2 (1998)
Spanish
-0.02
-0.1
-0.19
n.t.
Shook (2004)
Spanish
n.a.
n.a.
-0.30
Study
posttest
posttest
contrast
comprehension
n.a.
Jourdenais 3 (1998) Language
Spanish
-0.15
0.09
0.01
Mean
-0.01
-0.15
0.14
-0.29
n.a.
Jourdenais 1 (1998)
Spanish
-0.04
-0.1
-0.04
n.a.
n.t.
Kubota 1 (2000)
Spanish
-0.37
-0.45
SD
0.26
0.20
0.41
0.44
n.a.
Jourdenais 2 (1998)
Spanish
-0.02
-0.1
-0.19
n.a.
n.t.
Kubota 2 (2000)
Spanish
-0.1
-0.21
K
9
5
5
5
n.a.
Jourdenais 3 (1998)
Spanish
-0.15
0.09
0.01
n.t.
Leow (1997)
Spanish
-0.06
0.86
0.28
95%CI1upper
0.16
0.02
0.50
0.10
n.a.
n.t.
Kubota (2000)
Spanish
-0.37
-0.45
n.a.
n.t.
n.a.
Leow (2001)
Spanish
n.a.
95%CI lower
-0.18
-0.33
-0.22
-0.67
n.a.
n.t.
Kubota 2al (2003)
(2000)
Spanish
-0.1
-0.21
n.a.= not applicable because the study did not provide sufficient data
n.t.
n.a.
Leown.t. = not tested in theSpanish study; n.a.
et
n.a.
Note,
particular
n.t.
7
Leow (1997)(1998)
Spanish
-0.06
0.86
0.28
n.t.
for the effect
samples contributing to the effect size calculation.-0.94
Overstreet 1size calculations; k = number of0.07
Spanish
0.07
…
…

65. ! (a) What kinds of grammatical items are instructed? (Table 1)
Spotty, and biased toward some morpho-syntactic categories.
(b) 規則提示のタイミング
! (b) When is “explicit explanation” presented?
Sixty of 71 treatments gave some metalinguistic explanation. But
when? Before/while/after activities or training sessions (Table 2).
Table 2. The timing of metalinguistic explanation/feedback
Before
Before/while
While
After
Focus on FormS
12
26
4
0
Focus on Form
11
3
4
0
Biased toward giving rule description deductively, so we can’t get the
whole picture unless we construct and include rule discovery kind of
instruction with metalinguistic clues and/or summary more.
! (c) Which aspect(s) is “explicit explanation” focused on?
Grammar is related to 3 functions (Derewianka, 2007), e. g.,

66. Watari (2013)
•
N & O (2000)の再検討
•
(b) 規則提示のタイミング
•
規則の演繹的提示と誤りの指摘・訂正
に偏っている
•
規則発見・形成とか「まとめ」とか、
そういう発想ないの

67. (c) 焦点を当てている機能的側面
•
観念構成的:
•
•
•
The tiger was caught an hour ago.
対人関係的:
•
•
•
A man caught the tiger an hour ago.
You didn’t pay the bill last month.
The bill wasn’t paid last month.
テクスト形成的:
•
Two men have broken into the museum, and have been
arrested.

68. Watari (2013)
•
N & O (2000)の再検討
•
(c) 焦点を当てている機能的側面
•
•
•
•
観念構成的: 69 treatments (/71)
対人関係的: 11 treatments
テクスト形成的: 4 treatments
形式・意味・使用のバランス悪くないか

69. Watari (2013)
•
N & O (2000)の再検討
•
(d) 文法項目の選択・教材の配列の根拠
•
•
文法項目の選択理由: 4+α (/38)
教材・タスク・練習の配列の根拠: 2
•
•
Master (1994), Williams & Evans (1998)
これがなきゃ「指導」の評価できねッス

70. Cf. 浦野（40）in 広島
untimed GJT
timed GJT
メタ言語知識テスト
口頭物語テスト
模倣テスト
明示的知識を測定
暗示的知識を測定

71. で見てみた
•
N & O (2000)の再検討
•
(e) 指導の効果の測定方法
•
•
明示的知識を測っているもの: 71 (38)
暗示的知識を測っているもの: 10 (8)

72. Cf. 浦野（40）in 広島
外国語教育的には

73. 全部セットで考えていいけど
理解
暗示的知識
明示的知識
産出

74. Cf. 浦野（40）in 広島
けど、それって

75. Cf. 浦野（40）in 広島
第二言語習得研究
的には

76. 的知識
ここを見てないってことなので…
理解
暗示的知識
明示的知識
産出

77. 明示的文法指導が
明示的文法知識の獲得
に役立ったという

78. どうでもいい主張
をしているだけじゃ？
男性教授（41）in 秋田

79. 念のために全部みたょ
•
N & O (2000)の再検討
•
•
(e) 指導の効果の測定方法
（全46(49)研究中）
•
•
明示的知識を測っているもの: 43(45)
暗示的知識を測っているもの: 13(14)

80. つまり
•
synthesize された研究がそもそもひとつ
にまとめるべきものではない
•
あまりに異なる変数
•
•
e.g., 指導方法, 指導内容(言語規則), etc.
広くぼやけたことしか言えない

81. なぜ？
•
追試（replication）が少なすぎる
•
•
•
応用言語学; 重要性は何度か指摘
知見の蓄積に反映されていない
メタ分析を行えるほどの数の追試が行われな
いとダメ
•
たとえ synthesize しても、ものすごい広
く、そしてぼやけたことしか言えない。

82. 絞ったのもあるにはある
•
Russell & Spada (2006): corrective
feedback、文法諸形式について
•
Lyster & Saito (2010): Feedback in
classroom contexts
•
L2 writing: Truscott (2007)

83. 追試って？

84. 追試って？
•
Internal replication
•
•
•
carried out by the original researcher(s)
involves reassessing the data obtained,
typically through statistical re-sampling and
cross- validation techniques
External replication
•
concerned with new data from new subjects
and/or a new context. (Porte (Ed.), 2012, p. 7)

85. 追試って？
•
Exact replication
•
•
Approximate/systematic replication
•
•
Everything, including the subjects, is kept the same
One key variable (such as the learners' proficiency, L1
background, or learning context) is changed
Conceptual/constructive replications
•
involve changing the operationalization of a
construct, the study design, or a “nonmajor" variable.
(Language Teaching Review Panel, 2008, pp. 2-3)

86. 実際…
•
In Norris & Ortega (2000) ...
•
VanPatten & Cadierno (1993) の追試:
•
•
Cadierno (1995), VanPatten & Sanz (1995),
DeKeyser & Sokalski (1996), VanPatten & Oikkenon
(1996), (Kubota (1996)), Salaberry (1997)
Tomasello & Herron (1988, 1989) の追試:
•
(Fotos & Ellis (1991)), Ellis et al. (1994), Kubota
(1995a)

87. and (d) rationale for selection, sequence, and examples.
Table 1. Target languages & grammatical items (number of treatments)
English
Articles
2
Locative suffixes/Location phrase
Artificial
6
Japanese
French
5
2
Finnish
1
Number/Person/Gender
Spanish
Case
Pronouns/Relativization
Present/past participal
2
4
1
3
10
1
3
Dative alternation/SVO-order
1
12
1
2
Whether/if clause/Conditional
2
Implicature/Politeness
Derewiank
Cumm
York: S
3
3
6
2
1
" [5] “Yo
Refe
10
Particles/Honorifics
(Passive) voice
" [3] [2]
" [4] Yoi
{ stand
8
Interrogative/Imperative/Negative
" [1] Yoi
" [2] “Yo
3
Tense/Time phrases
Som
gave th
Besides
for exp
hypoth
e.g., tea
2
1
*This study doesn’t go into the methodological limitations of their meta-analysis (See
Doughty, C
H. Lon
Norris, J., &
analys
Shin, H. W.

88. 追試の重要性
•
追試の重要性に関する
これまでの指摘
•
Santos (1989), Polio &
Gass (1997)
•
Language Teaching
Review Panel (2008),
Oswald & Plonsky
(2010), Abbuhl (2012)
•
Porte (Ed.) (2012)

89. メタ分析のガイドライン
•
Plonsky & Oswald (2012):
•
Identification:
•
•
Author, Year, Source/venue, Journal, Title
Study context:
•
Second language/foreign language (SL/FL),
Classroom/ laboratory, Type of institution,
Age, Target language(s), first language (L1),
L2 proficiency level, Location of study

90. メタ分析のガイドライン
•
Plonsky & Oswald (2012):
•
Measures:
•
•
Dependent variable(s), Type(s) of outcomes
measures, Reliability
Outcomes:
•
Means and standard deviations for both control
and experimental groups, Effect size,
Frequencies, Percentages, p-values, Statistical
test results

91. 追試のガイドライン
•
Polio (2012):
•
•
•
•
•
追試した研究
RQs/トピック
変更点（文脈、独立・従属変数）
追試の種類
結果: 同じ？違う？Mixed？

92. 追試のガイドライン
•
Abbuhl (2012):
•
Step 1: Critically Review and Choose a Study
•
•
•
•
•
妥当性，必要性，強み弱み，実行可能性
Step 2: Decide on the Replication Type
Step 3: Formulate the Research Question
Step 4: Interpret the Results
Step 5: Write Up the Results

93. でも、これって…

94. 普通の研究のプロセスと
ほとんど同じじゃね？

95. 涙を拭いて
いい追試をしよう

96. 提案
•
Approximate replication のすすめ
•
•
•
先行研究から主な要因を1つだけ変更
Exact replication は実施が困難
あらたな要因が加わることで
publication の道も残る

97. 提案
•
大学院修士レベルでの replication の推
奨（期末プロジェクト等）
•
いきなり original study を行うのは
ハードルが高く、質の低いものになる
•
replication を行うことで研究手法全般
のトレーニングを積む

98. 提案
•
Replication を発表する場の設置・推奨
•
院生ワークショップなどを活用して
replication 発表枠を用意する
•
ジャーナルで replication セクションを
設ける(Language Teaching のように)

99. 提案だけしていても
何も変わらないことが多い世の中...

100. 何が実現可能か一緒に考え、
行動しましょう！

101. 引用文献
•
Abbuhl, R. (2012). Why, when, and how to replicate research. In A. Mackey ＆ S. M. Gass (Eds.), Research methods in second language acquisition: A practical guide (pp.
296-312). Chichester, West Sussex, UK: Wiley-Blackwell.
•
Gooßen, K., & Gräber, S. (2012). Longer term safety of dipeptidyl peptidase-4 inhibitors in patients with type 2 diabetes mellitus: Systematic review and meta-analysis.
Diabetes, Obesity, and Metabolism, 14, 1061–1072.
•
Griesdale D. E., de Souza, R.J., van Dam, R. M., Heyland, D.K., Cook, D.J., Malhotra, A., Dhaliwal, R., Henderson, W. R., Chittock, D. R., Finfer, S., & Talmor, D. (2009). Intensive
insulin therapy and mortality among critically ill patients: A meta-analysis including NICE-SUGAR study data. Canadian Medical Association Journal, 180(8), 821–827.
•
•
•
Language Teaching Review Panel (2008). Replication studies in language learning and teaching: Questions and answers. Language Teaching, 41, 1, 1–14.
•
•
•
Norris, J., & Ortega, L. (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50, 417–528.
•
Polio, C. (2012). Replication in published applied linguistics research: A historical perspective. In G. Porte (Ed.), Replication research in applied linguistics (pp. 47-91).
Cambridge: Cambridge University Press.
•
•
Polio, C., & Gass, S. (1997). Replication and reporting: A commentary. Studies in Second Language Acquisition, 19, 499–508.
•
Russell, J., & Spada, N. (2006). The effectiveness of corrective feedback for the acquisition of L2 grammar: A meta-analysis of the research. In J. M. Norris & L. Ortega
(Eds.), Synthesizing research on language learning and teaching (pp. 133-164). Philadelphia, PA: John Benjamins.
•
•
•
•
•
Santos, T. (1989). Replication in applied linguistics research. TESOL Quarterly, 23, 699–702.
Lyster, R., & Saito, K. (2010). Oral feedback in classroom SLA: A meta-analysis. Studies in Second Language Acquisition, 32, 2, 265–302.
Miller III, E. R., Pastor-Barriuso, R. Dalal, D., Riemersma, R. A., Appel, L. J., & Guallar, E. (2005). Meta-analysis: High-dosage vitamin E supplementation may Increase allcause mortality. Annual Internal Medicine, 142, 37–46.
Oswald, F. L., & Plonsky, L. (2010). Meta-analysis in second language research: Choices and challenges. Annual Review of Applied Linguistics, 30, 85–110.
Plonsky, L., & Oswald, F.L. (2012). How to do a meta-analysis. In A. Mackey ＆ S. M. Gass (Eds.), Research methods in second language acquisition: A practical guide (pp.
275–295). Chichester, West Sussex, UK: Wiley-Blackwell.
Reid, C. M., Martin, R. M., Sterne, J. A.C., Davies, A. N., & Hanks, G. W. (2006). Oxycodone for cancer-related pain: Meta-analysis of randomized controlled trials. Archives of
Internal Medicine, 166, 837–843.
Shin, H. W. (2010). Another look at Norris and Ortega (2000). Teachers College, Columbia University Working Papers in TESOL & Applied Linguistics, 10, 1, 15–38.
丹後俊郎. (2002). 『メタ・アナリシス入門: エビデンスの統合をめざす統計手法』. 東京: 朝倉書店.
Truscott, J. (2007). The effect of error correction on learners’ ability to write accurately. Journal of Second Language Writing, 16, 4, 255–272.
浦野研. (2013). 第二言語学習者の暗示的文法知識の測定法―構成概念妥当性の視点から―. 『外国語教育メディア学会（LET）関西支部メソドロジー研究部
会2012年度報告論集』, 36–45. Retrieved from http://www.mizumot.com/method/2012-03_Urano.pdf
•
Watari, Y. (2013). What do you mean by “explicit”?: A methodological consideration in explicit grammar teaching research. Poster session presented at the BAAL 2013
conference, Heriot-Watt University, Edinburgh, UK.