a review of hashemi Babae

1. Maryam Bolouri
Mixed method
Research
Toward New Designs
in
Applied Linguistics
Hashemi &Babaii
MLJ 97,4, 2013

2. Movement in inquiry in AL
• Emergence of Mixed method
• Driving forces behind this shift:
1.Advances in natural science
2.Development and challenges of social and
educational research
3.Criticism against the dichotomy of
qualitative and quantitive research
methods

3. Statement of problem:
• How these 2 methods can be integrated
usefully in a way to enhance the research
tools?
• How these 2 methods can be integrated
usefully in a way to enhance the quality of
obtained findings?

4. Purpose of the study:
• This study attempted to be a first move to
study
1.The current status of mixed methods
research in AL
2.Explore the nature of mixed research
designs, sampling designs and the
quality of inferences that can be
obtained

5. Review of literature
• Mixed method definition:
1.Simple definition by Bergman (2008): the combination
of at least one qualitative and at least one quantitive
component in a single research project or program
2.Technical definition by Hanson et. al (2005): the
collection, analysis and integration of quantitive and
qualitative data in a single or multiphase study
3.Extended definition by Tashakkori & Creswell
(2007a): research in which the investigator collects,
and analyzes data integrate the findings and draws
inferences using both qualitative and quantitive
approaches in a single study or program of inquiry

6. Mixed method origins
• Mixed method research has evolved from mono method
research.
• The first instances of mixing designs can be traced back to
1800s.
• It has been influenced by the emergence of the
“Triangulation”
• Navigation: taking bearings on 2 landmarks in order to
locate one’s position
• Land surveying: determine the position of point C if we
know the positions of points A and B
• Original definition: the use of multiple forms of qualitative
research methods in favor of richly developed
multidimensional descriptions of the world

7. Theoretical issues in mixed
method research
• Incompatibility thesis
• commensurability validity
• Pragmatist position (dialectical, balanced,
transformative or critical interpretive view):
• From practice oriented perspective. It is not a major
hindrance, the value is evident and they can coexist in
a dialectical relationship.
• The findings can enhance the understanding.
• The absent one would remain partially answered or
wholly unanswered questions.

8. Methodological issues:
in mixed method research
• Bazeley (2004): corruption of methods
“It more that using a variety of methods….it is an attempt
to serve the dual purposes of generalization and in-
depth understanding…”
• High quality mixed method research:
Integrations happens to RQ, units of analysis, sample,
instrumentation, data collection methods and analytic
strategies

9. Mixed method validity issues:
(Onwuegbuzie &Johnson, 2006): legitimation
• Complementary strength and non-overlapping
weakness of both are combined, so the validity
assessment is important and complex.
• (Tashakkori & Teddlie, 2008): inference quality
• It gathers info that result in meta inferences that
neither of method can do alone. They are an overall
conclusion through integration.
Methodological issues:
in mixed method research

10. Methodological issues:
in mixed method research
• (Tashakkori & Teddlie, 2003): Inference
• A researcher’s construction of the
relationship among people, events, and
variables as well as the his/her
construction of respondent’s perceptions,
behaviors, and feelings, and how there
relate to each other in a coherent and
systematic manner

11. Research questions:
1. What kinds of research designs are used
when method mixing is utilized in AL
research?
2. What kinds of sampling designs are used
in mixed methods research in the field of
AL?
3. How are general conclusions or meta
inferences developed in this type of
research?

12. Methodology
1. Data gathering
• How to select a representative sample
of all journals?
2. Data analysis
• What frameworks and typologies
inform data analysis?

13. Data gathering
• 7 comprehensive international peer reviewed
AL journals
• Over a period of 14 years 1995---2008
• Journal selection: based on 5 sources:
1. The list of professional journals published by MLJ- Weber&
Campbell, 2004
2. Evaluation of AL journals -Egbert (2007)
3. Examination of the frequency of the appearance of ELT
journals selected forpresentation in lg teaching btw 1996-
2002-Jung, (2004)
4. A study on the use of qualitative and quantitative methods in
332 articles overa 7 yearperiod- Lazarton (2000)
5. Expert judgment (2 PhDholders with research experience of

14. Data gathering (cont.)
• The first four sources yielded a comprehensive
list of 64 journals
• Expert judgment
• This exclusion was based on the following five
reasons:
1.An indirect relationship to the field of AL such as:
computers and compositions, British journal of
educational psychology.
2.Publishing in lg otherthan English
3.Insufficient info in the methodology section of
articles
4.Lesserprominence on the international scene

15. The final list of Journals
1. Applied linguistics-AL
2. English for specific purpose-ESP
3. Language learning-LL
4. Language teaching research-LTR
5. Language testing-LT
6. The modern language journal-MLJ
7. TESOL quarterly-TQ

16. The process of gathering the
data sources
• The electronic versions
• Screening phase (abstracts and
methodology)
• Search function of adobe reader
A total of 273 articles were identified

17. Review of Data gathering

18. Data Analysis
1. Rigorous qualitative content analysis
(rich description of mixing in the process of data
collection and data analysis)
Content analysis was based on both:
• Manifest content: the concrete terms used
for coding
• Latent content: the coder’s interpretation of
the content
2. Make use of code-sheets and respective
categories based on three FWs

19. Creswell et al. (2008)
analysis of research designs
• Concurrent designs and Sequential designs
• Concurrent triangulation design (parallel)
• Concurrent embedded design( btw pre and
post tests)
• Sequential explanatory design : quan->qual
• Sequential exploratory design: qual->quan
• Sequential embedded design: before and after
the intervention--qual

20. Collins et al. (2007)
2 dimensional model of mixed methods
sampling
• Time orientation of components
(concurrent or sequential)
• Relationship of the qualitative and quantitative
samples
1.Identical
2.Multilevel
3.Nested
4.Parallel

21. Data Analysis (cont.)
• Content analysis based on FWs
• Consider the arguments of validity and
inference quality
Tashakkori and Teddlie’s (2008)
integrative model of inference quality
in mixed method
• It consists of 2 broad categories of
1.Design quality
2.Integrative rigor

22. Data Analysis (cont.)
Design quality
1. Design suitability
2. Design adequacy
3. Within design
consistency
4. Analytic adequacy
Integrative rigor
1. Interpretive
consistency
2. Theoretical
consistency
3. Interpretive agreement
4. Interpretive
distinctiveness
5. Integrative efficacy

23. What is Integrative efficacy?
• The degree to which inferences made in
each strand of mixed methods study are
effectively integrated into a theoretically
consistent meta-inference.
• Simply speaking:
effective integration of both
qual and quan findings

24. extended definition of Tashkkori
& Creswell (2007a)
1. Mixing should happen at different
stages of a particular study
2. Meta inference must be developed
based on data integration both qual
and quan
It led to the exclusion of 68 articles and
the final sample included 205 articles

25. The reliability of the coding
• 1) An independent coder with qualitative
background coded a sample of articles
( 10% of the sample 20 articles)
• 2) Cohen’s Kappa was calculated . It was
• 0.82 for coding of research designs
• 0.71 for coding of sampling designs.
• 3) These indices were checked against Altman
(1991) criteria and proved to be acceptable.

26. Review of Data Analysis

27. Research question1: What kinds of research
designs are used when method mixing is
utilized in AL research?

28. Research question1: What kinds of research
designs are used when method mixing is
utilized in AL research?
• The direct use of mixed designs was directly
acknowledged in only a few studies
• . Triangulation at the level of analysis is
referred to as triangulation by analyses
(68 articles out of 136 articles that used triangulation)
• The term triangulation was used With different wording
such as: Multiple data source, multiple measures,
qualitative and quantitive data
• the emergence of mixed method designs that were not
in Creswell et al. typology (7 articles) next table

30. Research question2: What kinds of sampling
designs are used in mixed methods research in
the field of AL?

31. Research question3: How are general
conclusions or meta-inferences developed in
this type of research?
• The analyses of discussion and
conclusion sections revealed that data
integration and consolidation were not
presented
• These meta-inferences were offered by
researchers in order to:
1.enrich their discussions and
conclusions
2.present informative summaries of the
findings

32. Discussions:
1- Use of mixed research design
Concurrent triangulation design (66%)
1. Providing the supplementary data in multilevel D.A
2. Needs analysis
3. Program evaluation (collect data from various sources or by
multiple methods)
Lg Testing and ESP journals (Need assessment /program
evaluation)
Adopting a particular design could be a practical
considerations
Concurrent embedded designs(MLJ and lg learning)
Sequential explanatory design

33. Discussions:
1- Use of mixed research design
• Sequential exploratory design:
• development of questionnaire tests inventories
• validating the measurement instruments
• Sequential embedded designs
• examine cross sections of the test development
and validation process.
• ethnographical studies in order to develop
patterns in a more rigorous way.

34. Discussions
2-Use of mixed sampling design
This study (2013)
• Concurrent Identical designs
• multilevel concurrent>>>
Concurrent nested
• Why:
• different samples of articles in
terms of disciplines, size,
purpose, rationale for mixing
• Why concurrent identical:
• Practicality, feasibility, time
span
Collins et al (2007)
• Concurrent Identical
designs
• Concurrent
nested>>>multilevel
concurrent

35. Discussions :
3-Inference quality
• Why there was scant attention to the concept of
integrative efficacy among the articles?
• The concept hasn’t been introduced
appropriately to AL research
• It hasn’t been widely used in AL research
• It can be explained by Tashakkori and Creswell
distinction btw
• Methodology: integration of the 2 approaches
throughout the study
• Method: collecting and analyzing different types of
qual and quan data

36. Implications of the study
• 1) To explore complicated process and outcomes
• 2) Concurrent triangulation designs (DA, GA, NA,
…)
• 3) Concurrent embedded design (Lg development
or a process within an experiment)
• 4) Sequential explanatory designs:
• To explore a subsample from large samples used in qual
phase
• Analysis of outliers-extreme cases-inconsistent
components or incompatible subsystems
• Further analysis of performance on quan measurement
instruments and evaluate DIF items after statistical
analyses

37. Implications of the study
(Cont.)
• Sequential exploratory designs: validating
evaluation criteria such as tests, surveys,
questionnaire, scales or band descriptors
• Sequential embedded design:
longitudinal studies of development process in SLA
analyze cross sections of test development and
validation process
to investigate social aspects of lg learning such as T ID,
L ID, T D, L D, interaction patterns,…

38. Summary
• Innovative mixed method can be used in
design and sampling in AL based on
• Research purpose
• Requirements of the research context
High quality integration in mixed method
depends crucially on design quality and
quality of meta-inferences
• This can happen as result of integration of
qual and quan components in all of the
stages of the study

39. Penny for thoughts
• What are the evaluative criteria for
investigating the integration quality in AL
research?
• What is the tendency of some journals in
use of particular designs?
(As it was not the main focus of this study,
more accurate accounts of it would require
further research.)

40. Thank You
for your
patience 