Quality issues in mixed methods research (with an emphasis on teaching) - Alan Bryman

  • 2,376 views
Uploaded on

Presentation for the HEA-funded workshop ‘Teaching Research Methods in Business and Management’. …

Presentation for the HEA-funded workshop ‘Teaching Research Methods in Business and Management’.

Drawing on a mixture of practice and evidence, this one-day event provided an opportunity for those interested in the teaching of research methods in Business and Management – including qualitative, quantitative and mixed methods – to share experiences, insights, and good practice, and to discuss challenges and explore potential solutions.

This presentation forms part of a blog post reporting on the event which can be accessed via: http://bit.ly/1fcTwna

For further details of HEA Social Sciences work relating to teaching research methods in the Social Sciences please see http://bit.ly/15go0mh

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
2,376
On Slideshare
0
From Embeds
0
Number of Embeds
8

Actions

Shares
Downloads
13
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Quality issues in mixed methods research (with an emphasis on teaching) Alan Bryman Talk to Higher Education Academy event – Teaching Research Methods in Business and Management, 10 January 2014 ©Alan Bryman
  • 2. Is quality appraisal important for students?  Growing focus on quality criteria      Important for students to be familiar with aspects of quality that are especially valued      Need for consistent standards in assessments of research Interest among qualitative researchers has increased Use of systematic reviews Relevance for mixed methods research? May increase skill levels Provides vantage point for critical assessment of research May improve literature reviews for dissertations and theses In the case of mixed methods research, important to realise that it is more that just 2 separate components But might creativity and innovation be stifled?
  • 3. Creswell’s skill-based mixed methods seminar        Learning how to identify a mixed methods study Learning appropriate search terms for locating mixed methods studies Writing a rationale for mixing quantitative and qualitative research Writing research questions for mixed methods study Summarizing types of data collected in mixed methods research Transforming quantitative data into qualitative data and vice versa Combining quantitative and qualitative data to arrive at conclusions
  • 4. Creswell’s skill-based mixed methods seminar (contd.)       Identifying procedures used in software for analysis of quantitative and qualitative data Drawing visual model of mixed methods research design Identifying distinguishing features of mixed methods designs Comparing strengths and weaknesses of different mixed methods designs Evaluating a mixed methods study in terms of criteria Applying steps in design of a mixed methods study to a research proposal J. Creswell et al. (2003) ‘Teaching mixed methods research: practices, dilemmas and challenges’, in A. Tashakkori and C. Teddlie (eds), Handbook of Mixed Methods in Social & Behavioral Research, Thousand Oaks, CA: Sage.
  • 5. Do we expect too much of quality criteria? Two prominent roles: 1. Appraisal role.      Consideration of adequacy of research from the point of view of assessing research quality (e.g. for inclusion/exclusion in a systematic review) or as part of the refereeing function for research proposals and journal articles. With qualitative research, has led to long lists partly associated with lack of agreement on criteria (cf. Cassell & Symon, 2011) Difficult to argue for the removal of criteria. In the case of qualitative research, has led to long lists… C. Cassell & G. Symon (2011) ‘Assessing “good” qualitative research in the work psychology field: a narrative analysis’, Journal of Occupational and Organizational Psychology, 84: 633-50.
  • 6. Encyclopaedic approach to quality criteria for qualitative research? 1. How credible are the findings? 2. Has knowledge/understanding been extended by the research? 3. How well does the evaluation address its original aims and purposes? 4. Scope for drawing wider influences—how well is this explained? 5. How clear is the basis of the evaluative appraisal? 6. How defensible is the research design? 7. How well defended is the sample design/target selection of cases/documents? 8. Sample composition/case inclusion—how well is the eventual coverage described? 9. How well was the data collection carried out? 10. How well has the approach to, and formulation of, the analysis been conveyed? 11. Contexts of data sources—how well are they retained and portrayed? 12. How well has diversity of perspective and content been explored? 13. How well has detail, depth and complexity (richness?) of the data been conveyed? 14. How clear are the links between data, interpretation and conclusions—i.e. how well can the route to any conclusions be seen? 15. How clear and coherent is the reporting? 16. How clear are the assumptions/theoretical perspectives/values that have shaped the form and output of the evaluation? 17. What evidence is there of attention to ethical issues? 18. How adequately has the research process been documented? Spencer, L., et al. (2003), Quality in Qualitative Evaluation: A Framework for Assessing Research Evidence www.civilservice.gov.uk/wpcontent/uploads/2011/09/a_quality_framework_tcm6-38740.pdf
  • 7. Tracy’s 8 ‘must have’ criteria 1. Worthy topic—relevant, interesting, significant, etc. 2. Rich rigour—rich data supplied in abundance and appropriately 3. Sincerity—the researcher is reflexive about values and biases and is transparent in approach 4. Credibility—implements practices such as thick descriptions, triangulation, and respondent validation 5. Resonance—has an affecting impact on readers 6. Significant contribution—makes an impact in terms of such outcomes as theory, practice, and morality 7. Ethical—considers and engages in ethical practices 8. Meaningful coherence—addresses what it claims to address, uses appropriate methods, and links research questions, literature, findings and interpretations. S. J. Tracy (2010). ‘Qualitative Quality: Eight “Big Tent” Criteria for Excellent Qualitative Research’, Qualitative Inquiry, 16: 837–51.
  • 8. Do we expect too much of quality criteria? Teaching role 2.   Cohorts of students introduced to quantitative research quality criteria (internal, external, construct, conclusion, and ecological validity). Advantage of small number of issues around which criteria are grouped. From a teaching point of view, long lists may be unhelpful.     Lack of differentiation and therefore guidance concerning importance/significance. Potentially off-putting. Unrealisable. Therefore, for teaching purposes, shorter lists may be more helpful to students (and maybe lecturers).
  • 9. Trends in discussions of mixed methods criteria  Discussions of mixed methods research criteria may be treading a similar path to discussions of qualitative research criteria
  • 10. Mixed methods criteria The comprehensive approach (O’Cathain, 2010) – 6 domains + 2 Domain 1: Planning quality – 4 Domain 2: Design quality – 4 Domain 3: Data quality – 5 Domain 4: Interpretive rigour – 8 Domain 5: Inference transferability – 4 Domain 6: Reporting quality – 3 28 + 2 = 30 quality criteria A. O’Cathain (2010). ‘Assessing the quality of mixed methods research: toward a comprehensive framework’. In A. Tashakkori and C. Teddlie (Eds.), SAGE handbook of mixed methods in social and behavioral research (2nd edition, pp. 531-55). Los Angeles: Sage.
  • 11. Some recurring themes: Alan’s ‘must haves’       Need for quantitative and qualitative components to be appropriately implemented Need for transparency Use of mixed methods to be linked to research questions Need to be explicit about the mixed methods design and its appropriateness to research questions Importance of a rationale for the use of mixed methods research Importance of integration To appear in a forthcoming article which has been accepted for publication
  • 12. Some recurring themes: Alan’s ‘must haves’       Need for quantitative and qualitative components to be appropriately implemented Need for transparency Use of mixed methods to be linked to research questions Need to be explicit about the mixed methods design and its appropriateness to research questions Importance of a rationale for the use of mixed methods research Importance of integration To appear in a forthcoming article which has been accepted for publication
  • 13. Recent review of health services research  ‘Key methodological components that cut across qualitative and quantitative methodologies were often missing from mixed methods publications. Descriptions of sample selection and sampling procedures, the study context, and data-gathering procedures are essential aspects of interpreting study findings, and mixed methods studies should not be exempt from these basic research requirements. Many mixed methods studies did not include the level of detail that would likely be required for a qualitative or quantitative paper to be accepted in these high-ranking journals’. J.P. Wisdon et al. (2012) ‘Methodological reporting in qualitative, quantitative and mixed methods health services research articles’, Health Services Research, 47(2): 721-45.
  • 14. Some recurring themes: Alan’s ‘must haves’       Need for quantitative and qualitative components to be appropriately implemented Need for transparency Use of mixed methods to be linked to research questions Need to be explicit about the mixed methods design and its appropriateness to research questions Importance of a rationale for the use of mixed methods research Importance of integration To appear in a forthcoming article which has been accepted for publication
  • 15. Two common approaches to mixed methods research questions 1. 2. Separate quantitative and qualitative research questions. Overarching mixed research question; then expanded or broken down into quantitative and qualitative subquestions. From: Teddlie, C. and Tashakkori, A. Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences, Los Angeles: Sage, 2009, p. 133.
  • 16. Some recurring themes: Alan’s ‘must haves’       Need for quantitative and qualitative components to be appropriately implemented Need for transparency Use of mixed methods to be linked to research questions Need to be explicit about the mixed methods design and its appropriateness to research questions Importance of a rationale for the use of mixed methods research Importance of integration To appear in a forthcoming article which has been accepted for publication
  • 17. Types of mixed methods research design
  • 18. Some recurring themes: Alan’s ‘must haves’       Need for quantitative and qualitative components to be appropriately implemented Need for transparency Use of mixed methods to be linked to research questions Need to be explicit about the mixed methods design and its appropriateness to research questions Importance of a rationale for the use of mixed methods research Importance of integration To appear in a forthcoming article which has been accepted for publication
  • 19. Structure of the article: Parmelee et al.     Introduction Young people, voting, cynicism, and advertising Theoretical and mixed methods perspectives Method    Results      Focus groups Content analysis Short introduction Media-based frames and the failure to engage college-age voters Negative ads, audience-based frames, and cynical voters Students’ recommendations for how to build more engaging political ads Discussion   Summary and implications Limitations and future research J.H. Parmelee, et al. (2007) ‘ “What about people our age?”: Applying qualitative and quantitative methods to uncover how political ads alienate college students’, Journal of Mixed methods research, 1(2): 183-99
  • 20. May be more useful for research quality appraisal too   Lessons from qualitative research Pratt (2008) – reviewers of qualitative management research articles for North American journals believe qualitative standards are:      difficult to understand lack consensus/consistent standards lack clarity also, articles often evaluated against inappropriate (usually quantitative research) criteria Savall et al. (2008) – reviews of qualitative management research articles for a European journal:   Quality criteria proposed by literature not identified in a content analysis of reviews (9 criteria) Quality criteria used by reviewers but rarely mentioned in the literature (6 criteria) M.G. Pratt (2008) ‘Fitting oval pegs into round holes: tensions in evaluating and publishing qualitative research in top-tier North American journals’, Organizational Research Methods, 11: 481-509. H. Savall. et al. (2008) ‘The emergence of implicit criteria actually used by reviewers of qualitative research articles: case of a European journal’, Organizational Research Methods, 11: 510-40.
  • 21. May be more useful for research quality appraisal too    Focusing on a core set of criteria rather than a proliferation may mitigate some of these practices. For mixed methods research, concentrating on small number of core issues may help to focus attention on quality at a relatively early stage of its evolution. Importance of examples of mixed methods research
  • 22. Key points  Need to distinguish between    quality criteria as a screening or evaluation tool; and the possible pedagogical role of quality criteria In the context of mixed methods research, I’ve tried to suggest that a small number of criteria that capture widely-held positions may be more helpful in a teaching context.