Strategies to integrate qualitative and quantitative data in mixed methods research and evaluation. See an overview of the different types of mixed methods research; how NVivo handles combining qualitative and quantitative data and how specific analytical techniques can be used on any project to synthesize and summarize mixed methods data.

1. Presenters:
Lisa LeRoy, MBA, PhD
Lisa_leroy@abtassoc.com
Anna Jefferson, PhD
Anna_jefferson@abtassoc.com
Stuart Robertson, Ed.D.
stuart@robertsoneducational.com
September 2017
Conducting Integrated Mixed
Methods Research and Analysis
Using Software

2. Abt Associates | pg 2
Presentation Overview
 When are mixed methods most useful and how can
we better integrate qualitative and quantitative
methods? (Lisa LeRoy)
 Case Study: Understanding Consumers Financial
Decision Making? (Anna Jefferson)
 Mixed Methods Analytical and Reporting Capabilities
of Qualitative Data Analysis Software (Stuart
Robertson)

3. Abt Associates | pg 3
Mixed Methods Evaluation:
Conditions
 When different evaluation questions require different
methods, or when a single evaluation question requires
more than one method to answer all components
 When different methods are used to answer the same
elements of a single question, increasing confidence in
the validity and reliability of the evaluation results
 When the results from one method are used to help
design future phases of the evaluation using other
methods
U.S. AID. (June 2013). Technical Note: Conducting Mixed-Methods Evaluations. Washington, DC.

4. Abt Associates | pg 4
Mixed Methods Considerations
 Degree of interaction/independence between
methods
 Priority/precedence of each method
 Timing; sequential, concurrent, phased
 Sample; same sample for different methods, subset,
different samples
 Juncture; points at which methods will be mixed
(design, data collection, analysis, etc.)
Creswell and Plano Clark 2011

5. Abt Associates | pg 5
The bottom line…
 Use the methods that can best answer the research
and evaluation questions
 However, the options are expanding
 “Mixing” can occur at each stage of research:
– Study objectives - Analysis
– Design - Reporting
– Data collection - Dissemination
– Interpretation - Reflection and refinement

6. Abt Associates | pg 6
Why the increased attention on
mixed methods?

7. Abt Associates | pg 7
Why the increased attention on
mixed methods?
 Emphasis on best available evidence
 Technological advances in collecting and analyzing
data (e.g. social media, social network analysis, GIS)
 Quantity of available data is increasing
 Desire to improve the intervention with evaluation
learnings
 Desire to understand intervention “inner workings”
(implementation science)
 Understand how changing conditions affect an
intervention (e.g. environment)

8. Abt Associates | pg 8
Parallel versus Integrated
Mixed Methods
 Evaluation of program using RCT
 Qualitative method with a sample of program
participants or implementers
 Results analyzed and reported separately

9. Abt Associates | pg 9
Integrated Mixed Methods
 Qualitative and quantitative methods remain
independent
 Qualitative and quantitative methods are used in a
study but linkages between them are absent
 One type of evidence is privileged over another
 Research is driven by political agendas
Burch P and Heinrich C (2016) Mixed Methods for Policy Research and Program Evaluation

10. Abt Associates | pg 10
What are the useful mixed methods
frameworks?
The Qualitative Report - http://nsuworks.nova.edu/tqr

11. Abt Associates | pg 11
Mixed Methods Frameworks
(selected)
 A. Tashakkori & C. Teddlie
 P. Bazeley (analysis)
 J. Creswell & V. Plano Clark (design)
 S. Hesse-Biber & B. Johnson
 A. Onwuegbuzie & K. Collins (sampling)
 P. Burch & C. Heinrich (implementation)
 J. Maxwell (theory/philosophy)

12. Abt Associates | pg 12
Qualitative → Quantitative
Comparative Qualitative Analysis

13. Abt Associates | pg 13
Quantitative → Qualitative
Social Network Analysis

14. Abt Associates | pg 14
Case Study: Understanding Consumers
Financial Decision Making
 Started with the qualitative data (different than many
mixed methods studies)
 Both qualitative and quantitative data were collected
on every participant – full sample for both data types

15. Case Study:
Understanding
Consumers Financial
Decision Making
Anna Jefferson, PhD

16. Abt Associates | pg 16
Study Context
 Focus group study for Consumer Financial
Protection Bureau (2013-14)
 Goal: understand consumer decision-making about
four topics
– Credit reports and scores
– Auto financing and budgeting
– Comparison shopping for financial products
– Financial rules of thumb

17. Abt Associates | pg 17
Data Sources
 Qualitative: 32 consumer focus groups
– Boston, St. Louis, Atlanta, and Seattle
– 308 consumers
 Quantitative/Close-ended: Survey data
comprised of screening data and questionnaires
– Complete screening data for all 308; self-
administered questionnaires for 299 participants
– 100 data points
– Topics ranged from income/assets to financial
products, financial experiences, information sources,
and psychographics (e.g., stress about money)

18. Abt Associates | pg 18
Two Levels of Analysis
 Interpretive analysis: identifying themes based on
transcripts
 Attributes analysis: using questionnaire data to
identify patterns and segments among consumers.
 Mixed-methods goal: How do thematic findings
intersect with participants’ prior experiences,
attitudes, and characteristics?

19. Abt Associates | pg 19
Analysis Walk-Through
 Thematic analysis was done in NVivo 10 for each
topic and cutting across topics

20. Abt Associates | pg 20
Analysis Walk-Through
 Transcripts were auto-coded in NVivo so each
participant was a node
Note: all names are pseudonyms.

21. Abt Associates | pg 21
Analysis Walk-Through
 Survey data allowed us to associate each participant
node with the participants’ survey data
Note: all names are pseudonyms.

22. Abt Associates | pg 22
Analysis Walk-Through
 Auto-coding by participant in the transcripts allowed
every statement by a participant to also be classified
by its 100 close-ended data points
 This structure allowed for robust queries in NVivo
that identified patterns based on thematic
statements, participant characteristics, experiences,
and/or attitudes

23. Abt Associates | pg 23
Sample Output—Two Views

24. Abt Associates | pg 24
Final Product: Consumer Segments
 Defined on behavioral dimension
– Some segments further divided on motivational, situational,
or perceptual factors
 Utilized themes from qualitative data and participant
survey data
 Process:
– Qualitative analysts conducted first analysis, identified
themes and initial segments; qualitative and quantitative
analysts conferred regularly about findings, confirming
resonance between focus group and survey data, including
statistical analysis (outside NVivo)

25. Abt Associates | pg 25
Sample Thematic Findings Supporting
Segments: Credit Reports and Scores
 Respondents self-reported high levels of knowledge of
and engagement with credit reports and scores
– Awareness of credit scores’ implications for lending (generally),
employment, insurance, and housing
– However, many were often confused or incorrect in their
understanding of credit reports and scores
– “Checking credit” means various things to respondents and
may include over-reporting
 Consumers may cycle among different styles
– Expected purchases, negative financial events, or other life
changes
– Negative events trigger some people to become proactive
monitors while others continue to be more reactive

26. Abt Associates | pg 26
Sample Thematic Findings Supporting
Segments: Credit Reports and Scores
 Binary thinking around credit being “good” or “not so good”
 Some more report-driven (i.e., to look for errors) while
others score driven (i.e., to get “good” credit)
 Many reported identifying and disputing errors on their
reports
– Dispute process usually considered frustrating and confusing
 Confusion, frustration, and uncertainty were common
feelings associated with credit reports and scores
– Most do not feel in control of their credit reports
– Unsure what advice to trust about how to build credit

27. Abt Associates | pg 27
Sample Consumer Segments:
Credit Reports and Scores
Consumer segments for credit reports and scores
 Frequency of checking
– Segment 1a: Never checked
– Segment 1b: Past checkers
– Segment 1c: Frequent checkers
 How information is sought:
– Segment 1d: Passive (vs. active) checkers