Here are my slides for my session for the International Institute of Business Analysis (May 23, 2012). I focus on best practices of survey design, highlighting qualitative data.

1. Capturing & Analyzing
Qualitative Data in
Surveys
Karen Carleton, MEd, MS
www.performcorp.ca
International Institute of Business Analysts
April 23, 2012
© Karen Carleton 2012

2. Objectives
1. Identify organizational context (PI)
2. Overview of survey design
3. Data collection & analysis best practices
(Qualitative data focus)
Exercises:
 Question critique
 Question writing practice (scenarios)
 Survey plan (real)
 Review

3. An ounce of
analysis is worth
a pound of
objectives
–Joe Harless

4. BA Competencies
 Analytical thinking
 Problem-solving
 Organizational learning
 Systems thinking
Source: Business Analysis Book of Knowledge

5. Organizational
Performance Improvement
RSVP Approach
Results-oriented
Systems thinking
Value-add
Partnerships focus

6. Systematic Solution Design
ADDIE e.g. Frontline hours extended
Analyze unintended problems elsewhere
Design
Develop
Implement
Evaluate

7. Organizational Research
Process improvement
Re-engineering
Organizational change
Software development
Others?
Source: Business Analysis Books of Knowledge

8. Surveys
Think-Pair-Share
Why might you
conduct a survey in
your organization?

9. Survey Objectives
Problems
 Determine causes, impacts, optimal solution(s) &
solution performance
Opportunities
 Improvement (efficiency/effectiveness), define
business need or requirements
Stakeholders
 Elicit stakeholder needs/concerns, solution req’ts
Source: Business Analysis Book of Knowledge

10. Organizational Research
METHOD DESCRIPTOR
Survey/ Collects topical data from a representative
Questionnaire population
Focus group Discussion-based group interview, noting
ideas, themes & non-verbals
Interview One-on-one questioning (structured to
unstructured)
Document Background info, context (e.g. job
review descriptions, stats, annual report)
Observations Note details observed – situations/
behaviours (e.g. procedures)

11. Surveys/Questionnaires
Elicitation Techniques (methods)
 Interviews, case studies/scenarios,
interviews, document analysis, focus
groups, questionnaires/surveys
Attitudes, opinions, recommendations,
experiences
Source: Business Analysis Book of Knowledge

12. Organizational Research
More than 1 method (technique)
Triangulate

13. Organizational Research
More than 1
stakeholder
group
Triangulate
Example: BPI

14. Target Audience Example
 Select participants
 useful/relevant input
 Representative
Sample
 Determine precision
needed

15. Best Practices
 Informed consent
 Purpose,
confidentiality,
results
 Organized,
consistent
 Know audience &
resources

16. Survey Types
In-person
Phone
Web-based

17. Which type
Survey Types & why?
 Flexible, interactive
 Clarify, probe
 Body language/non-verbals
 Flexible, interactive
 Clarify, probe; no travel
 Voice tone, intonation
 Easy for large respondent group
 Cost-effective, convenient

18. Considerations
Target
Objective Audience
Resources Data Types

19. Surveys
 Objective
 Target audience(s)
 Location(s)
 Resources ($, time, help)
 Data needed (quantitative &/or
qualitative)
Examples?

20. Intro. Message
Objective/Purpose
Thank
Results
Confidential;
anonymous
Voluntary

21. Staff Satisfaction Survey
At ABC Company employee feedback is important to
us. We would appreciate it if you would take 5-10
minutes to complete this survey by Monday May 7,
2012, 2012. All responses are anonymous and will be
kept confidential.
Your participation is voluntary and you may skip any
questions or discontinue at any point. By June 15, 2012
we will post the aggregate results and action items on
the intranet. Thanks in advance for sharing your
thoughts and opinions about how ABC Company could
be a better workplace!

22. E-Surveys
Advantages Disadvantages
 Low cost, timely  Computer literate &
 Large group, dispersed access
 Data aggregation &  No follow-up/
storage (no data entry) probing
 Anonymity  No intonation,
gestures
 Format ease
 Little contact
 Skip logic, progress bar
 Mostly US servers

23. Quantitative Data
 Numerical (“objective”) data
 Measured, statistical, stable study
 Length, height, volume, weight, speed, time,
temperature, sound levels, cost, age range, #
users or groups, %, etc.

24. Qualitative Data
 Descriptive (“subjective”) data;
not traditionally measured
 Textual, flexible study
 Colour, texture, appearance, beauty, smell,
taste, etc.

25. Latte Example
Quantitative Qualitative
 12 ounce mug  Medium; see-
 7 “ mug height through mug
 Serving  Steaming hot
temperature 150ºF  Pricey drink
 Cost: $4.95  Robust aroma
 Frothy;
Delicious

26. Quantitative Data Analysis
 Trends
 Average (mean),
highest & lowest
scores
 Generalizable
sample -
Statistically
significant &
representative

27. Qualitative Data Analysis
 Descriptive responses
 Feelings, beliefs,
likes/dislikes, thoughts,
personal opinions,
comments
 Patterns/Themes,
Counts

28. Qualitative Data
Recommendation:
Couple quantitative
(numerical) & qualitative
(descriptive) data
“Confidence”, “Rigor”,
“Reliability”, “Validity”,
“Consistency”, “Truth”

29. Qualitative Data (open-ended
questions) Limit use!
Advantages Disadvantages
 Personally  Non-response – more
meaningful, cultural time/effort to do
 Deeper questions (higher skip rate)
 Rich, complex human  No follow up/clarifying
details explained  Longer to analyze
 Ideas not (more resources –
anticipated time, people)
Source: Business Analysis Book of Knowledge

30. Response Rate
# Surveys completed = Response rate
# Participants contacted
 Sufficient numbers important
 Too little data= may not have substantial
info.
Example: client survey

31. Increase Response Rates
1. User-friendly, brief
2. Rewards-Cost-Trust
 R=Value, positive regard, thanks, token
WIIFM (incentive)
 C=short, easy
 T=important, authority, confidentiality
3. Reminders
Example: BSU faculty/staff, & students

32. Questions
 Objectives –
ask targeted
questions
 Clear, concise

33. Closed Questions (Quantitative)
 Rating/Matrix (4, 5 or 7 point)
 Ranking, ordinal (1-4) – order
 Multiple choice (one or more responses)
 Likert scale (Agree…), descriptors

34. Open Questions (Qualitative)
 Fill-in-the-blank
(# or word)
 Text box
(comment or
paragraph)
Example: Staff Survey
“career development”

35. Question Critique
Under what circumstances Negative phrasing
do you not submit your
weekly project hours?
 Overtime still in progress
 Unknown factors
 NA  Other
Do you like our Double-barreled
Edmonton and Calgary
branches?
 Yes  No

36. Question Critique
What would make the Acronym or
EFAP better? jargon; unclear
If you are working on a Confusing/
late project and a team complex format
member is away, do you:
a) wait
b) continue
c) ask your supervisor for
advice

37. Question Critique
How fair was your Sensitive
performance bonus this year?
Strongly agree Agree Neutral Disagree Strongly disagree
5 4 3 2 1
Do you like the coffee? “Apple pie
Yes  No effect”
Do you like the washrooms?
Yes  No
Do you like the parking? Yes  No
Do you like your boss? Yes  No

38. Question Critique
How many copies did Lack of alternate
you make with the choice when
area’s photocopier last needed: “N/A”,
week? “no opinion” or
a) Less than 5 “prefer not to
b) Less than 20 answer”, ”don’t
c) Less than 50 know”
d) Less than 100
e) 100 or more

39. Question Critique
I generally feel informed about changes
that affect me at ABC Company.
1 2 3 4 5
Not at all Very
Informed Informed
Unclear language & descriptors

40. Question Critique
I can see the link between my work and
ABC Company’s objectives and
mission.
1 2 3 4 5
Not at all Very
Double-barreled question,
Presumes understanding (2),
Descriptors don’t fit.

41. Question Critique
Do you feel the efforts of the Social
Committee contribute to your
enjoyment in working at ABC
Company?
1 2 3 4 5
Not at all Very
Leading question: presumes
knowledge & enjoyment.
Descriptors don’t fit.

42. Question Critique
My working conditions are good.
1 2 3 4 5
Strongly Strongly
Disagree Agree
My relationship with my manager is good.
1 2 3 4 5
Strongly Strongly
Disagree Agree
Vague adjective

43. Question Critique
Is there anything else you would like to
mention that you feel could help improve
staff satisfaction at ABC Company?
Wordy

44. Question Tips
At start, interesting question
Easier before difficult ones
At end, personal & demographic
Consider analysis
Have others review
Pretest
Why?

45. Question Writing Practice
Scenario pairs:
1. Decide survey Objective
2. Choose a fitting Distribution Method
(e.g. phone, in-person, e-survey/web-
based)
3. Create 3 Questions Time: 20 minutes
4. Question Review (with another group)
Time: 5 minutes

46. Data Analysis
Quantitative
 Total responses,
frequency counts,
distribution, averages
Qualitative
 Pre-determined or
emergent categories

47. Qualitative
Triangulation
1. Cross-check
data accuracy
 Validate, verify
1. Other method(s)
2. Other
stakeholder(s)
e.g. survey employees,
interview supervisor

48. Qualitative Data Analysis
2. Constant Comparison
Method
 Compare & contrast text
items
 Sort, sift, group, match
 Categorize, order,
sub-divide
“Raw” “Cooked” Data

49. Qualitative Data
Analysis
3. Read-Reread Method
 Iterative review
 Make sense of findings
(determine meaning)
 Underline, sort, count,
code, highlight data
 Patterns/picture(s)
emerge

50. Qualitative Data
Analysis
4. Assign Codes
 Colours, numbers,
symbols, categories,
abbreviations
 e.g. F (female)
 e.g. MROLE
(manager role)

51. Survey Plan
1. Set objective & 5. Pre-test survey
target audience 6. Administer survey
2. Choose distribution 7. Analyze results
method 8. Communicate &
3. Create questions address results
4. Question
review/test Practice with
first 3 steps

52. Survey Plan (real)
1. Objective
2. Target audience(s)
3. Survey distribution method
4. Additional research method(s) (*to verify
survey data)
5. Draft 1 clear, open-ended question
*2-4 pairs to share plan with group.
Time: 15 minutes (pairs)

53. Communicating Results
 Honesty about
findings
 Timely results
 4-6 weeks
 = TRUST
 Act on findings
or what
happens?

54. Review
1. One BA competency involved in survey
analytical or systems
design is ____________________ thinking.
2. Another organizational research method you
could use to verify your findings’ accuracy is
Interviews, focus groups,
___________________________________
observations, secondary data
___________________________________.
3. By using more than one method or
riangulate
stakeholder group you can T____________
your results to validate them.

55. Review
4. Surveys/Questionnaires are considered an
licitation
E_________ technique by BABOK.
5. Knowing the characteristics of your
target audience
_______________ is important for survey
design.
6. Advantages of E-surveys are that they are:
cost-effective, good for dispersed
______________________________________
populations, and ease of analysis
_______________________________.

56. Review
7. Numerical or statistical data is
quantitative
____________, while descriptive data with rich
qualitative
detail is ___________.
8. Qualitative data analysis involves seeking
patterns/themes
________________ in the text, by using the
re-read constant comparison
read-________ or __________________
methods.
9. Accurate qualitative data is described as
having confidence, rigor, validity, consistency
_______________________________.

57. Summary
 Know your audience, objective & resources
 Consider question types & analysis
 Triangulate (*more than one method, &
stakeholder group)
 Follow best practices
 Empathize with your respondents
Questions or comments?

58. Extras & Presenter Info
1. Write #1 YES on back of your business card for
the FREE “Top Ten Reasons Surveys Fail”
2. Write #2 YES on back of your business card for a
FREE Strategy Session.
Karen Carleton, MEd, MS
Performance Solutions Corp.
 Survey design & consulting
 Needs analysis & evaluation
www.performcorp.ca Email: Karen@performcorp.ca

59. References
 Business Analysts’ Book of Knowledge (BABOK)
 Dillman, D.A. (2000). Mail and Internet surveys: The
total design method (2nd ed.). New York: Wiley.
 Greatbrook. Statistical Confidence. Retrieved April 18,
2012 from,
http://greatbrook.com/survey_statistical_confidence.htm
 Inquisite survey software, http://www.inquisite.com/
 Qualtrics survey software, http://www.qualtrics.com/
 Survey Monkey survey software,
http://www.surveymonkey.com/
 FREE account, up to 200 respondents. “How to” downloads.