Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Capturing & Analyzing   Qualitative Data in             Surveys       Karen Carleton, MEd, MS                www.performco...
Objectives1. Identify organizational context (PI)2. Overview of survey design3. Data collection & analysis best practices ...
An ounce ofanalysis is worth   a pound of   objectives    –Joe Harless
BA Competencies      Analytical thinking      Problem-solving      Organizational learning      Systems thinkingSource...
OrganizationalPerformance ImprovementRSVP ApproachResults-orientedSystems thinkingValue-addPartnerships focus
Systematic Solution DesignADDIE       e.g. Frontline hours extendedAnalyze     unintended problems elsewhereDesignDevelopI...
Organizational Research  Process improvement                     Re-engineering         Organizational change  Software de...
SurveysThink-Pair-ShareWhy might youconduct a survey inyour organization?
Survey Objectives   Problems    Determine causes, impacts, optimal solution(s) &     solution performance   Opportunities...
Organizational ResearchMETHOD           DESCRIPTORSurvey/       Collects topical data from a representativeQuestionnaire p...
Surveys/Questionnaires   Elicitation Techniques (methods)    Interviews, case studies/scenarios,      interviews, documen...
Organizational ResearchMore than 1 method (technique)Triangulate
Organizational Research                   More than 1                   stakeholder                   group               ...
Target Audience             Example   Select participants       useful/relevant input   Representative    Sample      ...
Best Practices Informed consent  Purpose,   confidentiality,   results Organized,  consistent Know audience &  resources
Survey Types     In-person     Phone     Web-based
Which typeSurvey Types             & why?      Flexible, interactive      Clarify, probe      Body language/non-verbals...
Considerations                    Target        Objective   Audience       Resources    Data Types
Surveys           Objective           Target audience(s)           Location(s)           Resources ($, time, help)    ...
Intro. MessageObjective/PurposeThankResultsConfidential; anonymousVoluntary
Staff Satisfaction SurveyAt ABC Company employee feedback is important tous. We would appreciate it if you would take 5-10...
E-SurveysAdvantages                     Disadvantages Low cost, timely              Computer literate & Large group, di...
Quantitative Data Numerical (“objective”) data Measured, statistical, stable study  Length, height, volume, weight, spe...
Qualitative Data Descriptive (“subjective”) data;  not traditionally measured Textual, flexible study  Colour, texture,...
Latte ExampleQuantitative          Qualitative 12 ounce mug         Medium; see- 7 “ mug height        through mug Ser...
Quantitative Data Analysis                      Trends                      Average (mean),                       highes...
Qualitative Data Analysis Descriptive responses   Feelings, beliefs,    likes/dislikes, thoughts,    personal opinions, ...
Qualitative DataRecommendation:Couple quantitative(numerical) & qualitative(descriptive) data“Confidence”, “Rigor”,“Reliab...
Qualitative Data (open-ended   questions)         Limit use!   Advantages                     Disadvantages    Personally...
Response Rate# Surveys completed      = Response rate# Participants contacted Sufficient numbers important  Too little d...
Increase Response Rates1. User-friendly, brief2. Rewards-Cost-Trust  R=Value, positive regard, thanks, token   WIIFM (inc...
Questions             Objectives –              ask targeted              questions             Clear, concise
Closed Questions (Quantitative)   Rating/Matrix (4, 5 or 7 point)   Ranking, ordinal (1-4) – order   Multiple choice (o...
Open Questions (Qualitative) Fill-in-the-blank  (# or word) Text box  (comment or  paragraph)  Example: Staff Survey  “c...
Question CritiqueUnder what circumstances       Negative phrasingdo you not submit yourweekly project hours? Overtime sti...
Question CritiqueWhat would make the          Acronym orEFAP better?                 jargon; unclearIf you are working on ...
Question CritiqueHow fair was your                           Sensitiveperformance bonus this year?Strongly agree   Agree N...
Question CritiqueHow many copies did       Lack of alternateyou make with the         choice whenarea’s photocopier last  ...
Question CritiqueI generally feel informed about changesthat affect me at ABC Company.1            2     3        4       ...
Question CritiqueI can see the link between my work andABC Company’s objectives andmission.1             2        3       ...
Question CritiqueDo you feel the efforts of the SocialCommittee contribute to yourenjoyment in working at ABCCompany?1    ...
Question CritiqueMy working conditions are good. 1         2         3           4       5Strongly                        ...
Question CritiqueIs there anything else you would like tomention that you feel could help improvestaff satisfaction at ABC...
Question TipsAt start, interesting questionEasier before difficult onesAt end, personal & demographicConsider analysis...
Question Writing PracticeScenario pairs:1. Decide survey Objective2. Choose a fitting Distribution Method   (e.g. phone, i...
Data AnalysisQuantitative Total responses,  frequency counts,  distribution, averagesQualitative Pre-determined or  emer...
QualitativeTriangulation1. Cross-checkdata accuracy Validate, verify  1. Other method(s)  2. Other     stakeholder(s)e.g....
Qualitative Data Analysis                2. Constant Comparison                Method                 Compare & contrast ...
Qualitative Data Analysis3. Read-Reread Method Iterative review Make sense of findings  (determine meaning)  Underline,...
Qualitative Data Analysis4. Assign Codes Colours, numbers,  symbols, categories,  abbreviations  e.g. F (female)  e.g. ...
Survey Plan1. Set objective &       5. Pre-test survey   target audience       6. Administer survey2. Choose distribution ...
Survey Plan (real)1. Objective2. Target audience(s)3. Survey distribution method4. Additional research method(s) (*to veri...
Communicating Results Honesty about  findings Timely results  4-6 weeks  = TRUST Act on findings   or what   happens?
Review1. One BA competency involved in survey             analytical or systems   design is ____________________ thinking....
Review4. Surveys/Questionnaires are considered an  licitationE_________ technique by BABOK.5. Knowing the characteristics ...
Review7. Numerical or statistical data is quantitative____________, while descriptive data with rich           qualitative...
Summary Know your audience, objective & resources Consider question types & analysis Triangulate (*more than one method...
Extras & Presenter Info1. Write #1 YES on back of your business card for   the FREE “Top Ten Reasons Surveys Fail”2. Write...
References Business Analysts’ Book of Knowledge (BABOK) Dillman, D.A. (2000). Mail and Internet surveys: The  total desi...
Upcoming SlideShare
Loading in …5
×

Capturing and Analyzing Qualitative Data in Surveys

22,869 views

Published on

Here are my slides for my session for the International Institute of Business Analysis (May 23, 2012). I focus on best practices of survey design, highlighting qualitative data.

Published in: Business, Technology

Capturing and Analyzing Qualitative Data in Surveys

  1. 1. Capturing & Analyzing Qualitative Data in Surveys Karen Carleton, MEd, MS www.performcorp.caInternational Institute of Business AnalystsApril 23, 2012 © Karen Carleton 2012
  2. 2. Objectives1. Identify organizational context (PI)2. Overview of survey design3. Data collection & analysis best practices (Qualitative data focus)Exercises: Question critique Question writing practice (scenarios) Survey plan (real) Review
  3. 3. An ounce ofanalysis is worth a pound of objectives –Joe Harless
  4. 4. BA Competencies  Analytical thinking  Problem-solving  Organizational learning  Systems thinkingSource: Business Analysis Book of Knowledge
  5. 5. OrganizationalPerformance ImprovementRSVP ApproachResults-orientedSystems thinkingValue-addPartnerships focus
  6. 6. Systematic Solution DesignADDIE e.g. Frontline hours extendedAnalyze unintended problems elsewhereDesignDevelopImplementEvaluate
  7. 7. Organizational Research Process improvement Re-engineering Organizational change Software development Others?Source: Business Analysis Books of Knowledge
  8. 8. SurveysThink-Pair-ShareWhy might youconduct a survey inyour organization?
  9. 9. Survey Objectives Problems  Determine causes, impacts, optimal solution(s) & solution performance Opportunities  Improvement (efficiency/effectiveness), define business need or requirements Stakeholders  Elicit stakeholder needs/concerns, solution req’tsSource: Business Analysis Book of Knowledge
  10. 10. Organizational ResearchMETHOD DESCRIPTORSurvey/ Collects topical data from a representativeQuestionnaire populationFocus group Discussion-based group interview, noting ideas, themes & non-verbalsInterview One-on-one questioning (structured to unstructured)Document Background info, context (e.g. jobreview descriptions, stats, annual report)Observations Note details observed – situations/ behaviours (e.g. procedures)
  11. 11. Surveys/Questionnaires Elicitation Techniques (methods)  Interviews, case studies/scenarios, interviews, document analysis, focus groups, questionnaires/surveys Attitudes, opinions, recommendations, experiencesSource: Business Analysis Book of Knowledge
  12. 12. Organizational ResearchMore than 1 method (technique)Triangulate
  13. 13. Organizational Research More than 1 stakeholder group Triangulate Example: BPI
  14. 14. Target Audience Example Select participants  useful/relevant input Representative Sample  Determine precision needed
  15. 15. Best Practices Informed consent  Purpose, confidentiality, results Organized, consistent Know audience & resources
  16. 16. Survey Types In-person Phone Web-based
  17. 17. Which typeSurvey Types & why?  Flexible, interactive  Clarify, probe  Body language/non-verbals  Flexible, interactive  Clarify, probe; no travel  Voice tone, intonation  Easy for large respondent group  Cost-effective, convenient
  18. 18. Considerations Target Objective Audience Resources Data Types
  19. 19. Surveys  Objective  Target audience(s)  Location(s)  Resources ($, time, help)  Data needed (quantitative &/or qualitative) Examples?
  20. 20. Intro. MessageObjective/PurposeThankResultsConfidential; anonymousVoluntary
  21. 21. Staff Satisfaction SurveyAt ABC Company employee feedback is important tous. We would appreciate it if you would take 5-10minutes to complete this survey by Monday May 7,2012, 2012. All responses are anonymous and will bekept confidential.Your participation is voluntary and you may skip anyquestions or discontinue at any point. By June 15, 2012we will post the aggregate results and action items onthe intranet. Thanks in advance for sharing yourthoughts and opinions about how ABC Company couldbe a better workplace!
  22. 22. E-SurveysAdvantages Disadvantages Low cost, timely  Computer literate & Large group, dispersed access Data aggregation &  No follow-up/ storage (no data entry) probing Anonymity  No intonation, gestures Format ease  Little contact  Skip logic, progress bar  Mostly US servers
  23. 23. Quantitative Data Numerical (“objective”) data Measured, statistical, stable study  Length, height, volume, weight, speed, time, temperature, sound levels, cost, age range, # users or groups, %, etc.
  24. 24. Qualitative Data Descriptive (“subjective”) data; not traditionally measured Textual, flexible study  Colour, texture, appearance, beauty, smell, taste, etc.
  25. 25. Latte ExampleQuantitative Qualitative 12 ounce mug  Medium; see- 7 “ mug height through mug Serving  Steaming hot temperature 150ºF  Pricey drink Cost: $4.95  Robust aroma  Frothy; Delicious
  26. 26. Quantitative Data Analysis  Trends  Average (mean), highest & lowest scores  Generalizable sample - Statistically significant & representative
  27. 27. Qualitative Data Analysis Descriptive responses  Feelings, beliefs, likes/dislikes, thoughts, personal opinions, comments  Patterns/Themes, Counts
  28. 28. Qualitative DataRecommendation:Couple quantitative(numerical) & qualitative(descriptive) data“Confidence”, “Rigor”,“Reliability”, “Validity”,“Consistency”, “Truth”
  29. 29. Qualitative Data (open-ended questions) Limit use! Advantages Disadvantages  Personally  Non-response – more meaningful, cultural time/effort to do  Deeper questions (higher skip rate)  Rich, complex human  No follow up/clarifying details explained  Longer to analyze  Ideas not (more resources – anticipated time, people)Source: Business Analysis Book of Knowledge
  30. 30. Response Rate# Surveys completed = Response rate# Participants contacted Sufficient numbers important  Too little data= may not have substantial info. Example: client survey
  31. 31. Increase Response Rates1. User-friendly, brief2. Rewards-Cost-Trust  R=Value, positive regard, thanks, token WIIFM (incentive)  C=short, easy  T=important, authority, confidentiality3. Reminders Example: BSU faculty/staff, & students
  32. 32. Questions  Objectives – ask targeted questions  Clear, concise
  33. 33. Closed Questions (Quantitative) Rating/Matrix (4, 5 or 7 point) Ranking, ordinal (1-4) – order Multiple choice (one or more responses) Likert scale (Agree…), descriptors
  34. 34. Open Questions (Qualitative) Fill-in-the-blank (# or word) Text box (comment or paragraph) Example: Staff Survey “career development”
  35. 35. Question CritiqueUnder what circumstances Negative phrasingdo you not submit yourweekly project hours? Overtime still in progress Unknown factors NA  OtherDo you like our Double-barreledEdmonton and Calgarybranches? Yes  No
  36. 36. Question CritiqueWhat would make the Acronym orEFAP better? jargon; unclearIf you are working on a Confusing/late project and a team complex formatmember is away, do you:a) waitb) continuec) ask your supervisor foradvice
  37. 37. Question CritiqueHow fair was your Sensitiveperformance bonus this year?Strongly agree Agree Neutral Disagree Strongly disagree5 4 3 2 1Do you like the coffee? “Apple pieYes  No effect”Do you like the washrooms?Yes  NoDo you like the parking? Yes  NoDo you like your boss? Yes  No
  38. 38. Question CritiqueHow many copies did Lack of alternateyou make with the choice whenarea’s photocopier last needed: “N/A”,week? “no opinion” ora) Less than 5 “prefer not tob) Less than 20 answer”, ”don’tc) Less than 50 know”d) Less than 100e) 100 or more
  39. 39. Question CritiqueI generally feel informed about changesthat affect me at ABC Company.1 2 3 4 5Not at all VeryInformed Informed Unclear language & descriptors
  40. 40. Question CritiqueI can see the link between my work andABC Company’s objectives andmission.1 2 3 4 5Not at all Very Double-barreled question, Presumes understanding (2), Descriptors don’t fit.
  41. 41. Question CritiqueDo you feel the efforts of the SocialCommittee contribute to yourenjoyment in working at ABCCompany?1 2 3 4 5Not at all Very Leading question: presumes knowledge & enjoyment. Descriptors don’t fit.
  42. 42. Question CritiqueMy working conditions are good. 1 2 3 4 5Strongly StronglyDisagree AgreeMy relationship with my manager is good.1 2 3 4 5Strongly StronglyDisagree Agree Vague adjective
  43. 43. Question CritiqueIs there anything else you would like tomention that you feel could help improvestaff satisfaction at ABC Company? Wordy
  44. 44. Question TipsAt start, interesting questionEasier before difficult onesAt end, personal & demographicConsider analysisHave others reviewPretest Why?
  45. 45. Question Writing PracticeScenario pairs:1. Decide survey Objective2. Choose a fitting Distribution Method (e.g. phone, in-person, e-survey/web- based)3. Create 3 Questions Time: 20 minutes4. Question Review (with another group) Time: 5 minutes
  46. 46. Data AnalysisQuantitative Total responses, frequency counts, distribution, averagesQualitative Pre-determined or emergent categories
  47. 47. QualitativeTriangulation1. Cross-checkdata accuracy Validate, verify 1. Other method(s) 2. Other stakeholder(s)e.g. survey employees,interview supervisor
  48. 48. Qualitative Data Analysis 2. Constant Comparison Method  Compare & contrast text items  Sort, sift, group, match  Categorize, order, sub-divide “Raw” “Cooked” Data
  49. 49. Qualitative Data Analysis3. Read-Reread Method Iterative review Make sense of findings (determine meaning)  Underline, sort, count, code, highlight data  Patterns/picture(s) emerge
  50. 50. Qualitative Data Analysis4. Assign Codes Colours, numbers, symbols, categories, abbreviations  e.g. F (female)  e.g. MROLE (manager role)
  51. 51. Survey Plan1. Set objective & 5. Pre-test survey target audience 6. Administer survey2. Choose distribution 7. Analyze results method 8. Communicate &3. Create questions address results4. Question review/test Practice with first 3 steps
  52. 52. Survey Plan (real)1. Objective2. Target audience(s)3. Survey distribution method4. Additional research method(s) (*to verify survey data)5. Draft 1 clear, open-ended question*2-4 pairs to share plan with group. Time: 15 minutes (pairs)
  53. 53. Communicating Results Honesty about findings Timely results  4-6 weeks  = TRUST Act on findings or what happens?
  54. 54. Review1. One BA competency involved in survey analytical or systems design is ____________________ thinking.2. Another organizational research method you could use to verify your findings’ accuracy is Interviews, focus groups, ___________________________________ observations, secondary data ___________________________________.3. By using more than one method or riangulate stakeholder group you can T____________ your results to validate them.
  55. 55. Review4. Surveys/Questionnaires are considered an licitationE_________ technique by BABOK.5. Knowing the characteristics of yourtarget audience_______________ is important for surveydesign.6. Advantages of E-surveys are that they are: cost-effective, good for dispersed______________________________________ populations, and ease of analysis_______________________________.
  56. 56. Review7. Numerical or statistical data is quantitative____________, while descriptive data with rich qualitativedetail is ___________.8. Qualitative data analysis involves seeking patterns/themes________________ in the text, by using the re-read constant comparisonread-________ or __________________methods.9. Accurate qualitative data is described ashaving confidence, rigor, validity, consistency _______________________________.
  57. 57. Summary Know your audience, objective & resources Consider question types & analysis Triangulate (*more than one method, & stakeholder group) Follow best practices Empathize with your respondentsQuestions or comments?
  58. 58. Extras & Presenter Info1. Write #1 YES on back of your business card for the FREE “Top Ten Reasons Surveys Fail”2. Write #2 YES on back of your business card for a FREE Strategy Session.Karen Carleton, MEd, MSPerformance Solutions Corp. Survey design & consulting Needs analysis & evaluationwww.performcorp.ca Email: Karen@performcorp.ca
  59. 59. References Business Analysts’ Book of Knowledge (BABOK) Dillman, D.A. (2000). Mail and Internet surveys: The total design method (2nd ed.). New York: Wiley. Greatbrook. Statistical Confidence. Retrieved April 18, 2012 from, http://greatbrook.com/survey_statistical_confidence.htm Inquisite survey software, http://www.inquisite.com/ Qualtrics survey software, http://www.qualtrics.com/ Survey Monkey survey software, http://www.surveymonkey.com/  FREE account, up to 200 respondents. “How to” downloads.

×