• Save
Capturing and Analyzing Qualitative Data in Surveys
Upcoming SlideShare
Loading in...5

Capturing and Analyzing Qualitative Data in Surveys



Here are my slides for my session for the International Institute of Business Analysis (May 23, 2012). I focus on best practices of survey design, highlighting qualitative data.

Here are my slides for my session for the International Institute of Business Analysis (May 23, 2012). I focus on best practices of survey design, highlighting qualitative data.



Total Views
Views on SlideShare
Embed Views



1 Embed 2

http://www.twylah.com 2



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Capturing and Analyzing Qualitative Data in Surveys Capturing and Analyzing Qualitative Data in Surveys Presentation Transcript

  • Capturing & Analyzing Qualitative Data in Surveys Karen Carleton, MEd, MS www.performcorp.caInternational Institute of Business AnalystsApril 23, 2012 © Karen Carleton 2012
  • Objectives1. Identify organizational context (PI)2. Overview of survey design3. Data collection & analysis best practices (Qualitative data focus)Exercises: Question critique Question writing practice (scenarios) Survey plan (real) Review
  • An ounce ofanalysis is worth a pound of objectives –Joe Harless
  • BA Competencies  Analytical thinking  Problem-solving  Organizational learning  Systems thinkingSource: Business Analysis Book of Knowledge
  • OrganizationalPerformance ImprovementRSVP ApproachResults-orientedSystems thinkingValue-addPartnerships focus
  • Systematic Solution DesignADDIE e.g. Frontline hours extendedAnalyze unintended problems elsewhereDesignDevelopImplementEvaluate
  • Organizational Research Process improvement Re-engineering Organizational change Software development Others?Source: Business Analysis Books of Knowledge
  • SurveysThink-Pair-ShareWhy might youconduct a survey inyour organization?
  • Survey Objectives Problems  Determine causes, impacts, optimal solution(s) & solution performance Opportunities  Improvement (efficiency/effectiveness), define business need or requirements Stakeholders  Elicit stakeholder needs/concerns, solution req’tsSource: Business Analysis Book of Knowledge
  • Organizational ResearchMETHOD DESCRIPTORSurvey/ Collects topical data from a representativeQuestionnaire populationFocus group Discussion-based group interview, noting ideas, themes & non-verbalsInterview One-on-one questioning (structured to unstructured)Document Background info, context (e.g. jobreview descriptions, stats, annual report)Observations Note details observed – situations/ behaviours (e.g. procedures)
  • Surveys/Questionnaires Elicitation Techniques (methods)  Interviews, case studies/scenarios, interviews, document analysis, focus groups, questionnaires/surveys Attitudes, opinions, recommendations, experiencesSource: Business Analysis Book of Knowledge
  • Organizational ResearchMore than 1 method (technique)Triangulate
  • Organizational Research More than 1 stakeholder group Triangulate Example: BPI
  • Target Audience Example Select participants  useful/relevant input Representative Sample  Determine precision needed
  • Best Practices Informed consent  Purpose, confidentiality, results Organized, consistent Know audience & resources
  • Survey Types In-person Phone Web-based
  • Which typeSurvey Types & why?  Flexible, interactive  Clarify, probe  Body language/non-verbals  Flexible, interactive  Clarify, probe; no travel  Voice tone, intonation  Easy for large respondent group  Cost-effective, convenient
  • Considerations Target Objective Audience Resources Data Types
  • Surveys  Objective  Target audience(s)  Location(s)  Resources ($, time, help)  Data needed (quantitative &/or qualitative) Examples?
  • Intro. MessageObjective/PurposeThankResultsConfidential; anonymousVoluntary
  • Staff Satisfaction SurveyAt ABC Company employee feedback is important tous. We would appreciate it if you would take 5-10minutes to complete this survey by Monday May 7,2012, 2012. All responses are anonymous and will bekept confidential.Your participation is voluntary and you may skip anyquestions or discontinue at any point. By June 15, 2012we will post the aggregate results and action items onthe intranet. Thanks in advance for sharing yourthoughts and opinions about how ABC Company couldbe a better workplace!
  • E-SurveysAdvantages Disadvantages Low cost, timely  Computer literate & Large group, dispersed access Data aggregation &  No follow-up/ storage (no data entry) probing Anonymity  No intonation, gestures Format ease  Little contact  Skip logic, progress bar  Mostly US servers
  • Quantitative Data Numerical (“objective”) data Measured, statistical, stable study  Length, height, volume, weight, speed, time, temperature, sound levels, cost, age range, # users or groups, %, etc.
  • Qualitative Data Descriptive (“subjective”) data; not traditionally measured Textual, flexible study  Colour, texture, appearance, beauty, smell, taste, etc.
  • Latte ExampleQuantitative Qualitative 12 ounce mug  Medium; see- 7 “ mug height through mug Serving  Steaming hot temperature 150ºF  Pricey drink Cost: $4.95  Robust aroma  Frothy; Delicious
  • Quantitative Data Analysis  Trends  Average (mean), highest & lowest scores  Generalizable sample - Statistically significant & representative
  • Qualitative Data Analysis Descriptive responses  Feelings, beliefs, likes/dislikes, thoughts, personal opinions, comments  Patterns/Themes, Counts
  • Qualitative DataRecommendation:Couple quantitative(numerical) & qualitative(descriptive) data“Confidence”, “Rigor”,“Reliability”, “Validity”,“Consistency”, “Truth”
  • Qualitative Data (open-ended questions) Limit use! Advantages Disadvantages  Personally  Non-response – more meaningful, cultural time/effort to do  Deeper questions (higher skip rate)  Rich, complex human  No follow up/clarifying details explained  Longer to analyze  Ideas not (more resources – anticipated time, people)Source: Business Analysis Book of Knowledge
  • Response Rate# Surveys completed = Response rate# Participants contacted Sufficient numbers important  Too little data= may not have substantial info. Example: client survey
  • Increase Response Rates1. User-friendly, brief2. Rewards-Cost-Trust  R=Value, positive regard, thanks, token WIIFM (incentive)  C=short, easy  T=important, authority, confidentiality3. Reminders Example: BSU faculty/staff, & students
  • Questions  Objectives – ask targeted questions  Clear, concise
  • Closed Questions (Quantitative) Rating/Matrix (4, 5 or 7 point) Ranking, ordinal (1-4) – order Multiple choice (one or more responses) Likert scale (Agree…), descriptors
  • Open Questions (Qualitative) Fill-in-the-blank (# or word) Text box (comment or paragraph) Example: Staff Survey “career development”
  • Question CritiqueUnder what circumstances Negative phrasingdo you not submit yourweekly project hours? Overtime still in progress Unknown factors NA  OtherDo you like our Double-barreledEdmonton and Calgarybranches? Yes  No
  • Question CritiqueWhat would make the Acronym orEFAP better? jargon; unclearIf you are working on a Confusing/late project and a team complex formatmember is away, do you:a) waitb) continuec) ask your supervisor foradvice
  • Question CritiqueHow fair was your Sensitiveperformance bonus this year?Strongly agree Agree Neutral Disagree Strongly disagree5 4 3 2 1Do you like the coffee? “Apple pieYes  No effect”Do you like the washrooms?Yes  NoDo you like the parking? Yes  NoDo you like your boss? Yes  No
  • Question CritiqueHow many copies did Lack of alternateyou make with the choice whenarea’s photocopier last needed: “N/A”,week? “no opinion” ora) Less than 5 “prefer not tob) Less than 20 answer”, ”don’tc) Less than 50 know”d) Less than 100e) 100 or more
  • Question CritiqueI generally feel informed about changesthat affect me at ABC Company.1 2 3 4 5Not at all VeryInformed Informed Unclear language & descriptors
  • Question CritiqueI can see the link between my work andABC Company’s objectives andmission.1 2 3 4 5Not at all Very Double-barreled question, Presumes understanding (2), Descriptors don’t fit.
  • Question CritiqueDo you feel the efforts of the SocialCommittee contribute to yourenjoyment in working at ABCCompany?1 2 3 4 5Not at all Very Leading question: presumes knowledge & enjoyment. Descriptors don’t fit.
  • Question CritiqueMy working conditions are good. 1 2 3 4 5Strongly StronglyDisagree AgreeMy relationship with my manager is good.1 2 3 4 5Strongly StronglyDisagree Agree Vague adjective
  • Question CritiqueIs there anything else you would like tomention that you feel could help improvestaff satisfaction at ABC Company? Wordy
  • Question TipsAt start, interesting questionEasier before difficult onesAt end, personal & demographicConsider analysisHave others reviewPretest Why?
  • Question Writing PracticeScenario pairs:1. Decide survey Objective2. Choose a fitting Distribution Method (e.g. phone, in-person, e-survey/web- based)3. Create 3 Questions Time: 20 minutes4. Question Review (with another group) Time: 5 minutes
  • Data AnalysisQuantitative Total responses, frequency counts, distribution, averagesQualitative Pre-determined or emergent categories
  • QualitativeTriangulation1. Cross-checkdata accuracy Validate, verify 1. Other method(s) 2. Other stakeholder(s)e.g. survey employees,interview supervisor
  • Qualitative Data Analysis 2. Constant Comparison Method  Compare & contrast text items  Sort, sift, group, match  Categorize, order, sub-divide “Raw” “Cooked” Data
  • Qualitative Data Analysis3. Read-Reread Method Iterative review Make sense of findings (determine meaning)  Underline, sort, count, code, highlight data  Patterns/picture(s) emerge
  • Qualitative Data Analysis4. Assign Codes Colours, numbers, symbols, categories, abbreviations  e.g. F (female)  e.g. MROLE (manager role)
  • Survey Plan1. Set objective & 5. Pre-test survey target audience 6. Administer survey2. Choose distribution 7. Analyze results method 8. Communicate &3. Create questions address results4. Question review/test Practice with first 3 steps
  • Survey Plan (real)1. Objective2. Target audience(s)3. Survey distribution method4. Additional research method(s) (*to verify survey data)5. Draft 1 clear, open-ended question*2-4 pairs to share plan with group. Time: 15 minutes (pairs)
  • Communicating Results Honesty about findings Timely results  4-6 weeks  = TRUST Act on findings or what happens?
  • Review1. One BA competency involved in survey analytical or systems design is ____________________ thinking.2. Another organizational research method you could use to verify your findings’ accuracy is Interviews, focus groups, ___________________________________ observations, secondary data ___________________________________.3. By using more than one method or riangulate stakeholder group you can T____________ your results to validate them.
  • Review4. Surveys/Questionnaires are considered an licitationE_________ technique by BABOK.5. Knowing the characteristics of yourtarget audience_______________ is important for surveydesign.6. Advantages of E-surveys are that they are: cost-effective, good for dispersed______________________________________ populations, and ease of analysis_______________________________.
  • Review7. Numerical or statistical data is quantitative____________, while descriptive data with rich qualitativedetail is ___________.8. Qualitative data analysis involves seeking patterns/themes________________ in the text, by using the re-read constant comparisonread-________ or __________________methods.9. Accurate qualitative data is described ashaving confidence, rigor, validity, consistency _______________________________.
  • Summary Know your audience, objective & resources Consider question types & analysis Triangulate (*more than one method, & stakeholder group) Follow best practices Empathize with your respondentsQuestions or comments?
  • Extras & Presenter Info1. Write #1 YES on back of your business card for the FREE “Top Ten Reasons Surveys Fail”2. Write #2 YES on back of your business card for a FREE Strategy Session.Karen Carleton, MEd, MSPerformance Solutions Corp. Survey design & consulting Needs analysis & evaluationwww.performcorp.ca Email: Karen@performcorp.ca
  • References Business Analysts’ Book of Knowledge (BABOK) Dillman, D.A. (2000). Mail and Internet surveys: The total design method (2nd ed.). New York: Wiley. Greatbrook. Statistical Confidence. Retrieved April 18, 2012 from, http://greatbrook.com/survey_statistical_confidence.htm Inquisite survey software, http://www.inquisite.com/ Qualtrics survey software, http://www.qualtrics.com/ Survey Monkey survey software, http://www.surveymonkey.com/  FREE account, up to 200 respondents. “How to” downloads.