Objectives1. Identify organizational context (PI)2. Overview of survey design3. Data collection & analysis best practices (Qualitative data focus)Exercises: Question critique Question writing practice (scenarios) Survey plan (real) Review
An ounce ofanalysis is worth a pound of objectives –Joe Harless
BA Competencies Analytical thinking Problem-solving Organizational learning Systems thinkingSource: Business Analysis Book of Knowledge
Staff Satisfaction SurveyAt ABC Company employee feedback is important tous. We would appreciate it if you would take 5-10minutes to complete this survey by Monday May 7,2012, 2012. All responses are anonymous and will bekept confidential.Your participation is voluntary and you may skip anyquestions or discontinue at any point. By June 15, 2012we will post the aggregate results and action items onthe intranet. Thanks in advance for sharing yourthoughts and opinions about how ABC Company couldbe a better workplace!
E-SurveysAdvantages Disadvantages Low cost, timely Computer literate & Large group, dispersed access Data aggregation & No follow-up/ storage (no data entry) probing Anonymity No intonation, gestures Format ease Little contact Skip logic, progress bar Mostly US servers
Quantitative Data Numerical (“objective”) data Measured, statistical, stable study Length, height, volume, weight, speed, time, temperature, sound levels, cost, age range, # users or groups, %, etc.
Qualitative Data Descriptive (“subjective”) data; not traditionally measured Textual, flexible study Colour, texture, appearance, beauty, smell, taste, etc.
Qualitative Data (open-ended questions) Limit use! Advantages Disadvantages Personally Non-response – more meaningful, cultural time/effort to do Deeper questions (higher skip rate) Rich, complex human No follow up/clarifying details explained Longer to analyze Ideas not (more resources – anticipated time, people)Source: Business Analysis Book of Knowledge
Response Rate# Surveys completed = Response rate# Participants contacted Sufficient numbers important Too little data= may not have substantial info. Example: client survey
Closed Questions (Quantitative) Rating/Matrix (4, 5 or 7 point) Ranking, ordinal (1-4) – order Multiple choice (one or more responses) Likert scale (Agree…), descriptors
Open Questions (Qualitative) Fill-in-the-blank (# or word) Text box (comment or paragraph) Example: Staff Survey “career development”
Question CritiqueUnder what circumstances Negative phrasingdo you not submit yourweekly project hours? Overtime still in progress Unknown factors NA OtherDo you like our Double-barreledEdmonton and Calgarybranches? Yes No
Question CritiqueWhat would make the Acronym orEFAP better? jargon; unclearIf you are working on a Confusing/late project and a team complex formatmember is away, do you:a) waitb) continuec) ask your supervisor foradvice
Question CritiqueHow fair was your Sensitiveperformance bonus this year?Strongly agree Agree Neutral Disagree Strongly disagree5 4 3 2 1Do you like the coffee? “Apple pieYes No effect”Do you like the washrooms?Yes NoDo you like the parking? Yes NoDo you like your boss? Yes No
Question CritiqueHow many copies did Lack of alternateyou make with the choice whenarea’s photocopier last needed: “N/A”,week? “no opinion” ora) Less than 5 “prefer not tob) Less than 20 answer”, ”don’tc) Less than 50 know”d) Less than 100e) 100 or more
Question CritiqueI generally feel informed about changesthat affect me at ABC Company.1 2 3 4 5Not at all VeryInformed Informed Unclear language & descriptors
Question CritiqueI can see the link between my work andABC Company’s objectives andmission.1 2 3 4 5Not at all Very Double-barreled question, Presumes understanding (2), Descriptors don’t fit.
Question CritiqueDo you feel the efforts of the SocialCommittee contribute to yourenjoyment in working at ABCCompany?1 2 3 4 5Not at all Very Leading question: presumes knowledge & enjoyment. Descriptors don’t fit.
Question CritiqueMy working conditions are good. 1 2 3 4 5Strongly StronglyDisagree AgreeMy relationship with my manager is good.1 2 3 4 5Strongly StronglyDisagree Agree Vague adjective
Question CritiqueIs there anything else you would like tomention that you feel could help improvestaff satisfaction at ABC Company? Wordy
Question TipsAt start, interesting questionEasier before difficult onesAt end, personal & demographicConsider analysisHave others reviewPretest Why?
Data AnalysisQuantitative Total responses, frequency counts, distribution, averagesQualitative Pre-determined or emergent categories
QualitativeTriangulation1. Cross-checkdata accuracy Validate, verify 1. Other method(s) 2. Other stakeholder(s)e.g. survey employees,interview supervisor
Qualitative Data Analysis 2. Constant Comparison Method Compare & contrast text items Sort, sift, group, match Categorize, order, sub-divide “Raw” “Cooked” Data
Qualitative Data Analysis3. Read-Reread Method Iterative review Make sense of findings (determine meaning) Underline, sort, count, code, highlight data Patterns/picture(s) emerge
Qualitative Data Analysis4. Assign Codes Colours, numbers, symbols, categories, abbreviations e.g. F (female) e.g. MROLE (manager role)
Survey Plan1. Set objective & 5. Pre-test survey target audience 6. Administer survey2. Choose distribution 7. Analyze results method 8. Communicate &3. Create questions address results4. Question review/test Practice with first 3 steps
Survey Plan (real)1. Objective2. Target audience(s)3. Survey distribution method4. Additional research method(s) (*to verify survey data)5. Draft 1 clear, open-ended question*2-4 pairs to share plan with group. Time: 15 minutes (pairs)
Communicating Results Honesty about findings Timely results 4-6 weeks = TRUST Act on findings or what happens?
Review1. One BA competency involved in survey analytical or systems design is ____________________ thinking.2. Another organizational research method you could use to verify your findings’ accuracy is Interviews, focus groups, ___________________________________ observations, secondary data ___________________________________.3. By using more than one method or riangulate stakeholder group you can T____________ your results to validate them.
Review4. Surveys/Questionnaires are considered an licitationE_________ technique by BABOK.5. Knowing the characteristics of yourtarget audience_______________ is important for surveydesign.6. Advantages of E-surveys are that they are: cost-effective, good for dispersed______________________________________ populations, and ease of analysis_______________________________.
Review7. Numerical or statistical data is quantitative____________, while descriptive data with rich qualitativedetail is ___________.8. Qualitative data analysis involves seeking patterns/themes________________ in the text, by using the re-read constant comparisonread-________ or __________________methods.9. Accurate qualitative data is described ashaving confidence, rigor, validity, consistency _______________________________.
Summary Know your audience, objective & resources Consider question types & analysis Triangulate (*more than one method, & stakeholder group) Follow best practices Empathize with your respondentsQuestions or comments?
Extras & Presenter Info1. Write #1 YES on back of your business card for the FREE “Top Ten Reasons Surveys Fail”2. Write #2 YES on back of your business card for a FREE Strategy Session.Karen Carleton, MEd, MSPerformance Solutions Corp. Survey design & consulting Needs analysis & evaluationwww.performcorp.ca Email: Karen@performcorp.ca
References Business Analysts’ Book of Knowledge (BABOK) Dillman, D.A. (2000). Mail and Internet surveys: The total design method (2nd ed.). New York: Wiley. Greatbrook. Statistical Confidence. Retrieved April 18, 2012 from, http://greatbrook.com/survey_statistical_confidence.htm Inquisite survey software, http://www.inquisite.com/ Qualtrics survey software, http://www.qualtrics.com/ Survey Monkey survey software, http://www.surveymonkey.com/ FREE account, up to 200 respondents. “How to” downloads.