New Media Research processAnalysisDesignImplementation
Qualitative Quantitative• Qualitative - deepunderstanding of theexperience. Noteverything can bereduced to numbers.• E.G.Observation, ethnography• Quantitative –objective,measurable. Helps toestablish patternsand relationships.• E.G. Survey
Which one to use?http://www.youtube.com/watch?v=ddx9PshVWXI
Qualitative Method?– User perspectives– Questions: how and why?– Captures perceptions, judgements, meanings,processes and reasons– Open ended questions, checklist of topics– Hypothesis and follow up questions are generatedduring data collection and analysis– Subjective – not easy to objectively verify– Interviews allow to probe– Provides deeper insight– Broader understanding and explanations, views
Qualitative Method?• Strengths– Participatory– Rich, detailed data– Considers users perspectives and the context for theirbehaviours• Weaknesses– Hard work– Time consuming– Smaller sample of users– Not easily verifiable– Not easy to group your responses/categorised
Quantitative Methods– Designed to ensure objectivity, reliability and ability togeneralise– Test predetermined hypothesis – explanations– Who, how much, how many?– Closed questions– Short answers– Averages, percentages, ranges, means, frequencies– Can generate accurate and precise data– Can test statistical relationship between variable– Can prove whether or not a particular problem exists– Can identify specific characteristics of a population
Quantitative Methods• Strengths– Robust– Objective– Verifiable• Weaknesses– Out of context – human behaviour, real worldsettings are not considered– Any variables left out of data collection are notused in analysis
Quantitative or Qualitative or Both?• Quantitative methods can highlight an issuesthat could then be studied in depth usingqualitative methods.• Qualitative methods might be used at thebeginning of a study to help the researcher todecide what closed questions could be used inthe bigger quantitative survey orquestionnaire.
New Media ResearchInterviews• Purpose• Characteristics• Advantages• Limitations• Process/Stages/Questions
Definitions / purposes“Qualitative research [such as interviewing]attempts to understand meanings thatpeople give to their deeds, experiences, or toother social phenomena”Silverman D (1997)‘The purpose of interviewing is to find out whatis in and on someone else’s mind. We findout from them those things which we cannotdirectly observe’May T (1997)
Advantages• Rich data - excellent for complex subjects• Meanings / understandings / perceptions explored• Powerful when territory unfamiliar• Unforeseen issues / experiences elicited• Clarification / Follow up Q&As possible• Captive subjects• Certainty over who is responding (unlike surveys)• Non-verbal communication / Observational opportunities
Disadvantages (1)Subjects may:• Conform to expectations – (social desirability)• try to be rationalInterviewer may:• Be inconsistent• (Unwittingly) bias respondents’ answers• Mis-interpret answers
Disadvantages (2)• Difficult to get quantitative data• Data analysis difficult / complicated• Unrepresentative - reliance on key figures• Time consuming / costly• Stressful for interviewer (& interviewee?)• Low validity / reliability
Reliability and validity• Validity: "By validity, I mean truth: interpreted as theextent to which an account accurately represents thesocial phenomena to which it refers. " (Hammersley,1992). Pg. 57. (e.g. does the data-gathering measurewhat you want it to measure?)• Reliability: the degree of consistency with whichinstances are assigned to the same category bydifferent observers or by the same observer ondifferent occasions". (Hammersley, 1990). Pg. 67 (e.g.does the data-gathering produce the same results ifrepeated?) .
Validating interviews• Triangulation• Purposive sampling - focus on specific population• Choose deviant case• Member check (refer findings back to subjects)• Co-researcher re-coding• Researcher ‘reflexivity’• ‘Fair dealing’ – ensure all viewpoints mentioned
But don’t forget …One case may be enough(!):‘What is wrong with samples of one? Why shouldresearchers have to apologise for them? ShouldPiaget apologise for studying his own children, aphysicist for splitting only one atom?’Mintzberg H (1973) The nature of managerial work NY:Harper & Row
Preparing the interview (1)Getting ethical approval:“The (UH) Ethics Committee will … carefully evaluate thefollowing aspects of your application:• the validity of the research• the welfare of the research subjects• the dignity of the research subjects• the ability of the researcher/team/supervisor to conduct theresearch”[see: http://perseus.herts.ac.uk/uhinfo/library/j34130_13.doc]
Preparing the interview (2)• Aims/objectives of the interview• Where these fit with overall research study• Interview style:– Structured unstructured– Formal informal– Open closed• Choose physical setting (if possible)• Atmosphere / environment
Recording the interviewTo tape or not to tape?? Consider:• Effect on interviewee• Listening / transcribing time• Reliability of machine / recordingBut also…• Scribbling whilst listening• Making sense of notes
Conducting the interview• Thank interviewee• Set the scene (why / how / where of study)• Give idea of question areas• Give ground rules (can refuse to answer/canterminate interview, anonymity etc.)• Start with demographic and ID questions• (usually) start with general question and thenfunnel
Types of question (1)Open• Opening stages in line of questioning (funnel)• Invites opinion, general knowledge.• Can cover areas where interviewer’s own knowledgelacking• No presumption about responseClosed• Elicit hard facts• Control pace/direction of interview
Types of question (2)Probing• Extracts more depth• Maintains line of enquiryLeading• Confirm interviewee’s answer• Help interviewee, by rephrasing answer• Bring a line of questioning to an end (summarising)
New Media ResearchQuestionnaires/Surveys• Purpose• Characteristics• Advantages• Limitations• Process/Stages/Questions
Purpose of questionnairesTo survey large number of people, todescribe/explain characteristics/opinion of apopulation, usually through a representativesample.Questionnaires measure generalities / theextent to which groups of people behave orthink in certain ways.
Characteristics of questionnaires• Cross-sectional• Mainly quantitative• ‘Snapshot’ in time• Qs and As standardised
Advantages• Reaches large / dispersed populations• Can generalise results• Provides quantitative, authoritative (?) data• Appears easy– work done by respondent• Relatively cheap and quick (per unit)• Removes personal influence• Respondent works in own time• Provides structure for report• Replicable
Limitations (1):Questionnaire construction• Low response rate (5-25%?)• Respondents may differ from non-respondents• No way to adapt add/remove questions• Little opportunity for respondents to explain• Can over-simplify issues• Respondents ‘shoehorned’
Limitations (2):Respondent issuesRespondents may:• Take middle position in polar responses• Lack recall / rely on recent experience• Want to please researcher by being:– socially responsible– compliant
Process/stages• Formulate study aims• Identify objectives that address aims• Decide what information is required, and from whom• Decide sampling frame (elements making up population)• Research for similar questionnaires• Formulate appropriate questions• Decide distribution method– Postal; Telephone; Clip board/street; Web/Internet; Email;f2f• Pilot (see next slide) and get feedback• Amend• Distribute / administer
The pilot• Use more “open ended” questions than in finalproduct (often interviews are undertaken first)• For “Pilot” responses look for:– Variation in type of answer– Redundancy – areas of no/limited response– Evidence of ambiguity etc.– Acquiesce• Amend questionnaire as appropriate
Question types (1)• Open questions– For further information (‘tell us more ‘)– For new information (‘what are your view on …’)• Multiple choice checklist– One answer only– All that apply (inc. Guttman scale)• Binary (yes/no)– Good for filtering• Rank order
Question types (2)• Likert scale– Agreement (strongly agree, agree, …)– Frequency (always, frequently, …)– Importance (very important, important …)– Quality (excellent, good, average …)– Likelihood (definitely, probably …)• Semantic differentialReactions to stimulus words / concepts in terms of ratings on‘bipolar’ scale with contrasting adjectives at each end:– Excellent ………………terrible– Helpful …………………unhelpful
Closed v open questions (1)closed …Advantages• Quick & easy for respondents• Less articulate for disadvantaged• Fewer irrelevant answers• Easy to code and analyseDisadvantages• Responses suggest ideas (e.g., shoehorning)• Frustrates respondents if categories not exhaustive• Misinterpretation goes unnoticed• Complex issues forced into simple categories
Closed v open questions (2)Open …Advantages• Permits detail, clarification• Unanticipated answers• Reveals the logic behind responseDisadvantages• Generalisation or comparison difficult• Coding and statistical analysis difficult• Irrelevant answers possible• Bias towards educated• Time consuming for respondents & researcher
Good practice• Explain purpose• Catch interest early• Question sequence logical / helpful - broad tospecific• Closed questions need "complete" set of responsealternatives• Use appropriate language• Include clear instructions• Use clear tick boxes• Keep short
Increasing response rate• Name recipient if possible• SAE• Prize (?)• Offer copy of the report• Good design/construction• Engender involvement/interest• Chase non-respondents• Target ignored/committed groups
Issues to considerQuestion wordingQuestion orderQuestion types
Question wordingWhat’s wrong with these questions:Do you like using the Internet and playing onlinegames?Do you agree with most people that onlinegaming is becoming more popular?
Question orderWhat’s wrong with these questions:1) Do you have any children?2) How old are you?3) Are you married?4) How long have you been studying at theUniversity?
Question types (1)Leading questions:• Why do you think the PC’s are worse than Apple computers?Social pressure:• Would you prefer to have more money spent on improving theeducation or more tax cuts?Presuming questions• How often do you search the Internet?
Question types (2)Ambiguous/imprecise questions• How much time to you spend playing online games?Double-barreled questions• How long have you been using computers and the Internet?Too much knowledge dependent questions• Do you agree with the … theory?
Question types (3)Memory dependent• How many times have you played a computer game in the lastmonth?Wish List/hypothetical questions• Would you like better computer games to be designed?
Match tools to your questions• For questions on factualclarification• Use polls and surveys• For questions on opinion• Use surveys, interviews, focusgroups• For questions on experience• Interviews, focus groups,observations, user testing, andethnographies• For questions on concepts• surveys, interviews,ethnographies, and user testing• For questions on emotions• Surveys, interviews, focus groups,and observations
You can mix tools• You need to map out each step you will take inyour research so that things follow in a logicalorder…• This is your method!
Spend the rest of the time thinking throughyour proposed question• Independent Variable– What is tested, measured or manipulated• Dependent Variable– What is observed – the outcome
References• Curasi CF (2001) A critical exploration of face-to-face interviewing vs.computer-mediated interviewing International Journal of MarketResearch 43(4)pp361-375• Hammersley, M. (1992). What’s wrong with Ethnography:Methodological Exploration, London: Routledge.• May T (1997) Social research: issues, methods, progress Buckingham:OUP• Rugg, D. (1941) Experiments in wording question. II Public OpinionQuarterly, 5:91-92.• Silverman D (1997) Qualitative research: theory, method,practiceLondon: Sage• Silverman D (2004) Doing Qualitative Research: A Practical HandbookLondon: Sage