Data collection and analysis

1,076 views
855 views

Published on

Week 4 lecture for im2044 2012-2013

Published in: Education, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,076
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
50
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Analyzing video and data logs can be time-consuming
  • Data collection and analysis

    1. 1. IM2044 – Week 4: Lecture Dr. Andres Baravalle 1
    2. 2. Interaction design • The next slides are based on the companion slides for the textbook • By the end of this week, you should have studied all chapters of the textbook up to chapter 8 2
    3. 3. Overview • Data gathering – Interviews – Questionnaires – Ethnographic observations and observations in a controlled environment • Qualitative and quantitative analysis 3
    4. 4. Key issues in data gathering • Setting goals – Decide how to analyze data once collected • Identifying participants – Decide who to gather data from • Relationship with participants – Clear and professional – Informed consent when appropriate • Triangulation – Look at data from more than one perspective • Pilot studies – Small trial of main study 4
    5. 5. Data recording • Typically supported by a combination of: – Notes – Audio recording – Video recording – Photography 5
    6. 6. Interviews • Unstructured - not directed by a script. Rich but not replicable. • Structured - tightly scripted, often like a questionnaire. Replicable but may lack richness. • Semi-structured - guided by a script but interesting issues can be explored in more depth. Can provide a good balance between richness and replicability. 6
    7. 7. Closed vs. open questions • ‘Closed questions’ have a predetermined answer format, e.g., ‘yes’ or ‘no’ – Easier to analyse • ‘Open questions’ do not have a predetermined format – Can allow to better explore research topics 7
    8. 8. Questions to avoid • Long questions • Compound sentences - split them into two • Jargon and language that the interviewee may not understand • Leading questions that make assumptions – e.g., why do you like …? Or – Asking a question that the respondent is not qualified to answer • Unconscious biases, e.g. gender stereotypes 8
    9. 9. Running the interview • Introduction – introduce yourself, explain the goals of the interview, reassure about the ethical issues, ask to record, present any informed consent form. • Warm-up – make first questions easy and nonthreatening. • Main body – present questions in a logical order • A cool-off period – include a few easy questions to defuse tension at the end • Closure – thank interviewee and signal the end, e.g. switch recorder off. 9
    10. 10. Enriching the interview process • Props - devices for prompting interviewee, e.g., a prototype, scenario 10
    11. 11. Questionnaires • Questions can be closed or open – Closed questions are easier to analyze, and may be done by computer • Can be administered to large populations – Paper, email and the web used for dissemination • Sampling can be a problem when the size of a population is unknown as is common online 11
    12. 12. Questionnaire design • Provide clear instructions on how to complete the questionnaire. • Decide on whether phrases will all be positive, all negative or mixed. • Different versions of the questionnaire might be needed for different populations. • The impact of a question can be influenced by question order. • Strike a balance between using white space and keeping the questionnaire compact. 12
    13. 13. Question and response format • ‘Yes’ and ‘No’ checkboxes • Checkboxes that offer many options • Rating scales – Likert scales – Semantic scales – 3, 5, 7 or more points • Open-ended responses 13
    14. 14. Encouraging a good response • Make sure purpose of study is clear • Ensure questionnaire is well designed – Consider offering a short version for those who do not have time to complete a long questionnaire • • • • Promise anonymity Follow-up with emails, phone calls, letters Provide an incentive 40% response rate is high, 20% is often acceptable 14
    15. 15. On-line questionnaires • Responses are usually received quickly • No copying and/or postage costs • Data can be easily collected in database for analysis • Time required for data analysis is reduced • Errors can be corrected easily 15
    16. 16. Problems with online questionnaires • Sampling is problematic if population size is unknown • Preventing individuals from responding more than once 16
    17. 17. Observation • Direct observation in the field – Structuring frameworks – Degree of participation (insider or outsider) – Ethnography • Direct observation in controlled environments • Indirect observation: tracking users’ activities – Diaries – Interaction logging 17
    18. 18. Structuring frameworks to guide observation • The Goetz and LeCompte (1984) framework: - Who is present? - What is their role? - What is happening? - When does the activity occur? - Where is it happening? - Why is it happening? - How is the activity organized? 18
    19. 19. Ethnographic observations (1) • Ethnography is a is a qualitative research method aimed to understand human behaviour – Techniques used include participant observation, interviews and questionnaires • Ethnographers immerse themselves in the culture that they study – Collections of comments, incidents, and artifacts are made – A researcher’s degree of participation can vary along a scale from ‘outside’ to ‘inside’ 19
    20. 20. Ethnographic observations (2) • Co-operation of people being observed is required – Informants are central – Questions get refined as understanding grows • Data analysis is continuous and typically time consuming • Reports usually contain examples 20
    21. 21. Online ethnography • On-line interaction differs from face-toface • Virtual worlds have a persistence that physical worlds do not have • Ethical considerations and presentation issues are different 21
    22. 22. Observation in a controlled environment • Direct observation: – Think-aloud technique • Indirect observation: – Diaries – Interaction logs – Web analytics 22
    23. 23. Choosing and combining data gathering techniques • Depends on: – The focus of the study – The participants involved – The nature of the techniques selected – The resources available 23
    24. 24. Quantitative and qualitative data • Quantitative data – expressed as numbers – Quantitative analysis – numerical methods to ascertain size, magnitude, amount • Qualitative data – difficult to measure sensibly as numbers e.g. dissatisfaction – Qualitative analysis – expresses the nature of elements and is represented as themes, patterns, stories • Be very careful on how you manipulate data and numbers! 24
    25. 25. Simple quantitative analysis • Averages – Mean: add up values and divide by number of data points – Median: middle value of data when ranked – Mode: figure that appears most often in the data • Percentages • Graphical representations give overview of data Number of errors made Internet use 10 < once a day 8 once a day 6 4 once a week 2 2 or 3 times a week 0 0 5 10 15 20 once a month User 25 Number of errors made Number of errors made Number of errors made 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 1 3 5 7 9 User 11 13 15 17
    26. 26. Visualising log data Interaction profiles of players in online game Log of web page activity 26
    27. 27. Visualising log data • Web analytics 27
    28. 28. Simple qualitative analysis • Recurring patterns or themes – Emergent from data, dependent on observation framework if used • Categorizing data – Categorization scheme may be emergent or pre-specified • Looking for critical incidents – Helps to focus in on key events 28
    29. 29. Tools to support data analysis • Spreadsheet – simple to use, basic graphs • Statistical packages - e.g. SPSS • Specialist qualitative data analysis tools – Categorization and theme-based analysis, e.g. N6 – Quantitative analysis of text-based data 29
    30. 30. Theoretical frameworks for qualitative analysis • Basing data analysis around theoretical frameworks provides further insight • Three such frameworks are: – Grounded Theory – Distributed Cognition – Activity Theory 30
    31. 31. Grounded Theory • Aims to derive theory from systematic analysis of data • Based on categorization approach (called here ‘coding’) • Three levels of ‘coding’ – Open: identify categories – Axial: flesh out and link to subcategories – Selective: form theoretical scheme • Researchers are encouraged to draw on own theoretical backgrounds to inform analysis 31
    32. 32. Distributed Cognition • The people, environment & artefacts are regarded as one cognitive system • Used for analyzing collaborative work • Focuses on information propagation & transformation 32
    33. 33. Activity Theory • Explains human behavior in terms of our practical activity with the world • Provides a framework that focuses analysis around the concept of an ‘activity’ and helps to identify tensions between the different elements of the system • Two key models: one outlines what constitutes an ‘activity’; one models the mediating role of artifacts 33
    34. 34. Presenting the findings • Only make claims that your data can support • The best way to present your findings depends on the audience, the purpose, and the data gathering and analysis undertaken • Graphical representations may be appropriate for presentation • Other techniques are: – Rigorous notations, e.g. UML – Using stories, e.g. to create scenarios 34

    ×