This document provides an overview of research methods taught in an empirical research methods course. It discusses both quantitative and qualitative research methods. For qualitative methods, it describes techniques like ethnography, case studies, and grounded theory research. It explains data collection methods for qualitative research including direct observation, participant observation, in-depth interviews, and analyzing documents. The document also covers qualitative data analysis including coding data and developing models and theories from the analysis.
3. Steps in the research process
⢠Identify a phenomenon of interest
⢠Iterate:
â Investigate current state of knowledge (lit. review ?)
â Narrow down your interest to a research question or
hypothesis
⢠Identify research method to employ (survey,
experiment, ethnography, case study)
4. Steps in the research process (cont.)
⢠Operationalize research question or hypothesis
â Define the source of your data
⢠Sample population and recruitment method (if relevant)
⢠And/or the location/activities to be observed
â Define variables and/or data collection methods and
instruments
â Identify how the analysis will be carried out
----YOU NOW HAVE A RESEARCH PROPOSAL ------
5. Steps in the research process (cont.)
⢠Carry out your observations
⢠Analyze your data
⢠Draw conclusions, write up the results
Example: Freshman retention â a major concern of
every University !!
A quantitative study: midterm exam question
A qualitative study (we will discuss that)
6. Quantitative vs. Qualitative
ďŽ Goals
ďŽ Kinds of data
ďŽ Data collection methods
ďŽ Kinds of analyses
ďŽ Kinds of explanations
ďŽ When used in system development
lifecycle?
7. Goals of qualitative research
⢠Identify/compare patterns of behavior
⢠Analyze beliefs/subjective experience
â How people view a situation or experience
â Distinguish multiple perspectives
⢠Compare beliefs/self-reports to actual behavior
⢠DESCRIBE a complex phenomenon
⢠EXPLAIN why things happen
8. 8
Case studies of failed system development
or deployment
Ethnographic studies of workgroup
practices prior to introduction of new
technology
Examples of qualitative studies
9. 9
⢠Case study research
â˘Explore in depth one activity or project
â˘Limited in time and place
â˘Data collection by observation, interviews, artifacts . . .
â˘Goal: tell a coherent âstoryâ with lessons learned
â˘Most common methodology for IT empirical studies
⢠Ethnography
â˘Observation in natural setting
â˘Observer may become part of the group to experience
directly how its members interact
â˘Goal: identify patterns of interaction (power structures,
problem-solving/goal achievement)
Some qualitative methods in brief
10. 10
⢠Grounded theory research
⢠Data collection from artifacts and/or interviews
⢠Develop a set of categories and a model telling how they
relate to each other
⢠Goal: explain the meaning of what is observed
⢠Involves an bottom-up iterative process of data
collection/theory formation
Some qualitative methods (cont.)
11. + Systematic rules and procedures already worked
out, and can be followed
+ Traditional, accepted as âproofâ
- Closed-ended questions may lead to ignoring
important factors and relationships
- Quantitative methods cannot handle phenomena
that are difficult to turn into variables
Advantages/disadvantages of
quantitative studies
12. Lecture 1 - Introduction 12
+ more innovative and creative
+ capable of addressing issues that do not lend themselves
to being described by variables
- conclusions may be less credible
Advantages/disadvantages of
qualitative methods
13. 13
Pragmatic philosophy â find out whatever you can
using whatever methods are possible
Involves both qualitative and quantitative elements
ď (at least 2 stages of research)
Advantages/Disadvantages
+ combines structure and flexibility
- requires more time and resources
Mixed Methods
14. 14
Pattern 1: âinstrumentâ data followed by in-depth interview
to get insight on the reasons for the observed
relationships and capture any insights you
overlooked in study design
Examples of Mixed Method Designs
Pattern 2: exploratory study followed by survey or experiment
to generalize the results â representative of a
long-term research program
15. â˘Studies of computer/supported learning
â˘Studies of IT impacts in medicine
â˘Computer-supported collaboration (in general)
â˘Case study or ethnography for groupware
â˘Grounded theory study of chat groups
Applications of the Methods
Which Method would you choose?
16. Goals of qualitative research
⢠Identify/compare patterns of behavior
⢠Analyze beliefs/subjective experience
â How people view a situation or experience
â Distinguish multiple perspectives
⢠Compare beliefs/self-reports to actual behavior
⢠Ideally, propose a model to describe or explain
17. What is a model ??
What counts as evidence ??
Elements/types of qualitative models
⢠Define a taxonomy â useful aggregation of data
â Rogersâ adopter categories
⢠Innovators
⢠Early adopters
⢠Early majority
⢠Late majority
⢠Laggards
18. Elements/types of qualitative models
⢠Define properties that affect outcomes
â Perceived attributes of innovation that help explain
their different rates of adoption
⢠Relative advantage
⢠Compatibility
⢠Complexity
⢠Trialability
⢠Observability
⢠Explains individualâs likelihood of adopting
19. Elements/types of qualitative models
⢠Identify a relationship of interest
â Rogers: homophily/heterophily and diffusion
⢠Homophily is the degree to which two or more
individuals who interact are similar in certain attributes
such as beliefs, education, socioeconomic status, etc.
⢠Although homophily enables better communication,
innovation requires heterophily, at least regarding
knowledge of and experience with an innovation, and it
is likely that heterophily in that area co-occurs with
other important differences.
⢠This explains why some groups are slower to
adopt new behaviors
20. Elements/types of qualitative models
⢠Characterizing a process as a sequence of steps
â Rogers: the innovation/diffusion process
⢠Knowledge
⢠Persuasion
⢠Decision
⢠Implementation
⢠Confirmation
⢠Characterizing the âshapeâ of a process
â Rogers: the S-shaped curve
21. 21
Identify/define the variables (and their coding)
Design the âinstrumentâ â a measurement process or
technique
Types of instruments:
a survey (paper, phone, Web)
a form for observer to fill in (experiment or field study)
a prototype system (with interaction capture)
Field testing/validating instruments part of quantitative
methods
Collecting data for quantitative studies
22. Qualitative Data and Collection
Methods
⢠Direct observation
â Participant observation
⢠In-depth interviews
â Focus groups
⢠âArtifactsâ â usually text or Databases
23. Direct Observation
â May be in person or use audio or videotape,
observe through a 1-way mirror
â Unlike participant observation, often focused on
specific events (how many, how often, by whom,
observe patterns â for example, interruptions at a
meeting)
24. What to observe
⢠Spatial relations
⢠Activities
⢠Communication
â Verbal
â Other
⢠Tasks
â How work is allocated
25. How to be an effective observer
⢠Preparation
⢠Stay in the background
⢠Be factual and objective in your notes
â (interpretation comes later)
⢠Taking notes:
â Hand written usually
â Type in to computer later
⢠EXPANDING NOTES (ASAP)
26. â(Participant) observationâ: in
natural setting
⢠âParticipantâ observation occurs when you
interact casually and/or form relationships with
informants
⢠How much you actually âparticipateâ depends
on the goals of the study.
27. Participant observation
⢠Advantages:
â Offers insights into complex behavior
â Identify the âright questionsâ for further study
â Verify/correct self-reports
⢠Disadvantage:
â Time consuming
â Data collection is difficult
â Problem of subjectivity
28. How to operationalize
⢠Field notes
â Text
â Diagrams, maps
â Can result in numerical data
⢠Interviews (interviewer more clueful)
⢠Focus groups (facilitator more clueful)
29. What to observe
⢠Spatial relations
⢠Activities
⢠Communication
â Verbal
â Other
⢠Tasks
â How work is allocated
â See Table 3 in reading
30. Ethics
⢠Do not disrupt the activity your are observing
versus
⢠Do not mislead
⢠No formal rules about disclosing your role as a
researcher when engaging in casual
conversation â article suggests a point where
you want to ask specific question
⢠Disclosure includes: right of refusal,
confidentiality
31. Protecting confidentiality when
data is unique
⢠Separate identify info from field notes entered
into the computer
⢠People, organizations/companies, should be
given fictitious names
32. How to be an effective observer
⢠Preparation
⢠Stay in the background
⢠Be factual and objective in your notes
â (interpretation comes later)
⢠Taking notes:
â Hand written usually
â Type in to computer later
⢠EXPANDING NOTES
33. Tips
⢠Leave space
⢠Take notes strategically
⢠Use abbreviations
⢠Cover a range of observations:
Body language, etc.
34. Tips
⢠Leave space
⢠Take notes strategically
⢠Use abbreviations
⢠Cover a range of observations:
Body language, etc.
35. Participant observation: in natural
setting
⢠âParticipantâ observation occurs when you
interact and/or form relationships with
informants
⢠Demanding and time-consuming
⢠How much you actually âparticipateâ depends
on the goals of the study.
⢠Subjects may âforgetâ you are a researcher
36. How to operationalize
direct/participant observation
⢠Field notes
â Text
â Diagrams, maps
â Can result in numerical data
⢠Interviews (interviewer more clueful in P.O.)
⢠Focus groups (facilitator more clueful in P.O.)
âWater coolerâ effect
37. Participant observation
⢠Advantages:
â Offers insights into complex behavior
â Identify the âright questionsâ for further study
â Verify/correct self-reports
⢠Disadvantage:
â Time consuming
â Data collection is difficult
â Problem of subjectivity
38. Ethics of direct observation/
participant observation
⢠Do not disrupt the activity your are observing
versus
⢠Do not mislead
⢠No formal rules about disclosing your role as a
researcher when engaging in casual
conversation â some authors suggest a point
where you want to ask specific question
⢠Disclosure includes: right of refusal,
confidentiality
39. Protecting confidentiality when
data is unique
⢠Separate identify info from field notes entered
into the computer
⢠People, organizations/companies, should be
given fictitious names
40. In-depth interview/focus group
⢠Probes the interviewee(s) views of the
phenomenon of interest
⢠Interviewer/facilitator should be neutral
⢠Data collected: transcript, audio/video
recording, notes
41. In-depth interview/focus group
Interviewer should:
⢠Start with some open-ended questions
⢠Follow up by asking âhowâ and âwhyâ
⢠Keep the discussion on track
42. Documents
⢠Memos and meeting notes
⢠Transcripts of conversations or speeches
⢠Manuals and policy handbooks
⢠Newspapers and magazines
⢠Internet-based research
â Email
â Web sites
â Blogs
⢠Especially important in case studies
43. Qualitative Data Analysis
by John V. Seidel
⢠Description of how to go about analyzing transcripts
of interviews, documents, and/or field notes.
⢠Focus on âcodingâ
â First identify âeventsâ
â Assign terms that represent concepts of interest
â Organizing codes into a scheme
â Building qualitative models using the coding scheme as the
model vocabulary
⢠Focus on iterative nature of QDA
44. Two perspectives on coding
⢠Objectivist perspective
â Condensed representation of facts
â Can be subjected to hypothesis testing
â Strong burden of consistency/completeness
⢠Heuristic perspective
â Signposts pointing to things you care about
â Foundation for further analysis
45.
46. Elements/types of qualitative models
⢠Examples from Rogersâ theory of innovation
diffusion
â VCRâs
â Cell phones
â Metric system
â Seat belts in cars
â Dvorak keyboard
47.
48. Three analogies to explain this
⢠Jigsaw puzzle analogy
⢠A little data and a lot of right brain
⢠Multi-threaded DNA (patterns among the
patterns)