This document discusses research design considerations for internet-mediated research (IMR). It addresses tools and procedures for quantitative and qualitative IMR, including surveys, experiments, interviews and observation. Key issues include balancing control, validity and accessibility. Careful attention to sampling frames, representation of populations, informed consent and anonymity is important. Further research is needed to validate IMR methods and better understand online populations and interaction effects. Overall, web-based approaches are preferable when control and functionality is required, though simpler methods may still have value depending on research goals.

1. Research Design and Tools for
Internet Research
Claire Hewson
University of Bolton
ESRC e-society Programme and Sage Handbook of Online
Research Methods Colloquium, 28-29 March, 2007

2. Aims
• Summarise some key issues and principles
involved in designing Internet-mediated
research (IMR) studies
• Consider some of the approaches possible
(tools, procedures), and reasons for
choosing between these alternatives
• Highlight areas which need further
exploration, evaluation and development in
relation to IMR design principles

3. Research Design in IMR
• Research design - the process of making
choices at various levels, or stages, within
the research (including developing a
research question, specifying a theoretical
orientation, selecting data collection and
analysis procedures)
• Research design in IMR - focus primarily
on the data-gathering stage

4. Why Use IMR?
• Advantages
– cost / time efficiency (no data input costs, or
multiple copies of materials, etc.)
– easy access to potentially vast, geographically
diverse participant pool (IUP) [enhanced reach,
statistical power, access to small specialist
populations]
– unique features of online environment can lead
to various potential benefits (e.g. reduced social
desirability effects, increased candour, balanced
power relationships, etc.)

5. Why Use IMR?
• Disadvantages
– biased nature of IUP(?) [inherent bias in
Internet-accessed samples; generalisability
issues]
– reduced levels of control (over procedures /
stimuli, participation context, participant
behaviour)
– features of online interaction could lead to
various disadvantages (e.g. ambiguities in
communication due to lack of extra-linguistic
cues)

6. Tools, Procedures, Design
Considerations in Quantitative IMR
• Design choices essentially linked to the criteria of
validity, reliability and generalisability
• Surveys, experiments, observation, and document
analysis, have been implemented to date

7. Internet Surveys
• Tools:
– www forms, email, ftp; lower or higher tech
solutions (e.g. javascript, server-side scripting);
audio / video capabilities
• Issues influencing design choices:
– levels of expertise / equipment available
– levels of control required (over presentation /
response format)
– sophistication of procedures / functionalities
required (ID-checking, etc.)
– data security (validity / ethical concerns)

8. Internet Surveys
• Overall, web-based approaches offer far greater
scope and are likely to be preferable for most
types of survey-based research, issues relating to
levels of technical support / ease of
implementation aside

9. Internet Experiments
• Tools:
– essentially same range of tools and approaches
as for survey-based research, though web-based
approaches likely to be required in many cases
(precise timings, control over presentation, etc.)
• Issues influencing design choices:
– essentially, the same issues as with survey-
based approaches emerge (ease of
implementation, levels of control,
functionalities required, data security)

10. Internet Experiments
• Overall, web-based approaches are almost always
to be preferred (and will often be necessary) in
experimental IMR, given the levels of control and
sophistication of functionalities that will often be
required. Though in some research contexts
alternative approaches may be possible (e.g.
emailing experimental text-based materials to
participants, e.g. Hewson, 1994)

11. Summary of IMR Design in Quantitative
Research
• A trade-off emerges between simple, low tech
implementations which may suffer reliability and
validity issues, and higher tech solutions which
allow greater levels of control (etc.), but place
greater demands on the researcher (and possibly
participant, perhaps restricting access and thus
affecting generalisability). Clearly specific
research aims and goals, as well as practical
considerations, will direct the design choices
made.

12. Tools, Procedures, Design
Considerations in Qualitative IMR
• Design choices essentially linked to criteria of
trustworthiness (dependability, etc.)
• Interviews, observation, and document analysis,
have been implemented to date

13. Internet Interviews
• Tools:
– asynchronous (email, mailing lists, discussion groups)
and synchronous (IM, online chat) approaches; audio /
video capabilities
• Issues influencing design choices:
– nature of interaction / data desired (rich, detailed,
reflective; factual, quick to obtain; candid, sensitive,
personal, etc. [relates to: timescale; levels of
anonymity; availability of extra-linguistic information])
– demands placed on participant and researcher (skill /
experience levels, etc.)
– reach, access and diversity (e.g. issue of different time
zones, accessibilty by e.g. disabled participants)

14. Internet Observation / Document Analysis
• Tools:
– essentially same range as for interviews, though often
archives of online interactions will be most useful,
which tend to involve asynchronous communications
(real-time observation is also feasible, and observation
beyond text-based interactions, e.g. in MU*s)
• Issues influencing design choices:
– specificity of material required (e.g. rumour
transmission, Bordia, 1996)
– resources / expertise available (e.g. chat software)
– disclosed / undisclosed (relates to research aims, and
ethics [e.g. public / private domain])
– naturalistic / solicited (e.g. Hessler et al, 2003)

15. Summary of IMR Design in Qualitative Research
• Trade-offs will tend to emerge, e.g. asynchronous
[depth, reflexivity, empowering participants,
accessibility / reach] versus synchronous
[enhanced flow, extra-linguistic information,
speedier]. Greater anonymity may enhance
candour, but inhibit rapport. Audio / video
approaches currently suffer reliability issues.
Cross-cultural approaches would seem to demand
asynchronous methods. Naturalistic (archives)
versus solicited approaches. Ethical considerations
(disclosure, consent, anonymity / confidentiality).

16. Sampling Issues
• Tools:
– email, discussion group / mailing list postings, www
adverts, dedicated IMR study pages, sampling offline
• Issues:
– inherent biases in IUP
– probability sampling problematic (of IUP, and of sub-
populations, e.g. dormant email addresses, etc.)
– sampling procedures may lead to biases, but difficulty
in measuring sampling frame / nature of biases
– how to maximise response (follow-ups; incentives;
issue salience; group readership / volume of postings)
– ethics (netiquette, e.g. permission of moderators, topic
relevance; email requests?)

17. IMR - Some Proposed Good Design
Principles
• Use procedures which can restrict variations on
parameters which must crucially remain constant
to ensure validity / reliability, but which maximise
accessibility
• Gather as much information as feasible to help
detect unwanted variations / validity threats
• Provide clear, explicit instructions, beyond what
may normally be necessary for offline approaches
• Pilot extensively (e.g. in different browsers)

18. IMR - Some Proposed Good Design
Principles
• Aim for procedures where sampling frame can be
estimated, and sample bias assessed, when
generalisability is important
• Carefully consider ethical issues, especially when
sensitive / personal information is being obtained
• Consider features of synchronous and
asynchronous interactions, and their effects, and
effects of greater or lower levels of anonymity,
and assess the relevant trade-offs in order to select
procedures that best meet research aims / goals

19. Issues for Further Research in IMR
Design
• Validation studies which support IMR procedures
(e.g. adaption of psychometric tests, experimental
procedures)
• Nature of IUP and relationship between sampling
procedures and sample characteristics
• Effects of online interactional medium
• Issues surrounding public / private domain online
• Reliability of technologies (e.g. audio / video)
• Further piloting and exploration of advantages /
disadvantages of novel approaches, e.g. mixed
methods studies