Presentation on research method in applied linguistics

1. Quantitative Data Collection
Research Methods in applied linguistics
By Zoltán Dörneyi
1/45

2. Introduction
 The most common instrument for
collecting quantitative data is the test.
tests have several types: language test or
psychological test (aptitude or personal
batteries)
 A frequent method of collecting
quantitative data is through conducting a
survey using some sort of questionnaire.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
2/45

3. Sampling in quantitative research
The most frequent questions asked by
novice researchers before starting
investigation:
 How many people do I need to include in
my study?
 Hoe large should my sample be?
 What sort of people shall I select?
Sampling decisions affect the necessary
arrangements, timing and scheduling of
the project as well as the costs.
3/45
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

4. Sample, population,
representativeness
Sample: The group of participants whom the
researcher actually examines in investigation.
Population: The group of people whom the
study is about.
 The sample is a subset of the population that
is representative of the whole population.
 The strength of the conclusions we can draw
from the results obtained from a selected
small group depends on how accurately the
particular sample represents the larger
population.
4/45
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

5. Sampling procedures
 Probability Sampling
 Random Sampling: selecting members of
population on a completely random basis
 Stratified random Sampling: combination of
randomization and categorization
 Systematic Sampling: selecting every nth member
of the target group
 Cluster Sampling: random selection of larger
groupings or units of population(especially when the
population is widely dispread)
5/45
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

6. Sampling procedures
 None-probability Sampling
 Quota Sampling and Dimensional
Sampling: Quota Sampling is similar to proportional
stratified random sampling without the ‘random’ element. In
Dimensional Sampling at least one representative of every
combination of the various parameters is included in
sample.
 Snowball Sampling: a few people having criteria
are asked to identify further members.
 Convenience or opportunity
Sampling:members are selected if they meet certain
pratical criteria.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
6/45

7. How large should the sample
be?
 Rule of thumb: a range of between 1% to 10% of
the population, with a minimum of 100 participants.
 Statistical consideration: sample should have
a normal distribution.
 Sample composition: identify any distinct sub
groups that may behave differently in advance.
 Safety margin: leave a decent margin for
unforeseen or unplanned circumstances.
 Reverse approach: first approximate the
expected magnitude of results then determine the
sample size.
7/45
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

8. The problem of respondent self-selection
The actual composition of sample is not only the
function of some systematic selection but also
of factors related to respondents' own
willingness to participate.
The problem can arise, for example when:
 Researcher invite volunteers to take part in
study
 The design allows for high degree of dropout
 Participants are free to choose to be in study or
not
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
8/45

9. Questionnaire Survey
Main methodological issues:
a) How to sample the participants
b) How to design and administer the research
tool
Positive point of questionnaires:
 Easy to construct
 Extremely Versatile
 Capable of gathering a large amount of
information quickly in a form that is readily
processible
9/45
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

10. What are questionnaires? what do they
measure?
It is used in at least 2 broad senses
1. Interview schedules/guides
2. Self–administered pencil-and-paper
questionnaire
“any written instruments that present respondents
with a series of questions to which they are to react
either by writing out or selecting their answers”
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
10/45

11. What do questionnaires
measure?
Questionnaires can yield 3 types of data:
 Factual questions: finding certain facts about the
respondents such as demographic characteristic,
occupation, residential location, marital and socio-
economical status, educational level,etc.
 Behavioral question: finding out what the
respondents are doing, have done in the past, habits,
lifestyles, focusing on action and personal history.
 Attitudinal question: finding out what people
think, covering attitudes, opinions, beliefs, iterests and
values
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
11/45

12. The difference between test and questionnaire
 A test takes a sample of the respondent`s
behavior/knowledge for the purpose of
evaluating the individual`s more general
underlying competence/abilities/skills. It
measures how well some one can do
something.
 Questionnaire items do not have good/bad
answers. They elicit information in a non-
evaluative manner, without gauging their
performance against a set of criteria.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
12/45

13. Multi-item scales
 Multi-item scales: a cluster of several
differently worded items that focus on the
same target.
 Item wording in general has a substantial
impact on the responses.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
13/45

14. Closed-ended items
 There`s no production of free writing;
respondents choose one of the given
alternatives.
 Most professional questionnaires are
made up of ‘closed-ended’ items.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
14/45

15. Common closed-ended item formats
 Likert scale: consisting of a characteristic statement.
Respondents are asked to indicate the extent to which
they ’agree’ or ’disagree’.
 Semantic differential scales: by using it researchers
can avoid writing statements.
 Numerical rating scale: ’Giving so many marks out
of so many’
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
15/45

16. Other closed-ended item types
 True-false items: Problems: simplifying things too
much; resulting highly reduced and distorted
 Multiple-Choice items: When asking about personal
information, such as level of education
 Rank order: Ordering items by assigning a number to
them
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
16/45

17. Open-ended question
 Here questions are not followed by
response options.
 They permit greater freedom of
expression.
 We use the when we do not know the
range of possible answers.
 They can work well if they are not
completely open but contain guidance.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
17/45

18. Four question types in open-ended
questions:
 Specific open questions: asking about concrete
pies of information
 Clarification questions: can be attached to
answers with special importance or after the “Other” in a
MC item
 Sentence completion: completing an unfinished
sentence
 Short-answer questions: more than a phrase
less than a paragraph
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
18/45

19. Rules about item wording
 Aim for short and simple items
 Use simple and natural language
 Avoid ambiguous or loaded words and
sentences(non specific adj/adv, universals, modifying
words, words having more than one meaning,Loaded
words )
 Avoid negative constructions
 Avoid double-barrelled questions
 Avoid items that are likely to be answered the same
way by everybody
 Include both positively and negatively worded items
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
19/45

20. The format of the questionnaire
Main parts
 Title: for identifying the domain of investigation,
providing initial orientation, activating relative
background knowledge and content expectations
 General introduction: describing the purpose,
sponsoring/conducting organization, emphasizing that
there`s no right or wrong answer, promising
confidentiality, requesting honest answers, saying ‘thank
you’
 Specific instructions: explaining how respondents
should answer the questions
20/45
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

21. The format of the questionnaire
Main parts
 Questionnaire items: main body of
questionnaire with the use of different typefaces
and font styles.
 Additional information: including a contact
name with a phone number or address. we can
include a note promising to send a summary of
findings, inviting for follow-up interviews
 Final ‘thank you’
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
21/45

22. The format of the questionnaire
Length
 The length depends on how important the topic
of questionnaire is for the respondent.
 Dörneyi agrees on a four-page well-designed
questionnaire that takes half an hour to be
completed.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
22/45

23. The format of the questionnaire
Layout
 Booklet Format: not only does the questionnaire have
to be short but also it has to look short.
 Appropriate density: we need to achieve a
compromise on how much material need to be put.
 Sequence marking: marking each main section with
roman numbers, each question with Arab figures and
lettering subparts of questions.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
23/45

24. Item sequence
Four principles:
 Mixing up the scales(items of different scales)
 Opening questions: need to be easy, interesting
and focusing on important aspects
 Factual(‘personal’,’classification’)
Questions: leave personal questions at the end
(especially culture issues)
 Open-ended questions ate the end: So
other items won`t be affected by negative
consequences of this question.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
24/45

25. Developing and piloting questionnaire
Developing and piloting questionnaire is a
stepwise process:
Drawing up an item pool:
‘Item pool’: letting our imagination to go free and create as
many potential items for each scale as we can think of.
2 sources for drawing ideas in doing so:
a) Qualitative, exploratory data gathered in interviews or
student essays focusing on the content of the
questionnaire
b) Established/published questionnaires in the area
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
25/45

26. Developing and piloting questionnaire
Initial piloting of the item pool: for reducing the
large of questions in the item pool to the intended final
number it is useful to ask 3-4 trusted and helpful
colleagues or friends to go through the items.
Final piloting(dress rehearsal): there`s only
one way to find out whether respondents will reply in
intended manner and it is by administrating the
questionnaire to about 50 respondents similar to the
target population.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
26/45

27. Developing and piloting questionnaire
Item analysis: it involves checking three
aspects:
 Missing responses that were not understood correctly
 The range of responses elicited by each item
 The internal consistency of multi-item scales
Post hoc item analysis: conducting a final item
analysis after administration of the final questionnaire
screen out any items that have not worked properly
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
27/45

28. Administering the questionnaire
In social research by mail
In educational research by hand
In applied linguistics by group administration
People in general do not mind answering the
questions as long as they think that the
survey is related to a worthy cause and that
their opinion matters.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
28/45

29. Strategies for achieving the cooperation of
our informants:
 Advance notice: announcing a few days in
advance and explaining the purpose and nature of the
survey
 Attitudes conveyed by teachers, parents,
and other authority figures: participants are
quick to pick up superiors’ attitude towards the survey.
Win the support of all these authority figures
 Respectable sponsorship: if we represent an
organization that is esteemed highly by respondents,
the positive reputation is likely to be projected onto the
survey
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
29/45

30. Strategies for achieving the cooperation of
our informants:
 The behavior of the survey administrators:
1.business-like cloths 2.friendlyness 3.smile for
breaking the ice 4.professional overall conduct
 Administrator attitudes: their behavior should exhibit
keen involvement in the project and show an obvious
interest in the outcome.
 Communicating the purpose and significance of
the survey: cover the following points in introductory
speech1. greeting and thanking 2. the purpose of the
survey 3.the reason of the selection of these
participants 4.assurance of confidentiality 5.the duration
6.’any question?’ 7.’thank you’
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
30/45

31. Strengths of questionnaires
 Collecting a huge amount of information in
less than an hour
 Straightforward and fast data processing
 Useable with a variety of people in variety
of situations targeting variety of topics
 Offering anonymity if needed
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
31/45

32. weaknesses of questionnaires
 Easy to produce unreliable and invalid
data
 Need of simple and straightforward items
 Unsuitable for probing deeply into an
issue
 Usually resulting in superficial data
 Including possible respondent literacy
problems
 Including social desirability bias
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
32/45

33. Experimental study
 Experimental study can establish
unambiguous cause-effect relationships.
 First take a group of learners and do something
special to them while measuring their progress.
Then compare the data with another group that
is similar to first group but it did not receive
special treatment.
 First group is “treatment” or “experimental
group” and the Second one is “control group”
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
33/45

34. Quasi-experimental study
They are similar to experiments except
they do not use random assignment to
create the comparisons
Two ways of improving its design:
 Avoiding students self-selection to be in
treatment group
 Minimizing pre-test differences between the two
groups
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
34/45

35. Methods for minimizing pre-test differences
 Matching participants in the treatment and
control groups: first determine particular
individual difference variables then identify
participants in two comparison groups with
similar parameters
 Using analysis of covariance(ANCOVA):it
offers statistical method for adjusting the post-
test scores for any pre-test differences; we can
statistically screen the unwanted effects out of
the outcome measure
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
35/45

36. Analyzing the results
Two ways of analyzing data obtained with a
‘pre-test-post-test control group design’:
1. ANOVA: computing ‘gain scores’
separately(by subtracting the pre-test scores
from post-test) then comparing with T-test or
‘analysis of variance’
2. ANCOVA: comparing the post-test scores
by controlling from the pre-test scores
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
36/45

37. ANCOVA offers more precise results
because:
 Gain scores are not sufficiently reliable as
they are systematically related to any
random error of measurement
 ANCOVA helps to reduce the initial group
differences(especially in quasi-
experimental studies)
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
37/45

38. Experimental studies in educational and
applied linguistic research
American Educational Research Journal(AERJ)
reports “not only has educational intervention
research been decreasing in quantity but there
also has been a decline in quality”.
Reasons for losing popularity in experiments:
a) Many of topics are not directly related to ‘treatment’ or
’intervention’
b) Narrow scope(only one or a few variables can be
altered at a time)
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
38/45

39. Strengths and weaknesses of experimental
design
Strengths
 Best method of establishing cause-effect
relationship and evaluating educational
innovations
 ‘pre-test-post-test control group design’ can
control the threats to its validity
Weaknesses
 High price for implementation
 We may end up with artificial framework in
laboratory conditions which reduce the external
validity
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
39/45

40. Strengths and weaknesses of quasi-
experimental design
Strengths
 We do not have to worry about external
validity
Weaknesses
 ‘selection bias’: inequality of the initial
treatment and control group
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
40/45

41. Collecting quantitative data via the
Internet
A web based study offers some tempting
benefits:
 Reduced costs
 Convenience of administration
 Automatic coding(by using ‘CGI script’)
 High level of anonymity(high level of honesty)
 International access (+for cross-cultural research)
 Access to specialized population(small, scattered,
or specialized)
Limitation of this approach:
1.Technial issues 2.Sampling issues
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
41/45

42. Technical issues
 Internet users have different computers,
systems, browsers, and monitors, so the
actual stimulus received may differ from
what the investigator has intended. For this
reason we can employ a single HTML page
and improve the user interface.
 Participation is limited by technical issues
like connection speed and quality of installed
software but given the speed of progress
these restrictions are likely to be temporary.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
42/45

43. Sampling issues
 There's a lack of control over who will
eventually participate in the study
 The actual sample that completes the
web-based survey or experiment may be
much more heterogeneous than in
traditional research.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
43/45

44. Strategies offering a partial solution for
sampling issues
1. Analysis the question separately within each sub
stream of the sample. If the same conclusions are
reached in each subgroups, this might lend some
external validity to the result.
2. Comparison the web-based results with the outcomes
of a similar, non-web-based survey or experiment.
Convergence of the findings can help to validate the
results.
3. Sampling in traditional way and asking them to log on
and complete the survey online at home or in
computer lab.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
44/45

45. Thank you for paying
attention
45/45