This document discusses key considerations for data collection in action research projects, including sampling, personal bias, validity, reliability, triangulation, member checking, and ethics. It defines these terms and provides examples and guidelines for how to address each consideration to ensure high-quality data collection and analysis. Specifically, it emphasizes the importance of minimizing bias, establishing validity and reliability, using multiple data sources and analysis methods (triangulation), having participants review findings (member checking), and protecting participants' identities and well-being (ethics).

1. Lecturer: Yee Bee Choo
IPGKTHO
Topic 10

2. Data Collection Considerations
1. • Sampling
2. • Personal bias
3. • Validity
4. • Reliability
5. • Triangulation
6. • Member checking
7. • Ethics

3. 1. Sampling
• A population is composed of the entire group
of people that could possibly be included in
your study.
• A sample is a sub-group of individuals selected
from that population.
• It is unrealistic to expect an entire population
to participate in a study (unless the
population is extremely small) therefore
sampling is an accepted alternative.

4. 1. Sampling
Sampling in Action Research
• Action research problems almost always focus on
only a particular group of individuals (a teacher’s
class, some of a counselor’s clients, an
administrator’s faculty), and hence the sample
and population are identical.
• Random sampling is often difficult in schools, but
this is not as critical as it would be in more
traditional research endeavors, because
generalising is not necessarily likely nor desired.

5. 1. Sampling
• In conducting an action research project, your
sample is "defined" by your classroom.
• There is no sampling technique but you select
your participants from the whole class you are
teaching.
• Avoid using the term ‘subjects’, ‘respondents’
for action research.

6. 2. Personal Bias
• If we conduct our research in a systematic,
disciplined manner, we will go a long way toward
minimising personal bias in our findings.
• It is a challenge to remain ‘objectivity’ and open,
to look into the mirror of our findings and reflect
on what we see.
• We may choose to review only the literature that
supports a particular thesis that we wish to
promote.
(Mills, 2014)

7. 2. Personal Bias
How to avoid personal bias in research?
• Spend a considerable amount of time at the site.
• Get to know your participants and the environment (both
physical and cultural) in which they live.
• Collect copious amounts of data and check your perceptions
against what the data reveal.
• Collect data from a variety of perspectives, using a variety of
formats.
• Prepare extremely detailed field notes and attempt to reflect
on your own subjectivity as a part of these field notes.
• Often work in teams so that you can check your observations
against another’s.
(Fraenkel, Wallen & Hyun, 2012, p.448)

8. 3. Validity
• Validity is a test of whether the data we
collect accurately gauge what we are trying to
measure (Mills, 2014).
• It refers to the accuracy and trustworthiness
of the data.
• A study has validity if the researcher's
inferences and interpretations of the data are
accurate.

9. 3. Validity
3 main types of validity
1. Content-related evidence of validity
• It refers to the content and format of the instrument.
– How appropriate is the content?
– How comprehensive?
– Does it logically get at the intended variable?
– How adequately does the sample of items or questions
represent the content to be assessed?
– Is the format appropriate?
• The content and format must be consistent with the
definition of the variable and the sample of subjects to be
measured.
(Fraenkel, Wallen & Hyun, 2012)

10. 2. Validity
2. Criterion-related evidence of validity
• It refers to the relationship between scores
obtained using the instrument and scores
obtained using one or more other instruments
or measures (often called a criterion) .
– How strong is this relationship?
– How well do such scores estimate, present, or
predict future performance of a certain type?
(Fraenkel, Wallen & Hyun, 2012)

11. 3. Validity
3. Construct-related evidence of validity
• It refers to the nature of the psychological
construct or characteristic being measured by
the instrument.
– How well does a measure of the construct explain
differences in the behavior of individuals or their
performance on certain tasks?
(Fraenkel, Wallen & Hyun, 2012)

12. (Fraenkel, Wallen & Hyun, 2012)

13. 3. Validity
INTERNAL VALIDITY IN ACTION RESEARCH
• Action research studies are subject to all of the
threats to internal validity such as age, gender,
maturity, mortality, etc.
• The researcher must avoid bias by not
overlooking results or responses he or she does
not want to see.
• Implementation and attitudinal effects are also a
strong possibility that distort the results of a
study.
(Fraenkel, Wallen & Hyun, 2012)

14. 3. Validity
EXTERNAL VALIDITY IN ACTION RESEARCH
• Action research studies are weak when it comes
to external validity (generalisability).
• One cannot recommend using a practice found to
be effective in only one classroom!
• Thus, action research studies that show a
particular practice to be effective, that reveal
certain types of attitudes, or that encourage
particular kinds of changes need to be replicated
if their results are to be generalised to other
individuals, settings, and situations.
(Fraenkel, Wallen & Hyun, 2012

15. 4. Reliability
• Reliability is the degree to which a test
consistently measures whatever it measures
(Mills, 2014).
• Reliability means that scores from an
instrument are stable and consistent. Scores
should be nearly the same when the
researchers administer the instrument
multiple times at different times (Creswell,
2012).

16. Types of Reliability and Ways of Enhancing Reliability
Type of Reliability Ways of Enhancing Reliability
Rater Reliability •Use experienced, trained raters
•Use more than one rater
•Raters should carry out their assessment independently
Person-related
Reliability
•Assess on several occasions
•Assess when person is prepared and best able to perform
well
•Ensure that person understands what is expected (that is,
that instructions are clear)
Instrument-related
Reliability
•Use different methods of assessment
•Use optimal assessment conditions, free from extraneous
distractions
•Keep assessment conditions constant
4. Reliability

17. 4. Reliability
• Reliability is the degree to which a test
consistently measures whatever it measures
(Mills, 2014).
• To achieve reliability, consistency and stability
must be present in the set of test scores.
• Remember, a valid test is always reliable, but a
reliable test is not necessarily valid. In other
words, in a test is measuring what it is supposed
to be measuring, it will be reliable and do so
every time; but a reliable test can consistently
measure the wrong thing and be invalid! (Mills,
2014)

20. 5. Triangulation
Golafshani (2003):
• In qualitative research reliability and validity is
replaced by the idea of trustworthiness, which
is defensible and recommended triangulation
as a way to establish this trustworthiness of
the results of a study.

21. 5. Triangulation
• Triangulation is the process of corroborating
evidence from different individuals (e.g. a
principal and a student), types of data (e.g.
observational field notes and interviews), or
methods of data collection (e.g. documents and
interviews) in descriptions and themes in
qualitative research.
• This ensures that the study will be accurate
because the information draws on multiple
sources of information, individuals, or processess.
(Creswell, 2012)

22. 5. Triangulation
Three types of triangulation:
1. Triangulation of methods/ data sources
• Use a combination of data collection methods
to confirm and validate the outcomes of the
action plan.
• This can mean using both quantitative and
qualitative data.

23. 5. Triangulation
Triangulation of three data sources
Interview
Tests Reflective Journal

24. 5. Triangulation
2. Triangulation of data analysis
• The same set of data collection can be
analysed using a different type of data
analysis method.
• The data findings will be reliable if the data
analysis methods that were used pointed
towards the same results.

25. 5. Triangulation
3. Triangulation of researchers
By comparing the data gathered by different
researcher, it can help increase the validity and
reliability of the research findings by eliminating
researcher bias.

26. 5. Triangulation
• On the following slide is a sample
triangulation matrix for an action research
project of a teacher who wanted to see if
some new instructional software would
improve students keyboarding skills

27. 5. Triangulation
Research questions Data sources
Students Students Students Parents Records
1.pre-existing
knowledge
Student
survey
Computer
knowledge
Pre-test
2.Keyboarding speed Pre-test Post-test Teacher
help
3.Appropriate use of
WP
Pre-test
software
Post-test
software
Timed typing
4. Time on
computers
Student
survey
Teacher
constructed
test
Parent
survey
School lab
records

28. 5. Triangulation
Five types of triangulation by Goh (2012):
1. Data sources of triangulation
2. Investigator triangulation
3. Methodologic triangulation
4. Theoretical triangulation
5. Data analysis triangulation

29. 6. Member Checking
• Asking one or more participants in the study
to review the accuracy of the research report.
This is frequently referred to as member
checking.
• Checking one informant’s descriptions of
something (a way of doing things or a reason
for doing something) against another
informant’s descriptions of the same thing.
• Discrepancies in descriptions may mean the
data are invalid.
(Fraenkel, Wallen & Hyun, 2012)

30. 6. Member Checking
• Member checking involves the sharing of
interview transcripts, analytic thoughts (such
as observation notes with observer’s
comments), and drafts with the participants of
the study.
• The purpose of sharing these data sources is
to make sure that you have represented your
participants and their ideas accurately
(Mertler, 2009)

31. 7. Ethics
1. The identities of all who participate in a
qualitative study should always be protected.
(e.g. participants’ real names should not be
revealed, pseudonyms or number such as
Participant 1 or P1 can be used.)
2. Participants should be respected.
3. No physical or psychological harm will come
to anyone who participates in the study.
(Fraenkel, Wallen & Hyun, 2012, p.438)

32. 7. Ethics
1. You should send your research proposal to the Education Planning
and Research Division (EPRD), MOE and request for approval and
permission for conducting research in the school.
2. You should obtain a letter of permission from the State Education
Department or the head of school granting you permission to
carry out the research in the school.
3. Avoid carrying our research actions that might in any way
adversely affect or cause serious damage (physically and mentally)
to the students and learning process.
4. Avoid biasness.
5. Be mindful of ethical behaviour towards the participants.
6. Respect the confidentiality and privacy of information of the
participants.
7. Be honest in reporting data, results, methods and procedures.
8. Avoid discrimination against colleagues or students on the basis of
sex, race, ethnicity, or other factors.
(Goh, 2012, p.90)

33. 7. Ethics
Obtain Permissions
• In most educational studies, you need to obtain
permissions from several individuals and groups before
you can gather data.
• Permissions may be required from:
– Institutions or organisations (e.g., school district)
– Specific sites (e.g., the secondary school)
– A participant or group of participants
– Parents of participants (e.g., 10th graders’ parents)
– The campus on which you conduct the research (i.e.,
permission from your university or college institutional
review board)
• The best way to seek permission from the necessary
individuals or groups is to ask for it formally in a letter.
(Creswell, 2012)

34. 7. Ethics
(Creswell, 2012, p.149)

35. 7. Ethics

36. Tutorial 10
• Complete a triangulation matrix for your proposed study.
(Remember: Don’t force the issue of three data sources for each question. In
some cases you may have more than three data sources that you wish to use,
and in other cases, it may not be possible to include more than one data
source. For example, standardised test scores.)
Research
questions
Data sources
1 2 3
1.
2.
3.

37. References
• Creswell, J.W. (2012). Educational research: planning, conducting
and evaluating quantitative and qualitative research (4th ed.).
Boston, MA: Pearson Education Inc.
• Fraenkel, J.R., Wallen, N.E., & Hyun, H. H. (2012). How to design and
evaluate research in education (8th ed.). London: McGraw-Hill
Higher Education.
• Goh, L. H. (2012). A practical guide to writing your action research.
Puchong: Penerbitan Multimedia.
• Mertler, C.A. (2009). Action research: Teachers as researchers in the
classroom. (2nd ed.). California: SAGE.
• Mills, G. E. (2014). Action research: A guide for the teacher
researcher (5th ed.). London: Pearson Education Limited.
• Golafshani, N. (2003). Understanding reliability and validity in
qualitative research. The Qualitative Report, 8(4), 597-607.
Retrieved from http://www.nova.edu/ssss/QR8-4/golafshani.pdf