‘Research and Health Care Skills 1’
Directed Study Activities
Critical Evaluation Frameworks
The activities in this booklet are designed to assist your understanding of research
methods. These activities will guide your learning. You are strongly advised to
participate fully in these activities which will advance your appraisal skills and
prepare you for the in class test.
These activities will supplement the key note sessions & tutor guided sessions
provided in the timetable. Other material can be found on sharepoint.
With thanks to the NUR244 development team.
ACTIVITY 1: Randomised Controlled Trials [RCTs]
‘The gold standard for demonstrating in a rigorously scientific manner that a
treatment or intervention is effective … the essential tool for a quantitative
assessment of the efficacy of an intervention’ (Donnan, cited in Cormack
Dominates medical thinking on research methods, and what constitutes
Involves conducting an experiment in which investigators don’t know the outcome
(verify theory/hunch – deductive method). In essence, RCTs are a simple idea –
compare the treatment group (taking the drug) with the placebo or control (not taking
the drug) to see by how much the treatment group’s blood pressure decreases
compared to the placebo groups BP (for example), when taking a new hypotensive.
Control: both groups must be similar in characteristics, so that if there is a treatment
effect after the intervention, it must only be the intervention that caused it (e.g. not
age, sex, class, race income, or any other variable).
Essential features of an experimental design
1. comparison and control groups
2. eligibility of subjects – clear protocols with exclusion and inclusion criteria
3. informed consent
4. randomisation to intervention and control groups
5. blinding of treatment and assessment: It should be impossible for
patients/staff/assessors to know who is receiving potential treatment and who
is not to avoid bias
6. analysis - methods and statistical tests
7. sample size, decided at planning stage. In general, statistical tests are more
reliable and powerful with larger numbers (thousands rather than tens).
Problems with RCTs
2. loss to follow-up – experimental mortality
3. statistically and/or clinically significant results can be difficult to interpret and
rely on statisticians to interpret results for us
4. tell us nothing about patient’s experience…
Identify and retrieve the following report of a Randomised Control Trial.
Feuchtinger, J. de Bie, R. Dassen,T. Halfens, R. (2006) A 4cm thermoactive
viscoelastic foam pad on the operating table to prevent pressure ulcer during cardiac
surgery. Journal of Clinical Nursing 15. 162 – 167 http://www.blackwell-
Examine the article using the tool attached. Think about its strengths and
1. Read the research paper up until the results section and consider what was
done & how it was done.
What was the aim of the study?
What were the objectives of the study? Was an hypothesis stated?
What was the precise design of the study? (e.g. experimental/quasi
How was the sample selected?
What was the sample size? Was a control group used? How were the
participants allocated to different groups? e.g. were experimental & control groups
What was the independent variable? How was this introduced?
Was there a placebo? What was provided to the control group?
What dependent variables (outcomes) were measured?
Describe how the data were collected and what measurement tools were used
What attempts were made to control for confounding variables in order to maintain
internal validity & reduce bias?
2. Read the results section. Do the results meet the objectives of the study
and are they statistically significant?
What statistical tests were used?
What were the results of the study? Were the results statistically significant?
How clearly were the results presented
What were the strengths & limitations of the study?
What inferences can be
drawn from this study?
ACTIVITY 2 Surveys
The survey technique involves the collection of descriptive data from a large
number of respondents. It commonly uses questionnaire and/or interview
methods of data collection.
Face to face
It is used to gain information about:
Attitudes and Values
Characteristics of groups/people
There are a number of different types of surveys:
Evaluative - consider the effectiveness of something retrospectively
Comparative - results from 2 different groups are compared
Field survey - is conducted in workplace/real world as it is happening
Longitudinal - follow a group over a long time period
Cross sectional - compare groups with same characteristics but with
varying length of exposure to thing under study
Advantages of surveys:
Can provide data that can form the basis of decision making about a larger
population than can be included in a single study
Generalisable (if random sample)
Accurate and objective
Provides data that can be compared/contrasted for particular groups.
Enables exploration of attitudes etc for particular groups.
Disadvantages of surveys:
Inappropriate generalisations because of sampling errors or inadequate
Self report nature - may be inaccurate responses.
Can be superficial and do not allow for intensive analysis
Difficult to show causal relationships (correlation does not mean
May require multiple workers - can be expensive.
Identify and retrieve the following report: Ylinen, E. Vehviläinen-Julkunen,K. Pietilä,A.
(2007) Nurses’ knowledge and skills in colonoscopy patients’ pain management.
Journal of Clinical Nursing 16, 1125 – 1133 http://www.blackwell-
Read the paper and answer the following questions
1. What is the research question and why is it important?
2. What was the precise design used in the study – was this the most appropriate
design to obtain the needed data?
3. Who comprised the sample and how was this selected?
4. What is the sample size and is this sufficient?
5. Is the response rate adequate?
6. What are you told about the questionnaire?
7. Are validity and reliability of the questionnaire described?
8. How was the data collected?
9. How was the data analysed?
10. What are the results and how are they presented?
11. Do the conclusions fit the findings of the analysis?
12. Are any limitations acknowledged and used to quantify statements?
13. Are recommendations for practice made from the results/discussion?
Activity 3: Examination of qualitative study
Qualitative research focuses on the every day life of people in their social context
and gains an understanding of their reality in relation to a particular experience.
Qualitative researchers endeavour ‘to document the world from the point of view of
the people studied’ (Hammersley 1992:165).
Researchers collect and analyse non-numerical data through the use of open
questioning and observations, representing their experiences and meanings to other
The researcher is the data collection instrument; analysis is performed with words
which are examined, organised, compared and categorised. Different interpretations
of the data may be possible, but findings are put forward for theoretical reasons or
on grounds of internal consistency.
In qualitative research the onus is on the researcher to make explicit the processes
of the study, and to demonstrate awareness of the relationship between the
researcher, conditions and participants of the study.
The question of reliability in quantitative research - consistency in collecting and
categorising data, and of validity - whether the research processes are ‘capturing’
social phenomena accurately - are addressed in qualitative research within the
concepts of rigour. Guba and Lincoln (1989) suggest three criteria by which the
trustworthiness (rigour) of qualitative inquiry may be assessed: credibility,
transferability and dependability. Credibility is established when researchers make
explicit their experience as researchers. The term transferability is used to decide
upon the ‘fittingness’ of the data to similar contexts and is therefore similar to
external validity found in quantitative research. Dependability of qualitative findings
concerns consistency in data collection and analysis, procedures recognised in the
‘reliability’ of quantitative research methods.
‘Dependable data’ are achieved through auditability, the condition in which another
researcher can clearly follow the analysis pattern used by the researcher in the
study. A ‘decision trail’ is provided detailing decisions about the methodological
and analytical choices throughout the study.
In qualitative research the researcher will carry certain value positions into the
research process, and readers (as interpreters) will also approach qualitative data
with their own preconceptions. It should be possible for the reader to judge the
credibility of the researcher’s account. Qualitative researchers should seek to
provide an account of method and data so that another researcher could analyse the
same data in the same way and come to essentially the same conclusions, and
produce a justifiable explanation of the phenomena under study.
Identify and retrieve the following qualitative research study:
Lohne,V. Severinsson, E. (2006) The power of hope: patients’ experience of hope a
year after acute spinal cord injury. Journal of Clinical Nursing 15. 315 - 323
Now consider the following questions:
What were the aims of the study?
Describe the research approach used
Was the research design clearly described?
How were the participants selected?
How many participants were studied?
What method of data collection was used?
How were the data analysed?
What were the main findings of the study?
Activity 4: Data collection methods
The purpose of this activity is to help you to develop your knowledge of advantages
and disadvantages of different methods of data collection in social research. .
Find two articles with different data collection methods (e.g. a questionnaire or
rating scale, observation or interview). You could approach this exercise by thinking
about an area of practice in which you are interested and trying to find studies with
different data collection methods by using key words from the electronic databases.
Note the full reference and, if possible, print the abstract.
Prepare a short summary and critique of the data collection methods used in
each article (you may need to refer to research textbooks to find out more about the
data collection methods). Think about the following:
How were the data collected in each study?
Was this consistent with the research questions being asked?
How do the methods of data collection influence the findings?
What are the strengths and weaknesses of the data collection methods in
relation to issues – reliability and validity?
Activity 5: Sampling issues in quantitative and qualitative research
The purpose of this activity is to help you understand the sampling issues that may
arise in studies of different research approaches.
Before you attempt the activity it is recommended that you read carefully the paper
Thomson, C. (1999) If you could just provide me with a sample; examining
sampling in qualitative and quantitative research papers. Evidence-
Based Nursing, 2, 3, 68-70.http://ebn.bmj.com/cgi/content/short/2/3/68
Select two original papers of different research designs (at least one must be a
quantitative study and at least one a qualitative study). You may use studies from
journals in library (e.g. Journal of Advanced Nursing, Journal of Clinical Nursing
etc.) or from an on-line database. You will need to take a copy of the articles
Identify the methods of sampling employed in the studies and consider the
Is the method of sampling appropriate?
Provide a definition of the method of sampling used
Is the size of sample adequate given the design of the study?
Consider whether there is likely to be sampling bias as a result of the choice of
sampling method. In order to assess this consider the following questions:
Who was included in the sample?
What was the source of recruitment for subjects (e.g. Hospital waiting list, Health
How were the subjects recruited? (e.g. convenience, random)
What were the inclusion and exclusion criteria for the sample? (e.g. demographic
characteristics, stage of disease, other medical conditions etc.)
How might the method of sampling affect generalisability of findings (for
quantitative research) or transferability/fittingness of findings (for qualitative
Once you have mastered the basics of research methodology, you can begin to
analyse the research critically. This will be examined in depth in Research and
Health Care Skills 2, but the following guidelines to critiquing will help to get you
started. Take one of your research reports and ask yourself the following questions
Is there a clear statement about the topic being investigated? Is there a clear
rationale for the research? Is there a clear statement about the limitations of the
research? (State what these were)
Do the researchers use contemporary material to put the topic being investigated in
Do the researchers link their work to a wider body of knowledge by citing key
Do the researchers link the topic to questions about theory?
Is the research question clear & unambiguous?
Are the terms used in the research question defined? And are they clearly
Is the research design clearly described?
Is the research design appropriate for the topic being investigated?
Are the advantages & disadvantages for the research design acknowledged by the
Is there a clear statement about who participated in the research?
Is there a clear statement about how participants (sample) were selected?
Is the selection of participants appropriate to the research design?
Is there a clear statement about the number of people participating in the research?
Is there a clear description about how the data were collected?
If it is a quantitative design, is there a clear statement describing the validity &
reliability of the measures used?
If it is a qualitative design, is there a clear statement about the role of the
Is there a clear statement about how the data were analysed?
Is the data analysis appropriate for the kind of data collected?
If the research is quantitative, were the statistical tests used appropriate for the kind
of data collected?
Is there a clear statement about ethical committee approval?
Is there a clear description about gaining informed consent, maintaining anonymity
Is there enough detail for the reader to be confident in the findings?
If the research is qualitative, is the voice of the participants present through quotes?
Is the presentation of the results clear and unambiguous?
Are the results related to the outcome of the literature review?
Are any weaknesses in the research design acknowledged and used to qualify
Are the clinical implications of the work acknowledged and discussed?
Are the implications of the research acknowledged and discussed?
Are further areas for research identified?
Are recommendations for practice made from the results/discussion?
Now that you have analysed the report in detail:
Think about what the authors are saying and what you have read about the topic
before in other books or research reports.
Think about your experiences of health care in relation to this topic.
This time you will be asking questions about the quality of the work.
And you will be looking at the relationships between argument, evidence &
Ask yourself the following questions as you read. You may need to go back and
forth between these questions and the text.
1. Can you trace how the researchers came to their conclusions from the argument
they make in the introduction/literature review and the evidence they present?
2. Does their argument, evidence and conclusion follow on from one to the other?
3. The researchers are implicitly claiming that theirs is the best interpretation of the
data they have recorded or measured. Can you see this from the methods/results
4. If their interpretation is not the best, can you suggest alternative reasons or
theories to explain the results or conclusions?
5. Is there enough information in the report to tell us whether the researchers were
aware of the effect of the research procedures on their results?
6. If the conclusions seem pertinent for the case or population sample being studied,
to what extent do the results relate to other groups or wide populations?
7. In whose interest was the research carried out? Do the participants gain any
benefit from being involved in the research?
8. Are the researchers presenting assumptions as unproblematic facts? Is the
research design based on assumptions which predetermine the results?
You have now actively read a piece of research, extracted the information you need
to prepare a critique and started to make an analysis of the worthiness of the report
to you and your practice. As you read more research reports, this process will take
less time and you will develop your own ways of recording your notes and thoughts.
Experiment with this format so that it suits your needs, but always remember the task
in hand is actively read, think and analyse.
Summary for Critique of Research Report
Author/s & date:
Journal Title: Volume: Issue number: Pages:
Aims of study
Methods of data collection
Discussion/implications for practice:
Attree, M. (2001) Patients’ and relatives’ experiences of “good” and “not so good”
quality care. Journal of Advanced Nursing 33, 4, 456-466.
Behi, R. Nolan, M. (1996) The basic experimental design. British Journal of
Nursing 5, 9, 563-566. [see also other articles on experimental research in this
Cormack, D. (2000) The Research Process in Nursing [4th ed.] Blackwell Science.
Guba, E. G. Lincoln, Y. S. (1989) Fourth Generation Evaluation. California. Sage
Hutchinson, A. Johnston, L. (2004) Bridging the divide: a survey of nurse’s opinions
regarding barriers to, and facilitators of, research utilisation in the practice setting
Journal of Clinical Nursing 13 (3) 304 – 315
Mays, N. & Pope, C. (1995) Rigour and qualitative research. BMJ, 311, 109-112.
Nelson, E. A. (1999) Randomised controlled trials: questions for valid evidence.
Nursing Times Learning Curve 3, 5, 6-8.
Pope, C. Mays, N. (1995) Qualitative research: reaching the parts other methods
cannot reach. BMJ 311: 42-45
Thomson, C. (1999) Appraising qualitative research. Nursing Times Learning
Curve. 3, 9, 7-9
Thomson, C. (1999) If you could just provide me with a sample; examining sampling
in qualitative and quantitative research papers. Evidence-Based Nursing, 2, 3, 68-