This document is a thesis presented by Sarah Kenehan to fulfill requirements for a BA Honours degree in psychology from Maynooth University. The thesis explores the effect of rule order manipulation on implicit self-esteem as measured by the Implicit Relational Assessment Procedure (IRAP). Forty-one undergraduate students participated in the study and were split into two groups where the order of rules presented in the IRAP was varied. Preliminary results showed some differences between groups, especially for self-positive and other-positive trial types, but these differences were not statistically significant. The findings suggest that implicit responding on the IRAP may be influenced by rule order effects.

1. Department of Psychology
Maynooth University, Maynooth
Final Year Research Project
Exploring the effect of rule order manipulation on implicit self-esteem in the Implicit
Relational Assessment Procedure (IRAP)
Thesis Presented in Part Fulfilment of the Requirements for the BA Honours Degree in
Psychology, Maynooth University, Maynooth
Submitted by: Sarah Kenehan
Supervisor: Dr. Yvonne Barnes-Holmes
Date February 2015.

2. i
Table of Contents
Table of Contents........................................................................................................................i
Acknowledgements....................................................................................................................ii
Abstract.....................................................................................................................................iii
Introduction................................................................................................................................1
Method.......................................................................................................................................7
Participants.....................................................................................................................7
Setting.............................................................................................................................7
Apparatus and Materials.................................................................................................7
Measurement..................................................................................................................7
Procedure......................................................................................................................12
Ethical Considerations..................................................................................................13
Results......................................................................................................................................14
Discussion................................................................................................................................17
References................................................................................................................................23
Appendix A: Consent Form
Appendix B: Rosenberg Self-Esteem Scale (RSES)
Appendix C: Depression Anxiety Stress Scales (DASS)

3. ii
Acknowledgements
I would like to take this opportunity to thank my supervisor Dr. Yvonne Barnes-Holmes for
her constant support, encouragement and guidance. Working with you has been a pleasure
and a valuable learning experience.
A special thanks to Deirdre Kennedy who kept me grounded, especially when things were not
going according to plan.
I wish to express my gratitude to all my participants who took time out of their busy
schedules to complete my experiment, sometimes multiple times, without your participation
this research would not have been possible.
I would also like to thank the Psychology Department of Maynooth University, who have
helped me greatly throughout my three years here.
Finally, I would like to thank my family and friends for their unconditional support, love, and
encouragement throughout my degree.

4. iii
Abstract
Self-esteem is an important and well-established concept particularly areas of
psychopathology such as depression. In the past explicit measures were used to assess self-
esteem however these measures are susceptible to different kinds of bias. In order to
overcome this implicit it has become increasingly more common to use implicit measures
such as the IAT (Implicit Association Test) and IRAP (Implicit Relational Assessment
Procedure). As with other measurement methodologies, it is essential to assess its validity
and to eliminate any variables that will confound the data. This experiment aims to evaluate
the impact of rule order manipulation on the IRAP effect during a self-esteem IRAP. The
experiment was conducted with 41 participants who were split into two group. In one group
the “self” aspect of the rule was presented first, whilst in the other group “others” was always
at the beginning of the rule. Results showed that there was a difference between the groups,
especially for the self positive and others positive trial types, however these differences did
not reach statistical significance. The findings suggest that implicit responding may be
influenced by rule order effects.

5. 1
Reporting high levels of self-esteem is thought to be of great importance to the mental
health and the well being of an individual (Dunning, Meyerowitz, & Holzberg, 1989). This
has been found as a result of extensive research that finds that people who suffer from
depression often report thinking negatively about the self (see Hagga, Dyck, & Ernst, 1991
for a review of the literature). This recurrent finding that depression is linked to negative self-
evaluations and low self-esteem stimulated the cognitive behavioural theory of depression.
This theory stated that it is these cognitively dysfunctional negative schemata that maintain
and increase vulnerability to depressive episodes (Williams, 1997). This theory has been
supported by the use of various questionnaires that demonstrate the tendency of clinically
depressed patients to report lower self-esteem than controls (Ingram, Miranda & Segal,
1998). In recent years however, the use of questionnaires to evaluate self-esteem has come
under increasing criticism (Young, 1994). Explicitly asking individuals about their self-
esteem can be gravely affected by multiple forms of bias and external factors such as demand
characteristics, social desirability, and self-representation (Ingram & Wisnicki, 1991). As
well as this, according to the cognitive behavioural theory, some schemata relating to the self
may not be consciously accessible to an individual so it is therefore impossible for the
individual to explicitly report these cognitions (Greenwald & Banaji, 1995). These flaws in
explicitly measured self-esteem have acted as a catalyst, triggering research in the
measurement of implicit attitudes (Greenwald, McGhee & Schwartz, 1998).

6. 2
The development of the Implicit Association Test (IAT) and other implicit tests have
brought about a new way to measure self-esteem and have begun to contradict already
established theories of self-esteem (Greenwald et al., 2002). These new paradigms have
improved the way in which underlying schema-processes are evaluated which is reflected in
their predictive power (Gremar , Segal, Sagrate, & Kennedy, 2001). The IAT measures
implicit attitudes by assessing the strength of associations between two concepts. It assumes
that if concepts are implicitly closely related and share the same response key that
participant’s response times will be quicker than those stimuli with weaker associations
(Pinter, & Greenwald, 2005). DeRaedt, Schacht, Franck and DeHouwer, (2006) were one of
the first researchers to use an implicit measure to assess self-esteem. They asked participants
to associate positive words (such as capable, competent and good) and negative words ( for
example inferior, failed and worthless) to a “self” and “not-self” category. They found that
both depressed and non-depressed participants displayed positive implicit self-esteem
(DeRaedt et al., 2006). These results contradicted the popular theory that depressed
individuals had lower self-esteem and stimulated the creation of other implicit measures.
One of the limitations of the IAT is that it only measures associations between
stimuli. The direction and relationship between the stimuli cannot be assessed, overcoming
this limitation was one of the main aims a newly developed implicit measure known as the
Implicit Relational Assessment Procedure (IRAP; Barnes-Holmes, Barnes-Holmes, Stewart
& Boles, 2010). The IRAP is based upon a recent account of human language known as
Relational Frame Theory (RFT). RFT uses the concept of stimulus equivalence but
hypothesises that the relations that arise as a result of stimulus equivalence are subsequently
framed and functionally categorised as part of a generalised operant response (Hayes, Barnes-
Holmes, & Roche, 2001). Stimulus equivalence occurs through the process of creating
relations about an object through a discrimination in language (Hayes, Barnes-Holmes &

7. 3
Smeets, 2000). These relations may be non-arbitrary based and on the physical properties of
the object, or arbitrary and formed by abstract factors which have been related in previous
interactions with the object (Torneke, Luciano & Salas, 2008). Non-arbitrary relations have
been found in the animal kingdom, but the arbitrary responding found in humans has yet to
be found in other species. It is hypothesised that the ability to create arbitrary relations is at
the core of complex human language (Barnes-Holmes et al., 2010).
The IRAP measures peoples’ implicit attitudes by asking them to respond to a sample
stimulus and a target item in line with a particular belief which is provided by the researcher
(Barnes-Holmes, Hayden & Stewart, 2008). Similar to the IAT, the IRAP assumes that the
response times will be quicker on trials where the rules being followed are consistent with the
implicit beliefs of the participant. The differences seen in the response times of participants
during consistent and inconsistent trials is hypothesised to be due to the participants having to
respond against their more probable implicit responses during an overt responding task
(O’Hora, Barnes-Holmes, Roche & Smeets, 2004). The first IRAP that was used to assess
self-esteem compared two groups that were known to have different levels of self-esteem,
prisoners and Irish students (Vahey, Barnes-Holmes, Barnes-Holmes, & Stewart, 2009).
Students have reliably been reported to have higher self-esteem than prisoners (Gullone,
Jones & Cummins, 2000). Vahey and colleagues (2009) found that students demonstrated did
in fact display higher self-esteem than prisoners which began to validate the IRAP as a
measure of self-esteem. Providing evidence of concurrent validity is of extreme importance
to any type of new measure, but this is particularly true of implicit measures as the dependent
variable is more abstract (Golijani-Moghaddam, Hart, & Dawson, 2013).
The IAT is the oldest and most validated measure of implicit self-esteem and as
discussed earlier when implicit self-esteem was measured using this method it was found that
there was no difference between depressed patients and non-depressed controls (DeRaedt et

8. 4
al., 2006). This differed from the explicit measures but this was difference was hypothesised
to be caused by bias and other confounding variables surrounding the explicit measure.
However, researchers began to question the validity of the IAT of self-esteem. They
postulated that as the IAT only assessed associations it was possible that it was measuring
one of two different variables in the two participant groups, either ideal or actual self-esteem
(Zentner & Renaud, 2007). In order to test this hypothesis the IRAP was used because of its
ability to capture propositional behaviour rather than simply measuring associations. Reume
and colleagues (2013) conducted an experiment whereby participants, who were either rated
as highly dysphoric (participants who scored highly on the Beck Depression Inventory) or
normal, completed two separate IRAPs; one where stimuli were related to the actual self ("I
am") or the ideal self ("I would like to be"). The results of the study showed that the
dysphoric group had a higher ideal self-esteem and a lower actual self-esteem when
compared to the control group (Reume et al., 2013). This research highlights the importance
of validating procedures and re-evaluating findings of previous research to provide the most
accurate data and interpretations possible. This is of considerable importance due to the
emphasis that has been put on self-esteem in relation to theories and treatment of people
suffering from depression (Roberts, Gotlib, & Kassel, 1996).
This process of constantly evaluating techniques used to measure psychological
concepts is of extreme importance and is reflected in the amount of research being conducted
in identifying and analysing variables that may influence IRAP data (see Golijani-
Moghaddam, Hart, & Dawson, 2013 for a review of the literature). One that is of particular
interest is the malleability of the IRAP effect (Hughes, Barnes-Holmes & Vahey, 2012).
Implicit measures are extremely influenced by context and it is important that effects like this
are controlled for during future research (Ferguson & Bargh, 2007). Hooper, Villatte,
Neofotistou & McHugh (2010) conducted an experiment in which participants were exposed

9. 5
to an IRAP which assessed their level of avoidance. Upon completing the first IRAP,
participants were shown a disturbing image of a dead body so that negative thoughts were
induced. Subsequently the participants were exposed to a 10 minute audio file which
promoted either the use of mindfulness or thought suppression to control negative thoughts.
The participants then completed the original IRAP again and it was found that the individuals
in the thought suppression group experienced an increase in implicit experiential avoidance
compared to the mindfulness group (Hooper et al., 2010). The introduction of an intervention
targeting the implicit attitude under investigation produced significant differences in the
results of the experiment clearly demonstrating the malleability of implicit attitudes. This
finding has been replicated in other clinically relevant populations such as people suffering
from depression (Hussey & Barnes-Holmes, 2012). The malleability of implicit attitudes has
also been found whereby once a context has been established implicit racist and ageist
attitudes have been altered (Fazio, Jackson, Dunton, & Williams, 1995; Cullen, Barnes-
Holmes & Barnes-Holmes, 2009).
Implicit attitudes are not only susceptible to be change through direct measures, such
as mood induction or a contextual intervention, they are also heavily influenced by more
technical and indirect variables such as screen presentation (Parris, Sharma, & Weekes,
2007). As participants during an IRAP task are under strict time pressure (in order to ensure
that results are due to implicit responding), what is presented on screen is of optimal
importance to allow participants to respond accurately and achieve mastery criteria (Deroost
& Saetens, 2006). In order to investigate the effects that screen presentation had on the IRAP
effect, four different groups of participants were designated to groups where screen
presentations were slightly varied (Campbell, Barnes –Holmes, Barnes-Holmes, & Stewart,
2011). They alternated sample stimuli and response options from being in a fixed position
throughout the trial to switching randomly on screen. They found that the randomisation of

10. 6
the sample stimuli (in the presence or absence of randomisation of response options)
produced the strongest IRAP effect. These results once again demonstrate the robustness of
IRAP responses and shows the lack of control the participant has over their responses
(McKenna, Barnes-Holmes, Barnes-Holmes, & Stewart, 2007). This process of investigating
the impact of multiple variables on the IRAP is of extreme importance to improve the
procedure and to validate the IRAP as a measure of implicit attitudes (Pothos, 2005).
Previous technical research surrounding the IRAP has led to notable changes in its
procedure, such as the decrease in the minimum response time from 300ms to 200ms
(Barnes-Holmes, Murphy, Barnes-Holmes, & Stewart, 2010). Alongside the changes in
screen presentation and minimum response latencies, it has been found that asking
participants to respond according to a rule before each block can help to reduce attrition rates
(Barnes-Holmes et al., 2010). During the IRAP trials, the order in which participants are
exposed to the rules (consistent or inconsistent first) is typically reversed, however rule order
is not typically manipulated. This research seeks to fill this gap in the research. To achieve
this aim participants will be split into two groups where both order presentation (inconsistent
versus consistent) and the rule order (self-first versus others-first) will be changed between
groups. One group will be exposed to the inconsistent trials first and with the rules reversed
so that the rule is “Others are good and I am bad”, while on the other IRAP the first rule will
be consistent with their current beliefs and will begin with the rule “I am good and others are
bad”. Our hypothesis predicts that there will be a significant difference between the groups
with the reversal of the anti-other effect in the others-first group.

11. 7
Method
Participants
This experiment consisted of 41 undergraduate students attending Maynooth
University (21 females and 20 males) aged between 18 and 26 (M=19.61, SD=2.02). The
participants were selected using random convenience sampling. The groups did not differ
significantly with regard to age. All participants were briefed on the nature of the experiment
and gave their written consent before taking part in any aspect of the research.
Setting
The current study took place in the experimental lab in the Department of Psychology
at Maynooth University. Each participant completed the experiment individually in a separate
cubicle containing no distractions. The experimenter was only present during the instructional
phase and practice blocks of the IRAP and vacated the room while the participants were
completing the test blocks and questionnaire measures.
Apparatus and Materials
All aspects of the experiment (including questionnaires) were completed on a DELL
desktop computer with a Pentium 4 processor with a 15” screen and a standard QWERTY
keyboard. The IRAP procedure was delivered via a programme called PsychoPy 1.81.03

12. 8
which controlled all aspects of stimulus presentation and the recording of participant
responses. For the purposes of statistical analysis a combination of SPSS 2.0, Microsoft Excel
2007 and StatView were used.
Measurement
Depression Anxiety Stress Scales. The Depression Anxiety Stress Scales 21 (DASS
21) consists of three different self-report scales which are used to evaluate the three negative
emotional states of depression, anxiety and stress. The scale consists of three main sections
pertaining to each of the emotions separately, with each of these sections containing 7 items.
Subjects are asked to indicate the severity/frequency to which they have experienced the
negative symptoms of these emotions over the past week. To calculate the scores for each of
the three emotional states the researcher simply adds the scores for the relevant items. The
DASS has been shown to have high internal consistency in a variety of settings and with
numerous populations and has been repeatedly validated and shown to be reliable (Brown,
Chorpita, Korotitsch & Barlowm 1997).
Rosenberg Self-Esteem Scale. The Rosenberg Self-Esteem Scale (RSES) scale is a
10 item questionnaire used to measure self-esteem. Participants are asked to indicate on a
four point scale the extent to which they agree or disagree with a statement. An individual’s
score is calculated by totalling the score of each item after the negatively worded items have
been reversed. The RSES has been shown to be extremely reliable with a test-retest reliability
correlation of 0.85. This scale has also shown strong concurrent, construct and predicative
validity suggesting that it is a useful explicit measure of self-esteem (Robins, Hendin, &
Trzesniewski, 2001).
IRAP Self-esteemmeasures. Each IRAP trial presented one of two sample sentence
stimuli (“I AM” and “OTHER PEOPLE ARE”) which appeared on the top of the computer

13. 9
screen. This was accompanied with one of the 6 positive or 6 negative target stimuli(see
Table 1 for stimulus arrangements). At the bottom of the screen were the two response
options, TRUE and FALSE. To select TRUE the participant pressed the “d” key and to select
FALSE the participant pressed the “k” key.
Table 1. The response combinations deemed consistent in the self-esteem IRAP
Sample 1 Positive
Targets
Sample 2 Negative
Targets
Sample 1 Negative
Targets
Sample 2 Positive
Targets
Loyal Manipulative Manipulative Loyal
Trustworthy Dishonest Dishonest Trustworthy
I am Kind Others are Cruel I am Cruel Others are Kind
Moral Horrible Horrible Moral
Generous Selfish Selfish Generous
Friendly Heartless Heartless Friendly
Response Option 1 Response Option 2
True False
During the IRAP participants would see one of the sample stimuli (“I AM” or
“OTHERS ARE”) together with a target stimulus such as “LOYAL”. The participant would
then be required to respond either “TRUE” or “FALSE” based on the rule they were given at
the start of the block of trials. During the consistent trials participants will be instructed to
respond as if “I am good and others are bad” or “Others are bad and I am good”, depending
on the group they have been randomly assigned to. In order to respond correctly (i.e.,
according to the rule) participants must select the “TRUE” response whenever the sample
stimulus “I AM” is paired with a positive target and select the “FALSE” response whenever
the sample stimulus “I AM” is paired with a negative target (See figure 1 for trial types).The
converse is deemed correct when the sample stimulus “OTHERS ARE” is paired with a
positive or negative target stimulus. During the inconsistent trials, responses that expressed

14. 10
positivity towards the self were deemed to be “FALSE”, whilst target stimuli paired with
positivity towards others should emit the “TRUE” response. The main assumption behind the
IRAP is that participants perform better on trials that are consistent with how they typically
relate the relevant stimuli. For example, if an individual generally responded quicker to “I
AM” and “FRIENDLY” with “TRUE” rather than “FALSE” they would be said to have a
self-is-positive implicit bias.
.
Figure 1: Examples of the four trial types employed in the self-esteem IRAP.
Building on previous IRAP research findings and recommendations (see Drake et al.,
2010 for a review) a maximum of four practice blocks had to be completed by the participant
before they progressed to the test blocks. Once the participant reached the mastery criteria
(80% accuracy and 2000ms response latency) they progressed onto the six tests blocks. In
I Am
Loyal
Select “d” Select “f”
For True For False
Consistent Inconsistent
Others Are
Manipulative
Select “d” Select “f”
For True For False
Consistent
I Am
Manipulative
Select “d” Select “f”
For True For False
InconsistentConsistent
Others Are
Loyal
Select “d” Select “f”
For True For False
Inconsistent
Consistent Inconsistent
Inconsistent Consistent

15. 11
line with previous research (Campbell, Barnes-Holmes, Barnes-Holmes & Stewart, 2011)
surrounding screen presentation IRAP effects, response options were fixed so participants
always pressed “d” for true and “f” for false.
The format of all blocks during the IRAP trials (both practice and test) were identical,
with all stimuli appearing simultaneously. If the participant responded correctly (i.e., in line
with the rule presented at the start of the trial), the response was followed by a blank screen
for 400ms which was followed by the presentation of the next pairing task. If the response
was not in accordance with the current rule a red “X” would appear on the screen and the
participant could not move onto the next trial until the correct response had been recorded.
During the trials latency feedback was also presented to the individual. If the participant went
beyond their allocated 2,000ms response latency a red exclamation mark would appear at the
bottom of the screen, informing them that they were required to respond quicker. Upon
completing the 24 IRAP trials the participant was presented with two types of feedback for
the previous block: the percentage of correct responses and the median response latency.
Once the participant pressed the spacebar to continue onto the next test block a screen
appeared displaying the rule for the next set of blocks.
The IRAP alternated between consistent and inconsistent blocks, meaning that
participants were required to switch their response patterns after each block. As this study
aimed to investigate rule order manipulation during the IRAP, participants were randomly
assigned to one of two IRAPs with rules presented in the following way:
IRAP 1
1. “I am good and others are bad”
2. “I am bad and others are good”
IRAP 2

16. 12
1. “Others are good and I am bad”
2. “Others are bad and I am good”
In this way half of the participants were exposed to consistent trial first and the other half
began with inconsistent. The IRAPs differed regarding both the order of the rules (self-first or
others-first) and in the presentation of the consistent or inconsistent rule first.
The end of the 6th block marked the end of the IRAP task and a screen displayed the
message “This task has been completed. Please contact the researcher”.
Procedure
The procedure for the experiment was identical for participants assigned to either
IRAP. When the participants arrived they were briefed on the nature of the experiment after
which they read and signed a consent form (See Appendix A). Each participant was
subsequently assigned an identification number which served to protect their anonymity.
Once the participant was satisfied to commence the experiment the IRAP was explained to
them. The experimenter emphasised to the participant that the task did not involve the
participant giving their own opinion but it was asking for the participant to respond to a given
rule. It was also made clear that it was crucial to respond accurately to the task and that speed
would come with practice. Once the experimenter was satisfied that the participant
understood the nature of the task the IRAP began. The experimenter interacted directly with
the participant during the practice blocks and drew their attention to their feedback so they
would achieve the mastery criteria to move onto the test blocks. The experimenter left the
room once the participant had progressed to the test blocks.
Once the IRAP was complete the participants completed two online questionnaires;
the Depression Anxiety Stress Scales (DASS; See Appendix B) and the Rosenberg Self-
Esteem Scale (RSES; See Appendix C). The experimenter left the room while the

17. 13
questionnaires were being completed for the privacy of the participant. Once the
questionnaires had been completed the participants were debriefed and thanked for their
participation. They were told to contact the experimenter if they had any questions or
problems regarding the study.
Ethical Considerations
This research was deemed to be ethical by the Ethics Department of Maynooth
University. All participants who took part in the current research were over the ages of 18
and gave their informed consent after being briefed by the researcher on exactly what would
be expected of them during the experiment. Participants were informed that if they wished to
withdraw their data they could do so. In order to provide anonymity to the participants each
person was randomly allocated an identification number. Upon the completion of the
experiment participants were fully debriefed and given contact information if they required
more information about the study.

18. 14
Results
The main dependent variable during an experiment that uses an IRAP is the response
latency, which is a measurement of the time taken between the onset of a trial and the length
of time taken for a correct response to be emitted. Although the accuracy of the participants is
also recorded, this information is used only in order to identify participants who display a low
accuracy score so that they can be emitted from the data analysis. If the participants did not
meet the criteria of 80% accuracy in under 2000ms they were excluded from analysis. These
response latencies were then converted into D-IRAP scores using an adaption of the D-
algorithm (Barnes-Holmes, Barnes-Holmes, Stewart, I., & Boles, 2011). The data from
participants’ practice blocks were not included in the practice blocks.
A repeated measures ANOVA was carried out to compare the D-scores of each of the
four trial types. The participants were split into two groups depending on the IRAP they were
exposed to (self-rule first or other-rule first). IRAP type was defined as the between
participant variable. There was no significant main effect for group on D-IRAP scores (F (1,
114)= 0.52, p= 0.8211). There was also no significant interaction effect found (F (3, 114)=
0.917, p=0.4350).
Figure 2 shows the inverted means for each trial type across both conditions. From the
graph it can be seen that there was a positive bias for the self positive trial for participants
who were exposed to the self rule first, with all other trials in this condition reporting a

19. 15
negative bias towards the self and others. In contrast to this the others-first group
demonstrated a positive bias towards both self and others across all trial types.
Figure 2. Graph showing the inverted means and response bias direction for each of the trial
types.
In total eight one-sample t-test were conducted to establish if the D-IRAP scores for
any of the individual trials types were significant from 0. From the participants who were
exposed to the self-rule first, only the self positive trial reached statistical significance ( t
(19)= 4.447 p=0.003). For the participants who were presented with the other-rule first, it was
found that the only trail type that was significantly different from zero was also the self
positive trial (t (19) = 4.074 p= 0.006), which mirrored the results found in the self-first
group.
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
Self
Positive
Self
Negative
Others
Positive
Others
Negative
Mean
Trial Type
Self Rule 1st
Other Rule 1st
Positive
Bias
Negative
Bias

20. 16
Twelve dependent t-tests were then conducted to see if trial types were significantly
different from each other in each of the groups. Statistically significant differences were
found in the self-rule first group between the self positive and self negative (t(19)= 4.111,
p=0.006), self positive and others positive (t(19)= 3.379, p=0.0031) and the self positive and
others negative (t(19)= 2.581, p=0.0183). Similarly to the self-rule first group, the other-rule
first group also displayed statistically significant differences between the self positive and
self negative (t(19)= 2.809, p=0.0112), the self positive and other positive (t(19)= 3.572,
p=0.002) and the self positive and others negative trial types (t (19)= 2.997 p= 0.0074).
Following on from this analysis, four independent t-tests were conducted to see if
there were a difference between the groups on each trial type. None of the t-tests reached
statistical significance. Although these differences did not reach statistical significance, it
must be noted that there does exist a distinct difference between the groups (See Figure 2).
These differences can be seen on the self positive trial type with the other-rule first
demonstrating a stronger self-positive bias (M=0.440, SD=0.484) than the self-rule first
group (M=0.329, SD= 0.330). A notable difference was also found between the groups on the
others positive trial, with the others-rule first showing a positive bias (M=0.147, SD=0.458)
and the self-first trial showing a negative bias (M=-0.004, SD=0.382). These differences did
not reach statistical significance, which means that caution must be exercised when
interpreting these results, however these differences must be noted.
To investigate the relationships between the implicit measures (IRAP) and the explicit
measures (DASS and RSES) a correlation matrix was produced. There was only one
relationship that showed a moderate correlation between variables. This was the relationship
between the stress score of the participants and the results of the others positive trial (r=-

21. 17
0.339). The relationship displayed a moderate negative relationship with higher D-IRAP
scores on this trial being associated with low levels of stress.
Discussion
From the current research it was found that the only significant differences lay within
comparison of the trial types within the groups. The differences between the groups in these
t-tests is similar to other self-esteem research and highlights the strong self-positive bias that
has been found time and time again in implicit research (Reume et al., 2013; Vahey et al.,
2009). Although these were the only tests that reached statistical significance, differences
between the groups were also noted. The most substantial differences were found between the
self-positive and others-positive trial types with the others-first group showing a stronger
self-positive and others-positive bias. This would suggest that there is a slight reversal of the
anti-others bias in the others-first group which was mediated by the manipulation of the order
of the rules. The correlation matrix did not show any significant correlations, with only one
moderate correlation being found between stress and the others-positive trial type. This
finding highlights the potential differences between explicit and implicit measures of self-
esteem.
The present research was conducted to investigate the influence of rule order on
implicit responding in the IRAP. In previous studies it has been found that small alterations to
screen presentations (Campbell et al., 2011) can influence the IRAP effect and it would hold
that as rules govern responding, the order in which they are presented would have a major
impact on implicit attitudes. Although the differences between the groups did not differ

22. 18
significantly, it is clear that rule order has an effect on the results of experiment. For both of
the self-positive trials there was a considerable positive bias seen in both groups which is
mirrored in all previous IRAP research concerning self-esteem (Remue, et al., 2013; Vahey et
al., 2009). This trend is not surprising due to how commonplace it has become in implicit
research concerning self-esteem (Franck, De Raedt, Dereu, Van den Abbelle, 2007; De
Raedt, Schacht, Franck, & DeHouwer, 2006; ),what is most interesting in the current research
is the difference that exists between the groups on the others positive trial (See figure 2). This
difference did not reach statistical significance so caution must be exercised when
interpreting results, but with this in mind it is hard to deny a clear difference does exist. The
lack of significance could be mediated by the small size of the sample (n=41) and it would
not be reckless to assume that if the population size was to increase the two means would be
pulled further apart making the statistic significant.
Our cautious optimism about future results is reinforced by previous research
surrounding the malleability of implicit attitudes (Cullen, Barnes-Holmes, & Barnes-Holmes,
2009). As in previous research, we also found that implicit attitudes as measured by the IRAP
are susceptible to modification depending on variables or interventions altered by the
researcher (Hussey & Barnes-Holmes, 2012). Cullen and colleagues (2009) conducted an
IRAP study to investigate the malleability of ageist attitudes. In this research two experiments
were conducted, in the first experiment participants were asked to complete an IRAP where
they were asked to respond to stimuli as similar or opposite to young and old people.
However, in this experiment the experimenters purposefully used negative words that are
typically associated with old people (for example, tired, and weary) and positive words (for
example, energetic, and enthusiastic) that are associated with young people. In this way they
primed participants to associate the negative stimuli with old people. This priming effect was
seen in the results which found that participants were faster on consistent trials (Similar-

23. 19
Positive-Young People) compared to inconsistent trials (Similar-Positive-Old People). As in
our experiment it was found that the results of the IRAP are malleable and are heavily
influenced by the wording of the IRAP. This highlights the importance of perfecting the
IRAP procedure before the research is applied in clinical settings or before forming theories
of psychopathological conditions such as depression.
The use of stereotypical words during the IRAP trials provided a reference point for
participants which arose from their history of responding and the surrounding context. The
above research contained a second experiment which investigated the direct influence of
context on implicit responding. To achieve this, participants were split into a pro-old or a pro-
young group. In the pro-old group the participants were presented with a booklet containing
images of admired old people and disliked young people, a booklet containing images of
admired young people and disliked old people was also presented to the pro-young group.
The subjects then completed an IRAP that was identical to the first experiment (still
containing the stereotypical words). The researchers found that although the results did not
reach statistical significance, there was a difference between the two groups after the
exemplar training, particularly in the pro-old group who showed a more positive older people
bias after the intervention (Cullen et al., 2009). The researchers stated this reversal in the
overall IRAP effect in the pro-old condition was not due to the creation of an anti-young bias,
but a reversal in the anti-old effect. From the above study it is clear that placing a stimulus, in
this case an old person, in context can change responding in the IRAP. These results strongly
resonate with the current research. As in our experiment the priming of a pro-others attitude
through the presentation of the others-first rule reversed the anti-others bias while
simultaneously increasing the self-positive bias.
The malleability of relational responses within the IRAP procedure has also
been seen in other experiments. One particular experiment investigated the effect of a mood

24. 20
induction procedure performance of depressed and non-depressed individuals in IRAP trials
(Hussey & Barnes-Holmes, 2012). During the experiment participants depressive symptoms
were measured using the Depression Anxiety and Stress Scale (DASS) and they were then
split into groups of high and normal levels of depressive symptoms. The researchers found
that after the mood induction procedure, only the high depression group showed a decrease in
the positivity of their emotional biases (Hussey & Barnes-Holmes, 2012). The experimenters
postulate that the results of this experiment demonstrate that mildly depressed individuals are
increasingly more susceptible to changes caused by mood state than non-depressed
counterparts and have postulated that this could be at the root of depression.
Although the IRAPs have been found to have strong predictive effects (Steinberg,
Karpinski & Alloy, 2007; Carpenter, Martinez, Vadham, Barnes-Holmes, & Nunes, 2012;
Nocks et al., 2010), it has also been found that the results of the experiments are extremely
malleable and can be influenced by many different variables (Campbell et al., 2011). Making
slight changes in what is presented on screen can increase or decrease the IRAP effect, even
in relation to stimuli which should have been consistently categorised as negative (abuse) or
positive (peace) in an individual’s responding history (Campbell et al., 2011). Just as screen
presentation influenced the IRAP effect, so too did the rule order in our current research
(albeit not significantly), once again demonstrating the malleability of implicit responding.
Self-esteem is a concept that is thought to be well-established due to the multiple exemplars
that an individual would have come across in their responding history and the impact of
altering rule-order on implicit self-esteem is stark. In previous studies it was found that
depressed individuals have been found to have a larger difference between self-esteem on
implicit actual and ideal self-esteem when compared to normal controls and this was
hypothesised to be one of the catalysts for their depressive symptoms (Reume et al., 2013).
Although this hypothesis could be true, the research on implicit is still in its youth and we

25. 21
must be very careful before making any grand claims about its utility. From this small piece
of research it is clear that the effects of the IRAP can be manipulated by the smallest changes
in the procedure. Before grand conclusions are drawn we must polish, perfect and eliminate
variables (such as rule order effects) which interfere with the results of the IRAP.
The robustness of IRAP responses is clear from the above evidence. Through
direct methods such as including some kind of intervention during the procedure, or through
indirect measures such as changing screen presentation or rule order, implicit attitudes are
extremely sensitive to minor changes (Campbell et al., 2011). The research becomes even
more interesting when you consider experiments that have found that the IRAP cannot be
readily faked. McKenna and colleagues (2007) found that when participants are asked
explicitly to slow down on consistent trials in order to fake the IRAP their results showed no
evidence of faking. Finding that implicit attitudes are more malleable after a small
manipulation of the rule order or screen presentation than when participants are explicitly
asked to alter responding, demonstrates the robust nature of IRAP effects and their immunity
to change as a result of the participant’s manipulations (Vahey et al., 2009).
The necessity of future research in the area of rule order manipulation has been
highlighted from the current research. Before implicit research delves into the treatment of
psychopathology, particularly depression, it must concern itself with the impact of rule order
on arbitrary and experimentally controlled relations. This could be achieved by conducting
IRAP research using stimuli such as shapes or nonsense syllables and investigating the effect
of the manipulation of rule order on these stimuli. Perfecting this method of research is of the
utmost importance if the research is to be applied in clinical settings. Although this current
research could act as a catalyst for future research, it is not without limitations. Firstly, the
study contained a small sample size of 41 and cannot be readily generalised to the general
population. This experiment, although demonstrating a difference, did not reach statistical

26. 22
significance and thus the results must be treated cautiously. It would also be conducive to
have both the groups start on a consistent or inconsistent trial types to ensure that this was not
a confounding factor in the research. However these limitations can be easily overcome by
increasing the sample size and altering procedures which should lead to the differences
between the groups being drawn out and reaching statistical significance. Research in this
field is important for both the validity of the IRAP and possibly for its application in clinical
settings.
In conclusion, although the differences between the groups did not differ significantly
there is enough evidence to suggest that further research in this area is warranted. The
manipulation of rule order could significantly influence implicit responding and in this way it
is extremely important that future research addresses this question in order to expose
participants to the most reliable and valid measure of implicit attitudes. As IRAP research has
been shown to have a predictive quality in matters such as suicide attempts (Nocks et al.,
2010) the importance of weeding out technical issues, such as rule order, cannot be
overemphasised.

27. 23
References
Barnes-Holmes, Y., Barnes-Holmes, D., & Roche, B. (2001). Exemplar training and a
derived transformation of function in accordance with symmetry: II. The
Psychological Record, 51, 589-603.
Barnes-Holmes, D., Barnes-Holmes, Y., Stewart, I., & Boles, S. (2011). A sketch of the
implicit relational assessment procedure (IRAP) and the relational elaboration and
coherence (REC) model. The Psychological Record, 60, 527-542.
Barnes-Holmes, D., Hayden, E., Barnes-Holmes, Y., & Stewart, I. (2008). The Implicit
Relational Assessment Procedure (IRAP) as a response-time and event-related
potentials methodology for testing natural verbal relations: A preliminary study. The
Psychological Record, 58, 497-515.
Barnes-Holmes, D., Murphy, A., Barnes-Holmes, Y., & Stewart, I. (2010). The Implicit
Relational Assessment Procedure (IRAP): Exploring the impact of private versus
public contexts and the response latency criterion on pro-white and anti-black
stereotyping among whit Irish individuals. The Psychological Record, 60, 57-66.
Brown, T. A., Chorpita, B. F., Korotitsch, W., & Barlow, D. H. (1997). Psychometric
properties of the Depression Anxiety Stress Scales (DASS) in clinical samples.
Behaviour Research and Therapy, 35(1), 79-89.

28. 24
Campbell, C., Barnes-Holmes, Y., Barnes-Holmes, D., & Stewart, I. (2011). Exploring screen
presentations in the Implicit Relational Assessment Procedure (IRAP). International
Journal of Psychology and Psychological Therapy, 11(3), 377-388.
Catania, A. C. (1999). Learning, 4th Edition. Upper Saddle River, New Jersey: Prentice-Hall.
Carpenter, K. M., Martinez, D., Vadham, N. L., Barnes-Holmes, D., & Nunes, V. E. (2012).
Performance based measures of attentional bias and relational responding are
associated with behavioural treatment outcome for cocaine dependence. The
American Journal of Drug and Alcohol Abuse, 38, 146-154.
Cullen, C., Barnes-Holmes, D., Barnes-Holmes, Y. (2009). The implicit relational assessment
procedure (IRAP) and the malleability of ageist attitudes. The Psychological Record,
59, 591-620.
De Raedt, Schacht, R., Franck, E., & De Houwer, J. (2006). Self-esteem and depression
revisited: Implicit positive self-esteem in depressed patients. Behaviour Research and
Therapy, 44(7), 1017-1028.
Deroost, N., & Saetens, E. (2006). Spatial processing and perceptual sequence learning in
SRT tasks. Experimental Psychology, 53, 16-30.
Dunning, D., Meyerowitz, J. A., & Holzberg, A. D. (1998). Ambiguity and self-evaluation:
The role of idiosyncratic trait definitions in self-serving assessments of ability.
Journal of Personality and Social Psychology, 57, 77-85.
Drake, C. E., Kellum, K. K., Wilson, K. G., Luoma, J. B., Weinstein, J. H., & Adams, C., H.
(2010). Examining the implicit relational assessment procedure: Four preliminary
studies. Psychological Record, 60(1), 81-100.
Ferguson, M. J., & Bargh, J. A. (2007). Beyond the attitude object: automatic attitudes spring
from object-centred-contexts. In B. Wittenbrink & N. Schwarz (Eds.), Implicit

29. 25
Measures of Attitudes. Guilford: New York pp.216-246.
Franck, E., De Raedt, R., Dereu, M., Van den Abbeele, D. (2007). Implicit and explicit self-
esteem in currently depressed individuals with and without suicidal ideation. Journal
of Behaviour Therapy and Experimental Psychiatry, 38, 75-85.
Golijani-Moghaddam, N., Hart, A., & Dawson, D. L. (2013). The Implicit Relational
Assessment Procedure: Emerging reliability and validity data. Journal of Contextual
Behavioural Science, 2(3), 105-119.
Greenwald, A. G., & Banaji, M. R. (1995). Implicit social cognition: Attitudes, self-esteem
and stereotypes. Psychological Review, 102, 4-27.
Greenwald, A. G., Banaji, M. R., Rudmanm L. A., Farnham, S. D., Nosek, B. A., & Mellot,
D. S. (2002). A unified theory of implicit attitudes, stereotypes, self-esteem and self-
concept. Psychological Review, 109, 2-25.
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. (1998). Measuring individual
differences in implicit cognition: the implicit association test. Journal of personality
and social psychology, 74(6), 1464.
Gremar, M. C., Segal, Z. V., Sagrati, S., & Kennedy, S. J. (2001). Mood-induced changes on
the implicit association test in recovered depressed patients. Journal of Abnormal
Psychology, 110, 282-289.
Gullone, E., Jones, T., & Cummins, R. (2000). Coping styles and prison experiences as
predictors of prychological well-being in male prisoners. Psychiatry, Psychology and
Law, 7, 170-181.
Hagga, D., Dyck, M., & Ernst, D. (1991). Empirical status of cognitive therapy of depression.
Psychological Bulletin, 110, 215-236.

30. 26
Hayes, S. C., Barnes-Holmes, D., Roche, B. (2001). Relational Frame Theory : A Post
Skinnerian Account of Human Language and Cognition. Hingham, MA: Kluwer
Academic Pulishers.
Healy, O., Barnes-Holmes, D., & Smeets, P. (2000). Derived generalised responding as a
generalised operant behaviour. Journal of Experimental Analysis of Behaviour, 74,
207-227.
Hooper, N., Villatte, M., Neofotistou & McHugh, L. (2010). The effects of mindfulness
versus thought suppression on implicit and explicit measures of experiential
avoidance. International Journal of Behavioural Consultation and Therapy, 6(3),
233-244.
Hughes, S., Barnes-Holmes, D., & Vahey, N. (2012). Holding on to our functional roots
when exploring new intellectual islands: A voyage through implicit cognition
research. Journal of Contextual Behavioural Science, 1, 17-30.
Hussey, I., & Barnes-Holmes, D. (2012). Implicit Relational Assessment Procedure as a
measure of implicit depression and the role of psychological flexibility. Cognitive and
Behavioural Practice, doi:10.1016/j.chpra.2012.03.0202.
Ingram, R. E., Miranda, J., & Segal, Z. V. (1998). Cognitive Vulnerability to Depression.
New York: Guilford Press.
Ingram, R. E., & Wisnicki, K. (1991). Cognition and depression. In P. A. Margo (Ed.),
Annual Review of Psychopathology, Vol. 1 (pp.187-230).
Luciano, C., Valdivia, S., Cabello, F., & Hernandez, M. (2004). Developing self-directed
rules. In R. Rehlfeld & Y. Barnes-Holmes (Eds). Applying Relational Frame Theory
to the Language Disabled. Oakland, CA: New Harbinger.

31. 27
McKenna, I., Barnes-Holmes, D., Barnes-Holmes, Y., & Stewart, I. (2007). Testing the fake-
ability of the Implicit Relational Assessment Procedure (IRAP): The first study.
International Journal of Psychology and Psychological Therapy, 7(2), 253-268.
Nock, M. K., Park, J. M., Finn, C. T., Deliberto, T. L., Dour, H. J., & Banaji, M. R. (2010).
Measuring the suicidal mind: Implicit cognition predicts suicidal behaviour.
Psychological Science, 21, 511-517.
O’Hora, D., Barnes-Holmes, D., Roche, B., & Smeets, P. M. (2004). Derived relational
networks and control by novel instructions: A possible model of generative verbal
responding. The Psychological Record, 54, 437-460.
Oser, C. B. (2006). The criminal offending-self-esteem nexus: Which version of the self-
esteem theory is supported? The Prison Journal, 86, 344-460.
Parris, B. A., Sharma, D., & Weekes, D. (2007). An optimal viewing position effect in the
Stroop task when only one letter is the colour carrier. Experimental Psychology, 54,
273-280.
Pinter, B., & Greenwald, A. G. (2005). Clarifying the role of the “other” category in the self-
esteem IAT. Experimental Psychology, 52(1), 74-79.
Pothos, E. M. (2005). Expectations about stimulus structure in implicit learning. Memory and
Cognition, 33, 171-181.
Remue, J., De Houwer, J., Barnes-Holmes, D., Vanderhasselt, A., & De Raedt, R. (2013).
Self-esteem revisited: Performance on implicit relational assessment procedure as a
measure of self-versus ideal self-related cognitions in dysphoria. Cognition and
Emotion, 27(8), 1441-1449.

32. 28
Roberts, J. E., Gotlib, I. H., & Kassel, J. D. (1996). Adult attachment security and symptoms
of depression: the mediating roles of dysfunctional attitudes and low self-esteem.
Journal of personality and social psychology, 70(2), 310-320.
Robins, R. W., Hendin, H. M., & Trezeniewski, K. H. (2001). Measuring global self-esteem:
Construct validation of a single-item measure and the Rosenberg Self-Esteem Scale.
Personality and Social Psychology Bulletin, 27(2), 151-161.
Torneke, N., Luciano, C., & Salas, S. V. (2008). Rule-governed behaviour and psychological
problems. International Journal of Psychology and Psychological Therapy, 8, 141-
156.
Steinberg, J. A., Karpinski, A., & Alloy,L. B. (2007). The exploration of implicit aspects of
self-esteem in vulnerability- Stress models of depression. Self and Identity, 6, 101-
117.
Vahey, N. A., Barnes-Holmes, D., Barnes-Holmes, Y., & Stewart, I. (2009). A first test of the
implicit relational assessment procedure as a measure of self-esteem: Irish prisoner
groups and university students. The Psychological Record, 59, 371-388.
Williams, J. M. G. (1997). Depression. In D. M Clark, & C. G. Fairburn (Eds.), Science and
Practice of Cognitive Behaviour Therapy (pp.259-283). Oxford: Oxford University
Press.
Young, J. E. (1994). Cognitive Therapy for Personality Disorders: A Schema-Focused
Approach. Sarasota: Professional Resource Press.
Zentner, M., & Renaud, O. (2007). Origins of adolescents’ ideal self: An intergenerational
perspective. Journal of Personality and Social Psychology, 92(3), 557-574.

33. Appendices
Appendix A: Consent Form
INFORMED CONSENT FORM FOR STUDENT RESEARCH
In agreeing to participate in this research I understand the following:
This research is being conducted by Sarah Kenehan, an undergraduate student at the
Department of Psychology, National University of Ireland, Maynooth. The method proposed for
this research project has been approved by the Departmental Ethics Committee, which means
that the Committee does not have concerns about the procedure itself as detailed by the student.
It is, however, the above-named student’s responsibility to adhere to ethical guidelines in their
dealings with participants and the collection and handling of data.
 If I have any concerns about participation I understand that I may refuse to participate
or withdraw at any stage.
 I have been informed as to the general nature of the study and agree voluntarily to
participate.
 I will complete a number of questionnaires, some of which will ask about symptoms of
psychological suffering, as well as a number of computer based pairing tasks where I will
be asked to pairs words related to self esteem .
 There are no known expected risks associated with any aspect of participation.
 All data from the study will be treated confidentially. The data from all participants will
be irrevocably anonymised, compiled, analysed, and submitted in a report to the
Psychology Department. The data will be retained for approximately 5 years before being
destroyed. No participant’s data will be identified by name at any stage of the data
analysis or in the final report.
 At the conclusion of my participation, any questions or concerns I have will be fully
addressed.
 I may withdraw from this study at any time without giving a reason, and may withdraw
my data at the conclusion of my participation if I still have concerns.

34. Participant’s signature Researcher’s signature
Participant’S name (PLEASE PRINT) Date
Should you be in distress experiencing any form of mental health complaint we encourage you to
contact the NUIM student counseling service. This professional, free, and confidential
counseling service can be contacted on 01-7083554 or counselling.nuim.ie to schedule an
appointment.

35. Appendix B: Rosenberg Self-Esteem Scale (RSES)
RSES
Belowisa listof statementsdealingwithyourgeneral feelingsaboutyourself. Please rate how much
youagree with each statementbycirclinganumbernexttoit. Use the scale below tomake your
choice.
Strongly Agree Agree Disagree Strongly Disagree
1 2 3 4
1.
I feel thatI am a personof worth,at leaston an equal plane
withothers.
1 2 3 4
2. I feel thatI have a numberof good qualities. 1 2 3 4
3. All inall,I am inclinedtofeel thatIam a failure. 1 2 3 4
4. I am able to do thingsaswell asmost otherpeople. 1 2 3 4
5. I feel Ido nothave much to be proudof. 1 2 3 4
6. I take a positive attitude towardmyself. 1 2 3 4
7. On the whole,Iam satisfiedwithmyself. 1 2 3 4
8. I wishI couldhave more respectformyself. 1 2 3 4
9. I certainlyfeel uselessattimes. 1 2 3 4
10. At timesIthinkI am no goodat all. 1 2 3 4

36. Appendix C: DepressionAnxiety Stress Scales 21 (DASS)