The document discusses situational effects in assessments. It finds that (1) simulations provide low incremental validity over context-free measures, (2) situational models provide a better fit than trait-only models for assessment center data, and (3) situational influences can be disentangled from traits using situational judgement tests. Cognitive models of situations show how encoding and judgments are affected by various situational dimensions. Improving assessments requires evaluating both situational and dispositional factors.

1. Representing Situations in Assessment –
extracting better value from our investment
Division of Occupational Psychology
Annual Conference 2011
Michael Burnett & Almuth McDowall

2. Overview
• The Problem with Situations
• Disentangling Situational Influences:
• Part 1: Predominance of situational effects in high stakes
Assessment Centres
• Part 2: Understanding trait v. situational effects in low stakes
Situational Judgement Tests
• Part 3: How cognitive Situational Models address sources of
situational effects
• Implications

3. The Problem with Situations
•Techniques used today range from ‘context free’ cognitive ability
and personality measures to simulations (ACs and SJTs) of work
or organisational situations
•The first problem is that simulations give significant (ACs .281,
SJTs .18-.382) but in absolute terms low additional incremental
validity over ‘context free’ measures
• The second problem is explaining what simulations actually
measure – e.g. the methods effect in ACs3, construct validity of
SJTs4
• The implication is that applied psychology needs to provide a
better account of individual differences in dealing with situations
*
1AC Criterion Validity –Harmelin et. al. (2007)
2SJT Criterion Validity - Christian et. al. (2010)
3Sackett & Dreher (1982)
4McDaniel & Whetzel (2005)

4. Multimedia based simulations will happen
• Multimedia-based ‘simulations’ in assessment will grow quickly
• Video-based, Avatar-based simulations (Second LifeTM), 2-D
Animations, Apps (i-phone/ipad/mobile), Games
• Occ. Psych. Design Methodology needs to adapt
• Maintain control over what is being measured!

5. Measures
Exercises
T Part 1: The predominance
r of the situation
a
i
t
s
Model x2 df GFI AGFI RMSEA NNFI SRMR
Study 1a
Traits Only 3034.6 67 0.69 0.51 0.22 0.53 0.17
(CT0E)
Exercises Only 96.6 57 0.99 0.97 0.027 0.99 0.014
(0TCE)
Correlated Traits 208.05 59 0.97 0.95 0.052 0.98 0.047
& Exercises
(CTCE)
Correlated 62.45 44 0.99 0.98 0.021 1.0 0.013
Exercises &
Traits (CECT)*

6. Situational effects in high stakes simulations
Structural Equation Models of assessment centre results based on
situations (exercises) provide a good fit to the data, trait-only
(competency dimensions) models do not fit
The best fit combines situations moderated by trait (competency
dimensions) factors
Adding in Cognitive ability and Big 5 Personality dimensions
showed best fit when added to the Exercise model, cognitive ability
predominated
Models that did not fit included Exercise Type model and Trait
Activation model (Lievens et. al., 2006)

7. Part 2: Exploring situational effects
• Use of low fidelity simulation based on SJT method to explore
comparative effect of situational and ‘trait’ effects
•Academic situations used with samples drawn from University
and job applicants seeking SJT practise (n. 339)
• Situational influence assessed by examining within-situation item
relationships (zero correlation expected if competency/traits
determine performance)
• Competency and personality trait influence assessed by
examining across-situation scale relationships between SJT items
and independent measures of competencies & traits

8. Design of Situational Judgement Items
Situation Situation
1 2
Item 1 Item 2 Item 3 Item 4 Item 1 Item 2 Item 3 Item 4
Plan & Relate & Persuade &
Analysis
Organise Network Influence
Conscientiousness Openness Agreeableness Extraversion

9. Situational Judgement Item
You have been selected to present a lecture to your course in
front of the head of Department and senior tutors
1. You set yourself objectives for what you are going to say to
include new and innovative ideas (Planning_Open)
2. You try and make sure that the content of your lecture will appeal
to both students and staff (Relating_Agreeable)
3. You aim to keep the audience captivated by questioning
established assumptions (Analysis_Extravert)
4. You check your material and rehearse the presentation to ensure
you appear credible (Persuading_Conscientious)

10. Likert what I ‘would do’ responses
Plan & Relate & Persuade &
Analysis
Organise Network Influence
.36 .33 .24 .10
Plan & Relate & Persuade &
Analysis
Organise Network Influence
Average .12
Situation (.06 to.20)
Conscientiousness Openness Agreeableness Extraversion
.31 .22 .05 .13
Conscientiousness Openness Agreeableness Extraversion

11. Part 3: Understanding situational effects
• Situations are multidimensional - consider effects on encoding & retrieval:
(Zwaan & Radvansky, 1998; Magliano et.al. 2008):
- Space – context, layout and movement
- Time – chronology of events, temporal shifts
- Protagonists - Actors & Objects
- Intention – Goals & Motivations
- Causation – inferences of what has caused events or what could
happen next

12. Cognitive Model of Situation
Assumption
about power
2. Time
relationship5
Goals & motivations4
Assumed ‘agreement’
Manager3 becomes disagreement Goals &
motivations4
Manager’s expectation
1. Space becomes “use of initiative”
Goals Supplier3
Assumption Participant
about power as
relationship5 Protagonist3 ‘Non-performing’ supplier
Goals & motivations4 becomes threat to supply chain
The Situational Judgement 3. Actors
4. Motivations
Causality5
determines these…………..
5. Causes

13. Personality & Competency effects on
encoding & judgements
Person Variables e.g. goals, domain knowledge
competences, personality traits, state
Situation Actual
as construed behaviour
& encoded Time enacted
Goals & Motivations
A a I i
Tutor Goals &
B b Motivations
II ii
C c Influence on future evaluation III iii
D d Space Goals Student IV iv
E e Protagonist
V v
F f Influence on future relationships VI vi
Goals & Motivations
G g VII vii
Objective Potential
characteristics behaviours
of Situation afforded
Adapted from Reis, 2008

14. Implications
•Performance in Simulation-based assessments is determined by
both situational and dispositional (trait & competency) factors
•AC feedback should reflect both aspects
•SJTs can be structured to return required facets
•Situational Models hold promise in improving the yield from
simulations (if we change the paradigm)
•assessing encoding as well as judgement to widen the range
of information gathered from assessments
•providing a systematic way of describing situations to
understand how these affect behaviour (IRT correlates)
•enabling greater control over how situations are used in
simulations including use of multimedia & gaming techniques
(impact on Occ Psych methodology)