Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

0

Share

Reducing Evaluative Bias in Measuring the Big Five by Dr Stewart Desson

What is evaluative bias, and what are we doing to reduce it from selection and development solutions? Lumina Learning CEO Dr Stewart Desson discusses his PhD research into this topic.

Watch a video of Stewart giving this presentation: https://youtu.be/86OOo3Kw8m0

Read his article on the topic: https://luminalearning.com/evaluative-bias-minimisation

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Reducing Evaluative Bias in Measuring the Big Five by Dr Stewart Desson

  1. 1. Exploring The Impact On Criterion Validity Of Reducing Evaluative Bias In Measuring the Big Five Dr. Stewart Desson, CEO Lumina Learning stewartdesson@luminalearning.com https://www.linkedin.com/in/stewartdesson/
  2. 2. Structure of presentation • 3 Main Achievements • Why is this research of interest? • 5 Aims & 9 Research questions • Highlights of the work • Key Findings • Limitations and Suggestions for Further Research • Conclusions
  3. 3. Three Main Achievements Lumina Spark personality instrument designed to: 1. Reduce Evaluative Bias - Improves “user validity” (MacIver, Anderson, Costa & Evers, 2014) - Values diversity 2. Measure Personality Adaptively and Maladaptively - Without pathologising - Discovered new blended personality trait 3. Improve our scientific understanding of performance at work - Bandwidth fidelity debate explored - Greater fidelity established MacIver, R., Anderson, N., Costa, A. C., & Evers, A. (2014). Validity of Interpretation: A user validity perspective beyond the test score. International Journal of Selection and Assessment, 22(2), 149-164.
  4. 4. Why is this research of interest? • Contribution to science - There is bias in current personality assessment tools - Need to measure dysfunctional behaviour at work, without recourse to mental health models - Greater fidelity in understanding the relationship between ‘personality’ and ‘performance at work’ - Through greater fidelity, measuring a new ‘blended trait’ • Contribution to organisational psychology - Enhanced interpersonal relationships - More effective organisations, through better developed leaders • This research is timely - Changing culture puts more focus on the need to avoid bias in the workplace - Need to value “deep diversity”
  5. 5. Five Aims 1. Measure both polarities of each Big Five dimensions - as scalar opposites and independent constructs - explore evaluative bias (Bäckström et al., 2014) and impact on user validity 2. Measure adaptive and maladaptive scales - explore their relationship with highly dysfunctional constructs (Judge, Piccolo & Kosalka, 2009) such as the "dark side" traits measured in the HDS (Hogan & Hogan, 1997; Hogan, Hogan & Kaiser, 2010) 3. Establish usefulness of the Spark model - understand convergent and divergent validity - locate scales in the periodic table of personality traits (Woods & Anderson, 2016) 4. Establish the criterion validity of the model - positive and negative correlations between workplace performance and adaptive / maladaptive traits at both poles of each Big Five construct 5. Explore bandwidth / fidelity debate - empirically test higher and lower level models on the data gathered
  6. 6. Why measure both ends? If you stick your head in the oven and your feet in the freezer, on average you'll be comfortable Attributed to statistician Bruce Grossman 19 January 1960, The Guthrian (Guthrie Center, IA), pg. 2, col. 4:
  7. 7. Methodology 1.Five Aims 2.Nine Research Questions 3.Forty-Two Hypothesis tests
  8. 8. Evaluative Bias in humour
  9. 9. Q. Why do we have Evaluative Bias in psychometrics? A. One reason is item selection based on Factor Analysis “When an evaluatively unbalanced set of descriptors such as the Big Five adjectival markers (Goldberg, 1992) is subjected to a simple structure rotation algorithm, the resulting factors almost invariably end up contrasting positive versus negative descriptors (Goldberg, 1992).” Pettersson, E., Mendle, J., Turkheimer, E., Horn, E. E., Ford, D. C., Simms, L. J., & Clark, L. A. (2014). Do maladaptive behaviors exist at one or both ends of personality traits? Psychological assessment, 26(2), 433.
  10. 10. Trait Descriptive Adjectives (TDA) The adjectives that describe “Extraversion”: • Extraverted • Unrestrained • Energetic • Active • Daring • Vigorous • Bold • Verbal • Assertive • Talkative Goldberg, L. R. (1992). The development of markers for the Big-Five factor structure. Psychological assessment, 4(1), 26. The adjectives that describe “Introversion”: • Introverted • Unexcitable • Inhibited • Untalkative • Timid • Withdrawn • Reserved • Bashful • Shy • Quiet
  11. 11. TDA scored using a 5-point Likert format (N = 40) 1 - ‘Highly Undesirable’ through to 5 - ‘Highly Desirable’ “Please score each of these statements according to how socially desirable you think they would be in another person. Do not consider whether you yourself possess them or not. Instead, just intuitively rate them based on how much you think others may find them desirable.”
  12. 12. TDA - Average Social Desirability Score Quantifiying Evaluative Bias Item Categories O C E A N Plus (O+,C+,E+,A+,N-) 3.8 4.0 3.7 4.3 1.9 Minus (O-,C-,E-,A-,N+) 2.1 1.8 2.4 1.8 3.2 Difference is a Measure of Evaluative Bias 1.8 2.2 1.3 2.6 -1.3
  13. 13. Evaluative Bias (Peabody, 1967) still an issue • Research shows many Big Five models have an evaluative bias (Bäckström, Björklund & Larsson, 2014) • Negative impacts: - on user validity - on construct validity • Questions: - has the construct been measured in a comprehensive and non biased way? - how does the user feel about reading about aspects of their personality in a biased and unbalanced personalised report?
  14. 14. A+ Agreeable A- Direct E- IntrovertedE+ Extraverted Measured Intimate Observing Purposeful Structured Reliable Accommodating Collaborative Empathetic Tough Competitive Logical Demonstrative Takes Charge Sociable Adaptable Flexible Spontaneous Cautious Practical Evidence Based Radical Conceptual Imaginative C+ Conscientious C- Flexible O- PragmaticO+ Open Agreeableness Conscientiousness Extraversion Openness to Experience Measuring the Big 5 Factors at ‘Both Ends’ – OCEAN – with Jungian Type Overlay Even-Tempered Optimistic Resilient Confident N- Emotional Stability Impassioned Vigilant Responsive Modest N+ Neurotic Neuroticism Feeling or Thinking Judging or Perceiving Extraversion or Introversion Intuition or Sensing ADAPTIVETRAITS ADAPTIVETRAITS Maladaptive Adaptive
  15. 15. A+ Agreeable to People Pleaser A- Direct to Aggressive E- Introverted to PassiveE+ Extraverted to Overbearing Measured Intimate Observing Purposeful Structured Reliable Accommodating Collaborative Empathetic Tough Competitive Logical Demonstrative Takes Charge Sociable Adaptable Flexible Spontaneous Cautious Practical Evidence Based Radical Conceptual Imaginative C+ Conscientious to Bureaucratic C- Flexible to Chaotic O- Pragmatic to ClosedO+ Open to Dreamer Agreeableness Conscientiousness Extraversion Openness to Experience Measuring the Big 5 Factors at ‘Both Ends’ – OCEAN – with Jungian Type Overlay Even-Tempered Optimistic Resilient Confident N- Emotional Stability Impassioned Vigilant Responsive Modest N+ Neurotic Neuroticism Feeling or Thinking Judging or Perceiving Extraversion or Introversion Intuition or Sensing MALADAPTIVETRAITS MALADAPTIVETRAITS Maladaptive Adaptive
  16. 16. I make new friends easily Illustrative Example: Measuring Extraversion Sometimes I listen too much and don’t give my view Extraversion Introversion
  17. 17. I make new friends easily Illustrative Example: Measuring Extraversion Sometimes I talk too much I choose my words carefully before I speak Sometimes I listen too much and don’t give my view Extraversion Adaptive Introversion Maladaptive Extraversion Maladaptive Introversion Adaptive
  18. 18. 4 Points of the Personality Compass • Aggregating the 4 measures reduces bias AND • Exploring the 4 measures supports richer interpretation and dialogue Extraversion Introversion Adaptive Maladaptive
  19. 19. Lumina Spark Reliability and Validity • Lumina Spark Psychometric is reliable - Cronbach’s Alphas - Test-re-test • Lumina Spark Psychometric is valid for use in the workplace - Construct Validity - Convergent and Divergent Validity - Criterion Validity - Consensual Validity - User Validity increased - Evaluative Bias decreased
  20. 20. IPIP NEO - Average Social Desirability Score Quantifiying Evaluative Bias Item Categories O C E A N Plus (O+,C+,E+,A+,N-) 3.4 4.3 3.7 3.9 2.0 Minus (O-,C-,E-,A-,N+) 2.6 2.1 2.5 2.0 3.8 Difference is a Measure of Evaluative Bias 0.7 2.2 1.2 1.8 -1.8
  21. 21. Lumina Spark - Average Social Desirability Score Quantifiying Evaluative Bias Item Categories O C E A N Plus (O+,C+,E+,A+,N-) 3.2 3.3 3.0 3.1 1.9 Minus (O-,C-,E-,A-,N+) 2.6 2.6 2.6 2.8 4.3 Difference is a Measure of Evaluative Bias 0.6 0.7 0.4 0.2 -2.4
  22. 22. Lumina Spark has Reduced Evaluative Bias Hypothesis Outcome H39: The TDA and IPIP-NEO measures of Openness will possess more evaluative bias than the Spark measure of Openness Supported H40: The TDA and IPIP-NEO measures of Conscientiousness will possess more evaluative bias than the Spark measure of Conscientiousness Supported H41: The TDA and IPIP-NEO measures of Extraversion will possess more evaluative bias than the Spark measure of Extraversion Supported H42: The TDA and IPIP-NEO measures of Agreeableness will possess more evaluative bias than the Spark measure of Agreeableness Supported
  23. 23. Great Eight • Kurz and Bartram (2002) defined the Great Eight • Kurz (2003) developed variation of Great Eight with more applied titles • Bartram (2005) meta analysis showed OPQ scale mappings - Double weight for top scales - Single weight for two further scales • Saville, Maclver, & Kurz (2009) repeated Bartram (2005) method - showed scale mappings of OPQ, Saville Wave Professional, Wave Focus, SPQ, NEO, HPI, 16PF onto Great Eight • This study applies Bartram (2005) and Saville et al. (2009) approach to Lumina Spark
  24. 24. Great Eight Items Analysing Situations: Demonstrating Analytical Thinking; Solving Complex Problems; Critically Evaluating Information Creating Concepts: Being Creative and Innovating; Thinking Strategically; Driving Organisational Change Relating to People: Displaying Good Interpersonal Skills; Exercising Active Listening; Communicating Effectively Controlling Resources: Leading and Directing Others; Managing People and Resources Effectively; Being Decisive; Making Sound Judgments Respecting People: Giving Support; Building Team Spirit; Showing Compassion and Being Approachable Adapting to Demands: Showing Composure; Working Effectively Under Pressure; Dealing with Ambiguity Delivering Results: Planning and Organising Efficiently; Working Diligently; Completing Tasks on Time Driving Performance: Having Career Ambition; Setting and Achieving Ambitious Work Objectives; Showing Business Acumen
  25. 25. 14 “Point to Point” Spark to Great Eight Hypotheses Created Analysing Situations Creating Concepts Relatingto People Controlling Resources Respecting People Adaptingto Demands Delivering Results Driving Performance SumofAll Eight O+ Big Picture Thinking .09 .22** -.07 .03 -.04 .01 -.12* .03 .03 O+ Overextended .10 .16* -.08 -.03 -.10 -.04 -.19** .03 -.02 O- Down to Earth .02 -.25** -.08 -.06 -.04 -.06 .13* -.06 -.08 O- Overextended -.03 -.26** .01 -.03 .00 -.01 .13* .01 -.04 C+ Discipline Driven .07 -.06 -.02 .04 .06 .08 .28** .08 .09 C+ Overextended .02 -.16* -.08 -.08 -.03 -.15* .11 -.02 -.07 C- Inspiration Driven -.13* .06 -.02 -.05 -.06 -.08 -.11 -.11 -.09 C- Overextended -.08 -.01 -.02 -.12 -.04 -.17** -.25** -.12* -.14* E+ Extarversion -.05 .12 .21** .06 .11 .10 .03 .08 .12* E+ Overextended .02 .09 .09 -.04 -.01 -.01 -.10 .10 .03 E- Introversion .10 -.10 -.18** -.02 -.07 -.09 .03 -.06 -.07 E- Overextended .09 -.10 -.20** -.06 -.14* -.14* .04 -.05 -.10 A+ People Focused -.09 .04 .19** -.05 .20** -.03 -.01 -.06 .03 A+ Overextended -.09 -.08 .01 -.16* .04 -.13* -.10 -.13* -.12 A- Outcome Focused .12* .02 -.14* .11 -.22** .05 .04 .12* .03 A- Overextended .13* -.10 -.22** .02 -.19** -.05 .02 .05 -.06 N+ Risk Reactor Overextend -.05 -.20** -.20** -.17** -.15* -.35** -.13* -.13* -.25** N- Reward Reactor .06 .22** .13* .21** .07 .25** .07 .13* .20** Note: N = 254; * p<0.05 ** p<0.01
  26. 26. Analysing Situations Creating Concepts Relatingto People Controlling Resources Respecting People Adaptingto Demands Delivering Results Driving Performance SumofAll Eight O+ Big Picture Thinking .22** -.12* O+ Overextended .16* -.19** O- Down to Earth -.25** .13* O- Overextended -.26** .13* C+ Discipline Driven .28** C+ Overextended -.16* -.15* C- Inspiration Driven C- Overextended -.17** -.25** -.12* -.14* E+ Extarversion .21** .12* E+ Overextended E- Introversion -.18** E- Overextended -.20** -.14* -.14* A+ People Focused .19** .20** A+ Overextended -.16* -.13* -.13* A- Outcome Focused .12* -.14* -.22** .12* A- Overextended .13* -.22** -.19** N+ Risk Reactor Overextend -.20** -.20** -.17** -.15* -.35** -.13* -.13* -.25** N- Reward Reactor .22** .13* .21** .25** .20** 12 out of 14 hypotheses supported (shaded green) 2 out of 14 hypotheses not supported (shaded red) 8 additional interesting and important findings (shaded orange)
  27. 27. Nine Research Questions 1. Is the proposed Spark model of personality compatible with the Big Five factor structure? 2. Where do the Spark scales sit in the personality periodic table (Woods & Anderson, 2016) of blended Big Five factors? 3. Do the Spark adaptive scales correlate more highly than the Spark maladaptive scales, with other “bright side” Big Five traits? 4. Do the Spark’s maladaptive scales correlate more highly than the Spark adaptive scales, with the HDS “dark side” traits? 5. What evidence is there to support the conceptualisation of Spark maladaptive scales as overplayed / overextended / extreme ends of the "bright side" Big Five traits? 6. How well does the Spark comply with a priori hypothesized criterion validity relationships with the Great Eight competency model? 7. Is there a differential pattern of criterion validities between the Spark adaptive and maladaptive scales and if so, what can be learnt from this? 8. Compared to the Spark five-dimensional bandwidth approach, can the higher fidelity Spark eighteen scales explain more of the variance in the personality criterion relationship? 9. Can the Spark reduce the impact of evaluative bias in the Big Five?
  28. 28. Limitations • Weird sample - “Western, educated, industrialised, rich, democratic countries” - Henrich, Heine and Lorentzian (2010) • All criterion measures are 360-degree observations • Causal inference?
  29. 29. Suggestions for further research • Replicate the confirmatory factor analysis on new samples • Explore if certain maladaptive traits may be - detrimental to performance in many roles - essential to performance in other specific roles e.g. a soldier about to go into combat • Can Neuroticism be expressed adaptively? • Could Emotional Stability be expressed in a maladaptive form?
  30. 30. Suggestions for further research • Build a maladaptive periodic table - measuring blends of opposite polarities - both adaptively and maladaptively • Use Lumina Spark as an instrument for dissecting and assessing the construct validity of existing instruments - in a way that other instruments may not be able to - as a compliment to the TDA periodic table analysis
  31. 31. Five Conclusions 1. Evaluative bias is (increasingly) an organisational issue 2. Issue addressed by measuring both ends of the Big Five dimensions 3. Conceptualising maladaptive traits as the more extreme ends of the Big Five (“too much of a good thing”) - Helps explain enablers and blockers to performance at work - Avoids risk of pathologising people 4. Spark approach does reduce evaluative bias - Compared to TDA and IPIP-NEO - Enables test user to crack open their Big Five dimensions and see their adaptive and maladaptive traits at both ends of the polarities 5. Spark has explored elements of the periodic table (Woods & Anderson, 2016) less well researched by other top psychometrics - Explored element not yet researched elsewhere - Blend that leads with Conscientiousness and is supported by Neuroticism stewartdesson@luminalearning.com https://www.linkedin.com/in/stewartdesson/

What is evaluative bias, and what are we doing to reduce it from selection and development solutions? Lumina Learning CEO Dr Stewart Desson discusses his PhD research into this topic. Watch a video of Stewart giving this presentation: https://youtu.be/86OOo3Kw8m0 Read his article on the topic: https://luminalearning.com/evaluative-bias-minimisation

Views

Total views

470

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

0

Shares

0

Comments

0

Likes

0

×