How one's gender can influence how they rate others and how they are rated.

1. © 2017 Peter Berry Consultancy Pty Ltd.
Dr Cate Borness
July 2017
Gender Bias in Multi-
Rater Feedback Surveys
How one’s gender can influence how they
rate others and how they are rated

2. © 2017 Peter Berry Consultancy Pty Ltd.

3. © 2017 Peter Berry Consultancy Pty Ltd.
Background
▪ Much research in multi-rater surveys has focused on factors affecting self-other
agreement, claiming that discrepancies can be explained by individual rater
characteristics such as gender, race and age, as well as contextual factors such as
rater-ratee similarity (Ostroff, et al, 2004).
▪ Bias involves an interpretative judgement that yields a rating that is unfair, illegitimate,
or unjustifiable, in the sense that it goes beyond the objective evidence of the situation
(Brewer & Brown, 1998; Turner & Reynolds, 2001).
▪ This research examined the impact of gender bias on 360 survey results using the
gender of both ratees and raters.

4. © 2017 Peter Berry Consultancy Pty Ltd.
Background
▪ Specific types of bias exist that affect the validity of 360° degree assessments. For
example,
▪ the halo/horn error occurs when an individual is rated as good or poor based on subjective
qualities (eg. appearance, popularity), rather than on actual behaviours observed (Yukl &
Lepsinger,1995).
▪ in-group bias which refers to the tendency to evaluate one’s own group and its members
more favourably than another group – the “out-group” – and its members (Hewstone, Rubin, & Willis,
2002).
▪ Men are less likely than women to have an in-group bias (Nosek & Banaji, 2001; Richeson & Ambady,
2001; Rudman & Goodwin, 2004).
▪ Based on previous research, it was hypothesised that there would be some sort of in-
group bias for female raters, whereby they would tend to rate other women more
positively than they rated males.

5. © 2017 Peter Berry Consultancy Pty Ltd.
About the Hogan 360 Survey
▪ Gathers performance feedback from managers,
peers, direct reports, other stakeholders (multi-
rater survey)
▪ Features rated items (1-7 scale), ranked items (top
strengths and opportunities) and qualitative data
▪ Uses a global benchmark that is updated annually
– as at January 2017:
▪ 10,567 managers
▪ 98,685 raters Hogan 360

6. © 2017 Peter Berry Consultancy Pty Ltd.

7. © 2017 Peter Berry Consultancy Pty Ltd.
Sample Information
Group Gender Age
Ratee
Female 31%
Male 69%
45 years
Range 24-72 years
Rater
Female 30%
Male 70%
45 years
Range 23-81 years
▪ 20,362 international managers from
both private and government
organisations who had participated in
the Hogan 360 multi-rater survey
between 2011 and 2015 as part of a
leadership development program.
▪ Raters classified as their manager,
peer, direct report or other stakeholder,
rated the manager.
▪ Each manager completed a self-
assessment.

8. © 2017 Peter Berry Consultancy Pty Ltd.
Data Analysis
50 items – 1 to 7 scale
▪ Missing values analysis / replacement techniques
(series mean)
▪ Standardised z-scores calculated for each quadrant and
competency
▪ ANOVA conducted on each quadrant and competency
by gender category
▪ Bonferroni adjustments made, p set at .05

9. © 2017 Peter Berry Consultancy Pty Ltd.
Data Analysis
Ranked Options
▪ Top 4 from a list of 26 Strengths
▪ Top 4 from a list of 26 Opportunities
Frequency count of those options most
frequently selected for both males and
females.

10. © 2017 Peter Berry Consultancy Pty Ltd.
Gender Category
Item Analysis Ranked Option Analysis
n
Male rated by Male 2,792
Male rated by Female 872
Female rated by Female 712
Female rated by Male 926
TOTAL 5,302
n
Male rated by Male 10,026
Male rated by Female 4,051
Female rated by Female 2,975
Female rated by Male 3,310
TOTAL 20,362

11. © 2017 Peter Berry Consultancy Pty Ltd.

12. © 2017 Peter Berry Consultancy Pty Ltd.
79%
78%
77%
75%
77%
76%
75%
73%
80%
79%
78%
77%
75%
74%
73%
72%
50%
60%
70%
80%
90%
100%
Female by Female Female by Male Male by Male Male by Female
Self Management Relationship Management
Working in the Business Working on the Business
Differences in Item Ratings by Hogan 360 Quadrant

13. © 2017 Peter Berry Consultancy Pty Ltd.
79%
78%
77%
75%
77%
76%
75%
73%
80%
79%
78%
77%
75%
74%
73%
72%
50%
60%
70%
80%
90%
100%
Female by Female Female by Male Male by Male Male by Female
Self Management Relationship Management
Working in the Business Working on the Business
Differences in Item Ratings by Hogan 360 Quadrant

14. © 2017 Peter Berry Consultancy Pty Ltd.
79%
78%
77%
75%
77%
76%
75%
73%
80%
79%
78%
77%
75%
74%
73%
72%
50%
60%
70%
80%
90%
100%
Female by Female Female by Male Male by Male Male by Female
Self Management Relationship Management
Working in the Business Working on the Business
Differences in Item Ratings by Hogan 360 Quadrant

15. © 2017 Peter Berry Consultancy Pty Ltd.
Differences in Ranked Strengths
▪ Women were seen as more organised, empathic and supportive.
▪ Men were seen as more steady and calm under pressure, innovative, visionary and
strategic.
▪ Women and men were both seen to:
▪ Work hard and have a strong work ethic
▪ Have solid technical ability, experience and knowledge
▪ Have a professional approach
▪ Have high ethical standards and integrity

16. © 2017 Peter Berry Consultancy Pty Ltd.
Differences in Ranked Opportunities
▪ Women needed to listen more, let others have their say and look at the big picture.
▪ Men needed to communicate better, improve their time management and
organisational skills, and improve their people and interpersonal skills.
▪ Women and men both needed to:
▪ Stop taking on too much and spreading selves too thin
▪ Challenge poor performance
▪ Delegate more
▪ Show leadership on issues

17. © 2017 Peter Berry Consultancy Pty Ltd.
Differences in Ranked Opportunities
where there was a Gender Interaction
▪ Compared to the views of men, women felt that both men and women needed to be
more available and visible in the workplace and share knowledge and resources more.
▪ Compared to the views of women, men felt that women needed to be more assertive,
look at the big picture more and acquire better job and/or industry knowledge.

18. © 2017 Peter Berry Consultancy Pty Ltd.
IMPLICATIONS

19. © 2017 Peter Berry Consultancy Pty Ltd.
Implications/Recommendations
Individual
▪ interpretation of 360 results for developmental or promotional purposes
HR Team
▪ introduce rater bias training/briefing pre-administration
▪ balance the gender mix in rater groups where a large number of raters
Organisation
▪ learning and development priorities
▪ acknowledge, understand and become aware of biases

20. © 2017 Peter Berry Consultancy Pty Ltd.
Conclusion
The size and breadth of this global sample suggests that
the findings should be generalisable across corporations
running multi-rater surveys as part of their leadership
development programs. As such, it is worth considering
how multi-rater information is collected and interpreted in
this context.

21. © 2017 Peter Berry Consultancy Pty Ltd.
QUESTIONS