This document summarizes a research study on the impact of algorithmic bias on decision-making in human-machine collaboration. The study finds that when human decision-makers collaborate with an artificially intelligent system that provides biased predictions, it leads the human-machine teams to make more biased decisions and lower organizational profits over time compared to teams using an unbiased AI system. However, the researchers also find that human decision-makers are able to implicitly recognize the AI's bias over repeated interactions and adapt their decision-making to improve organizational profits and reduce decision bias, especially in later periods. The researchers conclude that exposing human decision-makers to algorithmic bias mitigation over time can help decrease bias in human-machine collaborative decision-making.

1. Human-Machine Collaboration in Organizations: Impact
of Algorithm Bias on Decision Bias & Perceived Fairness
Anh Luonga
, Nanda Kumarb
, Karl R. Langb
a
Warwick Business School, University of Warwick
b
Zicklin School of Business, Baruch College, City University of New York
Workshop on Information Systems and Economics (WISE)
December 16, 2021
Austin, TX

2. 2
Research Context:
Fairness in Human-Machine Collaboration

3. Research Context:
Fairness in Human-Machine Collaboration
(Biased) AI
(Biased)
HUMANS
(Biased/Fair)
Decisions?
Human-Machine Collaborative Decision-Making
Expert-Driven Decision-Making
Fully Automated Decision-Making
Feedback Loop
3

4. Related Literature:
Focus on Machine Bias, not much on Human-Machine Bias
How our study is different:
(1) Human-Machine Collaboration
(2) Differential Impacts of AI Bias Levels on:
a. Decision Bias of Human-Machine Teams
b. Perceived Fairness of Human DMs
(3) Repeated Interaction over Time
(4) Aligned Incentive Structure
4
Extant Literature:
● Rhue, ICIS ‘19
● Vaccaro & Waldo, CACM 2019
● Khademi et al., ACM WWW ‘19
● Xu et al., ACM SIGKDD ‘20
● Lu et al., ICIS ‘19
● Kasy & Abebe, ACM FAccT ‘20
● Adomavicius & Yang, WP 2019

5. Research Questions
In the human-machine collaborative decision-making context:
1. What is the effect of AI prediction bias on human-machine teams’ decision bias and organizational
profit?
2. What is the effect of AI prediction bias on human decision-makers’ perceived fairness?
3. What is the effect of exposure between human and machines on organizational profit?
5

6. Preview of Experiment and Findings
Experiment:
● Task: Review 100 consumer loan
applications
○ Real world data: Lending Club
● Manipulated algorithmic
predictions: Bias vs No Bias
● 10 repeated decision rounds
6
Findings:
● DMs working with unbiased AI > DMs with biased AI
● DMs working with biased AI over time
○ Adapt more to biased predictions
○ Implicitly recognize the bias
○ Outperform biased AI alone during later periods
■ Improve org profit
■ Reduce decision bias

7. Theoretical Framework:
Terminology
● Bias (Group)
○ Objective, mathematical measure of differences in outcomes (disparity) among groups
● Fairness (Group)
○ Subjective, perceptual measure of whether the differences in outcomes among groups are considered fair
■ Different stakeholders have different views
○ Philosophy / ethics
■ Perceptual fairness depends on personal values and social norms regarding desirability of equal
outcomes

8. Theoretical Framework:
Measuring Bias
● Binary Classification Problem
○ Loan application review, Hiring decisions, Medical diagnoses, Predicting recidivism
○ Etc.
● Predictive Parity among groups
● Error Rate Balance among groups
○ False Negative (Type II Error) Rate Balance
○ False Positive (Type I Error) Rate Balance
● Accuracy Equity among groups

9. Theoretical Framework:
Measuring Fairness (Perceived)
● Fairness measured through perceptions (subjective) (Cowgill & Tucker, 2020)
● Different stakeholders have different, or contrasting views of perceived fairness
○ Babcock et al., AER 1995; Lee & Baykal, CSCW 2017; Konow, Soc Choice Welf 2009
○ Occurs even when using “objective”, mathematical measures
■ Northpointe/COMPASS
■ 21 Fairness Definitions and Their Politics, Narayanan ACM FAccT ‘18

10. Research Model
10
H1c (-)
H1a, b (+)
Exposure
Organizational
Profit
H-M Teams’ Bias
False Negative Imbalance
False Positive Imbalance
AI
Prediction Bias
H2 (-)
HDMs’ Perceived
Fairness
H3 (+)
Supported
Loans’ NPVs
Post-Experiment
Survey Question
H-M Teams: Human-Machine Teams HDMs: Human Decision-Makers

11. Methodology: Experimental Economics
11
Research Method Experimental Economics
Synchronous online Zoom experiments with financial incentives
Decision-making Platform oTree experimental platform
Participants Undergraduate Business Majors, SONA pool
Duration of Each Session 60–75 Minutes
Decision-Making Task Review 100 consumer-loan applications, split over 10 rounds
Payment Cash, USD $5–20 (Performance-Based)

12. Experimental Procedures
1. Zoom Meeting Room Welcome
2. Pre-Experiment Survey (competencies, personalities, cognitive styles)
3. Experiment Instructions Slides Read Aloud
a. Customized for each condition
b. Explains task & financial reward scheme (pay-off matrix)
4. Data Glossary URL & Slide
5. Experimental Task (Loan Applications Review)
6. Post-Experiment Survey (perceptions, demographic)
7. Cash Payment & Exit
12

13. Experimental Design
Prediction Bias
No Bias Bias
Decision
Time
Regular 28 participants 35 participants
Extended
(Pilot)
NA
9 participants
(Pilot)
13

14. Treatment Manipulation: (Context-Free) Prediction Bias
● Large scale, historic data -- Lending Club
● Manipulated AI predictions:
○ Bias Treatment: discriminate against Purple, favor Orange
■ FPR: Orange (0.2) < Purple (0.6)
■ FNR: Orange (0.6) > Purple (0.2)
○ No Bias Treatment: Treat Purple & Orange the same
■ FPR and FNR: Orange = Purple = 0.2
14

15. 15
Experimental Interface (Partial View): Initial Decision Page

16. 16
Experimental Interface (Partial View): AI Predictions and Final Decision Page

17. 17
. . .
Experimental Interface (Partial View): Feedback Page

18. Payoff Matrix
● Aligned Incentive Structure with assumed conservative lending strategy
Approve Reject
Loan
Performed Well + $ 0.2 $ 0
Loan
Performed Poorly - $ 0.4 + $ 0.2
18

19. Model 0 Model 1 Model 2
FPIPur - Or
FPIPur - Or
FPIPur - Or
AI Prediction Bias 0.081 ** 0.081 ** 0.080 **
Controls No Periods
Periods &
Other Controls
R Sq 0.008 0.342 0.347
19
Results: Impact of AI Bias on Human-Machine Teams’ Bias
(False Positive ImbalancePur - Or
)
Controls:
Quant Competence,
CollegeUpperLevel,
EthnMinority,
Male,
Intuition/Analytic (Cognitive Style)
**: < 0.05
H1a Supported

20. Model 0 Model 1 Model 2
FNIOr - Pur
FNIOr - Pur
FNIOr - Pur
AI Prediction Bias 0.104 *** 0.104 **** 0.104 ***
Controls No Periods
Periods & Other
Controls
R Sq 0.025 0.512 0.532
20
Results: Impact of AI Bias on Human-Machine Teams’ Bias
(False Negative ImbalanceOr - Pur
)
Controls:
Quant Competence,
CollegeUpperLevel,
EthnMinority,
Male,
Intuition/Analytic (Cognitive Style)
***: < 0.01; ****: < 0.001
H1b Supported

21. Model 0 Model 1 Model 2
Org. Profit
($)
Org. Profit
($)
Org. Profit
($)
AI Prediction Bias -5,764.93 **** -5,764.93 **** -6,035.51 ****
Controls No Periods
Periods &
Other Controls
R Sq 0.025 0.512 0.532
21
Results: Impact of AI Bias on Organizational Profit
Controls:
Quant Competence,
CollegeUpperLevel,
EthnMinority,
Male,
Intuition/Analytic (Cognitive Style)
***: < 0.01; ****: < 0.001
H1c Supported

22. Results:
Impact of AI Bias on DMs’ Perceived Fairness
(AI Predictions Overall)
m = 5.3
m = 4.8
m = 5

23. Results:
Impact of AI Bias on DMs’ Perceived Fairness
(AI’s Treatment of Borrowers)
m = 4.8
m = 5.1
m = 4.1

24. Results:
Impact of AI Bias on DMs’ Perceived Fairness
(Treatment of Borrowers - Composite Index)
m = 4.9
m = 5 m = 5

25. Results:
Impact of AI Bias on DMs’ Perceived Fairness
(DMs’ Outcome Fairness)
m = 12.6
m = 14.8
m = 13.6

26. Results:
Impact of AI Bias on DMs’ Perceived Fairness
(DMs’ Financial Reward - Composite Index)
m = 5.6 m = 5.7 m = 5.5

27. Additional Analysis:
Impact of AI Bias on DMs’ Trust in AI
m = 13.3
m = 11.3
m = 10.8
Human DMs seem to recognize
AI’s bias implicitly:
● trusting it significantly less
(than those working with
unbiased AI)

28. H3 Supported
Results:
Impact of Exposure on Organizational Outcomes
(Org Profit Diff Human Machine Teams vs. Machine)

29. Take-aways
● Human DMs working with Biased AI Predictions
○ Adapt to improve over time
○ Implicitly recognize bias
○ Perform better than Biased AI alone (in later periods)
■ DMs do not exhibit “borg” behavior, unlike Fügener et al. MISQ 2021
● Evidence that AI Bias Mitigation
○ does lead to decrease in bias in Human-Machine collaboration
29

30. THANK YOU