1. 1
WIS
Web
Information
Systems
Disparate Impact Diminishes
Consumer Trust Even for
Advantaged Users
Tim Draws1,2, Zoltán Szlávik1,3, Benjamin Timmermans1,4,
Nava Tintarev5, Kush R. Varshney4, Michael Hind4
t.a.draws@tudelft.nl
https://timdraws.net
1IBM Center for Advanced Studies Benelux
2Delft University of Technology
3myTomorrows
4IBM Research
5Maastricht University

2. 2
WIS
Web
Information
Systems
Trusting Persuasive Technology (PT)
+
+
References: Nickel & Spahn (2012); Purpura, Schwanda, Williams, Stubler, & Sengers (2011); Sattarov & Nagel (2019)

3. 3
WIS
Web
Information
Systems
PT’s Disparate Impact?
• Machine learning research documents biases
and unfairness of many different kinds
– Usually results from biased data
– Disparate impact: disproportionally negative effect on
some user groups (e.g., women)
• Disparate impact in PT  consumer trust?
References: Barocas & Selbst (2016); Baeckström, Silvester, & Pownall (2018); Rossi (2019); Mullainathan, Noeth, & Schoar (2012); Ntoutsi et al. (2020); Toreini et al. (2020)

4. 4
WIS
Web
Information
Systems
Our Study
RQ1: Disparate impact  consumer trust?
RQ2: Advantaged ≠ disadvantaged users?
Method: online between-subjects user study
Use-case: personal finance PT

5. 5
WIS
Web
Information
Systems
Method: Procedure Step 1/2
All participants:
• “AI Advisor”
• General usage statistics
• Baseline measurements
– Trust, perceived personal benefit, willingness to use

6. 6
WIS
Web
Information
Systems
Method: Procedure Step 2/2
Depending on condition (1 out of 4):
• Gender-specific statistics
– No bias
– Little bias
– Strong bias
– Extreme bias
• Second round of measurements
– Trust, perceived personal benefit, willingness to use
20%
20%
10%
10%
10%
20%
25%
15%
30%
10%
35%
5%

7. 7
WIS
Web
Information
Systems
Method
• 489 participants
– 49% male, 51% female
– Randomly distributed over four conditions
• Per participant: difference between
baseline and second measurement
– Change in trust
– Change in perceived personal benefit
– Change in willingness to use

8. 8
WIS
Web
Information
Systems
Results
RQ1: Disparate impact  consumer trust?
– H1a: Disparate impact decreases consumer trust.
– H1b: Disparate impact decreases willingness to use.
−0.50
−0.25
0.00
Control Little
Bias
Strong
Bias
Extreme
Bias
Condition
Change
in
trust
Difference between conditions
χ2 = 25.06, p < 0.001
Difference between conditions
F = 6.906, p < 0.001

9. 9
WIS
Web
Information
Systems
No evidence for interaction
between condition and gender
F = 2.094, p = 0.096
−0.8
−0.4
0.0
Control Little
Bias
Strong
Bias
Extreme
Bias
Condition
Change
in
trust
gender
female
male
Results
RQ2: Advantaged ≠ disadvantaged users?
– H2a: Gender moderates disparate impact  personal benefit.
– H2b: Gender moderates disparate impact  trust (see H1a).
Interaction between
condition and gender
F = 8.525, p < 0.001
−2.0
−1.5
−1.0
−0.5
0.0
Control Little
Bias
Strong
Bias
Extreme
Bias
Condition
Change
in
perceived
personal
benefit
gender
female
male

10. 10
WIS
Web
Information
Systems
Discussion & Conclusion
• Disparate impact in PT can decrease
consumer trust and willingness to use
• Despite users recognizing their respective
(dis-)advantage, all users may lose trust in
systems they use due to disparate impact
t.a.draws@tudelft.nl
https://timdraws.net

11. 11
WIS
Web
Information
Systems
References
• Baeckström, Y., Silvester, J., Pownall, R.A.: Millionaire investors: financial advi- sors, attribution theory and gender differences.
Eur. J. Financ. 24(15), 1333–1349 (2018). https://doi.org/10.1080/1351847X.2018.1438301
• Barocas, Solon and Selbst, A.D.: Big data’s disparate impact. Calif. Law Rev. 104(671), 671–732 (2016)
• Mullainathan, S., Noeth, M., Schoar, A.: The Market for Financial Advice: An Audit Study. SSRN Electron. J. (2012).
https://doi.org/10.2139/ssrn.1572334
• Nickel, P., Spahn, A.: Trust, Discourse Ethics, and Persuasive Technology. In: Persuas. Technol. Des. Heal. Safety; 7th Int. Conf.
Persuas. Technol. 2012. pp. 37–40. Linköping University Electronic Press (2012)
• Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M.E., Rug- gieri, S., Turini, F., Papadopoulos, S., Krasanakis, E.,
Kompatsiaris, I., Kinder- Kurlanda, K., Wagner, C., Karimi, F., Fernandez, M., Alani, H., Berendt, B., Kruegel, T., Heinze, C.,
Broelemann, K., Kasneci, G., Tiropanis, T., Staab, S.: Bias in data-driven artificial intelligence systems—An introductory sur- vey.
Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 10(3), 1–14 (2020). https://doi.org/10.1002/widm.1356
• Purpura, S., Schwanda, V., Williams, K., Stubler, W., Sengers, P.: Fit4Life: The Design of a Persuasive Technology Promoting
Healthy Behavior and Ideal Weight. In: Proc. SIGCHI Conf. Hum. factors Comput. Syst. pp. 423–432 (2011)
• Rossi, F.: Building trust in artificial intelligence. J. Int. Aff. 72(1), 127–133 (2019)
• Sattarov, F., Nagel, S.: Building trust in persuasive gerontechnology: User- centric and institution-centric approaches.
Gerontechnology 18(1), 1–14 (2019). https://doi.org/10.4017/gt.2019.18.1.001.00
• Toreini, E., Aitken, M., Coopamootoo, K., Elliott, K., Zelaya, C.G., van Moorsel, A.: The relationship between trust in AI and
trustworthy machine learning tech- nologies. FAT* 2020 - Proc. 2020 Conf. Fairness, Accountability, Transpar. pp. 272–283
(2020). https://doi.org/10.1145/3351095.3372834