1) Automated decision-making systems are opaque and can encode hidden biases, but human decisions also exhibit bias. It is mathematically impossible for algorithms to achieve both equal predictive value and equal false positive rates across groups with different base rates. 2) Studies show discrimination in online marketplaces, where requests from users with black-sounding names receive fewer responses from hosts on AirBnB and ads featuring arrest records appear more for searches with black-sounding names on Google. 3) Gerrymandering of electoral districts to benefit one party can violate equal treatment of voters, but defining and preventing partisan bias through redistricting is challenging, as natural

1. Algorithmic Bias and Fairness
1

2. Automated Decision Making: Pros
 Handles large volumes of data (Google search, airline
reservations, online markets, ..)
 Avoids certain kinds of bias
 Parole judges being more lenient after a meal
 Making hiring decisions based on the name of the person
 Subjectivity in evaluations of papers, music, teaching, etc.
 Human judgment in NYC stop and frisk policy
 4.4 M were stopped between 2004-2012
 88% of them led to no further action
 83% of the people stopped were Black or Hispanic – only
about half in the population are.
2

3. Complex and Opaque Decisions
 Hard to understand and make sense of
 Values, biases and potential discrimination built in
 The code is opaque and often trade secret
 Facebook’s newsfeed algorithm, recidivism algorithms, genetic testing
3

4. Gatekeeping Function
 Decide what gets attention, what is
published, and what is censored
 Google’s search results of geopolitical
queries might depend on location, e.g.,
different maps of Pakistan or India.
 Learning algorithms that make hiring
decisions.
 Pattern: Low commute time favors
low turnover
 Policy: Don’t hire from far off places
with bad public transportation
 Impact: People from poor and far off
neighborhoods may not be hired
4

5. Subjective Decision Making
 Algorithms to understand and translate language, drive cars, pilot
planes, and diagnose diseases.
 No right answer, but judgment and values.
 Detecting and removing terrorist content on the social networks.
 The definition of important words such as `terrorist’ and ‘extreme
content’ are controversial
 The scale makes it difficult for manual intervention.
 Algorithmic decisions may not be as good as people
5

6. Machine Learning
 Programs might be using protected attributes such as race and
gender to make predictions
 Even if the protected attributes are not used, they could be using
other “proxy” attributes which will have the same effect, e.g., zip
code.
 Recommendations based on earlier actions might create
bubbles, eg. Detecting trends on Twitter.
 Example: Predictive policing
 Predicting the neighborhoods most likely to be involved in
future crime based on crime statistics
 Rational but may be indistinguishable from racial profiling
 More police in the neighborhood lead to more arrests.
 Could lead to positive feedback loops and become a self-
fulfilling prophecy.
6

7. Data Privacy
 Who owns your browser data?
 Can your insurance company get access to your grocery list or
peek into your fridge?
 Can hospitals get access to consumer data to predict who is
going to get sick?
 Can your employer access your grades?
7

8. Transparency and Notification
 If the algorithm is opaque, there is no understanding or trust in
the program, e.g., medical decisions, hiring decisions
 Google’s search algorithm judged not demonstrably anti-
competitive in the US
 European Commission has successfully pursued an anti-
trust investigation
 Many points of trust: algorithm, input, learning data, control
surfaces, assumptions and models the algorithm uses, etc.
 Complete transparency makes it vulnerable to hacking. Does
not guarantee scrutiny.
 Consumers might demand the right to be notified when using
their information or demand excluding their personal information
8

9. Algorithmic Accountability
 How search engines censor violent/sexual search terms
 What influences Facebook’s newsfeed program or Google’s
advertisements
 Need causal explanations that link our digital experience with
data they are based upon
9

10. Government Regulation
 Destabilizing effect of high-speed trading systems led to
demands of transparency of these algorithms and ability to
modify them
 Should search algorithms be forced to follow some “search
neutrality rules”?
 Requires public officials to have access to the program and
modify it in the interest of public.
 There is no one right answer to the queries Google handles,
which makes it difficult.
10

11. Case Study: Recidivism Assessment
 COMPAS is a program to assess the recidivism of the prisoners –
their propensity to commit a crime in 3 years after the release.
 Propublica analyzed data of 10,000 prisoners in a Florida county
 There is one such table for Blacks and another for Whites. Θ is
chosen for each group separately.
 False Positive Rate 𝐹𝑃𝑅 =
𝐹𝑃
(𝐹𝑃+𝑇𝑁)
 Positive Predictive Value 𝑃𝑃𝑉 =
𝑇𝑃
(𝐹𝑃+𝑇𝑃)
 Propublica: FPR(Blacks) = 2 FPR(Whites)
 NorthPointe: PPV(Blacks) = PPV(Whites)
11
Recidivism Score ≤ θ Score > θ
False TN FP
True FN TP

12. Conflicting Demands on Fairness
12
 Red = False positives, FP; Blue= True positives TP
 Assumptions:
 Prevalence or rate of recidivism is higher for one group (say blacks)
 Positive Predictive Value 𝑃𝑃𝑉 =
𝑇𝑃
(𝐹𝑃+𝑇𝑃)
= same for both
 False Positive Rate 𝐹𝑃𝑅 =
𝐹𝑃
(𝐹𝑃+𝑇𝑁)
= higher for blacks.
White Black
Recidivism
=True
Prediction = Positive

13. Fairness of Recidivism Scores
Recidivism LowScore HighScore
False TN FP
True FN TP
13
 False Positive Rate 𝐹𝑃𝑅 =
𝐹𝑃
(𝐹𝑃+𝑇𝑁)
 False Negative Rate 𝐹𝑁𝑅 =
𝐹𝑁
(𝐹𝑁+𝑇𝑃)
 Prevalence p =
(𝐹𝑁+𝑇𝑃)
(𝐹𝑁+𝐹𝑃+𝑇𝑁+𝑇𝑃)
𝐹𝑃𝑅 =
𝑝
1 − 𝑝
1 − 𝑃𝑃𝑉
𝑃𝑃𝑉
(1 − FNR)
 Conclusion: If the prevalence p is different for two classes and
PPVs are the same then FNR or FPR or both must be different.
 The differences in FPR and FNR lead to disparate impacts –
more penalty for Blacks in both recidivism groups than Whites.

14. Summary
 It is mathematically impossible to achieve both equal PPV and
equal FPR across different groups.
 The differences in FPR and FNR persist in subgroups of
defendants.
 However, evidence suggests that data-driven risk assessment
tools (in medicine) are more accurate than human judgment.
 Human driven decisions are themselves prone to exhibiting
racial bias, eg, paroles, sentencing, stop and frisk, arrests, etc.
14

15. Case Study: Online Market Places
 How do we ensure that the sellers are honest about the quality of their
goods?
 Study: In early 2000’s eBay merchants misrepresented the quality
of their sports trading cards
 Problem largely solved by the feedback and reputation systems
 New development: demand for more information
 Study (2012): Subjects rated trustworthiness of potential borrowers
from photographs of them.
 People who looked trustworthy are more likely to get loans
 They are also more likely to repay their loans.
 More information leads to more freedom
 People can now choose whom to do business with based on looks
 A growing body of evidence suggests this leads to discrimination
15

16. Discrimination in Online Markets
 Air-BnB Study: 20 profiles sent to 6400 hosts
 The profiles are identical except 10 of them have names common to
white people and the rest to blacks
 Result: Requests for black-sounding names were 16% less
successful
 Discrimination was pervasive. Most of the people who rejected
never hosted a black guest.
 Other areas of discrimination: credit, labor markets, housing.
 Discrimination also occurs in algorithmic decisions.
 Searches for black sounding names on Google were more likely to
bring up ads about arrest records.
 Why?
 Learning from the past search data.
16

17. Principles and Recommendations
 Don’t Ignore potential discrimination
 Collect good data including race and gender stats
 Do regular reports and occasional audits
 Public disclosure of discrimination-related data
 Keep an experimental mindset to evaluate different design options
 Airbnb withholding host pictures from its ads
17

18. Design Decisions
 Control the information, its timing and salience
 When can you see the picture of Uber driver?
 Increase automation and charge for control
 Make instant book the default on AirBnB and charge a fee if the
host wants to approve the guest first
 Prioritize discrimination issues
 Remind the host about anti-discrimination policies at the time of
the transaction
 Make algorithms discrimination-aware
 Set explicit objectives: want my black and white customers to
be rejected at the same rate
18

19. Virtual Screens
 In mid 60’s less than 10% of the big 5 orchestras were women
 Moved away from face-to-face to behind-the-screen auditions
 Success rate of female musicians increased by 160%
 The online market allows virtual screens between buyers and sellers,
between employers and employees.
19

20. Case Study: Gerrymandering
20
 Background
 In the US, states are divided into
congressional districts every 10 years
 Each state is divided into precincts of
equal population
 The precincts are clustered into
congressional districts
 Whoever wins the majority of precincts
in the district wins that district
 Gerrymandering (named after Elbridge
Gerry) refers to manipulation of districts to
influence the outcome of an election.
 Packing: Pack most of the voters in the
opposing side into a small number of
districts
 Cracking: Split the voters of the
opposing side into several districts
where they are minority The original political cartoon on
Gerrymandered map of Essex
County Massachusetts, 1812

21. Impact of gerrymandering
 Racial gerrymandering that intentionally reduces minority
representation was ruled illegal in 1960.
 In 1980, voting rights act was amended to make states redraw
maps if they had racially discriminatory impact.
 Partisan gerrymandering has not been ruled illegal
 When republicans drew the maps (17 states) they won about 53 percent of
the vote and 72 percent of the seats.
 When democrats drew the maps (6 states), they won about 56 percent of
the vote and 71 percent of the seats.
 Proportional representation: Each party receives roughly the
same percent of votes as it wins the percent of the seats
 Wasted votes: Votes cast to the losing side or above the
minimum the winner needed to win.
 Efficiency gap: The difference in the wasted votes / total wasted.
It is intended to measure partisan bias.
21

22. Wisconsin’s redistricting in 2011
22
 Wisconsin’s Republican-led redistricting was struck
down by a 3 judge panel. It was heard by the supreme
court on October 3. A decision is pending.
 The arguments of the plaintiffs:
 Big efficiency gap indicates bias especially if it is
persistent. Wisconsin’s gap is the biggest ever.
 It violates voters’ right to equal treatment
 It discriminates against their views (first
amendment argument)
 Arguments of the defendants:
 Efficiency gaps arise naturally, e.g., when
democrats pack into cities
 Courts should stay out of it. States can appoint
independent commissions if they are concerned
 Justice Kennedy’s vote is probably going to be
decisive.

23. Discussion
Suppose you are heading an independent commission to
recommend a fair redistricting approach.
 How do you define fair redistricting? Why?
 How would you go about implementing your recommendation?
 What role do computer algorithms play?
23