1. Cool vs. Creepy: Ethics in Data Science
Cathy Cooper
3 February 2017

2. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 2Public
(Image of a hand with a credit card + JetBlue plane to Orlando?)
xxx
Planes, meals, theme
parks and … credit
cards?

3. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 3Public
Connection to ethics & data science
DATA SCIENCE
Data Analysis Applications
EXPERIENCE
Transparency Permission

4. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 4Public
Data, Analysis and Applications: Biased Data
Google tested machine learning analysis on their News data to try to find relationships between
concepts.
One of the examples was the following “gender + profession” comparison.
man + computer programmer vs. woman + _________
__________________ = Homemaker
Why? Because the system was trained with an inherently biased source.

5. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 5Public
Data, Analysis and Applications: Anonymized Data
How many time & location records does it
take to link cell phone usage data to an
individual?
Only four

6. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 6Public
Data, Analysis and Applications: Context
A financial institution was looking to predict customer churn.
Who did they identify as the prime target for their retention offers?
Spouses getting their finances in order before filing for divorce.
A test was conducted using just social media sentiment analysis to predict changes in US
unemployment rate.
It did predict an unemployment drop – but it was wrong. Why?
Relying only on the tool – and poor timing: Steve Jobs had just died.

7. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 7Public
Data, Analysis and Applications: Data/Predictions Change Behavior
and Vice Versa
GPS
Flu projection

8. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 8Public
Data, Analysis and Applications: Human Intervention Required
“… people have
too much trust in
data to be
intrinsically
objective, even
though it is in fact
only as good as
the human
processes that
collected it.” --
Cathy O’Neill
“Humans will need to check on the
outcomes and see if the models and the
algorithms and the rules are performing
as intended, and then intervene if they
don’t.” – Tom Davenport

9. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 9Public
Transparency: Try it for yourself
University of Cambridge The Psychometrics Centre uses your Facebook profile and activity for:
► YouAreWhatYouLike describe openness to new ideas, extraversion and introversion, your warmth
or competitiveness, and other personality traits.
► Apply Magic Sauce, predicts your politics, relationship status, sexual orientation, gender, and
more.
Pennebaker Conglomerate’s
► AnalyzeWords leverages linguistics to discover the personality you (other others) portray on
Twitter. You can analyze anyone, not just yourself.

10. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 10Public
Transparency: How It Can Work
Banks have to prove:
►their algorithms are not based on variables like race & gender
►their models don’t focus on patterns that disfavor specific demographic groups
and
they have to allow outside data scientists to assess their models for code or data that might have a
discriminatory effect.
Consider what it will takes to build transparency into AI system design: human access and
action plan

11. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 11Public
Permission/Fairness
Challenge and opportunity:
► Additional uses for previously
collected/aggregated data in
new types of analysis,
applications and businesses
► Coming legal and regulatory
requirements to audit and
correct algorithm-based
decision making

12. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 12Public
In closing…
“… algorithms have to be designed with fairness and legality in
mind, with standards that are understandable to everyone, from
the business leader to the people being scored.” – Cathy O’Neill,
“Unmasking Unconscious Bias in Algorithms”, The Digitalist,
data scientist and author Weapons of Math Destruction
“…a fundamental rethinking and a careful approach to software creation itself is needed.
Extra care has to be put into training the people who create the systems, and into
incorporating research into the machine learning algorithms so you don’t accidentally create
more bias. The system has to be re-trained to think differently.” – Yvonne Baur, “Is Machine
Learning Sexist?”, Techcrunch, and Head of Predictive Analytics/Machine Learning, SAP
SuccessFactors,

13. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 13Public
Additional Readings and Resources on Ethics & Data Science
http://www.digitalistmag.com/digital-supply-
networks/2017/01/16/cathy-oneil-
unmasking-unconscious-bias-in-algorithms-
04839140
http://www.digitalistmag.com/executive-
research/how-ai-can-end-bias
http://www.digitalistmag.com/executive-
research/an-ai-shares-my-office
http://www.digitalistmag.com/digital-
economy/digital-
futures/2017/01/23/machine-learning-real-
business-intelligence-04830225
http://www.digitalistmag.com/executive-
research/empathy-the-killer-app-for-
artificial-intelligence
http://www.digitaltrends.com/cool-
tech/women-in-artificial-intelligence/
https://techcrunch.com/2016/10/11/is-
machine-learning-sexist/
https://open.sap.com/courses/ds1
http://www.slate.com/articles/technology/fut
ure_tense/2014/10/youarewhatyoulike_find_
out_what_algorithms_can_tell_about_you_b
ased_on_your.html
http://searchcio.techtarget.com/opinion/Dat
a-products-introduce-ethical-dilemmas-for-
data-scientists
http://searchcio.techtarget.com/opinion/Big-
data-ethics-Why-the-CIO-needs-to-get-
involved
http://searchcloudapplications.techtarget.co
m/feature/Big-data-collection-efforts-spark-
an-information-ethics-debate
https://arstechnica.com/science/2015/11/ne
w-flu-tracker-uses-google-search-data-
better-than-google/
https://gigaom.com/2013/10/09/the-upside-
of-prism-at-least-were-talking-about-data-
privacy-or-lack-the-thereof/
http://nymag.com/thecut/2016/09/cathy-
oneils-weapons-of-math-destruction-math-
is-biased.html?mid=twitter-share-thecut
https://www.bloomberg.com/news/articles/2
016-06-23/artificial-intelligence-has-a-sea-of-
dudes-problem
http://boingboing.net/2016/01/06/weapons-
of-math-destruction-h.html
http://fortune.com/2015/12/04/ethics-
training-for-data-scientists/
https://gigaom.com/2013/03/25/why-the-
collision-of-big-data-and-privacy-will-
require-a-new-realpolitik/
http://www.nature.com/articles/srep01376
http://www.slate.com/articles/technology/fut
ure_tense/2016/02/how_to_bring_better_ethi
cs_to_data_science.htmlhttp://searchcio.tec
htarget.com/opinion/Big-data-bad-analytics
http://www.slate.com/articles/technology/fut
ure_tense/2016/02/how_to_bring_better_ethi
cs_to_data_science.html
https://open.sap.com/courses/ml1
https://motherboard.vice.com/en_us/article/
big-data-cambridge-analytica-brexit-trump

14. Thank you
Contact information:
Cathy Cooper
Sr. Director, Big Data Analytics
cathy.cooper@sap.com
212 653 9555
@catcooper1