Ethics of Analytics and Machine Learning

Ethics of Data Science
Issues Surfaced by IEEE P7003 WG*
Mark Underwood @knowlengr | Views my own | Creative Commons | *Thru 2018-07

Disclaimers
 My views only
 Not view of my employer
 Not view of IEEE or IEEE SA
 Not view of the IEEE P7003 WG
 P7003 standards work still early stage
Mark Underwood @knowlengr | Views my own | Creative Commons | *Thru 2018-07 | v1.6

“Minority Report” (2002)
 The “PreCogs” have landed
 Proprietary predictive models already deployed in several
states for
 Law enforcement
 Child welfare
 “Pockets of poverty” identification
 Educational / teacher assessment
 Credit: Philip K. Dick (1956)

Storm Clouds
 Broader social trends impacting application contexts
 ML weak transparency
 Some analytics practitioners may have weak statistical background
 Algorithms may not have faced usability or reproducibility tests
 Computation on encrypted data is still primitive and/or slow
 Anonymization practices may not take big data variety into account
 Lightweight practitioner training

Where Data Science Does Matter
A smattering of impact areas

Use Case Clusters
 Credit worthiness (Offers for balance transfers not extended to minority groups)
 Sales & marketing analytics (Some profitable prospects will be ignored)
 Autonomous vehicles (Fatality, injury accidents)
 Drug trials (Some patients will be rejected, even if potentially life-saving)
 Recommendation engines (Users not given opportunity to fine-tune – You, Netflix)
 Maintenance forecasting (Forecasts may not include loss of life risks)
 Operations Management (Cause cost overrun while managing AWS cost / performance)
 Cybersecurity (False positives could suggest you are an insider threat)

Solving Poverty through Data Science
It’s Magic!
https://www.marketplace.org/shows/marketplace-morning-report 2018-07-30

IEEE Initiative:
Value-Sensitive Design
“. . . to refine ethical system requirements in systems and software life cycles.”

IEEE P7000 WG: Marquis Group Charter
“Scope: The standard establishes a process model by which engineers and technologists can address
ethical consideration throughout the various stages of system initiation, analysis and design.
Expected process requirements include management and engineering view of new IT product
development, computer ethics and IT system design, value-sensitive design, and stakeholder
involvement in ethical IT system design. . .. The purpose of this standard is to enable the pragmatic
application of this type of Value-Based System Design methodology which demonstrates that
conceptual analysis of values and an extensive feasibility analysis can help to refine ethical system
requirements in systems and software life cycles.”

Related IEEE P70nn Groups
 IEEE P7000 Ethical Systems Design
 IEEE P7001 Transparency of Autonomous Systems
 IEEE P7002 Data Privacy Process
 IEEE P7003 Algorithmic Bias Considerations
 IEEE P7004 Standard for Child and Student Data Governance
 IEEE P7005 Standard for Transparent Employer Data Governance
 IEEE P7006 Standard for Personal AI Agent
 IEEE P7007 Ontological Standard for Ethically Driven Robotics and Automation Systems
 IEEE P7008 -Standard for Ethically Driven Nudging for Robotic, Intelligent and Autonomous Systems
 IEEE P7009 Standard for Fail-Safe Design of Autonomous and Semi-Autonomous Systems
 IEEE P7010 Wellbeing Metrics Standard for Ethical Artificial Intelligence and Autonomous Systems
 IEEE P7011 SSIE Standard for Trustworthiness of News Media
 IEEE P7012 SSIE Machine Readable Personal Privacy Terms
 IEEE P7013 Facial Analysis

IEEE P7003 Algorithmic Bias Considerations
“This standard is designed to provide individuals or organizations creating algorithms, largely in
regards to autonomous or intelligent systems, certification oriented methodologies to provide
clearly articulated accountability and clarity around how algorithms are targeting, assessing and
influencing the users and stakeholders of said algorithm. Certification under this standard will
allow algorithm creators to communicate to users, and regulatory authorities, that up-to-date
best practices were used in the design, testing and evaluation of the algorithm to avoid
unjustified differential impact on users.”
IEEE PAR https://standards.ieee.org/develop/project/7003.html

Who is IEEE SA?
Why care what it does?
• Affordable, volunteer-driven, int’l
• IEEE SA members voting rights
• Collaboration with ISO, NIST
• Key standards include ethernet

Broad Social Issues
Cross-cutting Concerns for all Technologists

Professional Pride, Public Disillusionment
Broader acceptance within IT & Evidence-based Practices
 Growth of data science inside many professions (R, Python)
 Extraordinary explosion of OSS tooling
 Big Data, ML, Real Time
 Watson, AlphaGo, Alexa “AI” (Gee Whiz factor)
Public Perspective
 “2017 was the year we fell out of love with algorithms.”
 Cambridge Analytica, Equifax

“When we fell out of love with algorithms.”

Ethical issues Already in Play
 Sustainability
 Environment
 Climate Change (*data center power consumption)
 Bias concerns in gender, race, free speech
 Social media technology responsibility
 As propaganda platforms
 Excessive use of cell phones by children: ADHD?
 Weakened critical thinking, F2F social skills (Sherry Turkle Reclaiming Conversation 2015)

Some Related ISO Standards
 ISO 26000 “ISO 26000 provides guidance on how businesses and organizations can operate in a socially responsible
way. This means acting in an ethical and transparent way that contributes to the health and welfare of society”
 ISO 27500:2016 is intended for executive board members and policy makers of all types of organizations (whether
large or small) in the private, public and non-profit sectors. It describes the values and beliefs that make an
organization human-centred, the significant business benefits that can be achieved, and explains the risks for the
organization of not being human-centered
 ISO 37101:2016 establishes requirements for a management system for sustainable development in communities,
including cities, using a holistic approach, with a view to ensuring consistency with the sustainable development
policy of communities.
 ISO/IEC JTC 1/SC 39 Standardization related to the intersection of resource efficiency and IT which supports
environmentally and economically viable development, application, operation and management aspects.
 ISO/IEC 31010:2009 – Risk Management – Risk Assessment Techniques

What do you know about these
standards?
To date, a bit hidden among the blockchain and TensorFlow conversations.

Algorithmic Bias in Data Science
*Including finance

Linkage to Privacy, Surveillance, Distrust
Algorithms are bad because they:
 Use data without our knowledge
 Are based on incorrect or misleading knowledge about us
 Are not accountable to individual citizens
 Are used by governments to spy on citizens
 Support drone warfare
 Are built by specialists who do what they are told without asking questions
 Represent a trend to automate jobs out of existence
 Are built by big companies with no public accountability

“Bias is the New Breach”
“Researchers from MIT and Stanford University
tested three commercially released facial-analysis
programs from major technology companies and
will present findings that the software contains
clear skin-type and gender biases. Facial
recognition programs are good at recognizing
white males but fail embarrassingly with females
especially the darker the skin tone. The news
broke last week but will be presented in full at the
upcoming Conference on Fairness, Accountability,
and Transparency.“
https://www.cio.com/article/3256272/artificial-intelligence/in-the-ai-revolution-bias-is-the-new-
breach-how-cios-must-manage-risk.html

Decisions vs. Decision Support:
Application Areas
Human-Computer Interactions in Decision-making

Undermining Specialists*
“The threat the electronic health records
and machine learning post for physicians’
clinical judgment – and their well-being.” – NYT
2018-05-16
“’Food poisoning’ was diagnosed because
the strangulated hernia in the groin was
overlooked, or patients were sent to the
catheterization lab for chest pain because
no one saw the shingles rash on the left
chest.”
*Or adversely changing specialist behavior.

“Rote Decision-Making”
“The authors, both emergency room physicians at
Brigham and Women’s Hospital in Boston, do a fine job
of sorting through most of the serious problems in
American medicine today, including the costs, over-
testing, overprescribing, overlitigation and general
depersonalization. All are caused at least in part, they
argue, by the increasing use of algorithms in medical
care.” -NYT 2018-04-01

Facial Recognition for Law Enforcement
 “AMZ touts its Rekognition facial recognition system as ‘simple and easy to
use,’ encouraging customers to ‘detect, analyze, and compare faces for a
wide variety of user verification, people counting, and public safety use
cases.’ And yet, in a study released Thursday by the American Civil Liberties
Union, the technology managed to confuse photos of 28 members of
Congress with publicly available mug shots. Given that Amazon actively
markets Rekognition to law enforcement agencies across the US, that’s
simply not good enough. The ACLU study also illustrated the racial bias that
plagues facial recognition today. ‘Nearly 40 percent of Rekognition’s false
matches in our test were of people of color, even though they make up only
20 percent of Congress,’ wrote ACLU attorney Jacob Snow. ‘People of color
are already disproportionately harmed by police practices, and it’s easy to
see how Rekognition could exacerbate that.’“ -Wired 2018-07-26

“Family” Impacts
“Charges of faulty forecasts have accompanied the
emergence of predictive analytics into public policy.
And when it comes to criminal justice, where
analytics are now entrenched as a tool for judges
and parole boards, even larger complaints have
arisen about the secrecy surrounding the workings
of the algorithms themselves — most of which are
developed, marketed and closely guarded by private
firms. That’s a chief objection lodged against two
Florida companies: Eckerd Connects, a nonprofit,
and its for-profit partner, MindShare Technology.” –
NYT “Can an algorithm tell when kids are in danger?” 2018-01-02

Lawsuit over Teacher Evaluation Algorithm
 Value-added measures for teacher evaluation, called the Education Value-
Added Assessment System, or EVAAS, in Houston, is a statistical method
that uses a student’s performance on prior standardized tests to predict
academic growth in the current year. This methodology—derided as
deeply flawed, unfair and incomprehensible—was used to make decisions
about teacher evaluation, bonuses and termination. It uses a secret
computer program based on an inexplicable algorithm (above).
 In May 2014, seven Houston teachers and the Houston Federation of
Teachers brought an unprecedented federal lawsuit to end the policy,
saying it reduced education to a test score, didn’t help improve teaching or
learning, and ruined teachers’ careers when they were incorrectly
terminated. Neither HISD nor its contractor allowed teachers access to the
data or computer algorithms so that they could test or challenge the
legitimacy of the scores, creating a ‘black box.’” http://kbros.co/2EvxjU9

Wells Fargo Credit Denial “Glitch”
CNN: “Hundreds of people
had their homes foreclosed
on after software used by
Wells Fargo incorrectly denied
them mortgage
modifications.” 2018-08-05
https://money.cnn.com/2018/08/04/news/companies/wells-fargo-mortgage-modification/index.html

. . . And not easy to “fix”
Risk mitigation for data science implementations is relatively immature.

Unintended Use Cases or Ethical Lapse?
• Algorithm corrected for color bias, but can
now be used for profiling
• “Red Teaming” or “Abuse User Stories” can
help
• Unintended use cases call for a safety vs. a
pure “assurance” framework

“Lite” AI Security/Reliability Frameworks
https://motherboard.vice.com/en_us/article/bjbxbz/researchers-tricked-ai-into-doing-free-computations-it-wasnt-trained-to-do
“Google researchers demonstrated that a
neural network could be tricked into
performing free computations for an
attacker. They worry that this could one
day be used to turn our smartphones into
botnets by exposing them to images.”

Machine Learning:
Expanding Impact
Impact across all industries may be difficult to assess.

The “Cambrian Explosion”
-- Ian Goodfellow
Staff Research Scientist
Google Brain
ACM Webinar 2018-07-24

Transparency & Interpretability
As Seen by Ian Goodfellow
Findings from adversarial machine learning

Why Ethical Issues Become Secondary
For many applications, ML benefits will
prove irresistible.
Goodfellow considers reliability needs as a
useful example.

Challenges of Interpretability
“Adversarial ML literature
suggests that ML models are
very easy to fool and even
linear models work in counter-
intuitive ways.” (Selvaraju et al, 2016)
• Reproducability
• Training sets including results of
other analytics (e.g., FICO)
• Provenance (think IoT)
• Opaque statistical issues

Transparency General Challenges
 Some data, algorithms are intellectual property
 Some training data includes PII
 Predictive analytical models are often “point in time”
 “Transparent” according to whose definition?
 Should algorithms have “opt-in?” Can they?
 Training set big data variety reidentification risks

Explainability / Interpretability
“[We need to] find ways of making techniques like
deep learning more understandable to their creators
and accountable to their users. Otherwise it will be
hard to predict when failures might occur—and it’s
inevitable they will. That’s one reason Nvidia’s car is
still experimental.”

Enterprise Level Risk
 Impact on reputation
 Litigation
 Unintentionally reveal sources, methods, data / interrupted data streams (e.g, web)
 Loss of consumer confidence, impact on public safety
 Misapplication of internally developed models
 Financial losses from data science #fail
 “. . . as long as our training is in the form of someone lecturing about the basics of gender or
racial bias in society, that training is not likely to be effective”.
Dr. Hanie Sedghi, Research Scientist, Google Brain

Risk Management
and Lessons from Others
Aligning with enterprise risk appetite, resilience

Insights from More Mature Settings
 AI Analytics for distributed military coalitions
 “. . . Research has recently started to address such concerns and
prominent directions include explainable AI [4], quantification of
input influence in machine learning algorithms [5], ethics
embedding in decision support systems [6], “interruptability” for
machine learning systems [7], and data transparency [8]. “
 “. . . devices that manage themselves and generate their own
management policies, discussing the similarities between such
systems and Skynet.”
S. Calo, D. Verma, E. Bertino, J. Ingham, and G. Cirincione, "How to prevent skynet
from forming (a perspective from Policy-Based autonomic device management),"
in 2018 IEEE 38th International Conference on Distributed Computing Systems
(ICDCS), Jul. 2018, pp. 1369-1376. [Online]. Available:
http://dx.doi.org/10.1109/ICDCS.2018.00137

Some Fairness Approaches (Sedghi, 2018)
 1) Post processing in terms of calibration of our model. What this means is that, we calibrate
classifiers parameters such that it has the same acceptance ratio for all subgroups of sensitive
features, e.g. race, sex, etc.
 2) Data resampling to remove skewed sample. But, for many reasons, collecting more data is not
very easy and sometimes causes problems for individuals.
 3) Causal reasoning: We capture different paths in a causal graph that can lead to the same
observational data. This basically means to model possible factors such that sex, race and other
sensitive features to make sure their impact is captured and does not directly affect the result
variable.

Decision Support for Bias Detection
“Things like transparency, intelligibility, and explanation are new enough
to the field that few of us have sufficient experience to know everything
we should look for and all the ways that bias might lurk in our models,”
says Rich Caruna, a senior researcher at Microsoft who is working on
the bias-detection dashboard.”
Technology Review, Will Knight 2018-05-25

“Fairness Flow”:
But will you share your ethics guidance?
https://www.cnet.com/news/facebook-starts-building-ai-with-an-ethical-compass/
“Bin Yu, a professor at UC Berkeley, says
the tools from Facebook and Microsoft
seem like a step in the right direction,
but may not be enough. She suggests
that big companies should have outside
experts audit their algorithms in order
to prove they are not biased. ‘Someone
else has to investigate Facebook's
algorithms—they can't be a secret to
everyone,” Yu says.’”
-Technology Review 2018-05-25

Professional Level Risk
 Skills obtained at considerable cost/effort will be deprecated
 Alternative approaches will be preferred
 Good, suitable approaches will be discouraged
 -- in order to avoid negative result approaches

Codes of Ethics
IEEE & ACM

Joint ACM-IEEE Software Engr Code
Assignment: Find the ethical trouble spots.
https://www.computer.org/web/education/code-of-ethics
 1. PUBLIC - Software engineers shall act consistently with the public interest.
 2. CLIENT AND EMPLOYER - Software engineers shall act in a manner that is in the best interests of their client and
employer consistent with the public interest.
 3. PRODUCT - Software engineers shall ensure that their products and related modifications meet the highest
professional standards possible.
 4. JUDGMENT - Software engineers shall maintain integrity and independence in their professional judgment.
 5. MANAGEMENT - Software engineering managers and leaders shall subscribe to and promote an ethical approach
to the management of software development and maintenance.
 6. PROFESSION - Software engineers shall advance the integrity and reputation of the profession consistent with the
public interest.
 7. COLLEAGUES - Software engineers shall be fair to and supportive of their colleagues.
 8. SELF - Software engineers shall participate in lifelong learning regarding the practice of their profession and shall
promote an ethical approach to the practice of the profession.

Transparency & Professional Ethics
 What connection to IEEE /ACM professional ethics?
 ACM: “The entire computing profession benefits when the ethical decision making process is accountable to
and transparent to all stakeholders. Open discussions about ethical issues promotes this accountability and
transparency.”
 ACM “A computing professional should be transparent and provide full disclosure of all pertinent system
limitations and potential problems. Making deliberately false or misleading claims, fabricating or falsifying data,
and other dishonest conduct are violations of the Code.”
 ACM “Computing professionals should establish transparent policies and procedures that allow individuals to
give informed consent to automatic data collection, review their personal data, correct inaccuracies, and, where
appropriate, remove data.”
 ACM “Organizational procedures and attitudes oriented toward quality, transparency, and the welfare of society
reduce harm to the public and raise awareness of the influence of technology in our lives. Therefore, leaders
should encourage full participation of all computing professionals in meeting social responsibilities and
discourage tendencies to do otherwise.”

State of Computing Profession Ethics
@ACM_Ethics
 Not part of professional training
 Enterprise ethics training is not technical enough
 Engineers are not challenged with realistic moral dilemmas
 Overlay of employment, profit, team loyalty as potential crosswinds
 Project Management practices dilute responsibility
 Requirements Engineering is more art than engineering

SDLC for Data Science
Life cycle of analytics | Analytics as Code

Analytics as Code
 Requirements Engineering
 Traceability and provenance (training data, code, constraints)
 Audit & Forensics
 Ethical tradeoffs tagged like PII (e.g., sustainability tradeoff for an oil company)
 Test automation (“left-shifted” test engineering, “Test-Driven Development”
 Dependency management
 Supporting infrastructure analogous to other application spaces
 Dashboard integration

Continuously Deployed Algorithms
Impacts of DevOps (IEEE P2675)
Test Harness construction
Workflow for retest, refinement, reproducibility

Training Set Scrutiny
 Training sets can get ethics off on the wrong foot
 Set responsibility for ethical ownership - e.g., similar to Institutional
Review Boards (IRB)
 Be prepared to defend incomplete, tentative, or experimental projects
 “Data” is part of data science “code”

Recognize Ethical, Abuse Scenarios
 For agile SDLC: User stories should include all stakeholders
Often supply chain, OSS, internal customers are omitted from
scenarios
 Add “abuse” or “adversarial” user stories / security stories (Secure Scrum
parlance)
 May require an external sensibility to discover & recite

Other Relevant IEEE Societies
IEEE Groups

IEEE Society on Social Implications
of Technology

IEEE Product Safety Engineering Society
• “Do no harm.” – It’s not
so easy.
• Do you know a system is
safe before it’s been fully
scaled up -- & possibly
federated?
• What constitutes “a
reasonable explanation”?

IEEE Reliability Society
See free reliability analytics
toolkit
https://kbros.co/2rugRij

Roadmap:
Algorithmic Bias Mitigation
A Roadmap for Specialists

Roadmaps: The Adversarial User Story
Image via Robert C. Seacord
writing about software security

Algorithmic Bias Risk Management
 1. Recognize, socialize groups protected by statute (e.g.,
Equal Credit Opportunity Act)
 2. Creatively consider other affected subpopulations
 Sight impaired – other disabilities
 Children, elderly
 Unusual household settings (elder care, multi-family
housing)
 Part time and workers
 Novice vs. Experienced users
 What counterfactuals are simply not being measured?

Roadmap Attributes
 Engage all stakeholders: consumers, co-workers, shareholders, specialists,
professional CoI’s
 Embrace or consult “traditional” AI
 Tie to enterprise risk appetite
 Align with professional Codes of Ethics
 “User Stories” should include adversarial ethical challenges
 Traceable responsibility for decision-making (e.g., tradeoff analysis)

Suggested Area of Study: Ethical Intent
 Rao, Anand; Georgeff, Michael. 1995. Belief-Desire-Intention Agents: From Theory to
Practice. Australian Artificial Intelligence Institute, Melbourne, Australia. In ICMAS-95.

Casali, A., Godo, L. and Sierra, C., 2011. A graded BDI agent model to represent and reason about
preferences. Artificial Intelligence, 175(7-8), pp.1468-1478.

Xiaowei Wang, Nicola Guarino, Giancarlo Guizzardi, and John Mylopoulos. 2014. Towards an Ontology
of Software: a Requirements Engineering Perspective. FOIS 2014.
https://www.academia.edu/8744511/Towards_an_Ontology_of_Software_A_Requirements_Engineering_
Perspective. [local]

Guarino, N., 2006. Ontology and Terminology. In How can formal ontology help concept modelling
and terminology. Powerpoint in Workshop «Terminology, Concept Modelling and Ontology. Which
approach for which problem.

A Final Rationale

• Co-Chair NIST Big Data Public WG Security & Privacy subgroup https://bigdatawg.nist.gov/
• Chair Ontology / Taxonomy subgroup for IEEE P7000. Occasional participant in IEEE Standards
WGs P7007, P7003, P7002, P7004, P7010
• IEEE Standard P1915.1 Standard for Software Defined Networking and Network Function
Virtualization Security (member)
• IEEE Standard P2675 WG Security for DevOps (member)
• Current: Finance, large enterprise: supply chain risk, complex playbooks, many InfoSec tools,
workflow automation, big data logging; risks include fraud and regulatory #fail
• Authored chapter “Big Data Complex Event Processing for Internet of Things Provenance:
Benefits for Audit, Forensics, and Safety” in Cyber-Assurance for IoT (Wiley, 2017)
https://kbros.co/2GNVHBv
• @knowlengr dark@computer.org knowlengr.com https://linkedin.com/in/knowlengr
About Me

This deck is released under
Creative Commons
Attribution-Share Alike.
See also: Stakeholders in Ethical Systems Design

Ethics of Analytics and Machine Learning

In this document