"Designing Trustworthy AI: A Human-Machine Teaming Framework to Guide Development" is a paper presented at the AAAI 2019 Fall Symposium on AI in Government and the Public Sector, (sponsored by the Association for the Advancement of Artificial Intelligence) in Washington, DC, November 7–9, 2019. Artificial intelligence (AI) holds great promise to empower us with knowledge and augment our effectiveness. We can -- and must -- ensure that we keep humans safe and in control, particularly with regard to government and public sector applications that affect broad populations. How can AI development teams harness the power of AI systems and design them to be valuable to humans? Diverse teams are needed to build trustworthy artificial intelligent systems, and those teams need to coalesce around a shared set of ethics. There are many discussions in the AI field about ethics and trust, but there are few frameworks available for people to use as guidance when creating these systems. The Human-Machine Teaming (HMT) Framework for Designing Ethical AI Experiences described in this paper, when used with a set of technical ethics, will guide AI development teams to create AI systems that are accountable, de-risked, respectful, secure, honest, and usable. To support the team's efforts, activities to understand people's needs and concerns will be introduced along with the themes to support the team's efforts. For example, usability testing can help determine if the audience understands how the AI system works and complies with the HMT Framework. The HMT Framework is based on reviews of existing ethical codes and best practices in human-computer interaction and software development. Human-machine teams are strongest when human users can trust AI systems to behave as expected, safely, securely, and understandably. Using the HMT Framework to design trustworthy AI systems will provide support to teams in identifying potential issues ahead of time and making great experiences for humans.

1. 1
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Software Engineering Institute
Carnegie Mellon University
Pittsburgh, PA 15213
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Designing Trustworthy AI:
A Human-Machine Teaming
Framework to Guide
Development
Carol J. Smith, @carologic
AAAI 2019 Fall Symposium on AI in Government and the
Public Sector, Washington, DC, November 7–9, 2019.
Sponsored by the Association for the Advancement
of Artificial Intelligence.

2. 2
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Related Resources
• Paper on arXiv: https://arxiv.org/abs/1910.03515 "Designing
Trustworthy AI: A Human-Machine Teaming Framework to Guide
Development"
• Checklist and Agreement - PDF on Google Drive:
https://drive.google.com/open?id=1rY8EF-
2orGquI23g7yQ5aWGJtUmzP4d6
• SEI page for Designing Trustworthy Artificial Intelligence:
https://sei.cmu.edu/research-capabilities/all-
work/display.cfm?customel_datapageid_4050=197910

3. 3
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
AI’s Great Promise
Empower us with knowledge
Augment our effectiveness
We can—and must—ensure that we keep humans
safe and in control

4. 4
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
How can AI development teams
harness the power
of AI systems
and design them
to be valuable to humans?

5. 5
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Designing Trustworthy AI
Diverse Teams
– Shared Ethics

6. 6
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Why diverse teams?
Focus more on facts
Process facts more carefully
More innovative
“They may also encourage greater scrutiny of each member’s actions,
keeping their joint cognitive resources sharp and vigilant.”
“…become more aware of their own potential biases”
Why Diverse Teams Are Smarter. Harvard Business Review. https://hbr.org/2016/11/why-diverse-teams-are-smarter

7. 7
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
What is a diverse team?
Diverse with regard to gender, race, education, thinking process,
disability status, and more…
An inclusive environment is required to make diversity successful.

8. 8
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Multi-Disciplinary
• Skill set and problem framing approach
• Machine learning experts, programmers, system architects, product managers,
etc. and…
• Curiosity experts
• Focus on understanding situation, constraints, and abilities
of people who will use system and how will be used
(includes: HCI, HMI, cognitive psychologists, digital anthropologists, and UX researchers)

9. 9
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
1. Well-being
2. Respect for autonomy
3. Protection of privacy and intimacy
4. Solidarity
5. Democratic participation
6. Equity
7. Diversity inclusion
8. Prudence
9. Responsibility
10. Sustainable development
Coalesce around a shared set of technical ethics
Montréal Declaration for a responsible development of artificial intelligence.
https://www.montrealdeclaration-responsibleai.com/the-declaration

10. 10
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Diversity + Ethics
Accountable
De-risked
Respectful
Secure
Honest
Usable
AI
Process
?
Montréal Declaration for a responsible development of artificial intelligence.
https://www.montrealdeclaration-responsibleai.com/the-declaration

11. 11
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
HMT Framework for Designing Ethical AI Experiences:
Themes
1) Accountable to humans
2) Cognizant of speculative risks and benefits
3) Respectful and secure
4) Honest and usable

12. 12
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Goals of HMT Framework for Designing Ethical
AI Experiences
Pair with technical ethics - bridge gap between “do no harm” and reality
Reduce risk and unwanted bias
Mitigation plans
Support inspection - ethical and trustable process and system

13. 13
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Conversations and Understanding
Guide AI development teams
Inform effort
Create AI systems that are accountable, de-risked, respectful, secure,
honest, and usable.

14. 14
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Accountable to Humans
Ensure humans are always in control, able to monitor and control risk
Designate responsibility for all decisions and outcomes to humans

15. 15
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Cognizant of Speculative Risks and Benefits
Identify full range of
•Harmful, malicious use, as well as good, beneficial use
•Blind spots and unintended consequences
Create communication and mitigation plans for misuse/abuse of AI system

16. 16
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Respectful and Secure
Value humanity, ethics, equity, fairness, accessibility, diversity and
inclusion
Be robust, valid and reliable
Respect privacy and data rights
Provide understandable security

17. 17
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Honest and Usable
Value transparency with the goal of engendering trust
Explicitly state identity as an AI system

18. 18
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Designing Trustworthy AI
Checklist and Agreement
PDF on Google Drive: https://drive.google.com/open?id=1rY8EF-
2orGquI23g7yQ5aWGJtUmzP4d6

19. 19
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
We are confident that we have designed our AI system
so that:
 Humans are always ultimately in control, able to monitor
and control risk
 Designate responsibility to humans for all decisions and outcomes
 Explicitly defined responsibility and who shares responsibility
 Preserve human responsibility for final decisions that affect
a person’s life, quality of life, health, or reputation
 Significant decisions made by the AI system are appealable,
able to be overridden, reversable

20. 20
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
We identified the full range of risks and benefits:
 Harmful, malicious use
 Good, beneficial use
 Blind spots and unintended consequences

21. 21
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
We have created plans:
 Communication plan(s) for misuse/abuse of AI system
 Mitigation plans for misuse/abuse of AI system

22. 22
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
The AI system is respectful and secure:
 Integrated values of humanity, ethics, equity, fairness, accessibility,
diversity and inclusion
 Respected privacy and data rights
 Provided understandable security methods
 AI system is robust, valid and reliable

23. 23
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
We value transparency with the goal of engendering
trust:
 Purpose and limitations of the AI system are explained in plain
language
 Data sources and training methods have unambiguous sources
and are verifiable
 Confidence and context are presented for humans to base decisions on
 Provided transparent justification for outcomes
 Straightforward, interpretable, monitoring systems

24. 24
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
The AI system explicitly states its identity, is honest and
usable:
 Easily discern when interacting with AI system vs. a human
 Easily discern when and why the AI system is taking action and/or
making decisions
 Improvements made regularly to meet human needs and technical
standards

25. 25
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Create AI systems that are accountable,
de-risked, respectful, secure, honest,
and usable

26. 26
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Make Trustworthy AI Systems
• Diverse team in inclusive environment
• Shared set of technology ethics
• Conduct activities to understand people’s needs and concerns
for the system
• Encourage deep conversations to align on clear expectations
and mitigation plans
• Use the HMT Framework’s Checklist and Agreement to design
ethical AI experiences

27. 27
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
https://sei.cmu.edu/research-capabilities/all-work/display.cfm?customel_datapageid_4050=197910

28. 28
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Carol J. Smith
cjsmith@sei.cmu.edu
Twitter: @carologic
SEI Emerging Technology Center
Twitter: @sei_etc