AI Design in High Risk Settings - Aligning business impact, risks, and innovation As AI becomes integral to business strategy, many organizations have struggled to navigate the delicate balancing act between technical complexity and business value, especially with the advent of generative AI. This is especially true in high-risk settings like healthcare, defense, and financial services. In this session, actionable strategies for enhancing AI and data literacy among stakeholders, aligning AI projects with business objectives, and fostering trust between AI builders and users will be explored. While many strategies are applicable to both traditional machine learning and generative AI projects, special considerations for projects incorporating generative AI will also be explored. Attendees will walk away with practical insights to propel AI projects from the design board to real-world impact and pave the way for a culture of informed and responsible AI innovation within their organizations. Session Outline: Participants will learn how to: · Develop greater AI and data literacy among stakeholders and users · Align between business outcomes and AI project goals · Build trust and effective communication channels between AI builders and users · Identify and communicate the potential risks and unintended consequences that arise from both traditional and generative ML models Background Knowledge: Suitable for all backgrounds - especially those looking to communicate more effectively with stakeholders, or stakeholders learning to get more comfortable with technical considerations.

1. Cal Al-Dhubaib,
CEO & AI Strategist
AI Design in High-
Risk Settings
Aligningbusinessimpact,
risks,andinnovation

2. AI is software that…
1. Recognizes patterns 2. Reacts to patterns

3. Pattern selection is critical to model performance

4. Left unchecked,
models can learn
the wrong things
from data
"Why Should I Trust You?": Explaining the
Predictions of Any Classifier
Marco Tulio Ribeiro, Sameer Singh, Carlos
Guestrin

5. Left unchecked, deep
learning models can
learn the wrong
things from data
Data Science Institute, American
College of Radiology (2022)

6. In some cases,
machines learn
patterns that
humans can’t see

7. Enterprise AI
Performant
Fair
Secure
Reliable
Transparent
Auditable
The pillars of
Enterprise AI

8. Fewer than 25% of
the workforce
perceive self as
data literate (HBR 2021)
Digital literacy
• Word processing
• Spreadsheets
2000s
Data literacy
• What data?
• Can we build?
2010s
AI literacy
• Should we build?
• When do we trust?
2020s

9. Enabling users saved the day
• 50 million+ patient lives
• The team reaches out to qualified (accepted) leads
• Missing potential accepts reduces potential revenue
• Accepting unqualified leads reduces profit
Ingest Outreach Qualify
Approve
Accept
Increased monthly
revenues by 18% (to date)
Outcome

10. Insight can be more powerful than prediction

11. Annual Solid-Tumor Cancer Readmissions
Hospitalizations
Readmissions (22k per
readmission)
Identified by model
(potential savings)
Prevented by intervention
(~$2.3M saved)
2.8 million cancer in-patient
visits in US
15 – 25%
(national avg)
~70%
~20%
• Resource
efficiency
• Patient
outcomes
• Care quality
• Care
coordination
and continuity

12. The value of data
science is limited by
the value of decisions
impacted

13. Aligning the critical humans
Builder Facilitator User

14. A simple data
labeling
exercise?

15. Appropriately
labeled data is
often a barrier
to successful AI
projects

16. Models struggle when humans don’t agree
Search
Follow-up
Pricing
Initial topic suggestions Humans verify topics

17. AI controversies are on the rise because it’s
hard to imagine how models might break

18. Machine-learning
has long suffered
from problematic
data and
unintended
consequences

19. Generative AI
introduces
some newer
risks

20. Prompt
poisoning is a
new threat
https://gandalf.lakera.ai/

21. Scams are
growing in
sophistication

22. Data
complexity
Goal
complexity
Measure Predict Recommend Create

23. Red-teaming: unexpected scenarios

24. Red Teaming
with Under-
Specified
Questions
Allen Institute
for AI
https://unqover.app
s.allenai.org/
Patrick The Asian
person
The Muslim
person
The Libyan
person

25. Red Teaming
with Under-
Specified
Questions
Allen Institute
for AI
https://unqover.app
s.allenai.org/`
More negativity
Less negativity

26. Guardrails & probability-based workflows
Filter inputs likely
to lead to bad
outputs
Filter problematic
outputs

27. Share these
decisions
with model
cards
(examples courtesy of Google & Twillio)

28. Enterprise AI Is Iterative
Develop
Design
1. Discover
2.
Experiment
& learn
6. Educate &
empower
5.
Integrate
& scale
4. Build resilience
3. Prototype
& evaluate
7. Monitor &
improve

29. Value
Complexity
Ideal,
but
rare
Ideal
Pilot
AVOID
Long-term
investment
The right starter projects unlock long-term value

30. Boring is
profitable & safe
Invest in AI
literacy
Account for
action
Adopt stress
testing

31. Let’s make AI
boring again
Cal Al-Dhubaib
CEO & AI Strategist
@caldhubaib
@ohpandata