Reforming Medical Device approval processes especially in software requires careful consideration of shifting risks to patients without adequate protections.

1. Adequate Directions for Use "in
the Age of AI and Watson" -
Compliance Online
Medical Device Summit 2017
Stephen Weitzman J.D. LL.M.
MedDATA Foundation

2. How I Got Here
Biology, Chemistry and Physics are all
the same, just depends on the size of the
Particle you are studying at the time.

3. WHAT IS Artificial Intelligence - AI
Development of computer systems able to perform tasks that normally
require human intelligence, such as visual perception, speech
recognition, decision-making.
Term coined in 1956 by John McCarthy at the 2nd Dartmouth
Conference

4. PATTERN RECOGNITION – MORE TRAINING

5. WHO & WHAT IS WATSON
IBM Quantum Chip 2016
Quantum Computer Housing
Think Pad - 1960
WATSON – JEOPARDY COMPUTER - 2011

6. OTHER BIG PLAYERS
• The Partnership on Artificial Intelligence to Benefit People and Society
- Microsoft, Amazon, IBM, Google and Facebook
• All the Traditional Chip Makers – Intel (CPU), AMD & NVidia (GPU)
• Quantum Chips – IBM, Google
• Optical HPE

7. APPLICATIONS IN MEDICINE – WELL KNOWN
TO FDA FOR YEARS
• The issue was recognized as far back as 1998 in part as a resource
issue in managing and dealing with databases
• FDA/CDRH’s perspective and approach on Digital health guidance's
(Webinar February 24, 2015)
• Software is becoming much more complicated as we moved from
statistics and data mining where you don’t need a computer to
crunch data to Neural Networks, and Machine Learning
• FDA now is forming a specific group in CDRH to deal with the issues of
Artificial Intelligence, Neural Networks, Pattern Recognition, and
Machine Learning under Bakul Patel – Associate Center Director for
Digital Health

8. Scientific Advances & Opportunities
• Genomics/proteomics
• Information technology
• High throughput technologies
• Mechanistic knowledge and biologic modeling
• Cancer, inflammation, cell signaling, etc.
• Artificial intelligence
• Noninvasive imaging – Pattern Recognition
• Radiological
• Microscopic – even down to the nano-molecular or subatomic level

9. DRIVERS OF AI
• Sensing devices
• Improved sensitivity
• Miniaturization
• Wearables
• Big Data
• Medical Records Systems
• Continuous monitoring
• Genomics

10. BIG DATA
BIG DATA challenges:
• Developing scalable algorithms for processing imperfect data
in distributed data stores
DARPA XDATA Program
• The OSTP Big Data initiative aims to:
• Advance state-of-the-art core technologies needed to collect, store, preserve, manage, analyze and share
huge quantities of data.
• Harness these technologies to accelerate the pace of discovery in science and engineering, strengthen our
national security, and transform teaching and learning; and
• Expand the workforce needed to develop and use big data technologies.
OSTP BIG DATA Research and Development Initiative (3/2012)
• DATA Reliability –
• Data Curation – the EHR Dilemma
• Good Bye Moore’s Law
• Data Storage - Economics
• CPU vs GPU
• The new chips and quantum computing

11. SOFTWARE ALGORITHMS
• Simple Process Flow
• Software that learns and improves
• Software that writes software based upon data and humans no longer
code
• Example – Microsoft Excel Spreadsheet – Voice Command – Tell it what chart
to create

12. User Control
… people - not apps - are in control.
Flexibility
… users complete, fine-grained control over their work.
Apple Design Guidelines

13. AI AND HUMAN PARTNERSHIP
• CAN WE RELY ON HUMAN JUDGEMENT IN MEDICINE?
• Too Much Literature to Absorb
• Carrying Watson to Medical School Class
• Keeping up with Journals and other sources of advances in clinical research
• WATSON AT ASCO
• Improving Diagnosis to 80% Correct
• Other decision support programs for selecting therapies
• Can we rely on Drug Labeling Alone?

14. RISKS - FEDS LEVY $155M FINE AGAINST SOFTWARE
VENDOR FOR FAULTY PATIENT RECORDS SYSTEM
• Charged that the software “failed reliably” to document and display medications and laboratory tests
resulting in “serious patient safety issues.”
• Selling faulty software while aware of the flaws but failed to fix them because fixes would require addressing
problems in the company’s core software and all of its modules, and because admitting the defects “would
put company at severe competitive disadvantage.”
• Flaws that may have exposed millions of patients to potential safety risks. For example, the company entered
in its programs only the limited number of drug codes required for testing rather than programming the
capability to retrieve any drug code from a complete database.
• Safety experts who worry that the computerized records have introduced new dangers to health care.
• Congress on the need for a health IT safety center and stronger supervision of IT products by the Office of
the National Coordinator for Health IT (?FDA?).
• Software used by the women’s hospital was malfunctioning dangerously.
• Patient records often overlapped on their computer screens, making it easy to mistake one patient’s
diagnosis or drug for another’s. Medication lists were error prone. Patients left the jail without proper
prescriptions or lab results. One patient’s HIV drugs weren’t listed on his medical report. Another’s tapering
methadone dosage was inaccurate.

15. PNAS Opinion: To mitigate the dangers of faulty, biased, or malicious algorithms requires independent oversight
(November 29. 2016) http://www.pnas.org/content/113/48/13538.full

16. 1. Awareness: Owners, designers, builders, users & other
stakeholders should be aware of… possible biases and harm
2. Access and redress: Regulators should adopt mechanisms that
enable questioning & redress
3. Accountability: Institutions should be held responsible…even
if it is not feasible to explain how the algorithms produce their results
4. Explanation: Systems & institutions that use algorithmic decision-
making are encouraged to produce explanations of the procedures &
decisions
Algorithmic Transparency & Accountability
Statement on Algorithmic Transparency and Accountability
by ACM U.S. Public Policy Council, approved January 12, 2017
ACM Europe Policy Committee, approved May 25, 2017

17. 5. Data Provenance: Algorithm builders should maintain a
description of how training data was collected
6. Auditability: Models, algorithms, data & decisions should be
recorded so that they can be audited
7. Validation & Testing: Institutions should use rigorous methods
to validate their models
Algorithmic Transparency & Accountability

18. Corporate internal audit committees & advisory boards
& External audits required by SEC
University Accreditation, NSF & EPSRC Advisory Boards
Zoning Boards, Planning Commissions, Environmental Impact Statements
NASA, FAA, FDA, DHS, Federal Reserve, etc.
ALGORITHMS

19. Clarifying responsibility accelerates quality
• Independent oversight: Open adversarial reviews
• Transparency: Open the black box
• Accountability: Open failure reporting
• Liability: No “hold harmless” contracts

20. SAFETY ASSESSMENT – RISK BASED APPROACH
Proposed Risk-Based Regulatory Framework and Strategy for Health
Information Technology, May 13-15, 2014
• Risk-based Approach in FDA Inspections
• The good guys vs. the problem companies – trust
• The severity of consequences of errors – frequency of checking
• Medical Device Software - Trusted Companies ?
• How Should Software Be Reviewed
• Class I – Low Risk
• Class II – Requirements Plus 510(k)-Like Review
• Class III – PMA – Manufacturer Immune from Liability?

21. THE SOFTWARE LABEL –
ADEQUATE DIRECTIONS FOR USE OF AI
• Highlights of Software Functionality
• Warnings and Precautions – Consequence of Errors
• Recent Major Changes to Software
• Uses
• Administration – User Interface Components
• Contradictions – Potential for Errors
• Use on specific populations – Variation ?
• Pregnancy
• Race
• Pediatric
• Physiological – Hepatic
• Clinical Studies – Details of Detailed Audits
• Company Certification – Assurances
• Liability Insurer