The Why And How Of Machine Learning And AI: An Implementation Guide For Healthcare Leaders

2. © 2017 Health Catalyst Proprietary and Confidential Eric Just • SVP at Health Catalyst in Product Development • Started career in healthcare at Northwestern University • Genomics • Clinical Data Warehouse • Health Catalyst • Early platform vision • Sales • Client Operations • Product

3. © 2017 Health Catalyst Proprietary and Confidential Ken Kleinberg • VP of Research at Chilmark Research, covering • Analytics, AI, machine learning, mobile/wireless, IoT, EHRs, medical devices… • Healthcare transformation, PHM, consumerism, payer-provider convergence… • 38 years in IT – last 20 in healthcare, including • Managing Director, Advisory Board • VP of Bus Dev, Health Language/Wolters Kluwer • VP and Hospital Strategist, Allscripts • Senior Director of Global Health, Symbol/Motorola/Zebra • VP and Editor-in-Chief of Healthcare, Gartner

4. © 2017 Health Catalyst Proprietary and Confidential Learning Objectives and Agenda 1) What Are Machine Learning (ML) and Artificial Intelligence (AI) and Why Should I Care • Getting past the hype to real definitions • Understanding use cases and benefits 2) How Do I Implement ML and AI in My Organization • Practical steps, resources, costs, timeframes, skills, and vendors • The importance of data management and an analytics platform • Real-time data and moving targets 3) Dealing with Challenges • Change management, regulation, liability, human touch 4) What Does the Future Hold for ML and AI (and for My Strategy) • Job augmentation and replacement, taking on greater challenges, scary (yet bright) futures What You Need to Know and Do

5. © 2017 Health Catalyst Proprietary and Confidential Executive Summary • Machine learning and the broader area of AI, inspired by biology and evolution to learn and adapt, have long histories. • Use cases in healthcare include improved predictions, patient clustering and classification, early detection/diagnosis, scheduling, medical image recognition, and natural language processing (NLP)/text mining. • Practical steps for getting started with ML and AI include: goal setting, allocating resources, establishing realistic timelines, costs, and benefits; managing complex data sources; running models; measuring results and taking action/operationalizing. • Existing investments in data management and analytics, as well as processes to identify needs and create operational decision models, give organizations an advantage. • Typical challenges include change management, regulation, liability, and concerns about human “touch.” • The future of ML and AI include increased job augmentation and replacement, tackling more complex problems (e.g., genomics), and the need to ensure these technologies are applied with the proper rigor and ethical oversight. 5

7. © 2017 Health Catalyst Proprietary and Confidential Understanding the Hype in Machine Learning and AI • World-leading companies have invested billions into voice assistants, facial recognition, shopping assistants, and semi-autonomous vehicles. • There has been a media frenzy over AI and job displacements, disruptions, and dangers – the rise of the intelligent machines! • It is difficult to discern the degree of healthcare vendor investments in these technologies and the sophistication of their solutions. • To more fully appreciate the opportunities and challenges requires some understanding of what machine learning and AI are. 7

8. © 2017 Health Catalyst Proprietary and Confidential Machine Learning and AI Definitions * ML and AI draw and advance approaches and principles from traditional statistics and mathematics, rule-based systems, and biological systems to create solutions that can learn and adapt. * They reduce the effort required by humans to manually choose algorithms, data sources, model parameters, and steps in order to test hypotheses, discern knowledge, and achieve goals. 8 Rules, inference engines, symbolic reasoning, programming Biology, parallel processing, genetics, evolution Machine Learning and AI Math, statistics, probability theory, analytics

9. © 2017 Health Catalyst Proprietary and Confidential Clarification of Definitions Artificial intelligence refers to systems or machines that can adapt and learn from data and their environments to exhibit the intelligent behavior found in living things. Machine learning is a subset of AI with its origins in math, statistics, computer science, and biology (e.g., neural networks, which simulate networks of neurons in the brain). Its models adjust themselves when presented with training data. This can be: • Unsupervised (meet an algorithmic goal, such as clustering data into discernible groups) • Supervised (the system is given labeled training data and adjusts itself to provide the right answer) Deep learning is a more sophisticated form of ML that incorporates multiple layers (e.g., of neurons) capable of pre-processing and filtering, to model more complex input and relationships. Cognitive computing attempts to model human thought processes, including generating ranked hypothesis. 9

10. © 2017 Health Catalyst Proprietary and Confidential Some Additional Definitions Expert systems refer to rule-based reasoning systems that either attempt to determine which possible answers are supported by data (backward chaining) or start with known information and work towards reaching a viable answer (forward chaining) – rules can be used to apply known knowledge and context to ML and AI. Descriptive, predictive, and prescriptive analytics are increasingly valuable approaches to determine what has happened (descriptive), what will or might happen (predictive), and what to do about it (prescriptive). Natural language processing uses a variety of ML and AI techniques to understand human text or speech. 10

11. © 2017 Health Catalyst Proprietary and Confidential Tackling (Big) Data and Problem Complexity 11 The volume, variety, and velocity of information can far exceed a care professional’s abilities Social determinants of health Microbiome Environmental data Genomic data Medical and family history Patient- generated health data Population health data Resource availability (care facilities, clinicians, medications) Policy and regulation Insurance coverage Medical knowledge Patient Journey Digital images, lab tests

12. © 2017 Health Catalyst Proprietary and Confidential What to Apply Where and Why Approach Best For Positives Risks Statistics • Good assumptions to start with • Standard models sufficient • Clean (structured) data • Skills and mature tools often already exist in the org • Acceptance by org • Techniques may not be powerful enough for difficult problems • Models may not transfer well to real and changing world • Lots of manual effort required Machine Learning • A hypothesis to test or pattern to recognize • Some assumptions to start with • Many models to try • May be able to utilize some existing organizational skills • Removes some of the manual burden of model development • Can identify most impactful input features • Models may appear to do well but be simply “memorizing” training data • Models may not be sophisticated enough for more difficult problems Deep Learning • A difficult problem to solve • Complex interactions • Messy and/or huge amounts of data • A “black-box” answer will do • May be able to provide answers or predictions difficult to discern by any other method • Skills and tools may not exist in the org • Users may require more specific explanation for model results • Even large datasets may not carry enough info for the most sophisticated systems to find insight Expert System • Extensive domain knowledge available • Processing order not predefined • Reasons for answers must be apparent • Answers can be traced to which rules have “fired” • Organizations can agree on rules or customize locally • Too many rules (over a few thousand) can be difficult to manage/curate • Has challenges with conflicting data 12

13. © 2017 Health Catalyst Proprietary and Confidential ML and AI Use Cases • Pattern recognition • Clustering • Prediction • Diagnosis • Scheduling • NLP • Smart network • Robotics • Facial recognition • Consumer buying patterns • Financial market trends • Car problems • Train schedules • Virtual assistant • Internet of things (home) • Assembly line • Tumor type • Disease variations • Length of stay, readmissions • Cancer care • Surgery and rehab • Chart abstraction • Internet of things (hospital room) • Medication delivery 13 Type of Goal Cross-Industry Example Healthcare Example

14. © 2017 Health Catalyst Proprietary and Confidential Setting Expectations While Accelerating Exponentially 14 Useful Too good? Promising Essential No way Geeky/Niche Advances have a way of creeping up on us – one day a joke, another day useful, and all of a sudden, Wow! We are here with AI Smart Phones Self-driving cars Virtual Assistants GPS “Doc in a box” Human brain interface Internet

16. © 2017 Health Catalyst Proprietary and Confidential Practical Steps Major Considerations include: 16 Strategy – Ensuring that ML and AI are applied to the rights kinds of problems for the right organizational goals Resources & Infrastructure – Determining and implementing analytics platforms, data warehouse capabilities, tools to access data, and security required to succeed Funding – Distributing reasonable costs for ML and AI infrastructure and initiatives across relevant stakeholders Timeframes and Benefits – Determining how long ML and AI initiatives will take and what are the expected benefits Skills – Ensuring the rights kinds of skills for project scoping, sourcing decisions, model development, and operational changes Vendors, Partners, and Tools – Choosing which vendors and partners will be able to provide the tools and skills to help

17. © 2017 Health Catalyst Proprietary and Confidential Approach to Strategy for ML and AI • A balanced approach supported and visible to executives and the board, desired or driven by (or they are at least willing to give it a chance) the departmental level (clinical, financial), and enabled by IT • Incorporated into the overall organizational strategy and initiatives (PHM, ACO, Quality, Consumerism) • Pursued within the process of operational governance to ensure priority, funding, ownership, results, P&L • Nurtured with a staged and natural progression growth that considers start-up costs, sourcing of skills and resources, knowledge transfer, risk, and scale 17 • Driven totally from the top or totally bottom up Good Worrisome • Considered an IT project with no specific links to business or clinical initiatives • Politics, lack of transparency, lack of accountability • Big bang, shoot for the moon, high expense

18. © 2017 Health Catalyst Proprietary and Confidential Flexible and Scalable Infrastructure Matching to an Organization’s Needs and Capabilities 18 Turnkey application (simpler use, more reproducible) Configurable application (varied needs, controlled by business user) Customizable application (specific needs, pilots) Typical Adoption Progression these can be incrementally added multiple approaches – mix and match full leveraged environment to support many needs web-based decision support ALGORITHM TOOLKIT BI PLATFORM WORKFLOW INTEGRATION

19. © 2017 Health Catalyst Proprietary and Confidential Real-Time Data and Moving Targets • One of the challenges with any prediction, decision support or analytics solution is its applicability/accuracy as input sources and situations change. • Should the system see something it was not trained on, depending on the influence/combinations of the inputs, the system might be able to come up with an accurate answer, semi-accurate, or it could be completely wrong. • ML and AI systems, by definition, should be readily adaptive to new training data and new situations. • It may be possible to monitor/identify different “alert levels” – that is, operating within a data stream thoroughly trained on, working with a different yet similar set of data, or working with data outside the training range. • Use of real-time data platforms can help ensure that systems are continually retrained on relevant data for successful operational deployment. 19 Retrospective training data Good for initial model development Real-time training data Needed for operational model deployment

20. © 2017 Health Catalyst Proprietary and Confidential The Cost Realities of Machine Learning and AI 20 Organizations should consider change management and cultural costs associated with implementing ML and AI Costs are mostly incremental for organizations that have already invested heavily in BI/Analytics and are using that data to affect improvements Organizations that have not gone this route will have much higher hills to climb The most important first step is to use data effectively, even at a basic level, to begin making decisions

21. © 2017 Health Catalyst Proprietary and Confidential Timeframes and Benefits 21 What will you do if benefits are not (completely or substantially) met?? Are you making interim measures at key points? Do you have buy-in from stakeholders about the benefits? Are you providing a range of (compelling) expected benefits? Do you have baseline measurements to compare results? Are you setting short, medium, and long-term goals? Are you measuring benefits in terms of costs, times, quality, or other measures (precision vs recall)?

22. © 2017 Health Catalyst Proprietary and Confidential Skillset Sourcing for Implementing ML and AI Job Function Description Internal External Clinical Identifying uses cases, discerning valuable outcomes, dismissing wrong/dangerous results, rallying support **** * Financial Recognizing opportunity costs, spreading funding across relevant stakeholders, evaluating ROI **** * IT Integrating data sources and data warehouse systems, monitoring performance, sourcing/hosting, upgrades ***** Security Ensuring that data sources are properly secured ***** Project Mgmt. Setting realistic timeframes, meeting resource requirements, and conveying progress to stakeholders **** * Bus Dev Identifying and contracting with suitable partners, data sharing, consortiums, co-development *** ** Marketing Ensuring visibility to achievements for purposes of organizational reputation and patient recruitment ** *** ML/AI Choosing the right algorithms and modeling approaches, running the models * **** BI/Data Science Ensuring value add, integration with existing techniques, dashboard design ** *** User Support Education, training, mentoring, answering questions, forwarding on more difficult problems, tracking concerns *** ** Legal Minimizing regulatory risks, meeting FDA requirements, ensuring HIPAA security, vetting business partners **** * Operations Minimizing disruption, scheduling, meeting reporting period requirements ***** 22

23. © 2017 Health Catalyst Proprietary and Confidential Considerations – Does Your Team Know the Answers? AI and ML Experience • Which ML algorithms and AI approaches (or combinations) are you going to pick? • Do you have experience with descriptive, predictive, and prescriptive analytics? • Has your team or your vendor any experience (and success) with competitions such as Kaggle? • Does your team have access to online courses to increase their expertise? Hardware Resources • Do you have the processing power (GPUs) and memory to run the models with the amount of data necessary? • Have you considered cloud and virtualization? Data Sets • Are you able to identify and have access to relevant data sources? • Do you have access to adequate sample datasets? • To what degree will you be attempting to transform/analyze unstructured data? Vendors and Products • Will you be using a turnkey solution, a configurable application system, or an application you have to customize or develop (and later support)? • Will you be using a proprietary commercial solution, or open source? If using open source, what support is available from who? • To what degree do you expect to use or need consulting services? 23

24. © 2017 Health Catalyst Proprietary and Confidential Questions to Ask Vendors of ML and AI Issue Description Concern ML and AI Experience How long has the vendor been using ML and AI? Do they know how to use it? Can they explain how it works? BI Platform Does the vendor have a platform for BI and analytics? Is it a modern architecture built with software engineering principles? Healthcare Experience Does the vendor have relevant/dedicated healthcare experience (with similar organizations)? Will you be teaching them about healthcare? ML and AI Source Are the ML and AI capabilities developed by the vendor, commercially available, or open source? Proprietary lock-in? Compelling Difference Can the vendor demonstrate a compelling difference ML and AI make to its solution? Is it worth it? Flexibility To what degree is the vendor able to switch in new ML and AI capabilities or customize their solution? In love with their hammer? – what does their roadmap look like? Integration with EHRs What approach and success does the vendor have with integrating (or competing) with major EHR vendors? Will there be cooperation? Scalability Are there any scalability issues regarding the amount of data, processing times, database structure (e.g., Hadoop), hardware requirements? Will you hit these limits? Business Model How does the vendor charge for their software, hosting, and services capabilities (over time)? Do they price the way you want? Self-Sufficiency Does the vendor offer tools or education to help make you self-sufficient? What skill levels will be required to use this solution? 24

26. © 2017 Health Catalyst Proprietary and Confidential Typical Challenges It would seem that after identifying the use cases and technologies, and then sourcing, developing and deploying the models, an organization would be home free. As hard as it might be to succeed on those fronts, there are additional challenges that are likely to be faced that could be more difficult to address and risk success of the system and worse. These include dealing with: 26 Change Management AI and ML systems may be discovering results, exposing weakness, taking away people’s jobs and more – often at a pace faster expected – as in any transformative project, a focus on change management is key. Regulation As ML and AI take on more complex situations, the may come under increasing regulatory scrutiny. Liability As ML and AI become more autonomous, a question arises as to who is liable when things go wrong. Human Touch As ML and AI systems and results touch more of our lives, how can we ensure a respectful and compassionate interaction while also balancing the needs of systems and society?

27. © 2017 Health Catalyst Proprietary and Confidential Change (Disruption/Transformative) Management • People generally don’t like change – particularly if their job, career, or ego is on the line. • The bar is not always high enough to overcome inertia – good enough my prevail. • Intelligent automation often unveils inefficiency, waste, fraud, abuse, bureaucracy. • Once projects are underway, results may come much faster than previously experienced. • Fear over job loss can lead to lack of cooperation, feet-dragging, passive aggressive behavior, sabotage, or open hate (Johnson and Johnson’s Sedasys for anesthesiology). • AI can be risky since the mechanisms are often not well understood, the skills and tools may be unfamiliar, and the costs and benefits can be uncertain. 27 How To Succeed • Successful change management requires open and transparent communication, a clear vision of the future, instilling belief that people will succeed under the new system, that there is no path backwards, and that everyone will benefit. • Successful projects require good planning, adequate resources, leadership, governance, accountability, realistic milestones, and the right tools and knowledge of how to use them (example, the right partners). What to Expect

28. © 2017 Health Catalyst Proprietary and Confidential Regulation • Privacy/Security - Because AI, BI, and analytics systems are increasingly being used to deal with complex data with many additional input sources, security and privacy concerns may be higher than with other systems. • Safety – While it is clear that medical devices are subject to a risk-based framework of regulation by the FDA, clinical software applications (e.g., EHRs), analytic models, and clinical decision support remain in a gray area. • Oversight - As systems move from being reference systems, to providing advice, to taking (autonomous) action, the reasons for them to come under regulatory control increase. 28 How To Succeed What to Expect • Ensure data operating systems are especially secure and able to manage multiple data sources. • Understand who will be responsible for regulatory approvals (if any) from the start – recognize that some vendors may tell you a system does not require approvals when it may.

29. © 2017 Health Catalyst Proprietary and Confidential How To Succeed What to Expect Liability • Crossing Boundaries – It’s one thing for a machine to carry out a mundane repetitive task, and another for it to start making decisions that affect health and lives. • With rule-based systems, the decisions can be traced to specific rules that “fire,” and the responsible parties for developing those rules are accountable. • With ML and AI, it’s possible for predictions, answers, and decisions to be made without a complete (or even substantial) understanding of how the result was reached. • As ML and AI take on more complex tasks, its only natural for people to be concerned about “who’s calling the shots” and to what degree ethics or compassion have been incorporated in the solution. • Media attention to such issues as which obstruction should a self-driving car hit if a crash is inevitable (school bus or wheelchair). 29 • Health systems should ensure, to the degree possible, AI systems can be transparent to the decisions they make, traceable to training sets, and accountable to human experts who deploy them – black boxes are a last resort. • Consider initially picking use cases that provide advisory results or are non-clinical – they may be less “sexy,” but will present fewer barriers and risks.

30. © 2017 Health Catalyst Proprietary and Confidential Human Touch – Moving Beyond Informing Decisions One final (future) challenge concern about AI systems as they start to play an increasing role in people’s lives is to what degree they can exhibit human touch vs. being (perceived as) cold and inhuman Fortunately, its possible to have them exhibit compassion and understanding, perhaps with more consistency than some humans, and to be attuned (via facial recognition, voice processing, historical data, etc.) to the emotional/behavioral state of the user. Given the nuance and complexity of human behavior, it’s more than easy to get this wrong, especially if the subject is intentionally trying to derail or game the solution, or exhibits conflicting or new behaviors outside the experience of the system. On the scale of population health, care rationing, costs, etc., these become more organizational, cultural/societal issues. We’re still some time away from machines being on equal footing with humans in deciding courses of action and the future. 30 How To Succeed • Recognize that how users interact with a system and to what degree that system design shows a concern for its results will become increasingly important as humans rely on these systems to a greater degree. What to Expect

32. © 2017 Health Catalyst Proprietary and Confidential What Role Will AI Play in the Future? 32 Will depend on your type of job (tech, care manager, nurse, PA, physician) and your training 3-5 yearsToday Future Minimal experience or skills Most experience or skills Medium experience or skills Assist Suggest Annoy? Super-charge Assist AssistSuggest Super-charge Replace

33. © 2017 Health Catalyst Proprietary and Confidential Dangers of AI 33 I think someone (or something) is trying to target our genetics I don’t know how to do that anymore There’s no future in that career anymore Which of these two systems should I trust? Over-reliance and trust Fewer new professionals enter the field Competing medical knowledge Loss of privacy, exposure of vulnerabilities

34. © 2017 Health Catalyst Proprietary and Confidential Advice • Include machine learning and AI in short and longer-term strategic planning – look to apply these approaches to gain tactical (labor saving), strategic (new markets), and transformational (breakthrough) effects. • Evaluate strategic options for introducing and expanding these technologies into the organization • Determine to what degree and pace to build internal teams and skills (especially quants who can identify and translate technology advances to clinical and business use cases), and to what degree to enlist vendors and external experts as partners. • Place increasing emphasis on first ensuring a solid BI platform to deal with more extensive and complex data sourcing and management. • Address change management – expect to deal with it at an increasing pace – especially in terms of changing job functions and staffing requirements across the continuum. • Continue to look for signposts in other industries and healthcare where advances are making a difference and disrupting existing business models. 34

35. © 2017 Health Catalyst Proprietary and Confidential The Bright Future 35 Awesome collaboration of human and machine Engineered health – genetics (remove disease) Significantly reduced healthcare costs More preventive care and treatment (nanobot repair) Cyborg-capabilities -– implants (senses, motor function, organs) Limitless life – brain scans Greater time for leisure, exploration

38. © 2017 Health Catalyst Proprietary and Confidential Healthcare Analytics Summit 17 Summit highlights Industry Leading Keynote Speakers We’ll hear from well-known healthcare visionaries. We’ll also hear from two C-level executives leading large healthcare organizations. CME Accreditation For Clinicians HAS 17 will again qualify as a continuing medical education (CME) activity. 30 Educational, Case Study, and Technical Sessions We have the most comprehensive set of breakout sessions of any analytics summit. Our primary breakout session focus is giving you detailed, practical “how to” learning examples combined with question and opportunities. The Analytics Walkabout Back by popular demand, the Analytics Walkabout will feature 24 new projects highlighting a variety of additional clinical, financial, operational, and workflow analytics and outcomes improvement successes. Analytics-driven, Hands-on Engagement for Teams and Individuals Analytics will continue to flow through the three-day summit touching every aspect of the agenda. Networking and Fun We’ll provide some new innovative analytics-driven opportunities to network while keeping our popular fun run and walk opportunities and dinner on the down. Sept. 12-14, 2017 Grand America Hotel Salt Lake City, UT ERIC J. TOPOL Author, The Patient Will See You Now and The Creative Destruction of Medicine. Director, Scripps Translational Science Institute DAVID B. NASH, MD. MBA Dean, Jefferson School of Population Health JOHN MOORE Founder and Managing Partner, Chilmark Research ROBERT A. DEMICHIEI Executive Vice President and Chief Financial Officer, University of Pittsburgh Medical Center THOMAS D. BURTON Co-Founder, Chief Improvement Officer, and Chief Fun Officer, Health Catalyst DALE SANDERS Executive Vice President, Product Development, Health Catalyst THOMAS DAVENPORT Author , Consultant Competing on Analytics*, , Analyitcs at Work, Big Data at Work, Only Humans Need Apply:Winners and Losers in the Age of Smart Machines. *Recognized by Harvard Business Review editors as one the most important management ideas of the past decade, one of HBR’s ten must-read articles in that magazine’s 90-year history.