Downloaded 10 times
![Stephen Hawking on AI
"Success in creating
effective AI could be the
biggest event in the history
of our civilization…[But]
unless we learn how to
prepare for and avoid the
potential risks, AI could be
the worst event in the history
of our civilization.“
November 2017
2©Hyperion Research 2018](https://image.slidesharecdn.com/theneedfordeeplearningtransparency-180510150100/85/The-Need-for-Deep-Learning-Transparency-2-320.jpg)
![Hyperion Definitions
AI: Machine Learning, Deep Learning
3©Hyperion Research 2018
Artificial Intelligence (AI): a broad, general term for
the ability of computers to do things human thinking
does (but NOT to think in the same way humans
think). AI includes machine learning, deep learning
and other methodologies.
Machine learning (ML): a process where examples are used to
train computers to recognize specified patterns, such as human blue eyes
or numerical patterns indicating fraud. The computers are unable to learn
beyond their training and human oversight is needed in the recognition
process. The computer follows the base rules given to it.
Deep Learning (DL): an advanced form of machine learning that uses
digital neural networks to enable a computer to go beyond its training and
learn on its own, without additional explicit programming or human
oversight. The computer develops its own rules.
“We still don’t have a universally accepted definition of what intelligence is, so it
would be hard to [define] artificial intelligence.” Larry Greenemeier, Associate
Editor for Technlogy Scientific American (March 2018)](https://image.slidesharecdn.com/theneedfordeeplearningtransparency-180510150100/85/The-Need-for-Deep-Learning-Transparency-3-320.jpg)
![The DL Transparency
Challenge Is Pervasive
“Deep learning methods are fairly
opaque to human inspection…
intelligence analysts are unlikely
to trust a system unless they
understand how its results are
achieved.” IARPA
8
“Deep neural networks are
notoriously opaque. If consumers
are to entrust their safety to AI-
driven vehicles or their health to AI-
assisted medical care, they will
want to know how these systems
make critical decisions.” University
of Oxford Future of Humanity
Institute
“The design of the [supercomputer
for oncology] system is such that it
cannot explain why it weighs one
treatment over another.” Stat
report on AI-driven system at
Memorial Sloan-Kettering
Hospital
DL medicine “is inherently
opaque.” Nicholas Price, Univ.
of Michigan health law scholar
©Hyperion Research 2018](https://image.slidesharecdn.com/theneedfordeeplearningtransparency-180510150100/85/The-Need-for-Deep-Learning-Transparency-8-320.jpg)
![Game
IBM Watson defeats best
humans at Jeopardy! game
show.
“I felt like ‘Quiz Show Contestant’
was now the first job that had
become obsolete under this new
regime of thinking computers.”
Ken Jennings, game show
champion who lost to Watson
Life
[The oncology supercomputer
at Memorial Sloan Kettering
Hospital] sometimes suggests a
chemotherapy drug for patients
whose cancer has spread to the
lymph nodes, even when it has
been given information about a
patient whose cancer has not
spread to the lymph nodes.
Slate.com, Septermber 6, 2017
10©Hyperion Research 2018](https://image.slidesharecdn.com/theneedfordeeplearningtransparency-180510150100/85/The-Need-for-Deep-Learning-Transparency-10-320.jpg)
![NHTSA Report: At Minimum, There
Must Be After-the-Fact Transparency
“Vehicles should
record…all available
information relevant to
the crash, so [it] can be
reconstructed.”
“No standard data
elements exist…to use
in determining why an
ADS-enabled vehicle
crashed.”
11
National Highway Transportation Safety Alliance
©Hyperion Research 2018](https://image.slidesharecdn.com/theneedfordeeplearningtransparency-180510150100/85/The-Need-for-Deep-Learning-Transparency-11-320.jpg)
Uploaded byinside-BigData.com
The Need for Deep Learning Transparency
The document discusses the need for transparency in deep learning (DL) and artificial intelligence (AI) due to their opaque nature, which complicates trust and accountability in critical applications like autonomous vehicles and medical systems. It highlights major challenges in the field, including limited data, the lack of standard benchmarks, and issues like 'catastrophic forgetting' in neural networks. Recommendations for addressing the transparency issue include manual and adversarial testing methods, as well as initiatives aimed at making AI systems more understandable to users.
More Related Content
Similar to The Need for Deep Learning Transparency
The Need for Deep Learning Transparency
- 1. The Need forDeep Learning Transparency with Speaker Notes Steve Conway, SVP-Research March 2018
- 2. Stephen Hawking onAI "Success in creating effective AI could be the biggest event in the history of our civilization…[But] unless we learn how to prepare for and avoid the potential risks, AI could be the worst event in the history of our civilization.“ November 2017 2©Hyperion Research 2018
- 3. Hyperion Definitions AI: MachineLearning, Deep Learning 3©Hyperion Research 2018 Artificial Intelligence (AI): a broad, general term for the ability of computers to do things human thinking does (but NOT to think in the same way humans think). AI includes machine learning, deep learning and other methodologies. Machine learning (ML): a process where examples are used to train computers to recognize specified patterns, such as human blue eyes or numerical patterns indicating fraud. The computers are unable to learn beyond their training and human oversight is needed in the recognition process. The computer follows the base rules given to it. Deep Learning (DL): an advanced form of machine learning that uses digital neural networks to enable a computer to go beyond its training and learn on its own, without additional explicit programming or human oversight. The computer develops its own rules. “We still don’t have a universally accepted definition of what intelligence is, so it would be hard to [define] artificial intelligence.” Larry Greenemeier, Associate Editor for Technlogy Scientific American (March 2018)
- 4. Forecast: HPDA Market andML/DL/AI Methods 4©Hyperion Research 2018
- 5. WW M/L, D/L,& AI Forecasts 5©Hyperion Research 2018
- 6. AI/Deep Learning MajorChallenges MARKET STATUS HPC has moved to the forefront of DL/AI research Ecosystem (including GPGPUs) formed around social media/Web giants DL needs massive data: not available yet in many markets Lack of standard benchmarks lengthens sales process Need for transparency HPC simulation! Lots of time/money being spent to get there 6 “The amount of data available today is miniscule compared to what we need for deep learning.” Marti Head, GlaxoSmithKline ©Hyperion Research 2018
- 7. The AI/Deep Learning(DL) Challenge We can’t teach machines to think like humans, because we don’t fully understand how humans think. DL machines can learn on their own -- beyond the instructions humans give them. DL machines are capable of learning from each other, i.e., they are capable of culture* * Aggregation of knowledge among individuals over time Today, DL learning is largely opaque to humans -- the basis for DL inferences/ decisions is unclear. DL “black boxes” need to be made transparent. 7©Hyperion Research 2018
- 8. The DL Transparency ChallengeIs Pervasive “Deep learning methods are fairly opaque to human inspection… intelligence analysts are unlikely to trust a system unless they understand how its results are achieved.” IARPA 8 “Deep neural networks are notoriously opaque. If consumers are to entrust their safety to AI- driven vehicles or their health to AI- assisted medical care, they will want to know how these systems make critical decisions.” University of Oxford Future of Humanity Institute “The design of the [supercomputer for oncology] system is such that it cannot explain why it weighs one treatment over another.” Stat report on AI-driven system at Memorial Sloan-Kettering Hospital DL medicine “is inherently opaque.” Nicholas Price, Univ. of Michigan health law scholar ©Hyperion Research 2018
- 9. Game Google’s DeepMind (AlphaGo) defeatsthe best humans. “We still can’t explain it…you could…review…every parameter in AlphaGo’s artificial brain, but even a programmer would not glean much from these numbers because what drives a neural net to make a decision is encoded in the billions of diffuse connections between nodes.” Alan Whitfield, robot ethicist, Univ. of the West of England Life Self-Driving Uber Vehicle Strikes and Kills Pedestrian Washington Post, March 19, 2018 Automakers, insurance companies & families need to know why. 9 ©Hyperion Research 2018
- 10. Game IBM Watson defeatsbest humans at Jeopardy! game show. “I felt like ‘Quiz Show Contestant’ was now the first job that had become obsolete under this new regime of thinking computers.” Ken Jennings, game show champion who lost to Watson Life [The oncology supercomputer at Memorial Sloan Kettering Hospital] sometimes suggests a chemotherapy drug for patients whose cancer has spread to the lymph nodes, even when it has been given information about a patient whose cancer has not spread to the lymph nodes. Slate.com, Septermber 6, 2017 10©Hyperion Research 2018
- 11. NHTSA Report: AtMinimum, There Must Be After-the-Fact Transparency “Vehicles should record…all available information relevant to the crash, so [it] can be reconstructed.” “No standard data elements exist…to use in determining why an ADS-enabled vehicle crashed.” 11 National Highway Transportation Safety Alliance ©Hyperion Research 2018
- 12. June 2017: GermanyPasses First Law Governing Autonomous Vehicles 14-person Ethics Commission led by Transportation Minister All cars operating in Germany must let humans take control. Accidents: • If car’s in control, automaker is liable • If person’s in control, person is liable Cars can’t be programmed demographically: • E.g., allow an elderly person to die before a baby March 2018: China invites member of German Ethics Commission to advise on China’s ADS law. 12 Report: “Autonomous and Networked Driving” ©Hyperion Research 2018
- 13. Approaches to theDL Transparency Problem Manual testing • Humans feed test images into the network until they spark a wrong decision 13 Adversarial testing • Computer automatically tweaks a specific image until it causes a wrong decision Deep Xplore • Assumes neural networks usually make the right decision. • Polls 3 or more neural networks and retrains the dissenting network to conform to the majority. “The goal of DARPA’s Explainable Artificial Intelligence (XIA) program is make AI-controlled systems more accountable to their human users.” ©Hyperion Research 2018
- 14. Another DL Issue:“Catastrophic Forgetting” When a neural network encounters something it wasn’t trained to recognize, it tends to make the same mistake over and over again. Attempts to retrain the network on the fly lead to “catastrophic forgetting”: learning the new thing disrupts the device’s prior knowledge. • With humans, moving a tennis net one foot wouldn’t require re-learning the game of tennis. © Hyperion Research 14 Working on it…
- 15. ADS Traffic: MultipleControl Levels 15©Hyperion Research 2018
- 16. HPC Simulation IsNot Going Away Any Time Soon Neural networks are not good today at simulating physics under varying conditions. Even if they improved, the transparency issue would make if difficult to validate scientific use cases. “There are many things we have been getting right in the simulation community based on 30-40 years of applied math, and so a black box, as good as it may be, will not be suitable until these things are worked out.” NERSC’s Prabhat. NextPlatform, 1/26/18 16©Hyperion Research 2018
- 17.
Editor's Notes
- #3 As world-renowned physicist Stephen Hawking has said, in coming years artificial intelligence will bring about a major transformation in the lives of human beings. This transformation could be strongly positive or strongly negative, depending on which decisions people make concerning AI. This dilemma is akin to the ones people have faced with other great scientific breakthroughs, such as the splitting of the atom or the discovery of the double-helix structure of DNA, except that AI may affect humanity even more profoundly and pervasively. Today, much of the world is understandably caught up in the “hype” cycle about AI, imagining AI’s benefits when applied to self-driving vehicles, precision medicine, smart power grids, cyber security and other fields. Much less attention is being given to the risks associated with these impressive advances. Behind the scenes, however, ethicists, researchers, politicians and others are working to address known risks associated with AI, so this emerging field can develop with maximum benefit and minimal harm. One of the largest risks is the lack of AI transparency—today we can’t “see” and understand how AI-enabled devices make important, sometimes life-changing decisions. This slide presentation attempts to describe the transparency problem and efforts that are under way to address it.
- #4 Let’s start by providing the definitions Hyperion Research uses for AI and for the associated terms machine learning and deep learning. We treat machine learning and deep learning as special cases of AI. We treat AI, machine learning and deep learning as methodologies rather than markets, because they can be applied across many vertical and horizontal markets. For that reason, we treat these terms much like we treat established methodologies such as computational fluid dynamics (CFD) or finite element analysis (FEA). But we also track HPC revenue associated with AI, ML and DL as if they were markets, because of the great interest on the part of our clients and the HPC community in following the growth of these methodologies.
- #5 Hyperion’s taxonomy goes like this: At the highest level, we divide the HPC server system market into compute-intensive and data-intensive systems, based on the majority (51% or more) of the work a system is acquired to perform. We call the data-intensive portion of the market high performance data analysis, or HPDA. It includes both data-intensive simulation and data-intensive analytics. We categorize AI, including AI methods such as machine learning and deep learning, as part of the HPDA market. Table 1 shows our forecast that the HPDA server market will grow at a robust 17% CAGR to reach about $4 billion in 2021. HPC systems used primarily (51% or more) to run AI methodologies will grow at an even-faster 29.5% CAGR to reach about $1.2 billion in 2021.
- #6 Figure 2 shows the AI forecast as a bar chart.
- #7 Not all AI activity uses HPC resources today. A lot happens on desktops and enterprise servers. But it’s clear the most advanced AI R&D is generally happening on HPC systems, because of their superior processing power, communication rates and memory subsystems. AI, and especially deep learning, needs extremely large data volumes to produce accurate results. Data volumes that large have been available to the giant Internet and social media companies, which is why the early deep learning ecosystem has evolved mostly around their requirements. Many other markets that will benefit from AI and deep learning, like precision medicine and automated driving systems, have requirements that are very different from those of the social media giants. And even the largest companies in many of these markets don’t have access today to nearly enough data to perform deep learning with the needed level of accuracy. As former Baidu executive Andrew Ng has said, “There’s a big difference between 96% and 99% accuracy.” Mega-companies like United HealthGroup are spending hundreds of millions of dollars to acquire more data for precision medicine. Another issue is the lack of useful industry-standard benchmarks for AI. Vendors tell us they often have to spend 3-4 weeks with each prospect to agree on what would constitute success in a project. We’ll get to the need for transparency in a minute. For now, it’s important to know that even though most AI activity is still in the R&D/exploratory phase, buyers are already spending a lot of money on HPC and other IT gear because they know they’ll need this capability to stay competitive in 2-3 years.
- #8 Some things about AI and deep learning Hyperion considers important to keep in mind: First, we can teach machines to do things the human brain does. We’ve already taught computers how to do math and to simulate physical phenomena. But we will not be able to teach machines how to think like humans until we understand better how humans think. That’s not likely to happen anytime soon, despite impressive progress in fields ranging from psychology to medical brain research. We do know, however, that machines can learn on their own. They can go beyond the instructions humans have programmed into them. Experiments have shown that machines can also learn from each other, and these lessons can be transmitted to still other computers. That ability is called culture. Humans have it. Some other great apes have it. And now, some computers have it. This should allow humans to train one computer and have that computer train others in the future. I said before that we humans don’t fully understand how humans think. When it comes to deep learning, humans also don’t understand yet how computers think. That’s a big problem when we’re entrusting our lives to self-driving vehicles or to computers that diagnose serious diseases, or to computers installed to protect national security. We need to find a way to make these “black box” computers transparent.
- #9 Here are just a few of many statements by experts on the problem of AI transparency. There’s a strong consensus that neural networks today are opaque to human inspection, and that this is a serious problem.
- #10 It’s important to distinguish between applying AI in games and applying it in life-threatening situations. As the quote in the left-hand panel says, when Google’s DeepMind computer beat the best humans at the AlphaGo game, even Google couldn’t explain how the computer had done this—because this deep learning computer had gone beyond its programming to learn and act on its own, and that process was not transparent. But if a self-driving car faces an unavoidable accident and decides to kill a pedestrian walking along the sidewalk instead of colliding with an oncoming vehicle, the pedestrian’s family, the owners of the colliding vehicles, their insurance companies, and the automakers in question will want to know exactly why the computer made this decision, so that decisions like this can be avoided in the future.
- #11 Another example. IBM Watson amazed the world by beating the best humans at the Jeopardy! television game show. That early version of Watson was less complex than today’s Watson and other deep learning computers, but even then Watson’s decision-making process wasn’t entirely transparent. But in recent years, deep learning computers have started to be deployed as decision-support systems that help physicians diagnose and recommend treatments for serious illnesses, based on evaluating enormous volumes of data from the medical literature, millions of anonymized patient records, and the history of the patient under investigation. A deep learning oncology supercomputer at Memorial Sloan-Kettering Hospital in New York produced suboptimal recommendations because the medical data entered by the hospital’s staff turned out to be biased. That could have jeopardized patients’ health if it hadn’t been discovered through an evaluation of the project. This is another example of why AI needs to become more transparent.
- #12 As I said before, there are people working to address the AI transparency problem. America’s National Highway Transportation Safety Alliance, or NHTSA, is part of the U.S. Department of Transportation. Last year, NHTSA published a report that described the transparency problem and made some recommendations for addressing it, though none that have been enacted into law yet. In essence, the report recommended that all self-driving vehicles contain a device that records data related to an accident, to help investigators reconstruct the accident. This is similar to the flight data recorders that are required in the aviation industry.
- #13 Germany has gone even further than this, by passing a comprehensive law governing self-driving vehicles that will operate in Germany. The committee that produced the report on which the law is based worked for two years. It was led by the Transportation Minister and also included an ethicist and a highly respected religious leader. In essence, the German law is designed to safeguard human autonomy and human primacy where self-driving vehicles are concerned. Humans must always have the option to take control of the vehicle. If a person is in control when an accident happens—and a recording device will determine who was in control—then the person is liable. If the car’s in control during an accident, the automaker is liable. That provision will put a lot of pressure on automakers, including foreign automakers selling into Germany, to design their self-driving vehicles for collision avoidance. It’s easy to imagine a rash of class-action lawsuits if a fatal accident exposes a design flaw in a self-driving vehicle. Self-driving cars operating in Germany also can’t be programmed to decide that an older person should die rather than a baby, or a limo driver rather than the driver’s wealthy passengers.
- #14 Here’s are some of the major approaches to addressing the AI transparency problem. The first is manual testing. Think of this as a kind of stress test, where humans keep presenting the computer with new, varying images until the computer makes a bad decision. This helps programmers go back and fix the problem. A relative of manual testing is adversarial testing. Here the same image is changed again and again until the computer makes a bad decision about it, meaning that the computer fails to recognize it when it ought to. Again, this lets programmers fine-tune the machine learning or deep learning program. Another promising approach basically involves giving the problem to several neural networks and then taking a vote—with the assumption that in most cases they will all agree with each other, or only one of them will make a different decision. When that happens, the majority decision is used and the dissenting network is retrained to conform to the majority. Like the other approaches I’ve just described, this polling method isn’t a foolproof solution to the problem but it reduces the risk that a bad decision will be acted on.
- #15 Transparency isn’t the only AI issue today. Another important one, though less potentially life-threatening for humans, is called catastrophic forgetting. This is based on the fact that neural networks used for deep learning sometimes make mistakes. When they do, they tend to repeat the same mistake again and again. In these cases, the neural network typically needs to be retrained. That’s where the problem emerges. It turns out that when you try to retrain a neural network in real time, the network has a tendency to forget all of its prior training. In the human realm, this is analogous to moving a tennis net one foot and having this cause the tennis players to forget how to play the game of tennis. DARPA and other are working to overcome this problem.
- #16 This is how an urban traffic management system might look in the era of self-driving vehicles. Each vehicle is equipped with a data recorder that continuously collects and reports trip data, including detailed data related to any accidents. Each vehicle is networked with other vehicles within a specified distance and is also networked to a citywide control center. (As the above photo shows, urban traffic control centers already exist in many cities and will presumably evolve their capabilities so they can handle self-driving traffic.) Analogous to flight control centers, when a self-driving vehicle leaves the management area of one traffic control center, it’s picked up by the center controlling the adjacent area. In real time, the control centers process data from vehicles and use this to optimize traffic flow and route around accidents and other obstructions. Urban traffic management will be an integral component of smart cities, along with smart power grid management and smart building management. In turn, smart cities will become dense locations on IoT infrastructures. Hyperion Research expects HPC systems to be important for managing these infrastructures at local, regional and national levels.
- #17 As this presentation suggests, the coming AI era promises to produce exciting new capabilities, based primarily on the use of advanced analytics. It’s important to keep in mind, however, that HPC-based simulation has made enormous contributions to science and engineering. Even as HPC-based advanced analytics have become more important in recent years, the market for HPC simulation has continued to grow at a healthy rate. For the foreseeable future, Hyperion expects modeling and simulation to continue to be a crucially important methodology for science and engineering and also to have an important role in advanced data analysis. Overcoming the AI transparency issue will liberate exciting new AI methodologies but will not replace the important roles occupied by simulation.