The failure of IBM Watson, disappointments of self-driving vehicles, slow diffusion of medical imaging, small markets for AI software, and scorching criticisms of Google’s research papers provide evidence for hype and disappointment in AI, which is consistent with negative social impact of Big Data and AI algorithms. There are some successes, but they are much smaller than the predictions, with virtual applications (advertising, news, retail sales, finance and e-commerce) having the largest success, building from previous Big Data usage in the past. Looking forward, AI will augment not replace workers just as past technologies did on farms, factories, and offices. Robotic process automation and natural language processing are likely to play important roles in this augmentation with RPA automating repetitive work, natural language processing summarizing information, and RPA also putting the information in the right bins for engineers, accountants, researchers, journalists, and lawyers. Big challenges include reductions in training time depending on faster computers, exponentially rising demands on computers for high accuracies in image recognition, a slowdown in supercomputer improvements, datasets riddled with errors, and reproducibility problems.

1. The State of AI
in Applications
AI Moonshots are failing and
most successes are in the virtual world.
Big challenges for moving forward.
BEYOND THE HYPE Organised by DATAWorkout
Thought Leader Forum
Dr Jeffrey Funk
Live Stream 15, 16 & 17 June 2021
Formerly National University of Singapore
https://dataworkout.com/beyond-the-hype-forum/on-demand/

2. BEYOND THE HYPE Organised by DATAWorkout
Thought Leader Forum
Dr Jeffrey Funk
Live Stream 15, 16 & 17 June 2021
Formerly National University of Singapore
PREDICTED
GDP
REAL
GDP
US$1.2 Trillion US$17 Billion
PwC’s 2016
predictions
for 2020
Revised projections for 2025
are US$37 billion
Market size is nowhere near the forecasts!

3. Failure of Healthcare Moonshot
• IBM Watson was hyped for years, until it wasn’t…
• And no form of AI has diffused widely in Healthcare
• Survey: only 1/3 of hospitals and imaging centers report using any type of AI “to aid
tasks associated with patient care imaging or business operations,”
• 2020 Mayo Clinic and Harvard survey: only 14% of staff said they would recommend the
system to another clinic
• Global market for AI-based imaging software was only $400 million in 2020, a tiny
fraction of $22.8 billion global healthcare software market

4. Disappointments in Radiology
• Turing Award Winner Geoffrey Hinton claimed in 2016 radiologists
would be replaced in five years, but their numbers are still rising
• In Survey of American radiologists
• Only 11% reported using AI for image interpretation in 2020 in clinical
practice, 33% if research and other applications are included
• Of those not using AI, 72% have no plans to do so while about 20% want
to adopt within five years
• “Concerns over inconsistent performance………have made the actual use
of AI in clinical practice modest."

5. Behind Inconsistent Performance, Andrew Ng
• “When we collect data and test from Stanford Hospital, we can show algorithms are
comparable to human radiologists in spotting certain conditions.” But, “when you
take same model/AI system to an older hospital/machine down the street, and
technician uses slightly different imaging protocol, data drifts to cause performance
of AI system to degrade significantly. In contrast, any human radiologist can walk
down the street to the older hospital and do just fine.“
• “All of AI, not just healthcare, has a proof-of-concept-to-production gap.”
• “A good rule of thumb is that you should estimate that for every $1 you spend developing an
#algorithm, you must spend $100 to deploy and support it.”

6. Scientists Question Google’s Healthcare Research
• Breast cancer paper
• “we see another very high-profile journal publishing a very exciting study that has
nothing to do with science. It’s more an advertisement for cool technology. We can’t
really do anything with it”
• Protein folding paper
• “Until DeepMind shares their code, nobody in the field cares and it’s just them
patting themselves on the back.”
• Remember Google Flu’s claims 5 years ago?
• It over-estimated number of flu cases for 100 of next 108 weeks, by an average of
nearly 100 percent, before being quietly retired

7. BEYOND THE HYPE Organised by DATAWorkout
Thought Leader Forum
Live Stream 15, 16 & 17 June 2021
Even Google’s Sundar Pichai now sounds pessimistic
“Still early days of AI,
real potential to come in place in 10-20 years“

8. Last year’s World Economic Forum:
The impact of AI could be “more profound than fire or electricity.”
This year’s WEF admits AI didn’t play a significant role
in devising a vaccine for Covid, instead backtracking:
“AI is laying a foundation to tackle future problems and it can
play a much bigger role in tackling future pandemics."
“Still early days of AI,
real potential to come in place in 10-20 years“
Even Google’s Sundar Pichai now Sounds Pessimistic

9. Failure of AV Moonshot
“I really consider AUTONOMOUS DRIVING a solved problem,” Musk
said in 2016. “I think we are probably less than two years away.”
• By late 2018, it was clear that self-driving cars were much harder than
originally thought, WSJ: “Driverless Hype Collides with Merciless
Reality.”
• By 2020, Zoox, Ike, Kodiak Robotics, Lyft, Uber, and Velodyne began
layoffs, bankruptcies, revaluations, and liquidations at deflated prices,
along with sale of AV units
• An MIT Task Force announced in mid-2020 that fully driverless
systems will take at least a decade to deploy over large areas
Already 100s
of $Billions
Have been
Spent on R&D

10. Open AI’s GPT-3
• GPT-3 interprets and creates text by observing statistical relationships
between words and phrases, but it doesn’t understand their meaning
• It will give nonsensical answers (“A pencil is heavier than a toaster”) or outright
dangerous replies
• “Some experts call language models “stochastic parrots” because they echo what they
hear, remixed by randomness, or call them “a mouth without a brain.”
• Nature Digital Medicine said that AI
• will not replace a conversation with another human being for the foreseeable future
in clinical settings
• But CEO of OpenAI continues hype in essay: “Moore’s Law for Everything.”
• Imagine a world where, for decades, housing, education, food, clothing, etc., all
halved [in cost] every two years”

11. Failure of AI Moonshot in Government
• Many organizations are using algorithms to decide which
• children enter foster care
• patients receive medical care
• families get access to stable housing
• and the bail amounts for arrested suspects
• Example: Government did not reveal an algorithm had determined
cutoff from Medicare until victim was in front of judge
• The witness, a nurse, couldn’t explain anything about the algorithm
because it was bought off the shelf.”
• There are hundreds of these stories documented in Weapons of
Math Destruction (2016) and other articles and books

12. There are Some Successes………..
• In virtual world of advertising, news, finance, retail, e-commerce
• as an outgrowth of data analytics and software automation in past
• Also, some successes in
• Robotic process automation for white-collar workers
• But more augmentation than replacement
• And overall, “Companies Are Rushing to Use AI—but Few See a Payoff,”
• only 11% of firms that have deployed AI are reaping “sizable” return on their
investments

13. Big Challenges Moving Forward
• Falling training time, but mostly from using more computers
• Supercomputer slowdown
• Exponentially rising demands on computers for high
accuracies
• Datasets riddled with errors
• Reproducibility
A lot of focus on getting more
data to reduce bias, but these
challenges suggest success will
take many years

14. Stanford
AI Index
Report
Training
Time
IMAGENET: TRAINING TIME and ACCELERATORS
(COMPUTING POWER) of BEST SYSTEM
Most reductions in training time
came from using more accelerators
(i.e., computers)
So 80% of improvements came
from more computing and only
20% from better other things
Training costs fell by 30%, partly
because cost of computers fell
But declines in computer costs are
slowing & higher accuracies
require much higher costs

15. Improvements in Supercomputers Have Slowed:
Annual Increases Since 2013 Are Much Smaller
Supercomputers
Source: After Moore’s Law:
How Will We Know How Much
Faster Computers Can Go?
(or how fast can AI progress)
Fractional
Increase
Per
Year

16. Benchmark Error Rate Computation
Required (Gflops)
Environmental
Cost (CO2)
Economic
Cost ($)
Image Net Today: 11.5% 10 14 10 4 10 6
Target 1:5% 10 19 10 10 10 11
Target 2: 1% 10 21 10 20 10 20
MS COCO Today: 47% 10 14 10 4 10 4
Target 1: 30% 10 23 10 14 10 15
Target 2: 10% 10 44 10 36 10 36
Exponential Increases in Computation and Cost to
Achieve Higher Accuracies
Conclusion
from
State
of
AI
(Artificial
Intelligence)
Report

17. “Datasets Riddled with Errors”
• Researchers found incorrect labels on 6% of images in ImageNet
• Big reason: data typically collected and labeled by low-paid workers
• Big data sets are essential to how AI systems built and tested
• Millions of road scene images help AVs perceive road obstacles
• Labeled medical records help algorithms predict chance of diseases
• Fixing this problem requires
• More expensive data collection, showing images to more people, or
even discarding notion that labels are useful

18. Reproducibility
• 85% of studies using machine learning to detect Covid in chest scans failed
reproducibility test (and none are ready for use in clinics)
• According to Nature editorial: "biomedical literature is littered with studies
that have failed test of reproducibility,” …… “due to failure to fully disclose
software and data.“
• A recent review (Nature) found only 15% of AI studies shared their code
• An international group of scientists (Nature) is demanding more transparency
• Problems extend to all scientific disciplines, particularly drug research

19. Moving Forward
• Reduce Hype
• Less Emphasis on Moonshots
• Focus on Success Stories
• Where is AI raising productivity now?
• Probably in less demanding applications that require lower accuracy
• More data is needed, but to be beneficial we must
• Fix Datasets: labels should be correct close to 100%
• Demand reproducibility: academic papers should disclose code, data sets, etc.