Stanford University: Artificial Intelligence Index Report, 2021
The AI Index is an effort to track, collate, distill and visualize data relating to artificial intelligence. It aspires to be a comprehensive resource of data and analysis for policymakers, researchers, executives, journalists, and the general public to develop intuitions about the complex field of AI.
Source: https://aiindex.stanford.edu/wp-content/uploads/2021/03/2021-AI-Index-Report_Master.pdf
Artificial Intelligence in Healthcare Market Forecast to 2024stella thomson
The report analyzes the artificial intelligence in healthcare market, By Component (Hardware, Software and Services), By Technology (Natural Language Processing, Machine Learning and Others), By End User (Healthcare Providers, Pharmaceutical Organizations and Others). Request free Sample of this Report @: http://azothanalytics.com/report/healthcare-pharma/artificial-intelligence-in-healthcare-market-world-market-analysis-by-component-hardware-software-services-technology-natural-language-processing-machine-learning-end-user-by-region-by-country-2019-edition-forecast-to-2024-r33199
Visit us: http://azothanalytics.com/research/healthcare-pharma-c1
McKinsey Quarterly
This Quarter
Regardless of whether China’s new leaders want to reshape the country’s economy, it is changing around them. The growth rate is slowing. Neither exports nor investment will be the engines that they once were. And public policies will inevitably reflect these shifts. China’s next chapter, in short, is going to look decidedly different from the one we’ve grown accustomed to.
MarketResearchReports.com has announced the addition of “Forecast of Global Power Distribution Units (PDU) Market 2024” research report to their offering. See more at- http://mrr.cm/w2a
$57.6 billion is expected to be invested in artificial intelligence and automation technologies by 2021. Here are 10 examples of this investment at work.
The promise — and potential — of blockchain to drive social impact is massive, but how much of it is hype and how much is reality? This slideshow includes highlights and case studies from the Stanford Graduate School of Business study “Blockchain for Social Impact: Moving Beyond the Hype.” It includes data on the landscape as a whole, as well as spotlights on eight sectors: Agriculture, Democracy, Identity, Energy, Financial Inclusion, Health, Land Rights, and Philanthropy.
The Amazing Ways YouTube Uses Artificial Intelligence And Machine LearningBernard Marr
YouTube uses AI and machine learning in several ways:
1) It automatically removes 76% of objectionable content using AI classifiers before it receives any views.
2) Its "Up Next" recommendations are powered by an AI system that analyzes user watch histories and assigns videos scores to determine recommendations.
3) Google researchers have used YouTube videos to train AI models to perform tasks like swapping backgrounds in videos and discerning depth in images.
Cisco Visual Networking Index: Forecast and Trends, 2017–2022ITSitio.com
Global IP traffic is projected to nearly triple from 2017 to 2022, reaching 4.8 ZB per year or 396 EB per month by 2022. Key drivers of this growth include the proliferation of internet-connected devices, rapid growth of mobile data usage, and increasing consumption of video content. The number of networked devices will exceed the global population by 2022, reaching 28.5 billion total connections worldwide. Mobile data traffic will increase sevenfold during this period due to rising smartphone and M2M usage. By 2022, IP video will account for 82% of all IP traffic, up from 75% in 2017, growing at a CAGR of 29%.
The Chinese government has set ambitious goals in its big data industry development to foster new economic drivers. One of these goals is e.g. to increase the annual sales of China’s big data industry (including related goods and services) to RMB 1 trillion by 2020 from an estimated RMB 280 billion in 2015. This report examines the Chinese big data industry and its innovators along with possible future opportunities and implications that China's expanding big data industry could entail for Finland.
Artificial Intelligence in Healthcare Market Forecast to 2024stella thomson
The report analyzes the artificial intelligence in healthcare market, By Component (Hardware, Software and Services), By Technology (Natural Language Processing, Machine Learning and Others), By End User (Healthcare Providers, Pharmaceutical Organizations and Others). Request free Sample of this Report @: http://azothanalytics.com/report/healthcare-pharma/artificial-intelligence-in-healthcare-market-world-market-analysis-by-component-hardware-software-services-technology-natural-language-processing-machine-learning-end-user-by-region-by-country-2019-edition-forecast-to-2024-r33199
Visit us: http://azothanalytics.com/research/healthcare-pharma-c1
McKinsey Quarterly
This Quarter
Regardless of whether China’s new leaders want to reshape the country’s economy, it is changing around them. The growth rate is slowing. Neither exports nor investment will be the engines that they once were. And public policies will inevitably reflect these shifts. China’s next chapter, in short, is going to look decidedly different from the one we’ve grown accustomed to.
MarketResearchReports.com has announced the addition of “Forecast of Global Power Distribution Units (PDU) Market 2024” research report to their offering. See more at- http://mrr.cm/w2a
$57.6 billion is expected to be invested in artificial intelligence and automation technologies by 2021. Here are 10 examples of this investment at work.
The promise — and potential — of blockchain to drive social impact is massive, but how much of it is hype and how much is reality? This slideshow includes highlights and case studies from the Stanford Graduate School of Business study “Blockchain for Social Impact: Moving Beyond the Hype.” It includes data on the landscape as a whole, as well as spotlights on eight sectors: Agriculture, Democracy, Identity, Energy, Financial Inclusion, Health, Land Rights, and Philanthropy.
The Amazing Ways YouTube Uses Artificial Intelligence And Machine LearningBernard Marr
YouTube uses AI and machine learning in several ways:
1) It automatically removes 76% of objectionable content using AI classifiers before it receives any views.
2) Its "Up Next" recommendations are powered by an AI system that analyzes user watch histories and assigns videos scores to determine recommendations.
3) Google researchers have used YouTube videos to train AI models to perform tasks like swapping backgrounds in videos and discerning depth in images.
Cisco Visual Networking Index: Forecast and Trends, 2017–2022ITSitio.com
Global IP traffic is projected to nearly triple from 2017 to 2022, reaching 4.8 ZB per year or 396 EB per month by 2022. Key drivers of this growth include the proliferation of internet-connected devices, rapid growth of mobile data usage, and increasing consumption of video content. The number of networked devices will exceed the global population by 2022, reaching 28.5 billion total connections worldwide. Mobile data traffic will increase sevenfold during this period due to rising smartphone and M2M usage. By 2022, IP video will account for 82% of all IP traffic, up from 75% in 2017, growing at a CAGR of 29%.
The Chinese government has set ambitious goals in its big data industry development to foster new economic drivers. One of these goals is e.g. to increase the annual sales of China’s big data industry (including related goods and services) to RMB 1 trillion by 2020 from an estimated RMB 280 billion in 2015. This report examines the Chinese big data industry and its innovators along with possible future opportunities and implications that China's expanding big data industry could entail for Finland.
This report begins with an examination of the global IoT industry and continues by looking into the Chinese IoT industry and its innovators. The report concludes with an analysis of the possible future opportunities and implications that China's expanding IoT industry could entail for Finland.
The document discusses the potential economic impact and value of artificial intelligence (AI) technologies. Some key points:
- Global GDP could be 14% higher by 2030 due to AI, equivalent to an additional $15.7 trillion in economic value. China and North America are expected to see the largest boosts of up to 26% and 14% respectively.
- The majority of GDP gains will come from increased productivity and consumption enabled by AI. Productivity gains will be driven by automation of processes and augmentation of human workers. Consumption gains will come from personalized and higher quality AI-enhanced products and services.
- Retail, financial services, and healthcare are identified as sectors that could see the biggest gains from AI
Future Watch: China's Digital Landscape and Rising Disruptors - Module 2.6 Ar...Team Finland Future Watch
AI is transforming industries in China just as we expect it to revolutionize our industries and organizations. As witnessed in many existing practical AI applications already available today, AI is not a futuristic concept anymore. The question today is, who will lead the next AI revolution? In many terms China can be the driving force of this development.
Hacktivists in Indonesia disrupted Malaysian websites in response to a misprinted flag by exploiting vulnerabilities in Malaysian web servers and replacing website content with protest messages. Ransomware like WCry and Petya have impacted organizations across Southeast Asia by leveraging vulnerabilities in outdated operating systems. Distributed denial of service (DDoS) attacks remain a frequent tactic used by threat actors to disrupt websites and draw attention to causes or hide other malicious activities.
Generation-IoT - The Key to Business Survival in 21st CenturyDr. Mazlan Abbas
This document discusses the importance of IoT (Internet of Things) for business survival in the 21st century. It notes that organizations need to reinvent themselves every 3-7 years to survive, and only 19% of S&P 500 companies from 50 years ago remain. The document outlines the three waves of internet (fixed, mobile, IoT) and explains IoT components and maturity phases from monitoring to autonomous. It argues that connecting assets benefits makers, users, and the operating environment. The large market potential of IoT in sectors like transportation, healthcare, manufacturing is also highlighted. The document emphasizes the growing demand for IoT professionals and calls the audience the "Generation IoT" that can help organizations succeed through IoT.
The United States government is a major supporter of artificial intelligence (AI) research and development through agencies like DARPA, NSF, NIH, ONR, and IARPA. (1) Finland and the U.S. are projected to see the largest economic gains from AI through 2035 with each seeing 2% higher economic growth. (2) Many industries like manufacturing, professional services, retail, and healthcare are expected to see large increases in value added from AI. (3)
Study: Analytics and Data Science Jobs in India: 2020 – By Great Learning & AIMSrishti Deoras
This report outlines the functional analytics skills and programming languages that are most in-demand in the market. The report provides insights for recruiters and hiring companies so that they can study the demand for skills across the analytics function, and can identify and close any capability gaps across workforces. By highlighting the talent hotspots in the country, the report enables organisations to build a steady talent pipeline.
State of Artificial Intelligence in India 2020Srishti Deoras
Report developed by AIMResearch, in association with Jigsaw Academy, on the Artificial Intelligence Market in India as AI emerges as one of the primary Data Science functions across Enterprises in India
Impact on Jobs across Emerging Technologies During the Current Pandemic Crisi...Srishti Deoras
Analytics India Magazine (AIM) along with Jigsaw Academy, has developed this study to focus on the impact on jobs across certain emerging technologies.
The document analyzes characteristics of the Chinese internet market. It finds that China has one of the world's largest internet markets in terms of users and online spending. The market has also experienced very strong growth. However, potential remains as internet penetration is still lower than other G20 countries. The market is characterized by young users who prefer mobile access and try new apps quickly. Competition is also intense as ecosystems form around major players.
Hasil penelitian (Economic Impact Study) terbaru oleh Google dan Deloitte Access Economics yang berisi temuan dan rekomendasi penting yang bermanfaat bagi pelaku bisnis dan pembuat kebijakan.
The document discusses how digital technologies and hyperconnectivity are transforming business and society. It notes that with the rise of the internet of things and big data, physical objects, information, and processes can now all be connected and share data. This new level of connectivity allows new ways of creating value for both businesses and society. The document advocates for a "human centric" approach where technology enhances people's experiences and helps address societal challenges. It also discusses how both digital and physical businesses can harness digital transformation to innovate and grow.
China is already the world’s second largest ICT market place. By 2020 the market is estimated to reach USD 850 billion. For the Chinese government, digital, ICT an AI market is not only seen as a business, but it is increasingly seen through national security and social stability lenses. Furthermore, policies to ensure security often appear to do so at the expense of foreign companies. The Cyber Security Law from 2017 has increasingly tightened the operational landscape for foreign companies in China.
China’s population is ageing rapidly and the country is looking into robotics and AI in order to offset the current and especially the future costs related to the country’s greying population. Finnish healthtech companies might be able to find opportunities related to this, especially those that provide digital or AI-enabled solutions for elderly care.
A white paper I authored about the major headwinds confronting the wearables market, with some interesting new data point around wearables abandonment and acquisition.
The document discusses how insurance is facing significant disruption from social, technological, economic, environmental, and political changes between now and 2020. These changes include a more fragmented customer base, rising digital connectivity and data availability, slowing economic growth in developed markets coupled with faster growth in emerging markets, increasing catastrophe risks, and greater political instability. Insurers will need to reinvent their business models to adapt to these trends and changing customer expectations in order to remain competitive. The document examines the implications of these changes and how insurers can design business strategies to succeed in this disrupted future.
Predictive TMT Analysis from CM ResearchCM Research
This document summarizes the past predictive analysis and investment ideas of CM Research, an independent research firm. Over the past two years, CM Research correctly predicted investment themes including mobile internet, connected devices, cybersecurity, 3D printing, mobile payments, robotics, internet advertising, big data, and UK tech stocks. Their only major mistake was in predicting the relaxation of net neutrality regulations. Going forward, CM Research believes technologies like the internet of things, software defined networks, and internet TV will be disruptive areas. They recommend their 2014 Global Trend Forecast report for predictions on technology, media and telecom sectors over the next year.
India is the world's largest sourcing destination for the IT industry, accounting for 67% of the $124-130 billion global market. The industry employs 10 million people and has transformed the Indian economy. India has a competitive advantage due to its lower costs, which are 3-4 times cheaper than the US. The IT industry has driven growth in education. Key segments include IT services, BPM, software products, and hardware. The sector is expected to grow at a CAGR of 9.5% to $300 billion by 2020. India is also becoming a major player in areas like internet services, e-commerce, cloud computing, and startups. The government is undertaking initiatives like Digital India to further promote the
The AI Index is an independent initiative at the Stanford Institute for Human-Centered Artificial Intelligence (HAI), led by the AI Index Steering Committee, an interdisciplinary group of experts from across academia and industry. The annual report tracks, collates, distills, and visualizes data relating to artificial intelligence, enabling decision-makers to take meaningful action to advance AI responsibly and ethically with humans in mind.
The AI Index Report 2023 provides the following key highlights from its research and development chapter:
1. The US and China have the most cross-country AI research collaborations, though the rate of growth has slowed in recent years.
2. Global AI research output has more than doubled since 2010, led by areas like machine learning, computer vision and pattern recognition.
3. China now leads in total AI research publications, while the US still leads in conference and repository citations but these leads are decreasing.
4. Industry now produces far more significant AI models than academia, as building state-of-the-art systems requires greater resources that industry can provide.
5. Large language models
This report begins with an examination of the global IoT industry and continues by looking into the Chinese IoT industry and its innovators. The report concludes with an analysis of the possible future opportunities and implications that China's expanding IoT industry could entail for Finland.
The document discusses the potential economic impact and value of artificial intelligence (AI) technologies. Some key points:
- Global GDP could be 14% higher by 2030 due to AI, equivalent to an additional $15.7 trillion in economic value. China and North America are expected to see the largest boosts of up to 26% and 14% respectively.
- The majority of GDP gains will come from increased productivity and consumption enabled by AI. Productivity gains will be driven by automation of processes and augmentation of human workers. Consumption gains will come from personalized and higher quality AI-enhanced products and services.
- Retail, financial services, and healthcare are identified as sectors that could see the biggest gains from AI
Future Watch: China's Digital Landscape and Rising Disruptors - Module 2.6 Ar...Team Finland Future Watch
AI is transforming industries in China just as we expect it to revolutionize our industries and organizations. As witnessed in many existing practical AI applications already available today, AI is not a futuristic concept anymore. The question today is, who will lead the next AI revolution? In many terms China can be the driving force of this development.
Hacktivists in Indonesia disrupted Malaysian websites in response to a misprinted flag by exploiting vulnerabilities in Malaysian web servers and replacing website content with protest messages. Ransomware like WCry and Petya have impacted organizations across Southeast Asia by leveraging vulnerabilities in outdated operating systems. Distributed denial of service (DDoS) attacks remain a frequent tactic used by threat actors to disrupt websites and draw attention to causes or hide other malicious activities.
Generation-IoT - The Key to Business Survival in 21st CenturyDr. Mazlan Abbas
This document discusses the importance of IoT (Internet of Things) for business survival in the 21st century. It notes that organizations need to reinvent themselves every 3-7 years to survive, and only 19% of S&P 500 companies from 50 years ago remain. The document outlines the three waves of internet (fixed, mobile, IoT) and explains IoT components and maturity phases from monitoring to autonomous. It argues that connecting assets benefits makers, users, and the operating environment. The large market potential of IoT in sectors like transportation, healthcare, manufacturing is also highlighted. The document emphasizes the growing demand for IoT professionals and calls the audience the "Generation IoT" that can help organizations succeed through IoT.
The United States government is a major supporter of artificial intelligence (AI) research and development through agencies like DARPA, NSF, NIH, ONR, and IARPA. (1) Finland and the U.S. are projected to see the largest economic gains from AI through 2035 with each seeing 2% higher economic growth. (2) Many industries like manufacturing, professional services, retail, and healthcare are expected to see large increases in value added from AI. (3)
Study: Analytics and Data Science Jobs in India: 2020 – By Great Learning & AIMSrishti Deoras
This report outlines the functional analytics skills and programming languages that are most in-demand in the market. The report provides insights for recruiters and hiring companies so that they can study the demand for skills across the analytics function, and can identify and close any capability gaps across workforces. By highlighting the talent hotspots in the country, the report enables organisations to build a steady talent pipeline.
State of Artificial Intelligence in India 2020Srishti Deoras
Report developed by AIMResearch, in association with Jigsaw Academy, on the Artificial Intelligence Market in India as AI emerges as one of the primary Data Science functions across Enterprises in India
Impact on Jobs across Emerging Technologies During the Current Pandemic Crisi...Srishti Deoras
Analytics India Magazine (AIM) along with Jigsaw Academy, has developed this study to focus on the impact on jobs across certain emerging technologies.
The document analyzes characteristics of the Chinese internet market. It finds that China has one of the world's largest internet markets in terms of users and online spending. The market has also experienced very strong growth. However, potential remains as internet penetration is still lower than other G20 countries. The market is characterized by young users who prefer mobile access and try new apps quickly. Competition is also intense as ecosystems form around major players.
Hasil penelitian (Economic Impact Study) terbaru oleh Google dan Deloitte Access Economics yang berisi temuan dan rekomendasi penting yang bermanfaat bagi pelaku bisnis dan pembuat kebijakan.
The document discusses how digital technologies and hyperconnectivity are transforming business and society. It notes that with the rise of the internet of things and big data, physical objects, information, and processes can now all be connected and share data. This new level of connectivity allows new ways of creating value for both businesses and society. The document advocates for a "human centric" approach where technology enhances people's experiences and helps address societal challenges. It also discusses how both digital and physical businesses can harness digital transformation to innovate and grow.
China is already the world’s second largest ICT market place. By 2020 the market is estimated to reach USD 850 billion. For the Chinese government, digital, ICT an AI market is not only seen as a business, but it is increasingly seen through national security and social stability lenses. Furthermore, policies to ensure security often appear to do so at the expense of foreign companies. The Cyber Security Law from 2017 has increasingly tightened the operational landscape for foreign companies in China.
China’s population is ageing rapidly and the country is looking into robotics and AI in order to offset the current and especially the future costs related to the country’s greying population. Finnish healthtech companies might be able to find opportunities related to this, especially those that provide digital or AI-enabled solutions for elderly care.
A white paper I authored about the major headwinds confronting the wearables market, with some interesting new data point around wearables abandonment and acquisition.
The document discusses how insurance is facing significant disruption from social, technological, economic, environmental, and political changes between now and 2020. These changes include a more fragmented customer base, rising digital connectivity and data availability, slowing economic growth in developed markets coupled with faster growth in emerging markets, increasing catastrophe risks, and greater political instability. Insurers will need to reinvent their business models to adapt to these trends and changing customer expectations in order to remain competitive. The document examines the implications of these changes and how insurers can design business strategies to succeed in this disrupted future.
Predictive TMT Analysis from CM ResearchCM Research
This document summarizes the past predictive analysis and investment ideas of CM Research, an independent research firm. Over the past two years, CM Research correctly predicted investment themes including mobile internet, connected devices, cybersecurity, 3D printing, mobile payments, robotics, internet advertising, big data, and UK tech stocks. Their only major mistake was in predicting the relaxation of net neutrality regulations. Going forward, CM Research believes technologies like the internet of things, software defined networks, and internet TV will be disruptive areas. They recommend their 2014 Global Trend Forecast report for predictions on technology, media and telecom sectors over the next year.
India is the world's largest sourcing destination for the IT industry, accounting for 67% of the $124-130 billion global market. The industry employs 10 million people and has transformed the Indian economy. India has a competitive advantage due to its lower costs, which are 3-4 times cheaper than the US. The IT industry has driven growth in education. Key segments include IT services, BPM, software products, and hardware. The sector is expected to grow at a CAGR of 9.5% to $300 billion by 2020. India is also becoming a major player in areas like internet services, e-commerce, cloud computing, and startups. The government is undertaking initiatives like Digital India to further promote the
The AI Index is an independent initiative at the Stanford Institute for Human-Centered Artificial Intelligence (HAI), led by the AI Index Steering Committee, an interdisciplinary group of experts from across academia and industry. The annual report tracks, collates, distills, and visualizes data relating to artificial intelligence, enabling decision-makers to take meaningful action to advance AI responsibly and ethically with humans in mind.
The AI Index Report 2023 provides the following key highlights from its research and development chapter:
1. The US and China have the most cross-country AI research collaborations, though the rate of growth has slowed in recent years.
2. Global AI research output has more than doubled since 2010, led by areas like machine learning, computer vision and pattern recognition.
3. China now leads in total AI research publications, while the US still leads in conference and repository citations but these leads are decreasing.
4. Industry now produces far more significant AI models than academia, as building state-of-the-art systems requires greater resources that industry can provide.
5. Large language models
AI ML 2023 Measuring Trends Analysis by Stanford UniversityIke Alisson
This year, the Stanford Institute for Human-Centered Artificial Intelligence (HAI) 386 pages Report introduces more original data than any previous edition, including a new Chapter on AI Public Opinion, a more thorough Technical Performance Chapter, Original Analysis about Large Language and Multimodal Models, detailed Trends in Global AI Legislation Records, a Study of the Environmental Impact of AI Systems, and more. On the "success" of AI ML on the Market, w.r.t. Chapter 4, indicating: "Corporate Investment (Mergers/Acquisitions, Minority Stakes, Private Investment, & Public Offerings) dipped in 2022 to 190 billion from 2021 amount of 276 billion highs, but the number has still increased 13-fold in the last decade. The Biggest Investment event of the Year was the Nuance Communications acquisition; the Computer SW Tech Company was picked up by Microsoft for $19.7 billion."
The AI Index 2023 Annual Report by Stanford University.pdfAI Geek (wishesh)
Top Ten Takeaways
1. Industry now leads in machine learning model production, with 32 significant models in 2022 compared to academia's three. Industry has more resources like data, computing power, and funding.
2. AI performance on traditional benchmarks is reaching saturation, showing marginal year-over-year improvements. New, more comprehensive benchmarking suites like BIG-bench and HELM are emerging.
3. AI has environmental impacts, with some models emitting significant carbon emissions. However, newer models like BCOOLER show potential for optimizing energy usage.
4. AI is accelerating scientific progress, contributing to advancements in fields like hydrogen fusion, matrix manipulation efficiency, and antibody generation.
5. Incidents of AI misuse are increasing, with 26 times more reported cases since 2012. Examples include deepfake videos and invasive use of call-monitoring technology in U.S. prisons.
6. Demand for AI-related skills is rising across various American industries, except for agriculture, forestry, fishing, and hunting.
7. Private investment in AI decreased by 26.7% in 2022, marking the first decline in the last decade. However, overall AI investment has seen significant growth since 2013.
8. While the proportion of companies adopting AI has plateaued at 50-60%, those that have adopted it are experiencing cost reductions and revenue increases.
9. Policymaker interest in AI is on the rise globally, with a notable increase in bills containing "artificial intelligence" passed into law from 2016 to 2022.
10. Chinese citizens have a highly positive view of AI products and services, with 78% believing in their benefits. In contrast, only 35% of sampled Americans share this sentiment.
The USA tops the global rankings in the Government AI Readiness Index 2021 due to its large and mature technology sector. Singapore ranks second thanks to its institutional strength and digital government capacity. For the first time, East Asian countries make up one quarter of the top 20 ranked countries due to skilled workforces and advanced research infrastructure. Finland and other Nordic countries outperform their economic size in the rankings as a result of effective governments and innovative business environments. The index finds clear inter-regional and intra-regional differences in AI readiness.
"AI's diversity is not just geographic...there is heightened awareness of gender and racial diversity's importance to progress in AI."
AI Index gathers data relating to AI with the goal of being a comprehensive resource of data and analysis for policymakers, researchers, executives, journalists, and the general public to develop intuitions about the complex field of AI.
This past year, AI Index made an effort to include more data from diversity-oriented organizations like AI4ALL and Women in Machine Learning (WiML). For example, the report highlights AI4ALL's increase of alumni in 2018: 900% more alumni than we had in 2015.
This document introduces the 2020 Government AI Readiness Index published by Oxford Insights and the International Development Research Centre. It finds that the US ranks first in overall readiness, while Western European nations like the UK and Germany also score highly. China ranks surprisingly low at 19th due to its focus on implementation over readiness factors. Sub-Saharan Africa, Latin America, and the Caribbean score lowest on average. A new sub-index on responsible AI use finds Nordic countries leading, while the US and UK score lower. The report aims to help governments understand gaps and strengths to improve their AI readiness and ensure its responsible development and use.
New powers, new responsibilities the journalism ai reportVittorio Pasteris
The report summarizes the findings of a survey of 71 news organizations from 32 countries about their use and perspectives on artificial intelligence (AI). Key findings include:
1) AI is already used significantly in journalism but unevenly, with potential for wide-ranging future impact.
2) Newsrooms see AI augmenting but not yet transforming journalism.
3) Just over a third of respondents had an active AI strategy, with different organizational approaches.
4) Newsroom roles were seen changing more through augmentation than replacement of jobs.
5) Biggest challenges to adopting AI were financial/skills resources and cultural resistance.
6) The report outlines elements that should be considered for an effective AI strategy.
This document provides an overview and summary of key findings from the AI Index 2017 Annual Report. It highlights increasing activity and progress in artificial intelligence as seen through metrics on academia, industry, technical performance, and efforts towards human-level performance. The number of academic papers, university course enrollment, conference attendance, startups, funding, and job openings related to AI have all increased substantially in recent years. Technical performance on tasks like object detection, machine translation, and question answering has also improved. However, the report notes that its analysis is limited and biased towards certain perspectives, and more data is needed for a complete picture of the field. It calls for community participation to help track and understand progress in AI.
Spring Splash 3.4.2019: When AI Meets Ethics by Meeri Haataja Saidot
Meeri Haataja's keyote 'When AI Meets Ethics' at Keväthumaus 2019 / Spring Splash 2019 (organised by Väestörekisterikeskus / Population Register Centre).
The document discusses responsible AI and the need for proper guidelines to ensure user trust and privacy as AI becomes more ubiquitous. It notes that while AI demonstrates potential for economic and social value, there are also risks that must be balanced. There is currently a lack of clear and consistent responsible AI benchmarks, standards, and compliance. However, awareness of ethics is growing as seen in the increasing number of frameworks and principles from different organizations. The document outlines India's national strategy for AI developed by NITI Aayog and NASSCOM's responsible AI toolkit which provides guidance and tools to help businesses leverage AI responsibly.
Relevance of Artificial Intelligence in Age of Industry 4.0 Prompt Science An...ijtsrd
The role of Artificial Intelligence AI in developing countries is transformative, offering unprecedented opportunities to address pressing challenges and foster sustainable development. This paper explores the multifaceted impact of AI across various sectors, emphasizing its potential to drive positive change in environmental sustainability, social well being, and governance practices. AI contributes to climate modeling, disaster prediction, and the optimization of renewable energy sources, aiding developing nations in mitigating the effects of climate change. In healthcare and education, AI enhances accessibility, providing innovative solutions for remote diagnostics, personalized learning, and skill development. Governance benefits from AI through improved transparency, efficiency, and accountability, with applications in e governance and supply chain traceability. Sustainable finance is advanced through AI driven risk assessments, directing investments towards environmentally and socially responsible projects. Resource management is optimized through precision agriculture, smart water management, and waste sorting, ensuring efficient utilization of resources. Community development sees advancements in the form of smart cities, improved infrastructure, and community engagement facilitated by AI. Furthermore, AI plays a crucial role in disaster response and resilience, supporting early warning systems and aiding post disaster recovery efforts. Education and upskilling opportunities for pupils are expanded through online courses, competitions, and practical experiences, preparing them for the evolving job market. While recognizing the immense potential, the paper also underscores the importance of responsible and ethical AI deployment, acknowledging cultural nuances and fostering collaboration between governments, businesses, and communities. As developing countries navigate the complexities of AI integration, a conscientious approach ensures that these technologies contribute meaningfully to sustainable development, economic growth, and improved quality of life. Manish Verma "Relevance of Artificial Intelligence in Age of Industry 4.0: Prompt Science Analysis" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-8 | Issue-1 , February 2024, URL: https://www.ijtsrd.com/papers/ijtsrd62387.pdf Paper Url: https://www.ijtsrd.com/computer-science/artificial-intelligence/62387/relevance-of-artificial-intelligence-in-age-of-industry-40-prompt-science-analysis/manish-verma
This Artificial Intelligence (AI) and Big Data Readiness Report
provides an analysis of a global survey of public relations
practitioners and academics and video/written evidence from
senior practitioners concerning the profession’s knowledge,
skills, adoption of and attitudes towards AI, and to a lesser
extent, Big Data. Its aim is to provide an overview of current AI
understanding and preparedness, but most importantly, provide
pointers to how the profession should equip itself to exploit the
potential and guard against the possible dangers of AI.
This report analyzes AI innovation across major banks based on publicly available data and interviews. It finds that a small number of large North American banks dominate areas like research publication, patent filing, and investment. These banks have grown their AI capabilities significantly in recent years through focused strategies and resources. European banks lag behind in most metrics of AI innovation. The report provides recommendations for how banks can strengthen their AI innovation efforts through activities like research, partnerships, and benchmarking against competitors.
The document discusses the GlobalAIIndex, which ranks nations according to their capacity for artificial intelligence. It does this by measuring national ecosystems across three pillars: implementation, innovation, and investment. The implementation pillar looks at talent, infrastructure, and operating environment. The innovation pillar focuses on research, development, and patents. The investment pillar examines commercial venture capital and government strategy. The index includes over 100 input metrics for 54 countries and is intended to highlight effective and ineffective AI practices among nations.
The document provides an overview of artificial intelligence and machine learning from White Star Capital. It discusses the history and evolution of AI from the 1950s to present day, including milestones like the development of neural networks, Deep Blue defeating Kasparov at chess, and AlphaGo defeating the Go champion. The document also defines key AI concepts like machine learning, deep learning, supervised vs unsupervised vs reinforcement learning. It analyzes the growth of the AI sector and increasing investment in areas like AI-first companies, fintech, mobility, healthtech, and more.
The Evolution of AI from Research to ROI in 2024.pdfCiente
Discover the transformative journey of artificial intelligence from theoretical research to strategic asset, unlocking substantial returns for organizations in 2024.
Mary Meeker Internet Trends Report for 2019Eric Johnson
Mary Meeker Internet Trends Report for 2019 - Mary Meeker has been a leading analyst and guru for all things internet business for many years. This report outline's Ms Meeker's review of trends for the year 2019. Topics include:
1) Users
2) E-Commerce + Advertising
3) Usage...
4) Freemium Business Models
5) Data Growth
6) ...Usage
7) Work
8) Education
9) Immigration + USA Inc.
10) Healthcare
11) China (Provided by Hillhouse Capital)
Similar to Artificial Intelligence Index Report, 2021 (20)
Emmy-awarded producer, director, and senior technologist with experience in startups, education, TV, VFX, and Fortune 500 companies. Lumiere-awarded storyteller specializing in AR and VR. Excel in creative partnerships, leading teams, and delivering high-quality content for brands, products, and visions. For a comprehensive review of my roles, with media, including technology research, analysis, and forecasting, please visit my LinkedIn profile.
"State of AI, 2019," from MMC Ventures, in partnership with Barclays.
The State of AI 2019: Divergence
As Artificial Intelligence (AI) proliferates, a divide is emerging. Between nations and
within industries, winners and losers are emerging in the race for adoption, the war
for talent and the competition for value creation.
The landscape for entrepreneurs is also changing. Europe’s ecosystem of 1,600 AI
startups is maturing and bringing creative destruction to new industries. While the
UK is the powerhouse of European AI, hubs in Germany and France are thriving and
may extend their influence in the decade ahead.
As new AI hardware and software make the impossible inevitable, we also face
divergent futures. AI offers profound benefits but poses significant risks. Which
future will we choose?
Our State of AI report for 2019 empowers entrepreneurs, corporate executives,
investors and policy-makers. While jargon-free, our Report draws on unique data
and 400 discussions with ecosystem participants to go beyond the hype and explain
the reality of AI today, what is to come and how to take advantage. Every chapter
includes actionable recommendations.
Artificial intelligence (AI) is a source of both huge excitement
and apprehension. What are the real opportunities and threats
for your business? Drawing on a detailed analysis of the business
impact of AI, we identify the most valuable commercial opening in
your market and how to take advantage of them.
Meta: Four Predictions for the Future of Work
Discover the trends shaping the future of hybrid working and work in the metaverse, and how they’ll redefine inclusion in the workplace. We spoke to 2,000 employees and 400 business leaders in the US and UK to understand the impact.
How could machines learn as eciently as humans and animals? How could machines
learn to reason and plan? How could machines learn representations of percepts
and action plans at multiple levels of abstraction, enabling them to reason, predict,
and plan at multiple time horizons? This position paper proposes an architecture and
training paradigms with which to construct autonomous intelligent agents. It combines
concepts such as congurable predictive world model, behavior driven through intrinsic
motivation, and hierarchical joint embedding architectures trained with self-supervised
learning.
This document is not a technical nor scholarly paper in the traditional sense, but a position
paper expressing my vision for a path towards intelligent machines that learn more like
animals and humans, that can reason and plan, and whose behavior is driven by intrinsic
objectives, rather than by hard-wired programs, external supervision, or external rewards.
Many ideas described in this paper (almost all of them) have been formulated by many
authors in various contexts in various form. The present piece does not claim priority for
any of them but presents a proposal for how to assemble them into a consistent whole. In
particular, the piece pinpoints the challenges ahead. It also lists a number of avenues that
are likely or unlikely to succeed.
The text is written with as little jargon as possible, and using as little mathematical
prior knowledge as possible, so as to appeal to readers with a wide variety of backgrounds
including neuroscience, cognitive science, and philosophy, in addition to machine learning,
robotics, and other fields of engineering. I hope that this piece will help contextualize some
of the research in AI whose relevance is sometimes difficult to see.
InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images
Abstract. We present a method for learning to generate unbounded flythrough videos of natural scenes starting from a single view, where this capability is learned from a collection of single photographs, without requiring camera poses or even multiple views of each scene. To achieve this, we propose a novel self-supervised view generation training paradigm, where we sample and rendering virtual camera trajectories, including cyclic ones, allowing our model to learn stable view generation from a collection of single views. At test time, despite never seeing a video during training, our approach can take a single image and generate long camera trajectories comprised of hundreds of new views with realistic and diverse contents. We compare our approach with recent state-of-the-art supervised view generation methods that require posed multi-view videos and demonstrate superior performance and synthesis quality.
Dialing up the danger: Virtual reality for the simulation of riskAlejandro Franceschi
There is a growing interest the use of virtual reality (VR) to simulate unsafe spaces, scenarios, and behaviours. Environments that might be difficult, costly, dangerous, or ethically contentious to achieve in real life can be created in virtual environments designed to give participants a convincing experience of “being there.” There is little consensus in the academic community about the impact of simulating risky content in virtual reality, and a scarcity of evidence to support the various hypotheses which range from VR being a safe place to rehearse challenging scenarios to calls for such content creation to be halted for fear of irreversible harm to users. Perspectives split along disciplinary lines, with competing ideas emerging from cultural studies and games studies, from psychology and neuroscience, and with industry reports championing the efficacy of these tools for information retention, time efficiency and cost, with little equivalence in information available regarding impact on the wellbeing of participants. In this study we use thematic analysis and close reading language analysis to investigate the way in which participants in a VR training scenario respond to, encode and relay their own experiences. We find that participants overall demonstrate high levels of “perceptual proximity” to the experience, recounting it as something that happened to them directly and personally. We discuss the impact of particular affordances of VR, as well as a participant’s prior experience on the impact of high-stress simulations. Finally, we consider the ethical mandate for training providers to mitigate the risk of traumatizing or re-traumatizing participants when creating high-risk virtual scenarios.
- The metaverse is still being defined but has the potential to be the next iteration of the internet by seamlessly combining digital and physical lives through immersion, interactivity, and use cases beyond gaming.
- Large technology companies, venture capital, private equity, start-ups, and brands have already invested over $120 billion in the metaverse in 2022 alone, driven by expectations of its economic impact and opportunities.
- The metaverse's potential economic impact is estimated to reach up to $5 trillion by 2030, generating new business models and engagement channels across industries like e-commerce, education, healthcare, and more.
Bank of England Staff Working Paper No 605 The Macroeconomics of Central Bank...Alejandro Franceschi
Bank of England Staff Working Paper No 605 The Macroeconomics of Central Bank Issued Digital Currencies (CBDC).
A study on the macroeconomic consequences of issuing a central bank digital currency (CBDC) - a universally accessible and interest-bearing central bank liability, implemented via distributed ledgers, that competes with bank deposits as a medium of exchange. Some summary results are: a possible rise in GDP by 3%, reductions in real interest rates, distortionary taxes, and monetary transaction costs. As a second monetary policy instrument, which could substantially improve the central bank's ability to stabilize the business cycle.
THE METAVERSE IS POTENTIALLY AN $8 TRILLION TO $13 TRILLION OPPORTUNITY (Citibank):
We believe the Metaverse may be the next generation of the
internet — combining the physical and digital world in a persistent
and immersive manner — and not purely a Virtual Reality world.
A device-agnostic Metaverse accessible via PCs, game consoles, and
smartphones could result in a very large ecosystem. Based on our
definition, we estimate the total addressable market for the Metaverse
economy could grow to between $8 trillion and $13 trillion by 2030.
METAVERSE USE CASES:
Gaming is viewed as a key Metaverse use case for the next several years due to the immersive and multi-player
experience of the space currently. But we believe that the Metaverse will eventually help us find new enhanced ways to
do all of our current activities, including commerce, entertainment and media, education and training, manufacturing and
enterprise in general. Enterprise use cases of the Metaverse in the coming years will likely include internal collaboration,
client contact, sales and marketing, advertising, events and conferences, engineering and design, and workforce train
METAVERSE INFRASTRUCTURE BUILDING:
the current state, the internet infrastructure is unsuitable for building a fully-immersive content streaming Metaverse
environment, that enables users to go seamlessly from one experience to another. To make the vision of Metaverse a reality, we
expect significant investment in a confluence of technology. Low latency — the time it takes a data signal to travel from one point
on the internet to another point and then come back — is critical to building a more realistic user experience.
MONEY IN THE METAVERSE:
We expect the next generation of the internet, i.e., the Metaverse, would encapsulate a range of form factors of money, including
the existing/traditional forms of money and also upcoming/digitally-native forms — cryptocurrency, stablecoins, central bank
digital currencies (CBDCs) — that were out of scope in a pre-blockchain virtual world
This document is the copyright of its respective holders. It is freely available on the Internet to anyone who searches for it independently. It is provided here under the "Fair Use Doctrine of U.S. Copyright Law."
Vitalik Buterin | Crypto Cities
In this whitepper, Vitalik outlines some concepts for how a new model of a city running on a blockchain, empoowers the community to essentially self-govern.
Everyone can vote on the blockchain, decisions are made as a collective on where to put monies to use, and work outside the community not unlike a centralized DAO (an oxymoron, but it is the proposal herein).
This is not science fiction. Thanks to a new law in Wisconsin that permits these types of collectives, or communities, etc., the radical (?), or rather, evolutionary step for this has already begun; and not just in Wisconsin, but Miami is a key location as well.
Mixed Reality: Pose Aware Object Replacement for Alternate RealitiesAlejandro Franceschi
This document explains how the technology involved, can semantically replace moving objects, humans, and other such visual input, and transform it, using mixed reality, into whatever the viewer would prefer the real world looks like instead.
From videogames to movies, to education, healthcare, commerce, communications, and industrial solutions, this will radically change the way we interact with the world, with others, and ourselves.
MaterialX has been integrated into MayaUSD and ArnoldUSD in the following ways:
- MaterialX definitions can be imported and exported from MayaUSD to enable interoperability via UsdShade and the MaterialX Preview Surface.
- The standard MaterialX shader generator is used to visualize MaterialX materials directly in Maya's viewport without translation, leveraging the same GLSL code generator as hdStorm.
- A plugin architecture allows custom MaterialX to Preview Surface translations to be defined and used for import and export between DCC tools and renderers.
Google Research Siggraph Whitepaper | Total Relighting: Learning to Relight P...Alejandro Franceschi
Google Research Siggraph Whitepaper | Total Relighting: Learning to Relight Portraits for Background Replacement
Abstract:
Given a portrait and an arbitrary high dynamic range lighting environment, our framework uses machine learning to composite the subject into a new scene, while accurately modeling their appearance in the target illumination condition. We estimate a high quality alpha matte, foreground element, albedo map, and surface normals, and we propose a novel, per-pixel lighting representation within a deep learning framework.
Intel, Intelligent Systems Lab: Syable View Synthesis WhitepaperAlejandro Franceschi
Intel, Intelligent Systems Lab:
Stable View Synthesis Whitepaper
We present Stable View Synthesis (SVS). Given a set
of source images depicting a scene from freely distributed
viewpoints, SVS synthesizes new views of the scene. The
method operates on a geometric scaffold computed via
structure-from-motion and multi-view stereo. Each point
on this 3D scaffold is associated with view rays and corresponding feature vectors that encode the appearance of
this point in the input images.
The core of SVS is view dependent on-surface feature aggregation, in which directional feature vectors at each 3D point are processed to produce a new feature vector for a ray that maps this point into the new target view.
The target view is then rendered by a convolutional network from a tensor of features synthesized in this way for all pixels. The method is composed of differentiable modules and is trained end-to-end. It supports spatially-varying view-dependent importance weighting and feature transformation of source images at each point; spatial and temporal stability due to the smooth dependence of on-surface feature aggregation on the target view; and synthesis of view-dependent effects such as specular reflection.
Experimental results demonstrate that SVS outperforms state-of-the-art view synthesis methods both quantitatively and qualitatively on three diverse realworld datasets, achieving unprecedented levels of realism in free-viewpoint video of challenging large-scale scenes.
PWC: Why we believe VR/AR will boost global GDP by $1.5 trillionAlejandro Franceschi
VR and AR have the potential to boost the global economy by $1.5 trillion by 2030. The document discusses how these technologies can transform business in areas like product development, training, process improvements, healthcare and retail. It provides examples of how different industries like automotive, energy, and healthcare are already using VR and AR to improve efficiency and customer experiences. The technologies are maturing and will be further enabled by 5G networks, with AR expected to have a bigger economic impact than VR through 2030. For businesses to fully realize the value, they need to address cultural concerns and privacy/security issues around the technologies.
This document provides an overview of 10 UK immersive technology companies that have raised significant funding. It discusses Anything World, a platform for creating 3D worlds using voice controls and AI; Bodyswaps, a VR platform for soft skills training that has worked with organizations like Save the Children; and Anthropic, an AI safety startup that has raised $50 million and works on techniques like constitutional AI. The document highlights the companies' technologies, customers, fundraising journeys, and growth opportunities in areas like enterprise training as immersive tech adoption increases.
1. The document presents an approach to enhance the realism of synthetic images rendered by game engines. A convolutional network is trained to modify rendered images using intermediate representations from the rendering process.
2. The network is trained with an adversarial objective to provide strong supervision at multiple perceptual levels. A new strategy is proposed for sampling image patches during training to address differences in scene layout distributions between datasets.
3. The approach significantly enhances photorealism over recent image-to-image translation methods and baselines, as shown in controlled experiments. It can add realistic details like gloss, vegetation, and road textures while keeping enhancements consistent with the input image content.
HAI Industry Brief: AI & the Future of Work Post Covid
Stanford University, Human-Centered Artificial Intelligence:
Researchers studying how AI can be used to help teams collaborate, improve workplace culture, promote employee well-being, assist humans in dangerous environments, and more.
Source: https://aiindex.stanford.edu/wp-content/uploads/2021/03/2021-AI-Index-Report_Master.pdf
The document provides an overview of hiring trends and salary expectations in 2021 for creative and marketing professionals in the United States and Canada, noting increased demand for digital skills as companies pivot more resources online and adopt hybrid remote/onsite models, while also discussing retention challenges and emerging jobs.
5th LF Energy Power Grid Model Meet-up SlidesDanBrown980551
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Digital Marketing Trends in 2024 | Guide for Staying AheadWask
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
A Comprehensive Guide to DeFi Development Services in 2024Intelisync
DeFi represents a paradigm shift in the financial industry. Instead of relying on traditional, centralized institutions like banks, DeFi leverages blockchain technology to create a decentralized network of financial services. This means that financial transactions can occur directly between parties, without intermediaries, using smart contracts on platforms like Ethereum.
In 2024, we are witnessing an explosion of new DeFi projects and protocols, each pushing the boundaries of what’s possible in finance.
In summary, DeFi in 2024 is not just a trend; it’s a revolution that democratizes finance, enhances security and transparency, and fosters continuous innovation. As we proceed through this presentation, we'll explore the various components and services of DeFi in detail, shedding light on how they are transforming the financial landscape.
At Intelisync, we specialize in providing comprehensive DeFi development services tailored to meet the unique needs of our clients. From smart contract development to dApp creation and security audits, we ensure that your DeFi project is built with innovation, security, and scalability in mind. Trust Intelisync to guide you through the intricate landscape of decentralized finance and unlock the full potential of blockchain technology.
Ready to take your DeFi project to the next level? Partner with Intelisync for expert DeFi development services today!
Generating privacy-protected synthetic data using Secludy and MilvusZilliz
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/temporal-event-neural-networks-a-more-efficient-alternative-to-the-transformer-a-presentation-from-brainchip/
Chris Jones, Director of Product Management at BrainChip , presents the “Temporal Event Neural Networks: A More Efficient Alternative to the Transformer” tutorial at the May 2024 Embedded Vision Summit.
The expansion of AI services necessitates enhanced computational capabilities on edge devices. Temporal Event Neural Networks (TENNs), developed by BrainChip, represent a novel and highly efficient state-space network. TENNs demonstrate exceptional proficiency in handling multi-dimensional streaming data, facilitating advancements in object detection, action recognition, speech enhancement and language model/sequence generation. Through the utilization of polynomial-based continuous convolutions, TENNs streamline models, expedite training processes and significantly diminish memory requirements, achieving notable reductions of up to 50x in parameters and 5,000x in energy consumption compared to prevailing methodologies like transformers.
Integration with BrainChip’s Akida neuromorphic hardware IP further enhances TENNs’ capabilities, enabling the realization of highly capable, portable and passively cooled edge devices. This presentation delves into the technical innovations underlying TENNs, presents real-world benchmarks, and elucidates how this cutting-edge approach is positioned to revolutionize edge AI across diverse applications.
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Building Production Ready Search Pipelines with Spark and MilvusZilliz
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfChart Kalyan
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providersakankshawande
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframePrecisely
Inconsistent user experience and siloed data, high costs, and changing customer expectations – Citizens Bank was experiencing these challenges while it was attempting to deliver a superior digital banking experience for its clients. Its core banking applications run on the mainframe and Citizens was using legacy utilities to get the critical mainframe data to feed customer-facing channels, like call centers, web, and mobile. Ultimately, this led to higher operating costs (MIPS), delayed response times, and longer time to market.
Ever-changing customer expectations demand more modern digital experiences, and the bank needed to find a solution that could provide real-time data to its customer channels with low latency and operating costs. Join this session to learn how Citizens is leveraging Precisely to replicate mainframe data to its customer channels and deliver on their “modern digital bank” experiences.
Dandelion Hashtable: beyond billion requests per second on a commodity serverAntonios Katsarakis
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Fueling AI with Great Data with Airbyte WebinarZilliz
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Tatiana Kojar
Skybuffer AI, built on the robust SAP Business Technology Platform (SAP BTP), is the latest and most advanced version of our AI development, reaffirming our commitment to delivering top-tier AI solutions. Skybuffer AI harnesses all the innovative capabilities of the SAP BTP in the AI domain, from Conversational AI to cutting-edge Generative AI and Retrieval-Augmented Generation (RAG). It also helps SAP customers safeguard their investments into SAP Conversational AI and ensure a seamless, one-click transition to SAP Business AI.
With Skybuffer AI, various AI models can be integrated into a single communication channel such as Microsoft Teams. This integration empowers business users with insights drawn from SAP backend systems, enterprise documents, and the expansive knowledge of Generative AI. And the best part of it is that it is all managed through our intuitive no-code Action Server interface, requiring no extensive coding knowledge and making the advanced AI accessible to more users.
Driving Business Innovation: Latest Generative AI Advancements & Success StorySafe Software
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...Alex Pruden
Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security.
Paper Link: https://eprint.iacr.org/2024/257
GraphRAG for Life Science to increase LLM accuracyTomaz Bratanic
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...alexjohnson7307
Predictive maintenance is a proactive approach that anticipates equipment failures before they happen. At the forefront of this innovative strategy is Artificial Intelligence (AI), which brings unprecedented precision and efficiency. AI in predictive maintenance is transforming industries by reducing downtime, minimizing costs, and enhancing productivity.
2. Artificial Intelligence
Index Report 2021
2
INTRODUCTION TO THE
2021 AI INDEX REPORT
Welcome to the fourth edition of the AI Index Report!
This year we significantly expanded the amount of data
available in the report, worked with a broader set of
external organizations to calibrate our data, and deepened
our connections with Stanford’s Institute for Human-
Centered Artificial Intelligence (HAI).
The AI Index Report tracks, collates, distills, and visualizes
data related to artificial intelligence. Its mission is to
provide unbiased, rigorously vetted, and globally sourced
data for policymakers, researchers, executives, journalists,
and the general public to develop intuitions about the
complex field of AI. The report aims to be the world’s most
credible and authoritative source for data and insights
about AI.
COVID AND AI
The 2021 report shows the effects of COVID-19 on AI
development from multiple perspectives. The Technical
Performance chapter discusses how an AI startup used
machine-learning-based techniques to accelerate COVID-
related drug discovery during the pandemic, and our
Economy chapter suggests that AI hiring and private
investment were not significantly adversely influenced
by the pandemic, as both grew during 2020. If anything,
COVID-19 may have led to a higher number of people
participating in AI research conferences, as the pandemic
forced conferences to shift to virtual formats, which in turn
led to significant spikes in attendance.
CHANGES FOR THIS EDITION
In 2020, we surveyed more than 140 readers from
government, industry, and academia about what they
found most useful about the report and what we should
change. The main suggested areas for improvement were:
•
Technical performance: We significantly expanded
this chapter in 2021, carrying out more of our own
analysis.
•
Diversity and ethics data: We gathered more data
for this year’s report, although our investigation
surfaced several areas where the AI community
currently lacks good information.
•
Country comparisons: Readers were generally
interested in being able to use the AI Index for cross-
country comparisons. To support this, we:
•
gathered more data to allow for comparison
among countries, especially relating to
economics and bibliometrics; and
•
included a thorough summary of the various AI
strategies adopted by different countries and
how they evolved over time.
PUBLIC DATA AND TOOLS
The AI Index 2021 Report is supplemented by raw data
and an interactive tool. We invite each member of the AI
community to use the data and tool in a way most relevant
to their work and interests.
•
Raw data and charts: The public data and high-
resolution images of all the charts in the report are
available on Google Drive.
•
Global AI Vibrancy Tool: We revamped the Global AI
Vibrancy Tool this year, allowing for better interactive
visualization when comparing up to 26 countries
across 22 indicators. The updated tool provides
transparent evaluation of the relative position of
countries based on users’ preference; identifies
relevant national indicators to guide policy priorities
at a country level; and shows local centers of AI
excellence for not just advanced economies but also
emerging markets.
•
Issues in AI measurement: In fall 2020, we
published “Measurement in AI Policy: Opportunities
and Challenges,” a report that lays out a variety of
AI measurement issues discussed at a conference
hosted by the AI Index in fall 2019.
3. Artificial Intelligence
Index Report 2021
3
Table of Contents
INTRODUCTION TO THE 2021 AI INDEX REPORT 2
TOP 9 TAKEAWAYS 4
AI INDEX STEERING COMMITTEE STAFF 5
HOW TO CITE THE REPORT 6
ACKNOWLEDGMENTS 7
REPORT HIGHLIGHTS 10
CHAPTER 1 Research and Development 14
CHAPTER 2 Technical Performance 41
CHAPTER 3 The Economy 80
CHAPTER 4 AI Education 107
CHAPTER 5 Ethical Challenges of AI Applications 125
CHAPTER 6 Diversity in AI 135
CHAPTER 7 AI Policy and National Strategies 151
APPENDIX 177
ACCESS THE PUBLIC DATA
4. Artificial Intelligence
Index Report 2021
4
AI investment in drug design and discovery increased significantly: “Drugs, Cancer, Molecular,
Drug Discovery” received the greatest amount of private AI investment in 2020, with more than USD
13.8 billion, 4.5 times higher than 2019.
The industry shift continues: In 2019, 65% of graduating North American PhDs in AI went into
industry—up from 44.4% in 2010, highlighting the greater role industry has begun to play in AI
development.
Generative everything: AI systems can now compose text, audio, and images to a sufficiently high
standard that humans have a hard time telling the difference between synthetic and non-synthetic
outputs for some constrained applications of the technology.
AI has a diversity challenge: In 2019, 45% new U.S. resident AI PhD graduates were white—by
comparison, 2.4% were African American and 3.2% were Hispanic.
China overtakes the US in AI journal citations: After surpassing the United States in the total
number of journal publications several years ago, China now also leads in journal citations;
however, the United States has consistently (and significantly) more AI conference papers (which are also
more heavily cited) than China over the last decade.
The majority of the US AI PhD grads are from abroad—and they’re staying in the US:
The percentage of international students among new AI PhDs in North America continued to rise in
2019, to 64.3%—a 4.3% increase from 2018. Among foreign graduates, 81.8% stayed in the United States
and 8.6% have taken jobs outside the United States.
Surveillance technologies are fast, cheap, and increasingly ubiquitous: The technologies
necessary for large-scale surveillance are rapidly maturing, with techniques for image classification,
face recognition, video analysis, and voice identification all seeing significant progress in 2020.
AI ethics lacks benchmarks and consensus: Though a number of groups are producing a range
of qualitative or normative outputs in the AI ethics domain, the field generally lacks benchmarks
that can be used to measure or assess the relationship between broader societal discussions about
technology development and the development of the technology itself. Furthermore, researchers and
civil society view AI ethics as more important than industrial organizations.
AI has gained the attention of the U.S. Congress: The 116th Congress is the most AI-focused
congressional session in history with the number of mentions of AI in congressional record more
than triple that of the 115th Congress.
TOP 9 TAKEAWAYS
1
2
3
4
5
6
7
8
9
5. Artificial Intelligence
Index Report 2021
5
AI Index Steering Committee
AI Index Staff
Co-Directors
Members
Research Manager and Editor in Chief
Program Manager
Jack Clark
OECD, GPAI
Daniel Zhang
Stanford University
Erik Brynjolfsson
Stanford University
John Etchemendy
Stanford University
Deep Ganguli
Stanford University
Barbara Grosz
Harvard University
Terah Lyons
Partnership on AI
James Manyika
McKinsey Global Institute
Juan Carlos Niebles
Stanford University
Michael Sellitto
Stanford University
Yoav Shoham (Founding Director)
Stanford University, AI21 Labs
Raymond Perrault
SRI International
Saurabh Mishra
Stanford University
6. Artificial Intelligence
Index Report 2021
6
How to Cite This Report
Daniel Zhang, Saurabh Mishra, Erik Brynjolfsson, John Etchemendy, Deep Ganguli, Barbara
Grosz, Terah Lyons, James Manyika, Juan Carlos Niebles, Michael Sellitto, Yoav Shoham,
Jack Clark, and Raymond Perrault, “The AI Index 2021 Annual Report,” AI Index Steering
Committee, Human-Centered AI Institute, Stanford University, Stanford, CA, March 2021.
The AI Index 2021 Annual Report by Stanford University is licensed under
Attribution-NoDerivatives 4.0 International. To view a copy of this license,
visit http://creativecommons.org/licenses/by-nd/4.0/.
The AI Index is an independent initiative at Stanford University’s
Human-Centered Artificial Intelligence Institute (HAI).
We thank our supporting partners
We welcome feedback and new ideas for next year.
Contact us at AI-Index-Report@stanford.edu.
The AI Index was conceived within the One Hundred Year Study on AI (AI100).
7. Artificial Intelligence
Index Report 2021
7
Acknowledgments
We appreciate the following organizations and individuals who provided data, analysis,
advice, and expert commentary for inclusion in the AI Index 2021 Report:
Organizations
arXiv
Jim Entwood, Paul Ginsparg,
Joe Halpern, Eleonora Presani
AI Ethics Lab
Cansu Canca, Yasemin Usta
Black in AI
Rediet Abebe, Hassan Kane
Bloomberg Government
Chris Cornillie
Burning Glass Technologies
Layla O’Kane, Bledi Taska, Zhou Zhou
Computing Research Association
Andrew Bernat, Susan Davidson
Elsevier
Clive Bastin, Jörg Hellwig,
Sarah Huggett, Mark Siebert
Intento
Grigory Sapunov, Konstantin Savenkov
International Federation of Robotics
Susanne Bieller, Jeff Burnstein
Joint Research Center, European
Commission
Giuditta De Prato, Montserrat López
Cobo, Riccardo Righi
LinkedIn
Guy Berger, Mar Carpanelli, Di Mo,
Virginia Ramsey
Liquidnet
Jeffrey Banner, Steven Nichols
McKinsey Global Institute
Brittany Presten
Microsoft Academic Graph
Iris Shen, Kuansan Wang
National Institute of Standards and
Technology
Patrick Grother
Nesta
Joel Klinger, Juan Mateos-Garcia,
Kostas Stathoulopoulos
NetBase Quid
Zen Ahmed, Scott Cohen, Julie Kim
PostEra
Aaron Morris
Queer in AI
Raphael Gontijo Lopes
State of AI Report
Nathan Benaich, Ian Hogarth
Women in Machine Learning
Sarah Tan, Jane Wang
8. Artificial Intelligence
Index Report 2021
8
Individuals
ActivityNet
Fabian Caba (Adobe Research); Bernard
Ghanem (King Abdullah University of
Science and Technology); Cees Snoek
(University of Amsterdam)
AI Brain Drain and Faculty Departure
Michael Gofman (University of Roches-
ter); Zhao Jin (Cheung Kong Graduate
School of Business)
Automated Theorem Proving
Geoff Sutcliffe (University of Miami);
Christian Suttner (Connion GmbH)
Boolean Satisfiability Problem
Lars Kotthoff (University of Wyoming)
Corporate Representation at
AI Research Conferences
Nuruddin Ahmed (Ivey Business School,
Western University); Muntasir Wahed
(Virginia Tech)
Conference Attendance
Maria Gini, Gita Sukthankar (AAMAS);
Carol Hamilton (AAAI); Dan Jurafsky
(ACL); Walter Scheirer, Ramin Zabih
(CVPR); Jörg Hoffmann, Erez Karpas
(ICAPS); Paul Oh (IROS); Pavlos Peppas,
Michael Thielscher (KR)
Ethics at AI Conferences
Pedro Avelar, Luis Lamb, Marcelo
Prates (Federal University of Rio
Grande do Sul)
ImageNet
Lucas Beyer, Alexey Dosovitskiy,
Neil Houlsby (Google)
MLPerf/DAWNBench
Cody Coleman (Stanford University),
Peter Mattson (Google)
Molecular Synthesis
Philippe Schwaller (IBM
Research–Europe)
Visual Question Answering
Dhruv Batra, Devi Parikh (Georgia
Tech/FAIR); Ayush Shrivastava
(Georgia Tech)
You Only Look Once
Xiang Long (Baidu)
9. Artificial Intelligence
Index Report 2021
9
Advice and Expert Commentary
Graduate Researchers
Report and Website
Alexey Bochkovskiy; Baidu’s PaddlePaddle Computer Vision Team; Chenggang Xu (Cheung Kong
Graduate School of Business); Mohammed AlQuraishi (Columbia University); Evan Schnidman
(EAS Innovation); Fanghzhen Lin (Hong Kong University of Science and Technology); David Kanter
(MLCommons); Sam Bowman (New York University); Maneesh Agrawala, Jeannette Bohg, Emma
Brunskill, Chelsea Finn, Aditya Grover, Tatsunori Hashimoto, Dan Jurafsky, Percy Liang, Sharon
Zhou (Stanford University); Vamsi Sistla (University of California, Berkeley); Simon King (University
of Edinburgh); Ivan Goncharov (Weights Biases)
Ankita Banerjea, Yu-chi Tsao (Stanford University)
Michi Turner (report graphic design and cover art); Nancy King (report editor); Michael Taylor
(report data visualization); Kevin Litman-Navarro (Global AI Vibrancy Tool design and development);
Travis Taylor (AI Index website design); Digital Avenues (AI Index website development)
10. Artificial Intelligence
Index Report 2021
10
CHAPTER 1: RESEARCH DEVELOPMENT
•
The number of AI journal publications grew by 34.5% from 2019 to 2020—a much higher percentage growth
than from 2018 to 2019 (19.6%).
•
In every major country and region, the highest proportion of peer-reviewed AI papers comes from academic
institutions. But the second most important originators are different: In the United States, corporate-
affiliated research represents 19.2% of the total publications, whereas government is the second most
important in China (15.6%) and the European Union (17.2%).
•
In 2020, and for the first time, China surpassed the United States in the share of AI journal citations in the
world, having briefly overtaken the United States in the overall number of AI journal publications in 2004 and
then retaken the lead in 2017. However, the United States has consistently (and significantly) more cited AI
conference papers than China over the last decade.
•
In response to COVID-19, most major AI conferences took place virtually and registered a significant increase
in attendance as a result. The number of attendees across nine conferences almost doubled in 2020.
•
In just the last six years, the number of AI-related publications on arXiv grew by more than sixfold, from 5,478
in 2015 to 34,736 in 2020.
•
AI publications represented 3.8% of all peer-reviewed scientific publications worldwide in 2019, up from
1.3% in 2011.
CHAPTER 2: TECHNICAL PERFORMANCE
•
Generative everything: AI systems can now compose text, audio, and images to a sufficiently high standard
that humans have a hard time telling the difference between synthetic and non-synthetic outputs for some
constrained applications of the technology. That promises to generate a tremendous range of downstream
applications of AI for both socially useful and less useful purposes. It is also causing researchers to invest in
technologies for detecting generative models; the DeepFake Detection Challenge data indicates how well
computers can distinguish between different outputs.
•
The industrialization of computer vision: Computer vision has seen immense progress in the past decade,
primarily due to the use of machine learning techniques (specifically deep learning). New data shows that
computer vision is industrializing: Performance is starting to flatten on some of the largest benchmarks,
suggesting that the community needs to develop and agree on harder ones that further test performance.
Meanwhile, companies are investing increasingly large amounts of computational resources to train
computer vision systems at a faster rate than ever before. Meanwhile, technologies for use in deployed
systems—like object-detection frameworks for analysis of still frames from videos—are maturing rapidly,
indicating further AI deployment.
REPORT HIGHLIGHTS
11. Artificial Intelligence
Index Report 2021
11
•
Natural Language Processing (NLP) outruns its evaluation metrics: Rapid progress in NLP has yielded AI
systems with significantly improved language capabilities that have started to have a meaningful economic impact
on the world. Google and Microsoft have both deployed the BERT language model into their search engines, while
other large language models have been developed by companies ranging from Microsoft to OpenAI. Progress in
NLP has been so swift that technical advances have started to outpace the benchmarks to test for them. This can
be seen in the rapid emergence of systems that obtain human level performance on SuperGLUE, an NLP evaluation
suite developed in response to earlier NLP progress overshooting the capabilities being assessed by GLUE.
•
New analyses on reasoning: Most measures of technical problems show for each time point the performance
of the best system at that time on a fixed benchmark. New analyses developed for the AI Index offer metrics that
allow for an evolving benchmark, and for the attribution to individual systems of credit for a share of the overall
performance of a group of systems over time. These are applied to two symbolic reasoning problems, Automated
Theorem Proving and Satisfiability of Boolean formulas.
•
Machine learning is changing the game in healthcare and biology: The landscape of the healthcare and
biology industries has evolved substantially with the adoption of machine learning. DeepMind’s AlphaFold
applied deep learning technique to make a significant breakthrough in the decades-long biology challenge of
protein folding. Scientists use ML models to learn representations of chemical molecules for more effective
chemical synthesis planning. PostEra, an AI startup used ML-based techniques to accelerate COVID-related drug
discovery during the pandemic.
CHAPTER 3: THE ECONOMY
•
“Drugs, Cancer, Molecular, Drug Discovery” received the greatest amount of private AI investment in 2020, with
more than USD 13.8 billion, 4.5 times higher than 2019.
•
Brazil, India, Canada, Singapore, and South Africa are the countries with the highest growth in AI hiring from
2016 to 2020. Despite the COVID-19 pandemic, the AI hiring continued to grow across sample countries in 2020.
•
More private investment in AI is being funneled into fewer startups. Despite the pandemic, 2020 saw a 9.3%
increase in the amount of private AI investment from 2019—a higher percentage increase than from 2018 to 2019
(5.7%), though the number of newly funded companies decreased for the third year in a row.
•
Despite growing calls to address ethical concerns associated with using AI, efforts to address these concerns
in the industry are limited, according to a McKinsey survey. For example, issues such as equity and fairness
in AI continue to receive comparatively little attention from companies. Moreover, fewer companies in 2020
view personal or individual privacy risks as relevant, compared with in 2019, and there was no change in the
percentage of respondents whose companies are taking steps to mitigate these particular risks.
•
Despite the economic downturn caused by the pandemic, half the respondents in a McKinsey survey said that
the coronavirus had no effect on their investment in AI, while 27% actually reported increasing their investment.
Less than a fourth of businesses decreased their investment in AI.
•
The United States recorded a decrease in its share of AI job postings from 2019 to 2020—the first drop in six years.
The total number of AI jobs posted in the United States also decreased by 8.2% from 2019 to 2020, from 325,724
in 2019 to 300,999 jobs in 2020.
12. Artificial Intelligence
Index Report 2021
12
CHAPTER 4: AI EDUCATION
•
An AI Index survey conducted in 2020 suggests that the world’s top universities have increased their investment
in AI education over the past four years. The number of courses that teach students the skills necessary to build
or deploy a practical AI model on the undergraduate and graduate levels has increased by 102.9% and 41.7%,
respectively, in the last four academic years.
•
More AI PhD graduates in North America chose to work in industry in the past 10 years, while fewer opted for jobs
in academia, according to an annual survey from the Computing Research Association (CRA). The share of new
AI PhDs who chose industry jobs increased by 48% in the past decade, from 44.4% in 2010 to 65.7% in 2019. By
contrast, the share of new AI PhDs entering academia dropped by 44%, from 42.1% in 2010 to 23.7% in 2019.
•
In the last 10 years, AI-related PhDs have gone from 14.2% of the total of CS PhDs granted in the United States,
to around 23% as of 2019, according to the CRA survey. At the same time, other previously popular CS PhDs have
declined in popularity, including networking, software engineering, and programming languages. Compilers all
saw a reduction in PhDs granted relative to 2010, while AI and Robotics/Vision specializations saw a substantial
increase.
•
After a two-year increase, the number of AI faculty departures from universities to industry jobs in North America
dropped from 42 in 2018 to 33 in 2019 (28 of these are tenured faculty and five are untenured). Carnegie Mellon
University had the largest number of AI faculty departures between 2004 and 2019 (16), followed by the Georgia
Institute of Technology (14) and University of Washington (12).
•
The percentage of international students among new AI PhDs in North America continued to rise in 2019, to
64.3%—a 4.3% increase from 2018. Among foreign graduates, 81.8% stayed in the United States and 8.6% have
taken jobs outside the United States.
•
In the European Union, the vast majority of specialized AI academic offerings are taught at the master’s level;
robotics and automation is by far the most frequently taught course in the specialized bachelor’s and master’s
programs, while machine learning (ML) dominates in the specialized short courses.
CHAPTER 5: ETHICAL CHALLENGES OF AI APPLICATIONS
•
The number of papers with ethics-related keywords in titles submitted to AI conferences has grown since 2015,
though the average number of paper titles matching ethics-related keywords at major AI conferences remains
low over the years.
•
The five news topics that got the most attention in 2020 related to the ethical use of AI were the release of the
European Commission’s white paper on AI, Google’s dismissal of ethics researcher Timnit Gebru, the AI ethics
committee formed by the United Nations, the Vatican’s AI ethics plan, and IBM’s exiting the facial-recognition
businesses.
13. Artificial Intelligence
Index Report 2021
13
CHAPTER 6: DIVERSITY IN AI
•
The percentages of female AI PhD graduates and tenure-track computer science (CS) faculty have remained low
for more than a decade. Female graduates of AI PhD programs in North America have accounted for less than
18% of all PhD graduates on average, according to an annual survey from the Computing Research Association
(CRA). An AI Index survey suggests that female faculty make up just 16% of all tenure-track CS faculty at several
universities around the world.
•
The CRA survey suggests that in 2019, among new U.S. resident AI PhD graduates, 45% were white, while 22.4%
were Asian, 3.2% were Hispanic, and 2.4% were African American.
•
The percentage of white (non-Hispanic) new computing PhDs has changed little over the last 10 years,
accounting for 62.7% on average. The share of Black or African American (non-Hispanic) and Hispanic computing
PhDs in the same period is significantly lower, with an average of 3.1% and 3.3%, respectively.
•
The participation in Black in AI workshops, which are co-located with the Conference on Neural Information
Processing Systems (NeurIPS), has grown significantly in recent years. The numbers of attendees and submitted
papers in 2019 are 2.6 times higher than in 2017, while the number of accepted papers is 2.1 times higher.
•
In a membership survey by Queer in AI in 2020, almost half the respondents said they view the lack of
inclusiveness in the field as an obstacle they have faced in becoming a practitioner in the AI/ML field. More than
40% of members surveyed said they have experienced discrimination or harassment at work or school.
CHAPTER 7: AI POLICY AND NATIONAL STRATEGIES
•
Since Canada published the world’s first national AI strategy in 2017, more than 30 other countries and regions
have published similar documents as of December 2020.
•
The launch of the Global Partnership on AI (GPAI) and Organisation for Economic Co-operation and Development
(OECD) AI Policy Observatory and Network of Experts on AI in 2020 promoted intergovernmental efforts to work
together to support the development of AI for all.
•
In the United States, the 116th Congress was the most AI-focused congressional session in history. The number
of mentions of AI by this Congress in legislation, committee reports, and Congressional Research Service (CRS)
reports is more than triple that of the 115th Congress.
14. TABLE OF CONTENTS
14
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
CHAPTER 1:
Research
Development
Artificial Intelligence
Index Report 2021
15. TABLE OF CONTENTS
15
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
CHAPTER 1:
RESEARCH
DEVELOPMENT
Overview 16
Chapter Highlights 17
1.1 PUBLICATIONS 18
Peer-Reviewed AI Publications 18
Overview 18
By Region 18
By Geographic Area 20
By Institutional Affiliation 21
Academic-Corporate Collaboration 23
AI Journal Publications 25
Overview 25
By Region 26
By Geographic Area 27
Citation 27
AI Conference Publications 28
Overview 28
By Region 29
By Geographic Area 30
Citation 30
AI Patents 31
Overview 31
arXiv Publications 32
Overview 32
By Region 32
By Geographic Area 33
By Field of Study 34
Highlight: Deep Learning Papers on arXiv 35
1.2 CONFERENCES 36
Conference Attendance 36
Highlight: Corporate Representation
at AI Research Conferences 38
1.3 AI OPEN-SOURCE SOFTWARE
LIBRARIES 39
GitHub Stars 39
Chapter Preview
CHAPTER 1:
ACCESS THE PUBLIC DATA
16. TABLE OF CONTENTS
16
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
Overview
OVERVIEW
The report opens with an overview of the research and development (RD)
efforts in artificial intelligence (AI) because RD is fundamental to AI
progress. Since the technology first captured the imagination of computer
scientists and mathematicians in the 1950s, AI has grown into a major
research discipline with significant commercial applications. The number
of AI publications has increased dramatically in the past 20 years. The rise
of AI conferences and preprint archives has expanded the dissemination of
research and scholarly communications. Major powers, including China, the
European Union, and the United States, are racing to invest in AI research.
The RD chapter aims to capture the progress in this increasingly complex
and competitive field.
This chapter begins by examining AI publications—from peer-reviewed
journal articles to conference papers and patents, including the citation
impact of each, using data from the Elsevier/Scopus and Microsoft
Academic Graph (MAG) databases, as well as data from the arXiv paper
preprint repository and Nesta. It examines contributions to AI RD
from major AI entities and geographic regions and considers how those
contributions are shaping the field. The second and third sections discuss
RD activities at major AI conferences and on GitHub.
CHAPTER 1:
RESEARCH
DEVELOPMENT
17. TABLE OF CONTENTS
17
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
CHAPTER HIGHLIGHTS
•
The number of AI journal publications grew by 34.5% from 2019 to 2020—a much higher
percentage growth than from 2018 to 2019 (19.6%).
•
In every major country and region, the highest proportion of peer-reviewed AI papers
comes from academic institutions. But the second most important originators are
different: In the United States, corporate-affiliated research represents 19.2% of the total
publications, whereas government is the second most important in China (15.6%) and the
European Union (17.2%).
•
In 2020, and for the first time, China surpassed the United States in the share of AI
journal citations in the world, having briefly overtaken the United States in the overall
number of AI journal publications in 2004 and then retaken the lead in 2017. However,
the United States has consistently (and significantly) more cited AI conference papers
than China over the last decade.
•
In response to COVID-19, most major AI conferences took place virtually and registered
a significant increase in attendance as a result. The number of attendees across nine
conferences almost doubled in 2020.
•
In just the last six years, the number of AI-related publications on arXiv grew by more
than sixfold, from 5,478 in 2015 to 34,736 in 2020.
•
AI publications represented 3.8% of all peer-reviewed scientific publications worldwide
in 2019, up from 1.3% in 2011.
CHAPTER
HIGHLIGHTS
CHAPTER 1:
RESEARCH
DEVELOPMENT
18. TABLE OF CONTENTS
18
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
0
20
40
60
80
100
120
Number
of
Peer-Reviewed
AI
Publications
(in
Thousands)
NUMBER of PEER-REVIEWED AI PUBLICATIONS, 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report
AI publications include peer-reviewed publications, journal articles, conference papers, and patents. To track trends among these
publications and to assess the state of AI RD activities around the world, the following datasets were used: the Elsevier/Scopus
database for peer-reviewed publications; the Microsoft Academic Graph (MAG) database for all journals, conference papers, and
patent publications; and arXiv and Nesta data for electronic preprints.
PEER-REVIEWED AI PUBLICATIONS
This section presents data from the Scopus database by
Elsevier. Scopus contains 70 million peer-reviewed research
items curated from more than 5,000 international publishers.
The 2019 version of the data shown below is derived from
an entirely new set of publications, so figures of all peer-
reviewed AI publications differ from those in previous years’
AI Index reports. Due to changes in the methodology for
indexing publications, the accuracy of the dataset increased
from 80% to 84% (see the Appendix for more details).
Overview
Figure 1.1.1a shows the number of peer-reviewed AI
publications, and Figure 1.1.1b shows the share of those
1.1 PUBLICATIONS
among all peer-reviewed publications in the world. The
total number of publications grew by nearly 12 times
between 2000 and 2019. Over the same period, the
percentage of peer-reviewed publications increased from
0.82% of all publications in 2000 to 3.8% in 2019.
By Region1
Among the total number of peer-reviewed AI publications
in the world, East Asia Pacific has held the largest share
since 2004, followed by Europe Central Asia, and North
America (Figure 1.1.2). Between 2009 and 2019, South Asia
and sub-Saharan Africa experienced the highest growth
in terms of the number of peer-reviewed AI publications,
increasing by eight- and sevenfold, respectively.
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.1a
1 Regions in this chapter are classified according to the World Bank analytical grouping.
19. TABLE OF CONTENTS
19
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
0%
1%
2%
3%
4%
Peer-Reviewed
AI
Publications
(%
of
Total)
3.8%
PEER-REVIEWED AI PUBLICATIONS (% of TOTAL), 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.1b
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
0%
10%
20%
30%
40%
Peer-Reviewed
AI
Publications
(%
of
Total)
5.5% Middle East North Africa
2.7% Latin America Caribbean
0.7% Sub-Saharan Africa
8.8% South Asia
17.0% North America
25.1% Europe Central Asia
36.9% East Asia Pacific
PEER-REVIEWED AI PUBLICATIONS (% of TOTAL) by REGION, 2000-19
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
Figure 1.1.2
20. TABLE OF CONTENTS
20
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
0%
5%
10%
15%
20%
25%
Peer-Reviewed
AI
Publications
(%
of
World
Total)
16.4% EU
14.6% US
22.4% China
PEER-REVIEWED AI PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report
By Geographic Area
To compare the activity among the world’s major AI
players, this section shows trends of peer-reviewed AI
publications coming out of China, the European Union,
and the United States. As of 2019, China led in the share
of peer-reviewed AI publications in the world, after
overtaking the European Union in 2017 (Figure 1.1.3).
It published 3.5 times more peer-reviewed AI papers
in 2019 than it did in 2014—while the European Union
published just 2 times more papers and the United States
2.75 times more over the same period.
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.3
21. TABLE OF CONTENTS
21
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
0
1,000
2,000
3,000
4,000
Number
of
Peer-Reviewed
AI
Publications
14 Other
382 Medical
4,352 Government
1,675 Corporate
NUMBER of PEER-REVIEWED AI PUBLICATIONS in CHINA by INSTITUTIONAL AFFILIATION, 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.4a
By Institutional Affiliation
The following charts show the number of peer-reviewed
AI publications affiliated with corporate, government,
medical, and other institutions in China (Figure 1.1.4a),
the European Union (Figure 1.1.4b), and the United States
(Figure 1.1.4c).2
In 2019, roughly 95.4% of overall peer-
reviewed AI publications in China were affiliated with the
academic field, compared with 81.9% in the European
Union and 89.6% in the United States. Those affiliation
categories are not mutually exclusive, as some authors
could be affiliated with more than one type of institution.
The data suggests that, excluding academia, government
institutions—more than those in other categories—
consistently contribute the highest percentage of peer-
reviewed AI publications in both China and the European
Union (15.6% and 17.2 %, respectively, in 2019), while
in the United States, the highest portion is corporate-
affiliated (19.2%).
2 Across all three geographic areas, the number of papers affiliated with academia exceeds that of government-, corporate-, and medical-affiliated ones; therefore, the academia affiliation is not shown,
as it would distort the graphs.
22. TABLE OF CONTENTS
22
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
0
1,000
2,000
3,000
Number
of
Peer-Reviewed
AI
Publications
120 Other
718 Medical
2,277 Gov.
3,513 Corporate
NUMBER of PEER-REVIEWED AI PUBLICATIONS in the UNITED STATES by INSTITUTIONAL AFFILIATION, 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.4c
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
0
1,000
2,000
3,000
Number
of
Peer-Reviewed
AI
Publications
187 Other
508 Medical
3,523 Government
1,594 Corporate
NUMBER of PEER-REVIEWED AI PUBLICATIONS in the EUROPEAN UNION by INSTITUTIONAL AFFILIATION, 2000-19
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report
Figure 1.1.4b
23. TABLE OF CONTENTS
23
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000
Number of Peer-Reviewed AI Publications
United States
European Union
China
United Kingdom
Germany
Japan
France
Canada
South Korea
Netherlands
Switzerland
India
Hong Kong
Spain
Italy
NUMBER of ACADEMIC-CORPORATE PEER-REVIEWED AI PUBLICATIONS by GEOGRAPHIC AREA, 2015-19 (SUM)
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.5
Academic-Corporate Collaboration
Since the 1980s, the RD collaboration between
academia and industry in the United States has grown
in importance and popularity, made visible by the
proliferation of industry-university research centers as
well as corporate contributions to university research.
Figure 1.1.5 shows that between 2015 and 2019, the
United States produced the highest number of hybrid
academic-corporate, co-authored, peer-reviewed AI
publications—more than double the amount in the
European Union, which comes in second, followed by
China in third place.
24. TABLE OF CONTENTS
24
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
5 10 20 50 100 200 500 1,000 2,000
Number of Academic-Corporate Peer-Reviewed AI Publications (Log Scale)
0
1
2
3
Peer-Reviewed
AI
Publications'
Field-Weighted
Citation
Impact
(FWCI)
Germany
Australia
Brazil
Canada
China
European Union
France
Hong Kong
India
Indonesia
Iran
Italy
Japan
Malaysia
Netherlands
Russia
Singapore
South Korea
Spain
Switzerland
Taiwan
Turkey
United Kingdom United States
PEER-REVIEWED AI PUBLICATIONS' FIELD-WEIGHTED CITATION IMPACT and NUMBER of ACADEMIC-CORPORATE
PEER-REVIEWED AI PUBLICATIONS, 2019
Source: Elsevier/Scopus, 2020 | Chart: 2021 AI Index Report
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
To assess how academic-corporate collaborations
impact the Field-Weighted Citation Impact (FWCI) of
AI publications from different geographic regions,
see Figure 1.1.6. FWCI measures how the number of
citations received by publications compares with the
average number of citations received by other similar
publications in the same year, discipline, and format
(book, article, conference paper, etc.). A value of 1.0
represents the world average. More than or less than 1
means publications are cited more or less than expected,
according to the world average. For example, an FWCI of
0.75 means 25% fewer citations than the world average.
The chart shows the FWCI for all peer-reviewed AI
publications on the y-axis and the total number (on a log
scale) of academic-corporate co-authored publications
on the x-axis. To increase the signal-to-noise ratio of the
FWCI metric, only countries that have more than 1,000
peer-reviewed AI publications in 2020 are included.
Figure 1.1.6
25. TABLE OF CONTENTS
25
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
0
20
40
60
80
Number
of
AI
Journal
Publications
(in
Thousands)
NUMBER of AI JOURNAL PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
0%
1%
2%
3%
AI
Journal
Publications
(%
of
All
Publications)
2.2%
AI JOURNAL PUBLICATIONS (% of ALL JOURNAL PUBLICATIONS), 2000-20
Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.7a
AI JOURNAL PUBLICATIONS
The next three sections chart the trends in the publication
of AI journals, conference publications, and patents, as
well as their respective citations that provide a signal
for RD impact, based on data from Microsoft Academic
Graph. MAG3
is a knowledge graph consisting of more than
225 million publications (at the end of November 2019).
Overview
Overall, the number of AI journal publications in 2020 is 5.4
times higher than it was in 2000 (Figure 1.1.7a). In 2020, the
number of AI journal publications increased by 34.5% from
2019—a much higher percentage growth than from 2018 to
2019 (19.6%). Similarly, the share of AI journal publications
among all publications in the world has jumped by 0.4
percentage points in 2020, higher than the average of 0.03
percentage points in the past five years (Figure 1.1.7b).
Figure 1.1.7b
3 See “An Overview of Microsoft Academic Service (MAS) and Applications” and “A Review of Microsoft Academic Services for Science of Science Studies” for more details.
26. TABLE OF CONTENTS
26
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
0%
10%
20%
30%
40%
AI
Journal
Publications
(%
of
World
Total)
0.3%, Sub-Saharan Africa
4.9%, South Asia
14.0%, North America
3.1%, Middle East North Africa
1.3%, Latin America Caribbean
13.3%, Europe Central Asia
26.7%, East Asia Pacific
AI JOURNAL PUBLICATIONS (% of WORLD TOTAL) by REGION, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
By Region
Figure 1.1.8 shows the share of AI journals—the dominant
publication entity in terms of numbers in the MAG
database—by region between 2000 and 2020. East Asia
Pacific, Europe Central Asia, and North America are
responsible for the majority of AI journal publications in
the past 21 years, while the lead position among the three
regions changes over time. In 2020, East Asia Pacific held
the highest share (26.7%), followed by Europe Central
Asia (13.3%) and North America (14.0%). Additionally,
in the last 10 years, South Asia, and Middle East North
Africa saw the most significant growth, as the number of
AI journal publications in those two regions grew six- and
fourfold, respectively.
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.8
27. TABLE OF CONTENTS
27
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
0%
5%
10%
15%
20%
25%
AI
Journal
Publications
(%
of
World
Total)
12.3%, US
8.6%, EU
18.0%, China
AI JOURNAL PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report
By Geographic Area
Figure 1.1.9 shows that among the three major AI powers, China has had the largest share of AI journal publications in the
world since 2017, with 18.0% in 2020, followed by the United States (12.3%) and the European Union (8.6%).
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.9
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
0%
10%
20%
30%
40%
AI
Journal
Citations
(%
of
World
Total)
19.8%
US
11.0%
EU
20.7%
China
AI JOURNAL CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
Citation
In terms of the highest share of AI journal citations, Figure 1.1.10 shows that China (20.7%) overtook the United States
(19.8%) in 2020 for the first time, while the European Union continued to lose overall share.
Figure 1.1.10
28. TABLE OF CONTENTS
28
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
0%
5%
10%
15%
20%
25%
AI
Conference
Publications
(%
of
All
Publications)
20.2%
AI CONFERENCE PUBLICATIONS (% of ALL CONFERENCE PUBLICATIONS), 2000-20
Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
0
10
20
30
40
50
Number
of
Publications
(in
Thousands)
NUMBER of AI CONFERENCE PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.11a
Figure 1.1.11b
AI CONFERENCE PUBLICATIONS
Overview
Between 2000 and 2019, the number of AI conference publications increased fourfold, although the growth flattened
out in the past ten years, with the number of publications in 2019 just 1.09 times higher than the number in 2010.4
4 Note that conference data in 2020 on the MAG system is not yet complete. See the Appendix for details.
29. TABLE OF CONTENTS
29
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
0%
10%
20%
30%
40%
AI
Conference
Publications
(%
of
World
Total)
0.3%, Sub-Saharan Africa
21.7%, North America
2.2%, Middle East North Africa
1.7%, Latin America Caribbean
5.1%, South Asia
18.6%, Europe Central Asia
27.3%, East Asia Pacific
AI CONFERENCE PUBLICATIONS (% of WORLD TOTAL) by REGION, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
By Region
Figure 1.1.12 shows that, similar to the trends in AI
journal publication, East Asia Pacific, Europe Central
Asia, and North America are the world’s dominant
sources for AI conference publications. Specifically, East
Asia Pacific took the lead starting in 2004, accounting
for more than 27% in 2020. North America overtook
Europe Central Asia to claim second place in 2018,
accounting for 20.1%, followed by 21.7% in 2020.
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.12
30. TABLE OF CONTENTS
30
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
0%
10%
20%
30%
40%
AI
Conference
Citations
(%
of
World
Total)
11.8% China
40.1% US
10.9% EU
AI CONFERENCE CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
0%
5%
10%
15%
20%
25%
AI
Conference
Publications
(%
of
World
Total)
12.8% EU
15.2% China
19.4% US
AI CONFERENCE PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20
Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report
By Geographic Area
China overtook the United States in the share of AI
conference publications in the world in 2019 (Figure
1.1.13). Its share has grown significantly since 2000.
China’s percentage of AI conference publications in 2019
is almost nine times higher than it was in 2000. The
share of conference publications for the European Union
peaked in 2011 and continues to decline.
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.13
Citation
With respect to citations of AI conference publications,
Figure 1.1.14 shows that the United States has held a
dominant lead among the major powers over the past
21 years. The United States tops the list with 40.1% of
overall citations in 2020, followed by China (11.8%) and
the European Union (10.9%).
Figure 1.1.14
31. TABLE OF CONTENTS
31
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
0
20
40
60
80
100
Number
of
AI
Patent
Publications
(in
Thousands)
NUMBER of AI PATENT PUBLICATIONS, 2000-20
Source: Microsoft Academic Graph, 2020 | Chart: 2021 AI Index Report
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.15a
AI PATENTS
Overview
The total number of AI patents published in the world
has been steadily increasing in the past two decades,
growing from 21,806 in 2000 to more than 4.5 times that,
or 101,876, in 2019 (Figure 1.1.15a). The share of AI patents
published in the world exhibits a lesser increase, from
around 2% in 2000 to 2.9% in 2020 (Figure 1.1.15b). The AI
patent data is incomplete—only 8% of the dataset in 2020
includes a country or regional affiliation. There is reason
to question the data on the share of AI patent publications
by both region and geographic area, and it is therefore not
included in the main report. See the Appendix for details.
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
0%
1%
2%
3%
AI
Patent
Publications
(%
of
All
Publications)
2.9%
AI PATENT PUBLICATIONS (% of ALL PATENT PUBLICATIONS), 2000-20
Source: Microso Academic Graph, 2020 | Chart: 2021 AI Index Report
Figure 1.1.15b
32. TABLE OF CONTENTS
32
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2015 2016 2017 2018 2019 2020
0
10
20
30
Number
of
AI-Related
Publications
on
arXiv
(in
Thousands)
NUMBER of AI-RELATED PUBLICATIONS on ARXIV, 2015-20
Source: arXiv, 2020 | Chart: 2021 AI Index Report
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.16
ARXIV PUBLICATIONS
In addition to the traditional avenues for
publishing academic papers (discussed
above), AI researchers have embraced the
practice of publishing their work (often
pre–peer review) on arXiv, an online
repository of electronic preprints. arXiv
allows researchers to share their findings
before submitting them to journals and
conferences, which greatly accelerates
the cycle of information discovery and
dissemination. The number of AI-related
publications in this section includes
preprints on arXiv under cs.AI (artificial
intelligence), cs.CL (computation and
language), cs.CV (computer vision), cs.NE
(neural and evolutionary computing),
cs.RO (robotics), cs.LG (machine learning in
computer science), and stat.ML (machine
learning in statistics).
Overview
In just six years, the number of AI-related publications on arXiv grew
more than sixfold, from 5,478 in 2015 to 34,736 in 2020 (Figure 1.1.16).
2015 2016 2017 2018 2019 2020 2021
0%
10%
20%
30%
40%
AI-Related
Publications
on
arXiv
(%
of
World
Total)
4.0% South Asia
2.5% Middle East North Africa
1.3% Latin America Caribbean
0.3% Sub-Saharan Africa
36.3% North America
22.9% Europe Central Asia
26.5% East Asia Paci c
ARXIV AI-RELATED PUBLICATIONS (% of WORLD TOTAL) by REGION, 2015-20
Source: arXiv, 2020 | Chart: 2021 AI Index Report
By Region
The analysis by region shows that while North America still holds the lead in the global share of arXiV AI-related
publications, its share has been decreasing—from 41.6% in 2017 to 36.3% in 2020 (Figure 1.1.17). Meanwhile, the share of
publications in East Asia Pacific has grown steadily in the past five years—from 17.3% in 2015 to 26.5% in 2020.
Figure 1.1.17
33. TABLE OF CONTENTS
33
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2015 2016 2017 2018 2019 2020
0
2,000
4,000
6,000
8,000
10,000
12,000
Number
of
AI-Related
Publications
on
arXiv
11,280
US
6,505
EU
5,440
China
NUMBER of AI-RELATED PUBLICATIONS on ARXIV by GEOGRAPHIC AREA, 2015-20
Source: arXiv, 2020 | Chart: 2021 AI Index Report
2015 2016 2017 2018 2019 2020
0%
10%
20%
30%
AI-Related
Publications
on
arXiv
(%
of
World
Total)
32.5% US
18.7% EU
15.7% China
ARXIV AI-RELATED PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2015-20
Source: arXiv, 2020 | Chart: 2021 AI Index Report
By Geographic Area
While the total number of AI-related publications on arXiv
is increasing among the three major AI powers, China is
catching up with the United States (Figure 1.1.18a and
Figure 1.1.18b). The share of publication counts by the
European Union, on the other hand, has remained largely
unchanged.
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.18a
Figure 1.1.18b
34. TABLE OF CONTENTS
34
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2015 2016 2017 2018 2019 2020
0
2,000
4,000
6,000
8,000
10,000
Number
of
AI-Related
Publications
on
arXiv
11,098 cs.LG
11,001 cs.CV
1,818 stat.ML
2,571 cs.RO
743 cs.NE
5,573 cs.CL
1,923 cs.AI
NUMBER of AI-RELATED PUBLICATIONS on ARXIV by FIELD of STUDY 2015-20
Source: arXiv, 2020 | Chart: 2021 AI Index Report
By Field of Study
Among the six fields of study related to AI on arXiv,
the number of publications in Robotics (cs.RO) and
Machine Learning in computer science (cs.LG) have
seen the fastest growth between 2015 and 2020,
increasing by 11 times and 10 times respectively
(Figure 1.1.19). In 2020, cs.LG and Computer Vision
(cs.CV) lead in the overall number of publications,
accounting for 32.0% and 31.7%, respectively, of all AI-
related publications on arXiv. Between 2019 and 2020,
the fastest-growing categories of the seven studied
here were Computation and Language (cs.CL), by
35.4%, and cs.RO, by 35.8%.
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.1.19
Among the six fields of
study related to AI on arXiv,
the number of publications
in Robotics (cs.RO) and
Machine Learning in
computer science (cs.
LG) have seen the fastest
growth between 2015 and
2020, increasing by 11 times
and 10 times respectively.
35. TABLE OF CONTENTS
35
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
Deep Learning Papers on arXiv
With increased access to data and significant
improvements in computing power, the field
of deep learning (DL) is growing at breakneck
speed. Researchers from Nesta used a topic
modeling algorithm to identify the deep learning
papers on arXiv by analyzing the abstract of
arXiv papers under the Computer Science (CS)
and Machine Learning in Statistics (state.ML)
categories. Figure 1.1.20 suggests that in the
last five years alone, the overall number of DL
publications on arXiv grew almost sixfold.
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
0
1
2
3
4
5
6
7
Number
of
Deep
Learning
Publications
on
arXiv
(in
Thousands)
NUMBER of DEEP LEARNING PUBLICATIONS on ARXIV, 2010-19
Source: arXiv/Nesta, 2020 | Chart: 2021 AI Index Report
Figure 1.1.20
1.1 PUBLICATIONS
CHAPTER 1:
RESEARCH
DEVELOPMENT
36. TABLE OF CONTENTS
36
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
Conference attendance is an indication of broader industrial and academic interest in a scientific field. In the past 20 years,
AI conferences have grown not only in size but also in number and prestige. This section presents data on the trends in
attendance at and submissions to major AI conferences.
CONFERENCE ATTENDANCE
Last year saw a significant increase in participation levels
at AI conferences, as most were offered through a virtual
format. Only the 34th Association for the Advancement
of Artificial Intelligence (AAAI) Conference on Artificial
Intelligence was held in person in February 2020.
Conference organizers report that a virtual format allows
for higher attendance of researchers from all over the
world, though exact attendance numbers are difficult to
measure.
Due to the atypical nature of 2020 conference attendance
data, the 11 major AI conferences in 2019 have been
split into two categories based on 2019 attendance data:
large AI conferences with over 3,000 attendees and small
AI conferences with fewer than 3,000 attendees. Figure
1.2.1 shows that in 2020, the total number of attendees
across nine conferences almost doubled.5
In particular,
the International Conference on Intelligent Robots
and Systems (IROS) extended the virtual conference
to allow users to watch events for up to three months,
which explains the high attendance count. Because the
International Joint Conference on Artificial Intelligence
(IJCAI) was held in 2019 and January 2021—but not in
2020—it does not appear on the charts.
1.2 CONFERENCES
1.2 CONFERENCES
CHAPTER 1:
RESEARCH
DEVELOPMENT
Conference organizers
report that a virtual
format allows for higher
attendance of researchers
from all over the world,
though exact attendance
numbers are difficult to
measure.
5 For the AAMAS conference, the attendance in 2020 is based on the number of users on site reported by the platform that recorded the talks and managed the online conference; For the KR conference,
the attendance in 2020 is based on the number of registrations; For the ICPAS conference, the attendance of 450 in 2020 is an estimate as some participants may have used anonymous Zoom accounts.
37. TABLE OF CONTENTS
37
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
0
1,000
2,000
3,000
4,000
5,000
Number
of
Attendees
469
KR
450
ICAPS
3,726
AAMAS
5,600
ICLR
3,972
ACL
ATTENDANCE at SMALL AI CONFERENCES, 2010-20
Source: Conference Data | Chart: 2021 AI Index Report
1.2 CONFERENCES
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.2.2
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
0
5,000
10,000
15,000
20,000
25,000
Number
of
Attendees
22,011 NeurIPS
25,719 IROS
3,015 IJCAI
3,050 ICRA
4,884 AAAI
10,800 ICML
7,500 CVPR
ATTENDANCE at LARGE AI CONFERENCES, 2010-20
Source: Conference Data | Chart: 2021 AI Index Report
Figure 1.2.1
38. TABLE OF CONTENTS
38
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
Corporate Representation at AI Research Conferences
Researchers from Virginia Tech and Ivey
Business School, Western University found
that large technology firms have increased
participation in major AI conferences. In
their paper, titled “The De-Democratization
of AI: Deep Learning and the Compute
Divide in Artificial Intelligence Research,” the
ressearchers use the share of papers affiliated
with firms over time at AI conferences to
illustrate the increased presence of firms
in AI research. They argue that the unequal
distribution of compute power in academia,
which they refer to as the “compute divide,”
is adding to the inequality in the era of deep
learning. Big tech firms tend to have more
resources to design AI products, but they also
tend to be less diverse than less elite or smaller
institutions. This raises concerns about bias and
fairness within AI. All 10 major AI conferences
displayed in Figure 1.2.3 show an upward trend
in corporate representation, which further
extends the compute divide.
0%
10%
20%
30%
40%
0%
10%
20%
30%
40%
0%
10%
20%
30%
40%
30.8% KDD
29.0% NeurIPS 28.7% ACL 28.5% EMNLP
27.9% ICML
25.6% ECCV 23.7% ICCV
21.9% CVPR
19.3% AAAI 17.5% IJCAI
SHARE of FORTUNE GLOBAL 500 TECH-AFFILIATED PAPERS
Source: Ahmed Wahed, 2020 | Chart: 2021 AI Index Report
%
of
Fortune
Global
500
Tech-Affiliated
Papers
2000 2019 2000 2019 2000 2019 2000 2019
Figure 1.2.3
1.2 CONFERENCES
CHAPTER 1:
RESEARCH
DEVELOPMENT
39. TABLE OF CONTENTS
39
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
A software library is a collection of computer code that is used to create applications and products. Popular AI-specific software
libraries—such as TensorFlow and PyTorch—help developers create their AI solutions quickly and efficiently. This section analyzes
the popularity of software libraries through GitHub data.
GITHUB STARS
GitHub is a code hosting platform that AI researchers and
developers frequently use to upload, comment on, and
download software. GitHub users can “star” a project
to save it in their list, thereby expressing their interests
and likes—similar to the “like’’ function on Twitter and
other social media platforms. As AI researchers upload
packages on GitHub that mention the use of an open-
source library, the “star” function on GitHub can be used
to measure the popularity of various AI programming
open-source libraries.
Figure 1.3.1 suggests that TensorFlow (developed by
Google and publicly released in 2017) is the most popular
AI software library. The second most popular library in
2020 is Keras (also developed by Google and built on top
of TensorFlow 2.0). Excluding TensorFlow, Figure 1.3.2
shows that PyTorch (created by Facebook) is another
library that is becoming increasingly popular.
1.3 AI OPEN-SOURCE
SOFTWARE LIBRARIES
1.3 AI OPEN-
SOURCE SOFTWARE
LIBRARIES
CHAPTER 1:
RESEARCH
DEVELOPMENT
TensorFlow (developed
by Google and publicly
released in 2017) is the
most popular AI software
library. The second most
popular library in 2020 is
Keras (also developed by
Google and built on top
of TensorFlow 2.0).
40. TABLE OF CONTENTS
40
CHAPTER 1 PREVIEW
Artificial Intelligence
Index Report 2021
2014 2015 2016 2017 2018 2019 2020
0
10
20
30
40
50
Cumulative
Github
Stars
(in
Thousand)
45 Sckit-learn
19 MXNet
51 Keras
8 Caffe2
9 Theano
46 PyTorch
17 Cntk
31 BVLC/caffe
NUMBER of GITHUB STARS by AI LIBRARY (excluding TENSORFLOW), 2014-20
Source: GitHub, 2020 | Chart: 2021 AI Index Report
1.3 AI OPEN-
SOURCE SOFTWARE
LIBRARIES
CHAPTER 1:
RESEARCH
DEVELOPMENT
Figure 1.3.2
2014 2015 2016 2017 2018 2019 2020
0
50
100
150
Cumulative
Github
Stars
(in
Thousand)
45 Sckit-learn
19 MXNet
51 Keras
8 Caffe2
9 Theano
153 TensorFlow
46 PyTorch
17 Cntk
31 BVLC/caffe
NUMBER of GITHUB STARS by AI LIBRARY, 2014-20
Source: GitHub, 2020 | Chart: 2021 AI Index Report
Figure 1.3.1
41. TABLE OF CONTENTS
41
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
CHAPTER 2:
Technical
Performance
Artificial Intelligence
Index Report 2021
42. TABLE OF CONTENTS
42
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
CHAPTER 2:
TECHNICAL
PERFORMANCE
Overview 43
Chapter Highlights 44
COMPUTER VISION 45
2.1 COMPUTER VISION—IMAGE 46
Image Classification 46
ImageNet 46
ImageNet: Top-1 Accuracy 46
ImageNet: Top-5 Accuracy 47
ImageNet: Training Time 48
ImageNet: Training Costs 49
Highlight: Harder Tests Beyond ImageNet 50
Image Generation 51
STL-10: Fréchet Inception Distance
(FID) Score 51
FID Versus Real Life 52
Deepfake Detection 53
Deepfake Detection Challenge (DFDC) 53
Human Pose Estimation 54
Common Objects in Context (COCO):
Keypoint Detection Challenge 54
Common Objects in Context (COCO):
DensePose Challenge 55
Semantic Segmentation 56
Cityscapes 56
Embodied Vision 57
2.2 COMPUTER VISION—VIDEO 58
Activity Recognition 58
ActivityNet 58
ActivityNet: Temporal Action
Localization Task 58
ActivityNet: Hardest Activity 59
Object Detection 60
You Only Look Once (YOLO) 60
Face Detection and Recognition 61
National Institute of Standards and
Technology (NIST) Face Recognition
Vendor Test (FRVT) 61
2.3 LANGUAGE 62
English Language Understanding Benchmarks 62
SuperGLUE 62
SQuAD 63
Commercial Machine Translation (MT) 64
Number of Commercially Available
MT Systems 64
GPT-3 65
2.4 LANGUAGE REASONING SKILLS 67
Vision and Language Reasoning 67
Visual Question Answering (VQA) Challenge 67
Visual Commonsense Reasoning (VCR) Task 68
2.5 SPEECH 69
Speech Recognition 69
Transcribe Speech: LibriSpeech 69
Speaker Recognition: VoxCeleb 69
Highlight: The Race Gap in
Speech Recognition Technology 71
2.6 REASONING 72
Boolean Satisfiability Problem 72
Automated Theorem Proving (ATP) 74
2.7 HEALTHCARE AND BIOLOGY 76
Molecular Synthesis 76
Test Set Accuracy for Forward Chemical
Synthesis Planning 76
COVID-19 and Drug Discovery 77
AlphaFold and Protein Folding 78
EXPERT HIGHLIGHTS 79
Chapter Preview
CHAPTER 2:
ACCESS THE PUBLIC DATA
43. TABLE OF CONTENTS
43
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
Overview
OVERVIEW
This chapter highlights the technical progress in various subfields of
AI, including computer vision, language, speech, concept learning, and
theorem proving. It uses a combination of quantitative measurements,
such as common benchmarks and prize challenges, and qualitative
insights from academic papers to showcase the developments in state-of-
the-art AI technologies.
While technological advances allow AI systems to be deployed more
widely and easily than ever, concerns about the use of AI are also growing,
particularly when it comes to issues such as algorithmic bias. The
emergence of new AI capabilities such as being able to synthesize images
and videos also poses ethical challenges.
CHAPTER 2:
TECHNICAL
PERFORMANCE
44. TABLE OF CONTENTS
44
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
CHAPTER HIGHLIGHTS
•
Generative everything: AI systems can now compose text, audio, and images to a sufficiently high
standard that humans have a hard time telling the difference between synthetic and non-synthetic
outputs for some constrained applications of the technology. That promises to generate a tremendous
range of downstream applications of AI for both socially useful and less useful purposes. It is
also causing researchers to invest in technologies for detecting generative models; the DeepFake
Detection Challenge data indicates how well computers can distinguish between different outputs.
•
The industrialization of computer vision: Computer vision has seen immense progress in the past
decade, primarily due to the use of machine learning techniques (specifically deep learning). New
data shows that computer vision is industrializing: Performance is starting to flatten on some of the
largest benchmarks, suggesting that the community needs to develop and agree on harder ones
that further test performance. Meanwhile, companies are investing increasingly large amounts
of computational resources to train computer vision systems at a faster rate than ever before.
Meanwhile, technologies for use in deployed systems—like object-detection frameworks for
analysis of still frames from videos—are maturing rapidly, indicating further AI deployment.
•
Natural Language Processing (NLP) outruns its evaluation metrics: Rapid progress in NLP has
yielded AI systems with significantly improved language capabilities that have started to have a
meaningful economic impact on the world. Google and Microsoft have both deployed the BERT
language model into their search engines, while other large language models have been developed
by companies ranging from Microsoft to OpenAI. Progress in NLP has been so swift that technical
advances have started to outpace the benchmarks to test for them. This can be seen in the rapid
emergence of systems that obtain human level performance on SuperGLUE, an NLP evaluation suite
developed in response to earlier NLP progress overshooting the capabilities being assessed by GLUE.
•
New analyses on reasoning: Most measures of technical problems show for each time point the
performance of the best system at that time on a fixed benchmark. New analyses developed for
the AI Index offer metrics that allow for an evolving benchmark, and for the attribution to individual
systems of credit for a share of the overall performance of a group of systems over time. These
are applied to two symbolic reasoning problems, Automated Theorem Proving and Satisfiability of
Boolean formulas.
•
Machine learning is changing the game in healthcare and biology: The landscape of the healthcare
and biology industries has evolved substantially with the adoption of machine learning. DeepMind’s
AlphaFold applied deep learning technique to make a significant breakthrough in the decades-long
biology challenge of protein folding. Scientists use ML models to learn representations of chemical
molecules for more effective chemical synthesis planning. PostEra, an AI startup used ML-based
techniques to accelerate COVID-related drug discovery during the pandemic.
CHAPTER
HIGHLIGHTS
CHAPTER 2:
TECHNICAL
PERFORMANCE
45. TABLE OF CONTENTS
45
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
Introduced in the 1960s, the field of computer vision has seen significant
progress and in recent years has started to reach human levels of
performance on some restricted visual tasks. Common computer
vision tasks include object recognition, pose estimation, and semantic
segmentation. The maturation of computer vision technology has unlocked
a range of applications: self-driving cars, medical image analysis, consumer
applications (e.g., Google Photos), security applications (e.g., surveillance,
satellite imagery analysis), industrial applications (e.g., detecting defective
parts in manufacturing and assembly), and others.
COMPUTER
VISION
CHAPTER 2:
TECHNICAL
PERFORMANCE
Computer Vision
46. TABLE OF CONTENTS
46
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
IMAGE CLASSIFICATION
In the 2010s, the field of image recognition and
classification began to switch from classical AI techniques
to ones based on machine learning and, specifically,
deep learning. Since then, image recognition has shifted
from being an expensive, domain-specific technology to
being one that is more affordable and applicable to more
areas—primarily due to advancements in the underlying
technology (algorithms, compute hardware, and the
utilization of larger datasets).
ImageNet
Created by computer scientists from Stanford University
and Princeton University in 2009, ImageNet is a dataset
of over 14 million images across 200 classes that expands
and improves the data available for researchers to train
AI algorithms. In 2012, researchers from the University of
Toronto used techniques based on deep learning to set
a new state of the art in the ImageNet Large Scale Visual
Recognition Challenge.
Since then, deep learning techniques have ruled
the competition leaderboards—several widely used
techniques have debuted in ImageNet competition
entries. In 2015, a team from Microsoft Research said it
had surpassed human-level performance on the image
classification task1
via the use of “residual networks”—an
innovation that subsequently proliferated into other AI
systems. Even after the end of the competition in 2017,
researchers continue to use the ImageNet dataset to test
and develop computer vision applications.
The image classification task of the ImageNet Challenge
asks machines to assign a class label to an image based
on the main object in the image. The following graphs
explore the evolution of the top-performing ImageNet
systems over time, as well as how algorithmic and
infrastructure advances have allowed researchers to
increase the efficiency of training image recognition
systems and reduce the absolute time it takes to train
high-performing ones.
ImageNet: Top-1 Accuracy
Top-1 accuracy tests for how well an AI system can
assign the correct label to an image, specifically whether
its single most highly probable prediction (out of all
possible labels) is the same as the target label. In recent
years, researchers have started to focus on improving
performance on ImageNet by pre-training their systems
on extra training data, for instance photos from
Instagram or other social media sources. By pre-training
on these datasets, they’re able to more effectively use
ImageNet data, which further improves performance.
Figure 2.1.1 shows that recent systems with extra training
data make 1 error out of every 10 tries on top-1 accuracy,
versus 4 errors out of every 10 tries in December 2012.
The model from the Google Brain team achieved 90.2%
on top-1 accuracy in January 2021.
2.1 COMPUTER VISION—IMAGE
2.1 COMPUTER
VISION—IMAGE
CHAPTER 2:
TECHNICAL
PERFORMANCE
1 Though it is worth noting that the human baseline for this metric comes from a single Stanford graduate student who took roughly the same test as the AI systems took.
Image recognition has
shifted from being an
expensive, domain-specific
technology to being one
that is more affordable
and applicable to more
areas—primarily due
to advancements in the
underlying technology.
47. TABLE OF CONTENTS
47
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
75%
80%
85%
90%
95%
100%
Top-5
Accuracy
98.8%, With Extra Training Data
97.9%, Without Extra Training Data
94.9% Human Performance
IMAGENET CHALLENGE: TOP-5 ACCURACY
Source: Papers with Code, 2020; AI Index, 2021 | Chart: 2021 AI Index Report
01/2013 01/2014 01/2015 01/2016 01/2017 01/2018 01/2019 01/2020 01/2021
90..
.
Human Performance
ImageNet: Top-5 Accuracy
Top-5 accuracy asks whether the correct label is in at least the classifier’s top five predictions. Figure 2.1.2 shows that
the error rate has improved from around 85% in 2013 to almost 99% in 2020.2
2.1 COMPUTER
VISION—IMAGE
CHAPTER 2:
TECHNICAL
PERFORMANCE
2 Note: For data on human error, a human was shown 500 images and then was asked to annotate 1,500 test images; their error rate was 5.1% for Top-5 classification. This is a very rough baseline, but it
gives us a sense of human performance on this task.
60%
70%
80%
90%
100%
Top-1
Accuracy
90.2%, With extra training data
86.5%, Without extra training data
IMAGENET CHALLENGE: TOP-1 ACCURACY
Source: Papers with Code, 2020; AI Index, 2021 | Chart: 2021 AI Index Report
01/2013 01/2014 01/2015 01/2016 01/2017 01/2018 01/2019 01/2020 01/2021
80..
.
Figure 2.1.1
Figure 2.1.2
48. TABLE OF CONTENTS
48
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
ImageNet: Training Time
Along with measuring the raw improvement in accuracy
over time, it is useful to evaluate how long it takes
to train image classifiers on ImageNet to a standard
performance level as it sheds light on advances in the
underlying computational infrastructure for large-scale
AI training. This is important to measure because the
faster you can train a system, the more quickly you
can evaluate it and update it with new data. Therefore,
the faster ImageNet systems can be trained, the more
productive organizations can become at developing and
deploying AI systems. Imagine the difference between
waiting a few seconds for a system to train versus waiting
a few hours, and what that difference means for the type
and volume of ideas researchers explore and how risky
they might be.
What follows are the results from MLPerf, a competition
run by the MLCommons organization that challenges
entrants to train an ImageNet network using a common
(residual network) architecture, and then ranks systems
according to the absolute “wall clock” time it takes them
to train a system.3
As shown in Figure 2.1.3, the training time on ImageNet
has fallen from 6.2 minutes (December 2018) to 47
seconds (July 2020). At the same time, the amount of
hardware used to achieve these results has increased
dramatically; frontier systems have been dominated by
the use of “accelerator” chips, starting with GPUs in the
2018 results, and transitioning to Google’s TPUs for the
best-in-class results from 2019 and 2020.
Distribution of Training Time: MLPerf does not just
show the state of the art for each competition period;
it also makes available all the data behind each
entry in each competition cycle. This, in turn, reveals
the distribution of training times for each period
(Figure 2.1.3). (Note that in each MLPerf competition,
competitors typically submit multiple entries that use
different permutations of hardware.)
Figure 2.1.4 shows that in the past couple of years,
training times have shortened, as has the variance
between MLPerf entries. At the same time, competitors
have started to use larger and larger numbers of
accelerator chips to speed training times. This is in line
with broader trends in AI development, as large-scale
training becomes better understood, with a higher
degree of shared best practices and infrastructure.
2.1 COMPUTER
VISION—IMAGE
CHAPTER 2:
TECHNICAL
PERFORMANCE
3 The next MLPerf update is planned for June 2021.
Imagine the difference
between waiting a few
seconds for a system to
train versus waiting a
few hours, and what that
difference means for the
type and volume of ideas
researchers explore and
how risky they might be.
49. TABLE OF CONTENTS
49
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
12/2017 06/2018 12/2018 06/2019 12/2019
1
2
5
10
20
50
100
200
500
1,000
2,000
Cost
(U.S.
Dollars;
Log
Scale)
$7.43
IMAGENET: TRAINING COST (to 93% ACCURACY)
Source: DAWNBench, 2020 | Chart: 2021 AI Index Report
ImageNet: Training Costs
How much does it cost to train a contemporary image-
recognition system? The answer, according to tests run by
the Stanford DAWNBench team, is a few dollars in 2020,
down by around 150 times from costs in 2017 (Figure
2.1.5). To put this in perspective, what cost one entrant
around USD 1,100 to do in October 2017 now costs about
USD 7.43. This represents progress in algorithm design as
well as a drop in the costs of cloud-computing resources.
2.1 COMPUTER
VISION—IMAGE
CHAPTER 2:
TECHNICAL
PERFORMANCE
Figure 2.1.5
12/2018 06/2019 07/2020
0
1,000
2,000
3,000
4,000
Number
of
Accelerators
0
1
2
3
4
5
6
Training
Time
4,096
1,024
640
6.2 Min
1.3 Min
47 Sec
IMAGENET: TRAINING TIME and HARDWARE of the BEST
SYSTEM
Source: MLPerf, 2020 | Chart: 2021 AI Index Report
Number of Accelerators Training Time
2018 2019 2020
1
10
100
1,000
10,000
Training
Time
(mins)
IMAGENET: DISTRIBUTION of TRAINING TIME
Source: MLPerf, 2020 | Chart: 2021 AI Index Report
Figure 2.1.3 Figure 2.1.4
50. TABLE OF CONTENTS
50
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
Harder Tests Beyond ImageNet
In spite of the progress in performance on ImageNet, current computer vision systems are still not
perfect. To better study their limitations, researchers have in recent years started to develop more
challenging image classification benchmarks. But since ImageNet is already a large dataset, which
requires a nontrivial amount of resources to use, it does not intuitively make sense to simply expand the
resolution of the images in ImageNet or the absolute size of the dataset—as either action would further
increase the cost to researchers when training systems on ImageNet. Instead, people have tried to figure
out new ways to test the robustness of image classifiers by creating custom datasets, many of which are
compatible with ImageNet (and are typically smaller). These include
IMAGENET ADVERSARIAL:
This is a dataset of images
similar to those found in
ImageNet but incorporating
natural confounders (e.g., a
butterfly sitting on a carpet with
a similar texture to the butterfly),
and images that are persistently
misclassified by contemporary
systems. These images “cause
consistent classification
mistakes due to scene
complications encountered
in the long tail of scene
configurations and by exploiting
classifier blind spots,” according
to the researchers. Therefore,
making progress on ImageNet
Adversarial could improve the
ability of models to generalize.
IMAGENET-C:
This is a dataset of common
ImageNet images with 75 visual
corruptions applied to them
(e.g., changes in brightness,
contrast, pixelations, fog
effects, etc.). By testing systems
against this, researchers can
provide even more information
about the generalization
capabilities of these models.
IMAGENET-RENDITION:
This tests generalization by
seeing how well ImageNet-
trained models can categorize
30,000 illustrations of 200
ImageNet classes. Since
ImageNet is designed to be built
out of photos, generalization
here indicates that systems
have learned something more
subtle about what they’re trying
to classify, because they’re able
to “understand” the relationship
between illustrations and the
photographed images they’ve
been trained on.
What is the Time Table for Tracking This Data? As these benchmarks are relatively new, the plan is to
wait a couple of years for the community to test a range of systems against them, which will generate
the temporal information necessary to make graphs tracking progress overtime.
2.1 COMPUTER
VISION—IMAGE
CHAPTER 2:
TECHNICAL
PERFORMANCE
51. TABLE OF CONTENTS
51
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
IMAGE GENERATION
Image generation is the task of generating images
that look indistinguishable from “real” images. Image
generation systems have a variety of uses, ranging from
augmenting search capabilities (it is easier to search for
a specific image if you can generate other images like
it) to serving as an aid for other generative uses (e.g.,
editing images, creating content for specific purposes,
generating multiple variations of a single image to help
designers brainstorm, and so on).
In recent years, image generation progress has
accelerated as a consequence of the continued
improvement in deep learning–based algorithms, as well
as the use of increased computation and larger datasets.
STL-10: Fréchet Inception Distance (FID) Score
One way to measure progress in image generation is via a
technique called Fréchet Inception Distance (FID), which
roughly correlates to the difference between how a given
AI system “thinks” about a synthetic image versus a real
image, where a real image has a score of 0 and synthetic
images that look similar have scores that approach 0.
Figure 2.1.6 shows the progress of generative models
over the past two years at generating convincing
synthetic images in the STL-10 dataset, which is designed
to test how effective systems are at generating images
and gleaning other information about them.
2.1 COMPUTER
VISION—IMAGE
CHAPTER 2:
TECHNICAL
PERFORMANCE
01/2018 04/2018 07/2018 10/2018 01/2019 04/2019 07/2019 10/2019 01/2020 04/2020 07/2020
20
25
30
35
40
45
Fréchet
Inception
Distance
(FID)
Score
25.4
STL-10: FRÉCHET INCEPTION DISTANCE (FID) SCORE
Source: Papers with Code, 2020 | Chart: 2021 AI Index Report
Figure 2.1.6
52. TABLE OF CONTENTS
52
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
2.1 COMPUTER
VISION—IMAGE
CHAPTER 2:
TECHNICAL
PERFORMANCE
FID Versus Real Life
FID has drawbacks as an evaluation technique—
specifically, it assesses progress on image generation
via quantitative metrics that use data from the model
itself, rather than other evaluation techniques. Another
approach is using teams of humans to evaluate the
outputs of these models; for instance, the Human eYe
Perceptual Evaluation (HYPE) method tries to judge image
quality by showing synthetically generated images to
humans and using their qualitative ratings to drive the
evaluation methodology. This approach is more expensive
and slower to run than typical evaluations, but it may
become more important as generative models get better.
Qualitative Examples: To get a sense of progress, you
can look at the evolution in the quality of synthetically
generated images over time. In Figure 2.1.7, you can
see the best-in-class examples of synthetic images of
human faces, ordered over time. By 2018, performance of
this task had become sufficiently good that it is difficult
for humans to easily model further progress (though it
is possible to train machine learning systems to spot
fakes, it is becoming more challenging). This provides a
visceral example of recent progress in this domain and
underscores the need for new evaluation methods to
gauge future progress. In addition, in recent years people
have turned to doing generative modeling on a broader
range of categories than just images of people’s faces,
which is another way to test for generalization.
Figure 2.1.7
GAN PROGRESS ON FACE GENERATION
2014 2015 2016
2017
2018
2020
Source: Goodfellow et al., 2014; Radford et al., 2016; Liu Tuzel, 2016; Karras et al., 2018; Karras et al., 2019; Goodfellow, 2019; Karras et al., 2020; AI Index, 2021
53. TABLE OF CONTENTS
53
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
DEEPFAKE DETECTION
Advances in image synthesis have created new
opportunities as well as threats. For instance, in recent
years, researchers have harnessed breakthroughs
in synthetic imagery to create AI systems that can
generate synthetic images of human faces, then
superimpose those faces onto the faces of other people
in photographs or movies. People call this application
of generative technology a “deepfake.” Malicious uses
of deepfakes include misinformation and the creation
of (predominantly misogynistic) pornography. To try
to combat this, researchers are developing deepfake-
detection technologies.
Deepfake Detection Challenge (DFDC)
Created in September 2019 by Facebook, the Deepfake
Detection Challenge (DFDC) measures progress on
deepfake-detection technology. A two-part challenge,
DFDC asks participants to train and test their models from
a public dataset of around 100,000 clips. The submissions
are scored on log loss, a classification metric based on
probabilities. A smaller log loss means a more accurate
prediction of deepfake videos. According to Figure
2.1.8, log loss dropped by around 0.5 as the challenge
progressed between December 2019 and March 2020.
2.1 COMPUTER
VISION—IMAGE
CHAPTER 2:
TECHNICAL
PERFORMANCE
1/6/2020 1/16/2020 1/26/2020 2/5/2020 2/15/2020 2/25/2020 3/6/2020 3/16/2020 3/26/2020
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Log
Loss
0.19
DEEPFAKE DETECTION CHALLENGE: LOG LOSS
Source: Kaggle, 2020 | Chart: 2021 AI Index Report
Figure 2.1.8
54. TABLE OF CONTENTS
54
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
HUMAN POSE ESTIMATION
Human pose estimation is the problem of estimating the
positions of human body parts or joints (wrists, elbows,
etc.) from a single image. Human pose estimation is a
classic “omni-use” AI capability. Systems that are good at
this task can be used for a range of applications, such as
creating augmented reality applications for the fashion
industry, analyzing behaviors gleaned from physical
body analysis in crowds, surveilling people for specific
behaviors, aiding with analysis of live sporting and
athletic events, mapping the movements of a person to a
virtual avatar, and so on.
Common Objects in Context (COCO):
Keypoint Detection Challenge
Common Objects in Context (COCO) is a large-scale
dataset for object detection, segmentation, and
captioning with 330,000 images and 1.5 million object
instances. Its Keypoint Detection Challenge requires
machines to simultaneously detect an object or a person
and localize their body keypoints—points in the image
that stand out, such as a person’s elbows, knees, and
other joints. The task evaluates algorithms based on
average precision (AP), a metric that can be used to
measure the accuracy of object detectors. Figure 2.1.9
shows that the accuracy of algorithms in this task has
improved by roughly 33% in the past four years, with the
latest machine scoring 80.8% on average precision.
2.1 COMPUTER
VISION—IMAGE
CHAPTER 2:
TECHNICAL
PERFORMANCE
07/2016 01/2017 07/2017 01/2018 07/2018 01/2019 07/2019 01/2020 07/2020
50%
60%
70%
80%
90%
Average
Precision
(AP)
80.8%
COCO KEYPOINT CHALLENGE: AVERAGE PRECISION
Source: COCO Leaderboard, 2020 | Chart: 2021 AI Index Report
Figure 2.1.9
55. TABLE OF CONTENTS
55
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
Common Objects in Context (COCO):
DensePose Challenge
DensePose, or dense human pose estimation, is the task
of extracting a 3D mesh model of a human body from a
2D image. After open-sourcing a system called DensePose
in 2018, Facebook built DensePose-COCO, a large-scale
dataset of image-to-surface correspondences annotated
on 50,000 COCO images. Since then, DensePose has
become a canonical benchmark dataset.
The COCO DensePose Challenge involves tasks of
simultaneously detecting people, segmenting their
bodies, and estimating the correspondences between
image pixels that belong to a human body and a
template 3D model. The average precision is calculated
based on the geodesic point similarity (GPS) metric,
a correspondence matching score that measures the
geodesic distances between the estimated points and
the true location of the body points on the image. The
accuracy has grown from 56% in 2018 to 72% in 2019
(Figure 2.1.10).
2.1 COMPUTER
VISION—IMAGE
CHAPTER 2:
TECHNICAL
PERFORMANCE
03/2018 05/2018 07/2018 09/2018 11/2018 01/2019 03/2019 05/2019 07/2019 09/2019
50%
55%
60%
65%
70%
75%
Average
Precision
(AP)
72%
COCO DENSEPOSE CHALLENGE: AVERAGE PRECISION
Source: arXiv CodaLab, 2020 | Chart: 2021 AI Index Report
Figure 2.1.10
56. TABLE OF CONTENTS
56
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
SEMANTIC SEGMENTATION
Semantic segmentation is the task of classifying each
pixel in an image to a particular label, such as person,
cat, etc. Where image classification tries to assign a
label to the entire image, semantic segmentation tries to
isolate the distinct entities and objects in a given image,
allowing for more fine-grained identification. Semantic
segmentation is a basic input technology for self-driving
cars (identifying and isolating objects on roads), image
analysis, medical applications, and more.
Cityscapes
Cityscapes is a large-scale dataset of diverse urban street
scenes across 50 different cities recorded during the
daytime over several months (during spring, summer, and
fall) of the year. The dataset contains 5,000 images with
high-quality, pixel-level annotations and 20,000 weekly
labeled ones. Semantic scene understanding, especially
in the urban space, is crucial to the environmental
perception of autonomous vehicles. Cityscapes is useful
for training deep neural networks to understand the urban
environment.
One Cityscapes task that focuses on semantic
segmentation is the pixel-level semantic labeling task.
This task requires an algorithm to predict the per-pixel
semantic labeling of the image, partitioning an image
into different categories, like cars, buses, people, trees,
and roads. Participants are evaluated based on the
intersection-over-union (IoU) metric. A higher IoU score
means a better segmentation accuracy. Between 2014 and
2020, the mean IoU increased by 35% (Figure 2.1.11). There
was a significant boost to progress in 2016 and 2017 when
people started using residual networks in these systems.
2.1 COMPUTER
VISION—IMAGE
CHAPTER 2:
TECHNICAL
PERFORMANCE
01/2015 01/2016 01/2017 01/2018 01/2019 01/2020
60%
65%
70%
75%
80%
85%
90%
Mean
Intersection-Over-Union
(mIoU)
85.1% With Extra Training Data
82.3% Without Extra Training Data
CITYSCAPES CHALLENGE: PIXEL-LEVEL SEMANTIC LABELING TASK
Source: Papers with Code, 2020 | Chart: 2021 AI Index Report
Figure 2.1.11
57. TABLE OF CONTENTS
57
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
EMBODIED VISION
The performance data so far shows that computer vision
systems have advanced tremendously in recent years.
Object recognition, semantic segmentation, and human
pose estimation, among others, have now achieved
significant levels of performance. Note that these visual
tasks are somewhat passive or disembodied. That
is, they can operate on images or videos taken from
camera systems that are not physically able to interact
with the surrounding environment. As a consequence
of the continuous improvement in those passive tasks,
researchers have now started to develop more advanced
AI systems that can be interactive or embodied—that
is, systems that can physically interact with and modify
the surrounding environment in which they operate: for
example, a robot that can visually survey a new building
and autonomously navigate it, or a robot that can learn
to assemble pieces by watching visual demonstrations
instead of being manually programmed for this.
Progress in this area is currently driven by the
development of sophisticated simulation environments,
where researchers can deploy robots in virtual spaces,
simulate what their cameras would see and capture, and
develop AI algorithms for navigation, object search, and
object grasping, among other interactive tasks. Because
of the relatively early nature of this field, there are few
standardized metrics to measure progress. Instead, here
are brief highlights of some of the available simulators,
their year of release, and any other significant feature.
•
Thor (AI2, 2017) focuses on sequential abstract
reasoning with predefined “magic” actions that are
applicable to objects.
• Gibson (Stanford, 2018) focuses on visual navigation
in photorealistic environments obtained with 3D
scanners.
• iGibson (Stanford, 2019) focuses on full interactivity
in large realistic scenes mapped from real houses and
made actable: navigation + manipulation (known in
robotics as “mobile manipulation”).
•
AI Habitat (Facebook, 2019) focuses on visual
navigation with an emphasis on much faster
execution, enabling more computationally expensive
approaches.
•
ThreeDWorld (MIT and Stanford, 2020) focuses on
photorealistic environments through game engines,
as well as adds simulation of flexible materials, fluids,
and sounds.
•
SEAN-EP (Yale, 2020) is a human-robot interaction
environment with simulated virtual humans that
enables the collection of remote demonstrations from
humans via a web browser.
•
Robosuite (Stanford and UT Austin, 2020) is a modular
simulation framework and benchmark for robot
learning.
2.1 COMPUTER
VISION—IMAGE
CHAPTER 2:
TECHNICAL
PERFORMANCE
58. TABLE OF CONTENTS
58
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
2016 2017 2018 2019 2020
0%
10%
20%
30%
40%
50%
Mean
Average
Precision
(mAP)
42.8%
ACTIVITYNET: TEMPORAL ACTION LOCALIZATION TASK
Source: ActivityNet, 2020 | Chart: 2021 AI Index Report
Video analysis is the task of making inferences over sequential image frames, sometimes with the inclusion of an audio feed.
Though many AI tasks rely on single-image inferences, a growing body of applications require computer vision machines to reason
about videos. For instance, identifying a specific dance move benefits from seeing a variety of frames connected in a temporal
sequence; the same is true of making inferences about an individual seen moving through a crowd, or a machine carrying out a
sequence of movements over time.
ACTIVITY RECOGNITION
The task of activity recognition is to identify various
activities from video clips. It has many important
everyday applications, including surveillance by video
cameras and autonomous navigation of robots. Research
on video understanding is still focused on short events,
such as videos that are a few seconds long. Longer-term
video understanding is slowly gaining traction.
ActivityNet
Introduced in 2015, ActivityNet is a large-scale video
benchmark for human-activity understanding. The
benchmark tests how well algorithms can label and
categorize human behaviors in videos. By improving
performance on tasks like ActivityNet, AI researchers are
developing systems that can categorize more complex
behaviors than those that can be contained in a single
image, like characterizing the behavior of pedestrians on
a self-driving car’s video feed or providing better labeling
of specific movements in sporting events.
ActivityNet: Temporal Action Localization Task
The temporal action localization task in the ActivityNet
challenge asks machines to detect time segments in a
600-hour, untrimmed video sequence that contains several
activities. Evaluation on this task focuses on (1) localization:
how well can the system localize the interval with the precise
start time and end time; and (2) recognition: how well can
the system recognize the activity and classify it into the
correct category (such as throwing, climbing, walking the
dog, etc.). Figure 2.2.1 shows that the highest mean average
precision of the temporal action localization task among
submissions has grown by 140% in the last five years.
2.2 COMPUTER VISION—VIDEO
2.2 COMPUTER
VISION—VIDEO
CHAPTER 2:
TECHNICAL
PERFORMANCE
Figure 2.2.1
59. TABLE OF CONTENTS
59
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
0% 2% 4% 6% 8% 10% 12% 14%
Mean Average Precision (mAP)
Drinking coffee
High jump
Polishing furniture
Putting in contact
lenses
Removing curlers
Rock-paper-scissors
Running a marathon
Shot put
Smoking a cigarette
Throwing darts
ACTIVITYNET: HARDEST ACTIVITIES, 2019-20
Source: ActivityNet, 2020 | Chart: 2021 AI Index Report
2019
2020
ActivityNet: Hardest Activity
Figure 2.2.2 shows the hardest activities of the temporal
action location task in 2020 and how their mean average
precision compares with the 2019 result. Drinking coffee
remained the hardest activity in 2020. Rock-paper-
scissors, though still the 10th hardest activity, saw the
greatest improvement among all activities, increasing by
129.2%—from 6.6% in 2019 to 15.22% in 2020.
2.2 COMPUTER
VISION—VIDEO
CHAPTER 2:
TECHNICAL
PERFORMANCE
Figure 2.2.2
60. TABLE OF CONTENTS
60
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
12/2016 04/2018 04/2020 11/2020
20
30
40
50
60
70
80
Mean
Average
Precision
(mAP50)
65.2
YOLOv2
(Resolution: Unclear)
YOLOv3
(Resolution: 608)
YOLOv4
(Resolution: 608)
PP-YOLO
(Resolution: 608)
YOU ONLY LOOK ONCE (YOLO): MEAN AVERAGE PRECISION
Source: Redmon Farhadi (2016 2018), Bochkovskiy et al. (2020), Long et al. (2020) | Chart: 2021 AI Index Report
OBJECT DETECTION
Object detection is the task of identifying a given object
in an image. Frequently, image classification and image
detection are coupled together in deployed systems.
One way to get a proxy measure for the improvement
in deployed object recognition systems is to study the
advancement of widely used object detection systems.
You Only Look Once (YOLO)
You Only Look Once (YOLO) is a widely used open source
system for object detection, so its progress has been
included on a standard task on YOLO variants to give a
sense of how research percolates into widely used open
source tools. YOLO has gone through multiple iterations
since it was first published in 2015. Over time, YOLO has
been optimized along two constraints: performance and
inference latency, as shown in Figure 2.2.3. What this
means, specifically, is that by measuring YOLO, one can
measure the advancement of systems that might not
have the best absolute performance but are designed
around real-world needs, like low-latency inference
over video streams. Therefore, YOLO systems might
not always contain the absolute best performance as
defined in the research literature, but they will represent
good performance when faced with trade-offs such as
inference time.
2.2 COMPUTER
VISION—VIDEO
CHAPTER 2:
TECHNICAL
PERFORMANCE
Figure 2.2.3
61. TABLE OF CONTENTS
61
CHAPTER 2 PREVIEW
Artificial Intelligence
Index Report 2021
2017 2018 2019 2020 2021 2022
0.002
0.005
0.01
0.02
0.05
0.1
0.2
0.5
1
False
Non-Match
rate
(FNMR;
Log
Scale)
0.0035, VISABORDER Photos FNMR@FMR ≤ 0.000001
0.0025, VISA Photos FNMR @ FMR ≤ 0.000001
0.0023, MUGSHOT Photos FNMR @ FMR ≤ 0.00001 DT=12 YRS
0.0022, MUGSHOT Photos FNMR @ FMR ≤ 0.00001
0.0293, WILD Photos FNMR @ FMR ≤ 0.00001
0.0064, BORDER Photos FNMR @ FMR = 0.000001
NIST FRVT 1:1 VERIFICATION ACCURACY by DATASET, 2017-20
Source: National Institute of Standards and Technology, 2020 | Chart: 2021 AI Index Report
FACE DETECTION AND
RECOGNITION
Facial detection and recognition is one of the use-cases
for AI that has a sizable commercial market and has
generated significant interest from governments and
militaries. Therefore, progress in this category gives
us a sense of the rate of advancement in economically
significant parts of AI development.
National Institute of Standards and Technology
(NIST) Face Recognition Vendor Test (FRVT)
The Face Recognition Vendor Tests (FRVT) by the
National Institute of Standards and Technology (NIST)
provide independent evaluations of commercially
available and prototype face recognition technologies.
FRVT measures the performance of automated face
recognition technologies used for a wide range of civil
and governmental tasks (primarily in law enforcement
and homeland security), including verification of visa
photos, mug shot images, and child abuse images.
Figure 2.2.4 shows the results of the top-performing 1:1
algorithms measured on false non-match rate (FNMR)
across several different datasets. FNMR refers to the
rate at which the algorithm fails when attempting to
match the image with the individual. Facial recognition
technologies on mug-shot-type and visa photos have
improved the most significantly in the past four years,
falling from error rates of close to 50% to a fraction of a
percent in 2020.4
2.2 COMPUTER
VISION—VIDEO
CHAPTER 2:
TECHNICAL
PERFORMANCE
Figure 2.2.4
4 You can view details and examples of various datasets on periodically updated FRVT 1:1 verification reports.