2. Economic Impact and Societal Considerations for Policy Decisions.

China AI Index：
China-US Comparison
Chenggang Xu
CKGSB
October 29 2019
HAI AI Index Workshop, Stanford

China AI Index (by CKGSB-Wuhan University)
Academia (Scopus) [US & China, 2018；US, UK, EU & China, 2019]
Journal publications Conference participation Citation counts AI sub-areas
Human Capital (LinkedIn；limited) [US & China, 2018]
Human capital in AI Human capital in AI academia
AI Industry [VentureXpert, US & China, 2019]
Startup companies in AI; All companies in AI related areas (keywords)
Development and use of open source package [US & China, 2018;+ UK, EU, 2019]
GitHub projects
Chinese National and Sub-nation Government Policies on AI [2019]
Policies on the paper; Implementations
Chinese Leading University AI Courses [Hand collected limited data; 2019]
Enrolments at different levels; Teaching hours at different levels
Trend of public opinions

Average Employment/Firm in AI Areas
（Tencent）
美国
Most Chinese AI firms are middle sized
Most US firms are small; or very large
US
China

Average Employment/Firm in AI Areas
（Tencent）
美国
Except chip-making all US AI firms are small
Almost all Chinese firms are middle sizedUS

AI Experts’ Work Experience: China vs. U.S.
LinkedIn, 2017
2.20%
（1100⼈）
11.50%
（5750⼈）
28.40%
（14200⼈）
19.20%
（9600⼈）
38.70%
（19350⼈）
0.60%
（5100⼈）
3.70%
（31450⼈）
11.20%
（95200⼈）
13%
（110500⼈）
71.50%
（607750⼈）
0-1年 2- 3年 4- 7年 8-10年 10年以上（不含 10年）
中国美国
US
China
Most US AI experts in US have more than 10 years
experience in AI
Most Chinese AI experts have less than 10 years
experience in AI

All AI Conference Participation:
China vs. US, 1995-2018
Figure3. Conference participation: China vs. US

ML Conference Participation:
China vs. US, 1995-2018

Computer Vision CVPR Conference
Participation: China vs. US, 1995-2018

AI Journal Publications: US, UK, China

No Citation AI Papers: US, UK, China

Total Journal Paper Citations: US, UK, China

Total Citations of Thousand-Citation Level Papers:
US, UK, China, 1995-2018

Most Chinese AI Projects are Applications based on
Open AI Platforms Developed by Google/Facebook
Chinese AI
using US (right)
US AI using US
(right)
0
5000
10000
15000
20000
25000
0
200
400
600
800
1000
1200
1400
1/1/2015
2/1/2015
3/1/2015
4/1/2015
5/1/2015
6/1/2015
7/1/2015
8/1/2015
9/1/2015
10/1/2015
11/1/2015
12/1/2015
1/1/2016
2/1/2016
3/1/2016
4/1/2016
5/1/2016
6/1/2016
7/1/2016
8/1/2016
9/1/2016
10/1/2016
11/1/2016
12/1/2016
1/1/2017
2/1/2017
3/1/2017
4/1/2017
5/1/2017
6/1/2017
7/1/2017
8/1/2017
9/1/2017
10/1/2017
11/1/2017
12/1/2017
1/1/2018
2/1/2018
3/1/2018
Chinese AI using Chinese
US AI using Chinese

Prasanna (Sonny) Tambe
Wharton School, U. Pennsylvania
October 30th, 2019
Stanford AI Index Roundtable Workshop
AI measurement: opportunities & challenges

Tambe
Why measure at all?
What should we be measuring?
Why is this challenging?
2

Tambe
Digital measurement problems have always been challenging.
You can see the computer age everywhere but in the productivity statistics.
… Robert Solow, 1987
More than thirty years later ...
Impact of IT by firm size?
Geography?
Entrepreneurship?
Innovation?
But in the AI era, the measurement of inputs and outputs has arguably become more
difficult - more service-based outputs and more intangible inputs (skills, software,
data, and culture)
3

Tambe
Why measure at all?
● Assignment of value is central to AI policy discussions.
How are the economic benefits of AI distributed?
● Environmental factors can affect who derives economic benefits.
Regulation (GDPR, Consumer taste for privacy, etc.)
● Effects of technological innovation may be difficult to anticipate.
Tableau, Azure studio, Tensorflow/Pytorch, AutoML
4

Tambe
What should we be measuring?
How much value is generated and who is capturing it, which requires measuring inputs
and outputs. Firms are still adjusting inputs to AI:
● Human capital
● Software
○ PyTorch, Mahout, AutoML ...
● Data
○ Quantity, quality, age, noise ...
● Hardware/compute
○ Cloud, VM, bare metal ...
● Management practices
○ Ethics councils, Reorganization of decision pathways ...
5

Tambe
1. Inputs - software, labor, data, computer cycles, management practices -- are rapidly
changing and often not transacted through external markets.
○ No payments to open source software, computation is often outsourced
○ Software tools (AutoML) have the potential to rapidly depreciate some human capital
1. AI is closer to decision-making, so environmental complements more variable:
○ Shifting constraints at region (GDPR), industry (Michigan laws against video-based hiring)
○ Changing consumer tastes for privacy, mounting concerns about bias and fairness
1. For outputs, the gap between investment and use is still large:
○ First order impact may be coming from new business models
○ “AI” vs. data-driven decision making?
○ Many firms outside of tech are in skunkworks mode
Why is this challenging? Both inputs and outputs are hard to measure
6

Tambe
Some inputs may matter more than others for explaining variability
How much value is generated and who is capturing it, which requires measuring inputs
and outputs. Firms are still adjusting inputs to AI:
● Human capital
● Software
○ PyTorch, Mahout, AutoML ...
● Data
○ Quantity, quality, age, noise ...
● Hardware/compute
○ Cloud, VM, bare metal ...
● Management practices
○ Ethics councils, Reorganization of decision pathways ...
7

Tambe
Competition:
AI-complementary assets in
firms of different sizes?**
- IT, AI, and Intangible Capital
- With Brynjolfsson, Rock, Hitt
- LinkedIn data
**Still a new topic for IT capital!
8

Tambe
Geography:
What will the geographic
distribution of derived
benefits look like for AI?
- AI literacy and the labor market
- BG data
9

Tambe
Education:
Which occupations will need
AI human capital?
10
- AI literacy and the labor market
- BG data

Tambe
Key points
Emerging policy questions require measuring inputs and outputs into AI-driven
processes. There are a variety of promising but incomplete markers of AI investment.
Firms are likely to vary considerably in their endowment of some of the hidden,
expensive, and hard to measure complements to AI.
This will probably show up first in a skewed distribution of AI returns but direct
measurement can offer a lot of insight.
Thanks! tambe@wharton.upenn.edu.
11

© 2019 Burning Glass Technologies—Proprietary and Confidential
Measuring the Diffusion of AI in the Labor
Market
Presentation for the AI Index Roundtable Workshop on Measurement
in AI Policy: Opportunities & Challenges
October 30, 2019
Bledi Taska, PhD
Chief Economist, Burning Glass Technologies

Presentation Summary
2
1. Artificial intelligence is transforming work and we need data to understand these
effects
2. Survey data suffer from significant lags and cost. Real-time labor market data can fill
some of these gaps
3. More researchers are using talent acquisition data as a proxy for technology adoption
4. Following the same approach, we show that the AI revolution in the labor market is
real
5. We find a large wage premium for AI but no effect on employment growth. There is
some evidence for distributional effects

Technology Is Transforming Work at an
Unprecedented Pace
3

We Need Data to Measure the Effects of AI
4

Survey Data Has Significant Lags and Cost
5

Using Talent Acquisition as a Proxy for
Technology Adoption
6
In a number of papers, researchers have emphasized that hiring decisions can be used to
understand technology adoption and diffusion. (e.g. Tambe 2014; Tambe and Hitt 2013).
P Tambe, LM Hitt, D Rock and E Brynjolfsson (2019), and D Rock (2019) use LinkedIn data
to understand the effects of investing in AI
D Acemoglu, D Autor, J Hazell, and P Restrepo (2019), A Goldfarb, B Taska and F
Teodoridis (2019), A Goldfarb, B Taska, F Teodoridis and C Tucker (2019), and LAlekseeva,
J Azar, M Gine and B Taska (2019) use Burning Glass data to understand the effects of AI
on labor markets, and adoption by different industries

Collecting Real-Time Labor Market Data
Visit Online
Job Sites
Collect &
Deduplicate
Job Postings
Read Postings to
Generate Detailed Data
• Job Title & Occupation
• Employer & Industry
• Technical Skills
• Foundational Skills
• Certifications
• Educational Requirements
• Experience Levels
• Salaries
Real-time job market data offer up-to-date insights not possible through
traditional sources

80%3.4 million
50,000 300 million
>1 billion
Dynamic Labor
Market Taxonomy
>1 million
Active unique jobs
collected daily
Sources across the web - job
boards and corporate sites
Historical job market
records
Firms represented, from
large corporations to SME’s
23 Career Areas
1700 Occupations
18,000 Skills
60,000 Skill Variants
Deduplication ensuring
integrity and consistency
Resumes processed
per annum
Burning Glass data have been built over 15 years with robust taxonomies and
the industry’s largest in-house data science team.
Real-Time Labor Market Data is Big Data

Measuring the Diffusion of AI in the Labor Market
9

The AI Revolution Is Real
10
Source: L Alekseeva, J Azar, M Gine and B Taska (2019)

We Find Evidence that AI Is a General-Purpose
Technology (GPT)
11
Source: A Goldfarb, B Taska and F Teodoridis (2019)

The AI Revolution Is Real: In Some Industries More than
Others
12

There Is an AI Adoption S-curve, and Larger Firms Are
Adopting AI First
13

There Is a Large Wage Premium for AI
14

We Find no Association Between Change in AI share and
Employment Growth
15

But There Are Distributional Effects: At-Risk Jobs Decline
16
Source: D Acemoglu, D Autor, J Hazell, and P Restrepo (2019)

Conclusions
17
1. The pace of change in jobs and skills is unprecedented
2. Traditional survey data cannot help us with understanding the current needs of the
labor market. Real-time labor market data can fill some of these gaps
3. AI is here to stay. The diffusion of AI in the labor market is immense and has real
effects on wages
4. Some initial results on distributional effects, but more research is needed
5. Workforce development strategies and education policies could benefit significantly
from the use of real-time labor market information on technology adoption
6. We need the research community and policymakers to push the boundaries of
understanding and utilizing big data even further

HAI Stanford: Workshop on Measurement in AI Policy
Macro-economic impact of AI
Ekkehard Ernst
30 October 2019
@ekkehardernst
ernste@ilo.org

Wg Progress in AI: Number of patentsg
Note: USPTO: United States Patent and Trademark Oce; SIPO: State Intellectual Property Oce of The People's Republic of China;
JPO: Japan Patent Oce; PCT: Patent Cooperation Treaty; EPO: European Patent Oce.
Source: Fujii and Managi, 2018

Wg Patents focus increasingly on automation
Note: Number of automation patents in the US, 1976-2014.
Source: Mann and Püttmann, 2017

Wg High impact of AI in developing countries
Dependent variable: World Developed Developing
Employment (in logs) countries countries
robot stock −0.055∗∗ −0.044∗∗ −0.029∗∗∗ −0.034∗∗∗ −0.343∗∗∗ -0.329
(0.028) (0.018) (0.009) (0.009) (0.112) (0.480)
robot stock × labour intensity -0.023 0.012 -0.011
(0.044) (0.019) (0.411)
labour intensity 0.007 0.015 0.002 0.001 -0.016 -0.010
(0.008) (0.015) (0.005) (0.005) (0.021) (0.217)
N 477 477 360 360 117 117
R2 0.84 0.85 0.80 0.80 0.35 0.38
Note: Regressions include country and industry xed eects. Trends are the coecients of regressions on a
linear trend. Robust standard error in parentheses. Signicance levels: ∗, ∗∗, ∗ ∗ ∗ indicate signicance at
0.10, 0.05 and 0.01. Controls: value added, wage. Estimates are weighted by sectoral employment in 2005.
Estimated equation:
Nij = β0 + β1robotsij + β2robotsij × li2005 + β3li2005 + β4VAij + β5Wij + uij .
Source: Carbonero, F.; Ernst, E.; Weber, E. 2019. Robots worldwide: The impact of automation on employment
and trade, Research Department Working Paper, no. 36 (Geneva, ILO).

Wg But AI is a General Purpose Technology!

Wg Transformation vs disruption of jobs (US)
1 Retail Salespersons
2 Cashiers
3 Oce Clerks
4
Food Preparation and
Serving Workers
5 Nurses
6 Waiters and Waitresses
7 Customer Service Representatives
8 Janitors and cleaners
9
Laborers and Freight, Stock,
and Material Movers
10
Secretaries and Adminis-
trative Assistants
Source: Fossen, F.; Sorgner, A. 2019. Mapping the Future of Occupations: Transformative and Destructive Eects
of New Digital Technologies on Jobs, Foresight and STI Governance, Vol. 13, No. 2, pp. 10-18

Wg Transformation vs disruption of jobs (Thailand)
1 Field crop and vegetable growers
2 Tree and shrub crop growers
3 Shopkeepers
4 Shop sales assistants
5 Crop farm labourers
6 Livestock and dairy producers
7 Car, taxi and van drivers
8 Stall and market salespersons
9 Cooks
10 Food service counter attendants
Source: ILO, Forthcoming, based on Fossen, F.; Sorgner, A. 2019. Mapping the Future of Occupations: Transfor-
mative and Destructive Eects of New Digital Technologies on Jobs, Foresight and STI Governance, Vol. 13, No.
2, pp. 10-18

Wg AI increases inequality, not unemployment
Computer storage cost fell dramatically...
...but so did job destruction...
But inequality
increased!
80
90
100
110
120 Job destruction
Income inequality
(Index, 2000=100)
-2
-1
0
1
2
3
4
5
6Storage costs: US$ per GB
(logarithmic scale)
1984 1988 1992 1996 2000 2004 2008 2012

Wg Job polarization across the globe g
Change in
employment
shares

(in pp)
5
0
5
10
High income
countries
Low income
countries
Lowermiddle income
countries
Uppermiddle income
countries
High
skill
Medium
skill
Low
skill
High
skill
Medium
skill
Low
skill
High
skill
Medium
skill
Low
skill
High
skill
Medium
skill
Low
skill
200013 201321

Wg The winner takes it all with AI!g

Wg Measuring AI: What is it that we don't know?
Smart cities
AI improves urban planning and
management
...might bring more spatial inequality
=⇒ Smart city index
Resource eciency
AI is highly enrgy intensive...
...but could help save resources
=⇒ AI energy eciency index
Sectoral productivity gains
AI has widely dierent application across
industries
=⇒ AI production potential index
Blockchain transforms capitalism
Legal infrastructure is evolving
...and encircling additional forms of
commons
=⇒ Smart contracts/Tokenization index

Wg Want to know more?g
Thank you!

Artiﬁcial Intelligence, Human Capital, and
Innovation
Michael Gofman (Michael.Gofman@simon.rochester.edu)
Zhao Jin (Zhao.Jin@simon.rochester.edu)
Simon Business School, University of Rochester
October 25, 2019

AI Brain Drain
AI professors are
a scarce resource and
highly specialized human capital
The top tech ﬁrms have hired many AI professors
Many AI professors also established their own startups
How does this AI Brain Drain affect AI knowledge transfer to
students?

Public Concerns of AI Brain Drain

The Unprecedented AI Brain Drain (1)
We document an unprecedented Brain Drain of 221 AI professors
from North American universities into the industry
180 AI faculty completely or partially started to work at another
company
41 professors founded startups
AI faculty are identified based on AI conferences and LinkedIn
profiles
The departures of AI faculty are based on LinkedIn profiles

010203040
NumberofDepartures
05101520
Percent
2004 2006 2008 2010 2012 2014 2016 2018
Year
Citation Proportion of Departures Number of Departures
AI Faculty Departures
AI Brain Drain Index

1 4
0 4
1 4
44
2 5
4 5
4 6
4 8
4 9
3 10
9 11
7 11
12 16
0 5 10 15
Harvard University
Purdue University
University of Illinois at Urbana-Champaign (UIUC)
University of Texas at Austin
University of California, San Diego
University of Southern California
University of Michigan
Stanford University
University of Toronto
Georgia Institute of Technology
University of California, Berkeley
University of Washington
Carnegie Mellon University
Professor Departures from 2004-2018
With University Afﬁliation Without University Afﬁliation

AI Entrepreneurs
AI Entrepreneurs’ data are from CrunchBase
AI knowledge is a critical determinant of AI startup formation
26% of AI startups have at least one PhD founder
10% for other IT startups
N(AI) N(IT) Mean(AI) Mean(IT) Diﬀ. SE
Single Founder 363 985 0.14 0.22 -0.07*** 0.024
Local Founder 363 985 0.17 0.19 -0.02 0.024
Bachelor 363 985 0.28 0.45 -0.17*** 0.030
Master (Non-MBA) 363 985 0.29 0.21 0.08*** 0.026
MBA 363 985 0.05 0.09 -0.04** 0.017
PhD 363 985 0.26 0.10 0.16*** 0.021
Found Lag 363 985 2.24 1.89 0.35*** 0.114
Observations 1348

The Causal Effect of AI Brain Drain on Indirect
Innovations
AI Brain Drain reduces the number of future AI entrepreneurs
who graduate in four to six years after AI faculty departure
It also reduces the early-stage funding of AI startups by affected
founders
The negative impact is most pronounced
for the top 10 universities
for entrepreneurs with PhD degrees
and for startups in the ﬁeld of deep learning

The data for our AI Brain Drain index can be downloaded at:
AIBrainDrain.org
We are looking for partners to expand our AI Brain Drain Index
Globally and study the consequences of this Brain Drain on:
Economic development
Innovations
Employment
Inequality

Appendix: The Unprecedented Brain Drain
0510152025
NumberofDepartures
2004 2006 2008 2010 2012 2014 2016 2018
Year
With Univeristy Affiliation Without Univeristy Affiliation
Departures by University Affiliation

Appendix: Professors Poached by Large Firms
1
1
1
1
1
1
1
2
2
2
3
4
4
7
7
7
12
17
23
0 5 10 15 20 25
Number of Professors
Adobe
American Express
Goldman Sachs
Illumina
JP Morgan
Lockheed Martin
Morgan Stanley
Netﬂix
Samsung
Toyota
Intel
Apple
IBM
Facebook
NVIDIA
Uber
Microsoft
Amazon
Google/DeepMind
Professors Poached by Public Firms from 2004-2018

Appendix: Extensive Margin
Dependent Variable ln(1+AI Entrepreneur) AI Entrepreneur
OLS OLS Poisson Poisson
(1) (2) (3) (4)
Untenured Leave[t−3,t−1] -0.705 -0.714 -1.049 -0.749
(0.588) (0.589) (2.438) (2.622)
Untenured Leave[t−6,t−4] -0.073 -0.070 1.062 1.129
(0.109) (0.111) (1.045) (1.181)
Tenured Leave[t−3,t−1] -0.571* -2.067
(0.334) (1.358)
Tenured Leave[t−6,t−4] -0.863** -2.480**
(0.360) (1.187)
Tenured Leave with Aﬃliation[t−3,t−1] -0.634 -1.703
(0.405) (1.262)
Tenured Leave with Aﬃliation[t−6,t−4] -0.738** -2.372***
(0.284) (0.840)
Rank -0.282 -0.299 -0.757 -0.803
(0.300) (0.300) (0.529) (0.534)
Year FE Yes Yes Yes Yes
University FE Yes Yes Yes Yes
N 621 621 621 621
R2 0.508 0.505
Pseudo R2 0.391 0.389

Appendix: Extensive Margin - Instrumental
Variable Approach
Dependent Variable ln(1+AI Entrepreneur) High Tenured Leave[t−6,t−4]
OLS 2SLS First Stage
(1) (2) (3)
High Tenured Leave[t−6,t−4] -0.183**
(0.085)
High Tenured Leave[t−6,t−4] -0.823***
(0.309)
Average Citation[t−6,t−4] 0.001***
(0.000)
Rank -0.280 -0.099 0.219
(0.304) (0.321) (0.150)
Year FE Yes Yes Yes
University FE Yes No Yes
N 621 621 621
R2 0.506 0.433 0.479

Appendix: Intensive Margin - Financing (First
Round)
Dependent Variable: Funding($MM) High Tenured Leave[t−6,t−4]
OLS OLS 2SLS First Stage
(1) (2) (3) (4)
Tenured Leave[t−6,t−4] -12.120***
(2.057)
(0.412)
(0.977)
(0.000)
Rank -7.067*** -6.618*** -4.662** 0.596***
(1.797) (1.762) (2.131) (0.081)
Number of Founders 1.403 1.456 1.619** 0.105
(0.918) (0.903) (0.662) (0.090)
Prior Exit 0.105 0.040 -0.039 -0.050
(1.067) (1.030) (0.790) (0.040)
Firm Age at Financing 1.299*** 1.304** 1.261*** 0.001
(0.449) (0.465) (0.356) (0.015)
Investor Count 0.248* 0.252* 0.267*** 0.013***
(0.128) (0.124) (0.097) (0.004)
Financing Year FE Yes Yes Yes Yes
Founding Location FE Yes Yes Yes Yes
N 199 199 199 199
R2 0.426 0.424 0.419 0.749

Appendix: Intensive Margin - Financing (Series-A
Round)
Dependent Variable: Funding($MM) High Tenured Leave[t−6,t−4]
OLS OLS 2SLS First Stage
(1) (2) (3) (4)
Tenured Leave[t−6,t−4] -63.086***
(15.075)
High Tenured Leave[t−6,t−4] -8.101**
(3.475)
(1.448)
(0.000)
Rank -3.774 -2.380 -4.360 0.521***
(10.118) (10.231) (5.446) (0.107)
Number of Founders 5.002* 5.721** 5.568*** 0.017
(2.307) (2.466) (1.756) (0.050)
Prior Exit -2.294* -2.547** -3.193*** 0.172***
(1.075) (0.973) (0.785) (0.038)
Firm Age at Financing 0.645 0.602 0.763*** -0.024*
(0.386) (0.489) (0.206) (0.011)
Investor Count 0.568*** 0.579*** 0.541*** 0.006
(0.177) (0.109) (0.121) (0.009)
Financing Year FE Yes Yes Yes Yes
Founding Location FE Yes Yes Yes Yes
N 113 113 113 113
R2 0.523 0.514 0.508 0.781

AI FOR FINANCIAL STABILITY
LARGEST PRIVATE AI LAB IN THE NORDICS
Peter Sarlin
S t a n f o r d , H A I - A I I n d e x W o r k s h o p
O c t o b e r , 2 0 1 9

For a world with safe human-centric AI that
frees the human mind for meaningful work.
PURPOSE:
A I F O R P E O P L E
F U L L A I S O L U T I O N C A P A B I L I T Y
I N M L , N L P C V
W I T H 5 0 + E X P E R T S 3 0 + P H D S
@ P E T E R S A R L I N | 1 0 / 2 0 1 9

@ P E T E R S A R L I N | 1 0 / 2 0 1 9
AI CHANGES FINANCE FUNDAMENTALLY
IT’S IMPACTING EVERY BUSINESS PROCESS

@ P E T E R S A R L I N | 1 0 / 2 0 1 9
MEET THE SHERLOCKS
AI CHANGES FINANCE FUNDAMENTALLY
IT’S IMPACTING EVERY BUSINESS PROCESS

G L O B A L R E S E A R C H P R O J E C T :
D i g g i n g I n t o H i g h - F r e q u e n c y D a t a
PROJECT INVESTIGATORS:
• EUROFIDAI
• UC Berkeley
• University of Massachusetts
• Goethe University Frankfurt
• London School of Economics
• Hanken School of Economics
HIGH-FREQUENCY TRADING:
• Speed of trading altered equity markets
• Market makers mostly replaced by HFTs
• Assure serving the real economy society
HIGH-FREQUENCY TRADING:
• Set up a transatlantic database
• Models of risk, contagion crises
• Network of EU US research policy

@ P E T E R S A R L I N | 1 0 / 2 0 1 9
CAN WE PREDICT FINANCIAL CRISES?
Predicting shocks vs. identifying vulnerability
Crises triggered by various shocks...
...but widespread imbalances build-up ex ante
B L A C K S W A N
( T A L E B 2 0 0 1 )
D R A G O N K I N G
( S O R N E T T E 2 0 0 9 )

I N D U S T R Y C O - O P E R A T I O N
A C A D E M I C C O - O P E R A T I O N
@ P E T E R S A R L I N | 1 0 / 2 0 1 9

@ P E T E R S A R L I N | 1 0 / 2 0 1 9
RISK QUANT
TOOL FOR MEASURING
INTERCONNECTED RISK
ANALYST
TOOL THAT UNDERSTANDS
NEWS LIKE A HUMAN ANALYST
SILOBRAIN
SOLUTIONS FOR ML WITH A
HUMAN IN THE LOOP
T H R E E A I S O L U T I O N S

2-CLASS CLASSIFICATION:
V U L N E R A B L E V S . T R A N Q U I L
DATA
• Historical distress events
• ~1000 risk indicators
• Bank interlinkages
CASE :
IDENTIFY VULNERABLE BANKS
S O L U T I O N :
R I S K Q U A N T
Lang JH, Peltonen TA, Sarlin P, 2018. A framework for early-warning modeling with an application to banks.. ECB WP series
Holopainen H, Sarlin P., 2017 Toward robust early-warning models: A horse race, ensembles and model uncertainty. QF.
Betz F., Oprică S., Peltonen T., Sarlin P, 2014. Predicting Distress in European Banks. Journal of Banking Finance.

ECB BANK ASSESSMENT
EXERCISE IN 2014
20 MOST VULNERABLE
BANKS IN 2014 Q2
M O D E L V S . 7 0 0 0
P E R S O N - Y E A R S
@ P E T E R S A R L I N | 1 0 / 2 0 1 9

S H E R L O C K S . A I –
T H E A R T I F I C I A L I N T E L L I G E N C E C O M P

T O O L F O R M E A S U R I N G I N T E R C O N N E C T E D R I S K
S O L U T I O N :
R I S K Q U A N T
@ P E T E R S A R L I N | 1 0 / 2 0 1 9

S O , W H A T A B O U T N E W S ?
@ P E T E R S A R L I N | 1 0 / 2 0 1 9

1. CAN WE DETECT DISTRESS BASED ON TEXT?
2. CAN WE OBTAIN DESCRIPTIONS FROM TEXT?
3. CAN WE IMPROVE NUMERICAL PREDICTIONS?
Step 1:
Learning semantic vectors,
through word prediction
Step 2:
Learning to predict events, with
articles' semantic vectors
Step 3:
Extend semantic vectors with
numerical risk indicators
D E E P L E A R N I N G B A N K
S T R E S S I N N E W S
@ P E T E R S A R L I N | 1 0 / 2 0 1 9
Rönnqvist S, Sarlin P, 2017. Bank distress in the news: Describing events through deep learning. Neurocomputing.
Cerchiello P, Nicola G, Rönnqvist S, Sarlin P, 2017. Deep Learning Bank Distress from News and Numerical Financial Data. DEM WP No. 140.
Rönnqvist S, Sarlin P, 2015. Bank Networks from Text: Interrelations, Centrality and Determinants. Quantitative Finance.

P E T E R @ S I L O . A I | 2 / 2 0 1 8

C O U N T R Y - L E V E L S T R E S S R E P O R T I N G : G E R M A N Y

S O L U T I O N :
A N A L Y S T
@ P E T E R S A R L I N | 1 0 / 2 0 1 9

I N F R A S T R U C T U R E T O S P E E D U P A I D E V E L O P M E N T
A I L I B R A R Y
SPEEDUP DEVELOPMENT OF
AI SOLUTIONS WITH AI MODULES
A I W O R K S P A C E
SCALE MACHINE LEARNING MODELS
AND ON-DEMAND DATA ACCESS
A I D E P L O Y M E N T
SPEED-UP DEPLOYMENT OF
AI SOLUTIONS TO PRODUCTION
A T O O L K I T R E A D Y F O R P R O D U C T I O N
Converts data into ML-ready form, speeds-up implementation of
production-ready models, and generates APIs.
NLP / TEXT
PROCESSING
MISSING VALUE
IMPUTATION
FEATURE
EXTRACTION
DOCKER
IMAGES
DATA PRE-
PROCESSING
MACHINE
LEARNING
COMPUTER
VISION
AI CODE
PACKAGES
PRE-TRAINED
MODELS
ONE-CLICK
PROJECT
SETUP
CLOUD
RESOURCES
LOGGING
VERSIONING
SCALABLE ML
WORKSPACE
ACTIVE DATA
MANAGEMENT
API OTHER
INTERFACES
HOST DEPLOYED
MODELS
ML TEACHING
INTERFACES
MAINTENANCE
DASHBOARDS
ONE-CLICK
DEPLOYMENT
@ P E T E R S A R L I N | 1 0 / 2 0 1 9

M L W I T H A H U M A N I N T H E L O O P
C AS E S T U D IES :
I M A G E A I
N U M E R A T I A I
D O C U M E N T A I
@ P E T E R S A R L I N | 1 0 / 2 0 1 9

M L W I T H A H U M A N I N T H E L O O P
C AS E S T U D IES :
@ P E T E R S A R L I N | 9 / 2 0 1 8

For a world with safe human-centric AI that
frees the human mind for meaningful work.
A I F O R P E O P L E
THANKS!
PETER SARLIN
CEO CO -FOUNDER
PETER@SILO.AI
@PETERSARLIN
@SILO_AI
@ P E T E R S A R L I N | 1 0 / 2 0 1 9

Measuring the Economic
Effects of Artificial Intelligence
Daniel Rock (MIT)
@danielrock
HAI – AI Index Roundtable Workshop
10/30/19

Lots of shifting market equilibria
2

Most of the talent-related AI data is all “Up and to the
Right”
3
Source: Google Trends

Deep learning libraries (PyTorch missing…)
4Source: Jeff Dean (Talk at MIT 2018)

LinkedIn Skills data can be useful for this kind of
benchmarking
5Rock (2019)

Where is AI/ML Talent?
6Source: Tambe, Hitt, Rock, and Brynjolfsson (2019)

These measurements are supply-side. What about
demand?
7Brynjolfsson, Frank, Mitchell, Rahwan, Rock (2019)

Investment equilibrium necessarily changes too
8Source: Hall (2001)

The Prediction Question: Is AI Intangible Capital
Priced?
9Rock (2019)

The Productivity –Curve
10Brynjolfsson, Rock, and Syverson (2019)

(1) (2) (3) (4) (5) (6)
Market Value (MM USD) AI Cluster +Data Science +Cloud Computing +Data Storage +Digital Literacy +Bus.Mgmt and Advertising
Log(Total Assets) 7,790*** 7,798*** 7,798*** 7,790*** 7,837*** 7,832***
(1,276) (1,277) (1,277) (1,275) (1,278) (1,275)
Log(Edu. Years) -1,582 -1,577 -1,578 -1,562 -1,535 -1,528
(1,014) (1,006) (1,009) (1,007) (997.5) (999.5)
Log(AI Index) -1,630 -1,616 -1,618 -1,610 -1,601 -1,592
(1,017) (1,022) (1,006) (1,006) (1,006) (1,002)
Log(AI Index x Post TF) 3,299*** 3,303*** 3,302*** 3,305*** 3,312*** 3,316***
(712.3) (712.0) (715.4) (715.7) (716.2) (719.5)
Log(Data Science Index) -133.0 -136.8 -104.9 187.4 176.2
(574.9) (559.0) (558.1) (446.5) (566.1)
Log(Cloud Computing Index) 16.68 46.40 80.63 81.93
(397.7) (398.8) (389.8) (419.9)
Log(Data Storage Technology
Index)
-29,129 -26,872 -27,442
(21,191) (21,370) (21,357)
Log(Digital Literacy Index) -639.1 -672.9
(595.3) (768.3)
Log(Bus. Management Index) 290.6
(1,444)
Log(Advertising Index) -235.1
(422.4)
Observations 6,440 6,440 6,440 6,440 6,440 6,440
Firm and Year FE Yes Yes Yes Yes Yes Yes
AI-intensive companies are repriced in 2016
11

Deep Learning per Skill Valuation Changes Over Time
12

Is the change contemporaneous with the TensorFlow
Launch?
13Rock (2019)

(1) (2) (3) (4) (5) (6)
Market Value (MM USD) AI Cluster +Data Science +Cloud Computing +Data Storage +Digital Literacy +Bus.Mgmt and Advertising
Log(Total Assets) 5,366*** 5,356*** 5,360*** 5,353*** 5,354*** 5,370***
(901.2) (900.5) (901.8) (900.6) (900.4) (902.9)
Log(Edu. Years) -740.1 -745.7 -763.0 -751.7 -751.5 -769.0
(555.1) (549.0) (546.8) (544.2) (540.6) (533.9)
Log(AI Index) 656.7 640.8 608.3 614.3 614.3 603.1
(643.3) (649.4) (647.9) (649.0) (648.4) (650.8)
Log(AI Index x Post TF) 980.0*** 974.3*** 958.5*** 958.4*** 958.5*** 944.9***
(290.8) (288.5) (289.6) (289.5) (289.7) (287.8)
Log(Data Science Index) 167.3 93.18 116.1 117.8 207.6
(465.3) (442.7) (440.6) (307.0) (473.0)
Log(Cloud Computing Index) 332.2 354.6 354.8 378.7
(284.7) (285.9) (275.4) (309.4)
Log(Data Storage Technology
Index)
-21,531 -21,518 -20,920
(13,919) (14,125) (14,272)
Log(Digital Literacy Index) -3.713 127.4
(470.2) (693.5)
Log(Bus. Management Index) -799.1
(1,311)
Log(Advertising Index) 352.5
(356.3)
Observations 5,864 5,864 5,864 5,864 5,864 5,864
Firm and Year FE Yes Yes Yes Yes Yes Yes
Removing the top quintile of AI-Using firms
14

Data Science per Skill Valuation Changes Over Time
15Rock (2019)

Linear Regression per Skill Valuation Changes Over Time
16Rock (2019)

Synthetic Difference-in-Differences Coefficients are
similar*
17
Synthetic Difference-in-Differences Results
*Arkhangelsky, Dmitry, Susan Athey, David A. Hirshberg, Guido W. Imbens, and Stefan Wager. Synthetic difference in
differences. No. w25532. National Bureau of Economic Research, 2019.

Data Challenges and Opportunities Abound..
• Sources some have tried
• Labor market data (job postings, resume data, etc.)
• Usually corporate and not representative à need to normalize
• Research and development data (papers, patents)
• Text is where the best indexing opportunities are, but this is also challenging. Can face sampling
issues.
• Progress on various benchmarks (ImageNet, GLUE, etc.)
• Performance to the metric
• Economics to Bear in Mind
• High fixed costs, low marginal costs of AI/IT mean that counting might not pick up value
• Current AI adopters are a special group
• Very early stages mean intangible investment critical
• Most measurement is supply-side so far
18

1
Michael Page
Research Fellow
michael.page@georgetown.edu
A short introduction to CSET and US
government spending on AI

2
About CSET
Goals
● Study the security impacts of emerging technologies; initial focus on
AI as test case
● Deliver nonpartisan analysis to the policy community
● Support academic work in security and technology studies
● Prepare a generation of policymakers, analysts, and diplomats to
wrestle with future technology dilemmas
Funding
● Initial $60M in 5-year grant-funding from three foundations
● Not accepting government or corporate funding

Competitiveness, state of play, and forecasting
● Overall
○ Metrics dashboard
○ Chinese tech transfer measures for AI
○ Prospects for cooperation with allies
● Talent
○ Immigration policy US AI
○ Retention of foreign AI students
○ Comparative immigration policies
○ Chinese mil-civ fusion: talent development
○ Semiconductor workforce
Applications and implications
○ Automation of cyber operations
○ AI’s effects on military power
○ Autonomous weapons and U.S. interests
○ Security implications of antitrust concerns
○ AI safety national security
● Investment
○ Chinese AI investments
○ Chinese public funding for AI
● Hardware
○ AI hardware trade policy
○ Chinese AI Chips: ASICs FPGAs
● Data
○ What it would mean to have a “data advantage”
● Algorithms/models
○ Brain-inspired AI research in China
Active Lines of Research
3

Foreign and domestic ST data sources
3. Workforce / Talent
○ Job Postings (English, Chinese)
○ CVs and Resume Data
○ FOIA Visa / Immigration Data / Port
of Entry Data (English)
● Analyst-directed data sources
○ Targeted Surveys
○ Human Annotated Data
○ Prioritized Translations
○ Intent / Policy Docs, etc.
4
1. Technical Text
○ Scholarly Literature (English,
Chinese, Russian)
○ Dissertations Theses (Chinese,
English, etc.)
○ Technical News (Chinese, English, +)
○ Patents (Worldwide)
2. Worldwide Funding
○ Grant Funding
○ Financial Transactions for Publicly
and Privately-held Corporations
○ Venture Financial Transactions
○ Spending by governments

US Government spending: Data sources (examples)
Transactions
• USA Spending. Data on federal grants, contracts, loans, and
other financial assistance from FY2008 to present.
– Federal Procurement Data System (FPDS). Federal primary
contracts. Reported by agencies.
– FFATA Sub-award Reporting System (FSRS). Federal
subcontracts. Reported by prime contractors.
Solicitations
• Federal Business Opportunities (FBO). Federal contract
solicitations.
Budgets
• Individual agency budget requests
• Defense Technical Information Center (DTIC). DoD budget
justifications for RDTE and procurement.
5

Some ﬁgures
6
Source: BGOV RDTE Dashboard
$850 million to civilian agencies under American AI
Initiative. Primary recipients: Energy, NIH, NIST, NSF
DoD budgeted over
$4bn to AI RDTE

Challenges Questions
• Data quality. Data in USA Spending is often missing or mislabeled.
• What’s AI? Some evidence DoD has relabeled existing programs as
“AI programs.”
• Linking contracts and solicitations. Contracts often contain little
information about the substance of the contract. Solicitations contain
significant descriptive information, but linking the two isn’t always
possible.
• International comparisons. Governments spend in different ways;
US-China spending comparisons might be misleading.
• What is the federal government’s role in AI funding? Given the
research community’s private-sector support and international
character, the role of the federal government might be evolving.
7

Where to Now?
Disruption, specialisation and diversity in AI RD
Juan Mateos-Garcia, Joel Klinger and Konstantinos Stathoulopoulos
HAI-AI Index Workshop on Measurement in AI Policy
Stanford University, 29th October 2019

Introduction Analysis Conclusions
The Economics of Diversity
● Technological diversity is valuable when there is uncertainty about the future
● The market might undersupply diversity if researchers opt for active
technologies with short term benefits
● Diversity in researcher capabilities beliefs could help preserve technological
diversity.
Also see Weitzman (1992, 1993)

It has happened before...

Will it happen again?
Some concerns about the dominant (deep learning driven) research trajectory
● About the methods (Marcus, 2019)
● About the ends (Russell, 2019)
● About the workforce (AI Now Institute, 2019)
● About the impacts (Acemoglu Restrepo, 2019)
Perception of competition between countries advancing their preferred variety of AI (Lee,
2018)
We need indicators that capture the composition of AI RD. This will give us more
detailed and policy relevant measures of disruption, diversity and specialisation.

Data Methods
● Data matched and enriched with researcher affiliation, location and gender
● AI papers identified using a keyword expanded approach
● Topic modelling to extract themes from the AI corpus
arXiv

The hockey stick

The paradigm shift?

A dominant design

From the classics...

...to the moderns...

... dual use technologies

● Publish emerging findings, data and code about thematic diversity in AI research
its drivers including...
■ Researcher affiliation (corporate / academia)
■ Gender diversity
■ International differences...
● Use full text in topic modelling
● Incorporate other data sources (patents, code, businesses)
● Consider economic and technological impacts
● Move from descriptive analyses to causal (and predictive?)
Use the results to inform AI policies!!

nesta.org.uk @nesta_uk
juan.mateos-garcia@nesta.org.uk
@JMateosGarcia

PwC Emerging Technology
October 2019
National AI
Strategy Radar
1

Emerging Technology
A number of major economic powers are launching long-term
AI strategies and initiatives
As AI continues to expand into other areas of our everyday lives as citizens, we
expect this trend to continue.
2
Source: Tim Dutton, July 2018

Emerging Technology
We have been supporting government entities across the globe
in the development of National AI strategies
3

Emerging Technology 4
Our National AI Strategy Radar (NAIS) uses Natural Language
Processing (NLP) techniques and automated data collection to
track the emerging priorities, investments, and concerns by
territory across 12 dimensions:
● Academic Partnership
● AI Governance
● AI RD
● Beneficial and Ethical AI
● Business Protection and Innovation
● Consumer Protection and Fairness
● Healthcare
● National Pre-eminence
● National Security
● Reskilling
● Safety
● Specialised AI tech
PwC’s National AI Strategy Radar can be used to understand
government concerns and initiatives on specific topics

PwC
For more information,
please contact:
5
Ilana Golbin
Director, Responsible AI Lead
PwC US Advisory
ilana.a.golbin@pwc.com
Anand Rao
Global AI Lead
PwC US Advisory
anand.s.rao@pwc.com

Corporate and Central Bank
Perceptions of AI
Evan A. Schnidman Ph.D.

● All primary source corporate
central bank communications
● 3000+ US companies
● 15 Central Banks
● Alerts, portal, API, native
mobile app, analytics
About Prattle
2

Proprietary NLP Data Science Tools
● Advanced linguistic mapping links
content directly to market response
● Delivery via API, user portal, mobile app
for both quant and fundamental investors
● Alert based system integrates directly
with existing research workflow
Automating Financial Research

AI Mentions by each Central Bank

Bloomberg: “machine learning is coming to every firm soon enough”*
The March of Machine Learning
*Quotes, survey, and
figures from The
implications of machine
learning in finance by
Bloomberg Professional
Services
How far along is your firm at incorporating Machine Learning into your investment strategy?*
Actively in Use: 18%
Researching: 24%
Haven’t Considered: 32%
Interested: 26%

Total AI Mentions on Earnings Calls

AI Mentions by Sector (2018-2019)

AI Mentions by Sector 2003-2019

AI Mentions by Sector as % (2003-2019)

Evan A. Schnidman Ph.D.
617-304-6807
evan@prattle.co

© 2019 Proprietary and confidential.
Measurement in AI Policy: Opportunities and Challenges
Stanford, CA
Presented by Daria Mehra with materials created by Julie Kim Emily Ralston.
2019

AGENDA
§ Introductions
§ The “How” Behind Quid
§ Case Study: AI Investment Landscape

Quid inspires full-picture thinking
by drawing connections
across massive amounts of written content.

The Technology Behind the Platform
Integrated
Custom
Open Source Data
Client Data
Output: Network
Map Derived AI
Insights
Integrated:
• Global News Blogs
(1.8 million sources,*
provided by
LexisNexis)
• Organizations (2.2
million,** provided by
Capital IQ and
Crunchbase)
• Global Patents (global
coverage of past 60
years,** provided by
Thomson Innovation)
Custom:
• Open Datasets
• Client Data
• Topic Detection/Modeling
• Document Clustering
• Entity Extraction
• Story Lines
• Anomaly Detection
• Sentiment Analysis
• Cross-Dataset Analysis
• Event Detection
• Data Summarization
• Machine Learning
PlatformData
2010 -2019
*Indexed every 15 minutes
**Indexed weekly

Methodology: Human Machine
Quid Machine
Global News Blogs
Organizations
Global Patents
Custom
Open Source Data
Client Data
HUMAN
Human Researcher HITL Curation
Peer Review
1. Determines relevant terms
for the question (e.g.
“Artificial Intelligence,”
“growth,” “funding”)
2. Arranges terms into a
Boolean search, thereby
translating human intent
into machine language
3. Determines search
parameters (timeframe,
geography, publication
type, etc.)
4. Iterates on the inputs to
match intent
1. Applies the Boolean
search to a corpus of
written documents and de-
duplicates results
2. Groups together
documents with semantic
similarity (Louvain
clustering)
3. Extracts representative
documents for each cluster
and chooses cluster names
(noun chunk methodology)
1. Researcher works in a
responsive UI to gain insights
from the data
2. Human corrects AI’s
gaffes
3. Researcher shares Quid
analysis with peers for
review and iteration on
methodology
4. “Human in the loop”
paradigm combines human
intuition and expertise with
machine scale and AI power

© 2019 Proprietary and confidential.
Dataset
Organizations
(Capitol IQ Crunchbase)
Search Input
(artificial intelligence OR AI
OR machine learning OR
deep learning)
Timeframe
2010 -2019
Result
4,403 companies
returned
Case Study: AI Investment Landscape

Sports, Games (2.5%)Facial Recognition (3.0%)
Medical Imaging (3.2%)
Drone, UAV (1.3%)
Bioinformatics, Life Science (3.0%)
Industrial Automation, Oil
and Gas (4.3%)
Marketing Tech, AdTech (3.5%)
Accounting, Payment, Invoices (2.8%)
Text Analytics (4.2%)
Energy Management (3.3%)
Chips, Semiconductor (2.8%)
EdTech (3.2%)
MedTech (4.1%)
Speech Recognition (2.5%)
Robotic Process Automation (3.0%)
Recruiting (2.3%)
Workforce, Human Resources (1.1%)
Real Estate (2.2%)
Data Center, Data Tools (5.5%)
Chatbots (3.5%)
Marketing/Sales Automation (2.3%)
Fashion, Retail
Tech (4.7%)
Hospitality,
Travel (1.6%)
FinTech (4.2%)
Food and Beverage, Healthy
Meal (1.7%)
Autonomous Vehicle (3.9%)
Fraud Detection, Money Laundering (1.9%)
Network Security (2.7%)
InsureTech (2.2%)
Legal Tech (1.3%)
Network Map
Company network with 4403 companies. Colored by clusters. Sized by degree. Labeled by clusters.

Overall AI Investment Landscape (2010-2019)
“Autonomous Vehicles” has the
highest total received
investment amount
“Text Analytics” has an older
median founding year and a high
investment received amount
“FinTech” is the newest AI
technology area for investment
Autonomous Vehicle
FinTech
Text Analytics
Fashion, Retail Tech
Facial Recognition
Bioinformatics, Life Science
Network Security
Chips, Semiconductor
Energy Management
MedTech
EdTechLegal Tech Mental Health, Wellness
Workforce, Human Resources
Company scatterplot with 36 clusters. Colored by cluster. Sized by investment received. Labeled by cluster name.

AI Investment Received Amount, Colored by Geography
Company bar chart with 1,576 companies. Categorized by cluster. Colored by where the company is based.
$12.1B
$6.8B
$6.5B
$4.8B
$4.7B
$4.0B
$3.0B
$3.0B
$3.0B
$2.6B
$2.5B
$2.3B
$2.1B
$2.1B
$2.0B
Country Key
● United States 39%
● China 11%
● United Kingdom 7.2%
● Canada 4.2%
● Japan 3.8%
● Israel 3.5%
● France 3.3%
● India 3.3%
● Germany 3.0%
● Singapore 2.1%
● Spain 1.2%
● Australia 1.1%
● Sweden 1.0%
From an investment
perspective, Chinese
companies dominate the
Autonomous Vehicles, Text
Analytics, Facial
Recognition, and Fintech
space while US companies
are less concentrated in one
area of investment.
US based companies
Databricks, Netskope,
and Sumo Logic share
the bulk of investment in
the Data Center/Tools
space.

Policy Implications
• Evidence points to unconditional divergence in AI startup activity across countries. There is also
evidence of unconditional convergence across investment in AI sectors at the global level, implying
sectors with low investment levels may thrive in the future.
• Countries, states, or sub-national regions that had higher initial level of startup activity, also witnessed
faster growth over the last five years.
• Emerging new sectors are still being funded.
• The increase in AI startup activity is not only concentrated in certain regions, but also concentrated
amongst select startups.
• Policymakers could ensure that AI does not just become a bubble and that it witnesses sustained growth
towards the development of a GPT ecosystem.
• A deeper understanding of how AI based service activities alter the nature of international trade and
investment could be highly informative for policymakers.

Quid Contacts For Further Questions
Q u i d I n c
• h t t p s : / / q u i d . c o m
• h e l l o @ q u i d . c o m
S p e a k e r :
• D a r i a M e h r a ( d m e h r a @ q u i d . c o m )
M a t e r i a l s C r e a t e d b y :
• Ju l i e K i m ( j k i m @ q u i d . c o m )
• E m i l y R a l s t o n ( E m i l y. R a l s t o n @ q u i d . c o m )

Network Methodology
The algorithm uses textual similarities to identify documents that are similar to each other. It then
creates a network based on these similarities so that the user can visualize these similarities as a network
of clusters. To do this, Quid leverages proprietary NLP algorithms and unsupervised machine learning
to automate the topical generation.
The dots, or nodes, represents individual articles and the links represent semantic similarity between two
nodes, with the clusters (groupings) differentiated by colors representing the topics.
Readers can refer to Olivia Fischer, Jenny Wang (2019). Innovation and Convergence/Divergence:
Searching the Technology Landscape for more details on the methodology.

2.2 M public and private company profiles from multiple data sources are indexed in order to search across company
descriptions, while filtering and including metadata ranging from investment information to firmographic information
such as founded year, HQ location, and more. Company information is updated on a weekly basis. Trends are based on
reading any text to identify key words, phrases, people, companies, and institutions; then compare different words from
each document (news article, company descriptions…etc) to develop links between these words based on similar
language. This process is repeated at an immense scale which produces a network that shows how similar all documents
are.
Search, Data Sources, and Scope

Organizational data from CapIQ and Crunchbase are embedded together. These companies include
all types of companies (private, public, operating, operating as subsidiary, out of business) in the world;
The investment data include private investment, MA, public offering, minority stake made by
PE/VCs, corporate venture arms, governments, institutions both in and out of the US. Some data is
simply unreachable when the investors are undisclosed, or the funding amounts by investors are
undisclosed. We also embed firmographic information such as founded year, HQ location.
We embed CapIQ data as a default, and add in data from Crunchbase for the ones that are not
captured in CapIQ. This way we not only have comprehensive and accurate data on all global
organizations, but also capture early-stage startups and funding events data. Company information is
uploaded on a weekly basis.
Data sourced from:
Quid indexes 2.2 M public and private company profiles from multiple data sources allowing you to
search the company descriptions, while filtering by included metadata ranging from investment
information to firmographic information such as founded year, HQ location, and more. Company
information is updated on a weekly basis. 7,000 companies can be analyzed within one network.
Company information is updated on a weekly basis.
Data

Cluster Color Key
●
Data Center, Data
Tools
5.6%
● Fashion, Retail Tech 4.8%
●
Industrial
Automation, Oil and
Gas
4.6%
● MedTech 4.4%
● FinTech 4.0%
● Text Analytics 4.0%
● Autonomous Vehicle 3.9%
●
Marketing Tech,
AdTech
3.6%
● Chatbots 3.5%
● Energy Management 3.3%
● Facial Recognition 3.1%
● EdTech 3.0%
● Medical Imaging 2.9%
●
Accounting,
Payment, Invoices
2.8%
●
Bioinformatics, Life
Science
2.7%
●
Chips,
Semiconductor
2.7%
● Network Security 2.7%
● Sports, Games 2.7%
● Recruiting 2.5%
●
Robotic Process
Automation
2.5%
● Speech Recognition 2.5%
●
Supply Chain
Management
2.5%
● Real Estate 2.4%
● AR/VR 2.4%
● InsureTech 2.4%
●
Marketing/Sales
Automation
2.3%
● Agriculture Tech 2.1%
● Credit Card, Lending 1.9%
●
Fraud Detection,
Money Laundering
1.9%
●
Food and Beverage,
Healthy Meal
1.8%
●
Mental Health,
Wellness
1.7%
● Hospitality, Travel 1.6%
●
Music, Media and
Entertainment
1.4%
● Drone, UAV 1.4%
● Legal Tech 1.3%
●
Workforce, Human
Resources
0.98%

D I G E S T L A N G U A G E ,
N O T J U S T N U M B E R S
Easily leverage our proprietary data
science for AI-driven insights
V I S U A L I Z E D A T A A T
S C A L E F O R C O N T E X T U A L
U N D E R S T A N D I N G
Quickly gain context around any topic with
long-form content analysis
Q U I D H E L P S Y O U
S U R F A C E S P E C I F I C
I N S I G H T S
Understand the data through unique
views for deep research and discovery
S H A R E Y O U R
A N A L Y S I S
Quid helps you build reports to influence
strategic decisions
H o w D o e s Q u i d W o r k ?
Quid software reads
millions of documents and
offers immediate insight
by organizing that content
visually.
We power human intuition
with machine intelligence,
enabling organizations to
make decisions that matter.

HOW TO
READ A
QUID
NETWORK
Nodes - Each node
represents one data point
Isolated nodes - Not
clustered due to lack of
available text or very niche
languageClusters – Nodes that
are bound together by
unique keywords
(semantic clustering
based on unstructured
text data)
Far proximity – Cluster is quite niche
Nodes in this cluster have few
connections to other clusters
Link – A link exists between
nodes if they share common
keywords in their text
Close proximity – Displays high
degree of relevancy and
integration between two themes
Node with potential
fresh new crossover
across multiple themes

Traditional search returns lists Quid identifies themes and organizes them visually
Enterprise IOT
(15%)
IoT Driving Change
(13%)
IoT Startups
(11%)
Cyber Security
(10%)
Performance Design
(10%)
International Innovation
(10%)
CES Previews
(4.9%)
IoT Opportunities
(4.6%)
Network Implications
(4.4%)
183,000,000 results
Events Conferences
(5.4%)

WHAT IS
QUID USED
FOR?
USE CASE USE CASE DESCRIPTION Sample Question
Emerging Tech
Identify new technology trends and “hot spots” emerging within
them
What does the technology landscape
for digital retail look like?
Adjacency Opportunities Understand how specific technologies may be applied in new ways
What are the applications of my
autonomous driving technology and
where can they be applied?
Competitive Intelligence
Understand where competitors are active to identify key
battlegrounds
What payments offerings are my
competitors providing?
Trend Analytics Identify novel trends that will shape the future at a macro-level
What are the key conversations
around ageing?
MA Screening Identify acquisition or partnership targets within a sector of interest
How can I leverage data from my
competitors’ acquisition strategy to
inform my MA strategy?
Partnerships Analysis
Identify subtle corporate partnerships to get a better pulse on
industry initiatives
Who is Tesla partnering with and are
they expanding beyond the automotive
industry?
Brand Perception
Map media conversations and understand how competitors are
differentiated
Which brands are leading the current
narrative around skin care?
Consumer Sentiment
Analyze online forums to see what customers are really saying or
thinking
What are the top concerns of diabetic
patients as discussed on online forums?
Survey Analysis
Map open-text survey responses to understand the key themes and
novel insights
How can I leverage the free-text survey
responses from my customers to
improve my service offerings?
KOL Analysis Identify Key Opinion Leaders in the news media
Is discussion of our brand influenced in
the media primarily by internal or
external stakeholders?

Vinod Bakthavachalam
Data Scientist
Coursera Global Skills
Index

46+ million
Registered learners
190
University and
industry partners
2,000+
Businesses,
Governments,
Universities
A global platform
for learning
2
Coursera Platform

We Partner with Leading
Universities Companies

Content Supplies Skills
Coursera is a skills marketplace
Learners Demand Skills

What can it tell us?
Benchmarking
Competency Popularity
Trending Skills
“How key countries and industries rank
on the skills of tomorrow”
“Which competencies are countries and
industries increasingly investing in learning”
“Which speciﬁc skills are countries
and industries looking to learn”
GLOBAL SKILLS INDEX
The Coursera Skills Graph

Industries
Automotive
Consulting
Finance
FMCG
Health/Med/Biotech
Insurance
Manufacturing
Media/Entertainment
Technology
Telecom
Middle East Africa
Egypt
Israel
Kenya
Nigeria
United Arab Emirates
Saudi Arabia
South Africa
GLOBAL SKILLS INDEX
60 countries across 5 regions + 10 industries
Europe
Austria
Belarus
Belgium
Czech Republic
Denmark
Finland
France
Germany
Greece
Ireland
Hungary
Italy
Netherlands
Norway
Poland
Portugal
US Canada
United States
Canada
Latin America
Argentina
Brazil
Chile
Colombia
Costa Rica
Dominican Republic
Ecuador
Mexico
Guatemala
Peru
Venezuela
Asia Paciﬁc
Australia
Bangladesh
China
Hong Kong, China
India
Indonesia
Japan
Korea (S)
Malaysia
New Zealand
Pakistan
Philippines
Singapore
Taiwan, China
Thailand
Vietnam
Romania
Russia
Spain
Sweden
Switzerland
Turkey
Ukraine
United Kingdom

What skills do we cover in the GSI?
GLOBAL SKILLS INDEX

Coursera Skills Taxonomy for AI
Benchmarking
Rankings,
Competency
Popularity
Trending
Skills
GLOBAL SKILLS INDEX

Benchmarking: How We Calculate It
Elo Model
Aggregate to
country or
industry level
GLOBAL SKILLS INDEX
80%, 90%, 50%,
100%, 90%, 75%
95%, 100%, ...
Learner
Assessment
Scores in Related
Courses
Individual Skill Ability

Correlation Between AI Skills Across Countries
GLOBAL SKILLS INDEX

GLOBAL SKILLS INDEX
AI Skills Ranking vs. HCI GDP per Capita

Benchmarking the LECCE Framework
We divide aggregated performance
into four categories, based on
quartiles (the “LECCE framework”):
LECCE Color Key for GSI
Artiﬁcial Intelligence
Cutting-Edge → for 76% to 100%
Lagging → for 25% or below
Emerging → 26% to 50%
Competitive → 51% to 75%
Performance
GLOBAL SKILLS INDEX

Section title
AI Skills Across the World
GLOBAL SKILLS INDEX

Track Country, Industry,
Company Progress in
AI skills over time
Understand the
Geographic Distribution
of AI Talent
Comparative
Benchmarks
Connect Coursera Data
to Economic, Financial,
and Business Data to
Understand how
Learning Drives
Performance
Identify Successful
Student Learning Paths
to Provide Educational
Opportunities Skill
Based Hiring Signals
Sample Uses
of GSI Data
GLOBAL SKILLS INDEX

How the
IMF is
using ML
as subfield
of AI
• Bachir Boukherouaa (ITD)
• Ashraf Khan (MCM)
• International Monetary Fund
• AI human index workshop: Stanford Oct 2019

Agenda • An Introduction to
the IMF
• Role of the IMF
• Use of ML within the
Fund
• The impact of
technology on the
financial system
• Conclusion
• AI Workshop Stanford
2019

How the IMF Works
Board of Governors IMFC Executive Board
Management Staff

ML
Initiatives/Projects
(i)
Global Trade Intelligence (GTI) – A
machine learning approach to
nowcast global trade activity.
AI Solutions to Assessing Capital and
Credit Flows in EMDCs.
Vulnerability Exercise (VE) upgrade -
The Vulnerability Exercise (VE) is a
multi-sectoral approach to assessing
factors that make a country
vulnerable to a crisis.

ML
(ii)
Causal Machine Learning. Use the
latest developments in ML to add
explanatory ML and causal inference
to address macroeconomic policies
counterfactuals. Proof of Concept.
Training “Fundamentals of Machine
Learning for Economists.” It is
delivered by internal IMF staff as part
of the regular calendar of the Internal
Economics
Macroeconomic variables nowcasting

ML
(iv)
• Cognitive Adviser System. Uses Natural
Language Processing to leverage better
knowledge from cross-country experience in
policy advice. Proof of Concept.
• Several NLP-related projects
– Sentiment analysis using Financial Times
data.
– Building of a database of major tax policy
changes in low-income and developing
countries by analyzing text from the
International Bureau of Fiscal
Documentation.
– Selecting Fintech-related news from a
variety of news feeds, blogs and tweets.
– Etc

ML
(v)
• Several geospatial data projects
– Filling data gaps for African
countries using nightlight data
– Assessing effects of climate change
– Transforming open-source Satellite
Images into sustainable
development indicators
– Etc.

IMF: Fintech IMF Policy Development
1) IMF Fintech Developments
- IMF Fintech
- Bali Fintech Agenda
- Fintech in practice
- AI/ML in the financial sector:
benefits risks
2) ML Example: Central Bank Legislation
Database (CBLD)

IMF Fintech
Source: IMF, 2017, Fintech and Financial Services: Initial Considerations, IMF Staff Discussion Note.
“A new wave of technological innovations, often called “fintech,” is accelerating change in
the financial sector. What impact might fintech have on financial services, and how should
regulation respond?”

Elements:
1) Embrace the promise of fintech
2) Enable new technologies to enhance financial service provision
3) Reinforce competition and commitment to open, free, and contestable markets
4) Foster fintech to promote financial inclusion and develop financial markets
5) Monitor developments closely to deepen understanding of evolving financial systems
6) Adapt regulatory framework and supervisory practices for orderly development and stability of
the financial system
7) Safeguard the integrity of financial systems
8) Modernize legal frameworks to provide an enabling legal landscape
9) Ensure the stability of domestic monetary and financial systems
10) Develop robust financial and data infrastructure to sustain fintech benefits
11) Encourage international cooperation and information-sharing
12) Enhance collective surveillance of the international monetary and financial system
Bali Fintech Agenda (1/2)

12
VI. Adapt Regulatory
Framework and Supervisory
Practices for Orderly Development
and Stability of the Financial System
and facilitate the safe entry of new
products, activities, and intermediaries;
sustain trust and confidence; and
respond to risks.
Many fintech risks might be addressed
by existing regulatory frameworks.
However, new issues may arise from
new firms, products, and activities that
lie outside the current regulatory
perimeter. This may require the
modification and adaptation of
regulatory frameworks to contain risks
of arbitrage, while recognizing that
regulation should remain proportionate
to the risks.
Holistic policy responses may be
needed at the national level, building
on guidance provided by standard-
setting bodies.
II. Enable New
Technologies to Enhance
Financial Service Provision by
facilitating foundational
infrastructures, fostering their
open and affordable access, and
ensuring a conducive policy
environment.
Foundational infrastructures
include telecommunications,
along with digital and financial
infrastructures (such as
broadband internet, mobile data
services, data repositories, and
payment and settlement
services).
The infrastructures should enable
efficient data collection,
processing, and transmission,
which are central in fintech
advances.
Bali Fintech Agenda (2/2)

The main challenges for
fintechs are gaps in
regulation, talent, and
private capital.
For Distributed Ledger
Technology (DLT) in specific,
legal risks and an unclear
regulatory environment are
key inhibitors.
Regulatory
sandboxes are
emerging in most
regions, though they
are limited in Africa.
Estimated cost savings from fintech
for banks in Asia are in the range of
10-20 percent of operating income.
Fintech in Practice

a) Enhanced machine-based processing of
operations: identifying customers’ needs
with tailored products and automating
routine business processes could increase
efficiency and lower operating costs.
b) Enhanced risk management: by means of
earlier and more accurate estimation of
financial and nonfinancial risks - though
tools could be “overtrained.”
c) Stronger collaboration: data intensity and
open-source character could facilitate
collaboration between financials,
regulators, and other institutions.
AI/ML the Financial Sector - Benefits
See: FSB, 2017, Artificial Intelligence and
Machine Learning in Financial Services.
Central banks
/ Financial
Supervisors
Financial
Institutions
Third Parties
Customers /
Users
Government

a) Complex algorithmic governance: decision-
making “black boxes” make it difficult to grasp
how decisions have been formulated, and
difficult to monitor them.
b) Unclear (legal) responsibility: lack of clarity
regarding allocation of responsibility for losses
(financial institution, third party, regulator,
users), as well as need to revisit of regulatory
and supervisory involvement.
AI/ML the Financial Sector - Risks
c) Increased financial risks: any uncertainty in the governance structure for using AI and machine learning could
increase risks. If investors makes investments without fully understanding the applications and possible
losses in tail events, the aggregate risks could be underestimated. In addition, any uncertainty in the
governance structure could substantially increase the costs for allocating losses, including the possible costs
of litigation.
d) Increased nonfinancial risks: high reliance on a relatively small number of third-party technological
developers and service providers could cause significant operational risk and might even be systemic. Also:
lack of regulation and supervision of third parties is a concern.

IMF Central Bank Legislation Database (CBLD)
• Most comprehensive central
bank law database in the world
- laws of 138 central banks and
monetary unions.
• Unique feature: 100+ categories
(developed by IMF), allowing for
detailed searches across
countries (region, income level,
exchange rate arrangement, and
monetary union).
• Managed by IMF; last updated
in 2014.
• Publicly accessible since 2019.
• Users: central banks,
governments, IMF/WB,
researchers / academics,
students, commercial entities.

1) Analyze CBLD data by means of NLP (English legal texts) and identify patterns,
allowing for identifying practices in legal texts, and assisting in drafting new
laws.
Contextual pre-training is highly relevant.
Example
The central bank is independent.
vs.
The central bank’s Board members are appointed by the Minister of
Finance and upon proposal of Parliament.
[appointed by… and] = central bank independence
2) Refine CBLD search methodology by better understanding of user queries.
CBLD: How can ML help?

AI Human index workshop: Stanford 2019
Thank you!

2. Economic Impact and Societal Considerations for Policy Decisions.

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to 2. Economic Impact and Societal Considerations for Policy Decisions.

Similar to 2. Economic Impact and Societal Considerations for Policy Decisions. (20)

More from Saurabh Mishra

More from Saurabh Mishra (6)

Recently uploaded

Recently uploaded (20)

2. Economic Impact and Societal Considerations for Policy Decisions.