Developing A Big Data Analytics Framework for Industry Intelligence

DEVELOPING A BIG DATA
ANALYTICS FRAMEWORK FOR
INDUSTRY INTELLIGENCE
Gene Moo Lee
SAUDER SCHOOL OF BUSINESS
UNIVERSITY OF BRITISH COLUMBIA

Gene Moo Lee, CAIDA, Jan 2019
I do research on Business Analytics
It is about analyzing Big Data for decision making
One of challenges is on unstructured data (80-90%)
• Text, photo, audio, video
Approach:
• Machine learning, text mining, deep learning
• Econometrics, network analysis
2

Research on business analytics
3
Social Media Cyber Security
Industry Dynamics Mobile Ecosystem

Analyzing industry dynamics using
network approach
• Industry can be modeled as a network, where node represents firm
and edge represents dyadic relationship between a pair of firms
• Inter-firm relationships: M&A, Competition, Opportunity, Alliance
4
Toward A Better Measure of Business Proximity:
Topic Modeling for Industry Intelligence
MIS Quarterly 2016
Opportunity Structures: A Machine Learning Approach for Analyzing
Industry Dynamics
Submitted to AOM 2019

Toward A Better Measure of
Business Proximity:
Gene Moo Lee (UBC)
Joint work with
Zhan Shi (Arizona State)
Andrew B. Whinston (UT Austin)
MIS Quarterly 2016

Business proximity: Motivation
• To measure firms’ dyadic relatedness in product, market, and technology
• Essential in strategy / industrial organization fields
• Existing methods
• Common industry code, patent holdings, geographic distance
• Limitations: strong data requirements, not fine-grained, static
• Our approach: text mining on business description text corpus
• Can be automated to capture industry dynamics
6

Our approach on business proximity
• Approach: LDA topic modeling [Blei et al. 2003]
• Unsupervised learning to discover latent “topics” from a large
collection of documents
• Business proximity = cosine similarity of topic distributions
7
LDA
Industry-wide topics
Company’s topics
Company
descriptions

CrunchBase data on high-tech industry
• CrunchBase: open database (“Wikipedia”) of high-tech industry
• Data collection time: April 2013 ~ April 2015
• 24,382 U.S. high-tech companies (1.4% public, 5.7 years old)
• HQ location, CB-defined industry sector, key personnels, M&A,
investments, business summary
• Leading states: CA, NY, MA, TX
• Leading industries: software, web, e-commerce, ad, mobile
8

LDA topic model with CrunchBase
9
Video/music
Energy
Sports
Healthcare

Empirical analysis on M&A network
• Objective: examine the relationship between likelihood of M&A link
formation and nodal/dyadic characteristics
• Challenges: incorporate inter-relatedness of M&As
• Logit/probit cannot capture dependency between obs.
• Model all M&A deals as a graph with ERGM or p* model
10

ERGM for M&A network
ERGM (Exponential Random Graph Model):
• Based on random graph [Erdos and Renyi 1959]
• Probability of realizing a graph = a function of the graph’s
statistics [Robins et al. 2007]
• Inter-firm proximity: business, geographic, social, co-invest
• Selective mixing: 50 states, 30 industry sectors
• Degree distribution: node degree, M&A experiences
11
degree selective mixing proximity

Estimation setup
• Dataset
• US firms founded from 2008 to 2012: |V| = 24,382
• All dyadic/nodal attributes collected in April 2013
• M&A transactions (April 2013~April 2015): |E| = 394
• Estimate our ERGM M&A model
• Randomly sample 25% companies for computational feasibility
• Run 100 condor jobs with 100 sample graphs
• Estimate model coefficients by Markov chain Monte Carlo
(MCMC) maximum likelihood estimation (MLE)
12

ERGM estimation from a sample
13

Empirical results on proximities
14
• Proximities are normalized for comparison
• 1.0 stdev increase in business proximity
= 3.64 stdev increase in social proximity
= 6.89 stdev increase in investment proximity
+
+
+

Empirical results on complementarity
15
• Original term (+) / Squared term (-) -> reverse U-curve
• Interpretation: M&A transactions between two firms that
have complementarity but not substitutes
• Can find this curvilinear effect because our proximity has
(1) comprehensiveness and (2) continuity
+
+
+
-

M&A matching platform w/ business proximity
• M&A matching platform for startups, VCs, and researchers
16

Platform UI: Find competitors
● M&A market is a two-sided platform
o buyers: established companies
o sellers: startups
● We can increase the efficiency of this two-sided market by
o building interface, VentureMap, to make data accessible
o recommending matchings with our M&A model
● Potential beneficiaries
o Established firms: intelligence/M&A department
o Startups: identify opportunities, potential buyers
o Venture capitalists
o Market intelligence firms
o Researchers in finance field
17

Search firms by business components
18

Search firms by business components
19

LDA results from U.S.
public firms (1995-
2016)
20

Service sector (SIC 7) in
1995-2016
21

Healthcare sector in 1995-2016
22

Toward A Better Measure of Business Proximity:
MIS Quarterly 2016
Big Data Analytics Special Issue
Opportunity Structures: A Machine Learning Approach for
Analyzing Industry Dynamics
Submitted to AOM 2019

Opportunity Structures:
A Machine Learning Approach for
Analyzing Industry Dynamics
Gene Moo Lee (UBC)
Joint work with
Myunghwan Lee (Yonsei), Hasan Cavusoglu (UBC),
Marc-David L. Seidel (UBC)

Structural hole as business opportunity
• In network perspective, we view industry as a network of firms
• Burt (1992) argues that as industry networks become
centralized, emerging “structural holes” serve as entry points
for new firms: new business opportunity
25
Publisher 1
Publisher 2 Subscriber B
Subscriber A
S.H.
S.H.
S.H.
S.H.
S.H.
S.H.
S.H.
Fully-connected Centralized

Definition of structural holes
• A pair of firms (I, J) has a structural hole w.r.t. firm K if
• (1) (I, J) are not connected
• (2) (I, K) and (J, K) are connected
• A pair of firms is connected if their 10-Ks are sufficiently similar to
each other (LDA or doc2vec model)
• A pair of firms can have 0 to N-2 holes (N=# firms in year t)
26

Data: Annual reports of U.S. public firms
• Form 10-K filings from SEC EDGAR: 165K reports (1995-2016)
• Use doc2vec [Le & Mikolov 2014] to get semantic vectors of 10-Ks
• Data: IPO/delisting events, financial and accounting metrics
27

Firm positioning with doc2vec: Link
28

Preliminary empirical results
• The likelihood of IPO entry between two firms has a positive relationship
with # structural holes between them. (Supported with GLM method)
• # of structural holes for an IPO has a U-shape relationship with the firm’s
ultimate mortality rate. (Partially supported with Cox hazard model)
29
Sector
Mining &
Construction
Wholesale &
Retail
Finance Services
Dependent
Variable
Hazard Rate (Delisting)
# structural
holes
n.s. negative* negative** negative*
# structural
holes
squared
positive+ n.s positive* n.s
# firm-year
obs.
1,215 1,578 3,322 2,786
# delisting
events
35 67 72 181

Concluding remarks
• Business Analytics is an emerging research area
• To apply AI, ML, and NLP on business data
• To gather insights for decision making
• Important business and societal decisions now depend on AI
• Many interesting research topics!
30

Contact Info: Gene Moo Lee
gene.lee@sauder.ubc.ca

Inter-firm relationships
• M&A: 1689 total
• cross-state: 62.6%
• cross-sector: 63.6%
• top 10 buyers: 14.3%
(skewed)
• Investments: 531 total
• Job mobility: 19K total
32

Validation: Leading effect on business networks
• Avg. business proximity
• 0.293 (394 M&A pairs)
• 0.224 (129 invests pairs)
• 0.218 (9792 job mobility pairs)
• 0.068 (random pairs)
33

M&A matching platform w/ business proximity
• “Data-driven” platform for M&A matching and startup search
1. M&A executives to find M&A targets
2. Entrepreneurs to position their products
3. Venture capitals to monitor niche markets
4. Analysts to examine the industry trends
• Implemented a cloud-based IS based on proposed business proximity
34

Cloud-based platform design
35
Big Data and Cloud technologies: Cronjob, NoSQL, Python, Scala,
Condor, Google Cloud (Storage, App Engine, Datastore) and more

Item Details
Item 1
“Business” –
Description of companies business, it’s main products and services, subsidiaries it
owns, market it operates in, recent events, competitions, etc.
Item 1A
“Risk Factors” –
Most significant risks that apply to company or its securities, listed in order of their
importance.
Item 1B
“Unresolved Staff Comments” –
Explain certain comments it received from SEC staff on previous filings that have not
been resolved over an extended period of time.
Item 2
“Properties”–
Information about companies significant properties like principal plants, mines, etc.
Item 3
“Legal Proceedings” –
Information about significant pending lawsuits or other legal proceedings, other than
ordinary litigation
Item 4 This item has no required information, but is reserved by SEC for future rule making
What are in 10-K annual reports?
36
Part 1 (Items1-4), Part 2 (Items 5-9), Part 3, Part 4

Information system design
37

SEC data from EDGAR
Electronic Data Gathering, Analysis, and Retrieval system
• U.S. Securities and Exchange Commission (SEC)
• 513K firms/individuals (CIK) in 451 industry sectors (SIC)
• 23K cities, 43K ZIP codes, 251 states (incl. non-US)
• 11.2M forms filed of 723 types (4, 8-K, 10-K/10-Q, etc.)
• 10-K documents’ text were parsed (Parts and Items)
• Duration: 1995 ~ on-going
38

Number of 10-K filers
1995-2016
39

Headquarters of 10-K filers
1995-2016
40

Analysis I: Trend analysis with LDA
• Approach: LDA topic modeling [Blei et al. 2003]
• Unsupervised learning to discover latent “topics” from a large
collection of documents
• Each document is represented as a distribution over the topics
41
LDA
Industry-wide topics
Company’s topics
Business
descriptions
(10-K)

Microsoft Topics, 1995 - 2016
42
Dominating topics:
software/tech/data,
president/vice/exec
utive, stores/retail,
systems/manufactu
ring/tech

Interactive visualization demo: Link
• Developed an interactive visualization tool to
identify industry topic trends by:
• Selected time windows (time-based)
• Industry sectors (SIC code-based)
• Geographic location (city, state-based)
• A particular firm (CIK-based)
43

Analysis II: Competitive analysis with word2vec
• Approach: word embedding called word2vec [Mikolov et al. 2013]
• Represent words in a high-dimensional vector space where
semantically similar words are nearby
• Train a model that maximizes prediction of words co-occurrence
(K words before/after the focal word)
• Competition level = distance between word vectors
44
Word2Vec
Business
descriptions
(10-K)

Static word2vec: Google
Nearest neighbors:
Yahoo, Linkedin,
Amazon, Netflix,
Facebook,
Youtube, Hulu,
Bing
45

Static word2vec: Android
Nearest neighbors:
apple_ios,
google_android,
symbian,
ipad_iphone
46

Temporal word2vec: Apple
Nearest neighbors:
• 1995: jbl, unisys, novell
• 2000:
information_technology,
isuzu, powertel_inc
• 2005: dell, panasonic,
emi, midi
• 2010: amazon, sony, dell
• 2015: google, iphone,
ipad, sony
47

Temporal word2vec: IBM
Nearest neighbors:
• 1995: microsoft,
nordstorm, apple
• 2000: oracle, sun,
sun_microsystems
• 2005: motorola, dell,
cisco, sun_microsystems
• 2010: hewlett_packard,
motorola, oracle, dell
• 2015: dell, oracle, nokia,
microsoft
48

Gene Moo Lee, WITS, Seoul, Korea, December 2017
Google Topics 1995 - 2016
49
Dominating topics:
software/tech/data,
advertising/televisio
n/media

Gene Moo Lee, WITS, Seoul, Korea, December 2017
Static word2vec results: iPhone
Nearest neighbors:
mobile_platforms,
ipad,
android_phones,
tablet_devices,
ipod_touch,
handheld_devices
50

Developing A Big Data Analytics Framework for Industry Intelligence

More Related Content

What's hot

Similar to Developing A Big Data Analytics Framework for Industry Intelligence

More from Gene Moo Lee

Recently uploaded

Developing A Big Data Analytics Framework for Industry Intelligence

Editor's Notes