Deep Learning based Emerging Technology Minining Model. Evaluating the importance of Quantitative Features of Technology. PICMET 2019, Portland, Oregon, USA Korea Institute of Science and Technology Information (KISTI) University of Science and Technolgy (UST) Seonho Kim

1. 2019년도
사업계획 및 예산(안)
한국과학기술정보연구원
“Can AI Tell Emerging Technologies?”
Evaluating the Importance of Quantitative
Features of Technology
PICMET 2019
Portland, Oregon
Seonho Kim (seonho@gmail.com)
Korea Institute of Science and Technology Information
University of Science and Technology

2. 2
Outline
• Motivations
• Research Goals
• Related Work
• Resource Data
• Quantitative Features of Technology
• AI-Based Emerging Technology Mining Model
• Feature Evaluation
• Empirical
• Statistical
• Discussion and Conclusions

3. • Purposes of technology forecasting
• To support Government’s strategic R&D planning
• To support SME searching for next business items
• Etc.
• Problems of Traditional Forecasting Process
• Subjective
• Inconsistent
• Expensive
• Slow
• Limited data
• Limited by knowledge
• Etc.
Motivations

4. • Build an AI-based (Deep Learning) model
• for emerging technology mining (from now on, ETM)
• objective, consistent, fast, affordable, …
• Focus is,
• on imitating human expert group conducting emerging technology
mining
• not on fully developing ETM algorithm
• Evaluate,
• the value, effectiveness, of various quantitative features of technology
• by deep learning way,
• by statistical way
Research Goals (1/4)

5. Research Goals (2/4): Turing Test
wall
Experiment task: Ask and answering

6. Human Expert Group
*ETM (Emerging Technology Mining)
Experiment A
Research Goals (3/4): AI based ETM
Experiment task: ETM

7. Human Expert Group
Features of Technologies
Experiment B
Research Goals (4/4): Feature Evaluation
Trained by
refer
Experiment task: ETM

8. • Emerging technology forecasting
• Delphi method, scenario planning, interview analysis, ..
• Limits: Subjective, inconsistent, slow, data is expensive
• Technology roadmapping
• Employs: market demand and customer needs, etc.
• Bibliometric methods
• Analyzing patents and papers
• Quantitative technology forecasting
• Artificial Intelligence
• Text mining, Machine learning
• Classify high-quality, promising, technology for each innovation attribute
• Selecting features and methods are still challenging
• Feature evaluation for Emerging technology Mining
• as Indexes to estimate the value of technologies
Related Work

9. Data
• Source Data for feature extraction
• US Trademark
• USPTO from 1993 to 2018 (granted 5,155,517 patents)
• Product (technology) Data
• Product network
• Contains 256,000 unique products
• Found about 450,000 of relations between about 54,000
product nodes
• Relations contain upstream, downstream, sibling, etc.
• Data for Training, Validation, and Testing
• ‘Top 2000 Emerging Products’
• by a human expert group for two years

10. Quantitative Features of Technology (1/2)
• Monopoly rate
෍
𝑖=1
𝑁
(
𝑛𝑜.𝑜𝑓 𝑖′ 𝑠 𝑝𝑎𝑡𝑒𝑛𝑡𝑠
𝑛𝑜.𝑜𝑓 𝑡𝑜𝑡𝑎𝑙 𝑝𝑎𝑡𝑒𝑛𝑡𝑠
× 100)2
, where i represents patents owned by the enterprise.
• Mean year
• Emerging rate
• Spread rate : forward influence on the product network
• Compound rate : backward influence on the product network
• Total number of related patent
• K-index
• Domestic rate

11. Quantitative Features of Technology (2/2)
• (last) 10-year patent
• Last 10-year slope (gradient)
• Last 5-year slope
• Last 2-year slope
• Stability

12. AI-based ETM model (1/3)
• 2 class, emerging or non-emerging, classifier with Multi-Layer
Perceptron
• 13 features for input
• 2 Hidden Layers, 100 nodes each
• A Softmax Layer

13. AI-based ETM model (2/3) : Experiment A
• 10-fold cross validation

14. AI-based ETM model (3/3) : Result of Experiment A
• Performance of classification: AUC-ROC (area under curve-
receiver operating characteristic)

15. Feature Evaluation
• Test the effectiveness of each quantitative features of technology
on ETM
• Empirical approach (Experiment B-1)
• Using AI-based (Deep Learning) model
• Statistical approach (Experiment B-2)
• Class comparisons

16. AI based Feature Evaluation : Experiment B-1 (1/2)
• Empirical approach
• 13 AI-ETM models
• 13 different training data for each AI-ETM model
• Analyze the performance changes
…

17. • Results
0.48
0.5
0.52
0.54
0.56
0.58
0.6
0.62
0.64
0.66
AUC-ROC (a feature excluded)
AI based Feature Evaluation : Experiment B-1 (2/2)

18. Statistics based Feature Evaluation : Experiment B-2 (1/3)
emerging non-emerging emerging non-emerging
Standard Deviation difference Mean Difference

19. • Mean Difference
0
0.1
0.2
0.3
0.4
0.5
0.6
Statistics based Feature Evaluation : Experiment B-2 (2/3)

20. • Standard Deviation Difference
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Statistics based Feature Evaluation : Experiment B-2 (3/3)

21. Conclusions, Discussion and Future Plan
• Observed potential of AI model for ETM
• Need more improvement to substitute human expert group
• Tested only the effectiveness of features
➔ Need to test detailed properties of feature
• Goal of performance was imitating the human group
➔ Further study for Post-evaluation based performance test
• More quantitative features needs to be examined
➔ Recommendation rate, influence rate, similarity rate, replaceability, etc.
• Features are not totally independent
➔ Delicate analysis technique is needed
• Different analysis results
➔ Need to sophisticate integration and interpretation

22. Questions? Comments?