Recommended
More Related Content
What's hot
What's hot (20)
Similar to Data science lab project
Similar to Data science lab project (20)
Recently uploaded
Recently uploaded (20)
Data science lab project
- 1. DATA SCIENCE LAB PROJECT Master Degree: Data Science Accomplished by: A. Portaluppi & L. Ravazzi & M. Spandri A.A. 2019-2020 1
- 2. INTRODUCTION DATA SET DEMS Publications (Dipartimento di Economia, Metodi Quantitativi e Strategie di Impresa). Find topics studied by DEMS universitary researcher. Multidimensional Scaling techniques and Cluster Analysis. 2 PURPOSES TOOLS
- 3. DATA MANAGEMENT: 1. Exploration 2. Preprocessing 3. Data Cleaning 4. NLP MULTI- DIMENSIONAL SCALING: 1. Common Multidimension al Scaling 2. Metric Scaling 3. Sammon Mapping CLUSTER ANALYSIS: Prototype-Based: Fuzzy Algorithm DATA VISUALIZATION: RShiny Application STEP BY STEP 3
- 4. DATA MANAGEMENT DATA VISUALIZATION CLUSTER ANALYSIS MULTIDIMENSIONAL SCALING ID TITLE JOURNAL ABSTRAC T ABSTRAC T_ENG KEYWOR DS KEYWOR DS_ENG 235 … … … ID DEMS_AUTHORS 235 … 235 … 235 … … HOW TO MANAGE THE DATA SETS? 4
- 5. DATA MANAGEMENT DATA VISUALIZATION CLUSTER ANALYSIS MULTIDIMENSIONAL SCALING HOW TO CHOOSE RECORDS? 30% DOCUMENTS LEFT 5 JOURNAL ARTICLES WRITTEN BY ASSISTANT PROFESSORS ETC. ENGLISH LANGUAGE
- 6. DATA MANAGEMENT DATA VISUALIZATION CLUSTER ANALYSIS MULTIDIMENSIONAL SCALING What does it means in English? Perfect! There is a field which specifies the language. The language of an article is the language of the abstract (textcat function). 6
- 7. DATA MANAGEMENT DATA VISUALIZATION CLUSTER ANALYSIS MULTIDIMENSIONAL SCALING NATURAL LANGUAGE PROCESSING Start with mixed texts (title, abstract, keywords and journal) Bag of words 1. Drop out punctuation, stop-words, non-letter character 2. All in lower case 3. Stemming process 1 2 3 4 Compute tf_idf 7
- 8. DATA MANAGEMENT DATA VISUALIZATION CLUSTER ANALYSIS MULTIDIMENSIONAL SCALING MULTIDIMENSIONAL SCALING WHAT? A function to project data from a N-dimensional space to 2 or 3 dimensions WHY? • Graphical approach (Clustering) • Increase Interpretability HOW? • Metric • Non Metric 8
- 9. DATA MANAGEMENT DATA VISUALIZATION CLUSTER ANALYSIS MULTIDIMENSIONAL SCALING Table with text and the number of terms into the bag of words. Choose a proximity measures. Apply the desidered technique. 9
- 10. DATA MANAGEMENT DATA VISUALIZATION CLUSTER ANALYSIS MULTIDIMENSIONAL SCALING 1. Common Multidimensional Scaling (Euclidean distance) 2. Metric Scaling 3. Sammon Mapping (Manhattan distance) We applied three techniques: and will describe only the last one due to the good results. 10
- 11. DATA MANAGEMENT DATA VISUALIZATION CLUSTER ANALYSIS MULTIDIMENSIONAL SCALING SAMMON MAPPING Minimize Sammon Stress: where is the distance between the i-th and j-th observation in the initial space, while refers to the final space. For metric and non metric data Non-linear trasformation approach (different from PCA) 11
- 12. DATA MANAGEMENT DATA VISUALIZATION CLUSTER ANALYSIS MULTIDIMENSIONAL SCALING Since an article can touch different topics, clustering must be of fuzzy type. CLUSTERING Labels of clusters rely on the fifteen words most frequent in the bag of words. 12 Manhattan distance is used in order to build clusters.
- 13. DATA MANAGEMENT DATA VISUALIZATION CLUSTER ANALYSIS MULTIDIMENSIONAL SCALING LABELS 13 DEMS ECONOMICS STATISTICS BUSINESS STRATEGY • Finance and Energy • Economic policy • Macroeconomics • Income Distribution • Game Theory • Health Statistics • Pure Statistics • Statistics and Finance • Social Issues • Industrial Economic • Corporate Finance
- 15. 1 2 3 SUMMARY CONCLUSIONS 15 Multidimensional scaling is a powerful tool to visualize data. We found the main topics studied by DEMS researches. MDS and Clustering can show interesting patterns in data.
- 16. FUTURE DEVELOPMENTS Other techniques for scaling, such as Self Organizing Maps. Other proximity measures for MDS. Consider not only singleton into the bag of words (Association Analysis). 16