From Genomics to NLP – One Algorithm to Rule Them All with Santi Adavani and Vinay Rao
Singular Value Decomposition (SVD) is a matrix decomposition technique with many applications in areas like genetics, natural language processing (NLP), and social network analysis. All these application areas result in very large matrices with millions of rows and Features. In genetics, matrix entries represent gene response for an individual, while in NLP these entries represent a term frequency per document. Datasets in these areas are growing rapidly and one processor cannot compute SVD in a feasible amount of time. Randomized algorithms for SVD with sketching have gained traction as they perform significantly better than classical deterministic algorithms in speed, accuracy, and robustness. In addition, these algorithms can be implemented to exploit multi-processor architectures and run on large-scale clusters with 1000s of cores on large datasets. This all sounds good. But there is a challenge! In data streaming applications data arrives in random order and is not directly suitable for randomized algorithms as they expect the whole dataset to be available. In this talk, we will present a hybrid approach of applying frequent directions algorithms that are well suited for data streaming applications and randomized algorithm for fast SVD computation. We will present results in various applications including video and NLP for datasets with billions of nonzero entries on clusters with 1000s of cores. We will also discuss how Spark performs for these large-scale machine learning challenge.
Recommended
Recommended
More Related Content
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
From Genomics to NLP – One Algorithm to Rule Them All with Santi Adavani and Vinay Rao
- 1. Santi Adavani/Vinay Rao Rocketml.net @rocketml From Genomics to NLP: One algorithm to rule them all #AI2SAIS
- 2. #AI2SAIS
- 3. Outline • Introduction • Making sense of SVD • One algorithm to rule them all • Scale out SVD #AI2SAIS
- 4. #AI2SAIS
- 5. Singular Value Decomposition (SVD) #AI2SAIS 𝑈, 𝑉 𝑎𝑟𝑒 𝑜𝑟𝑡ℎ𝑜𝑛𝑜𝑟𝑚𝑎𝑙 Σ is a diagonal matrix with singular values
- 6. Others in the same family Principal component analysis (PCA) Eigenvalue decomposition Latent Semantic Indexing (LSI) Latent Semantic Analysis (LSA) #AI2SAIS
- 7. Outline • Introduction • Making sense of SVD • One algorithm to rule them all • Scale out SVD #AI2SAIS
- 8. Making sense of SVD #AI2SAIS
- 9. Dimensionality reduction UABCDACCΣACCDACC 𝑉ACCDACC E 𝑈 Σ 𝑉′ #AI2SAIS
- 10. Reconstruct the matrix UABCDGΣGDG 𝑉GDACC E 𝑈 Σ 𝑉′ #AI2SAIS
- 11. Singular values and cumulative sum #AI2SAIS
- 12. UABCDHΣHDH 𝑉HDACC E UABCDIΣIDI 𝑉IDACC E 𝐴 #AI2SAIS
- 13. UHCΣHCDHC 𝑉HC E UKCCΣKCCDKCC 𝑉KCC E 180000 8510017020 9.4% 47% 𝐴 #AI2SAIS
- 15. Key points #AI2SAIS Reduce dimension without losing information Drop components that are noisy or do not contribute Identify components that contribute the most If eigenvalues are decaying fast then there is scope for dimensionality reduction
- 16. Outline • Introduction • Making sense of SVD • One algorithm to rule them all • Scale out SVD #AI2SAIS
- 20. #AI2SAIS 𝐴′ 𝑀 𝑠𝑎𝑚𝑝𝑙𝑒𝑠 𝑛 ≪ 𝑁 𝑠𝑖𝑛𝑔𝑢𝑙𝑎𝑟 𝑓𝑒𝑎𝑡𝑢𝑟𝑒𝑠 𝑀 𝑙𝑎𝑏𝑒𝑙𝑠 𝑦 𝐴 = UΣ𝑉′ 𝐴′ = 𝐴𝑉 Use the new matrix A’ to solve supervised learning problems using SVM, Logistic Regression, Decision Trees, Neural Networks etc.
- 23. #AI2SAIS 𝐴′ 𝑀 𝑠𝑎𝑚𝑝𝑙𝑒𝑠 𝑛 ≪ 𝑁 𝑠𝑖𝑛𝑔𝑢𝑙𝑎𝑟 𝑓𝑒𝑎𝑡𝑢𝑟𝑒𝑠 𝐴 = UΣ𝑉′ 𝐴′ = 𝐴𝑉 Use the new matrix A’ for clustering, nearest-neighbors, anomaly detection
- 24. 𝐴 → 𝑀 𝑥 𝑁 𝐵 −> 2𝑛 𝑥 𝑁 𝑈 Σ 𝑉′ Zeros K Randomized SVD 𝐵 = 𝛼𝑉′ Randomized SVD 𝑛 𝑛 K𝑛 𝑛 𝐵 = 𝛼𝑉′ n rows from A SVD on streaming data #AI2SAIS
- 25. #AI2SAIS
- 27. Steps Create a data matrix A using normal images and compute SVD Image 1 Image 2 Image N #AI2SAIS
- 28. Anomaly Score For a new image y compute anomaly score = | 𝐼 − 𝑉𝑉E 𝑦 | Intuition: Find the closest point to y that can be formed as a linear combination of vectors in V #AI2SAIS
- 29. 𝐴 → 𝑀 𝑥 𝑁 𝐵 −> 2𝑛 𝑥 𝑁 𝑈 Σ 𝑉′ Zeros K Randomized SVD 𝐵 = 𝛼𝑉′ Randomized SVD 𝑛 𝑛 K𝑛 𝑛 𝐵 = 𝛼𝑉′ n rows from A Anomaly detection #AI2SAIS Anomaly
- 31. AI to assist cardiologists #AI2SAIS
- 32. Problem formulation Forward Problem Inverse problem #AI2SAIS ab ac =Δ𝑢 + 𝑢(𝑢 − 𝑎)(𝑢 − 1) Solving for u Given a ab ac =Δ𝑢 + 𝑢(𝑢 − 𝑎)(𝑢 − 1) Given u (ECG) Solving for a
- 35. SVD based solver 100x faster! #AI2SAIS Use SVD to exploit the structure of the problem and design solvers ab ac =Δ𝑢 + 𝑢(𝑢 − 𝑎)(𝑢 − 1) Output ECG Solving for a
- 36. Outline • Introduction • Making sense of SVD • One algorithm to rule them all • Scale out SVD #AI2SAIS
- 37. SVD on 14M x 1M matrix #AI2SAIS
- 38. Distributed SVD on 1008 cores #AI2SAIS
- 39. #AI2SAIS