Image Compression and Semantic Search Theory
•
0 likes•7,240 views
Singular Value Decomposition (SVD) is a matrix decomposition technique developed during the 18th century and has been in use ever since. SVD has applications in several areas including image processing, natural language processing (NLP), genomics, and data compression. In NLP context, SVD is called latent semantic indexing (LSI) and used for concept based search and topic modeling. In this talk, we will describe the math and intuition behind eigenvalues, eigenvectors and their relation to SVD. We will also discuss specific applications of SVD in image processing and NLP with examples.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Image Compression and Semantic Search Theory
Similar to Image Compression and Semantic Search Theory (6)
Recently uploaded
Recently uploaded (20)
Image Compression and Semantic Search Theory
- 2. Bio • 2016 - → Co-founder, RocketML • 2008 - 2016 → Product Manager, Intel • 2003 - 2008 → Ph.D., University of Pennsylvania • 1999 - 2003 → B. Tech, IIT Madras 4/6/17 RocketML
- 3. • Singular Value Decomposition • Eigenvalue decomposition • Principal Component Analysis • Latent Semantic Analysis • Latent Semantic Index • Proper Orthogonal Decomposition 4/6/17 RocketML
- 5. Use cases across multiple disciplines • Natural Language Processing • Image Processing • Signal Processing • Genomics • Data compression • Search • Recommendation engines • Matrix inversion 4/6/17 RocketML
- 6. Topics • Vectors and Matrices • Singular value decomposition • Image Compression • Semantic search 4/6/17 RocketML
- 7. Vectors 4/6/17 RocketML x1 x2 [2,2] [2,1] x1 x2 x3 [2,2,2] x1, x2, x3, x4, … are features. In NLP, these are n-grams 2D 3D Hyper Space [2,3,3,5, … ]
- 8. Matrix Vector Multiplication 4/6/17 RocketML 2 0 0 1 1 1 = 2 1 x1 x2 [2,1][1,1] Ax x A x Ax 2 0 0 0 1 2 x1 x2 x3 [1,1,2] 1 1 2 2 5 = x1 x2 [2,5] 3D 2D A x Ax Stretching, Rotation Stretching, Rotation, dimension changes
- 9. 4/6/17 RocketML v A v A2v A3v Special vectors Only Stretching, No Rotation
- 11. 4/6/17 RocketML Example 5√5 𝑎𝑛𝑑 1/√5,2/√5 𝑓𝑜𝑟𝑚 𝑎𝑛 𝑒𝑖𝑔𝑒𝑛𝑣𝑎𝑙𝑢𝑒, 𝑒𝑖𝑔𝑒𝑛𝑣𝑒𝑐𝑡𝑜𝑟 𝑝𝑎𝑖𝑟 𝑜𝑓 𝐴 1 2 8 1 = 1 2 1/√5 2/√5 5√5𝐴𝑣 = 𝜆𝑣 5 10 =
- 12. Eigen decomposition for square matrices 4/6/17 RocketML Q is a square matrix whose ith column is the eigenvector qi of A L is a diagonal matrix where ith element is the ith eigenvalue If A is symmetric i.e, A = AT 𝐴 = 𝑄 Λ 𝑄?@ 𝐴 = 𝑄 Λ 𝑄A
- 13. Singular Value Decomposition of a matrix 4/6/17 RocketML 𝑀 = 𝑈 𝑆 𝑉 ∗ 𝑈, 𝑉 𝑎𝑟𝑒 𝑙𝑒𝑓𝑡 𝑎𝑛𝑑 𝑟𝑖𝑔ℎ𝑡 𝑠𝑖𝑛𝑔𝑢𝑙𝑎𝑟 𝑣𝑒𝑐𝑡𝑜𝑟𝑠 S 𝑖𝑠 𝑎 𝑑𝑖𝑎𝑔𝑜𝑛𝑎𝑙 𝑚𝑎𝑡𝑟𝑖𝑥 𝑤𝑖𝑡ℎ 𝑟𝑒𝑎𝑙 𝑠𝑖𝑛𝑔𝑢𝑙𝑎𝑟 𝑣𝑎𝑙𝑢𝑒𝑠 𝑈 𝑈∗ = 𝐼, 𝑉 𝑉∗ = 𝐼
- 14. SVD relation to eigenvalue decomposition 4/6/17 RocketML • 𝐶𝑜𝑙𝑢𝑚𝑛𝑠 𝑜𝑓 𝑉 𝑎𝑟𝑒 𝑒𝑖𝑔𝑒𝑛𝑣𝑒𝑐𝑡𝑜𝑟𝑠 𝑜𝑓 𝑀∗ 𝑀 • 𝐶𝑜𝑙𝑢𝑚𝑛𝑠 𝑜𝑓 𝑈 𝑎𝑟𝑒 𝑒𝑖𝑔𝑒𝑛𝑣𝑒𝑐𝑡𝑜𝑟𝑠 𝑜𝑓 𝑀𝑀∗ • 𝐸𝑙𝑒𝑚𝑒𝑛𝑡𝑠 𝑜𝑓 𝑆 𝑎𝑟𝑒 𝑠𝑞𝑢𝑎𝑟𝑒 𝑟𝑜𝑜𝑡𝑠 𝑜𝑓 𝑛𝑜𝑛 − 𝑧𝑒𝑟𝑜 𝑒𝑖𝑔𝑒𝑛𝑣𝑎𝑙𝑢𝑒𝑠 𝑜𝑓 𝑀∗ 𝑀 𝑜𝑟 𝑀𝑀∗ 𝑀∗ 𝑀 = 𝑉 Σ∗ 𝑈∗ 𝑈 Σ𝑉∗ = 𝑉 Σ∗ Σ 𝑉∗ 𝑀 𝑀∗ = 𝑈 Σ𝑉∗ 𝑉 Σ∗ 𝑈∗ = 𝑈 ΣΣ∗ 𝑈∗
- 15. Dimension reduction 4/6/17 RocketML Image 255 255255 255 255 255255 255 255 255255 255 255 255255 255 Matrix 200 pixels 200 pixels
- 16. Dimension reduction 4/6/17 RocketML 255 255255 255 255 255255 255 255 255255 255 255 255255 255 c c c c 51000 c c c c = U UTS C = -0.0707, Rank of this matrix = 1
- 17. Reconstruction 4/6/17 RocketML 450x400 pixels 90 90 89 90 … 90 90 89 90 … 123 94 101
- 20. Reconstruction using few singular values 4/6/17 RocketML 𝑈[: , 1: 2] 𝑆[1: 2] 𝑉[: , 1: 2] 𝑇 𝑈[: , 1: 3] 𝑆[1: 3] 𝑉[: , 1: 3] 𝑇
- 21. More singular values 4/6/17 RocketML 𝑈[: , 1: 20] ∗ 𝑆[1: 20] ∗ 𝑉[: , 1: 20] 𝑈[: , 1: 200] ∗ 𝑆[1: 200] ∗ 𝑉[: , 1: 200]
- 22. Normalize Cumulative Sum 4/6/17 RocketML 𝑆 = [ 𝜎 ] ^ 𝑠_ = 1 𝑆 [ 𝜎_ _?@ ^
- 25. Semantic Search • Take a collection of the following documents • Shipment of gold damaged in a fire. • Delivery of silver arrived in a silver truck • Shipment of gold arrived in a truck • Problem: Rank these documents for the query “gold silver truck” 4/6/17 RocketML
- 26. Step 1: Bag of words 4/6/17 RocketML • Shipment of gold damaged in a fire. • Delivery of silver arrived in a silver truck • Shipment of gold arrived in a truck 𝐴 = 11 𝑥 3 𝑎 𝑎𝑟𝑟𝑖𝑣𝑒𝑑 𝑑𝑎𝑚𝑎𝑔𝑒𝑑 𝑑𝑒𝑙𝑖𝑣𝑒𝑟𝑦 𝑓𝑖𝑟𝑒 𝑔𝑜𝑙𝑑 𝑖𝑛 𝑜𝑓 𝑠ℎ𝑖𝑝𝑚𝑒𝑛𝑡 𝑠𝑖𝑙𝑣𝑒𝑟 𝑡𝑟𝑢𝑐𝑘 1 1 1 0 1 1 1 0 0 0 1 0 1 0 0 1 0 1 1 1 1 1 1 1 1 0 1 0 2 0 0 1 1 𝐴 = Words Sentences
- 27. 4/6/17 RocketML 1 1 1 0 1 1 1 0 0 0 1 0 1 0 0 1 0 1 1 1 1 1 1 1 1 0 1 0 2 0 0 1 1 𝐴 = Step 2: Singular Value Decomposition (SVD) 𝑈 𝑖𝑠 11𝑥 3 𝑜𝑟𝑡ℎ𝑜𝑛𝑜𝑟𝑚𝑎𝑙 𝑚𝑎𝑡𝑟𝑖𝑥 𝑆 𝑖𝑠 3𝑥3 𝑚𝑎𝑡𝑟𝑖𝑥 𝑉 𝑖𝑠 3𝑥3 𝑚𝑎𝑡𝑟𝑖𝑥 11𝑥 3 3 𝑥 3 3 𝑥 3 𝑈 Σ Vc=
- 28. 4/6/17 RocketML 1 1 1 0 1 1 1 0 0 0 1 0 1 0 0 1 0 1 1 1 1 1 1 1 1 0 1 0 2 0 0 1 1 𝐴 = Step 3: Truncated SVD 𝑈′ 𝑖𝑠 11𝑥 2 𝑜𝑟𝑡ℎ𝑜𝑛𝑜𝑟𝑚𝑎𝑙 𝑚𝑎𝑡𝑟𝑖𝑥 𝑆′ 𝑖𝑠 2𝑥2 𝑚𝑎𝑡𝑟𝑖𝑥 𝑉′ 𝑖𝑠 2𝑥2 𝑚𝑎𝑡𝑟𝑖𝑥 11𝑥 2 2 𝑥 2 2 𝑥 2 𝑈′ Σ′ 𝑉eA=
- 29. Step 4: Find new query vector in reduced 2- dimension space 4/6/17 RocketML “𝑔𝑜𝑙𝑑 𝑠𝑖𝑙𝑣𝑒𝑟 𝑡𝑟𝑢𝑐𝑘” 0 0 0 0 0 1 0 0 0 1 1 q = 𝑎 𝑎𝑟𝑟𝑖𝑣𝑒𝑑 𝑑𝑎𝑚𝑎𝑔𝑒𝑑 𝑑𝑒𝑙𝑖𝑣𝑒𝑟𝑦 𝑓𝑖𝑟𝑒 𝑔𝑜𝑙𝑑 𝑖𝑛 𝑜𝑓 𝑠ℎ𝑖𝑝𝑚𝑒𝑛𝑡 𝑠𝑖𝑙𝑣𝑒𝑟 𝑡𝑟𝑢𝑐𝑘 𝑞e = 𝑞 𝑈eA 𝑆e?@ 𝑞′ = [−0.21, −0.1821]
- 30. Step 5: Rank documents based on cosine similarity 4/6/17 RocketML −0.4945 −0.6458 −0.5817 0.6492 −0.7914 0.2469 𝑞′ = [−0.21, −0.1821] Sentences [ −0.0541 0.9910 0.4478] 1 23
- 31. Search Results for “gold silver truck” using LSI 1. Delivery of silver arrived in a silver truck 2. Shipment of gold arrived in a truck 3. Shipment of gold damaged in a fire. 4/6/17 RocketML Semantic Search or Concept based search
- 33. Variations • Singular Value Decomposition • Eigenvalue decomposition • Principal Component Analysis • Latent Semantic Analysis • Latent Semantic Index • Proper orthogonal Decomposition 4/6/17 RocketML
- 34. Methods to compute SVD • Arnoldi method with explicit restart and deflation • Lanczos with explicit restart and deflation • Krylov-Schur • Generalized Davidson • Randomized SVD • Frequent Directions 4/6/17 RocketML Matrix Computations (Johns Hopkins Studies in Mathematical Sciences)(3rd Edition) 3rd Edition by Gene H. Golub (Author), Charles F. Van Loan (Author)
- 35. Packages • Numpy • Scikit-learn • Gensim • ARPACK • LAPACK 4/6/17 RocketML
- 36. References • An Introduction to the Conjugate Gradient Method Without the Agonizing Pain, Jonathan Richard Shewchuk • An introduction to Latent Semantic Analysis, Thomas K Landauer et. Al • Latent Semantic Indexing (LSI) An Example • Matrix Computations, Gene Golub and Charles F. Van Loan 4/6/17 RocketML
- 38. 4/6/17 RocketML
- 39. 4/6/17 RocketML