Image Classification and Retrieval logic
•
0 likes•425 views
strategy for implementing image classification pipeline with spark
Recommended
Recommended
More Related Content
What's hot
What's hot (9)
Viewers also liked
Viewers also liked (6)
Similar to Image Classification and Retrieval logic
Similar to Image Classification and Retrieval logic (20)
More from Gianvito Siciliano
Recently uploaded
Recently uploaded (20)
Image Classification and Retrieval logic
- 1. IMAGE CLASSIFICATION PIPELINE CF x IIF classification logic strategy
- 2. DONE • Information Retrieval (query the dataset) • TF-IDF • Machine Learning (classify new instances) • CF-IIF • Technologies: OpenIMAJ / Spark • Image Pipeline • Logica di classificazione • Implementazione in java (e test)
- 3. DA RIVEDERE • KNN with KDtree • Cosine similarities, and distance metrics • Improve cf-iif extractor (logica in spark) • Tuning with hyper parameter • Reduce features space: SVD (scegliendo lo 0,01% di cluster sono 6000+ features)
- 4. TODO • Re-engineering in spark • testare differenti SIFT features (pyramid?) • sostituire KNN con CNeuralNetworks • (GraphLab/Deep4J)
- 5. FEATURE EXTRACTOR LOADER MODEL TEST PREDICTION Build a classifier, based on salient-feature vocabulary created from the dataset 1. load images dataset inherent 3 distinct class 2. extract local features from each image, 3. create the codebook for classification 4. train and test the model IMAGE PIPELINE
- 6. FEATURES EXTRACTION • Extracting features is where cool image processing happens, and represent the key part of the pipeline. • Feature extractors make featureVector to represent an image in a vector space. • In a visual application systems, you need robust features for classification and search FEATURE EXTRACTOR IMAGES MODEL TEST PREDICTION
- 7. WHY FEATURES EXTRACTION • Typically, images features are numerical vectors that can be used with ML techniques. • FeatureVectors can be compared by measuring a distance • Is useful to groups similar similar features and reduce the dimension space • Indexing for Information Retrieval FEATURE EXTRACTOR
- 8. IMAGE FEATURES Image features can be: GLOBAL: single featureVector comes out FEATURE EXTRACTOR
- 9. IMAGE FEATURES Image features can be: GRID BASED: multiple featureVectors from each image blocks FEATURE EXTRACTOR
- 10. IMAGE FEATURES Image features can be: LOCAL: multiple featureVector from interest points (different from each image) FEATURE EXTRACTOR
- 11. IMAGE FEATURES Image features can be: SEGMENTED: multiple featureVector from region point (different from each image) FEATURE EXTRACTOR
- 12. SIFT FEATURES Scale Invariant FeatureTransformation is an advanced technique to extract local features from interest points of an images, that are invariant to rotation, lighting changes… Builds on the idea of a local gradient histogram by incorporating spatial binning which in essence creates multiple gradient histograms about interest points and appends them all together Standard SIFT geometry appends a spatial 4x4 grid of histograms with 8 orientations Leading a 128-dims features vector which is highly discriminant and robust128
- 13. OPENIMAJ • Image processing Java libraries, that includes a lot of feature extractor and other utilities for visual applications: • DoGSIFT • DenseSIFT • PyramidSIFT • …
- 14. SPARK • Spark is used to scale out the applications: big dataset, high feature dimensions… • MLlib
- 16. sift DATASET KNN KMEANS QUANTIZAT OR my EXTRACTOR IMAGE PIPELINE PATH LABEL img1 car img3 bycicle img17 car PATH LABEL siftFEAT img1 car img3 bycicle 128 6000+ PREDICTIONS 1) SIFT features extraction
- 17. sift EXTRACTOR DATASET KNN KME my EXTRACTOR IMAGE PIPELINE X siftFEAT 128 6000+ #images CLUSTER 128 PREDICTIONS 2) features quantisation
- 18. sift EXTRACTOR DATASET KNN KME my EXTRACTOR IMAGE PIPELINE X siftFEAT 128 6000+ #images CLUSTER 128 fixed K = 300 PREDICTIONS 2) features reduction
- 19. sift EXTRACTOR DATASET KNN KMEANS QUANTIZAT OR my IMAGE PIPELINE A questo punto associamo a ciascuna immagine un “cluster vector” (analogo del keyword vector nel testo) Cw=<w1, w2,...., wn> n=|C| Per ciascuna immagine, e per ciascun cluster di descrittori j, il corrispondente peso wj sara’ il prodotto di due fattori: • Cluster Frequency: percentuale di punti di quella immagine che sono stati mappati nel cluster j • Inverse Image Frequency: logaritmo del rapporto tra la cardinalita’ del database e il numero di immagini in cui descrittori mappati in quel cluster sono presenti PREDICTIONS
- 20. sift EXTRACTOR DATASET KNN KMEANS QUANTIZAT OR my IMAGE PIPELINE La logica è ripresa dalTF-IDF molto comune in text retrieval. Un cluster diventa discriminante se contiene poche immagini! PREDICTIONS
- 21. sift EXTRACTOR DATASET KNN KMEANS QUANTIZAT OR my IMAGE PIPELINE La logica è ripresa dalTF-IDF molto comune in text retrieval. Un cluster diventa discriminante se contiene poche immagini! Features-space: si passa dai descrittori SIFT ai cluster-vector PREDICTIONS
- 22. IMAGE PIPELINE PATH LABEL siftFEAT img can img gatt 128 CLUSTER V images my EXTR PATH LABEL cfiifFEAT img can img gatt
- 23. sift EXTRACTOR DATASET KNN KMEANS QUANTIZAT OR my IMAGE PIPELINE PATH LABEL myFEAT img1 bycicle img3 car #clusters #images PREDICTIONS 3) features transformation
- 24. sift EXTRACTOR DATASET KNN KMEANS QUANTIZAT OR my IMAGE PIPELINE PATH LABEL myFEAT img1 bycicle img3 car #clusters #images La cella wj del cluster-vector dell’imm.A rappresenta il suo peso nel cluster J PREDICTIONS
- 25. sift EXTRACTOR DATASET KNN KMEANS QUANTIZAT OR my IMAGE PIPELINE Tutte le immagini sono rappresentate dai cf-iif vector!PREDICTIONS PATH LABEL myFEAT img1 bycicle img3 car #clusters #images
- 26. sift EXTRACTOR DATASET KNN KMEANS QUANTIZAT OR my IMAGE PIPELINE This is own vocabulary! PREDICTIONS PATH LABEL myFEAT img1 bycicle img3 car
- 27. KNN sift EXTRACTOR KNN KMEANS QUANTIZAT OR my EXTRACTOR IMAGE PIPELINE cfiifFEAT TEST TRAIN 4) train a knn classifier
- 28. sift EXTRACTOR KNN KMEANS QUANTIZAT OR my EXTRACTOR IMAGE PIPELINE TEST TRAIN 4) learn the model KNN
- 29. sift EXTRACTOR KNN KMEANS QUANTIZAT OR my EXTRACTOR IMAGE PIPELINE PREDICTIONS TEST TRAIN myFEAT KNN 4) test LABEL car bycicle motorbike
- 30. sift EXTRACTOR KNN KMEANS QUANTIZAT OR my EXTRACTOR IMAGE PIPELINE PREDICTIONS TEST TRAIN myFEAT KNN This is the most similar! LABEL car bycicle motorbike LABEL car
- 31. sift EXTRACTOR KNN KMEANS QUANTIZAT OR my EXTRACTOR IMAGE PIPELINE PREDICTIONS TEST TRAIN myFEAT KNN This is the most similar! LABEL car bycicle motorbike LABEL car
- 32. IMAGE PIPELINES • Image Features Extraction: find interest points and extract discriminative and robust features • OPENIMAJ multiple algortihms • Learn large codebooks from features • Train the model (KNN) • SPARK scalable models (3 days to train a KNN model on 15 images with openimaj) • SPARK multiple models (Bayes models, Neural Networks) PAIN POINTS
- 33. • Efficient Nearest Neighbour Search (test) • KDTree • HyperParametersTuning (in own pipeline used for Kmeans, CFIIF and KNN) IMAGE PIPELINESPAIN POINTS
- 34. • Image Features can be used to match music! • Extractors can be used to find objects! (Face Detection) OPENIMAJ++