Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Flink Forward Berlin 2017: Trevor Grant - Do I know you? Real time facial recognition with an Apache Stack

238 views

Published on

Facial recognition in production is difficult because neural networks are slow and expensive to train, and must be retrained to recognize new faces added to the set. Older approaches which address these issues such as eigenfaces exist don’t scale as they require a matrix decomposition. Apache Mahout offers a distributed singular value decomposition method, which scales to matrices of arbitrary sizes on Apache Spark, making it possible to use the older yet still powerful Eigenfaces approach to recognize and add new faces in near real time (with the help of Apache Solr). In this talk we present a full stack lambda-style facial recognition system. The offline component uses Mahout/Spark to compute the eigenfaces. The online component leverages Apache Flink to identify faces in an image, decompose the face into a linear combination of the eigenfaces, search for a matching face using SOLR, and if no match is found add the face as a “new face”.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Flink Forward Berlin 2017: Trevor Grant - Do I know you? Real time facial recognition with an Apache Stack

  1. 1. Twitter: @rawkintrevo
  2. 2. Twitter: @rawkintrevo About me- • Trevor Grant • PMC Apache Mahout/Apache Streams • Open Source Evangelist and AI Engineer @IBM
  3. 3. Twitter: @rawkintrevo Simple Cylon • OpenCV for Facial Detection • Mahout / Eigenfaces for Facial Recognition • Problems Image Credit: Electric Alivia https://i.pinimg.com/736x/6a/be/0d/6abe0da5907a5f4923b0c94389a02ddc--pretty-tattoos-pin-up-tattoos.jpg
  4. 4. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  5. 5. Twitter: @rawkintrevo Mahout Decompose FaceRect into Linear Comb Vec 321 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Eigenfaces Ordinary Least Squares Linear Combination of Eigenfaces
  6. 6. Twitter: @rawkintrevo Search k-Length Vector (k is number of Eigenfaces ~ 130)
  7. 7. Twitter: @rawkintrevo That’s Apollo Solr Returns Closest Match Drone says “hi” or shoots to kill
  8. 8. Twitter: @rawkintrevo Cascade Filter Overview • Not a Neural Net (NNets are too slow) • OpenCV -> Haar Cascades are much more efficient • Scans for areas that match certain patterns.
  9. 9. Twitter: @rawkintrevo Cascade Filter Overview
  10. 10. Twitter: @rawkintrevo Cascade Filter
  11. 11. Twitter: @rawkintrevo Cascade Filter (Areawise)
  12. 12. Twitter: @rawkintrevo Eigenfaces
  13. 13. Twitter: @rawkintrevo Eigenfaces
  14. 14. Twitter: @rawkintrevo Eigenfaces (Pixelwise)
  15. 15. Twitter: @rawkintrevo Eigenfaces (Pixelwise)
  16. 16. Twitter: @rawkintrevo Eigenfaces (Pixelwise)
  17. 17. Twitter: @rawkintrevo Eigenfaces (Pixelwise) 22 85 54 123 56 187 92 91 111 204 103 245 8 247 155 212 239 87 99 84
  18. 18. Twitter: @rawkintrevo Eigenfaces (Pixelwise) 22 85 54 123 56 187 92 91 111 204 103 245 8 247 155 212 239 87 99 84
  19. 19. Twitter: @rawkintrevo Eigenfaces
  20. 20. Twitter: @rawkintrevo Eigenfaces
  21. 21. Twitter: @rawkintrevo Eigenfaces Matrix of Faces ith Image jth Pixel Position
  22. 22. Twitter: @rawkintrevo Eigenfaces: Singular Value Decomposition Matrix of FacesU Vx =
  23. 23. Twitter: @rawkintrevo Eigenfaces: Matrix V Matrix of FacesU V (Eigenfaces)x =
  24. 24. Twitter: @rawkintrevo Eigenfaces: Matrix U Matrix of FacesU Vx = Linear combinations of Eigenfaces required to form the Nth Face = 2.456 x - 7.2345 x + 0.4125 x
  25. 25. Twitter: @rawkintrevo New Faces y V Transpose (each column is eigenface)
  26. 26. Twitter: @rawkintrevo New Faces y X Simple Regression (OLS) Ordinary Least Squares β
  27. 27. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  28. 28. Twitter: @rawkintrevo Mahout Decompose FaceRect into Linear Comb Vec 321 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Eigenfaces Ordinary Least Squares Linear Combination of Eigenfaces
  29. 29. Twitter: @rawkintrevo Search k-Length Vector (k is number of Eigenfaces ~ 130)
  30. 30. Twitter: @rawkintrevo That’s Apollo Solr Returns Closest Match Drone says “hi” or shoots to kill
  31. 31. Twitter: @rawkintrevo Problems with Simple Cylon • OpenCV/Haar Cascades inconsistently boxes face • (Eigenfaces requires images be centered and same size as training) • Adds face every time angle changes (ideally we want multiple combinations assosciated with each person) • Will insert “bad” faces, and “not” faces. • To much ”radio chatter” with Kafka (esp with “who else is around” context clues)
  32. 32. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  33. 33. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  34. 34. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  35. 35. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  36. 36. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  37. 37. Twitter: @rawkintrevo OpenCV/ Haar Cascade Filter inconsistent
  38. 38. Twitter: @rawkintrevo Canopy Clustering • Create N Second Window • Cluster Faces in Window • Quick dirty clustering- but effective. • First point is “center” • All points within distance t2 are “in that cluster. • If a point is not within t2 of any cluster- it becomes a new cluster center.
  39. 39. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame t2= max square width
  40. 40. Twitter: @rawkintrevo Canopy Clustering To Remove “Ghost” Faces t2= max square widthFirst rect – new cluster Second Rect- within one width of first rect (same cluster) Third Rect- within one width of first rect (same cluster) Forth Rect- NOT within one width of first rect (new cluster) Fifth Rect- within one width of first rect (same cluster) Finally- any cluster with less than two entities in windows gets filtered out.
  41. 41. Twitter: @rawkintrevo Canopy Clustering To Remove “Ghost” Faces
  42. 42. Twitter: @rawkintrevo Remaining Faces (Stupid Way) HUMAN-0001 HUMAN-0002 HUMAN-0003 HUMAN-0004
  43. 43. Twitter: @rawkintrevo Remaining Faces (Smarter Way) HUMAN-0001 HUMAN-0001 HUMAN-0001 HUMAN-0001 All faces from a given cluster are likely from the same entity
  44. 44. Twitter: @rawkintrevo Flink Enabled • Windowing lets us use “Intelligence” • Removing Ghost Faces • Using Context Clues (Person in one part of frame doesn’t magically become someone else)
  45. 45. Twitter: @rawkintrevo Flink Plan
  46. 46. Twitter: @rawkintrevo Definition of Intelligence • Merriam Webster: • a (1) : the ability to learn or understand or to deal with new or trying situations : reason; also : the skilled use of reason • (2) : the ability to apply knowledge to manipulate one's environment or to think abstractly as measured by objective criteria (such as tests) • Oxford • The ability to acquire and apply knowledge and skills. • Contested subject among researches…
  47. 47. Twitter: @rawkintrevo Artificial Intelligence • Collect information from environment • Adapt to changing environment • Learn and understand / gain knowledge / improve skills • Apply knowledge
  48. 48. Twitter: @rawkintrevo IoT AI - A Diagram Edge Device Edge Device Edge Device … Edge devices produce data (sensors, drones, cars, etc.)
  49. 49. Twitter: @rawkintrevo A Diagram: Collecting from environment Edge Device Edge Device Edge Device … “Something” needs to collect and process the data
  50. 50. Twitter: @rawkintrevo A Diagram: Learn, understand, adapt, affect Edge Device Edge Device Edge Device … That “Something” needs to support learning and decision frame works Thoughts Actions
  51. 51. Twitter: @rawkintrevo A Diagram: Bonuses Edge Device Edge Device Edge Device … Perhaps it can learns offline and remember as well. Thoughts Actions Memories learning
  52. 52. Twitter: @rawkintrevo A Diagram: Apache AF Edge Device Edge Device Edge Device … Perhaps it can learns offline and remember as well. Thoughts Actions Memories learning
  53. 53. Twitter: @rawkintrevo A Diagram: Apache AF (Lambda) Thoughts Memories learning ”Stream of Conscious” ”Short Term Memory” * Windowing and CEP *
  54. 54. Twitter: @rawkintrevo A Diagram: Apache AF (Kappa) Thoughts Memories ”Stream of Conscious” ”Short Term Memory” * Windowing and CEP * learning
  55. 55. Twitter: @rawkintrevo Technically “Borg-style” AI, not Cylons • A finer technical point for those familiar with the Cylons and the Borg • “Hive Mind” Architecture
  56. 56. Twitter: @rawkintrevo NEW HUMAN-0001 Solr Returns Closest Match Drone says “hi” or shoots to kill OH, hai HUMAN-0001
  57. 57. Twitter: @rawkintrevo Shape of things to come. ”Science Fiction” of 10 years ago, today is domain of hobbyists Demo presented here is “Science Fair” grade AI. Vlad Putin’s recently talking about “it is undesirable for anyone to monopolize AI”. (Yay Apache!)

×