Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Twitter: @rawkintrevo
Twitter: @rawkintrevo
About me-
• Trevor Grant
• PMC Apache Mahout/Apache Streams
• Open Source Evangelist and AI Engineer...
Twitter: @rawkintrevo
Simple Cylon
• OpenCV for Facial Detection
• Mahout / Eigenfaces for Facial Recognition
• Problems
I...
Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame
Twitter: @rawkintrevo
Mahout Decompose FaceRect into Linear Comb Vec
321
4 5 6
7 8 9
1
2
3
4
5
6
7
8
9
Eigenfaces
Ordinary...
Twitter: @rawkintrevo
Search k-Length Vector
(k is number of Eigenfaces ~ 130)
Twitter: @rawkintrevo
That’s Apollo
Solr Returns Closest Match
Drone says “hi” or shoots to kill
Twitter: @rawkintrevo
Cascade Filter Overview
• Not a Neural Net (NNets are too slow)
• OpenCV -> Haar Cascades are much m...
Twitter: @rawkintrevo
Cascade Filter Overview
Twitter: @rawkintrevo
Cascade Filter
Twitter: @rawkintrevo
Cascade Filter (Areawise)
Twitter: @rawkintrevo
Eigenfaces
Twitter: @rawkintrevo
Eigenfaces
Twitter: @rawkintrevo
Eigenfaces (Pixelwise)
Twitter: @rawkintrevo
Eigenfaces (Pixelwise)
Twitter: @rawkintrevo
Eigenfaces (Pixelwise)
Twitter: @rawkintrevo
Eigenfaces (Pixelwise)
22 85 54 123
56 187 92 91
111 204 103 245
8 247 155 212
239 87 99 84
Twitter: @rawkintrevo
Eigenfaces (Pixelwise)
22 85 54 123
56 187 92 91
111 204 103 245
8 247 155 212
239 87 99 84
Twitter: @rawkintrevo
Eigenfaces
Twitter: @rawkintrevo
Eigenfaces
Twitter: @rawkintrevo
Eigenfaces
Matrix of Faces
ith Image
jth Pixel Position
Twitter: @rawkintrevo
Eigenfaces: Singular Value Decomposition
Matrix of FacesU Vx =
Twitter: @rawkintrevo
Eigenfaces: Matrix V
Matrix of FacesU V (Eigenfaces)x =
Twitter: @rawkintrevo
Eigenfaces: Matrix U
Matrix of FacesU Vx =
Linear combinations of Eigenfaces required to form the Nt...
Twitter: @rawkintrevo
New Faces
y
V Transpose
(each column is
eigenface)
Twitter: @rawkintrevo
New Faces
y X
Simple Regression (OLS)
Ordinary Least Squares
β
Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame
Twitter: @rawkintrevo
Mahout Decompose FaceRect into Linear Comb Vec
321
4 5 6
7 8 9
1
2
3
4
5
6
7
8
9
Eigenfaces
Ordinary...
Twitter: @rawkintrevo
Search k-Length Vector
(k is number of Eigenfaces ~ 130)
Twitter: @rawkintrevo
That’s Apollo
Solr Returns Closest Match
Drone says “hi” or shoots to kill
Twitter: @rawkintrevo
Problems with Simple Cylon
• OpenCV/Haar Cascades inconsistently boxes face
• (Eigenfaces requires i...
Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame
Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame
Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame
Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame
Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame
Twitter: @rawkintrevo
OpenCV/ Haar Cascade Filter inconsistent
Twitter: @rawkintrevo
Canopy Clustering
• Create N Second Window
• Cluster Faces in Window
• Quick dirty clustering- but e...
Twitter: @rawkintrevo
Open CV Detects Faces in Video Frame
t2= max square width
Twitter: @rawkintrevo
Canopy Clustering To Remove “Ghost” Faces
t2= max square widthFirst rect – new cluster
Second Rect- ...
Twitter: @rawkintrevo
Canopy Clustering To Remove “Ghost” Faces
Twitter: @rawkintrevo
Remaining Faces (Stupid Way)
HUMAN-0001
HUMAN-0002
HUMAN-0003
HUMAN-0004
Twitter: @rawkintrevo
Remaining Faces (Smarter Way)
HUMAN-0001
HUMAN-0001
HUMAN-0001
HUMAN-0001
All faces from a given clu...
Twitter: @rawkintrevo
Flink Enabled
• Windowing lets us use “Intelligence”
• Removing Ghost Faces
• Using Context Clues (P...
Twitter: @rawkintrevo
Flink Plan
Twitter: @rawkintrevo
Definition of Intelligence
• Merriam Webster:
• a (1) : the ability to learn or understand or to dea...
Twitter: @rawkintrevo
Artificial Intelligence
• Collect information from environment
• Adapt to changing environment
• Lea...
Twitter: @rawkintrevo
IoT AI - A Diagram
Edge
Device
Edge
Device
Edge
Device
…
Edge devices produce data (sensors, drones,...
Twitter: @rawkintrevo
A Diagram: Collecting from environment
Edge
Device
Edge
Device
Edge
Device
…
“Something” needs to co...
Twitter: @rawkintrevo
A Diagram: Learn, understand, adapt, affect
Edge
Device
Edge
Device
Edge
Device
…
That “Something” n...
Twitter: @rawkintrevo
A Diagram: Bonuses
Edge
Device
Edge
Device
Edge
Device
…
Perhaps it can learns offline and remember ...
Twitter: @rawkintrevo
A Diagram: Apache AF
Edge
Device
Edge
Device
Edge
Device
…
Perhaps it can learns offline and remembe...
Twitter: @rawkintrevo
A Diagram: Apache AF (Lambda)
Thoughts Memories
learning
”Stream of Conscious”
”Short Term Memory”
*...
Twitter: @rawkintrevo
A Diagram: Apache AF (Kappa)
Thoughts
Memories
”Stream of Conscious”
”Short Term Memory”
* Windowing...
Twitter: @rawkintrevo
Technically “Borg-style” AI, not Cylons
• A finer technical point for those familiar with the Cylons...
Twitter: @rawkintrevo
NEW HUMAN-0001
Solr Returns Closest Match
Drone says “hi” or shoots to kill
OH, hai HUMAN-0001
Twitter: @rawkintrevo
Shape of things to come.
”Science Fiction” of 10 years ago, today is domain of
hobbyists
Demo presen...
You’ve finished this document.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next

Share

Flink Forward Berlin 2017: Trevor Grant - Do I know you? Real time facial recognition with an Apache Stack

Facial recognition in production is difficult because neural networks are slow and expensive to train, and must be retrained to recognize new faces added to the set. Older approaches which address these issues such as eigenfaces exist don’t scale as they require a matrix decomposition. Apache Mahout offers a distributed singular value decomposition method, which scales to matrices of arbitrary sizes on Apache Spark, making it possible to use the older yet still powerful Eigenfaces approach to recognize and add new faces in near real time (with the help of Apache Solr). In this talk we present a full stack lambda-style facial recognition system. The offline component uses Mahout/Spark to compute the eigenfaces. The online component leverages Apache Flink to identify faces in an image, decompose the face into a linear combination of the eigenfaces, search for a matching face using SOLR, and if no match is found add the face as a “new face”.

  • Be the first to like this

Flink Forward Berlin 2017: Trevor Grant - Do I know you? Real time facial recognition with an Apache Stack

  1. 1. Twitter: @rawkintrevo
  2. 2. Twitter: @rawkintrevo About me- • Trevor Grant • PMC Apache Mahout/Apache Streams • Open Source Evangelist and AI Engineer @IBM
  3. 3. Twitter: @rawkintrevo Simple Cylon • OpenCV for Facial Detection • Mahout / Eigenfaces for Facial Recognition • Problems Image Credit: Electric Alivia https://i.pinimg.com/736x/6a/be/0d/6abe0da5907a5f4923b0c94389a02ddc--pretty-tattoos-pin-up-tattoos.jpg
  4. 4. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  5. 5. Twitter: @rawkintrevo Mahout Decompose FaceRect into Linear Comb Vec 321 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Eigenfaces Ordinary Least Squares Linear Combination of Eigenfaces
  6. 6. Twitter: @rawkintrevo Search k-Length Vector (k is number of Eigenfaces ~ 130)
  7. 7. Twitter: @rawkintrevo That’s Apollo Solr Returns Closest Match Drone says “hi” or shoots to kill
  8. 8. Twitter: @rawkintrevo Cascade Filter Overview • Not a Neural Net (NNets are too slow) • OpenCV -> Haar Cascades are much more efficient • Scans for areas that match certain patterns.
  9. 9. Twitter: @rawkintrevo Cascade Filter Overview
  10. 10. Twitter: @rawkintrevo Cascade Filter
  11. 11. Twitter: @rawkintrevo Cascade Filter (Areawise)
  12. 12. Twitter: @rawkintrevo Eigenfaces
  13. 13. Twitter: @rawkintrevo Eigenfaces
  14. 14. Twitter: @rawkintrevo Eigenfaces (Pixelwise)
  15. 15. Twitter: @rawkintrevo Eigenfaces (Pixelwise)
  16. 16. Twitter: @rawkintrevo Eigenfaces (Pixelwise)
  17. 17. Twitter: @rawkintrevo Eigenfaces (Pixelwise) 22 85 54 123 56 187 92 91 111 204 103 245 8 247 155 212 239 87 99 84
  18. 18. Twitter: @rawkintrevo Eigenfaces (Pixelwise) 22 85 54 123 56 187 92 91 111 204 103 245 8 247 155 212 239 87 99 84
  19. 19. Twitter: @rawkintrevo Eigenfaces
  20. 20. Twitter: @rawkintrevo Eigenfaces
  21. 21. Twitter: @rawkintrevo Eigenfaces Matrix of Faces ith Image jth Pixel Position
  22. 22. Twitter: @rawkintrevo Eigenfaces: Singular Value Decomposition Matrix of FacesU Vx =
  23. 23. Twitter: @rawkintrevo Eigenfaces: Matrix V Matrix of FacesU V (Eigenfaces)x =
  24. 24. Twitter: @rawkintrevo Eigenfaces: Matrix U Matrix of FacesU Vx = Linear combinations of Eigenfaces required to form the Nth Face = 2.456 x - 7.2345 x + 0.4125 x
  25. 25. Twitter: @rawkintrevo New Faces y V Transpose (each column is eigenface)
  26. 26. Twitter: @rawkintrevo New Faces y X Simple Regression (OLS) Ordinary Least Squares β
  27. 27. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  28. 28. Twitter: @rawkintrevo Mahout Decompose FaceRect into Linear Comb Vec 321 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Eigenfaces Ordinary Least Squares Linear Combination of Eigenfaces
  29. 29. Twitter: @rawkintrevo Search k-Length Vector (k is number of Eigenfaces ~ 130)
  30. 30. Twitter: @rawkintrevo That’s Apollo Solr Returns Closest Match Drone says “hi” or shoots to kill
  31. 31. Twitter: @rawkintrevo Problems with Simple Cylon • OpenCV/Haar Cascades inconsistently boxes face • (Eigenfaces requires images be centered and same size as training) • Adds face every time angle changes (ideally we want multiple combinations assosciated with each person) • Will insert “bad” faces, and “not” faces. • To much ”radio chatter” with Kafka (esp with “who else is around” context clues)
  32. 32. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  33. 33. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  34. 34. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  35. 35. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  36. 36. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame
  37. 37. Twitter: @rawkintrevo OpenCV/ Haar Cascade Filter inconsistent
  38. 38. Twitter: @rawkintrevo Canopy Clustering • Create N Second Window • Cluster Faces in Window • Quick dirty clustering- but effective. • First point is “center” • All points within distance t2 are “in that cluster. • If a point is not within t2 of any cluster- it becomes a new cluster center.
  39. 39. Twitter: @rawkintrevo Open CV Detects Faces in Video Frame t2= max square width
  40. 40. Twitter: @rawkintrevo Canopy Clustering To Remove “Ghost” Faces t2= max square widthFirst rect – new cluster Second Rect- within one width of first rect (same cluster) Third Rect- within one width of first rect (same cluster) Forth Rect- NOT within one width of first rect (new cluster) Fifth Rect- within one width of first rect (same cluster) Finally- any cluster with less than two entities in windows gets filtered out.
  41. 41. Twitter: @rawkintrevo Canopy Clustering To Remove “Ghost” Faces
  42. 42. Twitter: @rawkintrevo Remaining Faces (Stupid Way) HUMAN-0001 HUMAN-0002 HUMAN-0003 HUMAN-0004
  43. 43. Twitter: @rawkintrevo Remaining Faces (Smarter Way) HUMAN-0001 HUMAN-0001 HUMAN-0001 HUMAN-0001 All faces from a given cluster are likely from the same entity
  44. 44. Twitter: @rawkintrevo Flink Enabled • Windowing lets us use “Intelligence” • Removing Ghost Faces • Using Context Clues (Person in one part of frame doesn’t magically become someone else)
  45. 45. Twitter: @rawkintrevo Flink Plan
  46. 46. Twitter: @rawkintrevo Definition of Intelligence • Merriam Webster: • a (1) : the ability to learn or understand or to deal with new or trying situations : reason; also : the skilled use of reason • (2) : the ability to apply knowledge to manipulate one's environment or to think abstractly as measured by objective criteria (such as tests) • Oxford • The ability to acquire and apply knowledge and skills. • Contested subject among researches…
  47. 47. Twitter: @rawkintrevo Artificial Intelligence • Collect information from environment • Adapt to changing environment • Learn and understand / gain knowledge / improve skills • Apply knowledge
  48. 48. Twitter: @rawkintrevo IoT AI - A Diagram Edge Device Edge Device Edge Device … Edge devices produce data (sensors, drones, cars, etc.)
  49. 49. Twitter: @rawkintrevo A Diagram: Collecting from environment Edge Device Edge Device Edge Device … “Something” needs to collect and process the data
  50. 50. Twitter: @rawkintrevo A Diagram: Learn, understand, adapt, affect Edge Device Edge Device Edge Device … That “Something” needs to support learning and decision frame works Thoughts Actions
  51. 51. Twitter: @rawkintrevo A Diagram: Bonuses Edge Device Edge Device Edge Device … Perhaps it can learns offline and remember as well. Thoughts Actions Memories learning
  52. 52. Twitter: @rawkintrevo A Diagram: Apache AF Edge Device Edge Device Edge Device … Perhaps it can learns offline and remember as well. Thoughts Actions Memories learning
  53. 53. Twitter: @rawkintrevo A Diagram: Apache AF (Lambda) Thoughts Memories learning ”Stream of Conscious” ”Short Term Memory” * Windowing and CEP *
  54. 54. Twitter: @rawkintrevo A Diagram: Apache AF (Kappa) Thoughts Memories ”Stream of Conscious” ”Short Term Memory” * Windowing and CEP * learning
  55. 55. Twitter: @rawkintrevo Technically “Borg-style” AI, not Cylons • A finer technical point for those familiar with the Cylons and the Borg • “Hive Mind” Architecture
  56. 56. Twitter: @rawkintrevo NEW HUMAN-0001 Solr Returns Closest Match Drone says “hi” or shoots to kill OH, hai HUMAN-0001
  57. 57. Twitter: @rawkintrevo Shape of things to come. ”Science Fiction” of 10 years ago, today is domain of hobbyists Demo presented here is “Science Fair” grade AI. Vlad Putin’s recently talking about “it is undesirable for anyone to monopolize AI”. (Yay Apache!)

Facial recognition in production is difficult because neural networks are slow and expensive to train, and must be retrained to recognize new faces added to the set. Older approaches which address these issues such as eigenfaces exist don’t scale as they require a matrix decomposition. Apache Mahout offers a distributed singular value decomposition method, which scales to matrices of arbitrary sizes on Apache Spark, making it possible to use the older yet still powerful Eigenfaces approach to recognize and add new faces in near real time (with the help of Apache Solr). In this talk we present a full stack lambda-style facial recognition system. The offline component uses Mahout/Spark to compute the eigenfaces. The online component leverages Apache Flink to identify faces in an image, decompose the face into a linear combination of the eigenfaces, search for a matching face using SOLR, and if no match is found add the face as a “new face”.

Views

Total views

495

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

4

Shares

0

Comments

0

Likes

0

×