Downloaded 55 times
Uploaded byOntico
Face Recognition: From Scratch To Hatch / Эдуард Тянтов (Mail.ru Group)
The document discusses advancements in face recognition technology, detailing methods such as the Viola-Jones algorithm and deep learning-based approaches like MTCNN and Faster R-CNN, highlighting their performance metrics. It emphasizes the importance of data quality and diversity for effective model training, particularly in overcoming biases in recognition accuracy. Furthermore, it introduces various techniques for optimizing inference speed and enhancing feature discriminability, such as TensorRT and metric learning with center loss.
More Related Content
Similar to Face Recognition: From Scratch To Hatch / Эдуард Тянтов (Mail.ru Group)
Face Recognition: From Scratch To Hatch / Эдуард Тянтов (Mail.ru Group)
- 1. Face Recognition: From ScratchTo Hatch Tyantov Eduard, Mail.ru Group
- 2. Face Recognition inCloud@Mail.ru Users upload photos to Cloud Backend identifies persons on photos, tags and show clusters
- 3.
- 4.
- 5. edges object parts(combination of edges) object models
- 6.
- 7.
- 8. Auxiliary task: faciallandmarks – Face alignment: rotation – Goal: make it easier for Face Recognition Rotate
- 9. Train Datasets Wider – 32kimages – 494k faces Celeba – 200k images, 10k persons – Landmarks, 40 binary attributes
- 10. Test Dataset: FDDB FaceDetection Data Set and Benchmark – 2845 images – 5171 faces
- 11. Old school: Viola-Jones HaarFeature-based Cascade Classifiers Haar-like features eyes darker nose lighter Examples
- 12. Viola-Jones algorithm: training foreach patch AdaBoost ensemble features 160k valuable 6k weighted sum Face or NotDataset
- 13. Viola-Jones algorithm: inference foreach patch Stages Face Yes Yes Stage 1 Stage 2 Stage N Optimization – Features are grouped into stages – If a patch fails any stage => discard
- 14. Viola-Jones results OpenCV implementation –Fast: ~100ms on CPU FDDB results 0.45 – Not accurate
- 15. 1. Pre-trained network:extracting features New school: Region-based Convolutional Networks Faster RCNN, algorithm 1 CNN 2 RPN 4 Classifier Face ? 3 Roi-pooling Feature Maps 3 2. Region proposal network3. RoI-pooling: extract corresponding tensor4. Classifier: classes and the bounding box
- 16. Comparison: Viola-Jones vsR-FCN Results – 92% accuracy (R-FCN) FDDB results Viola-Jones (opencv) 0.45 HOG (dlib) 0.7 R-FCN 0.92 – 40ms on GPU (slow)
- 17. Face detection: howfast We need faster solution at the same accuracy! Target: < 10ms
- 18. Alternative: MTCNN 1 Different scales 2 Proposal CNN 3 Refine CNN 4 Output CNN Cascade of3 CNN 1. Resize to different scales2. Proposal -> candidates + b-boxes 3. Refine -> calibration4. Output -> b-boxes + landmarks
- 19. Comparison: MTCNN vsR-FCN MTCNN + Faster + Landmarks - Less accurate - No batch processing Model GPU Inference FDDB Precision (100 errors) R-FCN 40 ms 92% MTCNN 17 ms 90%
- 20.
- 21. What is TensorRT NVIDIATensorRT is a high-performance deep learning inference optimizer Features – Improves performance for complex networks – FP16 & INT8 support – Effective at small batch-sizes
- 22. TensorRT: layer optimizations 2.Horizontal fusion 3. Concat elision 1. Vertical layer fusion
- 23. TensorRT: downsides 1. Caffe+ TensorFlow supported 2. Fixed input/batch size 3. Basic layers support
- 24. Batch processing Problem Image sizeis fixed, but MTCNN works at different scales Solution Pyramid on a single image
- 25. Batch processing Results – Singlerun – Enables batch processing Model Inference ms MTCNN (Caffe, python) 17 MTCNN (Caffe, C++) 12.7 + batch 10.7
- 26. TensorRT: layers Problem No PReLUlayer => default pre-trained model can’t be used Retrained with ReLU from scratch Model GPU Inference ms FDDB Precision (100 errors) MTCNN, batch 10.7 90% +Tensor RT 8.8 91.2% -20%
- 27. Face detection: inference Target:< 10 ms Result: 8.8 ms Ingredients 1. MTCNN 2. Batch processing 3. TensorRT
- 28.
- 29. Face recognition task –Goal – to compare faces Latent SpaceCNN Embedding close distant Unseen – How? To learn metric – To enable Zero-shot learning
- 30. Training set: MSCeleb –Top 100k celebrities – 10 Million images, 100 per person – Noisy: constructed by leveraging public search engines
- 31. Small test dataset:LFW Labeled Faces in the Wild Home – 13k images from the web – 1680 persons have >= 2 photos
- 32. Large test dataset:Megaface – Identification under up to 1 million “distractors” – 530 people to find
- 33.
- 34.
- 35. Classification CNN Embedding Classify – Train CNNto predict classes – Pray for a good latent space close distant
- 36. Softmax – Learned featuresonly separable but not discriminative – The resulting features are not sufficiently effective close
- 37. We need metriclearning – Tightness of the cluster – Discriminative features
- 38. Triplet loss Features – Identity-> single point – Enforces a margin between persons Anchor Positive Negativepositive + α < negative minimize maximize
- 39. Choosing triplets Crucial problem Howto choose triplets ? Useful triplets = hardest errors Pick all positive Too easy Hard enough Solution Hard-mining within a large mini-batch (>1000)
- 40.
- 41. Choosing triplets: trap Anchor Positivepositive ~ negative minimize maximize Negative
- 42.
- 43. Choosing triplets: trap Selectinghardest negative may lead to the collapse early in training
- 44. Choosing triplets: semi-hard Pickall positive Too easy Semi-hard Too hard positive < negative < positive + α
- 45. Triplet loss: summary Overview –Requires large batches, margin tuning – Slow convergence Opensource Code – Openface (Torch) • suboptimal implementation – Facenet, not original (TensorFlow) LFW, % Megaface Openface (Torch) 92 - Our (Torch) 99.35 65 Google’s Facenet 99.63 70.5
- 46. Center loss Idea: pullthe points to class centroids
- 47. Center loss: structure –Without classification loss – collapses CNN Embedding Classify Softmax Loss λ Center Loss Pull – Final loss = Softmax loss + λ Center loss
- 48. Center Loss: differentlambdas λ = 10-7
- 49. Center Loss: differentlambdas λ = 10-6
- 50. Center Loss: differentlambdas λ = 10-5
- 51. Center loss: summary Overview –Intra-class compactness and inter-class separability – Good performance at several other tasks Opensource Code – Caffe (original, Megaface - 65%) LFW, % Megaface Triplet Loss 99.35 65 Center Loss (Torch, ours) 99.60 71.7
- 52. Tricks: augmentation Test timeaugmentation – Flip image Embedding Flipped Embedding Final Embedding Average – Average embeddings – Compute 2 embeddings
- 53. Tricks: alignment Rotation Kabsch algorithm- the optimal rotation matrix that minimizes the RMSD LFW, % Megaface Center Loss 99.6 71.7 Center Loss + Tricks 99.68 73
- 54. Shades on At onepoint we used shades augmentation How to – Construct several sunglass textures – Place them using landmarks
- 55. Tricks: adding Eyeloss CNN Embedding Person Softmax Loss Center Loss – We can force CNN to learn specific discriminative features – For celebrities eye colors are available in the Internet Eye Loss Eye Color
- 56. Eye loss: summary *Addingsimple features doesn’t help, i.e gender LFW, % Megaface Center Loss + Tricks 99.68 73 Center Loss + Eye 99.68 73.5
- 57. Angular Softmax ||X||= 1 Onsphere Angle discriminates ||W||= 1 b=0
- 58.
- 59. Angular Softmax: different«m» m=1 m=3
- 60. Angular softmax: summary Overview –As describes in the paper: doesn’t work at all – Works using sum of losses (m=1,N) over training • only on small datasets! LFW, % Megaface Center Loss 99.6 73 Center Loss + Eye 99.68 73.5 A-Softmax (Torch) 99.68 74.2 Opensource Code – Caffe (original) – Slight modification of the loss yields 74.2%
- 61. Metric learning: summary Softmax< Triplet < Center < A-Softmax A-Softmax – With bells and whistles better than center loss Center loss Overall – Rule of thumb: use Center loss – Metric learning may improve classification performance
- 62.
- 63. Errors after MSCeleb:children Problem Children all look alike Result Embeddings are almost single point in the space
- 64. Errors after MSCeleb:asian Problem Face Recognition’s intolerant to Asians Reason Dataset doesn’t contain enough photos of these categories
- 65. How to fixthese errors ? It’s all about data, we need diverse dataset! Natural choice – avatars of social networks
- 66. A way toconstruct dataset Face Detection Pick largest Face Recognition+ Clustering Cleaning algorithm 1. Face detection2. Face recognition -> embeddings3. Hierarchical clustering algorithm4. Pick the largest cluster as a personIterate after each model improvement
- 67. MSCeleb dataset’s errors MSCelebis constructed by leveraging search engines Joe Eszterhas Joe Eszterhas and Mel Gibson public confrontation leads to the error Mel Gibson =
- 68.
- 69.
- 70. MSCeleb dataset’s errors Datasethas been shrinked from 100k to 46k celebrities Random search engine Corrected
- 71. Results on newdatasets Datasets – Train: • MSCeleb (46k) • VK-train (150k) A-Softmax on dataset Megaface MSCeleb 74.2 MSCeleb cleaned 76.2 – Test • MegaVK • Sets for children and asians MegaVK 58.4 60 + VK 77.5 87.5
- 72. Ensemble Final Embedding Concat CNN-2 CNN-1 A-Softmax model MegafaceMegaVK Best single 77.5 87.5 Ensemble of 2 79.6 88.6
- 73. Workaround still … Childrenare still challenge for the model
- 74. Workaround Algorithm 1. Construct datasetwith children 2. Compute average embedding 3. Every point inside the sphere – a child 4. Tighten distance threshold there Results This allows softening the overall threshold
- 75. How to handlebig dataset It seems we can add more data infinitely, but no. Problems – Memory consumption (Softmax) – Computational costs – A lot of noise in gradients
- 76. Softmax Approximation Algorithm 1. PerformK-Means clustering using current FR model Dataset K-Means Children Women Men Smaller sets
- 77. Softmax Approximation Algorithm 1. PerformK-Means clustering using current FR model CNN Embedding Predict cluster Predict person MenPerson Softmax 2. Two Softmax heads: 1. Predicts cluster label 2. Class within the true cluster Cluster Softmax Men
- 78. Softmax Approximation Pros Push Push Harder negative 1. Preventsfusing of the clusters 2. Does hard-negative mining 3. Clusters can be specified • Children • Asian
- 79. Fighting errors onproduction
- 80. Errors: blur Problem • Detectoryields blurry photos • Recognition forms «blurry clusters» Solution Laplacian – 2nd order derivative of the image
- 81.
- 82. Errors: body parts Detection mistakesform clusters
- 83. Errors: diagrams &mushrooms
- 84. Fixing trash clusters Thereis similarity between “no faces”! CNN Embedding Embedding No «firing» features
- 85. Fixing trash clusters «Trash»has small norm Faces Trash Softmax loss Motivation Softmax encourage big embedding’s norms Results – ROC AUC 97% – Better then Laplacian for blurry
- 86.
- 87. Fun: new governors Recentlyappointed governors are almost twins, but FR distinguishes them Dmitriy Gleb
- 88. Over years Face recognitionalgorithm captures similarity across years Although we didn’t focus on the problem
- 89.
- 90. Summary 1. Use TensorRTto speed up inference 2. Metric learning: use Center loss by default 3. Clean your data thoroughly 4. Understanding CNN helps to fight errors
- 91.
- 92.
- 93. Best avatar Problem How topick an avatar for a person ? Solution Train model to predict awesomeness of photo
- 94. Predicting awesomeness: howto approach Social networks – not only photos, but likes too
- 95. Predicting awesomeness: dataset Awesomeness(A) = likes/audience A=18% A=27% A=75%
- 96. Results – Mean AveagePrecision @5: 25% – Data and metric are noisy => human evaluation Predicting awesomeness: summary High score Low score
- 97. Predicting awesomeness: incorporatinginto FR One more branch in Face Recognition CNN Small overhead awesomeness embedding face
- 98.
- 99. Histogram loss Idea – Computesimilarities positive and negative pairs – Maximize a probability that a randomly sampled positive pair has smaller similarity than negative one Loss = the integral of the product between the negative distribution and the cumulative density function for the positive distribution (shown with a dashed line)
- 100. Histogram loss Results – Gotnice distributions – Doesn’t improve the triplet results – LFW on 97.7%
Editor's Notes
- #8 «Detect all faces on the image at all scales and rotations»
- #10 http://vis-www.cs.umass.edu/fddb/
- #11 http://vis-www.cs.umass.edu/fddb/
- #12 https://docs.opencv.org/master/d7/d8b/tutorial_py_face_detection.html , http://blog.dlib.net/2014/02/dlib-186-released-make-your-own-object.html
- #16 R-FCN: https://arxiv.org/pdf/1605.06409.pdf , Faster: https://arxiv.org/pdf/1506.01497v3.pdf
- #17 «Another approach – HOG features (dlib)» «not far from state-of-the-art (Google reports 94%)»
- #19 Offtop: обучается первая сеть на 12x12, вторая – 24x24, 3 – 36x36 – hard negative mining для 2 и 3 для исправления ошибок выдает первая 5x1x1, а на пирамиде тензор, дальше составляется батч для 2 и 3 (батч над батчами не делали) https://kpzhang93.github.io/MTCNN_face_detection_alignment/paper/spl.pdf
- #20 «Open source Pre-trained model (Caffe) No original training code «
- #24 «but not Torch/PyTorch We’ve retrained the CNN because of PreLu absense» basic- actively devolping
- #25 TODO: возможно показать что мы не подряд засовываем в сеть, а сразу в кучу, хз не перегруз ли
- #26 табличка в одном месте
- #31 https://www.microsoft.com/en-us/research/project/ms-celeb-1m-challenge-recognizing-one-million-celebrities-real-world/
- #33 мб добавить
- #34 Top results ~ 80% Best – 91.7%
- #38 «All faces of person are project to a single point»
- #39 Chad Smith
- #45 TODO: про альфа лишнее ?
- #48 https://ydwen.github.io/papers/WenECCV16.pdf, 65% из статьи
- #53 TODO: убрать текст ?
- #54 (Better than dlib)
- #55 TODO: не выпилить но доделать нормально фотка чувака точки + очки
- #57 «пол например не является discriminative – пробовали» 71.5% -> 75.5% или +2 71.5 vs 73, 75,5 из-за датасета
- #58 СЛОВА «сказать что норма – это деление на сумму»
- #59 To discriminate better -> tighten clusters
- #61 http://proceedings.mlr.press/v48/liud16.pdf
- #62 http://proceedings.mlr.press/v48/liud16.pdf
- #65 “Рассказать про идею созданую по мотивам корпоратиа”
- #66 VK – easy API OK – hard access to API, rate-limits Facebook – just closed to mining
- #67 Cleaning algorithm Face Detection Face Recognition to get embeddings Hierarchical clustering algorithm with Soften the distance threshold Pick the largest cluster as a person iterate after each model improvement
- #68 this leads to improved perf on megaface = 75%?
- #75 цифра лучше ? чуть меньше текста или на 2 слайда разбить TODO: мы обнаружили что дети компанкто расположены
- #77 https://arxiv.org/pdf/1609.04309.pdf - paper: “Efficient softmax approximation for GPUs”
- #79 «какой профит помимо решения проблемы»
- #81 https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_imgproc/py_gradients/py_gradients.html It’s a face, but without enough facial information
- #83 “If the detector makes a mistake, recognizer gone crazy. Detection errors often form a cluster.”
- #85 «софтмакс стремится увеличить скалярное произведение»
- #86 maximizing scalar product <w_i, x> for i class We normalize embeddings only for distance
- #89 “Social networks contains avatars over users’ lives”
- #95 Data we know How many likes Who likes
- #97 http://people.cs.uchicago.edu/~larsson/fractalnet/overview.png
- #98 block 6 нарисовать
- #100 https://arxiv.org/pdf/1611.00822.pdf