Recommended
More Related Content
What's hot
What's hot (17)
Similar to Learning a multi-center convolutional network for unconstrained face alignment
Similar to Learning a multi-center convolutional network for unconstrained face alignment (20)
Recently uploaded
Recently uploaded (17)
Learning a multi-center convolutional network for unconstrained face alignment
- 1. Learning a Multi-Center Convolutional Network for Unconstrained Face Alignment Zhiwen Shao, Hengliang Zhu, Yangyang Hao, Min Wang, and Lizhuang Ma Shanghai Jiao Tong University
- 2. Introduction
- 3. Face Alignment Detecting facial landmarks like pupil centers, nose tip, mouth corners
- 4. Unconstrained scenarios including severe occlusions and large face variations Challenges
- 5. Methods based on low-level handcrafted features have a limited capacity to represent highly complex faces Deep convolutional network A nonlinear regression problem, which transforms appearance to shape Motivation
- 6. Cascaded CNN [1], Zhou et al. [2], CFAN [3], and CDAN [4] employ cascaded deep networks to refine predicted shapes Previous Deep Learning Methods time-consuming training processes high model complexity [1] Y. Sun, X. Wang, and X. Tang, “Deep convolutional network cascade for facial point detection,” in IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2013, pp. 3476–3483.[2] E. Zhou, H. Fan, Z. Cao, Y. Jiang, and Q. Yin, “Extensive facial landmark localization with coarse-to-fine convolutional network cascade,” in IEEE International Conference on Computer Vision Workshops. IEEE, 2013, pp. 386–391. [3] J. Zhang, S. Shan, M. Kan, and X. Chen, “Coarse-to-fine auto-encoder networks (cfan) for real- time face alignment,” in European Conference on Computer Vision. Springer, 2014, pp. 1–16. [4] R. Weng, J. Lu, Y.-P. Tan, and J. Zhou, “Learning cascaded deep auto-encoder networks for face alignment,” IEEE Transactions on Multimedia, vol. 18, no. 10, pp. 2066–2078, 2016. Multiple networks based
- 7. TCDCN [5] needs extra labels of facial attributes for samples one single network without auxiliary information Previous Deep Learning Methods limits the universality of this method Single network based [5] Z. Zhang, P. Luo, C. C. Loy, and X. Tang, “Learning deep representation for face alignment with auxiliary attributes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 5, pp. 918–930, 2016.
- 8. Methodology
- 9. Structural Correlations Chin is occluded Right contour is invisible Unconstrained faces with partial occlusion and large pose Landmarks in the same local region have similar properties including occlusion and visibility
- 10. Face Partition 29 landmarks 68 landmarks Partition of facial landmarks for different labeling patterns Left eye, right eye, nose, mouth, left contour, chin, and right contour
- 11. Network Architecture Shared layers Multiple center-specific shape prediction layers
- 12. Network Architecture Shared layers • Eight convolutional layers and one fully-connected layer • Each max-pooling layer follows a stack of two convolutional layers
- 13. Network Architecture • Each cluster of facial landmarks is treated as a separate center • Each layer estimates x and y coordinates of all n facial landmarks • Focusing on the shape estimation of a specific face region Multiple center-specific shape prediction layers
- 14. Loss Function ^ ^ 2 2 2 2 1 22 1 2 1 [( ) ( ) ]/ (2 ) n j j jj j j E w f f f f d− − = = − + −∑ Weighted inter-ocular distance normalized Euclidean loss jw weight of the j-th landmark ground truth coordinatesf predicted coordinates ^ f d ground truth inter-ocular distance the first center-specific layer: larger weights for landmarks around the left eye
- 18. Multi-Center Learning Reinforcement for Each Center
- 21. Weight Computation Multiple relationship ( ) ( )i c i m P P w wη= ( )i c P set of center-specific landmarks ( )i m P set of remaining minor landmarks amplification factor Different fine-tuning steps have different center- specific and minor facial landmarks Consistent with the basic model ( ) ( ) ( ) ( ) | | ( | |)i c i m i c i c P P w P w n P n+ − = | |× number of elements in a set During the i-th fine-tuning step
- 22. ( ) ( ) ( ) ( ) /[( 1) | | ] /[( 1) | | ] i c i m i c P i c P w n P n w n P n η η η = − + = − + other centers with relatively small weights rather than zeroutilize implicit structural correlations among different parts landmarks from the same cluster have similar properties share an identical weight search the solution smoothly Weight Computation During the i-th fine-tuning step
- 23. Combined Model high-level representation ( 1) 1 0 1( , , , ) ( 1024)T D Dx x x D+ × = ∈ =x L ¡ weight matrix ( 1) 2 1 2 2( , , , ) D n n + × = ∈W w w wL ¡ 0 1( , , , ) , 1, ,2T k k k Dkw w w k n= =w L L ^ 2 12 1 ^ 22 T jj T jj f f −− = = w x w x weight matrix of the i-th center-specific layer i W 2 1 2 1 2 2 combined i j j combined i j j − −= = w w w w ( ) 1, , , i c i m j P= ∈L
- 24. Combined Model Combined Model S combined Θ ∪ W complexity is as same as the basic model improves the location performance by exploiting the advantage of each center-specific solution Our multi-center learning algorithm takes full advantage of each stage and searches the optimal solution smoothly
- 25. Experiments
- 26. Datasets COFW occluded dataset in the wild 1345 training images 507 testing images IBUG large appearance variations 3148 training images 135 testing images
- 27. Evaluation Metric inter-ocular distance normalized mean error cumulative errors distribution (CED) curves failure rate failure: mean error larger than 10%
- 28. Validation of Multi-Center Learning Algorithm Method COFW IBUG Mean Failure Mean Failure Basic 6.26 3.16 9.23 33.33 Combined 6.08 2.96 8.87 25.93 Mean Error (%) and Failure Rate (%) improve the accuracy and robustness good performance of basic model effectiveness of our network reinforce the learning for each local face region
- 29. Validation of Multi-Center Learning Algorithm Mean error for different clusters on COFW
- 30. Comparison with Other Methods Method COFW IBUG ESR 11.2 17.00 SDM 11.14 15.40 RCPR 8.5 17.26 CFAN - 16.78 LBF - 11.98 cGPRT - 11.03 CFSS - 9.98 TCDCN 8.05 8.60 CFT 6.33 10.06 Wu et al. 5.93 - MCNet 6.08 8.87
- 31. Comparison with Other Methods
- 32. Comparison with Other Methods COFW
- 33. Comparison with Other Methods IBUG
- 34. Comparison with Other Methods Deep model Speed (FPS) CPU Cascaded CNN 5 single core, i5-6200U 2.3GHz CFAN* 43 i7-3770 3.4GHz CDAN* 50 i5 3.2GHz TCDCN 50 single core, i5-6200U 2.3GHz CFT 31 single core, i5-6200U 2.3GHz MCNet 67 single core, i5-6200U 2.3GHz Time of face detection is excluded
- 35. Conclusions We propose a novel multi-center convolutional network, which exploits the representation power of each center We propose the reinforcement for each center to improve the shape estimation precision of each facial part Comprehensive experiments demonstrate that our method achieves real-time and competitive performance compared to other state-of-the-art techniques
- 37. Thank you!
Editor's Notes
- Good morning, everyone. I am Zhiwen Shao. I come from Shanghai Jiao Tong University. In our paper, we propose a Multi-Center Convolutional Network to achieve face alignment.
- I first show the background of face alignment
- These images illustrate the results of face alignment.
- We can observe that these face images are very challenging. They have severe occlusions and large variations of pose, expression, illumination. Our goal is to develop an efficient method to handle unconstrained faces
- Face alignment can be regarded as a nonlinear regression problem, which transforms appearance to shape Most conventional methods are based on low-level handcrafted features, so they have a limited capacity to represent complex faces As we all know, a deep convolutional network has an outstanding representation ability. Therefore we use it to model the highly nonlinear function
- There are two types of deep learning methods. The first is multiple networks based. These methods employ cascaded deep networks to refine predicted shapes successively. Their training processes are complicated and time-consuming. And they have high computational cost and model complexity due to the use of multiple networks
- A very typical method is TCDCN. It trains only one deep network, but it needs extra labels of facial attributes for training samples. This limits the universality of this method. In contrast, our method uses one single network without auxiliary information
- Next I introduce our method in details
- Partial occlusion and large pose are main characteristics of unconstrained faces. We discover that each facial landmark is not isolated but highly correlated with adjacent landmarks. There are two examples. In the left figure, facial landmarks along the chin are all occluded. And the right figure shows that landmarks on the right side of the face are almost invisible. Therefore, landmarks in the same local face region have similar properties including occlusion and visibility.
- We analyze the structure of a face, and partition it into seven clusters: left eye, right eye, nose, mouth, left contour, chin, and right contour. As shown in these two figures, different labeling patterns of 29 and 68 facial landmarks are partitioned into 5 and 7 clusters respectively. Each cluster contains structurally relevant facial landmarks.
- This is the structure of our multi-center convolutional network. Our network consists of shared layers and multiple center-specific shape prediction layers.
- The shared layers contain eight convolutional layers and one fully-connected layer. Each max-pooling layer follows a stack of two convolutional layers The stack of convolutional layers is excellent in feature learning, which is proposed by VGGNet.
- According to the evaluation metric, we use weighted inter-ocular distance normalized Euclidean loss
- We first pre-train a basic model with shared layers and one shape prediction layer.
- Corresponding to Step 1
- We further fine-tune each center-specific layer respectively
- Corresponding to Step 2 to Step 6 Based on the pre-trained model, our network keeps shared layers and initializes each center-specific layer with the shape prediction parameters. There are m branches of center-specific layers at the end of our network. The fine-tuning of center-specific layers is mutually independent.
- Shared layers and integrated shape prediction layer constitute the combined model
- Corresponding to Step 7 We obtain the integrated shape prediction layer by combining corresponding parameters from each center-specific layer.
- We assume there is a multiple relationship between two weights To be consistent with the basic model, we keep weights conforming to this formula The summation of weights is ensured to equal n
- By solving two equations, we obtain the respective weights When emphasizing on the detection of current center, we still consider other centers with relatively small weights rather than zero. This is beneficial for utilizing implicit structural correlations among different facial parts and searching the solution smoothly
- Then I show the experiments
- Euclidean distance between two pupil centers
- We show the mean error of each cluster for basic model and combined model on COFW dataset It can be observed that the combined model improves the detection performance of each cluster
- We report the results of our method MCNet and previous works. We can see that our method outperforms most state-of-the-art methods It is worth noting that TCDCN obtains better performance than our method on IBUG partly owing to their larger training data. Although occlusions are not detected explicitly, we achieve an outstanding performance on par with Wu et al. on COFW benchmark.
- We plot the CED curves for our method and several state-of-the-art methods. It is observed that our method achieves competitive performance on both two benchmarks. Our method achieves better performance for high-level normalized mean error. Therefore, our method is strongly robust to unconstrained environments.
- There are several images from COFW We can see our method indicates higher accuracy than RCPR and CFT in the details Benefiting from utilizing structural correlations among different facial parts, our method is robust to severe occlusions.
- We also show example images from IBUG where our method MCNet outperforms LBF and CFSS Our method also achieves higher accuracy in the details. Therefore our method demonstrates superior capability of handling severe occlusions and complex variations of pose, expression, illumination.
- To obtain a more comprehensive comparison, we present the average running speed of different deep learning methods for face alignment We evaluate these methods on a single core i5-6200U 2.3GHz CPU with 1000 face images. Since CFAN and CDAN do not share their code, we use their published speed results. Both TCDCN and our method MCNet are based on only one network, so they show relatively quick speed. Cascaded CNN, CFAN and CDAN employ multiple networks, so they cost more running time. Our method only takes 15 ms on average to process one face, profiting from low model complexity and computational cost of our network. We believe that our method can be extended to real-time facial landmark tracking in unconstrained scenarios.
- Finally, I conclude our paper