Facial Feature Analysis For Model Based Coding

867 views

Published on

A genetic algorithm I contributed at the conference on evolutionary computation

  • Be the first to comment

  • Be the first to like this

Facial Feature Analysis For Model Based Coding

  1. 1. Eric Larson December 2007 Image Coding and Analysis Laboratory, Oklahoma State University
  2. 2. <ul><li>What is model-based coding? </li></ul><ul><ul><li>Facial Analysis </li></ul></ul><ul><ul><li>Dealing with Dynamic Bandwidths </li></ul></ul><ul><li>Solving a MOP quickly </li></ul><ul><ul><li>An application specific NSGA-II, with a deterministic search </li></ul></ul><ul><li>Results </li></ul><ul><li>Conclusion </li></ul>
  3. 3. <ul><li>Alternative to sending raw video footage </li></ul><ul><li>Creation of “essential” parameters needed to reconstruct a scene </li></ul><ul><li>A real-time analysis nightmare </li></ul>Copyright by Microsoft
  4. 4. <ul><li>Very Low Bit Rate Teleconferencing </li></ul><ul><li>Gaming </li></ul><ul><li>Man-Machine Interaction </li></ul><ul><li>Video Telephony </li></ul><ul><ul><li>Telephony for the deaf </li></ul></ul>Image Courtesy of Dr. Peter Eisert [3]
  5. 5. <ul><li>Analysis (by Synthesis) </li></ul>Image Courtesy of Dr. Peter Eisert [3]
  6. 6. Images Courtesy of Dr. Peter Eisert [4]
  7. 7. <ul><li>Generously, Instituto Superior Technico </li></ul>ISTface [22]
  8. 8. <ul><li>Gradient based approximation is not robust </li></ul><ul><li>Complication of direct optimization </li></ul><ul><ul><li>Handled by reducing FAPs </li></ul></ul><ul><li>Do not address problem of dynamic bandwidth </li></ul>Image Courtesy of J. Ahlberg [17]
  9. 10. <ul><li>Quality Objective Function: </li></ul><ul><li>FAP Number Objective Function: </li></ul>
  10. 11. <ul><li>Use NSGA-II for the multiple objective optimization </li></ul><ul><li>Assign a premature stopping criteria </li></ul><ul><li>Choose bandwidth </li></ul><ul><li>Select FAP sets </li></ul><ul><li>Use deterministic algorithm </li></ul>
  11. 12. <ul><li>Tournament selection used for crossover </li></ul><ul><li>Parents and children combined, sorted according to </li></ul><ul><ul><li>Domination </li></ul></ul><ul><ul><li>Nearest Neighbor </li></ul></ul><ul><li>Repeat </li></ul>From [7], NSGA-II
  12. 13. <ul><li>while {a search direction of improvement can be found} </li></ul><ul><ul><li>for {each dimension, step 20 units} </li></ul></ul><ul><ul><ul><li>-if the step is favorable , another step is made </li></ul></ul></ul><ul><ul><ul><li>-Else, choose next dimension </li></ul></ul></ul><ul><li>find direction of steepest descent from original point and improved point </li></ul><ul><li>  </li></ul><ul><li>while {step size scaling constant < 0.0001} </li></ul><ul><ul><li>take step in the steepest descent direction </li></ul></ul><ul><ul><ul><li>-if the new point is favorable , increase step size by two, </li></ul></ul></ul><ul><ul><ul><li>-else, decrease step size by a factor of ten. </li></ul></ul></ul><ul><ul><li>Update starting individual with new individual </li></ul></ul>
  13. 14. <ul><li>Pareto fronts </li></ul>
  14. 15. Max Bandwidth (Uncompressed) FRAME NO. Selected FAP Sets a Best PSNR Mean PSNR (Over 3 runs) Mean Function Evaluations Medium 0 0 (3) , 1 (2) , 2, 4, 5 (2) , 6, 9, 10, 11 (2) , 12, 13, 15 (3) , 16 (3) 30.57 dB 30.36 dB 779 (~4.8 Kbits/s 1 0 (3) , 1 (2) , 2, 4, 5 (3) , 6 (2) , 8, 9, 10, 11, 12, 13, 14 (2) , 15 (3) , 16 (3) , 17 (3) 35.14 dB 32.54 dB 690 At 25 fps) b 2 0 (2) , 1, 2, 4, 5 (2) , 6 (2) , 7, 8, 10, 11 (2) , 12 (2) , 13 (2) , 14 (3) , 15 (3) , 16 (3) , 17 (2) 32.09 dB 29.50 dB 392 3 0 (2) , 1, 2 (2) , 5, 6 (2) , 7 (2) , 8, 9 (2) , 11 (2) , 12, 13 (2) , 14 (2) , 15 (3) , 16 (3) , 17 33.20 dB 29.99 dB 561 4 0 (2) , 1, 2 (2) , 3, 5 (2) , 6 (2) , 7, 8, 10 (2) , 11 (2) , 13 (2) , 14 (2) , 15 (3) , 16 (2) , 17 (2) 32.98 dB 28.14 dB 415 5 0 (2) , 1 (2) , 2 (2) , 3, 6 (2) , 7 (2) , 8, 9 (2) , 10 (2) , 11, 12 (3) , 13, 14 (3) , 15 (3) , 16 (2) , 17 (2) 32.90 dB 28.73 dB 299 6 0 (2) , 1 (3) , 2, 5, 7 (2) , 8 (3) , 9, 10 (2) , 11, 12, 14, 15 (3) , 16 (3) , 17 32.13 dB 30.89 dB 748 7 0 (3) , 1 (2) , 4, 5, 6, 7 (3) , 8 (3) , 11 (2) , 12 (2) , 13, 14, 15 (3) , 16 (3) , 17 (2) 31.91 dB 29.51 dB 445 8 0 (3) , 2, 4, 5 (2) , 6(2), 8, 9, 11 (2) , 12 (2) , 13 (2) , 14 (2) , 15 (3) , 16 (3) , 17 (2) 30.97 dB 29.53 dB 726 9 0 (3) , 1 (2) , 3, 5 (2) , 6 (2) , 7, 8, 9 (2) , 10 (2) , 11 (2) , 12 (2) , 14, 15 (3) , 16 (3) , 17 30.96 dB 28.99 dB 451 10 0 (3) , 2, 3, 5, 6, 7, 8, 9, 10 (2) , 11 (2) , 12 (2) , 13 (2) , 14 (2) , 15 (3) , 16 (3) , 17 (2) 30.21 dB 28.80 dB 527 Low 0 0, 7, 8 (2) , 11 (2) , 14 (2) , 15 (3) , 16 (2) 29.95 dB 27.13 dB 573 (~2.4 Kbits/s 1 0, 5, 8, 11 (2) , 12, 14, 15 (3) , 16 (2) , 17 (3) 33.23 dB 29.46 dB 595 At 25 fps) b 2 8, 10, 11, 12 (2) , 13 (2) , 14, 15 (3) , 16 (2) , 17 (3) 32.02 dB 27.21 dB 773 3 2, 5, 6, 8, 9, 12 (2) , 14, 15 (3) , 16, 17 28.77 dB 24.34 dB 808 4 1, 9 (2) , 10, 11, 12 (2) , 14 (2) , 15 (3) , 17 (3) 22.99 dB 22.80 dB 745 5 1, 2, 4, 5, 6, 9, 11, 12, 14, 15 (3) , 16 (2) , 17 29.25 dB 26.93 dB 446 6 2, 5, 6, 9 (2) , 10, 11 (2) , 12, 14 (2) , 15 (2) , 16 (3) , 17 29.67 dB 25.75 dB 376 7 1, 2, 7, 8, 9, 10, 12, 14, 15 (3) , 16 (3) , 17 29.01 dB 28.41 dB 386 8 1, 3, 9, 12, 13, 15, 16 (3) 28.97 dB 23.98 dB 529 9 0, 5, 9, 10, 11, 12, 15 (2) , 16 (3) , 17 28.79 dB 25.93 dB 694 10 3, 5 (2) , 6 (2) , 9, 10 (2) , 12, 15, 16 (3) 27.56 dB 24.25 dB 226
  15. 16. <ul><li>Histogram of all resultant individuals </li></ul>
  16. 17. <ul><li>Video Sequence </li></ul>Frame 90 Low Medium
  17. 18. Frame 93 Low Medium
  18. 19. Frame 96 Low Medium
  19. 20. Frame 99 Low Medium
  20. 21. Frame 102 Low Medium
  21. 22. Frame 105 Low Medium
  22. 23. Frame 108 Low Medium
  23. 24. Frame 111 Low Medium
  24. 25. Frame 114 Low Medium
  25. 26. Frame 117 Low Medium
  26. 27. Frame 120 Low Medium
  27. 28. <ul><li>Deficiencies can be traced back to selection of PSNR </li></ul><ul><li>Future work should include error functions like SSIM or Eigen-faces </li></ul><ul><li>Algorithm works </li></ul><ul><ul><li>Accentuates the </li></ul></ul><ul><ul><li>details of PSNR </li></ul></ul>
  28. 29. <ul><li>D. Pearson, “Developments in model-based image coding,” Proceedings of the IEEE , Vol. 83, No. 6, June 1995. </li></ul><ul><li>I. Pandizic. J. Ahlberg, M. Wzorek, P. Rudol, and M. Mosmondor, “Faces Everywhere: Towards Ubiquitous Production and Delivery of Face Animation,” Proceedings of the 2 nd international conferenice on mobile and ubiquitous media , 2003 </li></ul><ul><li>P. Eisert, “MPEG-4 facial animation in video analysis and synthesis,” International Journal of Imaging Systems and Technology , June 2003. </li></ul><ul><li>P. Eisert, “Very Low Bit Rate Coding,” Doctoral Thesis, November 2000. </li></ul><ul><li>J. D. Schaffer, “Multiple objective optimization with vector evaluated genetic algorithms,” 1 st international conference on genetic algorithms , 1985. </li></ul><ul><li>K. Deb, “Multi-objective genetic algorithms: problems, difficulties, and construction of test problems,” Evolutionary Computation , 1999. </li></ul><ul><li>Deb, K., Pratap, A., Agarwal, S., and Meyarivan, T., A fast and elitist multiobjective genetic algorithm: NSGA-II , IEEE Transactions on Evolutionary Computation , 2002. </li></ul><ul><li>F. I. Parke, Parameterized Models for Facial Animation, IEEE Transactions on Computer Graphics and Animation , 1982. </li></ul><ul><li>R. Forchheimer and T. Kronander, “Image coding – from waveforms to animation,” IEEE Transactions on Acoustics, Speech, and Signal Processing , 37:1212, 1989. </li></ul><ul><li>C. S. Choi, K. Aizawa, H. Harashima, and T. Takebe, “Analysis and synthesis of facial image sequences in model-based image coding,” IEEE Transactions on Circuits and Systems for Video Technology , June 1994. </li></ul><ul><li>M. Buck, “Model based image sequence coding,” Motion Analysis and Image Sequence Coding, Ch. 10, Kluwer Academic Publishing, 1993, pp. 285-315. </li></ul><ul><li>N. Diehl, “Object motion estimation and segmentation on image sequences,” Signal Processing: Image Communications , Vol. 3, No. 1, February 1991, pp. 23-56. </li></ul><ul><li>K. Aizawa, H. Harashima and T. Saito, “Model-based analysis-synthesis image coding (MBASIC) system for a person’s face,” Signal Processing: Image Communication, vol. 1, pp. 139-152, 1989. </li></ul><ul><li>I. S. Pandizic and R. Forchheimer, “MPEG-4 Facial Animation: the Standard, Implementation, and Applications,” 1 st Ed. John Wiley and Sons, 2002, pp. 3-41. </li></ul><ul><li>J. Ahlberg and R. Forchheimer, “Face Tracking for model-based coding and face animation,” International Journal on Imaging Systems Technology , Wiley Periodicals, Vol. 13, pp. 8-22, 2003. </li></ul><ul><li>Dornaika, F., Ahlberg, J., Fast and Reliable Active Appearance Model Search for 3D Face Tracking, Proceedings of Mirage 2003, March 2003. </li></ul><ul><li>Dornaika, F., Ahlberg, J., Fitting 3D Face Models for Tracking and Active Appearance Model Training, Image and Vision Computing 24(2006), Science Direct, 2006. </li></ul><ul><li>Carter, E.F, 1994, The Generation and Application of Random Numbers, Forth Dimensions, Vol XVI, Nos 1 & 2, Forth Interest Group, Oakland California. </li></ul><ul><li>S. Kirkpatrick, C. D. Gelati, and M. P. Vecchi, “Optimization by simulated annealing,” Science , Vol. 220, No. 4598, pp. 671-680, 1983. </li></ul><ul><li>T. Edgar, D. Himmelblau, and Lasdon, L., Optimization of Chemical Processes , 2 nd Edition, McGraw-Hill, New York, NY, 2001. </li></ul><ul><li>G. Reklaitis, A. Ravindran, and Ragsdell, K., Engineering Optimization, Methods and Applications , 2 nd Edition, John Wiley and Sons, New York, NY, 2006. </li></ul><ul><li>ISTface, Program from Instituto Superior Technico, standard FAP animation sequence, “wow25.fap”. </li></ul><ul><li>J. Jiang, A. Alwan, P. A. Keating, and T. A. Edward Jr., “On the relationship between face movements, tongue movements, and speech acoustics,” EURASIP Journal on Applied Signal Processing , 2002. </li></ul><ul><li>Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity ,” IEEE Trans. Image Process . 13, 600–612 (2004). </li></ul>

×