Towards Embedded Computer Vision - New @ 2013

799 views

Published on

This is an academic talk for professors and graduate students. In addition to introducing recent trends in embedded computer vision (ECV), I also present our research experience in ECV.

Published in: Education, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
799
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
60
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Towards Embedded Computer Vision - New @ 2013

  1. 1. Wang, Yuan-Kai(王元凱) Electrical Engineering Department, Fu Jen University (輔仁大學電機工程系) Email: ykwang@mail.fju.edu.tw, URL: http://www.ykwang.tw Slideshare: Search "Towards Embedded Computer Vision" Towards Embedded Computer Vision 1
  2. 2. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Context Awareness 2 人 時 物 地 事 Situation High-Level Context Low-Level Contexts
  3. 3. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 3 MIT Oxygen http://oxygen.lcs.mit.edu/ 1999
  4. 4. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 4 The Aware Home http://awarehome.imtc.gatech.edu/
  5. 5. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Ambient Intelligence v.s. Object Intelligence Embedded system + Sensors 5 Wearable Computing Smart Home, Smart …
  6. 6. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 66 Embedded Systems "Without" Sensors 資料來源:資策會MIC ITIS計畫整理
  7. 7. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 77 Embedded Systems "With" Sensors GPS Navigation Wii Roomba Exoskeleton
  8. 8. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 8 Driver assist Front view Night vision Blind corner Blind spot Rear view Embedded Systems "With" Image Sensors
  9. 9. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 99 Embedded Systems "With" Image Sensors Google Glass
  10. 10. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Smart Watch 10
  11. 11. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Microsoft SenseCam Embedded Systems "With" Image Sensors 11
  12. 12. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Observation Miniaturization High Resolution 3D Capability 12
  13. 13. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 13 Embedded Computer Vision   Embedded System + Camera + Computer Vision Algorithm Image Processing Image Recognition Image Capturing
  14. 14. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 14 Embedded Computer Vision   Embedded compute vision   Implements computer vision algorithms on low-cost, low-power, constrained hardware   Constrained hardware   Low-speed CPU   Low capacity memory   No floating-point (FPU)   Low-resolution image sensor
  15. 15. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai A B Research Status (1/3) 15
  16. 16. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Research Status (2/3) H K K 16
  17. 17. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 1717 Research Status (3/3) DARPA Grand Challenge Google Car Google Taxi
  18. 18. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 18 International Activities (1/2)   Special conferences   IEEE Int. Workshop on Embedded Computer Vision   IEEE Int. Conf. Distributed Smart Cameras   Special journal issues   IEEE Journal of Selected Topics in Signal Processing, vol. 2, no. 4, Aug. 2008   EURASIP Journal on Embedded Systems   Short courses in important CV conferences   CVPR07&08: Distributed vision processing in smart camera networks   ESC07: Embedded CV and smart cameras   ICASSP09: Distributed processing in smart cameras
  19. 19. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 19 International Activities (2/2)   Research projects and Lab.   Princeton Univ./Georgia Tech.: Embedded Systems Lab., Wayne Wolf   Stanford Univ. Wireless sensor networks Lab.   UCLA, CMU, MIT   Delft Univ. of Technology SmartCam Project   Graz Univ. of Technology
  20. 20. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 20 Three Ways for ECV   CPU (Central Processing Unit)   ARM, PowerPC   DSP (Digital Signal Processor)   TI, ADI, NXP   FPGA (Field Programmable Gate Array)   Altera, Xilinx
  21. 21. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 21 CPU   Stanford MeshEye   ARM 7 (55MHz), ZigBee node   3 image sensors   30x30 grayscale x 2   640x480 color x 1 Object detection (http://wsnl.stanford.edu/smartcam.html) "MeshEye:a hybrid-resolution smart camera mote for applications in distributed intelligent surveillance", IPSN-SPOTS, 2007
  22. 22. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 22 DSP   "Visual surveillance on DSP-based embedded platform," Graz Univ. of Technology, 2008(Phd. dissertation)   TI C6414 (600MHz)   Applications: Adaboost face detection, vehicle detection, license plate detection
  23. 23. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 23 FPGA (2/3) "Hardware, Design and Implementation Issues on a FPGA-Based Smart Camera," ICDSC, 2007
  24. 24. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 24 Hybrid: CPU+DSP   "Distributed Embedded Smart Cameras for Surveillance Applications," IEEE Computer, 2006. Developed for traffic surveillance
  25. 25. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Hybrid: CPU+DSP by SoC   TI OMAP 25
  26. 26. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 26 Hybrid: CPU+FPGA   UCLA Cyclops   MCU: Atmega128   CPLD: image processing   ECV app.: Hand gesture recognition http://research.cens.ucla.edu/ projects/2007/ Multiscaled_Actuated_Sensing/ Cyclops/
  27. 27. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Hybrid: CPU+FPGA by SoC   Xilinx Zynq 27
  28. 28. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 28 Challenges for ECV   Algorithm refinement   Parallel computation   Function partition, Multi-threading   Stream processing   Memory hierarchy optimization   Hardware design   Pipeline, SIMD, board design   Optimized programming skills   Fixed-point arithmetic   Memory management   Intrinsic commands
  29. 29. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 29 Our Experiences Entertainment Robot (CPU) Vision Sensor Network (CPU) Smart Video Surveillance (DSP) Wearable Camera (DSP)
  30. 30. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 30 Case 1 Entertainment Robot CPU
  31. 31. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 31 Sony AIBO
  32. 32. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Rolling Dice 32
  33. 33. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 33 Face Detection
  34. 34. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 34 Face Recognition
  35. 35. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 35 Facial Expression Recognition
  36. 36. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 36 Case 2 Embedded Vision Sensor CPU
  37. 37. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 37 FJUCam • Power consumption: 1W • Power sources: • 5V DC current • 5V Cell Battery • Weight: 35gm • Size: 6 x 4.5 x 5 (cm) (W x H x D) Three Modules 1. Main board, 2. Lens module 3. Storage module
  38. 38. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 38 Face Detection FJUCam Face Detection Image Display
  39. 39. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 39 Cyclops Robot   Color tracking
  40. 40. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 40 Case 3 Video Surveillance FPGA
  41. 41. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 41 Mobile Video Surveillance FPGA 行動通訊網路 (3G/GPRS) 網際網路 網路攝影機 監控伺服器 行動裝置 (3G行動電話)GSM 數據機 Wap Push 傳遞視訊串流 PCBackground subtraction, ... •  2.8 GHz Intel CPU •  Software: C/C++ •  Frame rate: 10 fps
  42. 42. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 42 Background Subtraction Current Frame - Background Image Update Post Processing Motion Object Image Background Image 1+kB kB 1+kP 1+kM ),(),( ),( 1 1 yxByxP yxM kk k −= + + ),()1(),( ),( 1 1 yxPyxB yxB kk k + + −+= αα Speed up by (1) Circuit design, (2) Paralization
  43. 43. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 43 FPGA Speed Up   Comparisons   PC: 2.8GHz CPU, C implementation   FPGA can speed up 500 times Clock(Hz) FPS CPU FPGA 2.8G 25M 51 10
  44. 44. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 44 Case 4 Wearable Camera X-EYE ARM + DSP
  45. 45. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 45 Goal 45
  46. 46. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Demo Video 46
  47. 47. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 47 Development 滑鼠 鍵盤 SD卡 USB-WIFI卡 讀卡機 USB-RS232 控制線 微投影機 自製 外殼 USB 連接線 顯示器 觸控面板 筆電 移動電源 BeagleBoard Camera USB Hub
  48. 48. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 48 Hardware 4848 Peripheral I/O • USB Host • JTAG • DVI-D video out • S-Video out • SD/MMC+ • Stereo in/out • RS-232 serial1 • Alternate power • USB 2.0 HS OTG USER OMAP3530 Processor • 600MHz Cortex-A8 • NEON+VFPv3 • 16KB/16KB L1 • 256KB L2 • 430MHz C64x+ DSP • 32K/32K L1 • 48K L1D • 32K L2 • Power VR SGX GPU • 64K on-chip RAM POP Memory • 256MB LPDDR RAM • 256MB NAND flash RESET 7.6 cm
  49. 49. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 49 System 2010.04.25 49
  50. 50. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 50 Software
  51. 51. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Next : UAV 51
  52. 52. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Next : Sleep Monitoring 52
  53. 53. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Next : Vehicular Object Detection 53
  54. 54. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 54 Future of Embedded Vision
  55. 55. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 55 Summary   Embedded CPU is not appropriate for ECV   Pros   Great flexibility   Fast development   Cons   No float-point arithmatics   Interrupt-based architecture   Smart phone with 4/8 cores is OK?
  56. 56. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 56 I Think …   Embedded compute vision   Low-cost, low-power, minimal hardware   High-Performance computer vision   Fast speed without cost, power, and hardware constraints From contradiction to convergence ?! Constrained Resource Abundant Resource Yes by multicore Contradiction
  57. 57. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 57 Challenges (1/2)   Algorithm decomposition   Function decomposition   Partition serial part and parallel part   Data flow analysis and data dependency analysis   Parallelism   Loop unrolling   Multithreading   Pipeline
  58. 58. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 58 Challenges (2/2)   Performance analysis method   For efficiency improvement   Implementation efforts   Choose a good embedded platform for computer vision   Software issues   Hardware issues   Programming skills   Multi-threading
  59. 59. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 59 Future Research Directions Heterogeneous Multicore ARM + DSP ARM + FPGA ARM + GPGPU
  60. 60. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai I Think Again …   Is there a development platform for embedded computer vision with   Built-in camera   AE, AF, AWB   High-performance processor   Heterogeneous multicore   Software development environment:   C, Matlab   OpenCV   Sample codes   Labs 60
  61. 61. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Embedded Vision Alliance 61
  62. 62. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai   Ramesh Raskar MIT Platform 62
  63. 63. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai   Andrew Adams Stanford Platform 63
  64. 64. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Taiwan Platform for Embedded Vision ? 64
  65. 65. 王元凱 Towards Embedded Computer Vision p. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai 本簡報授權聲明  此簡報內容採用 Creative Commons 「姓名標示 - 非商業性台灣 3.0 版」 授權條款  歡迎非商業目的的重製、散布或修改本簡報的內容,但 請標明: (1)原作者姓名:王元凱; (2)圖標示:  簡報中所取用的部份圖形創作乃截取自網際網路,僅供 演講者於自由軟體推廣演講時主張合理使用,請讀者不 得對其再行取用,除非您本身自忖亦符合主張合理使用 之情狀,且自負相關法律責任。 66

×