Intelligent Video Surveillance with Cloud Computing

  • 3,898 views
Uploaded on

It is a presentation for acamedia talk about cloud computing for intelligent video surveillance, i.e. VSaaS, given in 2010. Some of our research results are also presented in this presentation.

It is a presentation for acamedia talk about cloud computing for intelligent video surveillance, i.e. VSaaS, given in 2010. Some of our research results are also presented in this presentation.

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,898
On Slideshare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
0
Comments
0
Likes
14

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. 智慧型視訊監控 與雲端運算技術 Wang, Yuan-Kai (王元凱) Electrical Engineering Department, Fu Jen Univ. Taiwan (輔仁大學電機工程系) Email: ykwang@mail.fju.edu.tw URL: http://www.ykwang.tw 2010/11/24本著作採用創用CC 「姓名標示」授權條款台灣3.0版
  • 2. Wang, Yuan-Kai (王元凱) Cloud Vision p. 2 What about this Talk  Intelligent video surveillance for  Public safety  Home security  High performance cloud computing  GPGPU  Algorithm parallelization  VSaaS: video surveillance as a service  Mobile cloud computing  All demo examples are done by our ISLab(www.islab.tw)
  • 3. Wang, Yuan-Kai (王元凱) Cloud Vision p. 3 Who Am I  輔仁大學  電機工程系副教授,資訊中心前主任  中央大學網路學習科技研究所兼任副教授  中華民國影像處理與圖形辨識學會 理事、監事、秘書長  中華民國大專校院資訊服務協會 理事  Who Is Who in the World Who Is Who in Science and Engineering Who Is Who in Asia
  • 4. Wang, Yuan-Kai (王元凱) Cloud Vision p. 4 What Did I Do  經濟部學界科專計畫(2004~2012)  以視覺為基礎之智慧型環境建構  國科會計畫(2010~2011)  以視覺為基礎之睡眠障礙分析  產學合作  智慧型行動視訊監控系統  智慧型嵌入式攝影機之研究(I)(II)  不受限環境人臉辨識系統於機器人之應用  電腦視覺於家庭照護之應用  智慧型運輸系統之研究
  • 5. Wang, Yuan-Kai (王元凱) Cloud Vision p. 5 Contents 1. Intelligent Video Surveillance 2. Cloud Computing and GPGPU 3. VSaaS: Video Surveillance as a Service 4. Conclusions and Discussions
  • 6. 6 1. Intelligent Video Surveillance 1.1 Video Surveillance 1.2 Intelligent Video Surveillance 1.3 Demos
  • 7. Wang, Yuan-Kai (王元凱) Cloud Vision p. 7 1.1 Video Surveillance  Video surveillance  Use video camera to monitor an area for crime investigation  Two applications  Police and public safety  Home seciruty
  • 8. Wang, Yuan-Kai (王元凱) Cloud Vision p. 8 Video Surveillance Market • CCTV has been a mass-product market • Since the 911 event, the market is continuously increasing(百萬美元) Source: JP Freeman
  • 9. Wang, Yuan-Kai (王元凱) Cloud Vision p. 9 Who’s Watching You?  UK has the most CCTV cameras in Europe  4.2 million cameras which is  20% of the worlds CCTV  1 camera for every 14 people in UK  On average a person can be caught on camera 200-300 times a day
  • 10. Wang, Yuan-Kai (王元凱) Cloud Vision p. 10 Crimes Breaking by CCTV 92年度 93年度前3季 統計 案類/年度 件數 人數 件數 人數 件數 人數 總計 610 689 720 796 1330 1485 竊盜 364 412 425 447 789 859 搶奪 91 69 104 104 195 173 強盜 44 53 39 65 83 118 殺人 18 36 13 20 31 56 擄人勒贖 5 15 1 2 6 17 重傷害 4 8 3 3 7 11 恐嚇取財 5 6 1 2 6 8 強制性交 5 5 3 3 8 8 其他 74 85 131 150 205 235
  • 11. Wang, Yuan-Kai (王元凱) Cloud Vision p. 11 CCTV v.s. Crime Breaking  監視系統對破獲刑案的助益 120000 7000 監視器數量 100000 6000 因監視器破獲件數 5000 80000 4000 60000 3000 40000 2000 20000 1000 0 0 92 93 94 95 96 97 年度  監視器數量和監視器破獲件數兩者間呈現 正向關係
  • 12. Wang, Yuan-Kai (王元凱) Cloud Vision p. 12 Important Crime Cases  近年來運用路口監視器 偵破社會矚目重大案件  白米炸彈客  汐止市殺警奪槍案  蠻牛千面人案  台南國道襲警奪槍案  新莊襲警奪槍案  英國倫敦地鐵爆炸案  台中角頭槍殺案
  • 13. Wang, Yuan-Kai (王元凱) Cloud Vision p. 13 Case Study  94年5月17日台中市蠻牛千面人案, 造成全省恐慌  破案關鍵在於  幾個放置毒蠻牛 的超商監視器 錄到千面人身影  歹徒車號被 提款機監視器 清楚拍下  動員500警員 觀看6000小時的錄影資料
  • 14. Wang, Yuan-Kai (王元凱) Cloud Vision p. 14 Home Security  Video surveillance for homecare  Use CCTV/IP cameras to monitor homes  Like千里眼@中華電信  Increasing demands for  Burglar care  Child care  Pet care  Elder care  Community/neighborhood care  Health care  Sleep care
  • 15. Wang, Yuan-Kai (王元凱) Cloud Vision p. 15 A Survey @ Taiwan 2009  台灣經濟部通訊產業推動小組委託 資策會進行調查  台北、台中、高雄等 3大都會區18歲以上 ,對社區大樓或居家 服務提供多媒體娛樂 與商務或智慧生活 應用有興趣的民眾  668份有效問卷
  • 16. Wang, Yuan-Kai (王元凱) Cloud Vision p. 16 Summary of the Survey  安全監控是3群年齡層都最重視的應 用  第二重視的應用:隨年齡有所不同
  • 17. Wang, Yuan-Kai (王元凱) Cloud Vision p. 17 Burglar Care  Care about burglar events when You are at home Nobody is at home (Vacation home)
  • 18. Wang, Yuan-Kai (王元凱) Cloud Vision p. 18 Child Care - Baby Monitor  Care about babys  Wake-up  Crying Audio/video  Suffocation Wireless Device
  • 19. Wang, Yuan-Kai (王元凱) Cloud Vision p. 19 Child Care – Nanny Monitor  Nanny may not appropriately take care of the baby  Use a hidden camera for monitoring
  • 20. Wang, Yuan-Kai (王元凱) Cloud Vision p. 20 Elder Care  Home : aging in place  Monitoring the elder by cameras  Fall detection by cameras  Institutional and nursing homes
  • 21. Wang, Yuan-Kai (王元凱) Cloud Vision p. 21 Pet Care • While the owner is away from home • A surveillance camera helps assure • Pets well-being • Dogs and cats dont cause damage
  • 22. Wang, Yuan-Kai (王元凱) Cloud Vision p. 22 Community Care  Parking lot, fence  Dangerous public place ex., swimming pool  Access control
  • 23. Wang, Yuan-Kai (王元凱) Cloud Vision p. 23 Health Care  Video conferencing at home with doctors for diagnostics
  • 24. Wang, Yuan-Kai (王元凱) Cloud Vision p. 24 Sleep Care  Feel not well for sleeping?  Go to sleep at hospitals sleep center  Bio-signal and video are recorded  OSA: Obstructive Sleep Apnea  Why not sleep at home? CPAP
  • 25. Wang, Yuan-Kai (王元凱) Cloud Vision p. 25 Video Surveillance Generations Paradigm shift of video surveillance  Role from security monitoring to the personalized video contents  Advent of the intelligent surveillance Changes in technology & desire 1. Network 2. Video compression 3. Live images Intelligent Surveillance IP Surveillance CCTV (DVR) CCTV (VCR) 1G 2G 3G
  • 26. Wang, Yuan-Kai (王元凱) Cloud Vision p. 26 CCTV Video Surveillance Video Display & Record VCR / DVR Analog Multiplexer components Centralized Monitoring Video Capture analogue analogue analogue analogue
  • 27. Wang, Yuan-Kai (王元凱) Cloud Vision p. 27 Digital Video Surveillance High scalibility IPCam + analog camera Network transmission Remote control Digital storgage digital Network digital digital analogue analogue analogue analogue
  • 28. Wang, Yuan-Kai (王元凱) Cloud Vision p. 28 Visual Surveillance Visual Surveillance = Digital CCTV + Video Analytics Smart/Intelligent Surveillance
  • 29. Wang, Yuan-Kai (王元凱) Cloud Vision p. 29 1.2 Video Analytics  Intelligent video surveillance  Use video camera to monitor an area for  Crime prevention  Intelligent ICT service  From video surveillance to visual surveillance  Impose video analytics by computer vision algorithm
  • 30. Wang, Yuan-Kai (王元凱) Cloud Vision p. 30 Why Visual Surveillance  Too many cameras, too few human guards  “After only 20 minutes, human attention to video monitors degenerates to an unacceptable level.” (Sandia National Laboratories)
  • 31. Wang, Yuan-Kai (王元凱) Cloud Vision p. 31 Applications of Visual Surveillance
  • 32. Wang, Yuan-Kai (王元凱) Cloud Vision p. 32 Video Analytics影像擷取 相機異常偵測 人臉辨識 查詢、過濾、聯防 Video Image Object Object Object Behavior Capture Enhance /Event Tracking /Event Analysis Retrieval Detection Recognition 影像強化 警戒線、路徑追蹤 流浪漢監控 跌倒、人潮行為分析
  • 33. Wang, Yuan-Kai (王元凱) Cloud Vision p. 33 IBM S3 Exploratory Computer Vision Group in IBM T.J. Watson Research Center. http://www.research.ibm.com/ecvg/
  • 34. Wang, Yuan-Kai (王元凱) Cloud Vision p. 34 ObjectVideo
  • 35. Wang, Yuan-Kai (王元凱) Cloud Vision p. 35 ITRI
  • 36. Wang, Yuan-Kai (王元凱) Cloud Vision p. 36 VBIE科專計畫 (1/3)  經濟部學界科專計畫  以視覺為基礎之智慧型環境建構 Construction of Vision-Based Intelligent Environment (VBIE)  第一期4年計畫: 2004/5 ~ 2008/4  第二期4年計畫: 2008/11 ~ 2012/10  參與人力  29 位教授,來自18 所大學與研究機構  110 位研究人員
  • 37. Wang, Yuan-Kai (王元凱) Cloud Vision p. 37 經濟部學界科專計畫 (2/3)  智慧型建築  目標:開發智慧型建築內部空間不可或缺的全 方位、主動式、機動性的智慧性視訊監控系統 A1 日夜活動式廣域安全監視 系統 A2 視訊監控中央管理系統 A3 室內突發事件分析系統  攝影機網路  感測網路
  • 38. Wang, Yuan-Kai (王元凱) Cloud Vision p. 38 經濟部學界科專計畫 (3/3)  智慧型社區與城市  目標:開發戶外社區及城市大範圍區域之穩定、成熟而 具產品面向的智慧性視覺監控系統 B1 人車偵測與辨識系統 B2 都會區人物追蹤系統 B3 室外事件分析與搜尋系統
  • 39. Wang, Yuan-Kai (王元凱) Cloud Vision p. 39 Activities in the VBIE Project  參加國際展覽  研究技術需  展示化:技術需能常駐展示  系統化:大型整合展示  指標化:技術有量化指標  市場化:建立產業鏈地圖、政策規劃  專利化:專利佈局、專利地圖分析  商品化:網路行銷百餘項技術  參與國際標準制訂(ONVIF)  引導業界投資  與警政機關合作
  • 40. Wang, Yuan-Kai (王元凱) Cloud Vision p. 40 Current Solutions of Police
  • 41. Wang, Yuan-Kai (王元凱) Cloud Vision p. 41 Equipments in the Pole
  • 42. Wang, Yuan-Kai (王元凱) Cloud Vision p. 42 Cameras on the Poles 固定式攝影機及密閉式雙層鋁製防護罩 固定式攝影機及密閉式雙層鋁製防護罩
  • 43. Wang, Yuan-Kai (王元凱) Cloud Vision p. 43 DVR in Closed Case 監視錄影設備
  • 44. Wang, Yuan-Kai (王元凱) Cloud Vision p. 44 A Proposed Architecture for Police Office (1/2) 鄧紹華、詹毓青,智慧型視訊監控技術在警政治安上之可行性研究, 中央警察大學資訊管理所碩士論文,2009
  • 45. Wang, Yuan-Kai (王元凱) Cloud Vision p. 45 A Proposed Architecture for Police Office (2/2) 鄧紹華、詹毓青,智慧型視訊監控技術在警政治安上之可行性研究, 中央警察大學資訊管理所碩士論文,2009
  • 46. Wang, Yuan-Kai (王元凱) Cloud Vision p. 46 Current Solutions for Home Security  Standalone system with three components  Camera: CCTV, IPCam, SmartCam  Storage: DVR (Digital Video Recorder) 2. DVR  View: PC, Smart Phone 3. Viewer 1.Camera
  • 47. Wang, Yuan-Kai (王元凱) Cloud Vision p. 47 Problems of Current Solution  Purchase, maintenance, update of  Facility & Storage  Cabling Rental service could be better than buying a home security system Like: electricity, gas, cable TV, ... ⇒ Cloud Computing
  • 48. Wang, Yuan-Kai (王元凱) Cloud Vision p. 48 Problems of Current Solution  Time to watch is unknown  Real-time event alert is helpful  Lack of smart sensors Instead of video recording in the cloud, How about video analytics? ⇒ Intelligent Video Surveillance by Cloud Computing
  • 49. Wang, Yuan-Kai (王元凱) Cloud Vision p. 49 Evolution to Cloud 監控 儲存 管理 分析 虛擬化 智慧分析 IP網路 監控 IP視訊監控 DVR類比CCTV(DVR儲存) Time Lapse 數位 VCR類比CCTV(VCR儲存) CCTV camera VSaaS 類比類比CCTV 時間 1950~ 1980 1990 2000 2010 2011 部份資料來源:拓璞產業研究所,2008年5
  • 50. Wang, Yuan-Kai (王元凱) Cloud Vision p. 50 1.3 DEMOS  1. Camera Tampering Detection  2. Tripwire  3. Face Recognition  4. Smart Building
  • 51. Wang, Yuan-Kai (王元凱) Cloud Vision p. 51 DEMO 1: Camera Tampering Detection  Possible tampering  Spray-painting  Replacement  Hit/collision  Defocus  Blocking  ...
  • 52. Wang, Yuan-Kai (王元凱) Cloud Vision p. 52 Motivation  For a large video surveillance installation  How to ensure every camera is OK?  A case study  A police IDC with more than 600 cameras  A person is responsible for checking  One Day One check in morning  Takes about 40 minutes  Then you never know it functions well until tomorrow
  • 53. Wang, Yuan-Kai (王元凱) Cloud Vision p. 53 Replacement  Intentionally by human  False alarm: Earthquake 5 3
  • 54. Wang, Yuan-Kai (王元凱) Cloud Vision p. 54 Spray Painting  Intentionally by human 5 4
  • 55. Wang, Yuan-Kai (王元凱) Cloud Vision p. 55 Defocusing  Intentionally by human  False alarm: water drops 5 5
  • 56. Wang, Yuan-Kai (王元凱) Cloud Vision p. 56 Blocking – Full Occlusion  Intentionally by human  False alarm: passing of large objects 556 6
  • 57. Wang, Yuan-Kai (王元凱) Cloud Vision p. 57 Blocking – Partial Occlusion  Intentionally by human 貼紙 廣告傳單 寶特瓶 包裝盒 雨傘 塑膠玩具 557 7
  • 58. Wang, Yuan-Kai (王元凱) Cloud Vision p. 58 The System  Sabotage detection before visual surveillance algorithms  Server-based solution for large-scale surveillance in a police IDC  > 400 cameras  Advantages  No more daily check with 40 minutes  Alert functions 24 hours
  • 59. Wang, Yuan-Kai (王元凱) Cloud Vision p. 59 Architecture (Current) Our System ... 攝影機異常 合成影像 偵測系統 偵測紀錄 ... 影像分配器 控制中心 ... 整段影像或 DVR 紀錄影像 5 9
  • 60. Wang, Yuan-Kai (王元凱) Cloud Vision p. 60 Demo 2: Tripwire  Tripwire detection is an important application for proactive crime prevention  Restricted ingress and egress  Unidirectional or bidirectional  Precondition: Precise Moving Object Detection d(Position) and Tracking
  • 61. Wang, Yuan-Kai (王元凱) Cloud Vision p. 61 Moving Object Detection in Day-and-Night  Background subtraction is the most important method  However, it can not work at night  Image processing techniques must be added  Night vision should be important because crimes usually happens at night
  • 62. Wang, Yuan-Kai (王元凱) Cloud Vision p. 62 Cant do tripwire detection in these two cases
  • 63. Wang, Yuan-Kai (王元凱) Cloud Vision p. 63 Example with GUI  Various Input Interface:  Webcam  IP Camera  Analog Camera  Customize Multi Tripwire  Self-defining direction  Active alarm & log(xml, video)
  • 64. Wang, Yuan-Kai (王元凱) Cloud Vision p. 64 Applications  Traffic control 警戒線:對由下而上的車 輛進行警報與計數
  • 65. Wang, Yuan-Kai (王元凱) Cloud Vision p. 65 DEMO 3: Face Recognition  Video-based method in unconstrained environment Training image Test video
  • 66. Wang, Yuan-Kai (王元凱) Cloud Vision p. 66 Applications  門禁管理  生物特徵認證  相簿管理  人員計數 http://picasa-readme.blogspot.com/2009/09/picasa-35-now-with-name-tags-build-7967.html
  • 67. Wang, Yuan-Kai (王元凱) Cloud Vision p. 67 DEMO 4: Smart Building  Integrated monitoring within building  Tracking target: person  11 techniques are integrated by top- down design  A long-term test site is built
  • 68. Wang, Yuan-Kai (王元凱) Cloud Vision p. 68 Heterogeneous Cameras  We use various kinds of cameras 環場攝影機 PTZ攝影機 紅外線熱像攝影機 固定式攝影機 活動攝影機 活動攝影機畫面
  • 69. Wang, Yuan-Kai (王元凱) Cloud Vision p. 69 The Scenario NTSC一般攝影機 PTZ網路攝影機 魚眼攝影機
  • 70. 2. Cloud Computing and GPGPU 2.1 Cloud Computing 2.2 High Performance Computing 2.3 HPC Cloud by GPGPU 2.4 CUDA 2.5 Parallel Computing 2.6 Demos
  • 71. Wang, Yuan-Kai (王元凱) Cloud Vision p. 71 2.1 Cloud Computing  Put computation and data in the cloud 1st Cloud Nth Cloud Computing Storage Application Computing Storage Application Web Server Web Server Provider 1 ..... Provider N Internet End-User PC Notebook Phone/PDA
  • 72. Wang, Yuan-Kai (王元凱) Cloud Vision p. 72 Technologies for Computing Cloud  It needs a data center in the cloud with high computing resources  CPU clusters (Server Farm)  Storage: SAN  Fiber networking  Also it needs  Virtualization  Web Service
  • 73. Wang, Yuan-Kai (王元凱) Cloud Vision p. 73 Services in Cloud Computing  AaaS Architecture as a Service  BaaS Business as a Service   CaaS DaaS Computing as a Service Data as a Service XaaS  DBaaS Database as a Service  EaaS Ethernet as a Service  FaaS Frameworks as a Service  GaaS Globalization or Governance as a Service  HaaS Hardware as a Service  IMaaS Information as a Service  IaaS Infrastructure or Integration as a Service  IDaaS Identity as a Service  LaaS Lending as a Service  MaaS Mashups as a Service  OaaS Organization or Operations as a Service  SaaS Software or Storage as a Service  PaaS Platform as a Service  TaaS Technology or Testing as a Service  VaaS Voice as a Service  VSaaS Video Surveillance as a Service
  • 74. Wang, Yuan-Kai (王元凱) Cloud Vision p. 74 Cloud Architecture User-Level 應用 Social Computing, Enterprise, ISV,… User-Level 程式語言 Web 2.0 介面, Mashups, Workflows, … S Middleware a P 控制 a a I Core Qos Neqotiation, Ddmission Control, Pricing, SLA Management, Metering… S a a Middleware 虛擬化 S a VM, VM management and Deployment S 硬體設施 System Level Infrastructure: Computer, Storage, Network
  • 75. Wang, Yuan-Kai (王元凱) Cloud Vision p. 75 Service Hierarchy More about SaaS for SaaS multimedia PaaS IaaS Hardware Data Storage Clusters Others Centers s
  • 76. Wang, Yuan-Kai (王元凱) Cloud Vision p. 76 SaaS for Multimedia  Challenges  Can IaaS afford multimedia computing, such as intelligent video surveillance?  Digital signal is continuously fed into a massive data set  Intensive computing power is necessary for ISP(Image and Signal Processing) We need High Performance Computing (HPC)
  • 77. Wang, Yuan-Kai (王元凱) Cloud Vision p. 77 2.2 High Performance Computing (HPC)  Classically it is called supercomputing  Supercomputing has been combined with networking technology and evolved into HPC Virtualization Utility Cluster Service Parallel Model Distributed High Cloud Web 2.0 Grid Performance Computing Computing
  • 78. Wang, Yuan-Kai (王元凱) Cloud Vision p. 78 Why HPC?  It is used for  Massive data processing  Intensive computation  Example application: video processing  Intelligent video surveillance, Computer vision, ...  Complexity of video processing  One video: 1 Megapixels, 30fps, 100 flops per pixel  ⇒ 3 Gigaflops per video
  • 79. Wang, Yuan-Kai (王元凱) Cloud Vision p. 79 New Approaches for HPC  Cluster/distributed computing  MAP-REDUCE (Google)  MPI  Multi-processing computing  Multicore CPU  GPGPU  FPGA/DSP  CellBE  Combination  GPU cluster
  • 80. Wang, Yuan-Kai (王元凱) Cloud Vision p. 80 HPC Architecture of IaaS The end user sees a finished application Compute Power End user Load Firewall Balancer Improved servers by Virtual Machine is deployed and started co-processing: Virtual Machine From multicore Automation to manycoreSoftware Virtual The virtual machine is Owner Machine uploaded to storage and configures to use storage. Storage IaaS Vendor
  • 81. Wang, Yuan-Kai (王元凱) Cloud Vision p. 81 Two Approaches to HPC  Symmetric multiprocessing (SMP)  Multicore CPU, GPGPU, multicore DSP  Homogeneous computing  Asymmetric multiprocessing (AMP)  CPU+GPGPU, CPU+FPGA, CPU+DSP  Heterogeneous computing
  • 82. Wang, Yuan-Kai (王元凱) Cloud Vision p. 82 SMP: Multicore CPU  Two or more CPUs on a chip Core i7: Quad cores Intel Core 2 Duo OneProcessor With multiple execution Cores
  • 83. Wang, Yuan-Kai (王元凱) Cloud Vision p. 83 SMP: Multicore CPU  Increased compute density  Increase performance with energy saving
  • 84. Wang, Yuan-Kai (王元凱) Cloud Vision p. 84 SMP: GPGPU  GPGPU has more homogeneous cores than CPU  120 ~ 512 cores  GPGPU is more powerful than multicore CPU  For DSP processing  Vendors: nVidia, ATI, Intel
  • 85. Wang, Yuan-Kai (王元凱) Cloud Vision p. 85 AMP: Co-processing  Multicore CPU + Stream processor CellBE ISP IO FPGA (Image and Signal GPGPU Processing)
  • 86. Wang, Yuan-Kai (王元凱) Cloud Vision p. 86 CellBE (1/2)  By IBM, Sony, Toshiba alliance from 2000  One PowerPC core, 8 compute cores (SPEs)  Sony PS3
  • 87. Wang, Yuan-Kai (王元凱) Cloud Vision p. 87 CellBE (2/2)  Toshiba: SpursEngine  IBM:  Blade server QS22  RoadRunner
  • 88. Wang, Yuan-Kai (王元凱) Cloud Vision p. 88 FPGA+CPU (1/2)  Ex.: XtremeData Inc. XD1000/2000
  • 89. Wang, Yuan-Kai (王元凱) Cloud Vision p. 89 FPGA+CPU (2/2)
  • 90. Wang, Yuan-Kai (王元凱) Cloud Vision p. 90 GPGPU+CPU  GPU has been a co-processor of CPU  GPU helps improve graphics processing  CPU focuses SMP + AMP on IO control
  • 91. Wang, Yuan-Kai (王元凱) Cloud Vision p. 91 GPU Cluster (1/2)  GPU cluster is a brand new architecture for HPC  GPU+CPU as a compute node  Many compute nodes are clustered as a unified HPC computer  Early experimented from 2010  UIUC NCSA: AC cluster  Maryland CPU-GPU cluster  CSIRO GPU cluster  Taiwan: NCHC, NTU  More researches are actively studied
  • 92. Wang, Yuan-Kai (王元凱) Cloud Vision p. 92 GPU Cluster (2/2)  Example: Maryland CPU-GPU cluster • If utility, load balancing, and virtualization can be incorporated into the GPU cluster architecture • Cloud computing can be more powerful •Especially for intelligent video surveillance
  • 93. Wang, Yuan-Kai (王元凱) Cloud Vision p. 93 Multi-Everything  Parallelism everywhere with SW & HW assistance Virtual •New Parallel Languages Container OS OS OS •New Threading tools VMM: •Thread Management & Abstraction layers Operating •Transactional memory App App App •Auto-threading compilers System: •Auto-threading hardware Application: Thread Thread Thread Code/Data Code/Data Code/Data Thread: Segment Segment Segment Chip: CPU GPU DSP
  • 94. Wang, Yuan-Kai (王元凱) Cloud Vision p. 94 2.3 HPC Cloud by GPGPU  HPC cloud is  Cloud computing with HPC architecture and performance  HPC As A Service (HPCaaS): HPC in the cloud
  • 95. Wang, Yuan-Kai (王元凱) Cloud Vision p. 95 Ex.: Amazon HPC  Five families of Amazon EC2  Standard, Micro, High-CPU, High- Memory, Cluster Compute  Cluster Compute  Large computational problems for business and researchers  Provide Amazon Elastic MapReduce service (Hadoop)  Cooperated with researchers at the Lawrence Berkeley National Laboratory to develop the HPC cloud offering
  • 96. Wang, Yuan-Kai (王元凱) Cloud Vision p. 96 HPC Cloud by GPGPU  In my viewpoint  GPGPU can be an enabling technology for HPC cloud  For single computer/one compute node  GPU can be used to greatly enhance performance  Personal supercomputer (nVidia)  If one compute node is not enough  Clustering many compute nodes
  • 97. Wang, Yuan-Kai (王元凱) Cloud Vision p. 97 Advantages of GPGPU for Cloud Computing  Compared with PC-based cloud computing  Improve space density  Improve compute performance  Compared with supercomputer  Application/software portability  Power saving  Similar teraflops performance
  • 98. Wang, Yuan-Kai (王元凱) Cloud Vision p. 98 Challenges in HPC Cloud  Virtualization  Software development  No more free lunch for software  Software has to be re-written  Multi-threading  Programming model  Multi-threading  SPMD(Single Program Multiple Data) Lets take GPGPU/CUDA model as an example
  • 99. Wang, Yuan-Kai (王元凱) Cloud Vision p. 99 2.4 CUDA  ex.: nVidia Fermi/GT200  Tesla 20-series/10-series Multicore Many-core
  • 100. Wang, Yuan-Kai (王元凱) Cloud Vision p. 100 Memory Hierarchy  Good for locality principle  Temporal locality  Spatial locality  Good for ISP
  • 101. Wang, Yuan-Kai (王元凱) Cloud Vision p. 101 CUDA  Parallel programming for nVidias GPGPU  Use C/C++ language  Java, Fortran, Matlab, ...  When executing CUDA programs, the GPU operates as coprocessor to the main CPU
  • 102. Wang, Yuan-Kai (王元凱) Cloud Vision p. 102 CUDA Hardware Environment: CPU+GPU  GPU PCI-E  Organizes, interprets, and CPU GPU communicates information  GPU  Handles the core processing on large quantities of parallel information  Compute-intensive portions of applications that are executed many times, but on different data, are extracted from the main application and compiled to execute in parallel on the GPU
  • 103. Wang, Yuan-Kai (王元凱) Cloud Vision p. 103 Processing Flow on CUDA Main 2 Memory CPU 3 Copy processing 5 Instruct the data Copy the processing result 4 1 Memory for GPU Execute Allocate parallel in device memory each core 6 Release device memory
  • 104. Wang, Yuan-Kai (王元凱) Cloud Vision p. 104 GPUs for Multimedia 3.5X 10 X 10 X PowerDirector7 Ultra CUDA JPEG Decoder DivideFrame GPU Decoder 26 X 10 X Hyperspectral Image GPU Decoder Motion Estimation for Compression on (Vegas/Premiere) - H.264/AVC on NVIDIA GPUs Using the Power of Multiple GPUs NVIDIA Graphic Card to Using NVIDIA CUDA Decode H.264 Video Files
  • 105. Wang, Yuan-Kai (王元凱) Cloud Vision p. 105 GPUs for Computer Vision(1/2) 87 X 26 X 200 X 100 XCUDA SURF – A Real-time Leukocyte Tracking: Real-time Spatiotemporal Image Denoising withImplementation for SURF ImageJ Plugin Stereo Matching Using the Bilateral Filter TU Darmstadt University of Virginia Dual-Cross-Bilateral Grid Wlroclaw University of Technology 85 X 100 X 8X 13 X Digital Breast Fast Optical Flow on GPU A Framework for Efficient Accelerating Advanced MRI Tomosynthesis At Video Rate for Full HD and Scalable Execution of Reconstructions Reconstruction Resolution Domain-specific Templates University of Illinois Massachusetts General Onera On GPU Hospital NEC Labs, Berkeley, Purdue
  • 106. Wang, Yuan-Kai (王元凱) Cloud Vision p. 106 GPUs for Computer Vision(2/2) 20 X 13 X 109 X 263 X GPU for Surveillance Fast Human Detection with Fast Sliding-Window GPU Acceleration of Object Cascaded Ensembles Object Detection Classification Algorithm Using NVIDIA CUDA 300 X 10 X 45 X 3X Audience Measurement – Real-time A GPU Accelerated Canny Edge Detection Real-time Video Analysis Visual Tracker by Evolutionary for Counting People, Face Stream Processing Computer Vision System Detection and Tracking
  • 107. Wang, Yuan-Kai (王元凱) Cloud Vision p. 107 Programming Challenges of CUDA  We have to manually parallelize the algorithm  We need expertise both in  Algorithms of image and signal processing  Filtering, frequency analysis, compression, feature extraction, recognition, ...  Theory, tools and methodology of parallel computing  Communication, synchronization, resource management, load balancing, debugging, ...
  • 108. Wang, Yuan-Kai (王元凱) Cloud Vision p. 108 2.5 Parallel Computing  Serial Computing  Parallel Computing
  • 109. Wang, Yuan-Kai (王元凱) Cloud Vision p. 109 What is Parallel Computing?  Solving a problem simultaneously with multiple processing elements (PE’s), or multiple cores  Dividing the problem into independent parts  Each PE executes its part concurrently with the other PE’s
  • 110. Wang, Yuan-Kai (王元凱) Cloud Vision p. 110 Parallelization  Multicore/Multi-threading  Data Parallelization  Data distribution  Parallel convolution  Reduction algorithm  Amdahl’s law  Memory Hierarchy Management  Locality principle  Program accesses a relatively small portion of the address space at any instant of time
  • 111. Wang, Yuan-Kai (王元凱) Cloud Vision p. 111 Four Steps to Create a Parallel Program Partitioning D O e A r c s c o s p p h p p M m i 0 1 e 0 1 a p0 p1 p g s p o n t p s i m p p r p p i p2 p3 e 2 3 a n t t 2 3 g i n i o t o n n Sequential Tasks Parallel Processors Processes programcomputation
  • 112. Wang, Yuan-Kai (王元凱) Cloud Vision p. 112 Methods to Decompose a Problem  Two basic approaches to partition computational work  Domain decomposition GPGPU  Partition the data used Cooperate in solving the problem  Function decomposition CPU  Partition the jobs (functions) from the overall work (problem)
  • 113. Wang, Yuan-Kai (王元凱) Cloud Vision p. 113 Domain Decomposition (1/3)  An image example  It is 2D data  Three popular partition ways
  • 114. Wang, Yuan-Kai (王元凱) Cloud Vision p. 114 Domain Decomposition (2/3)  Domain data are usually processed by loop  for (i=0; i<height; i++) for (j=0; j<width; j++) img2[i][j] = RemoveNoise(img1[i][j]); j i The X-ray image of a circuit board Original image(img1) Enhanced image(img2)
  • 115. Wang, Yuan-Kai (王元凱) Cloud Vision p. 115 Domain Decomposition (3/3) j i A three-block partition example OpenMP  // Thread 1 CUDA(SPMD) for (i=0; i<height/3; i++) for (j=0; j<width; j++) img2[i][j] = RemoveNoise(img1[i][j]);  // Thread 2 for (i=height/3; i<height*2/3; i++) fork(threads) subdomain 1 subdomain 2 subdomain 3 for (j=0; j<width; j++) i=0 i=4 i=8 img2[i][j] = RemoveNoise(img1[i][j]); i=1 i=5 i=9  // Thread 3 i=2 i=6 i=10 i=3 i=7 i=11 for (i=height*2/3; i<height; i++) for (j=0; j<width; j++) join(barrier) img2[i][j] = RemoveNoise(img1[i][j]);
  • 116. Wang, Yuan-Kai (王元凱) Cloud Vision p. 116 Function Decomposition (1/2)  Functional decomposition is a way to exploit task parallelism  Task parallelism by threading  Fork operation creates multiple threads  Each thread runs on a PE  A thread may perform the same or different operations on the same or different data
  • 117. Wang, Yuan-Kai (王元凱) Cloud Vision p. 117 Function Decomposition (2/2)  Ex.: Image Streaming  An image capture & streaming program  Partitioned into 3 tasks (functions)  Each function is set to be a thread, running in a PE, or CPU core Compress Capture Images, Stream the Internet Images Process Images Images Input Output Computer
  • 118. Wang, Yuan-Kai (王元凱) Cloud Vision p. 118 Issues with Parallelization  Good parallel programs  Execute correctly  with good speedup  Ideal speedup by Amdahls law  Speedup = N if you has N cores  However, no ideal speedup exists  Because parallel overhead, such as  Data communication  Data dependencies and synchronization  Other issues: design overhead  No free lunch for software development
  • 119. Wang, Yuan-Kai (王元凱) Cloud Vision p. 119 2.6 DEMOS  1. FPGA for Moving Object Detection  2. GPGPU for Image Restoration  3. GPGPU for Feature Extraction
  • 120. Wang, Yuan-Kai (王元凱) Cloud Vision p. 120 DEMO 1: FPGA for Moving Object Detection Background subtraction, ... • 2.8 GHz Intel CPU • Software: C/C++ FPGA • Frame rate: 10 fps for 1 channel
  • 121. Wang, Yuan-Kai (王元凱) Cloud Vision p. 121 Background Subtraction Current Frame B k + 1 Background M k +1 ( x, y ) P k+1 Image Update Background Image = Pk +1 ( x, y ) − Bk ( x, y ) - Bk M k + 1 Bk +1 ( x, y ) = αBk ( x, y ) + (1 − α ) Pk +1 ( x, y ) Post Processing Motion Object Image Speed up by (1) Circuit design, (2) Parallelization
  • 122. Wang, Yuan-Kai (王元凱) Cloud Vision p. 122 Parallelization by Hardware  Parallelism: 7-level pipeline  SIMD with stream processing
  • 123. Wang, Yuan-Kai (王元凱) Cloud Vision p. 123 Experimental Comparison  PC: 2.8GHz CPU, C implementation  FPGA: 25MHz, Verilog HDL 2.8G 51 Speedup ≈ 100 * 5 = 500 CPU FPGA 25M 10 Clock(Hz) FPS
  • 124. Wang, Yuan-Kai (王元凱) Cloud Vision p. 124 DEMO 2: GPGPU for Image Restoration  Restore and enhance an image  Its complexity is high for large images Original Complexity: Restored O(N2)
  • 125. Wang, Yuan-Kai (王元凱) Cloud Vision p. 125 The Method CPU GPGPU Copy Data to Illumination GPGPU Estimation Illumination Compensation Reduction Algorithm Copy Data Contrast from GPGPU Enhancement Intel Core 2 - 2 cores Tesla C1060 - 240 SPs (3.0GHZ) (1.296GHZ)
  • 126. Wang, Yuan-Kai (王元凱) Cloud Vision p. 126 Parallelization by GPGPU  Multicore/Multi-threading  Tesla C1060 : 240 SP (Stream Processor)  CUDA: , Thread , Block , Grid  Data Parallelization  Parallel convolution  Parallel convolution M pixels PE data time 1 pixels pixels 1 pixels t0 t1 t2 t3 t4 t5 A(0) A(0)+A(1) A(0)+A(1)+A(2)+A(3) sum 0 1 A(1) { M PE i PE i 2 A(2) A(2)+A(3)pixels pixels pixels pixels A(3) 3 { 4 A(4) A(4)+A(5) A(4)+A(5)+A(6)+A(7) pixels 5 A(5) 1 pixels 1 pixels 6 A(6) A(6)+A(7) pixels 7 A(7)
  • 127. Wang, Yuan-Kai (王元凱) Cloud Vision p. 127 Experimental Results (1/2) Original images CPU results GPGPU results
  • 128. Wang, Yuan-Kai (王元凱) Cloud Vision p. 128 Experimental Results (2/2) Original images CPU results GPGPU results
  • 129. Wang, Yuan-Kai (王元凱) Cloud Vision p. 129 Execution Time 18000 16000 14000 12000 CPU-SSR ms 10000 CUDA-SSR CPU-MSR 8000 CUDA-MSR 6000 CPU-MSRCR CUDA-MSRCR 4000 2000 0 256 x 256 512 x 512 1024 x 1024 2048 x 2048 Image Sizes
  • 130. Wang, Yuan-Kai (王元凱) Cloud Vision p. 130 GPGPU Speedup over CPU 35 30 30x speedup Speedup 25 20 CUDA-SSR 15 CUDA-MSR CUDA-MSRCR 10 5 0 Image 256 x 256 512 x 512 1024 x 1024 2048 x 2048 Sizes
  • 131. Wang, Yuan-Kai (王元凱) Cloud Vision p. 131 DEMO 3: GPGPU for SIFT Feature Extraction  SIFT  Scale Invariant Feature Transform  Invariance of feature points  Translation  Rotation  Scale
  • 132. Wang, Yuan-Kai (王元凱) Cloud Vision p. 132 Applications of SIFT Object recognition/tracking Image retrieval Autostitch
  • 133. Wang, Yuan-Kai (王元凱) Cloud Vision p. 133 Parallelize SIFT by GPGPUIntel Q9400 Geforce GTS 250Quad cores 128 SPs(2.66GHz) (1.836GHz)
  • 134. Wang, Yuan-Kai (王元凱) Cloud Vision p. 134 Experimental Results CPU GPU
  • 135. Wang, Yuan-Kai (王元凱) Cloud Vision p. 135 Execution Time CPU: 10 seconds in average ms GPGPU: 0.8 seconds in average
  • 136. Wang, Yuan-Kai (王元凱) Cloud Vision p. 136 Speedup 13x speedup in average
  • 137. 3. VSaaS Computing Cloud Mobile Cloud Demo
  • 138. Wang, Yuan-Kai (王元凱) Cloud Vision p. 138 VSaaS  Video Surveillance as a Service  Pay-as-you-go (Rental) Service  Bringing SaaS to physical security, hosted & managed video  Potential to make video surveillance cheaper and easier to deploy and use  Re-shape the home and small business market segments for video surveillance  Only few systems started from 2010
  • 139. Wang, Yuan-Kai (王元凱) Cloud Vision p. 139 Advantages of VSaaS  Economics  Reduced TCO (Total Cost of Ownership)  Eliminated time and effort  Efficiency  Fast and flexible deployment  Easy scalability and upgrade  Access anytime anywhere
  • 140. Wang, Yuan-Kai (王元凱) Cloud Vision p. 140 Architecture DVR by SaaS  SaaS Camera  VSaaS  Camera should be place in local places  DVR in clouds
  • 141. Wang, Yuan-Kai (王元凱) Cloud Vision p. 141 A VSaaS Example  Securitystation in UK  Functions  Live viewing, recording, storage, playback and system management  Web interface  Mobile access with iPhone or iPad  Plug and Play (PnP) support of IP cameras  No video analytics currently  Rental service: £5/camera/month
  • 142. Wang, Yuan-Kai (王元凱) Cloud Vision p. 142 A VSaaS Exmaple  Securitystation in UK Currently, 1.VSaaS can be seen only in industry 2.Video analytics has not yet been put into VSaaS
  • 143. Wang, Yuan-Kai (王元凱) Cloud Vision p. 143 Cloud Family  Impact of Cloud Computing  Death of the desktop  Transfer to thin client 1. 雲端的「雲」 : Computing cloud Public cloud Private cloud Enterprise cloud Hybrid cloud 2. 雲端的「端」 : Mobile Cloud
  • 144. Wang, Yuan-Kai (王元凱) Cloud Vision p. 144 Mobile Cloud  Accessing the “cloud” with mobile devices  Mobile devices becoming part of a larger cloud construct  Advantages  Offloading computation and save energy
  • 145. Wang, Yuan-Kai (王元凱) Cloud Vision p. 145 Mobile Cloud Computing  ABI research think it can be next disruptive force, because  More phones as connected nodes  Feature phones, but not smart phones  Applications are not related to carrier ABI Research in 2009 1. By 2008, 42.8 million users (1.1% of mobile cellular subscriber) 2. By 2015, 240 million users, (19%) $5.2 billion revenues
  • 146. Wang, Yuan-Kai (王元凱) Cloud Vision p. 146 Future Architecture of Mobile VSaaS Producer Consumer
  • 147. Wang, Yuan-Kai (王元凱) Cloud Vision p. 147 Problems of VSaaS •Video is transmitted and stored in a remote site Capture Record/Analysis View xDSL FTTH/Fiber •Transmission by ADSL is limited • 640x480 videos with H.264 • 2~4 cameras at a frame rate of 10 fps may be supported Front-end video analytics is necessary More research study is needed
  • 148. Wang, Yuan-Kai (王元凱) Cloud Vision p. 148 DEMO 1: Mobil Video Surveillance Event-driven Instant Messaging
  • 149. Wang, Yuan-Kai (王元凱) Cloud Vision p. 149 Current Mobile Surveillance  Mobile phone directly connects to the camera Mobile networks (circuit-switched & packet-switched) Remote monitor Camera client Drawbacks: 1. Not event-driven 2. High communication cost
  • 150. Wang, Yuan-Kai (王元凱) Cloud Vision p. 150 Instant Alarm Messaging Is Important 【連絡人】 【客戶端】 【保全監控中心】 3 1 異常訊息發報 通知連絡人 低速專線 或電話線 歹徒入侵 3 【勤務中心】 通知警察 保全主機 各式感知器 2 服務派遣處理
  • 151. Wang, Yuan-Kai (王元凱) Cloud Vision p. 151 Our System Architecture Keyframe Web Selection Server PC Object 1. Object Info. Video Detection 2. Keyframe Streaming &Tracking 3. Video Clip Mobile Phone Video SMS Transcoding Messaging
  • 152. Wang, Yuan-Kai (王元凱) Cloud Vision p. 152 Our System: Web-based Browsing Interface
  • 153. Wang, Yuan-Kai (王元凱) Cloud Vision p. 153 Our System – Event, Key Frame, Video Clip
  • 154. Wang, Yuan-Kai (王元凱) Cloud Vision p. 154 Our Mobile Interface SMS Message for Mobile Event Notification video streaming
  • 155. Wang, Yuan-Kai (王元凱) Cloud Vision p. 155 Our Protocol 1.監控 1.監控 新竹 視訊 台北 視訊 分析 分析 3. Video Analytic 確認 Video Analytic 2 3. 1 確認 2.5G 5. 2.5G 5. 3GP 3GP 確認 確認 2. SMS 2. SMS 3G 4.關鍵 畫面 4.關鍵 3G 畫面 6.監控 視訊 6.監控 串流 視訊 串流 輔仁大學
  • 156. 4. Conclusions and Suggestions
  • 157. Wang, Yuan-Kai (王元凱) Cloud Vision p. 157 Conclusions  Video surveillance can be widely applied to  Home security  Building/Campus security  Public safety  Video analytics, or intelligent video surveillance, can benefit from  Cloud computing  High performance computing
  • 158. Wang, Yuan-Kai (王元凱) Cloud Vision p. 158 Suggestions  HiCloud with Video Analytics  CaaS for Academic Research
  • 159. Wang, Yuan-Kai (王元凱) Cloud Vision p. 159 Hicloud with Video Analytics影像擷取 相機異常偵測 人臉辨識 查詢、過濾、聯防 Video Image Object Object Object Behavior Capture Enhance /Event Tracking /Event Analysis Retrieval Detection Recognition 影像強化 警戒線、路徑追蹤 流浪漢監控 跌倒、人潮行為分析
  • 160. Wang, Yuan-Kai (王元凱) Cloud Vision p. 160 CaaS for Academic Research  The academy needs CaaS  For data intensive computing  Models  The academy as users  Rent CaaS  The academy as technology sources  Technology transfer  Delegated research project  The academy as partners  A target: smart campus  Long range tracking within campus  Tracking target: car and person
  • 161. Wang, Yuan-Kai (王元凱) Cloud Vision p. 161 Smart Campus (1/2) 13.機器人導引訪客導引 6.智慧型車牌偵測 12.快速人臉辨識 7.行人交錯偵測與追蹤 11.人員連續追蹤 8.特定人員追蹤與特寫 10.軌跡式人員追蹤與檢索 3.循園多攝影機轉場監控 9.多頻道視訊平滑轉場 2.多攝影機整合式視訊監控 4.多重解析度顯示及轉換 5.手勢指定物件 1.串聯式車牌辨識 14.整合平台及信號傳輸 161
  • 162. Wang, Yuan-Kai (王元凱) Cloud Vision p. 162 Smart Campus (2/2)
  • 163. 163 The EndFree for Questions
  • 164. 本簡報授權聲明 此簡報內容採用 Creative Commons 「姓名標示 - 非商 業性台灣 3.0 版」授權條款 歡迎非商業目的的重製、散布或修改本簡報的內容,但 請標明: (1)原作者姓名:王元凱; (2)圖標示: 簡報中所取用的部份圖形創作乃截取自網際網路,僅供 演講者於自由軟體推廣演講時主張合理使用,請讀者不 得對其再行取用,除非您本身自忖亦符合主張合理使用 之情狀,且自負相關法律責任。