Ice ss2013


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Ice ss2013

  1. 1. ICE Summer School, July 2013 Visual Sensor Networks Bernhard Rinner Institut für Vernetzte und Eingebettete Systeme
  2. 2. B. Rinner. 2 Agenda Sensor Networks Smart Cameras Visual Sensor Networks
  3. 3. B. Rinner 333 Introduction to Sensor Networks
  4. 4. B. Rinner 4 Wireless Sensor Networks (WSNs) • Networks of typically small, battery-powered, wireless devices, (“sensor nodes”, “motes”) – On-board processing, – Communication, and – Sensing capabilities. Sensors Processor Radio Storage P O W E R Sensor node schematics [© Oracle Labs]
  5. 5. B. Rinner 5 Sensor Node Platforms • From research prototypes to commercial products The Vision „Smart Dust“ UC Berkeley late 1990‘s Commercial Products „Mote-on-a Chip“ Dust Networks, 2010 Research Prototypes „Mica-2“ Crossbow 2004
  6. 6. B. Rinner 6 Some Applications of Sensor Networks • Health • Structural Monitoring • Agriculture • Environmental [(c) University of Ghent][Kim et al. ACM SenSys, 2006 [AgriNet][M. Welsh, Harvard 2007]
  7. 7. B. Rinner 7 Communication is Key • Wireless communication is an enabling technology – Eases deployment – Enables mobility – Increases flexibility – Reduces costs • Communicate on demand (ad hoc, spontaneous) with dynamic infrastructure – Nodes organize themselves into network – Data is transferred via multiple hops source destination
  8. 8. B. Rinner 8 But also … • Advances in sensor technology – Micro electro-mechanical system (MEMS) revolutionized sensing – Integration of mechanical and electrical components on single chip – Example: 3D accelerometer (in your cell phone) • Embedded processors and integration – Moore’s Law still valid – Trade-off between processing performance and power consumption [© SensorDynamics] [© ARM]
  9. 9. B. Rinner 999 The “Energy Problem” • Major sources of energy consumption – Sensing, computing, communication – High temporal variation • Energy is the scarce resource for WSN. Several challenges – What energy reservoirs to exploit? Constraints: availability, max. power, size, … – How to distribute power over the network? Energy provider and consumer might be dislocated. – How to control the distribution? “The proper amount of energy in the right place at the right time” • Sensor networks have always been a “green” technology!
  10. 10. B. Rinner 10 Alternative Energy Reservoirs • Maybe micro-heat engines – Exploit MEMS technology to build „internal combustion engines“ – Expected power: 10-20 W – Still in early research/development phase • Or harvest energy from environment – Organic semiconductors for exploiting indoor ambient light – Thin film batteries for storing energy – EnHANTs : energy harvesting networked active tags [Handbook of Sensor Networks, Wiley] [Columbia University, CLUE]
  11. 11. B. Rinner. 111111 Smart Cameras
  12. 12. B. Rinner. 12 Principle of Smart Cameras • Smart cameras combine – sensing, – processing and – communication in a single embedded device • perform image and video analysis in real-time closely located at the sensor and transfer only the results • collaborate with other cameras in the network TrustEYE.M4 prototype on top of RaspberryPI
  13. 13. B. Rinner. 13 Differences to traditional Cameras Traditional Camera – Optics and sensor – Electronics – Interfaces delivers data in form of (encoded) images and videos, respectively Smart Camera – Optics and sensor – Onboard computer – Interfaces delivers abstracted image data and is configurable and programmable Sensor Electronics Image enhancement/ Compression Image Video Sensor Embedded Computer Image analysis „Events“ Programming Configuration Light Light
  14. 14. B. Rinner. 14 SmartCams look for important things • Examples for abstracted image data – compressed images and videos – features – detected events © CMU
  15. 15. B. Rinner 15 Be aware of scarce Resources • Major resource limitations – Processing power – Communication bandwidth – Onboard memory – Energy • Various Prototypes (with decreasing performance) Sony XCISX100C/XP x86 VIA Eden ULV @ 1 GHz TrustEYE.M4 ARM Cortex@ 168MHz SLR Engineering Atom Z530@ 1.6 GHz CITRIC PXA 270@ 13-640MHz [Rinner et al. The Evolution from Single to Pervasive Smart Cameras. Proc. ICDSC 2008]
  16. 16. B. Rinner 161616 Characteristics of Visual Sensor Networks
  17. 17. B. Rinner 17 Video Surveillance Network • 3rd generation – all-digital systems • 3+ generation – smart cameras – surveillance tasks run on-site on smart cameras, e.g., • video compression  traffic statistics • accident detection  wrong-way drivers • stationary vehicles (tunnels)  vehicle tracking • 1st and 2nd generation – primarily analog frontends – backend systems are digital [Regazzoni, Ramesh, Foresti. Special Issue on Video Communications, Processing and Understanding for Third Generation Surveillance Systems. Proceedings of the IEEE. October 2001]
  18. 18. B. Rinner 18 Video Surveillance Network (2) • Even third generation networks rely on “heavy” infrastructure. – Camera nodes: sensor, onboard processing (encryption) – Network: hierarchically structured, wired, large bandwidth – Energy: dedicated supply • Surveillance networks typically consist of large number of cameras • Processing in network is fixed; (compressed) data is streamed to control center
  19. 19. B. Rinner 19 Characteristics of VSN • Visual sensor networks lie somewhere in between wireless sensor networks (WSN) and multi-camera/surveillance networks. • VSN have unique characteristics (wrt. traditional WSN) • Resource limitations – Need to process and transfer large amounts of data – Energy and bandwidth • On-board processing (cp. Smart cameras) – Challenging vision algorithms – Adaptive behavior [Soro et al. A Survey of Visual Sensor Networks. Advances in Multimedia 2009]
  20. 20. B. Rinner 20 Characteristics of VSN (2) • Real-time operation – Most applications require real-time analysis (camera to user) • Location and orientation information (spatial calibration) – Absolute or relative coordinates and orientations – (Multi-)camera calibration • Time Synchronization (temporal calibration) • Data Storage – Access to historic data necessary, eg., frame buffer, detected events – Stored data may be discarded over time • Autonomous Camera Collaboration – cp. Distributed smart cameras (DSCs)
  21. 21. B. Rinner. 21 (Selected) VSN Problems • Sensor Placement – Eg., dynamic setting of PTZ parameters • Clustering, cluster head election – Eg., what cameras should “work together”, who is the “leader” • Synchronization and calibration – Eg., establish temporal and spatial correlation • Data (and energy) distribution – Eg., when and what data to exchange • …
  22. 22. B. Rinner. 222222 Coordinating Resources in Visual Sensor Networks
  23. 23. B. Rinner 23 Configuring Smart Camera Networks • Smart camera networks process data onboard can modify their functionality/execute actions during runtime to reflect changes – to the state of the environment – to the user criteria • A configuration describes what is processed/executed where; specified by – Description of camera network (including the available actions/tasks) – Specification of the objective • We study configuration methods to use scarce resources in these networks more efficiently
  24. 24. B. Rinner 24 Configuration Problem (example) • Configuring a camera network – Select a set of cameras to monitor an area of interest – Set the sensor (frame rate, resolution, PTZ) to achieve QoS – Assign monitoring functions to cameras – Optimize wrt. multiple criteria – Dependent on dynamics of environment s1 s2 s3 s4 s5 t1 t2 t3 p1, p2 p3 p4, p5
  25. 25. B. Rinner 25 Configuration Design Space Design space for configuration methods is given by: • Dynamics of environment (static vs. mobile observation points) • Configuration algorithm (centralized vs. distributed) • Tasks and sensors (homogeneous/heterogeneous; static/mobile cameras) • A priori knowledge (complete vs. no knowledge of environment/VSN) • Various alternatives for solving this optimization problem, eg. – Centralized configuration algorithms – Distributed configuration algorithms focus on resource-aware approaches
  26. 26. B. Rinner 26 Centralized Configuration with EA • Approximation with evolutionary algorithm satisfying all requirements along multiple criteria (eg., energy, data, QoS) • Smart Camera Network – Set of cameras at known position with fixed FoV – Sensor configurations (frame rate, resolution) • Observation Area – Static set of observation points with monitoring activity a at required QoS (pot, fps) • Monitoring tasks – Assign procedures for achieving – Required resources for },...{ 1 nSSS = [Dieber, Micheloni, Rinner. Resource-Aware Coverage and Task Assignment in Visual Sensor Networks IEEE Transactions on Circuits and Systems for Video Technology, Aug 2011] },...,{ 1 ki ddD = },...,{ 1 mttT = },...,{; 1 apaa ppPAa =∈ ),,(),( iiiii emcdPr →
  27. 27. B. Rinner 27 Self-aware Configuration • Adopted from proprioceptive computing systems – use proprioceptive sensors to monitor “one self” (concept from psychology, robotics/prosthetics, …, fiction) – reason about their behavior (self-awareness) – effectively and autonomously adapt their behavior to changing conditions (self-expression) • Demonstrate autonomous multi-object tracking in camera network – Exploit single camera object detector & tracker – Perform camera handover – Learn camera topology
  28. 28. B. Rinner 28 Bid C4 Virtual Market-based Handover • Initialize auctions for exchanging tracking responsibilities – Cameras act as self-interested agents, i.e., maximize their own utility – Selling camera (where object is leaving FOV) opens the auction – Other cameras return bids with price corresponding to “tracking” confidence – Camera with highest bid continues tracking; trading based on Vickrey auction Camera 1 Camera 2 Camera 3 Camera 4 Init auction Bid C3 Fully distributed approach no a-priori topology knowledge required
  29. 29. B. Rinner 29 Market-based Tracking Handover • Utility function (each camera) rpjvcOU iOj ijjii +−Φ⋅⋅= ∑∈ )]([)( tracking decision visibility confidence payments made payments received Simulation green: tracking yellow: shared FOV red: trading (handover)
  30. 30. B. Rinner 30 Tracking Performance • Tradeoff between utility and communication effort Scenario 1 (5 cameras, few objects) Scenario 2 (15 cameras, many objects) • Emerging Pareto front [Esterle et al. Socio-Economic Vision Graph Generation and Handover in Distributed Smart Camera Networks. ACM Trans. Sensor Networks. 2013]
  31. 31. B. Rinner 31 Learn Neighborhood Relationships • Gaining knowledge about the network topology (vision graph) by exploiting the trading activities • Temporal evolution of the vision graph
  32. 32. B. Rinner 32 Learning Heterogeneous Strategies • Heterogeneous strategies at cameras may improve Pareto front • Adapt camera behaviour by online learning using bandit solvers Homogeneous vs. heterogeneous handover strategies (offline) Online learning strategies with different bandit solvers [Lewis et al. Learning to be different: Heterogeneity and Efficiency in Distributed Smart Camera Networks. In Proc. IEEE SASO. 2013]
  33. 33. B. Rinner 333333 Applications
  34. 34. B. Rinner 343434 #1 Trustworthy Cameras • Smart cameras – Highly capable embedded systems (on-board video analysis) – Large software stacks – Networked devices using closed (CCTV) and public networks • Applications no longer only in public but also in private areas (assisted living, home monitoring, …) • Protection of sensitive image data – Protection against manipulation (e.g., enforcement applications; evidence at court) – Privacy of monitored people
  35. 35. B. Rinner 353535 Goals and Assumptions • We present a system level approach that addresses the following security issues: – Integrity: detect manipulation of image and video data – Authenticity: provide evidence about the origin of image and videos – Confidentiality: make sure that privacy sensitive image data cannot be accessed by an unauthorized party – Multi-level Access Control: support different abstraction levels and enforce access control for confidential data • Security and privacy protection as inherent features of the camera • Considered attack types: only software attacks [Winkler, Rinner. Securing Embedded Smart Cameras with Trusted Computing. EURASIP Journal on Wireless Communications and Networking, 2011 ]
  36. 36. Approach • Bringing of Trusted Computing concepts into cameras • Trusted Platform Modules (TPMs) are well defined, readily available and cheap • TC is an open industry standard • TPMs are available from many manufacturers B. Rinner 36
  37. 37. Hardware Security Anchor 37 • Trusted Platform Module (TPM) at a glance – Secure storage for cryptographic keys – Data encryption, digital signatures – System status monitoring and reporting (measurement + attestation) – Unique platform ID Security Chip (TPM) Image Sensor CPU RAM Bootloader Operating System (e.g., Embedded Linux) Software Libraries and Middleware Image Processing and Analysis Communication… Software Hardware B. Rinner
  38. 38. Implemented Security Features 38 • Trusted boot where camera software stack is “measured” and the status is securely reported to operator • Integrity and authenticity guarantees using non-migratable, TPM-protected RSA keys • Freshness/timestamping for outgoing images via TPM- protected tick (counter) sessions B. Rinner
  39. 39. B. Rinner 393939 Hardware Prototype • TI OMAP 3530 CPU: ARM @ 480MHz and DSP @ 430MHz • 256MB RAM, SD-Card as mass storage • VGA color image sensor • wireless: 802.11b/g WiFi and 802.15.4 (XBee) • LAN via USB (primarily used for debugging) • Atmel hardware TPM on I2C bus
  40. 40. Privacy Protection Approaches 40 • Protection as an inherent feature of the camera • Object-based protection: Identification of sensitive data (e.g., human faces) • Data abstraction and obfuscation • Global protection techniques: Uniform protection of entire frames (insensitive to misdetections of computer vision) B. Rinner
  41. 41. Multi-Level Protection 41 • Video stream contains sub streams • Every sub stream is encrypted – Hardware-bound cryptographic keys • Recovery of identities only via four eyes principle Video Stream Smart Camera Sub Streams B. Rinner
  42. 42. High-Level Processing Flow 42 B. Rinner
  43. 43. B. Rinner 434343 Privacy-aware Camera Networks • What about users (i.e., monitored people)? • users usually do not care much about integrity, authenticity of time stamping • users (hopefully!) care about confidentiality and privacy! • Question 1: How can we increase privacy awareness? • Question 2: How can we demonstrate that (our) cameras protect the privacy of users?
  44. 44. B. Rinner 444444 Raising Privacy Awareness • Let users know if there are cameras in their environment • Use user's handheld (e.g., smart phone) for location-based notifications
  45. 45. User Feedback • Goal: Trustworthy feedback to monitored persons about camera’s privacy protection • Visual communication for authentication – Direct line of sight – Intuitive way to select intended camera • Operator discloses applications to TrustCenter T. Winkler and B. Rinner, “User Centric Privacy Awareness in Video Surveillance,” Multimedia Systems Journal, vol. 18, no. 2, pp. 99–121, 2012. B. Rinner
  46. 46. Attestation Report 46 B. Rinner B. Rinner
  47. 47. B. Rinner 47 #2 Aerial Cameras for Disaster Mgm. • Develop autonomous multi-UAV system for aerial reconnaissance • Up-to-date aerial overview images are helpful in many situations: “Google Earth with up-to-date images in high resolution” • Small-scale quadcopter platform with onboard sensors and computation • GPS receiver for autonomous waypoint flights • Generic framework not bound to specific UAV
  48. 48. B. Rinner 48 Key Challenges • Increase autonomy – Control and coordination of multiple UAVs – High-level interaction with user • Provide prompt response to user – Provide preliminary results fast and improve over time • Deal with strong resource limitations – Flight time, payload, computation and communication – Limited sensing capabilities
  49. 49. B. Rinner 49 Autonomous UAV Operation Mission Planning Flight Real-World Simulator Image Analysis scenario specification waypoints Single/multiple UAV captured image/video stiching, detection user interface
  50. 50. B. Rinner 50 Key Questions • How to generate and update movement routes for the UAVs? – Achieve multiple optimization goals – Deal with changes in the environment • How to setup a wireless UAV network? – Provide networking coverage • How to generate the mosaic image? – Apply incremental image stiching – Combine RGB and thermal images • System integration and demonstration
  51. 51. 51 Demonstration Video B. Rinner
  52. 52. B. Rinner. SCVSN Tutorial (Chapter 3) 525252 Research Directions of Visual Sensor Networks
  53. 53. B. Rinner. 53 #1: Architecture • Low-power (high performance) camera nodes – Dedicated platforms: vision processors, PCBs, systems – Many examples: CITRIC, NXP • Visual/Multimedia Sensor Networks – Topology and (multi-tier) architecture – Multi-radio communication • Dynamic Power Management – For sensing, processing and communication How to design resource-aware nodes and networks
  54. 54. B. Rinner. 54 #2: Networking • Ad hoc, p2p communication over wireless channels – Providing RT and QoS – Eventing and/or streaming • Dynamic resource management – (local) computation, compression, communication, etc. – Degree of autonomy: dynamic, adaptive, self-organizing – Fault tolerance, scalability – Network-level software, middleware How to process and transfer data in the network
  55. 55. B. Rinner 55 #3: Deployment, Operation, Maintenance • Development support for applications – Model/simulate the application (function, resources, QoS) – Reuse/exchange of software/libraries – Software updates, debugging etc. • Autonomous calibration and scene adaption – Avoid manual procedures – Adapt to different scenes and settings • Network configuration Consider the entire life cycle of the camera network
  56. 56. B. Rinner. 56 #4: Distributed Sensing & Processing • Sensor placement, calibration & selection – Optimization problem – Distributed approaches eg., consensus, game theory, multi-agent systems • Compressive Sensing • Collaborative data analysis – Multi-view, multi-temporal, multi-modal – Sensor fusion • Online/real-time processing – Can not effort to store large amounts of data Where to place sensors and analyze the data
  57. 57. B. Rinner 57 #5: Mobility • Mobile cameras are ubiquitous – PTZ, vehicles, robotics etc. – Mobile phones • Advanced vision algorithms – Ego motion, online calibration – Closed-loop control, active vision How to exploit networks of mobile cameras
  58. 58. B. Rinner 58 #6: Usability • Ease of deployment, maintenance – Self-* functionality – “Smart cameras for dumb people” • Privacy and Security – Trust of the user – Control the privacy setting • Interaction with the camera network How to provide useful services to people
  59. 59. B. Rinner 59 #7: Applications • Demonstrations – Large scale networks eg., for surveillance – Small scale networks eg., for entertainment, home environments – Only single camera application? • Market opportunities • Killer Application What applications can (only) be solved by DSC
  60. 60. B. Rinner. SCVSN Tutorial (Chapter 3) 60 Summary • VSNs exploit various advantages of distributed camera sensors such as increased coverage, redundancy and 3D information. • Distributed cameras impose various challenges such as huge amount of data, required infrastructure and (network) topology. • VSN have unique characteristics (wireless sensor networks vs. surveillance camera networks) • Current research addresses signal processing, communications, architecture and middleware issues.
  61. 61. B. Rinner 61 Acknowledgements & Further Info Pervasive Computing @ AAU UAV Research • Tutorial site Most recent course material is available at