Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Large Scale On-Demand Image Processing For Disaster Relief


Published on

This is a status update (as of Feb 22, 2010) of a new Open Cloud Consortium project that will provide on-demand, large scale image processing to assist with disaster relief efforts.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Large Scale On-Demand Image Processing For Disaster Relief

  1. 1. Large Scale On-Demand Image Processing for Disaster Relief Robert Grossman Open Cloud Consortium February 22, 2010
  2. 2. <ul><li>501(3)(c) Not-for-profit corporation </li></ul><ul><li>Supports the development of standards, interoperability frameworks, and reference implementations. </li></ul><ul><li>Manages testbeds: Open Cloud Testbed and Intercloud Testbed. </li></ul><ul><li>Manages cloud computing infrastructure to support scientific research: Open Science Data Cloud. </li></ul><ul><li>Develops benchmarks. </li></ul>
  3. 3. Focus of OCC Large Data Cloud Working Group <ul><li>Developing APIs for this framework. </li></ul>Cloud Storage Services Cloud Compute Services (MapReduce, UDF, & other programming frameworks) Table-based Data Services Relational-like Data Services App App App App App App App App App
  4. 4. Storage Services Compute Services Applications Virtual Network Manager Data Services Network Transport Virtual Machine Manager IF-MAP (Metadata) Services Identity Manager IaaS PaaS Apps
  5. 5. Bridging the Gaps…A Small Step <ul><li>Infrastructure as a Service </li></ul><ul><ul><li>Virtual Data Centers (VDC) </li></ul></ul><ul><ul><li>Virtual Networks (VN) </li></ul></ul><ul><ul><li>Virtual Machines (VM) </li></ul></ul><ul><ul><li>Physical Resources </li></ul></ul><ul><li>Platform as a Service </li></ul><ul><ul><li>Cloud Compute Services </li></ul></ul><ul><ul><li>Data as a Service </li></ul></ul>Open Virtualization Format (OVF) Open Cloud Computing Interface (OCCI) SNIA Cloud Data Management Interface (CDMI) Large Data Cloud Interoperability Framework Metadata service linking IaaS and DaaS Metadata service naming and linking entities in the IaaS layers
  6. 6. Open Science Data Cloud Astronomical data Biological data (Bionimbus) Networking data Image processing for disaster relief
  7. 7. Image Processing on Large Data Clouds <ul><li>Data parallel applications </li></ul><ul><ul><li>Parallelism is often required at file or directory level </li></ul></ul><ul><ul><li>From a MapReduce perspective, often only Map operations are required. </li></ul></ul><ul><li>Data intensive applications </li></ul><ul><ul><li>The input data size can be at 10s or 100s of TB </li></ul></ul><ul><ul><li>Requires parallel disk IO & data locality is important </li></ul></ul>
  8. 8. Distributed File Systems <ul><li>Sector is broadly similar to the Hadoop Distributed File System </li></ul><ul><li>Main differences </li></ul><ul><ul><li>Hadoop directly implements a distributed block based file system </li></ul></ul><ul><ul><li>Sector is a layer over a native file system </li></ul></ul><ul><li>Sector does not split files </li></ul><ul><ul><li>A single image will not be split, therefore when it is being processed, the application does not need to read the data from other nodes via network </li></ul></ul><ul><ul><li>A directory can be kept together on a single node as well, as an option </li></ul></ul>
  9. 9. Sphere UDF <ul><li>Sphere allows a User Defined Function to be applied to each file (either it is a single image or multiple images) </li></ul><ul><li>Existing applications, such as OSSIM, can be wrapped up in a Sphere UDF or invoked via Sector streams </li></ul><ul><li>In many situations, Sphere streaming utility accepts a data directory and a application binary as inputs </li></ul>
  10. 10. Sector and OSSIM <ul><li>./sector_stream -i haiti -c ossim_foo -o results </li></ul><ul><li>“ -i” specifies the input data directory. In this example all images are located in the directory “haiti” </li></ul><ul><li>“ -c” refers to the command (or application) </li></ul><ul><li>“ -o” specifies the output location. This is a directory and the output of each input image is stored in a corresponding file </li></ul>
  11. 11. Next Steps <ul><li>Working group will set up persistent on-demand cloud for image processing to assist disaster relief using OSSIM and related open source software. </li></ul><ul><li>Will be used as a test case for Large Data Cloud and Intercloud Working Groups. </li></ul><ul><li>One rack of dedicated hardware will be available, with required high performance networking in place. </li></ul><ul><li>Initial operating capability by May 15,2010. </li></ul>
  12. 12. For More Information <ul><li>[email_address] </li></ul><ul><li> </li></ul>