Open Cloud Consortium: An Update (04-23-10, v9)


Published on

This is an overview of the Open Cloud Consortium that I gave at Cloud Lab '10 on April 23, 2010.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Open Cloud Consortium: An Update (04-23-10, v9)

  1. 1.<br />Open Cloud Consortium: An Update<br />Robert GrossmanOpen Cloud Consortium<br />April 23, 2010<br />
  2. 2. Part 1. Overview of theOpen Cloud Consortium (OCC)<br /><br />2<br />
  3. 3. 501(3)(c) Not-for-profit corporation<br />Supports the development of standards, interoperability frameworks, and reference implementations.<br />Manages testbeds: Open Cloud Testbed and IntercloudTestbed.<br />Manages cloud computing infrastructure to support scientific research: Open Science Data Cloud.<br />Develops benchmarks.<br />3<br /><br />
  4. 4. OCC Members<br />Companies: Aerospace, Booz Allen Hamilton, Cisco, InfoBlox, Open Data Group, Raytheon, Yahoo<br />Universities: CalIT2, Johns Hopkins, MIT Lincoln Lab, Northwestern Univ., University of Illinois at Chicago, University of Chicago<br />Government agencies: NASA<br />Open Source Projects: Sector Project<br />4<br />
  5. 5. OCC Working Groups<br />Large Data Cloud Working Group<br />Open Cloud Testbed Working Group.<br />Intercloud Testbed Working Group<br />Open Science Data Cloud Working Group<br />
  6. 6. Applications<br />Apps<br />Compute Services<br />Data Services<br />Metadata Services<br />PaaS<br />Storage Services<br />Identity Manager<br />Virtual Machine Manager<br />Virtual Network Manager<br />IaaS<br />Network Transport<br />
  7. 7. Part 2. Intercloud Testbed<br />7<br />Cloud 2<br />Cloud 1<br />
  8. 8. We have several cloud standards…<br /><ul><li>Platform as a Service
  9. 9. Cloud Compute Services
  10. 10. Data/Table Cloud Services
  11. 11. Cloud Storage Services</li></ul>Large Data Cloud Interoperability Framework<br />SNIA Cloud Data Management Interface (CDMI)<br /><ul><li>Infrastructure as a Service
  12. 12. Virtual Data Centers (VDC)
  13. 13. Virtual Networks (VN)
  14. 14. Virtual Machines (VM)</li></ul>Open Cloud Computing Interface (OCCI)<br />Open Virtualization Format (OVF)<br />
  15. 15. Where are the Gaps?<br /><ul><li>Naming entities in IaaS & PaaS
  16. 16. Bridging IaaS & DaaS
  17. 17. Services that span multiple VMs, ….
  18. 18. Platform as a Service
  19. 19. Cloud Compute Services
  20. 20. Data as a Service</li></ul>Large Data Cloud Interoperability Framework<br />SNIA Cloud Data Management Interface (CDMI)<br /><ul><li>Infrastructure as a Service
  21. 21. Virtual Data Centers (VDC)
  22. 22. Virtual Networks (VN)
  23. 23. Virtual Machines (VM)
  24. 24. Physical Resources</li></ul>Open Cloud Computing Interface (OCCI)<br />Open Virtualization Format (OVF)<br />
  25. 25. Bridging the Gaps…A Small Step<br /><ul><li>Platform as a Service
  26. 26. Cloud Compute Services
  27. 27. Data as a Service</li></ul>Large Data Cloud Interoperability Framework<br />SNIA Cloud Data Management Interface (CDMI)<br />Metadata service linking IaaS and DaaS<br />Metadata service naming and linking entities in the IaaS layers<br /><ul><li>Infrastructure as a Service
  28. 28. Virtual Data Centers (VDC)
  29. 29. Virtual Networks (VN)
  30. 30. Virtual Machines (VM)
  31. 31. Physical Resources</li></ul>Open Cloud Computing Interface (OCCI)<br />Open Virtualization Format (OVF)<br />
  32. 32. Part 3. Large Data Cloud Working Group<br />11<br />Standards for integrating and interoperating large data cloud services such as those provided by Hadoop and similar systems.<br />
  33. 33. Focus of Working Group<br />12<br />App<br />App<br />App<br />App<br />App<br />Table-based Data Services<br />Relational-like Data Services<br />App<br />App<br />Cloud Compute Services (MapReduce, UDF, & other programming frameworks)<br />App<br />App<br />Cloud Storage Services<br />Developing APIs for this framework.<br />
  34. 34. Benchmarks for Large Data Clouds<br />Until recently, the only benchmark used was Terasort (sorting 10 billion 100 byte records)<br />Replaced by Gray Sort and Minute Sort<br />Gray Sort tries to maximize TB / min sorted on 100 TB or more of data.<br />Hadoop holds the current Gray Sort and Minute Sort records.<br />Problem: sort is just one of the types of work load for analytic applications<br />
  35. 35. MalStone<br />MalGen – generates synthetic data with realistic distributions.<br />MalStone A & B – “stylized” computations that can be used as benchmarks for architectures, software and systems for large data clouds.<br />Open source and available at<br />14<br />
  36. 36. Part 4. Open Cloud Testbed<br />
  37. 37. Condominium Clouds<br />In a condominium cloud, you buy your own rack or bunch of racks.<br />The racks are managed and operated by the condominium association, in this case the OCC.<br />If your rack is 120 TB, you get the rights to c. 40 TB of storage in the cloud. The rest is a shared resource. <br />The Open Cloud Testbed is a condo cloud managed by the OCC.<br />16<br />
  38. 38. Open Cloud Testbed<br />C-Wave<br />CENIC<br />Dragon<br />Phase 2<br />9 racks<br />250+ Nodes<br />1000+ Cores<br />10+ Gb/s<br /><ul><li>Hadoop
  39. 39. Sector/Sphere
  40. 40. Thrift
  41. 41. KVM VMs
  42. 42. Eucalyptus VMs</li></ul>MREN<br />Phase 3 (2011) – we will stand up some 100 Gb/s links.<br />
  43. 43. Part 5. Open Science Data Cloud Working Group<br />18<br />
  44. 44. Open Science Data Cloud<br />Biological data (Bionimbus)<br />Astronomical data<br />Provide a long term home for selected scientific data sets and support elastic cloud-based analysis & integration of the data.<br />Networking data<br />19<br />
  45. 45. Part 6. Image Processing for Disaster Relief Using Elastic Clouds<br />
  46. 46. The Challenge<br />When a disaster strikes, there is usually an immediate and critical need for computing power to process images.<br />Example, there was a delay getting current images of Haiti to non-government organizations (NGO) after earthquake on January 12, 2009.<br />
  47. 47. The Idea …The OCC Elastic Cloud for Disaster Relief<br />Set up a permanent elastic cloud that is available to assist with disaster relief.<br />Establish connections to sources of images that can be enabled at times of need.<br />Set up a network of volunteers with accounts on the cloud and knowledge of the tools that can swarm when needed.<br />Use as a test of large data cloud standards and interoperability.<br />
  48. 48. Image Processing on Large Data Clouds<br />Data parallel applications<br />Parallelism is often required at file or directory level<br />Data locality is important<br />Parallel disk IO is also critical<br />Requirements<br />The input data size can be at 10+ TB per day<br />Want to integrate with open source libraries such as OSSIM<br />
  49. 49. Distributed File Systems & Image Processing<br />Sector is broadly similar to the Hadoop Distributed File System<br />Main differences<br />Hadoop directly implements a distributed block based file system<br />Sector is a layer over a native file system<br />Sector does not split files<br />A single image will not be split, therefore when it is being processed, the application does not need to read the data from other nodes via network<br />A directory can be kept together on a single node as well, as an option<br />
  50. 50. Get Involved… … Join our volunteer effort.<br />
  51. 51. Part 7. Virtual Networks<br />
  52. 52. How Long Does It Take…<br />
  53. 53. … To Move A Cloud Application Spanning Multiple VMs Between Clouds?<br />
  54. 54. … To Add A New Rack to a Cloud Service?<br />
  55. 55. … To Add Another Public Cloud to A Private/Public Cloud?<br />
  56. 56. We Have Several Ways of Defining Virtual Networks….<br />VN-Link<br />Open vSwitch<br />VPNs<br />CloudSwitch<br />vSwitch<br />VLAN<br />BGP<br />MPLS<br />OpenFlow<br />
  57. 57. But No Vendor Neutral VN Standard That<br /> That scales to 100,000+ VMs<br /> Supported by multiple vendors<br /> Spans multiple physical switches<br /> Supports VN Mobility<br /> Provides strong isolation of VN<br /> Is easy for VMs to join and leave VNs<br /> Includes management interfaces<br /> ….<br />
  58. 58. For More Information<br /><br /><br />