Your SlideShare is downloading. ×
0
appscale: open-source platform-level cloud computing Chandra Krintz Computer Science Dept. Univ. of California, Santa Barb...
cloud computing <ul><li>Software systems for accessing easily and transparently scalable CPU/storage/network resources via...
cloud computing <ul><li>3 types: as-a-Service (aaS) </li></ul><ul><ul><li>Infrastructure: Amazon Web Services (EC2, S3, EB...
many public cloud features useful on-premise <ul><li>Ease of use – of local cluster resources </li></ul><ul><ul><li>Develo...
many public cloud features useful on-premise <ul><li>Ease of use – of local cluster resources </li></ul><ul><ul><li>Develo...
many public cloud features useful on-premise <ul><li>Ease of use – of local cluster resources </li></ul><ul><ul><li>Develo...
cloud computing fabrics from UCSB <ul><li>Goal : Bring popular cloud fabrics to “on-premise” clusters that are easy to use...
cloud computing fabrics from UCSB <ul><li>Goal : Bring popular cloud fabrics to “on-premise” clusters that are easy to use...
cloud computing fabrics from UCSB <ul><li>Goal : Bring popular cloud fabrics to “on-premise” clusters that are easy to use...
google app engine GAE Application (your code here) private,  enterprise data,  Google apps Images IM Memcache Mail Users U...
sandbox restrictions GAE Application (your code here) Google App Engine (GAE) MyApp.appspot.com <ul><ul><ul><li>Pure Pytho...
google app engine: upload to google GAE Application (Python, Java) Google App Engine (GAE) SDC Administrator Console Free ...
from gae to appscale GAE Application (your code here) Open-source Google App Engine Software Development Kit (SDK) AppScal...
from gae to appscale <ul><li>GAE SDK extensions </li></ul><ul><ul><li>Pluggable using open-source distributed database tec...
appscale’s offerings <ul><li>Use of the Google APIs as a unifying interface </li></ul><ul><ul><li>Datastore/database backe...
appscale’s offerings <ul><li>Use of the Google APIs as a unifying interface </li></ul><ul><ul><li>Datastore/database backe...
appscale <ul><li>Distributed system with four key components </li></ul><ul><li>AppLoadBalancer (ALB)  Database Master/Peer...
appscale team foci <ul><li>Full system monitoring (low-overhead HPM/OS & profiling) </li></ul><ul><ul><li>Feedback driven ...
google app engine GAE Application (your code here) private,  enterprise data,  Google apps Images IM Memcache Mail Users U...
customized platform-as-a-service GAE, HPC, Data Intensive Application private,  enterprise data,  other apps public cloud ...
appscale and RDMA <ul><li>Shared-memory support for co-located components </li></ul><ul><li>PaaS-IaaS coordination, adapta...
appscale    http://appscale.cs.ucsb.edu <ul><li>Thanks! </li></ul><ul><ul><li>Leads: Chris Bunch, Navraj Chohan </li></ul>...
Upcoming SlideShare
Loading in...5
×

Appscale: Open-Source Platform-Level Cloud Computing

1,696

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,696
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
56
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Requests are accepted and resources granted via web services Accountability is e-commerce based
  • Scalable services: mapreduce, relational db, VM management, job queue support Rapid uptake in the commercial sector
  • Data APIs: JDO/JPA Structured data
  • Datastore - Flat file database Memcache - In-memory hash table URL Fetch - Works fine, basically a curl / wget Mail - Prints to standard out / error instead of sending mail XMPP (IM) - Same as Mail Images - Works fine once python-imaging library is installed (otherwise fails) Google Accounts - Allows you to log in as anyone with no password Task Queues - Puts tasks on the admin console for later execution Cron - Same as Task Queues Blobstore - Uses URL Fetch to post large files to a URL which then stores it in the flat-file DB
  • Data APIs: JDO/JPA Structured data
  • Data APIs: JDO/JPA Structured data
  • Transcript of "Appscale: Open-Source Platform-Level Cloud Computing"

    1. 1. appscale: open-source platform-level cloud computing Chandra Krintz Computer Science Dept. Univ. of California, Santa Barbara OpenFabrics Alliance, Sonoma Workshop March 15, 2010
    2. 2. cloud computing <ul><li>Software systems for accessing easily and transparently scalable CPU/storage/network resources via a network connection or web interface – “ as-a-service” </li></ul><ul><ul><li>Culmination of grid/cluster/utility/elastic computing </li></ul></ul><ul><ul><li>Advances in processor, virtualization, systems technology </li></ul></ul><ul><li>Remote access to distributed and shared cluster resources </li></ul><ul><ul><li>Has experienced a rapid uptake in the commercial sector </li></ul></ul><ul><ul><ul><li>Public clouds –your software/apps on others’ systems </li></ul></ul></ul><ul><ul><ul><li>Users rent a small fraction of vast resource pools </li></ul></ul></ul><ul><ul><ul><ul><li>Advertised service-level-agreements (SLAs) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Resources are opaque and isolated </li></ul></ul></ul></ul><ul><ul><li>Offer high availability, fault tolerance, and extreme scale </li></ul></ul>
    3. 3. cloud computing <ul><li>3 types: as-a-Service (aaS) </li></ul><ul><ul><li>Infrastructure: Amazon Web Services (EC2, S3, EBS) </li></ul></ul><ul><ul><ul><li>Virtualized, isolated (CPU, Network, Storage) systems on which users execute entire runtime stacks </li></ul></ul></ul><ul><ul><ul><ul><li>Fully customer self-service </li></ul></ul></ul></ul><ul><ul><ul><li>Open APIs (IaaS standard), scalable services </li></ul></ul></ul><ul><ul><li>Platform: Google App Engine, Microsoft Azure </li></ul></ul><ul><ul><ul><li>Scalable program-level abstractions via well-defined interfaces </li></ul></ul></ul><ul><ul><ul><li>Enable construction of network-accessible applications </li></ul></ul></ul><ul><ul><ul><li>Process-level (sandbox) isolation, complete software stack </li></ul></ul></ul><ul><ul><li>Software: Salesforce.com </li></ul></ul><ul><ul><ul><li>Applications provided to thin clients over a network </li></ul></ul></ul><ul><ul><ul><li>Customizable </li></ul></ul></ul>
    4. 4. many public cloud features useful on-premise <ul><li>Ease of use – of local cluster resources </li></ul><ul><ul><li>Development, configuration, deployment </li></ul></ul><ul><ul><li>Broaden participation </li></ul></ul><ul><li>Scale, performance and resource isolation </li></ul><ul><li>Concerns </li></ul><ul><ul><li>Vendor “lock-in”, pay-per-use, and availability reliance </li></ul></ul><ul><ul><li>Privacy of code and data </li></ul></ul><ul><ul><li>Resource constraints: storage, CPU/memory, communication </li></ul></ul><ul><li>Potential for hybrid and customized approaches </li></ul><ul><ul><li>One size may not fit all </li></ul></ul>
    5. 5. many public cloud features useful on-premise <ul><li>Ease of use – of local cluster resources </li></ul><ul><ul><li>Development, configuration, deployment </li></ul></ul><ul><ul><li>Broaden participation </li></ul></ul><ul><li>Difficult to test / try things out in public-only setting </li></ul><ul><ul><li>Developers visualize the system / have an intellectual model </li></ul></ul><ul><li>How to support other application domains: scale/availability </li></ul><ul><ul><li>HPC, data-intensive </li></ul></ul><ul><li>Clouds are opaque </li></ul><ul><ul><li>Open APIs, closed implementations </li></ul></ul><ul><ul><li>What is your application/workload doing? </li></ul></ul>
    6. 6. many public cloud features useful on-premise <ul><li>Ease of use – of local cluster resources </li></ul><ul><ul><li>Development, configuration, deployment </li></ul></ul><ul><ul><li>Broaden participation </li></ul></ul><ul><li>Difficult to test / try things out in public-only setting </li></ul><ul><ul><li>Developers visualize the system / have an intellectual model </li></ul></ul><ul><li>How to support other application domains: scale/availability </li></ul><ul><ul><li>HPC, data-intensive </li></ul></ul><ul><li>Clouds are opaque </li></ul><ul><ul><li>Open APIs, closed implementations </li></ul></ul><ul><ul><li>What is your application/workload doing? </li></ul></ul><ul><ul><li>If everyone does their own… </li></ul></ul><ul><ul><li>many non-standard APIs! </li></ul></ul>
    7. 7. cloud computing fabrics from UCSB <ul><li>Goal : Bring popular cloud fabrics to “on-premise” clusters that are easy to use and are transparent </li></ul><ul><ul><li>By emulating key cloud layers from the commercial sector </li></ul></ul><ul><ul><ul><li>Engender user community, access to real applications/users </li></ul></ul></ul><ul><ul><ul><ul><li>“ Standard” APIs (Amazon AWS, Google App Engine, MapReduce) </li></ul></ul></ul></ul><ul><ul><ul><li>Leverage extant software technologies </li></ul></ul></ul>
    8. 8. cloud computing fabrics from UCSB <ul><li>Goal : Bring popular cloud fabrics to “on-premise” clusters that are easy to use and are transparent </li></ul><ul><ul><li>By emulating key cloud layers from the commercial sector </li></ul></ul><ul><ul><ul><li>Engender user community, access to real applications/users </li></ul></ul></ul><ul><ul><ul><ul><li>“ Standard” APIs (Amazon AWS, Google App Engine, MapReduce) </li></ul></ul></ul></ul><ul><ul><ul><li>Leverage extant software technologies </li></ul></ul></ul><ul><ul><li>To facilitate investigation of </li></ul></ul><ul><ul><ul><li>Next-generation distributed / cloud computing software </li></ul></ul></ul><ul><ul><ul><ul><li>Services, underlying device technology, support technologies </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Customization (availability, performance, application behavior) </li></ul></ul></ul></ul><ul><ul><ul><li>Hybrid cloud solutions (public and on-premise) </li></ul></ul></ul><ul><ul><li>Not a replacement technology for any Public Cloud service </li></ul></ul>
    9. 9. cloud computing fabrics from UCSB <ul><li>Goal : Bring popular cloud fabrics to “on-premise” clusters that are easy to use and are transparent </li></ul><ul><ul><li>By emulating key cloud layers from the commercial sector </li></ul></ul><ul><ul><ul><li>Engender user community, access to real applications/users </li></ul></ul></ul><ul><ul><ul><ul><li>“ Standard” APIs (Amazon AWS, Google App Engine, MapReduce) </li></ul></ul></ul></ul><ul><ul><ul><li>Leverage extant software technologies </li></ul></ul></ul><ul><li>Platform-as-a-service (PaaS) </li></ul><ul><ul><li>Open-source implementation of Google App Engine APIs </li></ul></ul><ul><ul><li>Pluggable (services), scalable, fault tolerant </li></ul></ul><ul><ul><li>Runs over virtualization or IaaS layer: AWS, Eucalyptus </li></ul></ul><ul><ul><li>appscale.cs.ucsb.edu </li></ul></ul>
    10. 10. google app engine GAE Application (your code here) private, enterprise data, Google apps Images IM Memcache Mail Users URL Fetch Administrator Console Data Store Protobuf Data APIs SDC Google App Engine (GAE) Services Cron Tasks MyApp.appspot.com Blob store
    11. 11. sandbox restrictions GAE Application (your code here) Google App Engine (GAE) MyApp.appspot.com <ul><ul><ul><li>Pure Python or Java only, white list of library calls to framework </li></ul></ul></ul><ul><ul><ul><li>No thread/subprocess spawning, system calls </li></ul></ul></ul><ul><ul><ul><li>No writes to file system, reads only to static files uploaded w/app </li></ul></ul></ul><ul><ul><ul><li>Storage using key-value, schema-free datastore (Bigtable-based) </li></ul></ul></ul><ul><ul><ul><li>HTTP/S communication only, CGI to handle page requests </li></ul></ul></ul><ul><ul><ul><li>Limit on number of datastore elements accessed per request </li></ul></ul></ul><ul><ul><ul><li>Limit on response duration, task frequency, request rate </li></ul></ul></ul><ul><ul><ul><li>Enforced quotas (BW, CPU, requests/s, files, app size, …) </li></ul></ul></ul>
    12. 12. google app engine: upload to google GAE Application (Python, Java) Google App Engine (GAE) SDC Administrator Console Free w/ quotas Pay for additional scale: CPU, BW, emails, data BigTable: Automatic scaling High availability … GAE app users via the Internet MyApp.appspot.com private, enterprise data, Google apps Images IM Mem Cache Mail Users URL Fetch Cron Tasks Data store Blob Store
    13. 13. from gae to appscale GAE Application (your code here) Open-source Google App Engine Software Development Kit (SDK) AppScale Images IM Mem Cache Mail Users URL Fetch Cron Tasks Data store Blob Store
    14. 14. from gae to appscale <ul><li>GAE SDK extensions </li></ul><ul><ul><li>Pluggable using open-source distributed database technologies </li></ul></ul><ul><ul><ul><li>HBase, Hypertable, Cassandra, Voldemort, MongoDB, MemcacheDB, MySQL </li></ul></ul></ul><ul><ul><li>MemcacheD library </li></ul></ul><ul><ul><li>From console or as background thread (automatically) </li></ul></ul><ul><ul><ul><li>Interface to Hadoop (MapReduce) </li></ul></ul></ul><ul><ul><ul><ul><li>Multi-language support: Python, Java, Ruby, Perl, soon: X10 </li></ul></ul></ul></ul><ul><ul><li>Translator to Linux Cron job, similar to Tasks </li></ul></ul><ul><ul><li>Pluggable: built-in cloud-wide authentication via Rails, support for Eucalyptus and EC2 credentials </li></ul></ul>Mem Cache Users Cron Tasks Data store
    15. 15. appscale’s offerings <ul><li>Use of the Google APIs as a unifying interface </li></ul><ul><ul><li>Datastore/database backends, services </li></ul></ul><ul><ul><ul><li>Can grow this to include other standard APIs </li></ul></ul></ul><ul><li>Single framework that executes extant GAE applications </li></ul><ul><ul><li>Disparate API implementations automatically deploy/compare </li></ul></ul><ul><li>No restrictions on resource usage, library support </li></ul><ul><ul><li>Multiple high-level language front-ends and components </li></ul></ul><ul><ul><li>Optimized library support in the back-end </li></ul></ul>
    16. 16. appscale’s offerings <ul><li>Use of the Google APIs as a unifying interface </li></ul><ul><ul><li>Datastore/database backends, services </li></ul></ul><ul><ul><ul><li>Can grow this to include other standard APIs </li></ul></ul></ul><ul><li>Single framework that executes extant GAE applications </li></ul><ul><ul><li>Disparate API implementations automatically deploy/compare </li></ul></ul><ul><li>No restrictions on resource usage, library support </li></ul><ul><ul><li>Multiple high-level language front-ends and components </li></ul></ul><ul><ul><li>Optimized library support in the back-end </li></ul></ul><ul><li>Open-source international user community </li></ul><ul><li>Potential for integration with infrastructure level (IaaS) </li></ul><ul><ul><li>Adaptive SLA re-negotation </li></ul></ul><ul><ul><li>Growth/Shrinkage of the cloud </li></ul></ul><ul><ul><li>Application or workload specific customization of the platform </li></ul></ul>
    17. 17. appscale <ul><li>Distributed system with four key components </li></ul><ul><li>AppLoadBalancer (ALB) Database Master/Peer (DB M/P) </li></ul><ul><li>AppServer (AS) Database Slave/Peer (DB S/P) </li></ul><ul><li>Automatic deployment over Eucalyptus and EC2 </li></ul><ul><ul><li>Released as a single Xen or KVM image </li></ul></ul><ul><ul><li>Adaptive role configuration </li></ul></ul>GAE App Developer (AppScale Admin) GAE App Users HTTPS App Controller ALB DB S/P AS GAE App Users GAE App Users AppScale Cloud Compute tasks (e.g. Map Reduce) DB M/P AppScale tools
    18. 18. appscale team foci <ul><li>Full system monitoring (low-overhead HPM/OS & profiling) </li></ul><ul><ul><li>Feedback driven adaptation, grow/shrink the cloud </li></ul></ul><ul><ul><ul><li>Online workload characterization, behavior pattern recognition </li></ul></ul></ul><ul><ul><ul><li>Identify critical tasks, applications, components </li></ul></ul></ul><ul><li>Shared-memory support for co-located components </li></ul><ul><li>PaaS customization, PaaS-IaaS coordination, adaptation </li></ul>GAE App Developer (AppScale Admin) GAE App Users HTTPS App Controller ALB DB S/P AS GAE App Users GAE App Users AppScale Cloud Compute tasks (e.g. Map Reduce) DB M/P AppScale tools
    19. 19. google app engine GAE Application (your code here) private, enterprise data, Google apps Images IM Memcache Mail Users URL Fetch Administrator Console Data Store Protobuf Data APIs SDC Google App Engine (GAE) Services Cron Tasks MyApp.appspot.com Blob store
    20. 20. customized platform-as-a-service GAE, HPC, Data Intensive Application private, enterprise data, other apps public cloud fabrics GPU Map Reduce Collaboration Vector support Auth MPI Administrator Console Data Store Protobuf Data APIs SDC <ul><li>Specialized platform access to </li></ul><ul><li>standard APIs </li></ul><ul><li>Platforms for other app domains </li></ul>Services Stream support Tasks MyApp.your.url.gov Custom Libs Messaging
    21. 21. appscale and RDMA <ul><li>Shared-memory support for co-located components </li></ul><ul><li>PaaS-IaaS coordination, adaptation, scheduling </li></ul><ul><li>Compute/data intensive workloads (HPC, messages, streams) </li></ul><ul><li>Database system support for different workloads </li></ul><ul><li>Programmatic interface and tools for applications developers </li></ul><ul><ul><li>Improve ease of use access </li></ul></ul>GAE App Developer (AppScale Admin) GAE App Users HTTPS App Controller ALB DB S/P AS GAE App Users GAE App Users AppScale Cloud Compute tasks (e.g. Map Reduce) DB M/P AppScale tools
    22. 22. appscale http://appscale.cs.ucsb.edu <ul><li>Thanks! </li></ul><ul><ul><li>Leads: Chris Bunch, Navraj Chohan </li></ul></ul><ul><ul><li>Development and research team: Jovan Chohan, Nupur Garg, Matt Hubert, Jonathan Kupferman, Puneet Lakhina, Yiming Li, Nagy Mostafa, Yoshihide Nomura (Fujitsu), Michal Weigel </li></ul></ul><ul><ul><li>Support </li></ul></ul><ul><ul><ul><li>Google, IBM Research, National Science Foundation </li></ul></ul></ul><ul><ul><li>OpenFabrics Alliance </li></ul></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×