ESA and the Cloud Jason Brazile and Ronnie Brunner Final presentation of Study on Cloud Computing ESRIN/Contract Nr. 22700...
Why Cloud? Example: US Govt. “Cloud first” <ul><ul><li>All CIO’s  must   define ≥ 3 projects by Q2 2011 </li></ul></ul><ul...
Why Cloud? Example: Netflix    Amazon <ul><ul><li>Gains: Agility, Reduced Cost </li></ul></ul><ul><ul><li>Thousands  of E...
Cloud Computing in a Nutshell <ul><li>Cloud Computing supports those who wish to &quot;try first, justify second&quot; </l...
The Cloud Computing Stack ESA and the Cloud Platforms as a Service (PaaS) Infrastructure as a Service (IaaS) Software as a...
IaaS <ul><ul><li>Stop  worrying  about  hardware </li></ul></ul><ul><ul><ul><li>Computation, storage, backup, hopefully so...
PaaS <ul><li>If non-monopoly platform exists:  Use it ! </li></ul><ul><ul><li>[Web Apps - Java] Google App Engine/AppScale...
SaaS <ul><ul><li>Standardize  / raise capabilities </li></ul></ul><ul><ul><li>Browser-based  (work anywhere) </li></ul></u...
Cloud Economics <ul><li>Basic principles : </li></ul><ul><ul><li>Economies of scale </li></ul></ul><ul><ul><li>Higher pric...
Our Market Study <ul><ul><li>Goal: Enterprise-  and  pure- cloud </li></ul></ul><ul><ul><li>But only enterprise and/or  st...
Market Status <ul><ul><li>IaaS: Convergence only on basics (compute, storage) </li></ul></ul><ul><ul><li>PaaS: Domain cove...
ESA and the Cloud <ul><ul><li>Many projects already  started  (2001, 2008, 2009, 2010): Collaboration, Web, Distribution, ...
ESA Cloud Computing success stories ESA and the Cloud Contact  ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
Portal Edge Caching, Media Distribution ESA and the Cloud Contact  ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
ESAC’s GAIA/AGIS “Data Train” <ul><li>Since 2009 (prototype)… </li></ul><ul><ul><li>Amazon EC2 / S3 </li></ul></ul><ul><ul...
G-POD Framework ESA and the Cloud Contact  ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
Collaboration Tools ESA and the Cloud Contact  ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
Virtual Archive ESA and the Cloud Contact  ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
Learnings so far ESA and the Cloud Contact  ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
Benefits <ul><li>IaaS (mainly AWS) </li></ul><ul><ul><li>Generally happy with price/performance </li></ul></ul><ul><ul><li...
Caveats: Gaps / Limitations / Challenges <ul><li>IaaS (mainly AWS) </li></ul><ul><ul><li>S3 costs large if large volume da...
Risks and their Consequences ESA and the Cloud Risk Examples Result Re-invention  of wheel Portal proliferation; User acco...
Example: Dealing with a worst case scenario… <ul><ul><li>NASA’s loss of original Apollo 11 moon landing footage </li></ul>...
Conclusions <ul><li>Coordination : </li></ul><ul><ul><li>Experience  should be  shared   </li></ul></ul><ul><ul><li>Govern...
Upcoming projects at ESA ESA and the Cloud Contact  ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
And now? What’s next? <ul><li>Visualize where you want to be… </li></ul><ul><ul><li>Maybe a CIO “ cloud dashboard ” </li><...
Contact ESA and the Cloud Jason Brazile [email_address] +41-44-247 79 25 Ronnie Brunner [email_address] +41-44-247 79 79 N...
Upcoming SlideShare
Loading in …5
×

ESA and the Cloud

2,643 views

Published on

Netcetera consultants Ronnie Brunner and Jason Brazile present the results of a year long study of existing and potential uses of cloud computing at the European Space Agency. Some unpublished internal material was removed. Queries can be directed to the contract's Technical Officer at ESA ESRIN.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,643
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • ESA and the Cloud Quality Software Engineering ESA and the Cloud This presentation covers the „Study on Cloud Computing“ of the project: Study on Cloud Computing and Management of EOP Content Distribution Network (CDN) Services ESRIN/Contract No. 22700/09/I-SB The study was delivered as a document covering the following content: Part I: Cloud Computing Today 1 A Definition of Cloud Computing 2 Technical Features of Cloud Computing 3 The Economics of Cloud Computing 4 The Cloud Ecosystem 5 Current Limitations of Cloud Computing 6 Market Study 7 Research and Governmental Projects Leveraging Cloud Computing Part II: Recommendations For The Adoption Of Cloud Computing Services By ESA 8 Introduction 9 Identification Of Candidate ESA IT Activities 10 Key Aspects For Evaluation Of Cloud Computing Providers 11 Methodology For The Economic Evaluation Of Cloud Computing Services 12 Proposal For A Program-level Governance For Cloud Computing Initiatives
  • ESA and the Cloud Quality Software Engineering “ Cloud first” There’s a concerted US government effort to go “Cloud first” in many government agencies Office of Management and Budget Department of Interior US Geological Survey NASA Many others Main goal Aim to reduce the existing thousands of data centers (and associated costs – both in money and otherwise) Department of Agriculture is consolidating 21 different messaging and collaboration systems into one system that will support 120,000 users GSA is contracting with Unisys Corp. to move 17,000 government e-mail accounts to Google Apps for lowering costs by 50 percent and saving $15 million over five years Consolidation of 800 of the government&apos;s 2,094 data centers by 2015 Big players (Microsoft, Google) see this as significant – enough to sue the US government over their bidding process in some cases
  • ESA and the Cloud Quality Software Engineering Background YouTube changed the game w.r.t. what is an ISP, a CDN, and a content provider. So many network transit providers didn’t want to get on the wrong side of the bandwidth equation that they were willing to peer with Google “for free”. So although on paper, it long appeared that YouTube was operating at a huge loss, the savings in aggregate network transit costs and peering desirability compensated and allowed Google to catch up with competitors such as Microsoft. Now Netflix is changing the game w.r.t. TV/Film content: They snuck in cheaply: Originally they were primarily a DVD by physical post distributor, and only later got into Internet distribution Hollywood studios sold Internet viewing rights cheaply because it was free money on top of the more substantial DVD distribution licenses But then the same (cheap) license agreements remained even when Internet viewing vastly overtook DVD postal distribution This seems to be repeating the YouTube-style event – see Level 3 vs ComCast “more than a peering spat” In any case, Netflix’s huge and quick growth… included migration of several core business functions (not just website, data distribution and mastering/recoding) and put a large amount of trust in Amazon’s ability to provide scale, and geographic coverage
  • ESA and the Cloud Quality Software Engineering Cloud Computing has many definitions The aspects most often cited are: Self-service: No need to wait for anybody to set something up for me On-demand: I can get it now, if I need it now Pay-as-you-go: It only costs as long as I actually use it This already implies elasticity: I can get as many resources as I want/need Economy of scale A driving factor are economies of scale that make it possible to reduce cost per user to almost nil. Additional attributes And also, Cloud Computing is a more than just the these points, but they catch the essence of Cloud Computing pretty well. Other important aspects to make Cloud Computing work are: Broad network access Resource pooling Measured service No commitment Pay by credit card Programmatic/”Mashup-able” SLA-driven Retail-model discipline
  • ESA and the Cloud Quality Software Engineering The Stack SaaS builds on PaaS PaaS builds on IaaS IaaS builds on Virtualisation Does one size fit all? No! Content Delivery Networks (CDN) Storage &amp; backup providers (maybe IaaS?) DB providers (Can this be called SaaS? Or is it part of IaaS?) Deployment models The Cloud is the Cloud, right? No! Public Cloud: The infrastructure is “publicly” available for anyone to use (Does not imply that everybody can read everybody’s data) Virtual Private Cloud: A provider sets up some part of its Cloud to be accessible by you only (using encryption and VPN technology) Private Cloud: You own the iron, but set it up like a Cloud to share workload of different internal users Disclaimer: The logos of providers and services on this and in all the following slides don’t imply actual recommendations, but are for illustration purposes only, except where explicitly mentioned.
  • ESA and the Cloud Quality Software Engineering IaaS This is probably the most well-known cloud service model reported by the media. However, cloud compute-on-demand is also an easier to understand next step after in-house virtualization and co-location. When the resource need is known in advance but generally very irregular over time, IaaS is a viable approach to cost savings. However in many application the resources needed over time are not known especially if it depends on external demand (such as a web site that gets very popular because of a specific event). However, after initial “low-hanging fruit” of just moving a virtualized service into the “Cloud” next steps get tougher: Scaling a service needs manual effort or additional tools and measures There is likely a need to re-design (maybe even from scratch) to successfully move to an inherently more distributed platform. Only on PaaS (migrating to which may also need heavy application architecture change) do you get (in some cases at least) automatic reduction of scaling details
  • ESA and the Cloud Quality Software Engineering PaaS Platforms simplify architecture of applications by providing a more abstract model of its service, providing simpler deployment and hiding the complexity of scalability behind their interface to the application developer. In the best case (i.e. when there is a good fit to the service’s model) a user does not have to worry about scalability or availability simply by adhering to the platforms APIs. Lock-in The biggest drawback to consider however is “lock-in”: For IaaS, there are already many abstraction (cross-platform) APIs around that help mitigate the risk of locking the user to a specific provider. For platforms, this will be more difficult, although there are also already first open source implementations of existing platforms (e.g. AppScale, which is an open source implementation of the Google AppEngine)
  • ESA and the Cloud Quality Software Engineering SaaS A SaaS provider is basically a ASP, as it was called a couple of years ago. While the Cloud lets providers leverage existing infrastructure and the traditional ASP based his offering mostly on his own infrastructure, the difference for the actual user is insignificant. With SaaS too, the main characterization is: self-service, pay-as-you-go and on-demand. Moving from local installations to SaaS allows new capabilities that are not possible locally: e.g. simultaneous editing of documents. Variety Whenever a software can potentially be accessed with a browser, it is a candidate for SaaS. This implies that the number of offerings and services will explode even more in the near future. The difficult question when exploring possibilities is that the market is changing even faster than the IaaS and PaaS market.
  • ESA and the Cloud Quality Software Engineering Basic principles Economies of scale: cheaper to produce the service for the provider Pay a higher price, but for a (much) shorter time: Accommodate for variability in demand (for buyer) No idling resources (for buyer) Utilization and variability of demand are key factors: The better utilized any system is, the lower is the TCO Responsibility is transferred to Cloud provider: Seller needs to provide elasticity The higher in the (Cloud) stack, the more responsibility is transferred to the seller Moving from CapEx to OpEx: Makes it easier for buyer to budget (no investment) Opting-out is easier with OpEx (although it usually requires an alternative) The Cloud stack builds the economic basis It’s very easy for a provider to base its services on top of existing Cloud services: Write a self-service application for an existing platform and you can provide a SaaS offering that’s scales easily and where operations cost is strictly based on actual usage. The higher “up” a service, the more value it adds for the customer. It leverages the underlying services and allows for every provider in the value chain to earn money.
  • ESA and the Cloud Quality Software Engineering Initial goal The initial goal of the market study was to get a clear picture of the status of Enterprise Cloud providers as well as “pure” Cloud providers. While Enterprise providers were generously giving information, pure cloud providers did not respond to our inquiries. Concentration on IaaS providers and analysis As the offerings of Enterprise Cloud providers are very customer specific, as these providers traditionally do not only specific consulting, but very specific implementation, it was difficult to create a generalized view on their services. “Tell as what you need and we’ll implement it for you” is the basic summary for services offered by Enterprise cloud providers. For the study that was not helpful, so we concentrated on pure IaaS players and used publicly available information to enhance our study results. The big players The current big players (specifically in the infrastructure services area) are monitored by various sources like Guy Rosen’s “State of the Cloud”: they run queries on the top 500k sites identified by quantcast.com and assign them to infrastructure providers. Top players currently: Amazon EC2, Rackspace (closing the gap to Amazon) and Linode.
  • ESA and the Cloud Quality Software Engineering How big is the Cloud? In 2010: SaaS: ~$11.7 billion (Forrester) PaaS: ~$311 million (Forrester) IaaS: ~$1 billion (Forrester, Economist, UBS) Total: ~$13 billion In 2020: IaaS: ~$4 billion (Forrester) Total: ~$56 billion Source: Economist, “Tanks in the cloud”, 29. Dec 2010, (http://www.economist.com/node/17797794) Consolidation is happening now and even more in the near future Rackspace acquired Cloudkick in December 2010 Salesforce acquired Heroku in December 2010 Teradata to acquire Aprimo, December 1020 The “big ones” also heavily acquire Cloud companies IBM acquired Unica, Cast Iron Systems, Netezza, Coremetrics, … HP acquired Stratavia, ArcSight, Melodeo, … Dell acquired Everdream, SilverBack, MessageOne, Perot Systems, InSite One, Boomi, … Current developments The market is still changing at an enormous pace. New SaaS offerings pop up on a daily basis and even the current IaaS market leader Amazon regularly innovates with new services: In November 2010, Amazon Web Services (AWS) achieved Level 1 compliance with the Payment Card Industry (PCI) Data Security Standard, which means its cloud can process credit card transactions and store credit card data. In January 2011, Amazon launched a new PaaS: “Elastic Beanstalk”, an auto-scaling application platform to operate (primarily) Java applications.
  • ESA and the Cloud Quality Software Engineering Status Potential and existing Cloud projects at ESA Projects started already (since 2001, even before Cloud Computing was termed ;-) in the following areas: Collaboration Web Distribution Prototyping Software development … and others Main adoption areas Archive / Storage Collaboration Distribution Processing Current challenges Long term archive Large data transfer Cloud Management Centralized vs. Distributed? Public/Private Cloud? Lock-in / interoperability Governance … and others Office Projects / Topics involved ESTEC Gaia Operations EOP-GU EO Catalogue, UI, Web-Portal CEOS Cal/Val, Portal EOP-GG Data Reprocessing, Large Volume Data Processing, Processing on-demand with toolboxes Corp IT SaaS (SharePoint, WebEx) LEX Corporate Communications EOP-GTR G-Pod, Processing on-demand with toolboxes EOP-GQ Data Reprocessing, Large Volume Data Processing, EOP Data Distribution, L0 Cal/Val, QC, ESL OPS-L Space Situational Awareness Processing, Security EOP-GS EOP Data Distribution, L0 Cal/Val, QC, ESL TEC Simulation/Collaboration EOP-S Grid, Infrastructure, EO Catalogue, UI, Web-Portal, Processing on-demand EOP-GQ Data Reprocessing, Large Volume Data Processing, EOP Data Distribution, L0 Cal/Val, QC, ESL LEX-CCW Corporate Communications Web and Distribution Unit (Akamai, Flickr, YouTube) EOP-GTF Data Reprocessing, Large Volume Data Processing, EOP Data Distribution, L0 Cal/Val, QC, ESL OPS-C Operations directorate and Infrastructure EOP-GM Data Distribution, L0 Cal/Val, QC, ESL, Supersites Virtual Archive EOP-GU EO Catalogue, UI, Web-Portal EOP-GTF Data Reprocessing, Large Volume Data Processing, EOP Data Distribution, L0 Cal/Val, QC, ESL EOP-GT Data Reprocessing, Large Volume Data Processing OPS-GI EC2 for development ESAC Gaia Development and Testing
  • ESA and the Cloud Quality Software Engineering
  • ESA and the Cloud Quality Software Engineering LEX-CCW ESA&apos;s Communication &amp; Knowledge Department is responsible for keeping the world&apos;s media, decision makers and the public up to date with what is happening at ESA and for providing news on all its latest activities. Among this department&apos;s main activities is organizing large media events and press conferences, and prepare written and audiovisual material for specific target groups. The Communication &amp; Knowledge Department also organizes exhibits, maintains the main photo and video archives and the corporate web portal. Content distribution Since 2001, ESA is using Akamai to cache/deliver esa.int portal content, such as HTML, images, and video. (See: www.esa.int) Digital asset management The new “Digital Asset Management&amp;quot; (DAM) project, plans to use the eZ Publish platform to offer a video software as a service internally to scientific users within the Agency. The ESA Video Archive itself uses Highwinds CDN for content distribution (See: multimedia.esa.int) Social media accounts YouTube: ESA (as a partner) is allowed &gt;10min movies Flickr Pro: (unlimited #of images, and size) Why interesting for the Cloud? There is no real alternative to using a CDN to distribute data globally very fast.
  • ESA and the Cloud Quality Software Engineering ESAC The European Space Astronomy Centre (ESAC) is ESA’s centre for space science. It is located in Villanueva de la Cañada, close to Madrid in Spain . GAIA / AGIS Gaia is an astrometry mission with the primary goal of making the largest, most precise three-dimensional map of the Milky Way Galaxy by surveying more than one billion stars. One of the largest data processing components of that mission is the AGIS (Astrometric Global Iterative Solution) component, which is tasked with computing key motion and positional attributes of the stars observed by the mission&apos;s satellite, due to be launched in 2012. Amazon AWS prototype Only 20 person days (with UK-based The Server Labs) were needed to get the software running in Amazon cloud Oracle ASM Image based on Oracle Database 11g Release 1 Enterprise Edition - 64 Bit (Large instance) The prototype revealed and helped solve a scalability problem in code – The team never had 100 nodes to test before: Only 4 lines of code needed to be changed It ran at similar performance to existing in-house cheap cluster – E2C indeed is no super computer Availability of large number of nodes very interesting – not affordable in-house today Why interesting for the Cloud? Gaia needs to process data only twice per year, but then as massively as possible. Exclusive hardware would be used at very bad utilization this way. This means, that the more nodes can by used, the shorter the run cycle (faster delivery) and the cheaper AWS becomes in comparison to in-house hardware (if it cannot be used for other projects in the time between the runs).
  • ESA and the Cloud Quality Software Engineering
  • ESA and the Cloud Quality Software Engineering
  • ESA and the Cloud Quality Software Engineering
  • ESA and the Cloud Quality Software Engineering
  • ESA and the Cloud Quality Software Engineering Benefits There is a multitude of benefits in different categories. For IaaS the benefits concentrate around price (Cloud can be cheaper) and convenience. Delegation of responsibility for hardware and elasticity works. It’s clear however, that the case in question must properly be calculated as using the Cloud can also be more expensive than using an in-house alternative. For CDN the case is different: The service a CDN provide simply cannot be provided internally. In that sense, the only alternative would be to setup your own private global CDN, which by definition is then not really an in-house alternative. For content and media distribution the argumentation is similar. The benefits for SaaS in that context cannot easily be generalized. While there is no alternative to using external social media platforms to reach out globally, there are certainly alternatives for other types of services like collaboration etc.
  • ESA and the Cloud Quality Software Engineering IaaS While computation as the base Cloud service works pretty good, many of the main issues concern data input and output: Getting the data to where the compute capacity is a big problem. The fact that shipping physical devices is actually considered a valid alternative is already kind of a defeat reaction to the problem of not having enough bandwidth to transfer data online. Even with transfer rates in the Gigabit range, data volumes of 100 TB and more are infeasible for online transfer. Another issue of IaaS services is that scaling the resources is up to the user. This is one argument why a further abstraction (towards PaaS) is a good idea. CDN The main issue with CDNs is that they do not follow the Cloud paradigm. Whether this is really a problem is up to the reader. Obviously, as long as it fits the purpose, we should not care too much. However, a more flexible model with respect to on-demand, self-service, and pay-as-you-go would likely help to leverage the benefits even more. SaaS The user base for almost any software services is generally much larger than for underlying services such as PaaS and IaaS, because SaaS provides services to the end user. This implies that the acceptance and learning curve are very important factors to consider when evaluating a service. This is also the reason why switching providers is much more expensive than for lower level services: retraining and buy-in of a large user base to accept a new / different / other system incurs significant cost. General issues and (still) missing services Manageability and real-time tracking of cost often insufficient Long-term Archive solutions unclear and storage integrity rarely guaranteed Bandwidth / data transfer vs. data processing location is problematic (disk shipping although promising is not really a solution that will scale) Data deluge (e.g. Sentinel) requires even more capacity Security issues often not solved yet, governance often not clear Lock-in is likely (very few interoperability commitments between services) User expectations difficult to meet (iTunes-like convenience expected) Managing SLAs is difficult Scale Although one of the main driving factors for the economics of Cloud Computing are economies of scale, some aspects are just not ready now and they will become even greater bottlenecks specifically in respect to the future development of data volume and bandwidth needs
  • ESA and the Cloud Quality Software Engineering Typical risks The slides collects some critical risks that one might become more exposed to through the use of Cloud Computing. Being aware of what can go wrong and how bad it can be, is very important especially for first movers. It only takes one catastrophic early experience to cause potentially over-compensating measures which could limit the ability to profit from Cloud benefits for a long period of time before trust is again rebuilt or the technology becomes so mainstream that it overcomes the negative bias. Devising measures to mitigate the risks After having identified such risks, it is in everyone’s interest to define reasonable measures to prevent such cases from occurring. Easy to take measures include encryption of data or the introduction of checklists. Such measures are what many people think of as governance. Perhaps the easiest way to get buy-in is to elicit recommendations on such measures from the first movers. Such users are likely to recognize the danger of negative events and are more likely to actively participate in governance measures that they have helped to devise.
  • ESA and the Cloud Quality Software Engineering Worst case scenarios also happen outside of the cloud context
  • ESA and the Cloud Quality Software Engineering Summary of Learnings Cloud Computing works There are many opportunities Can be cheaper than in-house alternatives There are some challenges yet to be solved Some technological gaps have yet to be closed There is no need to wait because of that A sample from what others have learned: Conclusions from Netflix large move to Amazon Goals of the migration: Be Faster, Scalable, Available, Productive Main lesson: You only really learn by committing to it Development: Use latency tolerant protocols Can be faster to re-code than fixup datacenter-based apps Services (not jar files) as components Instrumented “service patterns” (rather than code) Operations: Datacenter oriented tools don’t work in the cloud Cloud tools don’t scale for the enterprise (yet) Lots of cloud tools available, but demand “deep-linking” ability Try to hide keys Source: Adrian Cockcroft, Netflix in the Cloud, Nov 2010
  • ESA and the Cloud Quality Software Engineering
  • ESA and the Cloud Quality Software Engineering Cloud Vision Obviously, the dashboard on this slide is just an idea for a teaser and the following discussion of where ESA wants to go and how to get there, but it shows some fundamental goals that should be high on the list: Leverage the first movers’ experience Be able to calculate the “true cost” of all IT services Have clear governance Know where the risks are and what measures to take to mitigate them From Virtualization to “Hypercloud” According to Jake Sorofman, it is possible to take a staged approach to adopting cloud computing where at each state you successfully improve upon the previous stage in terms of capabilities. However you also realize ROI with each step to justify the effort of handling the whole move in smaller steps instead of a giant leap. This of course also mitigates risks, assuming you don’t get stuck somewhere along the way. The ultimate goal is what he calls the “Hypercloud” which is defined by: Dynamic sharing of app workload Capacity arbitrage Self-service application provisioning Interpreting this pyramid considering the current market, ESA’s Cloud experience and learnings: Virtualization Cloud experimentation Cloud foundation (The missing PaaS experience) Cloud exploitation Hypercloud
  • ESA and the Cloud Quality Software Engineering Information For more information about the “Study on Cloud Computing”, the actual study document and other inquiries, please contact: ESA/ESRIN Via Galileo Galilei Casella Postale 64 00044 Frascati (Roma) Italy
  • ESA and the Cloud

    1. 1. ESA and the Cloud Jason Brazile and Ronnie Brunner Final presentation of Study on Cloud Computing ESRIN/Contract Nr. 22700/09/I-SB 2011-01-28 ESA and the Cloud
    2. 2. Why Cloud? Example: US Govt. “Cloud first” <ul><ul><li>All CIO’s must define ≥ 3 projects by Q2 2011 </li></ul></ul><ul><ul><li>By Q4, 1 must be in operation </li></ul></ul><ul><ul><li>By June 2012, all 3 must be </li></ul></ul><ul><ul><li>“ Security concerns not enough” </li></ul></ul>ESA and the Cloud
    3. 3. Why Cloud? Example: Netflix  Amazon <ul><ul><li>Gains: Agility, Reduced Cost </li></ul></ul><ul><ul><li>Thousands of EC2 nodes </li></ul></ul><ul><ul><li>Petabytes of S3 </li></ul></ul><ul><ul><li>Hadoop clusters </li></ul></ul><ul><ul><li>Akamai/Limelight(CDN) use </li></ul></ul>ESA and the Cloud Adrian Cockcroft, Netflix in the Cloud, Nov 2010
    4. 4. Cloud Computing in a Nutshell <ul><li>Cloud Computing supports those who wish to &quot;try first, justify second&quot; </li></ul><ul><li>- James Staten, Forrester </li></ul><ul><li>Cloud Computing is: </li></ul><ul><ul><li>Self-service </li></ul></ul><ul><ul><li>On-demand </li></ul></ul><ul><ul><li>Pay-as-you-go </li></ul></ul><ul><li>CC allows transforming CapEx to OpEx </li></ul><ul><li>- William Fellows, 451 Group </li></ul>ESA and the Cloud Judith Hurwitz et. al., Cloud Computing for Dummies, Wiley Publishing, Inc. 2010
    5. 5. The Cloud Computing Stack ESA and the Cloud Platforms as a Service (PaaS) Infrastructure as a Service (IaaS) Software as a Service (SaaS) Cloud Enablers / Cross platform solutions
    6. 6. IaaS <ul><ul><li>Stop worrying about hardware </li></ul></ul><ul><ul><ul><li>Computation, storage, backup, hopefully soon archive… </li></ul></ul></ul><ul><ul><ul><li>Interoperability, maintenance, provisioning etc. is still in your responsibility </li></ul></ul></ul><ul><ul><li>Our advice: try it, but deliberately … </li></ul></ul><ul><ul><ul><li>Abstract from proprietary interfaces </li></ul></ul></ul><ul><ul><ul><li>Plan toward platforms (esp. design and DB implications) </li></ul></ul></ul>ESA and the Cloud Infrastructure as a Service (IaaS) IaaS Abstraction APIs PaaS Strategic plan
    7. 7. PaaS <ul><li>If non-monopoly platform exists: Use it ! </li></ul><ul><ul><li>[Web Apps - Java] Google App Engine/AppScale </li></ul></ul><ul><ul><li>[Web Apps - RoR] Heroku/EngineYard </li></ul></ul><ul><ul><li>[Processing] Amazon-Hadoop/Rackspace-Hadoop/Cloudera </li></ul></ul>ESA and the Cloud $ ./appcfg.sh update ../../myapp/war $ ./appcfg.sh update_indexes ../../myapp/war $ ./appcfg.sh request_logs ../../myapp/war mylogs.txt $ heroku create myapp Created http://myapp.heroku.com/ git@heroku.com:myapp.git $ git remote add heroku git@heroku.com:myapp.git e.g. e.g. Platforms as a Service (PaaS) VS IaaS  Do-it-yourself = Complicated PaaS  Auto-scaling = Simpler
    8. 8. SaaS <ul><ul><li>Standardize / raise capabilities </li></ul></ul><ul><ul><li>Browser-based (work anywhere) </li></ul></ul><ul><ul><li>Higher “collaborativity” </li></ul></ul><ul><ul><li>e.g. simultaneous editing </li></ul></ul>ESA and the Cloud Software as a Service (SaaS) Project Mgmt CRM Office Desktop Sharing
    9. 9. Cloud Economics <ul><li>Basic principles : </li></ul><ul><ul><li>Economies of scale </li></ul></ul><ul><ul><li>Higher price, for a shorter time </li></ul></ul><ul><ul><li>Utilization/burstiness  key factors </li></ul></ul><ul><ul><li>Some responsibility  provider </li></ul></ul><ul><ul><li>CapEx  OpEx </li></ul></ul><ul><li>Logical progression in hosting: </li></ul><ul><li>Virtualization + co-location  </li></ul><ul><li>Cloud Computing </li></ul>ESA and the Cloud
    10. 10. Our Market Study <ul><ul><li>Goal: Enterprise- and pure- cloud </li></ul></ul><ul><ul><li>But only enterprise and/or startup providers responded </li></ul></ul><ul><ul><li>Unable to see clear outcome (Gartner did in Dec 2010 – yet very controversial ) </li></ul></ul><ul><li>Instead, we turned to “pure cloud” IaaS </li></ul><ul><ul><li>Top Websites  “real work” proxy </li></ul></ul><ul><ul><li>Amazon/Rackspace in lead </li></ul></ul><ul><ul><li>>20 more attempting to compete </li></ul></ul><ul><ul><li>Technically possible to abstract IaaS </li></ul></ul>ESA and the Cloud Guy Rosen, “State of the Cloud”, January 2011 Leong/Chamberlin, Gartner’s Cloud IaaS and Web Hosting, Dec 2010
    11. 11. Market Status <ul><ul><li>IaaS: Convergence only on basics (compute, storage) </li></ul></ul><ul><ul><li>PaaS: Domain coverage poor (limited to web, map/reduce) </li></ul></ul><ul><ul><li>SaaS: Very diverse, some already established, and “we’ve only seen the beginning” </li></ul></ul><ul><ul><li>Moving target : </li></ul></ul><ul><ul><li>Consolidation expected </li></ul></ul><ul><ul><li>New players expected </li></ul></ul><ul><ul><li>Platforms winning importance </li></ul></ul><ul><ul><li>Total market will grow from > $10 billion in 2010 to > $55 billion by 2020 </li></ul></ul>ESA and the Cloud Estimated Amazon VMs in US east cost data centers Source: Cloudkick Economist, “Tanks in the cloud”, 29 Dec 2010
    12. 12. ESA and the Cloud <ul><ul><li>Many projects already started (2001, 2008, 2009, 2010): Collaboration, Web, Distribution, Prototyping, Software development </li></ul></ul><ul><ul><li>More are coming soon … </li></ul></ul><ul><ul><li>Yet there are still technical, political and other challenges (see later slides) </li></ul></ul>ESA and the Cloud Contact ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
    13. 13. ESA Cloud Computing success stories ESA and the Cloud Contact ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
    14. 14. Portal Edge Caching, Media Distribution ESA and the Cloud Contact ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
    15. 15. ESAC’s GAIA/AGIS “Data Train” <ul><li>Since 2009 (prototype)… </li></ul><ul><ul><li>Amazon EC2 / S3 </li></ul></ul><ul><ul><li>Oracle as a service </li></ul></ul>ESA and the Cloud Parsons et al., Cloud Science or Astrometric Data Processing in Amazon EC2 May 2009 O’Mullane, GAIA Data Processing and Challenges, June 2010 O’Mullane, GAIA Data Processing and Challenges, June 2010
    16. 16. G-POD Framework ESA and the Cloud Contact ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
    17. 17. Collaboration Tools ESA and the Cloud Contact ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
    18. 18. Virtual Archive ESA and the Cloud Contact ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
    19. 19. Learnings so far ESA and the Cloud Contact ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
    20. 20. Benefits <ul><li>IaaS (mainly AWS) </li></ul><ul><ul><li>Generally happy with price/performance </li></ul></ul><ul><ul><li>Migrating to Cloud less effort than expected </li></ul></ul><ul><ul><li>EC2 (just computation) cost surprisingly small </li></ul></ul><ul><ul><li>Inclusion of software licenses in per node/hour price is convenient </li></ul></ul><ul><ul><li>In particular, “Oracle as a service” is attractive </li></ul></ul><ul><ul><li>Helped find code scalability problems (cheap scaled testing) </li></ul></ul><ul><ul><li>Financially interesting for uneven/low utilization (bursty loads) </li></ul></ul><ul><ul><li>Productized import/export (disk shipping) looks interesting (for > 1TB) </li></ul></ul><ul><li>CDN </li></ul><ul><ul><li>Speed / global reach, performance </li></ul></ul><ul><ul><li>Potentially reduced network transit costs </li></ul></ul><ul><ul><li>More data products available online </li></ul></ul><ul><li>SaaS </li></ul><ul><ul><li>Social media work: Twitter, Facebook, … </li></ul></ul><ul><ul><li>Distribution / Sharing works: Flickr, YouTube </li></ul></ul><ul><ul><li>Collaboration works: Webex, SharePoint </li></ul></ul>ESA and the Cloud Image Source: Level 3 Communications Identifying %utilization vs cost crossover points for e.g. 10 and 100 nodes
    21. 21. Caveats: Gaps / Limitations / Challenges <ul><li>IaaS (mainly AWS) </li></ul><ul><ul><li>S3 costs large if large volume data is “parked” </li></ul></ul><ul><ul><li>Data transfer often slower than needed </li></ul></ul><ul><ul><li>Contingency planning costly </li></ul></ul><ul><ul><li>Inconsistent inter-node performance </li></ul></ul><ul><ul><li>Often data is not &quot;already there&quot; </li></ul></ul><ul><ul><li>Manual scaling is needed (and apps must be changed for scalability) </li></ul></ul><ul><ul><li>Changing to a different instance (bigger, smaller) for EC2 is manual and off-line </li></ul></ul><ul><ul><li>AWS default storage is not persistent </li></ul></ul><ul><ul><li>Instances must be deleted to halt charges </li></ul></ul><ul><li>CDN </li></ul><ul><ul><li>Neither pay-as-you-go, on-demand, nor “no commitment”: ≥6 months contracts, usage caps </li></ul></ul><ul><ul><li>Not very self-service: emails needed for config changes, lead time higher than expected </li></ul></ul><ul><ul><li>Product pricing and structure complex (difficult to understand): needs sales engineer </li></ul></ul><ul><li>SaaS </li></ul><ul><ul><li>Often needs &quot;digital natives&quot; involved in execution to be effective (e.g. for social media) </li></ul></ul><ul><ul><li>Learning curve/acceptance varies greatly: many users compared to PaaS, and IaaS </li></ul></ul><ul><ul><li>Switching providers is expensive </li></ul></ul>ESA and the Cloud Source: AWS “When to use import/export” Internet connec-tion Min. time to transfer 1TB at 80% utilization When to consider import/ export? T1 (1.5Mbps) 82 days ≥ 100GB 10Mbps 13 days ≥ 600GB T3 (44.7Mbps) 3 days ≥ 2TB 100Mbps 1-2 days ≥ 5TB 1000Mbps < 1 day ≥ 60TB
    22. 22. Risks and their Consequences ESA and the Cloud Risk Examples Result Re-invention of wheel Portal proliferation; User account mess <ul><ul><li>Poor services, inefficiency </li></ul></ul>Individual “contracts” via credit card Critical service is down because key person‘s individual credit card expires Service failure , data mess (where’s what?) Single actor can chose wrong direction quickly Introduction of a proprietary SaaS solution that (only) provides a quick fix Unmanaged service portfolio, not reaching strategic goals Costs can‘t be tracked well Monthly bills unpredictable due to irregular demand. Lots of hard to track small transactions with many providers Financial exposure and uncertainty Costs slowly increase Nobody cleans up hard disks or gets rid of unused virtual machines More expensive over time, unclear what‘s still needed Data gets leaked Data protection violation, leak of industry partner’s (or member state‘s) secrets Financial liability, loss of trust Data loss NASA‘s moon landing tapes, hacker data vandalism, Provider default Image / brand damage
    23. 23. Example: Dealing with a worst case scenario… <ul><ul><li>NASA’s loss of original Apollo 11 moon landing footage </li></ul></ul><ul><ul><li>Headlines: &quot; Loss of tapes went unnoticed for 35 years ”, “ The scientific equivalent of recording an old episode of EastEnders over the prized video of your daughter's wedding day ” (The Guardian) </li></ul></ul><ul><ul><li>NASA’s image recovery plan? </li></ul></ul><ul><ul><ul><li>$230’000  Hollywood’s Lowry Studios for restoration </li></ul></ul></ul><ul><ul><ul><li>Press conference to “provide the public, future historians, […] the highest quality video of this historic event ” </li></ul></ul></ul><ul><ul><li>Lesson learned: You don’t want to go there! </li></ul></ul>ESA and the Cloud Original (lost or erased) Source: James Grahame, Retrothing After 2009 Lowry Restoration Source: NASA Multimedia Best existing copy Source: NASA Multimedia
    24. 24. Conclusions <ul><li>Coordination : </li></ul><ul><ul><li>Experience should be shared </li></ul></ul><ul><ul><li>Governance should encourage desirable behavior </li></ul></ul><ul><li>Migration : </li></ul><ul><ul><li>Lock-in avoidance requires effort </li></ul></ul><ul><ul><li>Each category (IaaS, PaaS, SaaS, CDN) has clear winning situations </li></ul></ul><ul><ul><li>“ Private Cloud for steady demand + public Cloud for peak ” looks promising </li></ul></ul><ul><li>Development : </li></ul><ul><ul><li>Differences between Cloud vs Enterprise application design can be substantial </li></ul></ul><ul><ul><li>Even software development via remote desktop access to a Cloud is feasible </li></ul></ul><ul><li>Operations : </li></ul><ul><ul><li>Provider management is not trivial </li></ul></ul><ul><ul><li>Governance needs to be deliberate </li></ul></ul><ul><ul><li>For operations: DR and BC measures must be designed (not a given) </li></ul></ul>ESA and the Cloud Image source: Aurora Meyer blog Image source: Peeraware.com Image source: Microsoft flight simulator
    25. 25. Upcoming projects at ESA ESA and the Cloud Contact ESRIN/Contract Nr. 22700/09/I-SB Technical Officer
    26. 26. And now? What’s next? <ul><li>Visualize where you want to be… </li></ul><ul><ul><li>Maybe a CIO “ cloud dashboard ” </li></ul></ul><ul><ul><li>Your 1st movers are actively sharing best practices experience </li></ul></ul><ul><ul><li>You know the “ true cost ” of IT </li></ul></ul><ul><ul><li>Governance measures known and respected by all </li></ul></ul><ul><li>From Virtualization to “ Hypercloud ”: </li></ul><ul><ul><li>Dynamic sharing of app workload </li></ul></ul><ul><ul><li>Capacity arbitrage </li></ul></ul><ul><ul><li>Self-service application provisioning </li></ul></ul>ESA and the Cloud Source: Jake Sorofman (rPath), “How to Achieve the Strategic Value of Cloud While Delivering Real ROI”, 3 March 2009
    27. 27. Contact ESA and the Cloud Jason Brazile [email_address] +41-44-247 79 25 Ronnie Brunner [email_address] +41-44-247 79 79 Netcetera Zypressenstrasse 71 CH-8040 Zürich +41-44-247 70 70

    ×