Every day we see an article or two written in some newspaper or magazine about CC. There are numerous conferences and camps being organized about this subject. There is a lot of buzz around and some people claim it is too much hype, while others opine that this is the new generation of computing. There are a lot of questions being asked. What are the implications to developers? What does it mean to next generation of datacenters? What does it mean to enterprises? The intent in the next 45 minutes is to try and clear some of the haze around CC. This is what I intent to cover. From a brief look at CC definition, to what are the various cloud layers, to who are the major players in this space. Then talk about how business can potentially take advantage of the technologies to innovate. Finally a look at some aspects of cloud evolution and adoption…
Here is an use case of how I have put together this slide deck. Besides my personal experience while I was at Yahoo!, I have leveraged the (web) cloud for a lot of data.
A quick look at how the term Cloud originated. In the early days of the internet, ‘cloud’ metaphor was used to represent a departmental network, a service providers private network or the internet itself. Typically architecture diagrams contained these networks which were represented as a cloud. In a number of cases it use to represent a network which was beyond ones control, like a service provider cloud (telecom). As we begin to leverage the internet more and more for a variety of needs, the ‘cloud’ is taking on more significance.
A map of the internet circa 2003 showing the connections between differen t internet routers, from the Opte Project .
Eucalyptus - Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems. Originated in the CS dept of Univ of CA at Sanata Barbara as a research project. An Emerging computational model in which applications, data and IT resources are provided ‘as services’ to users over the web (“Cloud”).
Here is a good representation of what is a cloud and what is a cloud service. Cloud Definition: Clouds are hardware-based services offering compute, network and storage capacity where: Hardware management is highly abstracted from the Buyer; Buyers incur infrastructure costs as variable OPEX; Infrastructure capacity is highly elastic (up or down) Characteristics of clouds Characteristic: Enterprises incur no infrastructure capital costs, just operational costs and operational costs are incurred on a pay-per-use basis, with no contractual obligations Characteristic: Capacity can be scaled up or down dynamically, and immediately, which differentiates from traditional hosting service providers Characteristic: The underlying hardware can be anywhere geographically Characteristic: Architecture specifics are abstracted. In addition, run in multi-tenancy mode with multiple users accessing the infrastructure simultaneously For the Cloud Service, the underlying infrastructure (platforms and hardware) are completely abstracted. The infrastructure/resource can scale elastically. An Emerging computational model in which applications, data and IT resources are provided ‘as services’ to users over the web (“Cloud”). How did the cloud providers come about? Amazon was one of the earliest and pioneers in this regard. The average utilization of servers in the data centers varied between 10 and 20%. The so called efficient dc’s were towards the 20% mark. Typically, companies provisioned for peak loads. This resulted in a lot of resources being under utilized. This was not only in production environments, but also in staging and other redundant environments like DR. The made the CXO’s think of ways to optimize investments. Amazon was the leader in this model with their EC2 and S3 Cloud capabilities.
Pre-cursor to CC was thin clients (ASP), grid computing and utility computing…Some people call it the 5 th generation of computing. We had Mainframes, PC, Client-Server, Internet, now Cloud. The emergence of standardized capabilities at various tiers is enabling companies to bring to market internet scale products quickly. At an extreme is the case where we have a lay person who can whip up an internet application without having to write a single line of code. This can be deployed easily on a robust infrastructure (platform and os) that is bound by SLA’s. We may not know what are resource requirements are upfront. In other cases, we do not have to provision for peak load. This will manage itself based on demand. The elastic nature of the cloud provider will handle the resource (storage, compute and network) needs. Since the cloud providers specialize in these domains, the quality and integrity of these services are high. They worry about the licenses, upgrading software administration etc.. Typically managing large scale data (especially web scale) is non-trivial. Storing and processing large amounts of data can be leveraged from service providers. A number of proof points exist today on this front. What are the implications to developers? While this enables faster and efficient means of developing applications, it still requires smart people to build the web service components, frameworks, the other underpinnings which are key to CC developing. Creates opportunities for technology transformation leading to newer business models. Small & Medium Businesses (Can take advantage of this paradigm to innovate and come up with new business models) Shorter time to market. Reduced investment and risk proposition Large Enterprises: Opportunity to improve utilization and focus on core competencies and drive innovation. Companies can take this opportunity to rewire their technology stack. A number of large companies are engaged in this activity. Companies have already made significant investment in resources (data center, technology etc..). This is an opportunity to increase the ROI. Cloud Providers:
Here is a look at how various cloud computing technologies can be arranged or stack up… In the earliest days, we had the mainframes which did pretty much all the heavy lifting. Then emerged the PC’s which revolutionized desktop computing. With the emergence of servers, we entered the multi-tier computing era. From 2-tier, 3-tier to multi-tier computing. Then came the internet and all the layers and services associated with it. Now let us see what the cloud and its layers are… At the foundation we have the infrastructure layer. This forms the hardware (Network, Systems and Storage) and the operating systems layer. This is referred to as the IaaS (Infrastructure as a Service) layer. This is normal terminology refers to the ‘Cloud’. There are technologies which enable abstraction of this layer. Virtual Machines are the main technology that is leveraged by the so called ‘cloud providers’ to offer this layer as a service. The IaaS layer typically comprises of nodes which range from 10^4 to 10^6 in number. Across these grids or clusters that are widely distributed, work loads needs to scheduled, processed and managed. Software Framework and tools that perform these tasks are classified as the Platform layer. The service offered by this layer has been PaaS. With this new paradigm, a new set of services that will utilize the platform and infrastructure forms the next layer. Services, like Billing, Security and Storage form part of this layer. The top most layer the Cloud applications layer which uses the services offered by the stack below. Since most of the heavy lifting is abstracted and offered as a service, building of a cloud application is simplified to a certain extent. This layer is called Saas or AaaS.
Some of these example companies have been operating in a difference model. High Performance, Grid and Cluster computing existed for some time now. The cloud offers a new model for them to evolve their products. Some of them have been operating in the enterprise space which is more of a private cloud environment. Ex: Gemstonre:- GemFire Enterprise is in-memory distributed data management platform that pools memory (and CPU, network and optionally local disk) across multiple processes to manage application objects and behavior. Ex: Aptana Cloud Connect: Develop, Host and Manage Web Apps in the cloud (PHP, Rails, Jaxer and Java) Ex: HP’s Magcloud:- We’re MagCloud and we want to help you to publish your own magazines. All you have to do is upload a PDF and we'll take care of the rest: printing, mailing, subscription management, and more. OpenID , the open online identity standard. OpenID is a free and easy way to use a single digital identity across the Internet. OAuth , the secure data portability standard. An open protocol to allow secure API authorization in a simple and standard method from desktop and web applications. [when a Website wants to import your contacts from another Web service, it asks for your login and password credentials. OAuth gets around that by sending you back to the original site where you login and authorize the one-time transfer of data. It is much more secure. And now it works with OpenID. ]
This slide shows a cloud taxonomy about 6 months ago. We can see the service providers classified in those four areas of I, P, S and A.
A number of use cases can be considered…. Social Apps (Rich and scales fast) Search Apps (Leverage the web/cloud) Analytics (Requires crunching and analysis of large quantities of data)
A Hype Cycle is a graphic representation of the maturity, adoption and business application of specific technologies . The term was coined by Gartner  , an analyst/research house, based in the United States , that provides opinions, advice and data on the global information technology industry.
&quot;...a five stage vision where virtualization is used first for test and development, then for server consolidation, then for infrastructure on demand, then for data center automation and finally for cloud computing.&quot;
Mainly highlighting the efforts that are underway to open cloud computing. Standards are key for interoperability and adoption.
Cloud Computing - Benefits and Challenges
Cloud Computing A New Computing Paradigm Viswanathan K
Cloud Computing SaaS Cloud Computing PaaS IaaS Amazon Web Services Hadoop AaaS Web Services Distributed Computing DFS Y!OS Google App Engine Windows Azure Silverlight
Some Disclaimers… <ul><li>I am not a CME! (Cloud Matter Expert) </li></ul>
Some Disclaimers… <ul><li>The following sites have been very useful… </li></ul>
What is Cloud Computing? <ul><li>Some Definitions… </li></ul><ul><ul><li>Forrester has defined cloud computing as “A form of standardized IT-based capability — such as Internet based services, software, or IT infrastructure — offered by a service provider that is accessible via Internet protocols from any computer, is always available and scales automatically to adjust to demand, is either pay-per-use or advertising-based, has Web- or programmatic-based control interfaces, and enables full customer self-service.” </li></ul></ul><ul><ul><li>EUCALYPTUS project : Elastic Computing, Utility Computing, and Cloud Computing are (possibly synonymous) terms referring to a popular SLA-based computing paradigm that allows users to "rent" Internet-accessible computing capacity on a for-fee basis. </li></ul></ul>
Cloud Computing <ul><li>A New Paradigm </li></ul><ul><ul><li>Building new generation internet applications in an efficient and optimal manner </li></ul></ul><ul><ul><li>SLA based (Reliability, Scalability and Availability) model </li></ul></ul><ul><ul><li>Optimizing Capital expenditure (pay-as-you-grow) </li></ul></ul><ul><ul><li>Scaling on demand with quality and integrity </li></ul></ul><ul><ul><li>Storing and managing large scale data </li></ul></ul>
Cloud Layers… CLOUD Infrastructure (IaaS) Public Cloud Private Cloud Network, Systems and Storage Platform (PaaS) Cloud Frameworks & Tools Services Cloud Services Application (SaaS) Cloud Applications
Cloud: Infrastructure (IaaS) <ul><li>Consists of the core computing resources of systems, storage and network </li></ul><ul><li>Infrastructure management is highly abstracted from the consumers </li></ul><ul><li>Capacity is elastic and made available on demand </li></ul><ul><li>Consumers pays infrastructure costs as variable operational expense (opex) </li></ul>
Cloud: Platforms (PaaS) <ul><li>Software (tools and services) for administration, deployment and hosting of business applications </li></ul><ul><li>Framework for managing distributed file systems </li></ul><ul><li>Framework and tools for distributed data processing on large clusters </li></ul><ul><li>Tools for managing Grids and Virtual appliances </li></ul>
Cloud: Services <ul><li>Services that are used by cloud applications. </li></ul><ul><ul><li>Storage </li></ul></ul><ul><ul><li>Integration </li></ul></ul><ul><ul><li>Billing </li></ul></ul><ul><ul><li>Security </li></ul></ul><ul><ul><li>Data </li></ul></ul>
Cloud: Applications (A/SaaS) <ul><li>There are a number of them today… </li></ul>
Cloud Layers filled out… CLOUD Infrastructure (IaaS) Public Cloud: Amazon EC2, Eucalyptus Private Cloud: Enterprise DC Platform (PaaS) Open Cloud: Aptana Custom Cloud: Google App Eng Framework: Hadoop Tools: Gemstone Services Storage: Amazon S3, Simple DB, Google Big Table Integration: Amazon SQS Security: OpenID, OAuth Application (SaaS) Salesforce.com, MagCloud, Snappy Fingers, Oracle OnDemand
Developing in the clouds <ul><li>Assume a plan is to create a website that has the following high level requirements amongst others </li></ul><ul><ul><li>Crawl and extract web content </li></ul></ul><ul><ul><li>Integrate content from other sources (listings, feeds, photos, maps etc..) </li></ul></ul><ul><ul><li>Need to store, analyze and manage large data sets </li></ul></ul><ul><ul><li>Non-functional requirements (Performance, Scalability, Availability etc..) </li></ul></ul><ul><ul><li>Core Business Logic </li></ul></ul>
An approach Cloud Service API’s Database Private Cloud Search Web as a DB Storage Storage Cloud Application Compute Grid Batch Processing Cloud Platform: Framework & Tools Queries to crawl & extract Elaborate extracted content Generate complex analysis Scalability Availability Provisioning Fault Tolerance Web Map
Cloud Benefits <ul><li>Productivity & Time to Market </li></ul><ul><ul><li>Building an internet scale application can be accomplished with speed and efficiency </li></ul></ul><ul><ul><li>Developers focus more on building functionality and business logic and not worry about non-functional requirements of scalability, availability etc. </li></ul></ul><ul><ul><li>Web and Platform services manages the heavy lifting required for scalability and fault tolerance </li></ul></ul><ul><ul><li>Managing hardware environments and software deployment is abstracted </li></ul></ul>
Cloud Challenges/Opportunities <ul><li>Trust, Privacy and Availability </li></ul><ul><li>Performance, Ownership & Cost Savings </li></ul><ul><li>Creation and adoption of new software development processes and training </li></ul><ul><li>New ways to manage Capacity (CapEx & OpEx) </li></ul><ul><li>Lack of standards for interoperability (supplier lock-in) </li></ul><ul><li>Security, Legal and Regulatory issues </li></ul>
Cloud events & conferences in India <ul><li>IEEE 2009 International Conference on Cloud Computing (CLOUD-II 2009), September 21-25, 2009, Bangalore, India </li></ul><ul><ul><li>http://www.thecloudcomputing.org/2009/2/ </li></ul></ul><ul><li>The Institution of Engineering and Technology (23-25 June 2009, Bangalore, India) </li></ul><ul><ul><li>http://conferences.theiet.org/cloud-computing/ </li></ul></ul><ul><li>IDC Mumbai - Friday, 24th April 2009 </li></ul><ul><li>Cloudcamp Bangalore, March 2009 </li></ul>
Cloud resources and useful links <ul><li>The Open Cloud Consortium (OCC) </li></ul><ul><ul><li>http://www.opencloudconsortium.org/index.html </li></ul></ul><ul><li>Open Cloud Manifesto </li></ul><ul><ul><li>http://www.opencloudmanifesto.org/ </li></ul></ul><ul><li>Distributed Management Task Force, Inc. </li></ul><ul><ul><li>http://www.dmtf.org/home </li></ul></ul><ul><li>EUCALYPTUS (Open-Source website) </li></ul><ul><ul><li>http://open.eucalyptus.com/ </li></ul></ul>
All questions and answers will we posted on thoughtworker.com/masterclass To know more about our events and up coming Webinars please visit www.thoughtworker.com/events To get in touch with this team mail us on [email_address]