Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cloud Computing With AWS


Published on

Published in: Technology
  • Be the first to comment

Cloud Computing With AWS

  1. 1. Cloud Computing With Amazon Web Service Munish K Gupta
  2. 2. Cloud Computing <ul><li>What </li></ul>June 8, 2009 © 2008 Wipro Ltd - Confidential
  3. 3. What is Cloud Computing ? <ul><li>Cloud computing refers to the use of Internet (&quot;cloud&quot;) based computer technology for a variety of services. It is a style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet . Users need not have knowledge of, expertise in, or control over the technology infrastructure &quot;in the cloud&quot; that supports them - Wikipedia </li></ul>CLOUD
  4. 4. Cloud Computing Comparisons <ul><li>Cloud computing is often confused with grid computing, utility computing and autonomic computing. </li></ul>
  5. 5. What is Grid Computing ? <ul><li>Multiple independent computing clusters which act like a “grid” because they are composed of resource nodes not located within a single administrative domain </li></ul><ul><li>Grid computing depends on software to divide and apportion pieces of a program among several computers, sometimes up to many thousands. </li></ul><ul><li>Grid computing can also be thought of as distributed and large-scale cluster computing, as well as a form of network-distributed parallel processing. </li></ul><ul><li>* Source - Wikipedia </li></ul>
  6. 6. What is Utility Computing ? <ul><li>Utility computing is the packaging of computing resources, such as computation and storage, as a metered service similar to a traditional public utility (such as electricity, water, natural gas, or telephone network). </li></ul><ul><li>“ Utility computing” usually envisions some form of virtualization so that the amount of storage or computing power available is considerably larger than that of a single time-sharing computer. Multiple servers are used on the “back end” to make this possible. These might be a dedicated computer cluster specifically built for the purpose of being rented out, or even an under-utilized supercomputer. The technique of running a single calculation on multiple computers is known as distributed computing. </li></ul><ul><li>* Source - Wikipedia </li></ul>
  7. 7. What is autonomic computing ? <ul><li>Autonomic Computing is an initiative started by IBM in 2001. Its ultimate aim is to develop computer systems capable of self-management, to overcome the rapidly growing complexity of computing systems management, and to reduce the barrier that complexity poses to further growth. </li></ul><ul><li>An autonomic system makes decisions on its own, using high-level policies; it will constantly check and optimize its status and automatically adapt itself to changing conditions. </li></ul><ul><li>* Source - Wikipedia </li></ul>
  8. 8. Cloud Computing as of 2009 <ul><li>The majority of cloud computing infrastructure consists of reliable services delivered through data centers and built on servers with different levels of virtualization technologies. The services are accessible anywhere in the world, with The Cloud appearing as a single point of access for all the computing needs of consumers. </li></ul><ul><li>Many cloud computing deployments depend on grids, have autonomic characteristics and bill like utilities but cloud computing can be seen as a natural next step from the grid-utility model </li></ul><ul><li>* Source - Wikipedia </li></ul>
  9. 9. Cloud Computing <ul><li>Why </li></ul>June 8, 2009 © 2009 Wipro Ltd - Confidential
  10. 10. Web Application Pains <ul><li>70% of Web development work is all about </li></ul><ul><ul><li>Data Centers </li></ul></ul><ul><ul><li>Bandwidth/Power/Cooling </li></ul></ul><ul><ul><li>Operations </li></ul></ul><ul><ul><li>Staffing </li></ul></ul><ul><li>Application Scaling is difficult and expensive </li></ul><ul><ul><li>Large up front investment </li></ul></ul><ul><ul><li>Invest ahead of time </li></ul></ul><ul><ul><li>Load is unpredictable (seasonal spikes) </li></ul></ul>
  11. 11. Why Cloud Computing ? <ul><li>Almost zero upfront infrastructure investment </li></ul><ul><li>Just-in-time Infrastructure </li></ul><ul><li>More efficient resource utilization </li></ul><ul><li>Usage-based costing </li></ul><ul><li>Potential for shrinking the processing time </li></ul><ul><li>Device and location independence </li></ul><ul><li>ilities (Reliability, Scalability, Sustainability, Maintainability) </li></ul>
  12. 12. Cloud Computing <ul><li>Where </li></ul>June 8, 2009 © 2009 Wipro Ltd - Confidential
  13. 13. Where to use Cloud Computing ? <ul><li>Examples of applications that could utilize the power of Cloud </li></ul><ul><li>Processing Pipelines </li></ul><ul><ul><li>Document processing pipelines – convert documents from Microsoft Word to PDF, OCR </li></ul></ul><ul><ul><li>Image processing pipelines - create thumbnails of an image, resize images </li></ul></ul><ul><ul><li>Video transcoding pipelines - transcode AVI to </li></ul></ul><ul><ul><li>MPEG movies </li></ul></ul><ul><ul><li>Indexing – create an index of web crawl data </li></ul></ul><ul><ul><li>Data mining – perform search over millions of records </li></ul></ul><ul><li>Batch Processing Systems </li></ul><ul><ul><li>Back-office applications (in financial, insurance or retail sectors) </li></ul></ul><ul><ul><li>Log analysis – analyze and generate daily/weekly reports </li></ul></ul><ul><ul><li>Nightly builds – perform nightly automated builds of source code repository every night in parallel </li></ul></ul><ul><ul><li>Automated Unit Testing and Deployment Testing – Test and deploy and perform automated unit testing (functional, load, quality) on different deployment configurations every night </li></ul></ul><ul><li>Websites </li></ul><ul><ul><li>Websites that “sleep” at night and auto-scale during the day </li></ul></ul><ul><ul><li>Instant Websites – websites for conferences or events (Super Bowl, sports tournaments) </li></ul></ul><ul><ul><li>Promotion websites </li></ul></ul><ul><ul><li>“ Seasonal Websites” - websites that only run during the tax season or the holiday season(“Black Friday” or Christmas) </li></ul></ul>
  14. 14. Cloud Computing <ul><li>Amazon Web Service (AWS) > Infrastructure Services </li></ul>June 8, 2009 © 2009 Wipro Ltd - Confidential
  15. 15. Amazon Web Service (AWS) <ul><li>What is Amazon Web Service </li></ul><ul><ul><li>A set of APIs and business models which give developer-level access to Amazon’s infrastructure and content </li></ul></ul>
  16. 16. AWS Products <ul><li>AWS provides the following </li></ul><ul><li>We will look at the Infrastructure Service as these form the core of the Cloud Computing infrastructure provided by AWS </li></ul><ul><li>Payments and Billing Services </li></ul><ul><ul><li>Amazon Flexible Payments Service </li></ul></ul><ul><ul><li>Amazon DevPay </li></ul></ul><ul><li>Alexa Web Services </li></ul><ul><ul><li>Alexa Web Information Service </li></ul></ul><ul><ul><li>Alexa Top Sites </li></ul></ul><ul><ul><li>Alexa Site Thumbnail </li></ul></ul><ul><li>Infrastructure Services </li></ul><ul><ul><li>Amazon Elastic Compute Cloud </li></ul></ul><ul><ul><li>Amazon Simple DB </li></ul></ul><ul><ul><li>Amazon Simple Storage Service </li></ul></ul><ul><ul><li>Amazon CloudFront </li></ul></ul><ul><ul><li>Amazon Simple Queue Service </li></ul></ul><ul><li>On Demand Workforce </li></ul><ul><ul><li>Amazon Mechanical Turk </li></ul></ul><ul><li>Fulfillment & Associates </li></ul><ul><ul><li>Amazon Fulfillment Service </li></ul></ul><ul><ul><li>Amazon Associate Web Service </li></ul></ul>
  17. 17. AWS - Infrastructure Services <ul><li>Amazon Elastic Compute Cloud (Amazon EC2) – A web service that provides resizable compute capacity in the cloud. One can configure an Amazon Machine Instance (AMI) and load it into the Amazon EC2 service. Allows to quickly scale capacity, both up and down, as your computing requirements change. </li></ul><ul><li>Amazon SimpleDB – A web service for running queries on structured data in real time. This service works in close conjunction with Amazon S3 and Amazon EC2, collectively providing the ability to store, process and query data sets in the cloud </li></ul><ul><li>Amazon Simple Storage Service (Amazon S3) – A simple web services interface that can be used to store and retrieve large amounts of data, at any time, from anywhere on the web. </li></ul><ul><li>Amazon CloudFront – A web service for content delivery. It integrates with other Amazon Web Services to provide an easy way to distribute content to end users with low latency and high data transfer speeds. </li></ul><ul><li>Amazon Simple Queue Service (Amazon SQS) – A reliable, highly scalable, hosted queue for storing messages as they travel between computers. By using Amazon SQS, developers can simply move data between distributed components of their applications that perform different tasks, without losing messages or requiring each component to be always available. </li></ul>
  18. 18. Amazon Elastic Compute Cloud (Amazon EC2) <ul><li>What > Concepts>Setup >API </li></ul>June 8, 2009 © 2009 Wipro Ltd - Confidential
  19. 19. Amazon Elastic Compute Cloud (Amazon EC2) <ul><li>Amazon EC2 is a web service that enables you to launch and manage server instances in Amazon's data centers using APIs or available tools and utilities </li></ul><ul><li>Instances are available in different sizes and configurations </li></ul><ul><li>For example, one can use an m1.small instance (one Amazon EC2 Compute Unit) as a web server, an m1.xlarge instance (eight Amazon EC2 Compute Units) as a database server, or an extra large High-CPU instance (twenty Amazon EC2 Compute Units) for processor intensive applications </li></ul><ul><li>All the instances can be managed using the Web Service APIs </li></ul><ul><li>This is akin to virtualization concept (albeit as a Web Service) </li></ul>
  20. 20. Concepts Amazon EC2 <ul><li>Amazon Machine Image (AMI): </li></ul><ul><ul><li>Bootable root disk </li></ul></ul><ul><ul><li>Pre-defined or user-built </li></ul></ul><ul><ul><li>Catalog of user-built AMIs </li></ul></ul><ul><ul><li>OS: Fedora, Centos, Gentoo, Debian, Ubuntu, Windows Server </li></ul></ul><ul><ul><li>App Stack: LAMP, mpiBLAST, Hadoop </li></ul></ul><ul><li>Instance: </li></ul><ul><ul><li>Running copy of an AMI </li></ul></ul><ul><ul><li>Launch in less than 2 minutes </li></ul></ul><ul><ul><li>Start/stop programmatically </li></ul></ul><ul><li>Network Security Model: </li></ul><ul><ul><li>Explicit access control </li></ul></ul><ul><ul><li>Security groups </li></ul></ul><ul><li>Inter-service bandwidth is free </li></ul>
  21. 21. Setting up Amazon EC2 <ul><li>One can create an AMI (Amazon Machine Images) from scratch or use an existing AMI </li></ul><ul><li>Next, you bundle the AMI and obtain an AMI ID so one can launch as many instances of the AMI  </li></ul><ul><li>Launch one or more instances of your AMI </li></ul><ul><li>One can administer and use your instances and you would with any servers but making use of Web Services </li></ul>
  22. 22. Amazon EC2 SOAP API <ul><li>AllocateAddress </li></ul><ul><li>AssociateAddress </li></ul><ul><li>AttachVolume </li></ul><ul><li>AuthorizeSecurityGroupIngress </li></ul><ul><li>BundleInstance </li></ul><ul><li>CancelBundleTask </li></ul><ul><li>ConfirmProductInstance </li></ul><ul><li>CreateKeyPair </li></ul><ul><li>CreateSecurityGroup </li></ul><ul><li>CreateSnapshot </li></ul><ul><li>CreateVolume </li></ul><ul><li>DeleteKeyPair </li></ul><ul><li>DeleteSecurityGroup </li></ul><ul><li>DeleteSnapshot </li></ul><ul><li>DeleteVolume </li></ul><ul><li>DeregisterImage </li></ul><ul><li>DescribeAddresses </li></ul><ul><li>DescribeAvailabilityZones </li></ul><ul><li>DescribeBundleTasks </li></ul><ul><li>DescribeImageAttribute </li></ul><ul><li>DescribeImages </li></ul><ul><li>DescribeInstances </li></ul><ul><li>DescribeKeyPairs </li></ul><ul><li>DescribeRegions </li></ul><ul><li>DescribeSecurityGroups </li></ul><ul><li>DescribeSnapshots </li></ul><ul><li>DescribeVolumes </li></ul><ul><li>DetachVolume </li></ul><ul><li>DisassociateAddress </li></ul><ul><li>GetConsoleOutput </li></ul><ul><li>ModifyImageAttribute </li></ul><ul><li>RebootInstances </li></ul><ul><li>RegisterImage </li></ul><ul><li>ReleaseAddress </li></ul><ul><li>ResetImageAttribute </li></ul><ul><li>RevokeSecurityGroupIngress </li></ul><ul><li>RunInstances </li></ul><ul><li>TerminateInstances </li></ul>The Amazon EC2 API consists of web service operations for every task the service can perform
  23. 23. Amazon EC2 At Work <ul><li>Startups </li></ul><ul><ul><li>Cruxy – Media transcoding </li></ul></ul><ul><ul><li>GigaVox Media – Podcast Management </li></ul></ul><ul><li>Fortune 500 clients: </li></ul><ul><ul><li>High-Impact, S hort-Term Projects </li></ul></ul><ul><ul><li>Development Host </li></ul></ul><ul><li>Science / Research: </li></ul><ul><ul><li>Hadoop / MapReduce </li></ul></ul><ul><ul><li>mpiBLAST </li></ul></ul><ul><li>Load-Management and Load Balancing Tools: </li></ul><ul><ul><li>Pound </li></ul></ul><ul><ul><li>Weogeo </li></ul></ul><ul><ul><li>Rightscale </li></ul></ul>
  24. 24. Amazon SimpleDB <ul><li>What > Setup >API </li></ul>June 8, 2009 © 2009 Wipro Ltd - Confidential
  25. 25. Amazon SimpleDB <ul><li>Amazon SimpleDB is a web service for running queries on structured data in real time </li></ul><ul><li>Provides the core functionality of a database - real-time lookup and simple querying of structured data - without the operational complexity </li></ul><ul><li>Amazon SimpleDB requires no schema, automatically indexes your data and provides a simple API for storage and access. This eliminates the administrative burden of data modeling, index maintenance, and performance tuning </li></ul>
  26. 26. Setting up Amazon SimpleDB <ul><li>Using Amazon SimpleDB, one can organize the structured data in domains within which one can put data, get data, or run queries </li></ul><ul><li>Domains consist of items which are described by  attribute  name-value pairs </li></ul><ul><li>Regardless of how the data is stored, Amazon SimpleDB automatically indexes your data for quick and accurate retrieval </li></ul>
  27. 27. Amazon SimpleDB API <ul><li>Amazon SimpleDB service consists of a small group of API calls that provide the core functionality one needs to build the application </li></ul><ul><li>Create Domain— Create domains to contain your data; you can create up to 100 domains </li></ul><ul><li>Delete Domain— Delete any of your domains </li></ul><ul><li>List Domains— List all domains within your account </li></ul><ul><li>Put— Add, modify, or remove data within your Amazon SimpleDB domains </li></ul><ul><li>Delete— Remove items, attributes, or attribute values from your domain </li></ul><ul><li>Get— Retrieve the attributes and values of any item ID that you specify </li></ul><ul><li>Select— Query the specified domain using a SQL SELECT expression </li></ul><ul><li>Query— Query the specified domain using an expression that performs value tests on one or more attributes </li></ul><ul><li>QueryWithAttributes— Query the specified domain and retrieve the results for matching items </li></ul>
  28. 28. Amazon Simple Storage Service (Amazon S3) <ul><li>What > Concepts >API </li></ul>June 8, 2009 © 2009 Wipro Ltd - Confidential
  29. 29. Amazon Simple Storage Service <ul><li>Amazon S3 has a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web </li></ul><ul><li>Amazon S3 provides a minimal feature set that focus on simplicity and robustness. Following are some of features of the Amazon S3 service: </li></ul><ul><ul><li>Enables you to write, read, and delete objects from 1 byte to 5 gigabytes in size with accompanying metadata. There is no fixed limit on the number of objects you can store. </li></ul></ul><ul><ul><li>Provides a straightforward flat object store model, where each object is stored and retrieved using a unique developer-assigned key. </li></ul></ul><ul><ul><li>Provides authentication mechanisms to ensure that data is kept secure from unauthorized access. Objects can be made private or public and rights can be granted to specific users. </li></ul></ul><ul><ul><li>Uses standards-based REST and SOAP interfaces designed to work with any Internet-development toolkit. </li></ul></ul>
  30. 30. Concepts Amazon S3 <ul><li>Objects: </li></ul><ul><ul><li>Opaque data to be stored (1 byte … 5 Gigabytes) </li></ul></ul><ul><ul><li>Authentication and access controls </li></ul></ul><ul><li>Buckets: </li></ul><ul><ul><li>Object container – any number of objects </li></ul></ul><ul><ul><li>100 buckets per account / buckets are “owned” </li></ul></ul><ul><li>Keys: </li></ul><ul><ul><li>Unique object identifier within bucket </li></ul></ul><ul><ul><li>Up to 1024 bytes long </li></ul></ul><ul><ul><li>Flat object storage model </li></ul></ul><ul><li>Standards-Based Interfaces: </li></ul><ul><ul><li>REST and SOAP </li></ul></ul><ul><ul><li>URL-Addressability – every object has a URL </li></ul></ul>
  31. 31. Amazon S3 SOAP API <ul><li>Service: </li></ul><ul><ul><li>ListAllMyBuckets </li></ul></ul><ul><li>Buckets: </li></ul><ul><ul><li>CreateBucket </li></ul></ul><ul><ul><li>DeleteBucket </li></ul></ul><ul><ul><li>ListBucket </li></ul></ul><ul><ul><li>GetBucketAccessControlPolicy </li></ul></ul><ul><ul><li>SetBucketAccessControlPolicy </li></ul></ul><ul><ul><li>GetBucketLoggingStatus </li></ul></ul><ul><ul><li>SetBucketLoggingStatus </li></ul></ul><ul><li>Objects: </li></ul><ul><ul><li>PutObject </li></ul></ul><ul><ul><li>PutObjectInline </li></ul></ul><ul><ul><li>GetObject </li></ul></ul><ul><ul><li>GetObjectExtended </li></ul></ul><ul><ul><li>DeleteObject </li></ul></ul><ul><ul><li>GetObjectAccessControlPolicy </li></ul></ul><ul><ul><li>SetObjectAccessControlPolicy </li></ul></ul>
  32. 32. Amazon CloudFront <ul><li>What > Setup >API </li></ul>June 8, 2009 © 2009 Wipro Ltd - Confidential
  33. 33. Amazon CloudFront <ul><li>Amazon CloudFront is a web service for content delivery </li></ul><ul><li>CloudFront delivers your content through a worldwide network of edge locations. End users are routed to the nearest  edge location , so content is delivered with the best possible performance. </li></ul>
  34. 34. Setting up Amazon CloudFront <ul><li>Objects - Objects are the files you want CloudFront to deliver. This typically includes web pages, images, and digital media files </li></ul><ul><li>Origin Server - An origin server is the location where you store the original, definitive version of your objects. One can place any objects that are required to be delivered through CloudFront in the bucket. </li></ul><ul><li>Distributions - After the objects are stored in the origin server, there is a link create between an Amazon S3 bucket (the origin server) and a domain name (which CloudFront automatically assigns) </li></ul><ul><li>Edge Locations - An edge location is a geographical site where CloudFront caches copies of your objects. </li></ul>
  35. 35. Amazon CloudFront API <ul><li>You create the Amazon S3 bucket that will be your origin server </li></ul><ul><li>You place your objects in the bucket and make them publicly readable </li></ul><ul><li>You create your CloudFront distribution and get the domain name that CloudFront assigns. </li></ul><ul><ul><li>Example distribution ID: PDFDVBD632BHDS5 </li></ul></ul><ul><ul><li>Example domain name: </li></ul></ul><ul><ul><li>The distribution ID will not necessarily match the domain name. </li></ul></ul><ul><li>You create the URLs that your end users will use to get your objects and include them as needed in your web application or web site. </li></ul><ul><li>Example URL: </li></ul>
  36. 36. Amazon Simple Queue Service <ul><li>What >Concept> Setup >API </li></ul>June 8, 2009 © 2009 Wipro Ltd - Confidential
  37. 37. Amazon Simple Queue Service (SQS) <ul><li>A queue is a temporary repository for messages that are awaiting processing </li></ul><ul><li>Amazon SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component </li></ul><ul><li>The queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing. This means the queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer are only intermittently connected to the network </li></ul><ul><li>SQS ensures delivery of each message at least once, and supports multiple readers and writers interacting with the same queue. </li></ul><ul><li>A single queue can be used simultaneously by many distributed application components, with no need for those components to coordinate with each other to share the queue </li></ul>
  38. 38. Concept Amazon SQS <ul><li>Redundant infrastructure— Guarantees delivery of the messages at least once, highly concurrent access to messages, and high availability for sending and retrieving messages </li></ul><ul><li>Multiple writers and readers— Multiple parts of the system can send or receive messages at the same time. SQS locks the message during processing, keeping other parts of the system from processing the message simultaneously. </li></ul><ul><li>Configurable settings per queue— All of the queues don't have to be exactly alike </li></ul><ul><li>For example, one queue can be optimized for messages that require a longer processing time than others. </li></ul><ul><li>Variable message size— the messages can be up to 8 KB in size </li></ul><ul><li>Unlimited queues and messages— One can have as many queues and messages in the Amazon SQS system as you want </li></ul>
  39. 39. Setting up Amazon SQS <ul><li>There are three main actors in the overall system: </li></ul><ul><ul><li>The components of your distributed system </li></ul></ul><ul><ul><li>Queues </li></ul></ul><ul><ul><li>Messages in the queues </li></ul></ul><ul><li>In the following diagram, your system has several components that send messages to the queue and receive messages from the queue. The diagram shows that a single queue, which has its messages (labeled A-E), is redundantly saved across multiple SQS servers. </li></ul>
  40. 40. Amazon SQS SOAP API <ul><li>Queues: </li></ul><ul><ul><li>ListQueues </li></ul></ul><ul><ul><li>DeleteQueue </li></ul></ul><ul><ul><li>SetVisibilityTimeout </li></ul></ul><ul><ul><li>GetVisibilityTimeout </li></ul></ul><ul><li>Messages: </li></ul><ul><ul><li>SendMessage </li></ul></ul><ul><ul><li>ReceiveMessage </li></ul></ul><ul><ul><li>DeleteMessage </li></ul></ul><ul><ul><li>PeekMessage </li></ul></ul><ul><li>Security: </li></ul><ul><ul><li>AddGrant </li></ul></ul><ul><ul><li>ListGrants </li></ul></ul><ul><ul><li>RemoveGrant </li></ul></ul>
  41. 41. Case Study GIGAVOX
  42. 42. GIGAVOX <ul><li>Implemented Amazon S3, Amazon EC2 and Amazon SQS in November 2006 </li></ul><ul><li>Created an infinitely scalable infrastructure for less than $100 - building the same infrastructure themselves would have cost thousands of dollars </li></ul><ul><li>Reduced staffing requirements - far less responsibility for 24x7 operations </li></ul>
  43. 43. Munish K Gupta Thank You Senior Consultant [email_address] June 8, 2009 © 2008 Wipro Ltd - Confidential