Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cloud Overview

789 views

Published on

A great overview of Cloud computing by Tom Creighton.

Published in: Technology
  • Login to see the comments

  • Be the first to like this

Cloud Overview

  1. 1. Cloud Computing An Introduction and Overview Tom Creighton CTO, Family Search tc@familysearch.org
  2. 2. Topic Areas •Cloud computing overview •Example large-scale cloud bursting application •Resource management in an enterprise cloud
  3. 3. Why Cloud Computing? •Cost Savings –Economies of scale –Use what’s needed; shut down what’s not •CapExto OpEx –Not necessarily an objective; –Projects move forward without large infrastructure investment •Scalability, Availability, Flexibility, Security •Opportunity benefit (maybe the biggest)
  4. 4. Cloud Computing Definitions •Essential Characteristics –On-demand self-service –Broad network access –Resource pooling –Rapid elasticity –Measured service •Service Models –Software as a Service –Platform as a Service –Infrastructure as a Service NIST defines five essential characteristics, three service models, and four deployment models.
  5. 5. Cloud Computing Definitions ACM CTO Roundtables What is Cloud Computing? Cloud computing is about moving services, computation and/or data—for cost and business advantage—off-site to an internal or external, location-transparent, centralized facility or contractor. By making data available in the cloud, it can be more easily and ubiquitously accessed, often at much lower cost, increasing its value by enabling opportunities for enhanced collaboration, integration, and analysis on a shared common platform. Three divisions (areas): •SaaS: WAN-enabled application services (eg. Google Apps, Salesforce.com, WebEx.) •PaaS: Foundational elements to develop new applications (eg. Coghead, Google Application Engine.) •Iaas: Providing computational and storage infrastructure in a centralized, location-transparent service (eg. Amazon.)
  6. 6. Cloud Computing Definitions •The term "Cloud Computing" refers to the on-demand delivery of IT resources via the Internet with pay-as- you-go pricing.
  7. 7. Cloud Computing Models •Deployment Models –Private cloud –Community cloud –Public cloud –Hybrid cloud NIST cloud computing reference architecture
  8. 8. Cloud Conceptual Framework
  9. 9. Cisco Domain Ten Framework
  10. 10. AWS Regions and Edge Locations
  11. 11. Regions and Availability Zones
  12. 12. Not All Regions Have Same Number of Zones
  13. 13. Cisco Domain Ten Framework
  14. 14. AWS Account Management
  15. 15. Cisco Domain Ten Framework
  16. 16. AWS Service Catalog
  17. 17. S3 Management Console
  18. 18. S3 Objects in a Bucket
  19. 19. An Example Cloud Use
  20. 20. CloudburstingExample •1 Trillion pairs to classify •500 TB of data •10 TB result set •CPU Bound –Single core: ~ 1000-1500 classify ops/s •11.5 K core-days
  21. 21. Approach •AWS c1.xlarge spot instances –8 core, 24GB RAM –$0.075 on AWS spot market –Best CPU/$ ratio •HP Blades in familysearch.org datacenter –8 core, 24 GB RAM –300 servers
  22. 22. Issues to Address •Machine Instability –Machine death from AWS spot instance volatility –Hardware failure (1000s of machines –certainty) •Coordination –Partitioning the work –Different server clusters •Different awsavailability zones •Fs.org datacenter –Restart failed processes
  23. 23. The Solution •Single Queue Multiple Reader Pattern –SQS with visibility timeouts to manage retries –Dead letter queue for messages failing > threshold •Claim Check Pattern –AWS S3 as data store •Homogeneous, Idempotent work units –15-20 minute target completion time –0 side effects –Work may get done multiple times
  24. 24. Claim Check -EIP
  25. 25. Results •Work Completed •64 hours •1000 machines avg.; ~2500 peak •$5000 –6X cost savings over previous Hadoop based solution •300 lines of Java/bash code •1 engineer / 1 week
  26. 26. Next Steps •Better metrics –Log aggregations via Splunk –Performance counters •Better deployment –CloudFormation •AWS Simple Workflow (SWF) for better server lifetime management •Investigate other OSS possibilities –Storm –Hadoop YARN –akka
  27. 27. Conclusion •Cloud technology offers new opportunities in scale, availability, security and management. •Properly using these tools is necessary to achieve business goals. •CapExto OpExis one reason to move to cloud computing. •A more important reason is the tremendous flexibility that the environment offers. •VPC provides the kinds of governance and security an enterprise needs to be comfortable with so much power under the hood. •Resource management is crucial in an enterprise environment.

×