Cloud Computing in Practice


King Chung Huang
Technical Solutions Analyst, University of Calgary
“Using services in the cloud,
 develop a large-scale
 video conversion system.”
What is Cloud Computing?
“Cloud computing is simply a buzzword used to repackage grid
computing and utility computing, both of which have existed f...
“The interesting thing about cloud computing is that we’ve
redefined cloud computing to include everything that we
already...
“The interesting thing about cloud computing is that we’ve
redefined cloud computing to include everything that we
already...
Getting Started
Infrastructure Services   Payments & Billing          Web Search & Information

Elastic Compute Cloud     Flexible Payment...
Infrastructure Services   Payments & Billing          Web Search & Information

Elastic Compute Cloud     Flexible Payment...
Infrastructure Services   Payments & Billing          Web Search & Information

Elastic Compute Cloud     Flexible Payment...
Amazon Mechanical Turk
• Marketplace for work that requires human intelligence
   ■ Enables programmatic distribution of t...
Ten Thousand Cents
• Drawing collected from November 2007 to March 2008
• Total labour paid: US$100
   ■ 1¢ per part × 10 ...
Video Conversion Plan
• Upload source content to Simple Storage Service (S3)
• Instantiate virtual machines on Elastic Com...
Accessing Infrastructure Services
• Web service APIs
   ■ SOAP and REST

• Many applications/plugins support S3
   ■ Trans...
Pricing
• Pay only for what you use
   ■ Data transfer

   ■ Data storage

   ■ Instance time

   ■ Transactions
Pricing
Simple Storage Service (United States)


                     Storage    15¢ per GB-month

             Data Trans...
Pricing
Simple Storage Service (Europe)


                    Storage    18¢ per GB-month

            Data Transfer In   ...
Pricing
Simple Queue Service


                    Storage    none

            Data Transfer In   10¢ per GB

          D...
Pricing
Simple Queue Service (Old API, prior to 2008)


                     Storage    none

             Data Transfer I...
Pricing
Simple Queue Service (Old API, prior to 2008)


Impact of the Price Change
We examined the effect that the new pri...
Pricing
Elastic Compute Service


                  Instances    10¢–80¢ per hour

            Data Transfer In   10¢ per ...
Pricing
Freebies

• Data transfer between EC2 instances are free
   ■ Instances must be in the same availability zone

   ...
Pricing
Elastic IP Addresses (EC2)

• IPv4 Addresses are scarce
• Instances are dynamically assigned a public IP address
 ...
Nuances
Simple Storage Service (S3)
• Storage is organized in buckets
    ■ Like a namespace for the objects it contains

    ■ Ac...
Elastic Compute Cloud (EC2)
• Machine images are templates
   ■ Run multiple instances from the same image

   ■ Appliance...
Beyond Raw Utilities
• Additional facilities around traditional infrastructure elements
  increases their usability
   ■ F...
Speed Bumps
Reality Bites
Simple Storage Service

• S3 objects have a maximum size of 5 GB
   ■ Split objects like machine images

• C...
Reality Bites
Elastic Compute Cloud

• Making a change means recreating an entire image
• Default storage is transient
   ...
All Together, Now!
Cumulus
• Completely implemented in JavaScript *
   ■ Calls Amazon services directly via REST APIs

• Contacts S3 to retri...
Demo
Project Timeline


                August 26
31
7
14
21
28          September 29
Project Timeline
                            8/26: project kickoff
                                       8/29: project pl...
Project Timeline

                            9/2: work begins
                August 26
31
7
14
21
28          September ...
Project Timeline


                August 26
31
7
14
21
28          September 29


                                   9/5:...
Project Timeline


                   August 26
31
7
14
21
28             September 29



         9/11: attempting video ...
Project Timeline
                 9/18: bandwidth increased
                                   9/18: first video converted...
Project Timeline
      9/22: creating front-end web app

                      August 26
31
7
14
21
28                Sept...
Project Timeline


                August 26
31
7
14
21
28          September 29




                   9/29: still workin...
Project Timeline


                August 26
31
7
14
21
28          September 29
By the Numbers
              19      working days
               2      weekends
          70 GB       transferred
       ...
Lessons Learned
Cloud Computing is more than
Raw Utilities
Cloud Computing
Enables Tinkering
Bandwidth Can Be Problematic
Centralization Results in
Centralization of Risk
Be Aware of
Service Level Agreements
More Information

King Chung Huang
Technical Solutions Analyst, University of Calgary
kchuang@ucalgary.ca

Patrick Mann
Ch...
Q&A
Cloud Computing in Practice
Cloud Computing in Practice
Cloud Computing in Practice
Cloud Computing in Practice
Cloud Computing in Practice
Cloud Computing in Practice
Upcoming SlideShare
Loading in...5
×

Cloud Computing in Practice

25,140

Published on

Experiences with Amazon Web Services and thoughts on Cloud Computing presented at Cybera Summit 08 in Banff, Canada.

Published in: Technology, Business
2 Comments
30 Likes
Statistics
Notes
No Downloads
Views
Total Views
25,140
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
2,168
Comments
2
Likes
30
Embeds 0
No embeds

No notes for slide

Cloud Computing in Practice

  1. 1. Cloud Computing in Practice King Chung Huang Technical Solutions Analyst, University of Calgary
  2. 2. “Using services in the cloud, develop a large-scale video conversion system.”
  3. 3. What is Cloud Computing?
  4. 4. “Cloud computing is simply a buzzword used to repackage grid computing and utility computing, both of which have existed for decades.” whatis.com definition of Cloud Computing
  5. 5. “The interesting thing about cloud computing is that we’ve redefined cloud computing to include everything that we already do. […] The computer industry is the only industry that is more fashion-driven than women’s fashion. Maybe I’m an idiot, but I have no idea what anyone is talking about. What is it? It’s complete gibberish. It’s insane. When is this idiocy going to stop? Larry Ellison during Oracle’s Analyst Day
  6. 6. “The interesting thing about cloud computing is that we’ve redefined cloud computing to include everything that we already do. […] The computer industry is the only industry that is more fashion-driven than women’s fashion. Maybe I’m an idiot, but I have no idea what anyone is talking about. What is it? It’s complete gibberish. It’s insane. When is this idiocy going to stop? Larry Ellison during Oracle’s Analyst Day
  7. 7. Getting Started
  8. 8. Infrastructure Services Payments & Billing Web Search & Information Elastic Compute Cloud Flexible Payments Service Web Search Simple Storage Service DevPay Web Information Service Simple Queue Service Top Sites SimpleDB Fulfillment & Associates Site Thumbnail Fulfillment Web Service Associates Web Service On-Demand Workforce Mechanical Turk
  9. 9. Infrastructure Services Payments & Billing Web Search & Information Elastic Compute Cloud Flexible Payments Service Web Search Simple Storage Service DevPay Web Information Service Simple Queue Service Top Sites SimpleDB Fulfillment & Associates Site Thumbnail Fulfillment Web Service Associates Web Service On-Demand Workforce Mechanical Turk
  10. 10. Infrastructure Services Payments & Billing Web Search & Information Elastic Compute Cloud Flexible Payments Service Web Search Simple Storage Service DevPay Web Information Service Simple Queue Service Top Sites SimpleDB Fulfillment & Associates Site Thumbnail Fulfillment Web Service Associates Web Service On-Demand Workforce Mechanical Turk
  11. 11. Amazon Mechanical Turk • Marketplace for work that requires human intelligence ■ Enables programmatic distribution of tasks that require human intelligence • Human Intelligence Tasks (HITs) ■ Desired output ■ Format of output ■ Pay rate • Only pay for quality work ■ Review results and pay for accepted work
  12. 12. Ten Thousand Cents • Drawing collected from November 2007 to March 2008 • Total labour paid: US$100 ■ 1¢ per part × 10 000 parts • Prints available for purchase ■ $100 each
  13. 13. Video Conversion Plan • Upload source content to Simple Storage Service (S3) • Instantiate virtual machines on Elastic Compute Cloud (EC2) to perform conversions ■ Be able to instantiate more conversion machines as needed to scale to demand ■ Use Simple Queue Service (SQS) to queue requests • Log status and results in SimpleDB (SDB) ■ Was not able to gain access during beta period • Deliver conversion results from S3
  14. 14. Accessing Infrastructure Services • Web service APIs ■ SOAP and REST • Many applications/plugins support S3 ■ Transmit on Mac OS X ■ S3Fox for Firefox • AWS Developer Community provides lots of documentation and sample code ■ JavaScript, PHP, Python • Many public machine images for EC2 are available ■ Use them as a starting point for custom machines • In actuality, you rarely have to start from scratch
  15. 15. Pricing • Pay only for what you use ■ Data transfer ■ Data storage ■ Instance time ■ Transactions
  16. 16. Pricing Simple Storage Service (United States) Storage 15¢ per GB-month Data Transfer In 10¢ per GB Data Transfer Out 10¢–17¢ per GB PUT, POST, or LIST 1.0¢ per 1000 requests GET 1.0¢ per 10000 requests
  17. 17. Pricing Simple Storage Service (Europe) Storage 18¢ per GB-month Data Transfer In 10¢ per GB Data Transfer Out 10¢–17¢ per GB PUT, POST, or LIST 1.2¢ per 1000 requests GET 1.2¢ per 10000 requests
  18. 18. Pricing Simple Queue Service Storage none Data Transfer In 10¢ per GB Data Transfer Out 10¢–17¢ per GB Requests 1.0¢ per 10000 requests
  19. 19. Pricing Simple Queue Service (Old API, prior to 2008) Storage none Data Transfer In 10¢ per GB Data Transfer Out 10¢–17¢ per GB Requests 1.0¢ per 10000 requests Messages 10¢ per 1000 messages
  20. 20. Pricing Simple Queue Service (Old API, prior to 2008) Impact of the Price Change We examined the effect that the new pricing would have had on Amazon SQS charges billed at the end of December 2007. Under the new plan, 76% of customers with bills greater than $1 would have received lower bills, saving an average of 71% each compared to their actual bill. source: http://aws.amazon.com/sqs/
  21. 21. Pricing Elastic Compute Service Instances 10¢–80¢ per hour Data Transfer In 10¢ per GB Data Transfer Out 10¢–17¢ per GB
  22. 22. Pricing Freebies • Data transfer between EC2 instances are free ■ Instances must be in the same availability zone ■ Instances must address each other by private IP addresses • Data transfer between EC2 instances and S3 are free • Data transfer between EC2 instances and SQS are free
  23. 23. Pricing Elastic IP Addresses (EC2) • IPv4 Addresses are scarce • Instances are dynamically assigned a public IP address and a private IP address • IPv4 addresses can be statically allocated and used for instances ■ No cost while in use ■ 1¢ per non-attached Elastic IP address per hour
  24. 24. Nuances
  25. 25. Simple Storage Service (S3) • Storage is organized in buckets ■ Like a namespace for the objects it contains ■ Accessible via http://bucketname.s3.amazonaws.com • It’s not file storage, it’s a key-value store ■ Like a big hash table or dictionary ■ Key-value pairs ■ Movies/My Movie.mp4 ➜ ftypmp42isom5eiomdat9af… ■ UserHomePage ➜ http://www.ucalgary.ca/~kchuang/ • Implicit BitTorrent seeding for all keys
  26. 26. Elastic Compute Cloud (EC2) • Machine images are templates ■ Run multiple instances from the same image ■ Appliance model • Data can be passed to instances at startup ■ Provide input to an instance ■ Negates the need to communicate to an instance • Storage is independent of the machine ■ By default, instance storage is transient ■ Elastic Block Storage can be attached/detached as needed ■ Lifecycle is independent of instances
  27. 27. Beyond Raw Utilities • Additional facilities around traditional infrastructure elements increases their usability ■ Fits well with on-demand, only what you need usage model ■ Enables uses that were previously not possible
  28. 28. Speed Bumps
  29. 29. Reality Bites Simple Storage Service • S3 objects have a maximum size of 5 GB ■ Split objects like machine images • Changing one byte means reinserting the entire object • Renaming (re-keying) an object also means reinserting ■ New beta API allows for object moves within a bucket • Various bugs in third-party apps and S3 itself ■ Inserting objects between 2 to 4 GB can be difficult • Bandwidth can be a significant barrier ■ Project goal of inserting 1 TB would have taken 168 days
  30. 30. Reality Bites Elastic Compute Cloud • Making a change means recreating an entire image • Default storage is transient ■ Lasts only as long as the instance it is attached to ■ Elastic Block Storage can only be attached to a single instance at a time • Instances are billed by the hour ■ Constantly starting and stopping short-run instances can be costly
  31. 31. All Together, Now!
  32. 32. Cumulus • Completely implemented in JavaScript * ■ Calls Amazon services directly via REST APIs • Contacts S3 to retrieve stored videos • Converts video on demand ■ Writes jobs to an SQS queue ■ Invokes EC2 instances, which reads from job queue and performs conversion * time constraints necessitated the use of some existing PHP sample code
  33. 33. Demo
  34. 34. Project Timeline August 26 31 7 14 21 28 September 29
  35. 35. Project Timeline 8/26: project kickoff 8/29: project planning August 26 31 7 14 21 28 September 29
  36. 36. Project Timeline 9/2: work begins August 26 31 7 14 21 28 September 29
  37. 37. Project Timeline August 26 31 7 14 21 28 September 29 9/5: slow upload speeds 9/8: daapd server abandoned
  38. 38. Project Timeline August 26 31 7 14 21 28 September 29 9/11: attempting video conversion 9/12: visiting clouds
  39. 39. Project Timeline 9/18: bandwidth increased 9/18: first video converted August 26 31 7 14 21 28 September 29
  40. 40. Project Timeline 9/22: creating front-end web app August 26 31 7 14 21 28 September 29 9/26: starting presentation
  41. 41. Project Timeline August 26 31 7 14 21 28 September 29 9/29: still working on presentation
  42. 42. Project Timeline August 26 31 7 14 21 28 September 29
  43. 43. By the Numbers 19 working days 2 weekends 70 GB transferred 321 instance hours $41.60 spent 16324 miles as of September 27, 2008
  44. 44. Lessons Learned
  45. 45. Cloud Computing is more than Raw Utilities
  46. 46. Cloud Computing Enables Tinkering
  47. 47. Bandwidth Can Be Problematic
  48. 48. Centralization Results in Centralization of Risk
  49. 49. Be Aware of Service Level Agreements
  50. 50. More Information King Chung Huang Technical Solutions Analyst, University of Calgary kchuang@ucalgary.ca Patrick Mann Chief Technology Officer, Cybera patrick.mann@cybera.ca Cloud Computing White Paper coming soon http://www.cybera.ca
  51. 51. Q&A
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×