Cloud Computing featuring Amazon Web Services an unofficial presentation
I am... Brian Kaney  from  Vermonster . We are a 3 person shop in the Back Bay. We've been around for over 8 years and mos...
Warning! There are lots of acronyms and abbreviations. I'll be bouncing from CLI, Web management console(s) and ruby code ...
Agenda TODO: Cram all this into 60 mins. <ul><li>Intro
Elastic Fundamentals
LoadBalance/AutoScale
Map Reduce
Monitoring
Other Services
Deploy / SCM ( System Configuration Management )
Architecture Thoughts
Closing </li></ul>
Agenda TODO: Cram all this into 60 mins. <ul><li>Intro
Elastic Fundamentals
LoadBalance/AutoScale
Map Reduce
Monitoring
Other Services
Deploy / SCM ( System Configuration Management )
Architecture Thoughts
Closing </li></ul>
Intro: Why Cloud?
Intro: Why Cloud?
Intro: Why Cloud?
Intro: Why Cloud?
Intro: Why Cloud?
Intro: Why Cloud?
Intro: Why Cloud?
Intro: Why Cloud? Temporary Events Unpredictability! Scalability Low Barrier to Entry Redundancy Batch Process
Intro: Uncontroversial Prediction Cloud computing is no longer vaporware and will continue growth .
Intro: Who Clouds? <ul><li>Web Applications
Financial (MS Windows)
Biotech / Modeling / Scientific Community
Testing (devver, sauce labs)
Who else here clouds? </li></ul>
Intro: Where can I get one? <ul><li>Engine Yard (Amazon)
Rightscale (Amazon and others)
Rackspace
GoGrid
Google
Amazon
Eucalyptus (DIY) </li></ul>
Intro: Where can I get one? <ul><li>Engine Yard (Amazon)
Rightscale (Amazon and others)
Rackspace
GoGrid
Google
Amazon
Eucalyptus (DIY) </li></ul>
Intro: Amazon AWS Currently... <ul><li>The most mature
The most fully featured
The most mature API
And the only one I've really used </li></ul>
Intro: How much does it Cost? <ul><li>Minimum hour increments
Min $0.10/hr  ($70/month/instance)
Max $0.80/hr  ($560/month/instance)
c1.medium $150-$200/month/instance
Reservations 1/3 the hourly price ($200 - $2000 per year)
NOTE: See aws.amazon.com for official pricing </li></ul>
Agenda <ul><li>Intro
Elastic Fundamentals
Upcoming SlideShare
Loading in...5
×

Bostonrb Amazon Talk

1,136

Published on

A talk covering some of the amazon AWS products for a Tuesday boston.rb meetup.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,136
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
26
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Bostonrb Amazon Talk

  1. 1. Cloud Computing featuring Amazon Web Services an unofficial presentation
  2. 2. I am... Brian Kaney from Vermonster . We are a 3 person shop in the Back Bay. We've been around for over 8 years and mostly do ruby, perl and python stuff. I do not work for Amazon. I don't know every Amazon offering – maybe you can help ?
  3. 3. Warning! There are lots of acronyms and abbreviations. I'll be bouncing from CLI, Web management console(s) and ruby code (and slides) This has been (somewhat) rehearsed but things happen... but at least you didn't have to pay and there's beer afterward.
  4. 4. Agenda TODO: Cram all this into 60 mins. <ul><li>Intro
  5. 5. Elastic Fundamentals
  6. 6. LoadBalance/AutoScale
  7. 7. Map Reduce
  8. 8. Monitoring
  9. 9. Other Services
  10. 10. Deploy / SCM ( System Configuration Management )
  11. 11. Architecture Thoughts
  12. 12. Closing </li></ul>
  13. 13. Agenda TODO: Cram all this into 60 mins. <ul><li>Intro
  14. 14. Elastic Fundamentals
  15. 15. LoadBalance/AutoScale
  16. 16. Map Reduce
  17. 17. Monitoring
  18. 18. Other Services
  19. 19. Deploy / SCM ( System Configuration Management )
  20. 20. Architecture Thoughts
  21. 21. Closing </li></ul>
  22. 22. Intro: Why Cloud?
  23. 23. Intro: Why Cloud?
  24. 24. Intro: Why Cloud?
  25. 25. Intro: Why Cloud?
  26. 26. Intro: Why Cloud?
  27. 27. Intro: Why Cloud?
  28. 28. Intro: Why Cloud?
  29. 29. Intro: Why Cloud? Temporary Events Unpredictability! Scalability Low Barrier to Entry Redundancy Batch Process
  30. 30. Intro: Uncontroversial Prediction Cloud computing is no longer vaporware and will continue growth .
  31. 31. Intro: Who Clouds? <ul><li>Web Applications
  32. 32. Financial (MS Windows)
  33. 33. Biotech / Modeling / Scientific Community
  34. 34. Testing (devver, sauce labs)
  35. 35. Who else here clouds? </li></ul>
  36. 36. Intro: Where can I get one? <ul><li>Engine Yard (Amazon)
  37. 37. Rightscale (Amazon and others)
  38. 38. Rackspace
  39. 39. GoGrid
  40. 40. Google
  41. 41. Amazon
  42. 42. Eucalyptus (DIY) </li></ul>
  43. 43. Intro: Where can I get one? <ul><li>Engine Yard (Amazon)
  44. 44. Rightscale (Amazon and others)
  45. 45. Rackspace
  46. 46. GoGrid
  47. 47. Google
  48. 48. Amazon
  49. 49. Eucalyptus (DIY) </li></ul>
  50. 50. Intro: Amazon AWS Currently... <ul><li>The most mature
  51. 51. The most fully featured
  52. 52. The most mature API
  53. 53. And the only one I've really used </li></ul>
  54. 54. Intro: How much does it Cost? <ul><li>Minimum hour increments
  55. 55. Min $0.10/hr ($70/month/instance)
  56. 56. Max $0.80/hr ($560/month/instance)
  57. 57. c1.medium $150-$200/month/instance
  58. 58. Reservations 1/3 the hourly price ($200 - $2000 per year)
  59. 59. NOTE: See aws.amazon.com for official pricing </li></ul>
  60. 60. Agenda <ul><li>Intro
  61. 61. Elastic Fundamentals
  62. 62. LoadBalance/AutoScale
  63. 63. Map Reduce
  64. 64. Monitoring
  65. 65. Other Services
  66. 66. Deploy / SCM ( System Configuration Management )
  67. 67. Architecture Thoughts
  68. 68. Closing </li></ul>
  69. 69. Fundamentals: Amazon API <ul><li>It is fairly consistent – REST or WSDL/SOAP
  70. 70. Security </li><ul><li>x509 key pairs for API Access
  71. 71. Separate keypairs installed on the instance </li></ul><li>Public API
  72. 72. Instance API via link-local address </li></ul>
  73. 73. Fundamentals: Ruby and Amazon <ul><li>aws-s3
  74. 74. EC2
  75. 75. RightScale
  76. 76. Tons of others (http://github.com/search?type=Repositories&language=rb&q=amazon) </li></ul>
  77. 77. Fundamentals: Ruby and Amazon <ul><li>aws-s3
  78. 78. EC2
  79. 79. RightScale
  80. 80. Tons of others (http://github.com/search?type=Repositories&language=rb&q=amazon) </li></ul>
  81. 81. Fundamentals: EC2 <ul><li>Elastic Compute Cloud
  82. 82. Virtualized Xen Servers
  83. 83. 6 sizes – c1.medium best bang for buck
  84. 84. Security Groups
  85. 85. Availability Zones ( different for everyone )
  86. 86. Launch from AMIs ( Amazon Machine Images ) </li></ul>
  87. 87. Fundamentals: Launch an Instance!
  88. 88. Fundamentals: AMIs <ul><li>Typical to find an existing, customize and then repackage
  89. 89. Ubuntu.com ( http://help.ubuntu.com/community/EC2StartersGuide )
  90. 90. Well-maintained Ubuntu ones at ( http://alestic.com )
  91. 91. Consider re-bundle as part of deployment </li></ul>
  92. 92. Fundamentals: Let's bundle !
  93. 93. Fundamentals: S3 <ul><li>Key-store
  94. 94. Buckets and blobs
  95. 95. AMIs live there
  96. 96. DNS mapping is cool
  97. 97. Pretty cheap and widely used – early offering </li></ul>
  98. 98. Fundamentals: EBS <ul><li>Elastic Block Storage
  99. 99. Network-attached drives
  100. 100. I use XFS
  101. 101. Snapshotting
  102. 102. Shared NEW </li></ul>
  103. 103. Fundamentals: Attach an EBS
  104. 104. Fundamentals: SDB <ul><li>Simple DB
  105. 105. Written in Erlang (rumors?)
  106. 106. “Offline” M/DB open-source clone (http://www.mgateway.com/mdb.html)
  107. 107. I haven't used SDB... anyone using it? </li></ul>
  108. 108. Fundamentals: SQS <ul><li>Simple Queue Service
  109. 109. Not AMPQ
  110. 110. v2 has message locking ( recommended )
  111. 111. Order is not FIFO – but you can provide sequence metadata
  112. 112. Up to 8k in size </li></ul>
  113. 113. Fundamentals: Create SQS Messages
  114. 114. Agenda <ul><li>Intro
  115. 115. Elastic Fundamentals
  116. 116. LoadBalance/AutoScale (ELB/AS)
  117. 117. Map Reduce
  118. 118. Monitoring
  119. 119. Other Services
  120. 120. Deploy / SCM ( System Configuration Management )
  121. 121. Architecture Thoughts
  122. 122. Closing </li></ul>
  123. 123. ELB/AS: A Proposal I have a random idea: Let's launch an automatically-scalable , load-balanced cloud running a trivial sinatra application!
  124. 124. ELB/AS: Loadbalance NEW <ul><li>Start with an Elastic LoadBalancer
  125. 125. Register Instances
  126. 126. Balanced across all availability zones
  127. 127. Create a real CNAME in real life </li></ul>
  128. 128. ELB/AS: AutoScaling NEW <ul><li>Create Groups of Instances
  129. 129. Launch Configuration for each Group
  130. 130. One or many Triggers create Scaling Activities ( add or remove instances ) </li><ul><li>Period – how often to measure
  131. 131. Breach Duration – how long the breach is affected </li></ul></ul>
  132. 132. ELB/AS: AutoScaling Concepts Trigger 1 Trigger 2 Trigger 3 Launch Config
  133. 133. ELB/AS: Launch Time!
  134. 134. Agenda <ul><li>Intro
  135. 135. Elastic Fundamentals
  136. 136. LoadBalance/AutoScale
  137. 137. Map Reduce
  138. 138. Monitoring
  139. 139. Other Services
  140. 140. Deploy / SCM ( System Configuration Management )
  141. 141. Architecture Thoughts
  142. 142. Closing </li></ul>
  143. 143. MapReduce <ul><li>Design pattern, coined by Google (?)
  144. 144. Functional Programming
  145. 145. For big data sets
  146. 146. Highly Parallelizable
  147. 147. Apache Hadoop
  148. 148. Uses S3 (HDFS?) </li></ul>
  149. 149. MapReduce: Theory <ul><li>First, map chunks up the data for processing
  150. 150. Then reduce combines based on keys
  151. 151. Remember math class: </li></ul>map(k, v) -> <k' v'> reduce(k', <v'>) -> <k',v'> <ul><li>Distributed key-stores … nosql. </li></ul>
  152. 152. MapReduce: EMR Implementation <ul><li>STDIN and STDOUT are the interfaces
  153. 153. Key - value separated by “: ”
  154. 154. Output of map and reduce should be similar.
  155. 155. JAR, Streaming , Pig or Hive </li></ul>
  156. 156. MapReduce: EMR Proposal I have another completely random idea: Let's process a data.gov feed of earthquake data to see the most seismically volatile regions using Elastic Map Reduce !
  157. 157. MapReduce: While we are Waiting... <ul><li>Minimum of 30 seconds to start a job.
  158. 158. I don't have time for that
  159. 159. Let's look at some monitoring in the meantime. </li></ul>
  160. 160. Agenda <ul><li>Intro
  161. 161. Elastic Fundamentals
  162. 162. LoadBalance/AutoScale
  163. 163. Map Reduce
  164. 164. Monitoring
  165. 165. Other Services
  166. 166. Deploy / SCM ( System Configuration Management )
  167. 167. Architecture Thoughts
  168. 168. Closing </li></ul>
  169. 169. Monitoring NEW <ul><li>Cloud Watch
  170. 170. API provides aggregation ( sweet! )
  171. 171. Amazon has a nice UI
  172. 172. But account management is crappy
  173. 173. We can write some code for that! </li></ul>
  174. 174. Monitoring: Code <ul>Start with a CLI: </ul>ec2-monitor-instances <instance-id> <ul><li>RightScale gem and a rake task
  175. 175. Then use some HTML5 canvas action </li></ul>Thanks to a jQuery visualize plugin by Filament!
  176. 176. Our Map-Reduce Should be Done!
  177. 177. Agenda <ul><li>Intro
  178. 178. Elastic Fundamentals
  179. 179. LoadBalance/AutoScale
  180. 180. Monitoring
  181. 181. Map Reduce
  182. 182. Other Services
  183. 183. Deploy / SCM ( System Configuration Management )
  184. 184. Architecture Thoughts
  185. 185. Closing </li></ul>
  186. 186. Other Services <ul><li>Cloud Front NEWISH
  187. 187. Mechanical Turk
  188. 188. VPN NEW
  189. 189. Payment System (DevPay, Flexible Payments)
  190. 190. Load up a truck and physically ship your data to Amazon
  191. 191. Anyone using these? </li></ul>
  192. 192. Agenda <ul><li>Intro
  193. 193. Elastic Fundamentals
  194. 194. LoadBalance/AutoScale
  195. 195. Map Reduce
  196. 196. Monitoring
  197. 197. Other Services
  198. 198. Deploy / SCM ( System Configuration Management )
  199. 199. Architecture Thoughts
  200. 200. Closing </li></ul>
  201. 201. Deployment (opinionated) <ul><li>Use a global registry – S3
  202. 202. Capistrano
  203. 203. EC2-on-Rails (shouldn't it be rails-on-ec2?)
  204. 204. Pool-Party
  205. 205. Chef
  206. 206. Puppet
  207. 207. Rubber
  208. 208. (many others) </li></ul>
  209. 209. Deployment v2 (also opinionated) <ul><li>That's fine for a handful of instances
  210. 210. But direct ssh does not scale (try deploying to 100, 500 or 1,000 instances)
  211. 211. Vertebra (dead? uses XMPP)
  212. 212. Build one on a messaging queue? Use Nanite? </li></ul>
  213. 213. Agenda <ul><li>Intro
  214. 214. Elastic Fundamentals
  215. 215. LoadBalance/AutoScale
  216. 216. Map Reduce
  217. 217. Monitoring
  218. 218. Other Services
  219. 219. Deploy / SCM ( System Configuration Management )
  220. 220. Architecture Thoughts
  221. 221. Closing </li></ul>
  222. 222. Architecture Redux <ul><li>Stateless – like the web and also like our ruby “best-practices” development.
  223. 223. Ephemeral Storage – /mnt
  224. 224. Messaging instead of forking/backgrounding (AMPQ)
  225. 225. Event Machine
  226. 226. Nanite (http://rubymanor.org/videos/nanite/) </li></ul>
  227. 227. Architecture Redux <ul><li>We need some state-full pieces
  228. 228. (DB, Proxy, configuration/settings)
  229. 229. But we can if we drink the kool-aid! </li></ul>SQS S3 SDB LoadBalancer EBS
  230. 230. Architecture Redux <ul><li>Vendor Lock-in
  231. 231. Predetermined technology ( SBD.. I'd rather have Postgres )
  232. 232. But we are agile, so less of an issue.
  233. 233. I can't back this up statistically, but my guess is most are using a hybrid architecture. </li></ul>
  234. 234. What about Exiting Rails Applications? <ul><li>We are fairly stateless to begin with
  235. 235. Configuration Management </li><ul><li>Inventory gems and plugins
  236. 236. External libraries </li></ul><li>Shared Storage </li><ul><li>Shared NEW EBS / S3 </li></ul><li>Messaging ( SQS / AMPQ, opposed to forking/backgrounding )
  237. 237. Log files ( Shared EBS helps ) </li></ul>
  238. 238. Consider a Third Party <ul><li>Simplify Architecture, Configuration Management and Deployment
  239. 239. Engine Yard
  240. 240. Right Scale
  241. 241. (others - lmgtfy)
  242. 242. Your Friendly Neighborhood Consultants </li></ul>
  243. 243. Thanks! <ul><li>Brian Kaney – twitter.com/ bkaney
  244. 244. IRC – bkaney
  245. 245. Vermonster – http://www. vermonster.com </li></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×