Bostonrb Amazon Talk
Upcoming SlideShare
Loading in...5
×
 

Bostonrb Amazon Talk

on

  • 1,942 views

A talk covering some of the amazon AWS products for a Tuesday boston.rb meetup.

A talk covering some of the amazon AWS products for a Tuesday boston.rb meetup.

Statistics

Views

Total Views
1,942
Views on SlideShare
1,932
Embed Views
10

Actions

Likes
0
Downloads
25
Comments
0

3 Embeds 10

http://www.slideshare.net 5
http://www.linkedin.com 4
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Bostonrb Amazon Talk Bostonrb Amazon Talk Presentation Transcript

  • Cloud Computing featuring Amazon Web Services an unofficial presentation
  • I am... Brian Kaney from Vermonster . We are a 3 person shop in the Back Bay. We've been around for over 8 years and mostly do ruby, perl and python stuff. I do not work for Amazon. I don't know every Amazon offering – maybe you can help ?
  • Warning! There are lots of acronyms and abbreviations. I'll be bouncing from CLI, Web management console(s) and ruby code (and slides) This has been (somewhat) rehearsed but things happen... but at least you didn't have to pay and there's beer afterward.
  • Agenda TODO: Cram all this into 60 mins.
    • Intro
    • Elastic Fundamentals
    • LoadBalance/AutoScale
    • Map Reduce
    • Monitoring
    • Other Services
    • Deploy / SCM ( System Configuration Management )
    • Architecture Thoughts
    • Closing
  • Agenda TODO: Cram all this into 60 mins.
    • Intro
    • Elastic Fundamentals
    • LoadBalance/AutoScale
    • Map Reduce
    • Monitoring
    • Other Services
    • Deploy / SCM ( System Configuration Management )
    • Architecture Thoughts
    • Closing
  • Intro: Why Cloud?
  • Intro: Why Cloud?
  • Intro: Why Cloud?
  • Intro: Why Cloud?
  • Intro: Why Cloud?
  • Intro: Why Cloud?
  • Intro: Why Cloud?
  • Intro: Why Cloud? Temporary Events Unpredictability! Scalability Low Barrier to Entry Redundancy Batch Process
  • Intro: Uncontroversial Prediction Cloud computing is no longer vaporware and will continue growth .
  • Intro: Who Clouds?
    • Web Applications
    • Financial (MS Windows)
    • Biotech / Modeling / Scientific Community
    • Testing (devver, sauce labs)
    • Who else here clouds?
  • Intro: Where can I get one?
    • Engine Yard (Amazon)
    • Rightscale (Amazon and others)
    • Rackspace
    • GoGrid
    • Google
    • Amazon
    • Eucalyptus (DIY)
  • Intro: Where can I get one?
    • Engine Yard (Amazon)
    • Rightscale (Amazon and others)
    • Rackspace
    • GoGrid
    • Google
    • Amazon
    • Eucalyptus (DIY)
  • Intro: Amazon AWS Currently...
    • The most mature
    • The most fully featured
    • The most mature API
    • And the only one I've really used
  • Intro: How much does it Cost?
    • Minimum hour increments
    • Min $0.10/hr ($70/month/instance)
    • Max $0.80/hr ($560/month/instance)
    • c1.medium $150-$200/month/instance
    • Reservations 1/3 the hourly price ($200 - $2000 per year)
    • NOTE: See aws.amazon.com for official pricing
  • Agenda
    • Intro
    • Elastic Fundamentals
    • LoadBalance/AutoScale
    • Map Reduce
    • Monitoring
    • Other Services
    • Deploy / SCM ( System Configuration Management )
    • Architecture Thoughts
    • Closing
  • Fundamentals: Amazon API
    • It is fairly consistent – REST or WSDL/SOAP
    • Security
      • x509 key pairs for API Access
      • Separate keypairs installed on the instance
    • Public API
    • Instance API via link-local address
  • Fundamentals: Ruby and Amazon
    • aws-s3
    • EC2
    • RightScale
    • Tons of others (http://github.com/search?type=Repositories&language=rb&q=amazon)
  • Fundamentals: Ruby and Amazon
    • aws-s3
    • EC2
    • RightScale
    • Tons of others (http://github.com/search?type=Repositories&language=rb&q=amazon)
  • Fundamentals: EC2
    • Elastic Compute Cloud
    • Virtualized Xen Servers
    • 6 sizes – c1.medium best bang for buck
    • Security Groups
    • Availability Zones ( different for everyone )
    • Launch from AMIs ( Amazon Machine Images )
  • Fundamentals: Launch an Instance!
  • Fundamentals: AMIs
    • Typical to find an existing, customize and then repackage
    • Ubuntu.com ( http://help.ubuntu.com/community/EC2StartersGuide )
    • Well-maintained Ubuntu ones at ( http://alestic.com )
    • Consider re-bundle as part of deployment
  • Fundamentals: Let's bundle !
  • Fundamentals: S3
    • Key-store
    • Buckets and blobs
    • AMIs live there
    • DNS mapping is cool
    • Pretty cheap and widely used – early offering
  • Fundamentals: EBS
    • Elastic Block Storage
    • Network-attached drives
    • I use XFS
    • Snapshotting
    • Shared NEW
  • Fundamentals: Attach an EBS
  • Fundamentals: SDB
    • Simple DB
    • Written in Erlang (rumors?)
    • “Offline” M/DB open-source clone (http://www.mgateway.com/mdb.html)
    • I haven't used SDB... anyone using it?
  • Fundamentals: SQS
    • Simple Queue Service
    • Not AMPQ
    • v2 has message locking ( recommended )
    • Order is not FIFO – but you can provide sequence metadata
    • Up to 8k in size
  • Fundamentals: Create SQS Messages
  • Agenda
    • Intro
    • Elastic Fundamentals
    • LoadBalance/AutoScale (ELB/AS)
    • Map Reduce
    • Monitoring
    • Other Services
    • Deploy / SCM ( System Configuration Management )
    • Architecture Thoughts
    • Closing
  • ELB/AS: A Proposal I have a random idea: Let's launch an automatically-scalable , load-balanced cloud running a trivial sinatra application!
  • ELB/AS: Loadbalance NEW
    • Start with an Elastic LoadBalancer
    • Register Instances
    • Balanced across all availability zones
    • Create a real CNAME in real life
  • ELB/AS: AutoScaling NEW
    • Create Groups of Instances
    • Launch Configuration for each Group
    • One or many Triggers create Scaling Activities ( add or remove instances )
      • Period – how often to measure
      • Breach Duration – how long the breach is affected
  • ELB/AS: AutoScaling Concepts Trigger 1 Trigger 2 Trigger 3 Launch Config
  • ELB/AS: Launch Time!
  • Agenda
    • Intro
    • Elastic Fundamentals
    • LoadBalance/AutoScale
    • Map Reduce
    • Monitoring
    • Other Services
    • Deploy / SCM ( System Configuration Management )
    • Architecture Thoughts
    • Closing
  • MapReduce
    • Design pattern, coined by Google (?)
    • Functional Programming
    • For big data sets
    • Highly Parallelizable
    • Apache Hadoop
    • Uses S3 (HDFS?)
  • MapReduce: Theory
    • First, map chunks up the data for processing
    • Then reduce combines based on keys
    • Remember math class:
    map(k, v) -> <k' v'> reduce(k', <v'>) -> <k',v'>
    • Distributed key-stores … nosql.
  • MapReduce: EMR Implementation
    • STDIN and STDOUT are the interfaces
    • Key - value separated by “: ”
    • Output of map and reduce should be similar.
    • JAR, Streaming , Pig or Hive
  • MapReduce: EMR Proposal I have another completely random idea: Let's process a data.gov feed of earthquake data to see the most seismically volatile regions using Elastic Map Reduce !
  • MapReduce: While we are Waiting...
    • Minimum of 30 seconds to start a job.
    • I don't have time for that
    • Let's look at some monitoring in the meantime.
  • Agenda
    • Intro
    • Elastic Fundamentals
    • LoadBalance/AutoScale
    • Map Reduce
    • Monitoring
    • Other Services
    • Deploy / SCM ( System Configuration Management )
    • Architecture Thoughts
    • Closing
  • Monitoring NEW
    • Cloud Watch
    • API provides aggregation ( sweet! )
    • Amazon has a nice UI
    • But account management is crappy
    • We can write some code for that!
  • Monitoring: Code
      Start with a CLI:
    ec2-monitor-instances <instance-id>
    • RightScale gem and a rake task
    • Then use some HTML5 canvas action
    Thanks to a jQuery visualize plugin by Filament!
  • Our Map-Reduce Should be Done!
  • Agenda
    • Intro
    • Elastic Fundamentals
    • LoadBalance/AutoScale
    • Monitoring
    • Map Reduce
    • Other Services
    • Deploy / SCM ( System Configuration Management )
    • Architecture Thoughts
    • Closing
  • Other Services
    • Cloud Front NEWISH
    • Mechanical Turk
    • VPN NEW
    • Payment System (DevPay, Flexible Payments)
    • Load up a truck and physically ship your data to Amazon
    • Anyone using these?
  • Agenda
    • Intro
    • Elastic Fundamentals
    • LoadBalance/AutoScale
    • Map Reduce
    • Monitoring
    • Other Services
    • Deploy / SCM ( System Configuration Management )
    • Architecture Thoughts
    • Closing
  • Deployment (opinionated)
    • Use a global registry – S3
    • Capistrano
    • EC2-on-Rails (shouldn't it be rails-on-ec2?)
    • Pool-Party
    • Chef
    • Puppet
    • Rubber
    • (many others)
  • Deployment v2 (also opinionated)
    • That's fine for a handful of instances
    • But direct ssh does not scale (try deploying to 100, 500 or 1,000 instances)
    • Vertebra (dead? uses XMPP)
    • Build one on a messaging queue? Use Nanite?
  • Agenda
    • Intro
    • Elastic Fundamentals
    • LoadBalance/AutoScale
    • Map Reduce
    • Monitoring
    • Other Services
    • Deploy / SCM ( System Configuration Management )
    • Architecture Thoughts
    • Closing
  • Architecture Redux
    • Stateless – like the web and also like our ruby “best-practices” development.
    • Ephemeral Storage – /mnt
    • Messaging instead of forking/backgrounding (AMPQ)
    • Event Machine
    • Nanite (http://rubymanor.org/videos/nanite/)
  • Architecture Redux
    • We need some state-full pieces
    • (DB, Proxy, configuration/settings)
    • But we can if we drink the kool-aid!
    SQS S3 SDB LoadBalancer EBS
  • Architecture Redux
    • Vendor Lock-in
    • Predetermined technology ( SBD.. I'd rather have Postgres )
    • But we are agile, so less of an issue.
    • I can't back this up statistically, but my guess is most are using a hybrid architecture.
  • What about Exiting Rails Applications?
    • We are fairly stateless to begin with
    • Configuration Management
      • Inventory gems and plugins
      • External libraries
    • Shared Storage
      • Shared NEW EBS / S3
    • Messaging ( SQS / AMPQ, opposed to forking/backgrounding )
    • Log files ( Shared EBS helps )
  • Consider a Third Party
    • Simplify Architecture, Configuration Management and Deployment
    • Engine Yard
    • Right Scale
    • (others - lmgtfy)
    • Your Friendly Neighborhood Consultants
  • Thanks!
    • Brian Kaney – twitter.com/ bkaney
    • IRC – bkaney
    • Vermonster – http://www. vermonster.com