Bostonrb Amazon Talk

Cloud Computing featuring Amazon Web Services an unofficial presentation

I am... Brian Kaney from Vermonster . We are a 3 person shop in the Back Bay. We've been around for over 8 years and mostly do ruby, perl and python stuff. I do not work for Amazon. I don't know every Amazon offering – maybe you can help ?

Warning! There are lots of acronyms and abbreviations. I'll be bouncing from CLI, Web management console(s) and ruby code (and slides) This has been (somewhat) rehearsed but things happen... but at least you didn't have to pay and there's beer afterward.

Agenda TODO: Cram all this into 60 mins.

Intro
Elastic Fundamentals
LoadBalance/AutoScale
Map Reduce
Monitoring
Other Services
Deploy / SCM ( System Configuration Management )
Architecture Thoughts
Closing

Agenda TODO: Cram all this into 60 mins.

Intro
Map Reduce
Monitoring
Other Services
Closing

Intro: Why Cloud?

Intro: Why Cloud? Temporary Events Unpredictability! Scalability Low Barrier to Entry Redundancy Batch Process

Intro: Uncontroversial Prediction Cloud computing is no longer vaporware and will continue growth .

Intro: Who Clouds?

Web Applications
Financial (MS Windows)
Biotech / Modeling / Scientific Community
Testing (devver, sauce labs)
Who else here clouds?

Intro: Where can I get one?

Engine Yard (Amazon)
Rightscale (Amazon and others)
Rackspace
GoGrid
Google
Amazon
Eucalyptus (DIY)

Intro: Where can I get one?

Engine Yard (Amazon)
Rightscale (Amazon and others)
Rackspace
GoGrid
Google
Amazon
Eucalyptus (DIY)

Intro: Amazon AWS Currently...

The most mature
The most fully featured
The most mature API
And the only one I've really used

Intro: How much does it Cost?

Minimum hour increments
Min $0.10/hr ($70/month/instance)
Max $0.80/hr ($560/month/instance)
c1.medium $150-$200/month/instance
Reservations 1/3 the hourly price ($200 - $2000 per year)
NOTE: See aws.amazon.com for official pricing

Agenda

Intro
Map Reduce
Monitoring
Other Services
Closing

Fundamentals: Amazon API

It is fairly consistent – REST or WSDL/SOAP
Security

x509 key pairs for API Access
Separate keypairs installed on the instance

Public API
Instance API via link-local address

Fundamentals: Ruby and Amazon

aws-s3
EC2
RightScale
Tons of others (http://github.com/search?type=Repositories&language=rb&q=amazon)

Fundamentals: Ruby and Amazon

aws-s3
EC2
RightScale
Tons of others (http://github.com/search?type=Repositories&language=rb&q=amazon)

Fundamentals: EC2

Elastic Compute Cloud
Virtualized Xen Servers
6 sizes – c1.medium best bang for buck
Security Groups
Availability Zones ( different for everyone )
Launch from AMIs ( Amazon Machine Images )

Fundamentals: Launch an Instance!

Fundamentals: AMIs

Typical to find an existing, customize and then repackage
Ubuntu.com ( http://help.ubuntu.com/community/EC2StartersGuide )
Well-maintained Ubuntu ones at ( http://alestic.com )
Consider re-bundle as part of deployment

Fundamentals: Let's bundle !

Fundamentals: S3

Key-store
Buckets and blobs
AMIs live there
DNS mapping is cool
Pretty cheap and widely used – early offering

Fundamentals: EBS

Elastic Block Storage
Network-attached drives
I use XFS
Snapshotting
Shared NEW

Fundamentals: Attach an EBS

Fundamentals: SDB

Simple DB
Written in Erlang (rumors?)
“Offline” M/DB open-source clone (http://www.mgateway.com/mdb.html)
I haven't used SDB... anyone using it?

Fundamentals: SQS

Simple Queue Service
Not AMPQ
v2 has message locking ( recommended )
Order is not FIFO – but you can provide sequence metadata
Up to 8k in size

Fundamentals: Create SQS Messages

Agenda

Intro
LoadBalance/AutoScale (ELB/AS)
Map Reduce
Monitoring
Other Services
Closing

ELB/AS: A Proposal I have a random idea: Let's launch an automatically-scalable , load-balanced cloud running a trivial sinatra application!

ELB/AS: Loadbalance NEW

Start with an Elastic LoadBalancer
Register Instances
Balanced across all availability zones
Create a real CNAME in real life

ELB/AS: AutoScaling NEW

Create Groups of Instances
Launch Configuration for each Group
One or many Triggers create Scaling Activities ( add or remove instances )

Period – how often to measure
Breach Duration – how long the breach is affected

ELB/AS: AutoScaling Concepts Trigger 1 Trigger 2 Trigger 3 Launch Config

ELB/AS: Launch Time!

Agenda

Intro
Map Reduce
Monitoring
Other Services
Closing

MapReduce

Design pattern, coined by Google (?)
Functional Programming
For big data sets
Highly Parallelizable
Apache Hadoop
Uses S3 (HDFS?)

MapReduce: Theory

First, map chunks up the data for processing
Then reduce combines based on keys
Remember math class:

map(k, v) -> <k' v'> reduce(k', <v'>) -> <k',v'>

Distributed key-stores … nosql.

MapReduce: EMR Implementation

STDIN and STDOUT are the interfaces
Key - value separated by “: ”
Output of map and reduce should be similar.
JAR, Streaming , Pig or Hive

MapReduce: EMR Proposal I have another completely random idea: Let's process a data.gov feed of earthquake data to see the most seismically volatile regions using Elastic Map Reduce !

MapReduce: While we are Waiting...

Minimum of 30 seconds to start a job.
I don't have time for that
Let's look at some monitoring in the meantime.

Agenda

Intro
Map Reduce
Monitoring
Other Services
Closing

Monitoring NEW

Cloud Watch
API provides aggregation ( sweet! )
Amazon has a nice UI
But account management is crappy
We can write some code for that!

Monitoring: Code

Start with a CLI: ec2-monitor-instances <instance-id>

RightScale gem and a rake task
Then use some HTML5 canvas action

Thanks to a jQuery visualize plugin by Filament!

Our Map-Reduce Should be Done!

Agenda

Intro
Monitoring
Map Reduce
Other Services
Closing

Other Services

Cloud Front NEWISH
Mechanical Turk
VPN NEW
Payment System (DevPay, Flexible Payments)
Load up a truck and physically ship your data to Amazon
Anyone using these?

Agenda

Intro
Map Reduce
Monitoring
Other Services
Closing

Deployment (opinionated)

Use a global registry – S3
Capistrano
EC2-on-Rails (shouldn't it be rails-on-ec2?)
Pool-Party
Chef
Puppet
Rubber
(many others)

Deployment v2 (also opinionated)

That's fine for a handful of instances
But direct ssh does not scale (try deploying to 100, 500 or 1,000 instances)
Vertebra (dead? uses XMPP)
Build one on a messaging queue? Use Nanite?

Agenda

Intro
Map Reduce
Monitoring
Other Services
Closing

Architecture Redux

Stateless – like the web and also like our ruby “best-practices” development.
Ephemeral Storage – /mnt
Messaging instead of forking/backgrounding (AMPQ)
Event Machine
Nanite (http://rubymanor.org/videos/nanite/)

Architecture Redux

We need some state-full pieces
(DB, Proxy, configuration/settings)
But we can if we drink the kool-aid!

SQS S3 SDB LoadBalancer EBS

Architecture Redux

Vendor Lock-in
Predetermined technology ( SBD.. I'd rather have Postgres )
But we are agile, so less of an issue.
I can't back this up statistically, but my guess is most are using a hybrid architecture.

What about Exiting Rails Applications?