Bostonrb Amazon Talk

Cloud Computing featuring Amazon Web Services an unofficial presentation

I am... Brian Kaney from Vermonster . We are a 3 person shop in the Back Bay. We've been around for over 8 years and mostly do ruby, perl and python stuff. I do not work for Amazon. I don't know every Amazon offering – maybe you can help ?

Warning! There are lots of acronyms and abbreviations. I'll be bouncing from CLI, Web management console(s) and ruby code (and slides) This has been (somewhat) rehearsed but things happen... but at least you didn't have to pay and there's beer afterward.

Agenda TODO: Cram all this into 60 mins.
Intro
Elastic Fundamentals
LoadBalance/AutoScale
Map Reduce
Monitoring
Other Services
Deploy / SCM ( System Configuration Management )
Architecture Thoughts
Closing

Intro: Why Cloud?

Intro: Why Cloud? Temporary Events Unpredictability! Scalability Low Barrier to Entry Redundancy Batch Process

Intro: Uncontroversial Prediction Cloud computing is no longer vaporware and will continue growth .

Intro: Who Clouds?
Web Applications
Financial (MS Windows)
Biotech / Modeling / Scientific Community
Testing (devver, sauce labs)
Who else here clouds?

Intro: Where can I get one?
Engine Yard (Amazon)
Rightscale (Amazon and others)
Rackspace
GoGrid
Google
Amazon
Eucalyptus (DIY)

Intro: Amazon AWS Currently...
The most mature
The most fully featured
The most mature API
And the only one I've really used

Intro: How much does it Cost?
Minimum hour increments
Min $0.10/hr ($70/month/instance)
Max $0.80/hr ($560/month/instance)
c1.medium $150-$200/month/instance
Reservations 1/3 the hourly price ($200 - $2000 per year)
NOTE: See aws.amazon.com for official pricing

Agenda
Intro
Map Reduce
Monitoring
Other Services
Closing

Fundamentals: Amazon API
It is fairly consistent – REST or WSDL/SOAP
Security
x509 key pairs for API Access
Separate keypairs installed on the instance
Public API
Instance API via link-local address

Fundamentals: Ruby and Amazon
aws-s3
EC2
RightScale
Tons of others (http://github.com/search?type=Repositories&language=rb&q=amazon)

Fundamentals: EC2
Elastic Compute Cloud
Virtualized Xen Servers
6 sizes – c1.medium best bang for buck
Security Groups
Availability Zones ( different for everyone )
Launch from AMIs ( Amazon Machine Images )

Fundamentals: Launch an Instance!

Fundamentals: AMIs
Typical to find an existing, customize and then repackage
Ubuntu.com ( http://help.ubuntu.com/community/EC2StartersGuide )
Well-maintained Ubuntu ones at ( http://alestic.com )
Consider re-bundle as part of deployment

Fundamentals: Let's bundle !

Fundamentals: S3
Key-store
Buckets and blobs
AMIs live there
DNS mapping is cool
Pretty cheap and widely used – early offering

Fundamentals: EBS
Elastic Block Storage
Network-attached drives
I use XFS
Snapshotting
Shared NEW

Fundamentals: Attach an EBS

Fundamentals: SDB
Simple DB
Written in Erlang (rumors?)
“Offline” M/DB open-source clone (http://www.mgateway.com/mdb.html)
I haven't used SDB... anyone using it?

Fundamentals: SQS
Simple Queue Service
Not AMPQ
v2 has message locking ( recommended )
Order is not FIFO – but you can provide sequence metadata
Up to 8k in size

Fundamentals: Create SQS Messages

Agenda
Intro
LoadBalance/AutoScale (ELB/AS)
Map Reduce
Monitoring
Other Services
Closing

ELB/AS: A Proposal I have a random idea: Let's launch an automatically-scalable , load-balanced cloud running a trivial sinatra application!

ELB/AS: Loadbalance NEW
Start with an Elastic LoadBalancer
Register Instances
Balanced across all availability zones
Create a real CNAME in real life

ELB/AS: AutoScaling NEW
Create Groups of Instances
Launch Configuration for each Group
One or many Triggers create Scaling Activities ( add or remove instances )
Period – how often to measure
Breach Duration – how long the breach is affected

ELB/AS: AutoScaling Concepts Trigger 1 Trigger 2 Trigger 3 Launch Config

ELB/AS: Launch Time!

Agenda
Intro
Map Reduce
Monitoring
Other Services
Closing

MapReduce
Design pattern, coined by Google (?)
Functional Programming
For big data sets
Highly Parallelizable
Apache Hadoop
Uses S3 (HDFS?)

MapReduce: Theory
First, map chunks up the data for processing
Then reduce combines based on keys
Remember math class:
map(k, v) -> <k' v'> reduce(k', <v'>) -> <k',v'>
Distributed key-stores … nosql.

MapReduce: EMR Implementation
STDIN and STDOUT are the interfaces
Key - value separated by “: ”
Output of map and reduce should be similar.
JAR, Streaming , Pig or Hive

MapReduce: EMR Proposal I have another completely random idea: Let's process a data.gov feed of earthquake data to see the most seismically volatile regions using Elastic Map Reduce !

MapReduce: While we are Waiting...
Minimum of 30 seconds to start a job.
I don't have time for that
Let's look at some monitoring in the meantime.

Agenda
Intro
Map Reduce
Monitoring
Other Services
Closing

Monitoring NEW
Cloud Watch
API provides aggregation ( sweet! )
Amazon has a nice UI
But account management is crappy
We can write some code for that!

Monitoring: Code
Start with a CLI:
ec2-monitor-instances <instance-id>
RightScale gem and a rake task
Then use some HTML5 canvas action
Thanks to a jQuery visualize plugin by Filament!

Our Map-Reduce Should be Done!

Agenda
Intro
Monitoring
Map Reduce
Other Services
Closing

Other Services
Cloud Front NEWISH
Mechanical Turk
VPN NEW
Payment System (DevPay, Flexible Payments)
Load up a truck and physically ship your data to Amazon
Anyone using these?

Agenda
Intro
Map Reduce
Monitoring
Other Services
Closing

Deployment (opinionated)
Use a global registry – S3
Capistrano
EC2-on-Rails (shouldn't it be rails-on-ec2?)
Pool-Party
Chef
Puppet
Rubber
(many others)

Deployment v2 (also opinionated)
That's fine for a handful of instances
But direct ssh does not scale (try deploying to 100, 500 or 1,000 instances)
Vertebra (dead? uses XMPP)
Build one on a messaging queue? Use Nanite?

Agenda
Intro
Map Reduce
Monitoring
Other Services
Closing

Architecture Redux
Stateless – like the web and also like our ruby “best-practices” development.
Ephemeral Storage – /mnt
Messaging instead of forking/backgrounding (AMPQ)
Event Machine
Nanite (http://rubymanor.org/videos/nanite/)

Architecture Redux
We need some state-full pieces
(DB, Proxy, configuration/settings)
But we can if we drink the kool-aid!
SQS S3 SDB LoadBalancer EBS

Architecture Redux
Vendor Lock-in
Predetermined technology ( SBD.. I'd rather have Postgres )
But we are agile, so less of an issue.
I can't back this up statistically, but my guess is most are using a hybrid architecture.

What about Exiting Rails Applications?
We are fairly stateless to begin with
Configuration Management
Inventory gems and plugins
External libraries
Shared Storage
Shared NEW EBS / S3
Messaging ( SQS / AMPQ, opposed to forking/backgrounding )
Log files ( Shared EBS helps )

Consider a Third Party
Simplify Architecture, Configuration Management and Deployment
Engine Yard
Right Scale
(others - lmgtfy)
Your Friendly Neighborhood Consultants

Thanks!
Brian Kaney – twitter.com/ bkaney
IRC – bkaney
Vermonster – http://www. vermonster.com

Bostonrb Amazon Talk

by Brian Kaney

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 6

Statistics

Bostonrb Amazon Talk Presentation Transcript

http://www.slideshare.net	5
http://www.linkedin.com	1