An overview of the cloud technologies that I've used and the nuances between them. This presentation was talking primarily about: https://github.com/riptano/ComboAMI/tree/2.2

1. February 7, 2012
Presented by:
Joaquin Casares
@joaquincasares Baking Your Own
AMI (and More)
A presentation for Austin Web
Architecture

2. Fair Warnings
› I am a clutterer
› Fast speech rate
› Stutter
› Exchange words
› Assume allthethings!
› Speak up!
› Ask for clarifications at any point J

3. The DataStax ComboAMI
› History
› Othervendors existed
› Some tools were out there
› Idea
› Ease-of-use

4. The DataStax ComboAMI
› Goals
› Think and it shall be created
› Test bed
› Best practice configurations
› Code
› https://github.com/riptano/ComboAMI/
tree/2.2
› Go through code line-by-line

5. The DataStax ComboAMI
› Goals
› Think and it shall be created
› Test bed
› Best practice configurations
› Code
› https://github.com/riptano/ComboAMI/
tree/2.2
› Go through code line-by-line (Joke)

6. Initial Setup
› pre_1.sh
› Setup
› Git Clone
› pre_2.py
› Dependencies + extra software
› Init.d
script
› Secure

7. Initial Setup (Reasoning)
› Say “No” to shell scripts
› subprocess.Popen(shlex.split(command))
› Minimize second startup time
› Init.d script is the pivotal key
› Clean all ssh keys

8. Baking in AWS (and Publishing)
› http://www.datastax.com/dev/blog/
personalizing-your-own-brisk-ami
› Get image and:
› Freezethe image to bundles in /mnt
› Uploads bundles to S3
› EBS (We don’t use it.)
› EBS
is easier at snapshotting since you don’t
have to… copy-paste

9. EBS Troubles
› EBS volumes contend directly for network
throughput with… a peer QoS policy to
standard packets.
› At some point… you will encounter
misbehaving EBS volumes.
› People have a tendency to bolt a large
number of EBS volumes to a host,…[this
will] outstrip… buffer caches and [ability
to] concurrently serve requests.
--Direct quotes from previous link.

10. The Boot
› ds0_updater.py
› git pull
› V2:git pull for dev and wget tarball for
‘release’
› Kick off ds1_launcher.py
› ds1_launcher.py
› Determine if this is the first launch

11. The Boot (Reasoning)
› ds0 will be your most stable and user
exposed code
› Keep it simple since no one wants to re-
image on 5, 6, 7… a lot of different zones
› ds1 can be updated, but this is a simple
starter of services

12. Configurations
› ds2_configure.py
› Read and parse EC2 data
› Setup repos
› Install software
› Construct configuration files
› RAID0 devices
› Sync clocks
› Return to ds1_launcher.py

13. Configurations (Reasoning)
› Primaryscript
› Makes the AMI
› Up to date
› No
need to stay up all night with the release
team
› Remove this after execution for highest
stability

14. The Boot
› ds1_launcher.py
› Always ensure the RAID device is mounted
› Set fleeting settings
› Allow the first node to launch
› Keep all other nodes waiting
› Once a seed is live, launch all other nodes
at the same time
› Kick off ds3_after_init.py

15. After Initialization
› ds3_after_init.py
› Waits 60 seconds and then:
› Emails logs to a personal email
› Smoke testing
› Starts OpsCenter which requires Cassandra to
be live and stable
› ds4_motd.py
› Prints
a quick diagnostic screen and useful
advice and getting started tips

16. After Initialization (Reasoning)
› ds3 will take care of other things that must
happen after certain services are up
› ds4 is a nice personalized welcome to the
customer describing why they are here
› ds4 also gets the customer up and
running today instead of:
› “Okay,
now I have a Cassandra ring, will it
blend?”

17. Other Methods
Using pre-built solutions

18. Chef
› Pros
› Functions are included for simple scripts
› Ruby code runs for complex scripts
› Incredibly helpful community on IRC
› Cons
› No simple way to launch clusters (intuitively)
› Seems to be more for single machine
deployment
› Data bags may be the key workaround

19. InfoChimps’ Cluster Chef
› Pros
› Already packages many chef scripts
› A hybrid of Chef and InfoChimps code
› Cons
› Seemsto require some Chef syntax
knowledge before jumping on board

20. Puppet
› Pros
› I liked the webcast video I’ve watched:
http://puppetlabs.com/misc/webinars/
› All operation based (Pro?)
› Unlike Chef that allows arbitrary Ruby coding
› 2.0’s code and GUI look amazing
› Cons
› Puppet 1.0 didn’t look like it was going to be
very friendly
› 1.0 had multiple ‘Getting Started Docs’
› TL;DR

21. RightScale
› Pros
› Incredibly deterministic
› Has the chef mentality of recipes
› Great support for official RightScripts
› Incredibly easy to use for the Ops team
› Idiot-proof settings system
› IRC is run by helpful RightScale employees
who are fast and friendly

22. RightScale
› Cons
› Not so easy to use for the Dev team
› No CLI
› No great version control
› No
github access (except for experimental
chef)
› No hosted chef
› Not the best refresh rate

23. RightScale
› QuickStart Tips:
› A Deployment contains ServerTemplates
which are RightScripts on top of MultiCloud
Images
› The idea is the developers write RightScripts
and compile them into a ServerTemplate
› The operations team then adds these
templates to a deployment

24. RightScale
› Good for:
› Multiple small projects
› Clusters that need expanding and deflating
based on rules
› Where Dev and Ops teams are separate
› A collaborative interface for EC2 and
Rackspace

25. Clouds
Choosing the right cloud

26. Amazon’s EC2
› EC2
› Instance-stores (ephemeral devices)
› S3
› Unlimited buckets on a redundant system
› Reactive GUI
› Things work pretty snappy
› When EC2 goes out
› Everyone went out as well (No sweat)
› Don’t be on EBS

27. Amazon’s EC2
› Pros
› Easy, simple
› Everybody’s doing it
› Integration is key
› Cons
› Live updates are constantly pushed and things
break
› Steal % (As seen in `top`; >15% is too high)
› Things you would expect… aren’t
› This does sometimes have to do with the images
themselves
› Happy fishing!

28. RackSpace Cloudservers
› Simple to read choices that aren’t hidden
by “instance types”
› More like real hardware
› Better I/O
› Time outs not seen as frequently as on
EC2 small instances
› Allows for hybrid hardware/cloud solutions
(Rack Connect 2.0) at a more reasonable
cost

29. Rack Connect 2.0
› After discussing the problems further with
multiple individuals, there are temporary
Rackspace users that need to be created
and external IP addresses that will change at
least once in the machine’s lifetime.
› As long as keeping track of external IPs is
workaround-able and machines are allowed
to create temporary ssh users, they typically
run well.
› Do note that the point at which these users will
be created and the IPs will change is at
random, but guaranteed to happen only once.

30. RackSpace Cloudservers
› Pros
› Prebuilt Rackspace Images with OS and version
selections
› Nobody’s doing it (less steal %)
› Support tickets with < 24 hour response time
› Static IPs
› Rax 256MB instances > EC2’s small instances
› Cons
› Sluggish interface
› No image ‘market’
› Lack of other integration utilities

31. Back to the ComboAMI
And some tools

32. The “Reflector”
› Having a constant server up at
reflector.datastax.com allows us to have
100+ nodes cluster together individually
› This also allows for the nodes to be a bit
more ‘self-aware’

33. The “Reflector”
› Life of a node:
› I know my launchindex
› I have a reservation-id
› You’ve told me how many nodes to expect
› Tell the reflector my launchindex and
private IP and match me up with others
from my reservation-id
› Retrieve the seed IPs

34. The “Reflector”
› Pros
› Easy to setup
› Easy to manage
› Cons
› Introduces a single point of failure at startup
› Oncethe nodes are up, there is nothing to
worry about

35. The New Launcher Interface
› https://github.com/joaquincasares/
cassandralauncher
› pip install cassandralauncher
› `cassandralauncher`

36. Plain Cluster Launcher
› https://github.com/joaquincasares/
cassandralauncher
› pip install cassandralauncher
› `clusterlauncher`
› Works on EC2 and Rax
› Simple, yet easily manipulatable
› Better than VMs

37. Future Plans
› Create the ComboAMI with an EBS stop-
like feature on instance-stores
› Adding instance size option for both
launchers
› Cassandra Chef scripts

38. February 7, 2012
Presented by:
Joaquin Casares
@joaquincasares
Questions?
joaquin@datastax.com