AWS Intro for Knight News Fellows
Upcoming SlideShare
Loading in...5
×
 

AWS Intro for Knight News Fellows

on

  • 1,250 views

I gave a brief intro about using AWS to our Mozilla Knight News Fellows.

I gave a brief intro about using AWS to our Mozilla Knight News Fellows.

Statistics

Views

Total Views
1,250
Views on SlideShare
398
Embed Views
852

Actions

Likes
0
Downloads
5
Comments
0

6 Embeds 852

http://jdotp.org 837
http://www.thegifreport.com 9
http://54.235.119.66 3
http://www.newsblur.com 1
http://webcache.googleusercontent.com 1
http://news.google.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

AWS Intro for Knight News Fellows AWS Intro for Knight News Fellows Presentation Transcript

  • AWS Fun
  • A short bit of history● Not that long ago, a "server" was :○ One piece of hardware○ One operating system○ Physically racked, powered, networked in amanaged datacenter● People started playing with "virtualization"○ One piece of hardware○ Multiple operating systems running independently○ Physically racked, powered, networked in amanaged datacenter
  • As virtualization was taking off...● Mid-2000s, Amazon.com buys a TON ofhardware.● The mantra for the folks building Amazon.com infrastructure:Provide service style endpoint access toinfrastructure management for internal use.EVERYTHING IS AN API
  • At the sametime....Marketing deptseverywhere goto town, asmarketing does...VIRTUALIZEITALL!!!!
  • Amazon Realizes...If we run the virtual server hosts...And we just open up our internal infrastructureAPIs to end users...
  • $$$$$$$$$$$$$$$$$$$$$$$$$$
  • Marketing took over,now everything is Cloud.
  • By Cloud, I mean....● Must be distributed.● Must be programatically accessible● Is multi-tenanted (you are not the only userof the hardware)
  • In general, what is AWS?● A collection of commonly used pieces ofsoftware, made easily accessible in:○ Distributed environment: Multiple Availability zonesper region, multiple regions○ Programatically accessible infrastructureFor example: Mysql, MS SQL, Memcached, Linux,Windows,CDN, DNS Management, User/Admin management,Firewalls, Load balancers...
  • Common componentsof infrastructurein your old datacenter
  • Common componentsof infrastructurein AWS
  • Some of what this buys us● We can spin up replica environments● Easier functional testing...in STAGING● Load test against prod without touching prod● Build in automated deployments and testing,making pushing to prod a breeze for alldevs.● This makes the feedback loop tighter, faster,and keeps changes and their inevitable bugsmore in context● This all wraps up to make you, the devs,more confident to try new things.
  • Controlling all of that infrastructure
  • Lots of configuration managementoptions....● Chef (Opscode)● Puppet (What I use)● AMIs (Server images)● Cloudformation (AWS Service)
  • But wait...isnt the cloud dangerous?● Yes! Just as dangerous as your datacenter● Secrets stores in S3, managed by puppet● Each app has its own key, security groups● Managing security via security groups, sshkeys
  • General scaling on AWS● Use autoscale groups (even if you neverhave to autoscale)● You can trigger autoscaling on any metric● Use EBS and instance store autoscalegroups○ 30 seconds to "traffic ready" prebuilt EBS instancevs. 2-10 min for a instance store○ Keep a baseline # of instance store nodes, for whenEBS has issues.○ You can have multiple autoscale groups load intoone ELB (so, app-ebs-fastscale-group and app-instancestore-noscale-group)
  • General scaling on AWS● For high IO data (RDS or self-managedEC2), use provisioned IOPS.● On EC2, EBS volumes can be RAID10d...need a 50k IOPS volume? :D Great way tovertically scale.●
  • General scaling on AWS● Adhere to 12factor.net rules so you canhoriziontally scale○ CNAME all resources, such as mysql servers. If youcan easily move a resource, you can easily verticallyscale it elsewhere and move to it.○ Store dependent content away from web tier nodes,ie media, user uploads. If a web node dies and youlost anything, you did it wrong.○ All pieces of app modular, independently scalableand revvable without retooling
  • General High Availability on AWS● Multi-Region (Each region has multiple AZs)● Multi-Availability Zone for○ RDS (built in) (takes ~3 min to failover)○ Load balancing○ Autoscaling groups (3 AZs recommended)● Dynamic DNS● Health Checks on apps
  • General High Availability on AWS● Mix in instance store baseline with EBS forfast scaling for when EBS has issues.● Health Checks on apps● Status updates to S3 file, updates app topoint to failover resources... No db? Writeto a SQS queue!● Oh yeah, use a lot of SQS!
  • CNAME for all the resources (12-factor friendly)
  • Easier to move, failover, rebuild
  • RDS Tricks● Multi-AZ, takes ~3 min to failover● EBS volumes of greater storage get betterperformance, always use 300gb for prod,even for small instances.● Read slaves have a lot of challenges withschema changes. It is usually faster to justrebuild slaves● For monitoring, grant repl client to user
  • Some other tricks● ELBs are EBS-backed EC2 Instances...when EBSalerts go out, be careful!● Setup ifttt alerts for AWS RSS status updates● Use New Relic. Please!● IAM Roles allow for interaction with AWSinfrastructure...think, a monitoring server that tells anautoscale group to respond to a problem by launchingnew nodes● Route53 is awesome. Alias A records, super reliable,you can keep low ttls
  • Pay Amazon Less● Reserved instances can save a lot of money● Spot instances are great for batch andprocessing, EMR, Cluster Compute● S3 static hosting is ridiculously inexpensive.Go that route for anything static.● For dev work, Heroku is great, no cost forapps that do not scale
  • Other random advice...
  • Good stuff● http://www.12factor.net/● http://paulstamatiou.com/how-to-getting-started-with-amazon-ec2● http://loggly.com/blog/2011/05/send-custom-metrics-to-cloudwatchs-api/● https://github.com/toolness/fleeting● AWS Marketplace has a lot of good stuff● My example repos: https://github.com/mozilla/sys_config_examples and https://github.com/mozilla/sys_scripts_examples● https://help.ubuntu.com/community/CloudInit● http://awsofa.info/●
  • Demo time (if there is time)-Building a new autoscale group/app?-Managing infrastructure via fabric, jenkins,puppet-Show off the puppet systems config setup?