Developer’s Guide to Amazon Web Services


Published on

Kinvey’s “Developer’s Guide to Amazon Web Services” eBook provides mobile app developers with everything they need to know to get started on AWS or any Infrastructure as a Service provider

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Developer’s Guide to Amazon Web Services

  1. 1. The Developer’s GuideTO AMAZON WEBSERVICES
  2. 2. Chapter 1Guide to Amazon Web Services 1
  3. 3. DEVELOPER’S GUIDE TO AMAZON WEB SERVICESWelcome to the latest Kinvey eBook. We is perhaps the most fundamental of theaim to help our community of develop- categories. Before diving into our guide,ers keep pace with the latest trends in let’s first review IaaS basics:app development, tools and marketing.Typically we share our perspective on According to TechTarget, Infrastructureour blog, but when our audience is as a Service is a cloud-based provisioninterested in a topic that can’t be done model that services cloud storage, virtualjustice in 500 words, we publish an servers and networking components toeBook instead. application owners for a usage-based cost. Its goal is to is to become aThis eBook curates some of the best foundation for Platform as a Servicethinking from Kinvey and outside (PaaS) and Software as a Service (SaaS)experts on how developers should providers by providing a flexibleapproach Infrastructure as a Service operating environment. In the IaaS(IaaS). The eBook emphasizes Amazon environment, the service provider owns,Web Services (AWS) because it’s the best runs and maintains the infrastructureknown vendor in the space, though we equipment, while the consumer takesdon’t recommend one provider over responsibility for configuration andanother. The eBook highlights the operations of the guest operatingbenefits of IaaS, helps in the IaaS vendor system, software and database.selection process, shares best practicesfor hosting an app on AWS, and provides A variety of technologies can benefittips for what to do in case of a service from IaaS: cloud-based CRM systems,outage. web, media and mobile applications, big data systems, and much more. But, IaaSWhat is Infrastructure as a Service? is not right for everyone. There are someThere are a handful of “__ as a service” important factors to consider before(*aaS) providers comprising the mobile determining if your application wouldand cloud computing ecosystems today. benefit from IaaS.Although individually each company maybe unique, collectively they share acommon goal: accelerating the rate ofinnovation by removing costs andbarriers to technology deployment. IaaS 2
  4. 4. Chapter 2How to Determine if IaaS is rightfor you – + 3
  5. 5. DEVELOPER’S GUIDE TO AMAZON WEB SERVICES Choosing an IaaS provider is a big IaaS may not be right for you if... decision. You want to trust the provider • usage is minimal or flat, and you have with your data and, essentially, your no plans to drive significant growth. entire application infrastructure. As with Maybe your product is built specifically any service, there are pros and cons for only a small subset of users (e.g., an associated with IaaS that must be internal application for a small company). assessed before deciding whether or not to use it for your application. You could benefit from IaaS if...• there is the potential for spikes in users “There are pros and cons or usage. Do you have upcoming press associated with IaaS coverage that may cause spikes in that must be assessed... ” downloads? Is your application seasonal?• you plan to expand your feature set. With an increase in features comes an • you don’t have a clear sense of your increased demand on infrastructure. application’s storage and networking “Scalability” doesn’t only apply to the needs. Successful IaaS deployments number of users, sometimes features benefit from clear user up-front too need to scale. requirements. • you are an enterprise concerned about• you’re an individual developer, small dev rogue users. (One risk of IaaS is rogue or shop or startup with no existing data unwarranted commandeering of center infrastructure or you’re an services. Because IaaS requires gover- established company taking on a large nance and usage monitoring, Vordels project that would require significant Mark ONeill recommends that additional data center infrastructure or enterprises establish cloud service staff. governance frameworks that help prevent employees from accessing• you have a low server-to-admin ratio information or services they are not and are looking to cut costs. permitted to use.) 4
  6. 6. DEVELOPER’S GUIDE TO AMAZON WEB SERVICES types, and supported operating The Major IaaS Players: A systems). Comparison Once you’ve decided to use an IaaS Another way to size-up vendors is by solution, it’s time to pick a vendor. While common “user concerns,” which is the AWS may be the most prevalent player thrust of this second TechRepublic chart. in the space - it is estimated to own Specifically, the table assesses vendors roughly 70 percent of the IaaS market - against security features (certifications there are several other reputable and protection), ease of migration (open vendors to consider, and many factors to standards and VM upload), and evaluate. Rest assured, there are plenty reliability [service age, Service Level of resources out there to help you Agreement (SLA), and support]. narrow down the options. This chart from TechRepublic evaluates the best-known IaaS providers against how they compare to common “cloud – promises.” The comparison takes into account a range of factors, including + pricing (variety, average, data transfer Pro Tips for IaaS Evaluation“ Beyond data, developers Regardless of whether you select an IaaS vendor based on its “cloud promises” or want to tap into many “user concerns,” Kinvey’s lead architect, tools and services that Shubhang Mani, advises you consider other clever minds have the following: created... ” Is IaaS really the solution you need? Depending on the complexity of your and storage costs), scalability (scaling up infrastructure needs, you might be and down, monitoring, and APIs), and better off opting for a higher layer in the choice / flexibility (number of data stack such as a Platform as a Service center locations, number of instance (PaaS), or even Backend as a Service 5
  7. 7. DEVELOPER’S GUIDE TO AMAZON WEB SERVICES(BaaS). The advantages of choosing Plan for the worst casethese alternatives are reduced opera- No system is infallible. Outages cantional complexity and decreased time to occur, even at your cloud provider ofmarket. The disadvantages are reduced choice. It is important that you have acontrol and a narrower set of choices as plan in place to address this were it toto the underlying infrastructure occur. For example, you should considercomponents. PaaS or BaaS might be a offsite backups for your critical data andgood starting point, allowing you to alternate deployments from your mainfocus on building the application / location.system until you deem it necessary toexert greater control on the choice of Understand your support requirementsinfrastructure components. Not all IaaS providers provide the same levels of support. In some cases, supportIaaS is not a silver bullet is priced independently from the actualSome key benefits of IaaS include ease in product and may end up being quitedeployment, redundancy, and the ability expensive. Most providers offer a freeto scale much faster than conventional support level via community forums.means. Deploying a distributed system, This may or may not be sufficientespecially across geographic regions, can depending on your needs.also be achieved in more easily. With so many factors to consider,However, it is important to note that selecting an IaaS vendor for your app iswhile IaaS gives you the means to arguably the most difficult part of theprovision, deploy and scale infrastruc- process. The next step is actuallyture, it is up to you to configure, monitor deploying your app on the chosen IaaSand maintain said infrastructure and use provider. Because AWS is the mostit in a manner that makes the most popular vendor it will be the main focussense to your system. You might be able moving build multiple redundancies into acomplex system sitting across geograph- 5 best practices for deploying anic regions in a matter of hours, but if application on AWSyour firewall rules are improperly Amazon Web Services is a majorconfigured, you’re still vulnerable to Infrastructure as a Service provider thatattack. provides elastic capacity, quick deploy- ment and automation for applications 6
  8. 8. DEVELOPER’S GUIDE TO AMAZON WEB SERVICESwithout using capital expenditure. Familiarize yourself with these toolsWithin AWS are several infrastructure before diving in.building blocks, including EC2, S3, andRDS, to name a few. Click here for a full Start smalllist of AWS products for mobile applica- Start by moving a small project to AWStion hosting. before your full project is underway. This way you can fully test and learnAWS may be a compelling choice for about the various components thatyour app’s infrastructure needs, but you’ll be using without worrying aboutremember, if you want more than just managing an entire, there are other categories of*aaS vendors, such as Platform as a Start free.Service, Software as a Service, and Consider starting with Amazon’s free tierBackend as a Service, that address to test and become familiar with thedifferent application needs. You maywant to consider vendors in “adjacent” “service categories as well. That said,below is a list of best practices for Know what yourhosting an app on AWS. project/appliccation isUse the right tool for the job. and the problem it solvesAlex Handy, Senior editor of SD Times, before you dig in.advises: “Know what your project/appli-cation is and the problem it solves ”before you dig in. Let AWS manage theinfrastructure so you can focus on the platform before jumping into a fullbusiness you do best.” As previously development effort. You get 5GB ofmentioned, there are several services storage free for a year on S3, so you canwithin the AWS offering. Alex suggests easily back something up for free to seecombining multiple: “For example, try if AWS is the way to go.Amazon Relational Database Service foryour database, AWS Elastic Beanstalk for Leverage multiple availabilityyour development environment, or zones.Amazon Elastic Map Reduce for your If you want your app to be fault-tolerant,Hadoop cluster and Big Data needs.” mirroring across availability zones is key 7
  9. 9. DEVELOPER’S GUIDE TO AMAZON WEB SERVICES for high availability and disaster unforeseen outages due to factors out of recovery. Ensure your design anticipates our control. If it wasn’t AWS, it would be and manages component failure to the other cloud provider you chose. But significantly reduce the chances of it there is still hope: when AWS is down, failing. you can still be up if you take the proper measures to prepare for possible outages in advance. 4 things you didn’t know about AWS outages“...always have a backup The eastern region of the US is the most likely area to suffer outages because ... plan in the event of an … it has the oldest data centers. outage … it is the largest in terms of data center ” footprint. … it is the default region for most customers, who don’t bother changing it because it’s cheaper and/or they aren’t Design for failure and nothing will fail aware they can change the region Understand Amazon’s disaster recovery principles, and always have a backup plan Availability zones are guaranteed to in the event of an outage. Our developers be distinct per customer only. have compiled multiple guides on recover- However, there are no guarantees as to ing from and preparing for an Amazon the composition of the availability zone outage - take a look at the next segment, across customers. To illustrate, if you and learn how to be proactive for the sake launch an instance in availability zone 1a of your application. and a different customer launches an instance in their availability zone 1a, the AWS outage survival guide two instances are not guaranteed to run An outage on your cloud service provider on the same subset of physical likely means an outage for your app and infrastructure. This is why Amazon does thus a delay in the experience on the not specify or call out names of availabil- user-end. This can be extremely detrimen- ity zones in their outage status updates, tal to your app’s ratings and overall usage. because 1a for one person could be 1d The unfortunate truth is that everyone has for another. 8
  10. 10. DEVELOPER’S GUIDE TO AMAZON WEB SERVICESSome outages are worse than others. control planes unusable, especially if theThis is because outages that affect core outage affects components that areinfrastructure components such as used to service the control planesAmazon EBS (Elastic Block Store) have a themselves. This results in customersripple effect on other AWS products that being unable to minimize downtimeutilize these components. For example, faced by their application / systems. Ait is possible to provision an EC2 potential solution to this would be toinstance using ephemeral storage that is have a hot / cold standby set of compo-bound to the instance. This instance nents ready in a different region. Whendoes not utilize EBS and should the outage occurs, it would be a mattertheoretically be impacted to a lesser of bringing these components onlinedegree. Let’s say you have an application with the understanding that outagesthat runs on this instance. This applica- that span regions are far less likely thantion also uses a MySQL database and those within a single’ve opted to use Amazon’s RelationalDatabase Service (RDS). Now you’re backto (potentially) being affected by an EBSoutage since RDS uses EBS for storage.In fact, a judicious use of these API’s viaclient libraries and scripts is what allowsone to be able to do things like launchnew instances based on traffic volumeand / or provision new infrastructure asnecessary. When an outage occurs,affected customers attempt to remedi-ate their situation by trying to provisionnew servers in other availability zonesand / or regions in order to redirect webtraffic and/or proceed with other tasks.This means that there’s a fairly largespike in traffic hitting the control planewhich invariably results in slowdownsand timeouts. Another side effect of theoutage could also be to render the 9
  11. 11. Chapter 3Surviving AWS Failures with anode.js & mongoDB Stack AWS SURVIVAL KIT 10
  12. 12. DEVELOPER’S GUIDE TO AMAZON WEB SERVICES In this segment, we’ll explain how to fully MongoDB instances should each be load prepare for an AWS EC2 outage with a balanced across multiple Availability node.js and MongoDB stack. Node+Mon- Zones. The more the better. go on EC2 is a very popular software stack among web services developers. 4. Place the node servers and the mongo There are many user guides on how to servers all in a security group, which design this system with built-in redun- allows only the Mongo ports internally dancy so that even coordinated failures and your application ports externally. don’t bring down the service. The This is trivial to set up and protects your absolute minimum for a resilient service database from external requests. requires a MongoDB replica set behind a load-balanced node farm. 5. MongoDB’s authentication provides additional protection. Mongo’s security You are not ready for an EC2 outage until model has limited robustness, but you have deliberately shut down having authentication in your MongoDB components in your system and verified store is still useful even if the application the expected behavior. As you periodical- and the database are inside an EC2 ly do this, you might discover that there security group. For your data to get are gaps you did not account for. Take exposed, you will have to make multiple the following steps to be as prepared as mistakes at the same time, which possible: happens, but the chances are greatly reduced.1. A Node.js single event will by default crash on an unhandled exception. Use 6. Ensure that failover happens smoothly. upstart or forever to restart the process. Shutdown the primary Mongo instance and see what happens as requests keep2. Use Monit, an external process on your coming in. The replicas notice the down server that makes liveness checks and primary and one of them takes over, but potentially restarts your service. Monit upon an incoming request you see this will also email you if and when it had to error message: “unauthorized db:mydb restart. While upstart ensures that your lock type:-1 client:” process is up, monit ensures that it is responsive. 7. What this error message means is that the failover happened, but your3. Your application instances and your application’s request is not authenticat- 11
  13. 13. DEVELOPER’S GUIDE TO AMAZON WEB SERVICES ed. This is an example of an esoteric bug GoDaddy, unfortunately this guide cannot that may not show up until you do a full help - you’ll just have to wait until end to end test. The bug is now in a pull GoDaddy is back online. request. Since pull requests don’t get 1. To start, log into the Route 53 dashboard released quickly, use npm git dependen- in the AWS Console. cies to install your app from your forked repo.8. While you have a down MongoDB instance, it may take a long time for your “...know how your appli- application to restart. The default in cation fails when a net- node-mongodb-native is “no timeout”, work service like DNS is which means leaving it down to the OS. unavailable To avoid yet another timeout cycle, use Mongo’s connectTimeoutMS setting. ”9. This ensures that your restarts will take 2. As the landing page describes, your first a little over 500ms, if you have a down step is to create a “hosted zone”. A instance. However, don’t assume that hosted zone is a concept created by your Mongo SDK supports it – Amazon to describe a collection of DNS node-mongodb-native doesn’t, unless records (i.e. A records, CNAME records, you use this github patch. MX records) that are managed together under a single parent domain name. You10. You are now prepared for an AWS can use the Route 53 dashboard to outage. Bring it on. manage these hosted zones. For a good explanation of the different types of DNS Migrating from GoDaddy DNS to records, check out Google’s guide. Amazon Route 53 This next guide walks through the 3. Click “Create Hosted Zone,” and fill in process of migrating from GoDaddy your domain name in the form that hosting to Amazon’s Route 53. This only appears on the right. applies to domain names that are controlled by another registrar, where 4. When you created the hosted zone for GoDaddy is used just for hosting. If the your domain, Route 53 filled in some domain name itself is controlled through basic DNS records. To see them, select 12
  14. 14. DEVELOPER’S GUIDE TO AMAZON WEB SERVICES your domain name from the list, and with your application. If your app relies click “Go to Record Sets” in the top right. on DNS to connect to another server (i.e. You’ll see that Route 53 populated your database), it may stop trying after a few domain with a set of NS records and failed lookups. At this point things SOA records. probably require manual intervention, even if the DNS service recovers. The key5. Your current site may have extra DNS takeaway from this is that you should records needed. For example, if you host know how your application fails when a a blog on Tumblr, you probably have a network service like DNS is unavailable. CNAME record setup to create that link. And you should know whether or not it Also, if you receive email at your domain, will require manual intervention to you probably have an MX record set for recover. that. Make a list of all these extra records you’ll need, and add them now To prevent manual intervention, you can (use the “Create Record Set” button). use a tool called Monit. Monit is a daemon that is responsible for monitor-6. Finally, it’s time to make the switch. ing your server and application health. It Head over to your current registar, and can also be configured to check the change the nameservers from the health of external services such as DNS. GoDaddy addresses to the ones provided in the NS record set on your Let’s take a look at a basic config to start: Route 53 dashboard. Note: because DNS servers around the world cache this value, it may take some time to see the change work while the update propa- gates through the DNS system. This config monitors an HTTP application Recovering from a DNS service outage that listens on on port 9009. in AWS using Monit When there is a failure, Monit alerts you Our final AWS outage survival guide uses and attempts to restart the process. a tool called Monit to recover from a We don’t want to constantly restart the DNS service outage. A DNS failure isn’t service if it is unhealthy, so if the app something you see everyday at AWS, but fails 10 times within 10 cycles, then when it happens it can cause problems Monit will leave the app off and stop 13
  15. 15. DEVELOPER’S GUIDE TO AMAZON WEB SERVICESmonitoring it. re-enables monitoring. It also disables our aws-dns-healthcheck since we knowWhen a network service fails like DNS, DNS is healthy.your application becomes unhealthy,tries to restart 10 times, and then You should now be able to recover yourbecomes unmonitored. Now you’re at a app during a DNS failure. To be 100%point where things require manual sure this works as intended, we canintervention to recover, even if DNS simulate another DNS outage usingbecomes available. Let’s configure Monit take care of everything for us. Run this command: sudo iptables -ABuilding on our previous configuration: OUTPUT -p udp –dport 53 -j DROP This command will prevent any DNS queries from completing. If your app relies on DNS you should see it fail once its DNS cache has expired. This should trigger the DNS check to be enabled via- We’re still monitoring our HTTP app on Monit. You can check the status of Monitport 9009 monitoring using the command: sudo- When the application fails now, it kicks monit statusoff the aws-dns-healthcheck monitor.- The aws-dns-healthcheck monitor To re-enable DNS run this:checks to see if it can resolve DNS on sudo iptables -F172.16.0.23 (the default AWS DNSserver). Your application should now try to- When DNS reports healthy for 3 cycles, recover after the DNS check returns to amonit execs a script to try to recover the healthy The examples above focus on how to approach a failure in the DNS service. InThe example recovery script is simple: reality there could be a myriad of external services that may affect the health of your app. You can use the same approach as described above andIt calls /usr/bin/monit start appsrv1 expand the aws-dns-healthcheck into awhich will attempt to start the app and generic health check for your applica- 14
  16. 16. DEVELOPER’S GUIDE TO AMAZON WEB SERVICEStion. This could include testing networkconnectivity, connectivity to externalservices (e.g: a database), and any otherprocesses that your app depends on.You can find examples for monitoring allkinds of services on the Monit websitehere.Again, the takeaway from this is to knowhow your application behaves underdifferent failure scenarios. Testingconnectivity loss, loss of networkservices, and other failures are veryimportant when building a highavailability application. 15
  17. 17. Written byKelly Rice and Shubhang Mani Designed by Jake McKibben Survival Guide AuthorsIvan Stoyanov, Dave Wasmer and Joey Imbasciano