Application Resiliency

Application Resiliency
Chris Westin
July 29, 2010

Definition
7/29/2010
2
What is “Application Resiliency?”
Qualitative concept
If something goes wrong, can can the application do better than serving a “404 – Not Found?”
Application can cope with the failure of systems it depends on
“Graceful degradation”

Motivation
7/29/2010
3
Why is application resiliency important?
Ad-supported businesses
No page shown, no ad shown
Ad impressions are tied to revenue, so down-time is lost revenue
Loss of user goodwill
“I can’t get to my email/calendar/movie showtimes/…, this service sucks.”
Subscription businesses
May need to make up lost service time to user as a credit
Need to build systems to track and redeem credits
Money, SOX compliance
Loss of user goodwill
“I couldn’t watch a movie/baseball game/… last night, this service sucks.”

Implementation
7/29/2010
4
General Principles
Break application up into loosely coupled components that can function independently
Page generation
Content management
Video streaming
Ratings
Commenting
Etc
Run redundant instances of application components
Have at least one extra instance in case another instance fails
Make the user interface do as much as it can with what is available
Use timeouts and check for errors
Work around dependent services that are unavailable

Example – A Content Portal
7/29/2010
5
This includes things like News, Sports, Front Page, etc
A classical implementation might do the following:
Connect to database(s)
If failure, return 404 or other static page
Get content by querying for it in database
Echo templated page structure
Format content into template
More template, content, etc
Failures anywhere along the way could result in 404/static page
Have to keep checking db result codes – connection could break at any time
Requires buffering the accumulating response in order to change the status code

Content Portal Content
7/29/2010
6
We found our content comes from a variety of sources
Populate pages with content from independent HTTP web services

Content Portal Pages
7/29/2010
7
We wrote a page generation engine to handle content modules
For each content module, declare a RESTful data source, and a content renderer
Raw content is fetched from the RESTful web service
The content renderer uses the raw content to create an HTML fragment
A page template specifies where to insert each module’s result
The result is the rendered HTML fragment
The page generation engine makes parallel asynchronous HTTP requests for the content
As each response comes in, run the renderer on it, and hang on to the HTML fragment
After all responses have been received, insert the HTML fragments into the template and serve the result
Modules specify action to take if an HTTP response times out or returns an error
Best practice is to serve static content…

Page Generation
7/29/2010
8
The page generator makes multiple parallel asynchronous HTTP requests to fetch content
Parallel calls take advantage of parallelism at the network level
Web service instances are independent of each other

Web Service Availability
7/29/2010
9
Within a colo, redundancy is used to make a web service available
Load balancers are used in pairs to avoid another SPOF
Each tier has one more host than necessary to meet load requirements
Database tiers have replicas to meet read load requirements

Web Service Caching
7/29/2010
10
Services for long-lived content can also employ caching to increase availability
Squid can be configured to return a stale result on error or timeout

Site Availability
7/29/2010
11
Presence in multiple colos avoids another SPOF
Colos are required to have sufficient capacity to handle the load when one colo fails

7/29/2010
12

http://timmorrow.org	1119
http://nosql.mypopescu.com	123
http://www.linkedin.com	8
http://translate.googleusercontent.com	7
http://static.slidesharecdn.com	3
http://webcache.googleusercontent.com	1
http://atm09td.tumblr.com	1
http://theoldreader.com	1

Application Resiliency

by Chris Westin

Accessibility

Categories

Upload Details

Usage Rights

8 Embeds 1,263

Statistics

Application Resiliency Presentation Transcript