Scaling APIs: Predict, Prepare for, Overcome the Challenges

Scaling APIsFeeding your SpeedsApril 5, 2012Greg Brail @gbrailBrian Pagano @brianpagano

@gbrail @brianpagano

groups.google.com/group/api-craft

youtube.com/apigee

New! IRC Channel #api-craft on freenode

What is scale to an API?Developers: Lots of developers building appsApps: Lots of apps for end users to useEnd users: Millions and millions of app usersVersions: Lots of API versions to manageAPI calls: All of these things result in API calls…

Today’s Topic: API CallsToday we are going to focus on handling huge numbersof API calls in an API infrastructure Ever been in a meeting where someone said,“let’s not talk about ‘speeds and feeds’ today?” This is not that meeting.

Why do APIs need to scale?Thousands of app developers… Each one must be managed, signed up, etc.Thousands of apps… Each one has credentials that need to be validatedMillions of end users… Each one has one or more OAuth access tokensResult in lots of API calls. For example: One million end users, Making 1000 API calls a day, Results in one billion API calls a day, Or about 11,000 API calls per second on average

Tracking API callsToday we’ll mainly talk about throughputMeasured in API calls or transactions per second (tps) For an API, usually as the number of users increases, throughput increasesAs throughput increases, latency often increases too It’s not enough just to handle lots of throughput – it’s important to handle it with a reasonable amount of latency

What Limits Scale?Disk DatabaseNetwork App serverCPU API ProxyMemory Load balancer Cache servers

What are some limits?Seek time Database design & tuningRotational speed App server coding & configTransfer speed Proxy configurationClock speed Load balancer policiesNumber of cores Cache configurationAmount of RAM And many more…

Some examplesWe’re going to talk about things to look at asthroughput grows from one level of traffic to another… 1 tps 10 tps 100 tps 1000 tps 10,000 tps 100,000 tps and beyond…..

Image from valdosta.edu

At 1 transaction per second86,400 per day / 2.5 million per monthAlmost everything can handle this.What about the database? If each API call makes several big SQL queries, it may not!

Strategies for 1 tpsTest (always)Tune the database installationTune the database designMonitor query performanceTest again!

image from istockphoto.com

At 10 transactions per second864,000 per day / 25 million per monthMost infrastructure can still handle this.What about the application server? Is the app well-designed enough? Does it make an excessive number of database calls?

Strategies for 10 tpsEnsure that the app server is properly optimized Do API calls make the minimum number of database calls? Do API calls depend on large numbers of external services?

image from istockphoto.com

At 100 transactions per second8.6 million per day / 259 million per month(Now we are starting to get somewhere)RDBMS systems may struggleLess-efficient app servers may struggle“Free” tiers on hosting platforms aren’t an option

Strategies for 100 tpsDatabase optimization and tuning is critical here Allocate fast storage, and lots of it Allocate lots of memory Tune the database to use it! Find bad queries and fix them or optimize themApp server tuning is critical here Are there enough threads in the thread pool? Are there enough processes?

Image from http://www.jigzone.com

At 1000 transactions per second86 million per day / 2.5 billion per monthNow everything may start to break…What is the mix between reads and writes?

Strategies for 1000 tpsUnderstand the mix between reads and writes Cache the reads as much as you can If you can cache them closer to the client, betterUnderstand your app server performance Faster app servers should still be able to handle (like Java) RoR, Python, PHP, etc will require much bigger clusters Stateless app servers are your friend!

More strategies for 1000 tpsCan the database handle the load? It can if most transactions are reads And you cache as much as you canOtherwise it’s time to scale the database layer Sharded RDBMSes Or a scalable NoSQL database works here

At 10,000 transactions per second864 million per day / 25 billion per monthIf most transactions are reads, caching is your friendOtherwise, this is serious business No single database can handle this Few single app servers can handle this If API calls are large, what will the bandwidth be?

Strategies for 10,000 tpsCaching is even more essential Even a simple cache can handle this load on one or two boxesDatabase writes are problematic No single database server can write 10,000 times per second Scalable, eventually-consistent databases can scale this big, (like Cassandra)

More for 10,000App servers You’ll need a cluster of app servers no matter what! What about session management? What about load balancing?

100,000 API calls per second8.6 billion per day!Now your API is truly impressive (either that or it is very poorly designed!)You will need racks of infrastructure no matter what!

Some other considerationsAPI designClient designLatencyBandwidth

What about API design?Every API call has overhead: TCP connection / SSL handshake / load balancer CPU / API proxy CPU / App server CPU and thread pool / database connections / disk I/O…Do you need to make so many? Can you design your APIs to support fewer high-value API calls? Can you have “batch” calls in your API?

What about the client?Can client apps use the API more efficiently? Don’t make the same API calls over and over Utilize compression Utilize conditional requests in HTTP Which means that the API server should support them! Request only the data that’s needed Which means that the API server should trim responses Or paginate them

What about latency?Latency kills user experience!How can the API server reduce it? Remove steps in the processing flow through caching Cache closer to the API clients

What about the network?What kind of network connection to you have? For instance, in EC2 you get 1Gbps, Or about 122 megabytes / second At 10,000 tps, for instance, that’s 12K per API call

THANK YOUQuestions and ideas to:@gbrail@brianpagano

http://blog.apigee.com	3136
http://blog.sonoasystems.com	83
http://apigee.com	65
http://feeds.apigee.com	35
http://mktg-dev.wearepropeople.md	5
http://webcache.googleusercontent.com	3
http://blog-dev.apigee.com	2
http://mktg-dev.apigee.com	1
http://blog-dev.wearepropeople.md	1
http://blog.local	1
http://conversation.ecairn.com	1
http://www.hanrss.com	1
http://ip54.216-86-157.static.steadfast.net	1

Scaling APIs: Predict, Prepare for, Overcome the Challenges

by Apigee

Accessibility

Categories

Upload Details

Usage Rights

13 Embeds 3,335

Statistics

Scaling APIs: Predict, Prepare for, Overcome the Challenges Presentation Transcript