1. Scaling APIs
Feeding your Speeds
April 5, 2012
Greg Brail @gbrail
Brian Pagano @brianpagano

2. @gbrail @brianpagano

3. groups.google.com/group/api-craft

4. youtube.com/apigee

5. New!
IRC Channel
#api-craft
on freenode

6. What is scale to an API?
Developers: Lots of developers building apps
Apps: Lots of apps for end users to use
End users: Millions and millions of app users
Versions: Lots of API versions to manage
API calls: All of these things result in API calls…

7. Today’s Topic: API Calls
Today we are going to focus on handling huge numbers
of API calls in an API infrastructure
Ever been in a meeting where someone said,
“let’s not talk about ‘speeds and feeds’ today?”
This is not that meeting.

8. Why do APIs need to scale?
Thousands of app developers…
Each one must be managed, signed up, etc.
Thousands of apps…
Each one has credentials that need to be validated
Millions of end users…
Each one has one or more OAuth access tokens
Result in lots of API calls. For example:
One million end users,
Making 1000 API calls a day,
Results in one billion API calls a day,
Or about 11,000 API calls per second on average

9. Tracking API calls
Today we’ll mainly talk about throughput
Measured in API calls or transactions per second (tps)
For an API, usually as the number of users increases,
throughput increases
As throughput increases, latency often increases too
It’s not enough just to handle lots of throughput – it’s
important to handle it with a reasonable amount of latency

10. What Limits Scale?
Disk Database
Network App server
CPU API Proxy
Memory Load balancer
Cache servers

11. What are some limits?
Seek time Database design & tuning
Rotational speed App server coding & config
Transfer speed Proxy configuration
Clock speed Load balancer policies
Number of cores Cache configuration
Amount of RAM And many more…

12. Some examples
We’re going to talk about things to look at as
throughput grows from one level of traffic to another…
1 tps
10 tps
100 tps
1000 tps
10,000 tps
100,000 tps
and beyond…..

13. Image from valdosta.edu

14. At 1 transaction per second
86,400 per day / 2.5 million per month
Almost everything can handle this.
What about the database?
If each API call makes several big SQL queries, it may not!

15. Strategies for 1 tps
Test (always)
Tune the database installation
Tune the database design
Monitor query performance
Test again!

16. image from istockphoto.com

17. At 10 transactions per second
864,000 per day / 25 million per month
Most infrastructure can still handle this.
What about the application server?
Is the app well-designed enough?
Does it make an excessive number of database calls?

18. Strategies for 10 tps
Ensure that the app server is properly optimized
Do API calls make the minimum number of database calls?
Do API calls depend on large numbers of external services?

19. image from istockphoto.com

20. At 100 transactions per second
8.6 million per day / 259 million per month
(Now we are starting to get somewhere)
RDBMS systems may struggle
Less-efficient app servers may struggle
“Free” tiers on hosting platforms aren’t an option

21. Strategies for 100 tps
Database optimization and tuning is critical here
Allocate fast storage, and lots of it
Allocate lots of memory
Tune the database to use it!
Find bad queries and fix them or optimize them
App server tuning is critical here
Are there enough threads in the thread pool?
Are there enough processes?

22. Image from http://www.jigzone.com

23. At 1000 transactions per second
86 million per day / 2.5 billion per month
Now everything may start to break…
What is the mix between reads and writes?

24. Strategies for 1000 tps
Understand the mix between reads and writes
Cache the reads as much as you can
If you can cache them closer to the client, better
Understand your app server performance
Faster app servers should still be able to handle (like Java)
RoR, Python, PHP, etc will require much bigger clusters
Stateless app servers are your friend!

25. More strategies for 1000 tps
Can the database handle the load?
It can if most transactions are reads
And you cache as much as you can
Otherwise it’s time to scale the database layer
Sharded RDBMSes
Or a scalable NoSQL database works here

26. At 10,000 transactions per second
864 million per day / 25 billion per month
If most transactions are reads, caching is your friend
Otherwise, this is serious business
No single database can handle this
Few single app servers can handle this
If API calls are large, what will the bandwidth be?

27. Strategies for 10,000 tps
Caching is even more essential
Even a simple cache can handle this load on one or two boxes
Database writes are problematic
No single database server can write 10,000 times per second
Scalable, eventually-consistent databases can scale this big,
(like Cassandra)

28. More for 10,000
App servers
You’ll need a cluster of app servers no matter what!
What about session management?
What about load balancing?

29. 100,000 API calls per second
8.6 billion per day!
Now your API is truly impressive
(either that or it is very poorly designed!)
You will need racks of infrastructure no matter what!

30. Some other considerations
API design
Client design
Latency
Bandwidth

31. What about API design?
Every API call has overhead:
TCP connection / SSL handshake / load balancer CPU / API
proxy CPU / App server CPU and thread pool / database
connections / disk I/O…
Do you need to make so many?
Can you design your APIs to support fewer high-value API calls?
Can you have “batch” calls in your API?

32. What about the client?
Can client apps use the API more efficiently?
Don’t make the same API calls over and over
Utilize compression
Utilize conditional requests in HTTP
Which means that the API server should support them!
Request only the data that’s needed
Which means that the API server should trim responses
Or paginate them

33. What about latency?
Latency kills user experience!
How can the API server reduce it?
Remove steps in the processing flow through caching
Cache closer to the API clients

34. What about the network?
What kind of network connection to you have?
For instance, in EC2 you get 1Gbps,
Or about 122 megabytes / second
At 10,000 tps, for instance, that’s 12K per API call

35. THANK YOU
Questions and ideas to:
@gbrail
@brianpagano