This document discusses performance testing of REST APIs. It provides lessons on using a non-blocking HTTP client like Akka for tests to avoid blocking connections, reusing logic from functional tests, devising a performance test plan with metrics and sign-offs, capturing results in a database for trending, running tests continuously in CI environments, and creating visualizations of performance test results with charts and dashboards. The goal is to share an approach for efficient, scalable API performance testing.

1. Performance Testing REST APIs
Selenium Conference 2013
Jason Weden @jweden
Principal Quality Engineer
Constant Contact, WebServices
Copyright © 2012 Constant Contact Inc. 1

2. Who We Are
 What do we do?
 Expose our internal web services externally to developers and
partners via an API for their own applications.
 http://developer.constantcontact.com
 Fun Digression: Very easy to get up and running with signing into API using
Mashery and using our console to hit the REST API
Copyright © 2012 Constant Contact Inc. 2

3. Problem Domain
 System Under Test: HTTP REST APIs
 Hitting the REST API with concurrent connections
 Defining your Performance Test approach
 Getting & Sharing results – performance metrics
Copyright © 2012 Constant Contact Inc. 3

4. Lesson 1: Be DRY
 Why in the heck repeat yourself?
 You already have the logic in your functional tests for REST calls so
reuse that logic for your performance tests
 http://techblog.constantcontact.com/api/what-lego-blocks-teach-us-
about-rest-api-testing/
 Reuse your data-seeding (e.g. for tests that have GET calls)
 Tools like LoadRunner and Jmeter have their place but
 They’re creating blocking connections that reduce scalability
 You are wet – you’re rewriting logic you already wrote in your functional
tests
 You confine the creation, execution, and general knowledge of the
performance testing arena to one or more toolsmiths
Copyright © 2012 Constant Contact Inc. 4

5. Lesson 2: Choose Carefully
 Choose your http client library with care
 Non-blocking
 Scale-up
 Seriously consider Akka (if on the JVM)
 Non-blocking but with high throughput
 Scales to multiple processors and prevents need to deal with shared state
 More on this later
 Other Examples on the JVM of non-blocking http clients
 http://spray.io (Scala) – Spray – Akka-based
 http://www.eclipse.org/jetty/ (Java) -- Jetty 9
 https://github.com/timboudreau/netty-http-client (Java) -- Netty
Copyright © 2012 Constant Contact Inc. 5

6. Lesson 2: Choose Carefully (cont.) – Our Choice…
 Written in Java but takes advantage of Akka for concurrency
 Integrated into test framework for everyone to use
 3rd Party libraries
 Apache HttpClient library
 Apache HttpCore library
 Akka concurrency framework
 Apache Commons Math3 library
 High performance
 Easy to use programmatic API – small learning curve
 Developers can go in and create performance tests
 Useful for any http functional OR performance testing
 Even functional tests have performance metrics logged
Copyright © 2012 Constant Contact Inc. 6

7. Lesson 2: Choose Carefully (cont.) – Our Choice…
 Found errors & significant bottlenecks
 Comparison with JMeter and LoadRunner
 Preliminary tests show faster than JMeter (even with our 8
threads vs JMeter's 1500)
 All performance tests can be checked into source control
 No dependency on toolsmiths for HTTP performance testing
 Performance testing experience shared across the team
 Likely faster than JMeter and LoadRunner -- still to be
scientifically determined
 Ability to tweak our home-grown solution vs. closed source
purchased solutions. (e.g. lots of concurrency properties to
tweak)
Copyright © 2012 Constant Contact Inc. 7

8. Lesson 3: Devise a Performance Test Plan
 What is your SLA? (e.g. avg. response time, avg. requests/sec)
 What kind of performance test? Options:
 Profile of production performance. Do the following
concurrently:
 X amount of POSTs to endpoint 1
 Y amount of GETs to endpoint 2 (data seeding needed)
 Z amount of DELETEs to endpoint 3 (data seeding needed)
 One type of Request: x amount of GETs
 Longevity – run test above for x amount of time
 What metrics will be recorded (client vs server)?
 How will results be captured and reported?
 Have sign-offs on your test plan (e.g. in a wiki section)
Copyright © 2012 Constant Contact Inc. 8

9. Lesson 4: Capture Perf. Test Results in a Database
 Capture:
 Performance Metrics
 Client: min/max/avg response time, standard deviation, avg
requests/second, percentiles, request failures
 Server: cpu, memory, open db connections, thread count, consider app.
profiling
 Test tool used and version
 Test name and description
 Freeform comments
 Concurrency level
 In what environment was test performed (server hostnames)
 When it was performed
 Version of Test tool and application under test
 Allows for posterity
 Allows for trending
Copyright © 2012 Constant Contact Inc. 9

10. Lesson 5: Run Performance Tests Continuously
 Lobby for a dedicated performance environment
 Run tests in non-production CI environments if only for trending relative
measurements
 Run daily
Copyright © 2012 Constant Contact Inc. 10

11. Lesson 6: Create Visualizations for Perf Test Results
 Charts, Charts, Charts: Have a webpage dashboard
 Preferably having ability to query or filter to obtain customized results
 Charts, Charts, Charts: Email results of each performance test run
 Visualization Research:
 Splunk w/ DB Connect
 D3: http://shop.oreilly.com/product/0636920026938.do
 Fun Digression: playing with this at: http://codepen.io
Copyright © 2012 Constant Contact Inc. 11

12. References – The end…
 http://akka.io/
 http://commons.apache.org/math/
 http://hc.apache.org/httpcomponents-client-ga/
 Slides available at: https://www.slideshare.net/jweden/rest-
performance-testing
Copyright © 2012 Constant Contact Inc. 12