Performance Testing REST APIs


Published on

Lessons learned from Rest Performance Testing at Constant Contact

Published in: Technology
  • @Burak Say Thanks for your comment. It's Apache HTTP Client that I tied in with Akka. However, lately I've been getting some really good results with SOASTA if you are looking for a COTS solution.
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello Jason, thanks for posting this slide and sorry if it's been too long ago. What tool you are using is not clear in the slide, maybe a missing logo or slide? I would love to figure out what you used (I can make an educated guess as, but really not clear in the slides)
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Performance Testing REST APIs

  1. 1. Performance Testing REST APIsSelenium Conference 2013Jason Weden @jwedenPrincipal Quality EngineerConstant Contact, WebServicesCopyright © 2012 Constant Contact Inc. 1
  2. 2. Who We Are What do we do? Expose our internal web services externally to developers andpartners via an API for their own applications. Fun Digression: Very easy to get up and running with signing into API usingMashery and using our console to hit the REST APICopyright © 2012 Constant Contact Inc. 2
  3. 3. Problem Domain System Under Test: HTTP REST APIs Hitting the REST API with concurrent connections Defining your Performance Test approach Getting & Sharing results – performance metricsCopyright © 2012 Constant Contact Inc. 3
  4. 4. Lesson 1: Be DRY Why in the heck repeat yourself? You already have the logic in your functional tests for REST calls soreuse that logic for your performance tests Reuse your data-seeding (e.g. for tests that have GET calls) Tools like LoadRunner and Jmeter have their place but They’re creating blocking connections that reduce scalability You are wet – you’re rewriting logic you already wrote in your functionaltests You confine the creation, execution, and general knowledge of theperformance testing arena to one or more toolsmithsCopyright © 2012 Constant Contact Inc. 4
  5. 5. Lesson 2: Choose Carefully Choose your http client library with care Non-blocking Scale-up Seriously consider Akka (if on the JVM) Non-blocking but with high throughput Scales to multiple processors and prevents need to deal with shared state More on this later Other Examples on the JVM of non-blocking http clients (Scala) – Spray – Akka-based (Java) -- Jetty 9 (Java) -- NettyCopyright © 2012 Constant Contact Inc. 5
  6. 6. Lesson 2: Choose Carefully (cont.) – Our Choice… Written in Java but takes advantage of Akka for concurrency Integrated into test framework for everyone to use 3rd Party libraries Apache HttpClient library Apache HttpCore library Akka concurrency framework Apache Commons Math3 library High performance Easy to use programmatic API – small learning curve Developers can go in and create performance tests Useful for any http functional OR performance testing Even functional tests have performance metrics loggedCopyright © 2012 Constant Contact Inc. 6
  7. 7. Lesson 2: Choose Carefully (cont.) – Our Choice… Found errors & significant bottlenecks Comparison with JMeter and LoadRunner Preliminary tests show faster than JMeter (even with our 8threads vs JMeters 1500) All performance tests can be checked into source control No dependency on toolsmiths for HTTP performance testing Performance testing experience shared across the team Likely faster than JMeter and LoadRunner -- still to bescientifically determined Ability to tweak our home-grown solution vs. closed sourcepurchased solutions. (e.g. lots of concurrency properties totweak)Copyright © 2012 Constant Contact Inc. 7
  8. 8. Lesson 3: Devise a Performance Test Plan What is your SLA? (e.g. avg. response time, avg. requests/sec) What kind of performance test? Options: Profile of production performance. Do the followingconcurrently: X amount of POSTs to endpoint 1 Y amount of GETs to endpoint 2 (data seeding needed) Z amount of DELETEs to endpoint 3 (data seeding needed) One type of Request: x amount of GETs Longevity – run test above for x amount of time What metrics will be recorded (client vs server)? How will results be captured and reported? Have sign-offs on your test plan (e.g. in a wiki section)Copyright © 2012 Constant Contact Inc. 8
  9. 9. Lesson 4: Capture Perf. Test Results in a Database Capture: Performance Metrics Client: min/max/avg response time, standard deviation, avgrequests/second, percentiles, request failures Server: cpu, memory, open db connections, thread count, consider app.profiling Test tool used and version Test name and description Freeform comments Concurrency level In what environment was test performed (server hostnames) When it was performed Version of Test tool and application under test Allows for posterity Allows for trendingCopyright © 2012 Constant Contact Inc. 9
  10. 10. Lesson 5: Run Performance Tests Continuously Lobby for a dedicated performance environment Run tests in non-production CI environments if only for trending relativemeasurements Run dailyCopyright © 2012 Constant Contact Inc. 10
  11. 11. Lesson 6: Create Visualizations for Perf Test Results Charts, Charts, Charts: Have a webpage dashboard Preferably having ability to query or filter to obtain customized results Charts, Charts, Charts: Email results of each performance test run Visualization Research: Splunk w/ DB Connect D3: Fun Digression: playing with this at: http://codepen.ioCopyright © 2012 Constant Contact Inc. 11
  12. 12. References – The end… Slides available at: © 2012 Constant Contact Inc. 12