Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Performance and Scalability Testing with Python and Multi-Mechanize


Published on

Performance and Scalability Testing
with Python and Multi-Mechanize.


Published in: Technology
  • Be the first to comment

Performance and Scalability Testing with Python and Multi-Mechanize

  1. 1. Performance and Scalability Testing with Python and Multi-Mechanize Corey Goldberg   2011 web: email: [email_address] twitter: @cgoldberg
  2. 2. Multi-Mechanize <ul><ul><li>Open Source (LGPLv3) </li></ul></ul><ul><ul><li>Started Development in 2010 </li></ul></ul><ul><ul><li>Python 2 (2.6+) </li></ul></ul><ul><ul><li>Project Hosted on Google Code (SVN) </li></ul></ul><ul><ul><li>Website: </li></ul></ul><ul><ul><ul><li>Download </li></ul></ul></ul><ul><ul><ul><li>Code Samples / Script Dev Guide </li></ul></ul></ul><ul><ul><ul><li>Wiki </li></ul></ul></ul><ul><ul><ul><li>Discussion Group </li></ul></ul></ul>
  3. 3. What is Multi-Mechanize? <ul><li>&quot;Python framework for performance and load testing remote API's&quot; </li></ul><ul><li>Enables you to: </li></ul><ul><ul><li>run simultaneous client scripts to generate workload against a remote service </li></ul></ul><ul><ul><li>run transactions at an increasing series of load levels against your API and measure performance </li></ul></ul><ul><ul><li>simulate realistic and projected API usage </li></ul></ul>
  4. 4. Who is Multi-Mechanize For? <ul><ul><li>Developers </li></ul></ul><ul><ul><li>Testers </li></ul></ul><ul><ul><li>Ops </li></ul></ul><ul><li>... who need to test and tune the performance and scalability of a web site or API service </li></ul><ul><li>Assumptions: </li></ul><ul><ul><li>You know how to code in Python </li></ul></ul><ul><ul><li>You understand basic performance testing </li></ul></ul>
  5. 5. Testing for Performance & Scalability <ul><li>You have a remote API or website and need to: </li></ul><ul><ul><li>stress test server and network infrastructure </li></ul></ul><ul><ul><li>find application bottlenecks </li></ul></ul><ul><ul><li>expose errors under load </li></ul></ul><ul><ul><li>performance benchmarks </li></ul></ul><ul><ul><li>capacity planning </li></ul></ul>
  6. 6. Which API's Can You Test? <ul><li>You write the virtual user (client) scripts in Python, so any remote protocol/lib you want: </li></ul><ul><ul><li>Web/HTTP </li></ul></ul><ul><ul><ul><li>Raw HTTP (urllib2, httplib) </li></ul></ul></ul><ul><ul><ul><li>Web Forms (mechanize) </li></ul></ul></ul><ul><ul><ul><li>REST/SOAP/XMLRPC </li></ul></ul></ul><ul><ul><li>Sockets </li></ul></ul><ul><ul><li>Database, NoSQL, Cache </li></ul></ul><ul><ul><li>Any Python API </li></ul></ul>
  7. 7. Things I Like <ul><ul><li>Free </li></ul></ul><ul><ul><li>Runs on multiple platforms (nix, win) </li></ul></ul><ul><ul><li>Python code base </li></ul></ul><ul><ul><li>Scripts are in Python (batteries included, python libraries) </li></ul></ul><ul><ul><li>Scalable load generator </li></ul></ul><ul><ul><li>Automatic report generation </li></ul></ul><ul><ul><li>Aggregate stats (response times, throughput, percentiles) </li></ul></ul><ul><ul><li>Graphs (scatter, time-series) </li></ul></ul>
  8. 8. Inside Multi-Mechanize <ul><ul><li>Multi-process/Multi-threaded Test Harness </li></ul></ul><ul><ul><li>Virtual User Scripting in Python </li></ul></ul><ul><ul><li>Configurable Test Runner </li></ul></ul><ul><ul><li>Report Generator </li></ul></ul><ul><ul><li>Input: Virtual User Test Scripts, Config File </li></ul></ul><ul><ul><li>Output:  HTML Report with PNG images (graphs) </li></ul></ul>
  9. 9. Virtual User Scripting <ul><li>You create test scripts in Python for Multi-Mechanize to run.   </li></ul><ul><li>Each test script must implement a Transaction() class . This class must implement a run() method . </li></ul><ul><li>So a basic test script consists of: </li></ul><ul><li>class Transaction: </li></ul><ul><li>    def run(self): </li></ul><ul><li>        # do something here </li></ul>
  10. 10. Virtual User Scripting <ul><li>During a test run, your Transaction() class is instantiated once, and then its run() method is called repeatedly: </li></ul><ul><li>class Transaction: </li></ul><ul><li>    def __init__(self): </li></ul><ul><li>        # this gets called once </li></ul><ul><li>     </li></ul><ul><li>    def run(self): </li></ul><ul><li>        # this gets called repeatedly </li></ul>
  11. 11. Virtual User Scripting <ul><li>A full script that will issue an HTTP GET using urllib2: </li></ul><ul><li>     </li></ul><ul><li>import urllib2 </li></ul><ul><li>class Transaction(object): </li></ul><ul><li>    def run(self): </li></ul><ul><li>        url = '' </li></ul><ul><li>        urllib2.urlopen(url).read() </li></ul>
  12. 12. Virtual User Scripting <ul><li>web request with timer and assertions: </li></ul><ul><li>import mechanize </li></ul><ul><li>import time </li></ul><ul><li>class Transaction: </li></ul><ul><li>    def run(self): </li></ul><ul><li>        self.custom_timers = {} </li></ul><ul><li>        br = mechanize.Browser() </li></ul><ul><li>        br.set_handle_robots(False) </li></ul><ul><li>         </li></ul><ul><li>        start_timer = time.time() </li></ul><ul><li>        resp ='') </li></ul><ul><li> </li></ul><ul><li>        latency = time.time() - start_timer </li></ul><ul><li>         </li></ul><ul><li>        self.custom_timers['Example_Homepage'] = latency </li></ul><ul><li>         </li></ul><ul><li>        assert (resp.code == 200), 'Bad HTTP Response' </li></ul><ul><li>        assert ('Example Web Page' in resp.get_data()),  </li></ul><ul><li>            'Failed Content Verification' </li></ul>
  13. 13. Test Configuration <ul><li>sample config file: </li></ul><ul><li>  [global] </li></ul><ul><li>  run_time: 300 </li></ul><ul><li>  rampup: 300 </li></ul><ul><li>  console_logging: off </li></ul><ul><li>  results_ts_interval: 60 </li></ul><ul><li>   </li></ul><ul><li>  [user_group-1] </li></ul><ul><li>  threads: 10 </li></ul><ul><li>  script: </li></ul><ul><li>   </li></ul><ul><li>  [user_group-2] </li></ul><ul><li>  threads: 5 </li></ul><ul><li>  script: </li></ul>
  14. 14. Running a Test
  15. 15. Results Reports
  16. 16. Results Reports (Stats) <ul><ul><li>test summary </li></ul></ul><ul><ul><li>transaction timers </li></ul></ul><ul><ul><li>custom timers (from instrumented client code) </li></ul></ul><ul><ul><li>time-series/interval data </li></ul></ul><ul><ul><li>counts </li></ul></ul><ul><ul><li>rate/throughput </li></ul></ul><ul><ul><li>response times </li></ul></ul><ul><ul><ul><li>average, min, max, stdev </li></ul></ul></ul><ul><ul><ul><li>percentiles (80th, 90th, 95th) </li></ul></ul></ul>
  17. 17. Results Reports (Graphs) <ul><li>response time scatter plot: </li></ul>
  18. 18. Results Reports (Graphs) <ul><li>throughput time-series plot: </li></ul>
  19. 19. Results Reports (Graphs) <ul><li>latency time-series plot: </li></ul>
  20. 20. The End <ul><li>visit: </li></ul>