Performance and Scalability Testing with Python and Multi-Mechanize

18,708 views
18,503 views

Published on

Performance and Scalability Testing
with Python and Multi-Mechanize.

visit http://multimechanize.com

Published in: Technology
1 Comment
15 Likes
Statistics
Notes
  • Excellent tutorials, however instead of mechanize package, requests package could be used for demo. For a person, new to python and new to multi-mechanize tool, one may get confuse on using a totally different package called mechanize as it sounds like the name of this tool
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
18,708
On SlideShare
0
From Embeds
0
Number of Embeds
42
Actions
Shares
0
Downloads
144
Comments
1
Likes
15
Embeds 0
No embeds

No notes for slide

Performance and Scalability Testing with Python and Multi-Mechanize

  1. 1. Performance and Scalability Testing with Python and Multi-Mechanize Corey Goldberg   2011 web: goldb.org email: [email_address] twitter: @cgoldberg multimechanize.com
  2. 2. Multi-Mechanize <ul><ul><li>Open Source (LGPLv3) </li></ul></ul><ul><ul><li>Started Development in 2010 </li></ul></ul><ul><ul><li>Python 2 (2.6+) </li></ul></ul><ul><ul><li>Project Hosted on Google Code (SVN) </li></ul></ul><ul><ul><li>Website: multimechanize.com </li></ul></ul><ul><ul><ul><li>Download </li></ul></ul></ul><ul><ul><ul><li>Code Samples / Script Dev Guide </li></ul></ul></ul><ul><ul><ul><li>Wiki </li></ul></ul></ul><ul><ul><ul><li>Discussion Group </li></ul></ul></ul>
  3. 3. What is Multi-Mechanize? <ul><li>&quot;Python framework for performance and load testing remote API's&quot; </li></ul><ul><li>Enables you to: </li></ul><ul><ul><li>run simultaneous client scripts to generate workload against a remote service </li></ul></ul><ul><ul><li>run transactions at an increasing series of load levels against your API and measure performance </li></ul></ul><ul><ul><li>simulate realistic and projected API usage </li></ul></ul>
  4. 4. Who is Multi-Mechanize For? <ul><ul><li>Developers </li></ul></ul><ul><ul><li>Testers </li></ul></ul><ul><ul><li>Ops </li></ul></ul><ul><li>... who need to test and tune the performance and scalability of a web site or API service </li></ul><ul><li>Assumptions: </li></ul><ul><ul><li>You know how to code in Python </li></ul></ul><ul><ul><li>You understand basic performance testing </li></ul></ul>
  5. 5. Testing for Performance & Scalability <ul><li>You have a remote API or website and need to: </li></ul><ul><ul><li>stress test server and network infrastructure </li></ul></ul><ul><ul><li>find application bottlenecks </li></ul></ul><ul><ul><li>expose errors under load </li></ul></ul><ul><ul><li>performance benchmarks </li></ul></ul><ul><ul><li>capacity planning </li></ul></ul>
  6. 6. Which API's Can You Test? <ul><li>You write the virtual user (client) scripts in Python, so any remote protocol/lib you want: </li></ul><ul><ul><li>Web/HTTP </li></ul></ul><ul><ul><ul><li>Raw HTTP (urllib2, httplib) </li></ul></ul></ul><ul><ul><ul><li>Web Forms (mechanize) </li></ul></ul></ul><ul><ul><ul><li>REST/SOAP/XMLRPC </li></ul></ul></ul><ul><ul><li>Sockets </li></ul></ul><ul><ul><li>Database, NoSQL, Cache </li></ul></ul><ul><ul><li>Any Python API </li></ul></ul>
  7. 7. Things I Like <ul><ul><li>Free </li></ul></ul><ul><ul><li>Runs on multiple platforms (nix, win) </li></ul></ul><ul><ul><li>Python code base </li></ul></ul><ul><ul><li>Scripts are in Python (batteries included, python libraries) </li></ul></ul><ul><ul><li>Scalable load generator </li></ul></ul><ul><ul><li>Automatic report generation </li></ul></ul><ul><ul><li>Aggregate stats (response times, throughput, percentiles) </li></ul></ul><ul><ul><li>Graphs (scatter, time-series) </li></ul></ul>
  8. 8. Inside Multi-Mechanize <ul><ul><li>Multi-process/Multi-threaded Test Harness </li></ul></ul><ul><ul><li>Virtual User Scripting in Python </li></ul></ul><ul><ul><li>Configurable Test Runner </li></ul></ul><ul><ul><li>Report Generator </li></ul></ul><ul><ul><li>Input: Virtual User Test Scripts, Config File </li></ul></ul><ul><ul><li>Output:  HTML Report with PNG images (graphs) </li></ul></ul>
  9. 9. Virtual User Scripting <ul><li>You create test scripts in Python for Multi-Mechanize to run.   </li></ul><ul><li>Each test script must implement a Transaction() class . This class must implement a run() method . </li></ul><ul><li>So a basic test script consists of: </li></ul><ul><li>class Transaction: </li></ul><ul><li>    def run(self): </li></ul><ul><li>        # do something here </li></ul>
  10. 10. Virtual User Scripting <ul><li>During a test run, your Transaction() class is instantiated once, and then its run() method is called repeatedly: </li></ul><ul><li>class Transaction: </li></ul><ul><li>    def __init__(self): </li></ul><ul><li>        # this gets called once </li></ul><ul><li>     </li></ul><ul><li>    def run(self): </li></ul><ul><li>        # this gets called repeatedly </li></ul>
  11. 11. Virtual User Scripting <ul><li>A full script that will issue an HTTP GET using urllib2: </li></ul><ul><li>     </li></ul><ul><li>import urllib2 </li></ul><ul><li>class Transaction(object): </li></ul><ul><li>    def run(self): </li></ul><ul><li>        url = 'http://www.example.com/' </li></ul><ul><li>        urllib2.urlopen(url).read() </li></ul>
  12. 12. Virtual User Scripting <ul><li>web request with timer and assertions: </li></ul><ul><li>import mechanize </li></ul><ul><li>import time </li></ul><ul><li>class Transaction: </li></ul><ul><li>    def run(self): </li></ul><ul><li>        self.custom_timers = {} </li></ul><ul><li>        br = mechanize.Browser() </li></ul><ul><li>        br.set_handle_robots(False) </li></ul><ul><li>         </li></ul><ul><li>        start_timer = time.time() </li></ul><ul><li>        resp = br.open('http://www.example.com/') </li></ul><ul><li>        resp.read() </li></ul><ul><li>        latency = time.time() - start_timer </li></ul><ul><li>         </li></ul><ul><li>        self.custom_timers['Example_Homepage'] = latency </li></ul><ul><li>         </li></ul><ul><li>        assert (resp.code == 200), 'Bad HTTP Response' </li></ul><ul><li>        assert ('Example Web Page' in resp.get_data()),  </li></ul><ul><li>            'Failed Content Verification' </li></ul>
  13. 13. Test Configuration <ul><li>sample config file: </li></ul><ul><li>  [global] </li></ul><ul><li>  run_time: 300 </li></ul><ul><li>  rampup: 300 </li></ul><ul><li>  console_logging: off </li></ul><ul><li>  results_ts_interval: 60 </li></ul><ul><li>   </li></ul><ul><li>  [user_group-1] </li></ul><ul><li>  threads: 10 </li></ul><ul><li>  script: search_simple.py </li></ul><ul><li>   </li></ul><ul><li>  [user_group-2] </li></ul><ul><li>  threads: 5 </li></ul><ul><li>  script: search_complex.py </li></ul>
  14. 14. Running a Test
  15. 15. Results Reports
  16. 16. Results Reports (Stats) <ul><ul><li>test summary </li></ul></ul><ul><ul><li>transaction timers </li></ul></ul><ul><ul><li>custom timers (from instrumented client code) </li></ul></ul><ul><ul><li>time-series/interval data </li></ul></ul><ul><ul><li>counts </li></ul></ul><ul><ul><li>rate/throughput </li></ul></ul><ul><ul><li>response times </li></ul></ul><ul><ul><ul><li>average, min, max, stdev </li></ul></ul></ul><ul><ul><ul><li>percentiles (80th, 90th, 95th) </li></ul></ul></ul>
  17. 17. Results Reports (Graphs) <ul><li>response time scatter plot: </li></ul>
  18. 18. Results Reports (Graphs) <ul><li>throughput time-series plot: </li></ul>
  19. 19. Results Reports (Graphs) <ul><li>latency time-series plot: </li></ul>
  20. 20. The End <ul><li>visit:   http://multimechanize.com </li></ul>

×