0
Performance and Scalability Testing with Python and Multi-Mechanize Corey Goldberg   2011 web:  goldb.org email:  [email_a...
Multi-Mechanize <ul><ul><li>Open Source (LGPLv3) </li></ul></ul><ul><ul><li>Started Development in 2010 </li></ul></ul><ul...
What is Multi-Mechanize? <ul><li>&quot;Python framework for performance and load testing remote API's&quot; </li></ul><ul>...
Who is Multi-Mechanize For? <ul><ul><li>Developers </li></ul></ul><ul><ul><li>Testers </li></ul></ul><ul><ul><li>Ops </li>...
Testing for Performance & Scalability <ul><li>You have a remote API or website and need to: </li></ul><ul><ul><li>stress t...
Which API's Can You Test? <ul><li>You write the virtual user (client) scripts in Python, so any remote protocol/lib you wa...
Things I Like <ul><ul><li>Free </li></ul></ul><ul><ul><li>Runs on multiple platforms (nix, win) </li></ul></ul><ul><ul><li...
Inside Multi-Mechanize <ul><ul><li>Multi-process/Multi-threaded Test Harness </li></ul></ul><ul><ul><li>Virtual User Scrip...
Virtual User Scripting <ul><li>You create test scripts in Python for Multi-Mechanize to run.   </li></ul><ul><li>Each test...
Virtual User Scripting <ul><li>During a test run, your Transaction() class is instantiated once, and then its run() method...
Virtual User Scripting <ul><li>A full script that will issue an HTTP GET using urllib2: </li></ul><ul><li>     </li></ul><...
Virtual User Scripting <ul><li>web request with timer and assertions: </li></ul><ul><li>import mechanize </li></ul><ul><li...
Test Configuration <ul><li>sample config file: </li></ul><ul><li>   [global] </li></ul><ul><li>   run_time: 300 </li></ul>...
Running a Test
Results Reports
Results Reports (Stats) <ul><ul><li>test summary </li></ul></ul><ul><ul><li>transaction timers </li></ul></ul><ul><ul><li>...
Results Reports (Graphs) <ul><li>response time scatter plot: </li></ul>
Results Reports (Graphs) <ul><li>throughput time-series plot: </li></ul>
Results Reports (Graphs) <ul><li>latency time-series plot: </li></ul>
The End <ul><li>visit:   http://multimechanize.com </li></ul>
Upcoming SlideShare
Loading in...5
×

Performance and Scalability Testing with Python and Multi-Mechanize

17,465

Published on

Performance and Scalability Testing
with Python and Multi-Mechanize.

visit http://multimechanize.com

Published in: Technology
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
17,465
On Slideshare
0
From Embeds
0
Number of Embeds
40
Actions
Shares
0
Downloads
132
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

Transcript of "Performance and Scalability Testing with Python and Multi-Mechanize"

  1. 1. Performance and Scalability Testing with Python and Multi-Mechanize Corey Goldberg   2011 web: goldb.org email: [email_address] twitter: @cgoldberg multimechanize.com
  2. 2. Multi-Mechanize <ul><ul><li>Open Source (LGPLv3) </li></ul></ul><ul><ul><li>Started Development in 2010 </li></ul></ul><ul><ul><li>Python 2 (2.6+) </li></ul></ul><ul><ul><li>Project Hosted on Google Code (SVN) </li></ul></ul><ul><ul><li>Website: multimechanize.com </li></ul></ul><ul><ul><ul><li>Download </li></ul></ul></ul><ul><ul><ul><li>Code Samples / Script Dev Guide </li></ul></ul></ul><ul><ul><ul><li>Wiki </li></ul></ul></ul><ul><ul><ul><li>Discussion Group </li></ul></ul></ul>
  3. 3. What is Multi-Mechanize? <ul><li>&quot;Python framework for performance and load testing remote API's&quot; </li></ul><ul><li>Enables you to: </li></ul><ul><ul><li>run simultaneous client scripts to generate workload against a remote service </li></ul></ul><ul><ul><li>run transactions at an increasing series of load levels against your API and measure performance </li></ul></ul><ul><ul><li>simulate realistic and projected API usage </li></ul></ul>
  4. 4. Who is Multi-Mechanize For? <ul><ul><li>Developers </li></ul></ul><ul><ul><li>Testers </li></ul></ul><ul><ul><li>Ops </li></ul></ul><ul><li>... who need to test and tune the performance and scalability of a web site or API service </li></ul><ul><li>Assumptions: </li></ul><ul><ul><li>You know how to code in Python </li></ul></ul><ul><ul><li>You understand basic performance testing </li></ul></ul>
  5. 5. Testing for Performance & Scalability <ul><li>You have a remote API or website and need to: </li></ul><ul><ul><li>stress test server and network infrastructure </li></ul></ul><ul><ul><li>find application bottlenecks </li></ul></ul><ul><ul><li>expose errors under load </li></ul></ul><ul><ul><li>performance benchmarks </li></ul></ul><ul><ul><li>capacity planning </li></ul></ul>
  6. 6. Which API's Can You Test? <ul><li>You write the virtual user (client) scripts in Python, so any remote protocol/lib you want: </li></ul><ul><ul><li>Web/HTTP </li></ul></ul><ul><ul><ul><li>Raw HTTP (urllib2, httplib) </li></ul></ul></ul><ul><ul><ul><li>Web Forms (mechanize) </li></ul></ul></ul><ul><ul><ul><li>REST/SOAP/XMLRPC </li></ul></ul></ul><ul><ul><li>Sockets </li></ul></ul><ul><ul><li>Database, NoSQL, Cache </li></ul></ul><ul><ul><li>Any Python API </li></ul></ul>
  7. 7. Things I Like <ul><ul><li>Free </li></ul></ul><ul><ul><li>Runs on multiple platforms (nix, win) </li></ul></ul><ul><ul><li>Python code base </li></ul></ul><ul><ul><li>Scripts are in Python (batteries included, python libraries) </li></ul></ul><ul><ul><li>Scalable load generator </li></ul></ul><ul><ul><li>Automatic report generation </li></ul></ul><ul><ul><li>Aggregate stats (response times, throughput, percentiles) </li></ul></ul><ul><ul><li>Graphs (scatter, time-series) </li></ul></ul>
  8. 8. Inside Multi-Mechanize <ul><ul><li>Multi-process/Multi-threaded Test Harness </li></ul></ul><ul><ul><li>Virtual User Scripting in Python </li></ul></ul><ul><ul><li>Configurable Test Runner </li></ul></ul><ul><ul><li>Report Generator </li></ul></ul><ul><ul><li>Input: Virtual User Test Scripts, Config File </li></ul></ul><ul><ul><li>Output:  HTML Report with PNG images (graphs) </li></ul></ul>
  9. 9. Virtual User Scripting <ul><li>You create test scripts in Python for Multi-Mechanize to run.   </li></ul><ul><li>Each test script must implement a Transaction() class . This class must implement a run() method . </li></ul><ul><li>So a basic test script consists of: </li></ul><ul><li>class Transaction: </li></ul><ul><li>    def run(self): </li></ul><ul><li>        # do something here </li></ul>
  10. 10. Virtual User Scripting <ul><li>During a test run, your Transaction() class is instantiated once, and then its run() method is called repeatedly: </li></ul><ul><li>class Transaction: </li></ul><ul><li>    def __init__(self): </li></ul><ul><li>        # this gets called once </li></ul><ul><li>     </li></ul><ul><li>    def run(self): </li></ul><ul><li>        # this gets called repeatedly </li></ul>
  11. 11. Virtual User Scripting <ul><li>A full script that will issue an HTTP GET using urllib2: </li></ul><ul><li>     </li></ul><ul><li>import urllib2 </li></ul><ul><li>class Transaction(object): </li></ul><ul><li>    def run(self): </li></ul><ul><li>        url = 'http://www.example.com/' </li></ul><ul><li>        urllib2.urlopen(url).read() </li></ul>
  12. 12. Virtual User Scripting <ul><li>web request with timer and assertions: </li></ul><ul><li>import mechanize </li></ul><ul><li>import time </li></ul><ul><li>class Transaction: </li></ul><ul><li>    def run(self): </li></ul><ul><li>        self.custom_timers = {} </li></ul><ul><li>        br = mechanize.Browser() </li></ul><ul><li>        br.set_handle_robots(False) </li></ul><ul><li>         </li></ul><ul><li>        start_timer = time.time() </li></ul><ul><li>        resp = br.open('http://www.example.com/') </li></ul><ul><li>        resp.read() </li></ul><ul><li>        latency = time.time() - start_timer </li></ul><ul><li>         </li></ul><ul><li>        self.custom_timers['Example_Homepage'] = latency </li></ul><ul><li>         </li></ul><ul><li>        assert (resp.code == 200), 'Bad HTTP Response' </li></ul><ul><li>        assert ('Example Web Page' in resp.get_data()),  </li></ul><ul><li>            'Failed Content Verification' </li></ul>
  13. 13. Test Configuration <ul><li>sample config file: </li></ul><ul><li>  [global] </li></ul><ul><li>  run_time: 300 </li></ul><ul><li>  rampup: 300 </li></ul><ul><li>  console_logging: off </li></ul><ul><li>  results_ts_interval: 60 </li></ul><ul><li>   </li></ul><ul><li>  [user_group-1] </li></ul><ul><li>  threads: 10 </li></ul><ul><li>  script: search_simple.py </li></ul><ul><li>   </li></ul><ul><li>  [user_group-2] </li></ul><ul><li>  threads: 5 </li></ul><ul><li>  script: search_complex.py </li></ul>
  14. 14. Running a Test
  15. 15. Results Reports
  16. 16. Results Reports (Stats) <ul><ul><li>test summary </li></ul></ul><ul><ul><li>transaction timers </li></ul></ul><ul><ul><li>custom timers (from instrumented client code) </li></ul></ul><ul><ul><li>time-series/interval data </li></ul></ul><ul><ul><li>counts </li></ul></ul><ul><ul><li>rate/throughput </li></ul></ul><ul><ul><li>response times </li></ul></ul><ul><ul><ul><li>average, min, max, stdev </li></ul></ul></ul><ul><ul><ul><li>percentiles (80th, 90th, 95th) </li></ul></ul></ul>
  17. 17. Results Reports (Graphs) <ul><li>response time scatter plot: </li></ul>
  18. 18. Results Reports (Graphs) <ul><li>throughput time-series plot: </li></ul>
  19. 19. Results Reports (Graphs) <ul><li>latency time-series plot: </li></ul>
  20. 20. The End <ul><li>visit:   http://multimechanize.com </li></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×