• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

Performance Testing your Kuali Student Product

on

  • 990 views

Faced with two critical goals and low resources, the Kuali Student Test Team developed and implemented a test automation framework based on an open source tool ? Tsung and Amazon?s Elastic Compute ...

Faced with two critical goals and low resources, the Kuali Student Test Team developed and implemented a test automation framework based on an open source tool ? Tsung and Amazon?s Elastic Compute Cloud (EC2). This automation framework netted success right away as it allowed the team to identify and regress critical performance issues through out development.

Kuali developers and test engineers, and technical staff from implementing institutions will learn how industry best practices for performance testing were applied to Kuali Student and will walk away with guidance on how to setup and run an open source performance testing tool to support their needs (live demonstration included).

Statistics

Views

Total Views
990
Views on SlideShare
990
Embed Views
0

Actions

Likes
0
Downloads
8
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Force Field Diagram is used to weigh up or lay out the points for and against a potential action or task

Performance Testing your Kuali Student Product Performance Testing your Kuali Student Product Presentation Transcript

  • open source administration software for education next generation student system
  • Agenda
    • Introductions
    • The Challenge
    • Approach
    • Performance Automaton Strategy
    • Picking a tool
    • Application Results
    • Demo
    • Next Steps
    • Lessons Learned
    • Practices that work for us
    open source administration software for education 2
  • Introduction
    • Kyle Campos : Performance Engineer for Kuali Student. 11 years experience testing and building automation frameworks, both functional and performance, for web, unix and mobile applications. QA Architect for several startups building processes, tooling and test automation frameworks.
    • Ben Clark , QA Manager for Kuali Student. 12 years of experience in both the computer software and banking industries. Have lead performance testing efforts using off-the-shelf and home grown solutions. ASQ Certified Six Sigma Black Belt and Software Quality Engineer. Earned a B.A. from the University of Montana.
    open source administration software for education 3
  • The Challenge: Performance Testing KS
    • Lacked common definitions of the space
    • Poorly defined requirements
    • Lacked institutional system environment information
    • No Kuali Student profile information (how many courses over a year, cycle info, user info, etc)
    • No test machines for perf testing
    • Lacked tools for the job
    • No qualified resources
    open source administration software for education 4
    • Experience
    • Management support
    • Implementor User Group support
    + Forces for achieving success - Forces against achieving success
  • The Approach open source administration software for education 5
    • Action Plan
    • Hire experienced performance engineer
    • Utilize Functional Council and IUG to obtain requirements and data
    • Define the performance automation strategy
    • Utilize cloud technologies for test infrastructure
    • Pick a tool that meets the need
    • Lacked common definitions of the space
    • Poorly defined requirements
    • Lacked institutional system environment information
    • No Kuali Student profile information (how many courses over a year, cycle info, user info, etc)
    • No test machines for perf testing
    • Lacked tools for the job
    • No qualified resources
  • Performance Automation Strategy
    • Develop test suites based on critical features list
    • Benchmark tags & measure performance gain/loss
    • Add suites as features are developed
    • Use usage targets provided by universities
    open source administration software for education 6
  • Open source tools – lots out there! open source administration software for education 7 Tool Description Pros Cons JMeter A java desktop app designed to load test functional behavior and measure performance
    • HTTP protocol Support
    • Documented
    • List Support
    • Didn ’ t work with GWT/Ajax
    httperf A software tool which is able to simulate load situations for web services, and calculates performance metrics against the results
    • Familiarity
    • Robust
    • Extensible
    • HTTP protocol Support
    • Documented
    • List Support
    No dynamic variable support The Grinder Java load testing framework that can run a distributed test using many load injector machines
    • Flexible
    • Extensible
    • Documented
    • List Support
    Requires Java expertise Tsung Built for high transaction Env
    • Built for high trans env
    • Extendable
    • HTTP protocol Support
    • Documented
    • Dynamic variable support
    Not being used across other KS products Homegrown Build a tool in-house to test performance Total control
    • Expensive
    • Time consuming
    • Proprietary
    • Reinventing the wheel
  • Tool Choice - Tsung
    • Tsung XML API is a homegrown Ruby framework that allows for more rapid load test development, maintenance and execution
    • Load Framework is tagged and released as validated against milestone releases
    • Can easily configure custom test suites and load profiles
    open source administration software for education 8 Tsung + Tsung XML API (Ruby) = Load Automation Framework
  • Tool Choice - Tsung open source administration software for education 8
  • Test Design
    • Test layer doesn ’ t generate any XML, all XML generated by API calls
    • Test layer environment agnostic (standalone/embedded)
    • Tests are added to suites and given a probability to run
    • API docs dynamically created through rdoc
    • Test configured & launched by homegrown utility
    open source administration software for education 6
  • EC2 Network Diagram open source administration software for education 9
  • Application Results - Key Metrics
    • Concurrent sessions
      • driven by launched sessions/min
    • Mean transaction times
      • feature defined transactions
      • highest 10sec mean time
    • Iterative growth changes (linear?)
    • System resources
      • cpu, mem
    open source administration software for education 10
  • Application Results open source administration software for education 10
    • Data collected into Numbers(excel) for calculations
    • Published on wiki
    • Demo
  • Demo
    • Driver Demo
    open source administration software for education 11
  • What ’ s Next?
    • Tuning guide documentation
    • Configuration
    open source administration software for education 12
  • Example problem walkthrough
    • During Create Proposal load test found 4 concurrent sessions
      • upped to 16-18 sessions by upping thread pool
      • Way too many threads being consumed per session
    • Several rounds of debugging through configuration changes
    • Traced bottleneck to disabling of Rice caching. Resolved after 5 months
    open source administration software for education 13
  • Lessons Learned
    • A proof of concept may not have really proved anything
    • Tsung is great for flexible load sessions but difficult to manage concurrent sessions
    • Your load tool must have runtime dynamic variable support to work with GWT (Tsung does)
    open source administration software for education 14
  • Practices that work for us
    • Know your goals and requirements
    • When in doubt, benchmark
    • Review the test approach with entire team
    • A coding effort needs a coder; hire the best
    • Pick a tool that ’ s within your resource capability
    • Separate data, tests and tool
    • Utilize the cloud – it ’ s cheap and easy
    open source administration software for education 15
  • Resources
    • Automation testing home
      • https://wiki.kuali.org/display/KULSTG/Automation+Testing
    • Load Test Framework tester ’ s guide
      • https://wiki.kuali.org/display/KULSTG/Load+Testing+-+Tester%27s+Guide
    • Load testing results
      • https://wiki.kuali.org/display/KULSTG/Automation+Testing+Results
    • Contact
      • Kyle Campos / Skype: kylecampos_kuali / [email_address]
      • Ben Clark / Skype: benclark_kuali / [email_address]
    open source administration software for education 16 d
  • Questions? open source administration software for education 16 d
  • Timeline open source administration software for education 17 d Perf. Bug Detected Perf. Bug Fixed 12/09 01/10 02/10 03/10 04/10 05/10 06/10 07/10 08/10 12/10 11/10 10/10 09/10 Hire Completed Strategy in place Tool Review Code Complete Tool Testing Benchmark Benchmark Benchmark Benchmark M4 M5 M6 M7 M8 M9 Release 1.1 Release 1.0 Benchmark Benchmark M3 EC2 Implemented EC2 Use Approved