This document discusses Viewpoint's approach to web API performance testing. It outlines three key checkpoints: (1) ensuring performance during agile sprints through design reviews and trend monitoring, (2) integrating and testing components from different teams, and (3) performing full regression testing before release. It also defines different types of performance testing and describes the tools and processes used, including load testing with Visual Studio, tracking performance metrics, and using dashboards to socialize goals.

1. Viewpoint Construction Software
QA Overview Series
Web API Performance

2. Presentation Overview
• Performance Checkpoints
– We list the top 3 instances of QA Performance
Testing and then break each one down.
• Performance Testing Definition
– We detail what Performance, Load, Stress testing
means here at Viewpoint.
• Tools and Processes
– We show the tools and process that we use for
performance testing.

3. Viewpoint Performance Checkpoints
• Agile Scrum Teams
– As part of each Scrum team QA ensures the team builds
with performance in mind through participation in design
meetings and build->build performance trending.
• Component Integration Milestones
– Components built by different teams are merge together
for integration tests which include end-to-end
performance testing.
• Pre-Release Regression
– As part of the final release the QA team works to provide a
full performance regression on release comparable
hardware.

4. Agile Scrum Teams
• Each Scrum Team has QA Resources assigned to
that team focused on Functionality, Security and
Performance.
• QA ensures that the design and implementation
of each product supports performance testing.
• QA provides per-build (at least 1 per 2 week
sprint) trends for common functionality to
ensures no performance impact from new work.

5. Component Integration Milestones
• As part of the Product Release Plan several
integration milestones will be identified and
QA will participate to ensure that
performance is reviewed and communicated
to all teams.
• Performance issues found during these
integrations will be marked as Bugs and QA
will work to ensure they are addressed and
retested before new work begins.

6. Pre-Release Regression
• Prior to a final release the QA Team will rerun
the entire performance test suite against each
Release Candidate.
• This performance profile will be made
available to the management team to assist in
the go-no-go decision.

7. Performance Testing Definitions
• Load Testing
– Models realistic usage scenarios for existing and
future customers to vet new Features.
• Volume Testing
– Focuses on high volume to track how the application
handles paging, chunking, retries and timeouts.
• Stress Testing
– Focus on “burst/spike” scenarios and tracks how the
system degrades with “too many users.”

8. Load Testing
• Focuses on “How Much” instead of “How Fast” and
models realist user scenarios, roles and functions.
• Goal is ensuring existing/future customer load is
supported with planned feature or system changes.
• Example Scenarios
– 300 concurrent users with 5% performing Admin
Functions.
– End of Month Report Generation – spiked on top of 300
User Load.
– V6 Ad-hoc Report Run – spiked on top of 300 Users Load

9. Volume Testing
• Focuses on high volume and does not need to
represent “real” usage. Typically used to simulate large
sets of data returned from the Database.
• Goal is ensuring application handles paging, chunking
of large files, retries and timeouts correctly.
• Example Scenarios
– Admin attempting to perform User Management on a
system with 10,000 users.
– Report Generation against a system with 100 times the
data of our largest customer.
– User attempting to upload and retrieve huge (multi-gig)
files.

10. Stress Testing
• Focuses on modeling “burst/spike” scenarios that
measure how the system behaves with “too many
users”.
• Goal is ensuring application degrades gracefully and
then successfully recovers.
• Example Scenarios
– Track user experience when system is overloaded.
• Verify each user was directed to a custom “System is Busy” Page.
• Verify the users browser did not hang and pages timed-out nicely.
– Verify system recovers correctly
• Ensure no services or jobs stopped responding
• Verify the users data was not corrupted or locked from future
access.

11. Performance Tools and Processes
• Microsoft Visual Studio Ultimate
– Viewpoint has invested in Microsoft’s complete Web Load and
Performance Test solution. This provides out of the box
capabilities as well as rich customization to support all of
Viewpoints needs.
• MSDN Overview on Performance Testing Applications
• Industry Standard Technologies
– Viewpoint has focused on a core set of standard technologies
which have enabled a tight Dev/QA integration.
• Performance Trends
– QA tracks performance metrics and trends starting as early in
the development cycle as possible which enables us to quickly
isolate performance issues and optimizations prior to release.

12. Visual Studio Ultimate
• Visual Studio Ultimate provides a complete suite of tools to enable
Performance Testing across the entire range of Viewpoint Services
• Web
– Visual Studios Web Performance and Load Testing Framework is an industry
standard in testing Web Applications and API’s
– Custom Adapter Framework built on top of the standard tools gives us the
ability to build a modern and maintainable automation framework customized
to our applications.
• Mobile
– Leveraging Visual Studios ability to record, parameterize and playback (via
multiple cloud test agents) gives us the power to simulate many 1000’s of
simultaneous mobile users.
• Windows
– Microsoft’s Coded UI automation framework allows us to control our own
(and 3rd party) windows applications to create very customized cutomer
workflows and play them back at scale.

13. Industry Standard Technologies
• Standard Data Access Layer
– By adopting oData as its secure web wrapper to our data,
QA was able to leverage this for data setup/cleanup for our
tests.
• REST API
– By provide REST interfaces for our public and private
services, QA is able to script and generate load very easily
against the majority of our application.
• Microsoft Stack – SQL Server, .Net, Windows Server
– With this stack we have the ability to get detailed
performance metrics from all layers of the system in a
standard format and relate that data to a specific test or
test run.

14. Performance Trends
• Visual Studio Performance Report Server
– Visual Studio gives us the ability to capture vast
amount of data (application, hardware, network) for
each of our tests runs and store it in a SQL Database
for analysis and comparisons.
• Web Dashboards
– To socialize our performance goals and how we are
meeting them across Sprints, Integration Milestones
and Releases we use a series of Web Dashboards.
• These dashboards are made available to the teams and
upper management so everyone is on the same page.