See more ways to improve application performance: https://www.castsoftware.com/use-cases/Improve-adm-quality This white paper presents a six-step Application Performance Modeling Process using software intelligence to identify potential performance issues earlier in the development lifecycle. Enriching dynamic testing with structural quality analysis gives ADM teams insight into the performance behavior of applications by highlighting critical application performance issues, especially when combined with runtime information. By adding structural quality analysis, ADM teams learn important information about violations of architectural and programming best practices earlier in the development lifecycle than with a pure dynamic testing approach. Structural quality analysis as part of the performance modeling process allows for fact-based insight into application complexity (e.g. multiple layers, dynamics of their interactions, complexity of SQL, etc.) and allows ADM managers to anticipate evolution of the runtime context (e.g. growing volume of data, higher number of transactions, etc.). The combined approach results in better detection of latent application performance issues within software. Resolving application performance issues early in the development cycle, these alerts help to not only save money but also prevent complete business disruptions. See more ways to improve application performance: https://www.castsoftware.com/use-cases/Improve-adm-quality

1. White Paper
6 Steps to Enhance Performance of
Critical Systems
Despite the fact that enterprise IT departments have invested heavily in
dynamic testing tools to verify and validate application performance and
scalability before releasing business applications into production, perfor-
mance issues and response time latency continue to negatively impact
the business. By supplementing dynamic performance testing with
automated structural quality analysis, development teams have the ability
to detect, diagnose, and analyze performance and scalability issues
more effectively. This white paper presents a six-step Performance
Modeling Process using automated structural quality analysis to identify
these potential performance issues earlier in the development lifecycle.

2. Page 2
www.castsoftware.com
6 Steps to Enhance Performance of Critical Systems
I. Introduction
Despite the fact that enterprise IT departments have invested heavily in
dynamic testing tools to verify and validate application performance and
scalability before releasing these business applications into production,
performance issues and response time latency continue to negatively
impact the business.
Application Development and Maintenance (ADM) teams often spot
performance issues in mission-critical applications during the dynamic
or “live” testing phase when an application is almost complete, and
theoretically “ready” for production. By the time they discover these
performance issues, it is too late to make the design or architectural
changes needed to address the issues without business disruption or
costly additional development cycles—resulting in significant delays and/
or business losses.
System-level structural quality analysis provides the ability to detect,
diagnose, and analyze performance and scalability issues. While
performance checks are still seen as the domain of dynamic testing (e.g.
“It does not make sense to solve performance problems by diving into
the code”), automated solutions to analyze and detect performance and
scalability issues based on structural quality analysis of the source code,
are emerging.
By supplementing dynamic performance testing with automated
structural quality analysis, development teams get early and important
information that might be missed with a pure dynamic approach, such
as inefficient loops or SQL queries. The combined approach results in
better detection of latent performance issues within application software.
This white paper presents a six-step Performance Modeling Process
using automated structural quality analysis to identify these potential
performance issues earlier in the development lifecycle. The paper also
presents cases studies to illustrate the proposed modeling process at
work.
II. Approach: Structural Quality Analysis of Source Code to Tackle
Performance Issues
Once upon a time, a seasoned software professional building advanced
military systems used to tell young developers this rather provocative
saying: “You should not optimize, but rather pessimize.”
The developers would laugh at him before finally understanding his
advice. He meant: Do not try to write a sophisticated and difficult
algorithm or query, but rather create a simple and functional one first.
Then optimize those that really need to perform at a very high speed.
Contents
I. Introduction
II. Approach: Structural Quality Analysis
of Source Code to Tackle Performance
Issues
III. Case Studies: Solving Performance Is-
sues with Structural Quality Analysis
IV. The Requirements of a Solid Structural
Quality Analysis Platform
V. Conclusion

3. Page 3
www.castsoftware.com
6 Steps to Enhance Performance of Critical Systems
His provocative advice should not lead developers to write “quick and
dirty” routines every time, only to optimize as an afterthought. Rather,
it should inspire them to think about performance from the outset. Just
like security, application performance should be taken seriously from the
beginning of the development lifecycle.
To achieve a performance perspective from the start of development, we
propose a six-step Performance Modeling Process that is in use today
by many professional developers and advanced ADM teams.
Performance Modeling Process
1. Identify high-level use cases: Focus on areas of high value such as
key functionalities and components (the performance sweet spot).
2. Run dynamic tests on transactions: Use different sets/ranges/sizes
of input data, and capture results/values. Some of these tests should be
intended to fail to expose performance issues.
3. Identify transactions with poor performance: Include use cases
where a performance hit/degradation is most pronounced.
4. Analyze the application source code: Use structural quality
analysis to identify specific poor performing transactions. (This step
has traditionally been performed manually with a high rate of mistakes;
however there are now tools that provide automation, so error rates are
virtually non-existent. We will discuss this more later.) Experience shows
that most bugs are due to poor coding standards or missing standard
requirements or best practices, which cannot be identified without
human intervention.
5. Identify violations of best practices: Determine violations of perfor-
mance best practices and performance coding standards (e.g. memory
leaks, resource leaks, poorly written SQL queries). Check compliance
to set baseline requirements or industry standard requirements (e.g.
performance standard requirements).
6. Fix the violations and re-test/repeat: Run dynamic tests on the
modified transactions again, and re-examine updated source code.
When used as an ongoing process during development of a structural
quality analysis platform combined with dynamic testing tools, ADM
teams can identify and eliminate performance issues before they reach
production, and with a high level of confidence and efficiency.
Following are two real-world examples where application development
teams have used this combined approach to fix or prevent performance
issues before they happen in production.
Highlight
The performance mod-
eling process allows
ADM teams to identify
and eliminate perfor-
mance issues before they
reach production, and
with a high level of con-
fidence and efficiency

4. Page 4
www.castsoftware.com
6 Steps to Enhance Performance of Critical Systems
Highlight
A structural quality
analysis solution capa-
ble of analyzing differ-
ent technologies such as
Java, XML, and SQL,
can understand how
each technology is inte-
grated through a par-
ticular framework
III. Case Studies: Solving Performance Issues with Structural Quality
Analysis
Case 1 - UPDATE Trigger Caused Major Troubles at a Global Travel
Company
At a global travel company, different travel providers and travel agencies
make reservations using a legacy system. Using mainframe applications
to manage the entire transaction has higher costs depending on its
duration, so the company decided to revamp all the reservation selection
routines for flights, hotels, and cars in Java EE. When the customer was
ready to buy, the Java EE application would call the mainframe to finalize
the transaction.
The application development and testing went well, but after putting
the system into production, the application had significant performance
latency issues that resulted in lost revenue since many customers
abandoned their transaction during processing. The ADM team was
forced to revert back to the legacy application and investigate the new
Java EE application.
The architects designed the Java EE application using Hibernate, Spring,
and Spring MVC and deployed on a Java EE 5 application server. The
team used the database as-is because of the legacy mainframe system,
and it could not be changed.
The team chose the architecture because it used well-known frameworks
with large communities, and permitted use of POJO (Plain Old Java
Object) to develop the application. In addition, Hibernate had features
to adapt to a specific legacy database, which would facilitate future
enhancements of the application. Furthermore, with this new Java EE
application, the company estimated a 30% reduction in the operational
cost of the mainframe system.
After testing the Java EE application and releasing it into production, the
team noticed a performance issue when a certain volume of transactions
occurred at one time (around 26 transactions per second).
Several days were spent to set-up an environment similar to the
production environment to simulate the transaction activity. The team
determined that it needed a structural quality analysis solution, which
was capable of analyzing different technologies such as Java, XML, and
SQL, and could understand how each technology is integrated through a
framework such as Hibernate, to help focus the investigation.
To reproduce the issue, the team simulated the number of transactions
that were resulting in performance issues to see what was happening in
the application and on the database. They saw abnormal activity on the
database due to an “on update” trigger that fired too frequently, which

5. Page 5
www.castsoftware.com
6 Steps to Enhance Performance of
Critical Systems
the architects kept in the database for use by other legacy applica-
tions. In turning on the Hibernate ‘show SQL property’ to see what was
happening, the team observed that the trigger was firing even if the data
had not changed.
This error was due to a specific parameter in Hibernate: select-before-
update on the entity that was set to false. When set to false, Hibernate
updated the table systematically. See Figure 1.
Figure 1 - UPDATE Trigger Firing
To fix the issue, the team simply needed to set select-before-update
to “true” so that when Hibernate selects the data from the table and
compares it, it performs an update only if the data are different.
The cost of this issue was estimated at about $400,000, which included
the sum of the transactions the company lost during the time period plus
the number of man-days lost to investigate and fix the issue. This does
not include impacts to the company’s reputation or other soft costs.
Using an automated structural quality analysis solution was instrumental
in solving the problem in this complex environment. In this example,
using a powerful Java EE framework like Hibernate to manage the
complexity of database transactions can be a great enabler; however, it
requires keen architectural skills to understand the ramifications of what
will happen in the back end. Similarly, it can be difficult to test all the
possible scenarios that might happen in reality. Structural quality analysis
filled in these critical knowledge gaps and provided speedy resolution to
the issue.

6. Page 6
www.castsoftware.com
6 Steps to Enhance Performance of Critical Systems
Case 2 - Critical SAP Transaction Suffers Huge Performance Hit
In a global chemical company, an enterprise-level implementation of SAP
ERP software manages the core business processes of the company.
The ADM team manages the system in a centralized company-managed
technical center, while end-users access the SAP applications around
the world.
To better meet the requirements of specific departments, multiple ADM
teams drive custom development to adapt standard applications, as well
as create new ones that extend functionality. Some of the applications,
which are not all defined as mission-critical, have several thousand users
and handle huge volumes of data daily. In some of these cases, the
amount of information is consistently large, and in others the volume of
information grows quickly before being processed, and then is removed
or archived.
As is commonly known, SAP is built on an RDBMS and uses many calls
to/from the database. In this SAP implementation, a custom designed
and developed application enabled the recording of technical data and
metadata, the calculations of new values based on previous information,
and the production of reports with statistics for technical managers.
The goal of this new application was to minimize the time spent record-
ing information by allowing a large number of employees to access
the system for management statistics, while mitigating the volume of
database calls.
Unfortunately, the development team for this application did not complete-
ly evaluate the quantity of information managed, and it did not realize that
the volume of data can grow very quickly in certain circumstances.
After some weeks in production, end-users began to complain about
abnormal response times for specific transactions in the new custom
application. The ADM team analyzed the production log files and it found
effectively abnormal execution times, up to 10 hours, for some transac-
tions connected to the application.
After using an automated structural quality analyzer, the team identified
the cause of the trouble. The performance defects were the conse-
quence of misuse of Open SQL statements regarding the volume of data
to process. The team found in some Open SQL queries, the addition
FOR ALL ENTRIES IN used without any check to control the internal
table content. As a result, the queries would end up performing a full
table scan, and thus cause severe latency issues, especially for very
large database tables.
In a few other cases, a SELECT - ENDSELECT statement had been used
instead of a SELECT INTO TABLE used in conjunction with a LOOP
AT statement. The SELECT - ENDSELECT worked as a loop fetching
a single record at a time, and caused a problem when this statement
Highlight
By using an automated
structural quality ana-
lyzer, the ADM team
identified performance
defects causing severe
latency issues

7. Page 7
www.castsoftware.com
6 Steps to Enhance Performance of Critical Systems
selected from large tables. The team revealed in its investigation on
the database that very big tables, with more than 1 million rows, were
common in the calls by the application. Figure 2 illustrates some of the
performance risk-laden transactions.
Figure 2 - SELECT Statement Errors
After the development team fixed these issues and retested, the situation
in production returned to normal, and response times decreased to less
than 3 hours for transactions with the largest volumes of data.
Even with the best test environment before production, unit tests and
integration tests are often not sufficient to prevent performance issues,
since load test cases devised to simulate the production environment
cannot address every possible scenario. Unfortunately, creating test
cases that are similar to transaction and data volumes used in production
is expensive and is often difficult to do in a short time window between
integration testing and production.
Therefore, one might conclude that the solution would be to perform
structural quality analysis in order to detect potential issues. However,
this technique is much more efficient when enriched with runtime infor-
mation, such as execution times, number of rows in database tables, etc.
Connecting both allows ADM teams to focus on the most critical results
that need to be fixed immediately.
IV. The Requirements of a Solid Structural Quality Analysis Platform
To tackle performance issues directly from source code before they
happen in production, it is necessary to analyze the application as a
whole, analyzing all layers of the application, especially when written
in different languages. System-level analysis is a requirement in most
IT domains as application layers are written in different programming
languages. For example, an application using a Java EE code-level
analyzer will only analyze the Java code and is unable to analyze the

8. Page 8
www.castsoftware.com
6 Steps to Enhance Performance of Critical Systems
SQL code that includes the dynamic SQL, the SQL stored procedures in
the database, and the table schema. Effective analysis of the application
should also take into account the framework information stored in XML
files. An effective structural quality analysis solution must also detect
violations of performance best practices or performance vulnerabilities,
so that the team is aware of the appropriate practices during develop-
ment.
Our experience has shown that to successfully implement the Perfor-
mance Modeling Process described earlier, it is important to use a
structural quality analytic approach that provides an end-to-end view of
the application—one that includes a system-level view of the applica-
tion’s transactions across all the technology layers.
Application owners, project managers, and ADM managers will gain
valuable insight from the information generated by structural quality
analysis, enabling them to address issues like the following:
• Manage and improve structural quality of applications with an
objective and data-driven approach
• Understand the risk impact of violations on specific modules and
systems
• Prioritize the violations to remediate
• Perform root cause analysis of production outages
• Quantify the technical debt being accumulated in applications
These types of issues can only be addressed with a structural quality
analysis platform that can relate performance vulnerabilities to known
transactions and the results from dynamic testing.
Highlight
To successfully imple-
ment the Performance
Modeling Process, it
is important to use
a structural quality
analytic approach that
provides an end-to-end
view of the application

9. V. Conclusion
Enriching dynamic testing with structural quality analysis gives ADM
teams insight into the performance behavior of applications by highlight-
ing critical performance issues, especially when combined with runtime
information.
By adding structural quality analysis, ADM teams learn important infor-
mation about violations of architectural and programming best practices
earlier in the development lifecycle than with a pure dynamic testing
approach. Structural quality analysis as part of the performance model-
ing process allows for fact-based insight into application complexity (e.g.
multiple layers, dynamics of their interactions, complexity of SQL, etc.)
and allows ADM managers to anticipate evolution of the runtime context
(e.g. growing volume of data, higher number of transactions, etc.).
The combined approach results in better detection of latent performance
issues within application software. Resolving these issues early in the
development cycle, these alerts help to not only save money but also
prevent complete business disruptions.
About the Authors
Jerome Chiampi, Product Manager, CAST
Manages mainframe and SAP application intelligence products at CAST,
researches software quality and best practices in legacy environments.
Frederic Kihm, Product Manager, CAST
Manages the Java EE software quality and application intelligence
products at CAST, author of innovative software risk ranking methodol-
ogy and tool.
Laurent Windels, Product Manager, CAST
Manages the implementation and deployment of CAST Application
Intelligence Platform in the development cycle.
About CAST
CAST is a pioneer and world leader in Software Analysis and Measure-
ment, with unique technology resulting from more than $100 million
in R&D investment. CAST introduces fact-based transparency into
application development and sourcing to transform it into a manage-
www.castsoftware.com
Europe 3 rue Marcel Allégot 92190 Meudon - France Phone: +33 1 46 90 21 00
North America 373 Park Avenue South New York, NY 10016 Phone:+1 212-871-8330
Questions?
Email us at contact@castsoftware.com