The document discusses testing in the oil and gas software industry. It focuses on testing PetroVR, an oil and gas business simulation software. Some key points: - Software in this industry handles large amounts of complex data and functional knowledge. Testing requires understanding specialized technical vocabulary and domain knowledge. - PetroVR is a comprehensive simulation suite that integrates knowledge from engineering, economics, planning, and finance. It has over 5,000 classes and 80,000 methods. - Testing includes reviewing documentation, defining test cases, executing tests, and domain experts performing ad-hoc testing to identify unintended behaviors. Testers require experience to understand statistical results and domain terminology.

1. 13
March 2011
The Magazine for Professional Testers
printed in Germany
print version 8,00 €
Testing @ Domains –
free digital version
How does Finance, Automotive,
Medical etc test?
Do we have to take care of the domains?
www.testingexperience.com
l
ia
ec
s Sp
ce
en
er
nf
Co
ISSN 1866-5705
© Alexander Raths - Fotolia.com

2. © Andrei Merkulov - Fotolia.com
“Testing PetroVR, a software solution for the Oil & Gas
Market” or “Testing in the Oil & Gas Market“
by Leandro Caniglia, Damian Famiglietti y Ernesto Kiszkurno
Technology is widely used in the oil & gas industry. Both software 65% of its methods take no arguments, 91% one at most.
packages and custom developments share unique features and
terminology that impact the way testing is done. Some of these Conventional indicators poorly reflect the scope of functionality.
characteristics are: For instance, the number of classes and LOC looks moderate. In
contrast, the suite supports interoperable modeling of enginee-
• The leading role of data. Software has to handle large
amounts of data, normally processing a significant set of in- ring, planning, production, risk assessment, financial and decis-
puts into an even larger number of outputs which contain ion analysis.
sensitive and valuable information. Data environments are
highly complex and interconnected. Product development is highly dynamic. Three annual releases
add on average 10 substantial enhancements plus 300+ improve-
• The complexity and extent of functional knowledge needed
to operate it. The software may contain complex calculati- ments each. A strong refactoring culture keeps the overall size of
ons, make use of statistical models for simulations or projec- the code fairly constant over time.
tions, and interact with multiple systems that feed its pro-
cesses with critical data. The development model is essentially agile: small team (5½ full-
time smalltalkers), ubiquitous domain language, short iterations,
• Specificity of the technical vocabulary used. This is seen both
in the dialog with the different types of users and also in the shared code ownership, pair programming, exhaustive peer revi-
software itself. sion, constant refactoring, continuous integration.
At the same time, if the software is to be installed at different cli- Automatic testing is rigorously applied: 7,000 unit tests are a pre-
ents and on a significant number of computers with heterogene- condition of daily integration, covering 80% of existing methods.
ous environments, the complexity of the development and of the
required quality levels increase. Documentation policies depart from agile values in that imple-
mentation-independent abstractions are extensively discussed
The product and the development process in 250 documents. This is often rendered necessary by the frag-
PetroVR is a comprehensive business simulation suite for the mentation of the available information; thus completed, the
upstream oil & gas industry. It integrates knowledge from dif- knowledge base provides developers, testers and domain experts
ferent domains: a) Engineering: reservoirs, drilling, facilities, flu- with a coherent background of mutual understanding.
ids, production and injection, routing and pipelines, resources,
maintenance; b) Economy: capital and operational expenditures,
The quality control process
depreciation; c) Planning: multi-decade scheduling of interde- The quality control process has control activities for every delive-
pendent activities influenced by inflation, learning and uncertain rable of the development process: specification documents, on-
events, and d) Finance: portfolio, efficient frontier. line help, release notes and executable code. The process includes
5 main activities:
A rule-based simulation engine supports hypothesis confirmati-
1. Specification document review
on and provides insights into emergent behavior. A Decision Ana-
lysis approach conjugates scheduling, risk modeling, sensitivity, 2. Test case definition
scenarios and finance. 3. Test case execution
4. Help & release notes review
PetroVR was written in Smalltalk in 1996 and rewritten in 2005-6.
It comprises 5,400 classes, 80,000 methods and 480,000 lines of 5. Ad-hoc testing performed by domain experts
code (LOC). Design and coding styles conform to well-known best
practice patterns, exhibiting excellent complexity metrics; e.g.,
30 The Magazine for Professional Testers www.testingexperience.com

3. Testers perform a review of specification documents in order to by business analysts in real-life conditions. Experts test the appli-
become familiar with the functionality to be tested. This activity cation with their specific problems in mind.
ensures the testability, completeness and consistency of the spe-
cification. During this activity testers, developers and domain ex- Experience has shown that placing this type of testing at the end
perts exchange opinions about the product and the specification of the quality control process allows detection of unintended be-
documents. In other domains this kind of activity is not necessary haviors that might otherwise go unnoticed. Note however that
because the functionality being tested is easily learned, even by real-life cases tend to be overly large and complex. In order to in-
less experienced testers. corporate this contribution into a systematic coverage, professio-
nal testers can reduce these cases to isolate emergent issues and
Test case definition comprises two levels: GUI testing and simula- transfer them to a controlled environment.
tion testing. GUI testing in this context is no different from that
performed in other domains. Simulation testing is a more com-
Some conclusions
plex task and is critical for the quality of the product. Over more than three years some interesting conclusions have
been drawn:
Less experienced testers usually work on GUI testing as a way to
learn about the product. Simulation testing requires more expe- The first concerns development. Whether completely error-free
rienced testers. In order to find unexpected behaviors during this software is attainable or not, it is a fact that it cannot be achie-
particular kind of testing, a tester has to be able to interpret stati- ved without traditional testing. Even if we have invested time
stical results and to understand specific terminology. and energy in carefully defining the development processes and
creating automatic tests that reach a high level of coverage, the
A divide-and-conquer strategy is used in simulation testing. Test involvement of a testing team remains essential.
cases are grouped by objects and their features. This strategy
allows two things: (1) to narrow down the model complexity to Secondly, project managers should believe in devoting time to
smaller units, and (2) to have a clear idea of how much functio- training testing teams in domain-related subjects. They should
nality is covered. After defining the cases for each root object, it is focus not only on selecting a proficient development team but
necessary to consider their combinations and discard those that also on securing the participation of a testing team with a sound
are redundant (or not relevant). Here the tester’s domain know- knowledge of the domain.
ledge is critical because it is impossible to test every combination
of features and objects. Third, investing in knowledge management is fundamental. The
corpus of knowledge in this type of industry is not static: it inevi-
The execution activity involves data preparation, model executi- tably undergoes constant change, and so do work teams. A good
on and result analysis. Data preparation and result analysis are knowledge management ensures that this ongoing renewal of
the most important tasks. both information and human resources does not result in loss of
knowledge for the project.
Data preparation consists of defining models in a way that de-
fined test cases are covered. These models should be complex Fourth, automatic testing can eliminate recidivism. Random
enough to test the functionality (test cases) and simple enough to samples of test cases showed that 0% of already-addressed fai-
easily detect errors. Unlike tests in other domains, testers spend lures reappear. (2% of functional tests that originally passed fail
most of their time designing the models as these constitute a lib- one year later.)
rary of knowledge that will be useful in performing tests not only
in this release, but also in upcoming ones. Finally, in a domain so characterized by the need for analytical
skills, the role that testers must assume in the process cannot be
Analyzing the results to determine whether they are correct is limited to early detection of failures. The active involvement of
hard. It involves the manipulation of a large number of formu- testers in exchanges with developers becomes indispensable, as
las and algorithms. A key issue here is the way testers document their feedback may even shed light on design issues.
their findings. It is necessary to document not only the data used
but also the calculation method involved. This is the only way the We are indebted to Diego Seguí for his assistance in putting this
developer can follow the computations in order to reproduce the paper together.
behavior.
Additionally, a regression test is executed in order to ensure back-
compatibility. This is accomplished by running models created
with previous versions of the product. The results obtained with
the current release must be equal to those of previous ones.
Help & release notes review is straightforward. As in any complex
and highly specific software product, technical documentation is
essential for users. Writing style, completeness, formula corres-
pondence between software and documentation are some of the
validation criteria used in this type of review.
The last activity of the quality control process is the ad-hoc tes-
ting carried out by domain experts. This step is usually performed
www.testingexperience.com The Magazine for Professional Testers 31

4. > biography
Leandro Caniglia
Your Ad here
has worked at Caesar Systems
since 2001 and serves as chief
technologist and director of
software development. For
more than a decade prior to
joining Caesar Systems, Ca-
niglia worked as a Smalltalk
consultant for several compa- www.testingexperience.com
nies in Argentina, Brazil and
Chile. He was professor at the
University of Buenos Aires for
more than 20 years. Caniglia
has also worked as a researcher in the CONICET, the official
office for scientific research in Argentina. In 1997, he foun-
ded the user group SUGAR. He has a Ph.D. in Mathematics
and has published extensively on Computational Algeb-
raic Geometry. In 2007 Caniglia became President of FAST,
the organizing board for the Annual Argentine Smalltalk
Conference.
Damián Famiglietti
has worked at Pragma Consul-
tores since 2000. He is a Qua-
lity Assurance expert. He has
10 years of solid professional
experience in Software Quali-
ty Assurance. Damián has led
testing teams in several pro-
jects of different industries.
He holds a Systems Analyst
degree from the University of
Buenos Aires (Argentina).
Ernesto Kiszkurno
has worked at Pragma Consul-
tores since 1996 and became
partner in 2007. He serves as
director of Research & Deve-
lopment and is a Quality As-
surance expert. Kiszkurno has
15 years of solid professional
experience in the areas of soft-
ware engineering, process im-
provement, project manage-
ment and quality assurance.
His main activity in recent
years has been the training, leadership and management
of testing teams at multiple companies and in various in-
dustries. He holds a Bachelor of Computer Science degree
from the University of Buenos Aires (Argentina) and a Mas-
ter in Business Administration degree from the Torcuato
Ditella University. He has also been vice president for His-
panic America of the Software Testing ISTQB Qualifications
Board since 2008. For more information, please visit www.
ernestokiszkurno.com.ar.
32 The Magazine for Professional Testers www.testingexperience.com