EuroSTAR Software Testing Conference 2012 presentation on Crafting our own Models of Software Quality by Henrik Emilsson. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/

1. Crafting Our Own Models of Software
Quality
Henrik Emilsson, Qamcom Research & Technology
www.eurostarconferences.com
@esconfs
#esconfs

2. Crafting Our Own Models of Software Quality
Henrik Emilsson
EuroSTAR 2012
Qamcom Research & Technology AB | www.qamcom.com | +46 31 721 1730

3. Abstract
In late 2010, thetesteye.com published a poster called Software Quality
Characteristics which was the result of trying to invent the best model of
quality characteristics. When we started this, there was only one realistic and
thorough model available – Bach’s CRUSSPIC STMPL – but we didn’t think
that this model was perfect for us. Instead we attacked this model and started
to question it in order to come up with a model that we thought were more
true and valid for us and in our context. In this talk I will describe what we did
during this interesting journey; a journey perhaps more important than the
result itself.

4. Snippet from http://literature.puertoricosupplier.com/023/CN22511.pdf

5. A short introduction to the poster…

6. Background & Context
"All models are wrong, but some are useful” (George E.P. Box)
If this is true, why aren't more models challenged? (Including Kaner's and
Bach's!)
You can create a model that is better in your context

7. Background & Context
The Test Eye – www.thetesteye.com
A collaborative blog about testing
Started in February, 2008, by Henrik Emilsson, Martin Jansson and Rikard Edgren
Later, Torbjörn Ryber and Henrik Andersson joined as contributors
Henrik, Martin & Rikard had been coworkers for more than 6 years
We discussed testing 24/7 (well, something like that…)
The blog became a way of keeping in touch when we split to three different
companies

8. Background & Context
Since 2007 we have all worked on different companies
Henrik as consultant
Rikard on a software company
Martin on a software/hardware company
We all used different quality models in different ways, e.g.
Henrik used information objectives and quality objectives in his work as a driver for
test planning
Rikard used Bach’s CRUSSPIC STMPL when writing and reviewing test specifications
By using it we identified difficulties and opportunities

9. The Journey
In 2008 Rikard used Bach’s CRUSSPIC STMPL; especially for detailed
functionality
It is a great model that inspires you to think for yourself
It’s not complete
Not too specific
Vague
Apply it in your context
Some areas were elaborated; some were on a too high level for us
Other models were too focused on metrics

10. Insights
We wanted more elaborations, Bach’s model vs. our model:

11. The Journey
Rikard started to extend Bach’s model – an extension written on paper
He then asked for help on extending the model;
Henrik and Martin joined
Rikard wanted to have the model on one page;
so Henrik and Martin did their best to extend it to two pages…
STMPL was not included from the start

12. Insights
Limit your model
Forced to focus on importance
Need to decide between merging or separating
We limited our model to one page, then when we discovered enough
important things that it became extended to two pages
We played with different dimensions in order to not break the boundary:
Font and font size
Formatting
Sentence lengths
Margins
By limiting the model and doing the above, we also thought more of the
meaning by examining synonyms

13. The Journey
Henrik added a new category “Charisma”
(This is sometimes referred to as Desirability but we did not know of that)
STMPL went back in after some hot discussions
Installability became IT-bility because there are more important things than
installation; we thought of quality characteristics an IT-department might value

15. The Journey
During the work with the model, we did a mix of
reading
thinking
using
discussing

16. Insights
Mix theoretical and empirical research
Theoretical research
Investigate previous models
Learn what is written about a subject e.g. usability
Look up definitions and the meaning of words and expressions
Empirical research
During your regular work, use what you learn and try to formulate quality
characteristics
Invent something by experiencing for real
If you find something important, see if it fits somewhere in the model
(e.g. Clarity came from health care project)
A mix of these methods over time was very good

17. The Journey
During the work with the model we used it with clients in their context
Revealed new categories
Improved existing categories
We discovered new categories by really thinking about the existing ones
applied in our context
Our Mnemonic became CRUCSPIC STMP
Scalability  Under Performance
Localization  Under Portability
Charisma has its own mnemonic: U SPACEHEADS

18. Insights
Understanding in your own context
Understanding in your own language
Nationality
Context (business language etc)
Understanding by applying it
There is always a better version of the model; not more true, but more useful
In order to be useful you need to be generic but specific in wordings

19. The Journey
In the middle of 2010, the model felt good enough
We wanted to incorporate James Bach and present this as an extended
CRUSSPIC STMPL
However, we decided to create our own model as much as possible;
we decided to rephrase all items on the list

20. Insights
It was actually good that we decided to create our own
This way we got rid of legacy words that we hadn’t written; instead we had to
come up with our own
It is important how you phrase things; in version 1.1 of the model, more items
will be phrased as questions
By putting your own words into it, you understand it better

21. Result
We published the model as a poster in November, 2010:
“With all due respect, this is the announcement of the perhaps most powerful
public two-page document in the history of software testing.”
Then it was remarkably quiet…
However, the poster is downloaded 100-500 times per month

23. Result
We often use the model in discussions with clients and prospect
It is a good way of letting the customer say, with their own words, what they
really value
Henrik has used them in a university course “Software Test Design” and in a
higher vocational study course “Manual Testing in Practice”
We continuously work with improving the model

24. Insights
Having all these in the back of your head when testing will make you a better
tester
You don’t have to remember everything;
you will recognize violations when you see them

25. Result
Comments on Charisma can be boiled down to:
“I like Charisma, but how do you test for it?”
We test for Charisma all the time…
You can do it
Use proxies to interpret the result
Compare with Usability
Not long ago Usability Testing was a branch of CHI
How do you test for usability without setting up a usability test lab and equip it with
eye-tracking machines etc?
"Everyone" agree that Charisma is important for the success of products, so
how come we don't test for it?

26. Usage
Test idea triggers
Define quality objectives
Write specifications
Review specifications
Evaluate test results
Report test results
Classify bugs
Talk about software testing
Learn about software testing
Identify risks

27. The Future
1.1 is coming up pretty soon
We are currently working on new two-page poster on sources for test ideas
And more to come…

28. Insights
The recipe for crafting our own models
1) This model can be improved
2) It is worth being improved
3) We are capable of doing it
4) Put in a lot of work

29. Questions
Software Quality Characteristics
http://thetesteye.com/posters/TheTestEye_SoftwareQualityCharacteristics.pdf
Blog
www.thetesteye.com
Contact
Email henrik.emilsson@qamcom.se
Twitter @henrikemilsson
Qamcom Research & Technology AB | www.qamcom.com | +46 31 721 1730