My personal lessons learned

1. Software Development
& Software Quality
Lessons Learned – Knowing The Limits
Herwig Habenbacher
herwig dot habenbacher at gmail dot com

2. Background: About Me
• Writing Software since over 30 years
• Broad Spectrum: PC Software, Supercomputers, Embedded Systems,
Cloud. Mainly Technical and Scientific Software
• Highly interested in Programming Languages
• Leading Software Group for Measuring Devices over 11 years:
customers must trust the Software or you are out of business
• No legal need (SIL Level) but still verified some parts:
Used PolySpace, played with SPIN, Maude, Frama-C and Coq.
Being aware of the limits …

3. Musings About Software Development
Software Development is not the same for all needs!
The best metaphor I found yet is to compare it to the Construction
Industry – their needs, material, planning, processes and tools.
What do we expect from the Construction Industry to get reliable
buildings? How do they adapt to the changing environment?

4. Built for Eternity: Pyramids
Are we building very long standing Software
which will be unchanged nearly forever?
Who use them over time?
They use a lot of cheap material, primitive tools,
even with not so complicated architecture.
It is pretty hard to adapt them to current needs.
On the other side they are very robust and
withstands nearly everything...
But built for dead God’s?

5. Sophisticated Architecture: Cathedrals
Cathedrals are mainly known for their beautiful
architecture.
They were in detail planned, but nearly always
unique pieces so that nothing was reused.
Once build (often over centuries) they last for
centuries - their use is not very adaptable, and
changing more on the architecture may lead to a
crash.
Everything is more an art than technique - with a
lot of handcrafted work and good tools.

6. Adapted To The Environment: Grown Cities
Most older cities are not planned
- they are grown.
We can see different styles of architectures and
materials. They are full of life and adapt
continuously on all changes as needed and
possible.
Planning and processes are applied mostly for
subprojects and is only partially done for the city
itself.
Replacing parts is common and natural.

7. Driven By Economics: Planned Cities
Most planned cities are driven by economic
needs: they should be cheap for the customers.
This is only possible by reusing the design with
only few adoptions or by using a modular
approach.
The material is rather cheap and standardized,
the building follow a standardized process and
tools.
Some life happens outside the city, otherwise the
city needs changes - and this means cost.

8. Special Needs: Skyscrapers
In some places space is a rare resource.
So people started to build into height -
Skyscrapers.
Here we need a good planning and design, good
materials and a well defined process.
We need special tools to build them - without it
would be impossible.
They need maintenance during their full lifetime,
otherwise they would become fragile and
dangerous, even for the Neighbors!

9. Comparison To Software Development
So far we have seen that we need:
• A Plan - Architecture is dominating material but depends on it
• Material - what the building is made of and stays there.
We cannot build a Skyscraper out of Sand only …
• Tools - defined by used material, they gone after job is finished
• Processes - defined rules and steps how to build it
We have to look at all of them if we want Quality …

10. Software Development: Actual State
So what is the actual state for Software Development?
- Plan / Architecture: Requirements with Use-Cases, UML
- Material: Mostly Imperative and Object-Oriented – Java, C++
- Tools: IDE’s (Eclipse, Visual Studio), Unit Tests, Jenkins, …
- Processes: Mostly own form of “Agile” – “Iterative V-Model”
Software is still developed by people, trend to Outsourcing still valid.

11. Development: V-Model vs. Iterative
The V-Model is mainly driven by the metaphor
compared to Mechanical Engineering:
• Detailed planning at the beginning
• Standardized components and processing methods
• Purpose and mission clear and stable
• Predictability: time, cost, performance
• Build on each other phase-oriented implementation
• Clear division of labor between specialists
• Independence from specific performing persons
• Communication over plans

12. Development: V-Model vs. Iterative
We know that in reality the V-Model has some
flaws:
Top-down approaches emphasize planning and a
complete understanding of the system, and it
means that the large-scale aspects of a program
must be engineered first.
It is inherent that no coding can begin until a
sufficient level of detail has been reached in the
design of at least some part of the system.
Iterative wants to shorten the feedback cycle.

13. How To Get Faster To The Product?
Iterative Development is mainly to get faster feedback for
doing the right thing.
“If everything seems under control,
you’re not going fast enough.”
(Mario Andretti)
But how can we develop faster without loosing Quality?

14. Planning And Specifications
Nearly all Software people agree about one fact:
As longer a Software Bug is undetected in the Development cycle,
as more expensive it becomes to correct the error
So you have to shorten the feedback cycle between Development and
Validation. This may be easy and cheap for a Web Application, but of
very high cost for e.g. medical devices (you may loose a life) or space
probes (not easy to correct the landing on a comet…)
So to get Software which behaved the way you want you first have to
specify it!

15. "A program that
has not been specified
cannot be incorrect,
it can only be surprising."

16. How To Get Specifications: Requirements
How do you know that you have the right Requirements?
Tools like Doors help for tracing Requirements, but not to get them.
Methodologies like such from the Sophist Group helps sometimes …
- Create a Vision
- Concentrate on customer value
- Gather Use-Cases
- Build a Risk List
- + Think how you want to test the Requirements

17. Test The Requirements: Validation
But here we are not talking how to get the right Requirements.
We want to talk how to validate them.
The most common way are Unit-Tests:
“For a well defined input we expect a well defined output.”
This may be a lot of work, because you have a lot of inputs to check.
How to check if the tests are good enough? Code Coverage?

18. Problems With Unit-Tests: Coverage
For simple cases Unit-Tests work – but the can give a wrong sense of
the correctness of the Software.
Assuming a simple function: f(x)=2/(3.4–x)
If you input vector for testing is [0,1,2,3,4,5,6] everything works
fine – you even get the metrics for Code Coverage of 100%.
You experience an exception if you input one time becomes 3.4

19. Finding Remaining Bugs: Fuzzing Tests
But how can we find such remaining bugs?
One way is “Fuzzing” and “Horror Cabinet”:
You generate random input values or use corrupt data.
We don’t look at the output values, just if the Software stays stable and
maybe detect the corruption.
In Embedded Systems this was a great way to handle wrong Sensor
values caused by Electronic Failure or Environment Influences.

20. Test The Test: Code Sabotage
For critical parts we wanted to know if the tests are “good enough”
One way to find out was that we were allowed to change the Code
afterwards: changing values, operations (e.g. switching from “+” to “-”)
and data flow – we made “Code Sabotage”.
If we run the tests they should detect the changes.
If the don’t find them, they are not good enough.
This works only on small critical parts, because it is very expensive.

21. Testing Properties
Instead of testing single values you can also check for properties or a
behavior you expect from a function:
- There is a trend: e.g. bigger inputs always give bigger outputs
- The delta is always the same: e.g. |sin(x)-sin(x+d)| < r
- There is a Symmetry: e.g. cos(x) = cos(-x)
- There is always a neutral element: e.g. to 0: sin(-x)+sin(x)
If we have to work with external libraries this is one way for checking

22. Example: Check If The Function Is Symmetric
This was an example from Anchor Calculation:
The whole layout is symmetric!
It doesn’t matter how big and in which direction
the shear forces are: the results should be the
same if we mirror the direction of the force.
We generate a shear force in direction and value
randomly and check if the results are the same
for mirrored directions. This can be automated.

23. Example: Check If The Function Is Continuous
This was an example from Anchor Calculation too:
We only change one value: the torque moment M
From a big negative value over zero to a big
positive value.
We can have check points (e.g. Zero), but mainly
we are not so much interested in the values but
looking at the previous and next point if the
results have only a small delta – they are
continuous. This can be automated too.

24. Generate Tests: QuickCheck
Writing tests is a lot of work – so it would be better to generate them.
QuickCheck is coming from Haskell and is heavily used in Erlang.
“In QuickCheck the programmer writes assertions about logical
properties that a function should fulfill. Then QuickCheck attempts to
generate a test case that falsifies these assertions. Once such a test
case is found, QuickCheck tries to reduce it to a minimal failing subset
by removing or simplifying input data that are not needed to make the
test fail.”
(Wikipedia)

25. Limits Of Testing: Ashby’s Law
Ashby was a Cybernetican who defined the
“The Law of Requisite Variety”
It is about how you want to control complex systems.
“If a system is to be stable, the number of states of its control
mechanism must be greater than or equal to the number of states in
the system being controlled.”
Can you manage the Effort to control your System?

26. Limits Of Testing: Bremermann’s Limit
Bremermann’s Limit is the maximum computational speed of a self-
contained system in the material universe.
Actually a lot of people look at it to judge cryptographic algorithms –
but we can look for testing effort too.
At first look it is a high number – but if you look how many variables
you have in you code it becomes clear that even for a few hundred bits
of variables you cannot fully test them – even if the whole Earth would
work as a Supercomputer!
This makes it very clear that we cannot bet on full testing…

27. Proof Of Absence Of Bugs: Verification
So if testing is not enough – are there other methods like verification?
- Can we use tools like PolySpace for Abstract Interpretation?
- If we have verified Code, do we need a verified Compiler?
- Do we need a verified Operating System?
- What about 3rd party libraries?
Actually it helps a lot to verify our own Code, but for big Systems there
is nothing yet available. Verification needs educated people!

28. How Can We Improve?
If we compare to Construction Industry an look for improvement,
do we need:
- Better Plans / Architecture?
- Better Materials?
- Better Tools?
- Better Processes?
Probably something from everywhere…

29. Better Architecture: Adapt To Future Needs
Truly complex things are grown, not build.
Examples are a tree which grows to adapt to wind, rain, and the pole
that was put too near to it. Or a city which grows as the focus of life
or commerce changes from the seaport to the railroad to the airport.
What does this mean for Software Architecture? Since we cannot
handle the full Feature Set in a Monolithic System, we have to think
about Extensibility: only if we divide we can conquer! With
decoupling we can bring the Complexity down to a manageable part.

30. Better Architecture: Divide And Conquer
One way is to decouple them – even the architecture.
- For different demands you need different architecture
- Different architecture may need different materials
- Different material needs different tools
- Different architectures may need different processes
Different needs for different parts may need different solutions.
How to work together? One Solution is Messaging!

31. Messaging Architectures: Trends
Messaging driven Architecture is not new (MOM)
They are used in Embedded Systems, in SOA or Microservices …
The common thing is that they share nothing and communicate via
Messaging – how close they are to Actor based Systems depends on
the history of the implementation.
The idea is coming from Biology where cells are some small
independent structures which communicates via Messages – this
influenced people like Alan Kay for their first Object-Oriented Systems.
One promising way is Domain Driven Design.

32. Sidenote: Does The UML Help?
Does Modeling help? Architecting with UML?
In my opinion very little – the UML is more a Notation than an Abstraction.
The Syntax is defined, but not the Semantic.
So it is very much bound to the underlying Programming Language.
For example in State Machines you can easily generate parallel regions.
The UML says nothing about Semantic or Implementation: are OS Threads
used, Green Threads or even no Threads? Ordering? You can only guess…

33. Sidenote: The UML Doesn’t Help!
A lot of things are not specified in UML – therefore you have Profiles (like
MARTE in Embedded Systems) and the OCL.
The aspect of Inheritance, Interfaces or Threading depends a lot of the
Language you want to use.
Notations can be useful – see arithmetic in Roman Notation versus the
Arabic Notation and their influence on multiplication!
But the UML doesn’t really help you to get a better architecture.
Even Patterns shows only the Defects of the Language…

34. Better Processes: Requirements & Specifications
What are the defined rules and steps how to build
the Software?
It is not only about designing and building Software:
- What are the Risks and Opportunities?
- What are the Steps for Risk Mitigation?
- How do we manage Change?
- How do we handle Technical Debts?
(Picture from «The Agile Samurai»)

35. Thinking Ahead: Anticipating The Future
Problem appeared
Decision taken
Problem determined
Cause determined
There are different Method-
ologies available which support
Decision making and Anticipating
of Future Problems.
FMEA and Sensitivity Models
may give you hints where
Problems may occur.
A systematic Approach is seldom
used in Software Development.

36. Use Better Material: Languages
Programming Languages are the same than material is for construction:
- They allow you a special or the wished architecture
- Architecture dominates material, but material defines what you can
build: there may also be some legal Requirements…
- They define a lot of tools you need to build
A lot of Managers overlook the Importance of Programming Languages.
They look more on processes and tools.

37. Languages: Features
Features of Languages can improve Quality and Productivity:
- If you don’t want to worry about Memory: use Memory Management
- If you don’t want to worry about Null Pointers: use Options
- If you don’t want to worry about Side Effects: use Pure Functions
- If you don’t want to worry about Type mismatch: use Type Checking
- If you don’t want to worry about Threads and Locks: use Actors
There is no Silver Bullet – but some Features may help a lot!

38. Languages: Java And Pointers
“Java has no pointers but
a NullPointerException”
Are null references the billion-dollar mistake? (By Tony Hoare)

39. Languages: Trends
Due to trends like MultiCore and Distributed Systems we can see a
trend to more “Functional” Languages:
Scala, Clojure, F#, Elixir, Elm, … and more advanced ones: Idris, Agda, …
They are going more to Strongly or Dependent Typed Languages.
Constructs like Algebraic Types allows to prevent the representation of
illegal states – a nice Example was given by Yaron Minsky “OCaml for
the Masses” in ACM queue 9/9 2011.

40. Languages: Things To Watch
Another trend is to bring the Language closer to the Specification.
Avoid verbose Constructs and work with the Human Short-Time Memory.
It is about composability of functions and how to handle both success
and errors. The term “Monadic” may loose its frightening sound over
time hopefully…
Scott Wlaschin created the term “Railway Oriented Programming”.
Look at his wonderful Slides on SlideShare!

41. Languages: Better Replacements?
What are better Language Alternatives Today?
- Use Rust instead of C++
- Use F# instead of C#
- Use Scala instead of Java
- Use Elm instead of JavaScript
For Distributed Systems: Look at Erlang OTP and Elixir!

42. Tools
What are tools for?
I found the best explanation on Bret Victor’s page:
A tool addresses human needs by amplifying
human capabilities.
That is, a tool converts what we can do into what
we want to do. A great tool is designed to fit both
sides.
Which tools do we need to build our Software?

43. Redundancy: Need Of Distributed Systems
To quote the father of Erlang, Joe Armstrong:
- The world is concurrent
- Things in the world don’t share data
- Things communicate with messages
- Things fail
What happens if your perfect, error-free Software is hit by a lightning?
(Happened to Google Datacenter in Belgium August 2015)

44. New Dragons Around The Corner
To minimize the Risk of Hardware Failure you need at least 2 Computers.
But now a full new can of worms are open:
- How to Synchronize the Systems?
- How to find Consensus? (Hint: Paxos or Raft Protocol)
- Maybe no common Clock, order of Messages…
- Some other Byzantine Failures …
- What happens to the CAP Theorem?

45. Erlang OTP and Akka
There are some Attempts how we can manage Distributed Systems.
A widely used one is based on the Actor Model.
Such Systems were pioneered by the Telecom Industry, namely Erlang.
Other Systems took the ideas to other platforms like JVM or .NET.
For integrating them users prefer to stay in their ecosystem – that is the
main reason why Akka get more focus than Erlang.
Maybe that changes with Elixir – the future will tell.

46. The Human Factor
Since Software is mainly written and used by Humans we have to take
the Human Nature into account.
We can train them, but better than that would be to know the Human
limits and try to prevent errors based on his weakness.
This doesn’t only affect users with Human Interface Guidelines, but
also Programmers and Designers who have to understand Mental
Models, Programming Languages and Operating Systems.

47. The Human Factor: Culture
According to Valentin Turchin Life had some Evolutionary Quanta’s:
1) Control of Position = Motion: the animal or agent develops the ability to control its
position in space
2) Control of Motion = Irritability: the movement of the agent is no longer given, but a
reaction to elementary sensations or stimuli
3) Control of Irritability = Reflex: different elementary sensations and their resulting
actions are integrated into a coordinated, but still rigid, reflex-like behavior
4) Control of Reflex = Association: behavioral routines become flexible or adaptive,
through the learning of new associations between experienced stimuli and actions
5) Control of Association = Thought: new routines no longer need to be learned
through experience; they can be developed by abstract, symbolic reasoning
6) Control of Thought = Culture: symbols and concepts are no longer fixed entities; they
adapt through a process of cultural evolution

48. The Human Factor: Culture Of Development
On the first moment it seems weird that culture is the control of
thought. But on a closer look it makes sense:
- You may come from the culture of Object-Oriented Programming
- You may come from the culture of Functional Programming
- Your development culture was to deliver the minimal valuable product
- Your development culture was from Aerospace or Medicine
Mixing cultural styles may create innovation – but can lead to conflicts

49. The Human Factor: Culture Of Developers
“The Problem Is Not The Problem.
The Problem Is Your Attitude
About The Problem.
Do You Understand?”
- Captain Jack Sparrow
Are the Developers aware of the Problem?
Some times they often don’t know that their Code can cause a bug…

50. The Human Factor: Environments
The Bullshit Asymmetry Principle:
“The amount of energy
needed to refute bullshit
is an order of magnitude
bigger than to produce it.”
- Alberto Brandolini
How to work with external libraries or legacy code?

51. The Human Factor: Users
But even if our Soft- and Hardware is perfect: our users are Humans
which can err and make mistakes:
- Habituation
- Complicated Non-Monotone Interfaces
- Missing or wrong Feedback
- Engineering Constraints leading to wrong Focus
- Not respecting the Human Nature
- Cognitive Psychology for different cultures
(Picture from «Cognitive Psychology»)

52. Conclusion
For writing correct and error-free Code there is still no Silver-Bullet.
Architecture, materials, tools and processes are important.
We still have to follow our journey for better Software…