Distributed development across countries creates both challenges and opportunities for the production of high quality software. We’ll look at new ways of achieving automation for testing software in a continuous delivery context, using parallelization techniques and automated analysis fully integrated with a reliable and scalable SCM system. A new optimal method of testing common code in similar branches is presented along with the semantic merging of testing results.

1. Software Testing in a
Distributed Environment
Neil Langmead
Unrestricted

2. 2
The Agile DevOps World
 Many stakeholders
 Distributed across teams
 Huge potential for
misunderstanding
 Lost In Translation…
• And this is just English!

3. 3
The Distributed Agile World
Requirement to:
 Merge in parallel
 Stop bugs proliferating
 Testing Early and Often
 Fix Before the Merge!
 Time Zero Fixing
 Fast Desktop Feedback
/main branch
BL130
Released to customer
devel: john
/main/task115
New loading form
devel: pat
devel: pablo
/main/task114
Typo in about form
/main/task116
GDI resource leak
This check-in introduces a bug,
but only “john” will get affected
by it until he merges! He has a
chance to fix it before affecting
anyone

4. 4
The Importance of Scalable SCM
DSCM systems require as a minimum:
 Scalability, to 80+ MLOC (e.g. PLM)
 Robustness, security
 Efficient branching and merging
 Automated Workflow, API
 Rapid Release Cadence

5. 5
Process Transformation- Siemens CT
Strong correlation in
net new coding
violations of
good practice (blue)
along normalized
code churn (orange).
Post Perforce, new
violations flatten to
zero- why?

6. 6
The Problem of Complexity
 Complexity Manager is now
an official role!
 LOC estimated to increase
by 20% in 2016
 Multiple build targets,
where only a few source
files may differ, complicate
the testing process

7. 7
The Problem of Technical Debt

8. 8
The Problem of Technical Debt
 Like financial debt, difficult to
control/ pay back
 Accumulates even when we do
nothing (e.g. lib versioning) !
 The importance of the NULL
Release
• How long does it take for your
organisation, if 1 LOC is changed, to
up issue to a new version?
• Can be weeks, or even
months…

9. 9
How Do We Test?
 Need Total Testing approach
• Consider FOSS implications
• Logic Flow/ Syntax / Semantic Analysis
• Architecture
• NFR (reliability ,security…)
• Unit Testing / Coverage
• BVA, Equivalence Partitioning
• Fuzzing
• Concurrency/ Threading issues
• Proliferation through branching

10. 10
How Do We Test?
 “Connected Desktop”
 Developer IDE’s connected
to build server AND SCM
 Time Zero Approach
• Don’t let the bug grow!
 On the Fly Analysis
• Eclipse, Visual Studio, etc

11. 11
How Do We Test?

12. 12
How Do We Test?

13. 13
Change Based Testing (CBT)
 How about we only test
what’s changed?
 BUT how do we identify
changed components?
 Tight integration between
SCM, Jenkins, JIRA
 High levels of Automation
 Continuous Testing

14. 14
CBT & Invasiveness
• Define & Measure
Invasiveness of a defect
 Compute transitive
closure of a change, to
assess impact on the
system
 Can be done on feature,
defect or enhancement
level from the SCM repo

15. 15
CBT & Invasiveness

16. 16
CBT – Issue Invasiveness
• Set of Scripts to
interrogate the Helix
repository
• Selects on IssueID,
Author
• Cross Referencing Matrix
yields the required
change metrics, which is
then passed to the unit
testing tools

17. 17
CBT – Least Common Ancestor

18. 18
CBT – Least Common Ancestor

19. 19
Insights from Repository Analytics (RA)
Other insights become
apparent from Helix
RA:
Comparison measures
contractor productivity
(violations of good
practice, contributions,
change impact) by
project & location.

20. 20
Insights from Repository Analytics
Other insights become
apparent from Helix
RA:
Comparison measures
contractor productivity
(violations of good
practice, contributions,
change impact) by
project & location.

21. 21
Test Driven Development Analysis
Outlying project reveals
one developer (ID
#1040) who is
contributing greatly
(i.e. skilled and
experienced) but adds
few new tests. “1040”
needs to on boarded
regarding the TDD
standard for for greater
compliance and project
quality.

22. 22
Do we now need a Dynamic Architecture?
 Architecture is the hardest
part of a SW system to
change
 Often set early, not very
adaptable
 Expensive, technically
challenging
 “Elastic Architecture”
methods required
• Component based
• Flexible API, “Plug and Play”

23. 23
Conclusions
 Complexity is here to stay
 Need advanced, efficient methods to reduce test effort
 Use Branch Per Task/ Test Per Task
 Change Based Testing Approach to reduce technical debt
 Time Zero Testing before the Merge
 Advanced Repository Analysis to gain additional insights
 Agility in the Architecture is the Next Frontier (Neil’s opinion…)
 “Continuous Testing” is here to stay

24. 24
Demonstration
demonstration of the Total Testing Platform, in use at Siemens
Thank you for your attention!