By Brian Leach and Nik Silver. First presented at London XPDay 2013, 25 Nov 2013.
Synopsis: Travis Perkins has introduced some very modern technologies in the last few years, but half of the 120-strong software engineering team is actively developing in BASIC on its 1960s green screen technology platform. Brian and Nik show how it is possible to introduce XP practices to technology that predates the moon landings, and will discuss lessons learned. They will cover unit testing, source control, and continuous integration, and will touch briefly on future objectives.
Synopsis: Travis Perkins has introduced some very modern technologies in the last few years, but half of the 120-strong software engineeringteam is actively developing in BASIC on its 1960s green screen technology platform. Brian and Nik show how it is possible to introduce XP practices to technology that predates the moon landings, and will discuss lessons learned. They will cover unit testing, source control, and continuous integration, and will touch briefly on future objectives.
Wider ContextTravis Perkins has embarked on a transformation of its IT practices with support and leadership from senior management.Where before small teams chipped away at the code face in single-function isolated silos, now they have grown their teams and embraced Agile principles, pairing and close working with stakeholders.
Does it have Global Variables?Yes.All variables have global scope within a program or subroutine.It also has ‘very global’ variables that can be shared between programs and subroutines.And ‘very, very global’ variables that exist for the duration of a login session (permanent).
What Engineers were SeeingThe engineers were learning java or sitting next to teams using java.So they were seeing those engineers working with TDD, frameworks and standard build processes.And they thought – does this apply to us?
The ChallengeStu had to change four lines of code in a small routine.Here is the context ..
.. And a bit more ..
.. And a bit more ..
.. And a bit more ..(This is one of the shorter routines. And Remember: every variable is global)
What We DidWe wrote our own unit test framework for UniVerse and taught engineers about unit testing and pair programming.Then we rewrote the framework to simplify and extend it as the engineers began to take ownership and to request features.The we extended it with configuration and faking.
Some LessonsUnits in UniVerse are different. The concept of a unit covers a wider block – you can’t build in Single Responsibility Principle in the same way. So we developed patterns for splitting and refactoring code to make the sections addressable for testing.The database is part of the code: we can’t separate it out as a dependency (though faking to isolate tests is possible).UniVerse engineers like TDD! And have taken it on board enthusiastically.. Eventually.
A framework has to have a name. All our unit test program names started “UNIT.” But UniVerse has a limit of 62 characters for a label, and the team was worried about wasting them. So they elected to drop one character, and thus the framework was named…
A token of RegardFrom the team to Nik...
The codebase did get source control in early 2013, but TFS had its problems. The Java teams’ move to git was the catalyst to move to the UniVerse teams, too.
Git AnguishMoving to git meant a change in mindset.UniVerse uses file locking to protect access to code. Previous source code control (TFS) also implemented locking.Moving to a distributed version control system with no visibility of who was working on what raised fears.
CI PipelineWe built a CI pipeline based around gitlab post-commit hooks. These are captured using HTML listeners in UniVerse and commit details logged to file. A cron job runs a CI process on commit changes, feeding a segregated release system and packager.Unusually, the decision was taken to email notifications of passes as well as failures (the opposite of many black box CI systems).The aim is to publicize and celebrate success as well as notify failures.
Consistent Reliable EnvironmentsShifting to TDD and CI led to changes in expectation about what would/could be run in a clean environment.In addition to data files, the application sets up configuration information on login. This is unavailable to the CI process.Many assumptions are made in the code about the existence of this and of standard parameter records.And this is a distributed system, so code expects to call routines on other machines.Fatal errors stop that UniVerse process: a problem as the CI runs in the same process (and cannot report back).And there were many programs that could not compile.
September 2013: Switch-over to git + CIAugust 2013: Start git training for UniVerseJuly 2013: Start building CI pipelineJune 2013: Target for git switch-over (missed)May 2013: Started discussion of git and CI for UniVerse Apr 2013: Started TDD for UniVerse, Cloudshop moves to gitFeb 2013: Source control introduced (TFS), teams start working in stepJan 2013: TDD training startsDec 2012: Scrum teams arrangedSep 2012: Senior team assembled
Using XP practices on 1960s green screen technology
Using XP practices on
1960s green screen
The system that time forgot
• Senior management buy-in
It’s not legacy, it’s a
• More team-working
Knowledge silos and
Pairing, independent teams,
breaking down of knowledge
silos, closer involvement of
What engineers were seeing
TDD + FitNesse training for
Sometimes learning Eclipse,
and Java, too
There was a basic test
With if/then instead of
Could not see how unit
testing applied to them:
Not designed to be tested
Not part of the culture
The Challenge – In Numbers
•40,000+ concurrent connections
Over 700 lines
of code added
day for 25 years
• 4x HP Superdomes
• 1 million+ reads/machine/second
• 12,000+ programs
• 4,644,481 lines of application code
• 31% have cyc. complexity > 50
• 12 TB changes every day
• 120 software engineers
• ... And growing
Lest we Forget
has powered a
over 25 years
to an annual
5 billion pounds
What we did
• Wrote our own framework
Created own refactoring catalogue
Started UniVerse-specific TDD training
• Rewrote the framework
• Extended the framework
Units are different in UniVerse.
Via refactoring catalogue
The database is under test
Via TDD champions
There are limits to breaking
down procedural code.
Critically, they drive each other
Starting to inherit tests
Community site for discussion of
Areas of resistance
“It’s a waste of time”
“It takes too long”
“We’re going to throw it away anyway”
“We can’t spend time on it if our product owner doesn’t
Effective pairing is more difficult with procedural code.
Experiences Introducing git for UniVerse
• February 2013:
TFS introduced for source control
File-driven/fragmented in a central repository
• April 2013:
CloudShop moved to git (practical and strategic),
trigger for similar move for UniVerse
Switch over in June firebreak, but lots of anguish…
• Change of workflow
Whole codebase snapshots
• Inclination to want all problems are solved.
“Git is not suitable
Git anguish resolutions… and disappointment
• Regular discussions and
• Some people still use
• Adjusting world view
• Difficulties with three
• Git training
• Adopting better tools
• Finally went live in…
Consistent reliable environments
• Chaining to/calling fatal errors or stop
• Calls which kill the CI process
• Lots of assumptions about the environment, and
continually challenging to ensure it’s clean.
• Non-compiling programs
• Needed to clarify the development process
• Engagement around non-compiling code
• Reaction to global CI email
Whole codebase compiled for the first time
Real collective ownership
Commits circulated to whole of Engineering
as a matter of pride.
• Human rewards…
• Engineers feel part of the modern world
• Feel invested-in
• Have more transferable skills
• TP held up as an exemplar
• “Best Agile Team” award 2013
Where Next ?
• Improve CI process
• Extend packaging
• And on and on…
Switch-over to git + CI
Start git training for UniVerse
Start building CI pipeline
Target for git switch-over (missed)
Planning git and CI for UniVerse
Started TDD for UniVerse
Cloudshop moves to git
Source control introduced (TFS)
teams start working in step
TDD training starts
Scrum teams arranged
Senior team assembled
New CIO arrives
1996: Migration to UniVerse
1987: Development starts
1987: Development starts
Tech appendix follows…