Hello, and welcome to our “Bug Wars” presentation.
I’m Ben Cripps, and to my SIDE is my colleague Martin. In this presentation, I’m going to look at bugs from a managerial view, be it project manager or development manager, while Martin is going to discuss bugs from a pizza-loving developer’s view.
Have you ever wondered how much time is spent fixing bugs?
As a manager, that has to be one of my worries – bugs can bring heat to me, the team and can burn through a lot of time trying to fix the bug. According to Tricentis, in 2017 we spent close to 269 years fixing bugs.
In that time alone, Tesla’s roadster would have made 215 round-trips to Mars.
Now imagine what you could have created with that amount of time.
Maybe some new functionality? Exciting new development projects? Cutting edge proof of concepts? Maybe self-investment – training, development practices, methodologies etc.
It certainly makes me think about my own missed opportunities.
Tricentis Software Fail Watch (2018) - https://www.tricentis.com/blog/2018/02/01/how-to-avoid-the-tricentis-software-fail-watch/ Mars One - https://www.mars-one.com/faq/mission-to-mars/how-long-does-it-take-to-travel-to-mars
Other than time, what are the bug costs are there?
The obvious one is financial.
Let’s imagine there is a loan company that found a bug that resulted in 60% of their loans not being collected on time. It already sounds quite serious, right?
Well, this happened in the real world and got introduced as part of department overhaul.
The bug caused a lot of financial damage:
Firstly, for 2017 it cost Provident Financial £120m in lost profit. But it also hit that market value – that dropped a whopping £1.7bn in one day.
Total cost was a whopping £1.8bn, plus the chief execs’ registration.
But there is more than just a financial and time cost. There is the potential brand or client damage; maybe internal trust will be lost with you or the rest of your development team.
Sub-prime loan firm Provident Financial. Chief exec Peter Crook resigned as the company's share price tumbled from £17.42 to just £4.50.
Will it ever happen again?
It has, and sadly it will continue to do so.
And the later you find a bug in the SDLC, the more it will cost in time, reputation and financial.
TSB have been suffering recently too, migrating to a their own, new platform. Intensive national news coverage, with open air social media posts on Twitter, Facebook etc. Days of outages which will no doubt cost millions in pounds in damages, compensation and most likely lost customers! Displaying on-screen errors to the customer – java.lang.NullPointerException, BeanCreationNotAllowedException and more. All in public display, stoked by news outlets and the power of social media. Not only that, TSB bank managers are reporting staff are close to collapse. 1.9m digital banking customers without access to their account. 40,000 complaints, 13x their usual levels. An interview with the TSB CEO about resolution time where introduced doubt over the technical teams capabilities to resolve by their predicted time. Soul destroying move for the already under pressure, under fire department, and it removes team morale and inter-company trust – bridges were burnt! Immense company, inter-departmental and development pressure.
This picture is of me drinking a hard-earnt glass of champagne with our client and my CEO, convincing them all is well with the new solution we’ve just released. Meanwhile, next door the QA team are finding new bugs, pushing them down into the basement for Martin and the team to deal with ASAP.
No matter your best efforts, some form of bug will always slip through the net.
So how do you capture and log exceptions? More importantly, how do you analyse and prioritise these?
So, what is the magic answer to bugs … in one word, “visibility”.
But as a development team, how can we achieve that? Here’s Martin with the answers!
The IDE experience while developing – everything is to hand:
Editor (syntax colour highlighting) Compiler Debugger Unit tests Code analysers Other tools
Production environment is usually locked down.
Developers cannot see under the covers.
Real users use the application in ways that were not anticipated.
20% of errors never make it to the logs in production: Research shows that 20% of catch blocks are empty! That means swallowing the exception without any trace. These are the silent killers of Java applications.
Research by OverOps into 1/2 million Java projects on GitHub:
Their research showed three groups:
Documenting what happened, through logging, printing a stack trace or printing out information to the console. Rethrowing an exception, probably a wider abstraction that one of the methods further up the call stack would know how to handle. And nothing! An empty block, simply swallowing the exception without any trace. And looks like it even happens at least as often as logging it.
Nearly two-thirds of logging statements are turned off in production: INFO and DEBUG logging accounts for 57.8% of logging statements and these are turned off in production. And in pre-prod environments where these are turned on, you will probably find you can't reproduce the issue.
More than 50% of logging statements don’t include ANY information about the variable state at the time of an error: 52.2% of log messages don't include any variables and 33.5% include only 1 variable. Considering the number of variables likely to be in scope that could have been responsible for the error then chances are, the logs won't have the detail you need.