© Lloyd Roden 1Top Challenges We Face in IT todayto Hinder Our Pursuit of Quality
© Lloyd Roden 2ContentsHow we react to challengesMy top challenges in testing todayHow to make a difference
© Lloyd Roden 3What is a challenge?climbing Mount Everest?running a marathon?cooking a dinner party for 20 people?flying?g...
© Lloyd Roden 4Good versus Bad Challengesbad challengeswhen they are harmfulwhen they have undesirableconsequencesgood cha...
© Lloyd Roden 5How we react…?
© Lloyd Roden 6ContentsHow we react to challengesMy top challenges in testing todayHow to make a difference
© Lloyd Roden 7Challenge #1: ban the use of “best practice”Best: the highest quality, excellence orstanding. Absolute qual...
© Lloyd Roden 8Why “best practice” can be harmfulDreyfus Model (for skills acquisition):NOVICE(needs practices)ADVANCED BE...
© Lloyd Roden 9So how does this help?where are “best practices” useful?• protecting people from themselves• helping advanc...
© Lloyd Roden 10Challenge #2: Test Cases…quantity or quality?Welcome to the “tester’s weakest link”Johnbeen with the compa...
© Lloyd Roden 1111John…your questionyour definition of a test case is:“detailed inputs and outputswith pre and post condit...
© Lloyd Roden 1212Carol…your questionyour definition of a test case is:“a set of input values, executionpre-conditions, ex...
© Lloyd Roden 1313Rick…your questionyour definition of a test case is:“one particular instance orvariation of a test or te...
© Lloyd Roden 1414Pam…your questionyour definition of a test case is:“a computer executable scriptwith a sequence of discr...
© Lloyd Roden 1515Counting test cases is meaningless…Without context• 30 test cases• 2 test cases• 300 test cases• 0 test ...
© Lloyd Roden 1616The problem with counting test cases…Public Course Booking Form - Please Complete All FieldsPlease read ...
© Lloyd Roden 1717We focus too much on quantity!Managers seem more interested in counting and monitoring theQUANTITY of te...
© Lloyd Roden 1818So how can we measure the quality ofthe test cases?GAINING CONFIDENCE• in what has been tested• the dept...
© Lloyd Roden 1919Be careful of the QUANTITY ofcases…without QUALITY!
© Lloyd Roden 20Challenge #3: Don’t lie with metrics…but I am sure not intentionallylet us look at some graphs…same data w...
© Lloyd Roden 21Lies, damned lies and statisticsconclusion: requirements are badconclusion: inconclusive, numbersare not s...
© Lloyd Roden 22Challenge # 4: Challenge complexity at every opportunitysimplicity seen as weak and uninterestingwho wants...
© Lloyd Roden 23Features and functions usedJim Johnson XP2002 Standish Study GroupFeatures and Functions Used16%13%7%19%45...
© Lloyd Roden 24Challenge # 5: Test Managers/Leads should test…and testers should improve their technical skillssome excus...
© Lloyd Roden 25Why is this non-negotiable?why do I believe Test Managers/Leads should test?you can identify & empathize w...
© Lloyd Roden 26ContentsHow we react to challengesMy top challenges in testing todayHow to make a difference
© Lloyd Roden 27What to do now…produce an action planwhich of these challenges are affecting you?what other challenges are...
© Lloyd Roden 28Summarywe need to understand howhumans react to challengesand the importance offacing them to shape ourper...
Upcoming SlideShare
Loading in …5
×

'Top Challenges We Face In IT Today To Hinder Our Pursuit Of Quality' by Lloyd Roden

306 views
224 views

Published on

Some people thrive on challenges, while others struggle with how to deal with them. Handled well, challenges can make us stronger in our passion, drive, and determination. Lloyd Roden describes the challenges we face today in software testing and how we can respond in a positive, constructive manner.



One of the challenges Lloyd often sees is identifying and eliminating metrics that lie. While we (hopefully) do not set out to deceive, we must endeavor to employ metrics that have significance, integrity, and operational value. Another challenge test leaders face is providing estimates that have clarity, accuracy, and meaning. Often we omit a vital ingredient when developing test estimates - the quality required in the product.



A third challenge is convincing test managers to actually test regularly to attain credibility and respect with the team they are leading. A further challenge is to see why the use of the term "best practice" can be so damaging.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
306
On SlideShare
0
From Embeds
0
Number of Embeds
25
Actions
Shares
0
Downloads
5
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Let us start by defining what a “challenge” is… there are various definitions: * call to battle (e.g. a duel) * contest that requires special skill (e.g. tennis competitions) * call into question (e.g. challenge the statement) Climbing Mount Everest: Has anyone done this? Would anyone like to do this? You need to train hard for this challenge – for most of us it would be something that is outside our reach or desire Running a Marathon Has anyone done this? Maybe for some of us it is a challenge just to…run  But maybe if you have already done this you want to beat your time or perhaps run 42 marathons in 52 days like Eddie Izzard! Running a marathon would certainly be a challenge for me. I have just taken up running on a treadmill…1 mile a day, but even that is a challenge for me Cooking Dinner for 20 people Not a real problem for me as I love to cook for dinner parties. 20 people would be manageable for me but for some of you – you would prefer to climb Mount Everest  Flying Not literally – that would be a challenge. But for some flying in an Airplane it would certainly be a challenge as this might be overcoming a fear that you have, This is a different sort of challenge Getting out of bed Anyone here have teenagers? Now you know what I mean! We spend the first 10 years getting them into bed and the next 10 years trying to get them out of bed
  • It is important for us to realise that challenges can be both “bad” and “good” Bad Challenges “ I challenge you to a game of squash” – not a problem if you are under 40 but if you are 70 and haven’t played the game before! “ I challenge you to run a marathon after this session” “ I challenge you to fight Mike Tyson I challenge you to cook a meal for 20 people all of whom are VIPs I challenge you to get out of bed before 10am each morning – result grumpy teenager Good Challenges These help improve our self or open our minds to other things I challenge myself to run a little further each day and a little faster (but not too much) I challenge myself to eat more healthily – smoothies taste really nice! I challenge myself to try anything once when it comes to culture and national food (sushi, bone, pigs ears to name but a few)
  • Our reactions will vary with challenges that we are faced with: “ Yes” or “No way” or “Boring” or “don’t really care” I hope the challenges I a going to share will invoke a positive reaction…
  • Go through the definitions of: BEST and PRACTICE Why is it that we use these terms? STOP IT! Who are we trying to impress? Vendors – STOP Saying that your product/process/methodology is best practice Managers – STOP saying you have adopted best practice What are you trying to prove? If you have best practice then it means that you cannot improve…best by definition is highest qualifier. Also are you trying to downgrade other methodologies? Ours is BEST practice – therefore yours must be inferior practice or WORST Practice
  • The Dreyfus Model. In the 1970s, the brothers Dreyfus (Hubert an Stuart) began their seminal research on how people can attain and master skills. The Dreyfus brothers looked at highly skilled practitioners, including commercial airline pilots and world chess masters. Their research showed quite a bit changes as you move from Novice to Expert. You just don’t “know more” or gain skill. Instead, you experience fundamental differences in how you perceive the world, how you approach problem solving and the mental models you form and use. Unlike other models or assessments that rate the whole person, the Dreyfus model is applicable per skill. In other words, it is a situational model and not a trait or talent model. You are neither “expert” nor “novice” at all things.; Rather you are at one of these stages in some particular skill domain. You might be a “novice” cook but an “expert” sky diver, or visa versa. Practices are the things we do…behaviour Let’s look at how Practices can help or hinder in the Dreyfus model… Go through the stages… Let us use the analogy of creating a test plan… (Subverts = to ruin or overturn) Novice – needs the practice/standard for test plans – they don’t know how to create one Advanced Beginner – uses the test pan procedure on a daily basis, starting to get to know how to do them Competent – knows all about test plans, so defines the procedures for other people to follow Proficient – they don’t need the procedure, they will be able to apply the test plan to different contexts. If they get stuck then they might refer to the procedure Expert – will challenge the use of test plans on every project, they might adapt and remove sections for projects since they are not relevant
  • So can practices ever be useful? well yes… Protecting Novices – they need practices because they don’t know what to do Helps Advanced Beginners as they will use the practices to guide them Provides work for competent people because they write the practices. These people write them with no contextual awareness BUT be warned If you continually use “best practice” and insist they are used then your proficient and expert people will leave Let’s take the concept of test scripts. Hands up who uses/ encourages the use/ writes Test Scripts within your organisation? (these are step by step sets of instructions which need to be followed) Well competent people will usually define them, novice and advanced people will use them BUT Expert and Proficient people WILL be bored and will leave! It is like baking a cake and giving the recipe to an expert chef (like Gordon Ramsay)
  • Welcome to the weakest link… This is the show in which we can decide, just by asking a few questions, who the weakest link is in our test team and then we can “fire them”…a fun game that can be played by all Test Managers with their staff… Let me introduce you to the testing team playing today’s weakest link… John…. Carol…. Rick…. Pam…
  • Read the slide… Once the answer has come up say… “ That is great John, well done. You must be pleased with that result!”
  • Read the slide… Once the answer has come up say… “ Well Carol that is most disappointing. I do hope you will try better on the next round!”
  • Read the slide… Once the answer has come up say… “ That is incredible Rick, truly amazing. You must be so pleased with that result and so must your management, excellent work”
  • Read the slide… Once the answer has come up say… “ Zero!! Pam what have you been doing. That result is appalling, even I could have run more than that and I don’t know anything about testing. I am sorry Pam but it is clearly obvious to everyone in this audience that YOU ARE THE WEAKEST LINK…GOODBYE!” I wonder if we sub-consciously think like this in our organisation. Or maybe our management does Now in the above scenario we have 4 different definitions of a test case and 4 definitely different answers. But I bet even if we have the same definition for the 4 contenstants we would have 4 different interpretations.
  • Without context on these test cases the only number that has meaning is ZERO I know that the person hasn’t run ANYTHING!!! With the others they have run SOMETHING, But we don’t know what Now let’s have a look at the same numbers …But with context… My conclusion is that the 2 test cases that were run by Carol were quite detailed and probably testing a lot of functionality. I now understand why Pam has not run anything This scenario REALLY happened during a recent consultancy assignment. Managers were judging the testers purely on the number of test cases being run….without knowing the context
  • Let us consider the problem…. This is a section of an on-line booking system for courses. If we were to test this…what would constitute a test case? Well reading the booking conditions…each hyperlink can be an individual test case testing each field for boundary conditions can each be a test case… Processing an entire booking can be a test case…etc etc THE ANALOGY OF A CASE IS A GOOD ONE BECAUSE We have many different size cases….and cases within cases…each having it’s own unique purpose. BUT the quality of the cases also has relevance as they wont last very long if you have poor quality cases…they break and also might damage the contents….
  • We have a major problem in our organisations with management focusing too much on the QUANTITY… How many have you run? What percentage of test cases are complete? I want our 1.5million test case script run tonight… Without understanding the QUALITY of what is being run.
  • So how can we measure quality of the tests….well with two aspects. There are only 2 reasons for us to tests systems… To Gain confidence…. and to find bugs This all leads to us providing information about the QUALITY of the system so that management/stakeholders can make informed decisions I shall leave you with one final thought…
  • We don’t lie intentionally but we do not spend time understanding the metrics we gather or the graphs we produce. Disraeli said there are 3 types of lies: lies, dammed lies and statistics Go through the graphs. Graph 1: number of bugs being raised (increasing) Graph 2: DMR = the rate at which we are finding them (this is coming down) Graph 3: What is wrong? scale starts with 20 – to make things look good…maybe
  • Pie charts Pie charts: These were created by Florence Nightingale in 1858. It is suggested that Florence Nightingale’s reputation was built on her ability to give clear and concise presentation of data. There were earlier uses Leon Lalanne used these diagrams in 1829 DDP Percentages are great but we need the actual numbers to draw any concrete conclusions. We must have SIGNIFICANT DATA – that means sufficient amount to draw conclusions. This is elementary statistics which we learnt at school. Test Cases The case against test case. The analogy of a case is a good one Show my suit case…one case….open it up and we have a laptop case…open it up ….another case….open it up…another case. Counting cases is meaningless, it is what is inside them that is important. This is why airlines weigh our cases rather than just count them! How many organisations count test cases? How many people encourage counting test cases? QUESTION – what does it actually mean? This was brought home to me recently when I taught an Exploratory Testing class and I got each group to “estimate” the number of tests they run during a 30minute ET session: Group 1: 35 Group 2: 50 Group 4: 90 Group 4: 1 Conclusion – group 4 should be fired!
  • Story about little Suzie…
  • Here are some facts about how we are increasing complexity unnecessarily.
  • NOTE : Testers should improve their technical skills if they want to progress. We must remember that there are some people who are quite happy to stay in the job that they are in and not progress. There are those who treat their work as a job (9-5) there are those who treat their work as a career and those who treat their work as a passion. Look at some of the excuses for Test Managers ….for not testing…
  • This is non-negotiable for me and managers/leads who use excuses for not testing are not worthy of the title “test manager” or “test Lead”. Too many managers it in their ivory towers looking down rather than rolling their sleeves up and getting stuck in. Identify and empathise you will go through similar experiences to the team, you will be involved with them. lead by example William Wallace lead from the front – he was not like the English gentry who lead from the back. His men were committed to their leader. testing yourself will enable you to help with testing when overtime is required improve your estimation skills When I started to test I soon realised that certain tasks took lots more time than I realised gain credibility and trust This is a by-product of being a tester. You can mentor others. Look at leading chefs, pilots, craftsmen
  • Good Generals will not fight every battle – they will choose the ones they think they can win This is the only piece of advice I give to parents…”choose your battles” Choose your battles…story about Hannah THANK YOU
  • 'Top Challenges We Face In IT Today To Hinder Our Pursuit Of Quality' by Lloyd Roden

    1. 1. © Lloyd Roden 1Top Challenges We Face in IT todayto Hinder Our Pursuit of Quality
    2. 2. © Lloyd Roden 2ContentsHow we react to challengesMy top challenges in testing todayHow to make a difference
    3. 3. © Lloyd Roden 3What is a challenge?climbing Mount Everest?running a marathon?cooking a dinner party for 20 people?flying?getting out of bed in the morning?Definition:something that by itsnature or characterserves as a call tobattle, a contest thatrequires special effortand skill. Demand toexplain, justify or callinto question
    4. 4. © Lloyd Roden 4Good versus Bad Challengesbad challengeswhen they are harmfulwhen they have undesirableconsequencesgood challengeswhen they improve yourselfwhen they open your mind toother things
    5. 5. © Lloyd Roden 5How we react…?
    6. 6. © Lloyd Roden 6ContentsHow we react to challengesMy top challenges in testing todayHow to make a difference
    7. 7. © Lloyd Roden 7Challenge #1: ban the use of “best practice”Best: the highest quality, excellence orstanding. Absolute qualifier, contextindependentPractice: habitual or customary performanceTHEREFORE:Best Practice: the highest quality of habitualperformance with no context!so why do we activelypromote “best practice inany lifecycle”?
    8. 8. © Lloyd Roden 8Why “best practice” can be harmfulDreyfus Model (for skills acquisition):NOVICE(needs practices)ADVANCED BEGINNER(uses practices)COMPETENT(defined practices)PROFICIENT(falls back on practices)EXPERT(subverts practices)usesinstinctcontextualawarenessbehavior from instinctmost people stop here
    9. 9. © Lloyd Roden 9So how does this help?where are “best practices” useful?• protecting people from themselves• helping advanced beginners• providing work for competent people suggestion: challenge any references to “Best Practices”replace with good practices and know when and whennot to use theminspired by Dan North, Øredev conferencehowever if “best practices” are usedit will…stifle the creative environmentfrustrate your best people 
    10. 10. © Lloyd Roden 10Challenge #2: Test Cases…quantity or quality?Welcome to the “tester’s weakest link”Johnbeen with the company for 10years and a tester for 5 yearsCarolbeen with the company for 1month and is new to testingRickbeen with the company as asenior tester for 25 yearsPambeen with the company for 3years and a tester for 2 years
    11. 11. © Lloyd Roden 1111John…your questionyour definition of a test case is:“detailed inputs and outputswith pre and post conditions”Question:In the last hour of testing, howmany test cases have yourun?Answer: 30
    12. 12. © Lloyd Roden 1212Carol…your questionyour definition of a test case is:“a set of input values, executionpre-conditions, expected resultsand execution post-conditions,developed to cover certain testconditions”Question:In the last hour of testing, howmany test cases have you run?Answer: 2
    13. 13. © Lloyd Roden 1313Rick…your questionyour definition of a test case is:“one particular instance orvariation of a test or test idea”Question:In the last hour of testing, howmany test cases have yourun?Answer: 300
    14. 14. © Lloyd Roden 1414Pam…your questionyour definition of a test case is:“a computer executable scriptwith a sequence of discretesteps ”Question:In the last hour of testing, howmany test cases have you run?Answer: zerogoodbye!
    15. 15. © Lloyd Roden 1515Counting test cases is meaningless…Without context• 30 test cases• 2 test cases• 300 test cases• 0 test cases…if we don’t know the backgroundWith context• 30 test cases = 30 different updates accepted• 2 test cases = 2 on-line orders processed and dispatched• 300 test cases = 300 button clicks• 0 test cases = install had not completed within the hourzero is the only numberwith meaning
    16. 16. © Lloyd Roden 1616The problem with counting test cases…Public Course Booking Form - Please Complete All FieldsPlease read our booking conditionsPublic course datesCourseDateContact Information for the person Invoice Detailsmaking the bookingCompany NameTitleCompany AddressFirst NameLast NameEmail Purchase Order needed for invoice?Telephone Yes NoContinueWhat is atest casein thisexample?The analogy of a“case” is a good one…Attendees
    17. 17. © Lloyd Roden 1717We focus too much on quantity!Managers seem more interested in counting and monitoring theQUANTITY of test cases to measure progress (and the larger thenumber the better!), without understanding the QUALITY of the testcases:• what are they doing?• do they add value?• what has been covered?… so before getting excited aboutthe sheer number of test cases beingexecuted (quantity)..…we must lookat what the test cases are doing(quality)
    18. 18. © Lloyd Roden 1818So how can we measure the quality ofthe test cases?GAINING CONFIDENCE• in what has been tested• the depth of testing• in how much has beentestedFINDING AND REMOVINGDEFECTS• effectively (number andseverity)• efficiently (quickly)• reducing product risksproviding timelyinformation onthe quality ofthe software –so thatmanagementcan makeinformeddecisions
    19. 19. © Lloyd Roden 1919Be careful of the QUANTITY ofcases…without QUALITY!
    20. 20. © Lloyd Roden 20Challenge #3: Don’t lie with metrics…but I am sure not intentionallylet us look at some graphs…same data with a difference…number of bugs isincreasing?010203040506070801 2 3 4 5 6 7 8 9 10 11 12Bugs203040506070801 2 3 4 5 6 7 8 9 10 11 12but what if we look at the datadifferently…how much effort isexpended in finding the bugs?024681012141 2 3 4 5 6 7 8 9 10 11 12
    21. 21. © Lloyd Roden 21Lies, damned lies and statisticsconclusion: requirements are badconclusion: inconclusive, numbersare not significantsuggestion: metrics we produce MUST be significant. Use basicstatistics to verify your numberswhat about test effectiveness after 1 monthof liveProject 1: DDP = 85%Project 2 : DDP = 60%Project 3 : DDP = 90%Project 4 : DDP = 10%conclusion: reward team 3, punish team 4?why don’t we have a look at the numbers850 15060 409 11 9RequirementsDesignCodeEnvironment1432 1RequirementsDesignCodeEnvironment
    22. 22. © Lloyd Roden 22Challenge # 4: Challenge complexity at every opportunitysimplicity seen as weak and uninterestingwho wants a “basic mobile phone?”complex is seen as goodI don’t understand this,so it must be really good(everyone else understands)$1m pensuggestion: challenge requirements and designdocuments at every opportunity to see whethercomplexity is needed
    23. 23. © Lloyd Roden 23Features and functions usedJim Johnson XP2002 Standish Study GroupFeatures and Functions Used16%13%7%19%45%Sometimes UsedOften UsedAlways UsedRarely UsedNever UsedFeatures and Functions Used20%64%16%Often and Alw aysUsedRarely or Never UsedSometimesthis means we have driven upcomplexity by putting in thingsthat are not required
    24. 24. © Lloyd Roden 24Challenge # 5: Test Managers/Leads should test…and testers should improve their technical skillssome excuses that have been given to me…“I can’t perform testing as Idon’t know the application”“I don’t have time to performtesting, I have meetings toattend, reports to write,schedules to monitor…”“I have done testing in the past, I havenow moved on. I have my team who doit much better”“I have 32 testersreporting to me, youare not seriouslysuggesting that Itest as well?
    25. 25. © Lloyd Roden 25Why is this non-negotiable?why do I believe Test Managers/Leads should test?you can identify & empathize with the team’sfrustrationsyou can lead by exampleassist when more testing is requiredavoid the “wool” being pulledyou will improve your estimation skillsyou will gain credibility and trust within theorganizationsuggestion: adopt Friday afternoon ExploratoryTesting/Bug hunting afternoons which includes theTest Lead/Manager
    26. 26. © Lloyd Roden 26ContentsHow we react to challengesMy top challenges in testing todayHow to make a difference
    27. 27. © Lloyd Roden 27What to do now…produce an action planwhich of these challenges are affecting you?what other challenges are you facing?produce a series of steps to takediscuss these with your manager/teamthis can be the hardest part!recognise that challenges will stir emotionsbut don’t give up!choose your battles
    28. 28. © Lloyd Roden 28Summarywe need to understand howhumans react to challengesand the importance offacing them to shape ourpersonality, character andunderstandingmy top challenges…theyare not exhaustive by anymeansthe step we now take isimportant to make adifference in ourworkplaces

    ×