Scrum Gathering 2012 Shanghai)_敏捷测试与质量管理分会场演讲话题:pros cons of system test only(康美国)


Published on

讲师: 康美国
康美国 has been doing Extreme Programming since 1999 when Kent Beck coached 康’s team at HP. Afterwards, he added Scrum to his XP practices after he joined SolutionsIQ in 2004 and has successfully blended the two together ever since. He is both a developer, a technical coach, and a process coached and has Agile consulted with teams both in the US and China, companies such as: AT&T Wireless, TMobile, Visa, HuaWei, and ZTE.

Some companies are trying to implement a System Test only strategy in lue of unit testing. Let’s look at the impacts of this decision using both real world data from development teams and conceptual models developed by 康。 Definition of a unit test. Definition of a system test. Look at some real world teams (two in the US and one in ChengDu) and their projects and how a system test only strategy affected their ability to deliver. Characteristics: application architecture system testing tools speed of test feedback to revealing a regression cost of test maintenance discovering defect location code quality Theoretical models to use for discussing impact to team productivity of doing ‘TDD & System Testing’ versus ‘System Test only’.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Scrum Gathering 2012 Shanghai)_敏捷测试与质量管理分会场演讲话题:pros cons of system test only(康美国)

  1. 1. System Testing Only Strategy 康美国(Lance Kind)
  2. 2. Pros/Cons of System Test ONLY
  3. 3. Background –康美国, Lance Kind• Working in a XP environment since 1999• Agile Coaching since 2006• Development Lead• Certified Scrum Master• Certified Professional Snowboard/Ski Instructor• TDD fanatic• 15 years in software development and Internet technologies• Science fiction author (
  4. 4. Why Automate Testing?• Developers using debuggers as manual unit testing tools takes a long time and it’s not repeated before each check in.• If testing isn’t significantly automated, we can’t easily check whether the system is running.• The two major types of automated testing are testing the whole system as a black box (System Testing) and testing each individual piece directly as small black boxes (Unit Tests).
  5. 5. What Automated Testing doesn’t do well!• Find bugs for which there ARE NOT automated tests.• There’ll always be a need for some amount of manual testing.
  6. 6. The difference between System Tests and Unit Tests• System tests tell us if the system does what the customer expects• Unit tests tell us if a *part* of the system does what the developers expect
  7. 7. System tests• Tell us whether the system does what the customer expects• Enable developers to know they’ve satisfied requirements• Can be run automatically at any time by anyone
  8. 8. Traditional balance of tests • Easy to Create Manual GUI • Very familiar – what we always do Acceptance • Typically tedious Tests • How do we know coverage? • There are thousands of these. • Need Automation specialists Automated • Automation good for performance GUI Tests • Seems like we always rewrite Unit • Sometimes fragile Tests • What is Development testing? • How do we know what these are? • How do we know when they fail?(Shamelessly borrowed from Mike Cohn)
  9. 9. Testing Pyramid •Fewest in number •Automate as much as possible GUI Acceptance •Less than 100 and take minutes to hours Tests •Drive development and acceptanceFitNesse •Hundreds and take hours to days •Do the most •Create Test Driven DesignUnit Tests •Thousands and run in minutes
  10. 10. Quality Decision MakersTraditional: Agile:• QA • Same as the left but Management add:• Architects • Development• Testing Teams engineers
  11. 11. Real World Data Collected from the following:• As a developer and consultant: – AT&T Wireless internal app – KeyBank College Loan web app• As a consultant: – Large Chinese Co. that makes router products.
  12. 12. AT&T Wireless• Project profile: – Internal web application – JEE, Java script, AJAX, SQL server – View heavy and DB heavy application – Six years of history (team of 8): • 600+ system tests (from view layer, down), 4 days for full system test pass, suite of intermittent failures • 300 unit tests, a minute for full unit test pass (JSUnit, JUnit)
  13. 13. AT&T Wireless Results• Good: – Shipped to customer every 4 months – Functional Quality very high• Impediments: – Code/design quality middle to low – View layer was thick as there was no technical/practical reason to make small – Developers were reluctant to improve design (refactoring) as the code coverage was low and the feedback loop slow (days to discover regressions). – When regression occurred, it took hours to a day to discover why the system tests failed.
  14. 14. China Router Co.’s• Project profile: – Embedded controller application to build routers used by telecoms – C++/VX Works – Separate system test team using Ruby system test scripts – 3 month project (team of 7 but system testers on external team): • High hundreds of system tests, days to a week for full system test pass • No unit tests.
  15. 15. China Router Co.’s biggest impediment• If a system test failed, it took days to weeks to resolve what caused the failure.• No unit tests were being developed.• Code/design quality was low with lots of duplicate code and commented out blocks of code.• No one ever refactored.• The team feared System Test failure reports!
  16. 16. KeyBank• Project profile: – External web application used by college students – Java Server Faces, TSQL – View heavy and DB heavy application – Four month project (team of 5): • 80 system tests (from view layer down, and at Web Service Layer), 2 hours days for full system test pass, suite of intermittent failures • 30unit tests, seconds for full unit test pass (JSUnit, JUnit)
  17. 17. KeyBank month #3 of 4 Results• Good: – The developers were very productive in creating system tests and functionality. – Code checkins were happening every 30 minutes to an hour.• Bad: – Few unit tests were being developed. – Code/design quality was only medium to low. Month #3 code review report was poor. – View layer was thick because unit testing was being skipped, so we didn’t have small, decoupled objects. – System Tests seemed unstable because upon each execution, different tests would fail. The fixes sometimes maybe were already checked in, but no one knew for sure until the next morning when they could view the results before work started again. – Everyone agreed that the situation was bad. The code review was a wakeup call.
  18. 18. We took action!• Even though we were in a hurry, (who isn’t?) we refactored like madmen!• Results after one week: – The big view layer classes were thinned and many small unit tested objects were created. – 80 unit tests were added – Duplicate code was reduced – We were proud of our source code again! – System tests still acted unstable as they couldn’t keep up with the pace of our changes. Only see green bar for system tests two times a week. – We stopped/slowed our changes 3 days before release and finally got GREEN BAR on the system test results – Our release was successful and no bugs reported by client.
  19. 19. Summary• System Test Only is easier to “roll out” in an organization” than System Test & TDD because: – Without TDD, developers don’t need to change their habits and be responsible for their own testing. – A separate System test team can create system tests against interfaces. – An organization doesn’t need to change itself very much. – Poor code design doesn’t have to be redesigned so it’s testable for unit tests.
  20. 20. Summary part 2• System Test Only has problems: – Slow feedback means we can’t quickly find out our project status. – When a system test fails, the scope is so large it takes half a day to weeks to find the problem where unit tests rarely take more than an hour and often 15-20 minutes. – Design quality is low as there’s nothing (other than code review) forcing developers to write decoupled code with good interface design. – Relying on code review to improve code is slow: (code->manual test (and maybe wait for system test report)->debug->have a code-review meeting->back to code step to fix code) Because it’s slow, people in a hurry will not improve code much. – Refactoring is infrequent because it’s not possible to discover in seconds to minutes if there’s a regression.
  21. 21. To make a System-Test-Only strategy work well• Even if it’s possible to get the same code coverage as TDD (90%+) … then there are still these problems:• Slow feedback--System Tests are too slow. Somehow, we need our full System Test suit to execute in seconds to minutes like Unit Tests.• Hard to FIND the bug: System Tests take hours to weeks to find the bug where Unit Tests take minutes to an hour.• Until then these problems are solved, I’ll keep using Test Driven Development to make Unit Tests. Acceptance Driven Development doesn’t help these problems at all.
  22. 22. Contact me: 康美国, Lance KindMobile: 18606926591LancerKind@gmail.comTwitter: lancerkindSkype: lancerkindAnd contact me through Bill Li (李国彪) of UPerform: 18602103637