Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
S Y S T E M AT I C U N I T
T E S T I N G
S C O T T G R A N T
A B O U T M E
• Scott Grant (hi!)
• Startup CTO in Kingston, Ontario
• Formerly @ Electronic Arts, Google
• Formerly Adjun...
A B O U T M E G A M A N 2
• Developed and published by
Capcom in 1988 for the Nintendo
Entertainment System
• In 200X, Dr....
W H AT I S S Y S T E M AT I C
T E S T I N G ?
W H AT I S S Y S T E M AT I C T E S T I N G ?
• An explicit discipline or procedure (a system) for:
W H AT I S S Y S T E M AT I C T E S T I N G ?
• An explicit discipline or procedure (a system) for:
• choosing and creatin...
W H AT I S S Y S T E M AT I C T E S T I N G ?
• An explicit discipline or procedure (a system) for:
• choosing and creatin...
W H AT I S S Y S T E M AT I C T E S T I N G ?
• An explicit discipline or procedure (a system) for:
• choosing and creatin...
W H AT I S S Y S T E M AT I C T E S T I N G ?
• An explicit discipline or procedure (a system) for:
• choosing and creatin...
U N I T T E S T S
• Units are, preferably, the smallest testable part of an
application
• Unit tests are designed to test ...
U N I T T E S T S
• Units are, preferably, the smallest testable part of an
application
• Unit tests are designed to test ...
U N I T T E S T S
• Units are, preferably, the smallest testable part of an
application
• Unit tests are designed to test ...
U N I T T E S T S
• Units are, preferably, the smallest testable part of an
application
• Unit tests are designed to test ...
A S S E R T I O N S
• In many ways, the heart of a unit test
A S S E R T I O N S
• In many ways, the heart of a unit test
• Checks for a predefined constraint about the program
state
...
A S S E R T I O N S
• In many ways, the heart of a unit test
• Checks for a predefined constraint about the program
state
...
P H P U N I T
• Provides a framework for creating and executing unit
tests easily, quickly, and reproducibly
• Based on at...
1 . C H O O S I N G A N D
C R E AT I N G T E S T C A S E S
W H AT M A K E S A G O O D T E S T ?
W H AT M A K E S A G O O D T E S T ?
• Independent: Each test needs to run independently
from other tests and environments
W H AT M A K E S A G O O D T E S T ?
• Independent: Each test needs to run independently
from other tests and environments...
W H AT M A K E S A G O O D T E S T ?
• Independent: Each test needs to run independently
from other tests and environments...
W H AT M A K E S A G O O D T E S T ?
• Up to date: Tests are written once, but code can be
changed or extended
W H AT M A K E S A G O O D T E S T ?
• Up to date: Tests are written once, but code can be
changed or extended
• Short: Te...
W H AT M A K E S A G O O D T E S T ?
• Up to date: Tests are written once, but code can be
changed or extended
• Short: Te...
T E S T S I N I S O L AT I O N
T E S T S I N I S O L AT I O N
• Before each test, initialize the appropriate state
T E S T S I N I S O L AT I O N
• Before each test, initialize the appropriate state
• Next, perform the test
T E S T S I N I S O L AT I O N
• Before each test, initialize the appropriate state
• Next, perform the test
• One test! O...
T E S T S I N I S O L AT I O N
• Before each test, initialize the appropriate state
• Next, perform the test
• One test! O...
T E S T S I N I S O L AT I O N
• Before each test, initialize the appropriate state
• Next, perform the test
• One test! O...
function is_even( $x ) {
if ( 0 == $x % 2 ) {
return true;
}
return false;
}
function is_odd( $x ) {
// ??
}
class Test_WCTO extends WP_UnitTestCase {
public function test_even_with_even() {
$this->assertTrue( is_even( 2 ) );
}
pub...
C H O O S I N G & O R G A N I Z I N G T E S T S
• The experimental model of software testing gives us
two important princi...
C H O O S I N G & O R G A N I Z I N G T E S T S
• The experimental model of software testing gives us
two important princi...
C H O O S I N G & O R G A N I Z I N G T E S T S
• The experimental model of software testing gives us
two important princi...
function is_odd( $x ) {
return ! is_even( $x );
}
function is_odd( $x ) {
$remainder = $x % 2;
if ( $remainder = 1 ) {
return true;
}
return false;
}
2 . E X E C U T I N G T H E T E S T S A N D
D O C U M E N T I N G T H E R E S U LT S
T E S T R E P O R T S
• Output of test execution should be summarized in a
readable report
T E S T R E P O R T S
• Output of test execution should be summarized in a
readable report
• Test reports should be design...
T E S T R E P O R T S
• Output of test execution should be summarized in a
readable report
• Test reports should be design...
PHPUnit 4.5.0 by Sebastian Bergmann
and contributors.
Configuration read from /tmp/
wordpress/src/wp-content/plugins/two-
...
PHPUnit 4.4.0 by Sebastian Bergmann.
Configuration read from /home/travis/build/scotchfield/arcadia/
test/phpunit_travis.x...
3 . E VA L U AT I N G T H E R E S U LT S ,
P O S S I B LY A U T O M AT I C A L LY
T E S T R U N S U C C E S S
• When has a test run successfully completed?
T E S T R U N S U C C E S S
• When has a test run successfully completed?
• When all of the unit tests pass!
T E S T R U N S U C C E S S
• When has a test run successfully completed?
• When all of the unit tests pass!
• When does a...
T E S T R U N S U C C E S S
• When has a test run successfully completed?
• When all of the unit tests pass!
• When does a...
T E S T R U N S U C C E S S
• When has a test run successfully completed?
• When all of the unit tests pass!
• When does a...
T E S T R U N S U C C E S S
• When has a test run successfully completed?
• When all of the unit tests pass!
• When does a...
S A F E T Y B L A N K E T
• When a programmer makes a change to the code (any
change), they can run the test suite
S A F E T Y B L A N K E T
• When a programmer makes a change to the code (any
change), they can run the test suite
• If al...
S A F E T Y B L A N K E T
• When a programmer makes a change to the code (any
change), they can run the test suite
• If al...
http://cybermoonstudios.deviantart.com/art/Mega-Man-2-Robot-Masters-194915393
A U T O M AT E D T E S T I N G
• A continuous build machine can run unit tests after
every commit
A U T O M AT E D T E S T I N G
• A continuous build machine can run unit tests after
every commit
• Useful even if a progr...
A U T O M AT E D T E S T I N G
• A continuous build machine can run unit tests after
every commit
• Useful even if a progr...
A U T O M AT E D T E S T I N G
• A continuous build machine can run unit tests after
every commit
• Useful even if a progr...
4 . D E C I D I N G W H E N W E A R E
D O N E ( E N O U G H T E S T I N G )
W H E N A R E W E D O N E ?
• When we've achieved 100% code coverage
W H E N A R E W E D O N E ?
• When we've achieved 100% code coverage
• Not an ideal statistic
W H E N A R E W E D O N E ?
• When we've achieved 100% code coverage
• Not an ideal statistic
• Does not check for input, ...
W H E N A R E W E D O N E ?
• When we've achieved 100% code coverage
• Not an ideal statistic
• Does not check for input, ...
W H E N A R E W E D O N E ?
• When we've achieved 100% code coverage
• Not an ideal statistic
• Does not check for input, ...
C O D E C O V E R A G E
• PHPUnit has a set of --coverage-* options
C O D E C O V E R A G E
• PHPUnit has a set of --coverage-* options
• Generate code coverage reports in various formats
C O D E C O V E R A G E
• PHPUnit has a set of --coverage-* options
• Generate code coverage reports in various formats
• ...
C O D E C O V E R A G E
• PHPUnit has a set of --coverage-* options
• Generate code coverage reports in various formats
• ...
C O D E C O V E R A G E
• PHPUnit has a set of --coverage-* options
• Generate code coverage reports in various formats
• ...
https://github.com/scotchfield/roller
https://github.com/scotchfield/roller
https://github.com/scotchfield/arcadia
https://github.com/scotchfield/arcadia
A systematic approach helps inform where new
tests are needed and provides one way to know
when a test suite is complete.
...
Systematic Unit Testing
Systematic Unit Testing
Systematic Unit Testing
Systematic Unit Testing
Systematic Unit Testing
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
Building and Maintaining A Remote Workforce - A Startup Story
Next
Upcoming SlideShare
Building and Maintaining A Remote Workforce - A Startup Story
Next
Download to read offline and view in fullscreen.

Share

Systematic Unit Testing

Download to read offline

Unit testing is an important part of verifying that code works as expected. However, how many tests do you need to write before you can be confident that you’ve done a good job? Where should you be spending your time writing tests, and how do you know when you’re done?

In this talk, we discuss a systematic approach to unit testing WordPress plugins using phpunit. We show how to set up a simple test suite, to examine the code coverage options, and to identify the most risky areas of our code that are in need of tests.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Systematic Unit Testing

  1. 1. S Y S T E M AT I C U N I T T E S T I N G S C O T T G R A N T
  2. 2. A B O U T M E • Scott Grant (hi!) • Startup CTO in Kingston, Ontario • Formerly @ Electronic Arts, Google • Formerly Adjunct Prof. @ Queen's University • Prof. James R. Cordy (CISC327: Software Quality Assurance)
  3. 3. A B O U T M E G A M A N 2 • Developed and published by Capcom in 1988 for the Nintendo Entertainment System • In 200X, Dr. Wily and his eight evil Robot Masters threaten Mega Man, Dr. Light, and the world
  4. 4. W H AT I S S Y S T E M AT I C T E S T I N G ?
  5. 5. W H AT I S S Y S T E M AT I C T E S T I N G ? • An explicit discipline or procedure (a system) for:
  6. 6. W H AT I S S Y S T E M AT I C T E S T I N G ? • An explicit discipline or procedure (a system) for: • choosing and creating test cases
  7. 7. W H AT I S S Y S T E M AT I C T E S T I N G ? • An explicit discipline or procedure (a system) for: • choosing and creating test cases • executing the tests and documenting the results
  8. 8. W H AT I S S Y S T E M AT I C T E S T I N G ? • An explicit discipline or procedure (a system) for: • choosing and creating test cases • executing the tests and documenting the results • evaluating the results, possibly automatically
  9. 9. W H AT I S S Y S T E M AT I C T E S T I N G ? • An explicit discipline or procedure (a system) for: • choosing and creating test cases • executing the tests and documenting the results • evaluating the results, possibly automatically • deciding when we are done (enough testing)
  10. 10. U N I T T E S T S • Units are, preferably, the smallest testable part of an application • Unit tests are designed to test small pieces of the code (units) independently
  11. 11. U N I T T E S T S • Units are, preferably, the smallest testable part of an application • Unit tests are designed to test small pieces of the code (units) independently function sum( $a, $b ) { return $a + $b; }
  12. 12. U N I T T E S T S • Units are, preferably, the smallest testable part of an application • Unit tests are designed to test small pieces of the code (units) independently function sum( $a, $b ) { return $a + $b; } sum( 1, 1 ) == 2
  13. 13. U N I T T E S T S • Units are, preferably, the smallest testable part of an application • Unit tests are designed to test small pieces of the code (units) independently function sum( $a, $b ) { return $a + $b; } sum( 1, 1 ) == 2 $this->assertEquals( 2, sum( 1, 1 ) );
  14. 14. A S S E R T I O N S • In many ways, the heart of a unit test
  15. 15. A S S E R T I O N S • In many ways, the heart of a unit test • Checks for a predefined constraint about the program state • "After sum is called with the arguments 1 and 1, it will return the value 2."
  16. 16. A S S E R T I O N S • In many ways, the heart of a unit test • Checks for a predefined constraint about the program state • "After sum is called with the arguments 1 and 1, it will return the value 2." • If the constraint does not hold, the test fails • The test asserts that the code is correct for a particular context if the assertion holds
  17. 17. P H P U N I T • Provides a framework for creating and executing unit tests easily, quickly, and reproducibly • Based on atomic unit tests with assertions • Open-source! • https://github.com/sebastianbergmann/phpunit
  18. 18. 1 . C H O O S I N G A N D C R E AT I N G T E S T C A S E S
  19. 19. W H AT M A K E S A G O O D T E S T ?
  20. 20. W H AT M A K E S A G O O D T E S T ? • Independent: Each test needs to run independently from other tests and environments
  21. 21. W H AT M A K E S A G O O D T E S T ? • Independent: Each test needs to run independently from other tests and environments • Fast: To have useful tests and to be able to run them as often as possible, tests need to be fast
  22. 22. W H AT M A K E S A G O O D T E S T ? • Independent: Each test needs to run independently from other tests and environments • Fast: To have useful tests and to be able to run them as often as possible, tests need to be fast • Repeatable: You should be able to run a test as many times as you want with the same result
  23. 23. W H AT M A K E S A G O O D T E S T ? • Up to date: Tests are written once, but code can be changed or extended
  24. 24. W H AT M A K E S A G O O D T E S T ? • Up to date: Tests are written once, but code can be changed or extended • Short: Tests should be just a few lines--easy to read and understand
  25. 25. W H AT M A K E S A G O O D T E S T ? • Up to date: Tests are written once, but code can be changed or extended • Short: Tests should be just a few lines--easy to read and understand • Resilient: Once written, tests shouldn't change until the behaviour of tested class/method changes
  26. 26. T E S T S I N I S O L AT I O N
  27. 27. T E S T S I N I S O L AT I O N • Before each test, initialize the appropriate state
  28. 28. T E S T S I N I S O L AT I O N • Before each test, initialize the appropriate state • Next, perform the test
  29. 29. T E S T S I N I S O L AT I O N • Before each test, initialize the appropriate state • Next, perform the test • One test! One!!
  30. 30. T E S T S I N I S O L AT I O N • Before each test, initialize the appropriate state • Next, perform the test • One test! One!! • After each test, clean up after yourself
  31. 31. T E S T S I N I S O L AT I O N • Before each test, initialize the appropriate state • Next, perform the test • One test! One!! • After each test, clean up after yourself • There are more tests after this one, we don't want to corrupt them with our test data
  32. 32. function is_even( $x ) { if ( 0 == $x % 2 ) { return true; } return false; } function is_odd( $x ) { // ?? }
  33. 33. class Test_WCTO extends WP_UnitTestCase { public function test_even_with_even() { $this->assertTrue( is_even( 2 ) ); } public function test_even_with_odd() { $this->assertFalse( is_even( 1 ) ); } }
  34. 34. C H O O S I N G & O R G A N I Z I N G T E S T S • The experimental model of software testing gives us two important principles for our test plan:
  35. 35. C H O O S I N G & O R G A N I Z I N G T E S T S • The experimental model of software testing gives us two important principles for our test plan: • Test inputs should be chosen to carefully isolate different causes of failure (the experimental variables)
  36. 36. C H O O S I N G & O R G A N I Z I N G T E S T S • The experimental model of software testing gives us two important principles for our test plan: • Test inputs should be chosen to carefully isolate different causes of failure (the experimental variables) • Test cases should be ordered such that each test only assumes features to be working that have already been tested by a previous test
  37. 37. function is_odd( $x ) { return ! is_even( $x ); }
  38. 38. function is_odd( $x ) { $remainder = $x % 2; if ( $remainder = 1 ) { return true; } return false; }
  39. 39. 2 . E X E C U T I N G T H E T E S T S A N D D O C U M E N T I N G T H E R E S U LT S
  40. 40. T E S T R E P O R T S • Output of test execution should be summarized in a readable report
  41. 41. T E S T R E P O R T S • Output of test execution should be summarized in a readable report • Test reports should be designed to be concise, easy to read, and to clearly point out failures or unexpectedly changed results
  42. 42. T E S T R E P O R T S • Output of test execution should be summarized in a readable report • Test reports should be designed to be concise, easy to read, and to clearly point out failures or unexpectedly changed results • Test results should be in a standardized form for easy comparison with future test executions
  43. 43. PHPUnit 4.5.0 by Sebastian Bergmann and contributors. Configuration read from /tmp/ wordpress/src/wp-content/plugins/two- factor/phpunit.xml ..................................... ................ Time: 11.37 seconds, Memory: 71.50Mb OK (53 tests, 90 assertions)
  44. 44. PHPUnit 4.4.0 by Sebastian Bergmann. Configuration read from /home/travis/build/scotchfield/arcadia/ test/phpunit_travis.xml.dist ............................................................... ............................................................... ......................................................F........ .................................... Time: 2.52 seconds, Memory: 9.50Mb There was 1 failure: 1) TestArcadiaUser::test_get_user_by_name_simple Failed asserting that false is not false. /home/travis/build/scotchfield/arcadia/test/tests/include/ test_user.php:43 FAILURES! Tests: 225, Assertions: 292, Failures: 1.
  45. 45. 3 . E VA L U AT I N G T H E R E S U LT S , P O S S I B LY A U T O M AT I C A L LY
  46. 46. T E S T R U N S U C C E S S • When has a test run successfully completed?
  47. 47. T E S T R U N S U C C E S S • When has a test run successfully completed? • When all of the unit tests pass!
  48. 48. T E S T R U N S U C C E S S • When has a test run successfully completed? • When all of the unit tests pass! • When does a unit test pass?
  49. 49. T E S T R U N S U C C E S S • When has a test run successfully completed? • When all of the unit tests pass! • When does a unit test pass? • When all of the assertions hold!
  50. 50. T E S T R U N S U C C E S S • When has a test run successfully completed? • When all of the unit tests pass! • When does a unit test pass? • When all of the assertions hold! • When does an assertion hold?
  51. 51. T E S T R U N S U C C E S S • When has a test run successfully completed? • When all of the unit tests pass! • When does a unit test pass? • When all of the assertions hold! • When does an assertion hold? • When that individual piece of expected behaviour captured in the assertion holds!
  52. 52. S A F E T Y B L A N K E T • When a programmer makes a change to the code (any change), they can run the test suite
  53. 53. S A F E T Y B L A N K E T • When a programmer makes a change to the code (any change), they can run the test suite • If all of the tests pass, and we believe the test suite covers enough functionality, we're fairly confident that the code works
  54. 54. S A F E T Y B L A N K E T • When a programmer makes a change to the code (any change), they can run the test suite • If all of the tests pass, and we believe the test suite covers enough functionality, we're fairly confident that the code works • If a test fails, we might have broken something (or exposed a bug!), and can fix it before checking in
  55. 55. http://cybermoonstudios.deviantart.com/art/Mega-Man-2-Robot-Masters-194915393
  56. 56. A U T O M AT E D T E S T I N G • A continuous build machine can run unit tests after every commit
  57. 57. A U T O M AT E D T E S T I N G • A continuous build machine can run unit tests after every commit • Useful even if a programmer isn't proactively running unit tests
  58. 58. A U T O M AT E D T E S T I N G • A continuous build machine can run unit tests after every commit • Useful even if a programmer isn't proactively running unit tests • Catch bugs as they're introduced
  59. 59. A U T O M AT E D T E S T I N G • A continuous build machine can run unit tests after every commit • Useful even if a programmer isn't proactively running unit tests • Catch bugs as they're introduced • Or provide confidence that the code isn't going to cause problems
  60. 60. 4 . D E C I D I N G W H E N W E A R E D O N E ( E N O U G H T E S T I N G )
  61. 61. W H E N A R E W E D O N E ? • When we've achieved 100% code coverage
  62. 62. W H E N A R E W E D O N E ? • When we've achieved 100% code coverage • Not an ideal statistic
  63. 63. W H E N A R E W E D O N E ? • When we've achieved 100% code coverage • Not an ideal statistic • Does not check for input, output, or functionality coverage
  64. 64. W H E N A R E W E D O N E ? • When we've achieved 100% code coverage • Not an ideal statistic • Does not check for input, output, or functionality coverage • Hard to achieve in practice
  65. 65. W H E N A R E W E D O N E ? • When we've achieved 100% code coverage • Not an ideal statistic • Does not check for input, output, or functionality coverage • Hard to achieve in practice • However, it's a clear and unambiguous goal
  66. 66. C O D E C O V E R A G E • PHPUnit has a set of --coverage-* options
  67. 67. C O D E C O V E R A G E • PHPUnit has a set of --coverage-* options • Generate code coverage reports in various formats
  68. 68. C O D E C O V E R A G E • PHPUnit has a set of --coverage-* options • Generate code coverage reports in various formats • Checks which lines of code are covered by at least one unit test
  69. 69. C O D E C O V E R A G E • PHPUnit has a set of --coverage-* options • Generate code coverage reports in various formats • Checks which lines of code are covered by at least one unit test • In addition, functions and classes
  70. 70. C O D E C O V E R A G E • PHPUnit has a set of --coverage-* options • Generate code coverage reports in various formats • Checks which lines of code are covered by at least one unit test • In addition, functions and classes • Does not check for input, output, or functionality coverage
  71. 71. https://github.com/scotchfield/roller
  72. 72. https://github.com/scotchfield/roller
  73. 73. https://github.com/scotchfield/arcadia
  74. 74. https://github.com/scotchfield/arcadia
  75. 75. A systematic approach helps inform where new tests are needed and provides one way to know when a test suite is complete. @ S C O T C H F I E L D @ S G R A N T S C O T T @ S C O O TA H . C O M

Unit testing is an important part of verifying that code works as expected. However, how many tests do you need to write before you can be confident that you’ve done a good job? Where should you be spending your time writing tests, and how do you know when you’re done? In this talk, we discuss a systematic approach to unit testing WordPress plugins using phpunit. We show how to set up a simple test suite, to examine the code coverage options, and to identify the most risky areas of our code that are in need of tests.

Views

Total views

1,549

On Slideshare

0

From embeds

0

Number of embeds

29

Actions

Downloads

7

Shares

0

Comments

0

Likes

0

×