Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Git and Testing
December, 2015
Christian Couder
chriscool@tuxfamily.org
A Distributed Version Control System (DVCS):
● created by Linus Torvalds
● maintained by Junio Hamano
● since 2005
● prefe...
● started developing Git in 2006
● worked especially on git bisect
● independent consultant
● started working on IPFS in 2...
● “The Permanent Web”, see http://ipfs.io/
● Content addressed, versioned, peer to peer
filesystem
● Like: Web + Git + Bit...
● In “http://example.com/foo/bar.png“,
“example.com” is translated into an IP address
like 10.20.30.40 which is a location...
● Pre-alpha software
● No black box tests, only some unit tests
● Heavy development
● A lot of regressions
=> Really neede...
● 10 years ago Junio Hamano created the Git test
framework
● Developed in shell (POSIX /bin/sh compatible)
● Extracted 4 y...
#!/bin/sh
test_description=”Sharness example”
. sharness/sharness.sh
test_expect_success “Testing echo” '
echo Hello world...
$ ./example.t
ok 1 - Testing echo
# passed all 1 test(s)
1..1
Sharness example (2)
Sharness feature: prerequisites
Some tests will be launched only if some condition is
satisfied, for example:
test "$TEST_...
Sharness feature: result aggregation
Other Sharness features
● Easily extensible with your own test functions
● Many options: verbose output, stop immediately ...
● Now IPFS has around 24 Sharness test scripts
● That means around 300 tests
● Run automatically on each commit of each pu...
● Add both unit tests and black box tests
● Avoid breaking backward compatibility even for
pre-alpha software
Avoid Regres...
● Regressions still happen sometimes as tests
don't cover everything
=> git bisect
● Each commit must compile and pass tes...
Commits in Git form a DAG
(Directed Acyclic Graph)
●
history direction is from left to right
●
new commits point to their ...
First Bad Commit
B
●
B introduces a bad behavior called "bug" or "regression"
●
red commits are called "bad"
●
blue commit...
Idea:
● help find a first bad commit
● use a binary search algorithm for efficiency if
possible
Benefits:
● manually verif...
2 ways to do it:
$ git bisect start
$ git bisect bad [<BAD>]
$ git bisect good [<GOOD>...]
or
$ git bisect start <BAD> <GO...
(toy example with the linux kernel)
$ git bisect start v2.6.27 v2.6.25
Bisecting: 10928 revisions left to test
after this ...
1.test the current commit
2.tell "git bisect" whether it is good or bad, for example:
$ git bisect bad
Bisecting: 5480 rev...
$ git bisect bad
2ddcca36c8bcfa251724fe342c8327451988be0d is the
first bad commit
commit 2ddcca36c8bcfa251724fe342c8327451...
When the first bad commit is found:
● you can check it out and tinker with it, or
● you can use "git bisect reset", like t...
At each bisection step a script or command will be
launched to tell if the current commit is good or bad.
Syntax:
$ git bi...
$ git bisect start v2.6.27 v2.6.25
Bisecting: 10928 revisions left to test after
this (roughly 14 steps)
[2ec65f8b89ea003c...
SUBLEVEL = 25
Bisecting: 2740 revisions left to test after
this (roughly 12 steps)
[671294719628f1671faefd4882764886f8ad08...
2ddcca36c8bcfa251724fe342c8327451988be0d is the
first bad commit
commit 2ddcca36c8bcfa251724fe342c8327451988be0d
Author: L...
0 => good
1-124 and 126-127 => bad
125 => skip
128-255 => stop
"skip": mark commit as "untestable",
"git bisect" will choo...
Problem when bisecting
Sometimes the commit that introduced a bug will be in an
untestable area of the graph.
For example:...
There are only 'skip'ped commits left to test.
The first bad commit could be any of:
15722f2fa328eaba97022898a305ffc8172db...
Possible solutions
Possible solutions to bisect anyway:
●
apply a patch before testing and remove it afterwards (can
be do...
A good solution
X X1 X2 X3W Y Z
X + Y X1' X2' X3' Z' Z1
The idea is that we will replace Z with Z' so that we bisect from ...
● ./example.t
● vi t0040-add-and-cat.sh
● TEST_EXPENSIVE=1 ./t0040-add-and-cat.sh -v
● cd test/sharness && make
● git reba...
● https://github.com/mlafeldt/sharness
● https://github.com/chriscool/sharnessify
● http://git-scm.com
● http://ipfs.io
Li...
Conclusion
● Black box tests with Sharness are quite easy and very useful
● Check every commit, not just every pull reques...
Many thanks to:
● Junio Hamano, Linus Torvalds,
● Juan Benet, Jeromy Johnson, Mathias Lafeldt
● many other great people in...
Questions?
Upcoming SlideShare
Loading in …5
×

Git and Testing

103 views

Published on

Describe Git related tools and practices to help testing.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Git and Testing

  1. 1. Git and Testing December, 2015 Christian Couder chriscool@tuxfamily.org
  2. 2. A Distributed Version Control System (DVCS): ● created by Linus Torvalds ● maintained by Junio Hamano ● since 2005 ● preferred VCS About Git
  3. 3. ● started developing Git in 2006 ● worked especially on git bisect ● independent consultant ● started working on IPFS in 2014 About myself
  4. 4. ● “The Permanent Web”, see http://ipfs.io/ ● Content addressed, versioned, peer to peer filesystem ● Like: Web + Git + BitTorrent ● Alpha software written using the Go language ● Has a Command Line Interface About IPFS (1)
  5. 5. ● In “http://example.com/foo/bar.png“, “example.com” is translated into an IP address like 10.20.30.40 which is a location; the Web is location addressed ● In “/ipns/example.com/foo/bar.png”, “example.com” is translated into a hash of some content like QmW98pJrc6FZ6; IPFS is content addressed About IPFS (2)
  6. 6. ● Pre-alpha software ● No black box tests, only some unit tests ● Heavy development ● A lot of regressions => Really needed black box tests IPFS in October 2014 (last year)
  7. 7. ● 10 years ago Junio Hamano created the Git test framework ● Developed in shell (POSIX /bin/sh compatible) ● Extracted 4 years ago by Mathias Lafeldt into a separate project called Sharness ● It's the reason why Git has always been very stable Sharness
  8. 8. #!/bin/sh test_description=”Sharness example” . sharness/sharness.sh test_expect_success “Testing echo” ' echo Hello world >actual && grep Hello actual ' test_done Sharness example (1)
  9. 9. $ ./example.t ok 1 - Testing echo # passed all 1 test(s) 1..1 Sharness example (2)
  10. 10. Sharness feature: prerequisites Some tests will be launched only if some condition is satisfied, for example: test "$TEST_EXPENSIVE" = 1 && test_set_prereq EXPENSIVE and then: test_expect_success EXPENSIVE,FUSE “Name” ' ...expensive code using fuse...'
  11. 11. Sharness feature: result aggregation
  12. 12. Other Sharness features ● Easily extensible with your own test functions ● Many options: verbose output, stop immediately if a test fails, ... ● Many builtin functions: test_expect_failure, test_pause, test_must_fail,... ● Output in the “test anything” protocol, so compatibility with other tools
  13. 13. ● Now IPFS has around 24 Sharness test scripts ● That means around 300 tests ● Run automatically on each commit of each pull request ● On MacOS and Linux ● Using Travis CI and CircleCI IPFS Sharness Tests now
  14. 14. ● Add both unit tests and black box tests ● Avoid breaking backward compatibility even for pre-alpha software Avoid Regressions (1)
  15. 15. ● Regressions still happen sometimes as tests don't cover everything => git bisect ● Each commit must compile and pass tests => GIT_EDITOR=true git rebase -i --exec “make test” master Avoid Regressions (2)
  16. 16. Commits in Git form a DAG (Directed Acyclic Graph) ● history direction is from left to right ● new commits point to their parents
  17. 17. First Bad Commit B ● B introduces a bad behavior called "bug" or "regression" ● red commits are called "bad" ● blue commits are called "good"
  18. 18. Idea: ● help find a first bad commit ● use a binary search algorithm for efficiency if possible Benefits: ● manually verifying the source code changes from only one commit is relatively easy ● the commit gives extra information: commit message, author, ... Git bisect
  19. 19. 2 ways to do it: $ git bisect start $ git bisect bad [<BAD>] $ git bisect good [<GOOD>...] or $ git bisect start <BAD> <GOOD> [<GOOD>...] where <BAD> and <GOOD> can be resolved to commits Starting a bisection and bounding it
  20. 20. (toy example with the linux kernel) $ git bisect start v2.6.27 v2.6.25 Bisecting: 10928 revisions left to test after this (roughly 14 steps) [2ec65f8b89ea003c27ff7723525a2ee335a2b393] x86: clean up using max_low_pfn on 32-bit $ => the commit you should test has been checked out Starting example
  21. 21. 1.test the current commit 2.tell "git bisect" whether it is good or bad, for example: $ git bisect bad Bisecting: 5480 revisions left to test after this (roughly 13 steps) [66c0b394f08fd89236515c1c84485ea712a157be] KVM: kill file->f_count abuse in kvm repeat step 1. and 2. until the first bad commit is found... Driving a bisection manually
  22. 22. $ git bisect bad 2ddcca36c8bcfa251724fe342c8327451988be0d is the first bad commit commit 2ddcca36c8bcfa251724fe342c8327451988be0d Author: Linus Torvalds <torvalds@linux- foundation.org> Date: Sat May 3 11:59:44 2008 -0700 Linux 2.6.26-rc1 :100644 100644 5cf8258195331a4dbdddff08b8d68642638eea57 4492984efc09ab72ff6219a7bc21fb6a957c4cd5 M Makefile First bad commit found
  23. 23. When the first bad commit is found: ● you can check it out and tinker with it, or ● you can use "git bisect reset", like that: $ git bisect reset Checking out files: 100% (21549/21549), done. Previous HEAD position was 2ddcca3... Linux 2.6.26-rc1 Switched to branch 'master'   to go back to the branch you were in before you started bisecting End of bisection
  24. 24. At each bisection step a script or command will be launched to tell if the current commit is good or bad. Syntax: $ git bisect run COMMAND [ARG...] Example to bisect a broken build: $ git bisect run make Driving a bisection automatically
  25. 25. $ git bisect start v2.6.27 v2.6.25 Bisecting: 10928 revisions left to test after this (roughly 14 steps) [2ec65f8b89ea003c27ff7723525a2ee335a2b393] x86: clean up using max_low_pfn on 32-bit $ $ git bisect run grep '^SUBLEVEL = 25' Makefile running grep ^SUBLEVEL = 25 Makefile Bisecting: 5480 revisions left to test after this (roughly 13 steps) [66c0b394f08fd89236515c1c84485ea712a157be] KVM: kill file->f_count abuse in kvm running grep ^SUBLEVEL = 25 Makefile Automatic bisect example (1)
  26. 26. SUBLEVEL = 25 Bisecting: 2740 revisions left to test after this (roughly 12 steps) [671294719628f1671faefd4882764886f8ad08cb] V4L/DVB(7879): Adding cx18 Support for mxl5005s ... ... running grep ^SUBLEVEL = 25 Makefile Bisecting: 0 revisions left to test after this (roughly 0 steps) [2ddcca36c8bcfa251724fe342c8327451988be0d] Linux 2.6.26-rc1 running grep ^SUBLEVEL = 25 Makefile Automatic bisect example (2)
  27. 27. 2ddcca36c8bcfa251724fe342c8327451988be0d is the first bad commit commit 2ddcca36c8bcfa251724fe342c8327451988be0d Author: Linus Torvalds <torvalds@linux- foundation.org> Date: Sat May 3 11:59:44 2008 -0700 Linux 2.6.26-rc1 :100644 100644 5cf8258195331a4dbdddff08b8d68642638eea57 4492984efc09ab72ff6219a7bc21fb6a957c4cd5 M Makefile bisect run success Automatic bisect example (3)
  28. 28. 0 => good 1-124 and 126-127 => bad 125 => skip 128-255 => stop "skip": mark commit as "untestable", "git bisect" will choose another commit to be tested "stop": bisection is stopped immediately, useful to abort bisection in abnormal situations Run script and exit code
  29. 29. Problem when bisecting Sometimes the commit that introduced a bug will be in an untestable area of the graph. For example: X X1 X2 X3W Y Z Commit X introduced a breakage, later fixed by commit Y.
  30. 30. There are only 'skip'ped commits left to test. The first bad commit could be any of: 15722f2fa328eaba97022898a305ffc8172db6b1 78e86cf3e850bd755bb71831f42e200626fbd1e0 e15b73ad3db9b48d7d1ade32f8cd23a751fe0ace 070eab2303024706f2924822bfec8b9847e4ac1b We cannot bisect more! Possible end of bisection
  31. 31. Possible solutions Possible solutions to bisect anyway: ● apply a patch before testing and remove it afterwards (can be done using "git cherry-pick"), or ● create a fixed up branch (can be done with "git rebase -i"), for example: X X1 X2 X3W Y Z X + Y X1' X2' X3' Z' Z1
  32. 32. A good solution X X1 X2 X3W Y Z X + Y X1' X2' X3' Z' Z1 The idea is that we will replace Z with Z' so that we bisect from the beginning using the fixed up branch. $ git replace Z Z'
  33. 33. ● ./example.t ● vi t0040-add-and-cat.sh ● TEST_EXPENSIVE=1 ./t0040-add-and-cat.sh -v ● cd test/sharness && make ● git rebase -i --exec “make test” master ● git bisect run ./test_script.sh Demos
  34. 34. ● https://github.com/mlafeldt/sharness ● https://github.com/chriscool/sharnessify ● http://git-scm.com ● http://ipfs.io Links
  35. 35. Conclusion ● Black box tests with Sharness are quite easy and very useful ● Check every commit, not just every pull request ● Use git bisect, automate it if possible
  36. 36. Many thanks to: ● Junio Hamano, Linus Torvalds, ● Juan Benet, Jeromy Johnson, Mathias Lafeldt ● many other great people in the IPFS, Git and Linux communities ● LinuxCon organizers and attendants, ● SoftAtHome, the company I am working for.
  37. 37. Questions?

×