My talk delivered at Wicked Good Ember Conference 2016 discussing strategies and techniques to help maintain your tests when working with a large test suite.

1. Maintaining Your Tests
At Scale

2. @trentmwillis

4. ln -s linkedin microsoft

6. Things that have happened
in recently

7. 📢🎵

8. 😡🎵

9. 😡🎵 for 😃

10. What is Scale?

11. 4000+ Tests

12. 1000+ Acceptance Tests

13. Dozens of commits
per day

14. Why is maintaining
tests at Scale hard?

15. Why is maintaining
tests at Scale difficult?
And how do we make
it not difficult?

16. Flakiness
(a.k.a. non-atomic, inter-dependent tests)

17. Common Causes

18. Common Causes
Not cleaning up async handlers
Previous tests modifying shared state
Environmental differences (looking at you
Phantom)

19. moduleForAcceptance(‘Some module’);
// passes fine
moduleForComponent(‘modified-by-initializer’);
// passes fine since acceptance test ran
// initializers

20. // moduleForAcceptance(‘Some module’);
// deleted due to feature removal
moduleForComponent(‘modified-by-initializer’);
// FAILS! No acceptance test to
// run initializers

21. Flakiness
(a.k.a. non-atomic, inter-dependent tests)

22. Randomization!
(chaos is good)

23. “[Reordering] can also lead to random
errors, in case your testsuite has non-
atomic tests, where the order is important.
You should fix those issues, instead of
disabling reordering!”
- QUnit documentation on config.reorder

24. QUnit.config.seed = ‘any string’;
QUnit.config.seed = true;

25. /tests?seed=1337
/tests?seed

26. ember install ember-exam
ember exam --random=<seed>

27. Slowness
(a.k.a. an excuse to slack off)

29. var numTests = 1000;
var timePerTest = 2;
var timePerMinute = 60;
var testTime= numTests * timePerTest;
console.log(testTime / timePerMinute);

30. 30 minutes!

31. Slowness
(a.k.a. an excuse to slack off)

32. Parallelization!
(of all the things)

33. Testem has supported parallelization for a
long time

34. {
“parallel”: “4”
}

35. Testem has supported parallelization for a
long time

36. Testem has supported parallelization for a
long time but it had limitations

37. {
“test_page”: [
“/tests/index.html?filter=acceptance”,
“/tests/index.html?filter=integration”,
“/tests/index.html?filter=unit”
]
}

38. ember exam --split=<num> --parallel

39. ok 1 PhantomJS 1.9 - Exam Partition #1 - some test
ok 2 PhantomJS 1.9 - Exam Partition #3 - some other other test
ok 3 PhantomJS 1.9 - Exam Partition #2 - some other test

40. {
“parallel”: “-1”
}

41. ember build
ember exam
--path dist
--split 20
--parallel
--random

42. Tracking Features
(a.k.a. did you add tests?)

43. Things We Should Stop Saying

44. Things We Should Stop Saying
“Tests will be in the next PR”
“I tested it manually”
“This is a blocker,
so tests will have to come later”

45. TDD
Write tests before implementing

46. import { skip } from ‘qunit’;
skip(‘Upcoming feature’);

47. Tracking Features
(a.k.a. did you add tests?)

48. Todo Lists
(ya, old school)

49. Feature: Awesome Feature
TestCase: Should Do Awesome Thing
TestCase: Should Do Less Awesome Thing

50. QUnit.todo();

51. Disclaimer: Not Ready Yet
(we have a todo for todo)

52. Disclaimer: Not Ready Yet
(we have a todo for todo)
PR: jquery/qunit#1006

53. import { module, test, todo } from ‘qunit’;
module(‘Awesome Feature’);
test(‘awesome thing’, function(assert) {
assert.ok(true, “I’m ready!”);
});
todo(‘less awesome thing’, function(assert) {
assert.ok(false, “I’m NOT ready yet!”);
});

54. testem/custom-reporter.js

55. testem/custom-reporter.js

56. Finding Things
(a.k.a, “I think I scrolled past it…”)

58. Cmd + Shift + F

59. Cmd + Shift + F + Hope For The Best

60. Finding Things
(a.k.a, “I think I scrolled past it…”)

61. Organization!
(conventions 😉)

62. <test-type> | <type> | <path>

63. Integration | Component | awesome-sauce

64. integration/component/awesome-sauce

66. Flakiness
Slowness
Tracking Features
Finding Things

67. Randomization
Parallelization
Todo Lists
Organization

68. One Final Thing
(we need you!)

69. Accessibility

70. Accessibility
ember-a11y-testing

71. Questions?