For the talk at shibuya.js on Test.js

1. How I stopped worrying
and
loved DumpRenderTree
2011/03/08
MORITA Hajime <omo@dodgson.org>
http://steps.dodgson.org/
Tuesday, March 8, 2011

2. How I stopped worrying
and
loved DumpRenderTree
a bit
2011/03/08
MORITA Hajime <omo@dodgson.org>
http://steps.dodgson.org/
Tuesday, March 8, 2011

3. • WebKit
Test Smell
• .js Test
• ( )
Tuesday, March 8, 2011

4. DumpRenderTree (DRT)
Demo
Tuesday, March 8, 2011

5. DumpRenderTree (DRT)
•
• HTML
( )
• “Golden File”
• OS
Tuesday, March 8, 2011

6. (-2002)
•
• DumpRenderTree, LayoutTest
• WebKit Mac
Tuesday, March 8, 2011

7. Mac
• JavaScript / DOM
• ex: ws://
• • OS/GUI
Win, Qt, GTK, EFL, Skia ...
(Ports)
ex: -webkit-transition:
• OS, • Mac
Tiger, Leopard, SL, Lion
ex: drag-n-drop, media, storage
• ex: <input type=”speech”>
• WYSIWYG
contentEditable
Tuesday, March 8, 2011

8. 3 Test Smells in WebKit
• Obscure Tests
• Fragile Tests
• High Test Maintenance Cost
Tuesday, March 8, 2011

9. (1/3): Obscure Tests
Dump?
• JavaScript
• HTML5-ish (File, XHR, WebSocket, etc.)
•
Tuesday, March 8, 2011

10. LayoutTestController.dumpAsText()
if (window.layoutTestController) {
layoutTestController.dumpAsText();
...
}
function log(msg)
{
document.getElementById('log').innerHTML += (msg + “n”);
}
...
sel = window.getSelection();
actualText = sel.getRangeAt(0).toString();
if (expectedText == actualText) {
log("Passed " + testId + " (with contentEditable)");
} else {
log("Failed " + testId + " (with contentEditable)");
log(" Expected: " + expectedText);
log(" Actual: " + actualText);
}
...
Tuesday, March 8, 2011

11. :
Passed totest_minimum
Passed totest_minimum (with contentEditable)
Passed totest_before_after
Passed totest_before_after (with contentEditable)
Passed totest_linebreak_out
Passed totest_linebreak_out (with contentEditable)
Passed totest_tab_out
Passed totest_tab_out (with contentEditable)
Passed totest_multiple_whitespaces_out
Passed totest_multiple_whitespaces_out (with contentEditable)
Passed totest_linebreak_in
Passed totest_linebreak_in (with contentEditable)
•
• Port
Tuesday, March 8, 2011

12. (1/3): Obscure Tests
• LayoutTestController.dumpAsText()
• js-test-pre.js/js-test-post.js
assertion
• dump-as-markup.js w/ dumpAsText())
DOM (built
contentEditable
Tuesday, March 8, 2011

13. (2/3): Fragile Tests
Golden File : Over specification
•
• OS
• OS
•
• GPU
•
• DRT
Tuesday, March 8, 2011

14. Tree Redness
http://build.webkit.org/waterfall
• ( )
•
Tuesday, March 8, 2011

15. “Skipped” list
Port
• (= )
•
• ( )
• Bug = TODO list
•
• Port Golden File ...
Tuesday, March 8, 2011

16. Rebaselining Tool (Chromium only)
script
Download Update
Commit (by dev)
... ...
expected.txt expected.txt
result.zip ... ...
Buildbot Local PC SVN
Server
Tuesday, March 8, 2011

17. (3/3): High Test Maintenance Cost
• Golden Files
•
•
• Port
• ...
Tuesday, March 8, 2011

18. Gardener (for Chromium)
• Rebaseline, Skip, Revert, ..
•
Tuesday, March 8, 2011

19. Reftests (work in progress)
• HTML
• Port
• Mozilla
https://developer.mozilla.org/en/Creating_reftest-based_unit_tests
• W3C
Tuesday, March 8, 2011

20. • WebKit
Test Smell
Tuesday, March 8, 2011

21. Backup Slides
Tuesday, March 8, 2011

22. 7 Test Smells in WebKit
• Obscure Tests
• Fragile Tests
• Erratic Tests
• Manual Intervention
• Slow Tests
• High Test Maintenance Cost
• Developers Not Writing (Unit) Tests
Tuesday, March 8, 2011

23. (x/x): Erratic Tests aka. Flaky Tests
•
• OS ( )
•
•
• ... ( )
Tuesday, March 8, 2011

24. (Chromium only)
unexpected flaky 1
http://build.chromium.org/p/chromium.webkit/waterfall
Tuesday, March 8, 2011

25. (x/x): Manual Intervention
JS
•
• DOM
• ex:
•
Tuesday, March 8, 2011

26. DRT API
// from editing/input/emacs-ctrl-o.html
...
if (window.eventSender)
eventSender.keyDown("o", ["ctrlKey"]);
...
// from editing/spelling/.../spelling-attribute-at-child.js
...
var marked = layoutTestController.hasSpellingMarker(1, 2);
...
// from fast/speech/input-onspeechchange-event.html
...
if (window.layoutTestController && window.eventSender) {
layoutTestController.addMockSpeechInputResult(....);
...
Tuesday, March 8, 2011

27. ...
Tuesday, March 8, 2011

28. (x/x): Slow Tests
• 2
• HTML/JS
Tuesday, March 8, 2011

29. ...
• GUI
: “Active Window”, “Clipboard”, ...
• (ex. localStorage)
• ( )
• Chromium WebKit
• Window-less by design
• ...
Tuesday, March 8, 2011

30. $ ./Tools/Scripts/run-webkit-tests --help
Usage: old-run-webkit-tests [options] [testdir|testpath ...]
....
--slowest Report the 10 slowest tests
....
Tuesday, March 8, 2011