Testing, Performance Analysis, and jQuery 1.4

  • 10,046 views
Uploaded on

This is the talk that I gave at JSConf.eu 2009, then modified slightly and given again at the December Bayjax meetup (the parts on jQuery and HTML 5 in IE were added).

This is the talk that I gave at JSConf.eu 2009, then modified slightly and given again at the December Bayjax meetup (the parts on jQuery and HTML 5 in IE were added).

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Very nice to see statistical methods applied to performance analysis in web development context!
    Are you sure you want to
    Your message goes here
  • Full video and transcript of this talk: http://developer.yahoo.com/yui/theater/video.php?v=resig-testing
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
10,046
On Slideshare
0
From Embeds
0
Number of Embeds
5

Actions

Shares
Downloads
438
Comments
2
Likes
27

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Testing, Performance Analysis, and jQuery 1.4 John Resig http://ejohn.org/ - http://twitter.com/jeresig
  • 2. Why Test JavaScript? ✦ Cross-browser issues. ✦ The possibility for causing an unforeseen problem is simply too great.
  • 3. JavaScript Testing Isn’t The Same As Desktop or Server-Side Testing
  • 4. What should I use?
  • 5. Looong Tail 300 225 150 75 0 Other
  • 6. Basic Components ✦ Writing and understanding a JavaScript test suite is easy. ✦ Test Suite ✦ Tests ✦ Assertions ✦ Async Tests ✦ Test Runner
  • 7. Assertions  <html>   <head>     <title>Test Suite</title>     <script>     function assert( value, desc ) {       var li = document.createElement("li");       li.className = value ? "pass" : "fail";       li.appendChild( document.createTextNode( desc ) );       document.getElementById("results").appendChild( li );     }          window.onload = function(){       assert( true, "The test suite is running." );     };     </script>     <style>       #results li.pass { color: green; }       #results li.fail { color: red; }     </style>   </head>   <body>     <ul id="results"></ul>   </body>   </html> 
  • 8. Tests      test("A test.", function(){         assert( true, "First assertion completed" );         assert( true, "Second assertion completed" );         assert( true, "Third assertion completed" );       });              test("Another test.", function(){         assert( true, "First test completed" );         assert( false, "Second test failed" );         assert( true, "Third assertion completed" );       }); 
  • 9. Tests    var results;          function assert( value, desc ) {       var li = document.createElement("li");       li.className = value ? "pass" : "fail";       li.appendChild( document.createTextNode( desc ) );       results.appendChild( li );       if ( !value ) {         li.parentNode.parentNode.className = "fail";       }       return li;     }          function test(name, fn){       results = document.getElementById("results");       results = assert( true, name ).appendChild(         document.createElement("ul") );       fn();     } 
  • 10. Async Tests  test("Async Test #1", function(){     pause();     setTimeout(function(){       assert( true, "First test completed" );       resume();     }, 1000);   });      test("Async Test #2", function(){     pause();     setTimeout(function(){       assert( true, "Second test completed" );       resume();     }, 1000);   }); 
  • 11. Async Tests  (function(){     var queue = [], paused = false;          this.test = function(fn){       queue.push( fn );       runTest();     };          this.pause = function(){       paused = true;     };          this.resume = function(){       paused = false;       setTimeout(runTest, 1);     };          function runTest(){       if ( !paused && queue.length ) {         queue.shift()();         if ( !paused ) {           resume();         }       }     }   })(); 
  • 12. People don’t test. :-( 900 675 450 225 0 None
  • 13. Popular Test Frameworks 300 225 150 75 0 QUnit JSUnit Selenium YUITest FireUnit Screw.Unit JSSpec
  • 14. Unit Testing ✦ Break code into logical chucks for testing. ✦ Focus on one method at a time. ✦ Good for testing APIs. ✦ Popular Frameworks: ✦ QUnit ✦ JSUnit ✦ YUITest ✦ FireUnit
  • 15. JSUnit ✦ One of the oldest JavaScript testing frameworks. ✦ A port of JUnit to JavaScript, circa 2001. ✦ Code feels very 2001 (frames!) ✦ http://www.jsunit.net/
  • 16. JSUnit        function coreSuite() {             var result = new top.jsUnitTestSuite();             result.addTestPage("tests/jsUnitAssertionTests.html");             result.addTestPage("tests/jsUnitSetUpTearDownTests.html");             result.addTestPage("tests/jsUnitRestoredHTMLDivTests.html");             result.addTestPage("tests/jsUnitFrameworkUtilityTests.html");             result.addTestPage("tests/jsUnitOnLoadTests.html");             result.addTestPage("tests/jsUnitUtilityTests.html");             return result;         }                  function serverSuite() {             var result = new top.jsUnitTestSuite();             result.addTestPage("tests/server/jsUnitVersionCheckTests.html");             result.addTestPage("tests/server/jsUnitServerAjaxTests.html");             return result;         }                  function librariesSuite() {             var result = new top.jsUnitTestSuite();             result.addTestPage("tests/jsUnitMockTimeoutTest.html");             return result;         }                  function suite() {             var newsuite = new top.jsUnitTestSuite();             newsuite.addTestSuite(coreSuite());             newsuite.addTestSuite(serverSuite());             newsuite.addTestSuite(librariesSuite());             return newsuite;         }
  • 17. JSUnit  function testAssertNotUndefined() {       assertNotUndefined("1 should not be undefined", 1);       assertNotUndefined(1);   }      function testAssertNaN() {       assertNaN("a string should not be a number", "string");       assertNaN("string");   }      function testAssertNotNaN() {       assertNotNaN("1 should not be not a number", 1);       assertNotNaN(1);   }      function testFail() {       var excep = null;       try {           fail("Failure message");       } catch (e) {           excep = e;       }       assertJsUnitException("fail(string) should throw a JsUnitException", excep);   }      function testTooFewArguments() {       var excep = null;       try {           assert();       } catch (e1) {           excep = e1;       }       assertNonJsUnitException("Calling an assertion function with too  few arguments should throw an exception", excep);   }
  • 18. JSUnit
  • 19. YUITest (2 & 3) ✦ Testing framework built and developed by Yahoo (released Oct 2008). ✦ Completely overhauled to go with YUI v3. ✦ Features: ✦ Supports async tests. ✦ Has good event simulation. ✦ v2: http://developer.yahoo.com/yui/ examples/yuitest/ ✦ v3: http://developer.yahoo.com/yui/3/test/
  • 20. YUITest 2   YAHOO.example.yuitest.ArrayTestCase = new YAHOO.tool.TestCase({       name : "Array Tests",              setUp : function () {           this.data = [0,1,2,3,4]       },              tearDown : function () {           delete this.data;       },                testPop : function () {           var Assert = YAHOO.util.Assert;                    var value = this.data.pop();                      Assert.areEqual(4, this.data.length);           Assert.areEqual(4, value);                   },                      testPush : function () {           var Assert = YAHOO.util.Assert;                      this.data.push(5);                      Assert.areEqual(6, this.data.length);           Assert.areEqual(5, this.data[5]);                   }  }); 
  • 21. YUITest 2
  • 22. YUITest 3  Y.example.test.DataTestCase = new Y.Test.Case({       name : "Data Tests",              setUp : function () {           this.data = {               name: "test",               year: 2007,               beta: true           };       },              tearDown : function () {           delete this.data;       },              testName : function () {           var Assert = Y.Assert;                      Assert.isObject(this.data);           Assert.isString(this.data.name);           Assert.areEqual("test", this.data.name);                   },              testYear : function () {           var Assert = Y.Assert;                      Assert.isObject(this.data);           Assert.isNumber(this.data.year);           Assert.areEqual(2007, this.data.year);                   }  }); 
  • 23. YUITest 3
  • 24. QUnit ✦ Unit Testing framework built for jQuery. ✦ Features: ✦ Supports asynchronous testing. ✦ Can break code into modules. ✦ Supports test timeouts. ✦ No dependencies. ✦ Painfully simple. ✦ http://docs.jquery.com/QUnit
  • 25. QUnit Style  test("a basic test example", function() {     ok( true, "this test is fine" );     var value = "hello";     equals( "hello", value, "We expect value to be hello" );   });      module("Module A");      test("first test within module", function() {     ok( true, "all pass" );   });      test("second test within module", function() {     ok( true, "all pass" );   });      module("Module B");      test("some other test", function() {     expect(2);     equals( true, false, "failing test" );     equals( true, true, "passing test" );   }); 
  • 26. QUnit
  • 27. FireUnit ✦ Unit testing extension for Firebug ✦ fireunit.ok( true, “...” ); ✦ http://fireunit.org/
  • 28. Standardization ✦ CommonJS: A unified cross-platform API for JavaScript. ✦ (Including the server-side!) ✦ Working to standardize a simple testing API. ✦ http://wiki.commonjs.org/wiki/CommonJS
  • 29. Server-Side ✦ Ignore the browser! Simulate it on the server-side. ✦ Almost always uses Java + Rhino to construct a browser. ✦ Some frameworks: ✦ Crosscheck ✦ Env.js ✦ Blueridge
  • 30. Server-Side ✦ Crosscheck ✦ Pure Java, even simulates browser bugs. ✦ http://www.thefrontside.net/crosscheck ✦ Env.js ✦ Pure JavaScript, focuses on standards support. ✦ http://github.com/thatcher/env-js/tree/ master ✦ Blueridge ✦ Env.js + Screw.Unit + Rhino ✦ http://github.com/relevance/blue-ridge/
  • 31. Env.js $ java -jar build/js.jar Rhino 1.6 release 6 2007 06 28 js> load('build/runtest/env.js'); js> window.location = 'test/index.html'; test/index.html js> load('dist/jquery.js'); // Add pretty printing to jQuery objects: js> jQuery.fn.toString = DOMNodeList.prototype.toString; js> $('span').remove(); [ <span#å°åŒ—Taibei>, <span#å°åŒ—>, <span#utf8class1>, <span#utf8class2>, <span#foo:bar>, <span#test.foo[5]bar> ] // Yes - UTF-8 is supported in DOM documents! js> $('span') [ ] js> $('div').append('<span><b>hello!</b> world</span>'); [ <div#main>, <div#foo> ] js> $('span') [ <span>, <span> ] js> $('span').text() hello! worldhello! world
  • 32. Browser Launching ✦ Automates the process of opening browser windows, running tests, and getting results. ✦ Frequently require a specific framework. ✦ Popular frameworks: ✦ WebDriver http://code.google.com/p/ webdriver/ (Java) ✦ Waitr http://wtr.rubyforge.org/ (Ruby) ✦ JsTestDriver http://code.google.com/p/ js-test-driver/ (Java) ✦ Selenium RC http://seleniumhq.org/ projects/remote-control/ (Java)
  • 33. Browser Launching
  • 34. Distributed ✦ Selenium Grid ✦ Push Selenium tests out to many machines (that you manage), simultaneously. ✦ Collect and store the results. ✦ http://selenium-grid.seleniumhq.org/ ✦ TestSwarm ✦ Push tests to a distributed swarm of clients. ✦ Results viewable on the server. ✦ http://testswarm.com/
  • 35. The Scaling Problem ✦ The Problem: ✦ jQuery has 6 test suites ✦ Run in 15 browsers ✦ (Not even including multiple platforms or mobile browsers!) ✦ All need to be run for every commit, patch, and plugin. ✦ JavaScript testing doesn’t scale well.
  • 36. Distributed Testing ✦ Hub server ✦ Clients connect and help run tests ✦ A simple JavaScript client that can be run in all browsers ✦ Including mobile browsers! ✦ TestSwarm
  • 37. FF 3.5 FF 3.5 FF 3.5 IE 6 IE 6 FF 3 IE 6 Op 9 FF 3 IE 7 TestSwarm IE 7 Test Suite Test Suite Test Suite
  • 38. TestSwarm.com ✦ Incentives for top testers (t-shirts, books) ✦ Will be opening for alpha testing very soon ✦ Help your favorite JavaScript library become better tested! ✦ http://testswarm.com
  • 39. Accurately Measuring JavaScript
  • 40. Major Cases ✦ Same Code, Different Platforms ✦ Compare V8 vs. SpiderMonkey vs. JavaScriptCore ✦ Different Code, Same Platform ✦ Compare CSS Selector Engines ✦ A/B testing a piece of code
  • 41. Same Code, Different Platform ✦ A number of suites analyzing JS perf: ✦ SunSpider (from WebKit) ✦ V8 Benchmark (from V8/Chrome) ✦ Dromaeo (from Mozilla) ✦ Statistical accuracy and reproducibility is paramount.
  • 42. SunSpider ✦ All tests were highly balanced. ✦ Provide some level of statistical accuracy. ✦ +/- 5ms (for example) ✦ Tests are run by loading an iframe with the test 5 times. ✦ getTime() is run before/after each test. ✦ Entire suite must be trashed in order to upgrade/fix a test.
  • 43. Error Rate? ✦ How do we get it? What does it mean? ✦ It’s how confident we are that we arrived at the result we want in the number of runs that we’ve done.
  • 44. Normal Distribution ✦ First: Assume that the results are coming back in a normal distribution. ✦ The “bell curve”
  • 45. Confidence ✦ Next: We need a confidence level. ✦ T-Distribution works well here. http://en.wikipedia.org/wiki/Student%27s_t-distribution
  • 46. Error Rate ✦ 5 runs ✦ (each run is potentially 1000s of individual test runs) ✦ 95% Confidence (t-distribution = 2.776) ✦ Standard Errors Mean = ✦ (std_dev / sqrt(runs)) * 2.776 ✦ Error = (err_mean / mean) * 100 ✦ This way you can get results like: ✦ 123ms +/- 5ms
  • 47. V8 Benchmark ✦ Tests are run, potentially, 1000s of times. ✦ Also provides an error rate. ✦ (Use a geometric mean to arrive at a result.)
  • 48. Small Time Accuracy ✦ Small time: ✦ 1ms, 1ms, 1ms, 1ms, 3ms ✦ huge error! ✦ Large time: ✦ 1234ms, 1234ms, 1234ms, 1234ms, 1238ms ✦ tiny error! ✦ Tests that run faster need to be run more times. ✦ Running more times = less potential for weird results. http://ejohn.org/blog/javascript-benchmark-quality/
  • 49. Runs/Second ✦ var start = (new Date).getTime(); while (time < 1000) { runTest(); time = (new Date).getTime() - start; } ✦ More test runs, more statistical accuracy. ✦ V8 & Dromaeo-style suites handle this. ✦ (Problem: getTime() is being run on every loop - it should be run less frequently in order to influence the numbers less.)
  • 50. Runs/Second ✦ You are now measuring tests/second rather than seconds per test. ✦ You run tests as many times in one second as you can. ✦ Then you do that multiple times (5?) ✦ THEN you analyze the final numbers: ✦ 1234run/s, 1230runs/s, 1240runs/s, ...
  • 51. Harmonic Mean ✦ A way to average rates ✦ Which is what we have! runs/second ✦ For example: ✦ 1234run/s, 1230runs/s, 1240runs/s, 1236runs/ms, 1232runs/s ✦ 5 / ( (1/1234) + (1/1230) + (1/1240) + (1/1236) + (1/1232) ) = ✦ 1234.39runs/s! http://en.wikipedia.org/wiki/Harmonic_mean
  • 52. Dromaeo ✦ All individual tests are versioned ✦ Makes it easy to update or fix a bug in a test ✦ Can only run tests of specific versions against each other ✦ Uses V8’s style of running tests. ✦ Also has DOM and framework tests. ✦ ...and hooks for doing Shark profiling.
  • 53. Bug Fixes ✦ Tests will, inevitably, have bugs that need to be fixed. ✦ Fixing a bug changes the result quality. ✦ Tests need to be versioned so that changes can be made. ✦ You look at Test v1 vs. Test v1 results. ✦ Not Test v2 vs. Test v1. ✦ Tip: Just use the last revision control commit # for the test file.
  • 54. Different Code, Same Platform ✦ Most solutions here are very poor. ✦ Run the test very few times, use getTime(). ✦ Highly inaccurate results, massive error.
  • 55. Garbage Collection ✦ Browsers periodically run garbage collectors to clean up old objects no longer referenced. ✦ This can take a long time and spike your test results. ✦ Example: ✦ 10ms, 13ms, 11ms, 12ms, 486ms, 12ms, ... ✦ When comparing engine to engine, this doesn’t matter. ✦ Comparing code vs. code, it does.
  • 56. Mean, Median, Mode? ✦ Mode! ✦ Run your tests a large number of times. ✦ What is the ‘mode’ (the result that occurs most frequently) ✦ Example: ✦ 10, 11, 11, 12, 12, 12, 13, 14 ✦ Mode = 12ms. ✦ Less accurate than mean, but gives you a more-consistent result. ✦ DON’T DISCARD “BAD” RESULTS!
  • 57. getTime() Accuracy http://ejohn.org/blog/accuracy-of-javascript-time/
  • 58. 15ms intervals ONLY! Error Rate of 50-750%!
  • 59. IE in Wine ✦ Running Internet Explorer in Wine (on Linux) gives fine-grained timer results ✦ Down to the millisecond! ✦ You can also run IE, in Wine, on OS X: ✦ ies4osx ✦ Huge Caveat: It gives you fine-grained time, but that doesn’t mean it’s accurate.
  • 60. Different Code, Same Platform ✦ How can we get good numbers? ✦ We have to go straight to the source: Use the tools the browsers provide.
  • 61. Firebug Profiler
  • 62. Safari 4 Profiler
  • 63. IE 8 Profiler
  • 64. IE 6-8: DynaTrace Ajax
  • 65. IE 6-8: DynaTrace Ajax
  • 66. Shark Profiling ✦ Extremely low-level ✦ Watch for all internal function calls ✦ See what gets called the most. ✦ Dromaeo includes shark profiling hooks.
  • 67. Interesting Problems Tackled in jQuery 1.4
  • 68. Tackled in 1.4 ✦ Complexity reduction ✦ Bubbling change, submit, focus, blur ✦ Required script loading
  • 69. Questions ✦ Contact: ✦ John Resig ✦ http://ejohn.org/ ✦ http://twitter.com/jeresig