Advertisement
Testing, Performance Analysis, and jQuery 1.4
Dec. 15, 2009•0 likes•12,105 views
28 likes
Be the first to like this
Show More
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Download to read offline
Report
Technology
This is the talk that I gave at JSConf.eu 2009, then modified slightly and given again at the December Bayjax meetup (the parts on jQuery and HTML 5 in IE were added).
jeresigFollow
Advertisement
Testing, Performance Analysis, and jQuery 1.4
- Testing, Performance Analysis, and jQuery 1.4 John Resig http://ejohn.org/ - http://twitter.com/jeresig
- Why Test JavaScript? ✦ Cross-browser issues. ✦ The possibility for causing an unforeseen problem is simply too great.
- JavaScript Testing Isn’t The Same As Desktop or Server-Side Testing
- What should I use?
- Looong Tail 300 225 150 75 0 Other
- Basic Components ✦ Writing and understanding a JavaScript test suite is easy. ✦ Test Suite ✦ Tests ✦ Assertions ✦ Async Tests ✦ Test Runner
- Assertions <html> <head> <title>Test Suite</title> <script> function assert( value, desc ) { var li = document.createElement("li"); li.className = value ? "pass" : "fail"; li.appendChild( document.createTextNode( desc ) ); document.getElementById("results").appendChild( li ); } window.onload = function(){ assert( true, "The test suite is running." ); }; </script> <style> #results li.pass { color: green; } #results li.fail { color: red; } </style> </head> <body> <ul id="results"></ul> </body> </html>
- Tests test("A test.", function(){ assert( true, "First assertion completed" ); assert( true, "Second assertion completed" ); assert( true, "Third assertion completed" ); }); test("Another test.", function(){ assert( true, "First test completed" ); assert( false, "Second test failed" ); assert( true, "Third assertion completed" ); });
- Tests var results; function assert( value, desc ) { var li = document.createElement("li"); li.className = value ? "pass" : "fail"; li.appendChild( document.createTextNode( desc ) ); results.appendChild( li ); if ( !value ) { li.parentNode.parentNode.className = "fail"; } return li; } function test(name, fn){ results = document.getElementById("results"); results = assert( true, name ).appendChild( document.createElement("ul") ); fn(); }
- Async Tests test("Async Test #1", function(){ pause(); setTimeout(function(){ assert( true, "First test completed" ); resume(); }, 1000); }); test("Async Test #2", function(){ pause(); setTimeout(function(){ assert( true, "Second test completed" ); resume(); }, 1000); });
- Async Tests (function(){ var queue = [], paused = false; this.test = function(fn){ queue.push( fn ); runTest(); }; this.pause = function(){ paused = true; }; this.resume = function(){ paused = false; setTimeout(runTest, 1); }; function runTest(){ if ( !paused && queue.length ) { queue.shift()(); if ( !paused ) { resume(); } } } })();
- People don’t test. :-( 900 675 450 225 0 None
- Popular Test Frameworks 300 225 150 75 0 QUnit JSUnit Selenium YUITest FireUnit Screw.Unit JSSpec
- Unit Testing ✦ Break code into logical chucks for testing. ✦ Focus on one method at a time. ✦ Good for testing APIs. ✦ Popular Frameworks: ✦ QUnit ✦ JSUnit ✦ YUITest ✦ FireUnit
- JSUnit ✦ One of the oldest JavaScript testing frameworks. ✦ A port of JUnit to JavaScript, circa 2001. ✦ Code feels very 2001 (frames!) ✦ http://www.jsunit.net/
- JSUnit function coreSuite() { var result = new top.jsUnitTestSuite(); result.addTestPage("tests/jsUnitAssertionTests.html"); result.addTestPage("tests/jsUnitSetUpTearDownTests.html"); result.addTestPage("tests/jsUnitRestoredHTMLDivTests.html"); result.addTestPage("tests/jsUnitFrameworkUtilityTests.html"); result.addTestPage("tests/jsUnitOnLoadTests.html"); result.addTestPage("tests/jsUnitUtilityTests.html"); return result; } function serverSuite() { var result = new top.jsUnitTestSuite(); result.addTestPage("tests/server/jsUnitVersionCheckTests.html"); result.addTestPage("tests/server/jsUnitServerAjaxTests.html"); return result; } function librariesSuite() { var result = new top.jsUnitTestSuite(); result.addTestPage("tests/jsUnitMockTimeoutTest.html"); return result; } function suite() { var newsuite = new top.jsUnitTestSuite(); newsuite.addTestSuite(coreSuite()); newsuite.addTestSuite(serverSuite()); newsuite.addTestSuite(librariesSuite()); return newsuite; }
- JSUnit function testAssertNotUndefined() { assertNotUndefined("1 should not be undefined", 1); assertNotUndefined(1); } function testAssertNaN() { assertNaN("a string should not be a number", "string"); assertNaN("string"); } function testAssertNotNaN() { assertNotNaN("1 should not be not a number", 1); assertNotNaN(1); } function testFail() { var excep = null; try { fail("Failure message"); } catch (e) { excep = e; } assertJsUnitException("fail(string) should throw a JsUnitException", excep); } function testTooFewArguments() { var excep = null; try { assert(); } catch (e1) { excep = e1; } assertNonJsUnitException("Calling an assertion function with too few arguments should throw an exception", excep); }
- JSUnit
- YUITest (2 & 3) ✦ Testing framework built and developed by Yahoo (released Oct 2008). ✦ Completely overhauled to go with YUI v3. ✦ Features: ✦ Supports async tests. ✦ Has good event simulation. ✦ v2: http://developer.yahoo.com/yui/ examples/yuitest/ ✦ v3: http://developer.yahoo.com/yui/3/test/
- YUITest 2 YAHOO.example.yuitest.ArrayTestCase = new YAHOO.tool.TestCase({ name : "Array Tests", setUp : function () { this.data = [0,1,2,3,4] }, tearDown : function () { delete this.data; }, testPop : function () { var Assert = YAHOO.util.Assert; var value = this.data.pop(); Assert.areEqual(4, this.data.length); Assert.areEqual(4, value); }, testPush : function () { var Assert = YAHOO.util.Assert; this.data.push(5); Assert.areEqual(6, this.data.length); Assert.areEqual(5, this.data[5]); } });
- YUITest 2
- YUITest 3 Y.example.test.DataTestCase = new Y.Test.Case({ name : "Data Tests", setUp : function () { this.data = { name: "test", year: 2007, beta: true }; }, tearDown : function () { delete this.data; }, testName : function () { var Assert = Y.Assert; Assert.isObject(this.data); Assert.isString(this.data.name); Assert.areEqual("test", this.data.name); }, testYear : function () { var Assert = Y.Assert; Assert.isObject(this.data); Assert.isNumber(this.data.year); Assert.areEqual(2007, this.data.year); } });
- YUITest 3
- QUnit ✦ Unit Testing framework built for jQuery. ✦ Features: ✦ Supports asynchronous testing. ✦ Can break code into modules. ✦ Supports test timeouts. ✦ No dependencies. ✦ Painfully simple. ✦ http://docs.jquery.com/QUnit
- QUnit Style test("a basic test example", function() { ok( true, "this test is fine" ); var value = "hello"; equals( "hello", value, "We expect value to be hello" ); }); module("Module A"); test("first test within module", function() { ok( true, "all pass" ); }); test("second test within module", function() { ok( true, "all pass" ); }); module("Module B"); test("some other test", function() { expect(2); equals( true, false, "failing test" ); equals( true, true, "passing test" ); });
- QUnit
- FireUnit ✦ Unit testing extension for Firebug ✦ fireunit.ok( true, “...” ); ✦ http://fireunit.org/
- Standardization ✦ CommonJS: A unified cross-platform API for JavaScript. ✦ (Including the server-side!) ✦ Working to standardize a simple testing API. ✦ http://wiki.commonjs.org/wiki/CommonJS
- Server-Side ✦ Ignore the browser! Simulate it on the server-side. ✦ Almost always uses Java + Rhino to construct a browser. ✦ Some frameworks: ✦ Crosscheck ✦ Env.js ✦ Blueridge
- Server-Side ✦ Crosscheck ✦ Pure Java, even simulates browser bugs. ✦ http://www.thefrontside.net/crosscheck ✦ Env.js ✦ Pure JavaScript, focuses on standards support. ✦ http://github.com/thatcher/env-js/tree/ master ✦ Blueridge ✦ Env.js + Screw.Unit + Rhino ✦ http://github.com/relevance/blue-ridge/
- Env.js $ java -jar build/js.jar Rhino 1.6 release 6 2007 06 28 js> load('build/runtest/env.js'); js> window.location = 'test/index.html'; test/index.html js> load('dist/jquery.js'); // Add pretty printing to jQuery objects: js> jQuery.fn.toString = DOMNodeList.prototype.toString; js> $('span').remove(); [ <span#å°åŒ—Taibei>, <span#å°åŒ—>, <span#utf8class1>, <span#utf8class2>, <span#foo:bar>, <span#test.foo[5]bar> ] // Yes - UTF-8 is supported in DOM documents! js> $('span') [ ] js> $('div').append('<span><b>hello!</b> world</span>'); [ <div#main>, <div#foo> ] js> $('span') [ <span>, <span> ] js> $('span').text() hello! worldhello! world
- Browser Launching ✦ Automates the process of opening browser windows, running tests, and getting results. ✦ Frequently require a specific framework. ✦ Popular frameworks: ✦ WebDriver http://code.google.com/p/ webdriver/ (Java) ✦ Waitr http://wtr.rubyforge.org/ (Ruby) ✦ JsTestDriver http://code.google.com/p/ js-test-driver/ (Java) ✦ Selenium RC http://seleniumhq.org/ projects/remote-control/ (Java)
- Browser Launching
- Distributed ✦ Selenium Grid ✦ Push Selenium tests out to many machines (that you manage), simultaneously. ✦ Collect and store the results. ✦ http://selenium-grid.seleniumhq.org/ ✦ TestSwarm ✦ Push tests to a distributed swarm of clients. ✦ Results viewable on the server. ✦ http://testswarm.com/
- The Scaling Problem ✦ The Problem: ✦ jQuery has 6 test suites ✦ Run in 15 browsers ✦ (Not even including multiple platforms or mobile browsers!) ✦ All need to be run for every commit, patch, and plugin. ✦ JavaScript testing doesn’t scale well.
- Distributed Testing ✦ Hub server ✦ Clients connect and help run tests ✦ A simple JavaScript client that can be run in all browsers ✦ Including mobile browsers! ✦ TestSwarm
- FF 3.5 FF 3.5 FF 3.5 IE 6 IE 6 FF 3 IE 6 Op 9 FF 3 IE 7 TestSwarm IE 7 Test Suite Test Suite Test Suite
- TestSwarm.com ✦ Incentives for top testers (t-shirts, books) ✦ Will be opening for alpha testing very soon ✦ Help your favorite JavaScript library become better tested! ✦ http://testswarm.com
- Accurately Measuring JavaScript
- Major Cases ✦ Same Code, Different Platforms ✦ Compare V8 vs. SpiderMonkey vs. JavaScriptCore ✦ Different Code, Same Platform ✦ Compare CSS Selector Engines ✦ A/B testing a piece of code
- Same Code, Different Platform ✦ A number of suites analyzing JS perf: ✦ SunSpider (from WebKit) ✦ V8 Benchmark (from V8/Chrome) ✦ Dromaeo (from Mozilla) ✦ Statistical accuracy and reproducibility is paramount.
- SunSpider ✦ All tests were highly balanced. ✦ Provide some level of statistical accuracy. ✦ +/- 5ms (for example) ✦ Tests are run by loading an iframe with the test 5 times. ✦ getTime() is run before/after each test. ✦ Entire suite must be trashed in order to upgrade/fix a test.
- Error Rate? ✦ How do we get it? What does it mean? ✦ It’s how confident we are that we arrived at the result we want in the number of runs that we’ve done.
- Normal Distribution ✦ First: Assume that the results are coming back in a normal distribution. ✦ The “bell curve”
- Confidence ✦ Next: We need a confidence level. ✦ T-Distribution works well here. http://en.wikipedia.org/wiki/Student%27s_t-distribution
- Error Rate ✦ 5 runs ✦ (each run is potentially 1000s of individual test runs) ✦ 95% Confidence (t-distribution = 2.776) ✦ Standard Errors Mean = ✦ (std_dev / sqrt(runs)) * 2.776 ✦ Error = (err_mean / mean) * 100 ✦ This way you can get results like: ✦ 123ms +/- 5ms
- V8 Benchmark ✦ Tests are run, potentially, 1000s of times. ✦ Also provides an error rate. ✦ (Use a geometric mean to arrive at a result.)
- Small Time Accuracy ✦ Small time: ✦ 1ms, 1ms, 1ms, 1ms, 3ms ✦ huge error! ✦ Large time: ✦ 1234ms, 1234ms, 1234ms, 1234ms, 1238ms ✦ tiny error! ✦ Tests that run faster need to be run more times. ✦ Running more times = less potential for weird results. http://ejohn.org/blog/javascript-benchmark-quality/
- Runs/Second ✦ var start = (new Date).getTime(); while (time < 1000) { runTest(); time = (new Date).getTime() - start; } ✦ More test runs, more statistical accuracy. ✦ V8 & Dromaeo-style suites handle this. ✦ (Problem: getTime() is being run on every loop - it should be run less frequently in order to influence the numbers less.)
- Runs/Second ✦ You are now measuring tests/second rather than seconds per test. ✦ You run tests as many times in one second as you can. ✦ Then you do that multiple times (5?) ✦ THEN you analyze the final numbers: ✦ 1234run/s, 1230runs/s, 1240runs/s, ...
- Harmonic Mean ✦ A way to average rates ✦ Which is what we have! runs/second ✦ For example: ✦ 1234run/s, 1230runs/s, 1240runs/s, 1236runs/ms, 1232runs/s ✦ 5 / ( (1/1234) + (1/1230) + (1/1240) + (1/1236) + (1/1232) ) = ✦ 1234.39runs/s! http://en.wikipedia.org/wiki/Harmonic_mean
- Dromaeo ✦ All individual tests are versioned ✦ Makes it easy to update or fix a bug in a test ✦ Can only run tests of specific versions against each other ✦ Uses V8’s style of running tests. ✦ Also has DOM and framework tests. ✦ ...and hooks for doing Shark profiling.
- Bug Fixes ✦ Tests will, inevitably, have bugs that need to be fixed. ✦ Fixing a bug changes the result quality. ✦ Tests need to be versioned so that changes can be made. ✦ You look at Test v1 vs. Test v1 results. ✦ Not Test v2 vs. Test v1. ✦ Tip: Just use the last revision control commit # for the test file.
- Different Code, Same Platform ✦ Most solutions here are very poor. ✦ Run the test very few times, use getTime(). ✦ Highly inaccurate results, massive error.
- Garbage Collection ✦ Browsers periodically run garbage collectors to clean up old objects no longer referenced. ✦ This can take a long time and spike your test results. ✦ Example: ✦ 10ms, 13ms, 11ms, 12ms, 486ms, 12ms, ... ✦ When comparing engine to engine, this doesn’t matter. ✦ Comparing code vs. code, it does.
- Mean, Median, Mode? ✦ Mode! ✦ Run your tests a large number of times. ✦ What is the ‘mode’ (the result that occurs most frequently) ✦ Example: ✦ 10, 11, 11, 12, 12, 12, 13, 14 ✦ Mode = 12ms. ✦ Less accurate than mean, but gives you a more-consistent result. ✦ DON’T DISCARD “BAD” RESULTS!
- getTime() Accuracy http://ejohn.org/blog/accuracy-of-javascript-time/
- 15ms intervals ONLY! Error Rate of 50-750%!
- IE in Wine ✦ Running Internet Explorer in Wine (on Linux) gives fine-grained timer results ✦ Down to the millisecond! ✦ You can also run IE, in Wine, on OS X: ✦ ies4osx ✦ Huge Caveat: It gives you fine-grained time, but that doesn’t mean it’s accurate.
- Different Code, Same Platform ✦ How can we get good numbers? ✦ We have to go straight to the source: Use the tools the browsers provide.
- Firebug Profiler
- Safari 4 Profiler
- IE 8 Profiler
- IE 6-8: DynaTrace Ajax
- IE 6-8: DynaTrace Ajax
- Shark Profiling ✦ Extremely low-level ✦ Watch for all internal function calls ✦ See what gets called the most. ✦ Dromaeo includes shark profiling hooks.
- Interesting Problems Tackled in jQuery 1.4
- Tackled in 1.4 ✦ Complexity reduction ✦ Bubbling change, submit, focus, blur ✦ Required script loading
- Questions ✦ Contact: ✦ John Resig ✦ http://ejohn.org/ ✦ http://twitter.com/jeresig
Advertisement